You are here: Home / Systems / Mistral / Programming / Working with Jupyter Notebook

Working with Jupyter Notebook

Jupyter Notebook is a powerful web-application which allows you to work with live python code and other content. Running a Jupyter Notebook on mistral is even more useful as it gives you access to all data on the file system and the parallel processing power of a super computer.

There are several options how to use Jupyter Notebook on mistral. You will have to decide which way fits best your requirements regarding supported clients and flexibility.

JupyterHub

JupyterHub gives you access to your notebooks by simply pointing your web browser to jupyterhub.dkrz.de. You can use your mobile device or the workstation in your office. Windows or MacOS or Linux all just work.

You have to chose from a predefined set of jobs which run your Notebook. You also have to stick to the Python installations we provide but we are working on making our JupyterHub installation even more flexible. For more information see our JupyterHub documentation.

Direct Jupyter Notebook

if you run Jupyter Notebook on your local machine it will start a web browser and you can immediately start working. Doing the same thing on mistral is not so straightforward because your local browser cannot directly connect to Jupyter Notebook on mistral. Furthermore, you may want to run Jupyter Notebook on dedicated resources inside a job.

To relieve you from having to set up ssh tunnels and write job scripts for Jupyter Notebook, we packaged everything in a simple bash script which you have to run on your local computer. It should work in most Linux environments and on macOS.

Download the script here

And add execute mode.

chmod a+x start-jupyter

Using jupyter on mistralpp

The most simple way to run it is without any options

./start-jupyter

This assumes that your username on mistral is the same as your local username and that jupyter can be started on mistral without any further configuration. I.e. you set up everything in ~/.profile or ~/.bashrc.

The script then starts jupyter on a mistralpp node and connects your local browser. Since it's running on an interactive node, no compute time is charged to your project's account but you have to share the node with many other users.

Suppose your username on mistral is different and you need to load a module before starting jupyter you wold do the following. Create a file to load the module. You can name it as you want and store it anywhere on mistral.

mlogin100:~$ cat jupyter_preload
module unload anaconda2 anaconda3 netcdf_c
module load anaconda3/bleeding_edge

You then reference the file with the -i option. Any relative path starts in your home directory. Absolute paths are also possible.

./start-jupyter -u u202020 -i jupyter_preload

To use your own miniconda installation you could put something like this into your include file.

export PATH=/work/bb1999/u202020/miniconda3/bin:$PATH
source activate my_env

Jupyter can't just run Notebook, it can also run Lab if it's installed in your environment. Use the option -c lab.

Running jupyter in a job

If you need dedicated resources like many compute cores and large amounts of memory it is better to run jupyter in a job. You don't have to write the job script yourself because start-jupyter will generate one on the fly.

./start-jupyter -u u202020 -i jupyter_preload -A bb1999

This will run jupyter on the shared partition with one task. You can request more tasks with the -n option or even use a different partition with the -p option.

Troubleshooting

Hanging script

The script may at times seem to hang because it is waiting for a job to start or for a python environment to activate. This will resolve itself after a while. However, it also may be hanging because jupyter couldn't be found with your configuration. This is usually not checked for performance reasons. If you are in doubt whether your set up is correct, then start the script with the -d option.

./start-jupyter -i some_setup -d
Looking for Jupyter.
which: no jupyter in (/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin)

Another option is to look into jupyter's output. This is redirected into a file ~/.jupyter/jupyter.XXXXX on mistral with random characters instead of Xs.

Orphaned jobs

When you terminate the script with Ctrl-C, it should kill all spawned processes and slurm jobs. If the script didn't get the chance to do that because the ssh connection was broken or it got killed -9, you may have to cancel the job by yourself. Log into mistral and run

squeue -u u202020

With your own username. If you see any jobs named Jupyter, you can cancel them with scancel.

Security considerations

Anyone who has access to your Jupyter Notebook server has full control over your user account. It is therefore important to secure your server. Any version of Jupyter starting with 4.3 should be secure by default. For more information on this topic visit the documentation.

Security in the Jupyter notebook server

Document Actions