You are here: Home / Systems / jupyterhub.dkrz.de

jupyterhub.dkrz.de

Jupyter offers the possibility to interactively work with Python in a web browser. Jupyterhub enables DKRZ users to run the workload directly on the Mistral HPC system - with access to all their data in the lustre filesystem.
DKRZ operates the jupyterhub server in extended testing phase, which means that we do not guarantee fulltime availabilty of the system and some more limitations documented here.

What are Jupyter Notebooks?

So called "notebooks" are documents that can contain Python (or other) code and further text elements like comments, figures, equations, etc. They are human-readable and executable by Jupyter in order to produce the desired results.

What is Jupyter?

Jupyter (formerly known as IPython) itself is a web application that allows editing and running notebooks via a web browser. Each notebook uses its own "kernel" that determines the used Python (or other language) version and associated modules.

What is JupyterHub?

JupyterHub is a multi-user server for Jupyter notebooks that allows to execute the notebook directly on the DKRZ HPC system Mistral. It therefore also supports the execution of parallel computation.

JupyterHub is available at https://jupyterhub.dkrz.de for all DKRZ users who have access to Mistral and who are allowed to submit batch jobs.

Using jupyterhub.dkrz.de

Navigate to the https://jupyterhub.dkrz.de URL using a web browser and log in with your DKRZ account.

After login to the jupyter web interface, press the 'Start My Server' button which opens the options menu for the notebook server to be spawned.

Several job profiles are predefined, chose one and specify the project account such that used CPUtime gets accounted. For testing purposes or small workload of the notebook to be executed, 1 to 4 cores in the shared partition might be enough. For larger workload (especially if a big amount of memory is needed) a full node of the compute partition should be used. The "Reservation" option just needs to be filled, if you wanna use a specific node reservation within SLURM, e.g. for workshops.

By clicking on 'Spawn' a new notebook server is submitted as a batch job on the HPC system Mistral. You will be redirected automatically once the SLURM allocation is ready to start the server.

If the batch queue is filled with waiting jobs, it might take some time to start the notebook server. There is an internal timelimit specified until which the resources should be ready. If not the spawner will reject and ask you to start again (maybe with reduced resources specified).

Once your session has started you should see the Jupyter file browser interface.

Use the New menu within the Files tab to start a new Notebook or Terminal. It will be opened in a new browser tab.

Managing Jupyter sessions

To see what Jupyter Notebooks and Terminals your Jupyter server is running, use the home browser tab and click on the Running tab. Here you can shutdown resources if needed.

To stop the entire Jupyter Notebook server itself, use the 'Control Panel' in the upper right part of the dashboard and click on 'Stop My Server'.

There are a few points you should be aware of when working with Jupyter Notebooks on the jupyterhub server
  • Sessions are persistent. If you close the browser tab without stopping the server before, you can revisit the jupyterhub site and carry on working
  • SLURM might cancel your server session if a timeout occurs after 8 hours
  • stdout and stderr of the SLURM batch job that is spawner by jupyterhub is written to your HOME directory on the HPC system in file "jupyterhub_slurmspawner_<id>.log"

More documentation needed?

For all details refer to the official jupyter documentation.

A collection of example notebooks on how to use this tool for postprocessing climate data is available at the DKRZ Gitlab server: jupyterhub_notebooks

Document Actions