User guide

Using Jupyter Notebooks for Model Data Analysis

Welcome to the DKRZ tutorials and use cases repository!

This repository collects and prepares Jupyter notebooks with coding examples on how to use state-of-the-art processing tools on big data collections. The Jupyter notebooks highlight the optimal usage of High-Performance Computing resources and adress data analysists and researchers which begin to work with resources of German Climate Computing Center DKRZ.

While jupyter notebooks with demonstrations are provided in the notebooks/demo directory, we also host notebooks for hands-on sessions in the notebooks/hands-on_* directories.

Getting a DKRZ account:

• for model data users working in EU:
  • for short and quick access, register at any time to the Climate Analytics Service, or
  • for longer access, check the calls for applications to the Analysis Platforms services, or
• for model data users with partners in the German earth systems research community, see here.

Visit our blog, we have animations on how to log in to the DKRZ resources and further explanations on the content of this repository.

Quick start

To run the notebooks, you only need a browser (like Firefox, Chrome, Safari,...) and internet connection.

1. Open the DKRZ Jupyterhub in your browser.

2. Login with your DKRZ account (if you do not have one account yet, see the links above).

3. Pick a profile (Preset -> Start from Preset Profile). You need a prepost node (they have internet access, more info here). Choose profile 5GB memory, prepost.

   NOTE: Everytime you run the notebook you will use some of that RAM, we recomend to click on Kernel -> Shutdown kernel often so the memory is released. If you want to run several notebooks at the same time or one notebook several times and you cannot shoutdown the kernel each time, please, choose a job profile with a larger memory.

4. Press "start" and your Jupyter server will start (which it is also known as spawning).

5. Open a terminal in Jupyter (New -> Terminal, on the right side)

6. A terminal window opens on the node where your Jupyter is running.

7. Clone the notebooks from the DKRZ GitLab:

$ git clone https://gitlab.dkrz.de/data-infrastructure-services/tutorials-and-use-cases.git

8. Go back to your Jupyter and open a notebook from the notebooks folder:

tutorials-and-use-cases/notebooks/

9. Make sure you use the Jupyter Python 3 unstable kernel (Kernel -> Change Kernel).

Advanced

Some notebooks need individual Jupyter kernel:

1. Open the terminal and run a script to make a new kernel:

$ bash make_kernel.sh

2. ... it takes a couple of minutes ...
3. When done then go to you Jupyter and choose the new Kernel we just created Notebook Demo.
4. Now you can run also the summer days notebook.

Content and structure

Tutorials

• notebooks/demo/tutorial_*

  1. We prepared a tutorial on how to use Intake in the DKRZ data pool.

  2. ESMVal-Tool

Use-cases

• notebooks/demo/use-case_*

Further Infos

• Find more in the DKRZ Jupyterhub documentation.
• prepost nodes at DKRZ have internet access info.
• Python 3 unstable kernel: This kernel already contains all the common geoscience packages that we need for our notebooks.
• See in this video the main features of the DKRZ Jupterhub and how to use it.
• Advanced users developing their own notebooks can find there how to create their own environments that are visible as kernels by the Jupyterhub.

Besides the information on the Jupyterhub, in these DKRZ docs you can find how to run Jupyter notebooks directly in the DKRZ server, that is, out of the Jupyterhub (it entails that you install the geoscience packages you need).

Exercises

In this hands-on we will find, analyze, and visualize data from our DKRZ data pool. The goal is to create two maps, one showing the number of tropical nights for 2014 (the most recent year of the historical dataset) and another one showing a chosen year in the past. The hands-on will be split into two exercises:

1_hands-on_find_data_intake.ipynb

• Search for an appropriate list of data files. The datasets should contain the variables tasmin on a daily basis.
• Save your selection as .csv file, so it can be used by another notebook.

2_hands-on_tropical_nights_intake_xarray_cmip6.ipynb

• Read the saved selection and open the two files, which are needed.
• Calculate the number of tropical nights for both years.
• Visualize the results on a map. You can use your preferred visualization package or stick to the example in the demo use-case_frost_days_intake_xarray_cmip6.ipynb.

Contact us

Reach us at data-pool@dkrz.de