Skip to content
Snippets Groups Projects

to zarr in storages

This small package enables writing cloud-optimized zarr-formatted data to e.g. DKRZ's institutional cloud swift

Installation

pip install git+https://gitlab.dkrz.de/data-infrastructure-services/tzis.git#egg=tzis

Usage Example

from tzis import tzis
import glob

#set credentials, tzis will ask for a password
project = "ik1017"
user = "k204210"

#the final location in swift is ACCOUNT/CONTAINER/PREFIX/
container_name="testtzis"
prefix_for_object_storage="testcmip6"

#note you have to `glob` yourself
path_var = "/work/ik1017/CMIP6/data/CMIP6/ScenarioMIP/DKRZ/MPI-ESM1-2-HR/ssp370/r10i1p1f1/Amon/tas/gn/v20190710/*"
varname = "tas"

chunkdim = "time"
target_mb = 10

#

#token=tzis.create_token("dkrz",project,user)
token=tzis.get_token("dkrz",project,user)

container = tzis.Tzis(token["OS_STORAGE_URL"],
                      token["OS_AUTH_TOKEN"],
                      container_name,
                      os_name=prefix_for_object_storage,
                      mf_dset=glob.glob(path_var),
                      varname=varname,
                      verbose=True,
                      open_kwargs=dict(
                          chunkdim=chunkdim,
                          target_mb=target_mb
                      )
)

container.write_zarr(chunkdim=chunkdim,
                       startchunk=0,
                       validity_check=False,
                       maxretries=3)
                      
container.write_catalog(catalogname="catalog.json",
                      pattern=None,
                      delim=".",
                      columns=None,
                      mode="a"
                     )

Tutorial Notebooks

  • You can find a detailed introduction here.
  • We provide a use case for CMIP6 data here

Service

On a Openstack VM, we run a jupyter server with the extension appmode. This allows to provide notebooks as interactive webapps.

We developed such a showcase service for tzis where the source data comes from a lustre-directory. When you are in the DKRZ internal network (via vpn), you can reach: http://jupyter-appmode.cloud.dkrz.de:8888/apps/tzis-service.ipynb

You have to put in a password which you can get by request from the DM team. You will get the output of the notebook provided in notebooks/tzis-service.ipynb.