to zarr in storages
This small package enables writing cloud-optimized zarr-formatted data to e.g. DKRZ's institutional cloud swift
Installation
pip install git+https://gitlab.dkrz.de/data-infrastructure-services/tzis.git#egg=tzisUsage Example
from tzis import tzis
import glob
#set credentials, tzis will ask for a password
project = "ik1017"
user = "k204210"
#the final location in swift is ACCOUNT/CONTAINER/PREFIX/
container_name="testtzis"
prefix_for_object_storage="testcmip6"
#note you have to `glob` yourself
path_var = "/work/ik1017/CMIP6/data/CMIP6/ScenarioMIP/DKRZ/MPI-ESM1-2-HR/ssp370/r10i1p1f1/Amon/tas/gn/v20190710/*"
varname = "tas"
chunkdim = "time"
target_mb = 10
#
#token=tzis.create_token("dkrz",project,user)
token=tzis.get_token("dkrz",project,user)
container = tzis.Tzis(token["OS_STORAGE_URL"],
token["OS_AUTH_TOKEN"],
container_name,
os_name=prefix_for_object_storage,
mf_dset=glob.glob(path_var),
varname=varname,
verbose=True,
open_kwargs=dict(
chunkdim=chunkdim,
target_mb=target_mb
)
)
container.write_zarr(chunkdim=chunkdim,
startchunk=0,
validity_check=False,
maxretries=3)
container.write_catalog(catalogname="catalog.json",
pattern=None,
delim=".",
columns=None,
mode="a"
)
Tutorial Notebooks
Service
On a Openstack VM, we run a jupyter server with the extension appmode. This allows to provide notebooks as interactive webapps.
We developed such a showcase service for tzis where the source data comes from a lustre-directory. When you are in the DKRZ internal network (via vpn), you can reach: http://jupyter-appmode.cloud.dkrz.de:8888/apps/tzis-service.ipynb
You have to put in a password which you can get by request from the DM team. You will get the output of the notebook provided in notebooks/tzis-service.ipynb.