to zarr in storages
This small package enables writing cloud-optimized zarr-formatted data to e.g. DKRZ's institutional cloud swift
Installation
pip install git+https://gitlab.dkrz.de/data-infrastructure-services/tzis.git#egg=tzisUsage Example
from tzis import swifthandling, openmf, tzis,catalog
import glob, os
import xarray as xr
#set credentials, tzis will ask for a password
project = "bm1344"
user = "k204210"
#the final location in swift is ACCOUNT/CONTAINER/PREFIX/
container_name="testtzis"
prefix_for_object_storage="testcmip6"
#note you have to `glob` yourself
glob_path_var = "/work/ik1017/CMIP6/data/CMIP6/ScenarioMIP/DKRZ/MPI-ESM1-2-HR/ssp370/r10i1p1f1/Amon/tas/gn/v20190710/*"
varname = "tas"
chunkdim = "time"
target_mb = 16
#
token=swifthandling.get_token(
"dkrz",
project,
user
)
target_fsmap=swifthandling.get_swift_mapper(
token["OS_STORAGE_URL"],
token["OS_AUTH_TOKEN"],
container_name,
os_name=prefix_for_object_storage
)
omo = openmf.open_mfdataset_optimize(
glob_path_var,
varname,
target_fsmap,
chunkdim=chunkdim,
target_mb=target_mb
)
outstore=tzis.write_zarr(
omo.target_fsmap,
omo.mf_dset,
omo.varname,
verbose=True,
target_mb=0
)
xr.open_zarr(
outstore,
consolidated=True
)
catalog.write_catalog(
omo.target_fsmap.fs,
os.path.dirname(omo.target_fsmap.root),
prefix_for_object_storage
)
Tutorial Notebooks
Service
On a Openstack VM, we run a jupyter server with the extension appmode. This allows to provide notebooks as interactive webapps.
We developed such a showcase service for tzis where the source data comes from a lustre-directory. When you are in the DKRZ internal network (via vpn), you can reach: http://jupyter-appmode.cloud.dkrz.de:8888/apps/tzis-service.ipynb
You have to put in a password which you can get by request from the DM team. You will get the output of the notebook provided in notebooks/tzis-service.ipynb.