README.md 2.29 KB
Newer Older
Fabian Wachsmann's avatar
Fabian Wachsmann committed
1
# to zarr in swift 
Marco Kulüke's avatar
Marco Kulüke committed
2

Fabian Wachsmann's avatar
Fabian Wachsmann committed
3
This small package enables writing cloud-optimized [zarr-formatted](https://zarr.readthedocs.io/en/stable/index.html) data to DKRZ's institutional cloud [swift](https://www.dkrz.de/up/systems/swift)
Fabian Wachsmann's avatar
Fabian Wachsmann committed
4

Marco Kulüke's avatar
Marco Kulüke committed
5
## Installation
Marco Kulüke's avatar
Marco Kulüke committed
6
```
Fabian Wachsmann's avatar
Fabian Wachsmann committed
7
pip install git+https://gitlab.dkrz.de/data-infrastructure-services/tzis.git#egg=tzis
Marco Kulüke's avatar
Marco Kulüke committed
8
```
Marco Kulüke's avatar
Marco Kulüke committed
9

Marco Kulüke's avatar
Marco Kulüke committed
10
11
12
13
14
## Usage Example

```
import tzis

Fabian Wachsmann's avatar
Fabian Wachsmann committed
15
16
token=tzis.get_token(ACCOUNT, USERNAME=USERNAME)

Fabian Wachsmann's avatar
Fabian Wachsmann committed
17
path_var = PATH_TO_VAR
Marco Kulüke's avatar
Marco Kulüke committed
18
19
mfs_towrite=[path_var +filename for filename in os.listdir(path_var)]

Fabian Wachsmann's avatar
Fabian Wachsmann committed
20
21
varname = pr

Fabian Wachsmann's avatar
Fabian Wachsmann committed
22
23
container = tzis.Tzis(token["OS_STORAGE_URL"],
                      token["OS_AUTH_TOKEN"],
Fabian Wachsmann's avatar
Fabian Wachsmann committed
24
25
26
27
28
29
                      container_name,
                      mf_dset=mfs_towrite,
                      varname=varname,
                      verbose=True             
)

Fabian Wachsmann's avatar
Fabian Wachsmann committed
30
prefix_for_object_storage=ZARR-DSET-NAME
Fabian Wachsmann's avatar
Fabian Wachsmann committed
31
container.open_store(prefix_for_object_storage)
Fabian Wachsmann's avatar
Fabian Wachsmann committed
32
33
34
35
36
37

container.write_to_swift(chunkdim="time",
                       target_mb=1000,
                       startchunk=0,
                       validity_check=False,
                       maxretries=3)
38
39
40
41
42
43
44
45
46
                      
print(container.store.url)
                       
container.write_catalog(catalogname="catalog.json",
                      pattern=None,
                      delim=".",
                      columns=None,
                      mode="a"
                     )
Marco Kulüke's avatar
Marco Kulüke committed
47
48

```
Marco Kulüke's avatar
Marco Kulüke committed
49

Marco Kulüke's avatar
Marco Kulüke committed
50
## Tutorial Notebooks
Marco Kulüke's avatar
Marco Kulüke committed
51

Fabian Wachsmann's avatar
Fabian Wachsmann committed
52
- You can find a detailed introduction [here](https://gitlab.dkrz.de/data-infrastructure-services/tzis/-/blob/master/notebooks/tzis-tutorial.ipynb).
Fabian Wachsmann's avatar
Fabian Wachsmann committed
53
- We provide a use case for CMIP6 data [here](https://gitlab.dkrz.de/data-infrastructure-services/tzis/-/blob/master/notebooks/use-case_climate-data-to-swift_tzis_cmip6.ipynb)
Marco Kulüke's avatar
Marco Kulüke committed
54

Fabian Wachsmann's avatar
Fabian Wachsmann committed
55
56
57
58
## Service

On a Openstack VM, we run a jupyter server with the extension [appmode](https://jupyter-tutorial.readthedocs.io/de/latest/web/dashboards/appmode/index.html). This allows to provide notebooks as interactive webapps.

Fabian Wachsmann's avatar
Fabian Wachsmann committed
59
We developed such a showcase service for **tzis** where the source data comes from a lustre-directory. When you are in the DKRZ internal network (via vpn), you can reach:
Fabian Wachsmann's avatar
Fabian Wachsmann committed
60
61
62
63
http://jupyter-appmode.cloud.dkrz.de:8888/apps/tzis-service.ipynb

You have to put in a password which you can get by request from the DM team.
You will get the output of the notebook provided in notebooks/tzis-service.ipynb.
Marco Kulüke's avatar
Marco Kulüke committed
64