data-infrastructure-services issueshttps://gitlab.dkrz.de/groups/data-infrastructure-services/-/issues2022-03-02T08:58:41Zhttps://gitlab.dkrz.de/data-infrastructure-services/tzis/-/issues/9Only metadata is written, no chunks2022-03-02T08:58:41ZHauke SchulzOnly metadata is written, no chunksI did not come up with a minimal reproducible example yet, but I was hoping that someone might already have a solution to the issue that I'm encountering. I'm not sure if this is actually an issue of this package or not though.
In any c...I did not come up with a minimal reproducible example yet, but I was hoping that someone might already have a solution to the issue that I'm encountering. I'm not sure if this is actually an issue of this package or not though.
In any case, this is my code and traceback:
```python
In [4]: import xarray as xr
...: from tzis import tzis
...: ACCOUNT="mh0010"
...: USERNAME="m300408"
...: CONTAINER_NAME="EUREC4A_LES"
...: ZARR_DSET_NAME="experiment_1/ICON_DOM01_radiation.zarr/"
...:
...: token=tzis.get_token(ACCOUNT, USERNAME=USERNAME)
...: container = tzis.Tzis(token["OS_STORAGE_URL"],token["OS_AUTH_TOKEN"],CONTAINER_NAME,use_fsspec=True)
...: prefix_for_object_storage=ZARR_DSET_NAME
...:
...: container.open_store(prefix_for_object_storage)
...:
...: zf=xr.open_zarr("ICON_DOM01_radiation.zarr")
...: zf.to_zarr(container.store,mode='w')
...:
---------------------------------------------------------------------------
PathNotFoundError Traceback (most recent call last)
<ipython-input-4-4b9c5e661c66> in <module>
13
14 zf=xr.open_zarr("ICON_DOM01_radiation.zarr")
---> 15 zf.to_zarr(container.store,mode='w')
~/.local/lib/python3.8/site-packages/xarray/core/dataset.py in to_zarr(self, store, chunk_store, mode, synchronizer, group, encoding, compute, consolidated, append_dim, region, safe_chunks)
2029 encoding = {}
2030
-> 2031 return to_zarr(
2032 self,
2033 store=store,
~/.local/lib/python3.8/site-packages/xarray/backends/api.py in to_zarr(dataset, store, chunk_store, mode, synchronizer, group, encoding, compute, consolidated, append_dim, region, safe_chunks)
1412 writer = ArrayWriter()
1413 # TODO: figure out how to properly handle unlimited_dims
-> 1414 dump_to_store(dataset, zstore, writer, encoding=encoding)
1415 writes = writer.sync(compute=compute)
1416
~/.local/lib/python3.8/site-packages/xarray/backends/api.py in dump_to_store(dataset, store, writer, encoder, encoding, unlimited_dims)
1122 variables, attrs = encoder(variables, attrs)
1123
-> 1124 store.store(variables, attrs, check_encoding, writer, unlimited_dims=unlimited_dims)
1125
1126
~/.local/lib/python3.8/site-packages/xarray/backends/zarr.py in store(self, variables, attributes, check_encoding_set, writer, unlimited_dims)
557 )
558 if self._consolidate_on_close:
--> 559 zarr.consolidate_metadata(self.zarr_group.store)
560
561 def sync(self):
~/.local/lib/python3.8/site-packages/zarr/convenience.py in consolidate_metadata(store, metadata_key)
1132 }
1133 store[metadata_key] = json_dumps(out)
-> 1134 return open_consolidated(store, metadata_key=metadata_key)
1135
1136
~/.local/lib/python3.8/site-packages/zarr/convenience.py in open_consolidated(store, metadata_key, mode, **kwargs)
1190 # pass through
1191 chunk_store = kwargs.pop('chunk_store', None) or store
-> 1192 return open(store=meta_store, chunk_store=chunk_store, mode=mode, **kwargs)
~/.local/lib/python3.8/site-packages/zarr/convenience.py in open(store, mode, **kwargs)
104 return open_group(_store, mode=mode, **kwargs)
105 else:
--> 106 raise PathNotFoundError(path)
107
108
PathNotFoundError: nothing found at path ''
```
The error occurs independent of the value of `use_fsspec`. When I investigate the objects on the swiftbrowser, all metafiles (.zattrs, .zarray, .zmetadata) are written, but no single chunk.
Cheers,
Haukehttps://gitlab.dkrz.de/data-infrastructure-services/tzis/-/issues/6rename it tzis2021-12-15T09:58:07ZFabian Wachsmannrename it tzishttps://gitlab.dkrz.de/data-infrastructure-services/tzis/-/issues/5Workflow2022-01-25T15:24:58ZMarco KulükeWorkflowPicture for AGUPicture for AGUMarco KulükeMarco Kulükehttps://gitlab.dkrz.de/data-infrastructure-services/tzis/-/issues/4Create Notebook for access and usage2021-12-15T09:58:20ZMarco KulükeCreate Notebook for access and usage... for Zarrdatasets which are not public... for Zarrdatasets which are not publicMarco KulükeMarco Kulükehttps://gitlab.dkrz.de/data-infrastructure-services/tzis/-/issues/3integration tests2021-09-29T16:16:34ZFabian Wachsmannintegration testsdoes tzis work for
- cmip6
- cordex
- era5
? select 3 files for each, put it in git lfs, setup a cidoes tzis work for
- cmip6
- cordex
- era5
? select 3 files for each, put it in git lfs, setup a cihttps://gitlab.dkrz.de/data-infrastructure-services/tzis/-/issues/2tracking_ids2021-09-20T10:26:06ZFabian Wachsmanntracking_idsCheck for global attributes like tracking_ids that need to be collected in the consolidated metadata.Check for global attributes like tracking_ids that need to be collected in the consolidated metadata.https://gitlab.dkrz.de/data-infrastructure-services/intake-esm/-/issues/11empty 0-size files: nbp_Lmon_EC-Earth3-CC_piControl_r1i1p1f12021-04-26T12:19:02ZAaron Springaaron.spring@mpimet.mpg.deempty 0-size files: nbp_Lmon_EC-Earth3-CC_piControl_r1i1p1f1I found files of size 0, which cannot be opened with CDO or xarray. Should I report such files also somewhere else? @k204210
Can we add a check avoid such files when creating in the catalogs?
```
ls -all /mnt/lustre02/work/ik1017/CMIP...I found files of size 0, which cannot be opened with CDO or xarray. Should I report such files also somewhere else? @k204210
Can we add a check avoid such files when creating in the catalogs?
```
ls -all /mnt/lustre02/work/ik1017/CMIP6/data/CMIP6/CMIP/EC-Earth-Consortium/EC-Earth3-CC/piControl/r1i1p1f1/Lmon/nbp/gr/v20210114
total 8
drwxr-sr-x 2 k204210 esgf 4096 Mar 16 16:20 ./
drwxr-sr-x 3 k204210 esgf 4096 Mar 16 16:20 ../
-r--r--r-- 1 k204210 esgf 0 Feb 15 19:36 nbp_Lmon_EC-Earth3-CC_piControl_r1i1p1f1_gr_191401-191412.nc
-r--r--r-- 1 k204210 esgf 0 Feb 15 19:36 nbp_Lmon_EC-Earth3-CC_piControl_r1i1p1f1_gr_199301-199312.nc
-r--r--r-- 1 k204210 esgf 0 Feb 15 19:36 nbp_Lmon_EC-Earth3-CC_piControl_r1i1p1f1_gr_200201-200212.nc
-r--r--r-- 1 k204210 esgf 0 Feb 15 19:36 nbp_Lmon_EC-Earth3-CC_piControl_r1i1p1f1_gr_201801-201812.nc
-r--r--r-- 1 k204210 esgf 0 Feb 15 19:36 nbp_Lmon_EC-Earth3-CC_piControl_r1i1p1f1_gr_202501-202512.nc
-r--r--r-- 1 k204210 esgf 0 Feb 15 19:36 nbp_Lmon_EC-Earth3-CC_piControl_r1i1p1f1_gr_205001-205012.nc
-r--r--r-- 1 k204210 esgf 0 Feb 15 19:36 nbp_Lmon_EC-Earth3-CC_piControl_r1i1p1f1_gr_208401-208412.nc
-r--r--r-- 1 k204210 esgf 0 Feb 15 19:36 nbp_Lmon_EC-Earth3-CC_piControl_r1i1p1f1_gr_216701-216712.nc
-r--r--r-- 1 k204210 esgf 0 Feb 15 19:36 nbp_Lmon_EC-Earth3-CC_piControl_r1i1p1f1_gr_218801-218812.nc
-r--r--r-- 1 k204210 esgf 0 Feb 15 19:36 nbp_Lmon_EC-Earth3-CC_piControl_r1i1p1f1_gr_221701-221712.nc
-r--r--r-- 1 k204210 esgf 0 Feb 15 19:36 nbp_Lmon_EC-Earth3-CC_piControl_r1i1p1f1_gr_224201-224212.nc
-r--r--r-- 1 k204210 esgf 0 Feb 15 19:36 nbp_Lmon_EC-Earth3-CC_piControl_r1i1p1f1_gr_226201-226212.nc
-r--r--r-- 1 k204210 esgf 0 Feb 15 19:36 nbp_Lmon_EC-Earth3-CC_piControl_r1i1p1f1_gr_226301-226312.nc
-r--r--r-- 1 k204210 esgf 0 Feb 15 19:36 nbp_Lmon_EC-Earth3-CC_piControl_r1i1p1f1_gr_229401-229412.nc
-r--r--r-- 1 k204210 esgf 0 Feb 15 19:36 nbp_Lmon_EC-Earth3-CC_piControl_r1i1p1f1_gr_233201-233212.nc
-r--r--r-- 1 k204210 esgf 0 Feb 15 19:36 nbp_Lmon_EC-Earth3-CC_piControl_r1i1p1f1_gr_234101-234112.nc
-r--r--r-- 1 k204210 esgf 0 Feb 15 19:36 nbp_Lmon_EC-Earth3-CC_piControl_r1i1p1f1_gr_234301-234312.nc
```
also opendap does seem to work:
```python
xarray.open_dataset('http://esgf3.dkrz.de/thredds/dodsC/cmip6/CMIP/EC-Earth-Consortium/EC-Earth3-CC/piControl/r1i1p1f1/Lmon/nbp/gr/v20210114/nbp_Lmon_EC-Earth3-CC_piControl_r1i1p1f1_gr_191401-191412.nc')
...
file, cached = self._acquire_with_cache_info(needs_lock)
File "/work/mh0727/m300524/conda-envs/s2s-ai/lib/python3.7/site-packages/xarray/backends/file_manager.py", line 205, in _acquire_with_cache_info
file = self._opener(*self._args, **kwargs)
File "netCDF4/_netCDF4.pyx", line 2291, in netCDF4._netCDF4.Dataset.__init__
File "netCDF4/_netCDF4.pyx", line 1855, in netCDF4._netCDF4._ensure_nc_success
OSError: [Errno -70] NetCDF: DAP server error: b'http://esgf3.dkrz.de/thredds/dodsC/cmip6/CMIP/EC-Earth-Consortium/EC-Earth3-CC/piControl/r1i1p1f1/Lmon/nbp/gr/v20210114/nbp_Lmon_EC-Earth3-CC_piControl_r1i1p1f1_gr_191401-191412.nc'https://gitlab.dkrz.de/data-infrastructure-services/tutorials-and-use-cases/-/issues/37Update tauc Gitlab repo with IS-ENES3 Github repo content2021-04-01T09:22:54ZGhost UserUpdate tauc Gitlab repo with IS-ENES3 Github repo contentnew folder hierarchy, spec, make_kernel.sh,...new folder hierarchy, spec, make_kernel.sh,...https://gitlab.dkrz.de/data-infrastructure-services/tutorials-and-use-cases/-/issues/36Model skills notebook with ERA52021-03-29T07:54:29ZGhost UserModel skills notebook with ERA5It was a nice idea, but I will not champion it.
Research question (thanks Iuliia Polkova!) **"Evaluation of prediction skill for surface air temperature and precipitation from the CMIP6 models using ERA5"**
**STEP 1**. DONE find ERA in...It was a nice idea, but I will not champion it.
Research question (thanks Iuliia Polkova!) **"Evaluation of prediction skill for surface air temperature and precipitation from the CMIP6 models using ERA5"**
**STEP 1**. DONE find ERA in dkrz data pool --> /pool/data/ERA5/ (thanks Fabi! also for the docs: https://confluence.ecmwf.int/display/CKB/ERA5%3A+data+documentation#ERA5:datadocumentation-Introduction)
**STEP 2**. DONE how to load ERA5 --> no problem with GRIB: http://xarray.pydata.org/en/stable/examples/ERA5-GRIB-example.html (thanks Marco!)
**STEP 3**. DONE load model data --Y intake and xarray, just copy-paste from the summer days nb's
**STEP 4**. TODO how to compare ERA-model: Iuliia's answer:
"load the data, calculate ensemble mean, calculate anomalies with respect to the long-term mean, calculate RMSE or CORR for different lead times et voila. If you have problems just let me know, I will help. The skill is usually calculated for anomalies and for lead times, so for prediction year 1, and multi-year averages such as years 2-5. Depending on the research problem, we calculate skill for each grid-point or for the anomalies averaged over a certain region. Here is one of my papers with the examples of skill calculation https://doi.org/10.1029/2018MS001439"https://gitlab.dkrz.de/data-infrastructure-services/tutorials-and-use-cases/-/issues/35Store Data Selection by Intake as csv/json - Notebook Example2021-02-19T12:35:47ZMarco KulükeStore Data Selection by Intake as csv/json - Notebook ExampleMarco KulükeMarco Kulükehttps://gitlab.dkrz.de/data-infrastructure-services/tutorials-and-use-cases/-/issues/34Create Lighweight Notebook which explains Data Pool/Intake for Webinar2021-03-11T14:05:19ZMarco KulükeCreate Lighweight Notebook which explains Data Pool/Intake for WebinarMarco KulükeMarco Kulükehttps://gitlab.dkrz.de/data-infrastructure-services/tutorials-and-use-cases/-/issues/33Update the cmip6 multimodel comparison nb to include intake2021-03-11T11:23:17ZGhost UserUpdate the cmip6 multimodel comparison nb to include intakeThe current cmip6 multimodel comparison nb points to the path where the datasets (directly, without intake). As the notebook calculations uses pre-processed data (the global means were calculated apart), the update also must rewrite the ...The current cmip6 multimodel comparison nb points to the path where the datasets (directly, without intake). As the notebook calculations uses pre-processed data (the global means were calculated apart), the update also must rewrite the calculations to include not only the annual means calculations but also the global means calculations in the notebook, so the intake call points to the "raw" cmip6 data.2021-02-08https://gitlab.dkrz.de/data-infrastructure-services/tutorials-and-use-cases/-/issues/31Set-up Gitlab-pages for tauc repo2021-07-14T13:55:37ZGhost UserSet-up Gitlab-pages for tauc repoLets try a config where all notebooks are executed and hosted. That has the advantage of having tests run at the same time.
Original comment:
> We already have gitlab pages build with Sphinx, which makes documentation-like pages (boring...Lets try a config where all notebooks are executed and hosted. That has the advantage of having tests run at the same time.
Original comment:
> We already have gitlab pages build with Sphinx, which makes documentation-like pages (boring :worried:) so we will set new gitlab pages with Jekyll for a more blog-like style.Fabian WachsmannFabian Wachsmannhttps://gitlab.dkrz.de/data-infrastructure-services/tutorials-and-use-cases/-/issues/30Re-Write Notebooks Climate-data-analysis-service from PyOphidia to xarray2021-02-19T12:35:02ZMarco KulükeRe-Write Notebooks Climate-data-analysis-service from PyOphidia to xarrayOld PyOphidia Notebooks https://github.com/ECAS-Lab/ecas-notebooks
New xarray Notebooks https://github.com/IS-ENES-Data/Climate-data-analysis-serviceOld PyOphidia Notebooks https://github.com/ECAS-Lab/ecas-notebooks
New xarray Notebooks https://github.com/IS-ENES-Data/Climate-data-analysis-serviceMarco KulükeMarco Kulükehttps://gitlab.dkrz.de/data-infrastructure-services/tutorials-and-use-cases/-/issues/29Upload Summer Day Notebook to iRODS2021-03-11T14:05:27ZMarco KulükeUpload Summer Day Notebook to iRODSGather more information on iRODSGather more information on iRODSMarco KulükeMarco Kulükehttps://gitlab.dkrz.de/data-infrastructure-services/tutorials-and-use-cases/-/issues/28Start Gitlab-pages for tauc repo - sphinx2021-01-25T09:32:17ZGhost UserStart Gitlab-pages for tauc repo - sphinxCreate a blog for tauc, probably using Sphinx, coordinated with the development of gitlab-pages for the Zarr repoCreate a blog for tauc, probably using Sphinx, coordinated with the development of gitlab-pages for the Zarr repo2021-01-22https://gitlab.dkrz.de/data-infrastructure-services/tutorials-and-use-cases/-/issues/27Add the cmip6 multimodel comparison notebook (is-enes3 demo) to the tauc repo2020-12-16T07:46:34ZGhost UserAdd the cmip6 multimodel comparison notebook (is-enes3 demo) to the tauc repoAdd to the taur repo the notebook used in the is-enes3 demo: https://portal.enes.org/data/data-metadata-service/analysis-platforms/example-of-how-to-run-server-side-data-near-multimodel-comparisons. From now on it will be updated here. F...Add to the taur repo the notebook used in the is-enes3 demo: https://portal.enes.org/data/data-metadata-service/analysis-platforms/example-of-how-to-run-server-side-data-near-multimodel-comparisons. From now on it will be updated here. First step: include intake. Second step: pre-process here (not in bash). Updates assigned to Maria but feel free to contribute :smile:2020-12-16https://gitlab.dkrz.de/data-infrastructure-services/tutorials-and-use-cases/-/issues/26Calculate Climate Extremes Indices according to the ETCCDI standard with CDOs2021-11-11T12:44:53ZFabian WachsmannCalculate Climate Extremes Indices according to the ETCCDI standard with CDOsFabian WachsmannFabian Wachsmannhttps://gitlab.dkrz.de/data-infrastructure-services/tutorials-and-use-cases/-/issues/25Analyse time series of a CMIP simulation with Xarray and Pandas2021-07-14T13:28:46ZFabian WachsmannAnalyse time series of a CMIP simulation with Xarray and Pandashttps://gitlab.dkrz.de/data-infrastructure-services/tutorials-and-use-cases/-/issues/24How to access data with intake catalogs2021-04-12T08:03:52ZFabian WachsmannHow to access data with intake catalogs