data-infrastructure-services issueshttps://gitlab.dkrz.de/groups/data-infrastructure-services/-/issues2023-06-14T11:53:52Zhttps://gitlab.dkrz.de/data-infrastructure-services/tutorials-and-use-cases/-/issues/19Zarr Notebook Draft2023-06-14T11:53:52ZMarco KulükeZarr Notebook DraftHow to use Zarr in your Data Processing WorkflowMarco KulükeMarco Kulükehttps://gitlab.dkrz.de/data-infrastructure-services/intake-esm/-/issues/4/work/ik1017/Catalogs structure2020-04-02T09:47:46ZAaron Springaaron.spring@mpimet.mpg.de/work/ik1017/Catalogs structureich merke grad dass ich da schreibrechte habe...
I think this folder should contain:
- builder folder? or do you want to hide this
- cataloging files for mistral: json and csv, the most current should always have the same filename for c...ich merke grad dass ich da schreibrechte habe...
I think this folder should contain:
- builder folder? or do you want to hide this
- cataloging files for mistral: json and csv, the most current should always have the same filename for compatibility
- pangeo-cmip.json is not too useful, rather a link to the pangeo catalogs in general for reference (maybe in small demo?)
- short demo of intake-esm in use
- environment to run intake-esm, we could even just take the default anaconda bleeding_edge, they have intake-esm installed from december 2019
- backup folder for older catalogs, to not overwrite older ones. rename and move.https://gitlab.dkrz.de/data-infrastructure-services/tzis/-/issues/5Workflow2022-01-25T15:24:58ZMarco KulükeWorkflowPicture for AGUPicture for AGUMarco KulükeMarco Kulükehttps://gitlab.dkrz.de/data-infrastructure-services/tutorials-and-use-cases/-/issues/29Upload Summer Day Notebook to iRODS2021-03-11T14:05:27ZMarco KulükeUpload Summer Day Notebook to iRODSGather more information on iRODSGather more information on iRODSMarco KulükeMarco Kulükehttps://gitlab.dkrz.de/data-infrastructure-services/tutorials-and-use-cases/-/issues/33Update the cmip6 multimodel comparison nb to include intake2021-03-11T11:23:17ZGhost UserUpdate the cmip6 multimodel comparison nb to include intakeThe current cmip6 multimodel comparison nb points to the path where the datasets (directly, without intake). As the notebook calculations uses pre-processed data (the global means were calculated apart), the update also must rewrite the ...The current cmip6 multimodel comparison nb points to the path where the datasets (directly, without intake). As the notebook calculations uses pre-processed data (the global means were calculated apart), the update also must rewrite the calculations to include not only the annual means calculations but also the global means calculations in the notebook, so the intake call points to the "raw" cmip6 data.2021-02-08https://gitlab.dkrz.de/data-infrastructure-services/tutorials-and-use-cases/-/issues/37Update tauc Gitlab repo with IS-ENES3 Github repo content2021-04-01T09:22:54ZGhost UserUpdate tauc Gitlab repo with IS-ENES3 Github repo contentnew folder hierarchy, spec, make_kernel.sh,...new folder hierarchy, spec, make_kernel.sh,...https://gitlab.dkrz.de/data-infrastructure-services/tzis/-/issues/2tracking_ids2021-09-20T10:26:06ZFabian Wachsmanntracking_idsCheck for global attributes like tracking_ids that need to be collected in the consolidated metadata.Check for global attributes like tracking_ids that need to be collected in the consolidated metadata.https://gitlab.dkrz.de/data-infrastructure-services/intake-esm/-/issues/6Subest Catalog(s) for CMIP62020-05-07T07:44:23ZFabian WachsmannSubest Catalog(s) for CMIP6The entire CMIP6 catalog requires at least 10GB memory so that 2 options of the jupyterhub server of dkrz cannot be used.
We therefore think about having subsets of the Catalogs to simplify its application.
Any suggestions?The entire CMIP6 catalog requires at least 10GB memory so that 2 options of the jupyterhub server of dkrz cannot be used.
We therefore think about having subsets of the Catalogs to simplify its application.
Any suggestions?https://gitlab.dkrz.de/data-infrastructure-services/tutorials-and-use-cases/-/issues/35Store Data Selection by Intake as csv/json - Notebook Example2021-02-19T12:35:47ZMarco KulükeStore Data Selection by Intake as csv/json - Notebook ExampleMarco KulükeMarco Kulükehttps://gitlab.dkrz.de/data-infrastructure-services/tutorials-and-use-cases/-/issues/28Start Gitlab-pages for tauc repo - sphinx2021-01-25T09:32:17ZGhost UserStart Gitlab-pages for tauc repo - sphinxCreate a blog for tauc, probably using Sphinx, coordinated with the development of gitlab-pages for the Zarr repoCreate a blog for tauc, probably using Sphinx, coordinated with the development of gitlab-pages for the Zarr repo2021-01-22https://gitlab.dkrz.de/data-infrastructure-services/tutorials-and-use-cases/-/issues/31Set-up Gitlab-pages for tauc repo2021-07-14T13:55:37ZGhost UserSet-up Gitlab-pages for tauc repoLets try a config where all notebooks are executed and hosted. That has the advantage of having tests run at the same time.
Original comment:
> We already have gitlab pages build with Sphinx, which makes documentation-like pages (boring...Lets try a config where all notebooks are executed and hosted. That has the advantage of having tests run at the same time.
Original comment:
> We already have gitlab pages build with Sphinx, which makes documentation-like pages (boring :worried:) so we will set new gitlab pages with Jekyll for a more blog-like style.Fabian WachsmannFabian Wachsmannhttps://gitlab.dkrz.de/data-infrastructure-services/tutorials-and-use-cases/-/issues/30Re-Write Notebooks Climate-data-analysis-service from PyOphidia to xarray2021-02-19T12:35:02ZMarco KulükeRe-Write Notebooks Climate-data-analysis-service from PyOphidia to xarrayOld PyOphidia Notebooks https://github.com/ECAS-Lab/ecas-notebooks
New xarray Notebooks https://github.com/IS-ENES-Data/Climate-data-analysis-serviceOld PyOphidia Notebooks https://github.com/ECAS-Lab/ecas-notebooks
New xarray Notebooks https://github.com/IS-ENES-Data/Climate-data-analysis-serviceMarco KulükeMarco Kulükehttps://gitlab.dkrz.de/data-infrastructure-services/tzis/-/issues/6rename it tzis2021-12-15T09:58:07ZFabian Wachsmannrename it tzishttps://gitlab.dkrz.de/data-infrastructure-services/tutorials-and-use-cases/-/issues/15Plotting CMIP6 data on unstructured grids with psyplot2021-08-25T15:26:39ZFabian WachsmannPlotting CMIP6 data on unstructured grids with psyplotTaking advantage of my work for ICON standardization and use it as well in this repo.
Introducing basic psyplot methods for ICON and AWI-FESOM.Taking advantage of my work for ICON standardization and use it as well in this repo.
Introducing basic psyplot methods for ICON and AWI-FESOM.Fabian WachsmannFabian Wachsmannhttps://gitlab.dkrz.de/data-infrastructure-services/tzis/-/issues/9Only metadata is written, no chunks2022-03-02T08:58:41ZHauke SchulzOnly metadata is written, no chunksI did not come up with a minimal reproducible example yet, but I was hoping that someone might already have a solution to the issue that I'm encountering. I'm not sure if this is actually an issue of this package or not though.
In any c...I did not come up with a minimal reproducible example yet, but I was hoping that someone might already have a solution to the issue that I'm encountering. I'm not sure if this is actually an issue of this package or not though.
In any case, this is my code and traceback:
```python
In [4]: import xarray as xr
...: from tzis import tzis
...: ACCOUNT="mh0010"
...: USERNAME="m300408"
...: CONTAINER_NAME="EUREC4A_LES"
...: ZARR_DSET_NAME="experiment_1/ICON_DOM01_radiation.zarr/"
...:
...: token=tzis.get_token(ACCOUNT, USERNAME=USERNAME)
...: container = tzis.Tzis(token["OS_STORAGE_URL"],token["OS_AUTH_TOKEN"],CONTAINER_NAME,use_fsspec=True)
...: prefix_for_object_storage=ZARR_DSET_NAME
...:
...: container.open_store(prefix_for_object_storage)
...:
...: zf=xr.open_zarr("ICON_DOM01_radiation.zarr")
...: zf.to_zarr(container.store,mode='w')
...:
---------------------------------------------------------------------------
PathNotFoundError Traceback (most recent call last)
<ipython-input-4-4b9c5e661c66> in <module>
13
14 zf=xr.open_zarr("ICON_DOM01_radiation.zarr")
---> 15 zf.to_zarr(container.store,mode='w')
~/.local/lib/python3.8/site-packages/xarray/core/dataset.py in to_zarr(self, store, chunk_store, mode, synchronizer, group, encoding, compute, consolidated, append_dim, region, safe_chunks)
2029 encoding = {}
2030
-> 2031 return to_zarr(
2032 self,
2033 store=store,
~/.local/lib/python3.8/site-packages/xarray/backends/api.py in to_zarr(dataset, store, chunk_store, mode, synchronizer, group, encoding, compute, consolidated, append_dim, region, safe_chunks)
1412 writer = ArrayWriter()
1413 # TODO: figure out how to properly handle unlimited_dims
-> 1414 dump_to_store(dataset, zstore, writer, encoding=encoding)
1415 writes = writer.sync(compute=compute)
1416
~/.local/lib/python3.8/site-packages/xarray/backends/api.py in dump_to_store(dataset, store, writer, encoder, encoding, unlimited_dims)
1122 variables, attrs = encoder(variables, attrs)
1123
-> 1124 store.store(variables, attrs, check_encoding, writer, unlimited_dims=unlimited_dims)
1125
1126
~/.local/lib/python3.8/site-packages/xarray/backends/zarr.py in store(self, variables, attributes, check_encoding_set, writer, unlimited_dims)
557 )
558 if self._consolidate_on_close:
--> 559 zarr.consolidate_metadata(self.zarr_group.store)
560
561 def sync(self):
~/.local/lib/python3.8/site-packages/zarr/convenience.py in consolidate_metadata(store, metadata_key)
1132 }
1133 store[metadata_key] = json_dumps(out)
-> 1134 return open_consolidated(store, metadata_key=metadata_key)
1135
1136
~/.local/lib/python3.8/site-packages/zarr/convenience.py in open_consolidated(store, metadata_key, mode, **kwargs)
1190 # pass through
1191 chunk_store = kwargs.pop('chunk_store', None) or store
-> 1192 return open(store=meta_store, chunk_store=chunk_store, mode=mode, **kwargs)
~/.local/lib/python3.8/site-packages/zarr/convenience.py in open(store, mode, **kwargs)
104 return open_group(_store, mode=mode, **kwargs)
105 else:
--> 106 raise PathNotFoundError(path)
107
108
PathNotFoundError: nothing found at path ''
```
The error occurs independent of the value of `use_fsspec`. When I investigate the objects on the swiftbrowser, all metafiles (.zattrs, .zarray, .zmetadata) are written, but no single chunk.
Cheers,
Haukehttps://gitlab.dkrz.de/data-infrastructure-services/tutorials-and-use-cases/-/issues/36Model skills notebook with ERA52021-03-29T07:54:29ZGhost UserModel skills notebook with ERA5It was a nice idea, but I will not champion it.
Research question (thanks Iuliia Polkova!) **"Evaluation of prediction skill for surface air temperature and precipitation from the CMIP6 models using ERA5"**
**STEP 1**. DONE find ERA in...It was a nice idea, but I will not champion it.
Research question (thanks Iuliia Polkova!) **"Evaluation of prediction skill for surface air temperature and precipitation from the CMIP6 models using ERA5"**
**STEP 1**. DONE find ERA in dkrz data pool --> /pool/data/ERA5/ (thanks Fabi! also for the docs: https://confluence.ecmwf.int/display/CKB/ERA5%3A+data+documentation#ERA5:datadocumentation-Introduction)
**STEP 2**. DONE how to load ERA5 --> no problem with GRIB: http://xarray.pydata.org/en/stable/examples/ERA5-GRIB-example.html (thanks Marco!)
**STEP 3**. DONE load model data --Y intake and xarray, just copy-paste from the summer days nb's
**STEP 4**. TODO how to compare ERA-model: Iuliia's answer:
"load the data, calculate ensemble mean, calculate anomalies with respect to the long-term mean, calculate RMSE or CORR for different lead times et voila. If you have problems just let me know, I will help. The skill is usually calculated for anomalies and for lead times, so for prediction year 1, and multi-year averages such as years 2-5. Depending on the research problem, we calculate skill for each grid-point or for the anomalies averaged over a certain region. Here is one of my papers with the examples of skill calculation https://doi.org/10.1029/2018MS001439"https://gitlab.dkrz.de/data-infrastructure-services/tzis/-/issues/3integration tests2021-09-29T16:16:34ZFabian Wachsmannintegration testsdoes tzis work for
- cmip6
- cordex
- era5
? select 3 files for each, put it in git lfs, setup a cidoes tzis work for
- cmip6
- cordex
- era5
? select 3 files for each, put it in git lfs, setup a cihttps://gitlab.dkrz.de/data-infrastructure-services/intake-esm/-/issues/7Integrate Cloud upload of catalog into cronjob2020-10-14T10:24:45ZFabian WachsmannIntegrate Cloud upload of catalog into cronjobhttps://gitlab.dkrz.de/data-infrastructure-services/tutorials-and-use-cases/-/issues/24How to access data with intake catalogs2021-04-12T08:03:52ZFabian WachsmannHow to access data with intake catalogshttps://gitlab.dkrz.de/data-infrastructure-services/intake-esm/-/issues/11empty 0-size files: nbp_Lmon_EC-Earth3-CC_piControl_r1i1p1f12021-04-26T12:19:02ZAaron Springaaron.spring@mpimet.mpg.deempty 0-size files: nbp_Lmon_EC-Earth3-CC_piControl_r1i1p1f1I found files of size 0, which cannot be opened with CDO or xarray. Should I report such files also somewhere else? @k204210
Can we add a check avoid such files when creating in the catalogs?
```
ls -all /mnt/lustre02/work/ik1017/CMIP...I found files of size 0, which cannot be opened with CDO or xarray. Should I report such files also somewhere else? @k204210
Can we add a check avoid such files when creating in the catalogs?
```
ls -all /mnt/lustre02/work/ik1017/CMIP6/data/CMIP6/CMIP/EC-Earth-Consortium/EC-Earth3-CC/piControl/r1i1p1f1/Lmon/nbp/gr/v20210114
total 8
drwxr-sr-x 2 k204210 esgf 4096 Mar 16 16:20 ./
drwxr-sr-x 3 k204210 esgf 4096 Mar 16 16:20 ../
-r--r--r-- 1 k204210 esgf 0 Feb 15 19:36 nbp_Lmon_EC-Earth3-CC_piControl_r1i1p1f1_gr_191401-191412.nc
-r--r--r-- 1 k204210 esgf 0 Feb 15 19:36 nbp_Lmon_EC-Earth3-CC_piControl_r1i1p1f1_gr_199301-199312.nc
-r--r--r-- 1 k204210 esgf 0 Feb 15 19:36 nbp_Lmon_EC-Earth3-CC_piControl_r1i1p1f1_gr_200201-200212.nc
-r--r--r-- 1 k204210 esgf 0 Feb 15 19:36 nbp_Lmon_EC-Earth3-CC_piControl_r1i1p1f1_gr_201801-201812.nc
-r--r--r-- 1 k204210 esgf 0 Feb 15 19:36 nbp_Lmon_EC-Earth3-CC_piControl_r1i1p1f1_gr_202501-202512.nc
-r--r--r-- 1 k204210 esgf 0 Feb 15 19:36 nbp_Lmon_EC-Earth3-CC_piControl_r1i1p1f1_gr_205001-205012.nc
-r--r--r-- 1 k204210 esgf 0 Feb 15 19:36 nbp_Lmon_EC-Earth3-CC_piControl_r1i1p1f1_gr_208401-208412.nc
-r--r--r-- 1 k204210 esgf 0 Feb 15 19:36 nbp_Lmon_EC-Earth3-CC_piControl_r1i1p1f1_gr_216701-216712.nc
-r--r--r-- 1 k204210 esgf 0 Feb 15 19:36 nbp_Lmon_EC-Earth3-CC_piControl_r1i1p1f1_gr_218801-218812.nc
-r--r--r-- 1 k204210 esgf 0 Feb 15 19:36 nbp_Lmon_EC-Earth3-CC_piControl_r1i1p1f1_gr_221701-221712.nc
-r--r--r-- 1 k204210 esgf 0 Feb 15 19:36 nbp_Lmon_EC-Earth3-CC_piControl_r1i1p1f1_gr_224201-224212.nc
-r--r--r-- 1 k204210 esgf 0 Feb 15 19:36 nbp_Lmon_EC-Earth3-CC_piControl_r1i1p1f1_gr_226201-226212.nc
-r--r--r-- 1 k204210 esgf 0 Feb 15 19:36 nbp_Lmon_EC-Earth3-CC_piControl_r1i1p1f1_gr_226301-226312.nc
-r--r--r-- 1 k204210 esgf 0 Feb 15 19:36 nbp_Lmon_EC-Earth3-CC_piControl_r1i1p1f1_gr_229401-229412.nc
-r--r--r-- 1 k204210 esgf 0 Feb 15 19:36 nbp_Lmon_EC-Earth3-CC_piControl_r1i1p1f1_gr_233201-233212.nc
-r--r--r-- 1 k204210 esgf 0 Feb 15 19:36 nbp_Lmon_EC-Earth3-CC_piControl_r1i1p1f1_gr_234101-234112.nc
-r--r--r-- 1 k204210 esgf 0 Feb 15 19:36 nbp_Lmon_EC-Earth3-CC_piControl_r1i1p1f1_gr_234301-234312.nc
```
also opendap does seem to work:
```python
xarray.open_dataset('http://esgf3.dkrz.de/thredds/dodsC/cmip6/CMIP/EC-Earth-Consortium/EC-Earth3-CC/piControl/r1i1p1f1/Lmon/nbp/gr/v20210114/nbp_Lmon_EC-Earth3-CC_piControl_r1i1p1f1_gr_191401-191412.nc')
...
file, cached = self._acquire_with_cache_info(needs_lock)
File "/work/mh0727/m300524/conda-envs/s2s-ai/lib/python3.7/site-packages/xarray/backends/file_manager.py", line 205, in _acquire_with_cache_info
file = self._opener(*self._args, **kwargs)
File "netCDF4/_netCDF4.pyx", line 2291, in netCDF4._netCDF4.Dataset.__init__
File "netCDF4/_netCDF4.pyx", line 1855, in netCDF4._netCDF4._ensure_nc_success
OSError: [Errno -70] NetCDF: DAP server error: b'http://esgf3.dkrz.de/thredds/dodsC/cmip6/CMIP/EC-Earth-Consortium/EC-Earth3-CC/piControl/r1i1p1f1/Lmon/nbp/gr/v20210114/nbp_Lmon_EC-Earth3-CC_piControl_r1i1p1f1_gr_191401-191412.nc'