This one-day workshop will introduce participants to ICON data analysis, covering data access, variables, grids, plotting, remapping, and analysis techniques.
**Notes:**
- Drinks will be available throughout the workshop.
- Participants should bring their laptops and have access to Levante
Detailed information of ICON-Sapphire simulations can be found [here](https://owncloud.gwdg.de/index.php/apps/onlyoffice/s/rgYtnOROmJkmZRg?fileId=3122041030).
<details><summary><b>Table of Sapphire projects and simulations </b></summary>
To access raw output, one needs to know the location of the files. Since most Ruby data has not been catalogued yet, we'll access them directly from their output path. Available Ruby simulations and their respective output locations are tabulated [here](#Ruby-as-of-10122024)
### Command line
Let's take the example from a CMIP7 experiment, `slo1826`.
One can use `query_yaml` to browse through the catalogue, look into its tree structure, available streams, output variables, possibly find location of output :smirk:, etc. Some details for the usage of `query_yaml` can be found on [easygems webpage](https://easy.gems.dkrz.de/Processing/Intake/query_yaml.html).
#### To use `query_yaml`, load the necessary modules
```bash
module use /work/k20200/k202134/hsm-tools/outtake/module
-`query_yaml -c ${rubycat} cmip7 slo1826 atm_2d_ml --var t_2m --cdo` gives path to data
> **Sapphire**
-`query_yaml -c ${sappcat} ICON` will list only ICON-Sapphire simulations that are part of the general catalogue. Note that this is from NextGEMS cycle3 onwards.
<details><summary>text dump snippet from "query_yaml -c ${sappcat} ICON" </summary>
<details><summary>text dump snippet from "query_yaml -c ${sappcat} ICON ngc3028" </summary>
```
Choices for this dataset:
name description type allowed default
0 time time resolution of the dataset str [PT30M, PT3H, P1D] P1D
1 zoom zoom resolution of the dataset int [10, 9, 8, 7, 6, 5, 4, 3, 2, 1, 0] 0
Variables:
a_tracer_v_to (time, depth_half, cell) sea water vertical diffusivity
atmos_fluxes_frshflux_evaporation (time, cell)
atmos_fluxes_frshflux_precipitation (time, cell)
......
```
</details>
-`query_yaml -c ${sappcat} ICON ngc3028 --search_args time=PT3H zoom=5` lists the variables available available at 3-hourly intervals and on zoom 5
-`query_yaml -c ${sappcat} ICON ngc3028 --search_args time=PT3H zoom=5 --var=hfls --uri --cdo` gives the path for latent heat flux at 3-hourly intervals and on zoom 5
> **EERIE**
-`query_yaml -c ${eeriecat} dkrz disk model-output icon-esm-er eerie-control-1950 v20231106 ocean native 2d_monthly_mean` gives list of variable in stream
<details><summary>text dump snippet from "query_yaml -c ${eeriecat} dkrz disk model-output icon-esm-er eerie-control-1950 v20231106 ocean native 2d_monthly_mean" </summary>
```
Choices for this dataset:
Variables:
Qbot (time, lev, ncells) Conductive heat flux at ice-ocean interface
Qtop (time, lev, ncells) Energy flux available for surface melting
Wind_Speed_10m (time, ncells) Wind Speed at 10m height
......
```
</details>
-`query_yaml -c ${eeriecat} dkrz disk model-output icon-esm-er eerie-control-1950 v20231106 atmos native 2d_monthly_mean --var tas --uri --cdo` gives path to data
-[EERIE data access on shell environment](https://pad.gwdg.de/nSKolAzGQY2zLz3oOr0QSQ?both#EERIE-data-access-on-shell-environment-for-cdo-operations) from hackathon starter pack
### Python (Sapphire)
[nextGEMS data access via intake](https://pad.gwdg.de/nSKolAzGQY2zLz3oOr0QSQ?both#NextGEMS-data-access-via-intake) from hackathon starter pack
[EERIE data access via intake](https://pad.gwdg.de/nSKolAzGQY2zLz3oOr0QSQ?both#EERIE-data-access-via-intake) from hackathon starter pack
The physics package of the atmospheric model used in ICON-XPP and ICON-Sapphire are different. Similarly, their naming convention for variables and parameters are different. Here's a table listing some of the variables from the atmospheric component of ICON-XPP.
The physics package of the atmospheric model used in ICON-XPP and ICON-Sapphire are different. Similarly, their naming convention for variables and parameters are different. Here's a table listing some of the variables from the atmospheric component of ICON-Sapphire.
The nomenclature (R*n*B*k*) for ICON resolution derives from how an icosahedron is divided. Taking a grid that originates from an icosahedron, the the edge of each triangle face is initially divided into *n* parts (thus `R02` in most of our cases), followed by *k* subsequent edge bisections. When a particular resolution grid is generated, a grid ID is officially tagged to it. Official list of gridIDs associated with [MPIM](http://icon-downloads.mpimet.mpg.de/mpim_grids.xml) and [DWD](http://icon-downloads.mpimet.mpg.de/dwd_grids.xml) are on their respective sites.
<details><summary><b> Abbreviated table of resolution and their gridID </b></summary>
As higher and higher resolution modelling are performed, efficiency is paramount for the post-processing of these very large dataset. To just view the data or compute area-average of of a large region, we don't necessarily need such high resolution. HEALpix co-ordinate system (heavily used in astronomy) allows for efficient access and computation of these humongous data set.
For plotting ICON data, we rely a lot on python and some available packages. Nils Brüggemann has developed an amazing package, called **pyICON**, that aids in the analysis of ICON data.
<details><summary><b> Modified snippet from <ahref="https://gitlab.dkrz.de/m300602/pyicon/-/blob/master/quickstart/notebook_plotting.ipynb?ref_type=heads"> pyICON example </a></b></summary>
```python
importxarrayasxr
importcartopy.crsasccrs
importmatplotlib.pyplotasplt
importnumpyasnp
importpyiconaspyic
#Example from CMIP7 experiment with r2b4atm, r2b6oce
We will use a script called `pyic_fig.py`, that is part of pyICON. Please refer to `pyic_fig.py --help` for the myraid options you can use to tweak your figure.
<details><summary><b> Quick view example using command line</b></summary>
[Next section on remapping](#Remapping) includes plotting of regridded data.
### Ocean sectional plots
**Python**: Please refer to [pyicon notebook example](https://gitlab.dkrz.de/m300602/pyicon/-/blob/master/quickstart/notebook_plotting.ipynb?ref_type=heads), cell 13 onwards.
**Command line**: A script called `pyic_sec.py`, that is part of pyICON, can be used for making quick sectional plots for 170W, 30W, global zonal average, Atlantic zonal average and Indo-Pacific zonal average. Please refer to `pyic_sec.py --help` for the myraid options you can use to tweak your figure.
<details><summary><b> Sample script using pyic_sec.py </b></summary>
<imgsrc="https://pad.gwdg.de/uploads/200b41b0-f4f4-4967-8ad2-10e79b3c877a.png"alt="ncview global 2m temperature at 1x1deg resolution"width="450"/>
</details>
## Caveats in ICON
Honorable mentions on some caveats in ICON that may matter for the analysis.
:::warning
1) 3D **time-mean** atmospheric output on model levels only. See next section to [interpolate to pressure level](#Interpolating-from-model-levels-to-pressure-levels)
2) Time shift in output (e.g. `cdo -shifttime,-14days` for monthly files)
3) For sea surface height, better to use `ssh` instead of `zos`, especially under sea-ice
4) ICON-A is non-hydrostatic, be careful with conversion of wa to omega
5) Ocean has time-variant vertical coordinate system (zstar)
:::
## Atmospheric 3D variables
### Interpolating from model levels to pressure levels
ICON can only output on pressure levels if they are instantaneous outputs. For time-mean variables, they have to be outputted on model levels. Hence, you may need to interpolate from model level to pressure level, which is easily done with cdo.
<details><summary><b> Command line example for interpolation (model to pressure levels) </b></summary>
We use C5 experiments (ICON-Sapphire) as an example.
```brainfuck=bash
#Find where files are located using `query_yaml`
module use /work/k20200/k202134/hsm-tools/outtake/module
[View quick plots on swiftbrowser](https://swift.dkrz.de/v1/dkrz_b70a6895-056b-4057-943d-2495ab773b09/m300466_DRAGON/dragon/qp-dap010_r3b7-r2b9/tave_1951-02-01-1954-01-01/qp_index.html)
### Climate Variability Diagnostic Package
CVDP is a package to obtain quick assessment of the different climate modes in the simulations. You can download the newest version v6.0.0 at [https://github.com/NCAR/CVDP-ncl?tab=readme-ov-file](https://github.com/NCAR/CVDP-ncl?tab=readme-ov-file)
The observational dataset used in CVDP can be downloaded from [https://www.cesm.ucar.edu/projects/cvdp/observations](https://www.cesm.ucar.edu/projects/cvdp/observations)
sed -i "s/\/project\/diag\/Comparison1/\/work\/mh1421\/m300466\/tools\/CVDP\/cvdpplots\/${expid}_cvdp_${ystart}-${yend}/" ${expid}_${ystart}-${yend}.ncl
sed -i "s/Title\ goes\ here/${expid}\ (${ystart}-${yend})/" ${expid}_${ystart}-${yend}.ncl
ncl ${expid}_${ystart}-${yend}.ncl
mv ${expid}_${ystart}-${yend}.ncl driverscripts/
rm -rf ${datadir}
rm ${plotdir}/*.nc
```
</details>
<details><summary><b> Upload figures to swiftbrowser and view </b></summary>
Using `cdo` to perform basic time series analysis on ICON output.
### Correlation and Regression
Correlation and regression coefficients can be simulataneously computed for each grid point on the native grid so that we can obtain 2D maps of the bi-variate relations.
<details><summary><b> Example script for computing correlation and regression </b></summary>
This example uses a control CMIP7 experiment (slo1774; 160km atm resolution).
Be sure to remove trend and largest "known" time scale, e.g. seasonal cycle.
While variance/covariance gives you the combined variability on all time scales, the power/cross spectra allows us to assess the variability at particular frequency. Furthermore, the squared coherence, phase spectra and gain functions tells us the correlation, lead-lag and regression, respectively, at specific frequency/timescales.
The spectra can be simulataneously computed for each grid point on the native grid so that we can obtain 2D maps of how variability and bi-variate relations may look like on different frequency/timescales.
<details><summary><b> Example script for spectral analysis </b></summary>
This example uses a control CMIP7 experiment (slo1774; 160km atm resolution). The script is called `timespectra_iconxpp.sh`
## Example notebooks for some scientific analysis from 2024 pre-hackathon
This was prepared for the 2024 hackathon (nextGEMS, EERIE, WarmWorld). Here's a [collection of examples](https://github.com/eerie-project/EERIE_hackathon_2023/blob/pre-joint-hackathon-2024/pre-joint-hackathon-2024/README.md) particularly for topics on mesoscale eddies and their impact on air-sea coupling, ocean state (physical and BGC) and atmospheric state.
<details><summary><b> Collection of example notebooks </b></summary>
-[Eddy identification, tracking and compositing](https://github.com/eerie-project/EERIE_hackathon_2023/blob/pre-joint-hackathon-2024/pre-joint-hackathon-2024/eddy_track_composite/README.md) of physical (ocean/atmos) and biogeochemical properties
- ICON (read-in netcdf; serial running): [identify and track eddies](https://github.com/eerie-project/EERIE_hackathon_2023/blob/main/ICON/ICON-O/howto-IDtrackcompeddy-daily.ipynb), [build eddy composites](https://github.com/eerie-project/EERIE_hackathon_2023/blob/main/ICON/ICON-O/howto-eddycompositeotherfields-daily.ipynb), [plot composites](https://github.com/eerie-project/EERIE_hackathon_2023/blob/main/ICON/ICON-O/howto-plot-eddycompositesalongtrack-dm.ipynb) by Dian Putrasahan
- ICON ( [parallelise eddy identification code](https://github.com/eerie-project/EERIE_hackathon_2023/blob/pre-joint-hackathon-2024/pre-joint-hackathon-2024/eddy_track_composite/ICON/identify_fast.py) and [compositing with higher res data](https://github.com/eerie-project/EERIE_hackathon_2023/blob/pre-joint-hackathon-2024/pre-joint-hackathon-2024/eddy_track_composite/ICON/composite_tracks.py) than 0.25deg) by Moritz Epke
- ICON ([read-in as xarray; parallel running](https://github.com/eerie-project/EERIE_hackathon_2023/tree/main/RESULTS/pyeddytracker_xarray_dask_parallel)) by Aaron Wienkers
-[IFS-AMIP](https://github.com/eerie-project/EERIE_hackathon_2023/blob/pre-joint-hackathon-2024/RESULTS/eddy_composites-short.ipynb)(atm response) by Matthias Aengenheyster
-[IFS/FESOM](https://github.com/eerie-project/EERIE_hackathon_2023/blob/pre-joint-hackathon-2024/pre-joint-hackathon-2024/eddy_track_composite/IFS-FESOM/IDtrackeddy-daily-intake_IFSFESOM.ipynb)(use intake and read-in as xarray; parallel running) by Aaron Wienkers and Dian Putrasahan
-[HAMOCC](https://github.com/eerie-project/EERIE_hackathon_2023/blob/pre-joint-hackathon-2024/pre-joint-hackathon-2024/eddy_track_composite/HAMOCC/README.md)(read-in netcdf; serial running) by Johann Jungclaus and Dian Putrasahan
-[Spatial filters](https://github.com/eerie-project/EERIE_hackathon_2023/blob/pre-joint-hackathon-2024/pre-joint-hackathon-2024/mesoscale-air-sea-coupling/Spatial_Filters/README.md) on regular grid
1.[Weighted area-average smoothing](https://github.com/eerie-project/EERIE_hackathon_2023/blob/pre-joint-hackathon-2024/pre-joint-hackathon-2024/mesoscale-air-sea-coupling/Spatial_Filters/README.md#weighted-area-average-smoothing-using-cdo) using `cdo -smooth` operator.
-[Spatial filtering on HEALpix data](https://github.com/eerie-project/EERIE_hackathon_2023/blob/pre-joint-hackathon-2024/pre-joint-hackathon-2024/mesoscale-air-sea-coupling/Spatial_Filters/README.md#bonus-spatial-filtering-on-healpix-data).
2.[Bessel filter](https://github.com/eerie-project/EERIE_hackathon_2023/blob/pre-joint-hackathon-2024/pre-joint-hackathon-2024/mesoscale-air-sea-coupling/Spatial_Filters/README.md#bessel-filter) using a function from py-eddy-tracker by Dian Putrasahan
3.[Gaussian filter](https://github.com/eerie-project/EERIE_hackathon_2023/blob/pre-joint-hackathon-2024/pre-joint-hackathon-2024/mesoscale-air-sea-coupling/Spatial_Filters/README.md#gaussian-filter) using a function from GCM filters by Matthias Aengenheyster and Dian Putrasahan
-[Scale dependency of air-sea coupling](https://github.com/eerie-project/EERIE_hackathon_2023/blob/pre-joint-hackathon-2024/pre-joint-hackathon-2024/mesoscale-air-sea-coupling/README.md)
-[ICON](https://github.com/eerie-project/EERIE_hackathon_2023/blob/pre-joint-hackathon-2024/pre-joint-hackathon-2024/mesoscale-air-sea-coupling/ICON/README.md) by Dian Putrasahan
-[IFS/FESOM](https://github.com/eerie-project/EERIE_hackathon_2023/blob/pre-joint-hackathon-2024/pre-joint-hackathon-2024/mesoscale-air-sea-coupling/IFS-FESOM/README.md) by Matthias Aengenheyster, Rohit Ghosh and Dian Putrasahan
-[Mean vs eddy fluxes in the ocean](https://github.com/eerie-project/EERIE_hackathon_2023/blob/pre-joint-hackathon-2024/pre-joint-hackathon-2024/mean_eddy_flux/README.md)
-[vertical heat fluxes in ICON](https://github.com/eerie-project/EERIE_hackathon_2023/blob/main/RESULTS/vertical_heat_flux_ICON_WP6_hackathon.ipynb) by Stella Bērziņa and Matthias Münnich
-[in-situ or potential density from potential temperature and salinity](https://github.com/eerie-project/EERIE_hackathon_2023/tree/pre-joint-hackathon-2024/pre-joint-hackathon-2024/mean_eddy_flux#in-situ-or-potential-density)
-[find depth of isopycnal surfaces](https://github.com/eerie-project/EERIE_hackathon_2023/tree/pre-joint-hackathon-2024/pre-joint-hackathon-2024/mean_eddy_flux#depth-of-isopycnal-surfaces)
-[Impact of eddies on atmosphere mean state (IFS-AMIP)](https://github.com/eerie-project/EERIE_hackathon_2023/tree/pre-joint-hackathon-2024/RESULTS)
-[2D atm clim state](https://github.com/eerie-project/EERIE_hackathon_2023/blob/pre-joint-hackathon-2024/RESULTS/IFS_AMIP_atmosresponse_to_SST_forcing_djf_clmdiff.ipynb) by Iuliia Polkova
-[3D atm clim state](https://github.com/eerie-project/EERIE_hackathon_2023/blob/pre-joint-hackathon-2024/RESULTS/IFS_AMIP_atmosresponse_to_SST_forcing_djf_clmdiff_3D.ipynb) by Iuliia Polkova
-[atm covariability/correlation](https://github.com/eerie-project/EERIE_hackathon_2023/blob/pre-joint-hackathon-2024/RESULTS/IFS_AMIP_atmosresponse_to_SST_forcing_djf_correlation.ipynb) by Iuliia Polkova
-[local atm response (composites)](https://github.com/eerie-project/EERIE_hackathon_2023/blob/pre-joint-hackathon-2024/RESULTS/IFS_AMIP_composites.ipynb) by Chris Roberts and Matthias Aengenheyster
