CDI consolidation

Goal

Consolidate all CDI developments into the develop branch of this repository so that there is a common codebase of the library that has all features that are currently implemented and maintained in different repositories and branches.

Apart from the obvious maintenance benefits (e.g. no need to back merge certain features to different versions) in the long run, the immediate (and quite urgent) goal is to have a version of the library that provide the following features:

  1. FDB5 support needed for the ACROSS project (available in develop@libcdi starting version 2.0.0).
  2. Certain GRIB2 compression features requested by DWD (available in develop@libcdi starting version 1.9.2).
  3. Parallel I/O (available in master@cdi-pio, which is based on version 1.8.2).
  4. Compatibility with the ICON build system (fully available only in cdi-1.8.x@libcdi, which is based on version 1.8.2).

Tasks

  1. Merge master@cdi-pio into develop@libcdi (not necessarily directly, see the steps below). Some time ago, Thomas (@jahns) reported on problems that hinder the merge (see 2021-06-11_CDI-PIO_merge_activities.pdf). However, according to Uwe (@m214003), most of them (if not all) are resolved, starting version 2.0.0. We need confirmation from Thomas on this.
  2. Make sure that all commits in cdi-1.8.x@libcdi starting 7ce36fce either have already found their way into develop@libcdi or are irrelevant.
  3. Update the build system of the library to make it compatible with the ICON build system. For example, the configure script should allow for better protection against invalid values of certain environment variables and do not assume that, for example, YAXT is already built and installed because that is not the case when configuring and building the bundled libraries of ICON.
  4. Set up a test infrastructure for develop@libcdi, make sure that all features and configurations of interest are covered with respective tests and that the HEAD of develop@libcdi passes them at all times. Ralf (@k202125) has already made the first steps in that direction (see https://bb-cdi.dkrz.de and https://gitlab.dkrz.de/mpim-sw/libcdi/-/pipelines) but they are not applicable to develop@libcdi yet.

Steps

We basically need to consolidate three branches — develop@libcdi, master@cdi-pio and cdi-1.8.x@libcdi — plus introduce certain changes to the build system. The tentative plan, which we have come up with in the meeting between MPI-M and DKRZ on the 25th of October, was to merge master@cdi-pio into develop@libcdi as the first step. However, in the follow-up meeting between me and Thomas on the 27th of October, we agreed to base our work on !8 (closed), which consolidates cdi-1.8.x@libcdi and master@cdi-pio plus almost fully covers tasks 3 and 4 (see above). Therefore, the plan is:

  • 1. Thomas reviews !8 (closed).
  • 2. I introduce changes to the merge request that are required to get Thomas's approval and do the merge into cdi-1.8.x@libcdi.
  • 2.1 Thomas review/tests !11 (merged) - approval required in case of positive result
  • 3. Ralf makes sure that the test pipelines of CDI work correctly for cdi-1.8.x@libcdi.
  • 4. I finalize the integration of cdi-1.8.x@libcdi into ICON (see https://gitlab.dkrz.de/icon/icon-cimd/-/merge_requests/32).
  • 5. review/test needed for !13 (merged) DKRZ merges cdi-1.8.x@libcdi into develop@libcdi. This step is currently the heaviest and the most uncertain one and will probably be split into several substeps.
  • 6. Ralf makes sure that the test pipelines of CDI work correctly for develop@libcdi (and they must never go "red" ever after).
  • 7. I integrate develop@libcdi into ICON from the build system perspective (to account for the new configure options and macros that currently exist only in develop@libcdi).
  • 8. Luis (@m214089) integrates develop@libcdi into ICON from the API perspective.
  • 9. Ralf updates and extends the BB of ICON to cover the parallel I/O functionality better (results-comparison with reference data)

Timeline

MPI-M are very interested in having this done by the end of this year and are ready to minimize any delays on their side. DKRZ are kindly asked to give this work higher priority and share their plans on when they can get item 1 on the list above done and start working on item 5.

Other people who might be interested in this but have not been mentioned yet: @k202061 @m218027 @m300910 @m300913 @m300196 @m300173 @b380563

Edited by Sergey Kosukhin

Imprint | Privacy Policy