CDI consolidation

Goal

Consolidate all CDI developments into the develop branch of this repository so that there is a common codebase of the library that has all features that are currently implemented and maintained in different repositories and branches.

Apart from the obvious maintenance benefits (e.g. no need to back merge certain features to different versions) in the long run, the immediate (and quite urgent) goal is to have a version of the library that provide the following features:

1. FDB5 support needed for the ACROSS project (available in develop@libcdi starting version 2.0.0).
2. Certain GRIB2 compression features requested by DWD (available in develop@libcdi starting version 1.9.2).
3. Parallel I/O (available in master@cdi-pio, which is based on version 1.8.2).
4. Compatibility with the ICON build system (fully available only in cdi-1.8.x@libcdi, which is based on version 1.8.2).

Tasks

1. Merge master@cdi-pio into develop@libcdi (not necessarily directly, see the steps below). Some time ago, Thomas (@jahns) reported on problems that hinder the merge (see 2021-06-11_CDI-PIO_merge_activities.pdf). However, according to Uwe (@m214003), most of them (if not all) are resolved, starting version 2.0.0. We need confirmation from Thomas on this.
2. Make sure that all commits in cdi-1.8.x@libcdi starting 7ce36fce either have already found their way into develop@libcdi or are irrelevant.
3. Update the build system of the library to make it compatible with the ICON build system. For example, the configure script should allow for better protection against invalid values of certain environment variables and do not assume that, for example, YAXT is already built and installed because that is not the case when configuring and building the bundled libraries of ICON.
4. Set up a test infrastructure for develop@libcdi, make sure that all features and configurations of interest are covered with respective tests and that the HEAD of develop@libcdi passes them at all times. Ralf (@k202125) has already made the first steps in that direction (see https://bb-cdi.dkrz.de and https://gitlab.dkrz.de/mpim-sw/libcdi/-/pipelines) but they are not applicable to develop@libcdi yet.

Steps

We basically need to consolidate three branches — develop@libcdi, master@cdi-pio and cdi-1.8.x@libcdi — plus introduce certain changes to the build system. The tentative plan, which we have come up with in the meeting between MPI-M and DKRZ on the 25th of October, was to merge master@cdi-pio into develop@libcdi as the first step. However, in the follow-up meeting between me and Thomas on the 27th of October, we agreed to base our work on !8 (closed), which consolidates cdi-1.8.x@libcdi and master@cdi-pio plus almost fully covers tasks 3 and 4 (see above). Therefore, the plan is:

• 1. Thomas reviews !8 (closed).
• 2. I introduce changes to the merge request that are required to get Thomas's approval and do the merge into cdi-1.8.x@libcdi.
• 2.1 Thomas review/tests !11 (merged) - approval required in case of positive result
• 3. Ralf makes sure that the test pipelines of CDI work correctly for cdi-1.8.x@libcdi.
• 4. I finalize the integration of cdi-1.8.x@libcdi into ICON (see https://gitlab.dkrz.de/icon/icon-cimd/-/merge_requests/32).
• 5. review/test needed for !13 (merged) DKRZ merges cdi-1.8.x@libcdi into develop@libcdi. This step is currently the heaviest and the most uncertain one and will probably be split into several substeps.
• 6. Ralf makes sure that the test pipelines of CDI work correctly for develop@libcdi (and they must never go "red" ever after).
• 7. I integrate develop@libcdi into ICON from the build system perspective (to account for the new configure options and macros that currently exist only in develop@libcdi).
• 8. Luis (@m214089) integrates develop@libcdi into ICON from the API perspective.
• 9. Ralf updates and extends the BB of ICON to cover the parallel I/O functionality better (results-comparison with reference data)

Timeline

MPI-M are very interested in having this done by the end of this year and are ready to minimize any delays on their side. DKRZ are kindly asked to give this work higher priority and share their plans on when they can get item 1 on the list above done and start working on item 5.

Other people who might be interested in this but have not been mentioned yet: @k202061 @m218027 @m300910 @m300913 @m300196 @m300173 @b380563

marked the checklist item 1. Thomas reviews !8 (closed). as completed

marked the checklist item 1. Thomas reviews !8 (closed). as incomplete

marked the checklist item 1. Thomas reviews !8 (closed). as completed

marked the checklist item 1. Thomas reviews !8 (closed). as incomplete

I propose to drop or revert 021117cd, it only works with MPI versions where all MPI subroutines are declared in the module (many don't). This is the reason, all YAXT code only uses the plain mpi module without an ONLY clause.

mentioned in merge request !8 (closed)

@jahns I suggest that we discuss the details of the merge (step 1) here: !8 (closed). See my reply there.

mentioned in commit 041f02eb

mentioned in commit cdcdf0ef

mentioned in commit c22851ae

mentioned in commit 2534cbf6

mentioned in commit 059593e6

mentioned in commit a1ff17a2

mentioned in commit e3b5348b

mentioned in commit fd1ffb7e

mentioned in commit 5db76225

mentioned in commit 9e1f7470

mentioned in commit 354c8d02

mentioned in commit 67f3fc5e

mentioned in commit 97f0bd87

mentioned in commit e30fdcc4

mentioned in commit 8123ab78

mentioned in commit 46d5ca5b

mentioned in commit a66d7405

mentioned in commit b4a15034

mentioned in commit fca3fc9a

mentioned in commit 2b4eb92e

mentioned in commit 963d6bf1

mentioned in commit 95c021fb

mentioned in commit aff53733

mentioned in commit f8375f03

mentioned in commit df4e9ffe

mentioned in commit 72f6138b

marked the checklist item 1. Thomas reviews !8 (closed). as completed

mentioned in commit a92e5e32

mentioned in commit 3f7c251f

mentioned in commit 5e58b994

mentioned in commit 6577564b

mentioned in commit 5e86d508

mentioned in commit 1ed9702a

marked this issue as related to #5 (closed)

mentioned in commit ace33f8d

mentioned in commit 6ac905cb

mentioned in commit 8fea86ee

mentioned in commit 4658fe28

changed the description

mentioned in commit e540f145

mentioned in commit dfe8308a

mentioned in commit 0960b084