Consolidation with CDI-PIO (develop)

Generally, this is !11 (merged) rebased onto the develop branch.

This is the first develop-based version accepted by the Buildbot of ICON: https://buildbot.dkrz.de/grid?branch=m300488/cdi-pio-2-integration

mentioned in issue #4 (closed)

added 803 commits

• 9e62932f...97e38865 - 236 commits from branch develop
• 574b1941 - Apply clang formatting to tests and examples.
• b21b9e4a - Add missing file data commit.
• 04a4b26b - Better handling of librt.
• d5fce445 - Cleanup of example programs.
• 3282b634 - Silence warning about symbol affecting external header.
• 0fddcaae - Improve log structure.
• 7d8a277f - Reduce number of print calls in printGridInfo and printZaxisInfo.
• 09a36384 - Fix missing include.
• ed29b510 - Remove redundant branch.
• 81f524cc - Use less indirection.
• a6801607 - Extract function to allocate time units string.
• f41b2937 - Move definition of variables to cdfDefVars.
• 2432f661 - Document polymorphic implementation for cdfDefTimestep.
• f1d0383a - Move cdfDefTimestep to stream_cdf_time.c.
• 444d35e9 - Move data to text segment.
• e45f0016 - Do not unnecessarily switch NetCDF files to data mode.
• f434d98e - Merge netcdf put operations.
• 7b647f99 - Narrow scope of variables in cdfDefZaxis.
• bd60c1b0 - Replace strlen with computation.
• 13b1bfd5 - Save short string copy.
• 56c4d2fc - Move invariant data to text segment.
• 7fba7235 - Replace multiple calls to same function with loop.
• 2e828ae8 - Replace table of pointers to strings with table of strings.
• 86707522 - Merge branches with identical outcome.
• 0b645c75 - Merge call into table-based loop.
• fe7594f6 - Use pointer to existing array instead of array copy.
• 39cb9e1d - Shorten assignment.
• be49ef2d - Restructure cdfInqTimestep.
• 98ac10cf - Reinstate scoping to allow temporal reuse.
• 0b1616e0 - Fix conditionally used declaration.
• c7d15f2f - Delay cdf_put_var calls until definition phase is finished.
• 87d79808 - Narrow scope of variables.
• a27d2875 - Reduce number of cdf_redef calls.
• 4e82a496 - Narrow variable scope.
• cf912b7d - Add query for grid property presence.
• 1008b10a - Use symbolic constant to clarify array access.
• c872f75a - Further prevent unnecessary loading of grid data.
• e6d18b05 - Use already stored result of query.
• fa384961 - Make table of namespace switches dynamic.
• 91383c73 - Extend endianness-invariant check-summing API for continuation.
• f06c1c0b - Add "this" pointer to valTxCodeFunc.
• 5d142f76 - Clarify control flow in grid comparison.
• 15d8389d - Clean examples for new API.
• 0e1ea690 - Fix incorrect order of operations.
• 87ae2413 - Remove static variable inhibiting generality.
• e96a1a5c - Merge declarations with initializations in namespaceAdaptKey.
• 4bd56c3c - Attach write mode cleanup to namespace.
• 372821c0 - Also check return value of fclose.
• 6813de73 - Store Xmap factory in namespace switches.
• fe64acf7 - Virtualize grid pack code for derived classes.
• 6ccc02b5 - Virtualize grid (un)packing functions.
• fda042d4 - Reduce number of branches and calls in gribapi interface.
• 0080620d - Express variation in C pre-processor flags via CPPFLAGS.
• 6cbe9bee - Extract creation of model grid to function.
• 83e74dc1 - Use signed arithmetic in test data generator.
• 8f931485 - Extract function to compose file names.
• 6282796a - Fix unnecessary indentation in test.
• 9255392e - Use increased flexibility of new CDI-PIO API in tests.
• 1097d35d - Clarify wrf_read_timestep.
• 7d1c671c - Extract function to open test streams.
• af72944a - Use no-fail malloc wrapper in test program.
• 41a900c6 - Make test value computations independent from grid coordinates.
• 772cc266 - Extend grid access with apply function.
• c4e67a68 - Document pio_write long options.
• a13c2bab - Fix oversight in C++ initializer.
• 593ee6e3 - Use no-fail wrappers in NetCDF test program.
• 7fae8f45 - Extract function to create lonlat grid.
• 12d128e3 - Re-use generator function in cksum_write.
• 14e491c2 - Move generation of curvilinear grid to utility function file.
• d1c72fdd - Change int to bool.
• 27c66ede - Move code parts for level computation together.
• 92f4d8de - Re-use grid generator function.
• f3c29b5e - Extend stream compose function for tests without sequence digits.
• 2d6a456a - Extend test grid generation function for chunks of curvilinear grids.
• fa5690a2 - Move invariant data to text segment.
• eb413cbf - Use size_t instead of unsigned as iteration counter.
• 099d4f6d - Use bit-flags instead of bools in test program configuration.
• f19e057b - Add support for curvilinear grids in PIO test program.
• cc35b26d - Fix bug in grid serialization.
• 96765cde - Fixup "Fix incorrect packing of positive axis flag."
• 8a65c58c - Fixup "Replaced gridInqXstdname()/gridInqYstdname() by cdiInqKeyInt() with CDI_KEY_STDNAME."
• d1270216 - Fixup "Added serializeKeysPack() and serializeKeysUnpack()."
• 016c0b7f - Renamed REDUCEDPOINTSSIZE to GRID_PACK_INT_IDX_REDUCED_POINTS_SIZE.
• 30545a88 - Sync vlistVarGetPackSize() and vlistVarPack().
• 5b3c8158 - Also print system error message where operating system call failed.
• 290dbb1c - Use SysError instead of perror and Error.
• 599d3451 - Improve gridInqParamRLL.
• 95125d29 - Virtualize grid xinc/yinc queries.
• bd7b5dfd - Move array to text segment when possible.
• 76bef1fc - Use no-fail wrappers in test.
• cc298aa6 - Add DATATYPE_{FLT|INT32} to MPI serialization.
• f4b79ed6 - Use more convenient predicates for grid array component packing.
• eada6655 - Allow NULL ops parameter for resh[GS]etStatus.
• 235cec88 - Shorten assignment.
• 5663e995 - Setup preferred xmap setup function on clients too.
• f0923be2 - Reduce overhead for YAXT calls on CDI PIO server processes.
• 70607e02 - Mark destroyed resources as unused.
• 3bfea65a - Fix protection of asynch I/O feature.
• cace861e - Revert "Removed unused grid_t member extraData."
• ce7c29ca - Merge handling of NetCDF X- and Y-axis for unstructured/curvilinear grids.
• 7f6443d0 - Use switch and strspn to simplify NetCDF axis parsing.
• 7a07becc - Prevent identical string compare.
• 86bbf119 - Shorten cdf_set_dimtype.
• 48907d76 - Put loop control in one place.
• 6be3769f - Fix missing predication.
• 1606da61 - Move CDF setup of grid properties to external function.
• 8548cd1e - Perform cheaper test first.
• 097d4c86 - Merge and re-order predicates by cost.
• e4af0e17 - Replace function calls with direct comparison.
• 42610306 - Improve temporal locality.
• 24a08b6e - Formulate doc-string more precisely.
• 77f2f51c - Make vectorization easier for compilers.
• 1ab5afb9 - Replace single-image operation with overrideable function call.
• 36bdb420 - Use temporary variable instead of redundant query.
• 0437d441 - Make internal functions available to parallel implementation.
• e7bbed81 - autotools update.
• c349cb6c - Fix copy&paste error in diagnostic message.
• 2bab4cdc - Add grib2 to tests.
• f58f6182 - Move variable closer to use and align datatype with use.
• fa836af7 - Move related code parts into corresponding block.
• 6539d223 - Prevent repeated calls with temporary.
• c2d903db - Merge duplicated calls.
• 9b8ac929 - Add distributed grid capability for parallel output.
• f5604bef - Support key queries for distributed grids.
• 2a318d25 - Add distributed curvilinear grids to the test suite.
• aeaa2da0 - Add configure-time check if Fortran (77) is to be tested for cfortran.h.
• df1f35be - Improve cfortran.h test when F77 and FC are used.
• cde154b0 - Add caching of redists for NetCDF writes.
• d4c7950b - Add other NetCDF variants in PIO code.
• e85ec100 - Fix incorrect checksumming code in 2D decomposed PIO test.
• a157fb95 - Use pointer to constant string instead of string copy.
• cacc4c35 - Remove old house-keeping data.
• 7872d2c8 - Replace redundant count function.
• bb30273a - Remove old institution resource house-keeping data.
• de7ae4a5 - Remove data dependency.
• 1721be87 - Encapsulate data structure in only user.
• edf8e68b - Fix doc-string.
• 5469d07e - Use size_t for address offset loop counter.
• d70cc021 - Clarify assignment.
• 627f4a83 - Eliminate failure mode where streamDefTimestep is called before streamDefVlist.
• 783117e2 - Clarify search loop.
• f29f203e - Fix string data copied twice.
• e2f8eb2b - Add testing of relative time axis for NetCDF.
• 8b9a8653 - Simplify error handling.
• fd7c9025 - Make necessary changes to open CDF-1/2 files in parallel mode when possible.
• f373cd8d - Remove incorrect declaration.
• 6bbcd139 - Improve documentation.
• c7e7d172 - Add attribute to clarify potential control flow.
• 1c988fcf - Switch size variable from int to size_t.
• 6c17e4c6 - Remove unnecessary cast.
• 6563ddb1 - Check distributed grid chunk ranges.
• fab444e8 - Add I/O decompositions with non-perfect distributions.
• 0fd997e7 - Eliminate unused data structure.
• 57d1aae3 - Fix missing documentation of pre-encode aggregation buffer configuration.
• 7339aba4 - Use pwrite instead of fseek/fwrite when available.
• 4b60bac5 - Revise pio fpguard code.
• 91a61785 - Clarify pio dbuffer implementation.
• ee377b73 - Remove unused tag.
• 9dcd4185 - Rename and move struct and functions for better cohesion.
• 04666c89 - Consolidate collector side naming.
• 633be51a - Make whole configuration object available in PIO.
• 14657811 - Eliminate unnecessary precaution and reduce number of print calls.
• 2a254f42 - Do not transfer zero terminator byte.
• 3da752a5 - Rename duplicate header inclusion guard macro.
• 39aa27d5 - Sort header alphabetically.
• c2689469 - Move struct declaration to earliest use.
• c625d947 - Shorten expression.
• 73af33c2 - Create stripes on destination side, too.
• b1712f1d - Add cache of destination index lists.
• c82a503e - Add xmap cache data structure.
• cf0a3c86 - Put prerequisites for Xmap caching in place.
• 7a010656 - Add caching of Xmaps to CDI PIO collector component.
• 7dc91464 - Minor improvements to async single writer output.
• e03fc8c6 - Remove redundant struct member.
• 8e469a50 - Convert dynamic size to configuration option.
• 722259af - Improve error handling of environment variable parser.
• 09ba07cc - Align data structure print-out with actual contents.
• 8572cc40 - Use temporary.
• 7d4811b6 - Eliminate unused buffer.
• 7ce0042b - Make AIO queue depth configurable.
• 85aa2164 - Use struct tag instead of typedef.
• f5e2c461 - Fix missing header include.
• d77e4e18 - Eliminate unused struct member.
• bd7f8aa9 - Use array instead of linked list.
• 2c97c02f - Assert non-aliasing property.
• 312480eb - Streamline disk buffer handling.
• 9291bdd4 - Improve error handling diagnostics.
• 2ecc70c0 - Beautify error diagnostic.
• 85137feb - Only execute query when needed.
• a0a5304c - Remove call overhead in non-edge case.
• d0d3cfef - Add handling for MPI functions filling array of statuses.
• 0d5bcc43 - Improve request handling for more communications overlap.
• d8ca677b - Move trivial routines to header for inlining.
• 1eff9611 - Move conditionally used variable into conditionally compiled block.
• f587887b - Rename type.
• cf02a316 - Convert variable length array struct member to pointer.
• ec849fb4 - Convert PIO STDIO file writer to stable file IDs.
• 0b4aae51 - Eliminate redundant file truncation.
• 20dbff92 - Rename struct.
• 818785b8 - Replace variable length array member with pointer.
• 151e2005 - Switch AIO funnel to reliable IDs.
• df0e5259 - Rename struct.
• 7d20dee5 - Switch to staged dynamic allocation of struct.
• eaccf6db - Improve error diagnostics.
• b370bed6 - Change variable length array struct member to pointer.
• 308ec061 - Use reliable file IDs in PIO MPI_File_write_at_all mode.
• a0d44c4a - Rename type and use struct tag instead of typedef.
• 4db25cd8 - Switch struct component to separate allocation.
• f67a79b2 - Make pointer from VLA struct member.
• c63561ac - Use reliable file IDs in PIO mode MPI_File_write_at with reblocking.
• d87e9fbd - Rename struct.
• 218ed261 - Change VLA member to pointer.
• e2bf84ed - Switch PIO MPI_File_write_ordered backend to reliable file IDs.
• 90048435 - Rename struct.
• cbac7810 - Change VLA struct member to separate allocation via pointer.
• 4e5866a7 - Eliminate superfluous struct member.
• b48df30c - Eliminate unneeded struct member.
• 0fb9e996 - Change PIO MPI shared file pointer method to reliable file IDs.
• 8a0e4e14 - Improve diagnostics.
• b279b2a8 - Convert VLA struct member to pointer.
• e2395682 - Use reliable file IDs in client part of PIO shared file pointer pwrite method.
• f552abc0 - Improve guard handling of file IDs.
• c92bf337 - Delete no longer used source file and corresponding header entries.
• 4183d00f - Fix representation of file offset.
• 529edc34 - Remove unnecessary branch.
• 371c7490 - Change funnel-like output methods to no longer rely on MPI_barrier.
• bdaf30af - Switch funnel request handling from MPI_Waitany to MPI_Waitsome.
• 68ada6d0 - Rename disk buffer functions.
• 86ab0608 - Remove pointless indirection.
• 58b8e988 - Remove effectively unused return value.
• 17c1a993 - Eliminate redundant storing of stream ID.
• 7e153dd5 - Add float variant of chunk write operation.
• acd73d42 - Fix loop-carried dependency.
• 49706a8b - Extend CDI-PIO writing of data to float.
• c94f1327 - Document CDI-PIO data write functions.
• 0b186500 - Document -qtaxis-type option of test program.
• aa2273b2 - Document -qtaxis-unit option of test program.
• 569c73de - Add option to change datatype of written files.
• c4f7b1f0 - Extract calendar test 1.
• 35dd5d1b - Move trivial function to header for inlining.
• dea96288 - Replace loops with O(1) computation.
• e0d3d6a5 - Prevent unnecessary fill value writing.
• 85a3edba - Use symbolic instead of numeric constants.
• 81397fb6 - Use symbolic instead of numeric constants.
• d460553b - Rename constant.
• fabbe1ee - Use symbolic instead of numeric constants.
• 62fbcf43 - Rename constant.
• 185fb3d9 - Use symbolic instead of numeric constants.
• f21280b4 - Use symbolic constants instead of counter.
• 863e9f85 - Eliminate effectively unused member.
• d035aac0 - Include taxis fc_period in check and pack/unpack.
• 6db37a53 - Remove pointless indirection.
• d0b7c3a8 - Merge partial initializations.
• a2aea8f0 - Replace arithmetic.
• 6980c28d - Eliminate no longer needed error check.
• 04ed55f7 - Disentangle and shorten test.
• 93932a6b - Fix redundant table lookup.
• 939c065e - Optimize layout of var_t struct.
• 10b69d5f - Use bool for two-state value.
• caa2dd47 - Perform initializations in memory-order of members.
• ee3865a4 - Use symbolic instead of numeric constants.
• 25de15c4 - Get rid of superfluous component.
• 1baca284 - Improve temporal locality and stack efficiency.
• 0c73813f - Convert two-state variable to bool.
• acb41dd4 - Switch xyz member to compressed representation.
• ad0b9394 - Adapt free F2003 interface to more closely match YAXT.
• f3a20d9a - Fix non-idiomatic loop-control.
• c28b538f - Use appropriate integer conversion.
• 4f82c0ef - Save redundant query.
• 97f2edd4 - Extend callbacks to streamClose Post action.
• e5fb732f - Align interface of unpacking functions.
• c590e4ea - Improve handling of empty comments in F77 interface generator.
• 8353b713 - Make verbatim paren match in F77 interface generator explicit.
• 2a22f1b2 - Enable handling of bool parameters and return values.
• caa39798 - Add optional post-processing hook to resource unpacking loop.
• ea84aa2f - Do not clear RMA buffers completely.
• 7a85f31a - Remove todo notice.
• 612cbbff - Add callback for server-side actions following a write batch.
• b8e84592 - wip: use distributed grids in local mode on clients.
• 8fbc425b - Fix project URL.
• 1baa07ff - Add detection of cfortran.h flag for G95.
• a4190711 - Mandate FC/F77 to be working when not explicitly disabled.
• 73c54830 - Also do configure-time compilations with libtool.
• 30171603 - Remove unused return value.
• aef28db0 - Remove unused function commInqRootGlob.
• 3fbd7df5 - Remove unused function commInqNProcsColl.
• 7a60da83 - Merge places of namespace creation in CDI-PIO initialization.
• 04c46929 - Remove unused function cdiPioInqCollectorOfClient.
• d728ba98 - Make PIO namespace available for more parts of initialization.
• 4eabc622 - Add virtualization to CDI-PIO comm scheme.
• f29b6324 - Make Intercomm query part of public API.
• f0a6edc6 - Use C99 bool for 2-state field.
• eebe8ed4 - Streamline wrapper generator string printing.
• 8ff801dd - Interface generator: extract function to create wrapper.
• 36d2b619 - Remove special case.
• 03d91c9f - Add doc-string.
• 32aacadd - Fix quadratic append.
• cfbb63f7 - Adjust for YAXT split of library into C and Fortran part.
• a9ab2b61 - Add flushing of unwritten data at streamClose for PIO modes.
• 2af6d54d - Add support for unstructured grids to PIO part.
• faa459b9 - Add const annotation.
• 357ebf49 - Rename function according to actual query.
• 68785f2f - Fix missing buffer size update.
• 34ecc346 - Use single instead of per-file buffer.
• 7374579f - Adapt distributed grids to stream-dependent netCDF parallelization.
• 06523733 - Move variables up in scope.
• 73b97e8c - Shorten block with existing variable.
• b02247d1 - Compute reductions for X- and Y-axis immediately.
• dec5bad2 - Do not activate server-side remote access unconditionally.
• 1b0dca69 - Enable parallel access for pnetcdf mode.
• 14ade388 - Consolidate initialization.
• 6f781521 - Improve parallel access implementation error handling.
• afdcb95d - Explicitly include header.
• 3fa12d46 - Move documentation comment to the line it refers to.
• 8495b813 - Use size_t for nmiss as long as possible.
• 75cfd660 - Use loop to shorten code.
• b0645a11 - Check that variables can be processed by YAXT.
• 801109e4 - Include stdbool.h in source declaring bool variables.
• b98523c9 - Add temporary work-around for large variables.
• e37e14e9 - Rename index list cache.
• 7240cfb8 - Use reliable PRNG instead of plain random.
• 0c2cf532 - Use AS_FOR instead of for loop.
• c3d0448c - Annotate room for improvement.
• 95e441cb - Add missing documentation.
• e209230c - Add alternative sequence of RMA and output.
• 546be1f8 - Add test for non-batched RMA mode.
• 722f21e1 - Rename index list cache.
• 8c7a5624 - Also rename index list cache interface functions.
• 849849c8 - Consolidate vlistDestroy error checking.
• 0b3e4eb1 - Make cdiVlistDestroy_ overridable.
• e8620df7 - Add caching for PIO chunk writes.
• c80c0c1e - Clear only first header in RMA buffer.
• 975fdf5b - Extract common functionality.
• d9e418d8 - Remove unused declaration.
• ae21891c - Add capability to specify decomposition in advance.
• d523d176 - Let ruby interface generator remove comments itself.
• 1e53a2d9 - Extend tests for advance decomposition setting
• 84f33ed4 - Split function.
• 4e913ee9 - Extract function.
• 941a99ca - Extract function.
• 811738cd - Remove unused functions.
• 48fa1124 - Extend preset decomposition with conversion.
• 6d4858ca - Extract function.
• 027970f4 - Extract function.
• 43486470 - Only perform computation and reallocation when actually needed.
• 4fc9344e - Extract function.
• 4a806c20 - Remove unused parameter.
• 75a8911f - Add missing const specifier.
• 27763d43 - Clarify function.
• 55b892bc - Add injection of xmaps into cache.
• bc876d6c - Refine NC_PNETCDF work-around.
• c7b15fcd - Add work-around for Bull X MPI bug.
• 30051478 - Spread test code cell centers more evenly.
• a07bb23f - Also account for eccodes when testing grib_api.
• e1669729 - Fix bug when only part of stream is written.
• 6570cbb4 - Fix incorrect counting of open files.
• 9641bf64 - Clarify responsibility of ID mapping cleanup.
• 4159e9cf - Add assertion to prevent double free problems.
• dd138828 - Regularly check the resource handle list.
• dcacd9bb - Fix resource leak.
• fe7f203b - Make zero params explicit.
• 508ee6bc - Clarify const property.
• e31ae8e4 - Fix for gcc warning.
• c4e95875 - Annotate that no changes happen through pointer component.
• 71b9e2c0 - Add client write call that uses advance declaration of decomposition.
• 24fade1b - Fix alignment of buffered float data.
• 9d5451b6 - Fix estimation of pack size of time axis.
• bc118777 - Add work-around for defect in OpenMPI 2.0.2.
• f63952e2 - Autotools files update.
• 9ed4ccf7 - Simplify Makefile conditionals.
• 14305f45 - Choose different type to get rid of compiler warning.
• c749ecc0 - Extend conditionally compiled region.
• 33549b90 - Change fallthrough comment so it is also recognized by newer GCC versions.
• a11f9492 - Add work-around for MPICH 3.3 defect.
• cd587df8 - Fix missing cleanup in parallel mode.
• 542c1ad9 - Fix missing free.
• c4b5708a - Move label so no variable scopy is jumped into.
• 15679240 - Prevent duplicate write of string NUL terminator.
• e8a278a2 - Clarify setting of institute ID.
• f8e8f858 - Extend buffer to prevent string truncation.
• a72a2901 - Fix condestruct pattern.
• 1c590d87 - Fix compiler warnings.
• 883bc07a - Use unsigned arithmetic to prepare for size_t switch.
• 602e14ca - Annotate intentional fall-through.
• 84534254 - Add another work-around for CCE.
• 8dce932a - Update to libtool 2.4.6.
• 76fa76d7 - Add work-around for Cray ftn+CC.
• 2b9ecc1e - Replace redundant library specifications.
• 01972c7c - Improve test for netcdf_par.h to also work without wrapper compiler.
• ed22e91e - update/replace a subset of m4 macros and Libtool patches (copy from the YAXT project)
• a552b08c - add a patch for ltmain.sh reflecting modifications in CDI-PIO repo
• 3cdb49f2 - update .gitignore
• abe79540 - add missing header file in src/Makefile.am
• 4de31737 - update condition for the dependency on librt
• 06b90074 - Fail the configuration when eccodes is requested but grib_api.h is not found.
• 912020db - Include netcdf_XXX.h files after including netcdf.h in the configure script.
• 227be1c1 - Unify YAXT location interface and check whether the library is really...
• 364464e3 - Refactoring for YAXT location logic in the configure script.
• 278a9dd3 - Unify PPM CORE location interface and check whether the library is really...
• cb6af166 - Unify the check for pnetcdf support.
• 5ad5a402 - Refactor CDI-PIO-related checks.
• 8dbaf838 - Switch from 'mpif.h' header to 'mpi' module in the code.
• 27923cea - Update cfortran-related m4 macros: make them work when either FC or F77 is disabled.
• 37824459 - Fix the nag pthread patch.
• 11142889 - Revert "test_cdf_const: deactivated (depends on app/cdi)"
• 9cabaf97 - Rewrite test scripts in the configure-like portable way.
• f523ddf3 - Skip 'pio_cksum_writer' when '--disable-grib'.
• b01f523e - Do not skip 'pio_cksum_cdf' when '--disable-grib'.
• e8821d8e - Do not switch to a subdirectory when running 'test_cdf_const'.
• aec448f7 - Avoid random names for the files generated by the tests.
• 43055715 - Do not remove temporary and log files generated by the tests.
• 5b8cec09 - Use '@top_builddir@' instead of '@abs_top_builddir@' and '..' in the test scripts.
• 88d1cb7c - Use '@srcdir@' instead of '${srcdir}' in the test scripts.
• 0f8e1c60 - Make 'pio_write_run' responsible for skipping tests for disabled file formats.
• 113711f3 - Make 'pio_write_run' responsible for setting the format argument '-f'.
• 36529217 - Copy format-related logic from 'pio_write_run' to 'pio_write_deco2d_run'.
• 3f50b4de - Skip MPI tests if MPI_LAUNCH is not available.
• 837aaa83 - Reformat the makefiles.
• 0af7c9c5 - Remove redundant EXTRA_DIST from the makefiles.
• 5fdd2cde - Remove redundant variables from the makefiles.
• 2ac9563e - Do not remove arbitrary files ending with '~' when 'make clean'.
• d4176d80 - Use generic Automake variable to declare a Fortran module to be installed to 'includedir'.
• eb6efeec - Treat the pkg-config files in a more laconical way.
• 0721d8e1 - Treat the cmake files in a more laconical way.
• 643797e1 - Optionally silent a few more rules.
• 66438be4 - Rename a variable in the makefile.
• 8cc51d04 - Update clean rules in the 'src' directory.
• 5c414e5c - Remove what seem to be redundancies for clean-related rules in the makefiles.
• 70f37379 - Remove suspicious install hooks from the makefile.
• 23908742 - Apply the '-all-static' flag more consistently.
• 19580866 - Treat the cfortran source files consistently.
• 61fe7ec5 - Introduce common LDADD in the 'app'.
• 0e7f2953 - Make 'pio' a SUBDIR of 'examples'.
• bda0cdd9 - Add missing header to the list of source files of 'libcdiresunpack'.
• a88fee99 - Declare a missing linker dependency for 'libcdiresunpack'.
• 61a96052 - Add missing 'crayftnpreproc-wrapper' to EXTRA_DIST.
• c064aae5 - Temporarily set USE_MPI macro in the makefile to guarantee it is accounted for...
• b1b8ce10 - Rewrite the configure script.
• 9aaed8d8 - Update the pkg-config files.
• faef062e - Update the post-configure message.
• e61fdf25 - Build 'libcdiresunpack' when 'make check'.
• 68bc8ca6 - Run the 'test_resource_copy_mpi_run' script in any configuration.
• d1a02209 - Refactor LDADD variables in the makefiles.
• 1deac613 - Refactor the dependency of test programs on the 'mo_cdi' module file.
• b98a7e09 - Update clean rules to all identified output files of the tests and examples.
• f613bd71 - Avoid building Fortran programs with CPPFLAGS.
• 62b21151 - Refactor compiler flags in the makefiles.
• f8b7474e - Build examples (that are not needed for testing) when 'make all'.
• 042d5e27 - Do not build example 'collectData2003' when the Fortran interface of YAXT is missing.
• 42d4f24a - Update .gitignore.
• 1d3ba1d2 - Temporarily remove the suspicious 'test_grib'.
• 8505fd33 - Reintroduce support for Ruby and Python interfaces.
• 1bcfbc7f - Remove obsolete Python interface files.
• 8ab926e1 - Make sure check programs can produce/consume i/o files with non-conflicting names.
• 36d8e75c - Switch to the parallel test harness.
• 3539c06d - Split 'pio_cksum_cdf' into three separate tests.
• c59d877e - Rename 'test_chunk_cksum' to 'test_cksum_nc_chunk' for consistency.
• a500a82c - Remove files that seem to be obsolete.
• 3861b84f - Do not use PPM in check programs if USE_MPI is not defined.
• 8a335ea0 - Build (and run) serial versions of the examples (and tests) also when MPI is enabled.
• 579469c7 - Optionally (and by default) run MPI tests one by one.
• 365f2a1c - Optionally (and by default) build check programs when 'make all'.
• c7a00f3b - Optionally skip building examples.
• 79916c0a - Copy more libtool patches from YAXT.
• f5f9d48f - Copy more m4 macros from YAXT.
• 7380861c - Address a peculiarity of ACX_TLS_XLC_RETRY in the configure script.
• c2dfa06f - Extract distributed array test to a separate script.
• 01c794b2 - Revisit libtool usage when compiling programs.
• 2024dd62 - Account for the case when YAXT and PPM are provided as '/path/to/lib/libname.a'.
• 85426474 - Introduce a workaround for PGI 20 in 'pio_server.c'.
• 7f8192de - Clean-up after ACX_FC_CHECK_STRPTR_CONVERT.
• 7b7b927b - Run ACX_SL_FC_CHECK_MOD_PATH_FLAG in the configure script also when '--disable-iso-c-interface'.
• 7cc7e583 - Add configure option '--enable-ppm-dist-array'.
• d30e438c - Get rid of 'rm: conftest.dSYM: is a directory when configuring on MacOS.
• 0c06ccc3 - Remove declaration of the 'BUILD_C_INCLUDE' and 'BUILD_FC_INCLUDE' precious...
• 20f45426 - Generate source files for parallel versions of the programs.
• 404e4c69 - Drop ONLY clause for MPI Fortran module.
• eea6c09f - Rewrite pkg-config and cmake rules as Automake would generate them.
• f51ebe51 - Chain commands with '&&' in the recipe for 'cdilib.c'.
• eb9b38ca - Add work-around for MPICH 3.3 defect.
• 17c78753 - Fix condestruct pattern.
• f1e04459 - Pass large struct by reference.
• b986d592 - Make test file name prefix configurable.
• 80c61192 - Autotools macro update.
• a385c635 - Make further information of the build available via pkg-config.
• a51a0a4c - Annotate unused argument.
• cb06947a - Eliminate unused temporaries.
• 2cede934 - Add new work-around for new fail dimension of PGI 20.
• b775a78f - Update autotooling.
• 805e510c - Use pio_write_run also for 2d-decomposed test.
• f321be0f - Use standard AC_RUN_IFELSE.
• 85c8dbce - Introduce cache variables 'acx_cv_fc_works' and 'acx_cv_f77_works'.
• 680a90c0 - Add 'acx_fortran_include_flag.m4' and use it in the configure script.
• 7a20969e - Introduce cache variable 'acx_cv_cxx_works'.
• fe0d1aeb - Introduce automake conditional BUILD_FC_PROGRAMS.
• ed0fd4f3 - Run ACX_FORTRAN_INCLUDE_FLAG for Fortran 77 (F77).
• 3b938bf7 - Avoid SIGPIPE when requesting the compilers for their versions at configure-time.
• ad45448e - Remove suffix '_serial' for all tests and examples.
• f9f5db2b - Concentrate all PIO test logic in 'pio_write_run.in' (again).
• 1d6ac92f - Delete more obsolete files.
• cc12fd46 - Fix skip conditions in 'pio_write_run.in'.
• 908aa780 - Skip 'test_cksum_grib2' when '--disable-grib'.
• e0cf6105 - Skip 'test_resource_copy' when '--disable-grib'.
• b225216e - Restore 'test_grib'.
• 9e39be4b - Generate source file for 'test_resource_copy_mpi'.
• 280c5dd2 - Refactor the skipping conditions in the tests.
• d1667f9a - Do not fail 'test_cdf_const' on a warning (as it was before).
• 593e3314 - Update ACX_CHECK_CFORTRAN: make it work when FC is disabled.
• 9612c9fc - Fix building and cleaning examples when '--without-example-programs'.
• b7e3ccff - Minimize the size of the generate source files of the parallel tests and examples.
• f71e8ec7 - Replace '_parallel' with '.parallel' in examples and tests.
• 9b110ab0 - Rename 'test_resource_copy_mpi_run' to 'test_resource_copy_run.parallel' for consistency.
• c7a8c17b - Introduce extension '.run' for tests and use it consistently.
• 9720d310 - Make 'test_cksum_base' responsible for skipping tests.
• c5d9ad37 - Update 'pio_write.run' before running tests that source it.
• 6646f955 - Polish the parallel cksum tests.
• 27b90fd2 - Update 'pio_write_dist_array.run' before running tests that source it.
• e455e109 - Avoid potentially unportable parameter substitution in the test scripts.
• 81d8657e - Do not rely on libtool overlinking.
• 17ca1d19 - Optionally download 'config.sub' and 'config.guess' with 'autogen.sh'.
• 4e0d59ff - Make './autogen.sh' switch to the top source directory.
• 73763262 - Do not overlink explicitly even though Libtool does it implicitly.
• 1aca9428 - Modify the naming scheme for libtool patches.
• 5c9d548d - Add a Libtool patch from Debian that prevents implicit overlinking.
• 61689cea - Generate SOURCE lists of the parallel versions of tests/examples automatically.
• 9d1b3462 - Append header search path flags to CPPFLAGS instead of CFLAGS when possible.
• 18b103d2 - Make configure-time OpenMP checks more robust.
• 1c4f3abd - Disable Fortran 77 interface by default for consistency.
• b6589b45 - Set compiler flags instead of command overriding to address unwanted double compilations.
• 2f8184b5 - Build and run make_fint with libtool.
• 1eea5ee0 - Create cdilib.c in the build directory.
• 7b15e4d7 - Rename and extend the Debian libtool overlinking patch.
• 703d5d4e - Keep and apply libtool patches like in YAXT.
• 79ab5abc - Fix change in config.status substitutions.
• 5463ed83 - Change cfortran default to yes.
• 7ca8bd59 - Align interface with declaration in cdi_uuid.h.
• 5be172f7 - Expand f2003 interface generator to cover partially specified array dimensions.
• 62a68b4d - Align interface with actual use.
• 46bde106 - Replace interface not exactly matching the implementation.
• a59e99b4 - Add pragma to prevent buggy gcc 11 diagnostic message.
• 797fe23c - Fix missing explicit integer to floating-point conversion.
• ff08ca32 - Convert patch of libtool.m4 into autoconf pattern substitution.
• 2241ecea - Merge initialization and declaration.
• 22c33bc1 - Switch lt CXX builds from explicit to implicit library dependencies.
• 505aa677 - Fix missing free.
• 03d6999a - Fix missing wrapper conversion.
• b9ac3385 - Reduce code divergence.
• fbba02ae - Improve CDI-PIO documentation.
• 423d6719 - Shorten array initializations.
• ebd5ee93 - Improve flexibility of cdfDefTimeValue for injection of bounds.
• 64f86561 - Clarify documentation.
• 9a469150 - Replace plain macro with function-like.
• 3452f699 - Prevent overwrites when unnecessary.
• 78bd5985 - Add back seam for PIO output directory.
• 1db3d714 - Prepare to make test program parameter ntfiles adjustable.
• a93dc9e1 - Configure with --enable-cf-interface=auto by default.
• 406588bf - Fix incorrect preprocessor conditional and variable scope.
• 4dff1e2f - Port work-around for Intel compiler.
• 3d681943 - Improve error message.
• cde67e1f - Fix misleading description of configure options.
• ca34360d - Do not apply Debian libtool overlinking patch.
• b01dbc1c - Update script language bindings.
• 2505df66 - Use correct macro for conditional compilation.
• e4f6a43f - Address NVHPC compiler warnings.
• 8be844a9 - Update .gitignore.
• 301e4cf0 - Update CI scripts.
• d97b964d - Add script for release branch preparation.

Compare with previous version

added 2 commits

• 3d930254 - Update CI scripts.
• f02b8b1e - Add script for release branch preparation.

Compare with previous version

So, I've gone through the rebasing exercise again. The pipeline is green but, of course, I have no idea how many bugs those ~500 commits have introduced. However, I don't think I will survive doing this one more time, therefore, I am waiting for https://gitlab.dkrz.de/icon/icon-mpim/-/merge_requests/27 to get sufficiently green (there are known problems with the time limits on Daint) and merging this today in the afternoon. Anyway, we are talking about the develop branch and, I guess, nothing stops anybody from testing this more thoroughly and fixing possible bugs later, before a release. The best thing is that we finally have only one main development branch once this is merged.

added 2 commits

• 7aa6b506 - Update CI scripts.
• d8612e57 - Add script for release branch preparation.

Compare with previous version

marked this merge request as ready

merged