mistral: missing shared libraries for cdo

Hi

When I use awicm-CMIP6 (release) on mistral with the esm-environment version 2626085, the following error occurs in the log/awicm/expname_date_expid_esmlog.dump:

Currently Loaded Modulefiles:
  1) gcc/4.8.2                  5) nco/4.7.5-gcc64
  2) jdk/1.8.0_20               6) intelmpi/2017.0.098
  3) intel/18.0.0               7) netcdf_c/4.3.2-gcc48
  4) cdo/1.9.5-magicsxx-gcc64   8) cmake/3.5.2
...
cdo: /sw/rhel6-x64/gcc/gcc-4.8.2/lib64/libstdc++.so.6: version `CXXABI_1.3.8' not found (required by cdo)
cdo: /sw/rhel6-x64/gcc/gcc-4.8.2/lib64/libstdc++.so.6: version `GLIBCXX_3.4.21' not found (required by cdo)
cdo: /sw/rhel6-x64/gcc/gcc-4.8.2/lib64/libstdc++.so.6: version `CXXABI_1.3.9' not found (required by cdo)
cdo: /sw/rhel6-x64/gcc/gcc-4.8.2/lib64/libstdc++.so.6: version `GLIBCXX_3.4.20' not found (required by cdo)
cdo: /sw/rhel6-x64/gcc/gcc-4.8.2/lib64/libgomp.so.1: version `GOMP_4.0' not found (required by cdo)
cdo: /sw/rhel6-x64/gcc/gcc-4.8.2/lib64/libgomp.so.1: version `OMP_4.0' not found (required by cdo)

Apparently, some shared library dependencies are missing for the new cdo version 1.9.5 and it cannot be executed. For example,

ldd /sw/rhel6-x64/cdo/cdo-1.9.5-magicsxx-gcc64/bin/cdo | grep gcc-
        libstdc++.so.6 => /sw/rhel6-x64/gcc/gcc-6.4.0/lib64/libstdc++.so.6 (0x00007f41abfa6000)
        libgomp.so.1 => /sw/rhel6-x64/gcc/gcc-6.4.0/lib64/libgomp.so.1 (0x00007f41abd79000)
        libgcc_s.so.1 => /sw/rhel6-x64/gcc/gcc-6.4.0/lib64/libgcc_s.so.1 (0x00007f4190a2f000)
        libgfortran.so.3 => /sw/rhel6-x64/gcc/gcc-4.8.2/lib64/libgfortran.so.3 (0x00007f418eca4000)
        libquadmath.so.0 => /sw/rhel6-x64/gcc/gcc-4.8.2/lib64/libquadmath.so.0 (0x00007f418ea69000)

shows, that different gcc versions were used for libraries cdo depends on (here gcc 6.4.0 and 4.8.2). So in this case, adding gcc version 6.4.0 to LD_LIBRARY_PATH somewhere in the esm-environment/mistral.dkrz.de solves the problem.

For doing this in a general way, i.e. not writing specific version numbers into esm-environment/mistral.dkrz.de, the following lines may work instead:

    echo "======== workaround ========"
    echo "   esm-environment/mistral.dkrz.de:"
    echo "   Load eventually missing shared library dependencies for cdo with LD_LIBRARY_PATH ..."
    echo "      execute 'strings -a `which cdo` | grep 'GCC: (' ..."
    strings -a `which cdo` | grep 'GCC: (' | while read -r strings_output ; do
        echo "         $strings_output"
        version=`echo $strings_output | awk -F ' ' '{print $3}'`
        echo "         add /sw/rhel6-x64/gcc/gcc-$version/lib64 to LD_LIBRARY_PATH ..."
        export LD_LIBRARY_PATH=/sw/rhel6-x64/gcc/gcc-$version/lib64:$LD_LIBRARY_PATH
    done
    echo "LD_LIBRARY_PATH: $LD_LIBRARY_PATH"
    echo "============================"

yields

======== workaround ========
   esm-environment/mistral.dkrz.de:
   Load eventually missing shared library dependencies for cdo with LD_LIBRARY_PATH ...
   LD_LIBRARY_PATH: /sw/rhel6-x64/intel/intel-18.0.0/mkl/lib/intel64:/sw/rhel6-x64/intel/intel-18.0.0/lib/intel64:/sw/rhel6-x64/gcc/gcc-4.8.2/lib64:
      execute 'strings -a /sw/rhel6-x64/cdo/cdo-1.9.5-magicsxx-gcc64/bin/cdo | grep 'GCC: (' ...
         GCC: (GNU) 4.4.7 20120313 (Red Hat 4.4.7-18)
         add /sw/rhel6-x64/gcc/gcc-4.4.7/lib64 to LD_LIBRARY_PATH ...
         GCC: (GNU) 6.4.0
         add /sw/rhel6-x64/gcc/gcc-6.4.0/lib64 to LD_LIBRARY_PATH ...
   LD_LIBRARY_PATH: /sw/rhel6-x64/gcc/gcc-6.4.0/lib64:/sw/rhel6-x64/gcc/gcc-4.4.7/lib64:/sw/rhel6-x64/intel/intel-18.0.0/mkl/lib/intel64:/sw/rhel6-x6
4/intel/intel-18.0.0/lib/intel64:/sw/rhel6-x64/gcc/gcc-4.8.2/lib64:
============================

in the log/awicm/expname_expdate_jobid_esmlog.dump and the errors disappear.

I dont know if this is a nice solution from a programers perspective - certainly not.
Note that using ldd would maybe the first choice for checking missing shared library dependencies, however, ldd executes the program to be checked (cdo in this case) by itself, which is not possible because of the same error as above.

Chris