yaxt GPU support
Currently, we rely on MPI to provide the support for user data that is in GPU memory. yaxt just passes data pointers and the appropriate MPI-datatypes to MPI. Internally, MPI determines the location of the data and apply the datatype accordingly. However, especially OpenMPI is terribly bad, when the data is in GPU memory und the MPI-datatype is not just a trivial vector. It will use individual cuMemcopy
for the packing/unpacking of each contiguous section of memory. When comparing mo_communication_orig
and mo_communication_yaxt
(from ICON) this can make a difference of 100-1000x (mo_communication_orig
does the packing/unpacking manually in the user code using OpenACC
and only passes a pointer to one contiguous section of memory to MPI).
Since, it does not seem like OpenMPI will fix this in the near future. There was the plan to overcome this issue directly in yaxt. That is why I am writing a new exchanger based on xt_exchanger_irecv_isend
that will do the packing/unpacking directly without the use of MPI_Pack
/MPI_Unpack
. Once this works for data in CPU memory, I am planing to add support for GPU memory by porting the packing/unpacking routines to GPU-code using OpenACC
.
@jahns @m300488 @k202076 How should yaxt handle this explicit support for GPUs?
- check for
OpenACC
-support inconfigure
--enable-gpu-support
-DHAVE_GPU_SUPPORT
- (de)activate GPU support by new exchanger or deactivate exchanger completely if there is no
OpenACC
-support?