JSC-Jupiter: Deadlock in the output coupling

There is a deadlock when running a hybrid job on the JSC-Jupiter machine with ICON-coupled model on GPU-CPU setup with Hiopy (YAC-ouput coupling).

The deadlock occurs when the number of variables written increases.

The cuda-gdb stacktrace shows the following:

#9  0x000040000f09c444 in xt_mpi_test_some (num_req=0x40039a5aa368, req=0x40039a5aa378, ops_completed=0x40039a5aa840, comm=0xe82301e0)
    at ./icon-mpim/externals/yaxt/src/xt_mpi.c:429
#10 0x000040000f0cae64 in xt_request_msgs_test (request=0x40039a5aa360)
    at ./icon-mpim/externals/yaxt/src/xt_request_msgs.c:183
#11 0x000040000f0a5eec in xt_request_test (request=0x13d67c6f0, flag=0xffffc4d9b66c)
    at ./icon-mpim/externals/yaxt/src/xt_request.c:71
#12 0x000040000f03fc0c in yac_interpolation_exchange_get_test (exchange=0x13d67c6d0, routine_name=<optimized out>)
    at ./icon-mpim/externals/yac/src/core/interpolation/interpolation_exchange.c:247
#13 0x000040000f0457d8 in yac_interp_operator_sum_mvp_at_src_execute_get_test (interp=<optimized out>)
    at ./icon-mpim/externals/yac/src/core/interpolation/operators/interp_operator_sum_mvp_at_src.c:635
#14 0x000040000f03f1b0 in yac_interpolation_execute_get_test (interp=0x13ba08540)
    at ./icon-mpim/externals/yac/src/core/interpolation/interpolation.c:348
#15 0x000040000efbb458 in yac_ctest (field_id=<optimized out>, flag=0xffffc4d9bf0c)
    at ./icon-mpim/externals/yac/src/mci/yac.c:3228
#16 0x000040000efa9654 in is_ready (this=0xb1beff0) at ./hiopy/src/field.cpp:73
#17 0x000040000efa7dc0 in run (this=0x4f3a350) at ./hiopy/src/coyoteenv.cpp:125
#18 0x000040000ef32210 in call (f=0x4467f38)

ICON has completed the send from its side. Hiopy seems to not be able to receive the timing of the output between the processes goes out of sync due to some unknown reason.

The deadlock does not occur when the number of variables are low.

The following flags seem to have no impact on the issue except for slowing down the simulation:

# export YAC_YAXT_EXCHANGER=irecv_isend
# export XT_CONFIG_DEFAULT_EXCHANGE_METHOD=irecv_isend
# export OMPI_MCA_coll_tuned_use_dynamic_rules=1
# export OMPI_MCA_coll_hcoll_enable=0
# export UCX_RNDV_THRESH=0 #default was intra:auto,inter:auto
# export UCX_RNDV_SCHEME=get_zcopy
# export UCX_ZCOPY_THRESH=0
# export UCX_IB_GPU_DIRECT_RDMA=no

Wanted to bring this to your attention as I am out of ideas to try at this point.

Would you be able to provide some hints to deal with this? Nils already helped out with a few hints which helped us exclude one or the other thing. The issue still persists.

What we do know so far is that the problem is not due to any of the following:

  1. Aggregations
  2. Variety of aggregations asked per variable
  3. Spatial coarsenings with NNN interp

Here is a file with defined couplings (debug output of coupling.yaml): couplings_defined_230982.yaml

@k202160 @k202077

Imprint | Privacy Policy