Skip to content
Snippets Groups Projects

Draft: Attempt at fuller exchanger interface

Open Draft: Attempt at fuller exchanger interface
new-exchanger into master
4 unresolved threads
Open Thomas Jahns requested to merge new-exchanger into master
4 unresolved threads

This branch contains my attempt to enable the following with the exchangers: Directly using per-rank basic MPI datatypes (and count) with the buffers (and an added displacement), such that

  • the packing exchangers can be modified to use the basic exchangers instead of hard-coded point-to-point message passing routines,
  • make the exchangers usable in cases where the MPI datatypes of the built-in redists simply have too high overhead.
  • extend exchangers to use MPI_Ibsend, MPI_Issend, MPI_Bsend, or MPI_Ssend, where applicable.
Edited by Thomas Jahns

Merge request reports

Loading
Loading

Activity

Filter activity
  • Approvals
  • Assignees & reviewers
  • Comments (from bots)
  • Comments (from users)
  • Commits & branches
  • Edits
  • Labels
  • Lock status
  • Mentions
  • Merge request status
  • Tracking
  • Thomas Jahns requested review from @k202077

    requested review from @k202077

    • Moritz Hanke
      Moritz Hanke @k202077 ·
      Maintainer 
      /home/moritz/DKRZ/yaxt/src/xt_exchanger_internal.h:65: warning: Example test_exchanger_parallel.c was already documented. Ignoring documentation found here.
/home/moritz/DKRZ/yaxt/src/xt/xt_idxlist.h:95: warning: unable to resolve reference to 'xt_idxlist_get_pack_size' for \ref command
/home/moritz/DKRZ/yaxt/src/xt/xt_idxlist.h:95: warning: unable to resolve reference to 'xt_idxlist_pack' for \ref command
/home/moritz/DKRZ/yaxt/src/xt/xt_idxlist.h:96: warning: unable to resolve reference to 'xt_idxlist_unpack' for \ref command
/home/moritz/DKRZ/yaxt/src/xt/xt_idxlist.h:119: warning: unable to resolve reference to 'xt_idxlist_get_pack_size' for \ref command
/home/moritz/DKRZ/yaxt/src/xt/xt_idxlist.h:121: warning: unable to resolve reference to 'xt_idxlist_get_pack_size' for \ref command
/home/moritz/DKRZ/yaxt/src/xt/xt_idxlist.h:121: warning: unable to resolve reference to 'xt_idxlist_pack' for \ref command
/home/moritz/DKRZ/yaxt/src/xt/xt_idxlist.h:121: warning: unable to resolve reference to 'xt_idxlist_unpack' for \ref command
/home/moritz/DKRZ/yaxt/src/xt/xt_idxlist.h:142: warning: unable to resolve reference to 'xt_idxlist_get_pack_size' for \ref command
/home/moritz/DKRZ/yaxt/src/xt/xt_idxlist.h:142: warning: unable to resolve reference to 'xt_idxlist_pack' for \ref command
/home/moritz/DKRZ/yaxt/src/xt/xt_idxlist.h:143: warning: unable to resolve reference to 'xt_idxlist_unpack' for \ref command
/home/moritz/DKRZ/yaxt/src/xt/xt_redist_p2p.h:316: warning: unable to resolve reference to 'xt_redist_p2p_blocks_off_new' for \ref command
/home/moritz/DKRZ/yaxt/src/xt/xt_redist_p2p.h:295: warning: unable to resolve reference to 'xt_redist_p2p_blocks_off_new' for \ref command
/home/moritz/DKRZ/yaxt/src/xt/xt_xmap.h:176: warning: unable to resolve reference to 'xt_xmap_iterator_get_transfer_pos' for \ref command
/home/moritz/DKRZ/yaxt/src/xt/xt_xmap.h:180: warning: unable to resolve reference to 'xt_xmap_iterator_get_transfer_pos' for \ref command
/home/moritz/DKRZ/yaxt/src/xt/xt_xmap.h:195: warning: unable to resolve reference to 'xt_xmap_iterator_get_transfer_pos_ext' for \ref command
/home/moritz/DKRZ/yaxt/src/xt/xt_xmap.h:199: warning: unable to resolve reference to 'xt_xmap_iterator_get_transfer_pos_ext' for \ref command
/home/moritz/DKRZ/yaxt/src/xt/xt_xmap.h:277: warning: unable to resolve reference to 'xt_xmap_get_max_dst_pos' for \ref command
/home/moritz/DKRZ/yaxt/src/xt_ddt.h:68: warning: The following parameter of xt_ddt_get_pack_size(size_t count, MPI_Datatype mpi_ddt) is not documented:
  parameter 'count'
/home/moritz/DKRZ/yaxt/src/xt_ddt_internal.h:129: warning: The following parameter of xt_ddt_get_pack_size_internal(size_t count, Xt_ddt ddt) is not documented:
  parameter 'count'
/home/moritz/DKRZ/yaxt/src/xt_ddt.h:76: warning: The following parameter of xt_ddt_pack(const void *src, size_t count, MPI_Datatype mpi_ddt, void *dst) is not documented:
  parameter 'count'
/home/moritz/DKRZ/yaxt/src/xt_ddt.h:86: warning: The following parameter of xt_ddt_unpack(const void *src, size_t count, MPI_Datatype mpi_ddt, void *dst) is not documented:
  parameter 'count'
/home/moritz/DKRZ/yaxt/src/xt_ddt.h:68: warning: The following parameter of xt_ddt_get_pack_size(size_t count, MPI_Datatype mpi_ddt) is not documented:
  parameter 'count'
/home/moritz/DKRZ/yaxt/src/xt_ddt.h:76: warning: The following parameter of xt_ddt_pack(const void *src, size_t count, MPI_Datatype mpi_ddt, void *dst) is not documented:
  parameter 'count'
/home/moritz/DKRZ/yaxt/src/xt_ddt.h:86: warning: The following parameter of xt_ddt_unpack(const void *src, size_t count, MPI_Datatype mpi_ddt, void *dst) is not documented:
  parameter 'count'
/home/moritz/DKRZ/yaxt/src/xt_ddt_internal.h:129: warning: The following parameter of xt_ddt_get_pack_size_internal(size_t count, Xt_ddt ddt) is not documented:
  parameter 'count'
/home/moritz/DKRZ/yaxt/src/xt_exchanger.h:143: warning: The following parameter of xt_exchanger_get_iterator(Xt_exchanger exchanger, Xt_config config) is not documented:
  parameter 'config'
/home/moritz/DKRZ/yaxt/src/xt_exchanger.h:129: warning: The following parameter of xt_exchanger_get_num_msg(Xt_exchanger exchanger, int direction) is not documented:
  parameter 'direction'
/home/moritz/DKRZ/yaxt/src/xt_exchanger.h:108: warning: The following parameters of xt_exchanger_new(int nsend, int nrecv, const struct Xt_msg_param *send_msgs, const struct Xt_msg_param *recv_msgs, MPI_Comm comm, int tag_offset, Xt_config config) are not documented:
  parameter 'nsend'
  parameter 'nrecv'
  parameter 'send_msgs'
  parameter 'recv_msgs'
  parameter 'tag_offset'
/home/moritz/DKRZ/yaxt/src/xt_exchanger.h:143: warning: The following parameter of xt_exchanger_get_iterator(Xt_exchanger exchanger, Xt_config config) is not documented:
  parameter 'config'
/home/moritz/DKRZ/yaxt/src/xt_exchanger.h:129: warning: The following parameter of xt_exchanger_get_num_msg(Xt_exchanger exchanger, int direction) is not documented:
  parameter 'direction'
/home/moritz/DKRZ/yaxt/src/xt_exchanger.h:108: warning: The following parameters of xt_exchanger_new(int nsend, int nrecv, const struct Xt_msg_param *send_msgs, const struct Xt_msg_param *recv_msgs, MPI_Comm comm, int tag_offset, Xt_config config) are not documented:
  parameter 'nsend'
  parameter 'nrecv'
  parameter 'send_msgs'
  parameter 'recv_msgs'
  parameter 'tag_offset'
/home/moritz/DKRZ/yaxt/src/xt/xt_idxlist.h:95: warning: unable to resolve reference to 'xt_idxlist_get_pack_size' for \ref command
/home/moritz/DKRZ/yaxt/src/xt/xt_idxlist.h:95: warning: unable to resolve reference to 'xt_idxlist_pack' for \ref command
/home/moritz/DKRZ/yaxt/src/xt/xt_idxlist.h:96: warning: unable to resolve reference to 'xt_idxlist_unpack' for \ref command
/home/moritz/DKRZ/yaxt/src/xt/xt_idxlist.h:119: warning: unable to resolve reference to 'xt_idxlist_get_pack_size' for \ref command
/home/moritz/DKRZ/yaxt/src/xt/xt_idxlist.h:121: warning: unable to resolve reference to 'xt_idxlist_get_pack_size' for \ref command
/home/moritz/DKRZ/yaxt/src/xt/xt_idxlist.h:121: warning: unable to resolve reference to 'xt_idxlist_pack' for \ref command
/home/moritz/DKRZ/yaxt/src/xt/xt_idxlist.h:121: warning: unable to resolve reference to 'xt_idxlist_unpack' for \ref command
/home/moritz/DKRZ/yaxt/src/xt/xt_idxlist.h:142: warning: unable to resolve reference to 'xt_idxlist_get_pack_size' for \ref command
/home/moritz/DKRZ/yaxt/src/xt/xt_idxlist.h:142: warning: unable to resolve reference to 'xt_idxlist_pack' for \ref command
/home/moritz/DKRZ/yaxt/src/xt/xt_idxlist.h:143: warning: unable to resolve reference to 'xt_idxlist_unpack' for \ref command
/home/moritz/DKRZ/yaxt/src/xt/xt_redist_p2p.h:316: warning: unable to resolve reference to 'xt_redist_p2p_blocks_off_new' for \ref command
/home/moritz/DKRZ/yaxt/src/xt/xt_redist_p2p.h:295: warning: unable to resolve reference to 'xt_redist_p2p_blocks_off_new' for \ref command
/home/moritz/DKRZ/yaxt/src/xt/xt_xmap.h:176: warning: unable to resolve reference to 'xt_xmap_iterator_get_transfer_pos' for \ref command
/home/moritz/DKRZ/yaxt/src/xt/xt_xmap.h:180: warning: unable to resolve reference to 'xt_xmap_iterator_get_transfer_pos' for \ref command
/home/moritz/DKRZ/yaxt/src/xt/xt_xmap.h:195: warning: unable to resolve reference to 'xt_xmap_iterator_get_transfer_pos_ext' for \ref command
/home/moritz/DKRZ/yaxt/src/xt/xt_xmap.h:199: warning: unable to resolve reference to 'xt_xmap_iterator_get_transfer_pos_ext' for \ref command
/home/moritz/DKRZ/yaxt/src/xt/xt_xmap.h:277: warning: unable to resolve reference to 'xt_xmap_get_max_dst_pos' for \ref command
warning: Call graph for '__yaxt_f90__' not generated, too many nodes (111), threshold is 50. Consider increasing DOT_GRAPH_MAX_NODES.
/home/moritz/DKRZ/yaxt/src/xt_config.c:459: warning: unable to resolve reference to 'xt_xmap_dist_dir_new' for \ref command
/home/moritz/DKRZ/yaxt/src/xt_config.c:460: warning: unable to resolve reference to 'xt_xmap_all2all_new' for \ref command
/home/moritz/DKRZ/yaxt/src/xt_config.c:460: warning: unable to resolve reference to 'xt_idxvec' for \ref command
/home/moritz/DKRZ/yaxt/src/xt_config.c:461: warning: unable to resolve reference to 'xt_idxstripes' for \ref command
/home/moritz/DKRZ/yaxt/src/xt_config.c:476: warning: unable to resolve reference to 'xt_config_set_redist_mthread_mode' for \ref command
/home/moritz/DKRZ/yaxt/src/xt_config.c:487: warning: unable to resolve reference to 'xt_config_set_sort_algorithm_by_id' for \ref command
/home/moritz/DKRZ/yaxt/src/xt_config.c:506: warning: unable to resolve reference to 'xt_config_set_xmap_stripe_align' for \ref command
/home/moritz/DKRZ/yaxt/doc/src/correct_use_of_addresses.dox:25: warning: unable to resolve reference to 'xt_redist_collection_static_new' for \ref command
/home/moritz/DKRZ/yaxt/doc/src/correct_use_of_addresses.dox:26: warning: unable to resolve reference to 'xt_redist_collection_new' for \ref command
/home/moritz/DKRZ/yaxt/doc/src/correct_use_of_addresses.dox:26: warning: unable to resolve reference to 'xt_redist_s_exchange' for \ref command
/home/moritz/DKRZ/yaxt/doc/src/correct_use_of_addresses.dox:27: warning: unable to resolve reference to 'xt_redist_a_exchange' for \ref command
/home/moritz/DKRZ/yaxt/doc/src/multi-threading.dox:27: warning: unable to resolve reference to 'xt_redist_s_exchange' for \ref command
/home/moritz/DKRZ/yaxt/doc/src/multi-threading.dox:27: warning: unable to resolve reference to 'xt_redist_a_exchange' for \ref command
/home/moritz/DKRZ/yaxt/doc/src/multi-threading.dox:28: warning: unable to resolve reference to 'xt_redist_s_exchange1' for \ref command
/home/moritz/DKRZ/yaxt/doc/src/multi-threading.dox:28: warning: unable to resolve reference to 'xt_redist_a_exchange1' for \ref command
/home/moritz/DKRZ/yaxt/doc/src/yaxt_concept.dox:71: warning: unable to resolve reference to 'xt_redist_s_exchange' for \ref command
/home/moritz/DKRZ/yaxt/doc/src/yaxt_concept.dox:72: warning: unable to resolve reference to 'xt_redist_s_exchange1' for \ref command
/home/moritz/DKRZ/yaxt/doc/src/yaxt_concept.dox:77: warning: unable to resolve reference to 'xt_idxlist_delete' for \ref command

      Maybe this should be checked automatically by a job in the gitlab-CI pipeline.

      Edited by Moritz Hanke
    • Thomas Jahns
      Thomas Jahns @k202069 ·
      Author Maintainer

      This problem (see https://github.com/doxygen/doxygen/issues/10994) is the reason the automated documentation build switched back to doxygen 1.9.2 in eeb47986 since 1.9.5 to 1.12 are broken (see discussion in chat on Jul 24).

    • Moritz Hanke
      Moritz Hanke @k202077 ·
      Maintainer

      The doxygen-error output still contains a couple of undocumented parameters, which were introduced in this MR.

    • Please register or sign in to reply
    • Moritz Hanke
      Moritz Hanke @k202077 ·
      Maintainer

      Why is there still the definition of SEND and RECV in xt_redist_internal.h. Couldn't xt_direction_send/recv be used everywhere instead?

      For example this now looks rather inconsistent:

         unsigned nmsg_send = xt_redist_agg_msg_count(
-      num_redists_, SEND, redists, num_ranks[SEND], ranks[SEND], config),
+      num_redists_, xt_direction_send, redists,
+      num_ranks[SEND], ranks[SEND], config),
    • Thomas Jahns
      Thomas Jahns @k202069 ·
      Author Maintainer

      The externally visible name is quite long, so I only switched the minimum needed to get the code to compile under all circumstances. SEND and RECV can then later be eliminated piecewise.

    • Please register or sign in to reply
    • Moritz Hanke
      Moritz Hanke @k202077 ·
      Maintainer

      With a public exchanger interface, yaxt can be used with varying degrees of complexity:

      • xmap+redist: yaxt does most of the work
      • xmap-intersection+redist: user has to generate communication matrix
      • exchanger: user has to generate communication matrix and MPI-datatypes
        • what is the application for this?
    • Thomas Jahns
      Thomas Jahns @k202069 ·
      Author Maintainer

      The third case is interesting when the datatype is very simple, i.e. a named datatype plus count. And because the exchanger does not generate its own communicator this leads to even less overhead then the use of a redist.

    • Moritz Hanke
      Moritz Hanke @k202077 ·
      Maintainer

      Out of curiosity: What is a practical application for this? I cannot seem to come up with one.

    • Please register or sign in to reply
  • Thomas Jahns added 54 commits

    added 54 commits

    • 6ea5fd4d...b72e4514 - 38 commits from branch master
    • b72e4514...13df6748 - 6 earlier commits
    • b8d309f3 - exchangers: Fix further clang warnings introduced by code changes.
    • a34c436d - exchangers: Add iterator interface for messages.
    • 0c64053c - exchangers: Add number of messages inquiry.
    • f9412607 - exchangers: Merge heap blocks.
    • 845ab60a - exchangers: Add Fortran symbols to public export list.
    • 7cec807a - exchangers: Move constructor query into individual tests.
    • 5c2521ae - exchangers: Extract function.
    • 973b728b - exchangers: Add constructor to API.
    • 789a8dc1 - exchangers: Use public functionality in exchanger test.
    • 04b91920 - exchanger: Add Fortran interface and test for exchanger.

    Compare with previous version

    Toggle commit list
  • Thomas Jahns added 35 commits

    added 35 commits

    • 04b91920...cff5708e - 25 earlier commits
    • f9ace6ea - exchangers: Unifiy place to set send mode.
    • 90526791 - exchangers: Shorten call.
    • 8073dd8a - exchangers: Retain configuration flags.
    • 536e218b - test suite: Move invariant data to text segment.
    • 3ef6a59f - test suite: Try alternative send modes.
    • 98eb5ca2 - exchangers: Add buffered and synchronous mode to irecv_isend exchanger.
    • c8366bbf - exchangers: Add buffered and synchronous mode to irecv_send exchanger.
    • 7b6c088d - exchangers: Add buffered and synchronous mode to packed irecv_isend exchanger.
    • 9de0463f - exchangers: Add buffered and synchronous mode to ddt packed exchanger.
    • ecc64c0d - exchangers: Make send mode feature optional.

    Compare with previous version

    Toggle commit list
  • Thomas Jahns changed the description

    changed the description

  • Thomas Jahns added 22 commits

    added 22 commits

    • ecc64c0d...d1d5817b - 12 earlier commits
    • adb2dca0 - exchangers: Unifiy place to set send mode.
    • df10199b - exchangers: Shorten call.
    • addc5ec7 - exchangers: Retain configuration flags.
    • 59a18362 - test suite: Move invariant data to text segment.
    • 3559b282 - test suite: Try alternative send modes.
    • af08fcc1 - exchangers: Add buffered and synchronous mode to irecv_isend exchanger.
    • 8ada9747 - exchangers: Add buffered and synchronous mode to irecv_send exchanger.
    • 71c4f0b2 - exchangers: Add buffered and synchronous mode to packed irecv_isend exchanger.
    • 4dc9be71 - exchangers: Add buffered and synchronous mode to ddt packed exchanger.
    • 183f43d7 - exchangers: Make send mode feature optional.

    Compare with previous version

    Toggle commit list
Please register or sign in to reply

Imprint | Privacy Policy