CarpetLib, Carpet: implement and use collective communication buffers

Collective buffers are used to gather all components' data on a processor
before it gets send off to other processors in one go. This minimizes the
number of outstanding MPI communications down to O(N-1) and thus improves
overall efficiency as benchmarks show.

Each processor allocates a pair of single send/recv buffers to communicate
with all other processors. For this the class (actually, the struct) comm_state
was extended by 3 more states:

 state_get_buffer_sizes:   accumulates the sizes for the send/recv buffers
 state_fill_send_buffers:  gathers all the data into the send buffers
 state_empty_recv_buffers: copies the data from the recv buffer back into
                           the processor's components

Send/recv buffers are exchanged during state_fill_send_buffers and
state_empty_recv_buffers. The constructor for a comm_state struct now takes
an argument <datatype> which denotes the CCTK datatype to use for the
attached collective buffers. If a negative value is passed here then it falls
back to using the old send/recv/wait communication scheme. The datatype
argument has a default value of -1 to maintain backwards compatibility to
existing code (which therefore will keep using the old scheme).

The new communication scheme is chosen by setting the parameter
CarpetLib::use_collective_communication_buffers to "yes". It defaults to "no"
meaning that the old send/recv/wait scheme is still used.

So far all the comm_state objects in the higher-level routines in thorn Carpet
(restriction/prolongation, regridding, synchronization) have been enabled to
use collective communication buffers.
Other thorns (CarpetInterp, CarpetIO*, CarpetSlab) will follow in separate
commits.

darcs-hash:20050330152811-776a0-51f426887fea099d1a67b42bd79e4f786979ba91.gz

1 files changed, 6 insertions, 0 deletions

diff --git a/Carpet/Carpet/src/variables.hh b/Carpet/Carpet/src/variables.hh
index ae5ade6df..7bab18432 100644
--- a/Carpet/Carpet/src/variables.hh
+++ b/Carpet/Carpet/src/variables.hh
@@ -119,6 +119,12 @@ namespace Carpet {
   };
   extern vector<groupdesc> groupdata; // [group]
   
+  // structure to hold a set of groups which all have the same CCTK vartype
+  struct group_set {
+    int vartype;                // eg. CCTK_VARIABLE_REAL, etc.
+    vector<int> members;        // members of this set
+  };                            // (given by their CCTK group indices)
+
   // Data for everything
   struct arrdesc {
     // points to hh etc. for GF, and is unique for SCALAR and ARRAY