CarpetLib, Carpet: implement and use collective communication buffers

Collective buffers are used to gather all components' data on a processor
before it gets send off to other processors in one go. This minimizes the
number of outstanding MPI communications down to O(N-1) and thus improves
overall efficiency as benchmarks show.

Each processor allocates a pair of single send/recv buffers to communicate
with all other processors. For this the class (actually, the struct) comm_state
was extended by 3 more states:

 state_get_buffer_sizes:   accumulates the sizes for the send/recv buffers
 state_fill_send_buffers:  gathers all the data into the send buffers
 state_empty_recv_buffers: copies the data from the recv buffer back into
                           the processor's components

Send/recv buffers are exchanged during state_fill_send_buffers and
state_empty_recv_buffers. The constructor for a comm_state struct now takes
an argument <datatype> which denotes the CCTK datatype to use for the
attached collective buffers. If a negative value is passed here then it falls
back to using the old send/recv/wait communication scheme. The datatype
argument has a default value of -1 to maintain backwards compatibility to
existing code (which therefore will keep using the old scheme).

The new communication scheme is chosen by setting the parameter
CarpetLib::use_collective_communication_buffers to "yes". It defaults to "no"
meaning that the old send/recv/wait scheme is still used.

So far all the comm_state objects in the higher-level routines in thorn Carpet
(restriction/prolongation, regridding, synchronization) have been enabled to
use collective communication buffers.
Other thorns (CarpetInterp, CarpetIO*, CarpetSlab) will follow in separate
commits.

darcs-hash:20050330152811-776a0-51f426887fea099d1a67b42bd79e4f786979ba91.gz

1 files changed, 22 insertions, 0 deletions

diff --git a/Carpet/CarpetLib/src/gdata.hh b/Carpet/CarpetLib/src/gdata.hh
index 7315bc763..ef384703c 100644
--- a/Carpet/CarpetLib/src/gdata.hh
+++ b/Carpet/CarpetLib/src/gdata.hh
@@ -158,6 +158,17 @@ private:
                        const gdata* src, const ibbox& box);
   void copy_from_wait (comm_state& state,
                        const gdata* src, const ibbox& box);
+
+  // Copy processor-local source data into communication send buffer 
+  // of the corresponding destination processor
+  // The case when both source and destination are local is also handled here.
+  void copy_into_sendbuffer (comm_state& state,
+                             const gdata* src, const ibbox& box);
+  // Copy processor-local destination data from communication recv buffer
+  // of the corresponding source processor
+  void copy_from_recvbuffer (comm_state& state,
+                             const gdata* src, const ibbox& box);
+
 #if 0
  protected:
   virtual void
@@ -213,6 +224,17 @@ private:
                               const ibbox& box, const CCTK_REAL time,
                               const int order_space,
                               const int order_time);
+
+  // Interpolate processor-local source data into communication send buffer
+  // of the corresponding destination processor
+  // The case when both source and destination are local is also handled here.
+  void interpolate_into_sendbuffer (comm_state& state,
+                                    const vector<const gdata*> srcs,
+                                    const vector<CCTK_REAL> times,
+                                    const ibbox& box,
+                                    const CCTK_REAL time,
+                                    const int order_space,
+                                    const int order_time);
  public:
   
 protected: