CarpetInterp: optimise global interpolator communication scheme

CarpetInterp used to use CarpetLib's data class to exchange interpolation
information (interpolation coordinates and a source map as inputs,
interpolation results and status/return codes as outputs) between processors.
This point-to-point communication has been replaced by explicit collective
MPI operations which should now make global interpolations faster.

For a general overview on the implementation see CarpetInterp's thorn documentation.

darcs-hash:20050826115157-776a0-d910b51d7a26cef12e13408a79f11ed2826f5ed1.gz

1 files changed, 82 insertions, 31 deletions

diff --git a/Carpet/CarpetInterp/doc/documentation.tex b/Carpet/CarpetInterp/doc/documentation.tex
index 52848ad40..44e000a13 100644
--- a/Carpet/CarpetInterp/doc/documentation.tex
+++ b/Carpet/CarpetInterp/doc/documentation.tex
@@ -75,14 +75,15 @@
 \begin{document}
 
 % The author of the documentation
-\author{Erik Schnetter \textless schnetter@uni-tuebingen.de\textgreater}
+\author{Erik Schnetter \textless schnetter@aei.mpg.de\textgreater\\
+        Thomas Radke \textless tradke@aei.mpg.de\textgreater}
 
 % The title of the document (not necessarily the name of the Thorn)
 \title{CarpetInterp}
 
 % the date your document was last changed, if your document is in CVS, 
 % please use:
-\date{April 29, 2003}
+\date{26 August 2005}
 
 \maketitle
 
@@ -95,6 +96,8 @@
 % Add an abstract for this thorn's documentation
 \begin{abstract}
 
+Thorn {\bf CarpetInterp} provides a parallel interpolator for Carpet.
+
 \end{abstract}
 
 % The following sections are suggestive only.
@@ -102,36 +105,84 @@
 
 \section{Introduction}
 
-\section{Physical System}
-
-\section{Numerical Implementation}
-
-\section{Using This Thorn}
-
-\subsection{Obtaining This Thorn}
-
-\subsection{Basic Usage}
-
-\subsection{Special Behaviour}
-
-\subsection{Interaction With Other Thorns}
-
-\subsection{Examples}
-
-\subsection{Support and Feedback}
-
-\section{History}
-
-\subsection{Thorn Source Code}
-
-\subsection{Thorn Documentation}
-
-\subsection{Acknowledgements}
-
-
-\begin{thebibliography}{9}
+{\bf CarpetInterp} works similar to {\bf PUGHInterp}, the interpolation
+thorn of the standard unigrid driver in Cactus.
+
+\begin{enumerate}
+  \item Firstly, the points to interpolate at (which can vary from processor
+        to processor) are mapped onto processors.
+        Each processor accumulates the number of coordinates it needs to
+        send to other processors (which own the corresponding points).
+
+  \item In a collective communication step, all processors exchange the
+        coordinates of the interpolation points which are not located locally.
+
+  \item After each processor received the coordinates of (local) points to
+        interpolate at, it maps them onto components (also counting the
+        number of points for each component).
+
+  \item Now all processors do -- in parallel -- the actual interpolation
+        by calling the local interpolator (via {\tt CCTK\_InterpLocalUnform()})
+        on each component.
+
+  \item Finally, the interpolation results are sent back to the requesting
+        processors in another collective communication step.
+\end{enumerate}
+
+
+\section{Using CarpetInterp}
+
+{\bf CarpetInterp} overloads the flesh interpolation C API routine
+\begin{verbatim}
+  int CCTK_InterpGridArrays
+                     (const cGH *GH,
+                      int N_dims,
+                      int local_interp_handle,
+                      int param_table_handle,
+                      int coord_system_handle,
+                      int N_interp_points,
+                        int interp_coords_type,
+                        const void *const interp_coords[],
+                      int N_input_arrays,
+                        const CCTK_INT input_array_indices[],
+                      int N_output_arrays,
+                        const CCTK_INT output_array_types[],
+                        void *const output_arrays[])
+\end{verbatim}
+which is described in detail in the Cactus Reference Manual.
+
+It also provides an equivalent aliased function
+\begin{verbatim}
+  CCTK_INT FUNCTION DriverInterpolate
+                     (CCTK_POINTER_TO_CONST IN cctkGH,
+                      CCTK_INT IN N_dims,
+                      CCTK_INT IN local_interp_handle,
+                      CCTK_INT IN param_table_handle,
+                      CCTK_INT IN coord_system_handle,
+                      CCTK_INT IN N_interp_points,
+                      CCTK_INT IN interp_coords_type,
+                      CCTK_POINTER_TO_CONST ARRAY IN interp_coords,
+                      CCTK_INT IN N_input_arrays,
+                      CCTK_INT ARRAY IN input_array_indices,
+                      CCTK_INT IN N_output_arrays,
+                      CCTK_INT ARRAY IN output_array_types,
+                      CCTK_POINTER ARRAY IN output_arrays)
+\end{verbatim}
+which can be called by symmetry thorns to implement symmetry interpolations
+(see the thorn documentation of {\bf CactusBase/SymBase} for details).
+
+
+\subsection{Limitations}
+
+The current implementation of {\bf CarpetInterp} supports only 3-dimensional
+grid variables as inputs ({\tt N\_dims $==$ 3}).
+
+Inperpolation coordinates and output arrays must be of type
+{\tt CCTK\_VARIABLE\_REAL}.
+
+A valid handle for a parameter table mus be passed in so that {\bf CarpetInterp}
+can store in it the overall status code of the local interpolator.
 
-\end{thebibliography}
 
 % Do not delete next line
 % END CACTUS THORNGUIDE