From c34a4a794529079827557d89d4bb65242980573f Mon Sep 17 00:00:00 2001 From: goodale Date: Tue, 2 Nov 2004 22:00:16 +0000 Subject: Restore old reduction docs so people can understand current codes and know how to migrate to new API. Patch from Yaakoub. git-svn-id: http://svn.cactuscode.org/flesh/trunk@3882 17b73243-c579-4c4c-a9d2-2d5706c11dac --- doc/UsersGuide/ThornWriters.tex | 396 +++++++++++++++++++++++++++++++++++++++- 1 file changed, 389 insertions(+), 7 deletions(-) (limited to 'doc/UsersGuide') diff --git a/doc/UsersGuide/ThornWriters.tex b/doc/UsersGuide/ThornWriters.tex index 2f2458c5..b5dcf10f 100644 --- a/doc/UsersGuide/ThornWriters.tex +++ b/doc/UsersGuide/ThornWriters.tex @@ -2585,15 +2585,43 @@ name. \section{Reduction Operators} \label{sec:reop} -The Flesh does not provide reduction routines by itself. Instead -it offers a general function API to thorns for the registration and -invocation of reduction operators. - -There are two different Flesh APIs for reduction, depending +A reduction operation can be defined as an operation on variables +distributed across multiple processor resulting in a single number. +Typical reduction operations are: sum, minimum/maximum value, boolean +operations. A typical application is, for example, +finding the maximum reduction from processor local error estimates, +therefor making the previous processor local error known to all processors. + +The exchange of +information across processors needs the functionality of a +communication layer e.g. {\tt CactusPUGH/PUGH}. For this reason, the +reduction operation itself is not part of the flesh, instead Cactus (again) +provides a registration mechanism for thorns to register routines they +provide as reduction operators. The different operators are +identified by their name and/or a unique number, called a {\em handle}. + +The registration mechanism gives the advantage of a common +interface while hiding the individual communication calls in the +layer. + +In Cactus, reduction operators can be applied to +grid functions, arrays and scalars, as well as to local arrays. Note that +different implementations of reduction operators may be limited in +the objects they can be applied to. +There is a fundamental difference between the reduction operation on +grid functions and quantities as arrays. + +Currently the Flesh supports the new and old reduction specification. +The old APIs will be deprecated in the next beta cycle in favour of the +new specification. + +{\bf New Reduction Specification API documentation} +\vskip .25cm +In the new reduction specification, there are two different Flesh APIs for reduction, depending on whether the data arrays are Cactus grid arrays or processor-local, programming language built-in arrays, and on what assumptions are made about the topology and spacing of the grid (these descriptions -are for 3-D, but the generalizations to other numbers of dimensions +are for 3-D, but the generalisations to other numbers of dimensions should be obvious): \begin{Lentry} \item[{\tt CCTK\_ReduceGridArrays()}] @@ -2623,6 +2651,360 @@ may be used to refer to the operator. \verb|CCTK_LocalArrayReductionHandle()| is used to get the handle corresponding to a given character string name. +{\bf Old Reduction Specification API documentation} +\vskip .25cm + +{\bf Obtaining the reduction handle} + +Before calling the routine which performs the reduction operation, +the handle, which identifies the operation must be derived from its +registered name. + +\begin{verbatim} + +int CCTK_ReductionHandle(const char *reduction_name); + +integer reduction_handle +character*(*) reduction_name +call CCTK_ReductionHandle(reduction_handle, reduction_name) + + +int CCTK_ReductionArrayHandle(const char *reduction_name); + +integer reduction_handle +character*(*) reduction_name +call CCTK_ReductionArrayHandle(reduction_handle, reduction_name) + +\end{verbatim} + + +\vskip .25cm + +\begin{Lentry} +\item[{\tt reduction\_handle}] +In Fortran the name of the variable will +contain the handle value after the call. In C this value is the +function value. +\item[{\tt reduction\_name}] +is the name under which the operator has +been registered by the providing thorn. The only thorn in the standard +Computational Toolkit release which provides reduction operators is +{\tt CactusPUGH/PUGHReduce}. +\item[{\bf error checking}] +negative handle value indicates failure to +identify the correct operator. +\end{Lentry} + +Get a integer handle corresponding to a given reduction operator. +The operator is identified by the name it was registered with. +(Note that although it would appear to be far more convenient to +pass the name of the reduction operator directly to the following +function call to {\t CCTK\_Reduce} this causes problems with the +translation of strings from {\t FORTRAN} to {\t C} with variable +argument lists). + +\vskip 0.25cm + + +{\bf The general reduction interface} +The main interfaces for reduction operations are quite powerful (and +hence rather complicated) To ease the use of these main interfaces, wrappers +designed for specific and more restricted use are described below. If +uncertain, you should use these. + +{\t +\begin{verbatim} + +int CCTK_Reduce( cGH *GH, + int proc, + int operation_handle, + int num_out_vals, + int type_out_vals, + void *out_vals, + int num_in_fields, + ...); + + +call CCTK_Reduce( int returnvalue, + cctkGH, + int processor, + int operation_handle, + int num_out_vals, + int type_out_vals, + out_vals, + int num_in_fields, + ... ) + +int CCTK_ReduceArray( cGH *GH, + int proc, + int operation_handle, + int num_out_vals, + int type_out_vals, + void *out_vals, + int num_dims, + int num_in_arrays, + int type_in_arrays, + ... ) + +call CCTK_ReduceArray(int returnvalue, + cctkGH, + int processor, + int operation_handle, + int num_out_vals, + int type_out_arrays, + void out_vals, + int num_dims, + int num_in_arrays, + int type_in_arrays, + ... ) +\end{verbatim} +} + +\begin{Lentry} +\item[{\tt int returnvalue}] +the return value of the operation. negative +value indicates failure to perform reduction. +Zero indicates successful operation. +\item[{\tt cctkGH}] +in Fortran the pointer to the grid hierarchy +structure. Cannot be used within Fortran but +can be used from within +C. Since this name is fixed write it out shown +above. +\item[{\tt cGH *GH}] + in the C the pointer to the grid hierarchy. +\item[{\tt int processor}] +the processor which collects the +information, negative value ($-1$) will distribute the data to all +processors. +\item[{\tt int operation\_handle}] the number of the reduction operation + handle, needs to be found by calling {\tt CCTK\_ReductionHandle} or + {\tt CCTK\_ReductionArrayHandle}. +\item[{\tt int num\_out\_vals}] integer defining the number of ?? +\item[{\tt int type\_out\_arrays}, {\tt type\_in\_arrays}] +specifies the type of the gridfunction +you are communicating. Use the values as specified in +\ref{sec:datyansi}. Note: do not {\em mix} datatypes: e.g. in +Fortran do not declare a variable as {\tt integer} and then specify +the type {\tt CCTK\_VARIABLE\_INT} in the reduction command. These +types need not be the same on some architectures and will conflict. +\item[{\tt out\_vals}] +\item[{\tt int num\_in\_fields}] specifies the number of to follow +\item[{\tt ...}] indicates a variable argument list: specify the size +of the array in each dimension, comma separated numbers or integer +variables. Specify the arrays which will be reduced, the number of +specified arrays must be the same as the value of the {\tt +num\_in\_fields} variable. +\item[{\bf error checking}] a return value other than zero indicates +failure to perform the operation. +\end{Lentry} + +\vskip 0.25cm + + +{\bf Special reduction interfaces} + +The routines are designed for the purpose of reducing scalars, arrays +and grid functions. They hide many of the options of the generic +interface described above.\\ + +{\bf Reduction of local scalars} across multiple processors, the result of +the reduction operation will be on the specified processor or on all processors. + +{\t +\begin{verbatim} +int CCTK_ReduceLocScalar (cGH *GH, + int processor, + int operation_handle, + void *in_scalar, + void *out_scalar, + int data_type) + +call CCTK_ReduceLocScalar(int returnvalue, + cctkGH, + int processor, + int operation_handle, + in_scalar, + out_scalar, + int data_type) +\end{verbatim} +} +\begin{Lentry} +\item[{\tt in\_scalar}] the processor local variable with local value to be reduced +\item[{\tt out\_scalar}] the reduction result: a processor local variable +with the global value (same on all procs) if {\tt processor} has been +set to $-1$. Otherwise {\tt processor} will hold the reduction result. +\item[{\tt data\_type}] +specifies the type of the gridfunction +you are communicating. Use the values as specified in +\ref{sec:datyansi}. +\end{Lentry} + +\vskip 0.25cm + +{\bf Reduction of local 1d arrays} to a local arrays, this reduction is carried +out element by element. The arrays need to have the same size on all +processors. +{\t +\begin{verbatim} +int CCTK_ReduceLocArrayToArray1D( cGH *GH, + int processor, + int operation_handle, + void *in_array1d, + void *out_array1d, + int xsize) + int data_type) + +call CCTK_ReduceLocArrayToArray1D( int returnvalue + cctkGH, + int processor, + int operation_handle, + in_array1d, + out_array1d, + int xsize, + int data_type) +\end{verbatim} +} + +\begin{Lentry} +\item[{\tt in\_array1d}] one dimensional local arrays, to be reduced across a +processors, element by element. +\item[{\tt out\_array1d}] array holding the reduction result. out\_array1d[1] += Reduction(in\_array[1]). +\item[{\tt xsize}] the size of the one dimensional array. +\end{Lentry} + +\vskip 0.25cm + +{\bf Reduction of local 2d arrays} to a local 2d array, this reduction is carried +out element by element. The arrays need to have the same size on all +processors. +{\t +\begin{verbatim} +int CCTK_ReduceLocArrayToArrayD( cGH *GH, + int processor, + int opertaion_handle, + in_array_2d, + out_array2d, + int xsize, + int ysize, + int data_type) + + +call CCTK_ReduceLocArrayToArray2D( int returnvalue + cctkGH, + int processor, + int operation_handle, + in_array2d, + out_array2d, + int xsize, + int ysize, + int data_type) + +\end{verbatim} +} + +\begin{Lentry} +\item[{\tt in\_array1d}] two dimensional local arrays, to be reduced across a +processors, element by element. +\item[{\tt out\_array1d}] two dimensional array holding the reduction +result. out\_array2d[i,j]= Reduction(in\_array2d[i,j]). +\item[{\tt xsize}] the size of the one dimensional array in x direction. +\item[{\tt ysize}] the size of the one dimensional array in y direction. +\end{Lentry} + +\vskip 0.25cm + +{\bf Reduction of local 3d arrays} to a local 3d array, this reduction is carried +out element by element. The arrays need to have the same size on all +processors. +{\t +\begin{verbatim} +int CCTK_ReduceLocArrayToArray3D(cGH *GH, + int processor, + int opertaion_handle, + in_array_3d, + out_array3d, + int xsize, + int ysize, + int zsize, + int data_type) + +call CCTK_ReduceLocArrayToArray3D(int returnvalue + cctkGH, + int processor, + int operation_handle, + in_array3d, + out_array3d, + int xsize, + int ysize, + int zsize, + int data_type) +\end{verbatim} +} + +\begin{Lentry} +\item[{\tt in\_array3d}] two dimensional local arrays, to be reduced across a +processors, element by element. +\item[{\tt out\_array3d}] two dimensional array holding the reduction +result. out\_array3d[i,j,k]= Reduction(in\_array3d[i,j,k]). +\item[{\tt xsize}] the size of the one dimensional array in x direction. +\item[{\tt ysize}] the size of the one dimensional array in y direction. +\item[{\tt ysize}] the size of the one dimensional array in z direction. +\end{Lentry} + +\vskip .25cm + +{\bf Some brief examples:} + +{\bf Reduction of a local scalars:} a local error is reduced across all +processors with the maximum operation. The variable {\tt tmp} will +hold the maximum of the error and is the same on all +processors. This quantity can then be reassigned to {\tt normerr}. +\begin{verbatim} + CCTK_REAL normerr, tmp + integer ierr, reduction_handle + + call CCTK_ReductionArrayHandle(reduction_handle,"maximum") + + if (reduction_handle.lt.0) then + call CCTK_WARN(1,"Cannot get reduction handle for maximum operation.") + endif + + call CCTK_ReduceLocScalar(ierr, cctkGH, -1, + . reduction_handle, + . normerr, tmp, CCTK_VARIABLE_REAL) + if (ierr.ne.0) then + call CCTK_WARN(1,"Reduction of norm failed!"); + endif + normerr = tmp +\end{verbatim} + + +{\bf Reduction of a local 2d array:} a two dimensional array $(2x3)$ is +reduced, reduction results (array of same size: {\tt bla\_tmp}) is seen +on all processors ($-1$ entry as the third argument); also demonstrates +some simple error checking with the {\tt CCTKi\_EXPECTOK} macro. +\begin{verbatim} + CCTK_REAL bla(2,3),bla_tmp(2,3); + integer ierr, sum_handle + + call CCTK_ReductionArrayHandle(sum_handle,"sum") + bla = 1.0d0 + write (*,*) "BLA ",bla + + call CCTK_ReduceLocArrayToArray2D(ierr, cctkGH, -1, sum_handle, + . bla, bla_tmp, 2, 3, CCTK_VARIABLE_REAL) + call CCTKi_EXPECTOK(ierr, 0, 1, "2D Reduction failed") + + bla = bla_tmp + write (*,*) "BLA ",bla +\end{verbatim} + +Note that the memory for the returned values must be allocated before +the reduction call is made. + + %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% @@ -3019,7 +3401,7 @@ ways (provided the underlying functionality is available on your system): \item[{\tt GetTimeOfDay}] Provides ``wall clock time'' via the unix \texttt{gettimeofday} function. \item[{\tt GetrUsage}] - Provides CPU usage time via the unix \texttt{getrusage} function. + Provides CPU usage time via the unix \texttt{getrusage} function. \end{Lentry} Additional clocks can be implemented by thorns and registered with the -- cgit v1.2.3