path: root/doc/UsersGuide/ThornWriters.tex

% /*@@
%   @file      ThornWriters.tex
%   @date      27 Jan 1999
%   @author    Gabrielle Allen, Tom Goodale, Gerd Lanfermann
%   @desc 
%   Thorn writer's guide part of the Cactus User's Guide
%   @enddesc 
%   @version $Header$      
% @@*/

\begin{cactuspart}{2}{Application thorn writing}{$RCSfile$}{$Revision$}

\renewcommand{\thepage}{\Alph{part}\arabic{page}}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\chapter{Overview}

This chapter goes into the nitty-gritty of writing a thorn. 
It introduces key concepts for thorns, then goes on to give
a brief outline of how to configure a thorn.
There is then some detail about concepts introduced by the configuration
step, followed by discussion of code which you must put into your files
in order to use Cactus functionality, and details of utility functions
you may use to gain extra functionality.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\chapter{Thorn concepts} 


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Thorns}

A thorn is the basic working module within Cactus.  All user-supplied
code goes into thorns, which are, by and large, independent of each other.
Thorns communicate with each other via calls to the flesh API, plus, more
rarely, custom APIs of other thorns.

The connection from a thorn to the flesh or to other thorns is specified in
configuration files which are parsed at compile time and used to generate
glue code which encapsulates the external appearence of a thorn.


\section{Arrangements}

Thorns are grouped into {\em arrangements}.  This is a logical grouping of
thorns which is purely for organisational purposes. For example, 
you might wish to keep all your initial data thorns in one arrangement,
and all your evolution thorns in another arrangement, or you may want 
to have separate arrangements for your developments, private and shared 
thorns. 

The arrangements live in the {\tt arrangements} directory off the main 
Cactus directory.  Arrangement names must be unique, and cannot start 
with a `\#', or finish
with `\~{}' or `.bak'.

Inside a arrangement directory there are directories for each thorn 
belonging to the arrangement.  
Thorn names have the same restrictions as arrangement names,
with the addition that a thorn cannot be called `doc'.  
This name is reserved for arrangement documentation.

\section{Implementations}

\label{sec:im}

One of the key concepts for thorns is the concept of the {\bf implementation}.
Relationships between thorns are all based upon relationship between the
{\bf implementations} they provide. 
In principle it should be possible to swap one thorn providing an 
implementation with another thorn providing that implementation,
without affecting any other thorn.

An {\bf implementation} defines a group of variables and parameters which
are used to implement some functionality.  For example the thorn 
{\tt CactusPUGH/PUGH} provides the implementation {\it driver}.  This 
implementation is responsible for providing memory for grid variables and
for communication.  Another thorn can also implement {\tt driver}, 
and both thorns can be compiled in {\em at the same time}.  
At runtime, the user can decide which thorn providing {\tt driver} is used.  
No other thorn should be affected by this choice.

When a thorn decides it needs access to a variable or a parameter provided by 
another thorn, it defines a relationship between itself and the other thorn's
{\bf implementation}, not explicitly with the other {\bf thorn}.  
This allows the transparent replacement, at compile or runtime, of one 
thorn with another thorn providing the same functionality as seen by 
the other thorns.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


\chapter{Anatomy of a thorn}
\label{sec:anofath}

\section{thorns}
\label{sec:th}

A thorn consists of a subdirectory of an arrangement containing three 
administrative files

\begin{Lentry}
\item[{\tt interface.ccl}] the cactus interface, which defines the grid 
functions, variables, etc. See \ref{sec:in}.
\item[{\tt param.ccl}] the parameters introduced by this thorn, and the
parameters needed from other thorns. See
\ref{sec:pa}.
\item[{\tt schedule.ccl}] scheduling information for routines called by
the flesh. See \ref{sec:sc}
\end{Lentry}

Thorns can also contain  
\begin{itemize}
\item a subdirectory called {\tt src}, which should hold source files
        and compilation instructions for the thorn
\item a subdirectory {\tt src/include} for include files 
\item a {\tt README} containing a brief description of the thorn
\item a {\tt doc} directory for documentation
\item a {\tt par} directory for example parameter files
\item a {\tt test} subdirectory may also be added, to hold the thorn's
    testsuite. See \ref{sec:adatesu} for details.
\end{itemize}

\section{Creating a thorn}


To simplify the creation of a thorn, a make target {\tt
gmake newthorn} has been provided. When this is run:

\begin{enumerate} 
\item{} You will be prompted for the name of the new thorn.
\item{} You will be prompted for the name of the arrangement you would
like to include your thorn in. Either enter a new arrangement name or pick 
one from the list of available arrangements that are shown.
\end{enumerate}


\section{Configuring your thorn}

The interaction of a thorn with the flesh and other thorns is controlled
by various configuration files.

These consist of:

\begin{Lentry}

\item [{\tt interface.ccl}]
This defines the {\bf implementation} (Section~\ref{sec:im}) the thorn 
provides, and the variables the thorn needs, along with their 
visibility to other implementations.

\item [{\tt param.ccl}]
This defines the parameters that are used to control the thorn, along
with their visibility to other implementations.

\item [{\tt schedule.ccl}]
This defines which functions from the thorn are called and when they are 
called. It also handles memory and communication assignment for grid variables.

\end{Lentry}

\subsection{General syntax of CCL files}

CCL {\bf Cactus Configuration Language} files are simple text files
used to define configuration information for a thorn.  CCL files are
case independent, and may contain comments introduced by the `\#' character,
which indicates that the rest of the line is a comment.

\subsection{The {\tt interface.ccl}}

The {\tt interface.ccl} file is used to declare the implementation 
provided by the thorn, and to define the variables provided by it.
The implementation is declared by a single line at the top of the file

\begin{verbatim}
implements: <name>
\end{verbatim}

Where {\tt <name>} can be any combination of alphanumeric 
characters and underscores, and is case independent.
There are three different access levels available for variables

\begin{Lentry}
\item[{\tt Public}]
Can be `inherited' by other implementations (see below).
\item[{\tt Protected}]
Can be shared with other implementations which declare themselves to
be friends of this one (see below).
\item[{\tt Private}]
Can only be seen by this thorn.
\end{Lentry}

Corresponding to the first two access levels there are two relationship
statements that can be used to get variables from other thorns.

\begin{Lentry}
\item [{\tt Inherits: <name>}]
This gets all {\tt Public} variables from implementation {\tt <name>}, and all
variables that {\tt <name>} has in turn inherited.  
An implementation may inherit from any number of thorns.
\item [{\tt Friend: <name>}]
This gets all {\tt Protected} variables from implementation {\tt <name>}, but, 
unlike {\tt inherits} it also pushes its own implementations {\tt Protected}
variables onto implementation {\tt <name>}.  This keyword is used to define
a group of implementations which all end up with the same {\tt Protected}
variables.
\end{Lentry}

So, for example, an {\tt interface.ccl} starting

\begin{verbatim}
implements: wavetoy
inherits:   grid
friend:     wave_extract
\end{verbatim}

declares that the thorn provides an implementation called `wavetoy', gets
all the {\tt public} variables declared by an implementation called `grid', and
shares all {\tt protected} variables with {\tt wave\_extract} and its friends.


For convenience variables are placed in groups of homogeneous variables.  
Currently, names of groups
and variables must be distinct. The group has several
attributes:

\begin{Lentry}

\item [{\tt variable type}]
e.g. REAL, INT, COMPLEX. The prefix ``CCTK\_'' is optional.

\item [{\tt name}]
The name of the group

\item [{\tt group type}]
This indicates the kind of variables help by the group (each group can
only hold one kind of variable), and defaults to {\tt GF}.

\begin{Lentry}
\item [{\tt SCALAR}]
This is a single number.
\item [{\tt GF}]
This is an array with the default grid size (a {\it grid function}).
\item [{\tt ARRAY}]
This is an array of any size.
\end{Lentry}

\item [{\tt Dim}]
This is the dimension of the arrays in a group.  (Meaningless for scalars.)
This defaults to 3

\item [{\tt Size}]
This is the grid size of the arrays in a group, which must be a 
comma-separated list of {\tt Dim} parameters. The grid size cannot be
 specified for grid functions, which take the default grid size.

\item [{\tt Ghostsize}]
This is the number of ghostzones used for the arrays in a group, which must 
be a comma-separated list of {\tt Dim} parameters. The ghostsize
cannot be specified for grid functions, whose ghostsize is specified 
by a driver thorn.

\item [{\tt TimeLevels}]
This is the number of timelevels the group has. This defaults to 1.

\item [{\tt Stagger}]
This attribute defines the stagger for a grid function. Specify a
string of length {\tt Dim}, where each character indicates the type of 
staggering in that direction (yxz ordered). If you use default
staggering ({\tt MMM}), you can omit this attribute.
\begin{Lentry}
\item[{\tt M}] no staggering (``minus'' face, relative to default gridpoint).
\item[{\tt C}] center staggering
\item[{\tt P}] full staggering by one grid spacing (``plus'' face,
relative to default)
\end{Lentry}
For a full discussion on staggering refer to \ref{sec:st}.

\item [{\tt Distrib}]
This attribute describes how an array should be distributed across 
multiple processors, it defaults to {\tt DEFAULT}.
\begin{Lentry}
\item[{\tt DEFAULT}] Distribute grid in default manner across processors. 
\item[{\tt CONSTANT}] Put a constant sized array on each processor.
\end{Lentry}

\end{Lentry}



A group specification consists of the variable type, followed by
the name of the group, then a space separated list of the form
`attribute = value' .  Then a brace delimited block containing
a comma or newline separated list of variables in the group.
A description of the group may be included on the line with the
closing brace.

For example, for a group of 5 grid functions on a 3D grid, with 3
timelevels:

\begin{verbatim}

REAL fields type=GF TimeLevels=3 Dim=3 stagger=MCM
{
  phi
  a,b,c,d
} "Wave fields"

\end{verbatim}
or, for a group consisting of just one 2D array of integers, 
\begin{verbatim}
INT afields type=ARRAY size=xsize,ysize ghostsize=gxsize,gysize
{
 anarray
} "My 2D arrays"

\end{verbatim}
where {\tt xsize}, {\tt ysize}, {\tt gxsize}, {\tt gysize} are all 
parameters defined in the thorns {\tt param.ccl}.


defines a group of real grid functions of dimension 3 each of which exists on
three time levels. The staggering of this group is of type {\tt MCM}:
no staggering {\tt M} in the first and third dimension, center
staggering {\tt C} for the second dimension.

By default all groups are {\tt private}, to change this an access
specification of the form {\tt public:} or {\tt protected:} (or 
{\tt private:} to change it back) may be placed on a line by itself.  This
changes the access level for any group defined in the file from that point on.

All variables seen by any one thorn must have distinct names.

\subsection{The {\tt param.ccl}}

Users control the operation of thorns via parameters given in a file
at runtime.  The {\tt param.ccl}
is used to specify the parameters used to control an individual thorn, and
to specify the values these parameters are allowed to take.  When the code
is run it reads a parameter file and sets the parameters if they fall
within the allowed values. If a parameter is not assigned in a parameter
file, it is given its default value.

There are three access levels available for parameters:

\begin{Lentry}
\item [{\tt Global}]
These parameters are seen by all thorns.
\item [{\tt Restricted}]
These parameters may be used by other implementations if they so desire.
\item [{\tt Private}]
These are only seen by this thorn.
\end {Lentry}

A parameter specification consists of:
\begin{itemize}

\item the parameter type (each may have an optional {\t CCTK\_} in front)
\begin{Lentry}
\item [{\tt REAL}]
\item [{\tt INT}]
\item [{\tt KEYWORD}]
A distinct string with only a few known allowed values.
\item [{\tt STRING}]
An arbitrary string, which must conform to a given regular expression.
\item [{\tt BOOLEAN}]
A boolean type which can take values 1, `t', `true', `yes' or 
0, `f', `false', `no'.
\end{Lentry}

\item a {\tt description} of the parameter

\item an allowed value block
This consists of a brace delimited block of lines
describing the allowed values of the parameter.  Each range may
have a description associated with it by placing a :: on the line and
putting the description afterwards.

\item the default value
This must be one of the allowed values.

\end{itemize}

For the numeric types {\tt INT} and {\tt REAL}, a range consists 
of a string of the
forms lower-bound:upper-bound:step where a missing number or a \* denotes
anything (i.e. infinite bounds or an infinitesimal step).

For example 

\begin{verbatim}

REAL Coeff "Important coefficient"
{
0:3.14 :: "Range has to be from zero to Pi, default is zero"
} 0.0

#No need to define a range for BOOLEAN
BOOLEAN nice "Nice weather ?"
{
}"yes"

# A example for a set of keywords and its default (which has to be
# defined in the body)
KEYWORD confused "Are we getting confused ?"
{
  "yes"    :: "absolutely positively"
  "perhaps" :: "we are not sure"
  "never"   :: "never"
} "never"
\end{verbatim}

defines a REAL parameter, a BOOLEAN parameter, and a KEYWORD.

By default all parameters are {\tt private}, to change this an access
specification of the form {\tt global:} or {\tt restricted:} (or 
{\tt private:} to change it back) may be placed on a line by itself.  This
changes the access level for any parameter defined in the file from that point on.

To access {\tt restricted} parameters from another implementation, a line
containing {\tt shares: <name>} declares that all parameters mentioned in
the file from now until the next access specification originate in 
implementation {\tt <name>}.  

In contrast to parameter declarations in other access blocks, the default
value must be omitted - it is impossible to set the default value of any
parameter not originating in this thorn.
For example, the following block adds possible values to the keyword
{\tt initial\_data} originally defined in the implementation {\tt einstein}.

\begin{verbatim}

shares:einstein

KEYWORD initial_data ""
{
  "bl_bh"         :: "Brill Lindquist black holes"
  "misner_bh"     :: "Misner black holes"
  "schwarzschild" :: "One Schwarzschild black hole"
}

\end{verbatim}

Note that you must compile at least one thorn which implements {\tt einstein}.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\subsection{The {\tt schedule.ccl}}

By default no routine of a thorn will be run.  The {\tt schedule.ccl} file
defines those that should be run and when they should be run.

The specification of this is via a schedule block which consists of
 lines of the form

\begin{verbatim}
schedule <name> at <time bin> [other options] 
{
  LANG:     <FORTRAN|C>
  STORAGE:  [group list]
  TRIGGERS: [group list]
  SYNC:     [group list]
} "A description"
\end{verbatim}
where {\tt <name>} is the name of the routine, and 

\begin{Lentry}

\item[{\tt <time bin>}] is one of
(in the following the initial {\tt CCTK\_} is optional):

\begin{Lentry}

\item [{\tt CCTK\_STARTUP}]
For routines run before the grid hierachy is set up, for example function
registration.

\item [{\tt CCTK\_PARAMCHECK}]
For routines which check parameter combinations, routines registered here
only have access to the grid size and the parameters.

\item [{\tt CCTK\_BASEGRID}]
Responsible for setting up coordinates etc.

\item [{\tt CCTK\_RECOVER}]
For recovery from checkpoint.

\item [{\tt CCTK\_INITIAL}]

For generating initial data.

\item [{\tt CCTK\_POSTINITIAL}]

Tasks which must be applied after initial data is created.

\item [{\tt CCTK\_CPINITIAL}]
For recovery of initial data from a checkpoint file.

\item [{\tt CCTK\_PRESTEP}]
Stuff done before the evolution step.

\item [{\tt CCTK\_EVOL}]      

The evolution step.

\item [{\tt CCTK\_POSTSTEP}]

Stuff done after the evolution step.

\item [{\tt CCTK\_CHECKPOINT}]
For checkpointing data

\item [{\tt CCTK\_ANALYSIS}]
For analysing data.

\item [{\tt CCTK\_TERMINATE}]
Called when cactus terminates.

\item [{\tt CCTK\_CONVERGENCE}]
Convergence stuff.

\end{Lentry}

These time bins can be given in the {\tt schedule.ccl} file with
or without the initial {\tt CCTK\_}, and in any case.

\item[{\tt STORAGE}] The {\tt STORAGE} keyword specifies any groups for 
which  memory should be
allocated for the duration of the routine.
The storage status reverts to its previous status after the 
routine returns.

\item[{\tt TRIGGERS}] {\tt TRIGGERS} is used when the routine is registered at {\tt ANALYSIS} --- 
this is a 
special time bin, a routine registered here will only be called if one of
the variables from a group in {\tt TRIGGERS} is due for output.

\item[{\tt SYNC}] 
The keyword {\tt SYNC} specifies groups of variables which should be
synchronised (that is, their ghostzones should be exchanged between 
processors) on exit from the routine. Specifying synchonisation of 
grid variables in {\tt schedule.ccl} is an alternative to calling the
function {\tt CCTK\_SyncGroup} from inside a routine. Using the {\tt SYNC}
keyword in the {\tt schedule.ccl} is the preferred method, since it 
provides the Flesh with more information about the behaviour of your code,
and in particular is a requirement for using a driver with adaptive mesh
refinement.

\item[{\tt other options}]
The `other options' allow finer grained control of the scheduling.  It is
possible to state that the routine must run {\tt BEFORE} or {\tt AFTER} 
another routine. It is also possible to schedule the routine under an
alias name by using {\tt AS <alias\_name>}.

\end{Lentry}

As well as schedule blocks it's possible to embed C style {\tt if/else} 
statements in the schedule.ccl.
These can be used to schedule things based upon the value of a parameter.

\vskip .5cm
{\bf Example I:}\\

The routine {\tt hydro\_predictor} is scheduled at {\em evolution}, after the 
routine {\tt metric\_predictor} and before {\tt metric\_corrector}, if
the parameter {\tt evolve\_hydro} has been set.
\begin{verbatim}

if(CCTK_Equals(evolve_hydro,''yes''))
{
  SCHEDULE hydro_predictor AT evol AFTER metric_predictor BEFORE metric_corrector 
  {
    LANG:     FORTRAN
    STORAGE:  hydro_variables
  } "Do a predictor step on the hydro variables"
}
\end{verbatim}

\vskip .5cm
{\bf Example II:}\\

The thorns {\tt WaveToy77} and {\tt WaveToyC} each provide an
routine to evolve the 3D wave equation: {\tt WaveToyF77\_Evolution} and
{\tt WaveToyC\_Evolution}. The routine names have to be different, so
that both thorns can be compiled at the same time, their functionlity
is identical though. Either one of them can then be activated at run
time  in the parameter file via {\tt ActiveThorns}.  

Since each evolution routine provides the same
functionality, it makes sense to schedule them under the common alias {\tt
WaveToy\_Evolution} to allow relative scheduling ({\tt
BEFORE/AFTER}) independent of the actual routine name (which may
change depending on the activation in the parameter file).

In both cases the group of variables {\tt scalarfield} are synchronised
across processes when the routine is exited.

\begin{verbatim}
schedule WaveToyF77_Evolution AS WaveToy_Evolution AT evol
{
  LANG: Fortran
  STORAGE: scalartmps
  SYNC: scalarfield
} "Evolution of 3D wave equation"
\end{verbatim}

\begin{verbatim}
schedule WaveToyC_Evolution AS WaveToy_Evolution AT evol
{
  LANG: C
  STORAGE: scalartmps
  SYNC: scalarfield
} "Evolution of 3D wave equation"
\end{verbatim}

The thorn {\tt IDScalarWave} schedules the routine {\tt WaveBinary}
after the alias {\em WaveToy\_Evolution}. It is scheduled independently of
the C or Fortran routine name.

\begin{verbatim}
schedule WaveBinary AT evol AFTER WaveToy_Evolution
{
  STORAGE: wavetoy::scalarevolve
  LANG:    Fortran
} "Provide binary source during evolution" 
\end{verbatim}

\subsubsection{Storage Outside of Schedule Blocks}

The keyword {\tt STORAGE} can also be used outside
of the schedule blocks to indicate that storage for these
groups should be switched on at the start of the run. Note that
the storage is only allocated in this way at the start,
a thorn could explicitly switch the storage off.




%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Naming Conventions for Source Files}
\label{nacofosofi}

The make system uses file extensions
to designate coding language. The following extensions can be handled:

\begin{center}
\begin{tabular}{|c|c|}
\hline
Extension & Coding Language \\
\hline
{\t .F} & Fortran90 fixed form \\
{\t .f} & (no preprocessing) Fortran90 fixed form\\
{\t .F90} & Fortran90 free form form\\
{\t .f90} & (no preprocessing) Fortran90 free form\\
{\t .F77} & Fortran77 \\
{\t .f77} & (no preprocessing) Fortran77\\
{\t .c} & C \\
{\t .cc} or {\t .C} & C++ \\
\hline
\end{tabular}
\end{center}

The following restrictions apply to file names:
\begin{itemize}
\item Currently all source files in different subroutines within a 
thorn must have distinct names.  We hope
to relax this in future.  Different thorns can have files with the same names. 
\end{itemize}

\section{Adding source files}

By default the CCTK looks in the {\tt src} directory of the thorn for source
files.

There are two ways in which to specify the sources.  The easiest is to use the 
{\tt make.code.defn} based method in which the CCTK does all the work, but you
may instead put a {\tt Makefile} in the {\tt src} directory and do everything
yourself.

\subsection{{\tt make.code.defn} based thorn building}
\label{sec:mabathbu}

This is the standard way to do compile your thorn's source files.
The Cactus make system looks for a file called {\tt make.code.defn} in that
directory (if there is no file called {\tt Makefile} in the {\tt src} directory).  At its simplest, this file contains two lines

\begin{itemize}
\item {\t SRCS = <list of all source files in this directory>}

\item {\t SUBDIRS = <list of all subdirectories, including subdirectories of subdirectories>}

\end{itemize}

Each subdirectory listed should then have a {\tt make.code.defn} file 
containing just a {\tt SRCS = } line, a {\tt SUBDIRS = } line will
be ignored.

In addition, each directory can have a {\tt make.code.deps} file, which,
for files in that directory, can contain additional make rules and dependencies
for files in that directory.  See the GNU Make documentation for complete details of the
syntax.

\subsection{{\tt Makefile} based thorn building}

This method gives you the ultimate responsibility.  
The only requirement is that
a library called {\tt \$NAME} be created by the {\tt Makefile}.

The makefile is passed the following variables

\begin{Lentry}

\item [{\tt \$(CCTK\_HOME)}]  the main Cactus directory

\item [{\tt \$(TOP)}] the configuration directory

\item [{\tt \$(SRCDIR)}] the directory in which the source files can be found

\item [{\tt \$(CONFIG)}]  the directory containing the configuration files

\item [{\tt \$(THORN)}]   the thorn name

\item [{\tt \$(SCRATCH\_BUILD)}]  the scratch directory where Fortran90 module
files should end up if they need to be seen by other thorns.

\item [{\tt \$(NAME)}]

\end{Lentry}

and has a working directory of {\tt <config>/build/<thorn\_name>} .

\subsection{Other makefile variables}

\begin{itemize}
\item CC
\item CXX
\item F90
\item F77
\item CFLAGS
\item CXXFLAGS
\item F90FLAGS
\item F77FLAGS
\item LD
\end{itemize}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\chapter{Cactus Variables}

Cactus variables are sorted into {\tt groups}.  All variables in a group have
the same attributes, and most cactus operations act on a group as a whole.

Cactus variables are used instead of local variables for two purposes.  Firstly
these variables can be made visable to other thorns, thus allowing thorns to
communicate and share data, secondly these variables can be communicated and
shared between processors, thus allowing parallel computation.

The full specification for a group declaration is

\begin{verbatim}
<data_type> <group_name> [TYPE=<group_type>] [DIM=<dim>] [TIMELEVELS=<num>] [SIZE=<size in each direction>] [DISTRIB=<distribution_type>] [GHOSTSIZE=<ghostsize>] [STAGGER=<stagger-specification>]
[{
   <variable_name>[,]<variable_name>
   <variable_name>
}] ["<group_description>"]
\end{verbatim}

\section{Data Type}

Cactus supports integer, real, complex and character variables, in various different
sizes.  (Sizes in the following refer to the number of bytes the type takes.)

\begin{Lentry}
\item[Integer]
CCTK\_INT, CCTK\_INT2, CCTK\_INT4, CCTK\_INT8.  CCTK\_INT defaults to being CCTK\_INT4, but
this can be changed at configuration time.
\item[REAL]
CCTK\_REAL, CCTK\_REAL4, CCTK\_REAL8, CCTK\_INT16.  CCTK\_REAL defaults to being CCTK\_REAL8, but
this can be changed at configuration time.
\item[COMPLEX]
CCTK\_COMPLEX, CCTK\_COMPLEX8, CCTK\_COMPLEX16, CCTK\_COMPLEX32.  CCTK\_COMPLEX defaults to being 
CCTK\_COMPLEX16, but this can be changed at configuration time.
\item[CHAR]
This is a 1 byte data type.
\end{Lentry}

Normally a thorn should use the default types --- CCTK\_INT, CCTK\_REAL, CCTK\_COMPLEX --- rather
than explicitly setting the size, as this gives maximum portability, and allows people to
compile the code in different precisions to test for round-off effects, or to run more
quickly with a lower accuracy.

\section{Group Types}

Groups can be either {\tt scalars}, {\tt grid functions (GFs)}, or {\tt grid arrays}.

\begin{Lentry}
\item[SCALAR] 
This is just a single number, e.g. the total energy of some field.  These
variables aren't communicated between processors --- what would be the
result of such communication ?
\item[GF]
This is the most common group type.  A GF is a an array with a specific size, set at
run-time in the parameter file, which is distributed across processors.  All GFs
have the same size, and the same number of ghostzones.
\item[ARRAY]
This is a more general form of the GF.  Each array can have a distinct size and number
of ghostzones.  The drawback of using an array over a GF is that a lot of data about the
array can only be determined by function calls, rather than the quicker methods available
for GFs.
\end{Lentry}

\section{Timelevels}

These are best introduced by an example.  People familier with
finite differencing can probably skip the next bit.

Take the 1-D wave equation

\begin{equation}
\frac{\partial^2 \phi}{\partial t^2} = \frac{\partial^2 \phi}{\partial x^2} 
\end{equation}

To solve this by partial differences, one discretises the derivatives, to get
an equation relating the solution at different times.  There are many ways
to do this, one of which produces the following difference equation

\begin{equation}
\label{equation:difference}
\phi(t+\Delta t,x) -2\phi(t,x) +\phi(t-\Delta t,x) = \frac{\Delta t}{\Delta x^2} \lbrace{\phi(t,x+\Delta x) -2\phi(t,x) +\phi(t,x-\Delta x)}\rbrace
\end{equation}

Which relates the three timelevels $t+\Delta t$, $t$, and $t-\Delta t$.  

Obviously the code could just use three variables, one for each timelevel.  This turns out,
however, to be inefficient as, as soon as the time is incremented to $t+\Delta t$, it
would be necessary to copy data from the $t$ variable to the $t-\Delta t$ variable and from
the $t+\Delta t$ variable to the $t$ variable.

To remove this extraneous copy, Cactus allows you to specify the number of timelevels used by
your numerical scheme.  It then generate variables with the base name (e.g. {\tt phi}) suffixed
by a qualifier for which timelevel is referred to --- {\tt \_n} for the {\tt n}ext timelevel, just 
the base name for the current timelevel, and an {\tt \_p} for each {\tt p}revious timelevel.  E.g.
a four timelevel scheme would produce phi\_n, phi, phi\_p and phi\_p\_p .  The only exception to this
is for a two timelevel scheme, which has no previous timelevel, just a current and a next.

\section{Size and Distrib}

A Cactus array can have a size set at runtime by parameters.  This size can either be the
total size of the array across all processors, or, if {\tt DISTRIB=CONSTANT} the specified
size on each processor.  If the size is split across processors, the driver thorn is
responsible for assigning the size on each processor.

\section{Ghost Size}

Cactus is based upon a {\tt distributed computing} paradigm.  I.e. the problem
domain is split into blocks, each of which is assigned to a processor.  For 
hyperbolic and parabolic problems the blocks only need to communicate at the edges.

To illustrate this, take the example of the wave equation again, equation \ref{equation:difference}  
describes a possible time-evolution scheme.  On examination you can see that to
generate the data at the point ($t+\Delta t$, $x$) we need data from the four points
($t$, $x$), ($t-\Delta t$, $x$), ($t$, $x+\Delta x$) and ($t$, $x-\Delta x$) only.
Ignoring the points at $x$, which are obviously always available, you can see that the
algorithm requires a point on either side of the point $x$, i.e. this algorithm
has {\tt stencil size} 1.  Similarly algorithms requiring two points on either side
have stencil size 2, etc.  

Now, if you evolve the above scheme, it becomes apparent that at each iteration the number
of grid points you can evolve decreases by one at each edge (see figure \ref{fig:noghost}).

\begin{figure}[h]
\begin{center}
\includegraphics[angle=0,width=8cm]{1dnoghost.eps}
\end{center}
%\caption[]{} 
% I know this figure is rubbish, but it's after 2am and I'll fix it up later.
\label{fig:noghost}
\end{figure}

At the outer boundary of the physical domain the data for the boundary point can be
generated by the boundary conditions, however at internal boundaries the data has to
be copied from the adjacent processor.  It would be inefficient to copy each point 
individually, so instead, a number of {\tt ghostzones} are created at the internal boundaries.
A ghostzone consists of a copy of the whole plane (in 3d, line in 2d, point in 1d) of the data 
from the adjacent processor.  I.e. the array on each processor is augmented with copies of
points from the adjacent processors, thus allowing the algorithm to proceed {\tt on the 
points owned by this processor} without having to worry about copying data.  Once the
data has been evolved one step, the data in the ghostzones can be exchanged (or 
{\tt synchronised}) between processors in one fell swoop before the next evolution step.
(See figure \ref{fig:withghost}.)  Note that you should have at least as many ghostzones as
your stencil-size requires.

\section{Staggering}

The staggering of a gridfunction or array describes the {\em physical} 
placement of that gridfunction relative to the supporting grid
structure. For example, a gridfunction does not have to 
be placed at the intersection 
of the ``grid lines''. It can moved by half a grid spacing in
any or all dimensions. In the latter case, it will be placed in
the center of a cell.  

The staggering of grid function is a pure {\em physical} property: 
the values will be calculated at a different position in physical
space. Still the indexing (or bookeeping)  is kept the same for all
types of staggerings: the indexing of the default unstaggered grids is 
used.

\vskip .25cm

{\bf Specifying the staggertype}

The type of staggering applied to a gridfunction can be specified in
the {\tt interface.ccl} file by the attribute {\tt stagger} (see
\ref{sec:in}). Cactus supports three kind of staggering
per dimension. The physical location of a gridfunction is shifted 
relative to the default position by adding the following values to the 
stagger attribute:
\begin{Lentry}
\item[{\tt M}] no staggering, default. Refers to the ``minus'' face
relative to  the default gridpoint. 
\item[{\tt C}] centre staggering. The physical location is offset by
half of the grid spacing in the positive direction (or to the right).
\item[{\tt P}] full staggerd. P refers to plus. The physical location
is offset by a full gridspacing in the positive direction (or the
right).
\end{Lentry}
For multi dimensional gridfunctions you concatenate the code
characters in xyz order. In picture xy we show four different
staggerings of a two dimensional grid functions. The solid black grid
circles show the default location of the grid functions at the
intersections of the grid lines. In (A) we show an additional grid
function of type {\tt stagger=MC}: no staggering in x direction,
center staggered in y direction. In (B) we have  {\tt stagger=CM} and
staggering each direction ({\tt stagger=CC}) is shown in (C). The full
staggering in (D) ({\tt stagger=PP}) obeyes the same rules, but is
rather unusual; included here for completeness. 

\begin{figure}[h]
  \def\epsfsize#1#2{0.45#1}
\begin{center}
\includegraphics[angle=0,width=8cm]{staggering1.eps}
%  \centerline{\epsfbox{./staggering1.eps}}
\end{center}
\caption[]{\small {\bf Staggered gridpoints in 2D} for several
staggerings. (a) : {\tt MC}, (b): {\tt CM}, (c): {\tt CC}, (d): {\tt
PP}. Note that the staggering of gridfunctions does change its
index. The staggered gridpoints and the corresponding unstaggered
points (arrows) are accessed by the same indices.} 
\label{fig:stagger2}
\end{figure}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\chapter{Cactus Parameters}

Parameters are the means by which the user specifies the run-time behaviour of
the code.  The user specifies values for parameters in the parameter file, and
then the flesh validates these values against the ranges the thorn writers have
specified to be valid.  Once validated, parameters values are fixed, and cannot
be changed (unless the parameter is specified to be steerable, see below).

The full specification for a parameter decalaration is

\begin{verbatim}
[EXTENDS|USES] <parameter_type> <parameter name> "<parameter description>" 
{
  <PARAMETER_RANGES>
} <default value>
\end{verbatim}

\section{Types and Ranges}

Parameters can be of several types:

\begin{Lentry}
\item[Integer]  Can take any integral value
\item[Real] Can take any floating point value
\item[Keyword] Can have a value consisting of one of a choice of strings
\item[Boolean] Can be true or false
\item[String] Can have any string value
\end{Lentry}

Each parameter can be validated against a set of allowed values or ranges, each 
of which has a description associated with it.  The type of range is determined 
by the type of parameter.

\subsection{Integer}

\subsection{Real}

\subsection{Keyword}

\subsection{Boolean}

\subsection{String}


\section{Scope}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\chapter{Scheduling}

Cactus contains a rule-based scheduling system, which determines which
routines from which thorn are run in which order.  The scheduler 
determines if the specifications are inconsistent, but does allow the
user to schedule a routine with respect to another routine which may not
exist.

The full specification for a schedule declaration is

\begin{verbatim}
schedule [GROUP] <function name> AT|IN <time> [BEFORE|AFTER <group>] [WHILE <variable>] [AS <alias>]
{
  LANG: <language>
  [STORAGE:       <group>,<group>...]
  [TRIGGER:       <group>,<group>...]
  [SYNC:          <group>,<group>...]
  [OPTIONS:       <option>,<option>...]
} "Description of function"
\end{verbatim}


This consists of a set of options, and the main body, referred to as the {\tt schedule block}.

\section{Schedule Options}


\section{The Schedule Block}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\chapter{Putting code into your thorn}

\section{What the Flesh provides}

The flesh provides for thorns.
\begin{Lentry}
\item [{\tt Variables}]
\item [{\tt Parameters}]
\item [{\tt Cactus Functions}]

\begin{itemize}
  \item{} driver (parallelisation) utilities
  \item{} IO utilities
  \item{} Coordinates utilities
  \item{} Reduction utilities
  \item{} Interpolation utilities
  \item{} Information utilities
\end{itemize}
\end{Lentry}


\subsection{Fortran Routines}

Any source file using Cactus infrastructure should include 
the header file {\tt cctk.h} using the line
\begin{verbatim}
#include "cctk.h"
\end{verbatim}
(Fortran programmers should not be put of by this being a C style
header file, most Cactus files are run through a C preprocessor
before compilation)

\subsubsection{Variables}

Any routine using Cactus argument lists (for example all 
routines called from the scheduler) should include at the
top of the file the header
\begin{verbatim}
#include "cctk_Arguments.h"
\end{verbatim}

A Cactus macro {\tt CCTK\_ARGUMENTS} is defined for each thorn
to contain
\begin{itemize}
\item General information about the grid hierarchy, for example
      the number of grid points used. See Section \ref{sec:cava} for a 
      complete list.
\item All the grid variables defined in the thorns {\tt interface.ccl}
\item All the grid variables required from other thorns as requested by
      the {\tt inherits} and {\tt friend} lines in the {\tt interface.ccl}
\end{itemize}
These variables must be declared at the start of the routine using
the macro {\tt DECLARE\_CCTK\_ARGUMENTS}. 

To pass the arguments to another routine in the same thorn use the macro 
{\tt CCTK\_PASS\_FTOF} in the calling routine, and again the macro 
{\tt CCTK\_ARGUMENTS} in the receiving routine.


\subsubsection{Parameters}

All parameters defined in a thorn's {\tt param.ccl} and all {\tt global} 
parameters appear as local variables in a thorn.  These variables are 
{\tt read only} and {\em changes should not be made to them}.  The effect of
changing a parameter is undefined (at best).

The parameters should be declared at the start of the routine
using them with the macro {\tt DECLARE\_PARAMETERS}

Any routine using Cactus parameters should include at
the top of the file the header
\begin{verbatim}
#include "cctk_Parameters.h"
\end{verbatim}

In Fortran, special care should be taken with string valued parameters.
This parameters are passed as C pointers, and can not be treated as
normal Fortran strings. To compare a string valued parameter and Fortran
string use the function {\tt CCTK\_Equals}. To print the value of a string 
valued parameter to screen, use the subroutine {\tt CCTK\_PrintString}.

\subsubsection{Fortran Example}

The Fortran routine {\tt MyFRoutine} is scheduled in the {\tt schedule.ccl} file,
doesn't use Cactus parameters, and calls another routine, in the same thorn,  
{\tt MyNewRoutine} which does use parameters. 
This routine needs to be passed an integer flag as 
well as the standard Cactus variables. The source file should look like

\begin{verbatim}
#include "cctk.h"
#include "cctk_Arguments.h"
#include "cctk_Parameters.h"

      subroutine MyFRoutine(CCTK_ARGUMENTS)

c     I'm very cautious, so I want to declare all variables
      implicit none 

      DECLARE_CCTK_ARGUMENTS

      integer flag

      flag = 1
      call MyNewRoutine(CCTK_PASS_FTOF,flag)

      return 
      end

      subroutine MyNewRoutine(CCTK_ARGUMENTS,flag)

      implicit none 

      DECLARE_CCTK_ARGUMENTS
      DECLARE_CCTK_PARAMETERS
      integer flag

c     Main code goes here

      return
      end
      
\end{verbatim}

\subsubsection{Cactus Fortran Functions}

Cactus Fortran functions, for example {\tt CCTK\_MyProc}, can
all be declared by adding before any executable code, the declaration

\begin{verbatim}
DECLARE_CCTK_FUNCTIONS
\end{verbatim}

\subsubsection{Fortran 90 Modules}

Fortran 90 modules should be included in a thorn's {\tt make.code.deps} file
(\ref{sec:mabathbu}) to ensure they are compiled before the
routines which use them. This is especially important for parallel 
building. For example, if a routine in {\tt MyRoutine.F} uses a module
in {\tt MyModule.F} add the line:
{\tt
\begin{verbatim}
$(SYS_OBJD)/MyRoutine.F.o:         $(SYS_OBJD)/MyModule.F.o  
\end{verbatim}
}

\subsubsection{The {\tt MOD} function}

The intrinsic function {\tt MOD} in Fortran takes two integer 
arguments, which should both be of the same type. This means
that it may be necessary to cast the arguements to {\it e.g}
{\tt INT} for some architectures. This can occur in particular
when a {\tt CCTK\_INT} parameter and the Cactus variable {\tt cctk\_iteration}
(which is declared to be {\tt INTEGER}) are used, 
in which case the correct code is
{\tt
\begin{verbatim}
MOD(cctk_iteration,INT(MyParameter))
\end{verbatim}
}

\subsection{C Routines}

Any source file using Cactus infrastructure should include 
the header file {\tt cctk.h} using the line
\begin{verbatim}
#include "cctk.h"
\end{verbatim}

\subsubsection{Variables}

Any routine using Cactus argument lists (for example all 
routines called from the scheduler) should include at the
top of the file the header
\begin{verbatim}
#include "cctk_Arguments.h"
\end{verbatim}

A Cactus macro {\tt CCTK\_ARGUMENTS} is defined for each thorn
to contain
\begin{itemize}
\item General information about the grid hierachy, for example
      the number of grid points on the processor. See Section \ref{sec:cava} for a 
      complete list.
\item All the grid variables defined in the thorns {\tt interface.ccl}
\item All the grid variables required from other thorns as requested by
      the {\tt inherits} and {\tt friend} lines in the {\tt interface.ccl}
\end{itemize}
These variables must be declared at the start of the routine using
the macro {\tt DECLARE\_CCTK\_ARGUMENTS}. This macro should always be the
first line of the routine.

To pass the arguments to another routine in the same thorn use the macro 
{\tt CCTK\_PASS\_CTOC} in the calling routine, and again the macro 
{\tt CCTK\_ARGUMENTS} in the receiving routine.


\subsubsection{Parameters}

All parameters defined in a thorn's {\tt param.ccl} and all {\tt global} 
parameters appear as local variables in a thorn.  These variables are 
{\tt read only} and {\em changes should not be made to them}.  The effect of
changing a parameter is undefined (at best).

Any routine using Cactus parameters should include at
the top of the file the header
\begin{verbatim}
#include "cctk_Parameters.h"
\end{verbatim}

The parameters should be declared at the start of the routine
using them with the macro {\tt DECLARE\_CCTK\_PARAMETERS}

\subsubsection{Example}

The C routine "MyCRoutine" is scheduled in the {\tt schedule.ccl} file,
and uses Cactus parameters. The source file should look like
\begin{verbatim}
#include "cctk.h"
#include "cctk_Arguments.h"
#include "cctk_Parameters.h"

void MyCRoutine(CCTK_ARGUMENTS)
{
  DECLARE_CCTK_ARGUMENTS
  DECLARE_CCTK_PARAMETERS

  /* Here goes your code */
}
\end{verbatim}

\subsubsection{Complex variables}

Cactus supports complex grid variables, and since there is no 
complex data type in C, Cactus provides a number
of functions for manipulating complex numbers to mirror the 
functionality available in Fortran. These functions are {\tt CCTK\_Cmplx},
{\tt CCTK\_CmplxReal}, {\tt CCTK\_CmplxImag}, {\tt CCTK\_CmplxConjg},
{\tt CCTK\_CmplxAdd}, {\tt CCTK\_CmplxSub}, {\tt CCTK\_CmplxMul},
{\tt CCTK\_CmplxDiv}, {\tt CCTK\_CmplxExp}, {\tt CCTK\_CmplSin},
{\tt CCTK\_CmplxAbs}, {\tt CCTK\_CmplxLog}, {\tt CCTK\_CmplSqrt}. 


\subsubsection{Specifically for C Programmers}

Grid functions are held in memory as 1D C arrays. These are laid
out in memory as in Fortran. This means that the first index should
be incremented through most rapidly. As can be seen in the example
below.


Cactus provides
macros to find the 1D index which is needed from the multidimensional
indices which are usually used. There is a macro for each dimension of
grid function.  Below is an articifial example to demonstrate this
using the 3D macro {\tt CCTK\_GFINDEX3D}:
\begin{verbatim}
for (k=0; k<cctk_lsh[2]; k++)
{
  for (j=0; j<cctk_lsh[1]; j++)
  {
    for (i=0; i<cctk_lsh[0]; i++)
    {
      My3D_GF[CCTK_GFINDEX3D(cctkGH,i,j,k)] = i*j*k;
    }
  }
}
\end{verbatim}

Here, {\tt CCTK\_GFINDEX3D(cctkGH,i,j,k)]} expands to 

\begin{verbatim}
((i) + cctkGH->cctk_lsh[0]*((j)+cctkGH->cctk_lsh[1]*(k)))
\end{verbatim}

\subsection{Cactus Variables}
\label{sec:cava}

The Cactus variables which are passed through the macros
{\tt CCTK\_ARGUMENTS}, {\tt CCTK\_ARGUMENTS}, etc are
\begin{Lentry}
\item [{\tt cctk\_dim}] An integer with the number of dimensions
      used for this grid hierarchy
\item [{\tt cctk\_lsh}] An array of {\tt cctk\_dim} integers
      with the local grid size on this processor
\item [{\tt cctk\_delta\_time}] A {\tt CCTK\_REAL} with the timestep 
\item [{\tt cctk\_time}] A {\tt CCTK\_REAL} with the current time
\item [{\tt cctk\_delta\_space}] An array of {\tt cctk\_dim} {\tt CCTK\_REAL}s with
         the grid spacing in each direction
\item [{\tt cctk\_nghostzones}] An array of {\tt cctk\_dim} integers with
         the number of ghostzones used in each direction
\item [{\tt cctkGH}] A C pointer identifying the grid hierachy
\end{Lentry}

The following variables describe the location of the local
grid (e.g. the grid treated on a given processor) within 
the global grid.
\begin{itemize}
\item {\tt cctk\_lbnd} 
      An array of {\tt cctk\_dim} integers 
      containing the lowest index (in each direction) 
      of the local grid, as seen on the global grid. Note that these indices
      start from zero, so you need to add one when using them in
      Fortran thorns.
\item {\tt cctk\_ubnd} 
      An array of {\tt cctk\_dim} integers 
      containing the largest index (in each direction) 
      of the local grid, as seen on the global grid.  Note that these indices
      start from zero, so you need to add one when using them in
      Fortran thorns.
\item {\tt cctk\_bbox} 
      An array of 2*{\tt cctk\_dim} integers 
      which indicate whether the boundaries are internal boundaries
      (e.g. between processors), or physical boundaries.
\end{itemize}

The following variable is needed for grid refinement methods
\begin{itemize}
\item {\tt cctk\_levfac} The factor by which the local grid is refined
        with respect to the base grid.
\end{itemize}

The following variable is used for identifing convergence levels. {\tt NOTE:} Convergence is not currently implemented by Cactus, so that {\tt cctk\_convlevel} is currently always 0.
\begin{itemize}
\item {\tt cctk\_convlevel} The convergence level of this grid hierachy. 
 	The base level is 0, and every level above that is currently coarsened 	       by a factor of 2.
\end{itemize}
 
The variables {\tt cctk\_delta\_space} and {\tt cctk\_delta\_time}
denote the grid spacings on the {\em base} grid. If you are using 
a grid refinement method, you need yo calculate the grid spacings
on the grid you are on. There are Cactus macros provided for this,
with the syntax {\tt CCTK\_DELTA\_SPACE(dir)} and {\tt CCTK\_DELTA\_TIME}
for both C and Fortran. It is recommended that these macros are
always used to provide the grid spacings in your thorns. 

\subsection{Cactus Data Types}

The Cactus grid variables and parameters are defined and
declared using Cactus data types, to provide portability 
across platforms. The most important of 
these data types are described below, for a full description
see Section~\ref{sec:datyansi}. These data types should 
be used to declare local variables where needed, and to
declare Cactus grid variables or parameters that need 
declarations. 

\begin{Lentry}

\item[{\tt CCTK\_INT}] default size 4 bytes
\item[{\tt CCTK\_REAL}] default size 8 bytes

\end{Lentry}

\subsubsection{Example}

In the following example {\tt MyScalar} is a grid scalar which
is declared in the {\tt interface.ccl} as {\tt CCTK\_REAL}.

\begin{verbatim}
      subroutine InitialData(CCTK_ARGUMENTS)

      DECLARE_CCTK_ARGUMENTS

      CCTK_REAL local_var
      
      local_var = 1.0/3.0      
      MyScalar = local_var

      return
      end 
\end{verbatim}

Declaring {\tt local\_var} to have a non-Cactus data type, e.g.
{\tt REAL*4}, or using one of the other Cactus real data types
described in Section~\ref{sec:datyansi} could give problems for 
different architectures or configurations.

\subsection{Staggering}
\label{sec:st}


{\bf Indexing, ghostzones, etc.}
Note that staggering does not make any changes to the indexing of a
gridfunction: the black solid circles in diagramm \ref{fig:stagger2} and their
associated staggered gridfunctions (connected by arrows) have the same index!

Since the gridfunction does not ``know'' anything about the physical
location (it's only addressed by indeces) why add staggering if the
indexing is the same ?  

Indeed, you could roll your own, but there compelling reasons:
Readabilty and you are able to query the staggertype of a
gridfunction. More important: in the way the grid is laid out, there is one grid
point {\em less} for {\tt M} and {\tt P} staggered grid functions. This is
illustrated in picture \ref{fig:stagger2}, which shows 15 gridpoints distributed
across 3 processors. The solid black cirlces show the default
location of the gridfunctions, the grey circles depict the ghostzones.
Note that the number of center staggered gridpoints (fat crosses)
corresponds to the number of default gridpoint on all processors but
the last one. (Same is true for full staggered gridpoints).

{\bf Staggertypes}
The string specifying the staggering is encoded in a number called
the {\em staggerindex}. With the 3 supported staggerings, the string
is converted into a base-3 number. Several routines exist, to extract the
staggering in a specific direction, called {\em directional
staggerindex}. E.g. {\tt stagger = MCM}: {\em staggerindex} = 3, in the 
x-direction: {\em directional staggerindex} = CCTK\_STAGGER\_M (value 0),  in the 
x-direction: {\em directional staggerindex} = CCTK\_STAGGER\_C (value 1).

\begin{Lentry}
\item[{\tt CCTK\_STAGGER\_M}]  value used for M-type staggering
\item[{\tt CCTK\_STAGGER\_C}]  value used for C-type staggering
\item[{\tt CCTK\_STAGGER\_P}]  value used for P-type staggering
\item[{\tt CCTK\_NO\_STAGGER}] value to indicate no staggering
\item[{\tt CCTK\_STAGGER}]    value to indicate staggering
\item[{\tt CCTK\_NSTAGGER}]   number of coded staggerings (3)
\item[{\tt CCTK\_STAGGER\_ERROR}] failed stagger operation , negative
\end{Lentry}


\begin{figure}[h]
  \def\epsfsize#1#2{0.45#1}
\begin{center}
\includegraphics[angle=0,width=10cm]{staggering2.eps}
%  \centerline{\epsfbox{./staggering2.eps}}
\end{center}
\caption[]{\small {\bf Unstaggered and center-staggered gridpoints} with
ghostzone size of one (above) and two (below). The points are
distributed across three processors. Note that the number of  
center staggered gridpoints (fat crosses) is one less on the outermost grid. How to
treat this case in a easy way is explained below. }
\label{fig:stagger2}
\end{figure}

When a thorn programmer uses staggered gridpoints, he has to be aware
of this gridpoint anomaly. This can be done easiest by using the {\tt
CCTL\_LSSH(<dir\_staggertype>,<direction>)} macro. For a given staggertype
and direction, this 2d array returns the local number of gridpoints,
including  ghostzones and the necessary change for the staggering on
the outermost processor. 

\begin{Lentry}
\item[{\tt CCTL\_LSSH(<dir\_staggertype>,<direction>)}]
for a given staggertype and a direction this macro returns the number
of processor local gridpoints, including ghostzones. 

\begin{itemize}
\item{macro has to be in capital letters}
\item{This macro is C/Fortran indexing aware:
can specify the dimension in C ranging from $0 \ldots$ and in Fortan
ranging from $1 \ldots$.}
\end{itemize}
\end{Lentry}

\vskip .25cm

Several functions exist to derive the staggertype for a given group
and for a certain direction.
\begin{Lentry}
\item[{\tt int CCTK\_GroupStaggerIndexGI(int group\_index)}] returns the
{\em staggerindex} for a given group index. 
\item[{\tt call CCTK\_GroupStaggerIndexGI(int staggerindex, int
group\_index)}]  returns the {\em staggerindex} for a given group index. 
\end{Lentry}
\vskip .45cm

\begin{Lentry}
\item[{\tt int CCTK\_GroupStaggerIndexGN(char *group\_name)}] returns the
{\em staggerindex} for a given group name. 
\item[{\tt call CCTK\_GroupStaggerIndexGN(int staggerindex, char *group\_name)}] returns the
{\em staggerindex} for a given group name. \vskip .25cm
\end{Lentry}
\vskip .45cm

\begin{Lentry}
\item[{\tt int CCTK\_GroupStaggerIndexVI(int variable\_index)}] returns the
{\em staggerindex} for a given variable index. 
\item[{\tt call CCTK\_GroupStaggerIndexVI(int staggerindex, int variable\_index)}] returns the
{\em staggerindex} for a given variable index. 
\end{Lentry}
\vskip .45cm

\begin{Lentry}
\item[{\tt int CCTK\_GroupStaggerIndexVN(char *variable\_name)}] returns the
{\em staggerindex} for a given variable name. 
\item[{\tt call CCTK\_GroupStaggerIndexVN(int staggerindex, char *variable\_name)}] returns the
{\em staggerindex} for a given variable name. 
\end{Lentry}
\vskip .45cm

\begin{Lentry}
\item[{\tt int CCTK\_StaggerIndex(char *stagger\_string)}] return the {\em
staggerindex} for a given stagger string.
\item[{\tt call CCTK\_StaggerIndex(int staggerindex, char *stagger\_string)}] return the {\em
staggerindex} for a given stagger string. 
\end{Lentry}
\vskip .45cm

\begin{Lentry}
\item[{\tt int CCTK\_StaggerIndex(char *stagger\_string)}] return the {\em
staggerindex} for a given stagger string.
\item[{\tt call CCTK\_StaggerIndex(int staggerindex, char *stagger\_string)}] return the {\em
staggerindex} for a given stagger string.
\end{Lentry}
\vskip .45cm

\begin{Lentry}
\item[{\tt int CCTK\_DirStaggerIndex(int direction, char *stagger\_string)}] 
returns the {\em directional staggerindex} for a given direction and
stagger string.
\item[{\tt call CCTK\_DirStaggerIndex(int dir\_staggerindex, int direction, char *stagger\_string)}] 
returns the {\em directional staggerindex} for a given direction and
stagger string.
\end{Lentry}
\vskip .45cm

\begin{Lentry}
\item[{\tt int CCTK\_DirStaggerIndexI(int direction, char *stagger\_type)}]
returns the {\em directional staggerindex} for a given direction and 
staggerindex.
\item[{\tt call CCTK\_DirStaggerIndexI(int dir\_direction, char *stagger\_type)}]
returns the {\em directional staggerindex} for a given direction and 
staggerindex.

\end{Lentry}

\subsection{Providing Run Time Information}

To write from thorns to standard output ({\it i.e.} the screen) 
at runtime, use the function {\tt CCTK\_INFO}. 
For example, from the Fortran thorn {\tt MyThorn},

\begin{verbatim}
call CCTK_INFO("Starting Tricky Calculation")
\end{verbatim}

will write the line:

\begin{verbatim}
INFO (MyThorn): Starting Tricky Calculation
\end{verbatim}

For a multiprocessor run, only information from processor zero 
will appear. If the "{\tt -r"} command line option is used 
(Section~\ref{sec:coliop}) the output from this command will be redirected.

To output a variable using {\tt CCTK\_INFO} currently more tricky. From Fortran you will need something like

\begin{verbatim}
write(infoline,'(A,1X,I)') 'The integer was ',inum
call CCTK_INFO(infoline)
\end{verbatim}

and for C

\begin{verbatim}
char *infoline;
infoline = (char *)malloc(18*sizeof(char));
sprintf(infoline,'The integer was %d',inum);
CCTK_INFO(infoline);
free(infoline);
\end{verbatim}



Notes
\begin{itemize}
\item{} {\tt CCTK\_INFO} is actually a macro for the function
        {\tt CCTK\_Info(<thorn name>,<message>)} which automatically
        includes the thorn name.

\item{} {\tt CCTK\_INFO} should be used rather than print statements,
       since it will give consistent behaviour on multiprocessors, and
       also provides a mechanism for switching the output to screen on 
       and off, even on a thorn-by-thorn basis. (Although this is 
       not yet implemented).
\end{itemize}

\section{Memory Tracing}
\label{sec:metr}

Cactus provides a mechanism for overriding the standard C memory allocation 
routines ({\tt malloc, free,} \ldots) with Cactus specific routines, 
that track the amount of memory allocated and from where the
allocation call was made. This information can be accessed by the user to
provide an understanding of the memory consumption between two instances and
to track down possible memory leaks. This feature is available in C only.

\subsection{Activating Memory Tracing}
\label{sec:acmetr}

Memory tracing has to be activated at configure time. The
standard malloc statements are overriden with macros ({\tt CCTK\_MALLOC}). 
To activate memory tracing use either

\begin{Lentry}
\item[{\tt DEBUG=all}]  Enables all debug options (compiler debug
flags, redefines malloc)
\item[{\tt DEBUG=memory}] Redefine malloc only.
\end{Lentry}

The {\tt CCTK\_MALLOC} statements can also be used directly in the C
code. But by employing them this way, only a fraction of the total
memory consumption is traced. Also, they cannot be turned off at
configure time. (this behavior might change). For example:
\begin{verbatim}
machine> gmake bigbuild DEBUG=yes
 
machine> gmake bigbuild-config DEBUG=memory
\end{verbatim}
The new configuration {\tt bigbuild} is configured with all debugging features
turned on. The already existing configuration {\tt bigbuild} is
reconfigured with memory tracing only.

\subsection{Using Memory Tracing}
\label{sec:usmetr}

You can  request Cactus to store the memory consumption at a certain
instance in the program flow and return the difference in memory
allocation some time later.

\begin{Lentry}
\item[{\tt int CCTK\_MemTicketRequest(void)}]
	Request a ticket: save the current total memory to a database.
	Return an integer (ticket). Use the ticket to calculate the
	difference in memory allocation between the two instances in
 	CCTK\_MemTicketCash.	
 
\item[{\tt long int CCTK\_MemTicketCash(int your\_ticket)}]
	Cash in your ticket: return the memory difference between now and the
     	time the ticket was requested. Tickets can be cashed in
	several times. See Example below.
     	This only tracks the real data memory, which is the same as in
     	undebug mode. It does not keep track of the internal allocations
     	done to provide the database, motivation is that this is not
 	allocated either if you compile undebugged.

\item[{\tt int CCTK\_MemTicketDelete(int your\_ticket)}]
	Delete the memory ticket. The ticket-id will not be reused, since
        it's incremented with every ticket request, but the memory of
	the memory datastructure is deallocated.

\item[{\tt unsigned long int CCTK\_TotalMemory(void)}]
	Returns the total allocated memory (not including the tracing
        data structures). 

\item[{\tt void CCTK\_MemStat}] Prints an info string, stating the current, 
 	past and total memory (in bytes) allocation between two
	successive calls to this routine, as well as the difference. 
\end{Lentry}

Sample C Code demonstrating the ticket handling. Two tickets are
requested during malloc operations. The {\tt CCTK\_MALLOC} statement is 
used directly. They are cashed in and the memory
difference is printed. Ticket 1 is cashed twice. The tickets are
deleted at the end.
\begin{verbatim}

int ticket1;
int ticket2;

/* store current memstate, ticket: t1*/
t1 = CCTK_MemTicketRequest();

/* allocate data */	
hi = (int*) CCTK_MALLOC(10*sizeof(int));

/* store current memstate, ticket: t2*/
t2 = CCTK_MemTicketRequest();      
	
/* cash ticket t1, print mem difference */	
printf("NOW1a: %+d \n",CCTK_MemTicketCash(t1)); 

/* allocte some more data */
wo = (CCTK_REAL*)CCTK_MALLOC(10*sizeof(CCTK_REAL));
	
/* cash ticket t1 and t2, print mem difference */	
printf("NOW1b: %+d \n",CCTK_MemTicketCash(t1));  
printf("NOW2 : %+d \n",CCTK_MemTicketCash(t2));    

/* delete the tickets from the database */
CCTK_MemTicketDelete(t1);
CCTK_MemTicketDelete(t2);

\end{verbatim}

\section{Error handling, Warnings and Code Termination}
\label{sec:erhawancote}
The Cactus function {\tt CCTK\_WARN} should be used to provide
warning messages during code execution. Along with the
warning message, an integer is given to indicate the severity
of the warning. The warning severity indicates whether the
message is printed to standard output and whether the code
should be stopped. A level 0 warning indicates the highest
severity, with higher numbers indicating lower severity.

By default, a Cactus run will abort on a level 0 warning
and will report level 1 and severer warnings to screen. 
This behaviour can be amended using command line arguments,
as described in Section~\ref{sec:coliop}.

The flesh will be implementing standard error return codes
which can be used by the thorns, although this is not 
yet ready. In general, thorns should attempt to handle errors
without terminating, and warning messages should be liberally
used.
      

\subsection{Adding Parallelisation}
\label{seap}

The flesh itself does not actually set up grid variables. This 
is done by a {\it driver} thorn. To allow the distribution of 
a grid over a number of processors, the driver thorn must 
also provide the grid decomposition, and routines to enable 
parallelisation. The method used to provide this parallelisation
(e.g. MPI, PVM) is not usually important for the thorn writer since
the driver thorn provides routines which are called by standard interfaces
from the flesh. Here we describe briefly the most important of these routines
for the application thorn writer. A more detailed description
of these interfaces with their arguments, is given in the Function Reference
Guide.
A complete description of the
routines a driver thorn must provide will be provided in the 
Interface Thorn Writers guide. The standard driver thorn is 
currently {\tt PUGH} in the {\tt CactusPUGH} package, which 
is a parallel unigrid driver. 

\begin{Lentry}
\item[{\tt CCTK\_nProcs}] Returns the number of processors being used
\item[{\tt CCTK\_MyProc}] Returns the processor number (this starts at
  processor number zero
\item[{\tt CCTK\_SyncGroup}] Synchronises a group of variables by 
  exchanging the values held in each processor ghostzones with the
  physical values of their neighbours
\item[{\tt CCTK\_Barrier}] Waits for all processors to reach this point
  before proceeding
\end{Lentry}

\subsection{Building Include Files}

Cactus provides a mechanism for thorns to add code to 
include files which can be used by any other thorn.
Such include files can contain executable source code, or header/declaration
information. A difference is made between these two cases, since included
executable code is protected from being run if a thorn is compiled but
not active by being wrapped by a call to {\tt CCTK\_IsThornActive}. 


Any thorn 
which uses the include file must declare this in its 
{\tt interface.ccl} with the line

\begin{verbatim}
USES INCLUDE [SOURCE|HEADER]: <file_name>
\end{verbatim}

Any thorn which wishes to add to this include file, 
declares in its own {\tt interface.ccl}

\begin{verbatim}
INCLUDE [SOURCE|HEADER]: <file_to_include> in <file_name>
\end{verbatim}

\subsubsection{Example}

For an example of this in practice, for the case of Fortran code, 
consider thorn A which
wants to gather terms for a calculation from any thorn
which wishes to provide them. Thorn A could have
the lines in its source code

\begin{verbatim}
c Get source code from other thorns
      allterms = 0d0
#include "AllSources.inc"
\end{verbatim}
and would then add to {\tt interface.ccl} the line
\begin{verbatim}
USES INCLUDE SOURCE: AllSources.inc
\end{verbatim}

If thorn B wants to add terms for the calculation, it would
create a file, say {\tt Bterms.inc} with the lines
\begin{verbatim}
c Add this to AllSources.inc
      allterms = allterms + 1d0
\end{verbatim}
and would add to its own {\tt interface.ccl}

\begin{verbatim}
INCLUDE SOURCE: Bterms.inc in AllSources.inc
\end{verbatim}

The final file for thorn A  which is compiled will contain the code
\begin{verbatim}
c Get source code from other thorns
      allterms = 0d0
      if (CCTK_IsThornActive("B").eq.0) then
c Add this to AllSources.inc
        allterms = allterms + 1d0       
      end if
\end{verbatim}

Any Fortran thorn routines which include source code must include 
the declaration {\tt DECLARE\_CCTK\_FUNCTIONS}.

\subsection{Commenting Source Code}

Note that since most source files (see Section~\ref{nacofosofi} for exceptions) 
pass through a C preprocessor, C style comments can be used in 
Fortran code. C++ comments (that is //), should not be used, since 
they indicate the string concatonation operator in Fortran. 

The Flesh and the Cactus thorns use {\tt grdoc} to document the code.

\subsection{Coordinates}
\label{sec:co}

The flesh provides utility routines for registering and querying coordinate 
information. 
The flesh does not provide any coordinates itself, these must be supplied 
by a thorn. 
Thorns are not required to register coordinates to the flesh, 
but registering coordinates provides a means 
for infrastructure thorns, to make use of coordinate information.

Coordinate support is still being developed in the Cactus flesh. 
At the moment, it is assumed that 
coordinates will usually be grid functions. 

Coordinates are grouped into {\it coordinate systems}, which have a specified dimension. Any number of 
coordinate systems can be registered with the flesh, and a coordinate system must be registered before
any coordinates can be registered, since they must be associated with their corresponding system. 
Coordinates can be registered, with any chosen name, with an existing coordinate system, along with
their direction or index in the coordinate system. Optionally, the coordinate can also be associated with
a given grid variable. A separate call can register the global range for a coordinate on a given grid hierachy. 

The registration utilities for thorns providing coordinates are:
\begin{Lentry}

\item[{\tt CCTK\_CoordRegisterSystem}]

Assigns a coordinate system with a chosen name and dimension

\item[{\tt CCTK\_CoordRegisterData}]

Defines a coordinate in a given coordinate system, with a given 
	direction and name, and optionally associates it to a grid variable.

\item[{\tt CCTK\_CoordRegisterRange}]

Assigns the global maximum and minimum for a coordinate 
            	on a grid hierachy, that is in a {\tt cctkGH}.

\end{Lentry}

The utilities for thorns using coordinates are:

\begin{Lentry}

\item[{\tt CCTK\_CoordSystemDim}]

Provides the dimension of a coordinate system

\item[{\tt CCTK\_CoordIndex}]

Provides the grid variable index for a given coordinate

\item[{\tt CCTK\_CoordDir}]

Provides the direction for a given coordinate

\item[{\tt CCTK\_CoordRange}]

Provides the global range of a coordinate on a given grid hierachy

\item[{\tt CCTK\_CoordLocalRange}]

Provides the local range of a coordinate on a processor for 
	a given grid hierachy. WARNING: This utility only currently works for regular cartesian grids!!

\end{Lentry}

\subsection{IO}
\label{sec:io}

To allow flexible IO, the flesh itself does not provide 
any output routines, however it provides a mechanism for 
thorns to register different routines as IO methods. Application thorns
can interact with the different IO methods through the following
function calls:

\begin{description}

\item[CCTK\_OutputGH]

This call loops over all registered IO methods, calling 
the routine that each method has registered for {\t OutputGH}.
The expected behaviour of any methods {\t OutputGH} is to
loop over all GH variables outputting them if the method 
contains appropriate routines (that is, not all methods will 
supply routines to output all different types of variables) 
and if the method decides it is an appropriate time to 
output. 

\item[CCTK\_OutputVarAsByMethod]

Output a variable {\t varname} using the method {\t methodname} if it is 
registered. Uses {\t alias} as the name of the variable for the purpose
of constructing a filename. The output should take place if at all possible,
if the appropriate file exists the data is appended, otheriwise a new
file is created.


\item[CCTK\_OutputVarByMethod]

Output a variable {\t varname} using the method {\t methodname} if it is 
registered. The output should take place if at all possible,
if the appropriate file exists the data is appended, otherwise a new
file is created.

\item[CCTK\_OutputVarAs]

Output a variable {\t varname} looping over all registered methods. 
The output should take place if at all possible,
if the appropriate file exists the data is appended, otherwise a new
file is created. Uses {\t alias} as the name of the variable for the purpose
of constructing a filename.

\item[CCTK\_OutputVar]

Output a variable {\t varname} looping over all registered methods. 
The output should take place if at all possible,
if the appropriate file exists the data is appended, otherwise a new
file is created.

\end{description}

\subsection{Reduction Operators}
\label{sec:reop}

A reduction operation can be defined as an operation on variables
distributed  across multiple processor resulting in a single number.
Typical reduction operations are: sum, minimum/maximum value, boolean
operations. A typical applications is e.g. the maximum reduction on
a processor local error estimates, therefor making the previous
processor local error known to all processors. 

The exchange of
information across processors needs the functionality of a
communication layer eg. PUGH. For this reason, the
reduction operation itself is not part of the flesh, instead Cactus (again)
provides a registration mechanism for thorns to register routines they 
provide as reduction operators. The different operators are
identified by their name and a unique number, called a {\em handle}.

The registration mechanism gives the advantage of a common
interface while hiding the individual communication calls in the
layer.

In Cactus, reduction operators can be applied to gridfunctions, grid
scalars and distributed arrays, as well as local variables/arrays. 
There is a fundamental difference between the reduction operation on
grid functions and quantities as arrays.


{\bf Obtaining the reduction handle}
Before calling the routine which performs the reduction operation,
the handle, which indentifies the operation must be derived from its
registered name.

\begin{verbatim}

int CCTK_GetReductionHandle(const char *reduction_name);

call CCTK_ReductionHandle(int reduction_handle, character reduction_name)


int CCTK_GetReductionArrayHandle(const char *reduction_name);

call CCTK_ReductionArrayHandle(int reduction_handle, character reduction_name)

\end{verbatim}


\vskip .25cm

\begin{Lentry}
\item[{\tt reduction\_handle}] 
In Fortran the name of the variable will
contain the handle value after the call. In C this value is the
function value.
\item[{\tt reduction\_name}] 
is the name under which the operator has
been registered by the providind thorn. Currently only PUGH is
providing reduction operations.
\item[{\bf error checking}] 
negative handle value indicates failure to
identify the correct operator. Possible causes: misspelled operator
name, operator not registered properly.
\end{Lentry}

Get a integer handle corresponding to a given reduction operator.
The operator is identified by the name it was registered with.
(Note that although it would appear to be far more convenient to 
pass the name of the reduction operator directly to the following
function call to {\t CCTK\_Reduce} this causes problems with the
translation of strings from {\t FORTRAN} to {\t C} with variable
argument lists).

\vskip 0.25cm


{\bf The general reduction interface}
The main interfaces for reduction operations are quite powerful (and
hence rather complicated) To ease the use of these main interfaces, wrappers
designed for specific and more restricted use are described below. If 
uncertain, you should use these.

{\t
\begin{verbatim}

int CCTK_Reduce(  cGH *GH, 
                  int retvaltype, 
                  int retvalnum, 
                  void *retval, 
                  int handle, 
                  int index, 
                  ...);


call CCTK_Reduce( int returnvalue, 
                  cctkGH, 
                  int processor,
                  int operation_handle,
                  int num_out_vals,
                  int type_out_vals,
                  out_vals,
                  int num_in_fields,
                  ... )

int CCTK_ReduceArray(  cGH *GH,
                       int proc,
                       int operation_handle,
                       int num_out_vals,
                       int type_out_vals,
                       void *out_vals,
                       int num_dims,
                       int num_in_arrays,
                       int type_in_arrays,
                       ... )

call CCTK_ReduceArray(int returnvalue,
                      cctkGH,
                      int processor,
                      int operation_handle,
                      int num_out_vals,
                      int type_out_arrays,
                      void out_vals,
                      int num_dims,
                      int num_in_arrays,
                      int type_in_arrays,
                      ... )
\end{verbatim}
}

\begin{Lentry}
\item[{\tt int returnvalue}] 
the return value of the operation. negative
value indicates failure to perform reduction.
Zero indicates successfull operation.
\item[{\tt cctkGH}]
in fortran the pointer to the grid hierachy
structure. Cannot be used within Fortran but
can be used from within 
C. Since this name is fixed write it out shown
above.
\item[{\tt cGH *GH}]
 in the C the pointer to the grid hierarchy.
\item[{\tt int processor}] 
the processor which collects the
information, negative value ($-1$) will distribute the data to all
processors.
\item[{\tt int operation\_handle}] the number of the reduction operation
                handle, needs to be found by calling {\tt CCTK\_ReductionHandle} or
                {\tt CCTK\_ReductionArrayHandle}.
\item[{\tt int num\_out\_vals}] integer defining the number of ??
\item[{\tt int type\_out\_arrays}, {\tt type\_in\_arrays}] 
specifies the type of the gridfunction 
you are communicating. Use the values as specified in
\ref{sec:datyansi}. Note: do not {\em mix} datatypes: e.g. in
Fortran do not declare a variable as {\tt integer} and then specify  
the type {\tt CCTK\_VARIABLE\_INT} in the reduction command. These
types need not be the same on some architectures and will conflict.
\item[{\tt out\_vals}]
\item[{\tt int num\_in\_fields}] specifies the number of  to follow
\item[{\tt ...}] indicates a varible argument list: specify the size
of the array in each dimension, comma separated numbers or integer
variables. Specify the arrays which will be reduced, the number of
specified arrays must be the same as the value of the {\tt
num\_in\_fields} variable.
\item[{\bf error checking}] a return value other than zero indicates
failure to perform the operation.
\end{Lentry}

\vskip 0.25cm


{\bf Special reduction interfaces}

The routines are designed for the purpose of reducing scalars, arrays
and grid functions. They hide many of the options of the generic
interface described above.\\

{\bf Reduction of local scalars} across multiple processors, the result of 
the reduction operation will be on the specified processor or on all processors.

{\t
\begin{verbatim}
int CCTK_ReduceLocScalar (cGH *GH, 
                          int processor, 
                          int operation_handle,
                          void *in_scalar, 
                          void *out_scalar, 
                          int data_type)

call CCTK_ReduceLocScalar(int returnvalue,
                          cctkGH,
                          int processor,
                          int operation_handle,
                          in_scalar,
                          out_scalar,
                          int data_type)
\end{verbatim}
}
\begin{Lentry}
\item[{\tt in\_scalar}] the processor local variable with local value to be reduced
\item[{\tt out\_scalar}] the reduction result: a processor local variable 
with the global value (same on all procs) if {\tt processor} has been
set to $-1$. Otherwise {\tt processor} will hold the reduction result.
\item[{\tt data\_type}]
specifies the type of the gridfunction 
you are communicating. Use the values as specified in
\ref{sec:datyansi}.
\end{Lentry}

\vskip 0.25cm

{\bf Reduction of local 1d arrays} to a local arrays, this reduction is carried
out element by element. The arrays need to have the same size on all
processors.
{\t
\begin{verbatim}
int CCTK_ReduceLocArrayToArray1D( cGH *GH, 
                                  int processor,
                                  int operation_handle,
                                  void *in_array1d, 
                                  void *out_array1d, 
                                  int xsize)
                                  int data_type)

call CCTK_ReduceLocArrayToArray1D( int returnvalue
                                  cctkGH,
                                  int processor,
                                  int operation_handle,
                                  in_array1d,
                                  out_array1d,
                                  int xsize,
                                  int data_type)
\end{verbatim}
}

\begin{Lentry}
\item[{\tt in\_array1d}] one dimensional local arrays, to be reduced across a 
processors, element by element. 
\item[{\tt out\_array1d}] array holding the reduction result. out\_array1d[1]
= Reduction(in\_array[1]).
\item[{\tt xsize}] the size of the one dimensional array.
\end{Lentry}

\vskip 0.25cm

{\bf Reduction of local 2d arrays} to a local 2d array, this reduction is carried
out element by element. The arrays need to have the same size on all
processors.
{\t
\begin{verbatim}
int CCTK_ReduceLocArrayToArrayD( cGH *GH,
                                 int processor,
                                 int opertaion_handle,
                                 in_array_2d,
                                 out_array2d,
                                 int xsize,
                                 int ysize,
                                 int data_type)
                                 

call CCTK_ReduceLocArrayToArray2D( int returnvalue
                                   cctkGH,
                                   int processor,
                                   int operation_handle,
                                   in_array2d,
                                   out_array2d,
                                   int xsize,
                                   int ysize,
                                   int data_type)
                                 
\end{verbatim}
}

\begin{Lentry}
\item[{\tt in\_array1d}] two dimensional local arrays, to be reduced across a 
processors, element by element. 
\item[{\tt out\_array1d}] two dimensional array holding the reduction
result. out\_array2d[i,j]= Reduction(in\_array2d[i,j]).
\item[{\tt xsize}] the size of the one dimensional array in x direction.
\item[{\tt ysize}] the size of the one dimensional array in y direction.
\end{Lentry}

\vskip 0.25cm

{\bf Reduction of local 3d arrays} to a local 3d array, this reduction is carried
out element by element. The arrays need to have the same size on all
processors.
{\t
\begin{verbatim}
int CCTK_ReduceLocArrayToArray3D(cGH *GH,
                                 int processor,
                                 int opertaion_handle,
                                 in_array_3d,
                                 out_array3d,
                                 int xsize,
                                 int ysize,
                                 int zsize,
                                 int data_type)

call CCTK_ReduceLocArrayToArray3D(int returnvalue
                                 cctkGH,
                                 int processor,
                                 int operation_handle,
                                 in_array3d,
                                 out_array3d,
                                 int xsize,
                                 int ysize,
                                 int zsize,
                                 int data_type)
\end{verbatim}
}

\begin{Lentry}
\item[{\tt in\_array3d}] two dimensional local arrays, to be reduced across a 
processors, element by element. 
\item[{\tt out\_array3d}] two dimensional array holding the reduction
result. out\_array3d[i,j,k]= Reduction(in\_array3d[i,j,k]).
\item[{\tt xsize}] the size of the one dimensional array in x direction.
\item[{\tt ysize}] the size of the one dimensional array in y direction.
\item[{\tt ysize}] the size of the one dimensional array in z direction.
\end{Lentry}

\vskip .25cm

{\bf Some brief examples:}

{\bf Reduction of a local scalars:} a local error is reduced across a
processors wiht the maximum operation. The variable {\tt tmp} will
hold the maximum of the error and  is the same on all
processors. This qunatity can then be reassigned to {\tt normerr}.
\begin{verbatim}      
         CCTK_REAL normerr, tmp
         integer   ierr, reduction_handle

         call CCTK_ReductionArrayHandle(reduction_handle,"max")

         if (reduction_handle.lt.0) then
            call CCTK_WARN(1,"Cannot get reduction handle for SUM operation.")
         endif

         call CCTK_ReduceLocScalar(ierr, cctkGH, -1, 
     .             reduction_handle,
     .             normerr, tmp, CCTK_VARIABLE_REAL)
         if (ierr.ne.0) then
            call CCTK_WARN(1,"Reduction of norm failed!");
         endif
         normerr = tmp
\end{verbatim}


{\bf Reduction of a local 2d array:} a two dimensional array $(2x3)$ is 
rdeuced, reduction results (aray of same size: {\tt bla\_tmp}) is seen
on all processors ($-1$ entry as the thid argument); also demonstrates
some simple error checking with the {\tt CCTKi\_EXPECTOK} macro.
\begin{verbatim}
      CCTK_REAL bla(2,3),bla_tmp(2,3);
      integer   ierr, sum_handle

      call CCTK_ReductionArrayHandle(sum_handle,"sum")
      bla         =  1.0d0
      write (*,*) "BLA ",bla

      call CCTK_ReduceLocArrayToArray2D(ierr, cctkGH, -1, sum_handle,
     .     bla, bla_tmp, 2, 3, CCTK_VARIABLE_REAL)
      call CCTKi_EXPECTOK(ierr, 0, 1, "2D Reduction failed")

      bla = bla_tmp
      write (*,*) "BLA ",bla
\end{verbatim}

Note that the memory for the returned values must be allocated before
the reduction call is made.


\section{Calls between different programming languages}
\label{sec:cabedipr}

\subsection{Calling C routines from FORTRAN}
\label{sec:cacrofr}

\subsection{Calling FORTRAN routines from C}
\label{sec:caforofr}

To call a utility Fortran routine from C use

{\tt
void CCTK\_FCALL CCTK\_FNAME(<Fortran routine name>)(<argument list>)
}


\section{Adding a test suite}
\label{sec:adatesu}

To add a test suite to your thorn, devise a series of parameter
files which use as many aspects of your thorn as possible. 
Make sure that the parameter files produce ascii output to files,
and that these files are in the directory 
{\tt ./<parameter file base name>}.

Run Cactus on each of the parameter files, and move the parameter files,
and the output directories they produced, to the {\tt test} directory
in your thorn. 

Document carefully any situations or architectures in which your test 
suite does not give the correct answers. 

For details on running the test suites, see Section~\ref{sec:te}.


\section{Naming conventions}

\begin{itemize}

\item{} Thorn names must not start with the word "Cactus" (in
        any case).
\item{} Arrangements will be ignored if their names start with \# or .
        or end in \~{} .bak or .BAK 
\item{} Thorns will be ignored if they are called doc or start with
        \# or . or end in \~{} .bak or .BAK
\item{} Routine names have to be unique among all thorns. 

\end{itemize}


\section{General Naming Conventions}

The following naming conventions are followed by the flesh and the
supported Cactus arrangements. They are not compulsory, but if followed
allow for a homogeneous code.

\begin{itemize}

\item Parameters: lower case (except for acronyms) with words separated
  by an underscore. Examples: {\tt my\_first\_parameter}, 
  {\tt solve\_PDE\_equation}.

\item Filenames and routine names: Prefixed by thorn name with an underscore, then capitalised words, with no spaces.
    Examples: {\tt MyThorn\_StartUpRoutine}, {\tt BestSolver\_InitialDataForPDE}.

\end{itemize}

\section{Data Types and Sizes}
\label{sec:datyansi}

Cactus supports the following fixed size data types

\begin{center}
\begin{tabular}{|c|c|c|c|}
\hline
Data Type & Size (bytes) & Variable Type & Fortran Equivalent\\
\hline
{\t CCTK\_INT2}      & 2  & {\t CCTK\_VARIABLE\_INT2}      & {\t integer*2}\\
{\t CCTK\_INT4}      & 4  & {\t CCTK\_VARIABLE\_INT4}      & {\t integer*4}\\
{\t CCTK\_INT8}      & 8  & {\t CCTK\_VARIABLE\_INT8}      & {\t integer*8}\\
{\t CCTK\_REAL4}     & 4  & {\t CCTK\_VARIABLE\_REAL4}     & {\t real*4}\\
{\t CCTK\_REAL8}     & 8  & {\t CCTK\_VARIABLE\_REAL8}     & {\t real*8}\\
{\t CCTK\_REAL16}    & 16 & {\t CCTK\_VARIABLE\_REAL16}    & {\t real*16}\\
{\t CCTK\_COMPLEX8}  & 8  & {\t CCTK\_VARIABLE\_COMPLEX8}  & {\t complex*8}\\
{\t CCTK\_COMPLEX16} & 16 & {\t CCTK\_VARIABLE\_COMPLEX16} & {\t complex*16}\\
{\t CCTK\_COMPLEX32} & 32 & {\t CCTK\_VARIABLE\_COMPLEX32} & {\t complex*32}\\
{\t CCTK\_CHAR}      & 1  & {\t CCTK\_VARIABLE\_CHAR}      & {\t character} \\ \hline
\end{tabular}
\end{center}

In addition Cactus provides three data types whose size is chosen
during the compilation process (at configuration time). This is to
allow the code to be easily run at different precisions. Note that
the effectiveness of running the code at a lower or higher precision
depends crucially on all thorns being used making consistent use
of the following data types:


\begin{center}
\begin{tabular}{|c|c|c|c|}
\hline
Data Type          & Default Size (bytes) & Variable Type & Configuration Option\\
\hline
{\t CCTK\_INT}     & 4 & {\t CCTK\_VARIABLE\_INT} & {\t INTEGER\_PRECISION}\\
{\t CCTK\_REAL}    & 8 & {\t CCTK\_VARIABLE\_REAL} & {\t REAL\_PRECISION}\\
{\t CCTK\_COMPLEX} & (8,8) & {\t CCTK\_VARIABLE\_COMPLEX} & Same as real precision\\
\hline
\end{tabular}
\end{center}

These variable types must be used by thorn writers to declare variables
 in the thorn interface files, and may be used to declare
variables in the thorn routines. Note that variable declarations in 
thorns should obviously match with definitions in the interface files
where appropriate.

Also provided, are a set of macros which
are interpreted by the preprocessor at compile time to signify which
data size is being used:

\begin{center}
\begin{tabular}{|c|c|}
\hline
Data Type & {\t \#define}\\
{\t CCTK\_INT2}      & {\t CCTK\_INT\_PRECISION\_2}      \\
{\t CCTK\_INT4}      & {\t CCTK\_INT\_PRECISION\_4}      \\
{\t CCTK\_INT8}      & {\t CCTK\_INT\_PRECISION\_8}      \\
{\t CCTK\_REAL4}     & {\t CCTK\_REAL\_PRECISION\_4}     \\
{\t CCTK\_REAL8}     & {\t CCTK\_REAL\_PRECISION\_8}     \\
{\t CCTK\_REAL16}    & {\t CCTK\_REAL\_PRECISION\_16}    \\
{\t CCTK\_COMPLEX8}  & {\t CCTK\_COMPLEX\_PRECISION\_8}  \\
{\t CCTK\_COMPLEX16} & {\t CCTK\_COMPLEX\_PRECISION\_16} \\
{\t CCTK\_COMPLEX32} & {\t CCTK\_COMPLEX\_PRECISION\_32} \\
\hline 
\end{tabular}
\end{center}

Note that the availability of these types, and the corresponding 
C data types are platform dependent.

\subsection{Fortran Thorn Writers}

Cactus provides a further data type {\tt CCTK\_POINTER} 
for use in Fortran code to declare a pointer passed from C. 
For example, the variable {\tt cctkGH} is of this type.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\chapter{Telling the makesystem what to do}

\section{Basic Recipe}

\section{Make Concepts}

\section{The Four Files}

\section{How your code is built}

\end{cactuspart}