path: root/doc/UsersGuide/ThornWriters.tex

\part{Application thorn writing}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\chapter{Thorn concepts} 





      (This section has to contain enough explanation to make the rest of
       the writers guide readable the first time through)
(The hardest bugs to find are
those arising from plausible but incorrect assumptions about the
behavior of someone else's thorn.)
      a) Again probably emphasize collaboration, what are thorns,
         packages, how to share them.
      b) Things to think about before you start programming:
             Language, read all the documentation, emphasize use of
             standard supported Cactus infrastructure
      c) Available data types
         i)   Scalars
         ii)  Arrays and GFs
         iii) Groups
      d) Ghost zones and parallelism
      e) Understanding the RFR concept
      f) Understanding the GH concept

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{packages}

Thorns are grouped into {\em packages}.  This is a logical grouping of
thorns which is purely for administrative purposes.  The compiled
code knows nothing about packages.

The packages live in the `packages' directory off the main CCTK directory.  
Package names must be unique, and cannot start with a `\#', or finish
with `~' or `.bak'.

Inside a package directory there is a directory for each thorn belonging
to the package.  Thorn names have the same restrictions as package names,
with the addition that a thorn cannot be called `doc'.  This name is
reserved for package documentation.

\section{Data Types and Sizes}

Cactus supports the following fixed size data types

\begin{center}
\begin{tabular}{|c|c|c|c|}
\hline
Data Type & Size (bytes) & Variable Type & Fortran Equivalent\\
\hline
{\t CCTK\_INT4}     & 4 & {\t CCTK\_VARIABLE\_INT4}     & {\t integer*4}\\
{\t CCTK\_INT8}     & 8 & {\t CCTK\_VARIABLE\_INT8}     & {\t integer*8}\\
{\t CCTK\_INT16}    & 16& {\t CCTK\_VARIABLE\_INT16}    & {\t integer*16}\\
{\t CCTK\_REAL4}    & 4 & {\t CCTK\_VARIABLE\_REAL4}    & {\t real*4}\\
{\t CCTK\_REAL8}    & 8 & {\t CCTK\_VARIABLE\_REAL8}    & {\t real*8}\\
{\t CCTK\_REAL16}   & 16& {\t CCTK\_VARIABLE\_REAL16}   & {\t real*16}\\
{\t CCTK\_COMPLEX4} & 4 & {\t CCTK\_VARIABLE\_COMPLEX4} & {\t complex*4}\\
{\t CCTK\_COMPLEX8} & 8 & {\t CCTK\_VARIABLE\_COMPLEX8} & {\t complex*8}\\
{\t CCTK\_COMPLEX16}& 16& {\t CCTK\_VARIABLE\_COMPLEX16}& {\t complex*16}\\
{\t CCTK\_CHAR}     & 1 & {\t CCTK\_VARIABLE\_CHAR}     & {\t character} \\ \hline
\end{tabular}
\end{center}

In addition Cactus provides three data types whose size is chosen
during the compilation process (at configuration time). This is to
allow the code to be easily run at different precisions. Note that
the effectiveness of running the code at a lower or higher precision
depends crucially on all thorns being used making consistent use
of the following data types:


\begin{center}
\begin{tabular}{|c|c|c|c|}
\hline
Data Type          & Default Size (bytes) & Variable Type & Configuration Option\\
\hline
{\t CCTK\_INT}     & 4 & {\t CCTK\_VARIABLE\_INT} & {\t INTEGER\_PRECISION}\\
{\t CCTK\_REAL}    & 8 & {\t CCTK\_VARIABLE\_REAL} & {\t REAL\_PRECISION}\\
{\t CCTK\_COMPLEX} & (8,8) & {\t CCTK\_VARIABLE\_COMPLEX} & Same as real precision\\
\hline
\end{tabular}
\end{center}

These variable types must be used by thorn writers to declare variables
 in the thorn interface files (FIXME: REF), and may be used to declare
variables in the thorn routines. Note that variable declarations in 
thorns should obviously match with definitions in the interface files
where appropriate.

Also provided, are a set of macros which
are interpreted by the preprocessor at compile time to signify which
data size is being used:

\begin{center}
\begin{tabular}{|c|c|}
\hline
Data Type & {\t \#define}\\
{\t CCTK\_INT4} & {\t CCTK\_INT\_PRECISION\_4} \\
{\t CCTK\_INT8} & {\t CCTK\_INT\_PRECISION\_8} \\
{\t CCTK\_INT16} & {\t CCTK\_INT\_PRECISION\_16} \\
{\t CCTK\_REAL4} & {\t CCTK\_REAL\_PRECISION\_4} \\
{\t CCTK\_REAL8} & {\t CCTK\_REAL\_PRECISION\_8} \\
{\t CCTK\_REAL16} & {\t CCTK\_REAL\_PRECISION\_16} \\
{\t CCTK\_COMPLEX4} & {\t CCTK\_COMPLEX\_PRECISION\_4} \\
{\t CCTK\_COMPLEX8} & {\t CCTK\_COMPLEX\_PRECISION\_8} \\
{\t CCTK\_COMPLEX16} & {\t CCTK\_COMPLEX\_PRECISION\_16} \\
\hline 
\end{tabular}
\end{center}

Note that Cactus does not provide integer types, and
that the corresponding C data types are platform dependent.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


\chapter{Anatomy of a thorn}

\section{thorns}

A thorn consists of a subdirectory of a package containing three 
administrative files

\begin{itemize}
\item{\tt interface.ccl}: the cactus interface, which defines the grid 
functions, variables, etc. See \ref{sec:in}.
\item{\tt param.ccl}: the parameters introduced by the thorn. See
\ref{param.ccl}.
\item{\tt schedule.ccl}: scheduling information for routines called by
the flesh.
\end{itemize}

and a subdirectory called `src', which should hold source files for
the thorn.

In addition each thorn should have a `README' containing a brief description
of the thorn, a `doc' directory for documentation, and a `par' directory
for example parameter files.

Once the thorn is finished a `test' subdirectory may also be added.  See \ref{sec:testsuites} for details.

\section{Creating a thorn}


To simplify the creation of a thorn, a make target {\tt
gmake newthorn} has been provided.

When this is run:

\begin{enumerate} 
\item{} You will be prompted for the name of the new thorn.
\item{} You will be prompted for the name of the package you would
like to include your thorn in. Either enter a new package name or pick 
one from the list of available packages that are shown.
\end{enumerate}


\section{Configuring your thorn}

\subsection{The {\tt param.ccl}}
The {\tt param.ccl} defines the parameters your thorn provides. To
avoid wrong assumptions by other thorns about your parameters, you have 
to speciy several informations, which we'll motivate here:
\begin{enumerate}
\item{} Are you creating a new parameter or are you extending
the value range of a parameter by a different thorns?
\item{} Is your parameter public to all other thorns, is it just
used by your thorn or is it available to a subset of thorns only ?
\item{} What are possible values your parameter can take: range of
numerical values or the set of keywords, what do they mean ?  What is
the parameter's default value ? 
\end{enumerate}
The {\em cactus computational language} lets you specify these
information in a straight forward manner. (For the actual syntax of the 
{\tt param.ccl} see \ref{sec:pa}.)
The parameters you define can be grouped in four classes: {\tt
private}, {\tt public},  {\tt protected} and {\tt friend}:{\em
thorn\_name}.  We'll go over them one by one: 
\begin{itemize}
\item{\tt private} This parameter can only be used by your thorn. It
it not even visible to other thorns: multiple private parameters of
the same name may exist in parallel. {\bf IS THIS TRUE ?} We give
examples of private parameters of type {\em integer}, {\em real}, {\em logical}
and {\em keyword}, which may have the following form (comments start with
{\tt \#}):
\begin{verbatim}

REAL Coeff "Important coefficient"
{
0:3.14 :: ``Range has to be from zero to Pi, default is zero''
} 0.0

INTEGER hitch ``The hitch-hiker parameter''
{
-42:42 :: ``Ranges from -42 to 42, default is zero''
} 0

#If you omitt the range, it will look a bit strange.
#better put a range!
INTEGER hitch_nolimits ``The confusing hitch-hiker parameter''
{
: :: ``We are confused, what is the range?''
} 0

#No need to define a range for LOGICAL
LOGICAL nice ``Nice weather ?''
{
}``yes''

# A example for a set of keywords and its default (which has to be
# defined in the body)
KEYWORD confused "Are we getting confused ?''
{
  "yes''    :: ``absolutley positively''
  "perhaps" :: "we are not sure''
  "never"   :: "never''
} "never''
\end{verbatim}
Please consult the syntax for the {\tt param.ccl} for other possible
combinations.
\item{\tt public} this class allows parameters to be read by {\em all} 
thorns a user will possibly compile with. You should make very
restricted use of the {\tt public} keyword since this can be the cause 
for incompatabilities and misunderstandings. {\bf CAN I HAVE A SECOND
(PRIVATE) PARAMETER OF THE SAME NAME ? IN A DIFFERENT THORN ?}
A typical declaration which further demonstrates the parameter syntax
could be: 
\begin{verbatim}
public:

# grid spacing has to be in the positive range 
REAL grid_dx "Coarse grid spacing in x-direction"
{
  0: :: ``grid spacing has to be positive, default is 0.3''
} 0.3
\end{verbatim}

\item{\tt protected} a parameter in this class is not only visible by
its own thorn, but can be {\em inherited} by friend: a different thorn 
may either check the value of this parameter or may extend its value
range. A typical application would be the generation of initial data
by different thorns, where each thorns extends an intial data
parameter value by a name indicating the type of data it implements.

\begin{verbatim}
protected:

KEYWORD initial_data "Physical spacetime model to evolve"
{
  "flat" :: "Flat Minkowski space"
} "flat"
\end{verbatim}

\item{\tt friend}:{\em thorn\_name} a parameter of type friend has been declared by an
other thorn and you are extending its value. You have
to specify which thorn has declared this parameter initially. This
``parent''-thorn has to identify the parameter as {\tt protected}
(see above) in order for you to modify its value range. A typical
delcaration for would look like:
\begin{verbatim}

#We inherit this parameter form thorn Einstein and add several models
#Only einstein and our thorn see this parameter. ******TRUE ???
#BTW: No need to add a new comment 

friend:einstein
EXTENDS KEYWORD initial_data ""
{
  "bl_bh"         :: "Brill Lindquist black holes"
  "misner_bh"     :: "Misner black holes"
  "schwarzschild" :: "One Schwarzshild black hole"
}
\end{verbatim}
Note, that you {\em cannot} reset the default value. The {\em ccl}
will report this as an error. Also, you cannot extend the default 
comment, rather document the individual extension in the body.
\end{itemize}

\subsection{The {\tt interface.ccl}}
{\bf CAN WE A CONVENTION FOR VARIABLES, GRIDFUNCTION, ...? }
The interface plays the inportant role of introducing new grid
function, their storage and communication concepts, new variables,
etc. to the CCTK infrastructure. While this interface convention looks 
difficult at first sight, it ensures the proper compatability of
thorns we can't even imagine at this point. And it is fairly easy to
learn. The thorn writer should question the impact and scope of his
implementation in the following way:
\begin{enumerate}
\item{What is the name of the implementation, so it can be referenced by other 
thorns.}
\item{On which other thorns (implementations) does the thorn depend,
because of parameter extension, grid functions, etc.}
\item{What are the grid functions to implement and how are they grouped ? A 
group should try to collect as many common feature as possible. Some
common features are mandatory, eg. the storage of the variables
belonging to the same group has to be equal.{\bf What else ?} That way
certain operation can be carried out on a group by group basis, instead
time consuming function by function.}.
\item{Do functions need to be communicated between different processors 
or are they local ?}
\item{Do variables need storage turned on all the time or are they
just used at a certain instance, for example for analysis.}
\end{enumerate}

The {\tt interface.ccl} defines all these specific information. The
syntax is described in appendix \ref{sec:in}, please see 
there for all possible options. Here we will introduce the general
idea behind the interface.ccl.

Here we show a sample taken from the thorn {\tt Einstein}, which
implements the ADM variables:
\begin{verbatim}
implements: einstein
inherits:   grid

public:

integer flags type = SCALAR
{
  conformal_state
  shift_state
} "State information"

real metric type = GF
{
  gxx,gxy,gxz,gyy,gyz,gzz
} "ADM 3-metric"

\end{verbatim}

The first lines in an {\em interface.ccl} file name the
implementation, which this thorns adds or expands and declare the
implementations this thorn depends on. In this case the thorn
implements {\em ``einstein''} and depends on the public and protected
variables of the {\em ``grid''} implementation, which add grids
specific data.

The gridfunctions and scalar variables, which will
be declared next, can be grouped in three classes -- very
similiar to the parameter declarations: the classes {\em public}, {\em 
protected} and {\em private} can be used to inform the system about
the scope of the individual varibles and gridfunctions:
\begin{enumerate}
\item{\tt public} grid functions, arrays or scalars defined in this
class can be {\em read and written} by all thorns, whitout having to
declare themselves as friends. This class should be used very
restrictive since unathourized altering of a varibable can cause sever 
confusion.
\item{\tt protected} grid functions of this can class can be read/modified by
other thorns, if these thorns have registered your implementation in
the {\tt inherit:} section.  In our example, you would be able to
access all variables of class protected of the {\tt grid}
implementation. The {\em heritage}-concept was developed in response
to the growing naming and access confusion in a growing pool of
thorns. It was not designed to give the thorn programmer a hard life,
but to make him think twice about possible sideeffects his
implementation will bring about.
\item{\tt private} grid functions in this class can only be
rad/modified by the {\bf IMPLEMENTATION} or {\bf THORN} itself.
\end{enumerate}

In this case all variables are of class {\em public} which means that
{\em all} other thorns can read and modify the values of these grids
functions/scalars. One should be very restrictive about the use of the 
{\em public} class. In general, declarations as {\em protected} and/pr 
{\em private} will serve as well but reduce the confusion on who owns
which variable significantly.

The variable/grid function can be of the following types: {\tt real}, {\tt
integer}, {\tt complex}. After the type declaration the {\em groupname} 
is given, so that certain operation can be carried out on a time
efficient group by group basis. In our example, the group names are
{\em ``flags''} and {\em ``metric''}. The next keywords denotes,
what kind of variable we have: {\tt type = }
\begin{enumerate}
\item{\tt SCALAR}: a scalar variable
\item{\tt GF}: a grid function of the size of the grid.
\item{\tt ARRAY}: an array of the size \ldots
{\bf {\tt grace} uses keywords DIM TIMELEVEL, how do they work and are 
there others ?}
\end{enumerate}

The actual varibables are declared in the body of the full variable
statement, enclosed by {\tt ``\{``} / {\tt ``\}''}. In our example,
for the first group, two scalars are defined ({\tt conformal\_state} and
{\tt shift\_state}), for the second group, the six components of the
metric tensor are defined. The declaration of the variables finishes
with a descriptive comment.

\subsection{The {\tt schedule.ccl}}
The {\tt schedule.ccl} is used to register the routines of your thorn
with the CCTK and defines the storage characteristics of a group. 

We will first describe the storage concepts. There is no good reason
to keep memory allocated for variables which are only used
intermittendly in the course of the simulation, eg. during output or
analysis, or are not used at all depending on parameter settings. The
CCTK allows for a flexible memory allocation:  
\begin{enumerate}
\item{} Variable memory can be turned on at all times, for example by
\begin{verbatim}
STORAGE: metric
\end{varbatim}, which would turn on storage for the group we defined
in the example above. 
\item{}All-time memory can turned on depending on parameter
settings. This can be done in the same way, the parameters are acessed 
within a C or Fortran code: by explicit use of the parameter variables 
in the case of integer/real/logical (see \ref{????}) or
by use of the {\tt CCTK\_Equals()} function (see
\ref{????}). For example:
\begin{verbatim}
# reserve memory for the group ``shift'' if parameter 
# shift NOT set to ``none'' 
if (!CCTK_Equals(shift,"none"))
{
  STORAGE: shift 
}

# allocate memory for the groups ``confac'',...
# if the LOGICAL parameter ``use_conformal'' was set to ``yes''
if (use_conformal) {
  STORAGE: confac,confac_1derivs,confac_2derivs
}
\end{verbatim}
\item{} The third options lets you associate the memory allocation
with a routine call: storage is allocated when the routine is called
by the CCTK. This options is explained in connection with the {\em
RFR} below.
\item{} A modification of \#3 is the options to have storage enabled,
when a certain variable is computed for output.  This options is
explained in connection with the {\em RFR} below.
\end{enumerate}


The {\em runtime function registry (RFR)} registers the different routines a 
a thorn provides with the CCTK depending on the timing keyword
chosen. You can assign timing keywords to your routines out of a 
set of ordered keywords and register your routine with the
CCTK. During the evolution the CCTK will loop through this ordered
set of keywords and execute the registered thorn routines. This way,
you can hook your thorns at well defined locations into the main
iteration loop without having 
to modify anybody's code. The CCTK currently features a fixed {\em
skeleton} of timing keywords, which mean that you cannot currently
register relative to a different thorn.

The current order of timing keywords is:

{\q Check order, let's review this }

\begin{itemize}

\item {\tt CACTUS\_BASEGRID}
Resposnible for setting up coordinates etc.

\item {\tt CACTUS\_RECOVER}
For recovery from checkpoint.

\item {\tt CACTUS\_INITIAL}

For generating initial data.

\item {\tt CACTUS\_CPINITIAL}
For recovery of initial data from a checkpoint file.

\item {\tt CACTUS\_PRESTEP}

Stuff done before the evolution step.
\item {\tt CACTUS\_EVOL}      

The evolution step.

\item {\tt CACTUS\_POSTSTEP}

Stuff done after the evolution step.

\item {\tt CACTUS\_BOUND}

Boundary stuff. {\q Is this used ? }
\item {\tt CACTUS\_CHECKPOINT}
For checkpointing data

\item {\tt CACTUS\_ANALYSIS}
For analysing data.

\item {\tt CACTUS\_TERMINATE}
Called when cactus teminates.

\item {\tt CACTUS\_CONVERGENCE}
Convergence stuff.

\end{itemize}
{\q are going to name them CCTK ? Are we going to make a difference
between system keywords and user keywords  ? better yes}

The scheduling together with the storage options in {\tt schedule.ccl}
is described here in examples (please the ccl-syntax
\ref{sec:sc} for all possible options).
\begin{verbatim}

schedule initial at CACTUS_INITIAL
{
  LANG: Fortran
} "Initialisation for Einstein methods"


schedule metric_carttosphere at CACTUS_ANALYSIS
{
  STORAGE: spherical_metric
  LANG: Fortran
} "Calculate the spherical metric in r,theta(q), phi(p)"

schedule evaltrK at CACTUS_ANALYSIS
{
  STORAGE: trace_of_K,detofg
  LANG: Fortran
  TRIGGERS: trK,detg
} "Compute the trace of the extrinsic curvature"

\end{verbatim}
In the first example, the initialization routine {\tt initial}  is
schedules at instance {\tt CACTUS\_INITIAL}. The body of the 
declaration specifies the language used for this routine: options are
{\tt C} or {\tt FORTRAN}. This routines has no impact 
on storage behavior.

The second example registers the routine {\tt metric\_carttosphere} is
scheduled at {\tt CACTUS\_ANALYSIS}. (This routine converts the
cartesian metric to a spherical metric). The routine requests storage
when it is executed by the RFR. Memory is freed after the calculation.

The third example demonstrates yet another way qualifying execution
behavior: while CACTUS\_ANALYSIS registered routines are called on
every iteration, you wish to calculate certain variables only when
they are output. And this may not be needed on every iteration. By
placing the {\tt TRIGGER} keyword in the schedule-declaration, you
only excute this routine at CACTUS\_ANALYSIS, if there will be actually 
output: that is, when the current iteration number matches the
parameters {\tt itout1}, ...
{\bf in EINSTEIN: TRIGGERS: trK,detg. Why don't we use the actual GF
names ? }


{\bf FIXME: talk about before/after. }

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Naming Conventions for Source Files}

The make system uses file extensions
to designate coding language. The following extensions are
can be handled:

\begin{center}
\begin{tabular}{|c|c|}
\hline
Extension & Coding Language \\
\hline
{\t .F} & Fortran90 fixed form \\
{\t .f} & (no preprocessing) Fortran90 fixed form [FIXME: To be implemented]\\
{\t .F90} & Fortran90 free form form [FIXME: To be implemented]\\
{\t .f90} & (no preprocessing) Fortran90 free form [FIXME: To be implemented]\\
{\t .F77} & Fortran77 \\
{\t .f77} & (no preprocessing) Fortran77  [FIXME: To be implemented]\\
{\t .c} & C \\
{\t .cc} or {\t .C} & C++ \\
\hline
\end{tabular}
\end{center}

The following restrictions apply to file names:
\begin{itemize}
\item Rootnames within each directory must be unique. For example, it is not possible 
      to have the files {\tt InitialData.c} and {\tt InitialData.F} in the same directory. 
\end{itemize}

\section{Adding source files}


By default the CCTK looks in the {\tt src} directory of the thorn for source
files.

There are two ways in which to specify the sources.  The easiest is to use the 
{\tt make.code.defn} based method in which the CCTK does all the work, but you
may instead put a {\tt Makefile} in the {\tt src} directory and do everything
yourself.

\subsection{{\tt make.code.defn} based thorn building}

This is the standard way to do it.

If there is no file called {\tt Makefile} in the {\tt src} directory,
the CCTK make system looks for a file called {\tt make.code.defn} in that
directory.  At its simplest, this file contains two lines

\begin{itemize}
\item
\item {\t SRCS = <list of all source files {\em in this directory}>}

\item {\t SUBDIRS = <list of all subdirectories, {\em including subdirectories of subdirectories}>}

\end{itemize}

Each subdirectory listed should then have a {\tt make.code.defn} file 
containing just a {\tt SRCS = } line, a {\tt SUBDIRS = } line will
be ignored.

In addition, each directory can have a {\tt make.code.deps} file, which,
for files in that directory, can contain additional make rules and dependencies
for files in that directory.  See the GNU Make documentation for details of the
syntax.

\subsection{{\tt Makefile} based thorn building}

This method gives you ultimate responsibility.  The only requirement is that
a library called {\tt \$NAME} be created by the Makefile.

The makefile is passed the following variables

\begin{itemize}

\item {\tt \$(CCTK\_HOME)} - the main CCTK directory

\item {\tt \$(TOP)}    - the CONF directory

\item {\tt \$(SRCDIR)} - the directory in which the source files can be found

\item {\tt \$(CONFIG)} - the directory containing the configuration files

\item {\tt \$(THORN)}  - the thorn name

\item {\tt \$(SCRATCH\_BUILD)} - the scratch directory where f90 module
files should end up if they need to be seen by other thorns.

\item {\tt \$(NAME)}.

\end{itemize}

and has a working directory of of <config>/build/<thorn\_name> .

\chapter{Putting code into your thorn}

\section{What the Flesh provides}

Header files, macros ...

\subsection{Reduction Operators}
\label{sec:reop}

Reduction operators are operators which process grid functions
and return, either on just one processor or on each processor,
a 1-Dimensional array of scalars. Examples of reduction operators
could include simple operations such as the maximum value of a 
grid function across the grid, or complex operations such as
{\bf FIXME: Think of a good example, Gab}.

The flesh itself does not provide any reduction operators,
instead providing a mechanism for thorns to register their
own routines as reduction operators, labelled by a given name.
For details of writing reduction operators see ????. Application
thorns can interact with the different registered reduction 
operators through the following function calls:

{\t
\begin{verbatim}

#include ``Reduction.h''
int CCTK_GetReductionHandle(const char *reduction);

\end{verbatim}
}
\vskip .25cm

Get a integer handle corresponding to a given reduction operator. 
(Note that although it would appear to be far more convenient to 
pass the name of the reduction operator directly to the following
function call to {\t CCTK\_Reduce} this causes problems with the
translation of strings from {\t FORTRAN} to {\t C} with variable
argument lists).

{\t
\begin{verbatim}

#include ``Reduction.h''
int CCTK_Reduce(cGH *GH, int retvaltype, int retvalnum, void *retval, int handle, int index, ...);

\end{verbatim}
}
\vskip .25cm
Note that the memory for {\t retval} must be assigned before the reduction
call is made.

\subsection{IO}
\label{sec:io}

To allow flexible IO, the flesh itself does not provide 
any output routines, however it provides a mechanism for 
thorns to register different routines as IO methods. For 
details of writing IO thorns see ????. Application thorns
can interact with the different IO methods through the following
function calls:

{\t
\begin{verbatim}

#include ``CactusIOFunctions.h''
int CCTK_OutputGH(cGH *GH);

#include ``cctk.h''
call CCTK_OutputGH(GH)
    CCTK_POINTER GH
\end{verbatim}
}
\vskip .25cm

This call loops over all registered IO methods, calling 
the routine that each method has registered for {\t OutputGH}.
The expected behaviour of any methods {\t OutputGH} is to
loop over all GH variables outputting them if the method 
contains appropriate routines (that is, not all methods will 
supply routines to output all different types of variables) 
and if the method decides it is an appropriate time to 
output. 


{\t
\begin{verbatim}

#include ``CactusIOFunctions.h''
int CCTK_OutputVarAsByMethod(cGH *GH, const char *varname, const char *alias, const char *methodname);

#include ``cctk.h''
call CCTK_OutputVarAsByMethod(GH,varname,alias,methodname)
    CCTK_POINTER GH
    char* varname
    char* alias
    char* methodname
\end{verbatim}
}

Output a variable {\t varname} using the method {\t methodname} if it is 
registered. Uses {\t alias} as the name of the variable for the purpose
of constructing a filename. The output should take place if at all possible,
if the appropriate file exists the data is appended, otheriwise a new
file is created.


{\t
\begin{verbatim}

#include ``CactusIOFunctions.h''
int CCTK_OutputVarByMethod(cGH *GH, const char *varname, const char *methodname);

#include ``cctk.h''
call CCTK_OutputVarByMethod(GH,varname,methodname)
    CCTK_POINTER GH
    char* varname
    char* methodname
\end{verbatim}
}

Output a variable {\t varname} using the method {\t methodname} if it is 
registered. The output should take place if at all possible,
if the appropriate file exists the data is appended, otherwise a new
file is created.

{\t
\begin{verbatim}

#include ``CactusIOFunctions.h''
int CCTK_OutputVarAs(cGH *GH, const char *varname, const char *alias);

#include ``cctk.h''
call CCTK_OutputVarAs(GH,varname,alias)
    CCTK_POINTER GH
    char* varname
    char* alias
\end{verbatim}
}

Output a variable {\t varname} looping over all registered methods. 
The output should take place if at all possible,
if the appropriate file exists the data is appended, otherwise a new
file is created. Uses {\t alias} as the name of the variable for the purpose
of constructing a filename.

{\t
\begin{verbatim}

#include ``CactusIOFunctions.h''
int CCTK_OutputVar(cGH *GH, const char *varname);

#include ``cctk.h''
call CCTK_OutputVarAs(GH,varname)
    CCTK_POINTER GH
    char* varname
\end{verbatim}
}

Output a variable {\t varname} looping over all registered methods. 
The output should take place if at all possible,
if the appropriate file exists the data is appended, otherwise a new
file is created.



\section{Argument lists and parameters}
\label{sec:arlianpa}

\section{A First Example (Baloney)}

\section{Programming language differences}

\section{A more complex example (WaveToy)}

\section{Error handling, Warnings and Code Termination}
      
There are two CCTK commands to use for stopping the code
from within your thorn:
\begin{itemize}
\item{} To shut the code down cleanly, use
{\t
\begin{verbatim}
CCTK_Stop(pointer GH, INTEGER return_code)
CCTK_Stop(cGH *GH, int return_code);
\end{verbatim}
}
\item{} To shut the code down more violently, use
{\t
\begin{verbatim}
CCTK_Abort(pointer GH, INTEGER return_code)
CCTK_Abort(cGH *GH, int return_code);
\end{verbatim}
}
\end{itemize}
In both cases, an error code should be returned indicating the
error that you trapped which precipitated the code shutdown.
The error codes are detailed in the following table:
{\bf ACTION: Fill in the table QUERY}.

\section{Calls between different programming languages}
\label{sec:cabedipr}

\subsection{Calling C routines from FORTRAN}
\label{sec:cacrofr}

\subsection{Calling FORTRAN routines from C}
\label{sec:caforofr}

To call a utility Fortran routine from C use

{\tt

void FORTRAN\_NAME(<Fortran routine name>)(<argument list>)
}

\section{Programming Style Guidelines and Recommendations}
\label{sec:prstguan}

\section{Adding a test suite}
\label{sec:adatesu}

\section{Sharing your thorns/packages with others}
\label{sec:shyothwi}

\section{Naming conventions}

{\bf FIXME I just want to get this down. Gab}

\begin{itemize}

\item{} Thorn names must not start with the word ``Cactus'' (in
        any case).
\item{} Packages will be ignored if their names start with \# or .
        or end in \~ .bak or .BAK 
\item{} Thorns will be ignored if they are called doc or start with
        \# or . or end in \~ .bak or .BAK
\item{} Routine names have to be unique among all thorns. 
\item{} Sourcecode filenames do not have to be unique, but you
        cannot have the same root filename used by different languages.
        (For example, you cannot have initial.c and initial.F)


\end{itemize}


\section{Providing a Reduction Operator}
\label{sec:prareop}

{\bf QUERY: Where should this go ... infrastructure or thorn writer?
     Interface with driver layer? Gab}

To provide a reduction operator, you must provide a routine with 
the following interface

{\t
\begin{verbatim}

int MyReductionOperator(cGH *GH, int proc, int retvaltype, int numretval, void *retvals, int numinfields, int *infields);

\end{verbatim}
}
\vskip .25cm

\begin{itemize}
\item{} {\t int proc} If this argument is negative, the result should
        be returned to all processors, if it is positive the result
        need {\bf QUERY : Must? Gab} only be calculated and returned
        on one processor.
\item{} {\t int retvaltype} {\bf FIXME}
\item{} {\t int numretval} The number of values of type {\t retvaltype}
        being returned in the array {\t retvals}.
\end{itemize}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%