\begin{cactuspart}{2}{Application thorn writing}{$RCSfile$}{$Revision$}
\renewcommand{\thepage}{\Alph{part}\arabic{page}}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\chapter{Thorn concepts} 

      (This section has to contain enough explanation to make the rest of
       the writers guide readable the first time through)
(The hardest bugs to find are
those arising from plausible but incorrect assumptions about the
behavior of someone else's thorn.)

\begin{itemize}

\item 
Again probably emphasize collaboration, what are thorns, packages, 
how to share them.
\item 
Things to think about before you start programming:
             Language, read all the documentation, emphasize use of
             standard supported Cactus infrastructure
\item  Available data types
\begin{itemize}
\item Groups
\item Scalars
\item Arrays and GFs
\end{itemize}

\item  Understanding the RFR concept
\item  Understanding the GH concept
\item  Ghost zones and parallelism

\end{itemize}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Packages}

Thorns are grouped into {\em packages}.  This is a logical grouping of
thorns which is purely for administrative purposes.  The compiled
code knows nothing about packages.

The packages live in the `packages' directory off the main CCTK directory.  
Package names must be unique, and cannot start with a `\#', or finish
with `~' or `.bak'.

Inside a package directory there is a directory for each thorn belonging
to the package.  Thorn names have the same restrictions as package names,
with the addition that a thorn cannot be called `doc'.  This name is
reserved for package documentation.

\section{Implementations}

One of the key concepts for thorns is the concept of the {\bf implementation}.
Relationships between thorns are all based upon relationship between
{\bf implementations} and in principle it should be possible to replace
one thorn providing an implementation with another thorn providing that
implementation without affecting any other thorn.

An {\bf implementation} defines a group of variables and parameters which
are used to implement some functionality.  For example the thorn 
{\tt CactusPUGH/PUGH} provides the implementation {\tt driver}.  This 
implementation is responsible for providing memory for grid variables and
for communication.  Another thorn can also implement {\tt driver}, 
and both thorns can be compiled in {\em at the same time}.  Then, at runtime,
the user can decide which thorn providing {\tt driver} is used.  No other
thorn should be affected by this choice.

When a thorn decides it needs access to a variable or a parameter provided by 
another thorn, it defines a relationship between itself and the other thorn's
implementation, not explicitly with the other thorn.  This allows the
transparent replacement, at compile or runtime, of one thorn with another
thorn providing the same functionality as seen by the other thorns.

\section{Data Types and Sizes}

Cactus supports the following fixed size data types

\begin{center}
\begin{tabular}{|c|c|c|c|}
\hline
Data Type & Size (bytes) & Variable Type & Fortran Equivalent\\
\hline
{\t CCTK\_INT2}     & 2 & {\t CCTK\_VARIABLE\_INT2}     & {\t integer*2}\\
{\t CCTK\_INT4}     & 4 & {\t CCTK\_VARIABLE\_INT4}     & {\t integer*4}\\
{\t CCTK\_INT8}     & 8& {\t CCTK\_VARIABLE\_INT8}      & {\t integer*8}\\
{\t CCTK\_REAL4}    & 4 & {\t CCTK\_VARIABLE\_REAL4}    & {\t real*4}\\
{\t CCTK\_REAL8}    & 8 & {\t CCTK\_VARIABLE\_REAL8}    & {\t real*8}\\
{\t CCTK\_REAL16}   & 16& {\t CCTK\_VARIABLE\_REAL16}   & {\t real*16}\\
{\t CCTK\_COMPLEX4} & 4 & {\t CCTK\_VARIABLE\_COMPLEX4} & {\t complex*4}\\
{\t CCTK\_COMPLEX8} & 8 & {\t CCTK\_VARIABLE\_COMPLEX8} & {\t complex*8}\\
{\t CCTK\_COMPLEX16}& 16& {\t CCTK\_VARIABLE\_COMPLEX16}& {\t complex*16}\\
{\t CCTK\_CHAR}     & 1 & {\t CCTK\_VARIABLE\_CHAR}     & {\t character} \\ \hline
\end{tabular}
\end{center}

In addition Cactus provides three data types whose size is chosen
during the compilation process (at configuration time). This is to
allow the code to be easily run at different precisions. Note that
the effectiveness of running the code at a lower or higher precision
depends crucially on all thorns being used making consistent use
of the following data types:


\begin{center}
\begin{tabular}{|c|c|c|c|}
\hline
Data Type          & Default Size (bytes) & Variable Type & Configuration Option\\
\hline
{\t CCTK\_INT}     & 4 & {\t CCTK\_VARIABLE\_INT} & {\t INTEGER\_PRECISION}\\
{\t CCTK\_REAL}    & 8 & {\t CCTK\_VARIABLE\_REAL} & {\t REAL\_PRECISION}\\
{\t CCTK\_COMPLEX} & (8,8) & {\t CCTK\_VARIABLE\_COMPLEX} & Same as real precision\\
\hline
\end{tabular}
\end{center}

These variable types must be used by thorn writers to declare variables
 in the thorn interface files (FIXME: REF), and may be used to declare
variables in the thorn routines. Note that variable declarations in 
thorns should obviously match with definitions in the interface files
where appropriate.

Also provided, are a set of macros which
are interpreted by the preprocessor at compile time to signify which
data size is being used:

\begin{center}
\begin{tabular}{|c|c|}
\hline
Data Type & {\t \#define}\\
{\t CCTK\_INT2} & {\t CCTK\_INT\_PRECISION\_2} \\
{\t CCTK\_INT4} & {\t CCTK\_INT\_PRECISION\_4} \\
{\t CCTK\_INT8} & {\t CCTK\_INT\_PRECISION\_8} \\
{\t CCTK\_REAL4} & {\t CCTK\_REAL\_PRECISION\_4} \\
{\t CCTK\_REAL8} & {\t CCTK\_REAL\_PRECISION\_8} \\
{\t CCTK\_REAL16} & {\t CCTK\_REAL\_PRECISION\_16} \\
{\t CCTK\_COMPLEX4} & {\t CCTK\_COMPLEX\_PRECISION\_4} \\
{\t CCTK\_COMPLEX8} & {\t CCTK\_COMPLEX\_PRECISION\_8} \\
{\t CCTK\_COMPLEX16} & {\t CCTK\_COMPLEX\_PRECISION\_16} \\
\hline 
\end{tabular}
\end{center}

Note that the availability of these types, and the corresponding 
C data types are platform dependent.

\subsection{Fortran Thorn Writers}

Cactus provides a further data type {\tt CCTK\_POINTER} 
for use in Fortran code to declare a pointer passed from C. 
For example, the variable {\tt cctkGH} is of this type.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


\chapter{Anatomy of a thorn}

\section{thorns}

A thorn consists of a subdirectory of a package containing three 
administrative files

\begin{itemize}
\item{\tt interface.ccl}: the cactus interface, which defines the grid 
functions, variables, etc. See \ref{sec:in}.
\item{\tt param.ccl}: the parameters introduced by the thorn. See
\ref{param.ccl}.
\item{\tt schedule.ccl}: scheduling information for routines called by
the flesh.
\end{itemize}

Thorns can also contain  
\begin{itemize}
\item a subdirectory called {\tt src}, which should hold source files
\item a subdirectory {\tt src/include} for include files 
\item a {\tt README} containing a brief description of the thorn
\item a {\tt doc} directory for documentation
\item a {\tt par} directory for example parameter files
\item a {\tt test} subdirectory may also be added.  See \ref{sec:testsuites} for details.
\end{itemize}

\section{Creating a thorn}


To simplify the creation of a thorn, a make target {\tt
gmake newthorn} has been provided.

When this is run:

\begin{enumerate} 
\item{} You will be prompted for the name of the new thorn.
\item{} You will be prompted for the name of the package you would
like to include your thorn in. Either enter a new package name or pick 
one from the list of available packages that are shown.
\end{enumerate}


\section{Configuring your thorn}

The interaction of a thorn with the flesh and other thorns is controlled
by various configuration files.

These consist of:

\begin{itemize}

\item {\tt interface.ccl}
This defines the {\bf implementation} the thorn provides, and the variables
the thorn needs, along with their visibility to other implementations.

\item {\tt param.ccl}
This defines the parameters that are used to control the thorn, along
with their visibility to other implementations.

\item {\tt schedule.ccl}
This defines which functions from the thorn are called and when they are 
called.

\end{itemize}

\subsection{General syntax of CCL files}

CCL {\bf Cactus Configuration Language} files are simple text files
used to define configuration information for a thorn.  CCL files are
case independent, and may contain comments introduced by the `\#' character,
which marks the rest of the line to the right of its appearence as a comment.

\subsection{The {\tt interface.ccl}}

The {\tt interface.ccl} file is used to declare the implementation 
provided by the thorn, and to define the variables provided by it.

The implementation is declared by a single line

\begin{verbatim}
implements: <name>
\end{verbatim}

at the top of the file.  <name> can be any combination of alphanumeric 
characters and underscores, and is case independent as per everything else
in a CCL file.

There are three different access levels available for variables

\begin{itemize}
\item {\tt Public}
Can be `inherited' by other implementations (see below).
\item {\tt Protected}
Can be shared with other implementations which declare themselves to
be friends of this one (see below).
\item {\tt Private}
Can only be seen by this thorn.
\end{itemize}

Corresponding to the first two access levels there are two relationship
statements that can be used to get variables from other thorns.

\begin{itemize}
\item {\tt Inherits: <name>}
This gets all {\tt Public} variables from implementation <name>, plus all
variables it has in turn inherited.  An implementation may inherit from as
many thorns as it likes.
\item {\tt Friend: <name>}
This gets all {\tt Protected} variables from implementation <name>, but, 
unlike {\tt inherits} it pushes this implementations {\tt Protected}
vaiables onto implementation <name>.  This keyword is used to define
a group of implementations which all end up with the same {\tt Protected}
variables.
\end{itemize}

So, for example, an interface.ccl starting

\begin{verbatim}
implements: wavetoy
inherits:   grid
friend:     wave_extract
\end{verbatim}

declares that the thorn provides an implementation called `wavetoy', gets
all {\tt public} variables declared by an implementation called `grid', and
shares all {\tt protected} variables with `wave\_extract' and its friends.

For convenience variables are placed in groups.  The group has several
attributes:

\begin{itemize}

\item {\tt variable type}
e.g. REAL, INT, COMPLEX

\item {\tt name}
The name of the group

\item {\tt group type}
\begin{itemize}
\item {\tt SCALAR}
This is a single number.
\item {\tt GF}
This is an array of the default grid size.
\item {\tt ARRAY}
This is an array of any size.
\end{itemize}

\item {\tt Dim}
This is the dimension of the group.  (Meaningless for scalars.)

\item {TimeLevels}
This is the number of timelevels the group has.
\end{itemize}

A group specification consists of the variable type, followed by
the name of the group, then a space seperated list of the form
`attribute = value' .  Then a brace delimited block containing
a comma or newline seperated list of variables in the group.
A description of the group may be included on the line with the
closing brace.

For example

\begin{verbatim}

REAL fields type=GF TimeLevels=3 Dim=3
{
  phi
  a,b,c,d
} "Wave fields"

\end{verbatim}

defines a group of real GFs of dimension 3 each of which exists on
three time levels.

By default all groups are {\tt private}, to change this an access
specification of the form {\tt public:} or {\tt protected:} (or 
{\tt private:} to change it back) may be placed on a line by itself.  This
changes the access level for any group defined in the file from that point on.

All variables seen by any one thorn must have distinct names.

\subsection{The {\tt param.ccl}}

Users control the operation of thorns via parameters.  The {\tt param.ccl}
is used to specify the parameters used to control an individual thorn, and
to specify the values these parameters are allowed to take.  When the code
is run it reads a parameter file and sets the parameters if they fall
within the allowed values.

There are three access levels available for parameters:

\begin{itemize}
\item {\tt Global}
These parameters are seen by all thorns.
\item {\tt Restricted}
These parameters may be used by other implementations if they so desire.
\item {\tt Private}
These are only seen by this thorn.
\end {itemize}

A parameter specification consists of:
\begin{itemize}

\item {\tt the parameter type}
\begin{itemize}
\item {\tt REAL}
\item {\tt INT}
\item {\tt KEYWORD}
A distinct string with only a few known allowed values.
\item {\tt STRING}
An arbitrary string, which must conform to a given regular expression.
\item {\tt LOGICAL}
A boolean type which can take values 1, `t', `true', `yes' or 
0, `f', `false', `no'.
\end{itemize}

\item {\tt A description of the parameter}

\item {An allowed value block}
This consists of a brace delimited block of lines
describing the allowed values of the parameter.  Each range may
have a description associated with it by placing a :: on the line and
putting the description afterwards.

\item {\tt The default value}
This must be one of the allowed values.

\end{itemize}

For the numeric types INT and REAL, a range consists of a string of the
forms lower-bound:upper-bound:step where a missing number or a \* denotes
anything (i.e. infinite bounds or an infinitesmal step).

For example 

\begin{verbatim}

REAL Coeff "Important coefficient"
{
0:3.14 :: "Range has to be from zero to Pi, default is zero"
} 0.0

#No need to define a range for LOGICAL
LOGICAL nice "Nice weather ?"
{
}"yes"

# A example for a set of keywords and its default (which has to be
# defined in the body)
KEYWORD confused "Are we getting confused ?"
{
  "yes"    :: "absolutley positively"
  "perhaps" :: "we are not sure"
  "never"   :: "never"
} "never"
\end{verbatim}

defines a REAL parameter, a LOGICAL parameter, and a KEYWORD.

By default all paramters are {\tt private}, to change this an access
specification of the form {\tt global:} or {\tt restricted:} (or 
{\tt private:} to change it back) may be placed on a line by itself.  This
changes the access level for any parameter defined in the file from that point on.

To access {\tt restricted} parameters from another implementation, a line
containing {\tt shares: <name>} declares that all parameters mentioned in
the file from now until the next access specification originate in 
implementation <name>.  

In contrast to parameter declarations in other access blocks, the default
value must be ommitted - it is impossible to set the default value of any
parameter not originating in this thorn.

For example

\begin{verbatim}

friend:einstein

KEYWORD initial_data ""
{
  "bl_bh"         :: "Brill Lindquist black holes"
  "misner_bh"     :: "Misner black holes"
  "schwarzschild" :: "One Schwarzshild black hole"
}

\end{verbatim}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\subsection{The {\tt schedule.ccl}}

By default no routine of a thorn will be run.  The schedule.ccl file
defines those that should be run and when they should be run.

The specification of this is via a schudule block which consists of
a line of the form

\begin{verbatim}

schedule <name> at <time bin> [other options] 
{
  LANG:     <FORTRAN|C>
  STORAGE:  [group list]
  COMM:     [group list]
  TRIGGERS: [group or variable list]
} "A description"

\end{verbatim}

where <name> is the name of the routine, and <time bin> is one of

\begin{itemize}

\item {\tt CACTUS\_BASEGRID}
Resposnible for setting up coordinates etc.

\item {\tt CACTUS\_RECOVER}
For recovery from checkpoint.

\item {\tt CACTUS\_INITIAL}

For generating initial data.

\item {\tt CACTUS\_CPINITIAL}
For recovery of initial data from a checkpoint file.

\item {\tt CACTUS\_PRESTEP}

Stuff done before the evolution step.
\item {\tt CACTUS\_EVOL}      

The evolution step.

\item {\tt CACTUS\_POSTSTEP}

Stuff done after the evolution step.

\item {\tt CACTUS\_BOUND}

Boundary stuff. {\q Is this used ? }
\item {\tt CACTUS\_CHECKPOINT}
For checkpointing data

\item {\tt CACTUS\_ANALYSIS}
For analysing data.

\item {\tt CACTUS\_TERMINATE}
Called when cactus teminates.

\item {\tt CACTUS\_CONVERGENCE}
Convergence stuff.

\end{itemize}

The STORAGE and COMM keywords specify any groups which must have memory 
allocated for them or communication enabled for the duration of that routine.
The storage or communication status reverts to its previous status after the 
routine returns.

TRIGGERS is used when the routine is registered at ANALYSIS --- this is a 
special time bin, a routine registered here will only be called if one of
the variables mentioned in TRIGGERS is due for output.

The `other options' allow finer grained control of the scheduling.  It is
possible to state that the routine must run BEFORE or AFTER another routine.

As well as schedule blocks it's possible to embed C code in the schedule.ccl.
This can be used to schedule things based upon the value of a parameter.

E.g.

\begin{verbatim}

if(evolve_hydro)
{
  SCHEDULE hydro_predictor AT evolve AFTER metric_predictor BEFORE metric_corrector 
  {
    LANG:     FORTRAN
    STORAGE:  hydro_variables
    COMM:     hydro_variables
  } "Do a predictor step on the hydro variables"
}
\end{verbatim}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Naming Conventions for Source Files}

The make system uses file extensions
to designate coding language. The following extensions are
can be handled:

\begin{center}
\begin{tabular}{|c|c|}
\hline
Extension & Coding Language \\
\hline
{\t .F} & Fortran90 fixed form \\
{\t .f} & (no preprocessing) Fortran90 fixed form [FIXME: To be implemented]\\
{\t .F90} & Fortran90 free form form [FIXME: To be implemented]\\
{\t .f90} & (no preprocessing) Fortran90 free form [FIXME: To be implemented]\\
{\t .F77} & Fortran77 \\
{\t .f77} & (no preprocessing) Fortran77  [FIXME: To be implemented]\\
{\t .c} & C \\
{\t .cc} or {\t .C} & C++ \\
\hline
\end{tabular}
\end{center}

The following restrictions apply to file names:
\begin{itemize}
\item Rootnames within each directory must be unique. For example, it is not possible 
      to have the files {\tt InitialData.c} and {\tt InitialData.F} in the same directory.
\item Currently all files within a thorn must have distinct names.  We hope
to relax this in future.  Different thorns can have files with the same names. 
\end{itemize}

\section{Adding source files}

By default the CCTK looks in the {\tt src} directory of the thorn for source
files.

There are two ways in which to specify the sources.  The easiest is to use the 
{\tt make.code.defn} based method in which the CCTK does all the work, but you
may instead put a {\tt Makefile} in the {\tt src} directory and do everything
yourself.

\subsection{{\tt make.code.defn} based thorn building}

This is the standard way to do it.

If there is no file called {\tt Makefile} in the {\tt src} directory,
the CCTK make system looks for a file called {\tt make.code.defn} in that
directory.  At its simplest, this file contains two lines

\begin{itemize}
\item
\item {\t SRCS = <list of all source files {\em in this directory}>}

\item {\t SUBDIRS = <list of all subdirectories, {\em including subdirectories of subdirectories}>}

\end{itemize}

Each subdirectory listed should then have a {\tt make.code.defn} file 
containing just a {\tt SRCS = } line, a {\tt SUBDIRS = } line will
be ignored.

In addition, each directory can have a {\tt make.code.deps} file, which,
for files in that directory, can contain additional make rules and dependencies
for files in that directory.  See the GNU Make documentation for details of the
syntax.

\subsection{{\tt Makefile} based thorn building}

This method gives you ultimate responsibility.  The only requirement is that
a library called {\tt \$NAME} be created by the Makefile.

The makefile is passed the following variables

\begin{itemize}

\item {\tt \$(CCTK\_HOME)} - the main CCTK directory

\item {\tt \$(TOP)}    - the CONF directory

\item {\tt \$(SRCDIR)} - the directory in which the source files can be found

\item {\tt \$(CONFIG)} - the directory containing the configuration files

\item {\tt \$(THORN)}  - the thorn name

\item {\tt \$(SCRATCH\_BUILD)} - the scratch directory where f90 module
files should end up if they need to be seen by other thorns.

\item {\tt \$(NAME)}.

\end{itemize}

and has a working directory of <config>/build/<thorn\_name> .

\subsection{Other makefile variables}

\begin{itemize}
\item CC
\item CXX
\item F90
\item F77
\item CFLAGS
\item CXXFLAGS
\item F90FLAGS
\item F77FLAGS
\item LD
\item \ldots FIXME
\end{itemize}


\chapter{Putting code into your thorn}

\section{What the Flesh provides}

The flesh provides various things to thorns.
\begin{itemize}
\item {\tt Variables}
\item {\tt Parameters}
\item {\tt IO functions}
\item {\tt Information functions}
\item {\tt Reduction}
\item {\tt Interpolation}
\end{itemize}

\subsection{Variables}

\begin{itemize}
\item {\tt Headers}
\item {\tt Core things from the GH }
\item {\tt Grid variables}
\end{itemize}

\subsection{Parameters}

All parameters defined in a thorn's {\tt param.ccl} and all {\tt global} 
parameters appear as local variables in a thorn.  These variables are 
{\tt read only} and changes should not be made to them.  The effect of
changing a parameter is undefined.

\begin{itemize}
\item {\tt Headers}
\item {\tt Appearence of parameters}
\item {\tt String valued parameters}
\end{itemize}

\subsection{IO}
\label{sec:io}

To allow flexible IO, the flesh itself does not provide 
any output routines, however it provides a mechanism for 
thorns to register different routines as IO methods. For 
details of writing IO thorns see ????. Application thorns
can interact with the different IO methods through the following
function calls:

{\t
\begin{verbatim}

#include ``CactusIOFunctions.h''
int CCTK_OutputGH(cGH *GH);

#include ``cctk.h''
call CCTK_OutputGH(GH)
    CCTK_POINTER GH
\end{verbatim}
}
\vskip .25cm

This call loops over all registered IO methods, calling 
the routine that each method has registered for {\t OutputGH}.
The expected behaviour of any methods {\t OutputGH} is to
loop over all GH variables outputting them if the method 
contains appropriate routines (that is, not all methods will 
supply routines to output all different types of variables) 
and if the method decides it is an appropriate time to 
output. 


{\t
\begin{verbatim}

#include ``CactusIOFunctions.h''
int CCTK_OutputVarAsByMethod(cGH *GH, const char *varname, const char *alias, const char *methodname);

#include ``cctk.h''
call CCTK_OutputVarAsByMethod(GH,varname,alias,methodname)
    CCTK_POINTER GH
    char* varname
    char* alias
    char* methodname
\end{verbatim}
}

Output a variable {\t varname} using the method {\t methodname} if it is 
registered. Uses {\t alias} as the name of the variable for the purpose
of constructing a filename. The output should take place if at all possible,
if the appropriate file exists the data is appended, otheriwise a new
file is created.


{\t
\begin{verbatim}

#include ``CactusIOFunctions.h''
int CCTK_OutputVarByMethod(cGH *GH, const char *varname, const char *methodname);

#include ``cctk.h''
call CCTK_OutputVarByMethod(GH,varname,methodname)
    CCTK_POINTER GH
    char* varname
    char* methodname
\end{verbatim}
}

Output a variable {\t varname} using the method {\t methodname} if it is 
registered. The output should take place if at all possible,
if the appropriate file exists the data is appended, otherwise a new
file is created.

{\t
\begin{verbatim}

#include ``CactusIOFunctions.h''
int CCTK_OutputVarAs(cGH *GH, const char *varname, const char *alias);

#include ``cctk.h''
call CCTK_OutputVarAs(GH,varname,alias)
    CCTK_POINTER GH
    char* varname
    char* alias
\end{verbatim}
}

Output a variable {\t varname} looping over all registered methods. 
The output should take place if at all possible,
if the appropriate file exists the data is appended, otherwise a new
file is created. Uses {\t alias} as the name of the variable for the purpose
of constructing a filename.

{\t
\begin{verbatim}

#include ``CactusIOFunctions.h''
int CCTK_OutputVar(cGH *GH, const char *varname);

#include ``cctk.h''
call CCTK_OutputVarAs(GH,varname)
    CCTK_POINTER GH
    char* varname
\end{verbatim}
}

Output a variable {\t varname} looping over all registered methods. 
The output should take place if at all possible,
if the appropriate file exists the data is appended, otherwise a new
file is created.

\subsection{Reduction Operators}
\label{sec:reop}

Reduction operators are operators which process grid functions
and return, either on just one processor or on each processor,
a 1-Dimensional array of scalars. Examples of reduction operators
could include simple operations such as the maximum value of a 
grid function across the grid, or complex operations such as
{\bf FIXME: Think of a good example, Gab}.

The flesh itself does not provide any reduction operators,
instead providing a mechanism for thorns to register their
own routines as reduction operators, labelled by a given name.
For details of writing reduction operators see ????. Application
thorns can interact with the different registered reduction 
operators through the following function calls:

{\t
\begin{verbatim}

#include ``Reduction.h''
int CCTK_GetReductionHandle(const char *reduction);

\end{verbatim}
}
\vskip .25cm

Get a integer handle corresponding to a given reduction operator. 
(Note that although it would appear to be far more convenient to 
pass the name of the reduction operator directly to the following
function call to {\t CCTK\_Reduce} this causes problems with the
translation of strings from {\t FORTRAN} to {\t C} with variable
argument lists).

{\t
\begin{verbatim}

#include ``Reduction.h''
int CCTK_Reduce(cGH *GH, int retvaltype, int retvalnum, void *retval, int handle, int index, ...);

\end{verbatim}
}
\vskip .25cm
Note that the memory for {\t retval} must be assigned before the reduction
call is made.

\subsection{Interpolation}


\section{A First Example (Baloney)}

\section{Programming language differences}

\section{A more complex example (WaveToy)}

\section{Error handling, Warnings and Code Termination}
      
There are two CCTK commands to use for stopping the code
from within your thorn:
\begin{itemize}
\item{} To shut the code down cleanly, use
{\t
\begin{verbatim}
CCTK_Stop(pointer GH, int return_code)
CCTK_Stop(cGH *GH, int return_code);
\end{verbatim}
}
\item{} To shut the code down more violently, use
{\t
\begin{verbatim}
CCTK_Abort(pointer GH, int return_code)
CCTK_Abort(cGH *GH, int return_code);
\end{verbatim}
}
\end{itemize}
In both cases, an error code should be returned indicating the
error that you trapped which precipitated the code shutdown.
The error codes are detailed in the following table:
{\bf ACTION: Fill in the table QUERY}.

\section{Calls between different programming languages}
\label{sec:cabedipr}

\subsection{Calling C routines from FORTRAN}
\label{sec:cacrofr}

\subsection{Calling FORTRAN routines from C}
\label{sec:caforofr}

To call a utility Fortran routine from C use

{\tt

void FMODIFIER FORTRAN\_NAME(<Fortran routine name>)(<argument list>)
}

\section{Programming Style Guidelines and Recommendations}
\label{sec:prstguan}

\section{Adding a test suite}
\label{sec:adatesu}

\section{Sharing your thorns/packages with others}
\label{sec:shyothwi}

\section{Naming conventions}

{\bf FIXME I just want to get this down. Gab}

\begin{itemize}

\item{} Thorn names must not start with the word ``Cactus'' (in
        any case).
\item{} Packages will be ignored if their names start with \# or .
        or end in \~ .bak or .BAK 
\item{} Thorns will be ignored if they are called doc or start with
        \# or . or end in \~ .bak or .BAK
\item{} Routine names have to be unique among all thorns. 
\item{} Sourcecode filenames do not have to be unique, but you
        cannot have the same root filename used by different languages.
        (For example, you cannot have initial.c and initial.F)


\end{itemize}


\section{Providing a Reduction Operator}
\label{sec:prareop}

{\bf QUERY: Where should this go ... infrastructure or thorn writer?
     Interface with driver layer? Gab}

To provide a reduction operator, you must provide a routine with 
the following interface

{\t
\begin{verbatim}

int MyReductionOperator(cGH *GH, int proc, int retvaltype, int numretval, void *retvals, int numinfields, int *infields);

\end{verbatim}
}
\vskip .25cm

\begin{itemize}
\item{} {\t int proc} If this argument is negative, the result should
        be returned to all processors, if it is positive the result
        need {\bf QUERY : Must? Gab} only be calculated and returned
        on one processor.
\item{} {\t int retvaltype} {\bf FIXME}
\item{} {\t int numretval} The number of values of type {\t retvaltype}
        being returned in the array {\t retvals}.
\end{itemize}

\section{General Naming Conventions}

The following naming conventions are followed by the flesh and the
supported Cactus packages. They are not compulsory, but if followed
allow for a homogeneous code.

\begin{itemize}

\item Parameters: lower case (except for acronyms) with words separated
  by an underscore. Examples: {\tt my\_first\_parameter}, 
  {\tt solve\_PDE\_equation}.

\item Filenames and routine names: Prefixed by thorn name with an underscore, then capitalised words, with no spaces.
    Examples: {\tt MyThorn\_StartUpRoutine}, {\tt BestSolver\_InitialDataForPDE}.

\end{itemize}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


\end{cactuspart}