path: root/doc/UsersGuide/ThornWriters.tex

\part{Application thorn writing}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\chapter{Thorn concepts} 

      (This section has to contain enough explanation to make the rest of
       the writers guide readable the first time through)
(The hardest bugs to find are
those arising from plausible but incorrect assumptions about the
behavior of someone else's thorn.)
      a) Again probably emphasize collaboration, what are thorns,
         packages, how to share them.
      b) Things to think about before you start programming:
             Language, read all the documentation, emphasize use of
             standard supported Cactus infrastructure
      c) Available data types
         i)   Scalars
         ii)  Arrays and GFs
         iii) Groups
      d) Ghost zones and parallelism
      e) Understanding the RFR concept
      f) Understanding the GH concept

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\chapter{Anatomy of a thorn}

Each thorn must be in a package. Packages are fairly informal. If you
want, you can just make your own package directory, and stick your new
thorn into it. The idea is to collect the interdependencies and
functionality of a set of thorns.

\section{Creating a thorn}
You can use the following gmake target to create a new thorn: {\tt
gmake newthorn}. 
\begin{enumerate} 
\item{} You will be prompted for the name of the new thorn.
\item{} You will be prompted for the name of the package you would
like to include your thorn in. Either enter a new package name or pick 
one from the list of available packages that are shown.
\end{enumerate}

Cactus creates the following files in {\tt ./package/<package\_name>/<thorn\_name>}:
\begin{itemize}
\item{\tt README}: please fill out 
\item{\tt interface.ccl}: the cactus interface, which defined the grid 
fucntions, variables, etc. See \ref{sec:in}.
\item{\tt param.ccl}: the parameter introduced by your thorn. See
\ref{param.ccl}.
\item{\tt schedule.ccl}: schedules the routines provided by your thorn 
with the CCTK registry. Depending on the timing keywords chosen for
the routines, they will be executed appropriately.
\end{itemize}
These sub-directories are created:
\begin{itemize}
\item{\tt ./par}: will hold sample parameter files to show the usage
of your thorn.
\item{\tt ./doc}: this directory shoudl be used to document your
thorn.
\item{\tt ./test}: this directory will hold a {\em testsuite}, a set
of parameter files and their respective output. This is very important 
for the maintainers and your colleagues  to be able to support your
thorn adequately in the case of changes made to the core code or key thorns.
\item{\tt ./src}: this directory will hold the source and include
files of your thorn. It comes with file {\tt ./src/make.code.defn}
which indentifies the source files in this directory to the make
process.  The can be either in {\em Fortran} ending with {\tt *.F} or
in {\em C} ending in {\tt *.c}.
\item{}You may create additional sub-directories in {\tt ./src}, but 
you have to mention them in {\tt make.code.defn}.
\end{itemize}


\section{The {\tt param.ccl}}
The {\tt param.ccl} defines the parameters your thorn provides. To
avoid wrong assumptions by other thorns about your parameters, you have 
to speciy several informations, which we'll motivate here:
\begin{enumerate}
\item{} Are you creating a new parameter or are you extending
the value range of a parameter by a different thorns?
\item{} Is your parameter public to all other thorns, is it just
used by your thorn or is it available to a subset of thorns only ?
\item{} What are possible values your parameter can take: range of
numerical values or the set of keywords, what do they mean ?  What is
the parameter's default value ? 
\end{enumerate}
The {\em cactus computational language} lets you specify these
information in a straight forward manner. (For the actual syntax of the 
{\tt param.ccl} see \ref{sec:pa}.)
The parameters you define can be grouped in four classes: {\tt
private}, {\tt public},  {\tt protected} and {\tt friend}:{\em
thorn\_name}.  We'll go over them one by one: 
\begin{itemize}
\item{\tt private} This parameter can only be used by your thorn. It
it not even visible to other thorns: multiple private parameters of
the same name may exist in parallel. {\bf IS THIS TRUE ?} We give
examples of private parameters of type {\em integer}, {\em real}, {\em logical}
and {\em keyword}, which may have the following form (comments start with
{\tt \#}):
\begin{verbatim}

REAL Coeff "Important coefficient"
{
0:3.14 :: ``Range has to be from zero to Pi, default is zero''
} 0.0

INTEGER hitch ``The hitch-hiker parameter''
{
-42:42 :: ``Ranges from -42 to 42, default is zero''
} 0

#If you omitt the range, it will look a bit strange.
#better put a range!
INTEGER hitch_nolimits ``The confusing hitch-hiker parameter''
{
: :: ``We are confused, what is the range?''
} 0

#No need to define a range for LOGICAL
LOGICAL nice ``Nice weather ?''
{
}``yes''

# A example for a set of keywords and its default (which has to be
# defined in the body)
KEYWORD confused "Are we getting confused ?''
{
  "yes''    :: ``absolutley positively''
  "perhaps" :: "we are not sure''
  "never"   :: "never''
} "never''
\end{verbatim}
Please consult the syntax for the {\tt param.ccl} for other possible
combinations.
\item{\tt public} this class allows parameters to be read by {\em all} 
thorns a user will possibly compile with. You should make very
restricted use of the {\tt public} keyword since this can be the cause 
for incompatabilities and misunderstandings. {\bf CAN I HAVE A SECOND
(PRIVATE) PARAMETER OF THE SAME NAME ? IN A DIFFERENT THORN ?}
A typical declaration which further demonstrates the parameter syntax
could be: 
\begin{verbatim}
public:

# grid spacing has to be in the positive range 
REAL grid_dx "Coarse grid spacing in x-direction"
{
  0: :: ``grid spacing has to be positive, default is 0.3''
} 0.3
\end{verbatim}

\item{\tt protected} a parameter in this class is not only visible by
its own thorn, but can be {\em inherited} by friend: a different thorn 
may either check the value of this parameter or may extend its value
range. A typical application would be the generation of initial data
by different thorns, where each thorns extends an intial data
parameter value by a name indicating the type of data it implements.

\begin{verbatim}
protected:

KEYWORD initial_data "Physical spacetime model to evolve"
{
  "flat" :: "Flat Minkowski space"
} "flat"
\end{verbatim}

\item{\tt friend}:{\em thorn\_name} a parameter of type friend has been declared by an
other thorn and you are extending its value. You have
to specify which thorn has declared this parameter initially. This
``parent''-thorn has to identify the parameter as {\tt protected}
(see above) in order for you to modify its value range. A typical
delcaration for would look like:
\begin{verbatim}

#We inherit this parameter form thorn Einstein and add several models
#Only einstein and our thorn see this parameter. ******TRUE ???
#BTW: No need to add a new comment 

friend:einstein
EXTENDS KEYWORD initial_data ""
{
  "bl_bh"         :: "Brill Lindquist black holes"
  "misner_bh"     :: "Misner black holes"
  "schwarzschild" :: "One Schwarzshild black hole"
}
\end{verbatim}
Note, that you {\em cannot} reset the default value. The {\em ccl}
will report this as an error. Also, you cannot extend the default 
comment, rather document the individual extension in the body.
\end{itemize}

\section{The {\tt interface.ccl}}
{\bf CAN WE A CONVENTION FOR VARIABLES, GRIDFUNCTION, ...? }
The interface plays the inportant role of introducing new grid
function, their storage and communication concepts, new variables,
etc. to the CCTK infrastructure. While this interface convention looks 
difficult at first sight, it ensures the proper compatability of
thorns we can't even imagine at this point. And it is fairly easy to
learn. The thorn writer should question the impact and scope of his
implementation in the following way:
\begin{enumerate}
\item{What is the name of the implementation, so it can be referenced by other 
thorns.}
\item{On which other thorns (implementations) does the thorn depend,
because of parameter extension, grid functions, etc.}
\item{What are the grid functions to implement and how are they grouped ? A 
group should try to collect as many common feature as possible. Some
common features are mandatory, eg. the storage of the variables
belonging to the same group has to be equal.{\bf What else ?} That way
certain operation can be carried out on a group by group basis, instead
time consuming function by function.}.
\item{Do functions need to be communicated between different processors 
or are they local ?}
\item{Do variables need storage turned on all the time or are they
just used at a certain instance, for example for analysis.}
\end{enumerate}

The {\tt interface.ccl} defines all these specific information. The
syntax is described in appendix \ref{sec:in}, please see 
there for all possible options. Here we will introduce the general
idea behind the interface.ccl.

Here we show a sample taken from the thorn {\tt Einstein}, which
implements the ADM variables:
\begin{verbatim}
implements: einstein
inherits:   grid

public:

integer flags type = SCALAR
{
  conformal_state
  shift_state
} "State information"

real metric type = GF
{
  gxx,gxy,gxz,gyy,gyz,gzz
} "ADM 3-metric"

\end{verbatim}

The first lines in an {\em interface.ccl} file name the
implementation, which this thorns adds or expands and declare the
implementations this thorn depends on. In this case the thorn
implements {\em ``einstein''} and depends on the public and protected
variables of the {\em ``grid''} implementation, which add grids
specific data.

The gridfunctions and scalar variables, which will
be declared next, can be grouped in three classes -- very
similiar to the parameter declarations: the classes {\em public}, {\em 
protected} and {\em private} can be used to inform the system about
the scope of the individual varibles and gridfunctions:
\begin{enumerate}
\item{\tt public} grid functions, arrays or scalars defined in this
class can be {\em read and written} by all thorns, whitout having to
declare themselves as friends. This class should be used very
restrictive since unathourized altering of a varibable can cause sever 
confusion.
\item{\tt protected} grid functions of this can class can be read/modified by
other thorns, if these thorns have registered your implementation in
the {\tt inherit:} section.  In our example, you would be able to
access all variables of class protected of the {\tt grid}
implementation. The {\em heritage}-concept was developed in response
to the growing naming and access confusion in a growing pool of
thorns. It was not designed to give the thorn programmer a hard life,
but to make him think twice about possible sideeffects his
implementation will bring about.
\item{\tt private} grid functions in this class can only be
rad/modified by the {\bf IMPLEMENTATION} or {\bf THORN} itself.
\end{enumerate}

In this case all variables are of class {\em public} which means that
{\em all} other thorns can read and modify the values of these grids
functions/scalars. One should be very restrictive about the use of the 
{\em public} class. In general, declarations as {\em protected} and/pr 
{\em private} will serve as well but reduce the confusion on who owns
which variable significantly.

The variable/grid function can be of the following types: {\tt real}, {\tt
integer}, {\tt complex}. After the type declaration the {\em groupname} 
is given, so that certain operation can be carried out on a time
efficient group by group basis. In our example, the group names are
{\em ``flags''} and {\em ``metric''}. The next keywords denotes,
what kind of variable we have: {\tt type = }
\begin{enumerate}
\item{\tt SCALAR}: a scalar variable
\item{\tt GF}: a grid function of the size of the grid.
\item{\tt ARRAY}: an array of the size \ldots
{\bf {\tt grace} uses keywords DIM TIMELEVEL, how do they work and are 
there others ?}
\end{enumerate}

The actual varibables are declared in the body of the full variable
statement, enclosed by {\tt ``\{``} / {\tt ``\}''}. In our example,
for the first group, two scalars are defined ({\tt conformal\_state} and
{\tt shift\_state}), for the second group, the six components of the
metric tensor are defined. The declaration of the variables finishes
with a descriptive comment.

\section{The {\tt schedule.ccl}}
The {\tt schedule.ccl} is used to register the routines of your thorn
with the CCTK and defines the storage characteristics of a group. 

We will first describe the storage concepts. There is no good reason
to keep memory allocated for variables which are only used
intermittendly in the course of the simulation, eg. during output or
analysis, or are not used at all depending on parameter settings. The
CCTK allows for a flexible memory allocation:  
\begin{enumerate}
\item{} Variable memory can be turned on at all times, for example by
\begin{verbatim}
STORAGE: metric
\end{varbatim}, which would turn on storage for the group we defined
in the example above. 
\item{}All-time memory can turned on depending on parameter
settings. This can be done in the same way, the parameters are acessed 
within a C or Fortran code: by explicit use of the parameter variables 
in the case of integer/real/logical (see \ref{????}) or
by use of the {\tt CCTK\_Equals()} function (see
\ref{????}). For example:
\begin{verbatim}
# reserve memory for the group ``shift'' if parameter 
# shift NOT set to ``none'' 
if (!CCTK_Equals(shift,"none"))
{
  STORAGE: shift 
}

# allocate memory for the groups ``confac'',...
# if the LOGICAL parameter ``use_conformal'' was set to ``yes''
if (use_conformal) {
  STORAGE: confac,confac_1derivs,confac_2derivs
}
\end{verbatim}
\item{} The third options lets you associate the memory allocation
with a routine call: storage is allocated when the routine is called
by the CCTK. This options is explained in connection with the {\em
RFR} below.
\item{} A modification of \#3 is the options to have storage enabled,
when a certain variable is computed for output.  This options is
explained in connection with the {\em RFR} below.
\end{enumerate}


The {\em runtime function registry (RFR)} registers the different routines a 
a thorn provides with the CCTK depending on the timing keyword
chosen. You can assign timing keywords to your routines out of a 
set of ordered keywords and register your routine with the
CCTK. During the evolution the CCTK will loop through this ordered
set of keywords and execute the registered thorn routines. This way,
you can hook your thorns at well defined locations into the main
iteration loop without having 
to modify anybody's code. The CCTK currently features a fixed {\em
skeleton} of timing keywords, which mean that you cannot currently
register relative to a different thorn.

The timing keywords are actually aliased to numbers, which can be looked 
up in {\tt ./CCTK/src/include/rfr\_constants}. The current order of
timing keywords is given here:
\begin{verbatim}

 CACTUS_BASEGRID 
 CACTUS_RECOVER  
 CACTUS_INITIAL0 
 CACTUS_INITIAL  
 CACTUS_INITIAL2 
 CACTUS_INITIAL3  
 CACTUS_INITIAL4  
 CACTUS_INITIAL5 
 CACTUS_INITIAL6 
 CACTUS_INITIAL7  
 CACTUS_INITIAL8 
 CACTUS_INITIAL9  

 CACTUS_CPINITIAL

 CACTUS_PRESTEP  
 CACTUS_PRESTEP2 
 CACTUS_PRESTEP3 
 CACTUS_PRESTEP4  
 CACTUS_PRESTEP5 


 CACTUS_POSTSTEP   
 CACTUS_POSTSTEP2 
 CACTUS_POSTSTEP3  
 CACTUS_POSTSTEP4 
 CACTUS_POSTSTEP5 
 CACTUS_POSTSTEP6  
 CACTUS_POSTSTEP7
 CACTUS_POSTSTEP8  
 CACTUS_POSTSTEP9  
 CACTUS_POSTSTEP10 

 CACTUS_EVOL      
 CACTUS_BOUND      
 CACTUS_CHECKPOINT 
 CACTUS_ANALYSIS   
 CACTUS_TERMINATE 
 CACTUS_CONVERGENCE
\end{verbatim}
{\bf are going to name them CCTK ? Are we going to make a difference
between system keywords and user keywords  ? better yes}

The scheduling together with the storage options in {\tt schedule.ccl}
is described here in examples (please the ccl-syntax
\ref{sec:sc} for all possible options).
\begin{verbatim}

schedule initial at CACTUS_INITIAL
{
  LANG: Fortran
} "Initialisation for Einstein methods"


schedule metric_carttosphere at CACTUS_ANALYSIS
{
  STORAGE: spherical_metric
  LANG: Fortran
} "Calculate the spherical metric in r,theta(q), phi(p)"

schedule evaltrK at CACTUS_ANALYSIS
{
  STORAGE: trace_of_K,detofg
  LANG: Fortran
  TRIGGERS: trK,detg
} "Compute the trace of the extrinsic curvature"

\end{verbatim}
In the first example, the initialization routine {\tt initial}  is
schedules at instance {\tt CACTUS\_INITIAL}. The body of the 
declaration specifies the language used for this routine: options are
{\tt C} or {\tt FORTRAN}. This routines has no impact 
on storage behavior.

The second example registers the routine {\tt metric\_carttosphere} is
scheduled at {\tt CACTUS\_ANALYSIS}. (This routine converts the
cartesian metric to a spherical metric). The routine requests storage
when it is executed by the RFR. Memory is freed after the calculation.

The third example demonstrates yet another way qualifying execution
behavior: while CACTUS\_ANALYSIS registered routines are called on
every iteration, you wish to calculate certain variables only when
they are output. And this may not be needed on every iteration. By
placing the {\tt TRIGGER} keyword in the schedule-declaration, you
only excute this routine at CACTUS\_ANALYSIS, if there will be actually 
output: that is, when the current iteration number matches the
parameters {\tt itout1}, ...
{\bf in EINSTEIN: TRIGGERS: trK,detg. Why don't we use the actual GF
names ? }

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\chapter{Putting code into your thorn}

\section{What the F***h provides}

Header files, macros ...

\subsection{IO}

To allow flexible IO, the flesh itself does not provide 
any output routines, however it provides a mechanism for 
thorns to register different routines as IO methods. For 
details of writing IO thorns see ????. Application thorns
can interact with the different IO methods through the following
function calls:

{\t
\begin{verbatim}

#include ``CactusIOFunctions.h''
int CCTK_OutputGH(cGH *GH);

#include ``cctk.h''
CCTK_OutputGH(GH)
    CCTK_POINTER GH
\end{verbatim}
}

\vskip .25cm

This call loops over all registered IO methods, calling 
the routine that each method has registered for {\t OutputGH}.
The expected behaviour of any methods {\t OutputGH} is to
loop over all GH variables outputting them if the method 
contains appropriate routines (that is, not all methods will 
supply routines to output all different types of variables) 
and if the method decides it is an appropriate time to 
output. 


{\t
\begin{verbatim}

#include ``CactusIOFunctions.h''
int CCTK_OutputVarAsByMethod(cGH *GH, const char *varname, const char *alias, const char *methodname);

#include ``cctk.h''
CCTK_OutputVarAsByMethod(GH,varname,alias,methodname)
    CCTK_POINTER GH
    char* varname
    char* alias
    char* methodname
\end{verbatim}
}

Output a variable {\t varname} using the method {\t methodname} if it is 
registered. Uses {\t alias} as the name of the variable for the purpose
of constructing a filename. The output should take place if at all possible,
if the appropriate file exists the data is appended, otheriwise a new
file is created.


{\t
\begin{verbatim}

#include ``CactusIOFunctions.h''
int CCTK_OutputVarByMethod(cGH *GH, const char *varname, const char *methodname);

#include ``cctk.h''
CCTK_OutputVarByMethod(GH,varname,methodname)
    CCTK_POINTER GH
    char* varname
    char* methodname
\end{verbatim}
}

Output a variable {\t varname} using the method {\t methodname} if it is 
registered. The output should take place if at all possible,
if the appropriate file exists the data is appended, otherwise a new
file is created.

{\t
\begin{verbatim}

#include ``CactusIOFunctions.h''
int CCTK_OutputVarAs(cGH *GH, const char *varname, const char *alias);

#include ``cctk.h''
CCTK_OutputVarAs(GH,varname,alias)
    CCTK_POINTER GH
    char* varname
    char* alias
\end{verbatim}
}

Output a variable {\t varname} looping over all registered methods. 
The output should take place if at all possible,
if the appropriate file exists the data is appended, otherwise a new
file is created. Uses {\t alias} as the name of the variable for the purpose
of constructing a filename.

{\t
\begin{verbatim}

#include ``CactusIOFunctions.h''
int CCTK_OutputVar(cGH *GH, const char *varname);

#include ``cctk.h''
CCTK_OutputVarAs(GH,varname)
    CCTK_POINTER GH
    char* varname
\end{verbatim}
}

Output a variable {\t varname} looping over all registered methods. 
The output should take place if at all possible,
if the appropriate file exists the data is appended, otherwise a new
file is created.



\section{Argument lists and parameters}

\section{A First Example (Baloney)}

\section{Programming language differences}

\section{A more complex example (WaveToy)}

\section{Error handling, Warnings and Code Termination}
      
There are two CCTK commands to use for stopping the code
from within your thorn:
\begin{itemize}
\item{} To shut the code down cleanly, use
{\t
\begin{verbatim}
CCTK_Stop(pointer GH, INTEGER return_code)
CCTK_Stop(cGH *GH, int return_code);
\end{verbatim}
}
\item{} To shut the code down more violently, use
{\t
\begin{verbatim}
CCTK_Abort(pointer GH, INTEGER return_code)
CCTK_Abort(cGH *GH, int return_code);
\end{verbatim}
}
\end{itemize}
In both cases, an error code should be returned indicating the
error that you trapped which precipitated the code shutdown.
The error codes are detailed in the following table:
{\bf ACTION: Fill in the table QUERY}.

\section{Calls between different programming languages}
\subsection{Calling C routines from FORTRAN}
\subsection{Calling FORTRAN routines from C}

\section{Programming Style Guidelines and Recommendations}

\section{Adding a test suite}

\section{Sharing your thorns/packages with others}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%