path: root/doc/UsersGuide/RunningCactus.tex

\begin{cactuspart}{1}{Installation and Running}{$RCSfile$}{$Revision$}
\renewcommand{\thepage}{\Alph{part}\arabic{page}}

\chapter{Overview of documentation}

This documentation splits naturally into these parts: 

\begin{itemize}

\item {\bf The history of Cactus and the Concepts behind the
    CCTK. Getting you up and running.}
  We give an overview on the required hardware and
  software and we will talk you through the installation of a working
  CCTK. You will be able to verify the correct installation by the
  CCTK test suite. We show how to perform some basic simulation and
  introduce the concepts behind data output. We provide brief
  information on how to visualize the 1D, 2D and 3D data.

\item {\bf Part2: How to write a physics thorn.} We introduce a
  sample thorn that illustrates the implementation of simple initial
  data and the subsequent evolution. You will learn how to use the
  programming interface to take advantage of parallelism and modularity.

\item {\bf Part3: How to writes an infrastructure thorn.} In this more
  advanced part, we talk about user supplied infrastructure routines
  as {\em additional output routines, boundary conditions, etc.}

\item {\bf The standard package.} 

\end{itemize}

Other topics will be discussed in separate documents:

\begin{itemize}

\item The numerical relativity package.

\item A description of the flesh, for the maintainers.

\end{itemize}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\chapter{About Cactus}

\section{A Cactus history}
\label{sec:acahi}

To begin with a one-phrase summary: Cactus is a numerical
infrastructure for solving partial differential equations (typically
in 3-dimensional space and time), a modular code for solving the
Einstein equations, and an international collaboration in numerical
general relativity pooling ideas in this code.

Cactus 1.0 was written in 1996 by Joan Mass\'o and Paul Walker out of
frustration with the numerical relativity codes they had been working
on. It has come a long way. The name is meant to suggest a modular
structure consisting of a ``flesh'' (or backbone, or core) and
``thorns'' (or modules, or collections of subroutines). Some essential
features of the code have been present from the beginning:

\begin{itemize}

\item Parallelism (the work is split between multiple processors, or
multiple computers) that is largely automatic and hidden from the
user.

\item A modular structure with a fixed interface between the flesh and the
thorns, which allows users to add new parts.

\item Giving the user a choice of C and Fortran for writing thorns, while
automating the interface between the two languages.

\item A spirit of open-source collaboration similar to the gnu
project.

\end{itemize}

Other aspects have changed completely since Cactus 1.0. The flesh
started out as a complete numerical relativity code, using one
particular formulation of the Einstein equations, with the thorns
providing only initial data, analysis, and other peripheral
functions. In 4.0 not only the Einstein equations and general
relativity basics, but even input/output and parallelism are provided
by thorns. The only function of the flesh is now to define the
interface between thorns, to hold varying collections of thorns, and
to provide an interface between C and Fortran routines. 

The aim of this radicalism is to allow for future growth. We now
distinguish informally between physics thorns and infrastructure
thorns. The main infrastructure project for now is adding different
flavors of adaptive mesh refinement -- while the user writing a
physics thorn can still large she is working a single-processor,
unigrid code. Similarly, an infrastructure thorn could be replaced by
one that does the same job, but simply better. Even the flesh could be
rewritten in the future, but the interface between the flesh and
thorns is to remain unchanged.

Cactus thorns can be written in F90, F77 and C (or a mixture). The
decision to support Fortran was made for the pragmatic reason that
many people like it, or don't write C. This decision does make the
flesh more complex. The flesh is written in C, and in perl that
generates C.

The change from Cactus 3.0 to Cactus 4.0 is the largest one yet. The
new interface forces modifications on all existing thorns, but it is
intended not to change again. 

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Cactus concepts} 
\label{sec:caco}

Cactus is designed to solve time evolution problems in three space
dimensions by finite differencing on a Cartesian grid. This typically
includes solving elliptic equations to obtain initial data, evolving
them by hyperbolic or parabolic equations, and perhaps solving more
elliptic equations at each time step.  One basic concept is therefore
a series of initialization steps followed by a do-loop that implements
the time evolution, along with periodic analysis and output. Problems
that do not naturally fit into this concept are methods which are
global in space or time (such as pseudo-spectral methods) or methods
which do not have a grid (such as finite elements). Thorns interact
with the flesh by, among other things, announcing when they want to be
called in this basic schedule.

Another basic concept is parallelism by distributing spatial domains
among processors. Finite differencing is an almost local procedure,
which means that each processor can work on its domain of space,
needing to communicate only with neighboring processors in order to
work on points of the boundary of its domain. The domain decomposition
is automatic and largely hidden. A physics thorn can pretend to work
on a single grid, which in reality is only a chunk of the complete
numerical domain. This chunk consists of an inner region of points the
processor ``owns'', and needs to update, surrounded by a ``ghost
zone'' of points which it can use but doesn't need to update -- they
are updated by neighboring processors. Clearly this concept is geared
towards explicit finite differencing schemes for the time-evolution
part. {\q Say something about elliptic equations here, and implicit
schemes}

The notion that we are dealing with a collection of fields that live
on the same (typically three-dimensional) space, and that evolve
together in time defines the third basic concept, that of ``grid
functions'' grouped together in a ``grid hierarchy''. Multiple
grid hierarchies arise, for example, when one wants to cover the
surface of a sphere with two Cartesian coordinate patches, or in
adaptive mesh refinement.  Grid functions are defined on Cartesian
grids that are equally spaced in each of the three coordinates $x$,
$y$ and $z$. (These could be spherical type coordinates, although that
was not the original intention.) Grid functions can be two or
one-dimensional as well as three dimensional.

Routines are grouped in thorns, and thorns informally in packages,
such as the base package, and the general relativity
package. The routines inside a thorn can interact in any way, but
thorns interact with the flesh and with each other only through a
well-defined interface. Confirming to this interface guarantees that
thorns will be compatible with, and useful for, all future extensions
of the code. The interface also helps users and developers to
understand what a given thorn does. 

The flesh and many thorns are public and can be used by anyone,
assuming ethical behavior. Public thorns and the flesh are distributed
by CVS . Public thorns are checked out in
packages. The flesh may change in the future but this should never
affect users, as Cactus 4.0 is making every effort to codify and
freeze the interface. Public thorns should of course not lose
important functionality they once had. (Nevertheless, CVS gives the
user access to all earlier versions.)

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\chapter{Installation} 
\label{sec:in}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Required software}
\label{sec:reqo}

In general, Cactus {\em requires} the following set of software to function
in a single processor mode. Please refer to the architecture section
\ref{sec:suar} for architecture specific items.
\begin{itemize}
\item{\tt Perl4.0} Perl is used extensively during the CCTK
  thorn configuration phase. Perl is available for nearly all
  operating systems known to man and can be obtained at
  http://www.perl.org
\item{\tt GNU make} The make
  process works with the GNU make utility (referred to as {\bf gmake} 
  henceforth) . While other make utilities may also work, this is not 
  guaranteed. Gmake can be obtained from your favorite GNU site.
\item{\tt C/C++} A C and C++ compiler. For example, the GNU compilers. These
 are available for most supported platforms.  Platform specific compilers 
 should also work. Currently the flesh uses no C++
 and so it should be possible to work on platforms where no C++ compiler is
 available.  
\item{\tt CVS} The {\em ``Concurrent Versioning System''} is not needed
  to run/compile the CCTK, but you are strongly encourage to install
  this software to take advantage of the update procedures. It can be
  downloaded from your favorite GNU site.  Tar files of each release are
  also available.
\end{itemize}

\noindent
To use the CCTK, with the default driver (CactusPUGH/PUGH) on multiple
processes you also need:
\begin{itemize}
\item{\tt MPI} the {\it Message Passing Interface (MPI)} 
which provides inter-processor communication. 
Supercomputering sites often supply a native MPI implementation with
which the CCTK is very likely to be compatible. Otherwise there are
various freely available ones available, e.g. the {\tt MPICH}
version of MPI is available for various architectures and operating
systems at {\tt http://www-unix.mcs.anl.gov/mpi/}. 
\end{itemize}

\noindent
If you are using any thorns containing routines 
written in {\tt FORTRAN} you also need
\begin{itemize}
\item{\tt F90/F77} For routines written in F77, either an F90 or an F77
 compiler can be used. For routines written in F90 an F90 compiler is
 required. There is a very limited set of free F90 compilers available
 for the different architectures.
\end{itemize}

\noindent
While not required for compiling or running the CCTK, for thorn development
it is useful to install
\begin{itemize}
\item{\tt ctags/etags}: Tags enables you browse through the calling structure
  of a program by help of function call database. Navigating the CCTK and
  packages becomes very easy. Emacs and vi both support this method. See
  \ref{sec:usta} for a short guide to ``tags''.
\end{itemize}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Supported architectures}
\label{suar}

Cactus runs on many machines, under a large number of Unix operating
systems. Here we list all the machines we have compiled and verified
Cactus on, including some architecture specific notes. 
\begin{itemize} 
\item{\bf SGI Origin 2000} running Irix
\item{\bf SGI} 32 or 64 bit running Irix
\item{\bf Cray T3E} 
\item{\bf Dec Alpha}: Dec operating system and Linux. Single processor
  mode and MPI supported. The Decs need to have the GNU {\tt C/C++} 
  compilers installed {\q Is this really true ?}
\item{\bf Intel Linux}: There is a
  free Linux F90 compiler available from  {\tt http://www.psrv.com}
  -- the only free we know of. Single processor mode and MPICH
  supported.
\item{\bf Windows NT}: successful compile with the following software:
  \begin{itemize}
  \item{\tt DIGITAL Visual Fortran Optimizing Compiler Version: V5.0C}
  \item{\tt Microsoft (R) 32-bit C/C++ Optimizing Compiler Version
        11.00.7022 for 80x86}
  \item{Cygnus Unix suite version 1.19}
  \end{itemize}
  {FIXME: update this}
\item{\bf HP Exemplar} (Soon)
\end{itemize}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Checkout procedure}
\label{sec:chpr}

The CCTK is distributed, extended, and maintained using the free CVS
software. CVS  allows many
people to work on a large software project together without getting
into a tangle. For the beginner, we summarize the basics in appendix
\ref{sec:uscv}, please refer to this section for a short description of
the CVS syntax.

The space required for an installation depends on the packages and
thorns used. The flesh on its own requires less than 5 MB.

\begin{itemize}
\item{\bf Login}: Prior to any CVS operation, you need to log into the CCTK
  repository. For an anonymous checkout, type:\\
  {\tt
    cvs -d :pserver:cvs\_anon@cvs.cactuscode.org:/usr/users/cactus login
    }
  You will be prompted for a password which is {\bf anon}. 
\item{\bf Checkout}: To obtain a fresh copy of the CCTK, move to a directory
  which does not contain a previously checked out version, and type
  {\t
    cvs -d :pserver:cvs\_anon@cvs.cactuscode.org:/usr/users/cactus checkout CCTK
    }
  The CVS checkout procedure will create a directory called {\bf
  CCTK} and install the CCTK inside this directory.  From now on we
  will reference all directory names relative to {\bf CCTK}. 

\noindent
  If you want to compile the CCTK with thorns, you now need to checkout separately 
  the required packages into the {\bf packages} directory. To see  the 
  available Cactus packages and thorns type
  {\t 
    cvs -d :pserver:cvs\_anon@cvs.cactuscode.org:/usr/users/cactus checkout -s
  }
  To check out a package or thorn type go to the packages directory,  {\t cd packages},
  and  for a package type
{\t 
  cvs checkout <package\_name>
  }
	or for just one thorn
{\t 
cvs checkout <package\_name/thorn\_name>
}
 
To simplify this procedure you may use {\t gmake checkout} in the CCTK
home directory which provides menus to pick packages and thorns from.
  
  
\item{\bf Update}: To update an existing CCTK checkout (to patch in
  possible changes, etc.), do the following {\em within} the {\tt CCTK} directory.
  {\t
    cvs update
    }
  The update process will operate recusrively downwards from your current position
  within the CCTK tree. To update only on certain directories, change
  into these directories and issue the update command.
\item{\bf CVS status}: to obtain a status report on the ``age'' of your
  CCTK or package routines (from your current directory position
  downward), type
  {\t
    cvs status
    }
\item{\bf non-anonymous CVS}: if you have an account at the central
  repository and would like to perform any of the operation above
  {\em non-anonymously}, replace {\tt cvs\_anon} by your login name
  and provide the appropriate password during the CVS login
  process. Depending on your permissions, you may then make commits to the CCTK
  or its packages.
\item{\bf Commits}: you need to perform a personalized login and have
  proper permissions to commit code to the repository. 
\end{itemize}
For more CVS commands on how to add files, etc. please refer to appendix 
\ref{sec:uscv}.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Directory structure}
\label{sec:dist}

A fresh CCTK checkout creates a directory {\tt CCTK} with the
following subdirectories:
\begin{itemize}

\item{\tt CVS} the CVS book-keeping directory, present in every subdirectory.

\item{\tt doc} CCTK documentation

\item{\tt lib} contains libraries.

\item{\tt src} contains the source code for the CCTK

\item {\tt packages} contains the Cactus packages. The packages
  (the actual ``physics'') are not supplied by the CCTK. If the packages
  you want to use are part of the central repository, they can be
  checked out in similar way the CCTK. 
\end{itemize}

When Cactus is first compiled it creates a new directory {\tt
CCTK/configs}. Disk space may be a problem on supercomputers where
home directories are small. A workaround is to first create a
configs directory on scratch space, say {\tt scratch/cactus\_configs/} (where
{\tt scratch/} is your scratch directory), and soft link ({\tt ln -s
scratch/cactus\_configs CCTK/configs/}) it to the Cactus directory.

Configurations are descibed in detail in section \ref{sec:coaco}.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Getting help}
\label{sec:gehe}

For tracking problem reports and bugs we use GNATS, which is bugtracking 
system published under the GNU license. We have set up a webinterface at 
{\tt http://www.cactuscode.org} which allows easy submission and browsing 
of problem reports.    

A description of the GNATS categories we feature is dscribed in the appendix 
\ref{sec:usgn}.

% OK, there is NO emacs at the moment, because the GNATS setup is really stupid
% and sendpr handles like c.... besides the fact, that the user has to go 
% through a make-process which installs stuff somewhere on his HD. gerd.
% \begin{itemize}
% \item {\tt A web interface}
% \item {\tt SendPR}
% {FIXME: Mention the emacs thing here too...}
% \end{itemize}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\chapter{Compilation} 

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


Cactus can be built in different configurations from the same copy of
the source files, and these different configurations coexist in the
{\tt CCTK} directory. Here are several cases, where this can be
useful:

\begin{enumerate}
\item{}Different configurations can be for {\em different
architectures}. You can keep executables for multiple architecures
based on a single copy of source code, shared on a common file
system.
\item{} You can compare different {\em compiler options, debug-modes}.
  You might want to compile different communication protocols
  (e.g. MPI/GLOBUS) or leave them out all together.
\item{} You can have different configurations for {\em different thorns
    collections} compiled into your executable.
\end{enumerate}

Once a configuration has been created, by {\tt gmake <configuration name>} as described
in detail in the next section, a single call to {\tt gmake} will compile the 
code.  The first time will display your compile ThornList, and give you the
chance to edit it before continuing.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Creating a configuration}
\label{sec:coaco}

At its simplest, this is done by {\tt gmake <confname>}.  This will generate
a configuration with the specified name, doing its best to automatically
determine the default compilers and compilation flags suitable for the current 
architecture.

There are a number of additional command line arguments which may be supplied 
to override some parts of the procedure.

\subsection{Configuration options}

There are two ways to pass options to the configuration process from the
gmake command line.  
\begin{enumerate}
\item{} Add the options to a configuration file and use,

{\tt gmake <configuration name>-config  options=<filename>} 

All available configuration options may be set in the file, any
which are not set will take the default value. The options file
should contain lines of the form:

{\tt <option> [=] ...}

The equals sign is optional.

\item{} Pass the options individually  on the command line,

{\tt gmake <configuration name>-config  <option name>=<chosen value>, ...}
Not all configuration options can be set on the command line. Those than can
be set are indicated in the table below.

\end{enumerate}

Note that if a configuration file is used, and options are also passed
on the command line, the configuration file will currently override the command line
options, although this behaviour will soon change.

It is important to note that these methods cannot be used to, for example add
options to the default values for {\tt CFLAGS}. Setting {\tt CFLAGS} in the
configuration file, or the command line will overwrite completely the 
default values.

\subsection{Available options}

There are a plethora of available options.

\begin{itemize}
\item {\tt Compiler and tool specification}

These are used to specify which compilers and other tools to use. Entries followed
by [CL] may be specified on the command line.

\begin{itemize}
\item {\tt CC}
[CL] The C compiler.
\item {\tt CXX}
The C++ compiler
\item {\tt F90}
[CL] The Fortran 90 compiler
\item {\tt F77}
[CL] The FORTRAN 77 compiler.
\item {\tt LD}
[CL] The linker.
\item {\tt AR}
The archiver used for generating libraries.
\item {\tt Ranlib}
The archive indexer to use.
\item {\tt MKDIR}
The program to use to create a directory.
\item {\tt SHELL}
The shell to use.

\end{itemize}

\item {\tt Compilation and tool flags}

Flags which are passed to the compilers and the tools.

\begin{itemize}
\item {\tt CFLAGS}
Flags for the C compiler

\item {\tt CXXFLAGS}
Flags for the C++ compiler

\item {\tt F90FLAGS}
[CL] Flags for the Fortran 90 compiler

\item {\tt F77FLAGS}
[CL] Flags for the FORTRAN 77 compiler

\item {\tt LDFLAGS}
[CL] Flags for the linker

\item {\tt ARLAGS}
Flags for the archiver

\end{itemize}

\item {\tt Library flags}

Used to specify auxiliary libraries and directories to find them in.

\begin{itemize}

\item {\tt LIBS}
The additional libraries.

\item {\tt LIBDIRS}
Any other library directories.

\end{itemize}

\item {\tt Extra include directories}

\begin{itemize}
\item {\tt SYS\_INC\_DIRS}
Used to specify any additional directories for system include files
\end{itemize}


\item {\tt Precision options}

\begin{itemize}

Used to specify the precision of the default real and integer data types,
specified as the number of bytes the data takes up.  Note that not all
values will be valid on all architectures.

\item {\tt REAL\_PRECISION}
[CL] Allowed values are 16,8,4 .

\item {\tt INTEGER\_PRECISION}
[CL] Allowed values are 8,4 and 2 .

\end{itemize}

\item {\tt Executable name}

\begin{itemize}
\item {\tt EXE }
The name of the executable.
\end{itemize}

\item {\tt Compile-only flags}

Used to specify options to compilers to make them only compile.

\begin{itemize}
\item {\tt CCOMPILEONLY}

\item {\tt FCOMPILEONLY}
\end{itemize}

\end{itemize}


\subsection{File layout after a configuration has been created }

The configuration process sets up various subdirectories and files in the 
{\tt configs} directory to contain the configuration specific files, these
are placed in a directory with the name of the configuration.

\begin{itemize}

\item {\tt config-data} : contains the files created by the configure
script:

The most important ones are

\begin{itemize}

\item {\tt make.config.defn} 
contains compilers and compilation flags for a configuration.  

\item {\tt config.h}
The main configuration header file, containing architecture specific
definitions.

\item {\tt cctk\_archdefs.h}
An architecture specific header file containing things which cannot be
automatically detected, and have thus been hand-coded for this architecture.
\end{itemize}

These are the first files which should be checked or modified to suite any
peculiarities of this configuration.

In addition the following files may be of some use.

\begin{itemize}
\item {\tt fortran\_name.pl} 
A perl script used to determine how the Fortran compiler names subroutines.  
This is used to make some C routines callable from Fortran, and Fortran 
routines callable from C.

\item {\tt make.config.deps}
Initially empty.  Can be edited to add extra architecture specific dependencies
needed to generate the executable.

\item {\tt make.config.rule} 
Make rules for generating object files from source files.

\end{itemize}

Finally, autoconf generates the following files.

\begin{itemize}

\item {\tt config.log}
A log of the autoconf process.

\item {\tt config.status}
A scrit which may be used to regenerate the configuration.

\item {\tt config.cache}
An internal file used by autoconf.

\end{itemize}

\item {\tt lib} 
An empty directory which will contain the libraries created for each thorn.

\item {\tt build} 
An empty directory which will contain the object files generated for this 
configuration, and preprocessed source files.

\end{itemize}


\section{Building and Administering a configuration}
\label{sec:buanadaco}

One you have created a new configuration, the command

{\tt gmake <configuration name>}

will build an executable, prompting you along the way for the 
thorns which should be included. There is a range of  gmake 
targets and options which are detailed in the following sections.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{gmake targets for building and adminstering configurations}
\label{sec:gmta}

A target for {\tt gmake} can be naively thought of as an argument
that tells it which of several things listed in the {\tt Makefile} it
is to do. The command {\tt gmake help} lists all gmake targets.

\begin{itemize}

\item {\tt gmake <CONF>} builds a configuration. If the configuation doesn't exist
			it will create it

\item {\tt gmake <CONF>-config} creates a new configuration or reconfigures an old one

\item {\tt gmake <CONF>-clean} removes all object and dependancy files from
  a configuration 

\item {\tt gmake <CONF>-realclean} removes from a configuration
  all object and dependency files, 
  as well as files generated from the CST (that is, only the files
  generated by configure and the ThornList files remain)

\item {\tt gmake <CONF>-cleanobjs} removes all object files from
  a configuration 

\item {\tt gmake <CONF>-cleandeps} removes all dependency files from
  a configuration 


\item {\tt gmake <CONF>-rebuild} rebuilds a configuration (reruns the CST -- the perl scripts which parse the thorn configuration files)

\item {\tt gmake <CONF>-delete} deletes a configuration ({\tt rm -r configs/<CONF>})

\item {\tt gmake <CONF>-thornlist} regenerates the ThornList for a configuration

\item {\tt gmake <CONF>-testsuite} runs the test programs associated with
 each thorn. See section \ref{sec:????} for information about the 
 test suite mechanism.

\end{itemize}



\subsection{Compiling in thorns}
\label{sec:cointh}

Cactus compiles in all thorns listed in {\tt configs/CONF/ThornList}, 
if they have been checked out.

The {\tt ThornList} file is simply a list of the form
{\t <package>/<thorn>}.  All text on appearing on a line after a \# sign
is ignored, so comments can be included.

Instead of using the editor to specify the thorns you want to
  have compiled, you can {\em edit} the {\em ThornList} outside
  the make process. It is located in {\tt configs/<CONF>/ThornList},
  wher {\tt <CONF>} refers to the name of your configuration.
  For a completely new configuration, this directory exists {\em
    after} the first  make phase.

\subsection{Notes and Caveats}
\begin{itemize}
\item{} If during the build you see the error ``{\tt missing
    seperator}'' you are probably not using GNU make. 
\item{} {\em The EDITOR environment variable}. You may not be aware of
  this, but this thing very often exists and may be set  by default to
  something scary like vi. If you don't know how to use vi or wish to
  use your favorite editor instead, reset this environment variable.
\end{itemize}

\subsection{gmake options for building configurations}
\label{sec:gmta}

An {\it option} for gmake can be thought of as an arguement which tells
it how it should make a target. Note that the final result is always
the same.

\begin{itemize}

\item {\tt gmake <target> SILENT=no} prints the commands that gmake is executing
\item {\tt gmake <target> TJOBS=<number>} compile in parallel, across thorns 
\item {\tt gmake <target> FJOBS=<number>} compile in parallel, across files within each thorn

\end{itemize}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%




%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Other gmake targets}

\begin{itemize}


\item {\tt gmake help} lists all make options

\item {\tt gmake tags} creates a {\tt vi} style tags file. See section
  \ref{sec:usta} for using TAGS within Cactus.

\item {\tt gmake TAGS} creates an Emacs style TAGS file. See section
  \ref{sec:usta} for using TAGS within Cactus.

\item {\tt gmake doc} places a postscript version of the Users Guide documentation in your Cactus home directory.

\item {\tt gmake checkout} allows you to easily checkout Cactus packages and thorns.

\item {\tt gmake distclean} delete your {\tt configs} directory and hence all your configurations.

\item {\tt gmake downsize} removes non essential files as documents
  and testsuites to allow for minimal installation size.

\end{itemize}


\section{Testing} 
\label{sec:te}

Some thorns come with a testsuite, consisting of example parameter files
and the output files generated by running these.

These testsuites serve the dual purpose of

\begin{itemize}
\item {\tt Regression testing}
i.e. making sure that changes to the thorn or the flesh don't affect the
output from a known parameter file.
\item {\tt Portability testing}
i.e. checking that the results are independent of the architecture - this
is also of use when trying to get Cactus to work on a new architecture.
\end{itemize}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\chapter{Running Cactus}

This chapter covers

\begin{itemize}
\item
 Command line options
\item
Simple parameter file syntax (will be explained fully in thorn
         writers part)
\item
Description of standard parameters (although maybe should be an
         appendix)
\item
Checkpointing and outputting
\item
Reference to web pages with machine dependant stuff
\item
Understanding the screen output (RFR tree, errors and warnings etc)
\item
Convergence mode
\item
Environment variables
\end{itemize}

\section{Command Line Options}
\label{sec:coliop}

The code accepts numerous command line arguments:

\begin{tabular}{ll}
{\t -O, -describe-all-parameters} &List all parameters and their descriptions\\
{\t -o, -describe-parameter <param>} & List description for one parameter\\
{\t -T, -list-thorns}& List all compiled thorns\\
{\t -t, -test-thorn-compiled <package/thorn>}& Query presence of a thorn\\
{\t -h, -help, -?} & Provide help on command line options\\
{\t -v, -version} & Compilation date of code\\
{\t -x, -test-parameters [<nprocs>]} & Run code as far a parameter checking, simulating being on {\t nprocs} processors\\
{\t -W, -warning-level <level>} & Set warning level to {\t level} (default is 1)\\
{\t -E, -error-level <level>} & Set error level to {\t level}, warnings of this level and above will stop the code (default is 0)\\
{\t -r, -redirect-stdout} & Redirect standard output to file on all processors\\
\end{tabular}

\begin{itemize}
\item {\tt -O}
Produces a full list of all parameters from all thorns which were compiled,
along with descriptions and allowed values.
\item {\tt -o} 
Produces the description and allowed values for a given parameter - takes one
argument.
\item {\tt -T} 
Produces a list of all the thorns which were compiled in.
\item {\tt -t} 
Checks if a given thorn was compiled in - takes one argument.
\item {\tt -h} or {\tt -?}
Produces a help message.
\item {\tt -v} 
Produces version information of the code.
\item {\tt -x} 
Runs the code far enough to check the consistency of the parameters.  If
given a numeric argument it will attempt to simulate being on that number 
of processors.
\item {\tt -W} 
Sets the warning level of the code.  All warning messages are given a level--
the lower the level the greater the severity.  This parameter controls the
level of messages to be seen.  0 indicates only fatal messages.
\item {\tt -E}
This works in concert with {\tt -W} - it controls which warning level is
treated as a fatal error.  This cannot be set to a higher value than 
{\tt -W}.
\item {\tt -r}
This redirects the standard output of each processor to a file.  By default
the output from processors other than processor 0 is discarded.
\end{itemize}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Parameter File Syntax}

The parameter file is used to control the behaviour of the code at runtime.
It consists of a text file the lines of which are either comments, denoted
by a `\#', or parameter statements.

A parameter statement consists of one or more parameter names, followed by
and `=', followed by the value(s) for this (these) parameter(s).

The name of a parameter consists of:

\begin{itemize}
\item {\tt Global parameters}
Just name of the parameter itself.
\item {\tt Restricted parameters}
The name of the {\em implementation} which defined the parameter, two colons,
and the name of the parameter --- e.g. `driver::global\_nx'.
\item {\tt Private parameters}
The name of the {\em thorn} which defined the parameter, two colons,
and the name of the parameter --- e.g. `wavetoy::amplitude'.
\end{itemize}

In addition there is a parameter {\em ActiveThorns} which tells the code
which {\em thorns} are to be used.  Only parameters from active thorns can
be set, and only active thorns get their {\tt startup} and {\tt RFR} routines
called.  By default all thorns are inactive.  {\bf This should be the first
parameter in your parameter file.}



%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


%\chapter{Looking at output}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Checkpointing}

\begin{itemize}
\item recover directive
\item parameter clobbering
\item controlling checkpoint times
\end{itemize}


\section{Screen output}

\begin{itemize}

\item Normal output
\begin{itemize}
\item Thorn activation
\item RFR stuff
\item Iteration stuff
\item Recovery stuff
\item Checkpoint stuff
\item Timing stuff
\item End stuff
\end{itemize}

\item Error outputs
\begin{itemize}
\item Thorn activation errors
\item Parameter error
\item Parameter file errors
\item Other errors
\end{itemize}

\end{itemize}


\section{Environment Variables}

\begin{itemize}
\item MPI ?
\item Anything else ?
\end{itemize}

\end{cactuspart}