% /*@@
%   @file      RunningCactus.tex
%   @date      27 Jan 1999
%   @author    Tom Goodale, Gabrielle Allen, Gerd Lanferman, Thomas Radke
%   @desc
%              How to run Cactus part of the Cactus User's Guide
%   @enddesc
%   @version   $Header$
% @@*/

\begin{cactuspart}{Installation and Running}{$RCSfile$}{$Revision$}
\label{part:RunningCactus}
\renewcommand{\thepage}{\Alph{part}\arabic{page}}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\chapter{Installation}
\label{cha:in}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Required software}
\label{sec:required_software}

In general, Cactus \emph{requires} the following set of software to function
in single processor mode. Please refer to the architecture section
\ref{sec:suar} for architecture specific items.
\begin{Lentry}
\item[Perl5.0] Perl is used extensively during the Cactus
  thorn configuration phase. Perl is available for nearly all
  operating systems known to man and can be obtained at
  \url{http://www.perl.org}
\item[GNU make] The make
  process works with the GNU make utility (referred to as \texttt{gmake}
  henceforth). While other make utilities may also work, this is not
  guaranteed. Gmake can be obtained from your favorite GNU site or
  from \url{http://www.gnu.org}
\item[C] C compiler. For example, the GNU compiler. This
 is available for most supported platforms.  Platform specific compilers
 should also work.
\item[CPP] C Preprocessor. For example, the GNU \texttt{cpp}.  These are
  normally provided on most platforms, and many C compilers have an option
  to just run as a preprocessor.
\item[CVS] The \textit{Concurrent Versions System} is not needed
  to run/compile Cactus, but you are strongly encourage to install
  this software to take advantage of the update procedures. It can be
  downloaded from your favorite GNU site.  Tar files of each release are
  also available.
\end{Lentry}

\noindent
To use Cactus, with the default driver\footnote{For help with unfamiliar terms, please consult the glossary, Appendix \ref{sec:glossary}.} (\texttt{CactusPUGH/PUGH}) on multiple
processors you also need:
\begin{Lentry}
\item[MPI] The \textit{Message Passing Interface}
which provides inter-processor communication.
Supercomputing sites often supply a native MPI implementation with
which Cactus is very likely to be compatible. Otherwise there are
various freely available ones available, e.g. the MPICH
version of MPI is available for various architectures and operating
systems at \url{http://www-unix.mcs.anl.gov/mpi/}.
\end{Lentry}

\noindent
If you are using any thorns containing routines
written in C++ you also need
\begin{Lentry}
\item[C++] C++ compiler. For example, the GNU compiler. This
 is available for most supported platforms.  Platform specific compilers
 should also work.  Note that if a C++ compiler is available then the
 \text{main()} routine in the Flesh is compiled with C++ to allow static
 class initialisations.
\end{Lentry}

\noindent
If you are using any thorns containing routines
written in Fortran you also need
\begin{Lentry}
\item[F90/F77] For routines written in Fortran 77, either an Fortran 90 or
 a Fortran 77 compiler can be used. For routines written in Fortran 90
 a Fortran 90 compiler is obviously required. There is a very limited set of
 free Fortran 90 compilers available for the different architectures.
\end{Lentry}

\noindent
While not required for compiling or running Cactus, for thorn development
it is useful to install
\begin{Lentry}
\item[\texttt{ctags/etags}] These programs enable you browse through the
  calling structure of a program by help of a function call database.
  Navigating the Flesh and arrangements becomes very easy. Emacs and
  \texttt{vi} both support this method. See \ref{sec:Appendix.tags} for a short
  guide to tags.
\end{Lentry}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Supported architectures}
\label{sec:suar}

Cactus runs on many machines, under a large number of operating
systems.  Here we list the machines we have compiled and verified
Cactus on, including some architecture specific notes.  A complete
list of architectures supported by Cactus, along with more notes, can
be found at
\begin{center}
\url{http://www.cactuscode.org/Documentation/Architectures.html}.
\end{center}

\begin{Lentry}
\item[\textbf{SGI}] 32 or 64 bit running Irix.
\item[\textbf{Cray T3E}]
\item[\textbf{Compaq Alpha}]  Compaq operating system and Linux.
  Single processor
  mode and MPI supported. The Alphas need to have the GNU C/C++
  compilers installed.
\item[\textbf{IA32}] running Linux, OpenBSD, FreeBSD, or Windows 2000/NT.
  Single processor mode and MPI (MPICH and LAM) supported.\\
  On Windows Cactus compiles with Cygwin.  MPI (WMPI, HPVM, and MPIPro)
  supported.  Please read \texttt{doc/README.NT} for details.
\item[\textbf{IA64}]  running Linux.
\item[\textbf{Macintosh PowerPC}] (MacOS X and Linux PPC)
\item[\textbf{IBM SP2,SP3,SP4}] 32 or 64 bit running AIX.
\item[\textbf{Hitachi SR8000-F1}]
\item[\textbf{Sun} Solaris]
\item[\textbf{Fujitsu}]
\item[\textbf{NEC SX-5, SX-6}]
\end{Lentry}

The following machines are only partially supported
\begin{Lentry}
\item[\textbf{HP Exemplar}]
\end{Lentry}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Checkout procedure}
\label{sec:checkout}

Cactus is distributed, extended, and maintained using the free CVS
software (\textit{Concurrent Versions System}: \url{http://www.cvshome.org}).
CVS  allows many people to work on a large software project
together without getting into a tangle.
Since Cactus thorns are distributed from several repositories on the
main CVS site, and from a growing number of user sites, we provide a
script, described below,  on our web site for checking out the Flesh and thorns.
The Cactus web site also provides a form interface for direct download.

CVS experts who want to use raw CVS commands are directed to
Appendix~\ref{sec:Appendix.cvs} for full instructions. For CVS novices,
we also summarize in this appendix basic CVS commands.

The space required for an installation depends on the arrangements and
thorns used. The Flesh on its own requires less than 5 MB.

The script for checking out the Flesh and distribution thorns,
\texttt{GetCactus}, is available from the web site at

\url{http://www.cactuscode.org/download/GetCactus}

The
script takes as an argument the name of a file containing a \textit{ThornList},
that is a list of thorns with the syntax

\begin{alltt}
<\var{arrangement name}>/<\var{thorn name}>
\end{alltt}

If no filename is given, only the Flesh is checked out.
Optional directives in the ThornList indicate which CVS repository to fetch
thorns from. The default is to take the thorns from the same repository as
the Flesh. A full description of ThornList syntax is provided in Appendix~\ref{chap:th}.
ThornLists for example applications are provided on the Cactus web site.

The same script can be used to checkout additional thorns.

Another script, \texttt{MakeThornList}, can be used to produce a minimal
ThornList from a given Cactus par file.  It needs a \emph{master} ThornList
to be copied into your \texttt{~\.cactus} directory.

See \url{http://www.cactuscode.org/toolkit/makeThornList/}.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Directory structure}
\label{sec:dist}

A fresh checkout creates a directory \texttt{Cactus} with the
following subdirectories:

\begin{Lentry}

\item[\texttt{CVS}] the CVS bookkeeping directory, present in every subdirectory

\item[\texttt{doc}] Cactus documentation

\item[\texttt{lib}] contains libraries

\item[\texttt{src}] contains the source code for Cactus

\item [\texttt{arrangements}] contains the Cactus arrangements. The arrangements
  (the actual ``physics'') are not supplied by checking out just Cactus.
  If the arrangements you want to use are standard Cactus arrangements, or
  reside on our CVS repository (\texttt{cvs.cactuscode.org}),
  they can be checked out in similar way to the Flesh.
\end{Lentry}

When Cactus is first compiled it creates a new directory
\texttt{Cactus/configs}, which will contain all the source code, object files
and libraries created during the build process.  Disk space may be a problem
on supercomputers where home directories are small.

A workaround is to first create a
configs directory on scratch space, say \texttt{scratch/cactus\_configs/} (where
\texttt{scratch/} is your scratch directory), and then either
\begin{itemize}
\item{} set the environment variable \texttt{CACTUS\_CONFIGS\_DIR} to point to
this directory
\end{itemize}
or
\begin{itemize}
\item{}  soft link this directory (\texttt{ln -s
scratch/cactus\_configs Cactus/configs/}) to the Cactus directory, if your
filesystem supports soft links.
\end{itemize}

Configurations are described in detail in section \ref{sec:configurations}.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Getting help}
\label{sec:gehe}

For tracking problem reports and bugs we use GNATS, which is a bug tracking
system published under the GNU license. We have set up a web interface at
\url{http://www.cactuscode.org} which allows easy submission and browsing
of problem reports.

A description of the GNATS categories which we use is provided in the appendix
\ref{sec:Appendix.gnats}.

% OK, there is NO emacs at the moment, because the GNATS setup is really stupid
% and sendpr handles like c.... besides the fact, that the user has to go
% through a make process which installs stuff somewhere on his HD. gerd.
% BUT, we could distribute our own, either copy cvsbug, or write a perl
% version.  Tom
% \begin{itemize}
% \item \texttt{A web interface}
% \item \texttt{SendPR}
% {FIXME: Mention the emacs thing here too...}
% \end{itemize}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\chapter{Compilation}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


Cactus can be built in different configurations from the same copy of
the source files, and these different configurations coexist in the
\texttt{Cactus/configs} directory. Here are several instances in which
 this can be useful:

\begin{enumerate}
\item{}Different configurations can be for \emph{different
architectures}. You can keep executables for multiple architectures
based on a single copy of source code, shared on a common file
system.
\item{} You can compare different \textit{compiler options, debug-modes}.
  You might want to compile different communication protocols
  (e.g. MPI or Globus) or leave them out all together.
\item{} You can have different configurations for \textit{different thorn
    collections} compiled into your executable.
\end{enumerate}

Once a configuration has been created, by \texttt{gmake <\var{config}>} as
described in detail in the next section, a single call to
\texttt{gmake <\var{config}>} will compile the code.  The first time it
generates a compile \texttt{ThornList},  and gives you the chance to edit
it before continuing.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Creating a configuration}
\label{sec:configurations}

At its simplest, this is done by \texttt{gmake <\var{config}>}\footnote
%
{A note on the Cactus make system --- if at any point it prompts you
to enter something, the default value, which will be assumed if you
simply press enter, is shown in parentheses.}
%
.  This generates a
configuration with the name \texttt{\var{config}}, doing its best to
automatically determine the default compilers and compilation flags
suitable for the current architecture.

There are a number of additional command line arguments which may be supplied
to override some parts of the procedure.

\subsection{Configuration options}

There are three ways to pass options to the configuration process.
% from the gmake command line.
\begin{enumerate}
  \item[1]{}
    Pass options individually in shell environment variables:

    {\tt export <option name>=<chosen value>    \# for bash}\\
    {\tt setenv <option name> <chosen value>    \# for (t)csh}\\
    {\tt gmake <configuration name>-config}

  \item[2a]{}
    Either: create a default configuration file \texttt{\$\{HOME\}/.cactus/config}.

    All available configuration options may be set in a default options file
    \texttt{\$\{HOME\}/.cactus/config}, any which are not set will take a
    default value. The file should contain lines of the form:

    \texttt{<\var{option}> [=] ...}

    The equals sign is optional. Spaces are allowed everywhere.
    Text starting wit a \texttt{'\#'} character will be ignored as a comment.

  \item[2b]{}
    Or: list your Cactus configuration files in an environment variable
    \texttt{CACTUS\_CONFIG\_FILES}:

    \texttt{gmake <\var{config name}>-config
                  CACTUS\_CONFIG\_FILES=$<$\var{list of config files}$>$}

    Multiple configuration files, with their file names separated by a
    \texttt{':'} character, will be processed in order.
    Each file should be given by its full path.
    The options file has the same format as \texttt{\$\{HOME\}/.cactus/config}.

  \item[3]{}
    Add the options to a configuration file and use,

    \texttt{gmake <\var{config name}>-config  options=<\var{filename}>}

    The options file has the same format as \texttt{\$\{HOME\}/.cactus/config}.
    (Note these options are \emph{added} to those from the
    \texttt{\$\{HOME\}/.cactus/config} file.)

  \item[4]{}
    Pass the options individually on the command line,

    \texttt{gmake <\var{config name}>-config
                           <\var{option name}>=<\var{chosen value}>, ...}

    Not all configuration options can be set on the command line.
    Those that can be set are indicated in the table below.
\end{enumerate}

They are listed here in order of increasing precedence, e.g. options
set on the command line will take priority over (potentially
conflicting) options set in \texttt{\$\{HOME\}/.cactus/config} or other
Cactus configuration files. Default options from
\texttt{\$\{HOME\}/.cactus/config} will only be read if the
environment variable \texttt{CACTUS\_CONFIG\_FILES} is not set.

It is important to note that these methods cannot be used to, for example, add
options to the default values for \texttt{CFLAGS}.  Setting any variable in the
configuration file or the command line will overwrite completely the
default values.

\subsection{Available options}
\label{subsec:Compilation-Available_Options}

There is a plethora of available options.

\begin{itemize}

\item {Cross Compiling}

If you are compiling on an architecture other than the one you are
producing an executable for, you will need to pass the

\begin{Lentry}
\item [\texttt{HOST\_MACHINE=\var{x-x-x}}]
\end{Lentry}
option, where \texttt{\var{x-x-x}} is the canonical name of the architecture
you are compiling for, such as \texttt{sx6-nec-superux};  the format is
\texttt{\var{processor}-\var{vendor}-\var{OS}}.

\item {Compiled Thorns}

These specify the chosen set of thorns for compilation. If the thorn choice is not provided
during configuration, a list containing all thorns in the
\texttt{arrangements} directory
is automatically created, and the users prompted for any changes.

\begin{Lentry}

\item [\texttt{THORNLIST}] Name of file containing a list of thorns with
the syntax \texttt{<\var{arrangement name}>/<\var{thorn name}>}, lines
beginning with \verb|#| or \texttt{!} are ignored.

\item [\texttt{THORNLIST\_DIR}] Location of directory containing
\texttt{THORNLIST}.
This defaults to the current working directory.


\end{Lentry}

\item {Compiler and tool specification}

These are used to specify which compilers and other tools to use. Entries followed
by * may be specified on the command line.

\begin{Lentry}
\item [\texttt{CC}]   * The C compiler.
\item [\texttt{CXX}]    The C++ compiler.
\item [\texttt{F90}]  * The Fortran 90 compiler.
\item [\texttt{F77}]  * The Fortran 77 compiler.
\item [\texttt{CPP}]    The preprocessor used to generate dependencies
for and to preprocess C and C++ code.
\item [\texttt{FPP}]    The preprocessor used to generate dependencies
for and to preprocess Fortran code.
\item [\texttt{LD}]   * The linker.
\item [\texttt{AR}]     The archiver used for generating libraries.
\item [\texttt{RANLIB}] The archive indexer to use.
\item [\texttt{MKDIR}]  The program to use to create a directory.
\item [\texttt{PERL}]   The name of the Perl executable.
\end{Lentry}

\item {Compilation and tool flags}

Flags which are passed to the compilers and the tools.

\begin{Lentry}
\item [\texttt{CFLAGS}]
Flags for the C compiler.

\item [\texttt{CXXFLAGS}]
Flags for the C++ compiler.

\item [\texttt{F90FLAGS}]
* Flags for the Fortran 90 compiler.

\item [\texttt{F77FLAGS}]
* Flags for the Fortran 77 compiler.

\item [\texttt{CPPFLAGS}]
Flags for the preprocessor (used to generate compilation dependencies
for and preprocess C and C++ code).

\item [\texttt{FPPFLAGS}]
Flags for the preprocessor (used to generate compilation dependencies
for and preprocess Fortran code).

\item [\texttt{MKDIRFLAGS}]
  Flags for \texttt{MKDIR} so that no error is given if the directory exists.

\item [\texttt{LDFLAGS}]
* Flags for the linker.  \emph{Warning:} This variable is ignored
while the compilers and linkers are autodetected.  This can lead to
strange errors while configuring.  You can pass the linker flags in
the variable \texttt{LD} instead.

\item [\texttt{ARFLAGS}]
Flags for the archiver.

\item [\texttt{C\_LINE\_DIRECTIVES}]
Whether error messages and debug information in the compiled C and C++
files should point to the original source file or to an internal file
created by Cactus.  The only options available are \texttt{yes} and
\texttt{no}, the default is \texttt{yes}.  Set this to \texttt{no} if your
compiler reports error messages about unrecognised \verb|#| directives.

\item [\texttt{F\_LINE\_DIRECTIVES}]
Whether error messages and debug information in the compiled Fortran
files should point to the original source file or to an internal file
created by Cactus.  The only options available are \texttt{yes} and
\texttt{no}, the default is \texttt{yes}.  Set this to \texttt{no} if your
compiler reports error messages about unrecognised \verb|#| directives.

\item [\texttt{DISABLE\_REAL16}] Disable support for the data type
\texttt{CCTK\_REAL16}.  The only options available are \texttt{yes} and
\texttt{no}, the default is \texttt{no}.  Cactus autodetects this data type
  only for C.  If the C compiler supports it, but the Fortran compiler
  does not, it may be necessary to disable \texttt{CCTK\_REAL16}
  altogether, since Cactus assumes that data types are fully supported
  if they exist.

\item [\texttt{DEBUG}]
* Specifies what type of debug mode should be used,
the default is no debugging.
Current options are \texttt{yes}, \texttt{no}, or \texttt{memory}. The option
\texttt{yes} switches on all debugging features, whereas \texttt{memory} just
employs memory tracing (\ref{sec:metr}).

\item [\texttt{C\_DEBUG\_FLAGS}]
Debug flags for the C compiler, their use depends on the type of
debugging being used.

\item [\texttt{CXX\_DEBUG\_FLAGS}]
Debug flags for the C++ compiler, their use depends on the type of
debugging being used.

\item [\texttt{F90\_DEBUG\_FLAGS}]
Debug flags for the Fortran 90 compiler, their use depends on the
type of debuggin being used.

\item [\texttt{F77\_DEBUG\_FLAGS}]
Debug flags for the Fortran 77 compiler, their use depends on the
type of debugging being used.

\item [\texttt{OPTIMISE, OPTIMIZE}]
* Specifies what type of optimisation should be used. The only options currently
available are \texttt{yes} and \texttt{no}.  The default is to use optimisation.\\
Note that the British spelling \texttt{OPTIMISE} will be checked first and, if set,
will override any setting of the American-spelled \texttt{OPTIMIZE}.

\item [\texttt{C\_OPTIMISE\_FLAGS}]
Optimisation flags for the C compiler, their use depends on the type of
optimisation being used.

\item [\texttt{CXX\_OPTIMISE\_FLAGS}]
Optimisation flags for the C++ compiler, their use depends on the type of
optimisation being used.

\item [\texttt{F90\_OPTIMISE\_FLAGS}]
Optimisation flags for the Fortran 90 compiler, their use depends on the
type of optimisation being used.

\item [\texttt{F77\_OPTIMISE\_FLAGS}]
Optimisation flags for the Fortran 77 compiler, their use depends on the
type of optimisation being used.

\item [\texttt{PROFILE}]
  * Specifies what type of profiling should be used.  The only options
  currently available are \texttt{yes} and \texttt{no}.  The default
  is to use no profiling.

\item [\texttt{C\_PROFILE\_FLAGS}]
  Profile flags for the C compiler, their use depends on the type of
  profiling being used.

\item [\texttt{CXX\_PROFILE\_FLAGS}]
  Profile flags for the C++ compiler, their use depends on the type of
  profiling being used.

\item [\texttt{F90\_PROFILE\_FLAGS}]
  Profile flags for the Fortran 90 compiler, their use depends on the
  type of profilin being used.

\item [\texttt{F77\_PROFILE\_FLAGS}]
  Profile flags for the Fortran 77 compiler, their use depends on the
  type of profiling being used.

\item [\texttt{WARN}]
  * Specifies what type of build warnings should be used.  The only
  options currently available are \texttt{yes} and \texttt{no}.  The
  default is to produce no warnings.

\item [\texttt{C\_WARN\_FLAGS}]
Warning flags for the C compiler, their use depends on the type of
warnings used during compilation (\ref{sec:gmopfobuco}).

\item [\texttt{CXX\_WARN\_FLAGS}]
Warning flags for the C++ compiler, their use depends on the type of
warnings used during compilation (\ref{sec:gmopfobuco}).

\item [\texttt{F90\_WARN\_FLAGS}]
Warning flags for the Fortran 90 compiler, their use depends on the type of
warnings used during compilation (\ref{sec:gmopfobuco}).

\item [\texttt{F77\_WARN\_FLAGS}]
Warning flags for the Fortran 77 compiler, their use depends on the type of
warnings used during compilation (\ref{sec:gmopfobuco}).

\end{Lentry}

\item {Architecture-specific flags}

\begin{Lentry}
\item [\texttt{IRIX\_BITS=32|64}] For Irix SGI systems: whether to build a 32- or 64-bit configuration.
\end{Lentry}

\begin{Lentry}
\item [\texttt{AIX\_BITS=32|64}] For IBM SP systems: whether to build a 32- or 64-bit configuration.
\end{Lentry}

\item {Library flags}

Used to specify auxiliary libraries and directories to find them in.

\begin{Lentry}
\item [\texttt{LIBS}]
Additional libraries.  \emph{Warning:} This variable is ignored while
the compilers and linkers are autodetected.  This can lead to strange
errors while configuring.  You can pass the additional libraries in
the variable \texttt{LD} instead.

\item [\texttt{LIBDIRS}] Any other library directories.
\end{Lentry}

\item {Extra include directories}

\begin{Lentry}
\item [\texttt{SYS\_INC\_DIRS}]
Used to specify any additional directories for system include files.
\end{Lentry}


\item {Precision options}

Used to specify the precision of the default real and integer data types,
specified as the number of bytes the data takes up.  Note that not all
values will be valid on all architectures.

\begin{Lentry}

\item [\texttt{REAL\_PRECISION}]
* Allowed values are \texttt{16, 8, 4}.

\item [\texttt{INTEGER\_PRECISION}]
* Allowed values are \texttt{8, 4, 2}.

\end{Lentry}

\item {Executable name}

\begin{Lentry}
\item [\texttt{EXEDIR}] The directory in which to place the executable.
\item [\texttt{EXE}] The name of the executable.
\end{Lentry}

\item{Extra packages}

Compiling with extra packages is described fully in
Section \ref{subsec:cowiexpa},
which should be consulted for the full range of configuration options.

\begin{Lentry}
\item [\texttt{MPI}] * The MPI package to use, if required. Supported values are
        \texttt{CUSTOM}, \texttt{NATIVE}, \texttt{MPICH} or \texttt{LAM}.

\item [\texttt{HDF5}]
Supported values are \texttt{yes, no}. A blank value is taken as \texttt{no}.

\item [\texttt{LAPACK}]
Supported values are \texttt{yes, no}. A blank value is taken as \texttt{no}.

\item [\texttt{PETSC}]
Supported values are \texttt{yes, no}. A blank value is taken as \texttt{no}.

\item [\texttt{PTHREADS}]
Supported values are \texttt{yes}.

\end{Lentry}

\item{Miscellaneous}

\begin{Lentry}
\item [\texttt{PROMPT}] Setting this to \texttt{no} turns off all prompts from the
make system.
\item [\texttt{SILENT}] Setting this to \texttt{no} instructs \texttt{gmake} to print the
commands that it is executing.
\end{Lentry}

\end{itemize}



%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\subsection{Compiling with extra packages}
\label{subsec:cowiexpa}


\subsubsection{MPI: Message Passing Interface}
\label{subsubsec:Compiling-MPI}

The \textit{Message Passing Interface} (MPI) provides inter-processor
communication. It can either be implemented natively on a machine
(this is usual on most supercomputers), or through a standard package
such as MPICH, LAM, WMPI, or PACX.

To compile with MPI, the configure option is

\texttt{MPI = <\var{MPI\_TYPE}>}

where \texttt{<\var{MPI\_TYPE}>} can take the values (entries followed by *
may be specified on the configuration command line):

\begin{Lentry}

\item[\texttt{CUSTOM}] For a custom MPI configuration set the variables
  \begin{Lentry}
  \item [\texttt{MPI\_LIBS}] * libraries.
  \item [\texttt{MPI\_LIB\_DIRS}] * library directories.
  \item [\texttt{MPI\_INC\_DIRS}] * include file directories.
  \end{Lentry}

\item[\texttt{NATIVE}] Use the native MPI for this machine, as indicated in
        the \texttt{known-architectures} directory
        (\texttt{lib/make/known-architectures}).

\item[\texttt{MPICH}]
Use MPICH (\url{http://www-unix.mcs.anl.gov/mpi/mpich}). This is controlled
by the options
  \begin{Lentry}
  \item [\texttt{MPICH\_ARCH}] * machine architecture.
  \item [\texttt{MPICH\_DIR} ] * directory in which MPICH is installed.
        If this option is not defined it will be searched for.
  \item [\texttt{MPICH\_DEVICE}] * the device used by MPICH. If not
        defined, the configuration process will search for this in a
        few defined places.
        Supported devices are currently \texttt{ch\_p4}, \texttt{ch\_shmem},
        \texttt{globus} and \texttt{myrinet}.
        For versions of MPICH prior to 1.2.0 the devices are searched for
         in this order, for 1.2.0 you may need to specify \texttt{MPICH\_DEVICE},
        depending on the installation.
  \end{Lentry}

If \texttt{MPICH\_DEVICE} is chosen to be \texttt{globus}
(\url{http://www.globus.org}), an additional variable must be set
  \begin{Lentry}
  \item[\texttt{GLOBUS\_LOCATION}] * directory in which Globus is installed.
  \end{Lentry}
The Globus flavor may be chosen optionally
  \begin{Lentry}
  \item[\texttt{GLOBUS\_FLAVOR}] * Globus flavor to build Cactus with
  \end{Lentry}
If it is not set, the first Globus flavor found will be used.

If \texttt{MPICH\_DEVICE} is chosen to be \texttt{ch\_gm},
(\url{http://www.myri.com}), an additional variable must be set
  \begin{Lentry}
  \item[\texttt{MYRINET\_DIR}] * directory in which Myrinet libraries are installed.
  \end{Lentry}

\item[\texttt{LAM}]
Use \texttt{LAM} (\textit{Local Area Multicomputer}, \url{http://www.lam-mpi.org/}).
This is controlled by the variables
  \begin{Lentry}
  \item[\texttt{LAM\_DIR} ] * directory in which LAM is installed. This
     will be searched for in a few provided places if not given.
  \end{Lentry}
if the \texttt{LAM} installation splits libraries and include files into different
directories, instead of setting \texttt{LAM\_DIR} set the two variables
  \begin{Lentry}
  \item[\texttt{LAM\_LIB\_DIR}] * directory in which LAM libraries are installed.
  \item[\texttt{LAM\_INC\_DIR}] * directory in which LAM include files are installed.
  \end{Lentry}


\item[\texttt{WMPI}]
Use WMPI (\textit{Win32 Message Passing Interface}, \url{http://dsg.dei.uc.pt/w32mpi/intro.html}). This is controlled by the variable
  \begin{Lentry}
  \item[\texttt{WMPI\_DIR}] * directory in which WMPI is installed.
  \end{Lentry}

\item[\texttt{HPVM}]
Use HPVM (\textit{High Performance Virtual Machine},
(\url{http://www-csag.ucsd.edu/projects/hpvm.html}).
This is controlled by the variable
  \begin{Lentry}
  \item[\texttt{HPVM\_DIR}] * directory in which HPVM is installed.
  \end{Lentry}

\item[\texttt{MPIPro}]
Use MPIPro (\url{http://www.mpi-softtech.com/}).

\item[\texttt{PACX}]
Use the PACX Metacomputing package (\textit{PArallel Computer eXtension},\\
\url{http://www.hlrs.de/structure/organisation/par/projects/pacx-mpi/}). This is controlled by the variables
  \begin{Lentry}
  \item[\texttt{PACX\_DIR}] * directory in which PACX is installed.
        If this option is not defined it will be searched for.
  \item[\texttt{PACX\_MPI}] * the MPI package PACX uses for node-local
        communication. This can be any of the above MPI packages.
  \end{Lentry}

\end{Lentry}

Note that the searches for libraries etc. mentioned above use the
locations given in the files in \texttt{lib/make/extras/MPI}.

\subsubsection{HDF5: Hierarchical Data Format version 5}
\label{subsec:hdf5}

To compile with HDF5 (\url{http://hdf.ncsa.uiuc.edu/whatishdf5.html}),
the configure options are

\texttt{HDF5 = yes/no [HDF5\_DIR = <\var{dir}>] [LIBZ\_DIR = <\var{dir}>] [LIBSZ\_DIR = <\var{dir}>]}

If \texttt{HDF5\_DIR} is not given the configuration process will search for an
installed HDF5 package in some standard places (defined in
\texttt{lib/make/extras/HDF5}).
If the found HDF5 library was built with the external deflate I/O filter,
the configuration process also searches for the \texttt{libz} library and adds
it to the linker flags. You may also point directly to the location of
\texttt{libz.a} by setting \texttt{LIBZ\_DIR}.
If the found HDF5 library was built with the external \texttt{szlib} I/O filter,
the configuration process also searches for the \texttt{szlib} library and adds
it to the linker flags. You may also point directly to the location of
\texttt{libsz.a} by setting \texttt{LIBSZ\_DIR}.


\subsubsection{LAPACK: Linear Algebra PACKage}

To compile with LAPACK (\url{http://www.netlib.org/lapack/}),
the configure options are

\begin{alltt}
LAPACK = yes | no | <blank>
[ LAPACK\_DIR = <\var{dir}> | none ]
[ LAPACK\_EXTRA\_LIBS\_DIRS = <\var{dir}> ]
[ LAPACK\_LIBS = <\var{libs}> ]
[ LAPACK\_EXTRA\_LIBS = <\var{libs}> ]
\end{alltt}

If \texttt{LAPACK\_DIR} is not given the configuration process will search for a
LAPACK library \texttt{liblapack.[\{a,so\}]} in some standard places (defined in
\texttt{lib/make/extras/LAPACK}). If \texttt{LAPACK\_DIR} is set to \texttt{no}
the LAPACK library path is assumed to be installed in a standard system location
(e.g. \texttt{/usr/lib/}) and thus the library path will not be added to the
linker's command line.

Because LAPACK doesn't come as a standardized system installation, there are
additional configuration variables to set the name of the lapack library
(\texttt{LAPACK\_LIBS}) as well as the name (\texttt{LAPACK\_EXTRA\_LIBS}) and
location (\texttt{LAPACK\_EXTRA\_LIBS\_DIRS}) of extra libraries that are
required by LAPACK itself.


\subsubsection{PETSc: Portable, Extensible Toolkit for Scientific Computation}

To compile with PETSc
(\url{http://www-unix.mcs.anl.gov/petsc/petsc-2/index.html}),
the configure options are

\begin{alltt}
PETSC = yes | no | <blank>
[ PETSC\_DIR = <\var{dir}> ]
[ PETSC\_ARCH = <\var{architecture}> ]
[ PETSC\_ARCH\_LIBS = <\var{architecture-specific libraries}> ]
\end{alltt}

If \texttt{PETSC\_DIR} is not given the configuration process will search for an
installed PETSc package in some standard places (defined in
\texttt{lib/make/extras/PETSC}).
If \texttt{PETSC\_ARCH} is not given the configuration process will choose the
first PETSc configuration found in \texttt{\$PETSC\_DIR/lib/libO/}.
If \texttt{PETSC\_ARCH\_LIBS} is not given the configuration process will choose
architecture-specific libraries as required by a PETSc configuration (usually
PETSc needs the LAPACK library).


\subsubsection{Pthreads: POSIX threads}

To enable multithreading support within Cactus using POSIX threads
the configure option is

\texttt{PTHREADS = yes}

The configuration process will check if a reentrant C library is available
and adds it to the linker flags. It will also search for the system's Pthreads
library (either \texttt{libpthread} or \texttt{libpthreads}) and set
preprocessor defines necessary for compiling multithreaded code.


\subsection{File layout}

The configuration process sets up various subdirectories and files in the
\texttt{configs} directory to contain the configuration specific files, these
are placed in a directory with the name of the configuration.

\begin{Lentry}

\item [\texttt{config-data}] contains the files created by the configure
script:

The most important ones are

\begin{Lentry}

\item [\texttt{make.config.defn}]
contains compilers and compilation flags for a configuration.

\item [\texttt{make.extra.defn}]
contains details about extra packages used in the configuration.

\item [\texttt{cctk\_Config.h}]
The main configuration header file, containing architecture specific
definitions.

\item [\texttt{cctk\_Archdefs.h}]
An architecture specific header file containing things which cannot be
automatically detected, and have thus been hand-coded for this architecture.
\end{Lentry}

These are the first files which should be checked or modified to suit any
peculiarities of this configuration.

In addition the following files may be informative:

\begin{Lentry}
\item [\texttt{fortran\_name.pl}]
A Perl script used to determine how the Fortran compiler names subroutines.
This is used to make some C routines callable from Fortran, and Fortran
routines callable from C.

\item [\texttt{make.config.deps}]
Initially empty.  Can be edited to add extra architecture specific dependencies
needed to generate the executable.

\item [\texttt{make.config.rule}]
The \texttt{make} rules for generating object files from source files.

\end{Lentry}

Finally, \texttt{autoconf} generates the following files.

\begin{Lentry}

\item [\texttt{config.log}]
A log of the \texttt{autoconf} process.

\item [\texttt{config.status}]
A script which may be used to regenerate the configuration.

\item [\texttt{config.cache}]
An internal file used by \texttt{autoconf}.

\end{Lentry}

\item [\texttt{lib}]
An empty directory which will contain the libraries created for each thorn.

\item [\texttt{build}]
An empty directory which will contain the object files generated for this
configuration, and preprocessed source files.

\item [\texttt{config-info}]
A file containing information about the configuration (including the options used to configure the configuration).

\item [\texttt{bindings}] A directory which contains all the files
generated by the CST from the \texttt{.ccl} files.

\item [\texttt{scratch}]
A scratch directory which is used to accommodate Fortran 90 modules.

\end{Lentry}


\section{Building and Administering a configuration}
\label{sec:buanadaco}

Once you have created a new configuration, the command
\\ \\
\texttt{gmake <\var{configuration name}>}
\\ \\
will build an executable, prompting you along the way for the
thorns which should be included. There is a range of \texttt{gmake}
targets and options which are detailed in the following sections.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{gmake targets for building and administering configurations}
\label{sec:gmtafobuanadco}

A target for \texttt{gmake} can be naively thought of as an argument
that tells it which of several things listed in the \texttt{Makefile} it
is to do. The command \texttt{gmake help} lists all \texttt{gmake} targets:
% colon clarifies that all (config) targets are listed here

\begin{Lentry}

\item [\texttt{gmake <\var{config}>}]
        builds a configuration. If the configuration doesn't exist
        it will create it.

\item [\texttt{gmake <\var{config}>-clean}] removes all object and dependency
files from a configuration.

\item [\texttt{gmake <\var{config}>-cleandeps}] removes all dependency files
from a configuration.

\item [\texttt{gmake <\var{config}>-cleanobjs}] removes all object files from
  a configuration.

\item [\texttt{gmake <\var{config}>-config}] creates a new configuration or
reconfigures an existing one overwriting any previous configuration options.\\
The configuration options are stored in a file
\texttt{configs/<\var{config}>/config-info}.

\item [\texttt{gmake <\var{config}>-configinfo}] displays the options
of the configuration (\texttt{cat configs/<\var{config}>/config-info}).

\item[\texttt{gmake <\var{config}>-cvsupdate}] updates the Flesh and this
configuration's thorns from the CVS repositories.

\item [\texttt{gmake <\var{config}>-delete}] deletes a configuration
(\texttt{rm -r configs/<\var{config}>}).

\item [\texttt{gmake <config>-editthorns}] edits the ThornList.

\item [\texttt{gmake <\var{config}>-examples}] copies all the example parameter files relevant for this configuration to the directory \texttt{examples} in the Cactus home directory. If a file of the same name is already there, it will not overwrite it.

\item [\texttt{gmake <\var{config}>-realclean}] removes from a configuration
all object and dependency files, as well as files generated from the
CST (stands for \textit{Cactus Specification Tool}, which is the set of Perl scripts
which parse the thorn configuration files).  Only the files generated
by configure and the \texttt{ThornList} file remain.

\item [\texttt{gmake <\var{config}>-rebuild}] rebuilds a configuration (reruns the CST).

\item [\texttt{gmake <\var{config}>-reconfig}] reconfigures an existing
configuration using its previous configuration options from the file
\texttt{configs/<\var{config}>/config-info}.

\item [\texttt{gmake <\var{config}>--testsuite}] runs the test programs
associated with each thorn in the configuration. See section
\ref{sec:testing} for information about the test suite mechanism.

\item[\texttt{gmake <\var{config}>-ThornGuide}] builds documentation for the
thorns in this configuration
  (see section \ref{sec:OtherGmakeTargetsDoc} on page
  \pageref{sec:OtherGmakeTargetsDoc} for other targets to build documentation
  for thorns).

\item [\texttt{gmake <\var{config}>-thornlist}] regenerates the
\texttt{ThornList} for a configuration.

\item [\texttt{gmake <\var{config}>-utils [UTILS$=$<\var{list}>]}] builds all
utility programs provided by the thorns of a configuration. Individual
utilities can be selected by giving their names in the \texttt{UTILS} variable.

\end{Lentry}



\subsection{Compiling in thorns}
\label{sec:cointh}

Cactus will try to compile all thorns listed in
\texttt{configs/<\var{config}>/ThornList}.
The \texttt{ThornList} file is simply a list of the form
\texttt{<\var{arrangement}>/<\var{thorn}>}.  All text after a pound sign
`\texttt{\#}' or exclamation mark `\texttt{!}'
on a line is treated as a comment and ignored.
The first time that you compile a configuration, if you did
not specify a ThornList already during configuration,
you will be shown a list of all the thorns in your arrangement
directory, and asked if you with to edit them. You can regenerate
this list at anytime by typing

\begin{alltt}
gmake <\var{config}>-thornlist
\end{alltt}

or you can edit it using

\begin{alltt}
gmake <\var{config}>-editthorns
\end{alltt}

Instead of using the editor to specify the thorns you want to
  have compiled, you can \emph{edit} the \texttt{ThornList} outside
  the make process. It is located in \texttt{configs/<\var{config}>/ThornList},
  where \texttt{<\var{config}>} refers to the name of your configuration.
  The directory, \texttt{./configs}, exists \emph{ after} the very first
  make phase for the first configuration.

\subsection{Notes and Caveats}
\begin{itemize}
\item{} If during the build you see the error ``\texttt{missing
    separator}'' you are probably not using GNU make.
\item{} \textit{The EDITOR environment variable}. You may not be aware of
  this, but this thing very often exists and may be set  by default to
  something scary like \texttt{vi}. If you don't know how to use \texttt{vi}
  or wish to
  use your favorite editor instead, reset this environment variable.
  (To exit \texttt{vi} type \texttt{<ESC> :q!})
\end{itemize}

\subsection{\texttt{gmake} options for building configurations}
\label{sec:gmopfobuco}

An \textit{option} for \texttt{gmake} can be thought of as an argument which tells
it how it should make a \textit{target}. Note that the final result is always
the same.

\begin{Lentry}
\item [\texttt{gmake <\var{target}> PROMPT=no}] turns off all prompts from the
make system.
\item [\texttt{gmake <\var{target}> SILENT=no}] print the commands that gmake
is executing.
\item [\texttt{gmake <\var{target}> WARN=yes}] show compiler warnings during
compilation.
\item [\texttt{gmake <\var{target}> FJOBS=<\var{number}>}] compile in parallel,
across files within each thorn.
\item [\texttt{gmake <\var{target}> TJOBS=<\var{number}>}] compile in parallel,
across thorns.

\end{Lentry}

Note that with more modern versions of gmake, it is sufficient to pass the
normal \texttt{-j <\var{number}>} flag to gmake to get parallel compilation.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%




%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Other gmake targets}

\begin{Lentry}

\item [\texttt{gmake help}] lists all make options.

\item [\texttt{gmake checkout}] allows you to easily checkout Cactus
arrangements and thorns.  For example it can checkout all the thorns
in any thornlist file found in the \texttt{thornlists} subdirectory of
the Cactus root directory. % (usually \texttt{Cactus}).

\item [\texttt{gmake cvsdiff}] shows differences between checked out version of Cactus and that in the CVS repositories.

\item [\texttt{gmake cvsstatus}] shows status of checked out version of Cactus, reporting which files have been modified or need updating.

\item [\texttt{gmake cvsupdate}] updates Flesh and all thorns from CVS repositories.

\item [\texttt{gmake configinfo}] prints configuration options for every
configuration found in user's \texttt{configs} subdirectory.

\item [\texttt{gmake default}] creates a new configuration with a default name.

\item [\texttt{gmake distclean}] deletes your \texttt{configs} directory and hence all your configurations.

\item [\texttt{gmake downsize}] removes non-essential files as documents
  and test suites to allow for minimal installation size.

\item [\texttt{gmake newthorn}] creates a new thorn, prompting for the necessary
  information and creating template files.

\item [\texttt{gmake TAGS}] creates an Emacs style TAGS file. See section
  \ref{sec:Appendix.tags} for using tags within Cactus.

\item [\texttt{gmake tags}] creates a \texttt{vi} style tags file. See section
  \ref{sec:Appendix.tags} for using tags within Cactus.

\end{Lentry}

{\bf Targets to generate Cactus documentation:}
\label{sec:OtherGmakeTargetsDoc}

\begin{Lentry}

\item[\texttt{gmake <\var{arrangement}>-ArrangementDoc}] builds the
documentation for the arrangement.

\item[\texttt{gmake ArrangementDoc}] builds the documentation for all
arrangements.

\item [\texttt{gmake MaintGuide}] runs LaTeX to produce a copy of the
Maintainers' Guide.

\item [\texttt{gmake ReferenceManual}] runs LaTeX to produce a copy of the
Reference Manual.

\item[\texttt{gmake <\var{thorn}>-ThornDoc}] builds the documentation for the
thorn.

\item[\texttt{gmake ThornDoc}] builds the documentation for all thorns.

\item [\texttt{gmake ThornGuide}] runs LaTeX to produce a copy of the Thorn
Guide, for all the thorns in the arrangements directory.

\item [\texttt{gmake UsersGuide}] runs LaTeX to produce a copy of the Users'
Guide.

\item [\texttt{gmake AllDoc}] creates all of the above documentations.

\end{Lentry}


\section{Testing}
\label{sec:testing}

Some thorns come with a test suite, consisting of example parameter files
and the output files generated by running these. To run the test suite
for the thorns you have compiled use

\texttt{gmake <\var{configuration}>--testsuite}

These test suite serve the dual purpose of

\begin{Lentry}
\item [Regression testing]
i.e. making sure that changes to the thorn or the Flesh don't affect the
output from a known parameter file.
\item [Portability testing]
i.e. checking that the results are independent of the architecture --- this
is also of use when trying to get Cactus to work on a new architecture.
\end{Lentry}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\chapter{Running Cactus}

Cactus executables always run from a parameter file (which may be a
specified as a command line argument taken from standard input), which
specifies which
thorns to use and sets the values of any parameters which are different
from the default values. There is no restriction on the name
of the parameter file, although it is conventional to use the file
extension \texttt{.par}.  Optional command line arguments can be used
to customise runtime behaviour, and to provide information about the
thorns used in the executable. The general syntax for running Cactus from
a parameter file is then

\texttt{./cactus\_<\var{config}> <\var{parameter file}>
[\var{command line options}]}

or if the parameter file should be taken from standard input by appending
a dash (``\texttt{-}'') on the command line like this:

\texttt{./cactus\_<\var{config}> [\var{command line options}] -}

The remainder of this chapter covers all aspects for running your
Cactus executable.  These include: command line options, parameter
file syntax, understanding screen output, environment variables, and
creating thorn documentation.

\section{Command Line Options}
\label{sec:command_line_options}

Cactus uses the standard GNU style of long-named command-line options;
many of these options also have traditional Unix single-letter short forms.
The options follow the usual GNU rules:
\begin{itemize}
\item	A long-named option \verb|--foo| which takes an argument \verb|bar|
	may be written as either \verb|--foo bar| or as \verb|--foo=bar|.
\item	A long-named option may be abbreviated so long as the abbreviation
	is unambiguous.
\item	The preferred way of spelling a long-named option is \verb|--foo|,
	but \verb|-foo| also accepted, though this is deprecated
	(we plan to remove it after the next beta release).
\item	A short option \verb|-X| which takes an argument \verb|bar|
	may be written as either \verb|-Xbar| or as \verb|-X=bar|.
\item	An option which can be interpreted as either a short option,
	or as an abbreviated \verb|-foo|-style long option, is interpreted
	as the former.  In particularl, \verb|-re| is interpreted as
	an abbreviation for \verb|-redirect|, rather than as \verb|-r=e|.
\end{itemize}

The Cactus command line options are specified in
table~\ref{tab:command-line-options}, and are as follows:

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{table}
\texttt{
\begin{tabular}{|l|l|}
\hline
Short Version & Long Version \\
\hline
 -O[v] & --describe-all-parameters \\
\hline
 -o<\var{param}> & --describe-parameter=<\var{param}> \\
\hline
 -S & --print-schedule\\
\hline
 -T & --list-thorns\\
\hline
 -t<\var{arrangement/thorn}>& --test-thorn-compiled=<\var{arrangement/thorn}>\\
\hline
 -h,-? & --help\\
\hline
 -v & --version \\
\hline
% -x [<nprocs>] & --test-parameters [<nprocs>] \\
%\hline
 -L<\var{level}> & --logging-level=<\var{level}> \\
\hline
 -W<\var{level}> & --warning-level=<\var{level}> \\
\hline
 -E<\var{level}> & --error-level=<\var{level}> \\
\hline
 -r[o|e|oe|eo] & --redirect=[o|e|oe|eo]\\
\hline
 & --logdir=<\var{directory}> \\
\hline
 -b[no|line|full] & --buffering=[no|line|full]\\
\hline
    & --parameter-level=<strict|normal|relaxed> \\
\hline
 -i & --ignore-next \\
\hline
\end{tabular}
}
\caption[Command-Line Options]
	{This table shows all the Cactus command-line options.}
\label{tab:command-line-options}
\end{table}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


\begin{Lentry}

\item [\texttt{-O} or \texttt{--describe-all-parameters}]
Prints a full list of all parameters from all thorns which were compiled,
along with descriptions and allowed values.  This can take an optional extra
parameter \texttt{v}  (i.e. \texttt{-Ov} to give verbose information about
all parameters).

\item [\texttt{-o<\var{param}>} or \texttt{--describe-parameter=<\var{param}>}]
Prints the description and allowed values for a given parameter --- takes one
argument.

\item [\texttt{-S} or \texttt{--print-schedule}]
Print only the schedule tree.

\item [\texttt{-T} or \texttt{--list-thorns}]
Prints a list of all the thorns which were compiled in.

\item [\texttt{-t<\var{arrangement or thorn}>} or \texttt{--test-thorn-compiled=<\var{arrangement or thorn>}} ]
Checks if a given thorn was compiled in --- takes one argument.

\item [\texttt{-h}, \texttt{-?} or \texttt{--help}]
Prints a help message.

\item [\texttt{-v} or \texttt{--version}]
Prints version information of the code.

%\item [\texttt{-x <nprocs>} or \texttt{--test-parameters <nprocs>}]
%Runs the code far enough to check the consistency of the parameters.  If
%given a numeric argument it will attempt to simulate being on that number
%of processors. [To be implemented.]

\item [\texttt{-L<\var{level}>} or \texttt{--logging-level=<\var{level}>}]
Sets the logging level of the code.  All warning messages are given a
level --- the lower the level the greater the severity.  This
parameter \texttt{-L} controls the level of messages to be seen, with all
warnings of level $\le$ \texttt{<\var{level}>} printed to standard output.  The
default is a logging level of~0, meaning that only level~0 messages
should be printed to standard output.

\item [\texttt{-W<\var{level}>} or \texttt{--warning-level=<\var{level}>}]
Similar to \texttt{-W}, but for standard error instead of
standard output.  All warnings of level $\le$ \texttt{<\var{level}>} are
printed to standard error.  The default is a warning level of~1,
meaning that level~0 and level~1 messages should be printed to
standard error.

\item [\texttt{-E<\var{level}>} or \texttt{--error-level=<\var{level}>}]
Similar to \texttt{-W}, but for fatal errors: Cactus treats all
warnings with level $\le$ \texttt{<\var{level}>} as fatal errors, and aborts
the Cactus run immediately (after printing the warning message%%%
\footnote{%%%
         Cactus imposes the constraints that
         $\hbox{the \texttt{-W} level} \ge \hbox{the \texttt{-E} level} \ge 0$,
         so any fatal-error message will always be printed (first).
         }%%%
).  The default value is zero, \ie{} only level~0 warnings
will abort the Cactus run.

\item [\texttt{-r[o|e|oe|eo]} or \texttt{--redirect=[o|e|oe|eo]}]
Redirects the standard output (`\texttt{o}') and/or standard error
(`\texttt{e}') of each processor to a file.  By default
the standard outputs from processors other than processor 0 are discarded.

\item [\texttt{--logdir=<\var{directory}>}]
Sets the output directory for logfiles created by the \texttt{-r} option.
If the directory doesn't exist yet, it will be created by Cactus.

\item [\texttt{-b[no|line|full]} or \texttt{--buffering=[no|line|full]}]
  Set the \texttt{stdout} buffering mode.  Buffered I/O is a
  standard feature of C programmes.  This delays writing the actual
  output; instead, the output is collected into an internal buffer,
  and is then written in large chunks.  This improves performance
  considerably.  Line buffering means that output is written whenever
  a newline character is encountered; full buffering means that output
  is written once e.g.\ 1000 characters have accmulated.  The default
  setting is line buffering for I/O that goes to a terminal, and full
  buffering for I/O that goes to a file.  For debugging purposes, it
  is sometimes useful to reduce the amount of buffering.  Error
  messages, i.e.\ the \texttt{stderr} stream, is always unbuffered
  (and hence usually slower than \texttt{stdout}).

\item [\texttt{--parameter-level=<strict|normal|relaxed>}]
Sets the level of parameter checking to be used, one of \texttt{strict}
(the default), \texttt{normal}, or \texttt{relaxed}.
See Section~\ref{sec:Parameter_File} for details.

\item [\texttt{-i} or \texttt{--ignore-next}]
Causes the next argument on the command line to be ignored.

\end{Lentry}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Parameter File Syntax}
\label{sec:Parameter_File}

A \textit{parameter file} ( or \textit{par file}) is used to control the
behaviour of a Cactus executable.  It specifies initial values for parameters
as defined in the various thorns' \texttt{param.ccl} files
(see Chapter~\ref{chap:Cactus_parameters}).
The name of a parameter file is often given the suffix \texttt{.par}, but
this is not mandatory.

A parameter file is a text file whose lines are either comments
or parameter statements.
Comments are blank lines or lines that begin with either
`\texttt{\#}' or `\texttt{!}'.
A parameter statement
consists of one or more parameter names, followed by
an `\texttt{=}', followed by the value(s) for this (these) parameter(s).
Note that all string parameters are case insensitive.

The first parameter statement in any parameter file should set \texttt{ActiveThorns},
which is a special parameter that tells the
program which \textit{thorns} are to be activated.  Only parameters from active
thorns can be set (and only those routines \textit{scheduled} by active thorns
are run).  By default all thorns are inactive. For example, the first
entry in a parameter file which is using just the two thorns
\texttt{CactusPUGH/PUGH} and \texttt{CactusBase/CartGrid3D} should be

\texttt{ActiveThorns = "PUGH CartGrid3D"}

Parameters following the \texttt{ActiveThorns} parameter all have names
whose syntax depends on the scope
(see Section~\ref{sec:Cactus_parameters.scope})
of the parameter:
\begin{Lentry}
\item [\texttt{Global parameters}]
Just the name of the parameter itself. Global parameters are to be avoided;
there are none in the Flesh and Cactus Toolkits.
\item [\texttt{Restricted parameters}]
The name of the \textit{implementation} which defined the parameter, followed
by two colons,
then the name of the parameter --- e.g. \texttt{driver::global\_nx}.
\item [\texttt{Private parameters}]
The name of the \textit{thorn} which defined the parameter, two colons,
and the name of the parameter --- e.g. \texttt{wavetoyF77::amplitude}.
\end{Lentry}

This notation is not currently strictly enforced in the code. It is
sufficient to specify the first part of the parameter name using either
the implementation name, or the thorn name. However, we recommend
that the above convention be followed.

The Cactus Flesh performs checks for consistency and range of parameters.
The severity of these checks is controlled by the command line argument
\texttt{--parameter-level} which can take the following values
\begin{Lentry}
\item[\texttt{relaxed}] Cactus will issue a level 0 warning (that is the
default behaviour will be to terminate) if
\begin{itemize}
\item{} The specified parameter value is outside of the allowed range.
\end{itemize}

\item [\texttt{normal}]
This is the default, and provides the same warnings as the
\texttt{relaxed} level, with in addition a level 0 warning issued for
\begin{itemize}
\item{} An implementation and/or thorn \texttt{foo} is active, but the
        parameter \texttt{foo::bar} was not defined.
\item{} The parameter \texttt{foo::bar} was successfully set for both an
        active implementation \texttt{foo} not implemented by a thorn \texttt{foo},
        and to a thorn \texttt{foo}.
\end{itemize}

\item [\texttt{strict}]
This provides the same warnings as the \texttt{normal} level, with in
addition a level 0 warning issued for
\begin{itemize}
\item{} The parameter \texttt{foo::bar} is specified in the parameter file,
        but no implementation or thorn with the name \texttt{bar} is active.
\end{itemize}
\end{Lentry}

Notes:

\begin{itemize}

\item{} You can obtain lists of the parameters associated with
each thorn using the command line options \texttt{-o} and \texttt{-O}
(Section~\ref{sec:command_line_options}).

\item{} The parameter file is read \emph{sequentially} from top to bottom,
        this means that if you set the value of a parameter twice in
        the parameter file, the second value will be used. (This is
        why the \texttt{ActiveThorns} parameter is always first in the file).

\item{} String parameter values can be specified either as unquoted tokens (not
        containing any whitespace) or as quoted values. If a quoted string
        parameter value spans multiple lines all whitespaces, including newline
        characters, are preserved.

\item{} Some parameters are \textit{steerable} and can be changed during
        the execution of a Cactus program using parameter steering interfaces
        For example, thorn \texttt{CactusConnect/HTTPD}, or using a
        parameter file when recovering from a checkpoint file.

\item{} For examples of parameter files, look in the \texttt{par} directory
        which can be found in most thorns.

\end{itemize}

\section{Thorn Documentation}
\label{sec:thdo}

The Cactus make system provides a mechanism for generating a
\textit{Thorn Guide} containing separate chapters for each thorn and
arrangement in your configuration. The documentation provided for an
individual thorn obviously depends on what the thorn authors added,
but the Thorn Guide is a good place to first look for special
instructions on how to run and interpret the output from a Thorn.
Details about parameters, grid variables and scheduling are
automatically included in from a thorns CCL files into the Thorn
Guide. To construct a Thorn Guide for the configuration
\texttt{$<$\var{config}$>$} use

\texttt{gmake $<$\var{config}$>$-ThornGuide}

or to make a Thorn Guide for all the thorns in the \texttt{arrangements} directory

\texttt{gmake $<$\var{config}$>$}.

See Section~\ref{sec:Adding_documentation} for a guide to adding
documentation to your own thorns.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


\chapter{Getting and looking at output}


\section{Screen output}

As your Cactus executable runs, standard output and standard error
are usually written to the screen. Standard output provides you
with information about the run, and standard error reports warnings
and errors from the Flesh and thorns.

As the program runs, the normal output provides the following information:

\begin{Lentry}

\item [Active thorns]
        A report is made as each of the thorns in the \texttt{ActiveThorns}
parameters from the parameter file (see Section~\ref{sec:Parameter_File})
is attempted to be activated. This report
shows whether the thorn activation was successful, and if successful gives the
thorn's implementation. For example

\begin{verbatim}
Activating thorn idscalarwave...Success -> active implementation idscalarwave
\end{verbatim}

\item [Failed parameters]
         If any of the parameters in the parameter file does not belong to any
of the active thorns, or if the parameter value is not in the allowed range
(see Section~\ref{sec:Parameters.Types_and_Ranges}),
an error is registered. For example, if the parameter is not recognised

\begin{verbatim}
Unknown parameter time::ddtfac
\end{verbatim}
or if the parameter value is not in the allowed range

\begin{verbatim}
Unable to set keyword CartGrid3D::type - ByMouth not in any active range
\end{verbatim}

\item [Scheduling information]
        The scheduled routines (see Section~\ref{chap:scheduling}),
are listed, in the order that they will be executed. For example

\begin{verbatim}
----------------------------------------------------------------------
  Startup routines
    Cactus: Register banner for Cactus
    CartGrid3D: Register GH Extension for GridSymmetry
    CartGrid3D: Register coordinates for the Cartesian grid
    IOASCII: Startup routine
    IOBasic: Startup routine
    IOUtil: IOUtil startup routine
    PUGH: Startup routine
    WaveToyC: Register banner

  Parameter checking routines
    CartGrid3D: Check coordinates for CartGrid3D
    IDScalarWave: Check parameters

  Initialisation
    CartGrid3D: Set up spatial 3D Cartesian coordinates on the GH
    PUGH: Report on PUGH set up
    Time: Set timestep based on speed one Courant condition
    WaveToyC: Schedule symmetries
    IDScalarWave: Initial data for 3D wave equation

  do loop over timesteps
    WaveToyC: Evolution of 3D wave equation
    t = t+dt
    if (analysis)
    endif
  enddo
----------------------------------------------------------------------
\end{verbatim}

\item [Thorn banners]
        Usually a thorn registers a short piece of text as a \emph{banner}.
        This banner of each thorn is displayed in the standard output when
        the thorn is initialised.

\end{Lentry}


\section{Output}
Output methods in Cactus are all provided by thorns.
Any number of output methods can be used for each run.
The behaviour of the output thorns in the
standard arrangements are described in those thorns' documentation.

In general, output thorns decide what to output by parsing a string parameter
containing the names of those grid variables, or groups of variables, for which
output is required. The names should be fully qualified with the
implementation and group or variable names.

There is usually a parameter for each method to denote how often, in evolution
iterations, this output should be performed.  There is also usually a parameter
to define the directory in which the output should be placed, defaulting to the
directory from which the executable is run.

See Chapter~\ref{chap:io_methods} for details on creating your own IO method.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Checkpointing/Recovery}
\label{sec:checkpointing}

Checkpointing is defined as saving the current state of a run (parameter
settings, contents of grid variables, and other relevant information) to a file.
At a later time, this run can then be restarted from that state by recovering
all the data from the checkpoint file.

Cactus checkpointing and recovery methods are provided by thorns.
In general, these thorns decide how often to generate a checkpoint.
They also register their recovery routines with the Flesh; these recovery
routines may then be called during initialisation of a subsequent run to
perform the recovery of the state of the run.
Such a recovery is requested by setting a parameter in the parameter file.

See Chapter~\ref{chap:cp_recovery_methods} for details of how to create
your own checkpointing and recovery methods.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\end{cactuspart}