% /*@@
%   @file      RunningCactus.tex
%   @date      27 Jan 1999
%   @author    Tom Goodale, Gabrielle Allen, Gerd Lanferman, Thomas Radke
%   @desc
%              How to run Cactus part of the Cactus User's Guide
%   @enddesc
%   @version   $Header$
% @@*/
\begin{cactuspart}{1}{Installation and Running}{$RCSfile$}{$Revision$}
\renewcommand{\thepage}{\Alph{part}\arabic{page}}



%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\chapter{Installation}
\label{cha:in}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Required software}
\label{sec:reqo}

In general, Cactus {\em requires} the following set of software to function
in single processor mode. Please refer to the architecture section
\ref{sec:suar} for architecture specific items.
\begin{Lentry}
\item[{\tt Perl5.0}] Perl is used extensively during the Cactus
  thorn configuration phase. Perl is available for nearly all
  operating systems known to man and can be obtained at
  {\tt http://www.perl.org}
\item[{\tt GNU make}] The make
  process works with the GNU make utility (referred to as {\bf gmake}
  henceforth). While other make utilities may also work, this is not
  guaranteed. Gmake can be obtained from your favorite GNU site or
  from {\tt www.gnu.org}
\item[{\tt C}] C compiler. For example, the GNU compiler. This
 is available for most supported platforms.  Platform specific compilers
 should also work.
\item[{\tt CPP}] C Pre-processor. For example, the GNU CPP.  These are
normally provided on most platforms, and many C compilers have an option
to just run as a preprocessor.
\item[{\tt CVS}] The {\em ``Concurrent Versioning System''} is not needed
  to run/compile Cactus, but you are strongly encourage to install
  this software to take advantage of the update procedures. It can be
  downloaded from your favorite GNU site.  Tar files of each release are
  also available.
\end{Lentry}

\noindent
To use Cactus, with the default driver\footnote{For help with unfamiliar terms, please consult the glossary, Appendix \ref{sec:glossary}.} ({\tt CactusPUGH/PUGH}) on multiple
processors you also need:
\begin{Lentry}
\item[{\tt MPI}] the {\it Message Passing Interface (MPI)}
which provides inter-processor communication.
Supercomputing sites often supply a native {\tt MPI} implementation with
which Cactus is very likely to be compatible. Otherwise there are
various freely available ones available, e.g. the {\tt MPICH}
version of {\tt MPI} is available for various architectures and operating
systems at {\tt http://www-unix.mcs.anl.gov/mpi/}.
\end{Lentry}

\noindent
If you are using any thorns containing routines
written in {\tt C++} you also need
\begin{Lentry}
\item[{\tt C++}] C++ compiler. For example, the GNU compiler. This
 is available for most supported platforms.  Platform specific compilers
 should also work.  Note that if a C++ compiler is available then the
 {\em main} routine in the flesh is compiled with C++ to allow static
 class initialisations.
\end{Lentry}

\noindent
If you are using any thorns containing routines
written in {\tt FORTRAN} you also need
\begin{Lentry}
\item[{\tt F90/F77}] For routines written in F77, either an F90 or an F77
 compiler can be used. For routines written in F90 a F90 compiler is
 obviously
 required. There is a very limited set of free F90 compilers available
 for the different architectures.
\end{Lentry}

\noindent
While not required for compiling or running Cactus, for thorn development
it is useful to install
\begin{Lentry}
\item[{\tt ctags/etags}] The program Tags enables you browse through the calling structure
  of a program by help of a function call database. Navigating the flesh and
  arrangements becomes very easy. Emacs and vi both support this method. See
  \ref{sec:usta} for a short guide to ``tags''.
\end{Lentry}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Supported architectures}
\label{sec:suar}

Cactus runs on many machines, under a large number of operating
systems.  Here we list the machines we have compiled and verified
Cactus on, including some architecture specific notes.  A complete
list of architectures supported by Cactus, along with more notes, can
be found at
\begin{center}
{\tt http://www.cactuscode.org/Documentation/Architectures.html}.
\end{center}

\begin{Lentry}
\item[{\bf SGI}] 32 or 64 bit running Irix.
\item[{\bf Cray T3E}]
\item[{\bf Compaq Alpha}]  Compaq operating system and Linux.  Single processor
  mode and {\tt MPI} supported. The Alphas need to have the GNU {\tt C/C++}
  compilers installed.
\item[\textbf{IA32}] running Linux or Windows 2000/NT.  Single
processor mode and MPI ({\tt MPICH} and {\tt LAM}) supported.\\On
Windows Cactus compiles with Cygwin.  MPI
({\tt WMPI}, {\tt HPVM}, and {\tt MPIPro}) supported.  Please read
doc/README.NT for details.
\item[\textbf{IA64}]  running Linux.
\item[{\bf Macintosh PowerPC}] (MacOS X and Linux PPC)
\item[{\bf IBM SP2}] 32 or 64 bit running AIX.
\item[{\bf Hitachi SR8000-F1}]
\item[{\bf Sun} Solaris]
\item[{\bf Fujitsu}]
\end{Lentry}

%\begin{Lentry}
%\item[{\bf SGI Origin 2000} running Irix]
%\item[{\bf SGI} 32 or 64 bit running Irix]
%\item[{\bf Cray T3E}]
%\item[{\bf Dec Alpha}]  Dec operating system and Linux. Single processor
%  mode and {\tt MPI} supported. The Decs need to have the GNU {\tt C/C++}
%  compilers installed
%\item[{\bf Linux (ia32, ia64, ppc, alpha)}] There is a
%  free Linux F90 compiler available from  {\tt http://www.psrv.com}
%  -- the only free we know of; please note the comment about installing this in
%the FAQ.
%  Single processor mode and MPI ({\tt MPICH} and {\tt LAM}) supported.
%\item[{\bf Windows NT}] Compiles with Cygwin. Single processor mode and MPI ({\t
%t WMPI},
%{\tt HPVM}, and {\tt MPIPro}) supported.  Please read doc/README.NT for details.
%\item[{\bf Macintosh PowerPC (MacOS X)}]
%\item[{\bf IBM SP2}]
%\item[{\bf Hitachi SR8000-F1}]
%\item[{\bf Sun Solaris}]
%\end{Lentry}

The following machines are only partially supported
\begin{Lentry}
\item[{\bf HP Exemplar}]
\item[{\bf NEC SX-5}]
\end{Lentry}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Checkout procedure}
\label{sec:chpr}

Cactus is distributed, extended, and maintained using the free CVS
software ({\it Concurrent Versioning System}: {\tt http://www.cvshome.org}).
CVS  allows many people to work on a large software project
together without getting into a tangle.
Since Cactus thorns are distributed from several repositories on the
main CVS site, and from a growing number of user sites, we provide a
script, described below,  on our website for checking out the flesh and thorns.
The Cactus website also provides a form interface for direct download.

CVS experts who want to use raw CVS commands are directed to Appendix~\ref{sec:uscv}
for full instructions. For CVS novices, we also summarize in this
appendix basic CVS commands.

The space required for an installation depends on the arrangements and
thorns used. The flesh on its own requires less than 5 MB.

The script for checking out the flesh and thorns, {\tt GetCactus}, is available
from the website at

{\tt http://www.cactuscode.org/Download/GetCactus}

The
script takes as an argument the name of a file containing a {\it ThornList},
that is a list of thorns with the syntax
{\tt
\begin{verbatim}
<arrangement name>/<thorn name>
\end{verbatim}
}
If no filename is given, only the flesh is checked out.
Optional directives in the ThornList indicate which CVS repository to fetch
thorns from. The default is to take the thorns from the same repository as
the flesh. A full description of ThornList syntax is provided in Appendix~\ref{chap:th}.
ThornLists for example applications are provided on the Cactus website.

The same script can be used to checkout additional thorns.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Directory structure}
\label{sec:dist}

A fresh checkout creates a directory {\tt Cactus} with the
following subdirectories:

\begin{Lentry}

\item[{\tt CVS}] the CVS book-keeping directory, present in every subdirectory

\item[{\tt doc}] Cactus documentation

\item[{\tt lib}] contains libraries

\item[{\tt src}] contains the source code for Cactus

\item [{\tt arrangements}] contains the Cactus arrangements. The arrangements
  (the actual ``physics'') are not supplied by checking out just Cactus.
  If the arrangements you want to use are standard Cactus arrangements, or
  reside on our CVS repository ({\tt cvs.cactuscode.org}),
  they can be checked out in similar way to the Flesh.
\end{Lentry}

When Cactus is first compiled it creates a new directory {\tt
Cactus/configs}, which will contain all the source code, object files and
libraries created during the build process.  Disk space may be a problem
on supercomputers where home directories are small.

A workaround is to first create a
configs directory on scratch space, say {\tt scratch/cactus\_configs/} (where
{\tt scratch/} is your scratch directory), and then either
\begin{itemize}
\item{} set the environment variable {\tt CACTUS\_CONFIGS\_DIR} to point to
this directory
\end{itemize}
or
\begin{itemize}
\item{}  soft link this directory ({\tt ln -s
scratch/cactus\_configs Cactus/configs/}) to the Cactus directory, if your
file-system supports soft-links.
\end{itemize}

Configurations are described in detail in section \ref{sec:coaco}.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Getting help}
\label{sec:gehe}

For tracking problem reports and bugs we use GNATS, which is a bugtracking
system published under the GNU license. We have set up a web interface at
{\tt http://www.cactuscode.org} which allows easy submission and browsing
of problem reports.

A description of the GNATS categories which we use is provided in the appendix
\ref{sec:usgn}.

% OK, there is NO emacs at the moment, because the GNATS setup is really stupid
% and sendpr handles like c.... besides the fact, that the user has to go
% through a make-process which installs stuff somewhere on his HD. gerd.
% BUT, we could distribute our own, either copy cvsbug, or write a perl
% version.  Tom
% \begin{itemize}
% \item {\tt A web interface}
% \item {\tt SendPR}
% {FIXME: Mention the emacs thing here too...}
% \end{itemize}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\chapter{Compilation}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


Cactus can be built in different configurations from the same copy of
the source files, and these different configurations coexist in the
{\tt Cactus/configs} directory. Here are several instances in which
 this can be useful:

\begin{enumerate}
\item{}Different configurations can be for {\em different
architectures}. You can keep executables for multiple architectures
based on a single copy of source code, shared on a common file
system.
\item{} You can compare different {\em compiler options, debug-modes}.
  You might want to compile different communication protocols
  (e.g. {\tt MPI} or {\tt GLOBUS}) or leave them out all together.
\item{} You can have different configurations for {\em different thorn
    collections} compiled into your executable.
\end{enumerate}

Once a configuration has been created, by {\tt gmake <config>} as described
in detail in the next section, a single call to {\tt gmake <config>}
will compile the code.  The first time it generates a compile
{\tt ThornList},  and gives you the chance to edit it before continuing.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Creating a configuration}
\label{sec:coaco}

At its simplest, this is done by {\tt gmake <config>}\footnote
%
{A note on the Cactus make system --- if at any point it prompts you
to enter something, the default value, which will be assumed if you
simply press enter, is shown in parantheses.}
%
.  This generates a
configuration with the name {\tt config}, doing its best to
automatically determine the default compilers and compilation flags
suitable for the current architecture.

There are a number of additional command line arguments which may be supplied
to override some parts of the procedure.

\subsection{Configuration options}

There are three ways to pass options to the configuration process.
% from the gmake command line.
\begin{enumerate}
\item{} Create a file \texttt{\~{ }/.cactus/config}.

All available configuration options may be set in the file {\tt
\~{ }/.cactus/config}, any which are not set will take a default value.
The file should contain lines of the form:

{\tt <option> [=] ...}

The equals sign is optional.

\item{} Add the options to a configuration file and use,

{\tt gmake <config>-config  options=<filename>}

The options file has the same format as {\tt \~{ }/.cactus/config}.

\item{} Pass the options individually on the command line,

{\tt gmake <configuration name>-config  <option name>=<chosen value>, ...}

Not all configuration options can be set on the command line.  Those that can
be set are indicated in the table below.
\end{enumerate}

They are listed here in order of increasing precedence, e.g. options
set on the command line will take priority over (potentially
conflicting) options set in \texttt{\~{ }/.cactus/config}.

It is important to note that these methods cannot be used to, for example, add
options to the default values for {\tt CFLAGS}.  Setting any variable in the
configuration file or the command line will overwrite completely the
default values.

\subsection{Available options}
\label{Compilation-Available_Options}

There is a plethora of available options.

\begin{itemize}

\item {\tt Compiled Thorns}

These specify the chosen set of thorns for compilation. If the thorn choice is not provided
during configuration, a list containing all thorns in the {\tt arrangements} directory
is automatically created, and the users prompted for any changes.

\begin{Lentry}

\item [{\tt THORNLIST}] Name of file containing a list of thorns with
the syntax {\tt <arrangement name>/<thorn name>}, lines beginning with
\# or ! are ignored.

\item [{\tt THORNLIST\_DIR}] Location of directory containing {\tt THORNLIST}.

%\item [{\tt THORNS}] List of thorns to use for compilation. This option can be used in
% 	conjunction with {\tt THORNLIST}. NOTE: In Beta 10 this will change to {\tt THORNLIST}.

\end{Lentry}

\item {\tt Compiler and tool specification}

These are used to specify which compilers and other tools to use. Entries followed
by * may be specified on the command line.

\begin{Lentry}
\item [{\tt CC}]   * The C compiler.
\item [{\tt CXX}]    The C++ compiler.
\item [{\tt F90}]  * The Fortran 90 compiler.
\item [{\tt F77}]  * The FORTRAN 77 compiler.
\item [{\tt CPP}]    The preprocessor used to generate dependencies and to preprocess Fortran code.
\item [{\tt LD}]   * The linker.
\item [{\tt AR}]     The archiver used for generating libraries.
\item [{\tt RANLIB}] The archive indexer to use.
\item [{\tt MKDIR}]  The program to use to create a directory.
\item [{\tt PERL}]   The name of the perl executable.
\end{Lentry}

\item {\tt Compilation and tool flags}

Flags which are passed to the compilers and the tools.

\begin{Lentry}
\item [{\tt CFLAGS}]
Flags for the C compiler.

\item [{\tt CXXFLAGS}]
Flags for the C++ compiler.

\item [{\tt F90FLAGS}]
* Flags for the Fortran 90 compiler.

\item [{\tt F77FLAGS}]
* Flags for the FORTRAN 77 compiler.

\item [{\tt MKDIRFLAGS}]
  Flags for MKDIR so that no error is given if the directory exists.
\item [{\tt LDFLAGS}]
* Flags for the linker.

\item [{\tt ARLAGS}]
Flags for the archiver.

\item [{\tt DEBUG}]
* Specifies what type of debug mode should be used,
the default is no debugging.
Current options are {\tt yes}, {\tt no}, or {\tt memory}. The option
{\tt yes} switches on all debugging features, whereas {\tt memory} just
employs memory tracing (\ref{sec:metr}).


\item [{\tt OPTIMISE, OPTIMIZE}]
* Specifies what type of optimisation should be used. The only options currently
available are {\tt yes} and {\tt no}.  The default is to use optimisation.\\
Note that the British spelling {\tt OPTIMISE} will be checked first and, if set,
will override any setting of the American-spelled {\tt OPTIMIZE}.

\item [{\tt C\_OPTIMISE\_FLAGS}]
Optimisation flags for the C compiler, their use depends on the type of
optimisation being used.

\item [{\tt CXX\_OPTIMISE\_FLAGS}]
Optimisation flags for the C++ compiler, their use depends on the type of
optimisation being used.

\item [{\tt F90\_OPTIMISE\_FLAGS}]
Optimisation flags for the FORTRAN 90 compiler, their use depends on the
type of optimisation being used.

\item [{\tt F77\_OPTIMISE\_FLAGS}]
Optimisation flags for the FORTRAN 77 compiler, their use depends on the
type of optimisation being used.

\item [{\tt C\_WARN\_FLAGS}]
Warning flags for the C compiler, their use depends on the type of
warnings used during compilation (\ref{sec:gmopfobuco}).

\item [{\tt CXX\_WARN\_FLAGS}]
Warning flags for the C++ compiler, their use depends on the type of
warnings used during compilation (\ref{sec:gmopfobuco}).

\item [{\tt F90\_WARN\_FLAGS}]
Warning flags for the FORTRAN 90 compiler, their use depends on the type of
warnings used during compilation (\ref{sec:gmopfobuco}).

\item [{\tt F77\_WARN\_FLAGS}]
Warning flags for the Fortran 77 compiler, their use depends on the type of
warnings used during compilation (\ref{sec:gmopfobuco}).

\end{Lentry}

\item {\tt Architecture-specific flags}

\begin{Lentry}
\item [{\tt IRIX\_BITS=32|64}] For Irix SGI systems: whether to build a 32- or 64-bit configuration.
\end{Lentry}

\begin{Lentry}
\item [{\tt AIX\_BITS=32|64}] For IBM SP systems: whether to build a 32- or 64-bit configuration.
\end{Lentry}

\item {\tt Library flags}

Used to specify auxiliary libraries and directories to find them in.

\begin{Lentry}
\item [{\tt LIBS}] The additional libraries.
\item [{\tt LIBDIRS}] Any other library directories.
\end{Lentry}

\item {\tt Extra include directories}

\begin{Lentry}
\item [{\tt SYS\_INC\_DIRS}]
Used to specify any additional directories for system include files.
\end{Lentry}


\item {\tt Precision options}

Used to specify the precision of the default real and integer data types,
specified as the number of bytes the data takes up.  Note that not all
values will be valid on all architectures.

\begin{Lentry}

\item [{\tt REAL\_PRECISION}]
* Allowed values are 16, 8, 4.

\item [{\tt INTEGER\_PRECISION}]
* Allowed values are 8, 4 and 2.

\end{Lentry}

\item {\tt Executable name}

\begin{Lentry}
\item [{\tt EXEDIR}] The directory in which to place the executable.
\item [{\tt EXE}] The name of the executable.
\end{Lentry}

\item{\tt Extra packages}

Compiling with extra packages is described fully in
Section \ref{sec:cowiexpa},
which should be consulted for the full range of configuration options.

\begin{Lentry}
\item [{\tt MPI} *] The {\tt MPI} package to use, if required. Supported values are
        {\tt CUSTOM}, {\tt NATIVE}, {\tt MPICH} or {\tt LAM}.

\item [{\tt HDF5}]
Supported values are {\it yes}.

\item [{\tt PTHREADS}]
Supported values are {\it yes}.

\end{Lentry}

\end{itemize}



%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\subsection{Compiling with extra packages}
\label{sec:cowiexpa}


\subsubsection{MPI: Message Passing Interface}

{\tt MPI} (the {\it Message Passing Interface}) provides inter-processor
communication. It can either be implemented natively on a machine
(this is usual on most supercomputers), or through a standard package
such as {\tt MPICH}, {\tt LAM}, {WMPI}, or {PACX}.

To compile with MPI, the configure option is

{\tt MPI = <MPI\_TYPE>}

where {\tt <MPI\_TYPE>} can take the values (entries followed by *
may be specified on the configuration command line):

\begin{Lentry}

\item[{\tt CUSTOM}] For a custom {\tt MPI} configuration set the variables
  \begin{Lentry}
  \item [{\tt MPI\_LIBS} *] libraries.
  \item [{\tt MPI\_LIB\_DIRS} *] library directories.
  \item [{\tt MPI\_INC\_DIRS} *] include file directories.
  \end{Lentry}

\item[{\tt NATIVE}] Use the native {\tt MPI} for this machine, as indicated in
        the {\tt known-architectures} directory
	({\tt lib/make/known-architectures}).

\item[{\tt MPICH}]
Use MPICH ({\tt http://www-unix.mcs.anl.gov/mpi/mpich}). This is controlled
by the options
  \begin{Lentry}
  \item [{\tt MPICH\_ARCH} *] machine architecture.
  \item [{\tt MPICH\_DIR} *] directory in which {\tt MPICH} is installed.
        If this option is not defined it will be searched for.
  \item [{\tt MPICH\_DEVICE} *] the device used by {\tt MPICH}. If not
        defined, the configuration process will search for this in a
        few defined places.
        Supported devices are currently {\tt ch\_p4}, {\tt ch\_shmem},
	{\tt globus} and {\tt myrinet}.
	For versions of MPICH prior to 1.2.0 the devices are searched for
 	in this order, for 1.2.0 you may need to specify {\tt MPICH\_DEVICE},
	depending on the installation.
  \end{Lentry}

If {\tt MPICH\_DEVICE} is chosen to be {\tt globus}, ({\tt www.globus.org}),
an additional variable must be set
  \begin{Lentry}
  \item[{\tt GLOBUS\_LIB\_DIR} *] directory in which Globus libraries are installed.
  \end{Lentry}

If {\tt MPICH\_DEVICE} is chosen to be {\tt ch\_gm}, ({\tt www.myri.com}),
an additional variable must be set
  \begin{Lentry}
  \item[{\tt MYRINET\_DIR} *] directory in which Myrinet libraries are installed.
  \end{Lentry}

\item[{\tt LAM}]
Use {\tt LAM} (Local Area Multicomputer, {\tt http://www.lam-mpi.org/}). This is
controlled by the variables
  \begin{Lentry}
  \item[{\tt LAM\_DIR} *] directory in which {\tt LAM} is installed. This
     will be searched for in a few provided places if not given.
  \end{Lentry}
if the {\tt LAM} installation splits libraries and include files into different
directories, instead of setting {\tt LAM\_DIR} set the two variables
  \begin{Lentry}
  \item[{\tt LAM\_LIB\_DIR} *] directory in which {\tt LAM} libraries are installed.
  \item[{\tt LAM\_INC\_DIR} *] directory in which {\tt LAM} include files are installed.
  \end{Lentry}


\item[{\tt WMPI}]
Use WMPI (Win32 Message Passing Interface, {\tt http://dsg.dei.uc.pt/w32mpi/intro.html}). This is controlled by the variable
  \begin{Lentry}
  \item[{\tt WMPI\_DIR} *] directory in which {\tt WMPI} is installed.
  \end{Lentry}

\item[{\tt HPVM}]
Use HPVM (High Performance Virtual Machine,\\{\tt http://www-csag.ucsd.edu/projects/hpvm.html}). This is controlled by the variable
  \begin{Lentry}
  \item[{\tt HPVM\_DIR} *] directory in which {\tt HPVM} is installed.
  \end{Lentry}

\item[{\tt MPIPro}]
Use MPIPro ({\tt http://www.mpi-softtech.com/}).

\item[{\tt PACX}]
Use the PACX Metacomputing package (PArallel Computer eXtension,\\
{\tt http://www.hlrs.de/structure/organisation/par/projects/pacx-mpi/}). This is controlled by the variables
  \begin{Lentry}
  \item[{\tt PACX\_DIR} *] directory in which {\tt PACX} is installed.
        If this option is not defined it will be searched for.
  \item[{\tt PACX\_MPI} *] the MPI package {\tt PACX} uses for node-local
        communication. This can be any of the above MPI packages.
  \end{Lentry}

\end{Lentry}

Note that the searches for libraries etc. mentioned above use the
locations given in the files in {\tt lib/make/extras/MPI}.

\subsubsection{HDF5: Hierarchical Data Format version 5}

To compile with HDF5 ({\tt http://hdf.ncsa.uiuc.edu/whatishdf5.html}),
the configure options are

{\tt HDF5 = YES [HDF5\_DIR = <dir>] [LIBZ\_DIR = <dir>]}

If HDF5\_DIR is not given the configuration process will search for an
installed HDF5 package in some standard places (defined in
{\tt lib/make/extras/HDF5}). If the found HDF5 library was compiled with
libz.a, the configuration process also searches for that library and adds it
to the linker flags. You may also point directly to an installation of libz.a
by setting LIBZ\_DIR.\\


\subsubsection{PTHREADS: POSIX threads}

To enable multithreading support within Cactus using POSIX threads
the configure option is

{\tt PTHREADS = yes}

The configuration process will check if a re-entrant C library is available
and adds it to the linker flags. It will also search for the system's Pthreads
library (either libpthread or libpthreads) and set preprocessor
defines necessary for compiling multithreaded code.


\subsection{File layout}

The configuration process sets up various subdirectories and files in the
{\tt configs} directory to contain the configuration specific files, these
are placed in a directory with the name of the configuration.

\begin{Lentry}

\item [{\tt config-data}] contains the files created by the configure
script:

The most important ones are

\begin{Lentry}

\item [{\tt make.config.defn}]
contains compilers and compilation flags for a configuration.

\item [{\tt make.extra.defn}]
contains details about extra packages used in the configuration.

\item [{\tt cctk\_Config.h}]
The main configuration header file, containing architecture specific
definitions.

\item [{\tt cctk\_Archdefs.h}]
An architecture specific header file containing things which cannot be
automatically detected, and have thus been hand-coded for this architecture.
\end{Lentry}

These are the first files which should be checked or modified to suit any
peculiarities of this configuration.

In addition the following files may be informative:

\begin{Lentry}
\item [{\tt fortran\_name.pl}]
A perl script used to determine how the Fortran compiler names subroutines.
This is used to make some C routines callable from Fortran, and Fortran
routines callable from C.

\item [{\tt make.config.deps}]
Initially empty.  Can be edited to add extra architecture specific dependencies
needed to generate the executable.

\item [{\tt make.config.rule}]
Make rules for generating object files from source files.

\end{Lentry}

Finally, autoconf generates the following files.

\begin{Lentry}

\item [{\tt config.log}]
A log of the autoconf process.

\item [{\tt config.status}]
A script which may be used to regenerate the configuration.

\item [{\tt config.cache}]
An internal file used by autoconf.

\end{Lentry}

\item [{\tt lib}]
An empty directory which will contain the libraries created for each thorn.

\item [{\tt build}]
An empty directory which will contain the object files generated for this
configuration, and preprocessed source files.

\item [{\tt config-info}]
A file containing information about the configuration.

\item [{\tt bindings}] A directory which contains all the files
generated by the CST from the \texttt{.ccl} files.

\item [{\tt scratch}]
A scratch directory which is used to accomodate Fortran 90 modules.

\end{Lentry}


\section{Building and Administering a configuration}
\label{sec:buanadaco}

Once you have created a new configuration, the command
\\ \\
{\tt gmake <configuration name>}
\\ \\
will build an executable, prompting you along the way for the
thorns which should be included. There is a range of  gmake
targets and options which are detailed in the following sections.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{gmake targets for building and administering configurations}
\label{sec:gmtafobuanadco}

A target for {\tt gmake} can be naively thought of as an argument
that tells it which of several things listed in the {\tt Makefile} it
is to do. The command {\tt gmake help} lists all gmake targets:
% colon clarifies that all (config) targets are listed here

\begin{Lentry}

\item [{\tt gmake <config>}]
	builds a configuration. If the configuration doesn't exist
        it will create it.

\item [{\tt gmake <config>-clean}] removes all object and dependency files from
  a configuration.

\item [{\tt gmake <config>-cleandeps}] removes all dependency files from
  a configuration.

\item [{\tt gmake <config>-cleanobjs}] removes all object files from
  a configuration.

\item [{\tt gmake <config>-config}] creates a new configuration or reconfigures an old one.

\item [{\tt gmake <config>-cvsupdate}] update the Flesh and Thorns for a configuration using CVS

\item [{\tt gmake <config>-delete}] deletes a configuration ({\tt rm -r configs/<config>}).

\item [{\tt gmake <config>-editthorns}] edits the ThornList.

\item [{\tt gmake <config>-examples}] copies all the example parameter files relevant for this configuration to the directory {\tt examples} in the Cactus home directory. If a file of the same name is already there, it will not overwrite it.

\item [{\tt gmake <config>-realclean}] removes from a configuration
all object and dependency files, as well as files generated from the
CST (stands for Cactus Specification Tool, which is the perl scripts
which parse the thorn configuration files).  Only the files generated
by configure and the ThornList file remain.

\item [{\tt gmake <config>-rebuild}] rebuilds a configuration (reruns the CST).

\item [{\tt gmake <config>-testsuite}] runs the test programs associated with
 each thorn in the configuration. See section \ref{sec:te} for information about the
 testsuite mechanism.

\item [{\tt gmake <config>-thornlist}] regenerates the ThornList for a configuration.

\item [{\tt gmake <config>-utils [UTILS$=$<list>]}] builds all utility programs provided by the thorns of a configuration. Individual utilities can be selected by giving their names in the {\tt UTILS} variable.

\item[{\tt gmake <config>-ThornGuide}] builds documentation for the thorns
in this configuration.

\item[{\tt gmake <config>-configinfo}] displays the options used to build the configuration.

\item[{\tt gmake <config>-cvsupdate}] updates the Flesh and this configuration's Thorns from the CVS repositories.

\end{Lentry}



\subsection{Compiling in thorns}
\label{sec:cointh}

Cactus will try to compile all thorns listed in
{\tt configs/<config>/ThornList}.
The {\tt ThornList} file is simply a list of the form
{\t <arrangement>/<thorn>}.  All text after a \# or ! sign
on a line is treated as a comment and ignored.
The first time that you compile a configuration, if you did
not specify a ThornList already during configuration,
you will be shown a list of all the thorns in your arrangement
directory, and asked if you with to edit them. You can regenerate
this list at anytime by typing

\begin{verbatim}
gmake <config>-thornlist
\end{verbatim}

or you can edit it using

\begin{verbatim}
gmake <config>-editthorns
\end{verbatim}

Instead of using the editor to specify the thorns you want to
  have compiled, you can {\em edit} the {\em ThornList} outside
  the make process. It is located in {\tt configs/<config>/ThornList},
  where {\tt <config>} refers to the name of your configuration.
  The directory, {\tt ./configs}, exists {\em
    after} the very first  make phase for the first configuration.

\subsection{Notes and Caveats}
\begin{itemize}
\item{} If during the build you see the error ``{\tt missing
    separator}'' you are probably not using GNU make.
\item{} {\em The EDITOR environment variable}. You may not be aware of
  this, but this thing very often exists and may be set  by default to
  something scary like {\tt vi}. If you don't know how to use {\tt vi}
  or wish to
  use your favorite editor instead, reset this environment variable.
  (To exit {\tt vi} type {\tt <ESC> :q!})
\end{itemize}

\subsection{gmake options for building configurations}
\label{sec:gmopfobuco}

An {\it option} for gmake can be thought of as an argument which tells
it how it should make a {\tt target}. Note that the final result is always
the same.

\begin{Lentry}
% This works as either a config or a build option:
\item [{\tt gmake <target> PROMPT=no}] turns off all prompts from the
make system.
% This should be a config option:
%\item [{\tt gmake <target> THORNLIST=<file> [THORNLIST\_DIR=<dir>]}] uses the file {\tt dir/file} as the ThornList for the configuration. The directory defaults to the current directory.
\item [{\tt gmake <target> SILENT=no}] print the commands that gmake is executing.
\item [{\tt gmake <target> WARN=yes}] show compiler warnings during compilation.
\item [{\tt gmake <target> FJOBS=<number>}] compile in parallel, across files within each thorn.
\item [{\tt gmake <target> TJOBS=<number>}] compile in parallel, across thorns.

\end{Lentry}

Note that with more modern versions of gmake, it is sufficient to pass the normal
 {\tt -j <number>} flag to gmake to get parallel compilation.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%




%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Other gmake targets}

\begin{Lentry}

\item [{\tt gmake help}] lists all make options.

\item [{\tt gmake checkout}] allows you to easily checkout Cactus
arrangements and thorns.  For example it can checkout all the thorns
in any thornlist file found in the \texttt{thornlists} subdirectory of
the Cactus root directory. % (usually \texttt{Cactus}).

\item [{\tt gmake cvsdiff}] differences between checked out version of Cactus and that in the CVS repositories.

\item [{\tt gmake cvsstatus}] status of checked out version of Cactus, reporting which files have been modified or need updating.

\item [{\tt gmake cvsupdate}] update Flesh and all Thorns from CVS repositories.

\item [{\tt gmake default}] creates a new configuration with a default name.

\item [{\tt gmake distclean}] delete your {\tt configs} directory and hence all your configurations.

\item [{\tt gmake downsize}] removes non-essential files as documents
  and testsuites to allow for minimal installation size.

\item [{\tt gmake newthorn}] creates a new thorn, prompting for the necessary
  information and creating template files.

\item [{\tt gmake TAGS}] creates an Emacs style TAGS file. See section
  \ref{sec:usta} for using TAGS within Cactus.

\item [{\tt gmake tags}] creates a {\tt vi} style tags file. See section
  \ref{sec:usta} for using TAGS within Cactus.

\item [{\tt gmake UsersGuide}] runs LaTeX to produce a copy of the Users' Guide.

\item [{\tt gmake ThornGuide}] runs LaTeX to produce a copy of the Thorn Guide, for all the thorns in the arrangements directory.

\item [{\tt gmake MaintGuide}] runs LaTeX to produce a copy of the Maintainers' Guide.

\item [{\tt gmake configinfo}] prints configuration options for every
configuration found in user's \texttt{configs} subdirectory.

\end{Lentry}


\section{Testing}
\label{sec:te}

Some thorns come with a testsuite, consisting of example parameter files
and the output files generated by running these. To run the testsuites
for the thorns you have compiled use

{\tt gmake <configuration>-testsuite}

These testsuites serve the dual purpose of

\begin{Lentry}
\item [Regression testing]
i.e. making sure that changes to the thorn or the flesh don't affect the
output from a known parameter file.
\item [Portability testing]
i.e. checking that the results are independent of the architecture --- this
is also of use when trying to get Cactus to work on a new architecture.
\end{Lentry}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\chapter{Running Cactus}

Cactus executables always run from a parameter file (which may be a
physical file or taken from standard input), which specifies which
Thorns to use and set the values of any parameters which are different
from the default values. Any accepted filename can be used for the name
of the parameter file, although standard convention is to use the file
extension {\tt .par}.  Optional command line arguments can be used
to customise runtime behaviour, and to provide information about the
Thorns used in the executable. The general syntax for running Cactus from
a physical parameter file is
then

{\tt ./cactus\_<config> <parameter file> [command line options]}

or if the parameter file should be taken from standard input

{\tt ./cactus\_<config> [command line options] -}

The remainder of this chapter covers all aspects for running your
Cactus executable.  These include: command line options, parameter
file syntax, understanding screen output, environment variables, and
creating thorn documentation.

\section{Command Line Options}
\label{sec:coliop}

The cactus executable accepts numerous command line arguments:

{\tt
\begin{tabular}{|l|l|}
\hline
Short Version & Long Version \\
\hline
 -O[v] & -describe-all-parameters \\
\hline
 -o <param> & -describe-parameter <param> \\
\hline
 -T & -list-thorns\\
\hline
 -t <arrangement/thorn>& -test-thorn-compiled <arrangement/thorn>\\
\hline
 -h,-? & -help\\
\hline
 -v & -version \\
\hline
 -x [<nprocs>] & -test-parameters [<nprocs>] \\
\hline
 -W <level> & -warning-level <level> \\
\hline
 -E <level> & -error-level <level> \\
\hline
 -r & -redirect-stdout \\
\hline
 -i & -ignore-next \\
\hline
    & -parameter-level <level> \\
\hline
\end{tabular}
}

\begin{Lentry}
\item [{\tt -O} or {\tt -describe-all-parameters}]
Produces a full list of all parameters from all thorns which were compiled,
along with descriptions and allowed values.  This can take an optional extra
parameter {\tt v}  (i.e. {\tt -Ov} to give verbose information about
all parameters).
\item [{\tt -o <param>} or {\tt -describe-parameter <param>}]
Produces the description and allowed values for a given parameter --- takes one
argument.
\item [{\tt -T} or {\tt -list-thorns}]
Produces a list of all the thorns which were compiled in.
\item [{\tt -t <arrangement or thorn>} or {\tt -test-thorn-compiled <arrangement or thorn>} ]
Checks if a given thorn was compiled in - takes one argument.
\item [{\tt -h}, {\tt -?} or {\tt -help}]
Produces a help message.
\item [{\tt -v} or {\tt -version}]
Produces version information of the code.
\item [{\tt -x <nprocs>} or {\tt -test-parameters <nprocs>}]
Runs the code far enough to check the consistency of the parameters.  If
given a numeric argument it will attempt to simulate being on that number
of processors. [To be implemented.]
\item [{\tt -W <level>} or {\tt -waring-level <level>}]
Sets the warning level of the code.  All warning messages are given a level ---
the lower the level the greater the severity.  This parameter controls the
level of messages to be seen, with all warnings of value {\tt <level>} or higher printed to standard out.  The default is a warning level of 1, with
0 indicating that only those messages which are (by default) fatal should
be seen.
\item [{\tt -E <level} or {\tt -error-level <level>}]
This works in concert with {\tt -W} --- it controls which warning level is
treated as a fatal error.  This cannot be set to a higher value than
{\tt -W}. The default value is zero.
\item [{\tt -r} or {\tt -redirect-stout}]
This redirects the standard output of each processor to a file.  By default
the output from processors other than processor 0 is discarded.
\item [{\tt -i} or {\tt -ignore-next}]
Ignore the next argument on the command line.
\item [{\tt -parameter-level}]
Set the level of parameter checking to be used, either {\tt strict}, {\tt normal} (the default), or {\tt relaxed}. See Section~\ref{sec:pafisy}.
\end{Lentry}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Parameter File Syntax}
\label{sec:pafisy}

The parameter file is used to control the behaviour of the code at runtime.
It is a text file with lines which are either comments, denoted
by a `\#' or `!', or parameter statements. A parameter statement consists
of one or more parameter names, followed by
an `=', followed by the value(s) for this (these) parameter(s).
Note that all string parameters are case insensitive.

The {\tt first parameter} in any parameter file should be {\tt ActiveThorns}.
This is a special parameter  which tells the
code which {\em thorns} are to be activated.  Only parameters from active
thorns can be set  (and only those routines {\it scheduled} by active thorns
are run).  By default all thorns are inactive. For example, the first
entry in a parameter file which is using just the two thorns
{\tt CactusPUGH/PUGH} and {\tt CactusBase/CartGrid3D} should be

{\tt ActiveThorns = ``PUGH CartGrid3D''}

Parameters following the {\tt ActiveThorns} parameter all have names
whose syntax depends on the scope of the parameter:
\begin{Lentry}
\item [{\tt Global parameters}]
Just the name of the parameter itself. Global parameters are avoided, and
there are none in the Flesh and Cactus Toolkits.
\item [{\tt Restricted parameters}]
The name of the {\em implementation} which defined the parameter, two colons,
and the name of the parameter --- e.g. {\tt driver::global\_nx}.
\item [{\tt Private parameters}]
The name of the {\em thorn} which defined the parameter, two colons,
and the name of the parameter --- e.g. {\tt wavetoyF77::amplitude}.
\end{Lentry}

This notation is not strictly enforced currently in the code. It is
sufficient to specify the first part of the parameter name using either
the implementation name, or the thorn name. However, it is suggested
that the convention above is followed.

The Cactus Flesh performs checks for consistency and range of parameters,
the severity of these checks is controlled by the command line argument
{\tt -parameter-level} which can take the following values
\begin{Lentry}
\item[{\tt relaxed}] Cactus will issue a level 0 warning (that is the
default behaviour will be to terminate) if
\begin{itemize}
\item{} The specified parameter value is outside of the allowed range.
\end{itemize}

\item [{\tt normal}]
This is the default, and provides the same warnings as the
{\tt relaxed} level, with in addition a level 0 warning issued for
\begin{itemize}
\item{} An implementation and/or thorn {\tt foo} is active, but the
	parameter {\tt foo::bar} was not defined.
\item{} The parameter {\tt foo::bar} was successfully set for both an
	active implementation {\tt foo} not implemented by a thorn {\tt foo},
	and to a thorn {\tt foo}.
\end{itemize}

\item [{\tt strict}]
This provides the same warnings as the {\tt normal} level, with in
addition a level 0 warning issued for
\begin{itemize}
\item{} The parameter {\tt foo::bar} is specified in the parameter file,		but no implementation or thorn with the name {\tt bar} is active.
\end{itemize}
\end{Lentry}

Notes:

\begin{itemize}

\item{} You can obtain lists of the parameters associated with
each thorn using the command line options {\tt -o} and {\tt -O}
(Section~\ref{sec:coliop}).

\item{} The parameter file is read {\it sequentially} from top to bottom,
	this means that if you set the value of a parameter twice in
	the parameter file, the second value will be used. (This is
	why the {\tt ActiveThorns} parameter is always first in the file).

\item{} Some parameters are {\it steerable} and can be changed during
	the execution of a code using parameter steering interfaces
	(for example, thorn {\tt CactusConnect/HTTPD}, or using a
	parameter file when recovering from a checkpoint file.

\item{} For examples of parameter files, look in the {\tt par} directory
	which can be found in most thorns.

\end{itemize}

\section{Thorn Documentation}
\label{sec:thdo}

The Cactus make system provides a mechanism for generating a {\it Thorn Guide}
containing separate chapters for each thorn in your configuration. The
provided documentation for an individual thorn obviously depends on what
the thorn authors added, but the Thorn Guide is a good place to first look 
for special instructions on how to run and interpret the output from a Thorn.
Details about parameters, grid variables and scheduling are automatically
included in from a thorns {\tt CCL} files into the Thorn Guide. To construct a 
Thorn Guide for the configuration {\tt $<$config$>$} use

{\tt gmake $<$config$>$-ThornGuide}

or to make a Thorn Guide for all the thorns in the {\tt arrangements} directory

{\tt gmake $<$config$>$}.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


\chapter{Getting and looking at output}


\section{Screen output}

As your Cactus executable runs, standard output and standard error
are usually written to the screen. Standard output provides you
with information about the run, and standard error reports warnings
and errors from the flesh and thorns.

As the program runs, the normal output provides the following information:

\begin{Lentry}

\item [Active thorns]
	A report is made as each of the thorns in the {\tt ActiveThorns} parameters from the parameter file is attempted to be activated. This report
shows whether the activation was successful, and if successful gives the
thorn's implementation. For example

{\tt
\begin{verbatim}
Activating thorn idscalarwave...Success -> active implementation idscalarwave
\end{verbatim}
}

\item [Failed parameters]
 	If any of the parameters in the parameter file does not belong to any of the active thorns, or if the parameter value is not in the allowed range, an
error is registered. For example, if the parameter is not recognised

{\tt
\begin{verbatim}
Unknown parameter time::ddtfac�
\end{verbatim}
}
or if the parameter value is not in the allowed range

{\tt
\begin{verbatim}
Unable to set keyword CartGrid3D::type - ByMouth not in any active range
\end{verbatim}
}

\item [Scheduling information]
	A complete list of all scheduled routines is given, in the
order that they will be executed. For example

{\tt
\begin{verbatim}
----------------------------------------------------------------------
  Startup routines
    Cactus: Register banner for Cactus
    CartGrid3D: Register GH Extension for GridSymmetry
    CartGrid3D: Register coordinates for the Cartesian grid
    IOASCII: Startup routine
    IOBasic: Startup routine
    IOUtil: IOUtil startup routine
    PUGH: Startup routine
    WaveToyC: Register banner

  Parameter checking routines
    CartGrid3D: Check coordinates for CartGrid3D
    IDScalarWave: Check parameters

  Initialisation
    CartGrid3D: Set up spatial 3D Cartesian coordinates on the GH
    PUGH: Report on PUGH set up
    Time: Set timestep based on speed one Courant condition
    WaveToyC: Schedule symmetries
    IDScalarWave: Initial data for 3D wave equation

  do loop over timesteps
    WaveToyC: Evolution of 3D wave equation
    t = t+dt
    if (analysis)
    endif
  enddo
----------------------------------------------------------------------
\end{verbatim}
}

\item [Thorn banners]
	Any banners registered from the thorns are displayed.

\end{Lentry}


\section{Output}
Output methods in Cactus are all provided by thorns. Any number of output
methods can be used for each run. The behaviour of the output thorns in the
standard arrangements are described in those thorns' documentation.

In general, these thorns decide what to output by parsing a string parameter
containing the names of those grid variables, or groups of variables, for which
output is required. The names should be fully qualified with the {\tt
implementation} and {\tt group} or {\tt variable} names.

There is usually a parameter for each method to denote how often, in evolution
iterations, this output should be performed.  There is also usually a parameter
to define the directory in which the output should be placed, defaulting to the
directory from which the executable is run.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Checkpointing/Recovery}

Checkpointing is defined as saving the current state of a run (parameter
settings, contents of grid variables, and other relevant information) to a file.
At a later time, this run can then be restarted from that state by recovering
all the data from the checkpoint file.

Checkpointing/recovery methods in Cactus are provided by thorns.

In general, these thorns decide how often to generate a checkpoint. They also
register their recovery routines with the flesh which then get called during
initialization to perform the recovery of parameters and grid variables if
requested in the parameter file.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\end{cactuspart}