% /*@@ % @file RunningCactus.tex % @date 27 Jan 1999 % @author Tom Goodale, Gabrielle Allen, Gerd Lanferman, Thomas Radke % @desc % How to run Cactus part of the Cactus User's Guide % @enddesc % @version $Header$ % @@*/ \begin{cactuspart}{Installation and Running}{$RCSfile$}{$Revision$} \label{part:RunningCactus} \renewcommand{\thepage}{\Alph{part}\arabic{page}} %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \chapter{Installation} \label{cha:in} %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \section{Required software} \label{sec:required_software} In general, Cactus \emph{requires} the following set of software to function in single processor mode. Please refer to the architecture section \ref{sec:suar} for architecture specific items. \begin{Lentry} \item[Perl5.0] Perl is used extensively during the Cactus thorn configuration phase. Perl is available for nearly all operating systems known to man and can be obtained at \url{http://www.perl.org} \item[GNU make] The make process works with the GNU make utility (referred to as \texttt{gmake} henceforth). While other make utilities may also work, this is not guaranteed. Gmake can be obtained from your favorite GNU site or from \url{http://www.gnu.org} \item[C] C compiler. For example, the GNU compiler. This is available for most supported platforms. Platform specific compilers should also work. \item[CPP] C Preprocessor. For example, the GNU \texttt{cpp}. These are normally provided on most platforms, and many C compilers have an option to just run as a preprocessor. \item[CVS] The \textit{Concurrent Versions System} is not needed to run/compile Cactus, but you are strongly encourage to install this software to take advantage of the update procedures. It can be downloaded from your favorite GNU site. Tar files of each release are also available. \end{Lentry} \noindent To use Cactus, with the default driver\footnote{For help with unfamiliar terms, please consult the glossary, Appendix \ref{sec:glossary}.} (\texttt{CactusPUGH/PUGH}) on multiple processors you also need: \begin{Lentry} \item[MPI] The \textit{Message Passing Interface} which provides inter-processor communication. Supercomputing sites often supply a native MPI implementation with which Cactus is very likely to be compatible. Otherwise there are various freely available ones available, e.g. the MPICH version of MPI is available for various architectures and operating systems at \url{http://www-unix.mcs.anl.gov/mpi/}. \end{Lentry} \noindent If you are using any thorns containing routines written in C++ you also need \begin{Lentry} \item[C++] C++ compiler. For example, the GNU compiler. This is available for most supported platforms. Platform specific compilers should also work. Note that if a C++ compiler is available then the \text{main()} routine in the Flesh is compiled with C++ to allow static class initialisations. \end{Lentry} \noindent If you are using any thorns containing routines written in Fortran you also need \begin{Lentry} \item[F90/F77] For routines written in Fortran 77, either an Fortran 90 or a Fortran 77 compiler can be used. For routines written in Fortran 90 a Fortran 90 compiler is obviously required. There is a very limited set of free Fortran 90 compilers available for the different architectures. \end{Lentry} \noindent While not required for compiling or running Cactus, for thorn development it is useful to install \begin{Lentry} \item[\texttt{ctags/etags}] These programs enable you browse through the calling structure of a program by help of a function call database. Navigating the Flesh and arrangements becomes very easy. Emacs and \texttt{vi} both support this method. See \ref{sec:Appendix.tags} for a short guide to tags. \end{Lentry} %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \section{Supported architectures} \label{sec:suar} Cactus runs on many machines, under a large number of operating systems. Here we list the machines we have compiled and verified Cactus on, including some architecture specific notes. A complete list of architectures supported by Cactus, along with more notes, can be found at \begin{center} \url{http://www.cactuscode.org/Documentation/Architectures.html}. \end{center} \begin{Lentry} \item[\textbf{SGI}] 32 or 64 bit running Irix. \item[\textbf{Cray T3E}] \item[\textbf{Compaq Alpha}] Compaq operating system and Linux. Single processor mode and MPI supported. The Alphas need to have the GNU C/C++ compilers installed. \item[\textbf{IA32}] running Linux, OpenBSD, FreeBSD, or Windows 2000/NT. Single processor mode and MPI (MPICH and LAM) supported.\\ On Windows Cactus compiles with Cygwin. MPI (WMPI, HPVM, and MPIPro) supported. Please read \texttt{doc/README.NT} for details. \item[\textbf{IA64}] running Linux. \item[\textbf{Macintosh PowerPC}] (MacOS X and Linux PPC) \item[\textbf{IBM SP2,SP3,SP4}] 32 or 64 bit running AIX. \item[\textbf{Hitachi SR8000-F1}] \item[\textbf{Sun} Solaris] \item[\textbf{Fujitsu}] \item[\textbf{NEC SX-5, SX-6}] \end{Lentry} The following machines are only partially supported \begin{Lentry} \item[\textbf{HP Exemplar}] \end{Lentry} %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \section{Checkout procedure} \label{sec:checkout} Cactus is distributed, extended, and maintained using the free CVS software (\textit{Concurrent Versions System}: \url{http://www.cvshome.org}). CVS allows many people to work on a large software project together without getting into a tangle. Since Cactus thorns are distributed from several repositories on the main CVS site, and from a growing number of user sites, we provide a script, described below, on our web site for checking out the Flesh and thorns. The Cactus web site also provides a form interface for direct download. CVS experts who want to use raw CVS commands are directed to Appendix~\ref{sec:Appendix.cvs} for full instructions. For CVS novices, we also summarize in this appendix basic CVS commands. The space required for an installation depends on the arrangements and thorns used. The Flesh on its own requires less than 5 MB. The script for checking out the Flesh and distribution thorns, \texttt{GetCactus}, is available from the web site at \url{http://www.cactuscode.org/download/GetCactus} The script takes as an argument the name of a file containing a \textit{ThornList}, that is a list of thorns with the syntax \begin{alltt} <\var{arrangement name}>/<\var{thorn name}> \end{alltt} If no filename is given, only the Flesh is checked out. Optional directives in the ThornList indicate which CVS repository to fetch thorns from. The default is to take the thorns from the same repository as the Flesh. A full description of ThornList syntax is provided in Appendix~\ref{chap:th}. ThornLists for example applications are provided on the Cactus web site. The same script can be used to checkout additional thorns. Another script, \texttt{MakeThornList}, can be used to produce a minimal ThornList from a given Cactus par file. It needs a \emph{master} ThornList to be copied into your \texttt{~\.cactus} directory. See \url{http://www.cactuscode.org/toolkit/makeThornList/}. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \section{Directory structure} \label{sec:dist} A fresh checkout creates a directory \texttt{Cactus} with the following subdirectories: \begin{Lentry} \item[\texttt{CVS}] the CVS bookkeeping directory, present in every subdirectory \item[\texttt{doc}] Cactus documentation \item[\texttt{lib}] contains libraries \item[\texttt{src}] contains the source code for Cactus \item [\texttt{arrangements}] contains the Cactus arrangements. The arrangements (the actual ``physics'') are not supplied by checking out just Cactus. If the arrangements you want to use are standard Cactus arrangements, or reside on our CVS repository (\texttt{cvs.cactuscode.org}), they can be checked out in similar way to the Flesh. \end{Lentry} When Cactus is first compiled it creates a new directory \texttt{Cactus/configs}, which will contain all the source code, object files and libraries created during the build process. Disk space may be a problem on supercomputers where home directories are small. A workaround is to first create a configs directory on scratch space, say \texttt{scratch/cactus\_configs/} (where \texttt{scratch/} is your scratch directory), and then either \begin{itemize} \item{} set the environment variable \texttt{CACTUS\_CONFIGS\_DIR} to point to this directory \end{itemize} or \begin{itemize} \item{} soft link this directory (\texttt{ln -s scratch/cactus\_configs Cactus/configs/}) to the Cactus directory, if your filesystem supports soft links. \end{itemize} Configurations are described in detail in section \ref{sec:configurations}. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \section{Getting help} \label{sec:gehe} For tracking problem reports and bugs we use GNATS, which is a bug tracking system published under the GNU license. We have set up a web interface at \url{http://www.cactuscode.org} which allows easy submission and browsing of problem reports. A description of the GNATS categories which we use is provided in the appendix \ref{sec:Appendix.gnats}. % OK, there is NO emacs at the moment, because the GNATS setup is really stupid % and sendpr handles like c.... besides the fact, that the user has to go % through a make process which installs stuff somewhere on his HD. gerd. % BUT, we could distribute our own, either copy cvsbug, or write a perl % version. Tom % \begin{itemize} % \item \texttt{A web interface} % \item \texttt{SendPR} % {FIXME: Mention the emacs thing here too...} % \end{itemize} %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \chapter{Compilation} %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% Cactus can be built in different configurations from the same copy of the source files, and these different configurations coexist in the \texttt{Cactus/configs} directory. Here are several instances in which this can be useful: \begin{enumerate} \item{}Different configurations can be for \emph{different architectures}. You can keep executables for multiple architectures based on a single copy of source code, shared on a common file system. \item{} You can compare different \textit{compiler options, debug-modes}. You might want to compile different communication protocols (e.g. MPI or Globus) or leave them out all together. \item{} You can have different configurations for \textit{different thorn collections} compiled into your executable. \end{enumerate} Once a configuration has been created, by \texttt{gmake <\var{config}>} as described in detail in the next section, a single call to \texttt{gmake <\var{config}>} will compile the code. The first time it generates a compile \texttt{ThornList}, and gives you the chance to edit it before continuing. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \section{Creating a configuration} \label{sec:configurations} At its simplest, this is done by \texttt{gmake <\var{config}>}\footnote % {A note on the Cactus make system --- if at any point it prompts you to enter something, the default value, which will be assumed if you simply press enter, is shown in parentheses.} % . This generates a configuration with the name \texttt{\var{config}}, doing its best to automatically determine the default compilers and compilation flags suitable for the current architecture. There are a number of additional command line arguments which may be supplied to override some parts of the procedure. \subsection{Configuration options} There are three ways to pass options to the configuration process. % from the gmake command line. \begin{enumerate} \item[1a]{} Either: create a default configuration file \texttt{\$\{HOME\}/.cactus/config}. All available configuration options may be set in a default options file \texttt{\$\{HOME\}/.cactus/config}, any which are not set will take a default value. The file should contain lines of the form: \texttt{<\var{option}> [=] ...} The equals sign is optional. Spaces are allowed everywhere. Text starting wit a \texttt{'\#'} character will be ignored as a comment. \item[1b]{} Or: list your Cactus configuration files in an environment variable \texttt{CACTUS\_CONFIG\_FILES}: \texttt{gmake <\var{config name}>-config CACTUS\_CONFIG\_FILES=$<$\var{list of config files}$>$} Multiple configuration files, with their file names separated by a \texttt{':'} character, will be processed in order. Each file should be given by its full path. The options file has the same format as \texttt{\$\{HOME\}/.cactus/config}. \item[2]{} Add options to a configuration file and use, \texttt{gmake <\var{config name}>-config options=<\var{filename}>} The options file has the same format as \texttt{\$\{HOME\}/.cactus/config}. (Note these options are \emph{added} to those from the \texttt{\$\{HOME\}/.cactus/config} file.) \item[3]{} Pass the options individually on the command line, \texttt{gmake <\var{config name}>-config <\var{option name}>=<\var{chosen value}>, ...} Not all configuration options can be set on the command line. Those that can be set are indicated in the table below. \end{enumerate} They are listed here in order of increasing precedence, e.g. options set on the command line will take priority over (potentially conflicting) options set in \texttt{\$\{HOME\}/.cactus/config} or other Cactus configuration files. Default options from \texttt{\$\{HOME\}/.cactus/config} will only be read if the environment variable \texttt{CACTUS\_CONFIG\_FILES} is not set. It is important to note that these methods cannot be used to, for example, add options to the default values for \texttt{CFLAGS}. Setting any variable in the configuration file or the command line will overwrite completely the default values. \subsection{Available options} \label{subsec:Compilation-Available_Options} There is a plethora of available options. \begin{itemize} \item {Cross Compiling} If you are compiling on an architecture other than the one you are producing an executable for, you will need to pass the \begin{Lentry} \item [\texttt{HOST\_MACHINE=\var{x-x-x}}] \end{Lentry} option, where \texttt{\var{x-x-x}} is the canonical name of the architecture you are compiling for, such as \texttt{sx6-nec-superux}; the format is \texttt{\var{processor}-\var{vendor}-\var{OS}}. \item {Compiled Thorns} These specify the chosen set of thorns for compilation. If the thorn choice is not provided during configuration, a list containing all thorns in the \texttt{arrangements} directory is automatically created, and the users prompted for any changes. \begin{Lentry} \item [\texttt{THORNLIST}] Name of file containing a list of thorns with the syntax \texttt{<\var{arrangement name}>/<\var{thorn name}>}, lines beginning with \verb|#| or \texttt{!} are ignored. \item [\texttt{THORNLIST\_DIR}] Location of directory containing \texttt{THORNLIST}. This defaults to the current working directory. \end{Lentry} \item {Compiler and tool specification} These are used to specify which compilers and other tools to use. Entries followed by * may be specified on the command line. \begin{Lentry} \item [\texttt{CC}] * The C compiler. \item [\texttt{CXX}] The C++ compiler. \item [\texttt{F90}] * The Fortran 90 compiler. \item [\texttt{F77}] * The Fortran 77 compiler. \item [\texttt{CPP}] The preprocessor used to generate dependencies for and to preprocess C and C++ code. \item [\texttt{FPP}] The preprocessor used to generate dependencies for and to preprocess Fortran code. \item [\texttt{LD}] * The linker. \item [\texttt{AR}] The archiver used for generating libraries. \item [\texttt{RANLIB}] The archive indexer to use. \item [\texttt{MKDIR}] The program to use to create a directory. \item [\texttt{PERL}] The name of the Perl executable. \end{Lentry} \item {Compilation and tool flags} Flags which are passed to the compilers and the tools. \begin{Lentry} \item [\texttt{CFLAGS}] Flags for the C compiler. \item [\texttt{CXXFLAGS}] Flags for the C++ compiler. \item [\texttt{F90FLAGS}] * Flags for the Fortran 90 compiler. \item [\texttt{F77FLAGS}] * Flags for the Fortran 77 compiler. \item [\texttt{CPPFLAGS}] Flags for the preprocessor (used to generate compilation dependencies for and preprocess C and C++ code). \item [\texttt{FPPFLAGS}] Flags for the preprocessor (used to generate compilation dependencies for and preprocess Fortran code). \item [\texttt{MKDIRFLAGS}] Flags for \texttt{MKDIR} so that no error is given if the directory exists. \item [\texttt{LDFLAGS}] * Flags for the linker. \emph{Warning:} This variable is ignored while the compilers and linkers are autodetected. This can lead to strange errors while configuring. You can pass the linker flags in the variable \texttt{LD} instead. \item [\texttt{ARFLAGS}] Flags for the archiver. \item [\texttt{C\_LINE\_DIRECTIVES}] Whether error messages and debug information in the compiled C and C++ files should point to the original source file or to an internal file created by Cactus. The only options available are \texttt{yes} and \texttt{no}, the default is \texttt{yes}. Set this to \texttt{no} if your compiler reports error messages about unrecognised \verb|#| directives. \item [\texttt{F\_LINE\_DIRECTIVES}] Whether error messages and debug information in the compiled Fortran files should point to the original source file or to an internal file created by Cactus. The only options available are \texttt{yes} and \texttt{no}, the default is \texttt{yes}. Set this to \texttt{no} if your compiler reports error messages about unrecognised \verb|#| directives. \item [\texttt{DISABLE\_REAL16}] Disable support for the data type \texttt{CCTK\_REAL16}. The only options available are \texttt{yes} and \texttt{no}, the default is \texttt{no}. Cactus autodetects this data type only for C. If the C compiler supports it, but the Fortran compiler does not, it may be necessary to disable \texttt{CCTK\_REAL16} altogether, since Cactus assumes that data types are fully supported if they exist. \item [\texttt{DEBUG}] * Specifies what type of debug mode should be used, the default is no debugging. Current options are \texttt{yes}, \texttt{no}, or \texttt{memory}. The option \texttt{yes} switches on all debugging features, whereas \texttt{memory} just employs memory tracing (\ref{sec:metr}). \item [\texttt{OPTIMISE, OPTIMIZE}] * Specifies what type of optimisation should be used. The only options currently available are \texttt{yes} and \texttt{no}. The default is to use optimisation.\\ Note that the British spelling \texttt{OPTIMISE} will be checked first and, if set, will override any setting of the American-spelled \texttt{OPTIMIZE}. \item [\texttt{C\_OPTIMISE\_FLAGS}] Optimisation flags for the C compiler, their use depends on the type of optimisation being used. \item [\texttt{CXX\_OPTIMISE\_FLAGS}] Optimisation flags for the C++ compiler, their use depends on the type of optimisation being used. \item [\texttt{F90\_OPTIMISE\_FLAGS}] Optimisation flags for the Fortran 90 compiler, their use depends on the type of optimisation being used. \item [\texttt{F77\_OPTIMISE\_FLAGS}] Optimisation flags for the Fortran 77 compiler, their use depends on the type of optimisation being used. \item [\texttt{C\_WARN\_FLAGS}] Warning flags for the C compiler, their use depends on the type of warnings used during compilation (\ref{sec:gmopfobuco}). \item [\texttt{CXX\_WARN\_FLAGS}] Warning flags for the C++ compiler, their use depends on the type of warnings used during compilation (\ref{sec:gmopfobuco}). \item [\texttt{F90\_WARN\_FLAGS}] Warning flags for the Fortran 90 compiler, their use depends on the type of warnings used during compilation (\ref{sec:gmopfobuco}). \item [\texttt{F77\_WARN\_FLAGS}] Warning flags for the Fortran 77 compiler, their use depends on the type of warnings used during compilation (\ref{sec:gmopfobuco}). \end{Lentry} \item {Architecture-specific flags} \begin{Lentry} \item [\texttt{IRIX\_BITS=32|64}] For Irix SGI systems: whether to build a 32- or 64-bit configuration. \end{Lentry} \begin{Lentry} \item [\texttt{AIX\_BITS=32|64}] For IBM SP systems: whether to build a 32- or 64-bit configuration. \end{Lentry} \item {Library flags} Used to specify auxiliary libraries and directories to find them in. \begin{Lentry} \item [\texttt{LIBS}] Additional libraries. \emph{Warning:} This variable is ignored while the compilers and linkers are autodetected. This can lead to strange errors while configuring. You can pass the additional libraries in the variable \texttt{LD} instead. \item [\texttt{LIBDIRS}] Any other library directories. \end{Lentry} \item {Extra include directories} \begin{Lentry} \item [\texttt{SYS\_INC\_DIRS}] Used to specify any additional directories for system include files. \end{Lentry} \item {Precision options} Used to specify the precision of the default real and integer data types, specified as the number of bytes the data takes up. Note that not all values will be valid on all architectures. \begin{Lentry} \item [\texttt{REAL\_PRECISION}] * Allowed values are \texttt{16, 8, 4}. \item [\texttt{INTEGER\_PRECISION}] * Allowed values are \texttt{8, 4} and \texttt{2}. \end{Lentry} \item {Executable name} \begin{Lentry} \item [\texttt{EXEDIR}] The directory in which to place the executable. \item [\texttt{EXE}] The name of the executable. \end{Lentry} \item{Extra packages} Compiling with extra packages is described fully in Section \ref{subsec:cowiexpa}, which should be consulted for the full range of configuration options. \begin{Lentry} \item [\texttt{MPI}] * The MPI package to use, if required. Supported values are \texttt{CUSTOM}, \texttt{NATIVE}, \texttt{MPICH} or \texttt{LAM}. \item [\texttt{HDF5}] Supported values are \texttt{yes, no}. A blank value is taken as \texttt{no}. \item [\texttt{LAPACK}] Supported values are \texttt{yes, no}. A blank value is taken as \texttt{no}. \item [\texttt{PETSC}] Supported values are \texttt{yes, no}. A blank value is taken as \texttt{no}. \item [\texttt{PTHREADS}] Supported values are \texttt{yes}. \end{Lentry} \item{Miscellaneous} \begin{Lentry} \item [\texttt{PROMPT}] Setting this to \texttt{no} turns off all prompts from the make system. \item [\texttt{SILENT}] Setting this to \texttt{no} instructs \texttt{gmake} to print the commands that it is executing. \end{Lentry} \end{itemize} %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \subsection{Compiling with extra packages} \label{subsec:cowiexpa} \subsubsection{MPI: Message Passing Interface} \label{subsubsec:Compiling-MPI} The \textit{Message Passing Interface} (MPI) provides inter-processor communication. It can either be implemented natively on a machine (this is usual on most supercomputers), or through a standard package such as MPICH, LAM, WMPI, or PACX. To compile with MPI, the configure option is \texttt{MPI = <\var{MPI\_TYPE}>} where \texttt{<\var{MPI\_TYPE}>} can take the values (entries followed by * may be specified on the configuration command line): \begin{Lentry} \item[\texttt{CUSTOM}] For a custom MPI configuration set the variables \begin{Lentry} \item [\texttt{MPI\_LIBS}] * libraries. \item [\texttt{MPI\_LIB\_DIRS}] * library directories. \item [\texttt{MPI\_INC\_DIRS}] * include file directories. \end{Lentry} \item[\texttt{NATIVE}] Use the native MPI for this machine, as indicated in the \texttt{known-architectures} directory (\texttt{lib/make/known-architectures}). \item[\texttt{MPICH}] Use MPICH (\url{http://www-unix.mcs.anl.gov/mpi/mpich}). This is controlled by the options \begin{Lentry} \item [\texttt{MPICH\_ARCH}] * machine architecture. \item [\texttt{MPICH\_DIR} ] * directory in which MPICH is installed. If this option is not defined it will be searched for. \item [\texttt{MPICH\_DEVICE}] * the device used by MPICH. If not defined, the configuration process will search for this in a few defined places. Supported devices are currently \texttt{ch\_p4}, \texttt{ch\_shmem}, \texttt{globus} and \texttt{myrinet}. For versions of MPICH prior to 1.2.0 the devices are searched for in this order, for 1.2.0 you may need to specify \texttt{MPICH\_DEVICE}, depending on the installation. \end{Lentry} If \texttt{MPICH\_DEVICE} is chosen to be \texttt{globus} (\url{http://www.globus.org}), an additional variable must be set \begin{Lentry} \item[\texttt{GLOBUS\_LOCATION}] * directory in which Globus is installed. \end{Lentry} The Globus flavor may be chosen optionally \begin{Lentry} \item[\texttt{GLOBUS\_FLAVOR}] * Globus flavor to build Cactus with \end{Lentry} If it is not set, the first Globus flavor found will be used. If \texttt{MPICH\_DEVICE} is chosen to be \texttt{ch\_gm}, (\url{http://www.myri.com}), an additional variable must be set \begin{Lentry} \item[\texttt{MYRINET\_DIR}] * directory in which Myrinet libraries are installed. \end{Lentry} \item[\texttt{LAM}] Use \texttt{LAM} (\textit{Local Area Multicomputer}, \url{http://www.lam-mpi.org/}). This is controlled by the variables \begin{Lentry} \item[\texttt{LAM\_DIR} ] * directory in which LAM is installed. This will be searched for in a few provided places if not given. \end{Lentry} if the \texttt{LAM} installation splits libraries and include files into different directories, instead of setting \texttt{LAM\_DIR} set the two variables \begin{Lentry} \item[\texttt{LAM\_LIB\_DIR}] * directory in which LAM libraries are installed. \item[\texttt{LAM\_INC\_DIR}] * directory in which LAM include files are installed. \end{Lentry} \item[\texttt{WMPI}] Use WMPI (\textit{Win32 Message Passing Interface}, \url{http://dsg.dei.uc.pt/w32mpi/intro.html}). This is controlled by the variable \begin{Lentry} \item[\texttt{WMPI\_DIR}] * directory in which WMPI is installed. \end{Lentry} \item[\texttt{HPVM}] Use HPVM (\textit{High Performance Virtual Machine}, (\url{http://www-csag.ucsd.edu/projects/hpvm.html}). This is controlled by the variable \begin{Lentry} \item[\texttt{HPVM\_DIR}] * directory in which HPVM is installed. \end{Lentry} \item[\texttt{MPIPro}] Use MPIPro (\url{http://www.mpi-softtech.com/}). \item[\texttt{PACX}] Use the PACX Metacomputing package (\textit{PArallel Computer eXtension},\\ \url{http://www.hlrs.de/structure/organisation/par/projects/pacx-mpi/}). This is controlled by the variables \begin{Lentry} \item[\texttt{PACX\_DIR}] * directory in which PACX is installed. If this option is not defined it will be searched for. \item[\texttt{PACX\_MPI}] * the MPI package PACX uses for node-local communication. This can be any of the above MPI packages. \end{Lentry} \end{Lentry} Note that the searches for libraries etc. mentioned above use the locations given in the files in \texttt{lib/make/extras/MPI}. \subsubsection{HDF5: Hierarchical Data Format version 5} \label{subsec:hdf5} To compile with HDF5 (\url{http://hdf.ncsa.uiuc.edu/whatishdf5.html}), the configure options are \texttt{HDF5 = yes/no [HDF5\_DIR = <\var{dir}>] [LIBZ\_DIR = <\var{dir}>] [LIBSZ\_DIR = <\var{dir}>]} If \texttt{HDF5\_DIR} is not given the configuration process will search for an installed HDF5 package in some standard places (defined in \texttt{lib/make/extras/HDF5}). If the found HDF5 library was built with the external deflate I/O filter, the configuration process also searches for the \texttt{libz} library and adds it to the linker flags. You may also point directly to the location of \texttt{libz.a} by setting \texttt{LIBZ\_DIR}. If the found HDF5 library was built with the external \texttt{szlib} I/O filter, the configuration process also searches for the \texttt{szlib} library and adds it to the linker flags. You may also point directly to the location of \texttt{libsz.a} by setting \texttt{LIBSZ\_DIR}. \subsubsection{LAPACK: Linear Algebra PACKage} To compile with LAPACK (\url{http://www.netlib.org/lapack/}), the configure options are \begin{alltt} LAPACK = yes | no | [ LAPACK\_DIR = <\var{dir}> | none ] [ LAPACK\_EXTRA\_LIBS\_DIRS = <\var{dir}> ] [ LAPACK\_LIBS = <\var{libs}> ] [ LAPACK\_EXTRA\_LIBS = <\var{libs}> ] \end{alltt} If \texttt{LAPACK\_DIR} is not given the configuration process will search for a LAPACK library \texttt{liblapack.[\{a,so\}]} in some standard places (defined in \texttt{lib/make/extras/LAPACK}). If \texttt{LAPACK\_DIR} is set to \texttt{no} the LAPACK library path is assumed to be installed in a standard system location (e.g. \texttt{/usr/lib/}) and thus the library path will not be added to the linker's command line. Because LAPACK doesn't come as a standardized system installation, there are additional configuration variables to set the name of the lapack library (\texttt{LAPACK\_LIBS}) as well as the name (\texttt{LAPACK\_EXTRA\_LIBS}) and location (\texttt{LAPACK\_EXTRA\_LIBS\_DIRS}) of extra libraries that are required by LAPACK itself. \subsubsection{PETSc: Portable, Extensible Toolkit for Scientific Computation} To compile with PETSc (\url{http://www-unix.mcs.anl.gov/petsc/petsc-2/index.html}), the configure options are \begin{alltt} PETSC = yes | no | [ PETSC\_DIR = <\var{dir}> ] [ PETSC\_ARCH = <\var{architecture}> ] [ PETSC\_ARCH\_LIBS = <\var{architecture-specific libraries}> ] \end{alltt} If \texttt{PETSC\_DIR} is not given the configuration process will search for an installed PETSc package in some standard places (defined in \texttt{lib/make/extras/PETSC}). If \texttt{PETSC\_ARCH} is not given the configuration process will choose the first PETSc configuration found in \texttt{\$PETSC\_DIR/lib/libO/}. If \texttt{PETSC\_ARCH\_LIBS} is not given the configuration process will choose architecture-specific libraries as required by a PETSc configuration (usually PETSc needs the LAPACK library). \subsubsection{Pthreads: POSIX threads} To enable multithreading support within Cactus using POSIX threads the configure option is \texttt{PTHREADS = yes} The configuration process will check if a reentrant C library is available and adds it to the linker flags. It will also search for the system's Pthreads library (either \texttt{libpthread} or \texttt{libpthreads}) and set preprocessor defines necessary for compiling multithreaded code. \subsection{File layout} The configuration process sets up various subdirectories and files in the \texttt{configs} directory to contain the configuration specific files, these are placed in a directory with the name of the configuration. \begin{Lentry} \item [\texttt{config-data}] contains the files created by the configure script: The most important ones are \begin{Lentry} \item [\texttt{make.config.defn}] contains compilers and compilation flags for a configuration. \item [\texttt{make.extra.defn}] contains details about extra packages used in the configuration. \item [\texttt{cctk\_Config.h}] The main configuration header file, containing architecture specific definitions. \item [\texttt{cctk\_Archdefs.h}] An architecture specific header file containing things which cannot be automatically detected, and have thus been hand-coded for this architecture. \end{Lentry} These are the first files which should be checked or modified to suit any peculiarities of this configuration. In addition the following files may be informative: \begin{Lentry} \item [\texttt{fortran\_name.pl}] A Perl script used to determine how the Fortran compiler names subroutines. This is used to make some C routines callable from Fortran, and Fortran routines callable from C. \item [\texttt{make.config.deps}] Initially empty. Can be edited to add extra architecture specific dependencies needed to generate the executable. \item [\texttt{make.config.rule}] The \texttt{make} rules for generating object files from source files. \end{Lentry} Finally, \texttt{autoconf} generates the following files. \begin{Lentry} \item [\texttt{config.log}] A log of the \texttt{autoconf} process. \item [\texttt{config.status}] A script which may be used to regenerate the configuration. \item [\texttt{config.cache}] An internal file used by \texttt{autoconf}. \end{Lentry} \item [\texttt{lib}] An empty directory which will contain the libraries created for each thorn. \item [\texttt{build}] An empty directory which will contain the object files generated for this configuration, and preprocessed source files. \item [\texttt{config-info}] A file containing information about the configuration (including the options used to configure the configuration). \item [\texttt{bindings}] A directory which contains all the files generated by the CST from the \texttt{.ccl} files. \item [\texttt{scratch}] A scratch directory which is used to accommodate Fortran 90 modules. \end{Lentry} \section{Building and Administering a configuration} \label{sec:buanadaco} Once you have created a new configuration, the command \\ \\ \texttt{gmake <\var{configuration name}>} \\ \\ will build an executable, prompting you along the way for the thorns which should be included. There is a range of \texttt{gmake} targets and options which are detailed in the following sections. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \subsection{gmake targets for building and administering configurations} \label{sec:gmtafobuanadco} A target for \texttt{gmake} can be naively thought of as an argument that tells it which of several things listed in the \texttt{Makefile} it is to do. The command \texttt{gmake help} lists all \texttt{gmake} targets: % colon clarifies that all (config) targets are listed here \begin{Lentry} \item [\texttt{gmake <\var{config}>}] builds a configuration. If the configuration doesn't exist it will create it. \item [\texttt{gmake <\var{config}>-clean}] removes all object and dependency files from a configuration. \item [\texttt{gmake <\var{config}>-cleandeps}] removes all dependency files from a configuration. \item [\texttt{gmake <\var{config}>-cleanobjs}] removes all object files from a configuration. \item [\texttt{gmake <\var{config}>-config}] creates a new configuration or reconfigures an existing one overwriting any previous configuration options.\\ The configuration options are stored in a file \texttt{configs/<\var{config}>/config-info}. \item [\texttt{gmake <\var{config}>-configinfo}] displays the options of the configuration (\texttt{cat configs/<\var{config}>/config-info}). \item[\texttt{gmake <\var{config}>-cvsupdate}] updates the Flesh and this configuration's thorns from the CVS repositories. \item [\texttt{gmake <\var{config}>-delete}] deletes a configuration (\texttt{rm -r configs/<\var{config}>}). \item [\texttt{gmake -editthorns}] edits the ThornList. \item [\texttt{gmake <\var{config}>-examples}] copies all the example parameter files relevant for this configuration to the directory \texttt{examples} in the Cactus home directory. If a file of the same name is already there, it will not overwrite it. \item [\texttt{gmake <\var{config}>-realclean}] removes from a configuration all object and dependency files, as well as files generated from the CST (stands for \textit{Cactus Specification Tool}, which is the set of Perl scripts which parse the thorn configuration files). Only the files generated by configure and the \texttt{ThornList} file remain. \item [\texttt{gmake <\var{config}>-rebuild}] rebuilds a configuration (reruns the CST). \item [\texttt{gmake <\var{config}>-reconfig}] reconfigures an existing configuration using its previous configuration options from the file \texttt{configs/<\var{config}>/config-info}. \item [\texttt{gmake <\var{config}>-testsuite}] runs the test programs associated with each thorn in the configuration. See section \ref{sec:testing} for information about the test suite mechanism. \item[\texttt{gmake <\var{config}>-ThornGuide}] builds documentation for the thorns in this configuration (see section \ref{sec:OtherGmakeTargetsDoc} on page \pageref{sec:OtherGmakeTargetsDoc} for other targets to build documentation for thorns). \item [\texttt{gmake <\var{config}>-thornlist}] regenerates the \texttt{ThornList} for a configuration. \item [\texttt{gmake <\var{config}>-utils [UTILS$=$<\var{list}>]}] builds all utility programs provided by the thorns of a configuration. Individual utilities can be selected by giving their names in the \texttt{UTILS} variable. \end{Lentry} \subsection{Compiling in thorns} \label{sec:cointh} Cactus will try to compile all thorns listed in \texttt{configs/<\var{config}>/ThornList}. The \texttt{ThornList} file is simply a list of the form \texttt{<\var{arrangement}>/<\var{thorn}>}. All text after a pound sign `\texttt{\#}' or exclamation mark `\texttt{!}' on a line is treated as a comment and ignored. The first time that you compile a configuration, if you did not specify a ThornList already during configuration, you will be shown a list of all the thorns in your arrangement directory, and asked if you with to edit them. You can regenerate this list at anytime by typing \begin{alltt} gmake <\var{config}>-thornlist \end{alltt} or you can edit it using \begin{alltt} gmake <\var{config}>-editthorns \end{alltt} Instead of using the editor to specify the thorns you want to have compiled, you can \emph{edit} the \texttt{ThornList} outside the make process. It is located in \texttt{configs/<\var{config}>/ThornList}, where \texttt{<\var{config}>} refers to the name of your configuration. The directory, \texttt{./configs}, exists \emph{ after} the very first make phase for the first configuration. \subsection{Notes and Caveats} \begin{itemize} \item{} If during the build you see the error ``\texttt{missing separator}'' you are probably not using GNU make. \item{} \textit{The EDITOR environment variable}. You may not be aware of this, but this thing very often exists and may be set by default to something scary like \texttt{vi}. If you don't know how to use \texttt{vi} or wish to use your favorite editor instead, reset this environment variable. (To exit \texttt{vi} type \texttt{ :q!}) \end{itemize} \subsection{\texttt{gmake} options for building configurations} \label{sec:gmopfobuco} An \textit{option} for \texttt{gmake} can be thought of as an argument which tells it how it should make a \textit{target}. Note that the final result is always the same. \begin{Lentry} \item [\texttt{gmake <\var{target}> PROMPT=no}] turns off all prompts from the make system. \item [\texttt{gmake <\var{target}> SILENT=no}] print the commands that gmake is executing. \item [\texttt{gmake <\var{target}> WARN=yes}] show compiler warnings during compilation. \item [\texttt{gmake <\var{target}> FJOBS=<\var{number}>}] compile in parallel, across files within each thorn. \item [\texttt{gmake <\var{target}> TJOBS=<\var{number}>}] compile in parallel, across thorns. \end{Lentry} Note that with more modern versions of gmake, it is sufficient to pass the normal \texttt{-j <\var{number}>} flag to gmake to get parallel compilation. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \section{Other gmake targets} \begin{Lentry} \item [\texttt{gmake help}] lists all make options. \item [\texttt{gmake checkout}] allows you to easily checkout Cactus arrangements and thorns. For example it can checkout all the thorns in any thornlist file found in the \texttt{thornlists} subdirectory of the Cactus root directory. % (usually \texttt{Cactus}). \item [\texttt{gmake cvsdiff}] shows differences between checked out version of Cactus and that in the CVS repositories. \item [\texttt{gmake cvsstatus}] shows status of checked out version of Cactus, reporting which files have been modified or need updating. \item [\texttt{gmake cvsupdate}] updates Flesh and all thorns from CVS repositories. \item [\texttt{gmake configinfo}] prints configuration options for every configuration found in user's \texttt{configs} subdirectory. \item [\texttt{gmake default}] creates a new configuration with a default name. \item [\texttt{gmake distclean}] deletes your \texttt{configs} directory and hence all your configurations. \item [\texttt{gmake downsize}] removes non-essential files as documents and test suites to allow for minimal installation size. \item [\texttt{gmake newthorn}] creates a new thorn, prompting for the necessary information and creating template files. \item [\texttt{gmake TAGS}] creates an Emacs style TAGS file. See section \ref{sec:Appendix.tags} for using tags within Cactus. \item [\texttt{gmake tags}] creates a \texttt{vi} style tags file. See section \ref{sec:Appendix.tags} for using tags within Cactus. \end{Lentry} {\bf Targets to generate Cactus documentation:} \label{sec:OtherGmakeTargetsDoc} \begin{Lentry} \item[\texttt{gmake <\var{arrangement}>-ArrangementDoc}] builds the documentation for the arrangement. \item[\texttt{gmake ArrangementDoc}] builds the documentation for all arrangements. \item [\texttt{gmake MaintGuide}] runs LaTeX to produce a copy of the Maintainers' Guide. \item [\texttt{gmake ReferenceManual}] runs LaTeX to produce a copy of the Reference Manual. \item[\texttt{gmake <\var{thorn}>-ThornDoc}] builds the documentation for the thorn. \item[\texttt{gmake ThornDoc}] builds the documentation for all thorns. \item [\texttt{gmake ThornGuide}] runs LaTeX to produce a copy of the Thorn Guide, for all the thorns in the arrangements directory. \item [\texttt{gmake UsersGuide}] runs LaTeX to produce a copy of the Users' Guide. \item [\texttt{gmake AllDoc}] creates all of the above documentations. \end{Lentry} \section{Testing} \label{sec:testing} Some thorns come with a test suite, consisting of example parameter files and the output files generated by running these. To run the test suite for the thorns you have compiled use \texttt{gmake <\var{configuration}>-testsuite} These test suite serve the dual purpose of \begin{Lentry} \item [Regression testing] i.e. making sure that changes to the thorn or the Flesh don't affect the output from a known parameter file. \item [Portability testing] i.e. checking that the results are independent of the architecture --- this is also of use when trying to get Cactus to work on a new architecture. \end{Lentry} %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \chapter{Running Cactus} Cactus executables always run from a parameter file (which may be a specified as a command line argument taken from standard input), which specifies which thorns to use and sets the values of any parameters which are different from the default values. There is no restriction on the name of the parameter file, although it is conventional to use the file extension \texttt{.par}. Optional command line arguments can be used to customise runtime behaviour, and to provide information about the thorns used in the executable. The general syntax for running Cactus from a parameter file is then \texttt{./cactus\_<\var{config}> <\var{parameter file}> [\var{command line options}]} or if the parameter file should be taken from standard input by appending a dash (``\texttt{-}'') on the command line like this: \texttt{./cactus\_<\var{config}> [\var{command line options}] -} The remainder of this chapter covers all aspects for running your Cactus executable. These include: command line options, parameter file syntax, understanding screen output, environment variables, and creating thorn documentation. \section{Command Line Options} \label{sec:command_line_options} The Cactus executable accepts these command line arguments: \texttt{ \begin{tabular}{|l|l|} \hline Short Version & Long Version \\ \hline -O[v] & -describe-all-parameters \\ \hline -o<\var{param}> & -describe-parameter=<\var{param}> \\ \hline -S & -print-schedule\\ \hline -T & -list-thorns\\ \hline -t<\var{arrangement/thorn}>& -test-thorn-compiled=<\var{arrangement/thorn}>\\ \hline -h,-? & -help\\ \hline -v & -version \\ \hline % -x [] & -test-parameters [] \\ %\hline -L<\var{level}> & -logging-level=<\var{level}> \\ \hline -W<\var{level}> & -warning-level=<\var{level}> \\ \hline -E<\var{level}> & -error-level=<\var{level}> \\ \hline -r[o|e|oe|eo] & -redirect=[o|e|oe|eo]\\ \hline -i & -ignore-next \\ \hline & -parameter-level=<\var{level}> \\ \hline \end{tabular} } \begin{Lentry} \item [\texttt{-O} or \texttt{-describe-all-parameters}] Prints a full list of all parameters from all thorns which were compiled, along with descriptions and allowed values. This can take an optional extra parameter \texttt{v} (i.e. \texttt{-Ov} to give verbose information about all parameters). \item [\texttt{-o<\var{param}>} or \texttt{-describe-parameter=<\var{param}>}] Prints the description and allowed values for a given parameter --- takes one argument. \item [\texttt{-S} or \texttt{-print-schedule}] Print only the schedule tree. \item [\texttt{-T} or \texttt{-list-thorns}] Prints a list of all the thorns which were compiled in. \item [\texttt{-t<\var{arrangement or thorn}>} or \texttt{-test-thorn-compiled=<\var{arrangement or thorn>}} ] Checks if a given thorn was compiled in --- takes one argument. \item [\texttt{-h}, \texttt{-?} or \texttt{-help}] Prints a help message. \item [\texttt{-v} or \texttt{-version}] Prints version information of the code. %\item [\texttt{-x } or \texttt{-test-parameters }] %Runs the code far enough to check the consistency of the parameters. If %given a numeric argument it will attempt to simulate being on that number %of processors. [To be implemented.] \item [\texttt{-L<\var{level}>} or \texttt{-logging-level=<\var{level}>}] Sets the logging level of the code. All warning messages are given a level --- the lower the level the greater the severity. This parameter \texttt{-L} controls the level of messages to be seen, with all warnings of level $\le$ \texttt{<\var{level}>} printed to standard output. The default is a logging level of~0, meaning that only level~0 messages should be printed to standard output. \item [\texttt{-W<\var{level}>} or \texttt{-warning-level=<\var{level}>}] Similar to \texttt{-W}, but for standard error instead of standard output. All warnings of level $\le$ \texttt{<\var{level}>} are printed to standard error. The default is a warning level of~1, meaning that level~0 and level~1 messages should be printed to standard error. \item [\texttt{-E<\var{level}>} or \texttt{-error-level=<\var{level}>}] Similar to \texttt{-W}, but for fatal errors: Cactus treats all warnings with level $\le$ \texttt{<\var{level}>} as fatal errors, and aborts the Cactus run immediately (after printing the warning message%%% \footnote{%%% Cactus imposes the constraints that $\hbox{the \texttt{-W} level} \ge \hbox{the \texttt{-E} level} \ge 0$, so any fatal-error message will always be printed (first). }%%% ). The default value is zero, \ie{} only level~0 warnings will abort the Cactus run. \item [\texttt{-r[o|e|oe|eo]} or \texttt{-redirect=[o|e|oe|eo]}] Redirects the standard output (`\texttt{o}') and/or standard error (`\texttt{e}') of each processor to a file. By default the standard outputs from processors other than processor 0 are discarded. \item [\texttt{-i} or \texttt{-ignore-next}] Causes the next argument on the command line to be ignored. \item [\texttt{-parameter-level=<\var{level}>}] Sets the level of parameter checking to be used, one of \texttt{strict}, \texttt{normal} (the default), or \texttt{relaxed}. See Section~\ref{sec:Parameter_File}. \end{Lentry} %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \section{Parameter File Syntax} \label{sec:Parameter_File} A \textit{parameter file} ( or \textit{par file}) is used to control the behaviour of a Cactus executable. It specifies initial values for parameters as defined in the various thorns' \texttt{param.ccl} files (see Chapter~\ref{chap:Cactus_parameters}). The name of a parameter file is often given the suffix \texttt{.par}, but this is not mandatory. A parameter file is a text file whose lines are either comments or parameter statements. Comments are blank lines or lines that begin with either `\texttt{\#}' or `\texttt{!}'. A parameter statement consists of one or more parameter names, followed by an `\texttt{=}', followed by the value(s) for this (these) parameter(s). Note that all string parameters are case insensitive. The first parameter statement in any parameter file should set \texttt{ActiveThorns}, which is a special parameter that tells the program which \textit{thorns} are to be activated. Only parameters from active thorns can be set (and only those routines \textit{scheduled} by active thorns are run). By default all thorns are inactive. For example, the first entry in a parameter file which is using just the two thorns \texttt{CactusPUGH/PUGH} and \texttt{CactusBase/CartGrid3D} should be \texttt{ActiveThorns = "PUGH CartGrid3D"} Parameters following the \texttt{ActiveThorns} parameter all have names whose syntax depends on the scope (see Section~\ref{sec:Cactus_parameters.scope}) of the parameter: \begin{Lentry} \item [\texttt{Global parameters}] Just the name of the parameter itself. Global parameters are to be avoided; there are none in the Flesh and Cactus Toolkits. \item [\texttt{Restricted parameters}] The name of the \textit{implementation} which defined the parameter, followed by two colons, then the name of the parameter --- e.g. \texttt{driver::global\_nx}. \item [\texttt{Private parameters}] The name of the \textit{thorn} which defined the parameter, two colons, and the name of the parameter --- e.g. \texttt{wavetoyF77::amplitude}. \end{Lentry} This notation is not currently strictly enforced in the code. It is sufficient to specify the first part of the parameter name using either the implementation name, or the thorn name. However, we recommend that the above convention be followed. The Cactus Flesh performs checks for consistency and range of parameters. The severity of these checks is controlled by the command line argument \texttt{-parameter-level} which can take the following values \begin{Lentry} \item[\texttt{relaxed}] Cactus will issue a level 0 warning (that is the default behaviour will be to terminate) if \begin{itemize} \item{} The specified parameter value is outside of the allowed range. \end{itemize} \item [\texttt{normal}] This is the default, and provides the same warnings as the \texttt{relaxed} level, with in addition a level 0 warning issued for \begin{itemize} \item{} An implementation and/or thorn \texttt{foo} is active, but the parameter \texttt{foo::bar} was not defined. \item{} The parameter \texttt{foo::bar} was successfully set for both an active implementation \texttt{foo} not implemented by a thorn \texttt{foo}, and to a thorn \texttt{foo}. \end{itemize} \item [\texttt{strict}] This provides the same warnings as the \texttt{normal} level, with in addition a level 0 warning issued for \begin{itemize} \item{} The parameter \texttt{foo::bar} is specified in the parameter file, but no implementation or thorn with the name \texttt{bar} is active. \end{itemize} \end{Lentry} Notes: \begin{itemize} \item{} You can obtain lists of the parameters associated with each thorn using the command line options \texttt{-o} and \texttt{-O} (Section~\ref{sec:command_line_options}). \item{} The parameter file is read \emph{sequentially} from top to bottom, this means that if you set the value of a parameter twice in the parameter file, the second value will be used. (This is why the \texttt{ActiveThorns} parameter is always first in the file). \item{} Some parameters are \textit{steerable} and can be changed during the execution of a Cactus program using parameter steering interfaces For example, thorn \texttt{CactusConnect/HTTPD}, or using a parameter file when recovering from a checkpoint file. \item{} For examples of parameter files, look in the \texttt{par} directory which can be found in most thorns. \end{itemize} \section{Thorn Documentation} \label{sec:thdo} The Cactus make system provides a mechanism for generating a \textit{Thorn Guide} containing separate chapters for each thorn and arrangement in your configuration. The documentation provided for an individual thorn obviously depends on what the thorn authors added, but the Thorn Guide is a good place to first look for special instructions on how to run and interpret the output from a Thorn. Details about parameters, grid variables and scheduling are automatically included in from a thorns CCL files into the Thorn Guide. To construct a Thorn Guide for the configuration \texttt{$<$\var{config}$>$} use \texttt{gmake $<$\var{config}$>$-ThornGuide} or to make a Thorn Guide for all the thorns in the \texttt{arrangements} directory \texttt{gmake $<$\var{config}$>$}. See Section~\ref{sec:Adding_documentation} for a guide to adding documentation to your own thorns. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \chapter{Getting and looking at output} \section{Screen output} As your Cactus executable runs, standard output and standard error are usually written to the screen. Standard output provides you with information about the run, and standard error reports warnings and errors from the Flesh and thorns. As the program runs, the normal output provides the following information: \begin{Lentry} \item [Active thorns] A report is made as each of the thorns in the \texttt{ActiveThorns} parameters from the parameter file (see Section~\ref{sec:Parameter_File}) is attempted to be activated. This report shows whether the thorn activation was successful, and if successful gives the thorn's implementation. For example \begin{verbatim} Activating thorn idscalarwave...Success -> active implementation idscalarwave \end{verbatim} \item [Failed parameters] If any of the parameters in the parameter file does not belong to any of the active thorns, or if the parameter value is not in the allowed range (see Section~\ref{sec:Parameters.Types_and_Ranges}), an error is registered. For example, if the parameter is not recognised \begin{verbatim} Unknown parameter time::ddtfac \end{verbatim} or if the parameter value is not in the allowed range \begin{verbatim} Unable to set keyword CartGrid3D::type - ByMouth not in any active range \end{verbatim} \item [Scheduling information] The scheduled routines (see Section~\ref{chap:scheduling}), are listed, in the order that they will be executed. For example \begin{verbatim} ---------------------------------------------------------------------- Startup routines Cactus: Register banner for Cactus CartGrid3D: Register GH Extension for GridSymmetry CartGrid3D: Register coordinates for the Cartesian grid IOASCII: Startup routine IOBasic: Startup routine IOUtil: IOUtil startup routine PUGH: Startup routine WaveToyC: Register banner Parameter checking routines CartGrid3D: Check coordinates for CartGrid3D IDScalarWave: Check parameters Initialisation CartGrid3D: Set up spatial 3D Cartesian coordinates on the GH PUGH: Report on PUGH set up Time: Set timestep based on speed one Courant condition WaveToyC: Schedule symmetries IDScalarWave: Initial data for 3D wave equation do loop over timesteps WaveToyC: Evolution of 3D wave equation t = t+dt if (analysis) endif enddo ---------------------------------------------------------------------- \end{verbatim} \item [Thorn banners] Usually a thorn registers a short piece of text as a \emph{banner}. This banner of each thorn is displayed in the standard output when the thorn is initialised. \end{Lentry} \section{Output} Output methods in Cactus are all provided by thorns. Any number of output methods can be used for each run. The behaviour of the output thorns in the standard arrangements are described in those thorns' documentation. In general, output thorns decide what to output by parsing a string parameter containing the names of those grid variables, or groups of variables, for which output is required. The names should be fully qualified with the implementation and group or variable names. There is usually a parameter for each method to denote how often, in evolution iterations, this output should be performed. There is also usually a parameter to define the directory in which the output should be placed, defaulting to the directory from which the executable is run. See Chapter~\ref{chap:io_methods} for details on creating your own IO method. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \section{Checkpointing/Recovery} \label{sec:checkpointing} Checkpointing is defined as saving the current state of a run (parameter settings, contents of grid variables, and other relevant information) to a file. At a later time, this run can then be restarted from that state by recovering all the data from the checkpoint file. Cactus checkpointing and recovery methods are provided by thorns. In general, these thorns decide how often to generate a checkpoint. They also register their recovery routines with the Flesh; these recovery routines may then be called during initialisation of a subsequent run to perform the recovery of the state of the run. Such a recovery is requested by setting a parameter in the parameter file. See Chapter~\ref{chap:cp_recovery_methods} for details of how to create your own checkpointing and recovery methods. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \end{cactuspart}