From e7650cc95121d3cda73dbf6f31965dd585230ef7 Mon Sep 17 00:00:00 2001 From: bentivegna Date: Tue, 10 Feb 2009 00:49:28 +0000 Subject: General restructuring. git-svn-id: http://svn.cactuscode.org/flesh/trunk@4548 17b73243-c579-4c4c-a9d2-2d5706c11dac --- doc/UsersGuide/Appendices.tex | 415 +++ doc/UsersGuide/ApplicationThorns.tex | 4621 ++++++++++++++++++++++++++++++ doc/UsersGuide/GettingStarted.tex | 383 +++ doc/UsersGuide/Infrastructure.tex | 677 ----- doc/UsersGuide/InfrastructureThorns.tex | 676 +++++ doc/UsersGuide/Notes.tex | 1500 ++++++++++ doc/UsersGuide/Preface.tex | 33 +- doc/UsersGuide/RunningCactus.tex | 1694 ----------- doc/UsersGuide/ThornWriters.tex | 4622 ------------------------------- doc/UsersGuide/UsersGuide.tex | 1 - doc/UsersGuide/UtilityRoutines.tex | 431 --- 11 files changed, 7611 insertions(+), 7442 deletions(-) create mode 100644 doc/UsersGuide/ApplicationThorns.tex create mode 100644 doc/UsersGuide/GettingStarted.tex delete mode 100644 doc/UsersGuide/Infrastructure.tex create mode 100644 doc/UsersGuide/InfrastructureThorns.tex create mode 100644 doc/UsersGuide/Notes.tex delete mode 100644 doc/UsersGuide/RunningCactus.tex delete mode 100644 doc/UsersGuide/ThornWriters.tex delete mode 100644 doc/UsersGuide/UtilityRoutines.tex (limited to 'doc') diff --git a/doc/UsersGuide/Appendices.tex b/doc/UsersGuide/Appendices.tex index 2db45a31..a7904e1a 100644 --- a/doc/UsersGuide/Appendices.tex +++ b/doc/UsersGuide/Appendices.tex @@ -1174,6 +1174,421 @@ No other lines should be output by the script. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +\chapter{Utility Routines} + +\section{Introduction} + +As well as the high-level \verb|CCTK_|* routines, Cactus also +provides a set of lower-level \verb|Util_|* utility routines, which +are mostly independent of the rest of Cactus. This chapter gives a +general overview of programming with these utility routines. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\section{Key/Value Tables} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subsection{Motivation} + +Cactus functions may need to pass information through a generic +interface. In the past, we have used various ad hoc means to do this, +and we often had trouble passing "extra" information that wasn't +anticipated in the original design. For example, for periodic output +of grid variables, +\verb|CCTK_OutputVarAsByMethod()| requires that +any parameters (such as hyperslabbing parameters) be appended as an option +string to the variable's character string name. Similarly, elliptic +solvers often need to pass various parameters, but we haven't had a +good way to do this. + +Key/value tables (\textit{tables} for short) provide a clean solution +to these problems. They're implemented by the \verb|Util_Table|* +functions (described in detail in the Reference Manual). + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subsection{The Basic Idea} + +Basically, a table is an object which maps strings to almost arbitrary +user-defined data. (If you know Perl, a table is very much like a +Perl hash table. Alternatively, if you know Unix shells, a table is +like the set of all environment variables. As yet another analogy, +if you know Awk, a table is like an Awk associative array.)%%% +\footnote{%%% + However, the present Cactus tables implementation + is optimized for a relatively small number of + distinct keys in any one table. It will still + work OK for huge numbers of keys, but it will be + slow. + }%%% + +More formally, a table is an object which stores a set of \textit{keys} +and a corresponding set of \textit{values}. We refer to a (key,value) +pair as a table \textit{entry}. + +Keys are C-style null-terminated character strings, with the slash +character `{\tt /}' reserved for future expansion.%%% +\footnote{%%% + Think of hierarchical tables for storing + tree-like data structures.%%% + }%%% + +Values are 1-dimensional arrays of any of the usual Cactus data types +described in Section~\ref{sect-ThornWriting/DataTypes}. +A string can be stored by treating it as a 1-dimensional array of +\verb|CCTK_CHAR| (there's an example of this below). + +The basic ``life cycle'' of a table looks like this: +\begin{enumerate} +\item Some code creates it with \verb|Util_TableCreate()| + or \verb|Util_TableClone()|. +\item Some code (often the same piece of code, but maybe some + other piece) sets entries in it using one or more of + the \verb|Util_TableSet*()|, \verb|Util_TableSet*Array()|, + \verb|Util_TableSetGeneric()|, \verb|Util_TableSetGenericArray()|, + and/or \verb|Util_TableSetString()| functions. +\item Some other piece or pieces of code can get (copies of) + the values which were set, using one or more of the + \verb|Util_TableGet*()|, \verb|Util_TableGet*Array()|, + \verb|Util_TableGetGeneric()|, \verb|Util_TableGetGenericArray()|, + and/or \verb|Util_TableGetString()| functions. +\item When everyone is through with a table, some (single) + piece of code should destroy it with \verb|Util_TableDestroy()|. +\end{enumerate} + +There are also convenience functions \verb|Util_TableSetFromString()| +to set entries in a table based on a parameter-file-style string, +and \verb|Util_TableCreateFromString()| to create a table and then +set entries in it based on a parameter-file-style string. + +As well, there are ``table iterator'' functions \verb|Util_TableIt*()| +to allow manipulation of a table even if you don't know its keys. + +A table has an integer ``flags word'' which may be used to specify +various options, via bit flags defined in \verb|util_Table.h|. +For example, the flags word can be used to control whether keys +should be compared as case sensitive or case insensitive strings. +See the detailed function description of \verb|Util_TableCreate()| +in the Reference Manual for a list +of the possible bit flags and their semantics. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subsection{A Simple Example} +\label{Tables_Simple_Example} + +Here's a simple example (in C)%%% +\footnote{%%% + All (or almost all) of the table routines + are also usable from Fortran. See the full + descriptions in the Reference Manual + for details. + }%%% +{} of how to use a table: +\begin{verbatim} +#include "util_Table.h" +#include "cctk.h" + +/* create a table and set some entries in it */ +int handle = Util_TableCreate(UTIL_TABLE_FLAGS_DEFAULT); +Util_TableSetInt(handle, 2, "two"); +Util_TableSetReal(handle, 3.14, "pi"); + +... + +/* get the values from the table */ +CCTK_INT two_value; +CCTK_REAL pi_value; +Util_TableGetInt(handle, &two_value, "two"); /* sets two_value = 2 */ +Util_TableGetReal(handle, &pi_value, "pi"); /* sets pi_value = 3.14 */ +\end{verbatim} + +Actually, you shouldn't write code like this---in the real world +errors sometimes happen, and it's much better to catch them close to +their point of occurrence, rather than silently produce garbage results +or crash your program. So, the \emph{right} thing to do is to always +check for errors. To allow this, all the table routines return a status, +which is zero or positive for a successful return, but negative if +and only if some sort of error has occurred.%%% +\footnote{%%% + Often (as in the examples here) you don't care + about the details of which error occurred. But if + you do, there are various error codes defined in + {\t util\_Table.h} and {\t util\_ErrorCodes.h}; + the detailed function descriptions in + the Reference Manual + say which error codes each function can return. + }%%% +{} So, the above example should be rewritten like this: + +\begin{verbatim} +#include "util_Table.h" + +/* create a table and set some entries in it */ +int handle = Util_TableCreate(UTIL_TABLE_FLAGS_DEFAULT); +if (handle < 0) + CCTK_WARN(CCTK_WARN_ABORT, "couldn't create table!"); + +/* try to set some table entries */ +if (Util_TableSetInt(handle, 2, "two") < 0) + CCTK_WARN(CCTK_WARN_ABORT, "couldn't set integer value in table!"); +if (Util_TableSetReal(handle, 3.14, "pi") < 0) + CCTK_WARN(CCTK_WARN_ABORT, "couldn't set real value in table!"); + +... + +/* try to get the values from the table */ +CCTK_INT two_value; +CCTK_REAL pi_value; +if (Util_TableGetInt(handle, &two_value, "two") < 0) + CCTK_WARN(CCTK_WARN_ABORT, "couldn't get integer value from table!"); +if (Util_TableGetReal(handle, &pi_value, "pi") < 0) + CCTK_WARN(CCTK_WARN_ABORT, "couldn't get integer value from table!"); + +/* if we get to here, then two_value = 2 and pi_value = 3.14 */ +\end{verbatim} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subsection{Arrays as Table Values} + +As well as a single numbers (or characters or pointers), tables can +also store 1-dimensional arrays of numbers (or characters or pointers).%%% +\footnote{%%% + Note that the table makes (stores) a \emph{copy} of the array + you pass in, so it's somewhat inefficient to store a large array + (e.g.~a grid function) this way. If this is a problem, consider + storing a \texttt{CCTK\_POINTER} (pointing to the array) in the table + instead. (Of course, this requires that you ensure that the array still exists whenever that \texttt{CCTK\_POINTER} is used.) + }%%% + +For example (continuing the previous example): +\begin{verbatim} +static const CCTK_INT a[3] = { 42, 69, 105 }; +if (Util_TableSetIntArray(handle, 3, a, "my array") < 0) + CCTK_WARN(CCTK_WARN_ABORT, "couldn't set integer array value in table!"); + +... + +CCTK_INT blah[10]; +int count = Util_TableGetIntArray(handle, 10, blah, "my array"); +if (count < 0) + CCTK_WARN(CCTK_WARN_ABORT, "couldn't get integer array value from table!"); +/* now count = 3, blah[0] = 42, blah[1] = 69, blah[2] = 105, */ +/* and all remaining elements of blah[] are unchanged */ +\end{verbatim} +As you can see, a table entry remembers the length of any array +value that has been stored in it.%%% +\footnote{%%% + In fact, actually \emph{all} table values are + arrays---setting or getting a single value is + just the special case where the array length is 1. + }%%% +{} + +If you only want the first few values of a larger array, just pass +in the appropriate length of your array, +that's OK: +\begin{verbatim} +CCTK_INT blah2[2]; +int count = Util_TableGetIntArray(handle, 2, blah2, "my array"); +if (count < 0) + CCTK_WARN(CCTK_WARN_ABORT, "couldn't get integer array value from table!"); +/* now count = 3, blah2[0] = 42, blah2[1] = 69 */ +\end{verbatim} +You can even ask for just the first value: +\begin{verbatim} +CCTK_INT blah1; +int count = Util_TableGetInt(handle, &blah1, "my array"); +if (count < 0) + CCTK_WARN(CCTK_WARN_ABORT, "couldn't get integer array value from table!"); +/* now count = 3, blah1 = 42 */ +\end{verbatim} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subsection{Character Strings} + +One very common thing you might want to store in a table is a +character string. While you could do this by explicitly storing +an array of \verb|CCTK_CHAR|, there are also routines +specially for conveniently setting and getting strings: +\begin{verbatim} +if (Util_TableSetString(handle, "black holes are fun", "bh") < 0) + CCTK_WARN(CCTK_WARN_ABORT, "couldn't set string value in table!"); + +... +char buffer[50]; +if (Util_TableGetString(handle, 50, buffer, "bh") < 0) + CCTK_WARN(CCTK_WARN_ABORT, "couldn't get string value from table!"); + +/* now buffer[] contains the string "black holes are fun" */ +\end{verbatim} + +\verb|Util_TableGetString()| guarantees that the string is +terminated by a null character (`\verb|\0|'), and also returns an +error if the string is too long for the buffer. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subsection{Convenience Routines} + +There are also convenience routines for the common case of setting +values in a table based on a string. + +For example, the following code sets up exactly the same table as the +example in Section \ref{Tables_Simple_Example}: + +\begin{verbatim} +#include + +/* create a table and set some values in it */ +int handle = Util_TableCreate(UTIL_TABLE_FLAGS_DEFAULT); +if (handle < 0) + CCTK_WARN(CCTK_WARN_ABORT, "couldn't create table!"); + +/* try to set some table entries */ +if (Util_TableSetFromString(handle, "two=2 pi=3.14") != 2) + CCTK_WARN(CCTK_WARN_ABORT, "couldn't set values in table!"); +\end{verbatim} + +There is also an even higher-level convenience function +\verb|Util_TableCreateFromString()|: this creates a table with the +case insensitive flag set (to match Cactus parameter file semantics), +then (assuming no errors occurred) calls \verb|Util_TableSetFromString()| +to set values in the table. + +For example, the following code sets up a table (with the case insensitive flag +set) with four entries: an integer number ({\tt two}), a real number ({\tt +pi}), a string ({\tt buffer}), and an integer array with three elements ({\tt +array}): + +\begin{verbatim} +#include + +int handle = Util_TableCreateFromString(" two = 2 " + " pi = 3.14 " + " buffer = 'Hello World' " + " array = { 1 2 3 }"); +if (handle < 0) + CCTK_WARN(CCTK_WARN_ABORT, "couldn't create table from string!"); +\end{verbatim} + +Note that this code passes a single string to +\verb|Util_TableCreateFromString()|%%% +\footnote{C automatically concatenates +adjacent character string constants separated only by whitespace.}, +which then gets parsed into key/value pairs, with the key separated from its +corresponding value by an equals sign. + +Values for numbers are converted into integers ({\tt CCTK\_INT}) if possible +(no decimal point appears in the value), otherwise into reals ({\tt CCTK\_REAL}). +Strings must be enclosed in either single or double quotes. String values in +single quotes are interpreted literally, strings in double quotes may contain +character escape codes which then will be interpreted as in C. +Arrays must be enclosed in curly braces, array elements must be single numbers +of the same type (either all integer or all real). + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subsection{Table Iterators} +\label{sect-UtilityRoutines/tables/table-iterators} + +In the examples up to now, the code, which wanted to get values from +the table, knew what the keys were. It's also useful to be able to +write generic code which can operate on a table without knowing the +keys. ``Table iterators'' (``iterators'', for short) are used for this. + +An iterator is an abstraction of a pointer to a particular table entry. +Iterators are analogous to the \verb|DIR *| pointers used by the POSIX +\verb|opendir()|, \verb|readdir()|, \verb|closedir()|, and similar +functions, to Perl hash tables' \verb|each()|, \verb|keys()|, +and \verb|values()|, and to the C++ Standard Template Library's +forward iterators. + +At any time, the entries in a table may be considered to be in some +arbitrary (implementation-defined) order; an iterator may be used to +walk through some or all of the table entries in this order. This +order is guaranteed to remain unchanged for any given table, so long +as no changes are made to that table, \ie{} so long as no +\verb|Util_TableSet*()|, \verb|Util_TableSet*Array()|, +\verb|Util_TableSetGeneric()|, \verb|Util_TableSetGenericArray()|, +\verb|Util_TableSetString()|, or \verb|Util_TableDeleteKey()| calls +are made on that table (making such calls on other tables doesn't +matter). The order may change if there is any change in the table, +and it may differ even between different tables with identical key/value +contents (including those produced by \verb|Util_TableClone()|).%%% +\footnote{%%% + For example, if tables were implemented by hashing, + the internal order could be that of the hash buckets, + and the hash function could depend on the internal + table address. + }%%% +{} + +Any change in the table also invalidates all iterators pointing +anywhere in the table; using any such iterator is an error. +Multiple iterators may point into the same table; they all use the +same order, and (unlike in Perl) they're all independent. + +The detailed function description +in the Reference Manual +for \verb|Util_TableItQueryKeyValueInfo()| has an example of +using an iterator to print out all the entries in a table. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subsection{Multithreading and Multiprocessor Issues} + +At the moment, the table functions are \emph{not} thread-safe +in a multithreaded environment. +%% However, this should change in +%%the not-too-distant future: then all the table functions will default +%to being thread-safe. That is, user code will be able call these +%%functions concurrently from multiple threads, with the table functions +%%automatically doing any necessary locking.%%% +%%\footnote{%%% +% For the implementation, this means that we will need a +% reader/writer lock for each table and for each iterator: +% any number of threads will be able to concurrently read +% the data structure, but any write will require exclusive +% access. +% }%%% +%{} (We may add a flags-word bit to suppress this for maximum +%performance if you know you won't be making concurrent calls from +%multiple threads.) + +Note that tables and iterators are process-wide, i.e. all +threads see the same tables and iterators (think of them as like the +Unix current working directory, with the various routines which modify +the table or change iterators acting like a Unix \verb|chdir()| system +call). + +In a multiprocessor environment, tables are always processor-local. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subsection{Metadata about All Tables} + +Tables do not \emph{themselves} have names or other +attributes. However, we may add some special +``system tables'' to be used by Cactus itself to store this sort of +information for those cases where it's needed. For example, we may add support for a +``checkpoint me'' bit in a table's flags word, so that if you want a +table to be checkpointed, you just need to set this bit. +In this case, the table will probably get a system generated name in +the checkpoint dump file. But if you want the table to have some +other name in the dump file, then you need to tell the checkpointing +code that, by setting an appropriate entry in a checkpoint table. +(You would find the checkpoint table by looking in a special +``master system table'' that records handles of other interesting tables.) + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + + \chapter{Schedule Bins} \label{sec:Appendix.schedule_bins} diff --git a/doc/UsersGuide/ApplicationThorns.tex b/doc/UsersGuide/ApplicationThorns.tex new file mode 100644 index 00000000..c174c551 --- /dev/null +++ b/doc/UsersGuide/ApplicationThorns.tex @@ -0,0 +1,4621 @@ +% /*@@ +% @file ThornWriters.tex +% @date 27 Jan 1999 +% @author Gabrielle Allen, Tom Goodale, Gerd Lanfermann +% @desc +% Thorn writer's guide part of the Cactus User's Guide +% @enddesc +% @version $Header$ +% @history +% @date Tue Feb 12 14:33:25 CET 2002 +% @author Jonathan Thornburg +% @desc Revise section "Interpolation Operators" to cover the +% new interpolation API (when we will have multiple APIs +% coexisting, e.g. CCTK_InterpLocalUniform() and +% CCTK_InterpLocalNonUniform()) +% @endhistory +% @@*/ + +\begin{cactuspart}{Thorn Writing} +\renewcommand{\thepage}{\Alph{part}\arabic{page}} +\label{part:ThornWriting} + +\chapter{Application thorns} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +This chapter goes into the nitty-gritty of writing a thorn. +It introduces key concepts for thorns, then goes on to give +a brief outline of how to configure a thorn. +There is then some detail about concepts introduced by the configuration +step, followed by discussion of code which you must put into your files +in order to use Cactus functionality, and details of utility functions +you may use to gain extra functionality. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\section{Thorn Concepts} +\label{chap:thorn_concepts} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subsection{Thorns} + +A thorn is the basic working module within Cactus. All user supplied +code goes into thorns, which are, by and large, independent of each other. +Thorns communicate with each other via calls to the flesh API, plus, more +rarely, custom APIs of other thorns. + +The connection from a thorn to the flesh, or to other thorns, is specified in +configuration files which are parsed at compile time and used to generate +glue code which encapsulates the external appearance of a thorn. + +Thorn names must be (case independently) unique, must start with a letter, +can only contain +letters, numbers or underscores, and must contain 27 characters or less. +In addition, a thorn cannot have the name \texttt{doc}, this is reserved +for arrangement documentation. Arrangement names which start with a +`\texttt{\#}', or finish with `\texttt{\~{}}' or `\texttt{.bak}' will be +ignored. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subsection{Arrangements} +\label{sec:arrangements} + +Thorns are grouped into \textit{arrangements}. This is a logical grouping of +thorns which is purely for organisational purposes. For example, +you might wish to keep all your initial data thorns in one arrangement, +and all your evolution thorns in another arrangement, or you may want +to have separate arrangements for your developments, private and shared +thorns. + +The arrangements live in the \texttt{arrangements} directory of the main +Cactus directory. Arrangement names must be (case independently) unique, +must start with a letter, +and can only contain +letters, numbers or underscores. Arrangement names which start with a +`\texttt{\#}', or finish with `\texttt{\~{}}' or `\texttt{.bak}' will be +ignored. + +Inside an arrangement directory there are directories for each thorn +belonging to the arrangement. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subsection{Implementations} + +\label{sec:implementations} + +One of the key concepts for thorns is the concept of the \textit{implementation}. +Relationships among thorns are all based upon relationships among the +implementations they provide. +In principle, it should be possible to swap one thorn providing an +implementation with another thorn providing that implementation, +without affecting any other thorn. + +An implementation defines a group of variables and parameters which +are used to implement some functionality. For example, the thorn +\texttt{CactusPUGH/PUGH} provides the implementation \texttt{driver}. This +implementation is responsible for providing memory for grid variables and +for communication. Another thorn can also implement \texttt{driver}, +and both thorns can be compiled in \emph{at the same time}. +At runtime, the user can decide which thorn providing \texttt{driver} is used. +No other thorn should be affected by this choice. + +When a thorn decides it needs access to a variable or a parameter provided by +another thorn, it defines a relationship between itself and the other thorn's +\emph{implementation}, not explicitly with the other \emph{thorn}. +This allows the transparent replacement, at compile or runtime, of one +thorn with another thorn providing the same functionality as seen by +the other thorns. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\section{Anatomy of a Thorn} +\label{chap:thorn_anatomy} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subsection{Thorns} + +A thorn consists of a subdirectory of an arrangement containing four +administrative files: + +\begin{Lentry} +\item[\texttt{interface.ccl}] the Cactus interface, which defines the grid +functions, variables, etc. See Appendix \ref{sec:Appendix.interface}. +\item[\texttt{param.ccl}] the parameters introduced by this thorn, and the +parameters needed from other thorns. See Appendix +\ref{sec:Appendix.param}. +\item[\texttt{schedule.ccl}] scheduling information for routines called by +the flesh. See Appendix \ref{sec:Appendix.schedule}. +\item[\texttt{configuration.ccl}] configuration options for the thorn. See + Appendix \ref{sec:Appendix.configuration.ccl}. +\end{Lentry} + +Thorns can also contain +\begin{itemize} +\item a subdirectory called \texttt{src}, which should hold source files + and compilation instructions for the thorn +\item a subdirectory \texttt{src/include} for include files +\item a \texttt{README} containing a brief description of the thorn +\item a \texttt{doc} directory for documentation +\item a \texttt{par} directory for example parameter files +\item a \texttt{test} subdirectory may also be added, to hold the thorn's + test suite. See Section \ref{sec:adding_test_suite} for details. +\end{itemize} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subsection{Creating a Thorn} + + +To simplify the creation of a thorn, a \texttt{make} target {\tt +gmake newthorn} has been provided. When this is run: + +\begin{enumerate} +\item{} You will be prompted for the name of the new thorn. +\item{} You will be prompted for the name of the arrangement in which you +would like to include your thorn. Either enter a new arrangement name or +pick one from the list of available arrangements that are shown. +\end{enumerate} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subsection{Configuring your Thorn} + +The interaction of a thorn with the flesh and other thorns is controlled +by certain configuration files. + +These are: + +\begin{Lentry} + +\item [\texttt{interface.ccl}] +This defines the \textit{implementation} (Section~\ref{sec:implementations}) +the thorn provides, and the variables the thorn needs, along with their +visibility to other implementations. + +\item [\texttt{param.ccl}] +This defines the parameters that are used to control the thorn, along +with their visibility to other implementations. + +\item [\texttt{schedule.ccl}] +This defines which functions are called from the thorn and when they are +called. It also handles memory and communication assignment for grid variables. + +\item [\texttt{configuration.ccl}] +This file is optional for a thorn. If it exists, it contains extra +configuration options of this thorn. + +\end{Lentry} + +\subsubsection{General Syntax of CCL Files} + +\textit{Cactus Configuration Language} (CCL) files are simple text files +used to define configuration information for a thorn. CCL files are +case independent, and may contain comments introduced by the hash `\texttt{\#}' +character, which indicates that the rest of the line is a comment. If the last +non-blank character of a line in a CCL file is a backslash +`\texttt{$\backslash$}', the +following line is treated as a continuation of the current line. + +\subsubsection{The \texttt{interface.ccl} File} +\label{subsec:interface_ccl} + +The \texttt{interface.ccl} file is used to declare + +\begin{itemize} +\item the implementation provided by the thorn +\item the variables provided by the thorn +\item the include files provided by the thorn +\item the functions provided by the thorn (in development) +\end{itemize} + + +The implementation is declared by a single line at the top of the file +\begin{alltt} +implements: <\var{name}> +\end{alltt} +Where \texttt{<\var{name}>} can be any combination of alphanumeric +characters and underscores, and is case independent. + +There are three different access levels available for variables + +\begin{Lentry} +\item[\texttt{Public}] +Can be `inherited' by other implementations (see below). +\item[\texttt{Protected}] +Can be shared with other implementations which declare themselves to +be friends of this one (see below). +\item[\texttt{Private}] +Can only be seen by this thorn. +\end{Lentry} + +Corresponding to the first two access levels there are two relationship +statements that can be used to get variables (actually groups of variables, +see below) from other implementations. + +\begin{Lentry} +\item [\texttt{Inherits: <\var{name}>}] +This gets all \texttt{Public} variables from implementation \texttt{<\var{name}>}, and all +variables that \texttt{<\var{name}>} has in turn inherited. +An implementation may inherit from any number of other implementations. +\item [\texttt{Friend: <\var{name}>}] +This gets all \texttt{Protected} variables from implementation \texttt{<\var{name}>}, but, +unlike \texttt{inherits}, it is symmetric and also defines a transitive relation by +pushing its own implementation's \texttt{Protected} variables onto implementation +\var{name}. This keyword is used to define a group of implementations which +all end up with the same \texttt{Protected} variables. +\end{Lentry} + +So, for example, an \texttt{interface.ccl} starting +\begin{alltt} +implements: wavetoy +inherits: grid +friend: wave_extract +\end{alltt} +declares that the thorn provides an implementation called \texttt{wavetoy}, gets +all the \texttt{public} variables declared by an implementation called \texttt{grid}, +and shares all \texttt{protected} variables with \texttt{wave\_extract} and its +friends. + +Cactus variables, described in Chapter~\ref{chap:cactus_variables}, are placed +in groups with homogeneous attributes, where +the attributes describe properties such as the data type, group type, +dimension, ghostsize, number of timelevels, type of staggering and +distribution. + +For example, a group, called \texttt{realfields} of 5 real grid +functions (\texttt{phi}, \texttt{a}, +\texttt{b}, \texttt{c}, \texttt{d}), on a 3D grid, would be defined by +\begin{verbatim} +CCTK_REAL realfields type=GF TimeLevels=3 Dim=3 +{ + phi + a,b,c,d +} "Example grid functions" +\end{verbatim} +or, for a group called \texttt{intfields} consisting of just one +distributed 2D array of integers, +\begin{verbatim} +CCTK_INT intfields type=ARRAY size=xsize,ysize ghostsize=gxsize,gysize dim=2 +{ + anarray +} "My 2D arrays" +\end{verbatim} +where \texttt{xsize}, \texttt{ysize}, \texttt{gxsize}, \texttt{gysize} are all +parameters defined in the thorn's \texttt{param.ccl}. + + +By default, all groups are \texttt{private}, to change this, an access +specification of the form \texttt{public:} or \texttt{protected:} (or +\texttt{private:} to change it back) may be placed on a line by itself. This +changes the access level for any group defined in the file from that point on. + +All variables seen by any one thorn must have distinct names. + +\subsubsection{The \texttt{param.ccl} File} +\label{subsec:param_ccl} + +Users control the operation of thorns via parameters given in a file +at runtime. The \texttt{param.ccl} file +is used to specify the parameters used to control an individual thorn, and +to specify the values these parameters are allowed to take. When the code +is run, it reads a parameter file and sets the parameters if they fall +within the allowed values. If a parameter is not assigned in a parameter +file, it is given its default value. + +There are three access levels available for parameters: + +\begin{Lentry} +\item [\texttt{Global}] +These parameters are seen by all thorns. +\item [\texttt{Restricted}] +These parameters may be used by other implementations if they so desire. +\item [\texttt{Private}] +These are only seen by this thorn. +\end {Lentry} + +A parameter specification consists of: +\begin{itemize} + +\item The parameter type (each may have an optional {\t CCTK\_} in front) +\begin{Lentry} +\item [\texttt{REAL}] +\item [\texttt{INT}] +\item [\texttt{KEYWORD}] +A distinct string with only a few known allowed values. +\item [\texttt{STRING}] +An arbitrary string, which must conform to a given regular expression. +\item [\texttt{BOOLEAN}] +A boolean type which can take values {\t 1}, {\t t}, {\t true}, {\t yes} or +{\t 0}, {\t f}, {\t false}, {\t no}. +\end{Lentry} + +\item The parameter name + +\item An optional size (in square brackets)--if this is present, the + parameter is a ``parameter array'', i.e.\ it will actually be an array + of parameters, each of which has the same properties, but a different + value. Such arrays appear as normal arrays in C or Fortran, 0-based + in C, and 1-based in Fortran. In the parameter file the value of + each element is specified with square brackets and is 0-based. The + size must be an integer. + +\item A description of the parameter + +\item An allowed value block--this consists of a brace-delimited +block of lines\footnote{The beginning brace (\texttt{\{}) must sit on a line by +itself; the ending brace (\texttt{\}}) must be preceded by a carriage return.} +describing the allowed values of the parameter. Each range may have a +description associated with it by placing a \texttt{::} on the line, and putting +the description afterwards. + +\item The default value--this must be one of the allowed values. + +\end{itemize} + +For the numeric types \texttt{INT} and \texttt{REAL}, a range consists +of a string of the +form lower-bound:upper-bound:step, where a missing number or an asterisk +`\texttt{*}' denotes anything (i.e.\ infinite bounds or an infinitesimal step). + +For example, +\begin{verbatim} +REAL Coeff "Important coefficient" +{ +0:3.14 :: "Range has to be from zero to Pi, default is zero" +} 0.0 + +#No need to define a range for BOOLEAN +BOOLEAN nice "Nice weather?" +{ +}"yes" + +# A example for a set of keywords and its default (which has to be +# defined in the body) +KEYWORD confused "Are we getting confused?" +{ + "yes" :: "absolutely positively" + "perhaps" :: "we are not sure" + "never" :: "never" +} "never" + +REAL Length[2] "Length in each direction" +{ +0:* :: "Range has to be from zero to infinity, default is one" +} 1.0 + +\end{verbatim} +defines a \texttt{REAL} parameter, a \texttt{BOOLEAN} parameter, a \texttt{KEYWORD}, +and an array of \texttt{REAL} parameters. + +By default, all parameters are \texttt{private}; to change this, an access +specification of the form \texttt{global:} or \texttt{restricted:} (or +\texttt{private:} to change it back) may be placed on a line by itself. This +changes the access level for any parameter defined in the file from that point on. + +To access \texttt{restricted} parameters from another implementation, a line +containing \texttt{shares: <\var{name}>} declares that all parameters mentioned in +the file, from now until the next access specification, originate in +implementation \texttt{<\var{name}>}. (Note that only one implementation can be +specified on each \texttt{shares:} line.) Each of these parameters must be qualified by the initial token \texttt{USES} or \texttt{EXTENDS}, where +\begin{Lentry} +\item[\texttt{USES}] indicates that the parameters range remains unchanged. +\item[\texttt{EXTENDS}] indicates that the parameters range is going to be extended. +\end{Lentry} + +In contrast to parameter declarations in other access blocks, the default +value must be omitted---it is impossible to set the default value of any +parameter not originating in this thorn. +For example, the following block adds possible values to the keyword +\texttt{initial\_data} originally defined in the implementation \texttt{einstein}, +and uses the \texttt{REAL} parameter \texttt{speed}. + +\begin{verbatim} +shares:einstein + +EXTENDS KEYWORD initial_data +{ + "bl_bh" :: "Brill Lindquist black holes" + "misner_bh" :: "Misner black holes" + "schwarzschild" :: "One Schwarzschild black hole" +} + +USES CCTK_REAL speed + +\end{verbatim} + +Note that you must compile at least one thorn which implements \texttt{einstein}. + + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subsubsection{The \texttt{schedule.ccl} File} +\label{subsec:schedule_ccl} +By default, no routine of a thorn will be run. The \texttt{schedule.ccl} +file defines those that should be run, and when and under which +conditions they should be run. + +The specification of routine scheduling is via a schedule block which +consists of lines of the form + +\begin{alltt} +schedule <\var{name}> at <\var{time bin}> [\var{other options}] +\{ + LANG: + STORAGE: [\var{group list with timelevels}] + TRIGGERS: [\var{group list}] + SYNC: [\var{group list}] +\} "\var{A description}" +\end{alltt} +where \texttt{<\var{name}>} is the name of the routine, and +\texttt{<\var{time bin}>} is the name of a schedule bin (the \texttt{CCTK\_} +prefix is optional). A list of the most useful schedule bins for application +thorns is given here, a complete and more descriptive list is provided in +Appendix~\ref{sec:Appendix.schedule_bins}: + +\begin{Lentry} + +\item [\texttt{CCTK\_STARTUP}] +For routines, run before the grid hierarchy is set up, for example, function +registration. + +\item [\texttt{CCTK\_PARAMCHECK}] +For routines that check parameter combinations, routines registered here +only have access to the grid size and the parameters. + +\item [\texttt{CCTK\_BASEGRID}] +Responsible for setting up coordinates, etc. + +\item [\texttt{CCTK\_INITIAL}] +For generating initial data. + +\item [\texttt{CCTK\_POSTINITIAL}] +Tasks which must be applied after initial data is created. + +\item [\texttt{CCTK\_PRESTEP}] +Stuff done before the evolution step. + +\item [\texttt{CCTK\_EVOL}] +The evolution step. + +\item [\texttt{CCTK\_POSTSTEP}] +Stuff done after the evolution step. + +\item [\texttt{CCTK\_ANALYSIS}] +For analysing data. + +\end{Lentry} + +The \var{other options} allow finer-grained control of the scheduling. It is +possible to state that the routine must run \texttt{BEFORE} or \texttt{AFTER} +another routine or set of routines. +It is also possible to schedule the routine under an +alias name by using \texttt{AS <\var{alias\_name}>}. + +\begin{Lentry} + +\item[\texttt{LANG}] +The \texttt{LANG} keyword specifies the linkage of the scheduled routine +which determines how to call it from the scheduler. +C and Fortran linkage are possible here. C++ routines should be defined as +\texttt{extern "C"} and registered as \texttt{LANG: C}. + +\item[\texttt{STORAGE}] The \texttt{STORAGE} keyword specifies any groups for +which memory should be allocated for the duration of the routine. The +storage status reverts to its previous status after the routine +returns. The format of the \texttt{STORAGE} statement includes specifying +the number of timelevels of each group for which storage should be activated. + +\texttt{STORAGE: <\var{group1}>[\var{timelevels1}], +<\var{group2}>[\var{timelevels2}]} + + This number can range from one to the maximum number of timelevels for the group, as specified in the group definition in its \texttt{interface.ccl} file. If this maximum number is one, the timelevel specification can be omitted from the +\texttt{STORAGE} statement. + +\item[\texttt{TRIGGERS}] \texttt{TRIGGERS} is used when the routine is +registered at \texttt{ANALYSIS}. This is a special time bin; a routine +registered here will only be called if one of the variables from a +group in \texttt{TRIGGERS} is due for output. (A routine without {\tt +TRIGGERS} declaration will always be called.) + +\item[\texttt{SYNC}] +The keyword \texttt{SYNC} specifies groups of variables which should be +synchronised (that is, their ghostzones should be exchanged between +processors) on exit from the routine. Specifying synchronisation of +grid variables in \texttt{schedule.ccl} is an alternative to calling the +functions \texttt{CCTK\_SyncGroup()} or \texttt{CCTK\_SyncGroupsI()} +(see the Reference Manual) from inside a routine. Using the \texttt{SYNC} +keyword in the \texttt{schedule.ccl} is the preferred method, since it +provides the flesh with more information about the behaviour of your code. + +\end{Lentry} + +Besides schedule blocks, it's possible to embed C style \texttt{if/else} +statements in the \texttt{schedule.ccl} file. These can be used to +schedule things based upon the value of a parameter. + +\vskip .5cm +{\bf Example I:} + +If the parameter \texttt{evolve\_hydro} is positively set, the Fortran +routine \texttt{hydro\_predictor} is scheduled to run in the \textit{evolution} +loop, after the routine \texttt{metric\_predictor} and before +\texttt{metric\_corrector}. The routine names \texttt{metric\_predictor} and +\texttt{metric\_corrector}, may either be real routine names from the same +or a different thorn, or they may be \texttt{aliased} routine names (see the +next example). + +Before entry to \texttt{hydro\_predictor}, +storage will be allocated for one timelevel for the group of grid +variables \texttt{hydro\_variables} on exit from the routine this storage +will be deallocated and the contents of the variables will be lost. +\begin{verbatim} +if(CCTK_Equals(evolve_hydro,"yes")) +{ + SCHEDULE hydro_predictor AT evol AFTER metric_predictor BEFORE metric_corrector + { + LANG: FORTRAN + STORAGE: hydro_variables[1] + } "Do a predictor step on the hydro variables" +} +\end{verbatim} + +If the parameter \texttt{evolve\_hydro} is set negatively, the {\tt +hydro\_predictor} routine will not be called by the scheduler. Note +that if the \texttt{evolve\_hydro} parameter is \texttt{STEERABLE}, it can be +dynamically scheduled and de-scheduled during a run if a steering +interface is available. + +\vskip .5cm +{\bf Example II:} + +The thorns \texttt{WaveToy77} and \texttt{WaveToyC}, each provide a +routine to evolve the 3D wave equation: \texttt{WaveToyF77\_Evolution} and +\texttt{WaveToyC\_Evolution}. The routine names have to be different, so +that both thorns can be compiled at the same time, their functionality +is identical though. Either one of them can then be activated at run +time in the parameter file via \texttt{ActiveThorns}. + +Since each evolution routine provides the same +functionality, it makes sense to schedule them under the common alias {\tt +WaveToy\_Evolution} to allow relative scheduling ({\tt +BEFORE/AFTER}) independent of the actual routine name (which may +change depending on the activation in the parameter file). + +In both cases, the group of variables \texttt{scalarfield} are synchronised +across processes when the routine is exited. +\begin{verbatim} +schedule WaveToyF77_Evolution AS WaveToy_Evolution AT evol +{ + LANG: Fortran + STORAGE: scalartmps + SYNC: scalarfield +} "Evolution of 3D wave equation" + +schedule WaveToyC_Evolution AS WaveToy_Evolution AT evol +{ + LANG: C + STORAGE: scalartmps + SYNC: scalarfield +} "Evolution of 3D wave equation" +\end{verbatim} +The thorn \texttt{IDScalarWave} schedules the routine \texttt{WaveBinary} +after the alias \texttt{WaveToy\_Evolution}. It is scheduled independently of +the C or Fortran routine name. +\begin{verbatim} +schedule WaveBinary AT evol AFTER WaveToy_Evolution +{ + STORAGE: wavetoy::scalarevolve + LANG: Fortran +} "Provide binary source during evolution" +\end{verbatim} + +\paragraph{Storage Outside of Schedule Blocks} +The keyword \texttt{STORAGE} can also be used outside of the schedule +blocks to indicate that storage for these groups should be switched on +at the start of the run. Note that the storage is only allocated in +this way at the start; a thorn could explicitly switch the storage off +(although this is not recommended practise). As for the \texttt{STORAGE} +statement in schedule blocks, each group must also specify how many +timelevels to activate storage for. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subsubsection{The \texttt{configuration.ccl}} +\label{subsec:configuration_ccl} + +[{\bf NOTE:} the \texttt{configuration.ccl} is a new feature, and not all the +features described in this section have been fully implemented yet. +PROVIDES, REQUIRES and SCRIPT should work, but OPTIONAL is still +being developed.] + +The \texttt{configuration.ccl} file is optional. It can be used for two +purposes: to detect certain features of the host system, such as the +presence or absence of libraries, variable types, etc, or the +location of libraries; or to provide access to certain functions too +complex or otherwise not suitable for function aliasing. + +The basic concept here is that a thorn can either provide or use a +{\bf capability}. A thorn providing a capability can specify a script +which is run by the CST to detect features and write any configuration +files; the script may output lines to its standard output to inform +the CST of features to: add to the header files included by thorns +using this capability; add to the make files used to build thorns +using this capability; or add to the main Cactus link line. The script +may also indicate that this capability is not present by returning a +non-zero exit code---e.g. if the thorn is providing access to an +external library, it should return an error if the library is not +installed on the system. + +A thorn may either require a capability to be present, in which case +it is an error if there is no thorn providing that capability in the +configuration's \texttt{ThornList}, or it may optionally use a capability, in +which case a macro is defined in the thorn's header file if a thorn +providing the capability is present. + +A configuration.ccl file has the form: + +\begin{alltt} +PROVIDES <\var{My_Capability}> +\{ + SCRIPT <\var{My_ConfigScript}> + LANG <\var{My_Language}> +\} + +REQUIRES <\var{Another_Capability}> + +OPTIONAL <\var{Yet_Another_Capability}> +\{ + DEFINE <\var{macro}> +\} + +\end{alltt} + +which states that this thorn provides the capability +\verb|My_Capability|, and a script \verb|MyConfigScript| should be run +to detect features of this capability; the script is in language +\verb|My_Language|---the CST will use the appropriate environment or +interpreter to invoke the script. + +The example requires a thorn providing \verb|Another_Capability| to be +in the \texttt{ThornList}, and, if a thorn providing +\verb|Yet_Another_Capability| is in the \texttt{ThornList}, the preprocessor +macro \verb|macro| will be defined, and set to 1. + +As an example, a thorn A may be able to use PVM for parallelism if it is +present, but can still work in the absence of it. There could be a +thorn providing PVM, and thorn A would then have + +\begin{alltt} +OPTIONAL PVM +\{ + DEFINE HAVE_PVM +\} + +\end{alltt} + +The code in thorn A could then have + +\begin{alltt} + +#ifdef HAVE_PVM + do something +#else + do something else +#endif + +\end{alltt} + +to pick the appropriate mode. + +The syntax of the output of the configure script is described in +Appendix \ref{sec:Appendix.configuration.ccl.configscript}. + + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subsection{Naming Conventions for Source Files} +\label{nacofosofi} + +The make system uses file extensions to designate coding language, +as well as other properties of the code in the file. + +The following extensions are understood: + +\begin{center} +\begin{tabular}{|l|c|c|c|} +\hline +Extension & Language & Format & Preprocess \\ +\hline +{\t .F} & Fortran 90 & fixed & yes \\ +{\t .f} & Fortran 90 & fixed & no \\ +{\t .F90} & Fortran 90 & free & yes \\ +{\t .f90} & Fortran 90 & free & no \\ +{\t .F77} & Fortran 77 & fixed & yes \\ +{\t .f77} & Fortran 77 & fixed & no \\ +{\t .c} & C & - & yes \\ +{\t .cc} or {\t .C} & C++ & - & yes \\ +\hline +\end{tabular} +\end{center} + +In order to use Cactus \texttt{\#include} directives in a file, it must be +preprocessed. + +A complete description of Fortran fixed and free format can be found in any +textbook on Fortran 90. The most obvious differences are that in fixed +format, code must begin after the 5th column and line continuations are +indicated by a character in column 5, while in free format, lines can begin +anywhere, and line continuations are indicated by an ampersand at the end of +the line to be continued. Also note that statement labels are handled +very differently. + +The following restrictions apply to file names: +\begin{itemize} +\item For portability across all operating systems, the base names + for any particular extension should not depend on the operating + system being case sensitive (e.g.\ having \texttt{MyFile.c} and + \texttt{MYFILE.f77} is allright, but \texttt{MyFile.c} and + \texttt{MYFILE.c} could cause problems). +\item Currently, all source files within a thorn must have distinct names, +regardless of whether they are placed in different subdirectories. We hope +to relax this in future. Different thorns may have files with the same names, +however. +\end{itemize} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subsection{Adding Source Files} + +By default, the CCTK looks in the \texttt{src} directory of the thorn for source +files. + +There are two ways in which to specify the sources. The easiest is to use the +\texttt{make.code.defn} based method in which the CCTK does all the work, but you +may instead put a \texttt{Makefile} in the \texttt{src} directory and do everything +yourself. + +\subsubsection{\texttt{make.code.defn} based thorn building} +\label{sec:mabathbu} + +This is the standard way to compile your thorn's source files. +The Cactus make system looks for a file called \texttt{make.code.defn} in that +directory (if there is no file called \texttt{Makefile} in the \texttt{src} directory). At its simplest, this file contains two lines + +\begin{itemize} +\item \texttt{SRCS = <\var{list of all source files in this directory}>} + +\item \texttt{SUBDIRS = <\var{list of all subdirectories, including +subdirectories of subdirectories}>} + +\end{itemize} + +Each subdirectory listed should then have a \texttt{make.code.defn} file +containing just a \texttt{SRCS = } line, a \texttt{SUBDIRS = } line will +be ignored. + +In addition, each directory can have a \texttt{make.code.deps} file, which, +for files in that directory, can contain additional make rules and dependencies +for files in that directory. See the GNU Make documentation for complete details of the +syntax. + +\subsubsection{\texttt{Makefile} based thorn building} + +This method gives you the ultimate responsibility. +The only requirement is that +a library called \texttt{\$NAME} be created by the \texttt{Makefile}. + +The makefile is passed the following variables +\begin{Lentry} + +\item [\texttt{\$(CCTK\_HOME)}] the main Cactus directory + +\item [\texttt{\$(TOP)}] the configuration directory + +\item [\texttt{\$(SRCDIR)}] the directory in which the source files can be found + +\item [\texttt{\$(CONFIG)}] the directory containing the configuration files + +\item [\texttt{\$(THORN)}] the thorn name + +\item [\texttt{\$(SCRATCH\_BUILD)}] the scratch directory where Fortran 90 module +files should end up if they need to be seen by other thorns. + +\item [\texttt{\$(NAME)}] the name of the library to be built + +\end{Lentry} + +and has a working directory of +\texttt{<\var{config}>/build/<\var{thorn\_name}>} . + +\subsubsection{Other makefile variables} + +\begin{itemize} +\item \texttt{CPP}: The C preprocessor which is used to preprocess C + and C++ source code, and to determine the C and C++ make + dependencies +\item \texttt{FPP}: The C preprocessor which is used to preprocess + Fortran source code +\item \texttt{CPPFLAGS}: Flags which are passed to \texttt{CPP}, to + \texttt{CC}, and to \texttt{CXX} +\item \texttt{FPPFLAGS}: Flags which are passed to \texttt{FPP} +\item \texttt{CC}: The C compiler +\item \texttt{CXX}: The C++ compiler +\item \texttt{F77}: The Fortran 77 compiler. This is only used if + there is no Fortran 90 compiler +\item \texttt{F90}: The Fortran 90 compiler. If this variable is + set, it is also used to compile Fortran 77 files +\item \texttt{CFLAGS}: Flags which are passed to \texttt{CC} +\item \texttt{CXXFLAGS}: Flags which are passed to \texttt{CXX} +\item \texttt{F77FLAGS}: Flags which are passed to \texttt{F77} +\item \texttt{F90FLAGS}: Flags which are passed to \texttt{F90} +\item \texttt{LD}: The binder. This should not be directly + \texttt{ld}, but should be a compiler driver such as C++. + Often, \texttt{LD} is the same as \texttt{CXX} +\item \texttt{LDFLAGS}: Flags which are passed to \texttt{LD} +\end{itemize} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\section{Cactus Variables} +\label{chap:cactus_variables} + +A \textit{grid variable} is a Cactus program variable passed among thorns, +(or routines belonging to the same thorn interface), +by way of calls to the flesh. +They are the only variables known to Cactus. +Such variables represent values of functions on the computational grid, +and are, therefore, often called \textit{grid functions}. + +In the Cactus context, grid variables are often referred +to simply as \textit{variables}. + +Cactus variables are used instead of local variables for a number of reasons: +\begin{itemize} +\item Cactus variables can be made visible to other thorns, allowing + thorns to communicate and share data. +\item Cactus variables can be distributed and communicated + among processors, allowing parallel computation. +\item A database of Cactus variables, and their attributes, is held by + the flesh, and this information is used by thorns, for example, for + obtaining a list of variables for checkpointing. +\item Many Cactus APIs and tools can only be used with Cactus variables. +\item Cactus provides features for error checking based on Cactus variables + and their attributes. +\end{itemize} + +Cactus variables are collected into \textit{groups}. All variables in a +group are of the same data type, and have the same attributes. Most Cactus +operations act on a group as a whole. A group must be declared in its +thorn's \texttt{interface.ccl} file. + +The specification for a group declaration +(fully described in Appendix~\ref{sec:Appendix.interface}) is, + +\begin{alltt} +<\var{data_type}> <\var{group_name}> [TYPE=<\var{group_type}>] [DIM=<\var{dim}>] [TIMELEVELS=<\var{num}>] + [SIZE=<\var{size in each direction}>] [DISTRIB=<\var{distribution_type}>] + [GHOSTSIZE=<\var{ghostsize}>] [STAGGER=<\var{stagger-specification}>] +[\{ + [ <\var{variable_name}>[,]<\var{variable_name}> + <\var{variable_name}> ] +\} ["<\var{group_description}>"] ] +\end{alltt} + +Currently, the names of groups and variables must be distinct. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subsection{Data Type} + +Cactus supports integer, real, complex and character variable types, in +various different sizes. (Sizes in the following refer to the number of +bytes occupied by the a variable of the type). + +\begin{Lentry} +\item[INTEGER] +\texttt{CCTK\_INT}, \texttt{CCTK\_INT1}, \texttt{CCTK\_INT2}, \texttt{CCTK\_INT4}, \texttt{CCTK\_INT8}. +(\texttt{CCTK\_INT} defaults to being \texttt{CCTK\_INT4}). +\item[REAL] +\texttt{CCTK\_REAL}, \texttt{CCTK\_REAL4}, \texttt{CCTK\_REAL8}, \texttt{CCTK\_REAL16}. (\texttt{CCTK\_REAL} defaults to being \texttt{CCTK\_REAL8}). +\item[COMPLEX] +\texttt{CCTK\_COMPLEX}, \texttt{CCTK\_COMPLEX8}, \texttt{CCTK\_COMPLEX16}, \texttt{CCTK\_COMPLEX32}. +(\texttt{CCTK\_COMPLEX} defaults to being \texttt{CCTK\_COMPLEX16}). +\item[BYTE] +This is a 1 byte data type. +\end{Lentry} + +Normally a thorn should use the default types---\texttt{CCTK\_INT}, \texttt{CCTK\_REAL}, \texttt{CCTK\_COMPLEX}---rather than explicitly setting the size, as this gives maximum +portability. Also, the defaults can be changed at configuration time (see Section +\ref{subsec:Compilation-Available_Options}), and this allows people to compile the +code with different precisions to test for roundoff effects, or to run more +quickly with a lower accuracy. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subsection{Group Types} +\label{sec:cactus_variables-groups} + +Groups can be either \textit{scalars}, \textit{grid functions} (GFs), or +\textit{grid arrays}. Different attributes are relevant for the different group +types. + + +\begin{Lentry} +\item[\texttt{SCALAR}] +This is just a single number, e.g.\ the total energy of some field. These +variables aren't communicated between processors---what would be the +result of such communication? +\item[\texttt{GF}] +This is the most common group type. A GF is an array with a +specific size, set at run time in the parameter file, which is distributed +across processors. All GFs have the same size, and the same number of +ghostzones. Groups of GFs can also specify a dimension, +number of timelevels, and stagger type. +\item[\texttt{ARRAY}] +This is a more general form of the GF. Each group of arrays can have +a distinct size and number of ghostzones, in addition to dimension, +number of timelevels and stagger type. +The drawback of using an array over a GF is that a lot of data about the +array can only be determined by function calls, rather than the +quicker methods available for GFs. +\end{Lentry} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subsection{Timelevels} + +These are best introduced by an example using finite differencing. +Consider the 1-D wave equation +\begin{equation} +\frac{\partial^2 \phi}{\partial t^2} = \frac{\partial^2 \phi}{\partial x^2} +\end{equation} +To solve this by partial differences, one discretises the derivatives to get +an equation relating the solution at different times. There are many ways +to do this, one of which produces the following difference equation +\begin{equation} +\label{equation:difference} +\phi(t+\Delta t,x) -2\phi(t,x) +\phi(t-\Delta t,x) = \frac{\Delta t^2}{\Delta x^2} \lbrace{\phi(t,x+\Delta x) -2\phi(t,x) +\phi(t,x-\Delta x)}\rbrace +\end{equation} +which relates the three timelevels $t+\Delta t$, $t$, and $t-\Delta t$. + +Obviously, the code could just use three variables, one for each timelevel. +This turns out, however, to be inefficient, because as soon as the time is +incremented to $t+\Delta t$, it would be necessary to copy data from the +$t$ variable to the $t-\Delta t$ variable and from the $t+\Delta t$ variable +to the $t$ variable. + +To remove this extraneous copy, Cactus allows you to specify the number +of timelevels used by your numerical scheme. It then generates variables +with the base name (e.g.\ \texttt{phi}) suffixed by a qualifier for +which timelevel is being referred to---no suffix for the +next timelevel, and \texttt{\_p} for each previous timelevel. + +The timelevels are rotated (by the driver thorn) at the start +of each evolution step, that is: +\begin{verbatim} +initial +poststep +analysis + +loop: + rotate timelevels + t = t + dt + it = it + 1 + prestep + evolve + poststep + analysis +\end{verbatim} + +Timelevel rotation means that, for example, +\texttt{phi\_p} now holds the values of the former \texttt{phi}, +and \texttt{phi\_p\_p} the values of the former \texttt{phi\_p}, etc. +Note that after rotation, \texttt{phi} is undefined, and its values should +not be used until they have been updated. + +All timelevels, except the current level, should be considered \emph{read-only} during evolution, that is, their values should not be changed by thorns. +The exception to this rule is for function initialisation, when the +values at the previous timelevels do need to be explicitly filled out. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subsection{Size and Distrib} + +A Cactus grid function or array has a size set at runtime by parameters. +This size can either be the global size of the array across all processors +(\texttt{DISTRIB=DEFAULT}), +or, if \texttt{DISTRIB=CONSTANT}, the specified +size on \emph{each} processor. +If the size is split across processors, the driver thorn is +responsible for assigning the size on each processor. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subsection{Ghost Zones} +\label{sec:ghost_size} + +Cactus is based upon a \textit{distributed computing} paradigm. That is, +the problem domain is split into blocks, each of which is assigned to +a processor. For hyperbolic and parabolic problems the blocks only +need to communicate at the edges. + +To illustrate this, take the example of the wave equation again. +Equation \ref{equation:difference} describes a possible time-evolution +scheme. On examination, you can see that to generate the data at the +point ($t+\Delta t$, $x$) we need data from the four points ($t$, +$x$), ($t-\Delta t$, $x$), ($t$, $x+\Delta x$), and ($t$, $x-\Delta x$) +only. Ignoring the points at $x$, which are obviously always +available on a given processor, you can see that the algorithm +requires a point on either side of the point $x$, i.e.\ this algorithm +has \textit{stencil size} 1. Similarly algorithms requiring two points +on either side have stencil size 2, etc. + +Now, if you evolve the above scheme, it becomes apparent that at each +iteration the number of grid points you can evolve decreases by one at +each edge (see Figure \ref{fig:noghost}). + +\begin{figure}[ht] +\begin{center} +\includegraphics[angle=0,width=8cm]{1dnoghost} +\end{center} +\caption{Distributed wave equation with no ghostzones} +\label{fig:noghost} +\end{figure} + +At the outer boundary of the physical domain, the data for the boundary +point can be generated by the boundary conditions, however, at internal +boundaries, the data has to be copied from the adjacent processor. It +would be inefficient to copy each point individually, so instead, a +number of \textit{ghostzones} are created at the internal boundaries. A +ghostzone consists of a copy of the whole plane (in 3D, line in 2D, +point in 1D) of the data from the adjacent processor. That is, the array +on each processor is augmented with copies of points from the adjacent +processors, thus allowing the algorithm to proceed \textit{on the points +owned by this processor} without having to worry about copying data. +Once the data has been evolved one step, the data in the ghostzones +can be exchanged (or \textit{synchronised}) between processors in one +fell swoop before the next evolution step. (See Figure +\ref{fig:withghost}.) Note that you should have at least as many +ghostzones as your stencil size requires. + +\begin{figure}[ht] +\begin{center} +\includegraphics[angle=0,width=8cm]{withghost} +\end{center} +\caption{Distributed wave equation with ghostzones} +\label{fig:withghost} +\end{figure} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subsection{Staggering} +\label{sec:staggering} + +The staggering of a grid function or array describes the \emph{physical} +placement of that grid function relative to the supporting grid +structure. For example, a grid function does not have to +be placed at the intersection +of the ``grid lines''. It can be moved by half a grid spacing in +any or all dimensions. In the latter case, it will be placed in +the center of a cell. + +The staggering of a grid function is a pure \emph{physical} property: +the values will be calculated at a different position in physical +space. Still the indexing (or bookkeeping) is kept the same for all +types of staggerings: the indexing of the default unstaggered grids is +used. + +\vskip .25cm + +{\bf Specifying the staggertype} + +The type of staggering applied to a grid function can be specified in +the \texttt{interface.ccl} file by the attribute \texttt{stagger} (see Appendix +\ref{sec:Appendix.interface}). Cactus supports three kinds of staggering +per dimension. The physical location of a grid function is shifted +relative to the default position by adding the following values to the +stagger attribute: +\begin{Lentry} +\item[\texttt{M}] no staggering, default. Refers to the ``minus'' face +relative to the default gridpoint. +\item[\texttt{C}] centre staggering. The physical location is offset by +half of the grid spacing in the positive direction (or to the right). +\item[\texttt{P}] full staggered. \texttt{P} refers to plus. The physical location +is offset by a full grid spacing in the positive direction (or the right). +\end{Lentry} +For multi-dimensional grid functions you concatenate the code +characters in $xyz$ order. In Figure \ref{fig:stagger1}, we show four different +staggerings of a two dimensional grid function. The solid black grid +circles show the default location of the grid function at the +intersections of the grid lines. In (A) we show an additional grid +function of type \texttt{stagger=MC}: no staggering in $x$ direction, +center staggered in $y$ direction. In (B) we have \texttt{stagger=CM}, and +staggering each direction (\texttt{stagger=CC}) is shown in (C). The full +staggering in (D) (\texttt{stagger=PP}) obeys the same rules, but is +rather unusual; it is included here for completeness. + +\begin{figure}[ht] + \def\epsfsize#1#2{0.45#1} +\begin{center} +\includegraphics[angle=0,width=8cm]{staggering1} +\end{center} +\caption[]{\small {\bf Staggered gridpoints in 2D} for several +staggerings. (a) : \texttt{MC}, (b): \texttt{CM}, (c): \texttt{CC}, (d): {\tt +PP}. Note that the staggering of grid functions does not change its +index. The staggered gridpoints and the corresponding unstaggered +points (arrows) are accessed by the same indices.} +\label{fig:stagger1} +\end{figure} + +\subsection{Information about Grid Variables} + +The flesh holds a database with information related to grid variables, and +provides a set of querying APIs. + +\subsubsection{Group Information} + +Fundamental information about grid functions (e.g.\ local grid size and +location, number of ghostzones) is passed through the argument list of +scheduled routines (see Section~\ref{sec:cactus_variables_c}). To obtain similar information +from non-scheduled routines, or for general grid variables, a set of +functions are provided, the last two letters of which specify whether +the information is requested using a group name (\texttt{GN}) or index +(\texttt{GI}), or a variable name (\texttt{VN}) or index (\texttt{VI}). + +\begin{Lentry} + +\item [\texttt{CCTK\_Grouplsh[GN|GI|VN|VI]}] An array of integers + with the local grid size on this processor. + +\item [\texttt{CCTK\_Groupgsh[GN|GI|VN|VI]}] An array of integers + with the global grid size. + +\item [\texttt{CCTK\_Groupbbox[GN|GI|VN|VI]}] An array of integers + which indicate whether the boundaries are internal boundaries + (e.g.\ between processors), or physical boundaries. + A value of \texttt{1} indicates + a physical (outer) boundary at the edge of the computational grid, + and \texttt{0} indicates an internal boundary. + +\item [\texttt{CCTK\_Groupnghostzones[GN|GI|VN|VI]}] An array of integers with + the number of ghostzones used in each direction. + +\item [\texttt{CCTK\_Grouplbnd[GN|GI|VN|VI]}] An array of integers + containing the lowest index (in each direction) + of the local grid, as seen on the global grid. Note that these indices + start from zero, so you need to add one when using them in + Fortran thorns. + +\item [\texttt{CCTK\_Groupubnd[GN|GI|VN|VI]}] An array of integers + containing the largest index (in each direction) + of the local grid, as seen on the global grid. Note that these indices + start from zero, so you need to add one when using them in + Fortran thorns. + +\end{Lentry} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\section{Cactus Parameters} +\label{chap:Cactus_parameters} + +Parameters are the means by which the user specifies the runtime behaviour of +the code. Each parameter has a data type and a name, as well as a +range of allowed values and a default value. These are declared in the thorn's +\texttt{param.ccl} file. + +The thorn determines which parameters can be used in other thorns by +specifying a \textit{scope} for the thorn, as explained in Section +\ref{sec:Cactus_parameters.scope}. + +The user may specify initial values for parameters in the parameter file +(see Section~\ref{sec:Parameter_File}); the flesh validates these values +against the parameters' allowed ranges. +Otherwise, the initial value of the parameter is taken to be its default. +Once validated, parameter values are fixed, and cannot be changed, +unless the parameter is specified to be \textit{steerable} +(see \ref{sec:Cactus_parameters.steerable}). +For a detailed discussion of the \texttt{param.ccl} syntax, see +Appendix~\ref{sec:Appendix.param}. + +The full specification for a parameter declaration is +\begin{alltt} +[EXTENDS|USES] <\var{parameter_type}>[[<\var{size}>]] <\var{parameter name}> "<\var{parameter description}>" +\{ + <\var{PARAMETER_RANGES}> +\} <\var{default value}> +\end{alltt} + +You can obtain lists of the parameters associated with +each thorn using the Cactus command line options \texttt{-o} and \texttt{-O} +(Section~\ref{sec:command_line_options}). + + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subsection{Types and Ranges} +\label{sec:Parameters.Types_and_Ranges} + +Parameters can be of these types: + +\begin{Lentry} +\item[Int] Can take any integral value +\item[Real] Can take any floating point value +\item[Keyword] Can have a value consisting of one of a choice of strings +\item[Boolean] Can be true or false ({\t 1}, {\t t}, {\t true}, or +{\t 0}, {\t f}, {\t false}) +\item[String] Can have any string value +\end{Lentry} + +Each parameter can be validated against a set of allowed +\textit{ranges}, each of which has a description associated with it. +The nature +of the range is determined by the type of parameter, as follows: + +\subsubsection{Int} + +The range specification is of the form + +\begin{alltt} +\var{lower}:\var{upper}:\var{stride} +\end{alltt} + +where \var{lower} and \var{upper} specify the lower and upper allowed +range, and \var{stride} allows numbers to be be missed out, e.g.\ +\begin{verbatim} +1:21:2 +\end{verbatim} +means the value must be an odd number between one and twenty-one +(inclusive). + +A missing end of range (or a `\texttt{*}') indicates negative or positive +infinity, and the default stride is one. + +\subsubsection{Real} + +The range specification is of the form +\begin{alltt} +\var{lower}:\var{upper} +\end{alltt} +where \var{lower} and \var{upper} specify the lower and upper allowed +range. A missing end of range (or a `\texttt{*}') implies negative or positive +infinity. The above is inclusive of the endpoints. A `\texttt{(}' +(or `\texttt{)}') before (or after) the lower (or upper) range specifies an open +endpoint. + +The numbers written in a \texttt{param.ccl} file are interpreted as C code. +To express a number in `scientific notation', use, +e.g.\ `\texttt{1e-10}', which is a double precision constant in C. (If the +floating precision of the variable to which it is assigned is not +double, then C will typecast appropriately. If you \emph{really} want to +specify a single precision floating constant, or a long double +constant, append the number with {\t f} or {\t l} respectively.) + +\subsubsection{Keyword} + +The range specification consists of a string, which will be matched in +a case insensitive manner. + +\subsubsection{Boolean} + +There is no range specification for this type of parameter. + +\subsubsection{String} + +The range is a POSIX regular expression. On some machines you may be +able to use extended regular expressions, but this is not guaranteed +to be portable. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subsection{Scope} +\label{sec:Cactus_parameters.scope} + +Parameters can be \texttt{GLOBAL}, \texttt{RESTRICTED}, or \texttt{PRIVATE}. +Global parameters are visible to all thorns. Restricted parameters +are visible to any thorn which chooses to \texttt{USE} or \texttt{EXTEND} +it. A private parameter is only visible to the thorn which declares +it. + +\subsection{Steerable} +\label{sec:Cactus_parameters.steerable} +A parameter can be changed dynamically if it is specified to be +\textit{steerable} (see +Section~\ref{subsec:Appendix.param.specification_items}). +It can then be changed by a call to the flesh function +\texttt{CCTK\_ParameterSet} (see the Reference Guide for a description +of this function). + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\section{Scheduling} +\label{chap:scheduling} + +Cactus contains a rule-based scheduling system, which determines which +routines, from which thorns are run in which order. The scheduler +determines if the specifications are inconsistent, but does allow the +user to schedule a routine with respect to another routine which may not +exist. +For a detailed discussion of the {\t schedule.ccl} syntax see +Appendix~\ref{sec:Appendix.schedule}. + +A usual simple specification for a schedule declaration is +\begin{alltt}schedule <\var{function name}> AT <\var{schedule bin}> +\{ + LANG: <\var{language}> + [STORAGE: <\var{group}>,<\var{group}>...] +\} "\var{Description of function}" +\end{alltt} + +The full specification for a schedule declaration is +\begin{alltt}schedule [GROUP] AT|IN <\var{schedule bin}|\var{group name}> + [AS <\var{alias}>] + [WHILE <\var{variable}>] [IF <\var{variable}>] + [BEFORE|AFTER <\var{item}>|(<\var{item}> <\var{item}> ...)] +\{ + LANG: <\var{language}> + [STORAGE: <\var{group}>,<\var{group}>...] + [TRIGGER: <\var{group}>,<\var{group}>...] + [SYNC: <\var{group}>,<\var{group}>...] + [OPTIONS: <\var{option}>,<\var{option}>...] +\} "\var{Description of function or schedule group}" +\end{alltt} + +This full schedule specification consists of a mandatory part, a set +of options, and the main body limited by braces, referred to as the +\texttt{schedule block}. + +Each schedule item is scheduled either \texttt{AT} a particular +\var{scheduling bin}, or \texttt{IN} a schedule \var{group}. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subsection{Schedule Bins} +\label{scheduling:schedule_bins} + +These are the main times at which scheduled functions are run, from +fixed points in the flesh and driver thorn (schedule bins can easily +be traversed from any thorn, although this is not usual). When a +schedule bin is \textit{traversed}, all the functions scheduled in that +particular are called, in the manner described in Section +\ref{scheduling:calling_scheduled_functions} and respecting the +requested ordering of functions(Section~\ref{scheduling:schedule_options}). In the absence of any ordering, functions in a particular schedule bin will be called in +an undetermined order. + +The schedule bins are described in Section \ref{subsec:schedule_ccl}. Note that +the preceding \texttt{CCTK\_} is optional for the use of the bin names +in the \texttt{schedule.ccl} file. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subsection{Groups} +\label{scheduling:groups} + +If the optional \texttt{GROUP} specifier is used, the item is a schedule +group rather than a normal function. Schedule groups are effectively +new, user-defined, schedule bins. Functions or groups may be +scheduled \texttt{IN} these, in the same way as they are scheduled {\tt +AT} the main schedule bins. (That is, groups may be nested.) + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subsection{Schedule Options} +\label{scheduling:schedule_options} +The options define various characteristics of the schedule item. + +\begin{Lentry} +\item[\texttt{AS}] +This assigns a new name to a function for scheduling purposes. This +is used, for instance, to allow a thorn to schedule something before +or after a routine from another implementation; two thorns providing this +implementation can schedule a routine \texttt{AS} the same thing, thus +allowing other thorns to operate independently of which one is active. + +\item[\texttt{WHILE}] +This specifies a \texttt{CCTK\_INT} grid scalar which is used to +control the execution of this item. As long as the grid scalar has +a nonzero value, the schedule item will be executed repeatedly. This +allows dynamic behaviour with scheduling. + +\item[\texttt{IF}] +This specifies a \texttt{CCTK\_INT} grid scalar which is used to +control the execution of this item. If the grid scalar has a +nonzero value, the schedule item will be executed, otherwise the item +will be ignored. This allows dynamic behaviour with scheduling. + +If both an \texttt{IF} and a \texttt{WHILE} clause are present, then +the schedule is executed according to the following pseudocode: +\begin{verbatim} +IF condition + WHILE condition + SCHEDULE item + END WHILE +END IF +\end{verbatim} + +\item[\texttt{BEFORE} or \texttt{AFTER}] +These specify either +\begin{itemize} +\item a function or group before or after which this item will be + scheduled, or +\item a list of functions and/or groups; this item will be scheduled + (once) before any of them or after all of them respectively. +\end{itemize} +Note that a single schedule item may have multiple \texttt{BEFORE} +and/or \texttt{AFTER} options; the scheduler will honor all of these +(or abort with a fatal error). For example, +\begin{alltt} +schedule FOO BEFORE A BEFORE B BEFORE C ... +\end{alltt} +schedules \texttt{FOO} before any of \texttt{A}, \texttt{B}, or \texttt{C}. +This can also be written +\begin{alltt} +schedule FOO BEFORE (A B C) ... +\end{alltt} + +Note that the set of all \texttt{BEFORE}/\texttt{AFTER} options in +all active schedule blocks of all active thorns, \emph{must} specify +a (directed) graph with no cycles; if there are any cycles, then the +scheduler will abort with a fatal error. +\end{Lentry} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subsection{The Schedule Block} +\label{scheduling:schedule_block} + +The schedule block specifies further details of the scheduled function +or group. + +\begin{Lentry} +\item[\texttt{LANG}] +This specifies the language of the routine. Currently this is either +C or Fortran. C++ routines should be defined as \texttt{extern "C"} +and registered as \texttt{LANG: C}. +\item[\texttt{STORAGE}] The \texttt{STORAGE} keyword specifies groups for +which memory should be allocated for the duration of the routine or +schedule group. The storage status reverts to its previous status +after completion of the routine or schedule group. +\item[\texttt{TRIGGER}] +This is only used for items scheduled at timebin \texttt{CCTK\_ANALYSIS}. +The item will only be executed if output is due for at least one +variable in one of the listed groups. (The item will also be called +if there is no group listed.) +\item[\texttt{SYNC}] +On exit from this item, the ghost zones of the listed groups will be +exchanged. +\item[\texttt{OPTIONS}] +This is for miscellaneous options. The list of accepted options is +given in Appendix~\ref{app:allopts}. +\end{Lentry} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subsection{How Cactus Calls Scheduled Functions} +\label{scheduling:calling_scheduled_functions} + +For each scheduled function called, the flesh performs a variety of jobs +at entry and exit. + +On entry to a scheduled routine, if the routine is being called at the +\texttt{CCTK\_ANALYSIS} timebin first, a check is made to see if the routine should +actually be called on this timestep. For this, all grid variables in the +trigger groups for the routine are checked with all registered output +methods to determine if it is time to output any triggers. The routine +will only be called if at least one is due to be output. Note that once +a grid variable has been analyzed, it gets marked as such, and will not +be analyzed again during this iteration. +(Note that a routine without any trigger groups will also be called.) +Thus, if more than one analysis +routine should be triggered on the same trigger variable(s), they must +be scheduled in a single group. Routines from all timebins, other than \texttt{ANALYSIS}, are always called. + +Next, storage is assigned for any required variables, remembering the +original state of storage. + +The routine is then called, and on exit, any required grid variables are +first synchronised. Following synchronization, any required output methods +are called for the triggers. Finally, the storage of grid variables is +returned to the original state. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\section{Writing a Thorn} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subsection{Thorn Programming Languages} + +When you start writing a new thorn, the first decision to make is +which programming language to use. The source code in Cactus thorns +can be written in any mixture of Fortran 77, Fortran 90, +C or C++. The following points should be considered when +choosing a language to work in +\begin{itemize} + +\item All functions designed for application thorn writers are available + in all languages, however, some interfaces for infrastructure + thorn writing are only available from C or C++. + +% This is no longer relevant? +%\item If you are writing in Fortran, use \texttt{F77} if you want +% to distribute your code to people who may not be able to afford +% to buy proprietary Fortran 90 compilers. + +\item Stick to C rather than C++, unless you really need + features from C++, this will help you with portability. + +\end{itemize} + +Whatever language you choose, if you want your thorn to be portable, and +compile and run on multiple platforms, stick to the standards and don't +use machine dependent extensions. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subsection{What the Flesh Provides} + +The flesh provides for thorns: +\begin{Lentry} +\item [\texttt{Variables}] +\item [\texttt{Parameters}] +\item [\texttt{Cactus Functions}] + +\begin{itemize} + \item{} Driver (parallelisation) utilities + \item{} I/O utilities + \item{} Coordinates utilities + \item{} Reduction utilities + \item{} Interpolation utilities + \item{} Information utilities +\end{itemize} +\end{Lentry} + + +\subsubsection{Fortran Routines} + +Any source file using Cactus infrastructure should include +the header file \texttt{cctk.h} using the line +\begin{verbatim} +#include "cctk.h" +\end{verbatim} +(Fortran programmers should not be put off by this being a C style +header file---most Cactus files are run through a C preprocessor +before compilation.) + +\paragraph{Variables} + +Any routine using Cactus argument lists (for example, all routines called from +the scheduler at time bins between {\t CCTK\_STARTUP} and {\t CCTK\_SHUTDOWN}) +should include at the top of the file the header +\begin{verbatim} +#include "cctk_Arguments.h" +\end{verbatim} + +A Cactus macro \texttt{CCTK\_ARGUMENTS} is defined for each thorn +to contain: +\begin{itemize} +\item General information about the grid hierarchy, for example, the number + of grid points used. See Section \ref{sec:cactus_variables_c} for a + complete list. +\item All the grid variables defined in the thorn's \texttt{interface.ccl} +\item All the grid variables required from other thorns as requested by + the \texttt{inherits} and \texttt{friend} lines in the \texttt{interface.ccl} +\end{itemize} +These variables must be declared at the start of the routine using +the macro \texttt{DECLARE\_CCTK\_ARGUMENTS}. + +To pass the arguments to another routine in the same thorn use the macro +\texttt{CCTK\_PASS\_FTOF} in the calling routine, and again the macro +\texttt{CCTK\_ARGUMENTS} in the receiving routine. + +Note that you cannot use Cactus argument lists in routines scheduled at the +{\t CCTK\_STARTUP} and {\t CCTK\_SHUTDOWN} time bins, because at this time +no grid hierarchy exists. + + +\paragraph{Parameters} + +All parameters defined in a thorn's \texttt{param.ccl} and all \texttt{global} +parameters, appear as local variables of the corresponding \texttt{CCTK} data type +in Fortran source code, i.e. Booleans and Integers appear as \texttt{CCTK\_INT} +types (with nonzero/zero values for boolean {\t yes/no}), +Reals as \texttt{CCTK\_REAL}, and Keywords and String parameters as +\texttt{CCTK\_STRING} (see +also note below). These variables are \emph{read only}, and \emph{changes should +not be made to them}. The effect of changing a parameter is undefined (at best). + +Any routine using Cactus parameters should include at +the top of the file the header +\begin{verbatim} +#include "cctk_Parameters.h" +\end{verbatim} + +The parameters should be declared at the start of the routine +using them with the macro \texttt{DECLARE\_CCTK\_PARAMETERS}. + +In Fortran, special care should be taken with string valued parameters. +These parameters are passed as C pointers, and can not be treated as +normal Fortran strings. +To compare a string valued parameter and Fortran +string, use the macro \texttt{CCTK\_EQUALS()} or the function \texttt{CCTK\_Equals()} +(see the reference manual for a description of the \texttt{CCTK\_} functions). +To print the value of a string valued parameter to screen, use the subroutine +\texttt{CCTK\_PrintString()}. A further function \texttt{CCTK\_FortranString} +provides a mechanism for converting a string parameter to a Fortran string. +For example, if \texttt{operator} is a Cactus string parameter holding the name of a reduction operator whose handle you need to find, you cannot pass it +directly into the subroutine \texttt{CCTK\_LocalArrayReductionHandle}, which is expecting +a Fortran string. Instead, the following is needed: +% +\begin{verbatim} + character*200 fortran_operator + CCTK_INT fortran_operator_len + integer handle + + call CCTK_FortranString(fortran_operator_len,operator,fortran_operator) + call CCTK_LocalArrayReductionHandle(handle,fortran_operator(1:fortran_operator_len)) +\end{verbatim} + + + +\paragraph{Fortran Example} + +The Fortran routine, \verb|MyFRoutine|, is scheduled in the {\tt +schedule.ccl} file, doesn't use Cactus parameters, and calls another +routine, in the same thorn, \verb|MyNewRoutine|, which does use +parameters. This routine needs to be passed an integer flag as well +as the standard Cactus variables. The source file should look like +% +\begin{verbatim} +#include "cctk.h" +#include "cctk_Arguments.h" +#include "cctk_Parameters.h" + + subroutine MyFRoutine(CCTK_ARGUMENTS) + +c I'm very cautious, so I want to declare all variables + implicit none + + DECLARE_CCTK_ARGUMENTS + + integer flag + + flag = 1 + call MyNewRoutine(CCTK_PASS_FTOF,flag) + + return + end + + subroutine MyNewRoutine(CCTK_ARGUMENTS,flag) + + implicit none + + DECLARE_CCTK_ARGUMENTS + DECLARE_CCTK_PARAMETERS + integer flag + +c Main code goes here + + return + end + +\end{verbatim} + +\paragraph{Cactus Fortran Functions} + +Cactus Fortran functions, for example, \texttt{CCTK\_MyProc} and {\tt +CCTK\_Equals}, can all be declared by adding the statement +% +\begin{verbatim} +#include "cctk_Functions.h" +\end{verbatim} +% +near the top of the file, and adding the declaration +% +\begin{verbatim} +DECLARE_CCTK_FUNCTIONS +\end{verbatim} +% +to a module or a subroutine after the \texttt{implicit none} +statement, but before any executable code. + +\paragraph{Fortran Modules} + +Fortran modules should be placed into source files that have the same +name as the module, followed by the corresponding file name suffix. A +module \texttt{metric} should thus be placed, e.g.\ into a file +\texttt{metric.F90}. This convention allows the Cactus build system +to automatically deduce the compile time dependencies. + +If you do not follow this convention, then you have to include the +modules into the thorn's \texttt{make.code.deps} file +(Section \ref{sec:mabathbu}) to ensure they are compiled before the routines +which use them. This is especially important for parallel building. +For example, if a routine in \texttt{MyRoutine.F90} uses a module in {\tt +MyModule.F90}, then add the line: +% +\begin{verbatim} +MyRoutine.F90.o: MyModule.F90.o +\end{verbatim} + +\paragraph{The \texttt{MOD} function} + +The intrinsic function \texttt{MOD} in Fortran takes two integer +arguments, which should both be of the same type. This means +that it may be necessary to cast the arguments to, e.g. +\texttt{INT} for some architectures. This can occur in particular +when a \texttt{CCTK\_INT} parameter and the Cactus variable \texttt{cctk\_iteration} +(which is declared to be \texttt{INTEGER}) are used, +in which case the correct code is +\begin{verbatim} +MOD(cctk_iteration,INT(MyParameter)) +\end{verbatim} + + +\subsubsection{C Routines} + +Any source file using Cactus infrastructure should include +the header file \texttt{cctk.h} using the line +\begin{verbatim} +#include "cctk.h" +\end{verbatim} + +\paragraph{Variables} + +Any routine using Cactus argument lists (for example, all routines called from +the scheduler at time bins between {\t CCTK\_STARTUP} and {\t CCTK\_SHUTDOWN}), +should include at the top of the file the header +\begin{verbatim} +#include "cctk_Arguments.h" +\end{verbatim} + +A Cactus macro \texttt{CCTK\_ARGUMENTS} is defined for each thorn +to contain +\begin{itemize} +\item General information about the grid hierarchy, for example, the +number of grid points on the processor. See Section \ref{sec:cactus_variables_c} +for a complete list. +\item All the grid variables defined in the thorn's \texttt{interface.ccl} +\item All the grid variables required from other thorns as requested by + the \texttt{inherits} and \texttt{friend} lines in the \texttt{interface.ccl} +\end{itemize} +These variables must be declared at the start of the routine using +the macro \texttt{DECLARE\_CCTK\_ARGUMENTS}. This macro should always be the +first line of the routine. + +To pass the arguments to another routine in the same thorn, use the macro +\texttt{CCTK\_PASS\_CTOC} in the calling routine, and again the macro +\texttt{CCTK\_ARGUMENTS} in the receiving routine. + +Note that you cannot use Cactus argument lists in routines scheduled at the +{\t CCTK\_STARTUP} and {\t CCTK\_SHUTDOWN} time bins, because at this time +no grid hierarchy exists. + + +\paragraph{Parameters} + +All parameters defined in a thorn's \texttt{param.ccl} and all \texttt{global} +parameters, appear as local variables of the corresponding \texttt{CCTK} data type +in C source code, i.e. Integers and Booleans appear as \texttt{CCTK\_INT} types +(with nonzero/zero values for boolean {\t yes/no}), Reals as +\texttt{CCTK\_REAL}, and Keywords and String parameters as \texttt{CCTK\_STRING}. +These variables are \emph{read only}, and \emph{changes should not be made to +them}. The effect of changing a parameter is undefined (at best). + +Any routine using Cactus parameters should include at +the top of the file the header +\begin{verbatim} +#include "cctk_Parameters.h" +\end{verbatim} + +The parameters should be declared as the last statement in the declaration part +of the routine using them with the macro \texttt{DECLARE\_CCTK\_PARAMETERS}. + +\paragraph{Example} + +The C routine \verb|MyCRoutine| is scheduled in the \texttt{schedule.ccl} file, +and uses Cactus parameters. The source file should look like +\begin{verbatim} +#include "cctk.h" +#include "cctk_Arguments.h" +#include "cctk_Parameters.h" + +void MyCRoutine(CCTK_ARGUMENTS) +{ + DECLARE_CCTK_ARGUMENTS + DECLARE_CCTK_PARAMETERS + + /* Here goes your code */ +} +\end{verbatim} + +\paragraph{Complex variables} + +Cactus supports complex grid variables, and since there is no +complex data type in C, Cactus provides a number +of functions for manipulating complex numbers to mirror the +functionality available in Fortran. These functions are \texttt{CCTK\_Cmplx}, +\texttt{CCTK\_CmplxReal}, \texttt{CCTK\_CmplxImag}, \texttt{CCTK\_CmplxConjg}, +\texttt{CCTK\_CmplxAdd}, \texttt{CCTK\_CmplxSub}, \texttt{CCTK\_CmplxMul}, +\texttt{CCTK\_CmplxDiv}, \texttt{CCTK\_CmplxExp}, \texttt{CCTK\_CmplSin}, +\texttt{CCTK\_CmplxAbs}, \texttt{CCTK\_CmplxLog}, and \texttt{CCTK\_CmplSqrt}. +See the Reference Manual for more details. + + +\paragraph{Specifically for C Programmers} + +Grid functions are held in memory as 1-dimensional C arrays. These are laid +out in memory as in Fortran. This means that the first index should +be incremented through most rapidly. This is illustrated in the example +below. + +Cactus provides +macros to find the 1-dimensional index which is needed from the multidimensional +indices which are usually used. There is a macro for each dimension of +grid function. Below is an artificial example to demonstrate this +using the 3D macro \texttt{CCTK\_GFINDEX3D}: +\begin{verbatim} +for (k=0; kcctk_lsh[0]*((j)+cctkGH->cctk_lsh[1]*(k))) +\end{verbatim} + +\subsubsection{Cactus Variables} +\label{sec:cactus_variables_c} + +The Cactus variables which are passed through the macro +\texttt{CCTK\_ARGUMENTS} are +\begin{Lentry} +\item [\texttt{cctkGH}] A C pointer identifying the grid hierarchy. +\item [\texttt{cctk\_dim}] An integer with the number of dimensions + used for this grid hierarchy. +\item [\texttt{cctk\_lsh}] An array of \texttt{cctk\_dim} integers + with the local grid size on this processor. +\item [\texttt{cctk\_gsh}] An array of \texttt{cctk\_dim} integers + with the \textit{global} grid size. +\item [\texttt{cctk\_iteration}] The current iteration number. +\item [\texttt{cctk\_delta\_time}] A \texttt{CCTK\_REAL} with the timestep. +\item [\texttt{cctk\_time}] A \texttt{CCTK\_REAL} with the current time. +\item [\texttt{cctk\_delta\_space}] An array of \texttt{cctk\_dim} {\tt +CCTK\_REAL}s with the grid spacing in each direction. +\item [\texttt{cctk\_nghostzones}] An array of \texttt{cctk\_dim} integers with + the number of ghostzones used in each direction. +%\item [\texttt{cctk\_from}] The index value from which the user should start loops. +%\item [\texttt{cctk\_to}] ... end loops. +\item [\texttt{cctk\_origin\_space}] An array of \texttt{cctk\_dim} {\tt + CCTK\_REAL}s with the spatial coordinates of the global origin + of the grid. +\item [\texttt{cctk\_lssh}] This is an internal array used to hold +array extents for staggering. One should use the macro CCTK\_LSSH(,) +to access its elements. + +\end{Lentry} + +The following variables describe the location of the local +grid (e.g.\ the grid treated on a given processor) within +the global grid. +\begin{Lentry} +\item [\texttt{cctk\_lbnd}] + An array of \texttt{cctk\_dim} integers + containing the lowest index (in each direction) + of the local grid, as seen on the global grid. Note that these indices + start from zero, so you need to add one when using them in + Fortran thorns. +\item [\texttt{cctk\_ubnd}] + An array of \texttt{cctk\_dim} integers + containing the largest index (in each direction) + of the local grid, as seen on the global grid. Note that these indices + start from zero, so you need to add one when using them in + Fortran thorns. +\item [\texttt{cctk\_bbox}] + An array of 2$*$\texttt{cctk\_dim} integers (in the order + $[\mbox{dim}_0^{\mbox{min}}, \mbox{dim}_0^{\mbox{max}}, + \mbox{dim}_1^{\mbox{min}}, \mbox{dim}_1^{\mbox{max}}, \ldots]$), + which indicate whether the boundaries are internal boundaries + (e.g.\ between processors), or physical boundaries. + A value of 1 indicates + a physical (outer) boundary at the edge of the computational grid, + and 0 indicates an internal boundary. +\end{Lentry} + +The following variable is needed for grid refinement methods +\begin{Lentry} +\item [\texttt{cctk\_levfac}] An array of \texttt{cctk\_dim} integer factors + by which the local grid is refined in the corresponding + direction with respect to the base grid. +\item [\texttt{cctk\_levoff}] and \texttt{cctk\_levoffdenom} Two arrays of + \texttt{cctk\_dim} integers describing the distance by which the + local grid is offset with respect to the base grid, measured + in local grid spacings. The distance in direction \texttt{dir} + is given by \texttt{1.0 * cctk\_levoff[dir] / + cctk\_levoffdenom[dir]}. +\item [\texttt{cctk\_timefac}] The integer factor + by which the time step size is reduced with respect to the + base grid. +\end{Lentry} + +The following variables are used for identifying convergence levels. + +\begin{Lentry} +\item [\texttt{cctk\_convlevel}] The convergence level of this grid hierarchy. + The base level is $0$, and every level above that is + coarsened by a factor of \texttt{cctk\_convfac}. +\item [\texttt{cctk\_convfac}] The factor between convergence levels. + The relation between the resolutions of different convergence + levels is $\Delta x_L = \Delta x_0 \cdot F^L$, where $L$ is the + convergence level and $F$ is the convergence factor. + The convergence factor defaults to $2$. +\end{Lentry} + +The variables \texttt{cctk\_delta\_space}, \texttt{cctk\_delta\_time}, and +\texttt{cctk\_origin\_space} denote the grid spacings, time step size, +and spatial origin on the \textit{base} grid. If you are using a grid +refinement method, you need to calculate these quantities on the grid +you are on. There are Cactus macros provided for this, with the +syntax \texttt{CCTK\_DELTA\_SPACE(dir)}, \texttt{CCTK\_ORIGIN\_SPACE(dir)}, +and \texttt{CCTK\_DELTA\_TIME} for both C and Fortran. It is recommended +that these macros are always used to provide the grid spacings, time +step sizes, and spatial origins in your thorns. In doing so, you +incorporate the effects of \texttt{cctk\_levfac}, \texttt{cctk\_levoff}, +\texttt{cctk\_levoffdenom}, and \texttt{cctk\_timefac}, so that you do not +explicitly have to take them into account. + +Putting the above information together, Figure~\ref{fig-global-xyz-coords} +shows two different ways to compute the global Cactus $xyz$ coordinates +of the current grid point. Because the ``alternate calculation'' (the +one using \verb|Grid::x|, \verb|Grid::y|, and \verb|Grid::z|) gives the +true global $xyz$ coordinates even in a multipatch/multiblock context, +this is generally the preferred form for general use. + +\begin{figure}[bp] +\begin{verbatim} +#include /* defines size_t */ +#include "cctk.h" + +void MyThorn_MyFunction(CCTK_ARGUMENTS) +{ +int i,j,k; + + for (k = 0 ; k < cctk_lsh[2] ; ++k) + { + for (j = 0 ; j < cctk_lsh[1] ; ++j) + { + for (i = 0 ; i < cctk_lsh[0] ; ++i) + { + const size_t posn = CCTK_GFINDEX3D(cctkGH, i,j,k); + + /* calculate the global xyz coordinates of the (i,j,k) grid point */ + /* (in a multipatch/multiblock context, this gives the per-patch coordinates) */ + const CCTK_REAL xcoord = CCTK_ORIGIN_SPACE(0) + (cctk_lbnd[0] + i)*CCTK_DELTA_SPACE(0); + const CCTK_REAL ycoord = CCTK_ORIGIN_SPACE(1) + (cctk_lbnd[1] + j)*CCTK_DELTA_SPACE(1); + const CCTK_REAL zcoord = CCTK_ORIGIN_SPACE(2) + (cctk_lbnd[2] + k)*CCTK_DELTA_SPACE(2); + + /* an alternate calculation, which requires that this thorn inherit from Grid */ + /* (in a multipatch/multiblock context, this gives the true global xyz coordinates) */ + const CCTK_REAL xxcoord = /* Grid:: */ x[posn]; + const CCTK_REAL yycoord = /* Grid:: */ y[posn]; + const CCTK_REAL zzcoord = /* Grid:: */ z[posn]; + } + } + } +} +\end{verbatim} +\caption{%%% + This figure shows two different ways to compute the global + Cactus $xyz$ coordinates of the current grid point. Because + the ``alternate calculation'' (the one one using \texttt{Grid::x}, + \texttt{Grid::y}, and \texttt{Grid::z}) gives the true global + $xyz$ coordinates even in a multipatch/multiblock context, + this is generally the preferred form for general use. + } +\label{fig-global-xyz-coords} +\end{figure} + +\subsubsection{Cactus Data Types} + +To provide portability across platforms, the Cactus grid variables and parameters are defined and +declared using Cactus data types. The most important of +these data types are described below, for a full description +see Section~\ref{sec:datyansi}. These data types should +be used to declare local variables where needed, and to +declare Cactus grid variables or parameters that need +declarations. + +\begin{Lentry} + +\item[\texttt{CCTK\_INT}] default size 4 bytes +\item[\texttt{CCTK\_REAL}] default size 8 bytes +\item[\texttt{CCTK\_COMPLEX}] consists of two \texttt{CCTK\_REAL} elements + +\end{Lentry} + +\paragraph{Example} + +In the following example, \verb|MyScalar| is a grid scalar which +is declared in the \texttt{interface.ccl} as \texttt{CCTK\_REAL}. +% +\begin{verbatim} + subroutine InitialData(CCTK_ARGUMENTS) + + DECLARE_CCTK_ARGUMENTS + + CCTK_REAL local_var + + local_var = 1.0/3.0 + MyScalar = local_var + + return + end +\end{verbatim} +% +Declaring \texttt{local\_var} to have a non-Cactus data type, e.g.\ +\texttt{REAL*4}, or using one of the other Cactus real data types +described in Section~\ref{sec:datyansi}, could give problems for +different architectures or configurations. + +\subsubsection{Staggering} +\label{sec:st} + +\paragraph{Indexing, ghostzones, etc.} +Note that staggering does not make any changes to the indexing of a +grid function: the black solid circles in diagram \ref{fig:stagger2} and their +associated staggered grid functions (connected by arrows) have the same index! + +Since the grid function does not ``know'' anything about the physical +location (it's only addressed by indices), why add staggering if the +indexing is the same? + +Indeed, you could roll your own, but there compelling reasons: +Readability and the fact that you are able to query the staggertype of a +grid function. More important: In the way the grid is laid out, there is one grid +point \emph{less} for \texttt{M} and \texttt{P} staggered grid functions. This is +illustrated in Figure \ref{fig:stagger2}, which shows 15 gridpoints distributed +across 3 processors. The solid black circles show the default +location of the grid functions, the grey circles depict the ghostzones. +Note that the number of center staggered gridpoints (fat crosses) +corresponds to the number of default gridpoints on all processors but +the last one. (The same is true for full staggered gridpoints). + +\paragraph{Staggertypes} +The string specifying the staggering is encoded in a number called +the \textit{staggerindex}. With the 3 supported staggerings, the string +is converted into a base 3 number. Several routines exist to extract the +staggering in a specific direction, called \textit{directional +staggerindex}. For example, \texttt{stagger = MCM}: \textit{staggerindex} = 3, in the +$x$-direction: \textit{directional staggerindex} = \texttt{CCTK\_STAGGER\_M} (value 0), +in the +$y$-direction: \textit{directional staggerindex} = \texttt{CCTK\_STAGGER\_C} (value 1). + +\begin{Lentry} +\item[\texttt{CCTK\_STAGGER\_M}] value used for M-type staggering +\item[\texttt{CCTK\_STAGGER\_C}] value used for C-type staggering +\item[\texttt{CCTK\_STAGGER\_P}] value used for P-type staggering +\item[\texttt{CCTK\_NO\_STAGGER}] value to indicate no staggering +\item[\texttt{CCTK\_STAGGER}] value to indicate staggering +\item[\texttt{CCTK\_NSTAGGER}] number of coded staggerings (3) +%\item[\texttt{CCTK\_STAGGER\_ERROR}] failed stagger operation, negative +\end{Lentry} + + +\begin{figure}[ht] + \def\epsfsize#1#2{0.45#1} +\begin{center} +\includegraphics[angle=0,width=10cm]{staggering2} +\end{center} +\caption[]{\small {\bf Unstaggered and center-staggered gridpoints} with +ghostzone size of one (above) and two (below). The points are +distributed across three processors. Note that the number of +center staggered gridpoints (fat crosses) is one less on the outermost grid. How to +treat this case in an easy way is explained in the text. } +\label{fig:stagger2} +\end{figure} + +When a thorn programmer uses staggered gridpoints, he has to be aware +of this gridpoint anomaly. This can be done most easily by using the +\texttt{CCTK\_LSSH(<\var{dir\_staggertype}>,<\var{direction}>)} macro. +For a given staggertype and direction, this 2D array returns the local +number of gridpoints, including ghostzones and the necessary change +for the staggering on the outermost processor. + +\begin{Lentry} +\item[\texttt{CCTK\_LSSH(<\var{dir\_staggertype}>,<\var{direction}>)}] +for a given staggertype and a direction, this macro returns the number +of processor local gridpoints, including ghostzones. + +\begin{itemize} +\item{For Fortran users, the macro has to be in capital letters}\item{This macro is C/Fortran indexing aware: +can specify the dimension in C ranging from $0 \ldots$ and in Fortran +ranging from $1 \ldots$.} +\end{itemize} +\end{Lentry} + +\vskip .25cm + +Several functions exist to derive the staggertype for a given group +and for a certain direction. +\begin{Lentry} +\item[\texttt{int CCTK\_GroupStaggerIndexGI(int \var{group\_index})}] %returns the +%\textit{staggerindex} for a given group index. +\item[\texttt{call CCTK\_GroupStaggerIndexGI(int \var{staggerindex}, int +\var{group\_index})}] returns the \var{staggerindex} for a given group index. +\end{Lentry} +\vskip .45cm + +\begin{Lentry} +\item[\texttt{int CCTK\_GroupStaggerIndexGN(char *\var{group\_name})}] %returns the +%\textit{staggerindex} for a given group name. +\item[\texttt{call CCTK\_GroupStaggerIndexGN(int \var{staggerindex}, char *\var{group\_name})}] returns the +\var{staggerindex} for a given group name. \vskip .25cm +\end{Lentry} +\vskip .45cm + +%\begin{Lentry} +%\item[\texttt{int CCTK\_GroupStaggerIndexVI(int variable\_index)}] %returns the +%%\textit{staggerindex} for a given variable index. +%\item[\texttt{call CCTK\_GroupStaggerIndexVI(int staggerindex, int variable\_index)}] returns the + +%\textit{staggerindex} for a given variable index. + +%\end{Lentry} +%\vskip .45cm +% +%\begin{Lentry} +%\item[\texttt{int CCTK\_GroupStaggerIndexVN(char *variable\_name)}] %returns the +%%\textit{staggerindex} for a given variable name. +%\item[\texttt{call CCTK\_GroupStaggerIndexVN(int staggerindex, char *variable\_name)}] returns the + +%\textit{staggerindex} for a given variable name. +%\end{Lentry} +%\vskip .45cm + +\begin{Lentry} +\item[\texttt{int CCTK\_StaggerIndex(char *\var{stagger\_string})}] %return the \textit{ +%staggerindex} for a given stagger string. +\item[\texttt{call CCTK\_StaggerIndex(int \var{staggerindex}, char *\var{stagger\_string})}] return the \var{staggerindex} for a given stagger string. +\end{Lentry} +\vskip .45cm + +\begin{Lentry} +\item[\texttt{int CCTK\_DirStaggerIndex(int \var{direction}, char *\var{stagger\_string})}] +%returns the \textit{directional staggerindex} for a given direction and +%stagger string. +\item[\texttt{call CCTK\_DirStaggerIndex(int \var{dir\_staggerindex}, int \var{direction}, char *\var{stagger\_string})}] +returns the \var{directional staggerindex} for a given direction and +stagger string. +\end{Lentry} +\vskip .45cm + +\begin{Lentry} +\item[\texttt{int CCTK\_DirStaggerIndexI(int \var{direction}, char *\var{stagger\_type})}] +%returns the \textit{directional staggerindex} for a given direction and +%staggerindex. +\item[\texttt{call CCTK\_DirStaggerIndexI(int \var{dir\_direction}, char *\var{stagger\_type})}] +returns the \var{directional staggerindex} for a given direction and +staggerindex. + +\end{Lentry} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subsection{Parallelisation} +\label{sec:parallelisation} + +The flesh itself does not actually set up grid variables. This +is done by a \textit{driver} thorn. To allow the distribution of +a grid over a number of processors, the driver thorn must +also provide the grid decomposition, and routines to enable +parallelisation. The method used to provide this parallelisation +(e.g.\ MPI, PVM) is not usually important for the thorn writer, since +the driver thorn provides routines which are called by standard interfaces +from the flesh. Here, we describe briefly the most important of these routines +for the application thorn writer. A more detailed description +of these interfaces with their arguments, is given in the Reference Manual. +A complete description of the +routines that a driver thorn must provide, will be provided in the +Infrastructure Thorn Writers Guide (Part \ref{chap:infrastructure}). The standard driver thorn is +currently \texttt{PUGH} in the \texttt{CactusPUGH} package, which +is a parallel unigrid driver. + +\begin{Lentry} +\item[\texttt{CCTK\_nProcs}] Returns the number of processors being used +\item[\texttt{CCTK\_MyProc}] Returns the processor number (this starts at + processor number zero) +\item[\texttt{CCTK\_SyncGroup, CCTK\_SyncGroupsI}] Synchronises either a single + group or a set of groups of grid arrays by + exchanging the values held in each processor ghostzones, with the + physical values of their neighbours (see the Reference Manual) +\item[\texttt{CCTK\_Barrier}] Waits for all processors to reach this point + before proceeding +\end{Lentry} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\section{Cactus Application Interfaces} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subsection{Coordinates} +\label{sec:CactusAPI.coordinates} + +The flesh provides utility routines for registering and querying +coordinate information. The flesh does not provide any coordinates +itself, these must be supplied by a thorn. Thorns are not required to +register coordinates to the flesh, but registering coordinates +provides a means for infrastructure thorns to make use of coordinate +information. + +Coordinates are grouped into \textit{coordinate systems}, which have a +specified dimension. Any number of coordinate systems can be +registered with the flesh, and a coordinate system must be registered +before any coordinates can be registered, since they must be +associated with their corresponding system. Coordinates can be +registered, with any chosen name, with an existing coordinate system, +along with their direction or index in the coordinate system. +Optionally, the coordinate can also be associated with a given grid +variable. A separate call can register the global range for a +coordinate on a given grid hierarchy. + +Following conventions for coordinate system and coordinate names, +provides a means for other thorns to use the physical properties of +coordinate systems, without being tied to a particular thorn. + +A registered coordinate system can be referred to by either its name or +an associated integer known as a \textit{handle}. Passing a handle instead +of the name string may be necessary for calling C routines from Fortran. + + +\subsubsection{Registering Coordinates and Coordinate Properties} + +\textbf{The APIs described in this section are deprecated, and will +probably be phased out fairly soon. +New code should use the APIs provided by the \texttt{CoordBase} thorn +instead (this lives in the \texttt{CactusBase} arrangement).} + +Coordinate systems and their properties can be registered at any time with the flesh. +The registration utilities for thorns providing coordinates are: +\begin{Lentry} + +\item[\texttt{CCTK\_CoordRegisterSystem}] + +Assigns a coordinate system with a chosen name and dimension. For example, +a 3-dimensional Cartesian coordinate system could be registered with the +name \texttt{cart3d} using the call from C +% +\begin{verbatim} +int ierr; +int dim=3; +ierr = CCTK_CoordRegisterSystem(dim,"cart3d"); +\end{verbatim} + +\item[\texttt{CCTK\_CoordRegisterData}] + +Defines a coordinate in a given coordinate system, with a given + direction and name, and optionally associates it to a grid variable. +The directions of the coordinates range from 1 to the dimension of the +coordinate system. For example, to register the grid variable \texttt{grid::y3d} +to have the coordinate name \texttt{y} in the \texttt{cart3d} system +% +\begin{verbatim} +int ierr; +int dir=2; +ierr = CCTK_CoordRegisterData(dir,"grid::y3d","y","cart3d"); +\end{verbatim} + +\item[\texttt{CCTK\_CoordRegisterRange}] + +Assigns the global computational maximum and minimum for a coordinate +on a grid hierarchy, that is in a \texttt{cctkGH}. At this time the +maximum and minimum values have to be of type \texttt{CCTK\_REAL}. For +example, if the \texttt{y} coordinate for the \texttt{cart3d} system ranges +between zero and one +% +\begin{verbatim} +CCTK_REAL lower=0; +CCTK_REAL upper=1; +int ierr; +ierr = CCTK_CoordRegisterRange(cctkGH, lower, upper, -1, "y", "cart3d"); +\end{verbatim} +% +Note that the API allows either the coordinate name or the direction to +be used, so that the following is also valid +% +\begin{verbatim} +CCTK_REAL lower=0; +CCTK_REAL upper=1; +int ierr; +ierr = CCTK_CoordRegisterRange(cctkGH, lower, upper, 2, NULL, "cart3d"); +\end{verbatim} + +\item[\texttt{CCTK\_CoordRegisterPhysIndex}] + +Implementing such things as symmetry properties for a grid leads to +the need to know the details of the \emph{physical} section of a grid. +Such information is typically needed by I/O thorns. The following call +illustrates how to register the +indices 3 and 25 as supplying the physical range of the \texttt{y} +coordinate in the \texttt{cart3d} system +% +\begin{verbatim} +int loweri=3; +int upperi=25; +int ierr; +ierr = CCTK_CoordRegisterPhysIndex(cctkGH, loweri, upperi, -1, "y", "cart3d"); +\end{verbatim} + + + +\end{Lentry} + +\subsubsection{Using Coordinates} + +\textbf{The APIs described in this section are deprecated, and will +probably be phased out fairly soon. +New code should use the APIs provided by the \texttt{CoordBase} thorn +instead (this lives in the \texttt{CactusBase} arrangement).} + +The utilities for thorns using coordinates are: + +\begin{Lentry} + +\item[\texttt{CCTK\_NumCoordSystems}] + +Returns the number of coordinate systems registered with the flesh. For example, +% +\begin{verbatim} +int num; +num = CCTK_NumCoordSystems(); +\end{verbatim} + +\item[\texttt{CCTK\_CoordSystemName}] + +Provides the name of a registered coordinate system, given the integer +handle (or index) for the system in the flesh's coordinate data base. +Note that the handle ranges between zero and the number of coordinate systems minus one: $0 \le \mbox{handle} \le \mbox{\texttt{CCTK\_NumCoordSystems()}}-1$. +It is important to remember that the handle given to a coordinate system +depends on the order in which systems are registered, and can be different +from one simulation to the next. + +For example, to print the names of all registered coordinate systems: +% +\begin{verbatim} +for (i=0; i$-ThornGuide}). Ideally this file should contain complete +(and \textit{up-to-date}) details about the thorn, exactly what is +relevant is for the authors to decide, but remember that the Cactus +make system automatically parses the thorn CCL files to include +information about all parameters, variables and scheduling. Suggested +sections include: + +\begin{itemize} + + \item{\bf Model.} A description of the system which the thorn is modelling, + including the equations, etc., which are being solved or implemented. + + \item{\bf Numerical implementation.} Details about how the model is + numerically implemented in the thorn. + + \item{\bf Using the thorn.} Any special details needed for using the + thorn, tricky parameters, particular operating systems or additional + required software, interactions with other thorns and examples of use. + + \item{\bf History.} Here is where you should describe why the thorn + was written, any previous software or experience which was made use of, + the authors of the thorn source code and documentation, how to get + hold of the thorn, etc. + + \item{\bf References.} A bibliography can be included, referencing papers + published using or about this thorn, or additional information about + the model or numerics used. + +\end{itemize} + +A LaTeX template for the Thorn Guide documentation can be found in the +flesh distribution at + +\texttt{doc/ThornGuide/template.tex}, + +this file is automatically copied to the correct location in a new thorn +which is created with \texttt{gmake newthorn}. + +Since Cactus scripts need to parse this documentation, and since the +LaTeX document should be consistent across all thorns included in a +Thorn Guide, please follow the guidelines below when filling in the +documentation: + +\begin{itemize} + + \item Use the Cactus Thorn Guide style file, located in the flesh + distribution at \texttt{doc/latex/cactus.sty}. This should be + included using a relative link, so that updates to the style file + are applied to all thorns. +% +\begin{verbatim} +\usepackage{../../../../doc/latex/cactus} +\end{verbatim} + + \item Aside from the \texttt{date}, \texttt{author}, and \texttt{title} fields, + all of the documentation to be included in a Thorn Guide should be + placed between the lines + + + \texttt{\% START CACTUS THORNGUIDE} + +and + + \texttt{\% END CACTUS THORNGUIDE} + + \item The command \texttt{$\backslash$def} can be used to define macros, but all + such definitions must lie between the \texttt{START} and \texttt{END} + line. Do not redefine any standard LaTeX command + + \item Do not use the {\t $\backslash$appendix} command, instead include any appendices + you have as standard sections. + + \item We only support PDF (\texttt{.pdf}) figures. + Graphics figures should be included using the \texttt{includegraphics} + command (not \texttt{epsffig}), with no file extension specified. For example, +% +\begin{verbatim} +\begin{figure}[ht] + \begin{center} + \includegraphics[width=6cm]{MyArrangement_MyThorn_MyFigure} + \end{center} + \caption{Illustration of this and that} + \label{MyArrangement_MyThorn_MyLabel} +\end{figure} +\end{verbatim} + + \item All {\bf labels}, {\bf citations}, {\bf references}, and {\bf + graphic images} names should conform to the following guidelines: + \texttt{ARRANGEMENT\_THORN\_LABEL}. For instance, if you arrangement is + called CactusWave, your thorn WaveToyC, and your original image + blackhole.eps, you should rename your image to be {\tt + CactusWave\_WaveToyC\_blackhole.eps} + + \item References should be formatted with the standard LaTeX {\bf + bibitem} command, for example, a bibliography section should + look like: +% +\begin{verbatim} +\begin{thebibliography}{9} + \bibitem{MyArrangement_MyThorn_Author99} + {J. Author, \textit{The Title of the Book, Journal, or periodical}, 1 (1999), + 1--16. \url{http://www.nowhere.com/}} +\end{thebibliography} +\end{verbatim} + +\end{itemize} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subsection{Adding a Test Suite} +\label{sec:adding_test_suite} + +To add a test suite to your thorn, devise a series of parameter +files which use as many aspects of your thorn as possible. +Make sure that the parameter files produce ASCII output to files, +and that these files are in the directory +\texttt{./}. + +Run Cactus on each of the parameter files, and move the parameter files, +and the output directories they produced, to the \texttt{test} directory +in your thorn. + +Document carefully any situations or architectures in which your test +suite does not give the correct answers. + +You can also specify options for running your testsuite by adding an +optional configuration file called \texttt{test.ccl} in the \texttt{test} +directory. These are simple text files and may contain comments +introduced by the hash `\texttt{\#}' character, which indicates that the +rest of the line is a comment. If the last non-blank character of a +line in a config file is a backslash `\texttt{$\backslash$}', the +following line is treated as a continuation of the current line. +Options include test specific absolute and relative tolerances, thorn +specific absolute and relative tolerances, the number of procesors required +to run, and file extensions. The configuration file has the form: + +\begin{alltt} +ABSTOL <\var{thorn_absolute_tolerance}> +RELTOL <\var{thorn_relative_tolerance}> +NPROCS <\var{thorn_nprocs}> +EXTENSIONS <\var{extension_1} \var{extension_2} \var{extension_3}> + +TEST <\var{test_example}> +\{ + ABSTOL <\var{absolute_tol}> + RELTOL <\var{relative_tol}> + NPROCS <\var{nprocs}> +\} +\end{alltt} + +which states that when comparing files of test \verb|test_example|, both +\verb|absolute_tol| and \verb|relative_tol| should be used as +the absolute and relative tolerances. For all other tests in the +thorn, the default value of absolute and relative tolerances are set +to \verb|thorn_absolute_tolerance| and \verb|thorn_relative_tolerance|. +The \texttt{NPROCS} option specifies the number of processors required to +run a given testsuite \var{test\_example} or all testsuites of a thorn +successfully. If no \texttt{NPROCS} option is present, the testsuite(s) +is (are) assumed to run with any number of processors. +The \texttt{EXTENSIONS} option adds +\verb|extension_1|, \verb|extension_2| and \verb|extension_3| to the +list of file extensions that are compared. This list is global over +all tests in a configuration. + +Test specific tolerances have precedence over all tolerances, next +come thorn wide tolerances, and then cactus default tolerances. +Absolute and relative tolerances are independent: you can choose to +use test specific absolute tolerance and thorn specific relative +tolerance when running a test. For example, + +\begin{alltt} +TEST \var{test_rad} +\{ + ABSTOL \var{1e-5} +\} + +ABSTOL \var{1e-8} +RELTOL \var{1e-12} +\end{alltt} + +would use an absolute tolerance of $10^{-5}$ and a relative tolerance of +$10^{-12}$ when running \verb|test_rad| and an absolute tolerance of $10^{-8}$ +and a relative tolerance of $10^{-12}$ when running all other tests. + +For details on running the test suites, see Section~\ref{sec:testing}. + +\subsubsection{Best Practices for Test Suites} + +When writing a test suite, there are a few things you should keep in +mind: + +\begin{itemize} +\item The test suite will be run together with many other test + suites. It should, therefore, finish quickly (say, in under two + minutes), and not use too much memory (so that it can run on a + ``normal'' workstation). +\item The test suite will be run automatically, often in situations + where no one checks the screen output. All important output should + be via grid variables that are written to files. Alternatively, if + the test suite tests some low-level infrastructure, it may just + abort the simulation if it fails; that will also be detected. +\item Downloading many files is slow on many systems. A test suite + should normally not have more than, say, hundred output files, and + normally the output files should be small, so that there are not + more than a few Megabytes of output files per test suite. +\item The test suite output files should always be the same. That + means that they should not contain time stamps, etc. It is, + therefore, best to use the option \verb|IO::out_fileinfo="none"|. +\item Norms are unfortunately quite insensitive to changes to a few + grid points only, even if the changes are significant. It is + necessary to output grid point values directly, not only norms. +\item Try to use as few thorns as possible in a test case. For example, do + not active 3D output thorns (unless you use it). The fewer thorns + you use, the easier it is to run the test suite. +\item It is not necessary that a test suite result is ``physically + correct'', or that it uses parameters that ensure a stable time + evolution. A test suite will usually take only a few time steps, so + that a grid size of, e.g.\ $20^3$ grid points \emph{without} + dissipation can be sufficient. Test suites cannot test whether the + result of a simulation is physically feasible; they only test + whether anything changed at all. Ensuring that the physics is still + correct has to be handled by different mechanisms. +\end{itemize} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\section{Advanced Thorn Writing} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subsection{Using Cactus Timers} +\label{sec:timers} + +\subsubsection{What are Timers?} + +%The standard timing information available during a simulation was +%described in Section~\ref{????}. +%[FIXME: This isn't filled in yet because it is being reworked] +Cactus provides a flexible mechanism for timing different sections of your +thorns using various \textit{clocks} which have been registered with the flesh. +By default, the flesh provides two clocks that measure time in different +ways (provided the underlying functionality is available on your system): + +\begin{Lentry} + +\item[\texttt{GetTimeOfDay}] + Provides ``wall clock time'' via the unix \texttt{gettimeofday} function. +\item[\texttt{GetrUsage}] + Provides CPU usage time via the unix \texttt{getrusage} function. +\end{Lentry} + +Additional clocks can be implemented by thorns and registered with the +flesh (see Chapter \ref{chap:clocks}). + +To use Cactus timing, you create a \textit{timer}, which provides time +information for all the registered clocks. + +You can add any number of timers to your thorn source code, providing +each with a name of your choice, and then use Cactus timing functions to +switch on the timers, stop or reset them, and recover timing information. + +Setting the flesh parameter \texttt{cactus::cctk\_timer\_output = "full"} +will cause some summary timing information to be printed at the end of a run. +Some other thorns have their own timer printing parameters as well. + +\subsubsection{Timing Calls} + +Many of the timing calls come in two versions, one whose name ends with +the letter \texttt{I}, and one without. The calls whose names end with the +letter \texttt{I} refer to the timer or clock by index, which is a +non-negative \texttt{int} value; the other calls refer to a timer by name. +If a timer is created without a name, it can be referred to only by its index, +otherwise, it can be referred to by name or by index. + +Typically, a negative return value from a timer function indicates an error. + +\begin{Lentry} + +\item[{\t CCTK\_TimerCreate}, {\t CCTK\_TimerCreateI}] + +Create a timer with a given name, or with no name (respectively) +and return a timer index or an error code. +Negative return values indicate errors. +Only one timer with a given name can exist at any given time. +% what about unnamed timers? % + +\item[{\t CCTK\_TimerDestroy}, {\t CCTK\_TimerDestroyI}] + +Reclaim resources used by a timer. + +\item[{\t CCTK\_TimerStart}, {\t CCTK\_TimerStartI}] + +Start the given timer, using all registered clocks. + +\item[{\t CCTK\_TimerStop}, {\t CCTK\_TimerStopI}] + +Stop the given timer on all registered clocks. + +\item[{\t CCTK\_TimerReset}, {\t CCTK\_TimerResetI}] + +Reset the given timer on all registered clocks. + +\item[{\t CCTK\_TimerCreateData}, {\t CCTK\_TimerDestroyData}] + +Allocate and reclaim (respectively) resources for a {\t cTimerData} structure, +which will be used to hold clock values. + +\item[{\t CCTK\_Timer}, {\t CCTK\_TimerI}] + +Fill the given {\t cTimerData} structure with clock values as of +the last call to {\t CCTK\_TimerStop}. + +\item[{\t CCTK\_NumTimers}] + +Return the number of created timers + +\item[{\t CCTK\_TimerName}] + +Return the name of the timer for a given timer index (or {\t NULL} if +the timer is unnamed or any other error occurs). + +\item[{\t CCTK\_NumTimerClocks}] + +Take a pointer to {\t cTimerData} and return the number of clocks +recorded in a timer measurement + +\item[{\t CCTK\_GetClockValue}, {\t CCTK\_GetClockValueI}] + +Given a clock referred to by name or index, respectively, and a +{\t cTimerData} pointer, return a {\t cTimerVal} pointer representing +the value of the clock when the timer was stopped + +\item[{\t CCTK\_TimerClockName}] + +Return the name of the clock given by the {\t cTimerVal} pointer argument. + +\item[{\t CCTK\_TimerClockResolution}] + +Return the floating-point value of the resolution in seconds +of the clock referred to by the {\t cTimerVal} pointer argument. +This is a lower bound for the smallest non-zero difference in values +between calls of {\t CCTK\_TimerClockSeconds}. + +\item[{\t CCTK\_TimerClockSeconds}] + +Return the floating-point value of the measurement in seconds +from the {\t cTimerVal} pointer argument. + +\end{Lentry} + +\subsubsection{How to Insert Timers in your Code} + +The function prototypes and structure definitions are contained in the +include file \texttt{cctk\_Timers.h}, which is included in the standard +thorn header file \texttt{cctk.h}. At the moment, the timer calls are only +available from C. + +The following example, which uses a timer to +instrument a section of code, illustrates how timers are used by +application thorns. A working example is available in the thorn +\texttt{CactusTest/TestTimers}. + +{\bf Creating a timer} + +The first action for any timer is to create it, using +\texttt{CCTK\_TimerCreate}. +This can be performed at any time prior to use of the timer: + +\begin{verbatim} +#include "cctk_Timers.h" +index = CCTK_TimerCreate("TimeMyStuff"); +\end{verbatim} + +{\bf Instrumenting a section of code} + +Code sections are instrumented using the Start, Stop and Reset functions. These +functions are applied to the chosen timer using all the registered clocks. +\begin{verbatim} +ierr = CCTK_TimerStart("TimeMyStuff"); +do_procedure_to_be_timed(); +ierr = CCTK_TimerStop("TimeMyStuff"); +\end{verbatim} + +{\bf Accessing the timer results} + +After calling {\t CCTK\_TimerStop}, you then get the time value from +any of the registered clocks. + +The procedure is to allocate a {\t cTimerData} structure, +and read the information from your timer into this structure +using {\t CCTK\_Timer}, then to extract time data of the desired clock from +this structure. After using the structure, you should destroy it. + +\begin{verbatim} + cTimerData *info = CCTK_TimerCreateData(); + int ierr = CCTK_Timer("TimeMyStuff",info); + + /* You can refer to a particular clock by name. */ + const cTimerVal *clock = CCTK_GetClockValue( "gettimeofday", info ); + if( clock ){ + printf ("\t%s: %.3f %s\n", "gettimeofday", + CCTK_TimerClockSeconds( clock ), "secs" ); + } + + /* To get results from all available clocks, refer to them by index.*/ + nclocks = CCTK_NumTimerClocks( info ); + + for (i = 0; i < numclocks; i++) { + const cTimerVal *clock = CCTK_GetClockValueI( i, info ); + printf ("\t%s: %.3f %s\n", CCTK_TimerClockName( clock ), + CCTK_TimerClockSeconds( clock ), "secs" ); + } + CCTK_TimerDestroyData (info); +\end{verbatim} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subsection{Include Files} +\label{sec:includefiles} + +Cactus provides a mechanism for thorns to add code to +include files which can be used by any other thorn. +Such include files can contain executable source code, or header/declaration +information. A distinction is made between these two cases, since included +executable code is protected from being run if a thorn is compiled, but +not active by being wrapped by a call to \texttt{CCTK\_IsThornActive}. + +Any thorn +which uses the include file must declare this in its +\texttt{interface.ccl} with the line + +\begin{alltt} +USES INCLUDE [SOURCE|HEADER]: <\var{file_name}> +\end{alltt} + +(If the optional \verb![SOURCE|HEADER]! is omitted, \verb|HEADER| is +assumed. Note that this can be dangerous, as included \emph{source} +code, which is incorrectly assumed to be \emph{header} code, will be +executed in another thorn \emph{even if the providing thorn is +inactive}. Thus, it is recommended to always include the optional +\verb![SOURCE|HEADER]! specification.) Any thorn that wishes to add +to this include file, declares in its own \texttt{interface.ccl} + +\begin{alltt} +INCLUDE [SOURCE|HEADER]: <\var{file_to_include}> in <\var{file_name}> +\end{alltt} + +\paragraph{Example} + +As an example of this in practice, for the case of Fortran code, +consider a thorn A, which +wants to gather terms for a calculation from any thorn +that wishes to provide them. Thorn A could have +the lines in its source code + +\begin{verbatim} +c Get source code from other thorns + allterms = 0d0 +#include "AllSources.inc" +\end{verbatim} +and would then add to \texttt{interface.ccl} the line +\begin{verbatim} +USES INCLUDE SOURCE: AllSources.inc +\end{verbatim} + +If thorn B wants to add terms for the calculation, it would +create a file, say \texttt{Bterms.inc} with the lines +\begin{verbatim} +c Add this to AllSources.inc + allterms = allterms + 1d0 +\end{verbatim} +and would add to its own \texttt{interface.ccl} + +\begin{verbatim} +INCLUDE SOURCE: Bterms.inc in AllSources.inc +\end{verbatim} + +The final file for thorn A which is compiled, will contain the code +\begin{verbatim} +c Get source code from other thorns + allterms = 0d0 + if (CCTK_IsThornActive("B").ne.0) then +c Add this to AllSources.inc + allterms = allterms + 1d0 + end if +\end{verbatim} + +Any Fortran thorn routines which include source code must include +the declaration \texttt{DECLARE\_CCTK\_FUNCTIONS}. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subsection{Memory Tracing} +\label{sec:metr} + +Cactus provides a mechanism for overriding the standard C memory +allocation routines (\texttt{malloc, free,} \ldots) with Cactus specific +routines that track the amount of memory allocated, and from where, the +allocation call was made. This information can be accessed by the user +to provide an understanding of the memory consumption between two +instances, and to track down possible memory leaks. This feature is +available in C only. + +\subsubsection{Activating Memory Tracing} +\label{sec:acmetr} + +Memory tracing has to be activated at configure time. The standard +\texttt{malloc} statements are overridden with macros (\texttt{CCTK\_MALLOC}). To +activate memory tracing use either + +\begin{Lentry} +\item[\texttt{DEBUG=all}] Enables all debug options (compiler debug +flags, redefines \texttt{malloc}) +\item[\texttt{DEBUG=memory}] Redefine \texttt{malloc} only. +\end{Lentry} + +The \texttt{CCTK\_MALLOC} statements can also be used directly in the C +code. But by employing them this way, only a fraction of the total +memory consumption is traced. Also, they cannot be turned off at +configure time. For example: +\begin{verbatim} +machine> gmake bigbuild DEBUG=yes + +machine> gmake bigbuild-config DEBUG=memory +\end{verbatim} +The new configuration \texttt{bigbuild} is configured with all debugging +features turned on. The already existing configuration \texttt{bigbuild} +is reconfigured with memory tracing only. + +\subsubsection{Using Memory Tracing} +\label{sec:usmetr} + +You can request Cactus to store the memory consumption at a certain +instance in the program flow, and return the difference in memory +allocation some time later. + +\begin{Lentry} +\item[\texttt{int CCTK\_MemTicketRequest(void)}] + Request a ticket: save the current total memory to a database. + Return an integer (ticket). Use the ticket to calculate the + difference in memory allocation between the two instances in + \texttt{CCTK\_MemTicketCash}. + +\item[\texttt{long int CCTK\_MemTicketCash(int your\_ticket)}] + Cash in your ticket: return the memory difference between now and the + time the ticket was requested. Tickets can be cashed in + several times. See example below. + This only tracks the real data memory, which is the same as in + undebug mode. It does not keep track of the internal allocations + done to provide the database, the motivation is that this is not + allocated either if you compile undebugged. + +\item[\texttt{int CCTK\_MemTicketDelete(int your\_ticket)}] + Delete the memory ticket. The ticket ID will not be reused, since + it's incremented with every ticket request, but the memory of + the memory datastructure is deallocated. + +\item[\texttt{unsigned long int CCTK\_TotalMemory(void)}] + Returns the total allocated memory (not including the tracing + data structures). + +\item[\texttt{void CCTK\_MemStat}] Prints an info string, stating the current, + past, and total memory (in bytes) allocation between two + successive calls to this routine, as well as the difference. +\end{Lentry} + +Sample C code demonstrating the ticket handling. Two tickets are +requested during \texttt{malloc} operations. The \texttt{CCTK\_MALLOC} statement is +used directly. They are cashed in, and the memory +difference is printed. Ticket 1 is cashed twice. The tickets are +deleted at the end. +\begin{verbatim} + +int ticket1; +int ticket2; + +/* store current memstate, ticket: t1*/ +t1 = CCTK_MemTicketRequest(); + +/* allocate data */ +hi = (int*) CCTK_MALLOC(10*sizeof(int)); + +/* store current memstate, ticket: t2*/ +t2 = CCTK_MemTicketRequest(); + +/* cash ticket t1, print mem difference */ +printf("NOW1a: %+d \n",CCTK_MemTicketCash(t1)); + +/* allocte some more data */ +wo = (CCTK_REAL*)CCTK_MALLOC(10*sizeof(CCTK_REAL)); + +/* cash ticket t1 and t2, print mem difference */ +printf("NOW1b: %+d \n",CCTK_MemTicketCash(t1)); +printf("NOW2 : %+d \n",CCTK_MemTicketCash(t2)); + +/* delete the tickets from the database */ +CCTK_MemTicketDelete(t1); +CCTK_MemTicketDelete(t2); + +\end{verbatim} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subsection[Calls to different language]{Calls between Different Programming Languages} +%\pagestyle{empty} + +\subsubsection{Calling C Routines from Fortran} +\label{sec:cacrofr} + +To make the following C routine, + +{\tt +int ()\\ +{\\ +...\\ +} +} + +also callable from Fortran, a new routine must be added, which is +declared using the \texttt{CCTK\_FCALL} and \texttt{CCTK\_FNAME} macros: + +{\tt +void CCTK\_FCALL CCTK\_FNAME()(int *ierr, )\\ + +} + +The convention used in Cactus, is that \texttt{} be the same as any +C routine name, and that this is mixed-case. The macros change +the case and number of underscores of the routine name to match that expected +by Fortran. + +All arguments passed by Fortran to the routine (except strings) are +pointers in C, e.g.\ a call from Fortran + +\begin{verbatim} +CCTK_INT arg1 +CCTK_REAL arg2 +CCTK_REAL arg3(30,30,30) + +... + +call MyCRoutine(arg1,arg2,arg3) +\end{verbatim} + +should appear in C as + +\begin{verbatim} +void CCTK_FCALL CCTK_FNAME(MyCRoutine)(CCTK_INT *arg1, + CCTK_REAL *arg2, + CCTK_REAL *arg3) +{ +... +} +\end{verbatim} + +\subsubsection{String Arguments from Fortran} + +Fortran passes string arguments in a special, compiler-dependent, way. +To facilitate this, the CCTK provides a set of macros to enable the +translation to C strings. +The macros are defined in \texttt{cctk\_FortranString.h}, which +should be included in your C file. + +String arguments \emph{must always come last} in the argument list for +these macros to be effective (some Fortran compilers automatically +migrate the strings to the end, so there is no portable workaround). + +The macros to use depend upon the number of string arguments--we +currently support up to three. The macros are +\texttt{\_FORTSTRING\_ARG}. +Corresponding to each of these are two macros +\texttt{\_FORTSTRING\_CREATE} and +\texttt{\_FORTSTRING\_PTR}, +which take one, two, or three arguments depending on the number of strings. +The latter set is only necessary if a string is to be modified. +In more detail: + +\begin{Lentry} + +\item[\texttt{\_FORTSTRING\_ARG}] + Used in the argument list of the C routine to which the Fortran + strings are passed. + +\item[\texttt{\_FORTSTRING\_CREATE}] + Used in the declaration section of the C routine to which the Fortran + strings are passed. These macros have one, two or three arguments + which are the variable names you choose to use for the strings in + the C routine, created + by null-terminating the passed-in Fortran strings. The \texttt{CREATE} + macros create new strings with the names you provide, and thus should + be treated as read-only and freed after use. + +\item[\texttt{\_FORTSTRING\_PTR}] + These macros, used in the declaration section of the C routine + \emph{after} the \texttt{CREATE} macro, + should be used if you need to modify one of the passed-in strings. + They declare and define pointers to the passed-in strings. + +\item[\texttt{cctk\_strlen<1|2|3>}] These integer variables, + automatically defined by + the \texttt{CREATE} macro, hold the lengths of the passed in + Fortran strings. +\end{Lentry} + +The use of the macros is probably best explained with examples. +For read-only access to the strings, only the first two macros are needed, +the following example compares two strings passed in from Fortran. + +\begin{verbatim} +#include +#include +#include + +int CCTK_FCALL CCTK_FNAME(CompareStrings)(TWO_FORTSTRING_ARG) +{ + int retval; + + /* Allocate and create C strings with \0 at end. */ + /* This makes variable declarations, so it must be before + any executable statements.*/ + + TWO_FORTSTRING_CREATE(arg1,arg2) + + /* Do some work with the strings */ + retval = strcmp(arg1,arg2); + + /* Important, these must be freed after use */ + free(arg1); + free(arg2); + + return retval; +} + +\end{verbatim} + +Since the null terminated strings may be copies of the strings passed +from Fortran, they should be treated as read-only. + +To change the data in a string passed from Fortran, you need to use +the \texttt{FORTSTRING\_PTR} macros, which declare and set up pointers +to the strings passed from C. Note that this macro must be used +\emph{after} the \texttt{FORTSTRING\_CREATE} macro. For example, the +following routine copies the contents of the second string to the +first string + +\begin{verbatim} +#include +#include +#include + +int CCTK_FCALL CCTK_FNAME(CopyStrings)(TWO_FORTSTRING_ARG) +{ + int retval; + + /* Allocate and create C strings with \0 at end. */ + /* This makes variable declarations, so it must be before + any executable statements. */ + + TWO_FORTSTRING_CREATE(arg1,arg2) + TWO_FORTSTRING_PTR(farg1,farg2) + + /* Do some work with the strings */ + + retval = strncpy(farg1,arg2,cctk_strlen1); + + /* Important, these must be freed after use */ + free(arg1); + free(arg2); + + return retval; +} + +\end{verbatim} + +Note that in the example above, two new variables, pointers to the +Fortran strings, were created. These are just pointers and +\emph{should not be freed}. The example also illustrates the +automatically-created variables, e.g.\ \texttt{cctk\_strlen1}, +which hold the sizes of original Fortran strings. +When writing to a string its length should never be exceeded. + +\subsubsection{Calling Fortran Routines from C} +\label{sec:caforofr} + +To call a utility Fortran routine from C, use +\begin{alltt} +void CCTK_FCALL CCTK_FNAME(<\var{Fortran routine name}>)(<\var{argument list}>) +\end{alltt} + +Note that Fortran expects all arguments (apart from strings) to be +pointers, so any non-array data should be passed by address. + +Currently, we have no support for calling Fortran routines which expect +strings from C. However, passing routines is supported when you use function aliasing, see Section \ref{sec:function_aliasing}. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subsection{Function aliasing} +\label{sec:function_aliasing} + +Like calling functions in a different language, Cactus offers a +mechanism for calling a function in a different thorn where you don 't need +to know which thorn is actually providing the function, nor what +language the function is provided in. The idea of \textit{function +aliasing} is similar to that of thorns; the routine that calls a +function should not need to know anything about it, except that the +function exists. + +Function aliasing is quite restrictive, because of the problems +involved in inter-language calling, as seen in the previous +section. Function aliasing is also comparatively inefficient, and +should not be used in a part of your code where efficiency is +important. + +Function aliasing is language-neutral, however, the syntax is strongly +based on C. In the future, the function aliasing +declarations may go into a new \texttt{functions.ccl} file, and will +have a format more similar to that of variable group and parameter +declarations. + +\subsubsection{Using an Aliased Function} + +To use an aliased function you must first declare it in your +\texttt{interface.ccl} file. Declare the prototype as, for example, +\begin{verbatim} +CCTK_REAL FUNCTION SumStuff(CCTK_REAL IN x, CCTK_REAL IN y) +\end{verbatim} +\noindent and that this function will be either required in your thorn by +\begin{verbatim} +REQUIRES FUNCTION SumStuff +\end{verbatim} +\noindent or optionally used in your thorn by +\begin{verbatim} +USES FUNCTION SumStuff +\end{verbatim} +A prototype of this function will be available to any C routine that +includes the \texttt{cctk.h} header file. +In a Fortran file, the declaration of the function will be included +in the \texttt{DECLARE\_CCTK\_FUNCTIONS} macro, which is available after +the statement \texttt{\#include "cctk\_Functions.h"}. +The keywords \texttt{IN}, \texttt{OUT}, and \texttt{INOUT} +work in the same fashion as \texttt{INTENT} statements in Fortran 90. +That is, the C prototype will expect an argument with intent \texttt{IN} +to be a value and one with intent \texttt{OUT} or \texttt{INOUT} to be a +pointer. There also exists the \texttt{ARRAY} keyword for passing arrays +of any dimension. +Functions which are required by some thorn (which doesn't provide it itself) +are checked at startup to be provided by some other thorn. + + +\subsubsection{Providing a Function} + +To provide an aliased function you must again add the prototype to +your \texttt{interface.ccl} file. A statement containing the name of the +providing function and the language it is provided in, must also be +given. For example, +\begin{verbatim} +CCTK_REAL FUNCTION SumStuff(CCTK_REAL IN x, CCTK_REAL IN y) +PROVIDES FUNCTION SumStuff WITH AddItUp LANGUAGE C +\end{verbatim} +The appropriate function must then be provided somewhere in this +thorn. Multiple thorns providing the same function can be compiled into the +same configuration; however, only one providing thorn may be activated at +runtime, otherwise, an error message is printed and the run is aborted. + +It is necessary to specify the language of the providing function; no +default will be assumed. + + +\subsubsection{Conventions and Restrictions} + +Various restrictions are necessary to make function aliasing +work. These are +\begin{itemize} +\item The return type of any function must be either \texttt{void} or one + of the Cactus data types \texttt{CCTK\_INT} or {\tt + CCTK\_REAL}. Standard types such as \texttt{int} are not allowed. +\item The type of an argument must be one of scalar types {\tt + CCTK\_INT, CCTK\_REAL, CCTK\_COMPLEX, CCTK\_STRING, CCTK\_POINTER, + CCTK\_FPOINTER}, or an array or pointer type \texttt{CCTK\_INT ARRAY, + CCTK\_REAL ARRAY, CCTK\_COMPLEX ARRAY, \texttt{CCTK\_POINTER ARRAY}}.%%% +\footnote{%%% + Unfortunately, neither \texttt{CCTK\_FPOINTER ARRAY}, + nor \texttt{CCTK\_STRING ARRAY} will work. + }%%% + {} The scalar types are assumed to be not + modifiable. Any changes made to a scalar argument by a providing + function may be silently lost, or may not; it is dependent on the + language of the providing and calling function. If you wish to + modify an argument, then it must have intent \texttt{OUT} or \texttt{INOUT} + (and hence must be either a \texttt{CCTK\_INT}, a + \texttt{CCTK\_REAL}, or a \texttt{CCTK\_COMPLEX}, + or an array of one of these types). +\item The name of both, the aliased and providing function, are + restricted. They must follow the standard C semantics (start with a + letter, contain only letters, numbers or underscores). Additionally, + they must be mixed case (that is, contain at least one uppercase and + one lowercase letter). The names of the aliased and providing + functions must be distinct. +\item If an argument is a function pointer, then the syntax looks like +\begin{verbatim} +CCTK_REAL Integrate(CCTK_REAL CCTK_FPOINTER func(CCTK_REAL IN x), \ + CCTK_REAL IN xmin, CCTK_REAL IN xmax) +\end{verbatim} + It is assumed that the function pointer argument has the same + language as the calling function. Function pointer arguments may not + be nested. +\item \texttt{CCTK\_STRING} arguments follow the same conventions as in + the previous section. That is, they must appear at the end of the + argument list, there must be at most three, and a function with a + \texttt{CCTK\_STRING} argument can only be provided in C, not Fortran, + although they may be called from either. +\end{itemize} + + +\subsubsection{Examples} + +\begin{itemize} +\item A C function is provided to add together two real numbers. The {\tt + interface.ccl} should read +\begin{verbatim} + CCTK_REAL FUNCTION SumStuff(CCTK_REAL IN x, CCTK_REAL IN y) + PROVIDES FUNCTION SumStuff WITH AddItUp LANGUAGE C + USES FUNCTION SumStuff +\end{verbatim} + +\item A Fortran function is provided to invert a real number. The {\tt + interface.ccl} should read +\begin{verbatim} + SUBROUTINE Invert(CCTK_REAL INOUT x) + PROVIDES FUNCTION Invert WITH FindInverse LANGUAGE Fortran + USES FUNCTION Invert +\end{verbatim} +\noindent Note that \texttt{SUBROUTINE} has the same meaning as \texttt{void + FUNCTION}. + +\item A Fortran function is provided to integrate any function over + an interval. The \texttt{interface.ccl} should read +\begin{verbatim} + CCTK_REAL Integrate(CCTK_REAL CCTK_FPOINTER func(CCTK_REAL IN x), \ + CCTK_REAL IN xmin, CCTK_REAL IN xmax) + PROVIDES FUNCTION Integrate WITH SimpsonsRule LANGUAGE Fortran + USES FUNCTION Integrate +\end{verbatim} + +\end{itemize} + +\subsubsection{Testing Aliased Functions} + +The calling thorn does not know if an aliased function is even +provided by another thorn. Calling an aliased function that has not +been provided, will lead to a level 0 warning message, stopping the +code. In order to check if a function has been provided by some thorn, +use the \texttt{CCTK\_IsFunctionAliased} function described in the +function reference section. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subsection{Naming Conventions} + +\begin{itemize} + +\item{} Thorn names must not start with the word ``Cactus'' (in + any case). +\item{} Arrangements will be ignored if their names start with a + hash mark `\texttt{\#}' or dot `\texttt{.}', + or end with a tilde `\texttt{\~{}}', \texttt{.bak} or \texttt{.BAK}. +\item{} Thorns will be ignored if they are called doc or start with a + hash mark `\texttt{\#}' or dot `\texttt{.}', + or end with a tilde `\texttt{\~{}}', \texttt{.bak} or \texttt{.BAK}. +\item{} Routine names have to be unique among all thorns. + +\end{itemize} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subsection{General Naming Conventions} + +The following naming conventions are followed by the flesh and the +supported Cactus arrangements. They are not compulsory, but if followed, will +allow for a homogeneous code. + +\begin{itemize} + +\item Parameters: lower case (except for acronyms) with words separated + by an underscore. Examples: \texttt{my\_first\_parameter}, + \texttt{solve\_PDE\_equation}. + +\item Filenames and routine names: Prefixed by thorn name with an underscore, then capitalised words, with no spaces. + Examples: \texttt{MyThorn\_StartUpRoutine}, \texttt{BestSolver\_InitialDataForPDE}. + +\end{itemize} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subsection{Data Types and Sizes} +\label{sect-ThornWriting/DataTypes} +\label{sec:datyansi} + +Cactus knows about the following fixed size data types: + +\begin{center} +\begin{tabular}{|l|l|l|l|} +\hline +Data Type & Size (bytes) & Variable Type & Fortran Equivalent\\ +\hline +{\t CCTK\_BYTE} & 1 & {\t CCTK\_VARIABLE\_BYTE} & {\t integer*1}\\ +{\t CCTK\_INT1} & 1 & {\t CCTK\_VARIABLE\_INT1} & {\t integer*1}\\ +{\t CCTK\_INT2} & 2 & {\t CCTK\_VARIABLE\_INT2} & {\t integer*2}\\ +{\t CCTK\_INT4} & 4 & {\t CCTK\_VARIABLE\_INT4} & {\t integer*4}\\ +{\t CCTK\_INT8} & 8 & {\t CCTK\_VARIABLE\_INT8} & {\t integer*8}\\ +{\t CCTK\_REAL4} & 4 & {\t CCTK\_VARIABLE\_REAL4} & {\t real*4}\\ +{\t CCTK\_REAL8} & 8 & {\t CCTK\_VARIABLE\_REAL8} & {\t real*8}\\ +{\t CCTK\_REAL16} & 16 & {\t CCTK\_VARIABLE\_REAL16} & {\t real*16}\\ +{\t CCTK\_COMPLEX8} & 8 & {\t CCTK\_VARIABLE\_COMPLEX8} & {\t complex*8}\\ +{\t CCTK\_COMPLEX16} & 16 & {\t CCTK\_VARIABLE\_COMPLEX16} & {\t complex*16}\\ +{\t CCTK\_COMPLEX32} & 32 & {\t CCTK\_VARIABLE\_COMPLEX32} & {\t complex*32}\\ +\hline +\end{tabular} +\end{center} + +The availability of these types, and the corresponding +C data types, are platform-dependent. For each fixed-size data type, there exists +a corresponding preprocessor macro {\t HAVE\_}, which should be used +to check whether the given CCTK data type is supported, e.g.\ + +\begin{verbatim} + /* declare variable with extended-precision complex data type if available, + otherwise, with default CCTK precision */ + #ifdef HAVE_CCTK_COMPLEX32 + CCTK_COMPLEX32 var; + #else + CCTK_COMPLEX var; + #endif +\end{verbatim} + +In addition, Cactus provides three generic numeric data types which map onto +the compilers' native data types used to represent integer, real, and complex +values. The size for these generic types can be chosen at configuration time +(see Section \ref{subsec:Compilation-Available_Options}). This is to allow the code to +be run easily at different precisions. Note that the effectiveness of running +the code, at a lower or higher precision, depends crucially on all thorns being +used making consistent use of the these generic data types: + +\begin{center} +\begin{tabular}{|l|l|l|l|} +\hline +Data Type & Variable Type & Configuration Option\\ +\hline +{\t CCTK\_INT} & {\t CCTK\_VARIABLE\_INT} & {\t INTEGER\_PRECISION}\\ +{\t CCTK\_REAL} & {\t CCTK\_VARIABLE\_REAL} & {\t REAL\_PRECISION}\\ +{\t CCTK\_COMPLEX} & {\t CCTK\_VARIABLE\_COMPLEX} & Same as real precision\\ +\hline +\end{tabular} +\end{center} + + +These variable types must be used by thorn writers to declare variables + in the thorn interface files, and may be used to declare +variables in the thorn routines. Note that variable declarations in +thorns should obviously match the definitions in the interface files +where appropriate. + +A set of macros, which +are interpreted by the preprocessor at compile time, to signify which +data size is being used, are also provided: + +\begin{center} +\begin{tabular}{|l|l|} +\hline +Data Type & {\t \#define}\\ +\hline +{\t CCTK\_INT1} & {\t CCTK\_INT\_PRECISION\_1} \\ +{\t CCTK\_INT2} & {\t CCTK\_INT\_PRECISION\_2} \\ +{\t CCTK\_INT4} & {\t CCTK\_INT\_PRECISION\_4} \\ +{\t CCTK\_INT8} & {\t CCTK\_INT\_PRECISION\_8} \\ +{\t CCTK\_REAL4} & {\t CCTK\_REAL\_PRECISION\_4} \\ +{\t CCTK\_REAL8} & {\t CCTK\_REAL\_PRECISION\_8} \\ +{\t CCTK\_REAL16} & {\t CCTK\_REAL\_PRECISION\_16} \\ +{\t CCTK\_COMPLEX8} & {\t CCTK\_COMPLEX\_PRECISION\_8} \\ +{\t CCTK\_COMPLEX16} & {\t CCTK\_COMPLEX\_PRECISION\_16} \\ +{\t CCTK\_COMPLEX32} & {\t CCTK\_COMPLEX\_PRECISION\_32} \\ +\hline +\end{tabular} +\end{center} + +Cactus also provides generic data and function pointers, which can +be used from either C or Fortran: + +\begin{center} +\begin{tabular}{|l|l|l|} +\hline + Data Type & + Variable Type & + C equivalent +\\\hline + {\t CCTK\_POINTER} & + {\t CCTK\_VARIABLE\_POINTER} & + {\t void *data\_ptr} +\\ + {\t CCTK\_POINTER\_TO\_CONST} & + {\t CCTK\_VARIABLE\_POINTER\_TO\_CONST} & + {\t const void *data\_ptr} +\\ + {\t CCTK\_FPOINTER} & + {\t CCTK\_VARIABLE\_FPOINTER} & + {\t void (*fn\_ptr)(void)} +\\\hline +\end{tabular} +\end{center} + + +\subsubsection{Fortran Thorn Writers} + +Cactus provides the data types \texttt{CCTK\_POINTER} and +\texttt{CCTK\_POINTER\_TO\_CONST} for use in Fortran code to declare a +pointer passed from C. For example, the variable \texttt{cctkGH} is +of the type \texttt{CCTK\_POINTER}. The data type +\texttt{CCTK\_STRING} is, in Fortran, also an opaque type; it corresponds +to a C pointer, and one has to use the function +\texttt{CCTK\_FortranString} to convert it to a Fortran string, or the +\texttt{CCTK\_Equals} to compare it to a Fortran String. + +Since the data types, \texttt{integer} in Fortran and \texttt{int} in C, +may be different\footnote{This is only a theoretical possibility, in practice, they + have to be the same type for Cactus to work at all.}, +many routines that can be called from both, C and Fortran, take +arguments of the type \texttt{CCTK\_INT}. This type can be different +from the type \texttt{integer}. Fortran does not convert routine +arguments automatically, and it is, therefore, necessary to pay +attention to the exact argument types that a routine expects, and to +convert between \texttt{integer} and \texttt{CCTK\_INT}, accordingly. +Currently, most flesh functions take \texttt{integer} arguments, while +all aliased functions take \texttt{CCTK\_INT} arguments. + +\begin{quote} + NOTE: If you make errors in passing Fortran arguments, and if there + are no interfaces (``prototypes'') available for the routines that + are called, then the compiler cannot detect these errors. Be + careful, when you write Fortran code yourself, consider placing + routines in modules, which implicitly define interfaces for all + contained routines. +\end{quote} + +There are two convenient ways to convert between these types. An easy +way, is to define parameters or to declare variables of the desired +type, assign a value to these parameters or variables, and then pass +the parameter or value. This makes for very readable code, since the +name of the parameter or variable serves as additional documentation: + +\begin{verbatim} +CCTK_INT, parameter : jtwo = 2 +integer :: vindex_gxx, vindex_kxx +CCTK_INT :: syncvars(jtwo) + +call CCTK_VarIndex (vindex_gxx, "ADMBase::gxx") +call CCTK_VarIndex (vindex_kxx, "ADMBase::kxx") + +syncvars(1) = vindex_gxx +syncvars(2) = vindex_kxx +call CCTK_SyncGroupsI (cctkGH, jtwo, syncvars) +\end{verbatim} + +(You have probably seen the strange Fortran convention, where people +introduce constants named \texttt{zero} or \texttt{two}---it is a convenient way to make sure that the +constant has the correct type.) + +Another possibility are explicit type conversions. They can rather +easily be added to existing code: + +\begin{verbatim} +! Boilerplate code to determine type kinds +integer, parameter :: izero = 0 ! A dummy variable of type integer +CCTK_INT, parameter :: jzero = 0 ! A dummy variable of type CCTK_ITN +integer, parameter :: ik = kind (izero) ! The kind of "integer" +integer, parameter :: jk = kind (jzero) ! The kind of "CCTK_INT" + +integer :: syncvars(2) + +call CCTK_VarIndex (syncvars(1), "ADMBase::gxx") +call CCTK_VarIndex (syncvars(2), "ADMBase::kxx") + +call CCTK_SyncGroupsI (cctkGH, int(2,jk), int(syncvars,jk)) +\end{verbatim} + +Fortran distinguishes between different integer \emph{kinds}. These +kinds are what is different between \texttt{integer} and +\texttt{CCTK\_INT}. The expression \texttt{int(EXPR,KIND)} converts +\texttt{EXPR} to an integer of kind \texttt{KIND}. Above, we use the +convention that the prefix \texttt{i} denotes things having to do with +\texttt{integer}, and the prefix \texttt{j} denotes +\texttt{CCTK\_INT}. + +Note that we declare the array \texttt{syncvars} with the type that is +necessary to set its values. Type conversions are only possible if +variables are read, not when they are written to. + + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\section{Telling the Make system What to Do} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subsection{Basic Recipe} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subsection{Make Concepts} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subsection{The Four Files} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subsection{How your code is built} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + diff --git a/doc/UsersGuide/GettingStarted.tex b/doc/UsersGuide/GettingStarted.tex new file mode 100644 index 00000000..2c8558a1 --- /dev/null +++ b/doc/UsersGuide/GettingStarted.tex @@ -0,0 +1,383 @@ +% /*@@ +% @file RunningCactus.tex +% @date 27 Jan 1999 +% @author Tom Goodale, Gabrielle Allen, Gerd Lanferman, Thomas Radke +% @desc +% How to run Cactus part of the Cactus User's Guide +% @enddesc +% @version $Header$ +% @@*/ + +\begin{cactuspart}{Introduction}{$RCSfile$}{$Revision$}\label{part:Introduction} +\renewcommand{\thepage}{\Alph{part}\arabic{page}} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\chapter{Getting Started} +\label{cha:gs} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\section{Obtaining Cactus} +\label{sec:checkout} + +Cactus is distributed, extended, and maintained using the free CVS +software (\textit{Concurrent Versions System}: \url{http://www.cvshome.org}). +CVS allows many people to work on a large software project +together without getting into a tangle. +Since Cactus thorns are distributed from several repositories on the +main CVS site, and from a growing number of user sites, we provide a +\texttt{GetCactus} script on our website for checking out the flesh +and thorns. The script is available at + +\url{http://www.cactuscode.org/download/GetCactus}. + +The +script takes as an argument the name of a file containing a \textit{ThornList}, +that is a list of thorns with the syntax +\begin{alltt} +<\var{arrangement name}>/<\var{thorn name}> +\end{alltt} + +If no filename is given, only the flesh is checked out. +Optional directives in the ThornList indicate which CVS repository to fetch +thorns from. The default is to take the thorns from the same repository as +the flesh. A full description of ThornList syntax is provided in Appendix~\ref{chap:th}. +ThornLists for example applications are provided on the Cactus website at + +%%% FIXME: url + +The same script can be used to checkout additional thorns, or to update +existing ones. + +The Cactus website also provides a form interface for direct download at: + +\url{http://www.cactuscode.org/cactus\_cgi-bin/download.pl} + +CVS experts who want to use raw CVS commands are directed to +Appendix~\ref{sec:Appendix.cvs} for full instructions. For CVS novices, +we also summarize in the Appendix basic CVS commands. + +The space required for an installation depends on the arrangements and +thorns used. The flesh on its own requires less than 5 MB. + +Another script, \texttt{MakeThornList}, can be used to produce a minimal +ThornList from a given Cactus par file. It needs a \emph{master} ThornList +to be copied into your \texttt{Cactus} directory. + +See \url{http://www.cactuscode.org/toolkit/makeThornList/}. + + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subsection{Directory Structure} +\label{sec:dist} + +A fresh checkout creates a directory \texttt{Cactus} with the +following subdirectories: + +\begin{Lentry} + +\item[\texttt{CVS}] the CVS bookkeeping directory, present in every subdirectory + +\item[\texttt{doc}] Cactus documentation + +\item[\texttt{lib}] contains libraries + +\item[\texttt{src}] contains the source code for Cactus + +\item [\texttt{arrangements}] contains the Cactus arrangements. The arrangements + (the actual ``physics'') are not supplied by just checking out just Cactus. + If the arrangements you want to use are standard Cactus arrangements, or + reside on our CVS repository (\texttt{cvs.cactuscode.org}), + they can be checked out in similar way to the flesh. +\end{Lentry} + +When Cactus is first compiled, it creates a new directory +\texttt{Cactus/configs}, which will contain all the source code, object files +and libraries created during the build process. + +Configurations are described in detail in Section \ref{sec:configurations}. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\section{Compiling a Cactus application} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + + +Cactus can be built in different configurations from the same copy of +the source files, and these different configurations coexist in the +\texttt{Cactus/configs} directory. Here are several instances in which + this can be useful: + +\begin{enumerate} +\item{}Different configurations can be for \emph{different +architectures}. You can keep executables for multiple architectures +based on a single copy of source code, shared on a common file +system. +\item{} You can compare different \textit{compiler options}, and \textit{debug-modes}. + You might want to compile different communication protocols + (e.g. MPI or Globus), or leave them out all together. +\item{} You can have different configurations for \textit{different thorn + collections} compiled into your executable. +\end{enumerate} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +\subsection{Creating a Configuration} +\label{sec:configurations} + +At its simplest, this is done by \texttt{gmake <\var{config}>}. +This generates a +configuration with the name \texttt{\var{config}}, doing its best to +automatically determine the default compilers and compilation flags +suitable for the current architecture. + +There are a number of additional command-line arguments which may be supplied +to override some parts of the procedure; they are listed in Section~\ref{sec:confopts}. + +Once you have created a new configuration, the command +\\ \\ +\texttt{gmake <\var{configuration name}>} +\\ \\ +will build an executable, prompting you along the way for the +thorns which should be included. There is a range of \texttt{gmake} +targets and options which are detailed in Section~\ref{sec:gmtafobuanadco}. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\section{Running a Cactus application} + +Cactus executables always run from a parameter file (which may be +provided as a command-line argument taken from standard input), which +specifies which thorns to use and sets the values of each thorn's +parameters (the parameters that are not set will take on default values, +see~\ref{sec:Appendix.param}). + +There is no restriction on the name +of the parameter file, although it is conventional to use the file +extension \texttt{.par}. Optional command-line arguments can be used +to customise runtime behaviour, and to provide information about the +thorns used in the executable. The general syntax for running Cactus from +a parameter file is then + +\texttt{./cactus\_<\var{config}> <\var{parameter file}> +[\var{command-line options}]} + +A parameter file is a text file whose lines are either comments +or parameter statements. +Comments are blank lines or lines that begin with either +`\texttt{\#}' or `\texttt{!}'. +A parameter statement consists of one or more parameter names, followed by +an `\texttt{=}', followed by the value(s) for this (these) parameter(s). +Note that all string parameters are case insensitive. + +The first parameter statement in any parameter file should set \texttt{ActiveThorns}, +which is a special parameter that tells the +program which \textit{thorns} are to be activated. Only parameters from active +thorns can be set (and only those routines \textit{scheduled} by active thorns +are run). By default all thorns are inactive. For example, the first +entry in a parameter file which is using just the two thorns +\texttt{CactusPUGH/PUGH} and \texttt{CactusBase/CartGrid3D} should be + +\texttt{ActiveThorns = "PUGH CartGrid3D"} + +Parameter specifications following \texttt{ActiveThorns} usually are +carried out by listing +the name of the \textit{thorn} which defined the parameter, two colons, +and the name of the parameter --- e.g.~\texttt{wavetoyF77::amplitude} +(see Section~\ref{sec:Cactus_parameters.scope} for more information). + +Notes: + +\begin{itemize} + +\item{} You can obtain lists of the parameters associated with +each thorn using the command-line options \texttt{-o} and \texttt{-O} +(Section~\ref{sec:command_line_options}). + +\item{} For examples of parameter files, look in the \texttt{par} directory + which can be found in most thorns. + +\item {} +The Cactus make system provides a mechanism for generating a +\textit{Thorn Guide} containing separate chapters for each thorn and +arrangement in your configuration. +Details about parameters, grid variables and scheduling are +automatically included in from a thorns CCL files into the Thorn +Guide. To construct a Thorn Guide for the configuration +\texttt{$<$\var{config}$>$} use + +\texttt{gmake $<$\var{config}$>$-ThornGuide} + +or to make a Thorn Guide for all the thorns in the \texttt{arrangements} directory + +\texttt{gmake $<$\var{config}$>$}. + +\end{itemize} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + + +\chapter{Getting and looking at output} + + +\section{Screen output} + +As your Cactus executable runs, standard output and standard error +are usually written to the screen. Standard output provides you +with information about the run, and standard error reports warnings +and errors from the flesh and thorns. + +As the program runs, the normal output provides the following information: + +\begin{Lentry} + +\item [Active thorns] + A report is made as each of the thorns in the \texttt{ActiveThorns} +parameters from the parameter file (see Section~\ref{sec:Parameter_File}) +is attempted to be activated. This report +shows whether the thorn activation was successful, and if successful gives the +thorn's implementation. For example + +\begin{verbatim} +Activating thorn idscalarwave...Success -> active implementation idscalarwave +\end{verbatim} + +\item [Failed parameters] + If any of the parameters in the parameter file does not belong to any +of the active thorns, or if the parameter value is not in the allowed range +(see Section~\ref{sec:Parameters.Types_and_Ranges}), +an error is registered. For example, if the parameter is not recognised + +\begin{verbatim} +Unknown parameter time::ddtfac +\end{verbatim} +or if the parameter value is not in the allowed range + +\begin{verbatim} +Unable to set keyword CartGrid3D::type - ByMouth not in any active range +\end{verbatim} + +\item [Scheduling information] + The scheduled routines (see Section~\ref{chap:scheduling}), +are listed, in the order that they will be executed. For example + +\begin{verbatim} +---------------------------------------------------------------------- + Startup routines + Cactus: Register banner for Cactus + CartGrid3D: Register GH Extension for GridSymmetry + CartGrid3D: Register coordinates for the Cartesian grid + IOASCII: Startup routine + IOBasic: Startup routine + IOUtil: IOUtil startup routine + PUGH: Startup routine + WaveToyC: Register banner + + Parameter checking routines + CartGrid3D: Check coordinates for CartGrid3D + IDScalarWave: Check parameters + + Initialisation + CartGrid3D: Set up spatial 3D Cartesian coordinates on the GH + PUGH: Report on PUGH set up + Time: Set timestep based on speed one Courant condition + WaveToyC: Schedule symmetries + IDScalarWave: Initial data for 3D wave equation + + do loop over timesteps + WaveToyC: Evolution of 3D wave equation + t = t+dt + if (analysis) + endif + enddo +---------------------------------------------------------------------- +\end{verbatim} + +\item [Thorn banners] + Usually a thorn registers a short piece of text as a \emph{banner}. + This banner of each thorn is displayed in the standard output when + the thorn is initialised. + +\end{Lentry} + + +\section{File output} +Output methods in Cactus are all provided by thorns. +Any number of output methods can be used for each run. +The behaviour of the output thorns in the +standard arrangements are described in those thorns' documentation. + +In general, output thorns decide what to output by parsing a string parameter +containing the names of those grid variables, or groups of variables, for which +output is required. The names should be fully qualified with the +implementation and group or variable names. + +There is usually a parameter for each method to denote how often, in evolution +iterations, this output should be performed. There is also usually a parameter +to define the directory in which the output should be placed, defaulting to the +directory from which the executable is run. + +See Chapter~\ref{chap:io_methods} for details on creating your own IO method. + + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\chapter{Checkpointing/Recovery} +\label{sec:checkpointing} + +Checkpointing is defined as saving the current state of a run (parameter +settings, contents of grid variables, and other relevant information) to a file. +At a later time, this run can then be restarted from that state by recovering +all the data from the checkpoint file. + +Cactus checkpointing and recovery methods are provided by thorns. +In general, these thorns decide how often to generate a checkpoint. +They also register their recovery routines with the flesh; these recovery +routines may then be called during initialisation of a subsequent run to +perform the recovery of the state of the run. +Such a recovery is requested by setting a parameter in the parameter file. + +See Chapter~\ref{chap:cp_recovery_methods} for details of how to create +your own checkpointing and recovery methods. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\chapter{Reporting bugs} +\label{sec:gehe} + +For tracking problem reports and bugs, we use GNATS, which is a bug tracking +system published under the GNU license. We have set up a web interface at +\url{http://www.cactuscode.org/BugReporting}, which allows easy submission and browsing +of problem reports. + +A description of the GNATS categories we use is provided in the Appendix +\ref{sec:Appendix.gnats}. + +% OK, there is NO emacs at the moment, because the GNATS setup is really stupid +% and sendpr handles like c.... besides the fact, that the user has to go +% through a make process which installs stuff somewhere on his HD. gerd. +% BUT, we could distribute our own, either copy cvsbug, or write a perl +% version. Tom +% \begin{itemize} +% \item \texttt{A web interface} +% \item \texttt{SendPR} +% {FIXME: Mention the emacs thing here too...} +% \end{itemize} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +%%%% FIXME: add something quick on required software and supported architectures + +%%%% FIXME: add an example + +\end{cactuspart} diff --git a/doc/UsersGuide/Infrastructure.tex b/doc/UsersGuide/Infrastructure.tex deleted file mode 100644 index fb48a8fb..00000000 --- a/doc/UsersGuide/Infrastructure.tex +++ /dev/null @@ -1,677 +0,0 @@ -% /*@@ -% @file Infrastructure.tex -% @date 27 Jan 1999 -% @author Tom Goodale, Gabrielle Allen, Gerd Lanferman, Thomas Radke -% @desc -% Infrastructure thorn writer's guide for the Cactus User's Guide -% @enddesc -% @version $Header$ -% @@*/ - -\begin{cactuspart}{Infrastructure Thorn Writer's Guide}{$RCSfile$}{$Revision$} -\label{part:Infrastructure} -\renewcommand{\thepage}{\Alph{part}\arabic{page}} - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\chapter{Introduction} - -\begin{itemize} - \item{} Concepts and terminology (Overloading and registration of functions) - \item{} The cGH structure --- what it is and how to use it - \item{} Extending the cGH structure - \item{} Querying group and variable information - \item{} Providing an I/O layer - \item{} Providing a communication layer - \item{} Providing a reduction operator - \item{} Providing an interpolation operator - \item{} Overloadable functions -\end{itemize} - -\chapter{Concepts and Terminology} -\label{chap:cote} - -\section{Overloading and Registration} - -The flesh defines a core API which guarantees the presence of a set of -functions. Although the flesh guarantees the presence of these functions, -they can be provided by thorns. Thorns do this by either the \textit{overloading} -or the \textit{registration} of functions. - -\subsection{Overloading} - -Some functions can only be provided by one thorn. The first thorn to -\textit{overload} this function succeeds, and any later attempt to overload -the function fails. For each overloadable function, there is a function -with a name something like {\tt CCTK\_Overload...} which is passed the -function pointer. - -\subsection{Registration} - -Some functions may be provided by several thorns. The thorns \textit{register} -their function with the flesh, and when the flesh-provided function is called, -the flesh calls all the registered functions. - -\section{GH Extensions} - -A GH extension is a way to associate data with each cGH. This data should be data -that is required to be associated with a particular GH by a thorn. - -Each GH extension is given a unique handle. - -\section{I/O Methods} - -An I/O method is a distinct way to output data. Each I/O method has a unique name, -and the flesh-provided I/O functions operate on all registered I/O methods. - -\chapter{GH Extensions} - -A GH extension is created by calling {\tt CCTK\_RegisterGHExtension}, with the -name of the extension. This returns a unique handle that identifies the extension. -(This handle can be retrieved at any time by a call to {\tt CCTK\_GHExtensionHandle}.) - -Associated with a GH extension are three functions - -\begin{Lentry} -\item[{\tt SetupGH}] -this is used to actually create the data structure holding the extension. It -is called when a new cGH is created. -\item[{\tt InitGH}] -this is used to initialise the extension. It is called after the scheduler has -been initialised on the cGH. -\item[{\tt ScheduleTraverseGH}] -this is called whenever the schedule tree is due to be traversed on the GH. It -should initialise the data on the cGH and the call {\tt CCTK\_ScheduleTraverse} to traverse -the schedule tree. -\end{Lentry} - -\chapter{Overloadable and Registerable Functions in Main} - - \begin{tabular}{|l|l|} - \hline {\bf Function} & {\bf Default} \\ - \hline {\t CCTK\_Initialise} &\\ - \hline {\t CCTK\_Evolve} &\\ - \hline {\t CCTK\_Shutdown} &\\ - \hline - \end{tabular} - -\chapter{Overloadable and Registerable Functions in Comm} - - \begin{tabular}{|l|l|} - \hline {\bf Function} & {\bf Default} \\ - \hline {\t CCTK\_SyncGroup} &\\ - \hline {\t CCTK\_SyncGroupsByDirI} &\\ - \hline {\t CCTK\_EnableGroupStorage} &\\ - \hline {\t CCTK\_DisableGroupStorage} &\\ - \hline {\t CCTK\_EnableGroupComm} &\\ - \hline {\t CCTK\_DisableGroupComm} &\\ - \hline {\t CCTK\_Barrier} &\\ - \hline {\t CCTK\_Reduce} &\\ - \hline {\t CCTK\_Interp} &\\ - \hline {\t CCTK\_ParallelInit} &\\ - \hline - \end{tabular} - -\chapter{Overloadable and Registerable Functions in I/O} - - \begin{tabular}{|l|l|} - \hline {\bf Function} & {\bf Default} \\ - \hline {\t CCTK\_OutputGH} & \\ - \hline {\t CCTK\_OutputVarAsByMethod} & \\ - \hline - \end{tabular} - -\chapter{Drivers} - -The flesh does not know about memory allocation for grid variables, about how -to communicate data when synchronisation is called for, or about -multiple patches or adaptive mesh refinement. All this is the job of a driver. - -This chapter describes how to add a driver to your code. - -\section{Anatomy} - -A driver consists of a Startup routine which creates a GH extension, -registers its associated functions, and overloads the communication functions. -It may optionally register interpolation, reduction, and I/O methods. - -A driver may also overload the default Initialisation and Evolution routines, -although a simple unigrid evolver is supplied in the flesh. - -\section{Startup} - -A driver consists of a GH extension, and the following overloaded -functions. - -\begin{enumerate} -\item{} {\tt CCTK\_EnableGroupStorage} -\item{} {\tt CCTK\_DisableGroupStorage} -\item{} {\tt CCTK\_ArrayGroupSizeB} -\item{} {\tt CCTK\_QueryGroupStorageB} -\item{} {\tt CCTK\_SyncGroup} -\item{} {\tt CCTK\_SyncGroupsByDirI} -\item{} {\tt CCTK\_EnableGroupComm} -\item{} {\tt CCTK\_DisableGroupComm} -\item{} {\tt CCTK\_Barrier} -\item{} {\tt CCTK\_OverloadParallelInit} -\item{} {\tt CCTK\_OverloadExit} -\item{} {\tt CCTK\_OverloadAbort} -\item{} {\tt CCTK\_OverloadMyProc} -\item{} {\tt CCTK\_OverloadnProcs} -\end{enumerate} - -The overloadable function {\tt CCTK\_SyncGroup} is deprecated, a driver should -instead provide a routine to overload the more general function -{\tt CCTK\_SyncGroupsByDirI}. - -\section{The GH Extension} - -The GH extension is where the driver stores all its grid-dependent information. -This is stuff like any data associated with a grid variable (e.g.\ storage and -communication state), how many grids if it is AMR, ... It is very difficult to -describe in general, but one simple example might be - -\begin{verbatim} - -struct SimpleExtension -{ - /* The data associated with each variable */ - /* data[var][timelevel][ijk] */ - void ***data -} ; - -\end{verbatim} - -with a {\tt SetupGH} routine like - -\begin{verbatim} - -struct SimpleExtension *SimpleSetupGH(tFleshConfig *config, int conv_level, cGH *GH) -{ - struct SimpleExtension *extension; - - extension = NULL; - - if(conv_level < max_conv_level) - { - /* Create the extension */ - extension = malloc(sizeof(struct SimpleExtension)); - - /* Allocate data for all the variables */ - extension->data = malloc(num_vars*sizeof(void**)); - - for(var = 0 ; var < num_vars; var++) - { - /* Allocate the memory for the time levels */ - extension->data[var] = malloc(num_var_time_levels*sizeof(void *)); - - for(time_level = 0; time_level < num_var_time_level; time_level++) - { - /* Initialise the data to NULL */ - extension->data[var][time_level] = NULL; - } - } - } - - return extension; -} - -\end{verbatim} - -Basically, what this example is doing is preparing a data array for use. The -function can query the flesh for information on every variable. Note that -scalars should always have memory actually assigned to them. - -An {\tt InitGH} function isn't strictly necessary, and in this case, it could -just be a dummy function. - -The {\tt ScheduleTraverseGH} function needs to fill out the cGH data, and -then call {\tt CCTK\_ScheduleTraverse} to have the functions scheduled at -that point executed on the grid - -\begin{verbatim} - -int SimpleScheduleTraverseGH(cGH *GH, const char *where) -{ - int retcode; - int var; - int gtype; - int ntimelevels; - int level; - int idir; - - extension = (struct SimpleExtension *)GH->extensions[SimpleExtension]; - - for (idir=0;idircctk_dim;idir++) - { - GH->cctk_levfac[idir] = 1; - GH->cctk_nghostzones[idir] = extension->nghostzones[idir]; - GH->cctk_lsh[idir] = extension->lnsize[idir]; - GH->cctk_gsh[idir] = extension->nsize[idir]; - GH->cctk_bbox[2*idir] = extension->lb[extension->myproc][idir] == 0; - GH->cctk_bbox[2*idir+1] = extension->ub[extension->myproc][idir] - == extension->nsize[idir]-1; - GH->cctk_lbnd[idir] = extension->lb[extension->myproc][idir]; - GH->cctk_ubnd[idir] = extension->ub[extension->myproc][idir]; - - } - - for(var = 0; var < extension->nvariables; var++) - { - gtype = CCTK_GroupTypeFromVarI(var); - ntimelevels = CCTK_MaxTimeLevelsVI(var); - - for(level = 0; level < ntimelevels; level++) - { - switch(gtype) - { - case CCTK_SCALAR : - GH->data[var][level] = extension->variables[var][level]; - break; - case CCTK_GF : - GH->data[var][level] = - ((pGF ***)(extension->variables))[var][level]->data; - break; - case CCTK_ARRAY : - GH->data[var][level] = - ((pGA ***)(extension->variables))[var][level]->data; - break; - default: - CCTK_WARN(CCTK_WARN_ALERT,"Unknown group type in SimpleScheduleTraverse"); - } - } - } - - retcode = CCTK_ScheduleTraverse(where, GH, NULL); - - return retcode; - -} - -\end{verbatim} - -The third argument to {\tt CCTK\_ScheduleTraverse} is actually a function -which will be called by the scheduler when it wants to call a function -scheduled by a thorn. This function is given some information about -the function to call, and is an alternative place where the cGH can be setup. - -This function is optional, but a simple implementation might be - -\begin{verbatim} - -int SimpleCallFunction(void *function, - cFunctionData *fdata, - void *data) -{ - void (*standardfunc)(void *); - - int (*noargsfunc)(void); - - switch(fdata->type) - { - case FunctionNoArgs: - noargsfunc = (int (*)(void))function; - noargsfunc(); - break; - case FunctionStandard: - switch(fdata->language) - { - case LangC: - standardfunc = (void (*)(void *))function; - standardfunc(data); - break; - case LangFortran: - fdata->FortranCaller(data, function); - break; - default : - CCTK_WARN(CCTK_WARN_ALERT, "Unknown language."); - } - break; - default : - CCTK_WARN(CCTK_WARN_ALERT, "Unknown function type."); - } - - /* Return 0, meaning didn't synchronise */ - return 0; -} - -\end{verbatim} - -The return code of the function signifies whether or not the function -synchronised the groups in this functions synchronisation list of not. - -The flesh will synchronise them if the function returns false. - -Providing this function is probably the easiest way to do multi-patch or -AMR drivers. - -\section{Memory Functions} - -These consist of -\begin{enumerate} -\item{} {\tt CCTK\_EnableGroupStorage} -\item{} {\tt CCTK\_DisableGroupStorage} -\item{} {\tt CCTK\_QueryGroupStorageB} -\item{} {\tt CCTK\_ArrayGroupSizeB} -\end{enumerate} - -\subsection{En/Disable Group Storage} - -These are responsible for switching the memory for all variables -in a group on or off. They should return the former state, e.g.\ -if the group already has storage assigned, they should return 1. - -In our simple example above, the enabling routine would look -something like - -\begin{verbatim} - -int SimpleEnableGroupStorage(cGH *GH, const char *groupname) -{ - - extension = (struct SimpleExtension *)GH->extensions[SimpleExtension]; - - if(extension->data[first][0][0] == NULL) - { - for(var = first; var <= last; var++) - { - allocate memory for all time levels; - } - retcode = 0; - } - else - { - retcode = 1; - } - - return retcode; -} - -\end{verbatim} - -The disable function is basically the reverse of the enable one. - -The {\tt QueryGroupStorage} function basically returns true or false if -there is storage for the group, and the {\tt ArrayGroupSize} returns the -size of the grid function or array group in a particular direction. - -\subsection{En/Disable Group Comm} - -These are the communication analogues to the storage functions. Basically, -they flag that communication is to be done on that group or not, and may -initialise data structures for the communication. - - -\chapter{I/O Methods} -\label{chap:io_methods} -% -The flesh by itself does not provide output for grid variables or other data. -Instead, it provides a mechanism for thorns to register their own -routines as I/O methods, and to invoke these I/O methods by either the -flesh scheduler or by other thorn routines. - -This chapter explains how to implement your own I/O methods and register -them with the flesh. -% -\section{I/O Method Registration} -% -All I/O methods have to be registered with the flesh before they can be used. - -The flesh I/O registration API provides a routine {\t CCTK\_RegisterIOMethod()} -to create a handle for a new I/O method which is identified by its name -(this name must be unique for all I/O methods). -With such a handle, a thorn can then register a set of functions (using the -{\t CCTK\_RegisterIOMethod*()} routines from the flesh) for doing -periodic, triggered, and/or unconditional output. - -The following example shows how a thorn would register an I/O method, -{\tt SimpleIO}, with routines to provide all these different types of output. -% -\begin{verbatim} - void SimpleIO_Startup (void) - { - int handle = CCTK_RegisterIOMethod ("SimpleIO"); - if (handle >= 0) - { - CCTK_RegisterIOMethodOutputGH (handle, SimpleIO_OutputGH); - - CCTK_RegisterIOMethodTimeToOutput (handle, SimpleIO_TimeToOutput); - CCTK_RegisterIOMethodTriggerOutput (handle, SimpleIO_TriggerOutput); - - CCTK_RegisterIOMethodOutputVarAs (handle, SimpleIO_OutputVarAs); - } - } -\end{verbatim} -% -% -\section{Periodic Output of Grid Variables} -% -The flesh scheduler will automatically call {\t CCTK\_OutputGH()} at every -iteration after the {\tt CCTK\_ANALYSIS} time bin. This function loops over all -I/O methods and calls their routines registered as {\t OutputGH()} (see also Section -\ref{subsec:schedule_ccl}). -% -\begin{alltt} - int SimpleIO_OutputGH (const cGH *\var{GH}); -\end{alltt} -% -The {\t OutputGH()} routine itself should loop over all variables living on the -\texttt{GH} grid hierarchy, and do all necessary output if requested -(this is usually determined by I/O parameter settings). - -As its return code, it should pass back the number of variables which were output -at the current iteration, or zero if no output was done by this I/O method. -% -% -\section{Triggered Output of Grid Variables} -% -Besides the periodic output at every so many iterations using {\t OutputGH()}, -analysis and output of grid variables can also be done via triggers. For this, -a {\t TimeToOutput()} routine is registered with an I/O method. -This routine will be called by the flesh scheduler at every iteration before -{\tt CCTK\_ANALYSIS} with the triggering variable(s) as defined in the schedule -block for all {\tt CCTK\_ANALYSIS} routines (see Section \ref{scheduling:schedule_block}). - -If the {\t TimeToOutput()} routine decides that it is now time to do output, the -flesh scheduler will invoke the corresponding analysis routine and also request -output of the triggering variable(s) using {\t TriggerOutput()}. -% -\begin{alltt} - int SimpleIO_TimeToOutput (const cGH *\var{GH}, int \var{varindex}); - int SimpleIO_TriggerOutput (const cGH *\var{GH}, int \var{varindex}); -\end{alltt} -% -Both routines get passed the index of a possible triggering grid variable. - -{\t TimeToOutput()} should return a non-zero value if analysis and output -for \texttt{\var{varindex}} should take place at the current iteration, and zero -otherwise. - -{\t TriggerOutput()} should return zero for successful output of variable -\texttt{\var{varindex}}, and a negative value in case of an error. -% -% -\section{Unconditional Output of Grid Variables} - -An I/O method's {\t OutputVarAs()} routine is supposed to do output for a -specific grid variable if ever possible. It will be invoked by the flesh I/O API -routines {\t CCTK\_OutputVar*()} for unconditional, non-triggered output of -grid variables (see also Section \ref{sec:io}). - -A function registered as an \texttt{OutputVarAs()} routine has the following -prototype: -% -\begin{alltt} - int SimpleIO_OutputVarAs (const cGH *\var{GH}, const char *\var{varname}, const char *\var{alias}); -\end{alltt} -% -The variable to output, \texttt{\var{varname}}, is given by its full name. -The full name may have appended an optional I/O options string enclosed in -curly braces (with no space between the full name and the opening curly brace). -In addition to that, an \texttt{\var{alias}} string can be passed which then -serves the purpose of constructing a unique name for the output file. - -The {\t OutputVarAs()} routine should return zero if output for -\texttt{\var{varname}} -was done successfully, or a negative error code otherwise. - - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -\chapter{Checkpointing/Recovery Methods} -\label{chap:cp_recovery_methods} - -Like for I/O methods, checkpointing/recovery functionality must be implemented -by thorns. The flesh only provides specific time bins at which the scheduler -will call thorns' routines, in order to perform checkpointing and/or recovery. - -This chapter explains how to implement checkpointing and recovery methods -in your thorn. For further information, see the documentation for thorn -\texttt{CactusBase/IOUtil}. - -\section{Checkpointing Invocation} - -Thorns register their checkpointing routines at {\t CCTK\_CPINITIAL} and/or -{\t CCTK\_CHECKPOINT} and/or {\t CCTK\_TERMINATE}. These time bins are -scheduled right after all initial data has been set up, after every -evolution timestep, and after the last time step of a simulation, -respectively. (See Section \ref{subsec:schedule_ccl} for a description of all timebins). -Depending on -parameter settings, the checkpoint routines decide whether to write an initial -data checkpoint, and when to write an evolution checkpoint. - -Each checkpoint routine should save all information to persistent storage, which -is necessary to restart the simulation at a later time from exactly the same -state. Such information would include -% -\begin{itemize} - \item the current settings of all parameters - \item the contents of all grid variables which have global storage assigned - and are not tagged with {\tt checkpoint="no"} (see also Section \ref{subsec:Appendix.interface-variables} on page \pageref{subsec:Appendix.interface-variables} for a list of possible tags)\\ - Note that grid variables should be synced before writing them to disk. - \item other relevant information such as the current iteration number and - physical time, the number of processors, etc. -\end{itemize} - - -\section{Recovery Invocation} - -Recovering from a checkpoint is a two-phase operation for which the flesh -provides distinct timebins for recovery routines to be scheduled at: -% -\begin{Lentry} - \item[{\t CCTK\_RECOVER\_PARAMETERS}] This time bin is executed before - {\t CCTK\_STARTUP}, in which the parameter file is parsed. From these parameter - settings, the recovery routines should decide whether recovery was - requested, and if so, restore all parameters from the checkpoint file and - overwrite those which aren't steerable.\\ - The flesh loops over all registered recovery routines to find out - whether recovery was requested. Each recovery routine should, therefore, - return a positive integer value for successful parameter recovery (causing - a shortcut of the loop over all registered recovery routines), - zero for no recovery, or a negative value to flag an error.\\ - If recovery was requested, but no routine could successfully recover - parameters, the flesh will abort the run with an error message. - If no routine recovered any parameters, i.e. if all parameter - recovery routines returned zero, then this indicates that this run - is not a recovery run.\\ - If parameter recovery was performed successfully, the scheduler will set the - {\tt recovered} flag which---in combination with the setting of the {\tt - Cactus::recovery\_mode} parameter---decides whether any thorn routine - scheduled for {\t CCTK\_INITIAL} and {\t CCTK\_POSTINITIAL} will be called. - The default is to not execute these initial time bins during recovery, - because the initial data will be set up from the checkpoint file during the - following {\t CCTK\_RECOVER\_VARIABLES} time bin. - \item[{\t CCTK\_RECOVER\_VARIABLES}] - Recovery routines scheduled for this time bin are responsible for restoring - the contents of all grid variables with storage assigned from the - checkpoint.\\ - Depending on the setting of {\tt Cactus::recovery\_mode}, they should also - decide how to treat errors in recovering individual grid variables. Strict - recovery (which is the default) requires all variables to be restored - successfully (and will stop the code if not), whereas a relaxed mode - could, e.g. allow for grid variables, which are not found in the checkpoint - file, to be gracefully ignored during recovery. -\end{Lentry} - - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -\chapter{Clocks for Timing} -\label{chap:clocks} - -To add a Cactus clock, you need to write several functions to provide the -timer functionality (see Section \ref{sec:timers}), and then register these -functions with the flesh as a named clock. - -The function pointers are placed in function pointer fields of a -\texttt{cClockFuncs} structure. The fields of this structure are are: - -\begin{Lentry} - -\item[{\t create}] {\t void *(*create)(int)} -\item[{\t destroy}] {\t void (*destroy)(int, void *)} -\item[{\t start}] {\t void (*start)(int, void *)} -\item[{\t stop}] {\t void (*stop)(int, void *)} -\item[{\t reset}] {\t void (*reset)(int, void *)} -\item[{\t get}] {\t void (*get)(int, void *, cTimerVal *)} -\item[{\t set}] {\t void (*set)(int, void *, cTimerVal *)} -\item[{\t n\_vals}] {\t int} - -\end{Lentry} - -The first \texttt{int} argument of the functions may be used in any way -you see fit. - -The \texttt{n\_vals} field holds the number of elements in the -\texttt{vals} array field of the \texttt{cTimerVal} structure -used by your timer (usually 1). - -The return value of the \texttt{create} function will be a pointer to -a new structure representing your clock. - -The second \texttt{void*} argument of all the other functions will be the -pointer returned from the \texttt{create} function. - -The \texttt{get} and \texttt{set} functions should write to and read from -(respectively) a structure pointed to by the \texttt{cTimerVal*} argument: - -\begin{verbatim} -typedef enum {val_none, val_int, val_long, val_double} cTimerValType; - -typedef struct -{ - cTimerValType type; - const char *heading; - const char *units; - union - { - int i; - long int l; - double d; - } val; - double seconds; - double resolution; -} cTimerVal; -\end{verbatim} - -The \texttt{heading} field is the name of the clock, the \texttt{units} -field holds a string describing the type held in the \texttt{val} field, -and the \texttt{seconds} field is the time elapsed in seconds. The -\texttt{resolution} field is the smallest non-zero difference in values -of two calls to the timer, in seconds. For example, it could reflect -that the clock saves the time value internally as an integer value -representing milliseconds. - -To name and register the clock with the flesh, call the function -\begin{alltt} - CCTK_ClockRegister( "\var{my_clock_name}", &\var{clock_func} ). -\end{alltt} - - - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\end{cactuspart} diff --git a/doc/UsersGuide/InfrastructureThorns.tex b/doc/UsersGuide/InfrastructureThorns.tex new file mode 100644 index 00000000..3e82c443 --- /dev/null +++ b/doc/UsersGuide/InfrastructureThorns.tex @@ -0,0 +1,676 @@ +% /*@@ +% @file Infrastructure.tex +% @date 27 Jan 1999 +% @author Tom Goodale, Gabrielle Allen, Gerd Lanferman, Thomas Radke +% @desc +% Infrastructure thorn writer's guide for the Cactus User's Guide +% @enddesc +% @version $Header$ +% @@*/ + +\renewcommand{\thepage}{\Alph{part}\arabic{page}} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\chapter{Infrastructure Thorns} +\label{chap:infrastructure} + +\begin{itemize} + \item{} Concepts and terminology (Overloading and registration of functions) + \item{} The cGH structure --- what it is and how to use it + \item{} Extending the cGH structure + \item{} Querying group and variable information + \item{} Providing an I/O layer + \item{} Providing a communication layer + \item{} Providing a reduction operator + \item{} Providing an interpolation operator + \item{} Overloadable functions +\end{itemize} + +\section{Concepts and Terminology} +\label{chap:cote} + +\subsection{Overloading and Registration} + +The flesh defines a core API which guarantees the presence of a set of +functions. Although the flesh guarantees the presence of these functions, +they can be provided by thorns. Thorns do this by either the \textit{overloading} +or the \textit{registration} of functions. + +\subsubsection{Overloading} + +Some functions can only be provided by one thorn. The first thorn to +\textit{overload} this function succeeds, and any later attempt to overload +the function fails. For each overloadable function, there is a function +with a name something like {\tt CCTK\_Overload...} which is passed the +function pointer. + +\subsubsection{Registration} + +Some functions may be provided by several thorns. The thorns \textit{register} +their function with the flesh, and when the flesh-provided function is called, +the flesh calls all the registered functions. + +\subsection{GH Extensions} + +A GH extension is a way to associate data with each cGH. This data should be data +that is required to be associated with a particular GH by a thorn. + +Each GH extension is given a unique handle. + +\subsection{I/O Methods} + +An I/O method is a distinct way to output data. Each I/O method has a unique name, +and the flesh-provided I/O functions operate on all registered I/O methods. + +\section{GH Extensions} + +A GH extension is created by calling {\tt CCTK\_RegisterGHExtension}, with the +name of the extension. This returns a unique handle that identifies the extension. +(This handle can be retrieved at any time by a call to {\tt CCTK\_GHExtensionHandle}.) + +Associated with a GH extension are three functions + +\begin{Lentry} +\item[{\tt SetupGH}] +this is used to actually create the data structure holding the extension. It +is called when a new cGH is created. +\item[{\tt InitGH}] +this is used to initialise the extension. It is called after the scheduler has +been initialised on the cGH. +\item[{\tt ScheduleTraverseGH}] +this is called whenever the schedule tree is due to be traversed on the GH. It +should initialise the data on the cGH and the call {\tt CCTK\_ScheduleTraverse} to traverse +the schedule tree. +\end{Lentry} + +\section{Overloadable and Registerable Functions in Main} + + \begin{tabular}{|l|l|} + \hline {\bf Function} & {\bf Default} \\ + \hline {\t CCTK\_Initialise} &\\ + \hline {\t CCTK\_Evolve} &\\ + \hline {\t CCTK\_Shutdown} &\\ + \hline + \end{tabular} + +\section{Overloadable and Registerable Functions in Comm} + + \begin{tabular}{|l|l|} + \hline {\bf Function} & {\bf Default} \\ + \hline {\t CCTK\_SyncGroup} &\\ + \hline {\t CCTK\_SyncGroupsByDirI} &\\ + \hline {\t CCTK\_EnableGroupStorage} &\\ + \hline {\t CCTK\_DisableGroupStorage} &\\ + \hline {\t CCTK\_EnableGroupComm} &\\ + \hline {\t CCTK\_DisableGroupComm} &\\ + \hline {\t CCTK\_Barrier} &\\ + \hline {\t CCTK\_Reduce} &\\ + \hline {\t CCTK\_Interp} &\\ + \hline {\t CCTK\_ParallelInit} &\\ + \hline + \end{tabular} + +\section{Overloadable and Registerable Functions in I/O} + + \begin{tabular}{|l|l|} + \hline {\bf Function} & {\bf Default} \\ + \hline {\t CCTK\_OutputGH} & \\ + \hline {\t CCTK\_OutputVarAsByMethod} & \\ + \hline + \end{tabular} + +\section{Drivers} + +The flesh does not know about memory allocation for grid variables, about how +to communicate data when synchronisation is called for, or about +multiple patches or adaptive mesh refinement. All this is the job of a driver. + +This chapter describes how to add a driver to your code. + +\subsection{Anatomy} + +A driver consists of a Startup routine which creates a GH extension, +registers its associated functions, and overloads the communication functions. +It may optionally register interpolation, reduction, and I/O methods. + +A driver may also overload the default Initialisation and Evolution routines, +although a simple unigrid evolver is supplied in the flesh. + +\subsection{Startup} + +A driver consists of a GH extension, and the following overloaded +functions. + +\begin{enumerate} +\item{} {\tt CCTK\_EnableGroupStorage} +\item{} {\tt CCTK\_DisableGroupStorage} +\item{} {\tt CCTK\_ArrayGroupSizeB} +\item{} {\tt CCTK\_QueryGroupStorageB} +\item{} {\tt CCTK\_SyncGroup} +\item{} {\tt CCTK\_SyncGroupsByDirI} +\item{} {\tt CCTK\_EnableGroupComm} +\item{} {\tt CCTK\_DisableGroupComm} +\item{} {\tt CCTK\_Barrier} +\item{} {\tt CCTK\_OverloadParallelInit} +\item{} {\tt CCTK\_OverloadExit} +\item{} {\tt CCTK\_OverloadAbort} +\item{} {\tt CCTK\_OverloadMyProc} +\item{} {\tt CCTK\_OverloadnProcs} +\end{enumerate} + +The overloadable function {\tt CCTK\_SyncGroup} is deprecated, a driver should +instead provide a routine to overload the more general function +{\tt CCTK\_SyncGroupsByDirI}. + +\subsection{The GH Extension} + +The GH extension is where the driver stores all its grid-dependent information. +This is stuff like any data associated with a grid variable (e.g.\ storage and +communication state), how many grids if it is AMR, ... It is very difficult to +describe in general, but one simple example might be + +\begin{verbatim} + +struct SimpleExtension +{ + /* The data associated with each variable */ + /* data[var][timelevel][ijk] */ + void ***data +} ; + +\end{verbatim} + +with a {\tt SetupGH} routine like + +\begin{verbatim} + +struct SimpleExtension *SimpleSetupGH(tFleshConfig *config, int conv_level, cGH *GH) +{ + struct SimpleExtension *extension; + + extension = NULL; + + if(conv_level < max_conv_level) + { + /* Create the extension */ + extension = malloc(sizeof(struct SimpleExtension)); + + /* Allocate data for all the variables */ + extension->data = malloc(num_vars*sizeof(void**)); + + for(var = 0 ; var < num_vars; var++) + { + /* Allocate the memory for the time levels */ + extension->data[var] = malloc(num_var_time_levels*sizeof(void *)); + + for(time_level = 0; time_level < num_var_time_level; time_level++) + { + /* Initialise the data to NULL */ + extension->data[var][time_level] = NULL; + } + } + } + + return extension; +} + +\end{verbatim} + +Basically, what this example is doing is preparing a data array for use. The +function can query the flesh for information on every variable. Note that +scalars should always have memory actually assigned to them. + +An {\tt InitGH} function isn't strictly necessary, and in this case, it could +just be a dummy function. + +The {\tt ScheduleTraverseGH} function needs to fill out the cGH data, and +then call {\tt CCTK\_ScheduleTraverse} to have the functions scheduled at +that point executed on the grid + +\begin{verbatim} + +int SimpleScheduleTraverseGH(cGH *GH, const char *where) +{ + int retcode; + int var; + int gtype; + int ntimelevels; + int level; + int idir; + + extension = (struct SimpleExtension *)GH->extensions[SimpleExtension]; + + for (idir=0;idircctk_dim;idir++) + { + GH->cctk_levfac[idir] = 1; + GH->cctk_nghostzones[idir] = extension->nghostzones[idir]; + GH->cctk_lsh[idir] = extension->lnsize[idir]; + GH->cctk_gsh[idir] = extension->nsize[idir]; + GH->cctk_bbox[2*idir] = extension->lb[extension->myproc][idir] == 0; + GH->cctk_bbox[2*idir+1] = extension->ub[extension->myproc][idir] + == extension->nsize[idir]-1; + GH->cctk_lbnd[idir] = extension->lb[extension->myproc][idir]; + GH->cctk_ubnd[idir] = extension->ub[extension->myproc][idir]; + + } + + for(var = 0; var < extension->nvariables; var++) + { + gtype = CCTK_GroupTypeFromVarI(var); + ntimelevels = CCTK_MaxTimeLevelsVI(var); + + for(level = 0; level < ntimelevels; level++) + { + switch(gtype) + { + case CCTK_SCALAR : + GH->data[var][level] = extension->variables[var][level]; + break; + case CCTK_GF : + GH->data[var][level] = + ((pGF ***)(extension->variables))[var][level]->data; + break; + case CCTK_ARRAY : + GH->data[var][level] = + ((pGA ***)(extension->variables))[var][level]->data; + break; + default: + CCTK_WARN(CCTK_WARN_ALERT,"Unknown group type in SimpleScheduleTraverse"); + } + } + } + + retcode = CCTK_ScheduleTraverse(where, GH, NULL); + + return retcode; + +} + +\end{verbatim} + +The third argument to {\tt CCTK\_ScheduleTraverse} is actually a function +which will be called by the scheduler when it wants to call a function +scheduled by a thorn. This function is given some information about +the function to call, and is an alternative place where the cGH can be setup. + +This function is optional, but a simple implementation might be + +\begin{verbatim} + +int SimpleCallFunction(void *function, + cFunctionData *fdata, + void *data) +{ + void (*standardfunc)(void *); + + int (*noargsfunc)(void); + + switch(fdata->type) + { + case FunctionNoArgs: + noargsfunc = (int (*)(void))function; + noargsfunc(); + break; + case FunctionStandard: + switch(fdata->language) + { + case LangC: + standardfunc = (void (*)(void *))function; + standardfunc(data); + break; + case LangFortran: + fdata->FortranCaller(data, function); + break; + default : + CCTK_WARN(CCTK_WARN_ALERT, "Unknown language."); + } + break; + default : + CCTK_WARN(CCTK_WARN_ALERT, "Unknown function type."); + } + + /* Return 0, meaning didn't synchronise */ + return 0; +} + +\end{verbatim} + +The return code of the function signifies whether or not the function +synchronised the groups in this functions synchronisation list of not. + +The flesh will synchronise them if the function returns false. + +Providing this function is probably the easiest way to do multi-patch or +AMR drivers. + +\subsection{Memory Functions} + +These consist of +\begin{enumerate} +\item{} {\tt CCTK\_EnableGroupStorage} +\item{} {\tt CCTK\_DisableGroupStorage} +\item{} {\tt CCTK\_QueryGroupStorageB} +\item{} {\tt CCTK\_ArrayGroupSizeB} +\end{enumerate} + +\subsubsection{En/Disable Group Storage} + +These are responsible for switching the memory for all variables +in a group on or off. They should return the former state, e.g.\ +if the group already has storage assigned, they should return 1. + +In our simple example above, the enabling routine would look +something like + +\begin{verbatim} + +int SimpleEnableGroupStorage(cGH *GH, const char *groupname) +{ + + extension = (struct SimpleExtension *)GH->extensions[SimpleExtension]; + + if(extension->data[first][0][0] == NULL) + { + for(var = first; var <= last; var++) + { + allocate memory for all time levels; + } + retcode = 0; + } + else + { + retcode = 1; + } + + return retcode; +} + +\end{verbatim} + +The disable function is basically the reverse of the enable one. + +The {\tt QueryGroupStorage} function basically returns true or false if +there is storage for the group, and the {\tt ArrayGroupSize} returns the +size of the grid function or array group in a particular direction. + +\subsubsection{En/Disable Group Comm} + +These are the communication analogues to the storage functions. Basically, +they flag that communication is to be done on that group or not, and may +initialise data structures for the communication. + + +\section{I/O Methods} +\label{chap:io_methods} +% +The flesh by itself does not provide output for grid variables or other data. +Instead, it provides a mechanism for thorns to register their own +routines as I/O methods, and to invoke these I/O methods by either the +flesh scheduler or by other thorn routines. + +This chapter explains how to implement your own I/O methods and register +them with the flesh. +% +\subsection{I/O Method Registration} +% +All I/O methods have to be registered with the flesh before they can be used. + +The flesh I/O registration API provides a routine {\t CCTK\_RegisterIOMethod()} +to create a handle for a new I/O method which is identified by its name +(this name must be unique for all I/O methods). +With such a handle, a thorn can then register a set of functions (using the +{\t CCTK\_RegisterIOMethod*()} routines from the flesh) for doing +periodic, triggered, and/or unconditional output. + +The following example shows how a thorn would register an I/O method, +{\tt SimpleIO}, with routines to provide all these different types of output. +% +\begin{verbatim} + void SimpleIO_Startup (void) + { + int handle = CCTK_RegisterIOMethod ("SimpleIO"); + if (handle >= 0) + { + CCTK_RegisterIOMethodOutputGH (handle, SimpleIO_OutputGH); + + CCTK_RegisterIOMethodTimeToOutput (handle, SimpleIO_TimeToOutput); + CCTK_RegisterIOMethodTriggerOutput (handle, SimpleIO_TriggerOutput); + + CCTK_RegisterIOMethodOutputVarAs (handle, SimpleIO_OutputVarAs); + } + } +\end{verbatim} +% +% +\subsection{Periodic Output of Grid Variables} +% +The flesh scheduler will automatically call {\t CCTK\_OutputGH()} at every +iteration after the {\tt CCTK\_ANALYSIS} time bin. This function loops over all +I/O methods and calls their routines registered as {\t OutputGH()} (see also Section +\ref{subsec:schedule_ccl}). +% +\begin{alltt} + int SimpleIO_OutputGH (const cGH *\var{GH}); +\end{alltt} +% +The {\t OutputGH()} routine itself should loop over all variables living on the +\texttt{GH} grid hierarchy, and do all necessary output if requested +(this is usually determined by I/O parameter settings). + +As its return code, it should pass back the number of variables which were output +at the current iteration, or zero if no output was done by this I/O method. +% +% +\subsection{Triggered Output of Grid Variables} +% +Besides the periodic output at every so many iterations using {\t OutputGH()}, +analysis and output of grid variables can also be done via triggers. For this, +a {\t TimeToOutput()} routine is registered with an I/O method. +This routine will be called by the flesh scheduler at every iteration before +{\tt CCTK\_ANALYSIS} with the triggering variable(s) as defined in the schedule +block for all {\tt CCTK\_ANALYSIS} routines (see Section \ref{scheduling:schedule_block}). + +If the {\t TimeToOutput()} routine decides that it is now time to do output, the +flesh scheduler will invoke the corresponding analysis routine and also request +output of the triggering variable(s) using {\t TriggerOutput()}. +% +\begin{alltt} + int SimpleIO_TimeToOutput (const cGH *\var{GH}, int \var{varindex}); + int SimpleIO_TriggerOutput (const cGH *\var{GH}, int \var{varindex}); +\end{alltt} +% +Both routines get passed the index of a possible triggering grid variable. + +{\t TimeToOutput()} should return a non-zero value if analysis and output +for \texttt{\var{varindex}} should take place at the current iteration, and zero +otherwise. + +{\t TriggerOutput()} should return zero for successful output of variable +\texttt{\var{varindex}}, and a negative value in case of an error. +% +% +\subsection{Unconditional Output of Grid Variables} + +An I/O method's {\t OutputVarAs()} routine is supposed to do output for a +specific grid variable if ever possible. It will be invoked by the flesh I/O API +routines {\t CCTK\_OutputVar*()} for unconditional, non-triggered output of +grid variables (see also Section \ref{sec:io}). + +A function registered as an \texttt{OutputVarAs()} routine has the following +prototype: +% +\begin{alltt} + int SimpleIO_OutputVarAs (const cGH *\var{GH}, const char *\var{varname}, const char *\var{alias}); +\end{alltt} +% +The variable to output, \texttt{\var{varname}}, is given by its full name. +The full name may have appended an optional I/O options string enclosed in +curly braces (with no space between the full name and the opening curly brace). +In addition to that, an \texttt{\var{alias}} string can be passed which then +serves the purpose of constructing a unique name for the output file. + +The {\t OutputVarAs()} routine should return zero if output for +\texttt{\var{varname}} +was done successfully, or a negative error code otherwise. + + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +\section{Checkpointing/Recovery Methods} +\label{chap:cp_recovery_methods} + +Like for I/O methods, checkpointing/recovery functionality must be implemented +by thorns. The flesh only provides specific time bins at which the scheduler +will call thorns' routines, in order to perform checkpointing and/or recovery. + +This chapter explains how to implement checkpointing and recovery methods +in your thorn. For further information, see the documentation for thorn +\texttt{CactusBase/IOUtil}. + +\subsection{Checkpointing Invocation} + +Thorns register their checkpointing routines at {\t CCTK\_CPINITIAL} and/or +{\t CCTK\_CHECKPOINT} and/or {\t CCTK\_TERMINATE}. These time bins are +scheduled right after all initial data has been set up, after every +evolution timestep, and after the last time step of a simulation, +respectively. (See Section \ref{subsec:schedule_ccl} for a description of all timebins). +Depending on +parameter settings, the checkpoint routines decide whether to write an initial +data checkpoint, and when to write an evolution checkpoint. + +Each checkpoint routine should save all information to persistent storage, which +is necessary to restart the simulation at a later time from exactly the same +state. Such information would include +% +\begin{itemize} + \item the current settings of all parameters + \item the contents of all grid variables which have global storage assigned + and are not tagged with {\tt checkpoint="no"} (see also Section \ref{subsec:Appendix.interface-variables} on page \pageref{subsec:Appendix.interface-variables} for a list of possible tags)\\ + Note that grid variables should be synced before writing them to disk. + \item other relevant information such as the current iteration number and + physical time, the number of processors, etc. +\end{itemize} + + +\subsection{Recovery Invocation} + +Recovering from a checkpoint is a two-phase operation for which the flesh +provides distinct timebins for recovery routines to be scheduled at: +% +\begin{Lentry} + \item[{\t CCTK\_RECOVER\_PARAMETERS}] This time bin is executed before + {\t CCTK\_STARTUP}, in which the parameter file is parsed. From these parameter + settings, the recovery routines should decide whether recovery was + requested, and if so, restore all parameters from the checkpoint file and + overwrite those which aren't steerable.\\ + The flesh loops over all registered recovery routines to find out + whether recovery was requested. Each recovery routine should, therefore, + return a positive integer value for successful parameter recovery (causing + a shortcut of the loop over all registered recovery routines), + zero for no recovery, or a negative value to flag an error.\\ + If recovery was requested, but no routine could successfully recover + parameters, the flesh will abort the run with an error message. + If no routine recovered any parameters, i.e. if all parameter + recovery routines returned zero, then this indicates that this run + is not a recovery run.\\ + If parameter recovery was performed successfully, the scheduler will set the + {\tt recovered} flag which---in combination with the setting of the {\tt + Cactus::recovery\_mode} parameter---decides whether any thorn routine + scheduled for {\t CCTK\_INITIAL} and {\t CCTK\_POSTINITIAL} will be called. + The default is to not execute these initial time bins during recovery, + because the initial data will be set up from the checkpoint file during the + following {\t CCTK\_RECOVER\_VARIABLES} time bin. + \item[{\t CCTK\_RECOVER\_VARIABLES}] + Recovery routines scheduled for this time bin are responsible for restoring + the contents of all grid variables with storage assigned from the + checkpoint.\\ + Depending on the setting of {\tt Cactus::recovery\_mode}, they should also + decide how to treat errors in recovering individual grid variables. Strict + recovery (which is the default) requires all variables to be restored + successfully (and will stop the code if not), whereas a relaxed mode + could, e.g. allow for grid variables, which are not found in the checkpoint + file, to be gracefully ignored during recovery. +\end{Lentry} + + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +\section{Clocks for Timing} +\label{chap:clocks} + +To add a Cactus clock, you need to write several functions to provide the +timer functionality (see Section \ref{sec:timers}), and then register these +functions with the flesh as a named clock. + +The function pointers are placed in function pointer fields of a +\texttt{cClockFuncs} structure. The fields of this structure are are: + +\begin{Lentry} + +\item[{\t create}] {\t void *(*create)(int)} +\item[{\t destroy}] {\t void (*destroy)(int, void *)} +\item[{\t start}] {\t void (*start)(int, void *)} +\item[{\t stop}] {\t void (*stop)(int, void *)} +\item[{\t reset}] {\t void (*reset)(int, void *)} +\item[{\t get}] {\t void (*get)(int, void *, cTimerVal *)} +\item[{\t set}] {\t void (*set)(int, void *, cTimerVal *)} +\item[{\t n\_vals}] {\t int} + +\end{Lentry} + +The first \texttt{int} argument of the functions may be used in any way +you see fit. + +The \texttt{n\_vals} field holds the number of elements in the +\texttt{vals} array field of the \texttt{cTimerVal} structure +used by your timer (usually 1). + +The return value of the \texttt{create} function will be a pointer to +a new structure representing your clock. + +The second \texttt{void*} argument of all the other functions will be the +pointer returned from the \texttt{create} function. + +The \texttt{get} and \texttt{set} functions should write to and read from +(respectively) a structure pointed to by the \texttt{cTimerVal*} argument: + +\begin{verbatim} +typedef enum {val_none, val_int, val_long, val_double} cTimerValType; + +typedef struct +{ + cTimerValType type; + const char *heading; + const char *units; + union + { + int i; + long int l; + double d; + } val; + double seconds; + double resolution; +} cTimerVal; +\end{verbatim} + +The \texttt{heading} field is the name of the clock, the \texttt{units} +field holds a string describing the type held in the \texttt{val} field, +and the \texttt{seconds} field is the time elapsed in seconds. The +\texttt{resolution} field is the smallest non-zero difference in values +of two calls to the timer, in seconds. For example, it could reflect +that the clock saves the time value internally as an integer value +representing milliseconds. + +To name and register the clock with the flesh, call the function +\begin{alltt} + CCTK_ClockRegister( "\var{my_clock_name}", &\var{clock_func} ). +\end{alltt} + + + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\end{cactuspart} diff --git a/doc/UsersGuide/Notes.tex b/doc/UsersGuide/Notes.tex new file mode 100644 index 00000000..30c01719 --- /dev/null +++ b/doc/UsersGuide/Notes.tex @@ -0,0 +1,1500 @@ +% /*@@ +% @file RunningCactus.tex +% @date 27 Jan 1999 +% @author Tom Goodale, Gabrielle Allen, Gerd Lanferman, Thomas Radke +% @desc +% How to run Cactus part of the Cactus User's Guide +% @enddesc +% @version $Header$ +% @@*/ + +\begin{cactuspart}{Additional notes}{$RCSfile$}{$Revision$}\label{part:Notes} +\renewcommand{\thepage}{\Alph{part}\arabic{page}} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + + + +\chapter{Installation} +\label{cha:in} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\section{Required Software} +\label{sec:required_software} + +In general, Cactus \emph{requires} the following set of software to function +in single processor mode. Please refer to the architecture section +\ref{sec:suar} for architecture specific items. +\begin{Lentry} +\item[Perl5.0] Perl is used extensively during the Cactus + thorn configuration phase. Perl is available for nearly all + operating systems known to man, and can be obtained at + \url{http://www.perl.org}. +\item[GNU make] The make + process works with the GNU make utility (referred to as \texttt{gmake} + henceforth). While other make utilities may also work, this is not + guaranteed. Gmake can be obtained from your favorite GNU site, or + from \url{http://www.gnu.org}. +\item[C] C compiler. For example, the GNU compiler. This + is available for most supported platforms. Platform specific compilers + should also work. +\item[CPP] C Preprocessor. For example, the GNU \texttt{cpp}. These are + normally provided on most platforms, and many C compilers have an option + to just run as a preprocessor. +\item[CVS] The \textit{Concurrent Versions System} is not needed + to run/compile Cactus, but you are strongly encouraged to install + this software to take advantage of the update procedures. It can be + downloaded from your favorite GNU site. Tar files of each release are + also available. +\end{Lentry} + +\noindent +To use Cactus, with the default driver\footnote{For help with unfamiliar terms, please consult the glossary, Appendix \ref{sec:glossary}.} (\texttt{CactusPUGH/PUGH}) on multiple +processors you also need: +\begin{Lentry} +\item[MPI] The \textit{Message Passing Interface}, +which provides inter-processor communication. +Supercomputing sites often supply a native MPI implementation +that is very likely to be compatible with Cactus. Otherwise, there are +various freely available ones available, e.g. the MPICH +version of MPI is available for various architectures and operating +systems at \url{http://www-unix.mcs.anl.gov/mpi/}. +\end{Lentry} + +\noindent +If you are using any thorns containing routines +written in C++ you also need +\begin{Lentry} +\item[C++] C++ compiler. For example, the GNU compiler. This + is available for most supported platforms. Platform specific compilers + should also work. Note that if a C++ compiler is available, then the + \text{main()} routine in the flesh is compiled with C++ to allow static + class initialisations. +\end{Lentry} + +\noindent +If you are using any thorns containing routines +written in Fortran you also need +\begin{Lentry} +\item[F90/F77] For routines written in Fortran 77, either a Fortran 90 or + a Fortran 77 compiler can be used. For routines written in Fortran 90, + a Fortran 90 compiler is obviously required. There is a very limited set of + free Fortran 90 compilers available for the different architectures. +\end{Lentry} + +\noindent +While not required for compiling or running Cactus, for thorn development +it is useful to install +\begin{Lentry} +\item[\texttt{ctags/etags}] These programs enable you browse through the + calling structure of a program by help of a function call database. + Navigating the flesh and arrangements becomes very easy. Emacs and + \texttt{vi} both support this method. See \ref{sec:Appendix.tags} for a short + guide to tags. +\end{Lentry} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\section{Supported Architectures} +\label{sec:suar} + +Cactus runs on many machines, under a large number of operating +systems. Here, we list the machines we have compiled and verified +Cactus on, including some architecture specific notes. A complete +list of architectures supported by Cactus, along with more notes, can +be found at +\begin{center} +\url{http://www.cactuscode.org/Documentation/Architectures.html}. +\end{center} + +\begin{Lentry} +\item[\textbf{SGI}] 32 or 64 bit running Irix. +\item[\textbf{Cray T3E}] +\item[\textbf{Compaq Alpha}] Compaq operating system and Linux. + Single processor + mode and MPI supported. The Alphas need to have the GNU C/C++ + compilers installed. +\item[\textbf{IA32}] running Linux, OpenBSD, FreeBSD, or Windows 2000/NT. + Single processor mode and MPI (MPICH and LAM) supported.\\ + On Windows, Cactus compiles with Cygwin. MPI (WMPI, HPVM, and MPIPro) + supported. Please read \texttt{doc/README.NT} for details. +\item[\textbf{IA64}] running Linux. +\item[\textbf{Macintosh PowerPC}] (MacOS X and Linux PPC) +\item[\textbf{IBM SP2,SP3,SP4}] 32 or 64 bit running AIX. +\item[\textbf{Hitachi SR8000-F1}] +\item[\textbf{Sun} Solaris] +\item[\textbf{Fujitsu}] +\item[\textbf{NEC SX-5, SX-6}] +\end{Lentry} + +The following machines are only partially supported, +\begin{Lentry} +\item[\textbf{HP Exemplar}] +\end{Lentry} + +\subsection{Note} +Disk space may be a problem on supercomputers where home directories are small. +A workaround is to first create a +configs directory on scratch space, say \texttt{scratch/cactus\_configs/} (where +\texttt{scratch/} is your scratch directory), and then either +\begin{itemize} +\item{} set the environment variable \texttt{CACTUS\_CONFIGS\_DIR} to point to +this directory +\end{itemize} +or +\begin{itemize} +\item{} soft link this directory (\texttt{ln -s +scratch/cactus\_configs Cactus/configs/}) to the Cactus directory, if your +filesystem supports soft links. +\end{itemize} + + +\chapter{Compilation} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\section{Configuration Options} +\label{sec:confopts} + +There are four ways to pass options to the configuration process. +% from the gmake command line. +\begin{enumerate} + \item[1]{} + Pass options individually in shell environment variables: + + {\tt export