General restructuring.

git-svn-id: http://svn.cactuscode.org/flesh/trunk@4548 17b73243-c579-4c4c-a9d2-2d5706c11dac

8 files changed, 1027 insertions, 858 deletions

diff --git a/doc/UsersGuide/Appendices.tex b/doc/UsersGuide/Appendices.tex
index 2db45a31..a7904e1a 100644
--- a/doc/UsersGuide/Appendices.tex
+++ b/doc/UsersGuide/Appendices.tex
@@ -1174,6 +1174,421 @@ No other lines should be output by the script.
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 
+\chapter{Utility Routines}
+
+\section{Introduction}
+
+As well as the high-level \verb|CCTK_|* routines, Cactus also
+provides a set of lower-level \verb|Util_|* utility routines, which
+are mostly independent of the rest of Cactus.  This chapter gives a
+general overview of programming with these utility routines.
+
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+
+\section{Key/Value Tables}
+
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+
+\subsection{Motivation}
+
+Cactus functions may need to pass information through a generic
+interface.  In the past, we have used various ad hoc means to do this,
+and we often had trouble passing "extra" information that wasn't
+anticipated in the original design.  For example, for periodic output
+of grid variables,
+\verb|CCTK_OutputVarAsByMethod()| requires that
+any parameters (such as hyperslabbing parameters) be appended as an option
+string to the variable's character string name.  Similarly, elliptic
+solvers often need to pass various parameters, but we haven't had a
+good way to do this.
+
+Key/value tables (\textit{tables} for short) provide a clean solution
+to these problems.  They're implemented by the \verb|Util_Table|*
+functions (described in detail in the Reference Manual).
+
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+
+\subsection{The Basic Idea}
+
+Basically, a table is an object which maps strings to almost arbitrary
+user-defined data.  (If you know Perl, a table is very much like a
+Perl hash table.  Alternatively, if you know Unix shells, a table is
+like the set of all environment variables.  As yet another analogy,
+if you know Awk, a table is like an Awk associative array.)%%%
+\footnote{%%%
+	 However, the present Cactus tables implementation
+	 is optimized for a relatively small number of
+	 distinct keys in any one table.  It will still
+	 work OK for huge numbers of keys, but it will be
+	 slow.
+	 }%%%
+
+More formally, a table is an object which stores a set of \textit{keys}
+and a corresponding set of \textit{values}.  We refer to a (key,value)
+pair as a table \textit{entry}.
+
+Keys are C-style null-terminated character strings, with the slash
+character `{\tt /}' reserved for future expansion.%%%
+\footnote{%%%
+	 Think of hierarchical tables for storing
+	 tree-like data structures.%%%
+	 }%%%
+
+Values are 1-dimensional arrays of any of the usual Cactus data types
+described in Section~\ref{sect-ThornWriting/DataTypes}.
+A string can be stored by treating it as a 1-dimensional array of
+\verb|CCTK_CHAR| (there's an example of this below).
+
+The basic ``life cycle'' of a table looks like this:
+\begin{enumerate}
+\item	Some code creates it with \verb|Util_TableCreate()|
+	or \verb|Util_TableClone()|.
+\item	Some code (often the same piece of code, but maybe some
+	other piece) sets entries in it using one or more of
+	the \verb|Util_TableSet*()|, \verb|Util_TableSet*Array()|,
+	\verb|Util_TableSetGeneric()|, \verb|Util_TableSetGenericArray()|,
+	and/or \verb|Util_TableSetString()| functions.
+\item	Some other piece or pieces of code can get (copies of)
+	the values which were set, using one or more of the
+	\verb|Util_TableGet*()|, \verb|Util_TableGet*Array()|,
+	\verb|Util_TableGetGeneric()|, \verb|Util_TableGetGenericArray()|,
+	and/or \verb|Util_TableGetString()| functions.
+\item	When everyone is through with a table, some (single)
+	piece of code should destroy it with \verb|Util_TableDestroy()|.
+\end{enumerate}
+
+There are also convenience functions \verb|Util_TableSetFromString()|
+to set entries in a table based on a parameter-file-style string,
+and \verb|Util_TableCreateFromString()| to create a table and then
+set entries in it based on a parameter-file-style string.
+
+As well, there are ``table iterator'' functions \verb|Util_TableIt*()|
+to allow manipulation of a table even if you don't know its keys.
+
+A table has an integer ``flags word'' which may be used to specify
+various options, via bit flags defined in \verb|util_Table.h|.
+For example, the flags word can be used to control whether keys
+should be compared as case sensitive or case insensitive strings.
+See the detailed function description of \verb|Util_TableCreate()|
+in the Reference Manual for a list
+of the possible bit flags and their semantics.
+
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+
+\subsection{A Simple Example}
+\label{Tables_Simple_Example}
+
+Here's a simple example (in C)%%%
+\footnote{%%%
+	 All (or almost all) of the table routines
+	 are also usable from Fortran.  See the full
+	 descriptions in the Reference Manual
+	 for details.
+	 }%%%
+{} of how to use a table:
+\begin{verbatim}
+#include "util_Table.h"
+#include "cctk.h"
+
+/* create a table and set some entries in it */
+int handle = Util_TableCreate(UTIL_TABLE_FLAGS_DEFAULT);
+Util_TableSetInt(handle, 2, "two");
+Util_TableSetReal(handle, 3.14, "pi");
+
+...
+
+/* get the values from the table */
+CCTK_INT two_value;
+CCTK_REAL pi_value;
+Util_TableGetInt(handle, &two_value, "two");    /* sets two_value = 2 */
+Util_TableGetReal(handle, &pi_value, "pi");     /* sets pi_value = 3.14 */
+\end{verbatim}
+
+Actually, you shouldn't write code like this---in the real world
+errors sometimes happen, and it's much better to catch them close to
+their point of occurrence, rather than silently produce garbage results
+or crash your program.  So, the \emph{right} thing to do is to always
+check for errors.  To allow this, all the table routines return a status,
+which is zero or positive for a successful return, but negative if
+and only if some sort of error has occurred.%%%
+\footnote{%%%
+	 Often (as in the examples here) you don't care
+	 about the details of which error occurred.  But if
+	 you do, there are various error codes defined in
+	 {\t util\_Table.h} and {\t util\_ErrorCodes.h};
+	 the detailed function descriptions in
+	 the Reference Manual
+	 say which error codes each function can return.
+	 }%%%
+{}  So, the above example should be rewritten like this:
+
+\begin{verbatim}
+#include "util_Table.h"
+
+/* create a table and set some entries in it */
+int handle = Util_TableCreate(UTIL_TABLE_FLAGS_DEFAULT);
+if (handle < 0)
+        CCTK_WARN(CCTK_WARN_ABORT, "couldn't create table!");
+
+/* try to set some table entries */
+if (Util_TableSetInt(handle, 2, "two") < 0)
+        CCTK_WARN(CCTK_WARN_ABORT, "couldn't set integer value in table!");
+if (Util_TableSetReal(handle, 3.14, "pi") < 0)
+        CCTK_WARN(CCTK_WARN_ABORT, "couldn't set real value in table!");
+
+...
+
+/* try to get the values from the table */
+CCTK_INT two_value;
+CCTK_REAL pi_value;
+if (Util_TableGetInt(handle, &two_value, "two") < 0)
+        CCTK_WARN(CCTK_WARN_ABORT, "couldn't get integer value from table!");
+if (Util_TableGetReal(handle, &pi_value, "pi") < 0)
+        CCTK_WARN(CCTK_WARN_ABORT, "couldn't get integer value from table!");
+
+/* if we get to here, then two_value = 2 and pi_value = 3.14 */
+\end{verbatim}
+
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+
+\subsection{Arrays as Table Values}
+
+As well as a single numbers (or characters or pointers), tables can
+also store 1-dimensional arrays of numbers (or characters or pointers).%%%
+\footnote{%%%
+	 Note that the table makes (stores) a \emph{copy} of the array
+	 you pass in, so it's somewhat inefficient to store a large array
+	 (e.g.~a grid function) this way.  If this is a problem, consider
+	 storing a \texttt{CCTK\_POINTER} (pointing to the array) in the table
+	 instead.  (Of course, this requires that you ensure that the array still exists whenever that \texttt{CCTK\_POINTER} is used.)
+	 }%%%
+
+For example (continuing the previous example):
+\begin{verbatim}
+static const CCTK_INT a[3] = { 42, 69, 105 };
+if (Util_TableSetIntArray(handle, 3, a, "my array") < 0)
+        CCTK_WARN(CCTK_WARN_ABORT, "couldn't set integer array value in table!");
+
+...
+
+CCTK_INT blah[10];
+int count = Util_TableGetIntArray(handle, 10, blah, "my array");
+if (count < 0)
+        CCTK_WARN(CCTK_WARN_ABORT, "couldn't get integer array value from table!");
+/* now count = 3, blah[0] = 42, blah[1] = 69, blah[2] = 105, */
+/*     and all remaining elements of blah[] are unchanged */
+\end{verbatim}
+As you can see, a table entry remembers the length of any array
+value that has been stored in it.%%%
+\footnote{%%%
+	 In fact, actually \emph{all} table values are
+	 arrays---setting or getting a single value is
+	 just the special case where the array length is 1.
+	 }%%%
+{}
+
+If you only want the first few values of a larger array, just pass
+in the appropriate length of your array,
+that's OK:
+\begin{verbatim}
+CCTK_INT blah2[2];
+int count = Util_TableGetIntArray(handle, 2, blah2, "my array");
+if (count < 0)
+        CCTK_WARN(CCTK_WARN_ABORT, "couldn't get integer array value from table!");
+/* now count = 3, blah2[0] = 42, blah2[1] = 69 */
+\end{verbatim}
+You can even ask for just the first value:
+\begin{verbatim}
+CCTK_INT blah1;
+int count = Util_TableGetInt(handle, &blah1, "my array");
+if (count < 0)
+        CCTK_WARN(CCTK_WARN_ABORT, "couldn't get integer array value from table!");
+/* now count = 3, blah1 = 42 */
+\end{verbatim}
+
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+
+\subsection{Character Strings}
+
+One very common thing you might want to store in a table is a
+character string.  While you could do this by explicitly storing
+an array of \verb|CCTK_CHAR|, there are also routines
+specially for conveniently setting and getting strings:
+\begin{verbatim}
+if (Util_TableSetString(handle, "black holes are fun", "bh") < 0)
+        CCTK_WARN(CCTK_WARN_ABORT, "couldn't set string value in table!");
+
+...
+char buffer[50];
+if (Util_TableGetString(handle, 50, buffer, "bh") < 0)
+        CCTK_WARN(CCTK_WARN_ABORT, "couldn't get string value from table!");
+
+/* now buffer[] contains the string "black holes are fun" */
+\end{verbatim}
+
+\verb|Util_TableGetString()| guarantees that the string is
+terminated by a null character (`\verb|\0|'), and also returns an
+error if the string is too long for the buffer.
+
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+
+\subsection{Convenience Routines}
+
+There are also convenience routines for the common case of setting
+values in a table based on a string.
+
+For example, the following code sets up exactly the same table as the
+example in Section \ref{Tables_Simple_Example}:
+
+\begin{verbatim}
+#include <util_Table.h>
+
+/* create a table and set some values in it */
+int handle = Util_TableCreate(UTIL_TABLE_FLAGS_DEFAULT);
+if (handle < 0)
+        CCTK_WARN(CCTK_WARN_ABORT, "couldn't create table!");
+
+/* try to set some table entries */
+if (Util_TableSetFromString(handle, "two=2 pi=3.14") != 2)
+        CCTK_WARN(CCTK_WARN_ABORT, "couldn't set values in table!");
+\end{verbatim}
+
+There is also an even higher-level convenience function
+\verb|Util_TableCreateFromString()|: this creates a table with the
+case insensitive flag set (to match Cactus parameter file semantics),
+then (assuming no errors occurred) calls \verb|Util_TableSetFromString()|
+to set values in the table.
+
+For example, the following code sets up a table (with the case insensitive flag
+set) with four entries: an integer number ({\tt two}), a real number ({\tt
+pi}), a string ({\tt buffer}), and an integer array with three elements ({\tt
+array}):
+
+\begin{verbatim}
+#include <util_Table.h>
+
+int handle = Util_TableCreateFromString(" two    = 2 "
+                                        " pi     = 3.14 "
+                                        " buffer = 'Hello World' "
+                                        " array  = { 1 2 3 }");
+if (handle < 0)
+        CCTK_WARN(CCTK_WARN_ABORT, "couldn't create table from string!");
+\end{verbatim}
+
+Note that this code passes a single string to
+\verb|Util_TableCreateFromString()|%%%
+\footnote{C automatically concatenates
+adjacent character string constants separated only by whitespace.}, 
+which then gets parsed into key/value pairs, with the key separated from its
+corresponding value by an equals sign.
+
+Values for numbers are converted into integers ({\tt CCTK\_INT}) if possible
+(no decimal point appears in the value), otherwise into reals ({\tt CCTK\_REAL}).
+Strings must be enclosed in either single or double quotes. String values in
+single quotes are interpreted literally, strings in double quotes may contain
+character escape codes which then will be interpreted as in C.
+Arrays must be enclosed in curly braces, array elements must be single numbers
+of the same type (either all integer or all real).
+
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+
+\subsection{Table Iterators}
+\label{sect-UtilityRoutines/tables/table-iterators}
+
+In the examples up to now, the code, which wanted to get values from
+the table, knew what the keys were.  It's also useful to be able to
+write generic code which can operate on a table without knowing the
+keys.  ``Table iterators'' (``iterators'', for short) are used for this.
+
+An iterator is an abstraction of a pointer to a particular table entry.
+Iterators are analogous to the \verb|DIR *| pointers used by the POSIX
+\verb|opendir()|, \verb|readdir()|, \verb|closedir()|, and similar
+functions, to Perl hash tables' \verb|each()|, \verb|keys()|,
+and \verb|values()|, and to the C++ Standard Template Library's
+forward iterators.
+
+At any time, the entries in a table may be considered to be in some
+arbitrary (implementation-defined) order; an iterator may be used to
+walk through some or all of the table entries in this order.  This
+order is guaranteed to remain unchanged for any given table, so long
+as no changes are made to that table, \ie{} so long as no
+\verb|Util_TableSet*()|, \verb|Util_TableSet*Array()|,
+\verb|Util_TableSetGeneric()|, \verb|Util_TableSetGenericArray()|,
+\verb|Util_TableSetString()|, or \verb|Util_TableDeleteKey()| calls
+are made on that table (making such calls on other tables doesn't
+matter).  The order may change if there is any change in the table,
+and it may differ even between different tables with identical key/value
+contents (including those produced by \verb|Util_TableClone()|).%%%
+\footnote{%%%
+	 For example, if tables were implemented by hashing,
+	 the internal order could be that of the hash buckets,
+	 and the hash function could depend on the internal
+	 table address.
+	 }%%%
+{}
+
+Any change in the table also invalidates all iterators pointing
+anywhere in the table; using any such iterator is an error.
+Multiple iterators may point into the same table; they all use the
+same order, and (unlike in Perl) they're all independent.
+
+The detailed function description
+in the Reference Manual
+for \verb|Util_TableItQueryKeyValueInfo()| has an example of
+using an iterator to print out all the entries in a table.
+
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+
+\subsection{Multithreading and Multiprocessor Issues}
+
+At the moment, the table functions are \emph{not} thread-safe
+in a multithreaded environment. 
+%% However, this should change in
+%%the not-too-distant future: then all the table functions will default
+%to being thread-safe.  That is, user code will be able call these
+%%functions concurrently from multiple threads, with the table functions
+%%automatically doing any necessary locking.%%%
+%%\footnote{%%%
+%	 For the implementation, this means that we will need a
+%	 reader/writer lock for each table and for each iterator:
+%	 any number of threads will be able to concurrently read
+%	 the data structure, but any write will require exclusive
+%	 access.
+%	 }%%%
+%{}  (We may add a flags-word bit to suppress this for maximum
+%performance if you know you won't be making concurrent calls from
+%multiple threads.)
+
+Note that tables and iterators are process-wide, i.e. all
+threads see the same tables and iterators (think of them as like the
+Unix current working directory, with the various routines which modify
+the table or change iterators acting like a Unix \verb|chdir()| system
+call).
+
+In a multiprocessor environment, tables are always processor-local.
+
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+
+\subsection{Metadata about All Tables}
+
+Tables do not \emph{themselves} have names or other
+attributes.  However, we may add some special
+``system tables'' to be used by Cactus itself to store this sort of
+information for those cases where it's needed. For example, we may add support for a
+``checkpoint me'' bit in a table's flags word, so that if you want a
+table to be checkpointed, you just need to set this bit.
+In this case, the table will probably get a system generated name in
+the checkpoint dump file.  But if you want the table to have some
+other name in the dump file, then you need to tell the checkpointing
+code that, by setting an appropriate entry in a checkpoint table.
+(You would find the checkpoint table by looking in a special
+``master system table'' that records handles of other interesting tables.)
+
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+
+
 \chapter{Schedule Bins}
 \label{sec:Appendix.schedule_bins}
 
diff --git a/doc/UsersGuide/ThornWriters.tex b/doc/UsersGuide/ApplicationThorns.tex
index 2b3bd6d9..c174c551 100644
--- a/doc/UsersGuide/ThornWriters.tex
+++ b/doc/UsersGuide/ApplicationThorns.tex
@@ -16,16 +16,16 @@
 %   @endhistory
 % @@*/
 
-\begin{cactuspart}{Application Thorn Writing}{$RCSfile$}{$Revision$}
-\label{part:ThornWriters}
+\begin{cactuspart}{Thorn Writing}
 \renewcommand{\thepage}{\Alph{part}\arabic{page}}
+\label{part:ThornWriting}
+
+\chapter{Application thorns}
 
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 
-\chapter{Overview}
-
 This chapter goes into the nitty-gritty of writing a thorn.
 It introduces key concepts for thorns, then goes on to give
 a brief outline of how to configure a thorn.
@@ -37,12 +37,12 @@ you may use to gain extra functionality.
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 
-\chapter{Thorn Concepts}
+\section{Thorn Concepts}
 \label{chap:thorn_concepts}
 
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 
-\section{Thorns}
+\subsection{Thorns}
 
 A thorn is the basic working module within Cactus.  All user supplied
 code goes into thorns, which are, by and large, independent of each other.
@@ -63,7 +63,7 @@ ignored.
 
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 
-\section{Arrangements}
+\subsection{Arrangements}
 \label{sec:arrangements}
 
 Thorns are grouped into \textit{arrangements}.  This is a logical grouping of
@@ -86,7 +86,7 @@ belonging to the arrangement.
 
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 
-\section{Implementations}
+\subsection{Implementations}
 
 \label{sec:implementations}
 
@@ -116,12 +116,12 @@ the other thorns.
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 
-\chapter{Anatomy of a Thorn}
+\section{Anatomy of a Thorn}
 \label{chap:thorn_anatomy}
 
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 
-\section{Thorns}
+\subsection{Thorns}
 
 A thorn consists of a subdirectory of an arrangement containing four
 administrative files:
@@ -152,7 +152,7 @@ Thorns can also contain
 
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 
-\section{Creating a Thorn}
+\subsection{Creating a Thorn}
 
 
 To simplify the creation of a thorn, a \texttt{make} target {\tt
@@ -167,7 +167,7 @@ pick one from the list of available arrangements that are shown.
 
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 
-\section{Configuring your Thorn}
+\subsection{Configuring your Thorn}
 
 The interaction of a thorn with the flesh and other thorns is controlled
 by certain configuration files.
@@ -195,7 +195,7 @@ configuration options of this thorn.
 
 \end{Lentry}
 
-\subsection{General Syntax of CCL Files}
+\subsubsection{General Syntax of CCL Files}
 
 \textit{Cactus Configuration Language} (CCL) files are simple text files
 used to define configuration information for a thorn.  CCL files are
@@ -205,7 +205,7 @@ non-blank character of a line in a CCL file is a backslash
 `\texttt{$\backslash$}', the
 following line is treated as a continuation of the current line.
 
-\subsection{The \texttt{interface.ccl} File}
+\subsubsection{The \texttt{interface.ccl} File}
 \label{subsec:interface_ccl}
 
 The \texttt{interface.ccl} file is used to declare
@@ -300,7 +300,7 @@ changes the access level for any group defined in the file from that point on.
 
 All variables seen by any one thorn must have distinct names.
 
-\subsection{The \texttt{param.ccl} File}
+\subsubsection{The \texttt{param.ccl} File}
 \label{subsec:param_ccl}
 
 Users control the operation of thorns via parameters given in a file
@@ -438,7 +438,7 @@ Note that you must compile at least one thorn which implements \texttt{einstein}
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 
-\subsection{The \texttt{schedule.ccl} File}
+\subsubsection{The \texttt{schedule.ccl} File}
 \label{subsec:schedule_ccl}
 By default, no routine of a thorn will be run.  The \texttt{schedule.ccl}
 file defines those that should be run, and when and under which
@@ -619,7 +619,7 @@ schedule WaveBinary AT evol AFTER WaveToy_Evolution
 } "Provide binary source during evolution"
 \end{verbatim}
 
-\subsubsection{Storage Outside of Schedule Blocks}
+\paragraph{Storage Outside of Schedule Blocks}
 The keyword \texttt{STORAGE} can also be used outside of the schedule
 blocks to indicate that storage for these groups should be switched on
 at the start of the run. Note that the storage is only allocated in
@@ -631,7 +631,7 @@ timelevels to activate storage for.
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 
-\subsection{The \texttt{configuration.ccl}}
+\subsubsection{The \texttt{configuration.ccl}}
 \label{subsec:configuration_ccl}
 
 [{\bf NOTE:} the \texttt{configuration.ccl} is a new feature, and not all the
@@ -725,7 +725,7 @@ Appendix \ref{sec:Appendix.configuration.ccl.configscript}.
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 
-\section{Naming Conventions for Source Files}
+\subsection{Naming Conventions for Source Files}
 \label{nacofosofi}
 
 The make system uses file extensions to designate coding language,
@@ -776,7 +776,7 @@ however.
 
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 
-\section{Adding Source Files}
+\subsection{Adding Source Files}
 
 By default, the CCTK looks in the \texttt{src} directory of the thorn for source
 files.
@@ -786,7 +786,7 @@ There are two ways in which to specify the sources.  The easiest is to use the
 may instead put a \texttt{Makefile} in the \texttt{src} directory and do everything
 yourself.
 
-\subsection{\texttt{make.code.defn} based thorn building}
+\subsubsection{\texttt{make.code.defn} based thorn building}
 \label{sec:mabathbu}
 
 This is the standard way to compile your thorn's source files.
@@ -810,7 +810,7 @@ for files in that directory, can contain additional make rules and dependencies
 for files in that directory.  See the GNU Make documentation for complete details of the
 syntax.
 
-\subsection{\texttt{Makefile} based thorn building}
+\subsubsection{\texttt{Makefile} based thorn building}
 
 This method gives you the ultimate responsibility.
 The only requirement is that
@@ -839,7 +839,7 @@ files should end up if they need to be seen by other thorns.
 and has a working directory of
 \texttt{<\var{config}>/build/<\var{thorn\_name}>} .
 
-\subsection{Other makefile variables}
+\subsubsection{Other makefile variables}
 
 \begin{itemize}
 \item \texttt{CPP}: The C preprocessor which is used to preprocess C
@@ -869,7 +869,7 @@ and has a working directory of
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 
-\chapter{Cactus Variables}
+\section{Cactus Variables}
 \label{chap:cactus_variables}
 
 A \textit{grid variable} is a Cactus program variable passed among thorns,
@@ -918,7 +918,7 @@ Currently, the names of groups and variables must be distinct.
 
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 
-\section{Data Type}
+\subsection{Data Type}
 
 Cactus supports integer, real, complex and character variable types, in
 various different sizes.  (Sizes in the following refer to the number of
@@ -945,7 +945,7 @@ quickly with a lower accuracy.
 
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 
-\section{Group Types}
+\subsection{Group Types}
 \label{sec:cactus_variables-groups}
 
 Groups can be either \textit{scalars}, \textit{grid functions} (GFs), or
@@ -975,7 +975,7 @@ quicker methods available for GFs.
 
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 
-\section{Timelevels}
+\subsection{Timelevels}
 
 These are best introduced by an example using finite differencing.
 Consider the 1-D wave equation
@@ -1032,7 +1032,7 @@ values at the previous timelevels do need to be explicitly filled out.
 
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 
-\section{Size and Distrib}
+\subsection{Size and Distrib}
 
 A Cactus grid function or array has a size set at runtime by parameters.
 This size can either be the global size of the array across all processors
@@ -1044,7 +1044,7 @@ responsible for assigning the size on each processor.
 
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 
-\section{Ghost Zones}
+\subsection{Ghost Zones}
 \label{sec:ghost_size}
 
 Cactus is based upon a \textit{distributed computing} paradigm.  That is,
@@ -1101,7 +1101,7 @@ ghostzones as your stencil size requires.
 
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 
-\section{Staggering}
+\subsection{Staggering}
 \label{sec:staggering}
 
 The staggering of a grid function or array describes the \emph{physical}
@@ -1160,12 +1160,12 @@ points (arrows) are accessed by the same indices.}
 \label{fig:stagger1}
 \end{figure}
 
-\section{Information about Grid Variables}
+\subsection{Information about Grid Variables}
 
 The flesh holds a database with information related to grid variables, and
 provides a set of querying APIs.
 
-\subsection{Group Information}
+\subsubsection{Group Information}
 
 Fundamental information about grid functions (e.g.\ local grid size and
 location, number of ghostzones) is passed through the argument list of
@@ -1210,7 +1210,7 @@ the information is requested using a group name (\texttt{GN}) or index
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 
-\chapter{Cactus Parameters}
+\section{Cactus Parameters}
 \label{chap:Cactus_parameters}
 
 Parameters are the means by which the user specifies the runtime behaviour of
@@ -1247,7 +1247,7 @@ each thorn using the Cactus command line options \texttt{-o} and \texttt{-O}
 
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 
-\section{Types and Ranges}
+\subsection{Types and Ranges}
 \label{sec:Parameters.Types_and_Ranges}
 
 Parameters can be of these types:
@@ -1266,7 +1266,7 @@ Each parameter can be validated against a set of allowed
 The nature
 of the range is determined by the type of parameter, as follows:
 
-\subsection{Int}
+\subsubsection{Int}
 
 The range specification is of the form
 
@@ -1285,7 +1285,7 @@ means the value must be an odd number between one and twenty-one
 A missing end of range (or a `\texttt{*}') indicates negative or positive
 infinity, and the default stride is one.
 
-\subsection{Real}
+\subsubsection{Real}
 
 The range specification is of the form
 \begin{alltt}
@@ -1305,16 +1305,16 @@ double, then C will typecast appropriately.  If you \emph{really} want to
 specify a single precision floating constant, or a long double
 constant, append the number with {\t f} or {\t l} respectively.)
 
-\subsection{Keyword}
+\subsubsection{Keyword}
 
 The range specification consists of a string, which will be matched in
 a case insensitive manner.
 
-\subsection{Boolean}
+\subsubsection{Boolean}
 
 There is no range specification for this type of parameter.
 
-\subsection{String}
+\subsubsection{String}
 
 The range is a POSIX regular expression.  On some machines you may be
 able to use extended regular expressions, but this is not guaranteed
@@ -1322,7 +1322,7 @@ to be portable.
 
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 
-\section{Scope}
+\subsection{Scope}
 \label{sec:Cactus_parameters.scope}
 
 Parameters can be \texttt{GLOBAL}, \texttt{RESTRICTED}, or \texttt{PRIVATE}.
@@ -1331,7 +1331,7 @@ are visible to any thorn which chooses to \texttt{USE} or \texttt{EXTEND}
 it.  A private parameter is only visible to the thorn which declares
 it.
 
-\section{Steerable}
+\subsection{Steerable}
 \label{sec:Cactus_parameters.steerable}
 A parameter can be changed dynamically if it is specified to be
 \textit{steerable} (see
@@ -1343,7 +1343,7 @@ of this function).
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 
-\chapter{Scheduling}
+\section{Scheduling}
 \label{chap:scheduling}
 
 Cactus contains a rule-based scheduling system, which determines which
@@ -1385,7 +1385,7 @@ Each schedule item is scheduled either \texttt{AT} a particular
 
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 
-\section{Schedule Bins}
+\subsection{Schedule Bins}
 \label{scheduling:schedule_bins}
 
 These are the main times at which scheduled functions are run, from
@@ -1403,7 +1403,7 @@ in the \texttt{schedule.ccl} file.
 
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 
-\section{Groups}
+\subsection{Groups}
 \label{scheduling:groups}
 
 If the optional \texttt{GROUP} specifier is used, the item is a schedule
@@ -1414,7 +1414,7 @@ AT} the main schedule bins.  (That is, groups may be nested.)
 
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 
-\section{Schedule Options}
+\subsection{Schedule Options}
 \label{scheduling:schedule_options}
 The options define various characteristics of the schedule item.
 
@@ -1476,7 +1476,7 @@ scheduler will abort with a fatal error.
 
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 
-\section{The Schedule Block}
+\subsection{The Schedule Block}
 \label{scheduling:schedule_block}
 
 The schedule block specifies further details of the scheduled function
@@ -1506,7 +1506,7 @@ given in Appendix~\ref{app:allopts}.
 
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 
-\section{How Cactus Calls Scheduled Functions}
+\subsection{How Cactus Calls Scheduled Functions}
 \label{scheduling:calling_scheduled_functions}
 
 For each scheduled function called, the flesh performs a variety of jobs
@@ -1536,11 +1536,11 @@ returned to the original state.
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 
-\chapter{Writing a Thorn}
+\section{Writing a Thorn}
 
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 
-\section{Thorn Programming Languages}
+\subsection{Thorn Programming Languages}
 
 When you start writing a new thorn, the first decision to make is
 which programming language to use. The source code in Cactus thorns
@@ -1569,7 +1569,7 @@ use machine dependent extensions.
 
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 
-\section{What the Flesh Provides}
+\subsection{What the Flesh Provides}
 
 The flesh provides for thorns:
 \begin{Lentry}
@@ -1588,7 +1588,7 @@ The flesh provides for thorns:
 \end{Lentry}
 
 
-\subsection{Fortran Routines}
+\subsubsection{Fortran Routines}
 
 Any source file using Cactus infrastructure should include
 the header file \texttt{cctk.h} using the line
@@ -1599,7 +1599,7 @@ the header file \texttt{cctk.h} using the line
 header file---most Cactus files are run through a C preprocessor
 before compilation.)
 
-\subsubsection{Variables}
+\paragraph{Variables}
 
 Any routine using Cactus argument lists (for example, all routines called from
 the scheduler at time bins between {\t CCTK\_STARTUP} and {\t CCTK\_SHUTDOWN})
@@ -1630,7 +1630,7 @@ Note that you cannot use Cactus argument lists in routines scheduled at the
 no grid hierarchy exists.
 
 
-\subsubsection{Parameters}
+\paragraph{Parameters}
 
 All parameters defined in a thorn's \texttt{param.ccl} and all \texttt{global}
 parameters, appear as local variables of the corresponding \texttt{CCTK} data type
@@ -1674,7 +1674,7 @@ a Fortran string. Instead, the following is needed:
 
 
 
-\subsubsection{Fortran Example}
+\paragraph{Fortran Example}
 
 The Fortran routine, \verb|MyFRoutine|, is scheduled in the {\tt
 schedule.ccl} file, doesn't use Cactus parameters, and calls another
@@ -1717,7 +1717,7 @@ c     Main code goes here
 
 \end{verbatim}
 
-\subsubsection{Cactus Fortran Functions}
+\paragraph{Cactus Fortran Functions}
 
 Cactus Fortran functions, for example, \texttt{CCTK\_MyProc} and {\tt
 CCTK\_Equals}, can all be declared by adding the statement
@@ -1735,7 +1735,7 @@ DECLARE_CCTK_FUNCTIONS
 to a module or a subroutine after the \texttt{implicit none}
 statement, but before any executable code.
 
-\subsubsection{Fortran Modules}
+\paragraph{Fortran Modules}
 
 Fortran modules should be placed into source files that have the same
 name as the module, followed by the corresponding file name suffix.  A
@@ -1754,7 +1754,7 @@ MyModule.F90}, then add the line:
 MyRoutine.F90.o:         MyModule.F90.o
 \end{verbatim}
 
-\subsubsection{The \texttt{MOD} function}
+\paragraph{The \texttt{MOD} function}
 
 The intrinsic function \texttt{MOD} in Fortran takes two integer
 arguments, which should both be of the same type. This means
@@ -1768,7 +1768,7 @@ MOD(cctk_iteration,INT(MyParameter))
 \end{verbatim}
 
 
-\subsection{C Routines}
+\subsubsection{C Routines}
 
 Any source file using Cactus infrastructure should include
 the header file \texttt{cctk.h} using the line
@@ -1776,7 +1776,7 @@ the header file \texttt{cctk.h} using the line
 #include "cctk.h"
 \end{verbatim}
 
-\subsubsection{Variables}
+\paragraph{Variables}
 
 Any routine using Cactus argument lists (for example, all routines called from
 the scheduler at time bins between {\t CCTK\_STARTUP} and {\t CCTK\_SHUTDOWN}),
@@ -1808,7 +1808,7 @@ Note that you cannot use Cactus argument lists in routines scheduled at the
 no grid hierarchy exists.
 
 
-\subsubsection{Parameters}
+\paragraph{Parameters}
 
 All parameters defined in a thorn's \texttt{param.ccl} and all \texttt{global}
 parameters, appear as local variables of the corresponding \texttt{CCTK} data type
@@ -1827,7 +1827,7 @@ the top of the file the header
 The parameters should be declared as the last statement in the declaration part
 of the routine using them with the macro \texttt{DECLARE\_CCTK\_PARAMETERS}.
 
-\subsubsection{Example}
+\paragraph{Example}
 
 The C routine \verb|MyCRoutine| is scheduled in the \texttt{schedule.ccl} file,
 and uses Cactus parameters. The source file should look like
@@ -1845,7 +1845,7 @@ void MyCRoutine(CCTK_ARGUMENTS)
 }
 \end{verbatim}
 
-\subsubsection{Complex variables}
+\paragraph{Complex variables}
 
 Cactus supports complex grid variables, and since there is no
 complex data type in C, Cactus provides a number
@@ -1858,7 +1858,7 @@ functionality available in Fortran. These functions are \texttt{CCTK\_Cmplx},
 See the Reference Manual for more details.
 
 
-\subsubsection{Specifically for C Programmers}
+\paragraph{Specifically for C Programmers}
 
 Grid functions are held in memory as 1-dimensional C arrays. These are laid
 out in memory as in Fortran. This means that the first index should
@@ -1888,7 +1888,7 @@ Here, \verb|CCTK_GFINDEX3D(cctkGH,i,j,k)| expands to
 ((i) + cctkGH->cctk_lsh[0]*((j)+cctkGH->cctk_lsh[1]*(k)))
 \end{verbatim}
 
-\subsection{Cactus Variables}
+\subsubsection{Cactus Variables}
 \label{sec:cactus_variables_c}
 
 The Cactus variables which are passed through the macro
@@ -2039,7 +2039,7 @@ int i,j,k;
 \label{fig-global-xyz-coords}
 \end{figure}
 
-\subsection{Cactus Data Types}
+\subsubsection{Cactus Data Types}
 
 To provide portability across platforms, the Cactus grid variables and parameters are defined and
 declared using Cactus data types. The most important of
@@ -2057,7 +2057,7 @@ declarations.
 
 \end{Lentry}
 
-\subsubsection{Example}
+\paragraph{Example}
 
 In the following example, \verb|MyScalar| is a grid scalar which
 is declared in the \texttt{interface.ccl} as \texttt{CCTK\_REAL}.
@@ -2081,10 +2081,10 @@ Declaring \texttt{local\_var} to have a non-Cactus data type, e.g.\
 described in Section~\ref{sec:datyansi}, could give problems for
 different architectures or configurations.
 
-\subsection{Staggering}
+\subsubsection{Staggering}
 \label{sec:st}
 
-\subsubsection{Indexing, ghostzones, etc.}
+\paragraph{Indexing, ghostzones, etc.}
 Note that staggering does not make any changes to the indexing of a
 grid function: the black solid circles in diagram \ref{fig:stagger2} and their
 associated staggered grid functions (connected by arrows) have the same index!
@@ -2104,7 +2104,7 @@ Note that the number of center staggered gridpoints (fat crosses)
 corresponds to the number of default gridpoints on all processors but
 the last one. (The same is true for full staggered gridpoints).
 
-\subsubsection{Staggertypes}
+\paragraph{Staggertypes}
 The string specifying the staggering is encoded in a number called
 the \textit{staggerindex}. With the 3 supported staggerings, the string
 is converted into a base 3 number. Several routines exist to extract the
@@ -2225,7 +2225,7 @@ staggerindex.
 
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 
-\section{Parallelisation}
+\subsection{Parallelisation}
 \label{sec:parallelisation}
 
 The flesh itself does not actually set up grid variables. This
@@ -2240,7 +2240,7 @@ for the application thorn writer. A more detailed description
 of these interfaces with their arguments, is given in the Reference Manual.
 A complete description of the
 routines that a driver thorn must provide, will be provided in the
-Infrastructure Thorn Writers Guide (Part \ref{part:Infrastructure}). The standard driver thorn is
+Infrastructure Thorn Writers Guide (Part \ref{chap:infrastructure}). The standard driver thorn is
 currently \texttt{PUGH} in the \texttt{CactusPUGH} package, which
 is a parallel unigrid driver.
 
@@ -2259,11 +2259,11 @@ is a parallel unigrid driver.
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 
-\chapter{Cactus Application Interfaces}
+\section{Cactus Application Interfaces}
 
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 
-\section{Coordinates}
+\subsection{Coordinates}
 \label{sec:CactusAPI.coordinates}
 
 The flesh provides utility routines for registering and querying
@@ -2293,7 +2293,7 @@ an associated integer known as a \textit{handle}. Passing a handle instead
 of the name string may be necessary for calling C routines from Fortran.
 
 
-\subsection{Registering Coordinates and Coordinate Properties}
+\subsubsection{Registering Coordinates and Coordinate Properties}
 
 \textbf{The APIs described in this section are deprecated, and will
 probably be phased out fairly soon.
@@ -2375,7 +2375,7 @@ ierr = CCTK_CoordRegisterPhysIndex(cctkGH, loweri, upperi, -1, "y", "cart3d");
 
 \end{Lentry}
 
-\subsection{Using Coordinates}
+\subsubsection{Using Coordinates}
 
 \textbf{The APIs described in this section are deprecated, and will
 probably be phased out fairly soon.
@@ -2556,7 +2556,7 @@ should usually be the case.
 
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 
-\section{I/O}
+\subsection{I/O}
 \label{sec:io}
 
 To allow flexible I/O, the flesh itself does not provide any output
@@ -2618,7 +2618,7 @@ name of the variable for the purpose of constructing a filename.
 
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 
-\section{Interpolation Operators}
+\subsection{Interpolation Operators}
 \label{sec:inop}
 
 The flesh does not provide interpolation routines by itself. Instead,
@@ -2712,7 +2712,7 @@ name.
 
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 
-\section{Reduction Operators}
+\subsection{Reduction Operators}
 \label{sec:reop}
 
 A reduction operation can be defined as an operation on variables
@@ -3133,11 +3133,11 @@ the reduction call is made.
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 
-\chapter{Completing a Thorn}
+\section{Completing a Thorn}
 
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 
-\section{Commenting Source Code}
+\subsection{Commenting Source Code}
 
 Note that since most source files (see Section~\ref{nacofosofi} for
 exceptions) pass through a C preprocessor, C style comments can be
@@ -3151,7 +3151,7 @@ source code.
 
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 
-\section{Providing Runtime Information}
+\subsection{Providing Runtime Information}
 \label{sec:prrutiin}
 
 To write from thorns to standard output (i.e. the screen)
@@ -3217,8 +3217,8 @@ Note that
 
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 
-\section{Error Handling, Warnings and Code Termination}
-\sectionmark{Error handling, ...}
+\subsection{Error Handling, Warnings and Code Termination}
+\subsectionmark{Error handling, ...}
 \label{sec:erhawancote}
 The Cactus function \texttt{CCTK\_VWarn()}, and its accompanying
 \texttt{CCTK\_WARN} macro, should be used to issue warning messages
@@ -3375,7 +3375,7 @@ CCTK_InfoCallbackRegister(void *data, cctk_infofunc my_infofunc);
 
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 
-\section{Adding Documentation}
+\subsection{Adding Documentation}
 \label{sec:Adding_documentation}
 
 Documentation is a vital part of your thorn, helping to ensure its
@@ -3385,13 +3385,13 @@ to a thorn (ideally in the \texttt{doc} directory), there are two
 standard places for adding thorn documentation, a \texttt{README} and a
 \texttt{doc/documentation.tex} file for including in Thorn Guides.
 
-\subsection{\texttt{README}}
+\subsubsection{\texttt{README}}
 
 The \texttt{README}, in the top level of a thorn, should contain brief
 and essential details about the thorn, such as the authors, any
 copyright details, and a synopsis of what the thorn does.
 
-\subsection{Contribution to Thorn Guide}
+\subsubsection{Contribution to Thorn Guide}
 
 The LaTeX file, \texttt{doc/documentation.tex}, is included in Thorn Guides
 built by the Cactus make system. (e.g.\ by \texttt{gmake
@@ -3504,7 +3504,7 @@ and
 
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 
-\section{Adding a Test Suite}
+\subsection{Adding a Test Suite}
 \label{sec:adding_test_suite}
 
 To add a test suite to your thorn, devise a series of parameter
@@ -3581,7 +3581,7 @@ and a relative tolerance of $10^{-12}$ when running all other tests.
 
 For details on running the test suites, see Section~\ref{sec:testing}.
 
-\subsection{Best Practices for Test Suites}
+\subsubsection{Best Practices for Test Suites}
 
 When writing a test suite, there are a few things you should keep in
 mind:
@@ -3622,14 +3622,14 @@ mind:
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 
-\chapter{Advanced Thorn Writing}
+\section{Advanced Thorn Writing}
 
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 
-\section{Using Cactus Timers}
+\subsection{Using Cactus Timers}
 \label{sec:timers}
 
-\subsection{What are Timers?}
+\subsubsection{What are Timers?}
 
 %The standard timing information available during a simulation was
 %described in Section~\ref{????}.
@@ -3661,7 +3661,7 @@ Setting the flesh parameter \texttt{cactus::cctk\_timer\_output = "full"}
 will cause some summary timing information to be printed at the end of a run. 
 Some other thorns have their own timer printing parameters as well.
 
-\subsection{Timing Calls}
+\subsubsection{Timing Calls}
 
 Many of the timing calls come in two versions, one whose name ends with 
 the letter \texttt{I}, and one without.  The calls whose names end with the 
@@ -3746,7 +3746,7 @@ from the {\t cTimerVal} pointer argument.
 
 \end{Lentry}
 
-\subsection{How to Insert Timers in your Code}
+\subsubsection{How to Insert Timers in your Code}
 
 The function prototypes and structure definitions are contained in the
 include file \texttt{cctk\_Timers.h}, which is included in the standard
@@ -3813,7 +3813,7 @@ this structure.  After using the structure, you should destroy it.
 
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 
-\section{Include Files}
+\subsection{Include Files}
 \label{sec:includefiles}
 
 Cactus provides a mechanism for thorns to add code to
@@ -3843,7 +3843,7 @@ to this include file, declares in its own \texttt{interface.ccl}
 INCLUDE [SOURCE|HEADER]: <\var{file_to_include}> in <\var{file_name}>
 \end{alltt}
 
-\subsubsection{Example}
+\paragraph{Example}
 
 As an example of this in practice, for the case of Fortran code,
 consider a thorn A, which
@@ -3888,7 +3888,7 @@ the declaration \texttt{DECLARE\_CCTK\_FUNCTIONS}.
 
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 
-\section{Memory Tracing}
+\subsection{Memory Tracing}
 \label{sec:metr}
 
 Cactus provides a mechanism for overriding the standard C memory
@@ -3899,7 +3899,7 @@ to provide an understanding of the memory consumption between two
 instances, and to track down possible memory leaks. This feature is
 available in C only.
 
-\subsection{Activating Memory Tracing}
+\subsubsection{Activating Memory Tracing}
 \label{sec:acmetr}
 
 Memory tracing has to be activated at configure time. The standard
@@ -3925,7 +3925,7 @@ The new configuration \texttt{bigbuild} is configured with all debugging
 features turned on. The already existing configuration \texttt{bigbuild}
 is reconfigured with memory tracing only.
 
-\subsection{Using Memory Tracing}
+\subsubsection{Using Memory Tracing}
 \label{sec:usmetr}
 
 You can  request Cactus to store the memory consumption at a certain
@@ -3999,10 +3999,10 @@ CCTK_MemTicketDelete(t2);
 
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 
-\section[Calls to different language]{Calls between Different Programming Languages}
+\subsection[Calls to different language]{Calls between Different Programming Languages}
 %\pagestyle{empty}
 
-\subsection{Calling C Routines from Fortran}
+\subsubsection{Calling C Routines from Fortran}
 \label{sec:cacrofr}
 
 To make the following C routine,
@@ -4051,7 +4051,7 @@ void CCTK_FCALL CCTK_FNAME(MyCRoutine)(CCTK_INT *arg1,
 }
 \end{verbatim}
 
-\subsection{String Arguments from Fortran}
+\subsubsection{String Arguments from Fortran}
 
 Fortran passes string arguments in a special, compiler-dependent, way.
 To facilitate this, the CCTK provides a set of macros to enable the
@@ -4177,7 +4177,7 @@ automatically-created variables, e.g.\ \texttt{cctk\_strlen1},
 which hold the sizes of original Fortran strings.
 When writing to a string its length should never be exceeded.
 
-\subsection{Calling Fortran Routines from C}
+\subsubsection{Calling Fortran Routines from C}
 \label{sec:caforofr}
 
 To call a utility Fortran routine from C, use
@@ -4193,7 +4193,7 @@ strings from C. However, passing routines is supported when you use function ali
 
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 
-\section{Function aliasing}
+\subsection{Function aliasing}
 \label{sec:function_aliasing}
 
 Like calling functions in a different language, Cactus offers a
@@ -4216,7 +4216,7 @@ declarations may go into a new \texttt{functions.ccl} file, and will
 have a format more similar to that of variable group and parameter
 declarations.
 
-\subsection{Using an Aliased Function}
+\subsubsection{Using an Aliased Function}
 
 To use an aliased function you must first declare it in your 
 \texttt{interface.ccl} file. Declare the prototype as, for example,
@@ -4246,7 +4246,7 @@ Functions which are required by some thorn (which doesn't provide it itself)
 are checked at startup to be provided by some other thorn.
 
 
-\subsection{Providing a Function}
+\subsubsection{Providing a Function}
 
 To provide an aliased function you must again add the prototype to
 your \texttt{interface.ccl} file. A statement containing the name of the
@@ -4265,7 +4265,7 @@ It is necessary to specify the language of the providing function; no
 default will be assumed.
 
 
-\subsection{Conventions and Restrictions}
+\subsubsection{Conventions and Restrictions}
 
 Various restrictions are necessary to make function aliasing
 work. These are
@@ -4311,7 +4311,7 @@ CCTK_REAL Integrate(CCTK_REAL CCTK_FPOINTER  func(CCTK_REAL IN x), \
 \end{itemize}
 
 
-\subsection{Examples}
+\subsubsection{Examples}
 
 \begin{itemize}
 \item A C function is provided to add together two real numbers. The {\tt
@@ -4343,7 +4343,7 @@ CCTK_REAL Integrate(CCTK_REAL CCTK_FPOINTER  func(CCTK_REAL IN x), \
 
 \end{itemize}
 
-\subsection{Testing Aliased Functions}
+\subsubsection{Testing Aliased Functions}
 
 The calling thorn does not know if an aliased function is even
 provided by another thorn. Calling an aliased function that has not
@@ -4354,7 +4354,7 @@ function reference section.
 
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 
-\section{Naming Conventions}
+\subsection{Naming Conventions}
 
 \begin{itemize}
 
@@ -4372,7 +4372,7 @@ function reference section.
 
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 
-\section{General Naming Conventions}
+\subsection{General Naming Conventions}
 
 The following naming conventions are followed by the flesh and the
 supported Cactus arrangements. They are not compulsory, but if followed, will
@@ -4391,7 +4391,7 @@ allow for a homogeneous code.
 
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 
-\section{Data Types and Sizes}
+\subsection{Data Types and Sizes}
 \label{sect-ThornWriting/DataTypes}
 \label{sec:datyansi}
 
@@ -4508,7 +4508,7 @@ be used from either C or Fortran:
 \end{center}
 
 
-\subsection{Fortran Thorn Writers}
+\subsubsection{Fortran Thorn Writers}
 
 Cactus provides the data types \texttt{CCTK\_POINTER} and
 \texttt{CCTK\_POINTER\_TO\_CONST} for use in Fortran code to declare a
@@ -4597,26 +4597,25 @@ variables are read, not when they are written to.
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 
-\chapter{Telling the Make system What to Do}
+\section{Telling the Make system What to Do}
 
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 
-\section{Basic Recipe}
+\subsection{Basic Recipe}
 
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 
-\section{Make Concepts}
+\subsection{Make Concepts}
 
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 
-\section{The Four Files}
+\subsection{The Four Files}
 
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 
-\section{How your code is built}
+\subsection{How your code is built}
 
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 
-\end{cactuspart}
diff --git a/doc/UsersGuide/GettingStarted.tex b/doc/UsersGuide/GettingStarted.tex
new file mode 100644
index 00000000..2c8558a1
--- /dev/null
+++ b/doc/UsersGuide/GettingStarted.tex
@@ -0,0 +1,383 @@
+% /*@@
+%   @file      RunningCactus.tex
+%   @date      27 Jan 1999
+%   @author    Tom Goodale, Gabrielle Allen, Gerd Lanferman, Thomas Radke
+%   @desc
+%              How to run Cactus part of the Cactus User's Guide
+%   @enddesc
+%   @version   $Header$
+% @@*/
+
+\begin{cactuspart}{Introduction}{$RCSfile$}{$Revision$}\label{part:Introduction}
+\renewcommand{\thepage}{\Alph{part}\arabic{page}}
+
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+
+\chapter{Getting Started}
+\label{cha:gs}
+
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+
+\section{Obtaining Cactus}
+\label{sec:checkout}
+
+Cactus is distributed, extended, and maintained using the free CVS
+software (\textit{Concurrent Versions System}: \url{http://www.cvshome.org}).
+CVS  allows many people to work on a large software project
+together without getting into a tangle.
+Since Cactus thorns are distributed from several repositories on the
+main CVS site, and from a growing number of user sites, we provide a
+\texttt{GetCactus} script on our website for checking out the flesh 
+and thorns. The script is available at
+
+\url{http://www.cactuscode.org/download/GetCactus}.
+
+The
+script takes as an argument the name of a file containing a \textit{ThornList},
+that is a list of thorns with the syntax
+\begin{alltt}
+<\var{arrangement name}>/<\var{thorn name}>
+\end{alltt}
+
+If no filename is given, only the flesh is checked out.
+Optional directives in the ThornList indicate which CVS repository to fetch
+thorns from. The default is to take the thorns from the same repository as
+the flesh. A full description of ThornList syntax is provided in Appendix~\ref{chap:th}.
+ThornLists for example applications are provided on the Cactus website at
+
+%%% FIXME: url
+
+The same script can be used to checkout additional thorns, or to update 
+existing ones.
+
+The Cactus website also provides a form interface for direct download at:
+
+\url{http://www.cactuscode.org/cactus\_cgi-bin/download.pl}
+
+CVS experts who want to use raw CVS commands are directed to
+Appendix~\ref{sec:Appendix.cvs} for full instructions. For CVS novices,
+we also summarize in the Appendix basic CVS commands.
+
+The space required for an installation depends on the arrangements and
+thorns used. The flesh on its own requires less than 5 MB.
+
+Another script, \texttt{MakeThornList}, can be used to produce a minimal
+ThornList from a given Cactus par file.  It needs a \emph{master} ThornList
+to be copied into your \texttt{Cactus} directory.
+
+See \url{http://www.cactuscode.org/toolkit/makeThornList/}.
+
+
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+
+\subsection{Directory Structure}
+\label{sec:dist}
+
+A fresh checkout creates a directory \texttt{Cactus} with the
+following subdirectories:
+
+\begin{Lentry}
+
+\item[\texttt{CVS}] the CVS bookkeeping directory, present in every subdirectory
+
+\item[\texttt{doc}] Cactus documentation
+
+\item[\texttt{lib}] contains libraries
+
+\item[\texttt{src}] contains the source code for Cactus
+
+\item [\texttt{arrangements}] contains the Cactus arrangements. The arrangements
+  (the actual ``physics'') are not supplied by just checking out just Cactus.
+  If the arrangements you want to use are standard Cactus arrangements, or
+  reside on our CVS repository (\texttt{cvs.cactuscode.org}),
+  they can be checked out in similar way to the flesh.
+\end{Lentry}
+
+When Cactus is first compiled, it creates a new directory
+\texttt{Cactus/configs}, which will contain all the source code, object files
+and libraries created during the build process.
+
+Configurations are described in detail in Section \ref{sec:configurations}.
+
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+
+\section{Compiling a Cactus application}
+
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+
+
+Cactus can be built in different configurations from the same copy of
+the source files, and these different configurations coexist in the
+\texttt{Cactus/configs} directory. Here are several instances in which
+ this can be useful:
+
+\begin{enumerate}
+\item{}Different configurations can be for \emph{different
+architectures}. You can keep executables for multiple architectures
+based on a single copy of source code, shared on a common file
+system.
+\item{} You can compare different \textit{compiler options}, and \textit{debug-modes}.
+  You might want to compile different communication protocols
+  (e.g. MPI or Globus), or leave them out all together.
+\item{} You can have different configurations for \textit{different thorn
+    collections} compiled into your executable.
+\end{enumerate}
+
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+\subsection{Creating a Configuration}
+\label{sec:configurations}
+
+At its simplest, this is done by \texttt{gmake <\var{config}>}.
+This generates a
+configuration with the name \texttt{\var{config}}, doing its best to
+automatically determine the default compilers and compilation flags
+suitable for the current architecture.
+
+There are a number of additional command-line arguments which may be supplied
+to override some parts of the procedure; they are listed in Section~\ref{sec:confopts}.
+
+Once you have created a new configuration, the command
+\\ \\
+\texttt{gmake <\var{configuration name}>}
+\\ \\
+will build an executable, prompting you along the way for the
+thorns which should be included. There is a range of \texttt{gmake}
+targets and options which are detailed in Section~\ref{sec:gmtafobuanadco}.
+
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+
+\section{Running a Cactus application}
+
+Cactus executables always run from a parameter file (which may be
+provided as a command-line argument taken from standard input), which
+specifies which thorns to use and sets the values of each thorn's 
+parameters (the parameters that are not set will take on default values, 
+see~\ref{sec:Appendix.param}).
+
+There is no restriction on the name
+of the parameter file, although it is conventional to use the file
+extension \texttt{.par}.  Optional command-line arguments can be used
+to customise runtime behaviour, and to provide information about the
+thorns used in the executable. The general syntax for running Cactus from
+a parameter file is then
+
+\texttt{./cactus\_<\var{config}> <\var{parameter file}>
+[\var{command-line options}]}
+
+A parameter file is a text file whose lines are either comments
+or parameter statements.
+Comments are blank lines or lines that begin with either
+`\texttt{\#}' or `\texttt{!}'.
+A parameter statement consists of one or more parameter names, followed by
+an `\texttt{=}', followed by the value(s) for this (these) parameter(s).
+Note that all string parameters are case insensitive.
+
+The first parameter statement in any parameter file should set \texttt{ActiveThorns},
+which is a special parameter that tells the
+program which \textit{thorns} are to be activated.  Only parameters from active
+thorns can be set (and only those routines \textit{scheduled} by active thorns
+are run).  By default all thorns are inactive. For example, the first
+entry in a parameter file which is using just the two thorns
+\texttt{CactusPUGH/PUGH} and \texttt{CactusBase/CartGrid3D} should be
+
+\texttt{ActiveThorns = "PUGH CartGrid3D"}
+
+Parameter specifications following \texttt{ActiveThorns} usually are
+carried out by listing
+the name of the \textit{thorn} which defined the parameter, two colons,
+and the name of the parameter --- e.g.~\texttt{wavetoyF77::amplitude}
+(see Section~\ref{sec:Cactus_parameters.scope} for more information).
+
+Notes:
+
+\begin{itemize}
+
+\item{} You can obtain lists of the parameters associated with
+each thorn using the command-line options \texttt{-o} and \texttt{-O}
+(Section~\ref{sec:command_line_options}).
+
+\item{} For examples of parameter files, look in the \texttt{par} directory
+        which can be found in most thorns.
+
+\item {}
+The Cactus make system provides a mechanism for generating a
+\textit{Thorn Guide} containing separate chapters for each thorn and
+arrangement in your configuration. 
+Details about parameters, grid variables and scheduling are
+automatically included in from a thorns CCL files into the Thorn
+Guide. To construct a Thorn Guide for the configuration
+\texttt{$<$\var{config}$>$} use
+
+\texttt{gmake $<$\var{config}$>$-ThornGuide}
+
+or to make a Thorn Guide for all the thorns in the \texttt{arrangements} directory
+
+\texttt{gmake $<$\var{config}$>$}.
+
+\end{itemize}
+
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+
+
+\chapter{Getting and looking at output}
+
+
+\section{Screen output}
+
+As your Cactus executable runs, standard output and standard error
+are usually written to the screen. Standard output provides you
+with information about the run, and standard error reports warnings
+and errors from the flesh and thorns.
+
+As the program runs, the normal output provides the following information:
+
+\begin{Lentry}
+
+\item [Active thorns]
+        A report is made as each of the thorns in the \texttt{ActiveThorns}
+parameters from the parameter file (see Section~\ref{sec:Parameter_File})
+is attempted to be activated. This report
+shows whether the thorn activation was successful, and if successful gives the
+thorn's implementation. For example
+
+\begin{verbatim}
+Activating thorn idscalarwave...Success -> active implementation idscalarwave
+\end{verbatim}
+
+\item [Failed parameters]
+         If any of the parameters in the parameter file does not belong to any
+of the active thorns, or if the parameter value is not in the allowed range
+(see Section~\ref{sec:Parameters.Types_and_Ranges}),
+an error is registered. For example, if the parameter is not recognised
+
+\begin{verbatim}
+Unknown parameter time::ddtfac
+\end{verbatim}
+or if the parameter value is not in the allowed range
+
+\begin{verbatim}
+Unable to set keyword CartGrid3D::type - ByMouth not in any active range
+\end{verbatim}
+
+\item [Scheduling information]
+        The scheduled routines (see Section~\ref{chap:scheduling}),
+are listed, in the order that they will be executed. For example
+
+\begin{verbatim}
+----------------------------------------------------------------------
+  Startup routines
+    Cactus: Register banner for Cactus
+    CartGrid3D: Register GH Extension for GridSymmetry
+    CartGrid3D: Register coordinates for the Cartesian grid
+    IOASCII: Startup routine
+    IOBasic: Startup routine
+    IOUtil: IOUtil startup routine
+    PUGH: Startup routine
+    WaveToyC: Register banner
+
+  Parameter checking routines
+    CartGrid3D: Check coordinates for CartGrid3D
+    IDScalarWave: Check parameters
+
+  Initialisation
+    CartGrid3D: Set up spatial 3D Cartesian coordinates on the GH
+    PUGH: Report on PUGH set up
+    Time: Set timestep based on speed one Courant condition
+    WaveToyC: Schedule symmetries
+    IDScalarWave: Initial data for 3D wave equation
+
+  do loop over timesteps
+    WaveToyC: Evolution of 3D wave equation
+    t = t+dt
+    if (analysis)
+    endif
+  enddo
+----------------------------------------------------------------------
+\end{verbatim}
+
+\item [Thorn banners]
+        Usually a thorn registers a short piece of text as a \emph{banner}.
+        This banner of each thorn is displayed in the standard output when
+        the thorn is initialised.
+
+\end{Lentry}
+
+
+\section{File output}
+Output methods in Cactus are all provided by thorns.
+Any number of output methods can be used for each run.
+The behaviour of the output thorns in the
+standard arrangements are described in those thorns' documentation.
+
+In general, output thorns decide what to output by parsing a string parameter
+containing the names of those grid variables, or groups of variables, for which
+output is required. The names should be fully qualified with the
+implementation and group or variable names.
+
+There is usually a parameter for each method to denote how often, in evolution
+iterations, this output should be performed.  There is also usually a parameter
+to define the directory in which the output should be placed, defaulting to the
+directory from which the executable is run.
+
+See Chapter~\ref{chap:io_methods} for details on creating your own IO method.
+
+
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+
+\chapter{Checkpointing/Recovery}
+\label{sec:checkpointing}
+
+Checkpointing is defined as saving the current state of a run (parameter
+settings, contents of grid variables, and other relevant information) to a file.
+At a later time, this run can then be restarted from that state by recovering
+all the data from the checkpoint file.
+
+Cactus checkpointing and recovery methods are provided by thorns.
+In general, these thorns decide how often to generate a checkpoint.
+They also register their recovery routines with the flesh; these recovery
+routines may then be called during initialisation of a subsequent run to
+perform the recovery of the state of the run.
+Such a recovery is requested by setting a parameter in the parameter file.
+
+See Chapter~\ref{chap:cp_recovery_methods} for details of how to create
+your own checkpointing and recovery methods.
+
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+
+\chapter{Reporting bugs}
+\label{sec:gehe}
+
+For tracking problem reports and bugs, we use GNATS, which is a bug tracking
+system published under the GNU license. We have set up a web interface at
+\url{http://www.cactuscode.org/BugReporting}, which allows easy submission and browsing
+of problem reports.
+
+A description of the GNATS categories we use is provided in the Appendix
+\ref{sec:Appendix.gnats}.
+
+% OK, there is NO emacs at the moment, because the GNATS setup is really stupid
+% and sendpr handles like c.... besides the fact, that the user has to go
+% through a make process which installs stuff somewhere on his HD. gerd.
+% BUT, we could distribute our own, either copy cvsbug, or write a perl
+% version.  Tom
+% \begin{itemize}
+% \item \texttt{A web interface}
+% \item \texttt{SendPR}
+% {FIXME: Mention the emacs thing here too...}
+% \end{itemize}
+
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+
+%%%% FIXME: add something quick on required software and supported architectures
+
+%%%% FIXME: add an example
+
+\end{cactuspart}
diff --git a/doc/UsersGuide/Infrastructure.tex b/doc/UsersGuide/InfrastructureThorns.tex
index fb48a8fb..3e82c443 100644
--- a/doc/UsersGuide/Infrastructure.tex
+++ b/doc/UsersGuide/InfrastructureThorns.tex
@@ -8,15 +8,14 @@
 %   @version   $Header$
 % @@*/
 
-\begin{cactuspart}{Infrastructure Thorn Writer's Guide}{$RCSfile$}{$Revision$}
-\label{part:Infrastructure}
 \renewcommand{\thepage}{\Alph{part}\arabic{page}}
 
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 
-\chapter{Introduction}
+\chapter{Infrastructure Thorns}
+\label{chap:infrastructure}
 
 \begin{itemize}
  \item{} Concepts and terminology (Overloading and registration of functions)
@@ -30,17 +29,17 @@
  \item{} Overloadable functions
 \end{itemize}
 
-\chapter{Concepts and Terminology}
+\section{Concepts and Terminology}
 \label{chap:cote}
 
-\section{Overloading and Registration}
+\subsection{Overloading and Registration}
 
 The flesh defines a core API which guarantees the presence of a set of
 functions.  Although the flesh guarantees the presence of these functions,
 they can be provided by thorns.  Thorns do this by either the \textit{overloading}
 or the \textit{registration} of functions.
 
-\subsection{Overloading}
+\subsubsection{Overloading}
 
 Some functions can only be provided by one thorn.  The first thorn to
 \textit{overload} this function succeeds, and any later attempt to overload
@@ -48,25 +47,25 @@ the function fails.  For each overloadable function, there is a function
 with a name something like {\tt CCTK\_Overload...} which is passed the
 function pointer.
 
-\subsection{Registration}
+\subsubsection{Registration}
 
 Some functions may be provided by several thorns.  The thorns \textit{register}
 their function with the flesh, and when the flesh-provided function is called,
 the flesh calls all the registered functions.
 
-\section{GH Extensions}
+\subsection{GH Extensions}
 
 A GH extension is a way to associate data with each cGH.  This data should be data
 that is required to be associated with a particular GH by a thorn.
 
 Each GH extension is given a unique handle.
 
-\section{I/O Methods}
+\subsection{I/O Methods}
 
 An I/O method is a distinct way to output data.  Each I/O method has a unique name,
 and the flesh-provided I/O functions operate on all registered I/O methods.
 
-\chapter{GH Extensions}
+\section{GH Extensions}
 
 A GH extension is created by calling {\tt CCTK\_RegisterGHExtension}, with the
 name of the extension.  This returns a unique handle that identifies the extension.
@@ -87,7 +86,7 @@ should initialise the data on the cGH and the call {\tt CCTK\_ScheduleTraverse}
 the schedule tree.
 \end{Lentry}
 
-\chapter{Overloadable and Registerable Functions in Main}
+\section{Overloadable and Registerable Functions in Main}
 
  \begin{tabular}{|l|l|}
    \hline  {\bf Function} & {\bf Default} \\
@@ -97,7 +96,7 @@ the schedule tree.
    \hline
  \end{tabular}
 
-\chapter{Overloadable and Registerable Functions in Comm}
+\section{Overloadable and Registerable Functions in Comm}
 
   \begin{tabular}{|l|l|}
    \hline {\bf Function} & {\bf Default}  \\
@@ -114,7 +113,7 @@ the schedule tree.
    \hline
   \end{tabular}
 
-\chapter{Overloadable and Registerable Functions in I/O}
+\section{Overloadable and Registerable Functions in I/O}
 
  \begin{tabular}{|l|l|}
    \hline {\bf Function} & {\bf Default}   \\
@@ -123,7 +122,7 @@ the schedule tree.
    \hline
  \end{tabular}
 
-\chapter{Drivers}
+\section{Drivers}
 
 The flesh does not know about memory allocation for grid variables, about how
 to communicate data when synchronisation is called for, or about
@@ -131,7 +130,7 @@ multiple patches or adaptive mesh refinement.  All this is the job of a driver.
 
 This chapter describes how to add a driver to your code.
 
-\section{Anatomy}
+\subsection{Anatomy}
 
 A driver consists of a Startup routine which creates a GH extension, 
 registers its associated functions, and overloads the communication functions.
@@ -140,7 +139,7 @@ It may optionally register interpolation, reduction, and I/O methods.
 A driver may also overload the default Initialisation and Evolution routines,
 although a simple unigrid evolver is supplied in the flesh.
 
-\section{Startup}
+\subsection{Startup}
 
 A driver consists of a GH extension, and the following overloaded
 functions.
@@ -166,7 +165,7 @@ The overloadable function {\tt CCTK\_SyncGroup} is deprecated, a driver should
 instead provide a routine to overload the more general function
 {\tt CCTK\_SyncGroupsByDirI}.
 
-\section{The GH Extension}
+\subsection{The GH Extension}
 
 The GH extension is where the driver stores all its grid-dependent information.
 This is stuff like any data associated with a grid variable (e.g.\ storage and
@@ -347,7 +346,7 @@ The flesh will synchronise them if the function returns false.
 Providing this function is probably the easiest way to do multi-patch or
 AMR drivers.
 
-\section{Memory Functions}
+\subsection{Memory Functions}
 
 These consist of
 \begin{enumerate}
@@ -357,7 +356,7 @@ These consist of
 \item{} {\tt CCTK\_ArrayGroupSizeB}
 \end{enumerate}
 
-\subsection{En/Disable Group Storage}
+\subsubsection{En/Disable Group Storage}
 
 These are responsible for switching the memory for all variables
 in a group on or off.  They should return the former state, e.g.\
@@ -397,14 +396,14 @@ The {\tt QueryGroupStorage} function basically returns true or false if
 there is storage for the group, and the {\tt ArrayGroupSize} returns the
 size of the grid function or array group in a particular direction.
 
-\subsection{En/Disable Group Comm}
+\subsubsection{En/Disable Group Comm}
 
 These are the communication analogues to the storage functions.  Basically,
 they flag that communication is to be done on that group or not, and may
 initialise data structures for the communication.
 
 
-\chapter{I/O Methods}
+\section{I/O Methods}
 \label{chap:io_methods}
 %
 The flesh by itself does not provide output for grid variables or other data.
@@ -415,7 +414,7 @@ flesh scheduler or by other thorn routines.
 This chapter explains how to implement your own I/O methods and register
 them with the flesh.
 %
-\section{I/O Method Registration}
+\subsection{I/O Method Registration}
 %
 All I/O methods have to be registered with the flesh before they can be used.
 
@@ -446,7 +445,7 @@ The following example shows how a thorn would register an I/O method,
 \end{verbatim}
 %
 %
-\section{Periodic Output of Grid Variables}
+\subsection{Periodic Output of Grid Variables}
 %
 The flesh scheduler will automatically call {\t CCTK\_OutputGH()} at every
 iteration after the {\tt CCTK\_ANALYSIS} time bin. This function loops over all
@@ -465,7 +464,7 @@ As its return code, it should pass back the number of variables which were outpu
 at the current iteration, or zero if no output was done by this I/O method.
 %
 %
-\section{Triggered Output of Grid Variables}
+\subsection{Triggered Output of Grid Variables}
 %
 Besides the periodic output at every so many iterations using {\t OutputGH()},
 analysis and output of grid variables can also be done via triggers. For this,
@@ -493,7 +492,7 @@ otherwise.
 \texttt{\var{varindex}}, and a negative value in case of an error.
 %
 %
-\section{Unconditional Output of Grid Variables}
+\subsection{Unconditional Output of Grid Variables}
 
 An I/O method's {\t OutputVarAs()} routine is supposed to do output for a
 specific grid variable if ever possible. It will be invoked by the flesh I/O API
@@ -519,7 +518,7 @@ was done successfully, or a negative error code otherwise.
 
 
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
-\chapter{Checkpointing/Recovery Methods}
+\section{Checkpointing/Recovery Methods}
 \label{chap:cp_recovery_methods}
 
 Like for I/O methods, checkpointing/recovery functionality must be implemented
@@ -530,7 +529,7 @@ This chapter explains how to implement checkpointing and recovery methods
 in your thorn.  For further information, see the documentation for thorn
 \texttt{CactusBase/IOUtil}.
 
-\section{Checkpointing Invocation}
+\subsection{Checkpointing Invocation}
 
 Thorns register their checkpointing routines at {\t CCTK\_CPINITIAL} and/or
 {\t CCTK\_CHECKPOINT} and/or {\t CCTK\_TERMINATE}. These time bins are
@@ -555,7 +554,7 @@ state. Such information would include
 \end{itemize}
 
 
-\section{Recovery Invocation}
+\subsection{Recovery Invocation}
 
 Recovering from a checkpoint is a two-phase operation for which the flesh
 provides distinct timebins for recovery routines to be scheduled at:
@@ -597,7 +596,7 @@ provides distinct timebins for recovery routines to be scheduled at:
 
 
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
-\chapter{Clocks for Timing}
+\section{Clocks for Timing}
 \label{chap:clocks}
 
 To add a Cactus clock, you need to write several functions to provide the
diff --git a/doc/UsersGuide/RunningCactus.tex b/doc/UsersGuide/Notes.tex
index d16a921c..30c01719 100644
--- a/doc/UsersGuide/RunningCactus.tex
+++ b/doc/UsersGuide/Notes.tex
@@ -8,14 +8,15 @@
 %   @version   $Header$
 % @@*/
 
-\begin{cactuspart}{Installation and Running}{$RCSfile$}{$Revision$}
-\label{part:RunningCactus}
+\begin{cactuspart}{Additional notes}{$RCSfile$}{$Revision$}\label{part:Notes}
 \renewcommand{\thepage}{\Alph{part}\arabic{page}}
 
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 
+
+
 \chapter{Installation}
 \label{cha:in}
 
@@ -25,8 +26,8 @@
 \label{sec:required_software}
 
 In general, Cactus \emph{requires} the following set of software to function
-in single processor mode. Please refer to the architecture section,
-\ref{sec:suar}, for architecture specific items.
+in single processor mode. Please refer to the architecture section
+\ref{sec:suar} for architecture specific items.
 \begin{Lentry}
 \item[Perl5.0] Perl is used extensively during the Cactus
   thorn configuration phase. Perl is available for nearly all
@@ -56,7 +57,8 @@ processors you also need:
 \begin{Lentry}
 \item[MPI] The \textit{Message Passing Interface},
 which provides inter-processor communication.
-Supercomputing sites often supply a native MPI implementation that is very likely to be compatible with Cactus . Otherwise, there are
+Supercomputing sites often supply a native MPI implementation
+that is very likely to be compatible with Cactus. Otherwise, there are
 various freely available ones available, e.g. the MPICH
 version of MPI is available for various architectures and operating
 systems at \url{http://www-unix.mcs.anl.gov/mpi/}.
@@ -133,87 +135,8 @@ The following machines are only partially supported,
 \item[\textbf{HP Exemplar}]
 \end{Lentry}
 
-%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
-
-\section{Checkout Procedure}
-\label{sec:checkout}
-
-Cactus is distributed, extended, and maintained using the free CVS
-software (\textit{Concurrent Versions System}: \url{http://www.cvshome.org}).
-CVS  allows many people to work on a large software project
-together without getting into a tangle.
-Since Cactus thorns are distributed from several repositories on the
-main CVS site, and from a growing number of user sites, we provide a
-script, described below,  on our website for checking out the flesh and thorns.
-The Cactus website also provides a form interface for direct download.
-
-CVS experts who want to use raw CVS commands are directed to
-Appendix~\ref{sec:Appendix.cvs} for full instructions. For CVS novices,
-we also summarize in the appendix basic CVS commands.
-
-The space required for an installation depends on the arrangements and
-thorns used. The flesh on its own requires less than 5 MB.
-
-The script for checking out the flesh and distribution thorns,
-\texttt{GetCactus}, is available from the website at
-
-\begin{center}
-\url{http://www.cactuscode.org/download/GetCactus}.
-\end{center}
-
-The
-script takes as an argument the name of a file containing a \textit{ThornList},
-that is a list of thorns with the syntax
-
-\begin{alltt}
-<\var{arrangement name}>/<\var{thorn name}>
-\end{alltt}
-
-If no filename is given, only the flesh is checked out.
-Optional directives in the ThornList indicate which CVS repository to fetch
-thorns from. The default is to take the thorns from the same repository as
-the flesh. A full description of ThornList syntax is provided in Appendix~\ref{chap:th}.
-ThornLists example applications are provided on the Cactus website.
-
-The same script can be used to checkout additional thorns.
-
-Another script, \texttt{MakeThornList}, can be used to produce a minimal
-ThornList from a given Cactus par file.  It needs a \emph{master} ThornList
-to be copied into your \texttt{Cactus} directory.
-
-See \url{http://www.cactuscode.org/toolkit/makeThornList/}.
-
-
-%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
-
-\section{Directory Structure}
-\label{sec:dist}
-
-A fresh checkout creates a directory \texttt{Cactus} with the
-following subdirectories:
-
-\begin{Lentry}
-
-\item[\texttt{CVS}] the CVS bookkeeping directory, present in every subdirectory
-
-\item[\texttt{doc}] Cactus documentation
-
-\item[\texttt{lib}] contains libraries
-
-\item[\texttt{src}] contains the source code for Cactus
-
-\item [\texttt{arrangements}] contains the Cactus arrangements. The arrangements
-  (the actual ``physics'') are not supplied by just checking out Cactus.
-  If the arrangements you want to use are standard Cactus arrangements, or
-  reside on our CVS repository (\texttt{cvs.cactuscode.org}),
-  they can be checked out in similar way to the flesh.
-\end{Lentry}
-
-When Cactus is first compiled, it creates a new directory
-\texttt{Cactus/configs}, which will contain all the source code, object files
-and libraries created during the build process.  Disk space may be a problem
-on supercomputers where home directories are small.
-
+\subsection{Note}
+Disk space may be a problem on supercomputers where home directories are small.
 A workaround is to first create a
 configs directory on scratch space, say \texttt{scratch/cactus\_configs/} (where
 \texttt{scratch/} is your scratch directory), and then either
@@ -228,82 +151,13 @@ scratch/cactus\_configs Cactus/configs/}) to the Cactus directory, if your
 filesystem supports soft links.
 \end{itemize}
 
-Configurations are described in detail in Section \ref{sec:configurations}.
-
-%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
-
-\section{Getting Help}
-\label{sec:gehe}
-
-For tracking problem reports and bugs, we use GNATS, which is a bug tracking
-system published under the GNU license. We have set up a web interface at
-\url{http://www.cactuscode.org}, which allows easy submission and browsing
-of problem reports.
-
-A description of the GNATS categories we use is provided in Appendix
-\ref{sec:Appendix.gnats}.
-
-% OK, there is NO emacs at the moment, because the GNATS setup is really stupid
-% and sendpr handles like c.... besides the fact, that the user has to go
-% through a make process which installs stuff somewhere on his HD. gerd.
-% BUT, we could distribute our own, either copy cvsbug, or write a perl
-% version.  Tom
-% \begin{itemize}
-% \item \texttt{A web interface}
-% \item \texttt{SendPR}
-% {FIXME: Mention the emacs thing here too...}
-% \end{itemize}
-
-%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
-%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 
 \chapter{Compilation}
 
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 
-
-Cactus can be built in different configurations from the same copy of
-the source files, and these different configurations coexist in the
-\texttt{Cactus/configs} directory. Here are several instances in which
- this can be useful:
-
-\begin{enumerate}
-\item{}Different configurations can be for \emph{different
-architectures}. You can keep executables for multiple architectures
-based on a single copy of source code, shared on a common file
-system.
-\item{} You can compare different \textit{compiler options}, and \textit{debug-modes}.
-  You might want to compile different communication protocols
-  (e.g. MPI or Globus), or leave them out all together.
-\item{} You can have different configurations for \textit{different thorn
-    collections} compiled into your executable.
-\end{enumerate}
-
-Once a configuration has been created by \texttt{gmake <\var{config}>}, as
-described in detail in the next section, a single call to
-\texttt{gmake <\var{config}>} will compile the code.  The first time, it
-generates a compile \texttt{ThornList},  and gives you the chance to edit
-it before continuing.
-
-%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
-\section{Creating a Configuration}
-\label{sec:configurations}
-
-At its simplest, this is done by \texttt{gmake <\var{config}>}\footnote
-%
-{A note on the Cactus make system---if at any point it prompts you
-to enter something, the default value, which will be assumed if you
-simply press enter, is shown in parentheses.}
-%
-.  This generates a
-configuration with the name \texttt{\var{config}}, doing its best to
-automatically determine the default compilers and compilation flags
-suitable for the current architecture.
-
-There are a number of additional command line arguments which may be supplied
-to override some parts of the procedure.
-
-\subsection{Configuration Options}
+\section{Configuration Options}
+\label{sec:confopts}
 
 There are four ways to pass options to the configuration process.
 % from the gmake command line.
@@ -319,7 +173,7 @@ There are four ways to pass options to the configuration process.
     Either: create a default configuration file \texttt{\$\{HOME\}/.cactus/config}.
 
     All available configuration options may be set in a default options file
-    \texttt{\$\{HOME\}/.cactus/config}, any option which is not set will take a
+    \texttt{\$\{HOME\}/.cactus/config}, any option which are not set will take a
     default value. The file should contain lines of the form:
 
     \texttt{<\var{option}> [=] ...}
@@ -358,7 +212,7 @@ There are four ways to pass options to the configuration process.
     Those that can be set are indicated in the table below.
 \end{enumerate}
 
-They are listed here in order of increasing precedence, e.g. options
+The options are listed here in order of increasing precedence, e.g. options
 set on the command line will take priority over (potentially
 conflicting) options set in \texttt{\$\{HOME\}/.cactus/config} or other
 Cactus configuration files. Default options from
@@ -456,7 +310,7 @@ Flags for the preprocessor (used to generate compilation dependencies
 for and preprocess Fortran code).
 
 \item [\texttt{MKDIRFLAGS}]
-  Flags for \texttt{MKDIR}, so that no error is given if the directory exists.
+Flags for \texttt{MKDIR}, so that no error is given if the directory exists.
 
 \item [\texttt{LDFLAGS}]
 * Flags for the linker.  \emph{Warning:} This variable is ignored
@@ -483,7 +337,7 @@ compiler reports error messages about unrecognised \verb|#| directives.
 
 \item [\texttt{CROSS\_COMPILE}] Enables cross compilation.
   Available options are \texttt{yes} and \texttt{no}, the default is
-  \texttt{no}. To create a cross-compiled configuration one must explicitly set   this option to \texttt{yes}. 
+  \texttt{no}. To create a cross-compiled configuration one must explicitly set   this option to \texttt{yes}.
 
 \item [\texttt{DISABLE\_REAL16}] Disable support for the data type
 \texttt{CCTK\_REAL16}.  The only options available are \texttt{yes} and
@@ -516,27 +370,6 @@ type of debugging being used.
 Debug flags for the Fortran 77 compiler, their use depends on the
 type of debugging being used.
 
-\item [\texttt{OPENMP}]
-* Specifies whether OpenMP support should be enabled. The only options
-  currently available are \texttt{yes} and \texttt{no}. The default
-  is to disable OpenMP support.
-
-\item [\texttt{C\_OPENMP\_FLAGS}]
-OpenMP flags for the C compiler, their use depends on the type of
-OpenMP support being used.
-
-\item [\texttt{CXX\_OPENMP\_FLAGS}]
-OpenMP flags for the C++ compiler, their use depends on the type of
-OpenMP support being used.
-
-\item [\texttt{F90\_OPENMP\_FLAGS}]
-OpenMP flags for the Fortran 90 compiler, their use depends on the
-type of OpenMP support being used.
-
-\item [\texttt{F77\_OPENMP\_FLAGS}]
-OpenMP flags for the Fortran 77 compiler, their use depends on the
-type of OpenMP support being used.
-
 \item [\texttt{OPTIMISE, OPTIMIZE}]
 * Specifies what type of optimisation should be used. The only options currently
 available are \texttt{yes} and \texttt{no}.  The default is to use optimisation.\\
@@ -629,7 +462,7 @@ errors while configuring.  You can pass the additional libraries in
 the variable \texttt{LD} instead.
 
 \item [\texttt{LIBDIRS}] Any other library directories.
-This variable can also contain linker options.  (Cactus adds a
+This variable can also contain linker options.  (Cactus adds an
 \verb+-L+ prefix to library directories, but does not modify linker
 options.)
 \end{Lentry}
@@ -644,7 +477,8 @@ Used to specify any additional directories for system include files.
 
 \item {Precision options}
 
-Used to specify the precision of the default real and integer data types, by the number of bytes the data takes up.  Note that not all
+Used to specify the precision of the default real and integer data types,
+by the number of bytes the data takes up.  Note that not all
 values will be valid on all architectures.
 
 \begin{Lentry}
@@ -703,11 +537,11 @@ commands that it is executing.
 
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 
-\subsection{Compiling with Extra Packages}
+\section{Compiling with Extra Packages}
 \label{subsec:cowiexpa}
 
 
-\subsubsection{MPI: Message Passing Interface}
+\subsection{MPI: Message Passing Interface}
 \label{subsubsec:Compiling-MPI}
 
 The \textit{Message Passing Interface} (MPI) provides inter-processor
@@ -813,10 +647,10 @@ Use the PACX Metacomputing package (\textit{PArallel Computer eXtension},\\
 
 \end{Lentry}
 
-Note that the searches for libraries, etc., mentioned above, use the
+Note that the searches for libraries, etc. mentioned above use the
 locations given in the files in \texttt{lib/make/extras/MPI}.
 
-\subsubsection{HDF5: Hierarchical Data Format version 5}
+\subsection{HDF5: Hierarchical Data Format version 5}
 \label{subsec:hdf5}
 
 To compile with HDF5 (\url{http://hdf.ncsa.uiuc.edu/whatishdf5.html}),
@@ -824,7 +658,7 @@ the configure options are
 
 \texttt{HDF5 = yes/no [HDF5\_DIR = <\var{dir}>] [LIBZ\_DIR = <\var{dir}>] [LIBSZ\_DIR = <\var{dir}>]}
 
-If \texttt{HDF5\_DIR} is not given the configuration process will search for an
+If \texttt{HDF5\_DIR} is not given, the configuration process will search for an
 installed HDF5 package in some standard places (defined in
 \texttt{lib/make/extras/HDF5}).
 If the found HDF5 library was built with the external deflate I/O filter,
@@ -837,7 +671,7 @@ it to the linker flags. You may also point directly to the location of
 \texttt{libsz.a} by setting \texttt{LIBSZ\_DIR}.
 
 
-\subsubsection{LAPACK: Linear Algebra PACKage}
+\subsection{LAPACK: Linear Algebra PACKage}
 
 To compile with LAPACK (\url{http://www.netlib.org/lapack/}),
 the configure options are
@@ -852,7 +686,7 @@ LAPACK = yes | no | <blank>
 
 If \texttt{LAPACK\_DIR} is not given, the configuration process will search for a
 LAPACK library \texttt{liblapack.[\{a,so\}]} in some standard places (defined in
-\texttt{lib/make/extras/LAPACK}). If \texttt{LAPACK\_DIR} is set to \texttt{no},
+\texttt{lib/make/extras/LAPACK}). If \texttt{LAPACK\_DIR} is set to \texttt{no}
 the LAPACK library path is assumed to be installed in a standard system location
 (e.g. \texttt{/usr/lib/}), and thus the library path will not be added to the
 linker's command line.
@@ -864,7 +698,7 @@ location (\texttt{LAPACK\_EXTRA\_LIBS\_DIRS}) of extra libraries that are
 required by LAPACK itself.
 
 
-\subsubsection{PETSc: Portable, Extensible Toolkit for Scientific Computation}
+\subsection{PETSc: Portable, Extensible Toolkit for Scientific Computation}
 
 To compile with PETSc
 (\url{http://www-unix.mcs.anl.gov/petsc/petsc-2/index.html}),
@@ -887,7 +721,7 @@ architecture-specific libraries, as required by a PETSc configuration (usually
 PETSc needs the LAPACK library).
 
 
-\subsubsection{Pthreads: POSIX threads}
+\subsection{Pthreads: POSIX threads}
 
 To enable multithreading support within Cactus using POSIX threads,
 the configure option is
@@ -900,7 +734,7 @@ library (either \texttt{libpthread} or \texttt{libpthreads}), and set
 preprocessor defines necessary for compiling multithreaded code.
 
 
-\subsection{File Layout}
+\section{File Layout}
 
 The configuration process sets up various subdirectories and files in the
 \texttt{configs} directory to contain the configuration specific files; these
@@ -1000,7 +834,8 @@ targets and options which are detailed in the following sections.
 \label{sec:gmtafobuanadco}
 
 A target for \texttt{gmake} can be naively thought of as an argument
-that tells it which of several things listed in the \texttt{Makefile} is to do. The command \texttt{gmake help} lists all \texttt{gmake} targets:
+that tells it which of several things listed in the \texttt{Makefile} it
+is to do. The command \texttt{gmake help} lists all \texttt{gmake} targets:
 % colon clarifies that all (config) targets are listed here
 
 \begin{Lentry}
@@ -1049,12 +884,12 @@ configuration using its previous configuration options from the file
 \texttt{configs/<\var{config}>/config-info}.
 
 \item [\texttt{gmake <\var{config}>--testsuite}] runs the test programs
-associated with each thorn in the configuration. See Section
+associated with each thorn in the configuration. See section
 \ref{sec:testing} for information about the test suite mechanism.
 
 \item[\texttt{gmake <\var{config}>-ThornGuide}] builds documentation for the
 thorns in this configuration
-  (see Section \ref{sec:OtherGmakeTargetsDoc}, page
+  (see section \ref{sec:OtherGmakeTargetsDoc}, page
   \pageref{sec:OtherGmakeTargetsDoc}, for other targets to build documentation
   for thorns).
 
@@ -1076,10 +911,12 @@ Cactus will try to compile all thorns listed in
 \texttt{configs/<\var{config}>/ThornList}.
 The \texttt{ThornList} file is simply a list of the form
 \texttt{<\var{arrangement}>/<\var{thorn}>}.  All text after a pound sign
-`\texttt{\#}', or exclamation mark `\texttt{!}'
-on a line is treated as a comment and is ignored. If you did
-not specify a ThornList already during configuration the first time that you compile a configuration, you will be shown a list of all the thorns in your arrangement
-directory, and asked if you want to edit them. You can regenerate
+`\texttt{\#}' or exclamation mark `\texttt{!}'
+on a line is treated as a comment and ignored.
+If you did
+not specify a ThornList already, the first time that you compile a configuration
+you will be shown a list of all the thorns in your arrangement
+directory, and asked if you with to edit them. You can regenerate
 this list at anytime by typing
 
 \begin{alltt}
@@ -1096,7 +933,7 @@ Instead of using the editor to specify the thorns you want to
   have compiled, you can \emph{edit} the \texttt{ThornList} outside
   the make process. It is located in \texttt{configs/<\var{config}>/ThornList},
   where \texttt{<\var{config}>} refers to the name of your configuration.
-  The directory \texttt{./configs} exists \emph{after} the very first
+  The directory \texttt{./configs} exists \emph{ after} the very first
   make phase for the first configuration.
 
 \subsection{Notes and Caveats}
@@ -1104,14 +941,14 @@ Instead of using the editor to specify the thorns you want to
 \item{} If during the build you see the error ``\texttt{missing
     separator}'', you are probably not using GNU make.
 \item{} \textit{The EDITOR environment variable}. You may not be aware of
-  this, but this variable usually exists, and may be set by default to
+  this, but this thing very often exists and, may be set  by default to
   something scary like \texttt{vi}. If you don't know how to use \texttt{vi},
   or wish to
   use your favorite editor instead, reset this environment variable.
   (To exit \texttt{vi} type \texttt{<ESC> :q!})
 \end{itemize}
 
-\subsection{\texttt{gmake} Options for Building Configurations}
+\subsection{\texttt{gmake} Options for building configurations}
 \label{sec:gmopfobuco}
 
 An \textit{option} for \texttt{gmake} can be thought of as an argument which tells
@@ -1171,10 +1008,10 @@ configuration found in user's \texttt{configs} subdirectory.
 \item [\texttt{gmake newthorn}] creates a new thorn, prompting for the necessary
   information and creating template files.
 
-\item [\texttt{gmake TAGS}] creates an Emacs style TAGS file. See Section
+\item [\texttt{gmake TAGS}] creates an Emacs style TAGS file. See section
   \ref{sec:Appendix.tags} for using tags within Cactus.
 
-\item [\texttt{gmake tags}] creates a \texttt{vi} style tags file. See Section
+\item [\texttt{gmake tags}] creates a \texttt{vi} style tags file. See section
   \ref{sec:Appendix.tags} for using tags within Cactus.
 
 \end{Lentry}
@@ -1235,27 +1072,9 @@ is also of use when trying to get Cactus to work on a new architecture.
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 
-\chapter{Running Cactus}
-
-Cactus executables always run from a parameter file (that may be a
-specified as a command line argument taken from standard input), which
-specifies which
-thorns to use and sets the values of any parameters which are different
-from the default values. There is no restriction on the name
-of the parameter file, although it is conventional to use the file
-extension \texttt{.par}.  Optional command line arguments can be used
-to customise runtime behaviour, and to provide information about the
-thorns used in the executable. The general syntax for running Cactus from
-a parameter file is then
+\chapter{Runtime options}
 
-\texttt{./cactus\_<\var{config}> <\var{parameter file}>
-[\var{command line options}]}
-
-or if the parameter file should be taken from standard input, the syntax includes a dash (``\texttt{-}'') on the command line like this:
-
-\texttt{./cactus\_<\var{config}> [\var{command line options}] -}
-
-The remainder of this chapter covers all aspects for running your
+This chapter covers all aspects for running your
 Cactus executable.  These include: command-line options, parameter
 file syntax, understanding screen output, environment variables, and
 creating thorn documentation.
@@ -1273,7 +1092,7 @@ The options follow the usual GNU rules:
 	is unambiguous.
 \item	The preferred way of spelling a long-named option is \verb|--foo|,
 	but \verb|-foo| also accepted, though this is deprecated.
-\item	A short option, \verb|-X|, which takes an argument \verb|bar|,
+\item	A short option, \verb|-X|, which takes an argument \verb|bar|
 	may be written as either \verb|-Xbar| or as \verb|-X=bar|.
 \item	An option which can be interpreted as either a short option,
 	or as an abbreviated \verb|-foo|-style long option, is interpreted
@@ -1366,7 +1185,7 @@ Prints version information of the code.
 
 \item [\texttt{-L<\var{level}>} or \texttt{--logging-level=<\var{level}>}]
 Sets the logging level of the code.  All warning messages are given a
-level---the lower the level, the greater the severity.  This
+level---the lower the level the greater the severity.  This
 parameter \texttt{-L} controls the level of messages to be seen, with all
 warnings of level $\le$ \texttt{<\var{level}>} printed to standard output.  The
 default is a logging level of~0, meaning that only level~0 messages
@@ -1385,7 +1204,7 @@ warnings with level $\le$ \texttt{<\var{level}>} as fatal errors, and aborts
 the Cactus run immediately (after printing the warning message%%%
 \footnote{%%%
          Cactus imposes the constraint,
-         $\hbox{ \texttt{-W} level} \ge \hbox{ \texttt{-E} level} \ge 0$,
+         $\hbox{\texttt{-W} level} \ge \hbox{\texttt{-E} level} \ge 0$,
          so any fatal-error message will always be printed (first).
          }%%%
 ).  The default value is zero, \ie{} only level~0 warnings
@@ -1394,7 +1213,7 @@ will abort the Cactus run.
 \item [\texttt{-r[o|e|oe|eo]} or \texttt{--redirect=[o|e|oe|eo]}]
 Redirects the standard output (`\texttt{o}') and/or standard error
 (`\texttt{e}') of each processor to a file.  By default,
-the standard outputs from processors, other than processor 0, are discarded.
+the standard outputs from processors other than processor 0 are discarded.
 
 \item [\texttt{--logdir=<\var{directory}>}]
 Sets the output directory for logfiles created by the \texttt{-r} option.
@@ -1407,7 +1226,7 @@ If the directory doesn't exist yet, it will be created by Cactus.
   and is then written in large chunks.  This improves performance
   considerably.  Line buffering means that output is written whenever
   a newline character is encountered; full buffering means that output
-  is written, say, once 1000 characters have accumulated.  The default
+  is written, say, once 1000 characters have accmulated.  The default
   setting is line buffering for I/O that goes to a terminal, and full
   buffering for I/O that goes to a file.  For debugging purposes, it
   is sometimes useful to reduce the amount of buffering.  Error
@@ -1421,9 +1240,15 @@ See Section~\ref{sec:Parameter_File} for details.
 
 \item [\texttt{-i} or \texttt{--ignore-next}]
 Causes the next argument on the command line to be ignored.
-
 \end{Lentry}
 
+A dash (``\texttt{-}'') appended at the end of the command line like this:
+
+\texttt{./cactus\_<\var{config}> [\var{command-line options}] -}
+
+lets the user specify parameter values from standard input rather than
+from a parameter file.
+
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 
@@ -1452,11 +1277,11 @@ program which \textit{thorns} are to be activated.  Only parameters from active
 thorns can be set (and only those routines \textit{scheduled} by active thorns
 are run).  By default, all thorns are inactive. For example, the first
 entry in a parameter file which is using just the two thorns
-\texttt{CactusPUGH/PUGH} and \texttt{CactusBase/CartGrid3D}, should be
+\texttt{CactusPUGH/PUGH} and \texttt{CactusBase/CartGrid3D} should be
 
 \texttt{ActiveThorns = "PUGH CartGrid3D"}
 
-All parameters following the \texttt{ActiveThorns} parameter have names
+All parameters following the \texttt{ActiveThorns} parameter all have names
 whose syntax depends on the scope
 (see Section~\ref{sec:Cactus_parameters.scope})
 of the parameter:
@@ -1479,7 +1304,7 @@ the implementation name, or the thorn name. However, we recommend
 that the above convention be followed.
 
 The Cactus flesh performs checks for consistency and range of parameters.
-The severity of these checks is controlled by the command line argument
+The severity of these checks is controlled by the command-line argument
 \texttt{--parameter-level}, which can take the following values
 \begin{Lentry}
 \item[\texttt{relaxed}] Cactus will issue a level 0 warning (that is, the
@@ -1527,12 +1352,12 @@ each thorn using the command-line options \texttt{-o} and \texttt{-O}
         characters, are preserved.
 
 \item{} Some parameters are \textit{steerable}, and can be changed during
-        the execution of a Cactus program by using parameter steering interfaces,
-        for example, thorn \texttt{CactusConnect/HTTPD}, or by using a
+        the execution of a Cactus program using parameter steering interfaces,
+        for example, thorn \texttt{CactusConnect/HTTPD}, or using a
         parameter file when recovering from a checkpoint file.
 
-\item{} For examples of parameter files, look in the \texttt{par} directory,
-        which can be found in most thorns.
+\item{} For examples of parameter files, look in the \texttt{par} directory
+        contained in most thorns.
 
 \end{itemize}
 
@@ -1546,13 +1371,13 @@ individual thorn, obviously depends on what the thorn authors added,
 but the Thorn Guide is a good place to first look for special
 instructions on how to run and interpret the output from a thorn.
 Details about parameters, grid variables and scheduling are
-automatically included from the thorns' CCL files into the Thorn
+automatically read from a thorn's CCL files and included in the Thorn
 Guide. To construct a Thorn Guide for the configuration
 \texttt{$<$\var{config}$>$} use
 
 \texttt{gmake $<$\var{config}$>$-ThornGuide}
 
-or to make a Thorn Guide for all the thorns in the \texttt{arrangements} directory,
+or to make a Thorn Guide for all the thorns in the \texttt{arrangements} directory
 
 \texttt{gmake $<$\var{config}$>$}.
 
@@ -1583,7 +1408,7 @@ As the program runs, the normal output provides the following information:
 parameters from the parameter file (see Section~\ref{sec:Parameter_File})
 is attempted to be activated. This report
 shows whether the thorn activation was successful, and if successful, gives the
-thorn's implementation. For example,
+thorn's implementation. For example
 
 \begin{verbatim}
 Activating thorn idscalarwave...Success -> active implementation idscalarwave
@@ -1606,7 +1431,7 @@ Unable to set keyword CartGrid3D::type - ByMouth not in any active range
 
 \item [Scheduling information]
         The scheduled routines (see Section~\ref{chap:scheduling})
-are listed in the order that they will be executed. For example,
+are listed, in the order that they will be executed. For example,
 
 \begin{verbatim}
 ----------------------------------------------------------------------
@@ -1668,25 +1493,6 @@ See Chapter~\ref{chap:io_methods} for details on creating your own I/O method.
 
 
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
-
-\section{Checkpointing/Recovery}
-\label{sec:checkpointing}
-
-Checkpointing is defined as saving the current state of a run (parameter
-settings, contents of grid variables, and other relevant information) to a file.
-At a later time, this run can then be restarted from that state by recovering
-all the data from the checkpoint file.
-
-Cactus checkpointing and recovery methods are provided by thorns.
-In general, these thorns decide how often to generate a checkpoint.
-They also register their recovery routines with the flesh; these recovery
-routines may then be called during initialisation of a subsequent run to
-perform the recovery of the state of the run.
-Such a recovery is requested by setting a parameter in the parameter file.
-
-See Chapter~\ref{chap:cp_recovery_methods} for details of how to create
-your own checkpointing and recovery methods.
-
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
diff --git a/doc/UsersGuide/Preface.tex b/doc/UsersGuide/Preface.tex
index 69d24f8d..66f227f9 100644
--- a/doc/UsersGuide/Preface.tex
+++ b/doc/UsersGuide/Preface.tex
@@ -22,8 +22,10 @@
  
 \vskip .5cm
 
-This document is a complete guide to using and
-developing with the Cactus Code. 
+This document contains a quick-start guide to installing and running
+a Cactus application. In subsequent chapters, it provides more detailed
+information on advanced user's topics, as well as an introduction to 
+thorn writing.
 Please report omissions, errors, or suggestions to 
 any of our contact addresses below. 
 
@@ -33,29 +35,26 @@ any of our contact addresses below.
 
 \begin{Lentry}
 
-\item [\textbf{Part~\ref{part:RunningCactus}: Installation and Running.}]
-  A description of required hardware and software and an installation guide
-  for Cactus. 
+\item [\textbf{Part~\ref{part:Introduction}: Introduction to Cactus.}]
+  A guide through the process of obtaining and installing Cactus and running
+  a simple example application with it. 
+
+\item [\textbf{Part~\ref{part:Notes}: Additional Notes.}]
+  A more in-depth description of required hardware and software, along with 
+  configuration, installation and running options. 
   Describes how to check the installation with Cactus test suites. 
 
-\item [\textbf{Part~\ref{part:ThornWriters}: Application Thorn Writing.}]
+\item [\textbf{Part~\ref{part:ThornWriting}: Thorn Writing.}]
   An introduction to thorn concepts and description of how to create, write
   and maintain application thorns. Explanation of use of the
   programming interface to take advantage of parallelism and modularity.
-
-\item [\textbf{Part~\ref{part:UtilityRoutines}: Utility Routines.}]
-  A discussion of low-level utility routines available in Cactus, such as
-  key/value tables.
-
-\item [\textbf{Part~\ref{part:Infrastructure}:
-       Infrastructure Thorn Writer's Guide.}]
-  A more advanced discussion of user supplied infrastructure routines such
+  This is followed by a more advanced discussion of user supplied 
+  infrastructure routines such
   as additional \textit{output routines}, \textit{drivers}, etc.
- 
 
 \item [\textbf{Part~\ref{part:Appendices}: Appendices.}]
-        These contain a description of the Cactus Configuration Language,
-        a glossary,
+        These contain a glossary, a description of the Cactus Configuration 
+	Language, the Utility routines
         and other odds and ends, such as how to use GNATS and TAGS.
 
 \end{Lentry}
diff --git a/doc/UsersGuide/UsersGuide.tex b/doc/UsersGuide/UsersGuide.tex
index 8562f81a..aeb3922b 100644
--- a/doc/UsersGuide/UsersGuide.tex
+++ b/doc/UsersGuide/UsersGuide.tex
@@ -84,7 +84,6 @@ urlcolor=blue
 
 \include{ApplicationThorns}
 \include{InfrastructureThorns}
-\include{UtilityRoutines}
 %\include{FunctionReference}
 
 \include{Appendices}
diff --git a/doc/UsersGuide/UtilityRoutines.tex b/doc/UsersGuide/UtilityRoutines.tex
deleted file mode 100644
index fb685d8f..00000000
--- a/doc/UsersGuide/UtilityRoutines.tex
+++ /dev/null
@@ -1,431 +0,0 @@
-% /*@@
-%   @file      UtilityRoutines.tex
-%   @date      27 Jan 1999
-%   @author    Jonathan Thornburg
-%   @desc
-%   Description of the Util_* utility routines for the Cactus User's Guide.
-%   @enddesc
-%   @version $Header$
-% @@*/
-
-\begin{cactuspart}{Utility Routines}{$RCSfile$}{$Revision$}
-\label{part:UtilityRoutines}
-\renewcommand{\thepage}{\Alph{part}\arabic{page}}
-
-%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
-%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
-%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
-
-\chapter{Introduction}
-
-As well as the high-level \verb|CCTK_|* routines, Cactus also
-provides a set of lower-level \verb|Util_|* utility routines, which
-are mostly independent of the rest of Cactus.  This chapter gives a
-general overview of programming with these utility routines.
-
-%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
-
-\chapter{Key/Value Tables}
-
-%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
-
-\section{Motivation}
-
-Cactus functions may need to pass information through a generic
-interface.  In the past, we have used various ad hoc means to do this,
-and we often had trouble passing "extra" information that wasn't
-anticipated in the original design.  For example, for periodic output
-of grid variables,
-\verb|CCTK_OutputVarAsByMethod()| requires that
-any parameters (such as hyperslabbing parameters) be appended as an option
-string to the variable's character string name.  Similarly, elliptic
-solvers often need to pass various parameters, but we haven't had a
-good way to do this.
-
-Key/value tables (\textit{tables} for short) provide a clean solution
-to these problems.  They're implemented by the \verb|Util_Table|*
-functions (described in detail in the Reference Manual).
-
-%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
-
-\section{The Basic Idea}
-
-Basically, a table is an object which maps strings to almost arbitrary
-user-defined data.  (If you know Perl, a table is very much like a
-Perl hash table.  Alternatively, if you know Unix shells, a table is
-like the set of all environment variables.  As yet another analogy,
-if you know Awk, a table is like an Awk associative array.)%%%
-\footnote{%%%
-	 However, the present Cactus tables implementation
-	 is optimized for a relatively small number of
-	 distinct keys in any one table.  It will still
-	 work OK for huge numbers of keys, but it will be
-	 slow.
-	 }%%%
-
-More formally, a table is an object which stores a set of \textit{keys}
-and a corresponding set of \textit{values}.  We refer to a (key,value)
-pair as a table \textit{entry}.
-
-Keys are C-style null-terminated character strings, with the slash
-character `{\tt /}' reserved for future expansion.%%%
-\footnote{%%%
-	 Think of hierarchical tables for storing
-	 tree-like data structures.%%%
-	 }%%%
-
-Values are 1-dimensional arrays of any of the usual Cactus data types
-described in Section~\ref{sect-ThornWriting/DataTypes}.
-A string can be stored by treating it as a 1-dimensional array of
-\verb|CCTK_CHAR| (there's an example of this below).
-
-The basic ``life cycle'' of a table looks like this:
-\begin{enumerate}
-\item	Some code creates it with \verb|Util_TableCreate()|
-	or \verb|Util_TableClone()|.
-\item	Some code (often the same piece of code, but maybe some
-	other piece) sets entries in it using one or more of
-	the \verb|Util_TableSet*()|, \verb|Util_TableSet*Array()|,
-	\verb|Util_TableSetGeneric()|, \verb|Util_TableSetGenericArray()|,
-	and/or \verb|Util_TableSetString()| functions.
-\item	Some other piece or pieces of code can get (copies of)
-	the values which were set, using one or more of the
-	\verb|Util_TableGet*()|, \verb|Util_TableGet*Array()|,
-	\verb|Util_TableGetGeneric()|, \verb|Util_TableGetGenericArray()|,
-	and/or \verb|Util_TableGetString()| functions.
-\item	When everyone is through with a table, some (single)
-	piece of code should destroy it with \verb|Util_TableDestroy()|.
-\end{enumerate}
-
-There are also convenience functions \verb|Util_TableSetFromString()|
-to set entries in a table based on a parameter-file-style string,
-and \verb|Util_TableCreateFromString()| to create a table and then
-set entries in it based on a parameter-file-style string.
-
-As well, there are ``table iterator'' functions \verb|Util_TableIt*()|
-to allow manipulation of a table even if you don't know its keys.
-
-A table has an integer ``flags word'' which may be used to specify
-various options, via bit flags defined in \verb|util_Table.h|.
-For example, the flags word can be used to control whether keys
-should be compared as case sensitive or case insensitive strings.
-See the detailed function description of \verb|Util_TableCreate()|
-in the Reference Manual for a list
-of the possible bit flags and their semantics.
-
-%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
-
-\section{A Simple Example}
-\label{Tables_Simple_Example}
-
-Here's a simple example (in C)%%%
-\footnote{%%%
-	 All (or almost all) of the table routines
-	 are also usable from Fortran.  See the full
-	 descriptions in the Reference Manual
-	 for details.
-	 }%%%
-{} of how to use a table:
-\begin{verbatim}
-#include "util_Table.h"
-#include "cctk.h"
-
-/* create a table and set some entries in it */
-int handle = Util_TableCreate(UTIL_TABLE_FLAGS_DEFAULT);
-Util_TableSetInt(handle, 2, "two");
-Util_TableSetReal(handle, 3.14, "pi");
-
-...
-
-/* get the values from the table */
-CCTK_INT two_value;
-CCTK_REAL pi_value;
-Util_TableGetInt(handle, &two_value, "two");    /* sets two_value = 2 */
-Util_TableGetReal(handle, &pi_value, "pi");     /* sets pi_value = 3.14 */
-\end{verbatim}
-
-Actually, you shouldn't write code like this---in the real world
-errors sometimes happen, and it's much better to catch them close to
-their point of occurrence, rather than silently produce garbage results
-or crash your program.  So, the \emph{right} thing to do is to always
-check for errors.  To allow this, all the table routines return a status,
-which is zero or positive for a successful return, but negative if
-and only if some sort of error has occurred.%%%
-\footnote{%%%
-	 Often (as in the examples here) you don't care
-	 about the details of which error occurred.  But if
-	 you do, there are various error codes defined in
-	 {\t util\_Table.h} and {\t util\_ErrorCodes.h};
-	 the detailed function descriptions in
-	 the Reference Manual
-	 say which error codes each function can return.
-	 }%%%
-{}  So, the above example should be rewritten like this:
-
-\begin{verbatim}
-#include "util_Table.h"
-
-/* create a table and set some entries in it */
-int handle = Util_TableCreate(UTIL_TABLE_FLAGS_DEFAULT);
-if (handle < 0)
-        CCTK_WARN(CCTK_WARN_ABORT, "couldn't create table!");
-
-/* try to set some table entries */
-if (Util_TableSetInt(handle, 2, "two") < 0)
-        CCTK_WARN(CCTK_WARN_ABORT, "couldn't set integer value in table!");
-if (Util_TableSetReal(handle, 3.14, "pi") < 0)
-        CCTK_WARN(CCTK_WARN_ABORT, "couldn't set real value in table!");
-
-...
-
-/* try to get the values from the table */
-CCTK_INT two_value;
-CCTK_REAL pi_value;
-if (Util_TableGetInt(handle, &two_value, "two") < 0)
-        CCTK_WARN(CCTK_WARN_ABORT, "couldn't get integer value from table!");
-if (Util_TableGetReal(handle, &pi_value, "pi") < 0)
-        CCTK_WARN(CCTK_WARN_ABORT, "couldn't get integer value from table!");
-
-/* if we get to here, then two_value = 2 and pi_value = 3.14 */
-\end{verbatim}
-
-%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
-
-\section{Arrays as Table Values}
-
-As well as a single numbers (or characters or pointers), tables can
-also store 1-dimensional arrays of numbers (or characters or pointers).%%%
-\footnote{%%%
-	 Note that the table makes (stores) a \emph{copy} of the array
-	 you pass in, so it's somewhat inefficient to store a large array
-	 (e.g.~a grid function) this way.  If this is a problem, consider
-	 storing a \texttt{CCTK\_POINTER} (pointing to the array) in the table
-	 instead.  (Of course, this requires that you ensure that the array still exists whenever that \texttt{CCTK\_POINTER} is used.)
-	 }%%%
-
-For example (continuing the previous example):
-\begin{verbatim}
-static const CCTK_INT a[3] = { 42, 69, 105 };
-if (Util_TableSetIntArray(handle, 3, a, "my array") < 0)
-        CCTK_WARN(CCTK_WARN_ABORT, "couldn't set integer array value in table!");
-
-...
-
-CCTK_INT blah[10];
-int count = Util_TableGetIntArray(handle, 10, blah, "my array");
-if (count < 0)
-        CCTK_WARN(CCTK_WARN_ABORT, "couldn't get integer array value from table!");
-/* now count = 3, blah[0] = 42, blah[1] = 69, blah[2] = 105, */
-/*     and all remaining elements of blah[] are unchanged */
-\end{verbatim}
-As you can see, a table entry remembers the length of any array
-value that has been stored in it.%%%
-\footnote{%%%
-	 In fact, actually \emph{all} table values are
-	 arrays---setting or getting a single value is
-	 just the special case where the array length is 1.
-	 }%%%
-{}
-
-If you only want the first few values of a larger array, just pass
-in the appropriate length of your array,
-that's OK:
-\begin{verbatim}
-CCTK_INT blah2[2];
-int count = Util_TableGetIntArray(handle, 2, blah2, "my array");
-if (count < 0)
-        CCTK_WARN(CCTK_WARN_ABORT, "couldn't get integer array value from table!");
-/* now count = 3, blah2[0] = 42, blah2[1] = 69 */
-\end{verbatim}
-You can even ask for just the first value:
-\begin{verbatim}
-CCTK_INT blah1;
-int count = Util_TableGetInt(handle, &blah1, "my array");
-if (count < 0)
-        CCTK_WARN(CCTK_WARN_ABORT, "couldn't get integer array value from table!");
-/* now count = 3, blah1 = 42 */
-\end{verbatim}
-
-%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
-
-\section{Character Strings}
-
-One very common thing you might want to store in a table is a
-character string.  While you could do this by explicitly storing
-an array of \verb|CCTK_CHAR|, there are also routines
-specially for conveniently setting and getting strings:
-\begin{verbatim}
-if (Util_TableSetString(handle, "black holes are fun", "bh") < 0)
-        CCTK_WARN(CCTK_WARN_ABORT, "couldn't set string value in table!");
-
-...
-char buffer[50];
-if (Util_TableGetString(handle, 50, buffer, "bh") < 0)
-        CCTK_WARN(CCTK_WARN_ABORT, "couldn't get string value from table!");
-
-/* now buffer[] contains the string "black holes are fun" */
-\end{verbatim}
-
-\verb|Util_TableGetString()| guarantees that the string is
-terminated by a null character (`\verb|\0|'), and also returns an
-error if the string is too long for the buffer.
-
-%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
-
-\section{Convenience Routines}
-
-There are also convenience routines for the common case of setting
-values in a table based on a string.
-
-For example, the following code sets up exactly the same table as the
-example in Section \ref{Tables_Simple_Example}:
-
-\begin{verbatim}
-#include <util_Table.h>
-
-/* create a table and set some values in it */
-int handle = Util_TableCreate(UTIL_TABLE_FLAGS_DEFAULT);
-if (handle < 0)
-        CCTK_WARN(CCTK_WARN_ABORT, "couldn't create table!");
-
-/* try to set some table entries */
-if (Util_TableSetFromString(handle, "two=2 pi=3.14") != 2)
-        CCTK_WARN(CCTK_WARN_ABORT, "couldn't set values in table!");
-\end{verbatim}
-
-There is also an even higher-level convenience function
-\verb|Util_TableCreateFromString()|: this creates a table with the
-case insensitive flag set (to match Cactus parameter file semantics),
-then (assuming no errors occurred) calls \verb|Util_TableSetFromString()|
-to set values in the table.
-
-For example, the following code sets up a table (with the case insensitive flag
-set) with four entries: an integer number ({\tt two}), a real number ({\tt
-pi}), a string ({\tt buffer}), and an integer array with three elements ({\tt
-array}):
-
-\begin{verbatim}
-#include <util_Table.h>
-
-int handle = Util_TableCreateFromString(" two    = 2 "
-                                        " pi     = 3.14 "
-                                        " buffer = 'Hello World' "
-                                        " array  = { 1 2 3 }");
-if (handle < 0)
-        CCTK_WARN(CCTK_WARN_ABORT, "couldn't create table from string!");
-\end{verbatim}
-
-Note that this code passes a single string to
-\verb|Util_TableCreateFromString()|%%%
-\footnote{C automatically concatenates
-adjacent character string constants separated only by whitespace.}, 
-which then gets parsed into key/value pairs, with the key separated from its
-corresponding value by an equals sign.
-
-Values for numbers are converted into integers ({\tt CCTK\_INT}) if possible
-(no decimal point appears in the value), otherwise into reals ({\tt CCTK\_REAL}).
-Strings must be enclosed in either single or double quotes. String values in
-single quotes are interpreted literally, strings in double quotes may contain
-character escape codes which then will be interpreted as in C.
-Arrays must be enclosed in curly braces, array elements must be single numbers
-of the same type (either all integer or all real).
-
-%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
-
-\section{Table Iterators}
-\label{sect-UtilityRoutines/tables/table-iterators}
-
-In the examples up to now, the code, which wanted to get values from
-the table, knew what the keys were.  It's also useful to be able to
-write generic code which can operate on a table without knowing the
-keys.  ``Table iterators'' (``iterators'', for short) are used for this.
-
-An iterator is an abstraction of a pointer to a particular table entry.
-Iterators are analogous to the \verb|DIR *| pointers used by the POSIX
-\verb|opendir()|, \verb|readdir()|, \verb|closedir()|, and similar
-functions, to Perl hash tables' \verb|each()|, \verb|keys()|,
-and \verb|values()|, and to the C++ Standard Template Library's
-forward iterators.
-
-At any time, the entries in a table may be considered to be in some
-arbitrary (implementation-defined) order; an iterator may be used to
-walk through some or all of the table entries in this order.  This
-order is guaranteed to remain unchanged for any given table, so long
-as no changes are made to that table, \ie{} so long as no
-\verb|Util_TableSet*()|, \verb|Util_TableSet*Array()|,
-\verb|Util_TableSetGeneric()|, \verb|Util_TableSetGenericArray()|,
-\verb|Util_TableSetString()|, or \verb|Util_TableDeleteKey()| calls
-are made on that table (making such calls on other tables doesn't
-matter).  The order may change if there is any change in the table,
-and it may differ even between different tables with identical key/value
-contents (including those produced by \verb|Util_TableClone()|).%%%
-\footnote{%%%
-	 For example, if tables were implemented by hashing,
-	 the internal order could be that of the hash buckets,
-	 and the hash function could depend on the internal
-	 table address.
-	 }%%%
-{}
-
-Any change in the table also invalidates all iterators pointing
-anywhere in the table; using any such iterator is an error.
-Multiple iterators may point into the same table; they all use the
-same order, and (unlike in Perl) they're all independent.
-
-The detailed function description
-in the Reference Manual
-for \verb|Util_TableItQueryKeyValueInfo()| has an example of
-using an iterator to print out all the entries in a table.
-
-%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
-
-\section{Multithreading and Multiprocessor Issues}
-
-At the moment, the table functions are \emph{not} thread-safe
-in a multithreaded environment. 
-%% However, this should change in
-%%the not-too-distant future: then all the table functions will default
-%to being thread-safe.  That is, user code will be able call these
-%%functions concurrently from multiple threads, with the table functions
-%%automatically doing any necessary locking.%%%
-%%\footnote{%%%
-%	 For the implementation, this means that we will need a
-%	 reader/writer lock for each table and for each iterator:
-%	 any number of threads will be able to concurrently read
-%	 the data structure, but any write will require exclusive
-%	 access.
-%	 }%%%
-%{}  (We may add a flags-word bit to suppress this for maximum
-%performance if you know you won't be making concurrent calls from
-%multiple threads.)
-
-Note that tables and iterators are process-wide, i.e. all
-threads see the same tables and iterators (think of them as like the
-Unix current working directory, with the various routines which modify
-the table or change iterators acting like a Unix \verb|chdir()| system
-call).
-
-In a multiprocessor environment, tables are always processor-local.
-
-%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
-
-\section{Metadata about All Tables}
-
-Tables do not \emph{themselves} have names or other
-attributes.  However, we may add some special
-``system tables'' to be used by Cactus itself to store this sort of
-information for those cases where it's needed. For example, we may add support for a
-``checkpoint me'' bit in a table's flags word, so that if you want a
-table to be checkpointed, you just need to set this bit.
-In this case, the table will probably get a system generated name in
-the checkpoint dump file.  But if you want the table to have some
-other name in the dump file, then you need to tell the checkpointing
-code that, by setting an appropriate entry in a checkpoint table.
-(You would find the checkpoint table by looking in a special
-``master system table'' that records handles of other interesting tables.)
-
-%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
-%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
-%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
-
-\end{cactuspart}