Added section on how to add a new checkpointing/recovery method using the

I/O infrastructure provided by IOUtil.


git-svn-id: http://svn.cactuscode.org/arrangements/CactusBase/IOUtil/trunk@156 b32723a9-ab3a-4a60-88e2-2e5d99d7c17a

1 files changed, 545 insertions, 343 deletions

diff --git a/doc/documentation.tex b/doc/documentation.tex
index a740e5b..c7d6c8a 100644
--- a/doc/documentation.tex
+++ b/doc/documentation.tex
@@ -5,226 +5,241 @@
 
 \title{IOUtil}
 \author{Thomas Radke, Gabrielle Allen}
-\date{June 2000}
+\date{June 2000\\$ $Revision$ $Date$ $}
 \maketitle
 
 \section{Introduction}
 
-This document details the thorns provided in the standard Cactus 
-distribution for the output of grid variables, and describes how to 
-set parameters for I/O and checkpointing using these thorns.
-
-Input and Output of data (I/O) in Cactus is provided by infrastructure thorns,
-which interact with the flesh via a fixed interface, which is described in 
-the Users Guide. 
-The standard release of Cactus contains a number of thorns 
-which provide parallel I/O using a variety of data formats and methods. 
-All these provided I/O methods use thorn {\tt CactusBase/IOUtil} which 
-provides general utilities for I/O (such as parsing parameter strings to
-decide which variables to output), and a general set of parameters which
-are inherited by the different I/O methods (such as the output directory). 
-Thorn {\tt IOUtil} by itself provides no I/O methods.
-
-More information about I/O and Visualisation of Cactus data can be found in 
-the individual I/O thorns, and in the {\tt Visualization-HOWTO} available 
-on the Cactus web pages.
-
-\section{I/O methods in Cactus}
-
-Cactus has several I/O modes for the output of grid variables in different 
-formats.
-Each I/O method comes with its own parameters by which it can be customised,
-and all methods are registered with the flesh, satisfying the 
-Cactus API, allowing them to be called directly from application thorns. 
-An I/O method registers itself with the flesh along with it's name, and 
-these registered names are the labels we now use to describe the various
-methods. 
+This document details the thorns provided in the standard Cactus distribution
+for the output of grid variables, and describes how to set parameters for I/O
+and checkpointing/recovery using these thorns.
+
+Input and output of data (I/O) in Cactus is provided by infrastructure thorns,
+which interact with the flesh via a fixed interface, which is described in the
+Users' Guide. The standard release of Cactus contains a number of thorns which
+provide so-called I/O methods implementing the actual I/O in a variety of data
+formats and styles. All these provided I/O methods use thorn {\tt
+CactusBase/IOUtil} which provides general utilities for I/O (such as parsing
+parameter strings to decide which variables to output), and a general set of
+parameters which are inherited by the different I/O methods (such as the output
+directory). Thorn {\tt IOUtil} by itself provides no I/O methods.
+
+More information about I/O and visualisation of Cactus data can be found in the
+individual I/O thorns, and in the {\tt Visualization-HOWTO} available on the
+Cactus web pages.
+
+\section{I/O Methods in Cactus}
+
+Cactus has several I/O methods for the output of grid variables in different
+data formats and styles. Each I/O method comes with its own parameters by which
+it can be customised, and all methods are registered with the flesh, satisfying
+the Cactus API, allowing them to be called directly from application thorns.
+An I/O method registers itself with the flesh along with it's name, and these
+registered names are the labels we now use to describe the various methods.
 
 
 \begin{table}
-\begin{tabular}{|l|p{4.5cm}|l|}
+\begin{center}
+\begin{tabular}{|l|l|l|}
   \hline
   {\bf I/O method} & {\bf Description} & {\bf Providing Thorn}\\
   \hline
-  {\tt Scalar} & 
-  output of scalars or grid array reductions in xgraph or gnuplot format & 
+  {\tt Scalar} &
+  output of scalars or grid array reductions in xgraph or gnuplot format &
   {\tt CactusBase/IOBasic}
 \\
-  {\tt Info}      & 
-  screen output of scalars or grid array reductions & 
+  {\tt Info}      &
+  screen output of scalars or grid array reductions &
   {\tt CactusBase/IOBasic}
 \\
 &&\\
-  {\tt IOASCII\_1D} & 
-  1D line output of grid arrays in xgraph or gnuplot format & 
+  {\tt IOASCII\_1D} &
+  1D line output of grid arrays in xgraph or gnuplot format &
   {\tt CactusBase/IOASCII}
 \\
-  {\tt IOASCII\_2D} & 
-  2D slice output of grid arrays in gnuplot format & 
+  {\tt IOASCII\_2D} &
+  2D slice output of grid arrays in gnuplot format &
   {\tt CactusBase/IOASCII}
 \\
-&&\\
-  {\tt IOHDF5\_2D} & 
-  2D slice output of grid arrays in HDF5 format & 
-  {\tt CactusPUGHIO/IOHDF5}
+  {\tt IOASCII\_3D} &
+  full 3D output of grid arrays in gnuplot format &
+  {\tt CactusBase/IOASCII}
 \\
-  {\tt IOHDF5\_3D} & 
-  full 3D output of grid arrays in HDF5 format & 
+&&\\
+  {\tt IOHDF5\_3D} &
+  full 3D output of grid arrays in HDF5 format &
   {\tt CactusPUGHIO/IOHDF5}
 \\
 &&\\
-  {\tt IOFlexIO\_2D} & 
-  2D slice output of grid arrays in FlexIO format & 
+  {\tt IOFlexIO\_2D} &
+  2D slice output of grid arrays in FlexIO format &
   {\tt CactusPUGHIO/IOFlexIO}
 \\
-  {\tt IOFlexIO\_3D} & 
-  full 3D output of grid arrays in FlexIO format & 
+  {\tt IOFlexIO\_3D} &
+  full 3D output of grid arrays in FlexIO format &
   {\tt CactusPUGHIO/IOFlexIO}
 \\
   \hline
 \end{tabular}
 \caption{Standard I/O methods provided with the Cactus distribution}
 \label{one}
+\end{center}
 \end{table}
 
 The standard provided Cactus I/O methods are shown in Table~\ref{one}.
-As described above, each of these I/O thorns inherit parameters from thorn 
+As described above, each of these I/O thorns inherit parameters from thorn
 {\tt CactusBase/IOUtil}, which must be included in your ThornList
-and activated in your parameter files before any of these I/O methods 
-can be used. {\tt CactusBase/IOUtil} allows you to set the default 
+and activated in your parameter files before any of these I/O methods
+can be used. {\tt CactusBase/IOUtil} allows you to set the default
 behaviour for all the I/O methods described above, for example, setting
-the parameter {\tt IO::out\_every=1} will result in any chosen I/O method
+the parameter {\tt IO::out\_every = 1} will result in any chosen I/O method
 providing output on each iteration. The default behaviour can be overriden
 by specific parameters for each method.
 For example, you may want scalar and 1D output at every iteration
-but computationally expensive 3D output only every 10th iteration, 
+but computationally expensive 3D output only every 10th iteration,
 with these files going into another directory on a scratch partition.
 
-\section{External Packages}
-
-NOTE: This section will not be filled out properly until the new treatment of external 
-packages is implemented.
 
-Some of the I/O methods used by Cactus require that external packages are installed and available 
-on your machine. 
-
-\begin{itemize}
-
-\item{\tt HDF5} {\bf Hierachical Data Format}\\
-{\tt http://hdf.ncsa.uiuc.edu/}
-
-\item{\tt IEEEIO} \\
-{\tt http://zeus.ncsa.uiuc.edu/}
-
-\end{itemize}
+\section{Providing Your own I/O Method}
+
+If you as a Cactus developer have your own input/output routines and want to
+share this functionality with other people you should do this by putting them
+into an new I/O thorn and register them as an I/O method.
+A description on how to register a new I/O method with the flesh's I/O subsystem
+can be found in the {\it Infrastructure Thorn Writer's Guide} (as part of the
+{\it Cactus User's Guide}).
+
+New I/O thorns should always inherit from thorn {\tt CactusBase/IOUtil} in order
+to reuse as much of the existing I/O infrastructure as possible, and to maintain
+a uniform interface on how to use the I/O methods.
+
+
+%\section{External Packages}
+%
+%NOTE: This section will not be filled out properly until the new treatment of external
+%packages is implemented.
+%
+%Some of the I/O methods used by Cactus require that external packages are installed and available
+%on your machine.
+%
+%\begin{itemize}
+%
+%\item{\tt HDF5} {\bf Hierachical Data Format}\\
+%{\tt http://hdf.ncsa.uiuc.edu/}
+%
+%\item{\tt IEEEIO} \\
+%{\tt http://zeus.ncsa.uiuc.edu/}
+%
+%\end{itemize}
 
 \section{Standard Parameters}
 
-Here we describe a few of the standard parameters used by {\tt IOUtil}
-to control output. 
+Here we describe a few of the standard parameters used by {\tt IOUtil} to
+control output.
 \begin{itemize}
 
-\item{\tt IO::outdir}\\
-  The name of the directory to be used for output. All the I/O 
-  methods described here will write by default to this directory. 
-  Individual methods have parameters
-which can direct their output to a different directory.
+  \item{\tt IO::outdir}\\
+    The name of the directory to be used for output. All the I/O methods
+    described here will write by default to this directory (which itself
+    defaults to the current working directory). Individual methods have
+    parameters which can direct their output to a different directory.
 
-\item{\tt IO::out\_every}\\
-  How often, in terms of iterations, each of the Cactus I/O methods will
-  write output. Again, individual methods can set there own parameter 
-  to override this. The default is to never write output.
+  \item{\tt IO::out\_every}\\
+    How often, in terms of iterations, each of the Cactus I/O methods will write
+    output. Again, individual methods can set there own parameter to override
+    this. The default is to never write output.
 \end{itemize}
 
-\section{Manipulating Parameter Files}
 
-Thorn {\tt CactusBase/IOUtil} can also manage and manipulate parameter 
-files, using the parameter {\tt IO::parfile\_write} to 
+\section{Saving/Generating Parameter Files}
+
+Thorn {\tt IOUtil} can save a copy of the parameter file of a run, or can also
+automatically generate a parameter file from all current parameter settings.
+This is controlled by the {\tt IO::parfile\_write} parameter:
 
 \begin{itemize}
 
-  \item{\tt IO::parfile\_write="copy"} This is the default option,
-  and makes an exact replica of the input parameter file in the
-  standard output directory (this is particularly useful when the
-  output directory is going to be archived).
+  \item{\tt IO::parfile\_write="copy"}\\
+    This is the default option, and makes an exact replica of the input
+    parameter file in the standard output directory (this is particularly useful
+    when the output directory is going to be archived).
 
-  \item{\tt IO::parfile\_write="generate"} Generate a new parameter
-  file from runtime information, containing the Cactus version, the
-  name of the original parameter file, the run time/date, the host to
-  run on, and the number of processors - all on comment
-  lines. Following this the parameter file contains the ActiveThorns
-  list plus a sorted list of all active thorns' parameters which have
-  been set in the original parameter file.
+  \item{\tt IO::parfile\_write="generate"}\\
+    Generate a new parameter file from runtime information, containing the
+    Cactus version, the name of the original parameter file, the run time/date,
+    the host to run on, and the number of processors - all on comment lines.
+    Following this the parameter file contains the ActiveThorns list plus a
+    sorted list of all active thorns' parameters which have been set in the
+    original parameter file.
 
-  \item{\tt IO::parfile\_write="no"} Switch off writing of a new 
-	parameter file.
+  \item{\tt IO::parfile\_write="no"}\\
+    Switch off writing of a new parameter file.
 
 \end{itemize}
 
-The name of the new parameter file defaults to the original filename, 
-unless the parameter {\tt IO::parfile\_name} is set. Note that if a file
-already exists with the chosen name it will not be overwritten.
-  
+The name of the new parameter file defaults to the original filename, unless the
+parameter {\tt IO::parfile\_name} is set. Note that if a file already exists
+with the chosen name it will not be overwritten.
+
 
-\section{I/O modes}
+\section{I/O Modes}
 \label{iomodes}
-For a run on multiple processors, scalar, 1D, and 2D output will always be 
+For a run on multiple processors, scalar, 1D, and 2D output will always be
 written from only processor zero (that is, required data from all other
 processors will be sent to processor zero, who then outputs all the gathered
-data).
-For 3D output this may become a quite expensive operation since output
-by only a single processor will probably result in a I/O bottleneck and delay
-further computation. 
-For this reason Cactus offers different I/O modes for 3D output which can be
-controlled by the {\tt IO::out\_3Dmode} parameter, in combination with
- {\tt IO::out\_unchunked} and {\tt IO::out3D\_np}. These parameters allow
-I/O to be optimised for your particular machine architecture and needs: 
+data). For 3D output this may become a quite expensive operation since output
+by only a single processor will probably result in an I/O bottleneck and delay
+further computation. For this reason Cactus offers different I/O modes for 3D
+output which can be controlled by the {\tt IO::out\_3Dmode} parameter, in
+combination with {\tt IO::out\_unchunked} and {\tt IO::out3D\_np}. These
+parameters allow I/O to be optimised for your particular machine architecture
+and needs:
 
 \begin{itemize}
   \item {\tt IO::out3D\_mode = "onefile"}\\
-    As for the 1D and 2D I/O methods, writing to file is performed only 
+    As for the 1D and 2D I/O methods, writing to file is performed only
     by processor zero.
     This processor gathers all the output data from the other processors
-    and writes to a single file. The gathered grid array data from each
-    processor can be either
-    written in chunks ({\tt IO::out3D\_unchunked = "no"}) with each chunk
-    containing the data from a single processor, or collected into a
-    single global array before writing ({\tt IO::out3D\_unchunked = "yes"}).
-    The default is to write the data in chunks.
+    and then writes to a single file. The gathered grid array data from each
+    processor can be either written in chunks ({\tt IO::out3D\_unchunked =
+    "no"}) with each chunk containing the data from a single processor, or
+    collected into a single global array before writing ({\tt
+    IO::out3D\_unchunked = "yes"}). The default is to write the data in chunks.
+
   \item {\tt IO::out3D\_mode = "np"}\\
     3D output is written in parallel for groups of processors. Each group
-    consists of {\tt IO::out3D\_procs} processors which have assigned one
-    I/O processor which gathers data from the group and writes it
-    to file. The chunked output will go into {\tt IO::out3D\_procs} files. 
+    consists of {\tt IO::out3D\_procs} processors which have assigned one I/O
+    processor which gathers data from the group and writes it to file. The
+    chunked output will go into {\tt IO::out3D\_procs} files.
     The default number of processors in a group is eight.
+
   \item {\tt IO::out3D\_mode = "proc"}\\
     This is the default method for 3D output.
     Every processor writes its own chunk of data into a separate output file.
 \end{itemize}
 
-Probably the single processor {\tt proc} mode is the most efficient 3D output mode on 
-machines with a fast I/O subsystem and many I/O nodes (eg. an Origin 2000) 
-because it provides the highest
+Probably the single processor {\tt "proc"} mode is the most efficient 3D output
+mode on machines with a fast I/O subsystem and many I/O nodes (eg. a Linux
+cluster with local disks attached to each node) because it provides the highest
 parallelity for outputting data. Note that on very large numbers of processors
-you mave have to fall back to {\tt np}, output from groups of processors, 
-mode if the system limit of
-maximum open file descriptors is exceeded (this is true for large jobs on a
- T3E).\\
-While the {\tt np} and {\tt proc} I/O modes are fast for outputting large
+you mave have to fall back to {\tt "np"}, output from groups of processors,
+mode if the system limit of maximum open file descriptors is exceeded (this is
+true for large jobs on a T3E).
+
+While the {\tt "np"} and {\tt "proc"} I/O modes are fast for outputting large
 amounts of data from all or a group of processors in parallel, they have
-the disadvantage of writing chunked files. These files have to be 
-{\tt recombined}
-during a postprocessing phase so that the final unchunked data can be
+the disadvantage of writing chunked files. These files then have to be
+recombined during a postprocessing phase so that the final unchunked data can be
 visualized by standard tools. For that purpose a recombiner utility program is
-provided by thorn {\tt CactusPUGHIO/IOFlexIO}.
-Note that {\tt CactusPUGHIO/IOHDF5} does not provide such a utility yet. 
+provided by the thorns offering parallel I/O methods.
+
 
 \section{Output of Hyperslab Data}
 
 While some I/O methods ({\tt IOHDF5\_3D, IOFlexIO\_3D}) can dump the full
-contents of a multidimensional CCTK variable, others such as
-{\tt IOASCII\_1D} and {\tt IOASCII\_2D} will output only a subset of the data
-(eg. 1D lines or 2D planes of 3D grid funtions).
+contents of a multidimensional CCTK variable, others such as {\tt IOASCII\_1D}
+and {\tt IOASCII\_2D} will output only a subset of the data (eg. 1D lines or 2D
+planes of 3D grid functions).
 Such a subset (called a {\it hyperslab}) is generally defined as an orthogonal
 region into the multidimensional dataset, with a startpoint and a length in any
 direction, and an optional downsampling factor.
@@ -251,8 +266,8 @@ which then will overwrite the defaults provided by {\tt IOUtil}.
     specifies the downsampling factor for 3D output in every direction
 \end{itemize}
 
-Setting the index points for the slice centers in a parameter file
-has precedence over setting their location by coordinate values.
+Setting the index points for the slice centers in a parameter file has
+precedence over setting their location by coordinate values.
 If nothing was specified the default values of {\tt IO::out\_[xyz]line\_[xyz]}
 and {\tt IO::out\_[\{xy\}\{xz\}\{yz\}]plane\_[xyz]} will be used.
 These are set to be all zeros which causes the output to go through the
@@ -268,11 +283,11 @@ coordinate system's origin or the closest grid point to this (see figure
 \label{default_1D_output}
 \end{figure}
 
-Although they are not hyperslabs by the above definition, output of 1D
-diagonals for 3D grid arrays is also supported by I/O method {\tt IOASCII\_1D}
-but has the restriction that the line will always start in the bottom-left
-corner of the computational grid and steadily rise by one grid point in
-every direction (see figure \ref{default_diagonal_output} for an example).
+Although they are not hyperslabs by the above definition, output of 1D diagonals
+for 3D grid arrays is also supported by I/O method {\tt IOASCII\_1D} but has the
+restriction that the line will always start in the bottom-left corner of the
+computational grid and steadily rise by one grid point in every direction (see
+figure \ref{default_diagonal_output} for an example).
 
 \begin{figure}[ht]
 \begin{center}
@@ -285,29 +300,86 @@ computational grid}
 \end{figure}
 
 
+\section{Data Filenames and Filename Extensions}
+
+The standard I/O thorns in Cactus make use of a consistent set of filenames
+and extensions, which identify the variables and data format used in the file.
+The filenames are listed in the following tables.
+%The filenames are listed in Table~\ref{filename_table},
+%and the extensions in Table~\ref{filename_extensions_table}
+
+\begin{table}[htb]
+\begin{center}
+\label{filename_table}
+\begin{tabular}{|l|l|}
+  \hline
+  {\bf I/O method}   & {\bf Filename for output of variable {\tt phi}}\\
+  \hline
+  {\tt Scalar}       & phi\_nm1.tl, phi\_nm2.tl phi\_max.tl phi\_min.tl\\
+  {\tt Info}         & Provides no output file (the same data is available with {\tt Scalar} output)\\
+  {\tt IOASCII\_1D}  & phi.xl, phi.yl, phi.zl, phi.dl\\
+  {\tt IOASCII\_2D}  & phi\_2d\_xy.gnuplot, phi\_2d\_xz.gnuplot, phi\_2d\_yz.gnuplot\\
+  {\tt IOASCII\_3D}  & phi\_3d.gnuplot\\
+  {\tt IOHDF5\_3D}   & phi\_3d.h5\\
+  {\tt IOFlexIO\_2D} & phi\_2d.ieee\\
+  {\tt IOFlexIO\_3D} & phi\_3d.ieee\\
+  \hline
+\end{tabular}
+\caption{Filenames used by standard I/O thorns}
+\end{center}
+\end{table}
+
+\begin{table}[htb]
+\begin{center}
+\label{filename_extensions_table}
+\begin{tabular}{|c|l|l|}
+\hline
+{\bf Extension} & {\bf Description} & {\bf Thorn} \\
+  \hline
+ {\tt .xl} & 1D line plots in {\it x}-direction & {\tt CactusBase/IOASCII}\\
+ {\tt .yl} & 1D line plots in {\it y}-direction & {\tt CactusBase/IOASCII}\\
+ {\tt .zl} & 1D line plots in {\it z}-direction & {\tt CactusBase/IOASCII}\\
+ {\tt .dl} & 1D diagonal line plots             & {\tt CactusBase/IOASCII}\\
+ {\tt .tl} & traceline plots over time          & {\tt CactusBase/IOBasic}\\
+  \hline
+\end{tabular}
+\caption{File extensions used by the standard I/O thorns}
+\end{center}
+\end{table}
+
+
 \section{Checkpointing and Recovery in Cactus}
+\label{cp_recovery}
 
 The I/O methods for 3D output also provide functionality for {\it checkpointing}
-and {\it recovery}. A checkpoint is a snapshot of the current state of the 
-simulation ({\it ie} the contents of all the grid variables and the parameter 
-settings) at a chosen timestep. Each checkpoint is saved into a 
+and {\it recovery}. A checkpoint is a snapshot of the current state of the
+simulation ({\it ie} the contents of all the grid variables and the parameter
+settings) at a chosen timestep. Each checkpoint is saved into a
 {\it checkpoint file} which can be used to restart a new simulation at a later
-time, recreating the exact state at which it was checkpointed.\\
-Checkpointing is especially useful when running Cactus in batch queue systems 
-where jobs get only limited CPU time. A more advanced use of checkpointing 
+time, recreating the exact state at which it was checkpointed.
+
+Checkpointing is especially useful when running Cactus in batch queue systems
+where jobs get only limited CPU time. A more advanced use of checkpointing
 would be to restart your simulation after a crash or a problem had developed,
-using a  different parameter set recovering from the latest stable timestep. 
+using a  different parameter set recovering from the latest stable timestep.
 Additionally, for performing parameter studies,  compute-intensive
-initial data can be calculated just once and saved in a checkpoint file 
-from which each job can be started. \\
+initial data can be calculated just once and saved in a checkpoint file
+from which each job can be started.
 
 Again, thorn {\tt CactusBase/IOUtil} provides general checkpoint \& recovery
 parameters. The most important ones are:
 \begin{itemize}
   \item {\tt IO::checkpoint\_every}\\
-    specifies how often to checkpoint in terms of iteration number
+    specifies how often to write a periodic checkpoint in terms of iteration
+    number
+  \item {\tt IO::checkpoint\_next}\\
+    triggers a checkpoint at the end of the current iteration
+  \item {\tt IO::checkpoint\_next}\\
+    triggers a checkpoint at the end of the last iteration of a simulation run
   \item {\tt IO::checkpoint\_file}\\
-    holds the checkpoint filename
+    holds the basename for checkpoint file(s) to create\\
+    Iteration number and file extension are appended by the individual I/O
+    method used to write the checkpoint.
   \item {\tt IO::checkpoint\_dir}\\
     names the directory where checkpoint files are stored
   \item {\tt IO::checkpoint\_ID}\\
@@ -316,64 +388,69 @@ parameters. The most important ones are:
     keyword parameter telling if/how to recover.\\
     Choices are {\tt "no"}, {\tt "manual"}, {\tt "auto"}, and {\tt "autoprobe"}.
   \item {\tt IO::recover\_file}\\
-    filename of the recovery file
+    basename of the recovery file\\
+    Iteration number and file extension are appended by the individual I/O
+    method used to recover from the recovery file.
   \item {\tt IO::recovery\_dir}\\
     directory where the recovery file is located
 \end{itemize}
 
 To checkpoint your simulation, you need to enable checkpointing by setting
-the Boolean parameter {\tt checkpoint}, for one of the 3D I/O methods to
+the boolean parameter {\tt checkpoint}, for one of the 3D I/O methods to
 {\tt yes}.
-Checkpoint filenames consist of a basename (as specified in
-{\tt IO::checkpoint\_file}) followed by {\tt ".chkpt.it\_$<$iteration\_number$>$"}
+Checkpoint filenames consist of a basename (as specified in {\tt
+IO::checkpoint\_file}) followed by {\tt ".chkpt.it\_$<$iteration\_number$>$"}
 plus the file extension indicating the file format ({\tt ".ieee"} for IEEEIO
 data from {\tt CactusPUGHIO/IOFlexIO}, or {\tt ".h5} for HDF5 data from
-{\tt CactusPUGHIO/IOHDF5}).\\
-Use the {\tt "manual"} mode to recover from a specific checkpoint file
-by adding the iteration number to the basename parameter.
-
-The {\tt "auto"} recovery mode will automatically
-recover from the latest checkpoint file found in the recovery directory.
-In this case {\tt IO::recover\_file} should contain the basename only.
-
-The {\tt "autoprobe"} recovery mode is similar to the {\tt "auto"} mode
-except that it would not stop the code if no checkpoint file was
-found but only print a warning message and then continue with the
-simulation. This mode allows you to enable checkpointing and recovery
-in the same parameter file and use that without any changes to restart your
-simulation. On the other hand, you are responsible now for making the
-checkpoint/recovery directory/file parameters match -- a mismatch will not
-be detected by Cactus in order to terminate it. Instead the simulation would
-always start from initial data without any recovery.\\
-
-Because the same I/O methods implement both output of 3D data and
-checkpoint files, the same I/O modes are used (see Section~\ref{iomodes}). 
-Note that the recovery routines in Cactus can process both chunked and 
-unchunked checkpoint files if you restart on the same number of processors.
-That's why you should always use one of the parallel I/O modes for 
-checkpointing. If you want to restart on a different number of processors, you first need to 
-{\it recombine} the data in the checkpoint file(s) to create a single file with
-unchunked data.
-Note that you can restart from your checkpoint file on a different machine.
+{\tt CactusPUGHIO/IOHDF5}).
+
+Use the {\tt "manual"} mode to recover from a specific checkpoint file by adding
+the iteration number to the basename parameter.
+
+The {\tt "auto"} recovery mode will automatically recover from the latest
+checkpoint file found in the recovery directory.
+In this case {\tt IO::recover\_file} should contain the basename only (without
+any iteration number).
+
+The {\tt "autoprobe"} recovery mode is similar to the {\tt "auto"} mode except
+that it would not stop the code if no checkpoint file was found but only print
+a warning message and then continue with the simulation. This mode allows you
+to enable checkpointing and recovery in the same parameter file and use that
+without any changes to restart your simulation. On the other hand, you are
+responsible now for making the checkpoint/recovery directory/file parameters
+match -- a mismatch will not be detected by Cactus in order to terminate it.
+Instead the simulation would always start from initial data without any
+recovery.
+
+Because the same I/O methods implement both output of 3D data and checkpoint
+files, the same I/O modes are used (see Section~\ref{iomodes}).
+Note that the recovery routines in Cactus can process both chunked and unchunked
+checkpoint files if you restart on the same number of processors -- no
+recombination is needed here. That's why you should always use one of the
+parallel I/O modes for checkpointing. If you want to restart on a different
+number of processors, you first need to recombine the data in the checkpoint
+file(s) to create a single file with unchunked data.
+Note that Cactus checkpoint files are platform-independent so you can restart
+from your checkpoint file on a different machine/architecture.
 
 \vskip .5cm
 \noindent{\bf WARNING:}
 
 \noindent
-Checkpointing and recovery should {\bf always} be tested for a new thorn set. This is because
-only Cactus grid variables and parameters are saved in a checkpoint file. If a thorn has made 
-use of saved local
-variables, the state of a recovered simulation may differ from the original run. To test 
-checkpointing and recovery, simply perform one run of say 10 timesteps, and compare output data
-with a checkpointed and recovered run at say the 5th timestep. The output data should match 
-exactly if recovery was successful.
-
-%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+Checkpointing and recovery should {\bf always} be tested for a new thorn set.
+This is because only Cactus grid variables and parameters are saved in a
+checkpoint file. If a thorn has made use of saved local variables, the state of
+a recovered simulation may differ from the original run. To test checkpointing
+and recovery, simply perform one run of say 10 timesteps, and compare output
+data with a checkpointed and recovered run at say the 5th timestep.
+The output data should match exactly if recovery was successful.
+
+
 \section{Reading Data from Files into Cactus}
 
 The very same routines which implement checkpointing/recovery functionality
 in the {\tt IOHDF5} and {\tt IOFlexIO} thorns are also used to provide
-filereader capabilities within Cactus. They enable a user to read variables,
+filereader capabilities within Cactus. They enable users to read variables,
 which contents was written to files in HDF5 or IEEEIO data format, back into
 Cactus at a later time. This is especially useful if compute-intensive
 initial data is calculated only once and stored in a file. Such data
@@ -385,213 +462,338 @@ read from file(s) as initial data:
 
 \begin{itemize}
   \item {\tt IO::recover\_ID\_files}\\
-  list of files to read in as initial data (multiple filenames must be separated by spaces)\\
-  The same file naming conventions (what I/O mode used, which iteration number)
-  apply as for checkpoint files.
+    list of files to read in as initial data (multiple filenames must be
+    separated by spaces)\\
+    The same file naming conventions (what I/O mode used, which iteration
+    number) apply as for checkpoint files.
   \item {\tt IO::recover\_ID\_vars}\\
-  list of CCTK variables to read in from the given initial data files (variables
-  are identified by their full name, multiple variable names must be separated by spaces)\\
-  This is useful if a datafile contains multiple variables but only some of
-  them should be read. Thus it is possible to recover distinguished
-  variables from a full checkpoint file. Note that if the file contains several
-  timesteps of the same variable only the last one is taken.
+    list of CCTK variables to read in from the given initial data files
+    (variables are identified by their full name, multiple variable names must
+    be separated by spaces)\\
+    This is useful if a datafile contains multiple variables but only some of
+    them should be read. Thus it is possible to recover distinguished variables
+    from a full checkpoint file. Note that if the file contains several
+    timesteps of the same variable only the last one is taken.
 \end{itemize}
 
-Thorn {\tt IOUtil} also provides a filereader routine which can be called
+Thorn {\tt IOUtil} also provides a filereader API which can be called
 by any application thorn at any time. It gets passed the equivalent information
-to the filereader parameters, plus a pointer to the underlying CCTK grid hierarchy.
+to the filereader parameters, plus a pointer to the underlying CCTK grid
+hierarchy:
 
 \begin{verbatim}
-void IOUtil_RecoverVarsFromDatafiles (cGH *GH,
-                                      const char *in_files,
-                                      const char *in_vars);
+  void IOUtil_RecoverVarsFromDatafiles (cGH *GH,
+                                        const char *in_files,
+                                        const char *in_vars);
 \end{verbatim}
 
-If data is to be imported from files which were not created by
-{\tt IOHDF5} or {\tt IOFlexIO} it needs to be converted first into the
-appropriate HDF5 or IEEEIO file format and the file layout
-which either one of these thorns uses. This is described in detail in the
-thorns' documentation, along with a simple C source file which can be used as a
-template to build your own data converter program.
+If data is to be imported from files which were not created by {\tt IOHDF5} or
+{\tt IOFlexIO} it needs to be converted first into the appropriate HDF5 or
+IEEEIO file format and the file layout which either one of these thorns uses.
+This is described in detail in the thorns' documentation, along with a simple C
+source file which can be used as a template to build your own data converter
+program.
 
 
-%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
-\section{Data Filenames and Filename Extensions}
-
-The standard I/O thorns in Cactus make use of a consistent set of filenames
-and extensions, which identify the variables and data format used in the file.
-The filenames are listed in the following tables.
-%The filenames are listed in Table~\ref{filename_table},
-%and the extensions in Table~\ref{filename_extensions_table}
-
-\begin{center}
-\begin{table}[htb]
-\label{filename_table}
-\begin{tabular}{|l|l|}
-  \hline
-  {\bf I/O method}   & {\bf Filename for output of variable {\tt phi}}\\
-  \hline
-  {\tt Scalar}       & phi\_nm1.tl, phi\_nm2.tl phi\_max.tl phi\_min.tl\\
-  {\tt Info}         & Provides no output file (the same data is available with {\tt Scalar} output)\\
-  {\tt IOASCII\_1D}  & phi.xl, phi.yl, phi.zl, phi.dl\\
-  {\tt IOASCII\_2D}  & phi\_2d\_xy.gnuplot, phi\_2d\_xz.gnuplot, phi\_2d\_yz.gnuplot\\
-  {\tt IOHDF5\_3D}   & phi\_3d.h5\\
-  {\tt IOFlexIO\_2D} & phi\_2d.ieee\\
-  {\tt IOFlexIO\_3D} & phi\_3d.ieee\\
-  \hline
-\end{tabular}
-\caption{Filenames used by standard I/O thorns}
-\end{table}
-\end{center}
-
-\begin{center}
-\begin{table}[htb]
-\label{filename_extensions_table}
-\begin{tabular}{|c|p{4.5cm}|l|}
-\hline
-{\bf Extension} & {\bf Description} & {\bf Thorn} \\
-  \hline
- {\tt .xl} & 1D line plots in {\it x}-direction & {\tt CactusBase/IOASCII}\\
- {\tt .yl} & 1D line plots in {\it y}-direction & {\tt CactusBase/IOASCII}\\
- {\tt .zl} & 1D line plots in {\it z}-direction & {\tt CactusBase/IOASCII}\\
- {\tt .dl} & 1D diagonal line plots             & {\tt CactusBase/IOASCII}\\
- {\tt .tl} & traceline plots over time          & {\tt CactusBase/IOBasic}\\
-  \hline 
-\end{tabular}
-\caption{File extensions used by the standard I/O thorns in the Cactus distribution}
-\end{table}
-\end{center}
- 
 \section{Example Parameter Files}
 
 Here we give examples of the parameters for the different I/O methods.
 
 \begin{itemize}
-\item{\bf Screen Information {\tt info}}
+\item{\bf Output information to screen using {\tt IOBasic's "Info"}} I/O method
+\begin{verbatim}
+  ActiveThorns = "IOBasic IOUtil PUGHReduce ..."
+
+  # Output using all methods on iteration 0, 10, 20, ...
+  IO::out_every = 10
+
+  # Group of variables to output to screen
+  IOBasic::outInfo_vars = "evolve::vars"
+\end{verbatim}
+
+\item{\bf Scalar Output from {\tt IOBasic's "Scalar"} I/O method}
 \begin{verbatim}
-ActiveThorns = "iobasic ioutil ..."
+  ActiveThorns = "IOBasic IOUtil PUGHReduce ..."
 
-# Output using all methods on iteration 0, 10, 20, ...
-io::out_every = 10            
+  # Output vars using scalar method on iteration 0, 10, 20, ...
+  IOBasic::outScalar_every = 10
 
-# Group of variables to output to screen
-iobasic::outinfo_vars = "evolve::vars"
+  # Group of variables to output to file
+  IOBasic::outScalar_vars = "evolve::vars"
 \end{verbatim}
 
-\item{\bf Scalar Output {\tt scalar}}
+\item{\bf ASCII 1D and 2D Output with {\tt IOASCII's "IOASCII\_1D"} and
+{\tt "IOASCII\_2D"} I/O methods}
 \begin{verbatim}
-ActiveThorns = "iobasic ioutil ..."
+  ActiveThorns = "IOASCII IOUtil PUGHSlab ..."
 
-# Output vars using scalar method on iteration 0, 10, 20, ...
-iobasic::outscalar_every = 10            
+  # Output vars in 1D on iteration 0, 10, 20, ...
+  IOASCII::out1D_every = 10
 
-# Group of variables to output to file
-iobasic::outscalar_vars = "evolve::vars"
+  # Output vars in 2D on iteration 0, 50, 100, ...
+  IOASCII::out2D_every = 50
 \end{verbatim}
 
-\item{\bf ASCII 1D and 2D Output {\tt ioascii\_1d}, {\tt ioascii\_2d}}
+\item{\bf HDF5 Output with {\tt IOHDF5's "IOHDF5\_3D"} I/O method}
 \begin{verbatim}
-ActiveThorns = "ioascii ioutil pughslab ..."
+  ActiveThorns = "IOHDF5 IOUtil PUGHSlab ..."
+
+  # Output vars in HDF5 format on iteration 0, 5, 10, ...
+  IOHDF5::out3D_every = 5
 
-# Output vars using ascii 1d method on iteration 0, 10, 20, ...
-ioascii::out1d_every = 10            
+  # Group of variables to output
+  IOHDF5::out3D_vars = "evolve::vars"
 
-# Output vars using ascii 2d method on iteration 0, 50, 100, ...
-ioascii::out2d_every = 50
+  # Special I/O directory for HDF5 output
+  IOHDF5::outdir3D = "/scratch/tmp"
 
-# Group of variables to output to file for each method
-iobasic::outscalar_vars = "evolve::vars"
+  # 3D Output unchunked to one file
+  # (Only using a small number of processors)
+  IO::out3D_mode      = "onefile"
+  IO::out3D_unchunked = "yes"
+
+  # Downsample 3D data by a factor of 3 in each direction
+  IO::out3D_downsample_x = 3
+  IO::out3D_downsample_y = 3
+  IO::out3D_downsample_z = 3
 \end{verbatim}
 
-\item{\bf HDF5 3D Output {\tt iohdf5\_3d}}
+\item{\bf IEEEIO 2D and 3D output using {\tt IOFlexIO's "IOFlexIO\_2D"} and
+{\tt "IOFlexIO\_3D"} I/O methods}
 \begin{verbatim}
-ActiveThorns = "iohdf5 ioutil pughslab ..."
+  ActiveThorns = "IOFlexIO FlexIO IOUtil PUGHSlab ..."
 
-# Output vars using hdf5 2d method on iteration 0, 100, 200, ...
-iohdf5::out2d_every = 100            
+  # Output vars in 2D IEEEIO format on iteration 0, 100, 200, ...
+  IOFlexIO::out2D_every = 100
 
-# Output vars using hdf5 3d method on iteration 0, 5, 10, ...
-iohdf5::out3d_every = 5
+  # Output vars in 3D IEEEIO format on iteration 0, 5, 10, ...
+  IOFlexIO::out3D_every = 5
 
-# Standard I/O directory for 2D output
-io::outdir = "test"
+  # Group of variables to output to file for each method
+  IOFlexIO::out2D_vars = "evolve::vars"
+  IOFlexIO::out3D_vars = "evolve::vars"
 
-# Special I/O directory for 3D output
-iohdf5::outdir3D = "/scratch/tmp"
+  # 2D output goes into standard I/O directory
+  IO::outdir = "test"
 
-# 3D Output unchunked to one file 
-# (Only using a small number of processors)
-io::out3D_mode = "onefile"
-io::out3D_unchunked = "yes"
+  # Special I/O directory for 3D output
+  IOFlexIO::outdir3D = "/scratch/tmp"
 
-# Downsample 3D data by a factor of 3 in each direction
-io::out3D_downsample_x = 3
-io::out3D_downsample_y = 3
-io::out3D_downsample_z = 3
+  # 3D Output chunked to one file for every eight processors
+  # (Run on large number of processors)
+  IO::out3D_mode      = "np"
+  IO::out3D_nproc     = 8
+  IO::out3D_unchunked = "no"
 
-# Group of variables to output to file for each method
-iohdf5::out2D_vars = "evolve::vars"
-iohdf5::out3D_vars = "evolve::vars"
+  # Downsample 3D data by a factor of 3 in each direction
+  IO::out3D_downsample_x = 3
+  IO::out3D_downsample_y = 3
+  IO::out3D_downsample_z = 3
 \end{verbatim}
 
+\item{\bf Checkpointing using thorn {\tt IOFlexIO}}
+\begin{verbatim}
+  ActiveThorns = "IOFlexIO FlexIO IOUtil PUGHSlab ..."
+
+  # Use IEEEIO data format for checkpoint files
+  IOFlexIO::checkpoint = "yes"
+
+  # Make a new checkpoint file every 300 iterations
+  IO::checkpoint_every = 300
 
-\item{\bf IEEEIO 2D and 3D Output {\tt ioflexio\_2d}, {\tt ioflexio\_3d}}
+  # Name and directory of checkpoint file
+  IO::checkpoint_file = "run5"
+  IO::checkpoint_dir  = "/scratch/tmp"
+\end{verbatim}
+
+\item{\bf Recovering from a checkpoint file}
 \begin{verbatim}
-ActiveThorns = "ioflexio ieeeio ioutil pughslab ..."
+  ActiveThorns = "IOFlexIO FlexIO IOUtil PUGHSlab ..."
 
-# Output vars using 2D FlexIO method on iteration 0, 100, 200, ...
-ioflexio::out2d_every = 100            
+  # automatically choose the latest checkpoint file
+  IO::recover = "auto"
 
-# Output vars using 3D FlexIO method on iteration 0, 5, 10, ...
-ioflexio::out3d_every = 5
+  # Name and directory of checkpoint file to recover from
+  IO::recover_file = "run5"
+  IO::recover_dir  = "/scratch/tmp"
+\end{verbatim}
 
-# Standard I/O directory for 2D output
-io::outdir = "test"
+\end{itemize}
 
-# Special I/O directory for 3D output
-ioflexio::outdir3D = "/scratch/tmp"
 
-# 3D Output chunked to one file for every eight processors
-# (Run on large number of processors)
-io::out3D_mode = "np"
-io::out3D_nproc = 8
-io::out3D_unchunked = "no"
+\section{Providing Your own Checkpointing/Recovery Method}
 
-# Downsample 3D data by a factor of 3 in each direction
-io::out3D_downsample_x = 3
-io::out3D_downsample_y = 3
-io::out3D_downsample_z = 3
+This section is for Cactus developers to describe how they can add a new method
+for checkpointing/recovery using the existing I/O parameters and the function
+API of thorn {\tt CactusBase/IOUtil}.
+Inheriting this functionality from thorn {\tt IOUtil} helps you to reuse
+existing code and maintain a uniform user interface on how to invoke different
+checkpointing/recovery methods.
 
-# Group of variables to output to file for each method
-ioflexio::out2D_vars = "evolve::vars"
-ioflexio::out3D_vars = "evolve::vars"
-\end{verbatim}
+\subsection{Adding a Checkpointing Method}
 
-\item{\bf Checkpointing using IEEEIO}
-\begin{verbatim}
-ActiveThorns = "ioflexio ieeeio ioutil pughslab ..."
+Checkpointing is similar to performing full output of an individual grid
+variable, except that
+
+\begin{enumerate}
+  \item the output is done for all grid variables existing in a grid hierarchy
+  \item in addition to the contents of all variables, also the current setting
+    of all parameters is saved as well as some other information neccessary for
+    recovering from the checkpoint at a later time
+\end{enumerate}
 
-# Use IEEEIO data format 
-ioflexio::checkpoint = "yes"
+A thorn routine providing this checkpointing capability should register itself
+with the flesh's scheduler at the {\tt CCTK\_CPINITIAL} (for initial data
+checkpoints), {\tt CCTK\_CHECKPOINT} (for periodic checkpoints of evolution
+data), and {\tt CCTK\_TERMINATE} time bins (for checkpointing the last timestep
+of a simulation).
 
-# Make a new checkpoint file every 300 iterations
-io::checkpoint_every = 300
+It should also decide whether checkpointing is needed by evaluating the
+corresponding checkpoint parameters of {\tt IOUtil} (see section
+\ref{cp_recovery}).
 
-# Name and directory of checkpoint file
-io::checkpoint_file = "run5"
-io::checkpoint_dir  = "/scratch/tmp"
+Before dumping the contents of a distributed grid array into a checkpoint
+file the variable should be synced in case synchronization was not done before
+implicitly by the scheduler.
+
+To gather the current parameter values you can use the C routine
+\begin{verbatim}
+  char *IOUtil_GetAllParameters (const cGH *GH, int all);
 \end{verbatim}
+from thorn {\tt IOUtil}. This routine returns the parameter settings in an
+allocated single large string. Its second argument {\tt all} flags whether all
+parameter settings should be gathered ($!= 0$) or just the ones which have been
+set before ($== 0$). Note that you should always save all parameters in a
+checkpoint in order to reproduce the same state after recovery.
+
+As additional data neccessary for proper recovery, the following information
+must be saved in a checkpoint file:
+\begin{itemize}
+  \item the current main loop index (used by the driver as the main evolution
+    loop index)
+  \item the current CCTK iteration number ({\tt GH->cctk\_iteration})
+  \item the physical simulation time ({\tt GH->cctk\_time})
+\end{itemize}
+
+Moreover, information about the I/O mode used to create the checkpoint
+(chunked/unchunked, serial versus parallel I/O), the active thorns list, or
+this run's Cactus version ID (for compatibilty checking at recovery time) could
+be relevant to save in a checkpoint.
+
+
+\subsection{Adding a Recovery Method}
+
+Recovering from a checkpoint is a two-step operation:
+\begin{enumerate}
+  \item Right after reading the parameter file and finding out if recovery was
+    requested, all the parameters are restored from the checkpoint file
+    (overwriting any previous settings for non-steerable parameters).
+  \item After the flesh has created the grid hierarchy with all containing grid
+    variables and the driver has set up storage for these, their contents is
+    restored from the checkpoint (overwriting any previously initialized
+    contents).
+\end{enumerate}
+
+The flesh provides the special time bins {\tt CCTK\_RECOVER\_PARAMETERS} and
+{\tt CCTK\_RECOVER\_VARIABLES} for these two steps (see also the chapter on
+{\it Adding a Checkpointing/Recovery Method} in the {\it Infrastructure Thorn
+Writer's Guide} as part of the {\it Cactus User's Guide}).\\
+
+Thorn {\tt IOUtil} evaluates the recovery parameters (determines the recovery
+mode to use, construct the name(s) of the recovery file(s) etc.).
+It also controls the recovery process by invoking the recovery methods of
+other I/O thorns, one after another until one method succeeded.\\
+
+A recovery method must provide a routine with the following prototype:
 
-\item{\bf Recovering from a checkpoint file}
 \begin{verbatim}
-ActiveThorns = "ioflexio ieeeio ioutil pughslab ..."
+  int Recover (cGH *GH, const char *basefilename, int called_from);
+\end{verbatim}
+
+This routine will be invoked by {\tt IOUtil} with the following arguments:
+
+\begin{itemize}
+  \item a GH pointer refer to the grid hierarchy and its grid variables\\
+    Note that this can also be a {\tt NULL} pointer if the routine was called
+    at {\tt CP\_RECOVER\_PARAMETERS} when no grid hierarchy exists yet.
+
+  \item the basename of the checkpoint file to recover from\\
+    This name is constructed by {\tt IOUtil} from the settings of {\tt
+    IO::recovery\_dir} and {\tt IO::recover\_file}. It will also include an
+    iteration number if one of the {\tt auto} recovery modes is used.
+    The filename extension by which checkpoint files of different recovery
+    methods are distinguished must be appended explicitly.
+
+  \item a flag identifying when the routine was called\\
+    This can be one of the keywords {\tt CP\_INITIAL\_DATA, CP\_EVOLUTION\_DATA,
+    CP\_RECOVER\_PARAMETERS, CP\_RECOVER\_DATA}, or {\tt FILEREADER\_DATA}).
+    This flag tells the routine what it should do when it is being called by
+    {\tt IOUtil}. Note that {\tt IOUtil} assumes that the recovery method can
+    also be used as a filereader routine here which is essentially the same
+    as recovering (individual) grid variables from (data) files.
+\end{itemize}
+
 
-io::recover = "auto"
+To perform the first step of recovery process, a recovery method must register
+a routine with the flesh's scheduler at the {\tt CCTK\_RECOVER\_PARAMETERS}
+time bin. This routine itself should call the {\tt IOUtil} API
 
-# Name and directory of checkpoint file
-io::recover_file = "run5"
-io::recover_dir  = "/scratch/tmp"
+\begin{verbatim}
+  int IOUtil_RecoverParameters (int (*recover_fn) (cGH *GH,
+                                                   const char *basefilename,
+                                                   int called_from),
+                                const char *file_extension,
+                                const char *file_type)
 \end{verbatim}
 
+which will determine the recovery filename(s) and in turn invoke the actual
+recovery method's routine as a callback function. The arguments to pass to this
+routine are
+
+\begin{itemize}
+  \item a function pointer {\tt recover\_fn} as a reference to the recovery
+    method's actual recovery routine (as described above)
+
+  \item {\tt file\_extension} -- the filename extension for recovery files
+    which are accepted by this recovery method\\
+    When {\tt IOUtil} constructs the recovery filename and searches for
+    potential recovery files (in the {\tt auto} recovery modes) it will only
+    match filenames with the basename as given in the {\tt IO::recovery\_file}
+    parameter, appended by {\tt file\_extension}.
+
+  \item {\tt file\_type} -- the type of checkpoint files which are accepted by
+    this recovery method\\
+    This is just a descriptive string to print some info output during recovery.
 \end{itemize}
+
+The routine registered at {\tt CCTK\_RECOVER\_PARAMETERS} should return to
+the scheduler a negative value if parameter recovery failed for this recovery
+method for some reason (eg. if no appropriate recovery file was found).
+The scheduler will then continue with the next recovery method until one
+finally succeeds (a positive value is returned). If none of the available
+recovery methods were successful the flesh would stop the code.
+
+A value of zero should be returned to the scheduler to indicate that no
+recovery was requested.\\
+
+The second step during recovery --- restoring the contents of grid variables
+from the recovery file --- is invoked by thorn {\tt IOUtil} which registers
+a routine at {\tt CCTK\_RECOVER\_VARIABLES}. This routine calls all recovery
+methods which were registered before with {\tt IOUtil} via the API
+
+\begin{verbatim}
+  int IOUtil_RegisterRecover (const char *name,
+                              int (*recover_fn) (cGH *GH,
+                                                 const char *basefilename,
+                                                 int called_from));
+\end{verbatim}
+
+With this registration, all recovery method's actual recovery routines are made
+known to {\tt IOUtil}, along with a descriptive name under which they are
+registered.
+
+At {\tt CCTK\_RECOVER\_VARIABLES} thorn {\tt IOUtil} will loop over all 
+available recovery routines (passing the same arguments as for parameter
+recovery) until one succeeds (returns a positive value).
+
 \end{document}