New version of documentation for IO and Checkpointing from Thomas, still some things to add and tidy up

git-svn-id: http://svn.cactuscode.org/arrangements/CactusBase/IOUtil/trunk@75 b32723a9-ab3a-4a60-88e2-2e5d99d7c17a

1 files changed, 295 insertions, 59 deletions

diff --git a/doc/documentation.tex b/doc/documentation.tex
index 012b1b3..16ccf37 100644
--- a/doc/documentation.tex
+++ b/doc/documentation.tex
@@ -1,71 +1,307 @@
-% Thorn documentation template
 \documentclass{article}
 \begin{document}
 
 \title{IOUtil}
-\author{Gabrielle Allen, Tom Goodale, Thomas Radke}
-\date{1999}
+\author{Thomas Radke}
+\date{June 2000}
 \maketitle
 
-\abstract{Thorn IOUtil provides basic IO parameters as well as a function
-interface for registration and scheduling of recovery routines supplied by
-other IO thorns.}
-%
-\section{Purpose}
-%
-Together with the flesh, thorn IOUtil provides the infrastructure for
-implementing IO in Cactus.\\
-For that purpose IOUtil declares a set of basic parameters that are commonly
-used to control IO in general, eg. saying
-%
+\section{Introduction}
+
+This document details the thorns provided in the standard Cactus 
+distribution for the output of grid variables, and describes how to 
+set parameters for IO and checkpointing using these thorns.
+
+Input and Output of data (IO) in Cactus is provided by infrastructure thorns,
+which interact with the flesh via a fixed interface, which is described in 
+the Users Guide. 
+The standard release of Cactus contains a number of thorns 
+which provide parallel IO using a variety of data formats and methods. 
+All these provided IO methods use thorn {\tt CactusBase/IOUtil} which 
+provides general utilities for IO (such as parsing parameter strings to
+decide which variables to output), and a general set of parameters which
+are inherited by the different IO methods (such as the output directory). 
+Thorn {\tt IOUtil} by itself provides no IO methods.
+
+
+\section{IO methods in Cactus}
+
+Cactus has several IO modes for the output of grid variables in different 
+formats.
+Each IO method comes with its own parameters by which it can be customised,
+and all methods are registered with the flesh, satisfying the 
+Cactus API, allowing them to be called directly from application thorns. 
+An IO method registers itself with the flesh along with it's name, and 
+these registered names are the labels we now use to describe the various
+methods.
+
+
+\begin{table}
+\label{one}
+\begin{tabular}{|l|p{4.5cm}|l|}
+  \hline
+  {\bf IO method} & {\bf Description} & {\bf Providing Thorn}\\
+  \hline
+  {\tt Scalar} & 
+  output of scalars or grid array reductions in xgraph or gnuplot format & 
+  {\tt CactusBase/IOBasic}
+\\
+  {\tt Info}      & 
+  screen output of scalars or grid array reductions & 
+  {\tt CactusBase/IOBasic}
+\\
+&&\\
+  {\tt IOASCII\_1D} & 
+  1D line output of grid arrays in xgraph or gnuplot format & 
+  {\tt CactusBase/IOASCII}
+\\
+  {\tt IOASCII\_2D} & 
+  2D slice output of grid arrays in gnuplot format & 
+  {\tt CactusBase/IOASCII}
+\\
+&&\\
+  {\tt IOHDF5\_2D} & 
+  2D slice output of grid arrays in HDF5 format & 
+  {\tt CactusPUGHIO/IOHDF5}
+\\
+  {\tt IOHDF5\_3D} & 
+  full 3D output of grid arrays in HDF5 format & 
+  {\tt CactusPUGHIO/IOHDF5}
+\\
+&&\\
+  {\tt IOFlexIO\_2D} & 
+  2D slice output of grid arrays in FlexIO format & 
+  {\tt CactusPUGHIO/IOFlexIO}
+\\
+  {\tt IOFlexIO\_3D} & 
+  full 3D output of grid arrays in HDF5 format & 
+  {\tt CactusPUGHIO/IOFlexIO}
+\\
+  \hline
+\end{tabular}
+\caption{Standard IO methods provided with the Cactus distribution}
+\end{table}
+
+The standard provided Cactus IO methods are shown in Table~\ref{one}.
+As described above, each of these IO thorns inherit parameters from thorn 
+{\tt CactusBase/IOUtil}, which must be included in your ThornList
+and activated in your parameter files before any of these IO methods 
+can be used. {\tt CactusBase/IOUtil} allows you to set the default 
+behaviour for all the IO methods described above, for examples, setting
+the parameter {\tt IO::out\_every=1} will result in any chosen IO method
+providing output on each iteration. The default behaviour can be overriden
+by specific parameters for each method.
+For example, you may want scalar and 1D output at every iteration
+but computationally expensive 3D output only every 10th iteration, 
+with these files going into another directory on a scratch partition.
+
+\section{Standard Parameters}
+
+Here we describe a few of the standard parameters used by {\tt IOUtil}
+to control output. 
 \begin{itemize}
-  \item when to do output
-  \item where output files should be put
-  \item what output modes should be used, etc.
+
+\item{\tt IO::outdir}\\
+  The name of the directory to be used for output. All the IO 
+  methods described here will write by default to this directory. 
+  Individual methods have parameters
+which can direct their output to a different directory.
+
+\item{\tt IO::out\_every}\\
+  How often, in terms of iterations, each of the Cactus IO methods will
+  write output. Again, individual methods can set there own parameter 
+  to override this. The default is to never write output.
 \end{itemize}
-%
-These parameters should be used by other IO thorns
-to define the default behaviour for the IO methods they implement.
-Of course, those thorns can also declare their own parameters in addition
-to the general ones from IOUtil, giving the user a chance to override the
-defaults and provide some specific control over individual IO methods.\\
-%
-\newline
-{\bf Example}\\
-You want to output some grid function in 1D, 2D, and 3D format at
-every iteration. Simply tell the different IO methods what they should
-output, and set the single IOUtil parameter {\tt IO::out\_every = 1}. This saves
-you setting the specific {\tt out\_every} parameter for each of the 3 formats.\\
-If you want to restrict 3D output to only every 10th iteration (because it takes
-too long to do it at every iteration) you just change the {\tt out\_every}
-parameter there -- and you're done.\\
-%
-\newline
-IOUtil also provides a skeleton for implementing checkpointing/recovery.
-There are parameters to
-%
+
+
+\section{IO modes}
+\label{iomodes}
+For a run on multiple processors, scalar, 1D, and 2D output will always be 
+written from only processor zero (that is, required data from all other
+processors will be sent to processor zero, who then outputs all the gathered
+data).
+For 3D output this may become a quite expensive operation since output
+by only a single processor will probably result in a IO bottleneck and delay
+further computation. 
+For this reason Cactus offers different IO modes for 3D output which can be
+controlled by the {\tt IO::out\_3Dmode} parameter, in combination with
+ {\tt IO::out\_unchunked} and {\tt IO::out3D\_np}. These parameters allow
+IO to be optimised for your particular machine architecture and needs: 
+
 \begin{itemize}
-  \item enable/disable checkpointing \& recovery
-  \item specify how often checkpointing should take place
-  \item tell where checkpoint files should go to or can be found 
+  \item {\tt IO::out3D\_mode = "onefile"}\\
+    As for the 1D and 2D IO methods, writing to file is performed only 
+    by processor zero.
+    This processor gathers all the output data from the other processors
+    and writes to a single file. The gathered grid array data from each
+    processor can be either
+    written in chunks ({\tt IO::out3D\_unchunked = "no"}) with each chunk
+    containing the data from a single processor, or collected into a
+    single global array before writing ({\tt IO::out3D\_unchunked = "yes"}).
+    The default is to write the data in chunks.
+  \item {\tt IO::out3D\_mode = "np"}\\
+    3D output is written in parallel for groups of processors. Each group
+    consists of {\tt IO::out3D\_procs} processors which have assigned one
+    IO processor which writes the gathers data from the group and writes it
+    to file. The chunked output will go into {\tt IO::out3D\_procs} files. 
+    The default number of processors in a group is eight.
+  \item {\tt IO::out3D\_mode = "proc"}\\
+    This is the default method for 3D output.
+    Every processor writes its own chunk of data into a separate output file.
 \end{itemize}
-%
-Like with IO, checkpointing/recovery is implemented by other IO thorns. These
-register their checkpoint routines with the RFR if checkpointing was
-explicitely enabled for them.\\
-To recover from a checkpoint file you normally would have to know what data
-format the file is, and which IO thorn can process that format. To free the user
-from this burden, thorn IOUtil provides its own generic recovery routine and
-a function interface to register other IO thorns' recovery routines.
-IOUtil's generic routine will then just loop over all registered recovery
-routines (passing it the name of the recovery file) until one of them succeeded
-to process it.
-%
-%
-% Automatically created from the ccl files 
-% Do not worry for now.
-\include{interface}
-\include{param}
-\include{schedule}
 
+Probably the single processor {\tt proc} mode is the most efficient 3D output mode on 
+machines with a fast IO subsystem and many IO nodes (eg. an Origin 2000) 
+because it provides the highest
+parallelity for outputting data. Note that on very large numbers of processors
+you mave have to fall back to {\tt np}, output from groups of processors, 
+mode if the system limit of
+maximum open file descriptors is exceeded (this is true for large jobs on a
+ T3E).\\
+While the {\tt np} and {\tt proc} IO modes are fast for outputting large
+amounts of data from all or a group of processors in parallel, they have
+the disadvantage of writing chunked files. These files have to be 
+{\tt recombined}
+during a postprocessing phase so that the final unchunked data can be
+visualized by standard tools. For that purpose a recombiner utility program is
+provided by thorn {\tt CactusPUGHIO/IOFlexIO}.
+Note that {\tt CactusPUGHIO/IOHDF5} does not provide such a utility yet. 
+
+\section{Checkpointing and Recovery in Cactus}
+
+The IO methods for 3D output also provide functionality for {\it checkpointing}
+and {\it recovery}. A checkpoint is a snapshot of the current state of the 
+simulation ({\it ie} the contents of all the grid variables and the parameter 
+settings) at a chosen timestep. Each checkpoint is saved into a 
+{\it checkpoint file} which can be used to restart a new simulation at a later
+time, recreating the exact state at which it was checkpointed.\\
+Checkpointing is especially useful when running Cactus in batch queue systems 
+where jobs get only limited CPU time. A more advanced use of checkpointing 
+would be to restart your simulation after a crash or a problem had developed,
+using a  different parameter set recovering from the latest stable timestep. 
+Additionally, for performing parameter studies,  compute-intensive
+initial data can be calculated just once and saved in a checkpoint file 
+from which each job can be started. \\
+
+Again, thorn {\tt CactusBase/IOUtil} provides general checkpoint \& recovery
+parameters. The most important ones are:
+\begin{itemize}
+  \item {\tt IO::checkpoint\_every}\\
+    specifies how often to checkpoint in terms of iteration number
+  \item {\tt IO::checkpoint\_file}\\
+    holds the checkpoint filename
+  \item {\tt IO::checkpoint\_dir}\\
+    names the directory where checkpoint files are stored
+  \item {\tt IO::checkpoint\_ID}\\
+    boolean flag for checkpointing only initial data
+  \item {\tt IO::recover}\\
+    keyword parameter telling if/how to recover.\\
+    Choices are {\tt "auto"}, {\tt "manual"}, and {\tt "no"}.
+  \item {\tt IO::recover\_file}\\
+    filename of the recovery file
+  \item {\tt IO::recovery\_dir}\\
+    directory where the recovery file is located
+\end{itemize}
+
+To checkpoint your simulation, you need to enable checkpointing by setting
+the Boolean parameter {\tt checkpoint}, for one of the 3D IO methods to
+{\tt yes}.
+Checkpoint filenames consist of a basename (as specified in
+{\tt IO::checkpoint\_file}) followed by {\tt ".chkpt.it\_$<$iteration\_number$>$"}
+plus the file extension indicating the file format ({\tt ".ieee"} for IEEEIO
+data from {\tt CactusPUGHIO/IOFlexIO}, or {\tt ".h5} for HDF5 data from
+{\tt CactusPUGHIO/IOHDF5}).\\
+The {\tt "auto"} recovery mode will automatically
+recover from the latest checkpoint file found in the recovery directory.
+In this case {\tt IO::recover\_file} should contain the basename only.
+Use the {\tt "manual"} mode to recover from a specific checkpoint file
+by adding the iteration number to the basename parameter.\\
+Because the same IO methods implement both output of 3D data and
+checkpoint files, the same IO modes are used (see Section~\ref{iomodes}). 
+Note that the recovery routines in Cactus can process both chunked and 
+unchunked checkpoint files if you restart on the same number of processors.
+That's why you should always use one of the parallel IO modes for 
+checkpointing.
+
+<CAN RESTART ON DIFFERENT NUMBER OF PROCESSORS OR DIFFERENT MACHINES>
+
+\section{Data Filename Extensions}
+
+\begin{table}
+\label{two}
+\begin{tabular}{|c|p{4.5cm}|l|}
+\hline
+{\bf Extension} & {\bf Description} & {\bf Thorn} \\
+ {\tt .xl}
+ &
+ &
+ {\tt CactusBase/IOASCII}
+\\
+ {\tt .yl}
+ &
+ &
+ {\tt CactusBase/IOASCII}
+\\
+{\tt  .zl}
+ &
+ &
+ {\tt CactusBase/IOASCII}
+\\
+{\tt .dl}	
+ &
+ &
+ {\tt CactusBase/IOASCII}
+\\
+{\tt .tl}
+ &
+ &
+ {\tt CactusBase/IOASCII}
+\\ \hline 
+\end{tabular}
+\caption{File extensions used by the standard IO thorns in the Cactus distribution}
+\end{table}
+
+ 
+\section{Example Parameter Files}
+
+Here we give examples of the parameters for the different IO methods.
+
+\begin{itemize}
+\item{Screen Information {\tt info}}
+\begin{verbatim}
+ActiveThorns = ``iobasic ioutil ...''
+
+# Output using all methods on iteration 0, 10, 20, ...
+io::out_every = 10            
+
+# Group of variables to output to screen
+iobasic::outinfo_vars = ``evolve::vars''
+\end{verbatim}
+
+\item{Scalar Output {\tt scalar}}
+\begin{verbatim}
+ActiveThorns = ``iobasic ioutil ...''
+
+# Output vars using scalar method on iteration 0, 10, 20, ...
+iobasic::outscalar_every = 10            
+
+# Group of variables to output to file
+iobasic::outscalar_vars = ``evolve::vars''
+\end{verbatim}
+
+\item{ASCII 1D and 2D Output {\tt ioascii\_1d}, {\tt ioascii\_2d}}
+\begin{verbatim}
+ActiveThorns = ``ioascii ioutil hyperslab ...''
+
+# Output vars using ascii 1d method on iteration 0, 10, 20, ...
+ioascii::out1d_every = 10            
+
+# Output vars using ascii 2d method on iteration 0, 50, 100, ...
+ioascii::out1d_every = 100
+
+# Group of variables to output to file for each method
+iobasic::outscalar_vars = ``evolve::vars''
+\end{verbatim}
+
+\end{itemize}
 \end{document}