\documentclass{article}
\begin{document}

\title{IOUtil}
\author{Thomas Radke, Gabrielle Allen}
\date{June 2000}
\maketitle

\section{Introduction}

This document details the thorns provided in the standard Cactus 
distribution for the output of grid variables, and describes how to 
set parameters for IO and checkpointing using these thorns.

Input and Output of data (IO) in Cactus is provided by infrastructure thorns,
which interact with the flesh via a fixed interface, which is described in 
the Users Guide. 
The standard release of Cactus contains a number of thorns 
which provide parallel IO using a variety of data formats and methods. 
All these provided IO methods use thorn {\tt CactusBase/IOUtil} which 
provides general utilities for IO (such as parsing parameter strings to
decide which variables to output), and a general set of parameters which
are inherited by the different IO methods (such as the output directory). 
Thorn {\tt IOUtil} by itself provides no IO methods.

More information about IO and Visualisation of Cactus data can be found in 
the individual IO thorns, and in the {\tt Visualization-HOWTO} available 
on the Cactus web pages.

\section{IO methods in Cactus}

Cactus has several IO modes for the output of grid variables in different 
formats.
Each IO method comes with its own parameters by which it can be customised,
and all methods are registered with the flesh, satisfying the 
Cactus API, allowing them to be called directly from application thorns. 
An IO method registers itself with the flesh along with it's name, and 
these registered names are the labels we now use to describe the various
methods. 


\begin{table}
\label{one}
\begin{tabular}{|l|p{4.5cm}|l|}
  \hline
  {\bf IO method} & {\bf Description} & {\bf Providing Thorn}\\
  \hline
  {\tt Scalar} & 
  output of scalars or grid array reductions in xgraph or gnuplot format & 
  {\tt CactusBase/IOBasic}
\\
  {\tt Info}      & 
  screen output of scalars or grid array reductions & 
  {\tt CactusBase/IOBasic}
\\
&&\\
  {\tt IOASCII\_1D} & 
  1D line output of grid arrays in xgraph or gnuplot format & 
  {\tt CactusBase/IOASCII}
\\
  {\tt IOASCII\_2D} & 
  2D slice output of grid arrays in gnuplot format & 
  {\tt CactusBase/IOASCII}
\\
&&\\
  {\tt IOHDF5\_2D} & 
  2D slice output of grid arrays in HDF5 format & 
  {\tt CactusPUGHIO/IOHDF5}
\\
  {\tt IOHDF5\_3D} & 
  full 3D output of grid arrays in HDF5 format & 
  {\tt CactusPUGHIO/IOHDF5}
\\
&&\\
  {\tt IOFlexIO\_2D} & 
  2D slice output of grid arrays in FlexIO format & 
  {\tt CactusPUGHIO/IOFlexIO}
\\
  {\tt IOFlexIO\_3D} & 
  full 3D output of grid arrays in HDF5 format & 
  {\tt CactusPUGHIO/IOFlexIO}
\\
  \hline
\end{tabular}
\caption{Standard IO methods provided with the Cactus distribution}
\end{table}

The standard provided Cactus IO methods are shown in Table~\ref{one}.
As described above, each of these IO thorns inherit parameters from thorn 
{\tt CactusBase/IOUtil}, which must be included in your ThornList
and activated in your parameter files before any of these IO methods 
can be used. {\tt CactusBase/IOUtil} allows you to set the default 
behaviour for all the IO methods described above, for examples, setting
the parameter {\tt IO::out\_every=1} will result in any chosen IO method
providing output on each iteration. The default behaviour can be overriden
by specific parameters for each method.
For example, you may want scalar and 1D output at every iteration
but computationally expensive 3D output only every 10th iteration, 
with these files going into another directory on a scratch partition.

\section{External Packages}

NOTE: This section will not be filled out properly until the new treatment of external 
packages is implemented.

Some of the IO methods used by Cactus require that external packages are installed and available 
on your machine. 

\begin{itemize}

\item{\tt HDF5} {\bf Hierachical Data Format}\\
{\tt http://hdf.ncsa.uiuc.edu/}

\item{\tt IEEEIO} \\
{\tt http://zeus.ncsa.uiuc.edu/}

\end{itemize}

\section{Standard Parameters}

Here we describe a few of the standard parameters used by {\tt IOUtil}
to control output. 
\begin{itemize}

\item{\tt IO::outdir}\\
  The name of the directory to be used for output. All the IO 
  methods described here will write by default to this directory. 
  Individual methods have parameters
which can direct their output to a different directory.

\item{\tt IO::out\_every}\\
  How often, in terms of iterations, each of the Cactus IO methods will
  write output. Again, individual methods can set there own parameter 
  to override this. The default is to never write output.
\end{itemize}


\section{IO modes}
\label{iomodes}
For a run on multiple processors, scalar, 1D, and 2D output will always be 
written from only processor zero (that is, required data from all other
processors will be sent to processor zero, who then outputs all the gathered
data).
For 3D output this may become a quite expensive operation since output
by only a single processor will probably result in a IO bottleneck and delay
further computation. 
For this reason Cactus offers different IO modes for 3D output which can be
controlled by the {\tt IO::out\_3Dmode} parameter, in combination with
 {\tt IO::out\_unchunked} and {\tt IO::out3D\_np}. These parameters allow
IO to be optimised for your particular machine architecture and needs: 

\begin{itemize}
  \item {\tt IO::out3D\_mode = "onefile"}\\
    As for the 1D and 2D IO methods, writing to file is performed only 
    by processor zero.
    This processor gathers all the output data from the other processors
    and writes to a single file. The gathered grid array data from each
    processor can be either
    written in chunks ({\tt IO::out3D\_unchunked = "no"}) with each chunk
    containing the data from a single processor, or collected into a
    single global array before writing ({\tt IO::out3D\_unchunked = "yes"}).
    The default is to write the data in chunks.
  \item {\tt IO::out3D\_mode = "np"}\\
    3D output is written in parallel for groups of processors. Each group
    consists of {\tt IO::out3D\_procs} processors which have assigned one
    IO processor which writes the gathers data from the group and writes it
    to file. The chunked output will go into {\tt IO::out3D\_procs} files. 
    The default number of processors in a group is eight.
  \item {\tt IO::out3D\_mode = "proc"}\\
    This is the default method for 3D output.
    Every processor writes its own chunk of data into a separate output file.
\end{itemize}

Probably the single processor {\tt proc} mode is the most efficient 3D output mode on 
machines with a fast IO subsystem and many IO nodes (eg. an Origin 2000) 
because it provides the highest
parallelity for outputting data. Note that on very large numbers of processors
you mave have to fall back to {\tt np}, output from groups of processors, 
mode if the system limit of
maximum open file descriptors is exceeded (this is true for large jobs on a
 T3E).\\
While the {\tt np} and {\tt proc} IO modes are fast for outputting large
amounts of data from all or a group of processors in parallel, they have
the disadvantage of writing chunked files. These files have to be 
{\tt recombined}
during a postprocessing phase so that the final unchunked data can be
visualized by standard tools. For that purpose a recombiner utility program is
provided by thorn {\tt CactusPUGHIO/IOFlexIO}.
Note that {\tt CactusPUGHIO/IOHDF5} does not provide such a utility yet. 

\section{Checkpointing and Recovery in Cactus}

The IO methods for 3D output also provide functionality for {\it checkpointing}
and {\it recovery}. A checkpoint is a snapshot of the current state of the 
simulation ({\it ie} the contents of all the grid variables and the parameter 
settings) at a chosen timestep. Each checkpoint is saved into a 
{\it checkpoint file} which can be used to restart a new simulation at a later
time, recreating the exact state at which it was checkpointed.\\
Checkpointing is especially useful when running Cactus in batch queue systems 
where jobs get only limited CPU time. A more advanced use of checkpointing 
would be to restart your simulation after a crash or a problem had developed,
using a  different parameter set recovering from the latest stable timestep. 
Additionally, for performing parameter studies,  compute-intensive
initial data can be calculated just once and saved in a checkpoint file 
from which each job can be started. \\

Again, thorn {\tt CactusBase/IOUtil} provides general checkpoint \& recovery
parameters. The most important ones are:
\begin{itemize}
  \item {\tt IO::checkpoint\_every}\\
    specifies how often to checkpoint in terms of iteration number
  \item {\tt IO::checkpoint\_file}\\
    holds the checkpoint filename
  \item {\tt IO::checkpoint\_dir}\\
    names the directory where checkpoint files are stored
  \item {\tt IO::checkpoint\_ID}\\
    boolean flag for checkpointing only initial data
  \item {\tt IO::recover}\\
    keyword parameter telling if/how to recover.\\
    Choices are {\tt "auto"}, {\tt "manual"}, and {\tt "no"}.
  \item {\tt IO::recover\_file}\\
    filename of the recovery file
  \item {\tt IO::recovery\_dir}\\
    directory where the recovery file is located
\end{itemize}

To checkpoint your simulation, you need to enable checkpointing by setting
the Boolean parameter {\tt checkpoint}, for one of the 3D IO methods to
{\tt yes}.
Checkpoint filenames consist of a basename (as specified in
{\tt IO::checkpoint\_file}) followed by {\tt ".chkpt.it\_$<$iteration\_number$>$"}
plus the file extension indicating the file format ({\tt ".ieee"} for IEEEIO
data from {\tt CactusPUGHIO/IOFlexIO}, or {\tt ".h5} for HDF5 data from
{\tt CactusPUGHIO/IOHDF5}).\\
The {\tt "auto"} recovery mode will automatically
recover from the latest checkpoint file found in the recovery directory.
In this case {\tt IO::recover\_file} should contain the basename only.
Use the {\tt "manual"} mode to recover from a specific checkpoint file
by adding the iteration number to the basename parameter.\\
Because the same IO methods implement both output of 3D data and
checkpoint files, the same IO modes are used (see Section~\ref{iomodes}). 
Note that the recovery routines in Cactus can process both chunked and 
unchunked checkpoint files if you restart on the same number of processors.
That's why you should always use one of the parallel IO modes for 
checkpointing. If you want to restart on a different number of processors, you first need to 
{\it recombine} the data in the checkpoint file(s) to create a single file with
unchunked data.
Note that you can restart from your checkpoint file on a different machine.

\vskip .5cm
\noindent{\bf WARNING:}

\noindent
Checkpointing and recovery should {\bf always} be tested for a new thorn set. This is because
only Cactus grid variables and parameters are saved in a checkpoint file. If a thorn has made 
use of saved local
variables, the state of a recovered simulation may differ from the original run. To test 
checkpointing and recovery, simply perform one run of say 10 timesteps, and compare output data
with a checkpointed and recovered run at say the 5th timestep. The output data should match 
exactly if recovery was successful.

\section{Data Filenames and Filename Extensions}

The standard IO thorns in Cactus make use of a consistent set of filenames and extensions, which 
identify the variables and data format used in the file. The filenames are listed in Table~\ref{two},
and the extensions in Table~\ref{three}

\begin{center}
\begin{table}
\label{two}
\begin{tabular}{|l|l|}
  \hline
  {\bf IO method} & {\bf Filename for output of variable {\tt phi}}\\
  \hline
  {\tt Scalar} & 
phi\_nm1.tl, phi\_nm2.tl phi\_max.tl phi\_min.tl
\\
  {\tt Info}      & 
Provides no output file (the same data is available with {\tt Scalar} output)
\\
  {\tt IOASCII\_1D} & 
phi.xl, phi.yl, phi.zl, phi.tl
\\
  {\tt IOASCII\_2D} & 
??
\\
  {\tt IOHDF5\_2D} & 
phi2d.h5
\\
  {\tt IOHDF5\_3D} & 
phi3d.h5
\\
  {\tt IOFlexIO\_2D} & 
phi2d.ieee
\\
  {\tt IOFlexIO\_3D} & 
phi3d.ieee
\\
  \hline
\end{tabular}
\caption{Filenames used by standard IO thorns}
\end{table}
\end{center}
\begin{center}
\begin{table}
\label{three}
\begin{tabular}{|c|p{4.5cm}|l|}
\hline
{\bf Extension} & {\bf Description} & {\bf Thorn} \\
 {\tt .xl}
 &
 &
 {\tt CactusBase/IOASCII}
\\
 {\tt .yl}
 &
 &
 {\tt CactusBase/IOASCII}
\\
{\tt  .zl}
 &
 &
 {\tt CactusBase/IOASCII}
\\
{\tt .dl}	
 &
 &
 {\tt CactusBase/IOASCII}
\\
{\tt .tl}
 &
 &
 {\tt CactusBase/IOASCII}
\\ \hline 
\end{tabular}
\caption{File extensions used by the standard IO thorns in the Cactus distribution}
\end{table}
\end{center}
 
\section{Example Parameter Files}

Here we give examples of the parameters for the different IO methods.

\begin{itemize}
\item{\bf Screen Information {\tt info}}
\begin{verbatim}
ActiveThorns = "iobasic ioutil ..."

# Output using all methods on iteration 0, 10, 20, ...
io::out_every = 10            

# Group of variables to output to screen
iobasic::outinfo_vars = "evolve::vars"
\end{verbatim}

\item{\bf Scalar Output {\tt scalar}}
\begin{verbatim}
ActiveThorns = "iobasic ioutil ..."

# Output vars using scalar method on iteration 0, 10, 20, ...
iobasic::outscalar_every = 10            

# Group of variables to output to file
iobasic::outscalar_vars = "evolve::vars"
\end{verbatim}

\item{\bf ASCII 1D and 2D Output {\tt ioascii\_1d}, {\tt ioascii\_2d}}
\begin{verbatim}
ActiveThorns = "ioascii ioutil hyperslab ..."

# Output vars using ascii 1d method on iteration 0, 10, 20, ...
ioascii::out1d_every = 10            

# Output vars using ascii 2d method on iteration 0, 50, 100, ...
ioascii::out1d_every = 100

# Group of variables to output to file for each method
iobasic::outscalar_vars = "evolve::vars"
\end{verbatim}

\item{\bf HDF5 2D and 3D Output {\tt iohdf5\_2d}, {\tt iohdf5\_3d}}
\begin{verbatim}
ActiveThorns = "iohdf5 ioutil hyperslab ..."

# Output vars using ascii 2d method on iteration 0, 100, 200, ...
iohdf5::out2d_every = 100            

# Output vars using ascii 3d method on iteration 0, 5, 10, ...
iohdf5::out3d_every = 5

# Standard IO directory for 2D output
io::outdir = "test"

# Special IO directory for 3D output
iohdf5::outdir3D = "/scratch/tmp"

# 3D Output unchunked to one file 
# (Only using a small number of processors)
io::out3D_mode = "onefile"
io::out3D_unchunked = "yes"

# Downsample 3D data by a factor of 3 in each direction
io::out3D_downsample_x = 3
io::out3D_downsample_y = 3
io::out3D_downsample_z = 3

# Group of variables to output to file for each method
iohdf5::out2D_vars = "evolve::vars"
iohdf5::out3D_vars = "evolve::vars"
\end{verbatim}


\item{\bf IEEEIO 2D and 3D Output {\tt ioflexio\_2d}, {\tt ioflexio\_3d}}
\begin{verbatim}
ActiveThorns = "ioflexio ieeeio ioutil hyperslab ..."

# Output vars using 2D FlexIO method on iteration 0, 100, 200, ...
ioflexio::out2d_every = 100            

# Output vars using 3D FlexIO method on iteration 0, 5, 10, ...
ioflexio::out3d_every = 5

# Standard IO directory for 2D output
io::outdir = "test"

# Special IO directory for 3D output
ioflexio::outdir3D = "/scratch/tmp"

# 3D Output chunked to one file for every eight processors
# (Run on large number of processors)
io::out3D_mode = "np"
io::out3D_nproc = 8
io::out3D_unchunked = "no"

# Downsample 3D data by a factor of 3 in each direction
io::out3D_downsample_x = 3
io::out3D_downsample_y = 3
io::out3D_downsample_z = 3

# Group of variables to output to file for each method
ioflexio::out2D_vars = "evolve::vars"
ioflexio::out3D_vars = "evolve::vars"
\end{verbatim}

\item{\bf Checkpointing using IEEEIO}
\begin{verbatim}
ActiveThorns = "ioflexio ieeeio ioutil hyperslab ..."

# Use IEEEIO data format 
ioflexio::checkpoint = "yes"

# Make a new checkpoint file every 300 iterations
io::checkpoint_every = 300

# Name and directory of checkpoint file
io::checkpoint_file = "run5"
io::checkpoint_dir  = "/scratch/tmp"
\end{verbatim}

\item{\bf Recovering from a checkpoint file}
\begin{verbatim}
ActiveThorns = "ioflexio ieeeio ioutil hyperslab ..."

io::recover = "auto"

# Name and directory of checkpoint file
io::recover_file = "run5"
io::recover_dir  = "/scratch/tmp"
\end{verbatim}

\end{itemize}
\end{document}