From 7608b9fcc86d225130fecf890aecd71a13b667de Mon Sep 17 00:00:00 2001 From: allen Date: Fri, 30 Jun 2000 12:48:07 +0000 Subject: New version of documentation for IO and Checkpointing from Thomas, still some things to add and tidy up git-svn-id: http://svn.cactuscode.org/arrangements/CactusBase/IOUtil/trunk@75 b32723a9-ab3a-4a60-88e2-2e5d99d7c17a --- doc/documentation.tex | 354 +++++++++++++++++++++++++++++++++++++++++--------- 1 file changed, 295 insertions(+), 59 deletions(-) (limited to 'doc') diff --git a/doc/documentation.tex b/doc/documentation.tex index 012b1b3..16ccf37 100644 --- a/doc/documentation.tex +++ b/doc/documentation.tex @@ -1,71 +1,307 @@ -% Thorn documentation template \documentclass{article} \begin{document} \title{IOUtil} -\author{Gabrielle Allen, Tom Goodale, Thomas Radke} -\date{1999} +\author{Thomas Radke} +\date{June 2000} \maketitle -\abstract{Thorn IOUtil provides basic IO parameters as well as a function -interface for registration and scheduling of recovery routines supplied by -other IO thorns.} -% -\section{Purpose} -% -Together with the flesh, thorn IOUtil provides the infrastructure for -implementing IO in Cactus.\\ -For that purpose IOUtil declares a set of basic parameters that are commonly -used to control IO in general, eg. saying -% +\section{Introduction} + +This document details the thorns provided in the standard Cactus +distribution for the output of grid variables, and describes how to +set parameters for IO and checkpointing using these thorns. + +Input and Output of data (IO) in Cactus is provided by infrastructure thorns, +which interact with the flesh via a fixed interface, which is described in +the Users Guide. +The standard release of Cactus contains a number of thorns +which provide parallel IO using a variety of data formats and methods. +All these provided IO methods use thorn {\tt CactusBase/IOUtil} which +provides general utilities for IO (such as parsing parameter strings to +decide which variables to output), and a general set of parameters which +are inherited by the different IO methods (such as the output directory). +Thorn {\tt IOUtil} by itself provides no IO methods. + + +\section{IO methods in Cactus} + +Cactus has several IO modes for the output of grid variables in different +formats. +Each IO method comes with its own parameters by which it can be customised, +and all methods are registered with the flesh, satisfying the +Cactus API, allowing them to be called directly from application thorns. +An IO method registers itself with the flesh along with it's name, and +these registered names are the labels we now use to describe the various +methods. + + +\begin{table} +\label{one} +\begin{tabular}{|l|p{4.5cm}|l|} + \hline + {\bf IO method} & {\bf Description} & {\bf Providing Thorn}\\ + \hline + {\tt Scalar} & + output of scalars or grid array reductions in xgraph or gnuplot format & + {\tt CactusBase/IOBasic} +\\ + {\tt Info} & + screen output of scalars or grid array reductions & + {\tt CactusBase/IOBasic} +\\ +&&\\ + {\tt IOASCII\_1D} & + 1D line output of grid arrays in xgraph or gnuplot format & + {\tt CactusBase/IOASCII} +\\ + {\tt IOASCII\_2D} & + 2D slice output of grid arrays in gnuplot format & + {\tt CactusBase/IOASCII} +\\ +&&\\ + {\tt IOHDF5\_2D} & + 2D slice output of grid arrays in HDF5 format & + {\tt CactusPUGHIO/IOHDF5} +\\ + {\tt IOHDF5\_3D} & + full 3D output of grid arrays in HDF5 format & + {\tt CactusPUGHIO/IOHDF5} +\\ +&&\\ + {\tt IOFlexIO\_2D} & + 2D slice output of grid arrays in FlexIO format & + {\tt CactusPUGHIO/IOFlexIO} +\\ + {\tt IOFlexIO\_3D} & + full 3D output of grid arrays in HDF5 format & + {\tt CactusPUGHIO/IOFlexIO} +\\ + \hline +\end{tabular} +\caption{Standard IO methods provided with the Cactus distribution} +\end{table} + +The standard provided Cactus IO methods are shown in Table~\ref{one}. +As described above, each of these IO thorns inherit parameters from thorn +{\tt CactusBase/IOUtil}, which must be included in your ThornList +and activated in your parameter files before any of these IO methods +can be used. {\tt CactusBase/IOUtil} allows you to set the default +behaviour for all the IO methods described above, for examples, setting +the parameter {\tt IO::out\_every=1} will result in any chosen IO method +providing output on each iteration. The default behaviour can be overriden +by specific parameters for each method. +For example, you may want scalar and 1D output at every iteration +but computationally expensive 3D output only every 10th iteration, +with these files going into another directory on a scratch partition. + +\section{Standard Parameters} + +Here we describe a few of the standard parameters used by {\tt IOUtil} +to control output. \begin{itemize} - \item when to do output - \item where output files should be put - \item what output modes should be used, etc. + +\item{\tt IO::outdir}\\ + The name of the directory to be used for output. All the IO + methods described here will write by default to this directory. + Individual methods have parameters +which can direct their output to a different directory. + +\item{\tt IO::out\_every}\\ + How often, in terms of iterations, each of the Cactus IO methods will + write output. Again, individual methods can set there own parameter + to override this. The default is to never write output. \end{itemize} -% -These parameters should be used by other IO thorns -to define the default behaviour for the IO methods they implement. -Of course, those thorns can also declare their own parameters in addition -to the general ones from IOUtil, giving the user a chance to override the -defaults and provide some specific control over individual IO methods.\\ -% -\newline -{\bf Example}\\ -You want to output some grid function in 1D, 2D, and 3D format at -every iteration. Simply tell the different IO methods what they should -output, and set the single IOUtil parameter {\tt IO::out\_every = 1}. This saves -you setting the specific {\tt out\_every} parameter for each of the 3 formats.\\ -If you want to restrict 3D output to only every 10th iteration (because it takes -too long to do it at every iteration) you just change the {\tt out\_every} -parameter there -- and you're done.\\ -% -\newline -IOUtil also provides a skeleton for implementing checkpointing/recovery. -There are parameters to -% + + +\section{IO modes} +\label{iomodes} +For a run on multiple processors, scalar, 1D, and 2D output will always be +written from only processor zero (that is, required data from all other +processors will be sent to processor zero, who then outputs all the gathered +data). +For 3D output this may become a quite expensive operation since output +by only a single processor will probably result in a IO bottleneck and delay +further computation. +For this reason Cactus offers different IO modes for 3D output which can be +controlled by the {\tt IO::out\_3Dmode} parameter, in combination with + {\tt IO::out\_unchunked} and {\tt IO::out3D\_np}. These parameters allow +IO to be optimised for your particular machine architecture and needs: + \begin{itemize} - \item enable/disable checkpointing \& recovery - \item specify how often checkpointing should take place - \item tell where checkpoint files should go to or can be found + \item {\tt IO::out3D\_mode = "onefile"}\\ + As for the 1D and 2D IO methods, writing to file is performed only + by processor zero. + This processor gathers all the output data from the other processors + and writes to a single file. The gathered grid array data from each + processor can be either + written in chunks ({\tt IO::out3D\_unchunked = "no"}) with each chunk + containing the data from a single processor, or collected into a + single global array before writing ({\tt IO::out3D\_unchunked = "yes"}). + The default is to write the data in chunks. + \item {\tt IO::out3D\_mode = "np"}\\ + 3D output is written in parallel for groups of processors. Each group + consists of {\tt IO::out3D\_procs} processors which have assigned one + IO processor which writes the gathers data from the group and writes it + to file. The chunked output will go into {\tt IO::out3D\_procs} files. + The default number of processors in a group is eight. + \item {\tt IO::out3D\_mode = "proc"}\\ + This is the default method for 3D output. + Every processor writes its own chunk of data into a separate output file. \end{itemize} -% -Like with IO, checkpointing/recovery is implemented by other IO thorns. These -register their checkpoint routines with the RFR if checkpointing was -explicitely enabled for them.\\ -To recover from a checkpoint file you normally would have to know what data -format the file is, and which IO thorn can process that format. To free the user -from this burden, thorn IOUtil provides its own generic recovery routine and -a function interface to register other IO thorns' recovery routines. -IOUtil's generic routine will then just loop over all registered recovery -routines (passing it the name of the recovery file) until one of them succeeded -to process it. -% -% -% Automatically created from the ccl files -% Do not worry for now. -\include{interface} -\include{param} -\include{schedule} +Probably the single processor {\tt proc} mode is the most efficient 3D output mode on +machines with a fast IO subsystem and many IO nodes (eg. an Origin 2000) +because it provides the highest +parallelity for outputting data. Note that on very large numbers of processors +you mave have to fall back to {\tt np}, output from groups of processors, +mode if the system limit of +maximum open file descriptors is exceeded (this is true for large jobs on a + T3E).\\ +While the {\tt np} and {\tt proc} IO modes are fast for outputting large +amounts of data from all or a group of processors in parallel, they have +the disadvantage of writing chunked files. These files have to be +{\tt recombined} +during a postprocessing phase so that the final unchunked data can be +visualized by standard tools. For that purpose a recombiner utility program is +provided by thorn {\tt CactusPUGHIO/IOFlexIO}. +Note that {\tt CactusPUGHIO/IOHDF5} does not provide such a utility yet. + +\section{Checkpointing and Recovery in Cactus} + +The IO methods for 3D output also provide functionality for {\it checkpointing} +and {\it recovery}. A checkpoint is a snapshot of the current state of the +simulation ({\it ie} the contents of all the grid variables and the parameter +settings) at a chosen timestep. Each checkpoint is saved into a +{\it checkpoint file} which can be used to restart a new simulation at a later +time, recreating the exact state at which it was checkpointed.\\ +Checkpointing is especially useful when running Cactus in batch queue systems +where jobs get only limited CPU time. A more advanced use of checkpointing +would be to restart your simulation after a crash or a problem had developed, +using a different parameter set recovering from the latest stable timestep. +Additionally, for performing parameter studies, compute-intensive +initial data can be calculated just once and saved in a checkpoint file +from which each job can be started. \\ + +Again, thorn {\tt CactusBase/IOUtil} provides general checkpoint \& recovery +parameters. The most important ones are: +\begin{itemize} + \item {\tt IO::checkpoint\_every}\\ + specifies how often to checkpoint in terms of iteration number + \item {\tt IO::checkpoint\_file}\\ + holds the checkpoint filename + \item {\tt IO::checkpoint\_dir}\\ + names the directory where checkpoint files are stored + \item {\tt IO::checkpoint\_ID}\\ + boolean flag for checkpointing only initial data + \item {\tt IO::recover}\\ + keyword parameter telling if/how to recover.\\ + Choices are {\tt "auto"}, {\tt "manual"}, and {\tt "no"}. + \item {\tt IO::recover\_file}\\ + filename of the recovery file + \item {\tt IO::recovery\_dir}\\ + directory where the recovery file is located +\end{itemize} + +To checkpoint your simulation, you need to enable checkpointing by setting +the Boolean parameter {\tt checkpoint}, for one of the 3D IO methods to +{\tt yes}. +Checkpoint filenames consist of a basename (as specified in +{\tt IO::checkpoint\_file}) followed by {\tt ".chkpt.it\_$<$iteration\_number$>$"} +plus the file extension indicating the file format ({\tt ".ieee"} for IEEEIO +data from {\tt CactusPUGHIO/IOFlexIO}, or {\tt ".h5} for HDF5 data from +{\tt CactusPUGHIO/IOHDF5}).\\ +The {\tt "auto"} recovery mode will automatically +recover from the latest checkpoint file found in the recovery directory. +In this case {\tt IO::recover\_file} should contain the basename only. +Use the {\tt "manual"} mode to recover from a specific checkpoint file +by adding the iteration number to the basename parameter.\\ +Because the same IO methods implement both output of 3D data and +checkpoint files, the same IO modes are used (see Section~\ref{iomodes}). +Note that the recovery routines in Cactus can process both chunked and +unchunked checkpoint files if you restart on the same number of processors. +That's why you should always use one of the parallel IO modes for +checkpointing. + + + +\section{Data Filename Extensions} + +\begin{table} +\label{two} +\begin{tabular}{|c|p{4.5cm}|l|} +\hline +{\bf Extension} & {\bf Description} & {\bf Thorn} \\ + {\tt .xl} + & + & + {\tt CactusBase/IOASCII} +\\ + {\tt .yl} + & + & + {\tt CactusBase/IOASCII} +\\ +{\tt .zl} + & + & + {\tt CactusBase/IOASCII} +\\ +{\tt .dl} + & + & + {\tt CactusBase/IOASCII} +\\ +{\tt .tl} + & + & + {\tt CactusBase/IOASCII} +\\ \hline +\end{tabular} +\caption{File extensions used by the standard IO thorns in the Cactus distribution} +\end{table} + + +\section{Example Parameter Files} + +Here we give examples of the parameters for the different IO methods. + +\begin{itemize} +\item{Screen Information {\tt info}} +\begin{verbatim} +ActiveThorns = ``iobasic ioutil ...'' + +# Output using all methods on iteration 0, 10, 20, ... +io::out_every = 10 + +# Group of variables to output to screen +iobasic::outinfo_vars = ``evolve::vars'' +\end{verbatim} + +\item{Scalar Output {\tt scalar}} +\begin{verbatim} +ActiveThorns = ``iobasic ioutil ...'' + +# Output vars using scalar method on iteration 0, 10, 20, ... +iobasic::outscalar_every = 10 + +# Group of variables to output to file +iobasic::outscalar_vars = ``evolve::vars'' +\end{verbatim} + +\item{ASCII 1D and 2D Output {\tt ioascii\_1d}, {\tt ioascii\_2d}} +\begin{verbatim} +ActiveThorns = ``ioascii ioutil hyperslab ...'' + +# Output vars using ascii 1d method on iteration 0, 10, 20, ... +ioascii::out1d_every = 10 + +# Output vars using ascii 2d method on iteration 0, 50, 100, ... +ioascii::out1d_every = 100 + +# Group of variables to output to file for each method +iobasic::outscalar_vars = ``evolve::vars'' +\end{verbatim} + +\end{itemize} \end{document} -- cgit v1.2.3