path: root/Carpet/CarpetIOHDF5/doc/documentation.tex

\documentclass{article}

% Use the Cactus ThornGuide style file
% (Automatically used from Cactus distribution, if you have a 
%  thorn without the Cactus Flesh download this from the Cactus
%  homepage at www.cactuscode.org)
\usepackage{../../../../doc/ThornGuide/cactus}

\begin{document}

% The author of the documentation
\author{Erik Schnetter \textless schnetter@uni-tuebingen.de\textgreater\\
        Christian D. Ott \textless cott@aei.mpg.de\textgreater\\
        Thomas Radke \textless tradke@aei.mpg.de\textgreater}

% The title of the document (not necessarily the name of the Thorn)
\title{CarpetIOHDF5}

% the date your document was last changed, if your document is in CVS, 
% please use:
\date{1 December 2004}

\maketitle

% Do not delete next line
% START CACTUS THORNGUIDE

\newcommand{\ThisThorn}{{\it CarpetIOHDF5}}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{abstract}
Thorn \ThisThorn\ provides HDF5-based output to the {\em Carpet} mesh
refinement driver in {\em Cactus}.
This document explains \ThisThorn's usage and contains a specification
of the HDF5 file format that was adapted from John Shalf's FlexIO library.
\end{abstract}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Introduction}

Having encountered various problems with the Carpet I/O thorn
{\bf CarpetIOFlexIO} and the underlying FlexIO library,
Erik Schnetter decided to write this thorn \ThisThorn\ which bypasses
any intermediate binary I/O layer and outputs in HDF5\footnote{Hierarchical
Data Format version 5, see {\tt http://hdf.ncsa.uiuc.edu/whatishdf5.html}
for details} file format directly.

\ThisThorn\ provides output for the {\em Carpet} Mesh Refinement driver
within the Cactus Code. Christian D. Ott added  a file reader (analogous to
Erik Schnetter's implementation present in {\bf CarpetIOFlexIO}) 
as well as checkpoint/recovery functionality to \ThisThorn.
Thomas Radke has taken over maintainence of this I/O thorn and is continuously
working on fixing known bugs and improving the code functionality and
efficiency.

The \ThisThorn\ I/O method can output any type of CCTK grid variables
(grid scalars, grid functions, and grid arrays of arbitrary dimension);
data is written into separate files named {\tt "<varname>.h5"}.
It implements both serial and full parallel I/O --
data files can be written/read either by processor 0 only or by all processors.
Such datafiles can be used for further postprocessing (eg. visualization with
OpenDX or DataVault\footnote{see our VizTools page at \url{http://www.cactuscode.org/VizTools.html}
for details}) or fed back into Cactus via the filereader capabilities of thorn
{\bf IOUtil}.

This document aims at giving the user a first handle on how to use
\ThisThorn. It also documents the HDF5 file layout used.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{\ThisThorn\ Parameters}

Parameters to control the \ThisThorn\ I/O method are:

\begin{itemize}
  \item {\tt IOHDF5::out\_every} (steerable)\\
        How often to do periodic \ThisThorn\ output. If this parameter is set
        in the parameter file, it will override the setting of the shared
        {\tt IO::out\_every} parameter. The output frequency can also be set
        for individual variables using the {\tt out\_every} option in an option
        string appended to the {\tt IOHDF5::out\_vars} parameter.

  \item {\tt IOHDF5::out\_dt} (steerable)\\
        output in intervals of that much coordinate time (overwrites {\tt IO::out\_dt})

  \item {\tt IOHDF5::out\_criterion} (steerable)\\
        criterion to select output intervals (overwrites {\tt IO::out\_criterion})

  \item {\tt IOHDF5::out\_vars} (steerable)\\
        The list of variables to output using the \ThisThorn\ I/O method.
        The variables must be given by their fully qualified variable or group
        name. The special keyword {\it all} requests \ThisThorn\ output for
        all variables. Multiple names must be separated by whitespaces.

        Each group/variable name can have an option string attached in which you
        can specify a different output frequency for that individual variable
	or a set of individual refinement levels to be output, e.g.
\begin{verbatim}
  IOHDF5::out_vars = "wavetoy::phi{ out_every = 4 refinement_levels = { 1 2 } }"
\end{verbatim}

  \item {\tt IOHDF5::out\_dir}\\
        The directory in which to place the \ThisThorn\ output files.
        If the directory doesn't exist at startup it will be created.
        If parallel output is enabled and the directory name contains the
        substring {\tt "\%u"} it will be substituted by the processor ID.
        By this means each processor can have its own output directory.\\
        If this parameter is set to an empty string \ThisThorn\ output will go
        to the standard output directory as specified in {\tt IO::out\_dir}.

  \item {\tt IO::out\_single\_precision (steerable)}\\
        whether to output double-precision data in single precision

\end{itemize}


\section{Serial versus Parallel Output}

According to the ouptput mode parameter settings of ({\tt IO::out\_mode},
{\tt IO::out\_unchunked},\newline{\tt IO::out\_proc\_every}) of thorn
{\bf IOUtil}, thorn \ThisThorn\ will output distributed grid variables either

\begin{itemize}
  \item in serial from processor 0 into a single unchunked file
\begin{verbatim}
  IO::out_mode      = "onefile"
  IO::out_unchunked = "yes"
\end{verbatim}

  \item in serial from processor 0 into a single chunked file
\begin{verbatim}
  IO::out_mode      = "onefile"
  IO::out_unchunked = "no"
\end{verbatim}

  \item in parallel, that is, into separate chunked files (one per processor)
        containing the individual processors' patches of the
        distributed grid variable
\begin{verbatim}
  IO::out_mode      = "proc"
\end{verbatim}
\end{itemize}

For unchunked data all interprocessor ghostzones are excluded from the output.
The entire grid variable in contained in a single HDF5 dataset.
Chunked output includes all information from all processors as chunks in
separate HDF5 datasets (thus adding some overhead in storing metadata).
When visualising chunked data files, they probably need to be recombined
for a global view on the data.

The default is to output distributed grid variables in parallel, each processor
writing a file
{\tt \textless varname\textgreater.file\_\textless processor ID\textgreater.h5}.
Grid scalars
and {\tt DISTRIB = CONST} grid arrays are always output as unchunked data
on processor 0 only.\\
Parallel output in a parallel simulation will ensure maximum I/O
performance. Note that changing the output mode to serial I/O might only be
necessary if the data analysis and visualisation tools cannot deal with
chunked output files. Cactus itself, as well as many of the tools to
visualise Carpet HDF5 data (see \url{http://www.cactuscode.org/VizTools.html}),
can process both chunked and unchunked data. For instance, to visualise parallel
output datafiles with DataVault, you would just send all the individual files
to the DV server: {\tt hdf5todv phi.file\_*.h5}. In OpenDX the {\tt
ImportCarpetIOHDF5} module can be given any filename from the set of parallel
chunked files; the module will determine the total number of files in the set
automatically and read them all.


\section{Using the flesh I/O API to produce HDF5 output}

Periodic output of grid variables is usually specified via I/O parameters
in the parameter file and then automatically triggered by the flesh scheduler
at each iteration step after analysis. If output should also be triggered
at a different time, one can do that from within an application thorn by
invoking one of the {\tt CCTK\_OutputVar*()} I/O routines provided
by the flesh I/O API (see chapter B8.2 ``IO'' in the Cactus Users Guide).
In this case, the application thorn routine which calls {\tt CCTK\_OutputVar*()}
must be scheduled in level mode.

It should be noted here that -- due to a restriction in the naming scheme of
objects in an HDF5 data file -- \ThisThorn\ can output a given grid variable
with given refinement level only once per timestep. Attempts of application
thorns to trigger the output of the same variable multiple times during an iteration
will result in a runtime warning and have no further effect.
If output for a variable is required also for intermediate timesteps
this can be achieved by calling {\tt CCTK\_OutputVarAs*()} with a different
{\tt alias} name; output for the same variable is then written into
different HDF5 files based on the {\tt alias} argument.


\section{Checkpointing \& Recovery and Importing Data}

Thorn \ThisThorn\ can also be used to create HDF5 checkpoint files and
to recover from such files later on. In addition it can read HDF5 datafiles
back in using the generic filereader interface described in the thorn
documentation of {\bf IOUtil}.

Checkpoint routines are scheduled at several timebins so that you can save
the current state of your simulation after the initial data phase,
during evolution, or at termination. Checkpointing for thorn \ThisThorn\ 
is enabled by setting the parameter {\tt IOHDF5::checkpoint = "yes"}.

A recovery routine is registered with thorn {\bf IOUtil} in order to restart
a new simulation from a given HDF5 checkpoint.
The very same recovery mechanism is used to implement a filereader
functionality to feed back data into Cactus.

Checkpointing and recovery are controlled by corresponding checkpoint/recovery
parameters of thorn {\bf IOUtil} (for a description of these parameters please
refer to this thorn's documentation).


\section{Example Parameter File Excerpts}

\subsection{Serial (unchunked) Output of Grid Variables}

\begin{verbatim}
  # how often to output and where output files should go
  IO::out_every = 2
  IO::out_dir   = "wavetoy-data"

  # request output for wavetoy::psi at every other iteration for timelevel 0,
  #                for wavetoy::phi every 4th iteration with timelevels 1 and 2
  IOHDF5::out_vars = "wavetoy::phi{ out_every = 4 refinement_levels = { 1 2 } }
                      wavetoy::psi"

  # we want unchunked output
  # (because the visualisation tool cannot deal with chunked data files)
  IO::out_mode      = "onefile"
  IO::out_unchunked = 1
\end{verbatim}

\subsection{Parallel (chunked) Output of Grid Variables}

\begin{verbatim}
  # how often to output
  IO::out_every = 2

  # each processor writes to its own output directory
  IOHDF5::out_dir = "wavetoy-data-proc%u"

  # request output for wavetoy::psi at every other iteration for timelevel 0,
  #                for wavetoy::phi every 4th iteration with timelevels 1 and 2
  IOHDF5::out_vars = "wavetoy::phi{ out_every = 4 refinement_levels = { 1 2 } }
                      wavetoy::psi"

  # we want parallel chunked output (note that this already is the default)
  IO::out_mode = "proc"
\end{verbatim}

\subsection{Checkpointing \& Recovery}

\begin{verbatim}
  # say how often we want to checkpoint, how many checkpoints should be kept,
  # how the checkpoints should be named, and they should be written to
  IO::checkpoint_ID   = 100
  IO::checkpoint_keep = 2
  IO::checkpoint_file = "wavetoy"
  IO::checkpoint_dir  = "wavetoy-checkpoints"

  # enable checkpointing for CarpetIOHDF5
  IOHDF5::checkpoint = "yes"

  #######################################################

  # recover from the latest checkpoint found
  IO::recover_file = "wavetoy"
  IO::recover_dir  = "wavetoy-checkpoints"
  IO::recover      = "auto"
\end{verbatim}

\subsection{Importing Grid Variables via Filereader}

\begin{verbatim}
  # which data files to import and where to find them
  IO::filereader_ID_files = "phi psi"
  IO::filereader_ID_dir   = "wavetoy-data"

  # what variables and which timestep to read
  # (if this parameter is left empty, all variables and timesteps found
  #  in the data files will be read)
  IO::filereader_ID_vars  = "WaveToyMoL::phi{ cctk_iteration = 0 }
                             WaveToyMoL::psi"
\end{verbatim}


\iffalse
\section{CarpetIOHDF5's HDF5 file layout}

The HDF5 file layout of {\bf CarpetIOHDF5} is quite simple.
There are no groups besides the standard HDF5 root data object group:

Each dataset is named according to this template:

\begin{verbatim}
  <group::varname> it=<cctk_iteration> tl=<timelevel> [ml=<mglevel>] [m=<map>]
  [rl=<reflevel>] [c=<component>]}
\end{verbatim}

where optional parts only contribute to the name if they vary (if there is
more than one multigrid level, map, refinement level, component respectively).

Each HDF5 dataset has the following attributes associated with it:

\begin{itemize}
  \item {\tt level} : Carpet::reflevel
  \item {\tt origin} : 1-D array of length vdim. \\
        origin[d] = CCTK\_ORIGIN\_SPACE(d) + cctk\_lbnd[d] * delta[d]
  \item {\tt delta} : 1-D array of length vdim. \\
        delta[d] = CCTK\_DELTA\_SPACE(d)
  \item {\tt time} : cctk\_time
  \item {\tt timestep} : cctk\_iteration
  \item {\tt iorigin} : 1-D array of length vdim. \\ iorigin[d] = (Carpet::ext.lower() / Carpet::ext.stride())[d]
  \item {\tt name} : CCTK\_FullName(variable index)
  \item {\tt cctk\_bbox} : 1-D array of length 2*Carpet::dim. cctk\_box
  \item {\tt cctk\_nghostzones} : 1-D array of length Carpet::dim. cctk\_nghostzones
  \item {\tt carpet\_mglevel} : Carpet::mglevel
  \item {\tt carpet\_reflevel} : Carpet::reflevel
\end{itemize}


\subsection{Attributes needed by the file reader}

The number of attributes needed by the CarpetIOHDF5 file reader is much smaller then the total
number of attributes attached to each dataset:

\begin{itemize}
  \item {\tt name}
  \item {\tt level}
  \item {\tt iorigin}
\end{itemize}

\fi


% Do not delete next line
% END CACTUS THORNGUIDE

\end{document}