\documentclass{article}

% Use the Cactus ThornGuide style file
% (Automatically used from Cactus distribution, if you have a 
%  thorn without the Cactus Flesh download this from the Cactus
%  homepage at www.cactuscode.org)
\usepackage{../../../../doc/ThornGuide/cactus}

\begin{document}

\title{NaNChecker}
\author{Thomas Radke}
\date{$ $Date$ $}

\maketitle

% Do not delete next line
% START CACTUS THORNGUIDE

\begin{abstract}
Thorn NaNChecker reports NaN values found in CCTK grid variables.
\end{abstract}

% Do not delete next line
% END CACTUS THORNGUIDE

\section{Purpose}

The NaNChecker thorn can be used to analyze Cactus grid variables (that is grid
functions, arrays or scalars) of real or complex data type for NaN
(Not-a-Number) and (on availability of {\tt finite(3)}) infinite
values. Grid variables can be periodically checked, or a call can 
be inserted into a thorn to check at a specific point.

This thorn is a utility thorn, designed to be used for debugging and
testing code for uninitialised variables, or for variables which 
become corrupted during a simulation, for example following a division
by zero or illegal memory usage.

On many architectures, uninitialised variables will be given the value
zero, and simulations using such variables will seemingly run perfectly 
well. However, not only is it dubious programming practise to assume such
behaviour, but also moving to a new machine may well cause pathalogical 
problems (for example, with Alpha processors used in Compaq or Cray machines).
It is thus recommended to test codes periodically with the NaNChecker, 
and to fix any problems as soon as they are seen.

\section{Periodic Testing}

Periodic testing of variables can easily be achieved by adding NaNChecker
to the {\tt ActiveThorns} parameter, and setting the parameters 

\noindent
{\tt NaNChecker::check\_every, NaNChecker::check\_after,} and {\tt NaNChecker::check\_vars} 

\noindent
to the required
values. (For most testing purposes these can be set to {\tt 1, 0,} and {"all"} 
respectively).

The NaNChecker then registers a routine at {\bf CCTK\_ANALYSIS} which checks
at every {\tt NanChecker::check\_every} iteration -- starting at iteration
number {\tt NaNChecker::check\_after} -- all the variables listed in
{\tt NaNChecker::check\_vars} for NaN or infinite values (depending on {\tt
NaNChecker::check\_for}) and --- if such a value is found --- performs an
action as specified in {\tt NaNChecker::action\_if\_found}.\\

Currently these actions can be to
\begin{itemize}
 \item{\tt just\_warn} (the default) 

   just print a level 1 warning message telling you where NaNs/Infs were found
   and how many (for grid array variables).\\
   If the keyword parameter {\tt verbose} is set to {\tt "all"} then for each
   grid array it will also print the grid indices (in Fortran notation)
   and the physical coordinates for all NaN/Inf elements found.
   You can limit the number of
   such warnings by setting the {\tt NanChecker::report\_max} parameter.

 \item{\tt terminate} 

also set the CCTK termination flag so that Cactus will stop the evolution
   loop and gracefully terminate at the next time possible (giving you the
   choice of outputting the data from the last evolution timestep),

 \item{\tt abort} 

print the warning message(s) and immediately terminate Cactus after checking
all variables from {\tt NaNChecker::check\_vars} by a call to {\tt CCTK\_Abort()}
\end{itemize}

By default, the current timelevel of the variables given in
{\tt NaNChecker::check\_vars} will be checked. This can be overwritten by
an optional string {\tt [timelevel=<timelevel>]} appended to the
variable/group name. For example, to apply the NaNChecker to timelevel 0
of the variable grid::x, timelevel 1 of {\tt grid::y} and timelevel 2 of 
{\tt grid::z} you would use the parameter

\noindent
{\tt NaNChecker::check\_vars = "grid::x grid::y[timelevel=1] grid::z[timelevel=2]"}

%
\section{Tracking and Visualizing NaNs Positions}
%
The NaNChecker thorn can also mark the positions (in grid index points)
of all the NaNs found for a given list of CCTK grid functions in a mask array
and save this array to an HDF5 file.

The mask array is declared as a grid function {\tt NaNChecker::NaNmask} with
data type INTEGER. Each bit $i$ in an integer element is used to flag a NaN
value found in grid function $i$ at the corresponding grid position
(the counting for $i$ starts at 0 and is incremented for each grid function
as it appears in {\tt NaNChecker::check\_vars}).
Thus the NaN locations of up to 32 individual grid functions can be coded
in the NaNmask array.

In order to activate the NaNmask output you need to set the parameter
{\tt NaNChecker::out\_NaNmask} to {\tt "yes"} (which is already the default)
and have the IOHDF5 thorn activated.

The NaN locations can be visualized with OpenDX. An example DX network
{\tt VisualizeNaNs.net} and a sample NaNmask HDF5 output file {\tt NaNmask.h5}
are available via anonymous CVS from the NumRel CVS server:
\begin{verbatim}
  # this is for (t)csh; use export CVSROOT for bash
  setenv CVSROOT :pserver:cvs_anon@cvs.aei.mpg.de:/numrelcvs

  # CVS pserver password is 'anon'
  cvs login
  cvs checkout AEIPhysics/Visualization/OpenDX/Networks/Miscellaneous
\end{verbatim}

%
\section{NaNChecker API}
%
Thorn NaNChecker also provides a function API which can be used by other code
to invoke the NaNChecker routines to test for NaN/Inf values or to set NaN
values for a list of variables:

\vskip .2cm
\noindent{\bf C API}

\begin{verbatim}
  int NaNChecker_CheckVarsForNaN (const cGH *cctkGH,
                                  int report_max,
                                  const char *vars,
                                  const char *check_for,
                                  const char *action_if_found);

  int NaNChecker_SetVarsToNaN (const cGH *cctkGH,
                               const char *vars);
\end{verbatim}

\vskip.2cm
\noindent
{\bf Fortran API}

\begin{verbatim}
  call NaNChecker_CheckVarsForNaN (ierror, cctkGH, report_max,
                                   vars, check_for, action_if_found)

                                   integer ierror
                                   CCTK_POINTER cctkGH
                                   integer report_max
                                   character*(*) vars
                                   character*(*) check_for
                                   character*(*) action_if_found

  call NaNChecker_SetVarsToNaN (ierror, cctkGH, vars)

                                integer ierror
                                CCTK_POINTER cctkGH
                                character*(*) vars
\end{verbatim}

The {\tt report\_max, check\_vars, check\_for} and {\tt action\_if\_found}
arguments have the same semantics as their parameter counterparts.\\
If {\tt action\_if\_found} is given as a NULL pointer (C API) or as an empty
string (Fortran API) the routine will be quiet and just return the number of
NaN values found.\\

The C function {\tt NaNChecker\_CheckVarsForNaN()} returns the total number of
NaN/Inf values found, {\tt NaNChecker\_SetToNaN()} returns the total number of
variables set to NaN; this return value is stored in the {\tt ierror} argument
for the corresponding fortran wrapper routines.

% Do not delete next line
% END CACTUS THORNGUIDE

\end{document}