1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
|
% Thorn documentation template
\documentclass{article}
\begin{document}
\title{IOFlexIO}
\author{Paul Walker}
\date{1998-1999}
\maketitle
\abstract{Thorn IOFlexIO provides I/O methods to output variables in IEEEIO
file format. It also implements checkpointing/recovery functionality.}
%
\section{Purpose}
%
Thorn IOFlexIO uses John Shalf's FlexIO library (see {\tt
http://bach.ncsa.uiuc.edu/FlexIO/} for details) to output any type of grid
variables (grid scalars, grid functions, and arrays of arbitrary dimension)
in the IEEEIO file format.\\
The thorn registers two I/O methods with the flesh's I/O interface at startup:
%
\begin{itemize}
\item method {\tt IOFlexIO\_3D} outputs all types of grid variables with
arbitrary dimensions
\item method {\tt IOFlexIO\_2D} outputs two-dimensional slices (xy-, xz-,
and yz-slice) of three-dimensional grid functions and arrays
\end{itemize}
Data is written into files named {\tt "<varname>.ieee"} (for method {\tt IOFlexIO\_3D}) and {\tt "<varname>\_2d\_<plane>.ieee"} (for method {\tt IOFlexIO\_2D}).
Such datafiles can be used for further postprocessing (eg. visualization)
or fed back into Cactus via the filereader capabilities of thorn IOUtil.\\[3ex]
{\bf Parallel File I/O}\\
According to the ouptput mode parameter settings ({\tt IO::out3D\_mode,
IO::out3D\_unchunked, IO::out3D\_procs}) of thorn IOUtil, thorn IOFlexIO
will output distributed data either
\begin{itemize}
\item in serial into a single unchunked file
\begin{verbatim}
IO::out3D_mode = "onefile"
IO::out3D_unchunked = "yes"
\end{verbatim}
\item in parallel, that is, into separate files containing chunks of the
individual processors' patches of the distributed array
\begin{verbatim}
IO::out3D_mode = "proc | np"
\end{verbatim}
\end{itemize}
The default is to output data in parallel, in order to get maximum I/O
performance. If needed, you can recombine the resulting chunked datafiles
into a single unchunked file using the recombiner utility program provided
in {\tt IOFlexIO/src/util/}.\\
To build the recombiner just do a
\begin{verbatim}
make <configuration>-utils
\end{verbatim}
in the Cactus toplevel directory. The recombiner executable {\tt
ieee\_recombiner} will be placed in the {\tt exe/<configuration>/}
subdirectory.\\
Its usage is:
\begin{verbatim}
~/Cactus/exe/wave> ieee_recombiner
-------------------------------
Cactus 4 IEEEIO File Recombiner
-------------------------------
Usage: recombiner <chunked_infile0> <unchunked_outfile>
eg, recombiner alp_3d.file_0.ieee alp_3d.ieee
\end{verbatim}
If you have a lot of different variables to recombine you can use the following
Bourne shell commands to recombine them.
This assumes that the chunked output files for each variable are located in a
subdirectory {\tt <varname>\_3d/}.
The recombined output file {\tt <varname>\_3d.ieee} would then be placed into
the current working directory:
\begin{verbatim}
for var in *_3d ; \
do \
{ \
if [ ! -r $var.ieee ] ; then \
ieee_recombiner $var/$var.file_0.ieee $var.ieee; \
fi; \
}; \
done
\end{verbatim}
\vspace*{3ex}
{\bf Checkpointing \& Recovery}\\
Thorn IOFlexIO can also be used for creating IEEEIO checkpoint files and
recovering from such files later on.\\
Checkpoint routines are scheduled at several timebins so that you can save
the current state of your simulation atfer the initial data phase,
during evolution, or at termination.
A recovery routine is registered with thorn IOUtil in order to restart
a new simulation from a given IOFlexIO checkpoint.
The very same recovery mechanism is used to implement a filereader
functionality to feed back data into Cactus.\\
Checkpointing and recovery are controlled by corresponding checkpoint/recovery
parameters of thorn IOUtil (for a description of these parameters please refer
to this thorn's documentation).
\section{Comments}
{\bf Importing external data into Cactus with IOFlexIO}\\
In order to import external data into Cactus (eg. to initialize some variable)
you first need to convert this data into an IEEEIO datafile which then can be
processed by the registered recovery routine of thorn IOFlexIO.\\
The following description explains the IEEEIO file layout of an unchunked
datafile which thorn IOFlexIO expects in order to restore Cactus variables
from it properly. There is also a well-documented example C program provided
({\tt IOFlexIO/doc/CreateIOFlexIOdatafile.c}) which illustrates how to create
a datafile with IEEEIO file layout. This working example can be used as a
template for building your own data converter program.\\
\begin{enumerate}
\item Actual data is stored as multidimensional datasets in an IEEEIO file.
\item The type of your data as well as its dimensions are already
inherited by a dataset itself as metainformation. But this is not
enough for IOFlexIO to savely match it against a specific Cactus
variable.
For that reason, the variable's name, its groupname, its grouptype, the
timelevel to restore, and the
total number of timelevels must be attached to every dataset
as attribute information.
\item Finally, the recovery routine needs to know how the datafile to
recover from was created:
\begin{itemize}
\item Does the file contain chunked or unchunked data ?
\item How many processors were used to produce the data ?
\item How many I/O processors were used to write the data ?
\end{itemize}
Such information is attached as attributes to the very first dataset
in the file. Since we assume unchunked data here
the processor information isn't relevant -- unchunked data can
be fed back into a Cactus simulation running on an arbitrary
number of processors.
\end{enumerate}
The example C program goes through all of these steps and creates a datafile
{\tt x\_3d.ieee} in IEEEIO file layout which contains a single dataset named
{\tt "grid::x"}, with groupname {\tt "grid::coordinates"}, grouptype {\tt
CCTK\_GF} (thus identifying the variable as a grid function), the timelevel
to restore set to 0, and the total number of timelevels set to 1.\\
The global attributes are set to
{\tt "unchunked" $=$ "yes", nprocs $=$ 1,} and {\tt ioproc\_every $=$ 1}.\\
Once you've built and ran the program you can easily verify if it worked
properly with
\begin{verbatim}
ioinfo -showattrdata x_3d.ieee
\end{verbatim}
which lists all objects in the datafile along with their values.
Since the single dataset in it only contains zeros
it would probably not make much sense to feed this datafile into Cactus for
initializing your x coordinate grid function :-)
%
% Automatically created from the ccl files
% Do not worry for now.
\include{interface}
\include{param}
\include{schedule}
\end{document}
|
