extended IOHDF5 which can do n dimensional output plus n-1 dimensional hyperslabs - testing phase

git-svn-id: http://svn.cactuscode.org/arrangements/CactusPUGHIO/IOHDF5/trunk@2 4825ed28-b72c-4eae-9704-e50c059e567d

36 files changed, 6596 insertions, 0 deletions

diff --git a/COPYRIGHT b/COPYRIGHT
new file mode 100644
index 0000000..57b2896
--- /dev/null
+++ b/COPYRIGHT
@@ -0,0 +1,352 @@
+
+
+This thorn is (c) Copyright the authors listed in the sources files. This
+thorn is distributed under the GNU GPL with the specific exception that
+the GNU GPL does not migrate to code linking to or using infrastructure
+from this thorn.
+
+- The Cactus Team <cactus@cactuscode.org>
+
+**************************************************************************
+
+		    GNU GENERAL PUBLIC LICENSE
+		       Version 2, June 1991
+
+ Copyright (C) 1989, 1991 Free Software Foundation, Inc.
+                          675 Mass Ave, Cambridge, MA 02139, USA
+ Everyone is permitted to copy and distribute verbatim copies
+ of this license document, but changing it is not allowed.
+
+			    Preamble
+
+  The licenses for most software are designed to take away your
+freedom to share and change it.  By contrast, the GNU General Public
+License is intended to guarantee your freedom to share and change free
+software--to make sure the software is free for all its users.  This
+General Public License applies to most of the Free Software
+Foundation's software and to any other program whose authors commit to
+using it.  (Some other Free Software Foundation software is covered by
+the GNU Library General Public License instead.)  You can apply it to
+your programs, too.
+
+  When we speak of free software, we are referring to freedom, not
+price.  Our General Public Licenses are designed to make sure that you
+have the freedom to distribute copies of free software (and charge for
+this service if you wish), that you receive source code or can get it
+if you want it, that you can change the software or use pieces of it
+in new free programs; and that you know you can do these things.
+
+  To protect your rights, we need to make restrictions that forbid
+anyone to deny you these rights or to ask you to surrender the rights.
+These restrictions translate to certain responsibilities for you if you
+distribute copies of the software, or if you modify it.
+
+  For example, if you distribute copies of such a program, whether
+gratis or for a fee, you must give the recipients all the rights that
+you have.  You must make sure that they, too, receive or can get the
+source code.  And you must show them these terms so they know their
+rights.
+
+  We protect your rights with two steps: (1) copyright the software, and
+(2) offer you this license which gives you legal permission to copy,
+distribute and/or modify the software.
+
+  Also, for each author's protection and ours, we want to make certain
+that everyone understands that there is no warranty for this free
+software.  If the software is modified by someone else and passed on, we
+want its recipients to know that what they have is not the original, so
+that any problems introduced by others will not reflect on the original
+authors' reputations.
+
+  Finally, any free program is threatened constantly by software
+patents.  We wish to avoid the danger that redistributors of a free
+program will individually obtain patent licenses, in effect making the
+program proprietary.  To prevent this, we have made it clear that any
+patent must be licensed for everyone's free use or not licensed at all.
+
+  The precise terms and conditions for copying, distribution and
+modification follow.
+
+		    GNU GENERAL PUBLIC LICENSE
+   TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION
+
+  0. This License applies to any program or other work which contains
+a notice placed by the copyright holder saying it may be distributed
+under the terms of this General Public License.  The "Program", below,
+refers to any such program or work, and a "work based on the Program"
+means either the Program or any derivative work under copyright law:
+that is to say, a work containing the Program or a portion of it,
+either verbatim or with modifications and/or translated into another
+language.  (Hereinafter, translation is included without limitation in
+the term "modification".)  Each licensee is addressed as "you".
+
+Activities other than copying, distribution and modification are not
+covered by this License; they are outside its scope.  The act of
+running the Program is not restricted, and the output from the Program
+is covered only if its contents constitute a work based on the
+Program (independent of having been made by running the Program).
+Whether that is true depends on what the Program does.
+
+  1. You may copy and distribute verbatim copies of the Program's
+source code as you receive it, in any medium, provided that you
+conspicuously and appropriately publish on each copy an appropriate
+copyright notice and disclaimer of warranty; keep intact all the
+notices that refer to this License and to the absence of any warranty;
+and give any other recipients of the Program a copy of this License
+along with the Program.
+
+You may charge a fee for the physical act of transferring a copy, and
+you may at your option offer warranty protection in exchange for a fee.
+
+  2. You may modify your copy or copies of the Program or any portion
+of it, thus forming a work based on the Program, and copy and
+distribute such modifications or work under the terms of Section 1
+above, provided that you also meet all of these conditions:
+
+    a) You must cause the modified files to carry prominent notices
+    stating that you changed the files and the date of any change.
+
+    b) You must cause any work that you distribute or publish, that in
+    whole or in part contains or is derived from the Program or any
+    part thereof, to be licensed as a whole at no charge to all third
+    parties under the terms of this License.
+
+    c) If the modified program normally reads commands interactively
+    when run, you must cause it, when started running for such
+    interactive use in the most ordinary way, to print or display an
+    announcement including an appropriate copyright notice and a
+    notice that there is no warranty (or else, saying that you provide
+    a warranty) and that users may redistribute the program under
+    these conditions, and telling the user how to view a copy of this
+    License.  (Exception: if the Program itself is interactive but
+    does not normally print such an announcement, your work based on
+    the Program is not required to print an announcement.)
+
+These requirements apply to the modified work as a whole.  If
+identifiable sections of that work are not derived from the Program,
+and can be reasonably considered independent and separate works in
+themselves, then this License, and its terms, do not apply to those
+sections when you distribute them as separate works.  But when you
+distribute the same sections as part of a whole which is a work based
+on the Program, the distribution of the whole must be on the terms of
+this License, whose permissions for other licensees extend to the
+entire whole, and thus to each and every part regardless of who wrote it.
+
+Thus, it is not the intent of this section to claim rights or contest
+your rights to work written entirely by you; rather, the intent is to
+exercise the right to control the distribution of derivative or
+collective works based on the Program.
+
+In addition, mere aggregation of another work not based on the Program
+with the Program (or with a work based on the Program) on a volume of
+a storage or distribution medium does not bring the other work under
+the scope of this License.
+
+  3. You may copy and distribute the Program (or a work based on it,
+under Section 2) in object code or executable form under the terms of
+Sections 1 and 2 above provided that you also do one of the following:
+
+    a) Accompany it with the complete corresponding machine-readable
+    source code, which must be distributed under the terms of Sections
+    1 and 2 above on a medium customarily used for software interchange; or,
+
+    b) Accompany it with a written offer, valid for at least three
+    years, to give any third party, for a charge no more than your
+    cost of physically performing source distribution, a complete
+    machine-readable copy of the corresponding source code, to be
+    distributed under the terms of Sections 1 and 2 above on a medium
+    customarily used for software interchange; or,
+
+    c) Accompany it with the information you received as to the offer
+    to distribute corresponding source code.  (This alternative is
+    allowed only for noncommercial distribution and only if you
+    received the program in object code or executable form with such
+    an offer, in accord with Subsection b above.)
+
+The source code for a work means the preferred form of the work for
+making modifications to it.  For an executable work, complete source
+code means all the source code for all modules it contains, plus any
+associated interface definition files, plus the scripts used to
+control compilation and installation of the executable.  However, as a
+special exception, the source code distributed need not include
+anything that is normally distributed (in either source or binary
+form) with the major components (compiler, kernel, and so on) of the
+operating system on which the executable runs, unless that component
+itself accompanies the executable.
+
+If distribution of executable or object code is made by offering
+access to copy from a designated place, then offering equivalent
+access to copy the source code from the same place counts as
+distribution of the source code, even though third parties are not
+compelled to copy the source along with the object code.
+
+  4. You may not copy, modify, sublicense, or distribute the Program
+except as expressly provided under this License.  Any attempt
+otherwise to copy, modify, sublicense or distribute the Program is
+void, and will automatically terminate your rights under this License.
+However, parties who have received copies, or rights, from you under
+this License will not have their licenses terminated so long as such
+parties remain in full compliance.
+
+  5. You are not required to accept this License, since you have not
+signed it.  However, nothing else grants you permission to modify or
+distribute the Program or its derivative works.  These actions are
+prohibited by law if you do not accept this License.  Therefore, by
+modifying or distributing the Program (or any work based on the
+Program), you indicate your acceptance of this License to do so, and
+all its terms and conditions for copying, distributing or modifying
+the Program or works based on it.
+
+  6. Each time you redistribute the Program (or any work based on the
+Program), the recipient automatically receives a license from the
+original licensor to copy, distribute or modify the Program subject to
+these terms and conditions.  You may not impose any further
+restrictions on the recipients' exercise of the rights granted herein.
+You are not responsible for enforcing compliance by third parties to
+this License.
+
+  7. If, as a consequence of a court judgment or allegation of patent
+infringement or for any other reason (not limited to patent issues),
+conditions are imposed on you (whether by court order, agreement or
+otherwise) that contradict the conditions of this License, they do not
+excuse you from the conditions of this License.  If you cannot
+distribute so as to satisfy simultaneously your obligations under this
+License and any other pertinent obligations, then as a consequence you
+may not distribute the Program at all.  For example, if a patent
+license would not permit royalty-free redistribution of the Program by
+all those who receive copies directly or indirectly through you, then
+the only way you could satisfy both it and this License would be to
+refrain entirely from distribution of the Program.
+
+If any portion of this section is held invalid or unenforceable under
+any particular circumstance, the balance of the section is intended to
+apply and the section as a whole is intended to apply in other
+circumstances.
+
+It is not the purpose of this section to induce you to infringe any
+patents or other property right claims or to contest validity of any
+such claims; this section has the sole purpose of protecting the
+integrity of the free software distribution system, which is
+implemented by public license practices.  Many people have made
+generous contributions to the wide range of software distributed
+through that system in reliance on consistent application of that
+system; it is up to the author/donor to decide if he or she is willing
+to distribute software through any other system and a licensee cannot
+impose that choice.
+
+This section is intended to make thoroughly clear what is believed to
+be a consequence of the rest of this License.
+
+  8. If the distribution and/or use of the Program is restricted in
+certain countries either by patents or by copyrighted interfaces, the
+original copyright holder who places the Program under this License
+may add an explicit geographical distribution limitation excluding
+those countries, so that distribution is permitted only in or among
+countries not thus excluded.  In such case, this License incorporates
+the limitation as if written in the body of this License.
+
+  9. The Free Software Foundation may publish revised and/or new versions
+of the General Public License from time to time.  Such new versions will
+be similar in spirit to the present version, but may differ in detail to
+address new problems or concerns.
+
+Each version is given a distinguishing version number.  If the Program
+specifies a version number of this License which applies to it and "any
+later version", you have the option of following the terms and conditions
+either of that version or of any later version published by the Free
+Software Foundation.  If the Program does not specify a version number of
+this License, you may choose any version ever published by the Free Software
+Foundation.
+
+  10. If you wish to incorporate parts of the Program into other free
+programs whose distribution conditions are different, write to the author
+to ask for permission.  For software which is copyrighted by the Free
+Software Foundation, write to the Free Software Foundation; we sometimes
+make exceptions for this.  Our decision will be guided by the two goals
+of preserving the free status of all derivatives of our free software and
+of promoting the sharing and reuse of software generally.
+
+			    NO WARRANTY
+
+  11. BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY
+FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW.  EXCEPT WHEN
+OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES
+PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED
+OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
+MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.  THE ENTIRE RISK AS
+TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU.  SHOULD THE
+PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING,
+REPAIR OR CORRECTION.
+
+  12. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
+WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR
+REDISTRIBUTE THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES,
+INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING
+OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED
+TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY
+YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER
+PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE
+POSSIBILITY OF SUCH DAMAGES.
+
+		     END OF TERMS AND CONDITIONS
+
+	Appendix: How to Apply These Terms to Your New Programs
+
+  If you develop a new program, and you want it to be of the greatest
+possible use to the public, the best way to achieve this is to make it
+free software which everyone can redistribute and change under these terms.
+
+  To do so, attach the following notices to the program.  It is safest
+to attach them to the start of each source file to most effectively
+convey the exclusion of warranty; and each file should have at least
+the "copyright" line and a pointer to where the full notice is found.
+
+    <one line to give the program's name and a brief idea of what it does.>
+    Copyright (C) 19yy  <name of author>
+
+    This program is free software; you can redistribute it and/or modify
+    it under the terms of the GNU General Public License as published by
+    the Free Software Foundation; either version 2 of the License, or
+    (at your option) any later version.
+
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU General Public License for more details.
+
+    You should have received a copy of the GNU General Public License
+    along with this program; if not, write to the Free Software
+    Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+
+Also add information on how to contact you by electronic and paper mail.
+
+If the program is interactive, make it output a short notice like this
+when it starts in an interactive mode:
+
+    Gnomovision version 69, Copyright (C) 19yy name of author
+    Gnomovision comes with ABSOLUTELY NO WARRANTY; for details type `show w'.
+    This is free software, and you are welcome to redistribute it
+    under certain conditions; type `show c' for details.
+
+The hypothetical commands `show w' and `show c' should show the appropriate
+parts of the General Public License.  Of course, the commands you use may
+be called something other than `show w' and `show c'; they could even be
+mouse-clicks or menu items--whatever suits your program.
+
+You should also get your employer (if you work as a programmer) or your
+school, if any, to sign a "copyright disclaimer" for the program, if
+necessary.  Here is a sample; alter the names:
+
+  Yoyodyne, Inc., hereby disclaims all copyright interest in the program
+  `Gnomovision' (which makes passes at compilers) written by James Hacker.
+
+  <signature of Ty Coon>, 1 April 1989
+  Ty Coon, President of Vice
+
+This General Public License does not permit incorporating your program into
+proprietary programs.  If your program is a subroutine library, you may
+consider it more useful to permit linking proprietary applications with the
+library.  If this is what you want to do, use the GNU Library General
+Public License instead of this License.
+
+
diff --git a/README b/README
new file mode 100644
index 0000000..8b7643d
--- /dev/null
+++ b/README
@@ -0,0 +1,22 @@
+Cactus Code Thorn IOHDF5
+Authors    : Thomas Radke
+CVS info   : $Header$
+--------------------------------------------------------------------------
+
+1. Purpose of the thorn
+
+This thorn does output of 2D and 3D variables in HDF5 format.
+
+It also provides checkpointing/recovery functionality -
+you can create HDF5 checkpoint files and recover from them.
+
+The IO methods "IOHDF5_2D" and "IOHDF5_3D" are registered
+which can be called by other thorns via CCTK_OutputVarAsByMethod()
+to output a variable without specifying it in the parameter file.
+
+
+2. Additional information
+
+Currently only 3D output is implemented.
+
+Fore more information on HDF5 see http://hdf.ncsa.uiuc.edu.
diff --git a/doc/documentation.tex b/doc/documentation.tex
new file mode 100644
index 0000000..1a617ab
--- /dev/null
+++ b/doc/documentation.tex
@@ -0,0 +1,97 @@
+% Thorn documentation template
+\documentclass{article}
+\begin{document}
+
+\title{IOHDF5}
+\author{Thomas Radke}
+\date{1999}
+\maketitle
+
+\abstract{Thorn IOHDF5 provides an IO method for output of variables in HDF5,
+the Hierarchical Data Format version 5 ({\tt http://hdf.ncsa.uiuc.edu/whatishdf5.html}).\\
+It also implements checkpointing/recovery functionality using HDF5.}
+%
+\section{Purpose}
+%
+Thorn IOHDF5 makes HDF5 available to the Cactus user by registering an IO method
+called {\tt IOHDF5\_3D} with the IO interface in the flesh.\\
+This method creates three-dimensional output of 3D grid functions as well as
+plane output of scalar variables.
+Data is written in HDF5 file format and goes into files named
+{\tt "<varname>\_3D.h5"}. Such datafiles can be further processed by
+visualization tools like Amira or AVS.\\
+%
+\newline
+%
+You obtain output by an IO method by either
+%
+\begin{itemize}
+  \item setting the appropriate IO parameters
+  \item calling one the routines of the IO function interface provided by the flesh
+\end{itemize}
+%
+%
+IOHDF5 also provides checkpointing/recovery functionality by registering
+%
+\begin{itemize}
+  \item a checkpoint method with the Runtime Function Repository in the flesh
+     if checkpointing was requested for this thorn.
+     This method is then scheduled at {\em CCTK\_CPINITIAL} and/or
+     {\em CCTK\_CHECKPOINT} (to checkpoint initial and/or evolution data).\\
+     Controled via checkpoint parameters, it decides when to save the current
+     state of simulation by dumping the contents of all Cactus variables and
+     parameters into a checkpoint file which is in HDF5 file format.\\
+     To write this file the thorn's IO method {\tt IOHDF5\_3D} is used.
+  \item a recovery method with the generic recovery function interface of thorn IOUtil.\\
+     At recovery time this method is called by IOUtil's generic recovery
+     routine. It gets passed a filename which is tested to identify a checkpoint
+     file in HDF5 format. If successful the method will then restore the
+     contents of all Cactus variables and parameters from the given checkpoint
+     file.
+\end{itemize}
+%
+For a description of IO and checkpoint/recovery parameters and the generic
+recovery function interface please see also the documentation of thorn IOUtil.\\
+For a description of IO function interface to invoke IO methods by application
+thorns please see the flesh documentation.
+%
+%
+\section{Comments}
+
+Since IOHDF5 uses parameters and the recovery function interface from IOUtil
+it also needs this I/O skeleton thorn compiled into Cactus and activated.\\
+%
+%
+\newline
+{\bf Building Cactus with HDF5}\\
+%
+The Cactus distribution does not contain the HDF5 header files and library which
+is used by thorn IOHDF5. So you need to configure it as an external software
+package via:
+%
+\begin{verbatim}
+  make <configuration>-config HDF5=YES
+                             [HDF5_DIR=<path to HDF5 package>]
+\end{verbatim}
+%
+The configuration script will look in some default places for an installed
+HDF5 package. If nothing is found this way you can explicitely specify it with
+the {\tt HDF5\_DIR} option.\\
+%
+Configure also checks which library version is contained within the HDF5
+package: it can be either serial or parallel. The latter version includes the parallel IO extensions of the MPI 2 standard. To make use of these extensions
+you need to configure Cactus with both HDF5 and MPI. Please make also sure then
+that the parallel HDF5 library was built with the same MPI version as is used
+for Cactus.\\
+%
+If Cactus was not configured to use HDF5 but has thorn IOHDF5 compiled in
+it will give a warning message each time a thorn's routine is called
+saying HDF5 I/O is not available.
+%
+% Automatically created from the ccl files 
+% Do not worry for now.
+\include{interface}
+\include{param}
+\include{schedule}
+
+\end{document}
diff --git a/interface.ccl b/interface.ccl
new file mode 100644
index 0000000..7f68414
--- /dev/null
+++ b/interface.ccl
@@ -0,0 +1,5 @@
+# Interface definition for thorn IOHDF5
+# $Header$
+
+implements: IOHDF5
+inherits:   CACTUS
diff --git a/par/hdf5.par b/par/hdf5.par
new file mode 100644
index 0000000..ec67555
--- /dev/null
+++ b/par/hdf5.par
@@ -0,0 +1,31 @@
+# hdf5.par: HDF5 output 
+
+ActiveThorns = "CartGrid3D IOUtil IDScalarWave WavetoyF77 PUGH IOHDF5"
+
+Cactus::cctk_brief_output	= "yes"
+
+IDScalarWave::radius = 0.0 
+IDScalarWave::amplitude = 1.0 
+IDScalarWave::initial_data = "gaussian"
+
+cactus::cctk_itlast = 10
+
+# PUGH parameters
+driver::global_nx = 10
+driver::global_ny = 10
+driver::global_nz = 10
+
+
+####################### IO parameters ###################################
+
+# flag to enable verbose output
+IO::verbose = "yes"
+
+# say how often to output
+IO::out_every = 1
+IO::out3D_mode       = "onefile"
+IO::out3D_unchunked    = "yes"
+
+# say what GF to output
+IOHDF5::out3D_vars = "all"
+IOHDF5::outdir3D = "./hdf5"
diff --git a/param.ccl b/param.ccl
new file mode 100644
index 0000000..f222e18
--- /dev/null
+++ b/param.ccl
@@ -0,0 +1,290 @@
+# Parameter definitions for thorn IOHDF5
+
+#############################################################################
+### declare IOHDF5 parameters
+#############################################################################
+private:
+
+########################
+# How often to do output
+########################
+INT out1D_every "How often to do 1D output, overrides out_every" STEERABLE = ALWAYS
+{
+  -1:* ::
+} -1
+INT out2D_every "How often to do 2D output, overrides out_every" STEERABLE = ALWAYS
+{
+  -1:* ::
+} -1
+INT out3D_every "How often to do 3D output, overrides out_every" STEERABLE = ALWAYS
+{
+  -1:* ::
+} -1
+
+####################
+# Output directories
+####################
+STRING outdir1D "Name of IO 1D output directory, overrides outdir"
+{
+  .* :: A regex which matches everything
+} "."
+STRING outdir2D "Name of IO 2D output directory, overrides outdir"
+{
+  .* :: A regex which matches everything
+} "."
+STRING outdir3D "Name of IO 3D output directory, overrides outdir"
+{
+  .* :: A regex which matches everything
+} "."
+
+
+#####################
+# Variables to output
+#####################
+STRING out1D_vars "Variables to output in 1D HDF5 file format" STEERABLE = ALWAYS
+{
+  .* :: A regex which matches everything
+} ""
+STRING out2D_vars "Variables to output in 2D HDF5 file format" STEERABLE = ALWAYS
+{
+  .* :: A regex which matches everything
+} ""
+STRING out3D_vars "Variables to output in 3D HDF5 file format" STEERABLE = ALWAYS
+{
+  .* :: A regex which matches everything
+} ""
+
+##############################################
+# Downsampling parameters specific to IOHDF5
+##############################################
+# 1D 
+######
+STRING origin1D "Default origin"
+{
+  .* :: "Comma separated list of positive integer values"
+} "0,0,0"
+STRING downsampling1D "Default downsampling"
+{
+  .* :: "Comma separated list of positive integer values"
+} "1,1,1"
+STRING length1D "Default length of the 1D hyperslab"
+{
+  .* :: "Comma separated list of integer values"
+} "-1,-1,-1"   
+
+######
+# 2D 
+######
+STRING origin2D "Default origin 2D"
+{
+  .* :: "Comma separated list of positive integer values"
+} "0,0,0"
+STRING downsampling2D "Default downsampling 2D"
+{
+  .* :: "Comma separated list of positive integer values"
+} "1,1,1"
+STRING length2D "Default length 2D"
+{
+  .* :: "Comma separated list of integer values"
+} "-1,-1,-1" 
+
+######
+# 3D 
+######
+STRING origin3D "Default origin 3D"
+{
+  .* :: "Comma separated list of positive integer values"
+} "0,0,0"
+STRING downsampling3D "Default downsampling 3D"
+{
+  .* :: "Comma separated list of positive integer values"
+} "1,1,1"
+STRING length3D "Default length of the 3D hyperslab"
+{
+  .* :: "Comma separated list of integer values 3D"
+} "-1,-1,-1"
+
+######
+# 4D 
+######
+STRING origin4D "Default origin"
+{
+  .* :: "Comma separated list of positive integer values"
+} "0,0,0"
+STRING downsampling4D "Default downsampling"
+{
+  .* :: "Comma separated list of positive integer values"
+} "1,1,1"
+STRING length4D "Default length of the hyperslab to stream"
+{
+  .* :: "Comma separated list of integer values"
+} "-1,-1,-1"
+
+##########
+# General 
+##########
+STRING origin "Default origin"
+{
+  .* :: "Comma separated list of positive integer values"
+} "0,0,0"
+STRING downsampling "Default downsampling"
+{
+  .* :: "Comma separated list of positive integer values"
+} "1,1,1"
+STRING length "Default length of the hyperslab to stream"
+{
+  .* :: "Comma separated list of integer values"
+} "-1,-1,-1" 
+
+#########################
+# Checkpointing requested
+#########################
+BOOLEAN checkpoint "Do checkpointing with HDF5"
+{
+} "no"
+
+
+#############################################################################
+### import IOUtil parameters
+#############################################################################
+shares: IO
+
+####################
+# Output directories
+####################
+USES STRING outdir ""
+{
+}
+
+
+########################
+# How often to do output
+########################
+USES INT out_every ""
+{
+  : ::
+}
+
+
+################
+# various things
+################
+USES BOOLEAN verbose ""
+{
+}
+USES BOOLEAN print_timing_info ""
+{
+}
+USES BOOLEAN out3D_datestamp ""
+{
+}
+USES BOOLEAN out3D_parameters ""
+{
+}
+USES BOOLEAN out2D_datestamp ""
+{
+}
+USES BOOLEAN out2D_parameters ""
+{
+}
+USES BOOLEAN out1D_datestamp ""
+{
+}
+USES BOOLEAN out1D_parameters ""
+{
+}
+
+#######################
+# Specific to 1/2/3D output
+#######################
+USES KEYWORD out3D_mode ""
+{
+}
+USES INT out3D_procs ""
+{
+ : :: 
+} 
+USES BOOLEAN out3D_septimefiles ""
+{
+}
+USES BOOLEAN out3D_unchunked ""
+{
+}
+
+USES KEYWORD out2D_mode ""
+{
+}
+USES INT out2D_procs ""
+{
+ : :: 
+} 
+USES BOOLEAN out2D_septimefiles ""
+{
+}
+USES BOOLEAN out2D_unchunked ""
+{
+}
+USES KEYWORD out1D_mode ""
+{
+}
+USES INT out1D_procs ""
+{
+ : :: 
+} 
+USES BOOLEAN out1D_septimefiles ""
+{
+}
+USES BOOLEAN out1D_unchunked ""
+{
+}
+##############################################
+# Downsampling parameters based on IOUTIL
+##############################################
+USES INT out3D_downsample_x ""
+{
+  : ::
+}
+USES INT out3D_downsample_y ""
+{
+  : ::
+}
+USES INT out3D_downsample_z ""
+{
+  : ::
+}
+
+###################################
+# Checkpointing/recovery parameters
+###################################
+USES BOOLEAN checkpoint_ID ""
+{
+}
+USES BOOLEAN checkpoint_keep_all ""
+{
+}
+USES KEYWORD recover ""
+{
+}
+USES INT checkpoint_every ""
+{
+  : ::
+}
+USES INT checkpoint_keep ""
+{
+  : ::
+}
+USES STRING checkpoint_file ""
+{
+}
+USES STRING checkpoint_ID_file ""
+{
+}
+USES STRING recover_file ""
+{
+}
+USES STRING checkpoint_dir ""
+{
+}
+USES STRING recovery_dir ""
+{
+}
diff --git a/schedule.ccl b/schedule.ccl
new file mode 100644
index 0000000..03fd80f
--- /dev/null
+++ b/schedule.ccl
@@ -0,0 +1,46 @@
+# Schedule definitions for thorn IOHDF5
+# $Header$
+
+########################################################################
+### register IOHDF5 routines
+########################################################################
+schedule IOHDF5_Startup at STARTUP after (IOUtil_Startup Driver_Startup)
+{
+  LANG:C
+} "IOHDF5 startup routine"
+
+
+########################################################################
+### register checkpointing routines
+########################################################################
+if (checkpoint && checkpoint_ID)
+{
+  schedule IOHDF5_InitialDataDumpGH at CCTK_CPINITIAL
+  {
+    LANG:C
+  } "Initial data checkpoint routine"
+}
+
+if (checkpoint && checkpoint_every > 0)
+{
+  schedule IOHDF5_ConditionallyDumpGH at CCTK_CHECKPOINT
+  {
+    LANG:C
+  } "Regular checkpoint routine"
+}
+
+#if (checkpoint)
+#{
+#  schedule IOHDF5_TerminationDumpGH at CCTK_TERMINATE BEFORE Driver_Terminate
+#  {
+#    LANG:C
+#  } "Termination checkpoint routine"
+#}
+
+if (! CCTK_Equals (recover, "no") && *recover_file)
+{
+  schedule IOHDF5_RecoverParameters at CCTK_RECOVER_PARAMETERS
+  {
+    LANG:C
+  } "Parameter recovery routine"
+}
diff --git a/src/ChooseOutput.c b/src/ChooseOutput.c
new file mode 100644
index 0000000..39ac818
--- /dev/null
+++ b/src/ChooseOutput.c
@@ -0,0 +1,145 @@
+/*@@
+  @file ChooseOutput.c
+  @author Gabrielle Allen
+  @date July 6 2000
+  @desc
+  Choose what data to write for different IO methods in IOHDF5
+  @enddesc 
+
+  @history
+  @hauthor 
+  @hdesc 
+  @version $Header$
+@@*/
+
+#include <stdlib.h>
+
+#include "cctk.h"
+#include "cctk_Arguments.h"
+#include "cctk_Parameters.h"
+#include "CactusBase/IOUtil/src/ioGH.h"
+#include "ioFlexGH.h"
+
+static char *rcsid = "$Header$";
+CCTK_FILEVERSION(CactusPUGHIO_IOHDF5_ChooseOutput_c)
+
+
+/*@@
+  @routine IOFlexIO_Choose2D
+  @author Gabrielle Allen
+  @date July 6 2000
+  @desc
+  Use parameters to choose the 2D planes through the output data. 
+  @enddesc
+
+  @history
+  @hauthor 
+  @hdesc 
+  @version $Header$
+@@*/
+
+void IOFlexIO_Choose2D(CCTK_ARGUMENTS)
+{
+  DECLARE_CCTK_PARAMETERS
+
+  int i;
+  int maxdim;
+  flexioGH *myGH;                   /* FlexIO extension handle */
+  int *origin_index;           /* Specify output planes by indices */
+  CCTK_REAL *origin_phys;           /* Specify output planes by coordinates */
+
+  myGH     = (flexioGH *) cctkGH->extensions 
+    [CCTK_GHExtensionHandle ("IOFlexIO")];
+
+  /* Set up lines to be output */
+  maxdim = CCTK_MaxDim();
+  origin_phys     = (CCTK_REAL *)calloc(maxdim,sizeof(CCTK_REAL ));
+  origin_index    = (int  *)calloc(maxdim,sizeof(int ));
+    
+  /* Set parameters using 
+     1. Indices from IOFlexIO
+     2. Indices from IOUtil
+     3. Coords from IOFlexIO
+     4. Coords from IOUtil
+  */
+
+  if (CCTK_ParameterQueryTimesSet("out2D_yzplane_xi","IOFlexIO")>0)
+  {
+    origin_index[0] = out2D_yzplane_xi;
+  }
+  else if (CCTK_ParameterQueryTimesSet("out_yzplane_xi","IOUtil")>0)
+  {
+    origin_index[0] = out_yzplane_xi;
+  }
+  else
+  {
+    origin_index[0]  = -1;
+    if (CCTK_ParameterQueryTimesSet("out2D_yzplane_x","IOFlexIO")>0)
+    {
+      origin_phys[0] = out2D_yzplane_x;
+    }
+    else
+    {
+      origin_phys[0] = out_yzplane_x;
+    }
+  }
+
+  if (CCTK_ParameterQueryTimesSet("out2D_xzplane_yi","IOFlexIO")>0)
+  {
+    origin_index[1] = out2D_xzplane_yi;
+  } 
+  else if (CCTK_ParameterQueryTimesSet("out_xzplane_yi","IOUtil")>0)
+  {
+    origin_index[1] = out_xzplane_yi;
+  }
+  else
+  {
+    origin_index[1]  = -1;
+    if (CCTK_ParameterQueryTimesSet("out2D_xzplane_y","IOFlexIO")>0)
+    {
+      origin_phys[1] = out2D_xzplane_y;
+    }
+    else
+    {
+      origin_phys[1] = out_xzplane_y;
+    }
+  }
+
+  if (CCTK_ParameterQueryTimesSet("out2D_xyplane_zi","IOFlexIO")>0)
+  {
+    origin_index[2] = out2D_xyplane_zi;
+  }
+  else if (CCTK_ParameterQueryTimesSet("out_xyplane_zi","IOUtil")>0)
+  {
+    origin_index[2] = out_xyplane_zi;
+  }
+  else
+  {
+    origin_index[2]  = -1;
+    if (CCTK_ParameterQueryTimesSet("out2D_xyplane_z","IOFlexIO")>0)
+    {
+      origin_phys[2] = out2D_xyplane_z;
+    }
+    else
+    {
+      origin_phys[2] = out_xyplane_z;
+    }
+  }
+
+  if (myGH) 
+  {
+    for (i=1;i<=CCTK_MaxDim();i++)
+    {
+      IOUtil_2DPlanes (cctkGH, i, origin_index, 
+		       origin_phys, myGH->sp2xyz[i-1]);
+    }
+  }
+  else
+  {
+    CCTK_WARN(1,"IOFlexIO_Choose2D: FlexIO GH extension not found");
+  }
+
+  return;
+}
+
+
diff --git a/src/DumpGH.c b/src/DumpGH.c
new file mode 100644
index 0000000..74d7729
--- /dev/null
+++ b/src/DumpGH.c
@@ -0,0 +1,494 @@
+ /*@@
+   @file      DumpGH.c
+   @date      Wed Jun 10 14:13:35 1998
+   @author    Paul Walker
+   @desc 
+   DumpGH dumps an entire grid hierarchy (except for 1D arrays) to a 
+   checkpoint file. This file also contains the different wrappers for
+   IOHDF5_DumpGH: in case of initial data, termination or regular checkpointing
+   @enddesc 
+   @version $Id$
+ @@*/
+
+
+#include <stdio.h>
+#include <string.h>
+#include <stdlib.h>
+
+#include "cctk.h"
+#include "cctk_Parameters.h"
+#include "CactusBase/IOUtil/src/ioGH.h"
+#include "ioHDF5GH.h"
+
+/* the rcs ID and its dummy funtion to use it */
+static char *rcsid = "$Id$";
+CCTK_FILEVERSION(CactusPUGHIO_IOHDF5_DumpGH_c)
+
+
+/* local function prototypes */
+void IOHDF5i_DumpParameters (cGH *GH, hid_t group);
+
+
+ /*@@
+   @routine    IOHDF5_ConditionallyDumpGH
+   @date       Fri Aug 21 14:38:25 1998
+   @author     Gabrielle Allen
+   @desc 
+     registered as CCTK_CHECKPOINT, checks if it's time for 
+     checkpointing, sets the checkpoint type and calls IOHDF5_DumpGH 
+   @enddesc 
+   @calledby   CCTK Scheduler
+   @history 
+ 
+   @endhistory 
+   @var        GH
+   @vdesc      Pointer to CCTK grid hierarchy
+   @vtype      cGH
+   @vio        in
+   @endvar
+@@*/
+
+void IOHDF5_ConditionallyDumpGH (cGH *GH)
+{
+  DECLARE_CCTK_PARAMETERS
+
+  if (checkpoint_every > 0 && GH->cctk_iteration % checkpoint_every == 0) {
+    if (verbose) {
+      CCTK_INFO("------------------------------------------------------------");
+      CCTK_VInfo (CCTK_THORNSTRING, "Dumping periodic checkpoint file at "
+                  "iteration %d", GH->cctk_iteration);
+    }
+    IOHDF5_DumpGH (GH, CP_EVOLUTION_DATA);
+  }
+}
+
+
+ /*@@
+   @routine    IOHDF5_TerminationDumpGH
+   @date       Fri Aug 21 14:40:21 1998
+   @author     Gabrielle Allen
+   @desc 
+     This routine is registered as CCTK_TERMINATE, it checks if a
+     termination signal is raised (that is: global var cactus_terminate
+     set to TERMINATION_RAISED_BRDCAST) and checkpoints all available 
+     grid hierarchies.
+   @enddesc 
+   @calledby   CCTK Scheduler
+   @history 
+ 
+   @endhistory 
+   @var        GH
+   @vdesc      Pointer to CCTK grid hierarchy
+   @vtype      cGH
+   @vio        in
+   @endvar
+@@*/
+
+void IOHDF5_TerminationDumpGH (cGH *GH)
+{
+/*** FIXME: no cactus_terminate flag anymore ? ***/
+#if 0
+  if (cactus_terminate == TERMINATION_RAISED_BRDCAST) {
+    IOHDF5_DumpGH (GH, CP_EVOLUTION_DATA); 
+  }
+#else
+  CCTK_WARN (1, "IOHDF5_TerminationDumpGH() not yet implemented !");
+#endif
+}
+
+ /*@@
+   @routine    IOHDF5_InitialDataDumpGH
+   @date       Fri Aug 21 14:46:28 1998
+   @author     Gerd Lanfermann
+   @desc 
+     This routine dumps the initial data of the GHs, registered 
+     as CCTK_CPINITIAL; sets 'called_from' flag to CP_INITIAL_DATA
+     and calls IOHDF5_DumpGH.
+     BoxInBox needs special treatment since for now.
+   @enddesc 
+   @calledby   CCTK Scheduler
+   @history 
+ 
+   @endhistory 
+   @var        GH
+   @vdesc      Pointer to CCTK grid hierarchy
+   @vtype      cGH
+   @vio        in
+   @endvar
+@@*/
+
+void IOHDF5_InitialDataDumpGH (cGH *GH)
+{
+
+/*** FIXME ***/
+#if 0
+#ifdef THORN_BOXINBOX
+  /* This will go once we reorganize the CINitial calling structure! */
+  /* or have a function to deregister a rfr routine */
+  /* BoxinBox does its own: get maxlevel (once!) and do dumpGH for all GH*/
+  /* when this is called by the highest box */
+  if (Contains("boxinbox","yes")) {
+    int l;
+    cGH *helperGH;
+    static int maxlev = 0;
+
+    if (maxlev == 0) while (GHbyLevel(0,maxlev+1,0)) maxlev++;
+    if (maxlev == 0) return; 
+    
+    if (GH->level == maxlev) 
+      for (l = 0; l <= maxlev; l++) {
+        CCTK_VInfo (CCTK_THORNSTRING, "Dumping BoxinBox level %d of %d",
+                    l, maxlev);
+        IOHDF5_DumpGH (GHbyLevel (0, l, 0), CP_INITIAL_DATA);
+      }
+    return;
+  }
+#endif
+#endif
+  IOHDF5_DumpGH (GH, CP_INITIAL_DATA);
+}
+
+
+ /*@@
+   @routine    IOHDF5i_DumpParameters
+   @date       Thu Jan 20 2000
+   @author     Thomas Radke
+   @desc 
+     Gets the parameters of all active implementations as a single string
+     and writes it as an attribute into a group of an already opened HDF5 file.
+   @enddesc 
+   @calledby   IOHDF5_Write3D, IOHDF5_DumpGH
+   @history 
+ 
+   @endhistory 
+   @var        GH
+   @vdesc      Pointer to CCTK grid hierarchy
+   @vtype      cGH
+   @vio        in
+   @endvar
+   @var        group
+   @vdesc      the group where to dump the parameters to
+   @vtype      hid_t
+   @vio        in
+   @endvar
+@@*/
+void IOHDF5i_DumpParameters (cGH *GH, hid_t group)
+{
+  ioHDF5GH *myGH;
+  char *parameters;
+
+
+  /* Get the parameter string buffer */
+  parameters = IOUtil_GetAllParameters (GH);
+
+  if (parameters) {
+    /* Get the handle for IOHDF5 extensions */
+    myGH = (ioHDF5GH *) GH->extensions [CCTK_GHExtensionHandle ("IOHDF5")]; 
+
+    WRITE_ATTRIBUTE (GLOBAL_PARAMETERS, parameters, group,
+                     myGH->scalarDataspace, 0, myGH->IOHDF5_STRING);
+    free (parameters);
+  }
+}
+
+
+ /*@@
+   @routine    IOHDF5i_DumpGHExtensions
+   @date       Thu Jan 20 2000
+   @author     Thomas Radke
+   @desc 
+     Dumps the important variables from the grid hierarchy, PUGH, and IO
+     into a group of an already opened HDF5 file.
+   @enddesc 
+   @calledby   IOHDF5_Write3D, IOHDF5_DumpGH
+   @history 
+ 
+   @endhistory 
+   @var        GH
+   @vdesc      Pointer to CCTK grid hierarchy
+   @vtype      cGH
+   @vio        in
+   @endvar
+   @var        group
+   @vdesc      the group where to dump the GH extensions to
+   @vtype      hid_t
+   @vio        in
+   @endvar
+@@*/
+void IOHDF5i_DumpGHExtensions (cGH *GH, hid_t group)
+{
+  int attrValue, main_loop_index;
+  ioGH *ioUtilGH;
+  ioHDF5GH *myGH;
+  extern int CCTK_MainLoopIndex (void);
+
+  /* Get the handles for IOUtil and IOHDF5 extensions */
+  ioUtilGH = (ioGH *) GH->extensions [CCTK_GHExtensionHandle ("IO")];
+  myGH = (ioHDF5GH *) GH->extensions [CCTK_GHExtensionHandle ("IOHDF5")]; 
+
+  main_loop_index = CCTK_MainLoopIndex ();
+
+  attrValue = CCTK_nProcs (GH);
+  WRITE_ATTRIBUTE ("nprocs", &attrValue, group,
+                   myGH->scalarDataspace, 0, H5T_NATIVE_INT);
+  WRITE_ATTRIBUTE ("ioproc_every", &ioUtilGH->ioproc_every, group,
+                   myGH->scalarDataspace, 0, H5T_NATIVE_INT);
+  WRITE_ATTRIBUTE ("unchunked", &ioUtilGH->unchunked, group,
+                   myGH->scalarDataspace, 0, H5T_NATIVE_INT);
+  WRITE_ATTRIBUTE ("cctk_time", &GH->cctk_time, group,
+                   myGH->scalarDataspace, 0, IOHDF5_REAL);
+  WRITE_ATTRIBUTE ("cctk_iteration", &GH->cctk_iteration, group,
+                   myGH->scalarDataspace, 0, H5T_NATIVE_INT);
+  WRITE_ATTRIBUTE ("main_loop_index", &main_loop_index, group,
+                   myGH->scalarDataspace, 0, H5T_NATIVE_INT);
+
+}
+
+
+ /*@@
+   @routine    IOHDF5_DumpGH
+   @date       Fri Aug 21 15:13:13 1998
+   @author     Paul Walker
+   @desc 
+     The heart of checkpointing. Called by the different wrappers, this
+     routines get the appropriate filename by IOUtil_PrepareFilename() and 
+     then dumps away using the dump routines from IO ...
+   @enddesc 
+   @calledby   IOHDF5_ConditionallyDumpGH, IOHDF5_TerminationDumpGH,
+               IOHDF5_InitialDataDumpGH
+   @history 
+   @hdate Oct 4 1998 @hauthor Gabrielle Allen
+   @hdesc Removed code which checked and reset unchunked, assume that this 
+          is done when the variable is first set.
+   @hdate Nov 4 1998 @hauthor Gabrielle Allen
+   @hdesc Do a forced synchronization before checkpointing
+   @hdate Apr 16 1999 @hauthor Thomas Radke
+   @hdesc Removed forced sync before checkpointing (just do a normal sync)
+          Introduced a ring buffer for keeping last <checkpoint_keep> files
+   @endhistory 
+   @var        GH
+   @vdesc      Pointer to CCTK grid hierarchy
+   @vtype      cGH
+   @vio        in
+   @endvar
+   @var        called_from
+   @vdesc      flag indicating where this routine was called from
+               (either CHECKPOINT_ID, CHECKPOINT, or filereader)
+   @vtype      int
+   @vio        in
+   @endvar
+@@*/
+
+void IOHDF5_DumpGH (cGH *GH, int called_from)
+{
+  DECLARE_CCTK_PARAMETERS
+  char dumpfname [1024], tmpfname [1024];
+  hid_t iof;
+  int index,idim;
+  int timelevel, current_timelevel;
+  int *old_downsample;
+  int old_out_single;
+  ioGH *ioUtilGH;
+  ioHDF5GH *myGH;
+  ioHDF5Geo_t geo;
+  static char **dumpfnames = NULL;  /* dump filename ring buffer */
+  static int findex = 0;            /* index into ring buffer */
+
+
+  /* Get the handles for IOUtil and IOHDF5 extensions */
+  if ((index = CCTK_GHExtensionHandle ("IO")) < 0)
+    return;
+  ioUtilGH = (ioGH *) GH->extensions [index];
+  if ((index = CCTK_GHExtensionHandle ("IOHDF5")) < 0)
+    return;
+  myGH = (ioHDF5GH *) GH->extensions [index];
+
+  /* allocate the ring buffer for filenames */
+  if (! dumpfnames)
+    dumpfnames = (char **) calloc (checkpoint_keep, sizeof (char *));
+
+  /* disable downsampling after saving original downsampling params */
+  old_downsample = (int *) malloc (CCTK_MaxDim () * sizeof (int));
+  for (index = 0; index < CCTK_MaxDim (); index++) {
+    old_downsample [index] = ioUtilGH->downsample [index];
+    ioUtilGH->downsample [index] = 1;
+  }
+
+  /* disable output in single precision */
+  old_out_single = ioUtilGH->out_single;
+  ioUtilGH->out_single = 0;
+
+  /* start the total timer */
+  if (myGH->print_timing_info)
+    CCTK_TimerStartI (myGH->timers[2]);
+
+  /* sync all groups */
+  for (index = 0; index < CCTK_NumGroups (); index++)
+    if (CCTK_IsImplementationActive (CCTK_ImpFromVarI (
+                                       CCTK_FirstVarIndexI (index))))
+      CCTK_SyncGroupI (GH, index);
+
+  /* get the base filename ... */
+  IOUtil_PrepareFilename (GH, NULL, dumpfname, called_from,
+                          CCTK_MyProc (GH) / ioUtilGH->ioproc_every,
+                          ioUtilGH->unchunked);
+
+  /* ... and append the extension */
+  sprintf (tmpfname,  "%s.tmp.h5", dumpfname);
+  sprintf (dumpfname, "%s.h5",     dumpfname);
+
+  /* Now open the file */
+  if (CCTK_MyProc (GH) == ioUtilGH->ioproc) {
+    if (verbose)
+      CCTK_VInfo (CCTK_THORNSTRING, "Creating checkpoint file '%s'", tmpfname);
+
+    iof = H5Fcreate (tmpfname, H5F_ACC_TRUNC, H5P_DEFAULT, H5P_DEFAULT);
+    if (iof < 0) {
+      CCTK_VWarn (1, __LINE__, __FILE__, CCTK_THORNSTRING,
+                  "Can't open checkpoint file '%s'. Checkpointing is skipped",
+                  tmpfname);
+      return;
+    }
+  } else
+    iof = -1;
+
+  /* Great; Now start dumping away! */
+
+  /* start timer for dumping parameters */
+  if (myGH->print_timing_info) {
+    CCTK_TimerResetI (myGH->timers[0]);
+    CCTK_TimerStartI (myGH->timers[0]);
+  }
+
+  if (iof >= 0) {
+    hid_t group;
+
+    /* all GH extension variables and parameter stuff which is not
+       specific to any dataset goes into dedicated groups */
+    if (out3D_parameters) {
+      if (verbose)
+        CCTK_INFO ("Dumping Parameters ...");
+
+      CACTUS_IOHDF5_ERROR (group = H5Gcreate (iof, GLOBAL_PARAMETERS_GROUP, 0));
+      IOHDF5i_DumpParameters (GH, group);
+      CACTUS_IOHDF5_ERROR (H5Gclose (group));
+    }
+
+    if (verbose)
+      CCTK_INFO ("Dumping GH extensions ...");
+
+    CACTUS_IOHDF5_ERROR (group = H5Gcreate (iof, GHEXTENSIONS_GROUP, 0));
+    IOHDF5i_DumpGHExtensions (GH, group);
+    CACTUS_IOHDF5_ERROR (H5Gclose (group));
+  }
+
+  /* stop parameter timer and start timer for dumping datasets */
+  if (myGH->print_timing_info) {
+    CCTK_TimerStopI (myGH->timers[0]);
+    CCTK_TimerResetI (myGH->timers[1]);
+    CCTK_TimerStartI (myGH->timers[1]);
+  }
+
+  /* Set the geometry variables manually, do not use the 
+     geometry structures set by parameters; vdim,sdim are set below */
+  for (idim=0; idim<IOHDF5_MAXDIM;idim++) {
+    geo.direction[idim]  = idim;
+    geo.slab_start[idim] = 0;
+    geo.length[idim]     = -1;
+    geo.downs[idim]      = 1;
+  }
+
+  if (verbose)
+    CCTK_INFO ("Dumping variables ...");
+
+  /* ... now the variables */
+  for (index = 0; index < CCTK_NumVars (); index++) {
+    char oldFlag;
+    const char *thorn, *impl = CCTK_ImpFromVarI (index);
+
+    /* find out the thorn implementing variable with index */
+    thorn = CCTK_ImplementationThorn (impl);
+    if (! thorn)
+      thorn = impl;
+
+    /* let only variables pass which belong to an active thorn and
+       have storage assigned */
+    if (! CCTK_IsThornActive (thorn) ||
+        ! CCTK_QueryGroupStorageI (GH, CCTK_GroupIndexFromVarI (index)))
+      continue;
+
+    if (verbose && iof >= 0)
+      CCTK_VInfo (CCTK_THORNSTRING, "  %s", CCTK_VarName (index));
+
+    /* get the current timelevel */
+    current_timelevel = CCTK_NumTimeLevelsFromVarI (index) - 1;
+    if (current_timelevel > 0)
+      current_timelevel--;
+    
+    /* Set dimension information based on the variable */
+    geo.vdim = CCTK_GroupDimFromVarI(index);
+    geo.sdim = geo.vdim;
+
+    /* reset this flag because checkpoint files are always created anew */
+    oldFlag = myGH->checkForExistingObjects [index];
+    myGH->checkForExistingObjects [index] = 0;
+
+    /* now dump all timelevels up to the current */
+    for (timelevel = 0; timelevel <= current_timelevel; timelevel++)
+      IOHDF5_DumpVar (GH, index, timelevel, &geo, iof);
+
+    /* restore the original value of the flag */
+    myGH->checkForExistingObjects [index] = oldFlag;
+
+  } /* end of loop over all variables */
+
+  /* stop timer for dumping datasets */
+  if (myGH->print_timing_info)
+    CCTK_TimerStopI (myGH->timers[1]);
+
+  if (iof >= 0)
+    CACTUS_IOHDF5_ERROR (H5Fclose (iof));
+  
+  if (CCTK_MyProc (GH) == ioUtilGH->ioproc) {
+
+    /* rename successfully created temporary file
+       and overwrite the old checkpoint file */
+    if (verbose)
+      CCTK_VInfo (CCTK_THORNSTRING, "Closing and renaming checkpoint file "
+                  "into '%s'", dumpfname);
+
+    if (rename (tmpfname, dumpfname))
+      CCTK_VWarn (0, __LINE__, __FILE__, CCTK_THORNSTRING,
+                  "Could not rename temporary checkpoint file '%s' to '%s'",
+                  tmpfname, dumpfname);
+
+    /* delete the oldest checkpoint file if checkpoint_keep_all isn't set
+       and put the new filename into the ring buffer */
+    if (dumpfnames [findex]) {
+      if (! checkpoint_keep_all)
+        remove (dumpfnames [findex]);
+      free (dumpfnames [findex]);
+    }
+    dumpfnames [findex] = strdup (dumpfname);
+    findex = (findex + 1) % checkpoint_keep;
+  }
+
+  /* restore output precision flag */
+  ioUtilGH->out_single = old_out_single;
+
+  /* restore original downsampling params */
+  memcpy (ioUtilGH->downsample, old_downsample, CCTK_MaxDim () * sizeof (int));
+  free (old_downsample);
+
+  /* stop total checkpoint timer and print timing info */
+  if (myGH->print_timing_info) {
+    const char *timer_descriptions [3] = {"Time to dump parameters: ",
+                                          "Time to dump datasets:   ",
+                                          "Total time to checkpoint:"};
+
+
+    CCTK_TimerStopI (myGH->timers[2]);
+    IOUtil_PrintTimings ("Timing information for checkpointing in IOHDF5:", 3,
+                         myGH->timers, timer_descriptions);
+  }
+}
diff --git a/src/DumpVar.c b/src/DumpVar.c
new file mode 100644
index 0000000..dd616af
--- /dev/null
+++ b/src/DumpVar.c
@@ -0,0 +1,1070 @@
+/*@@
+   @file      DumpVar.c
+   @date      Fri May 21 1999
+   @author    Thomas Radke
+   @desc 
+   Do the actual writing of a variable, for output or for checkpointing.
+   @enddesc 
+   @history
+   @hauthor Thomas Radke @hdate May 21 1999
+   @hdesc Just copied from thorn FlexIO.
+   @hendhistory
+ @@*/
+
+
+#include "cctk.h"
+#include "cctk_Parameters.h"
+#include "CactusPUGH/PUGH/src/include/pugh.h"
+#include "CactusPUGH/PUGHSlab/src/PUGHSlab.h" 
+#include "CactusBase/IOUtil/src/ioGH.h"
+#include "ioHDF5GH.h"
+
+#include <stdio.h>
+#include <stdlib.h>
+#ifdef SGI
+#include <time.h>
+#endif
+
+#define IOTAGBASE 20000 /* This may break on more than 2000 processors */
+
+#define IOHDF5_DEBUG
+
+/* info structure describing how to dump a given CCTK variable type */
+typedef struct {
+  int iohdf5_type;              /* the HDF5 type to use */
+  int element_size;             /* size of a single data element */
+#ifdef CCTK_MPI
+  MPI_Datatype mpi_type;        /* the data type for MPI communication */
+#endif
+} dumpInfo_t;
+
+static char *char_time_date = NULL;
+
+/* local function prototypes */
+static void IOHDF5_DumpGS (cGH *GH, int index, int timelevel, hid_t iof,
+                           int iohdf5_type);
+
+static int  IOHDF5_DumpGA (cGH *GH, int index, int timelevel, ioHDF5Geo_t *slab_geo, 
+			   hid_t iof,  dumpInfo_t *info);
+
+static int  IOHDF5_getDumpData_ND (cGH *GH, int index, int timelevel, ioHDF5Geo_t *slab_geo, 
+				   void **outme, int *free_outme, 
+				   CCTK_INT *geom, int element_size);
+
+static void IOHDF5_procDump (cGH *GH, int index, int timelevel, ioHDF5Geo_t *slab_geo, 
+			     void *outme, CCTK_INT *geom, int proc, 
+			     int iohdf5_type, hid_t iof);
+
+static void IOHDF5_AddCommonAttributes (cGH *GH, int index, int timelevel, 
+					CCTK_INT *global_shape, 
+					hid_t dataset);
+
+static void IOHDF5_AddCommonAttributes_ND (cGH *GH, int index, ioHDF5Geo_t *slab_geo, 
+					   CCTK_INT *global_shape, hid_t dataset);
+
+#ifdef CCTK_MPI
+#ifdef HAVE_PARALLEL
+static void IOHDF5_collectiveDump (cGH *GH, int index, int timelevel, ioHDF5Geo_t *slab_geo,
+				   void *outme, CCTK_INT *geom, 
+				   int hdf5io_type, hid_t iof);
+#endif /* HAVE_PARALLEL */
+#endif /* MPI */
+
+
+
+/*@@
+  @routine    IOHDF5_DumpVar
+  @date       16 Apr 1999
+  @author     Thomas Radke
+  @desc
+              Generic dump routine, just calls the appropriate dump routine
+  @enddesc
+  @calledby   IOHDF5_DumpGH IOHDF5_Write3D
+  @history
+  @endhistory
+  @var        GH
+  @vdesc      Pointer to CCTK grid hierarchy
+  @vtype      cGH
+  @vio        in
+  @endvar
+  @var        index
+  @vdesc      global index of the variable to be dumped
+  @vtype      int
+  @vio        in
+  @endvar
+  @var        timelevel
+  @vdesc      the timelevel to store
+  @vtype      int
+  @vio        in
+  @endvar
+  @var        iof
+  @vdesc      the HDF5 file handle
+  @vtype      hid_t
+  @vio        in
+  @endvar
+@@*/
+int IOHDF5_DumpVar (cGH *GH, int index, int timelevel, ioHDF5Geo_t *geo, hid_t iof)
+{
+  pGH *pughGH;
+  ioHDF5GH *h5GH;
+  ioGH *ioUtilGH;
+  int variable_type;
+  dumpInfo_t info;
+  int retval = 0;
+
+  /* Get the handle for PUGH, IOUtil, and IOHDF5 extensions */
+  pughGH = PUGH_pGH (GH);
+  ioUtilGH = (ioGH *) GH->extensions [CCTK_GHExtensionHandle ("IO")];
+  h5GH = (ioHDF5GH *) GH->extensions [CCTK_GHExtensionHandle ("IOHDF5")];
+
+  variable_type = CCTK_VarTypeI (index);
+
+  switch (variable_type) {
+    case CCTK_VARIABLE_CHAR:
+      info.iohdf5_type = IOHDF5_CHAR;
+      info.element_size = sizeof (CCTK_CHAR);
+#ifdef CCTK_MPI
+      info.mpi_type = PUGH_MPI_CHAR;
+#endif
+      break;
+    case CCTK_VARIABLE_INT:
+      info.iohdf5_type = IOHDF5_INT;
+      info.element_size = sizeof (CCTK_INT);
+#ifdef CCTK_MPI
+      info.mpi_type = PUGH_MPI_INT;
+#endif
+      break;
+    case CCTK_VARIABLE_REAL: 
+      info.iohdf5_type  = ioUtilGH->out_single ? IOHDF5_REAL4 : IOHDF5_REAL;
+      info.element_size = ioUtilGH->out_single ?
+                              sizeof (CCTK_REAL4) : sizeof (CCTK_REAL);
+#ifdef CCTK_MPI
+      info.mpi_type     = ioUtilGH->out_single ? PUGH_MPI_REAL4 : PUGH_MPI_REAL;
+#endif
+      break;
+    case CCTK_VARIABLE_COMPLEX:
+      info.iohdf5_type  = h5GH->IOHDF5_COMPLEX;
+      info.element_size = sizeof (CCTK_COMPLEX);
+#ifdef CCTK_MPI
+      info.mpi_type     = pughGH->PUGH_mpi_complex;
+#endif
+      break;
+    default:
+      CCTK_VWarn (1, __LINE__, __FILE__, CCTK_THORNSTRING,
+                  "Unsupported variable type %d", variable_type);
+      return(-1);
+  }
+
+  switch (CCTK_GroupTypeFromVarI (index)) {
+    case GROUP_SCALAR:
+      IOHDF5_DumpGS (GH, index, timelevel, iof, info.iohdf5_type);
+      break;
+    case GROUP_ARRAY:
+    case GROUP_GF:
+      retval = IOHDF5_DumpGA (GH, index, timelevel, geo, iof, &info);
+      break;
+    default:
+      CCTK_VWarn (1, __LINE__, __FILE__, CCTK_THORNSTRING,
+                  "Invalid group type %d in IOHDF5_DumpVar",
+                  CCTK_GroupTypeFromVarI (index));
+      retval = -1;
+  }
+  return(retval);
+}
+
+
+/*@@
+  @routine    IOHDF5_DumpGS
+  @date       May 21 1999
+  @author     Thomas Radke
+  @desc       Dumps a SCALAR type variable into a HDF5 file
+  @enddesc 
+  @calledby   IOHDF5_DumpVar
+  @var        GH
+  @vdesc      Pointer to CCTK grid hierarchy
+  @vtype      cGH
+  @vio        in
+  @endvar
+  @var        index
+  @vdesc      global index of the variable to be dumped
+  @vtype      int
+  @vio        in
+  @endvar
+  @var        timelevel
+  @vdesc      the timelevel to store
+  @vtype      int
+  @vio        in
+  @endvar
+  @var        iof
+  @vdesc      the HDF5 file to dump to
+  @vtype      hid_t
+  @vio        in
+  @endvar
+  @var        iohdf5_type
+  @vdesc      the HDF5 datatype
+  @vtype      int
+  @vio        in
+  @endvar
+  @history 
+  @endhistory 
+
+@@*/
+static void IOHDF5_DumpGS (cGH *GH, int index, int timelevel, hid_t iof,
+                           int iohdf5_type)
+{
+  CCTK_INT global_shape [1] = {0};
+  ioGH *ioUtilGH;
+  ioHDF5GH *h5GH;
+  hid_t dataset, dataspace;
+  char *name, *datasetname;
+
+  /* Get the handles for IOHDF5 and IOUtil extensions */
+  ioUtilGH = (ioGH *) GH->extensions [CCTK_GHExtensionHandle ("IO")];
+  h5GH = (ioHDF5GH *) GH->extensions [CCTK_GHExtensionHandle ("IOHDF5")];
+
+  if (CCTK_MyProc (GH) != ioUtilGH->ioproc)
+    return;
+
+  CACTUS_IOHDF5_ERROR (dataspace = H5Screate (H5S_SCALAR));
+  name = CCTK_FullName (index);
+  datasetname = (char *) malloc (strlen (name) + 30);
+  sprintf (datasetname, "%s@%d@%d", name, GH->cctk_iteration, timelevel);
+  free (name);
+
+  /* if restart from recovery delete an already existing dataset
+     so that it can be created as anew */
+  if (h5GH->checkForExistingObjects [index]) {
+    CACTUS_IOHDF5_ERROR (H5Eset_auto (NULL, NULL));
+    H5Gunlink (iof, datasetname);
+    CACTUS_IOHDF5_ERROR(H5Eset_auto(h5GH->printErrorFn, h5GH->printErrorFnArg));
+  }
+  CACTUS_IOHDF5_ERROR (dataset = H5Dcreate (iof, datasetname,
+                       iohdf5_type, dataspace, H5P_DEFAULT));
+  free (datasetname);
+  CACTUS_IOHDF5_ERROR (H5Dwrite (dataset, iohdf5_type, H5S_ALL, H5S_ALL,
+                       H5P_DEFAULT,
+                       CCTK_VarDataPtrI (GH, timelevel, index)));
+    
+  IOHDF5_AddCommonAttributes (GH, index, timelevel, global_shape, dataset);
+
+  CACTUS_IOHDF5_ERROR (H5Dclose (dataset));
+  CACTUS_IOHDF5_ERROR (H5Sclose (dataspace));
+}
+
+
+/*@@
+  @routine    IOHDF5_DumpGA
+  @date       May 21 1999
+  @author     Paul Walker
+  @desc       Dumps a GA type variable into a HDF5 file
+  @enddesc 
+  @calledby   IOHDF5_DumpVar
+  @var        GH
+  @vdesc      Pointer to CCTK grid hierarchy
+  @vtype      cGH
+  @vio        in
+  @endvar
+  @var        index
+  @vdesc      global index of the variable to be dumped
+  @vtype      int
+  @vio        in
+  @endvar
+  @var        timelevel
+  @vdesc      the timelevel to store
+  @vtype      int
+  @vio        in
+  @endvar
+  @var        iof
+  @vdesc      the HDF5 file to dump to
+  @vtype      hid_t
+  @vio        in
+  @endvar
+  @var        info
+  @vdesc      info structure describing
+                - the HDF5 datatype to store
+                - the size of an element of variable
+                - the MPI datatype to communicate
+  @vtype      dumpInfo_t
+  @vio        in
+  @endvar
+
+  @history 
+  @endhistory 
+
+@@*/
+static int IOHDF5_DumpGA (cGH *GH, int index, int timelevel, ioHDF5Geo_t *slab_geo,  
+			   hid_t iof, dumpInfo_t *info)
+{
+  DECLARE_CCTK_PARAMETERS
+  ioGH *ioUtilGH;
+  pGH *pughGH;
+  int myproc;
+  int nprocs;
+  int dim;
+  CCTK_INT *geom; /* Lower bounds, sizes and global shape of data */
+  void *outme;    /* The data pointer to dump ... */
+  int free_outme; /* and whether it needs freeing */
+  int retval=0;
+#ifdef CCTK_MPI
+  int i, j;
+  MPI_Status ms;
+#endif
+
+  /* Get the handles for PUGH and IOUtil extensions */
+  pughGH   = PUGH_pGH (GH);
+  ioUtilGH = (ioGH *) GH->extensions [CCTK_GHExtensionHandle ("IO")];
+
+  dim = slab_geo->vdim;
+
+  /* allocate the geometry buffer */
+  geom = (CCTK_INT *) malloc (3*dim * sizeof (CCTK_INT));
+  
+  /* Get the pointer to the data we want to output (includes downsampling) */
+  if (IOHDF5_getDumpData_ND(GH, index, timelevel, slab_geo, 
+			    &outme, &free_outme, geom,
+			    info->element_size)<0)
+  {
+    return(-1);
+  }
+
+  myproc = CCTK_MyProc (GH);
+  nprocs = CCTK_nProcs (GH);
+
+#ifdef CCTK_MPI
+#ifdef HAVE_PARALLEL
+  if (ioUtilGH->unchunked) {
+    IOHDF5_collectiveDump (GH, index, timelevel, slab_geo,
+			   outme, geom,
+                           info->iohdf5_type, iof);
+    if (free_outme)
+      free (outme);
+    free (geom);
+    return;
+  }
+#endif
+#endif
+
+  /* Dump data held on IO processor */
+  if (myproc == ioUtilGH->ioproc)
+    IOHDF5_procDump (GH, index, timelevel, slab_geo, 
+		     outme, geom, myproc,
+                     info->iohdf5_type,iof);
+
+#ifdef CCTK_MPI
+  if (myproc == ioUtilGH->ioproc) {
+
+    /* Dump data from all other processors */
+    for (i = myproc+1; i < myproc+ioUtilGH->ioproc_every && i < nprocs; i++) {
+
+      /* Allocate temp to the right size (based on GH->ub [i] et..) */
+      void *tmpd;
+      int incoming;
+
+      /* receive geometry */
+      CACTUS_MPI_ERROR (MPI_Recv (geom, 3*dim, PUGH_MPI_INT, i, 2*i+IOTAGBASE+1,
+                        pughGH->PUGH_COMM_WORLD, &ms));
+
+      incoming = geom [dim];
+      for (j = 1; j < dim; j++)
+        incoming *= geom [dim + j];
+
+      /* receive data */
+      tmpd = malloc (incoming * info->element_size);
+      CACTUS_MPI_ERROR (MPI_Recv (tmpd, incoming, info->mpi_type, i,
+                        2*i+IOTAGBASE, pughGH->PUGH_COMM_WORLD, &ms));
+
+      IOHDF5_procDump (GH, index, timelevel, slab_geo,
+		       tmpd, geom, i,
+                       info->iohdf5_type, iof);
+      free (tmpd);
+      
+    } /* End loop over processors */
+
+  } else {
+
+    /* Send the geometry and data from the non-IO processors
+     * to the IO processors
+     */
+
+    int outgoing = geom [dim];
+
+    for (i = 1; i < dim; i++)
+      outgoing *= geom [dim + i];
+      
+    /* send geometry */
+    CACTUS_MPI_ERROR (MPI_Send (geom, 3*dim, PUGH_MPI_INT, ioUtilGH->ioproc,
+                      2*myproc+IOTAGBASE+1, pughGH->PUGH_COMM_WORLD));
+    /* send data */
+    CACTUS_MPI_ERROR (MPI_Send (outme, outgoing, info->mpi_type, ioUtilGH->ioproc,
+                      2*myproc+IOTAGBASE, pughGH->PUGH_COMM_WORLD));
+
+#ifdef IOHDF5_DEBUG
+    printf ("Processor %d sent %d data points\n", myproc, outgoing);
+#endif
+  }
+
+  CCTK_Barrier (GH);
+#endif
+
+  if (free_outme)
+    free (outme);
+
+  free (geom);
+  return(retval);
+}
+
+
+/***************************** local functions ****************************/
+
+/*@@
+  @routine    IOHDF5_AddCommonAttributes
+  @date       May 21 1999
+  @author     Thomas Radke
+  @desc 
+  Add "Common" attributes, these are:
+  <ul>
+    <li> the variable's groupname
+    <li> the grouptype
+    <li> number of timelevels
+    <li> the current date
+         Note that the datestamp should be turned of if you are byte 
+         comparing two output files.
+    <li> simulation time
+    <li> origin
+    <li> bounding box
+    <li> gridspacings (both downsampled and evolution)
+    <li> global grid size
+  </ul>
+  @enddesc 
+  @calledby   IOHDF5_DumpGS, IOHDF5_collectiveDump, IOHDF5_procDump
+  @history 
+  @endhistory 
+  @var        GH
+  @vdesc      Pointer to CCTK grid hierarchy
+  @vtype      cGH
+  @vio        in
+  @endvar
+  @var        index
+  @vdesc      global index of the variable to be dumped
+  @vtype      int
+  @vio        in
+  @endvar
+  @var        timelevel
+  @vdesc      the timelevel to store
+  @vtype      int
+  @vio        in
+  @endvar
+  @var        global_shape
+  @vdesc      the global shape of the dataset
+  @vtype      CCTK_INT [dim]
+  @vio        in
+  @endvar
+  @var        dataset
+  @vdesc      the dataset handle where the attributes should be attached to
+  @vtype      hid_t
+  @vio        in
+  @endvar
+@@*/
+static void IOHDF5_AddCommonAttributes (cGH *GH, int index, int timelevel,
+                                        CCTK_INT *global_shape, hid_t dataset)
+{
+  DECLARE_CCTK_PARAMETERS
+  int vdim;
+  CCTK_INT intAttr;
+  CCTK_REAL dummy;
+  CCTK_REAL realAttr [6];
+  char *groupname;
+  ioGH *ioUtilGH;
+  ioHDF5GH *h5GH;
+
+  /* Get the handles for IOUtil and IOHDF5 extensions */
+  ioUtilGH = (ioGH *) GH->extensions [CCTK_GHExtensionHandle ("IO")];
+  h5GH = (ioHDF5GH *) GH->extensions [CCTK_GHExtensionHandle ("IOHDF5")];
+
+  /* get the dimension of the variable */
+  vdim = CCTK_GroupDimI (CCTK_GroupIndexFromVarI (index));
+
+  groupname = CCTK_GroupNameFromVarI (index);
+  WRITE_ATTRIBUTE ("groupname", groupname,
+                   dataset, h5GH->scalarDataspace, 0, h5GH->IOHDF5_STRING);
+  free (groupname);
+  intAttr = CCTK_GroupTypeFromVarI (index);
+  WRITE_ATTRIBUTE ("grouptype", &intAttr, dataset, h5GH->scalarDataspace, 0,
+                   IOHDF5_INT);
+  intAttr = CCTK_NumTimeLevelsFromVarI (index);
+  WRITE_ATTRIBUTE ("ntimelevels", &intAttr, dataset, h5GH->scalarDataspace, 0,
+                   IOHDF5_INT);
+
+  if (char_time_date && out3D_datestamp) 
+    WRITE_ATTRIBUTE ("date", char_time_date, dataset, h5GH->scalarDataspace, 0,
+                     h5GH->IOHDF5_STRING);
+
+  WRITE_ATTRIBUTE ("time", &GH->cctk_time, dataset, h5GH->scalarDataspace, 0,
+                   IOHDF5_REAL);
+
+  /* NOTE: the attributes "origin", "min_ext", "max_ext", and "delta"
+           are always stored as 3-dimensional points */
+  CCTK_CoordRange (GH, &realAttr [0], &dummy, -1, "x", "cart3d");
+  CCTK_CoordRange (GH, &realAttr [1], &dummy, -1, "y", "cart3d");
+  CCTK_CoordRange (GH, &realAttr [2], &dummy, -1, "z", "cart3d");
+  WRITE_ATTRIBUTE ("origin", realAttr, dataset, h5GH->arrayDataspace, 3,
+                   IOHDF5_REAL);
+
+  CCTK_CoordRange (GH, &realAttr [0], &realAttr [3], -1, "x", "cart3d");
+  CCTK_CoordRange (GH, &realAttr [1], &realAttr [4], -1, "y", "cart3d");
+  CCTK_CoordRange (GH, &realAttr [2], &realAttr [5], -1, "z", "cart3d");
+  WRITE_ATTRIBUTE ("min_ext", realAttr, dataset, h5GH->arrayDataspace, 3,
+                   IOHDF5_REAL);
+  WRITE_ATTRIBUTE ("max_ext", realAttr + 3, dataset, h5GH->arrayDataspace, 3,
+                   IOHDF5_REAL);
+ 
+
+  realAttr [0] = GH->cctk_delta_space [0] * ioUtilGH->downsample [0]; 
+  realAttr [1] = GH->cctk_delta_space [1] * ioUtilGH->downsample [1]; 
+  realAttr [2] = GH->cctk_delta_space [2] * ioUtilGH->downsample [2];
+  WRITE_ATTRIBUTE ("delta", realAttr, dataset, h5GH->arrayDataspace, 3,
+                   IOHDF5_REAL);
+
+  if (ioUtilGH->downsample [0] > 1 ||
+      ioUtilGH->downsample [1] > 1 ||
+      ioUtilGH->downsample [2] > 1)
+    WRITE_ATTRIBUTE ("evolution_delta", GH->cctk_delta_space, dataset,
+                     h5GH->arrayDataspace, 3, IOHDF5_REAL);
+
+  WRITE_ATTRIBUTE ("global_size", global_shape, dataset, h5GH->arrayDataspace,
+                   vdim, IOHDF5_INT);
+
+}
+
+static void IOHDF5_AddCommonAttributes_ND (cGH *GH, int index, ioHDF5Geo_t *geo,
+					   CCTK_INT *global_shape, hid_t dataset)
+{
+  DECLARE_CCTK_PARAMETERS
+  CCTK_INT intAttr;
+  CCTK_REAL dummy;
+  CCTK_REAL realAttr [6];
+  char *groupname;
+
+  ioGH *ioUtilGH;
+  ioHDF5GH *h5GH;
+
+  int idim;
+
+  /* Get the handles for IOUtil and IOHDF5 extensions */
+  ioUtilGH = (ioGH *) GH->extensions [CCTK_GHExtensionHandle ("IO")];
+  h5GH = (ioHDF5GH *) GH->extensions [CCTK_GHExtensionHandle ("IOHDF5")];
+
+  /* VARIABLE INFO */
+  groupname = CCTK_GroupNameFromVarI (index);
+  WRITE_ATTRIBUTE ("groupname", groupname,
+                   dataset, h5GH->scalarDataspace, 0, h5GH->IOHDF5_STRING);
+  free (groupname);
+  intAttr = CCTK_GroupTypeFromVarI (index);
+  WRITE_ATTRIBUTE ("grouptype", &intAttr, dataset, h5GH->scalarDataspace, 0,
+                   IOHDF5_INT);
+
+  /* TIMLEVELS (grid) */
+  intAttr = CCTK_NumTimeLevelsFromVarI (index);
+  WRITE_ATTRIBUTE ("ntimelevels", &intAttr, dataset, h5GH->scalarDataspace, 0,
+                   IOHDF5_INT);
+
+  /* DATE  */
+  if (char_time_date && out3D_datestamp) 
+    WRITE_ATTRIBUTE ("date", char_time_date, dataset, h5GH->scalarDataspace, 0,
+                     h5GH->IOHDF5_STRING);
+
+  /* TIME (grid) */
+  WRITE_ATTRIBUTE ("time", &GH->cctk_time, dataset, h5GH->scalarDataspace, 0,
+                   IOHDF5_REAL);
+
+  /* ORIGIN (grid) */
+  CCTK_CoordRange (GH, &realAttr [0], &dummy, -1, "x", "cart3d");
+  CCTK_CoordRange (GH, &realAttr [1], &dummy, -1, "y", "cart3d");
+  CCTK_CoordRange (GH, &realAttr [2], &dummy, -1, "z", "cart3d");
+  WRITE_ATTRIBUTE ("origin", realAttr, dataset, h5GH->arrayDataspace, geo->vdim,
+                   IOHDF5_REAL);
+
+  /* ORIGIN (slab) -  slab may be offset by (slab_start) grid points */
+  for (idim=0;idim<geo->vdim;idim++)
+    realAttr[idim]=geo->slab_start[idim]*GH->cctk_delta_space[idim];
+  WRITE_ATTRIBUTE ("origin_slab", realAttr, dataset, h5GH->arrayDataspace, geo->vdim,
+                   IOHDF5_REAL);
+    
+  /* MIN/MAX extent (grid) */
+  CCTK_CoordRange (GH, &realAttr [0], &realAttr [3], -1, "x", "cart3d");
+  CCTK_CoordRange (GH, &realAttr [1], &realAttr [4], -1, "y", "cart3d");
+  CCTK_CoordRange (GH, &realAttr [2], &realAttr [5], -1, "z", "cart3d");
+  WRITE_ATTRIBUTE ("min_ext", realAttr, dataset, h5GH->arrayDataspace, geo->vdim,
+                   IOHDF5_REAL);
+  WRITE_ATTRIBUTE ("max_ext", realAttr + 3, dataset, h5GH->arrayDataspace, geo->vdim,
+                   IOHDF5_REAL);
+
+  /* MIN/MAX extent (slab) -  slab may be smaller than grid*/
+  for (idim=0;idim<geo->vdim;idim++) {
+    realAttr[idim  ]=realAttr[idim]+(geo->slab_start[idim]*GH->cctk_delta_space[idim]);
+    realAttr[idim+3]=realAttr[idim]+(geo->slab_start[idim]+geo->actlen[idim]-1)*
+                                    (GH->cctk_delta_space[idim]*geo->downs[idim]);
+  }
+  WRITE_ATTRIBUTE ("min_ext_slab", realAttr, dataset, h5GH->arrayDataspace, geo->sdim,
+                   IOHDF5_REAL);
+  WRITE_ATTRIBUTE ("max_ext_slab", realAttr + 3, dataset, h5GH->arrayDataspace, geo->sdim,
+                   IOHDF5_REAL);
+
+  /* DELTA SPACING (grid) */
+  realAttr [0] = GH->cctk_delta_space [0];
+  realAttr [1] = GH->cctk_delta_space [1];
+  realAttr [2] = GH->cctk_delta_space [2];
+  WRITE_ATTRIBUTE ("delta", realAttr, dataset, h5GH->arrayDataspace, geo->vdim,
+                   IOHDF5_REAL);
+
+  /* DELTA SPACING (slab) */
+  for (idim=0;idim<geo->sdim;idim++)
+    realAttr[idim]*=geo->downs[idim];
+  WRITE_ATTRIBUTE ("delta_slab", realAttr, dataset, h5GH->arrayDataspace, geo->sdim,
+                   IOHDF5_REAL);
+
+  /* FIXME: WHAT IS THIS ? */
+  if (ioUtilGH->downsample [0] > 1 ||
+      ioUtilGH->downsample [1] > 1 ||
+      ioUtilGH->downsample [2] > 1)
+    WRITE_ATTRIBUTE ("evolution_delta", GH->cctk_delta_space, dataset,
+                     h5GH->arrayDataspace, 3, IOHDF5_REAL);
+
+  /* GLOBAL SIZE (slab) */
+  WRITE_ATTRIBUTE ("global_size_grid", GH->cctk_gsh, dataset, h5GH->arrayDataspace,
+                   geo->vdim, IOHDF5_INT);
+
+  /* GLOBAL SIZE (slab) */
+  WRITE_ATTRIBUTE ("global_size", global_shape, dataset, h5GH->arrayDataspace,
+                   geo->sdim, IOHDF5_INT);
+
+  /* GLOBAL SIZE (slab) */
+  WRITE_ATTRIBUTE ("global_size_slab", geo->actlen, dataset, h5GH->arrayDataspace,
+                   geo->sdim, IOHDF5_INT);
+}
+
+
+static int IOHDF5_getDumpData_ND (cGH *GH, int index, int timelevel, ioHDF5Geo_t *slab_geo, 
+				   void **outme, int *free_outme, 
+				   CCTK_INT *geom, int element_size)
+{
+  ioGH *ioUtilGH;
+  pGExtras *extras=NULL;
+  
+  CCTK_REAL4 *single_ptr;
+
+  int myproc, do_downsample;
+  int idim, sdim, vdim;                      /*iteration, slab dim, variable dim */
+  int *hsizes,*hsizes_global,*hsizes_offset; /* geometry information */
+  int sdir[3];                               /* slab direction FIXME */
+  int ip;
+
+  void *data = CCTK_VarDataPtrI (GH, timelevel, index);
+  
+
+  myproc = CCTK_MyProc (GH);
+
+  /* get GH extension for IOUtil */
+  ioUtilGH = (ioGH *) GH->extensions [CCTK_GHExtensionHandle ("IO")];
+
+  /* get the pointer to PUGH specific structures */
+  extras = ((pGA***) PUGH_pGH(GH)->variables) [index][timelevel]->extras;
+
+  /* Shortcuts */
+  vdim = slab_geo->vdim;
+  sdim = slab_geo->sdim;  
+
+  /* Set downsample flag */
+  do_downsample = 0;
+  for (idim = 0; idim < sdim; idim++)
+    do_downsample |= slab_geo->downs[idim] > 1;
+
+  /* Simple case if no downsampling and sdim==vdim */
+  if ((! do_downsample)&& (sdim==vdim)) 
+  {
+    if (ioUtilGH->out_single) {
+      single_ptr = (CCTK_REAL4 *) malloc (extras->npoints*sizeof (CCTK_REAL4));
+      for (ip = 0; ip < extras->npoints; ip++)
+        single_ptr [ip] = (CCTK_REAL4) ((CCTK_REAL *) data) [ip];
+
+      *outme      = single_ptr;
+      *free_outme = 1;
+    } 
+    else 
+    {
+      *outme      = data;
+      *free_outme = 0;
+    }
+
+    for (idim = 0; idim < sdim; idim++) {
+      geom [idim + 0*sdim] = extras->lb [myproc][idim];
+      geom [idim + 1*sdim] = extras->lnsize [idim];        
+      geom [idim + 2*sdim] = extras->nsize [idim];
+    }
+  }
+  else  /* Call Hyperslab else */ {  
+
+    /* allocate array to hold size information of slab */
+    hsizes = (int*)malloc(sdim*sizeof(int));
+    hsizes_global= (int*)malloc(sdim*sizeof(int));
+    hsizes_offset= (int*)malloc(sdim*sizeof(int));
+
+    /* TEMPORARY FIX DUE TO INCONSISTENT DIR SPECIFICATION */
+    /* direction vector of 1d line in 3d volume */
+    if (sdim==1) {
+      sdir[0] = (slab_geo->direction[0]==0) ? 1:0;
+      sdir[1] = (slab_geo->direction[0]==1) ? 1:0;
+      sdir[2] = (slab_geo->direction[0]==2) ? 1:0;
+    }
+    /* norm vector of 2d surface in 3d volume */
+    else if (sdim==2)
+      {
+	sdir[0] = ((slab_geo->direction[0]==1)&&(slab_geo->direction[1]==2)) ? 1:0;
+	sdir[1] = ((slab_geo->direction[0]==0)&&(slab_geo->direction[1]==2)) ? 1:0;
+	sdir[2] = ((slab_geo->direction[0]==0)&&(slab_geo->direction[1]==1)) ? 1:0;
+      }
+    /* spanning directions for 3d */
+    else if (sdim==3)
+      {
+	sdir[0] = slab_geo->direction[0];
+	sdir[1] = slab_geo->direction[1];
+	sdir[2] = slab_geo->direction[2];
+      }
+    
+    if (Hyperslab_GetLocalHyperslab (GH, index, timelevel, sdim, slab_geo->slab_start,
+				     sdir, slab_geo->length, slab_geo->downs, outme,
+				     hsizes, hsizes_global, hsizes_offset)< 0) {
+      char *fullname = CCTK_FullName (index);
+      CCTK_VWarn (1, __LINE__, __FILE__, CCTK_THORNSTRING,
+		  "Failed to extract hyperslab for variable '%s'", fullname);
+      free (fullname); 
+      *free_outme = 0;
+      return(-1);
+      
+    }
+    
+    *free_outme = 1;
+
+    for (idim = 0; idim < sdim; idim++) {
+      geom [idim + 0*sdim]  = hsizes_offset[idim];
+      geom [idim + 1*sdim]  = hsizes[idim];        
+      geom [idim + 2*sdim]  = hsizes_global[idim];
+      slab_geo->actlen[idim]= hsizes_global[idim];
+    }
+  }
+#ifdef IOHDF5_DEBUG
+  printf ("Lower bound: %d", (int) geom [0]);
+  for (idim = 1; idim < sdim; idim++)
+    printf (" %d", (int) geom [idim]);
+  printf ("\n");
+  printf ("Chunk size : %d", (int) geom [1*sdim + 0]);
+  for (idim = 1; idim < sdim; idim++)
+    printf (" %d", (int) geom [1*sdim + idim]);
+  printf ("\n");
+  printf ("Global size: %d", (int) geom [2*sdim + 0]);
+  for (idim = 1; idim < sdim; idim++) 
+    printf (" %d", (int) geom [2*sdim + idim]);
+  printf ("\n");
+  printf ("Downsampling: ");
+  for (idim = 0; idim < sdim; idim++)
+    printf (" %d", slab_geo->downs[idim]);
+  printf ("\n");
+#endif
+
+  return(1);
+}
+
+
+/*@@
+  @routine IOHDF5_procDump
+  @author  Thomas Radke
+  @date    May 21 1999
+  @desc    All IO processors dump the data of their group into the file,
+           either as one chunk per processor or unchunked
+           (depending on parameter "unchunked").
+  @enddesc
+  @calledby   IOHDF5_DumpGA
+  @history 
+  @endhistory 
+  @var        GH
+  @vdesc      Pointer to CCTK grid hierarchy
+  @vtype      cGH
+  @vio        in
+  @endvar
+  @var        index
+  @vdesc      global index of the variable to be dumped
+  @vtype      int
+  @vio        in
+  @endvar
+  @var        timelevel
+  @vdesc      the timelevel to store
+  @vtype      int
+  @vio        in
+  @endvar
+  @var        outme
+  @vdesc      pointer to the chunk to dump
+  @vtype      void *
+  @vio        in
+  @endvar
+  @var        geom
+  @vdesc      bounds and sizes of the output
+  @vtype      CCTK_INT [3*dim]
+  @vio        in
+  @vcomment   geom [0*dim..1*dim-1]: lower bounds
+              geom [1*dim..2*dim-1]: (local) data sizes
+              geom [2*dim..3*dim-1]: global space
+  @endvar 
+  @var        proc
+  @vdesc      the processor which's chunk is to be dumped
+  @vtype      int
+  @vio        in
+  @endvar
+  @var        hdf5io_type
+  @vdesc      the HDF5 datatype identifier
+  @vtype      int
+  @vio        in
+  @endvar
+  @var        iof
+  @vdesc      the HDF5 file handle
+  @vtype      hid_t
+  @vio        in
+  @endvar
+@@*/
+static void IOHDF5_procDump (cGH *GH, int index, int timelevel, ioHDF5Geo_t *slab_geo,
+			     void *outme, CCTK_INT *geom, int proc, 
+			     int iohdf5_type, hid_t iof)
+{
+  DECLARE_CCTK_PARAMETERS
+  int i, vdim,sdim, idim;
+  ioGH *ioUtilGH;
+  ioHDF5GH *h5GH;
+  hid_t group, dataset, memspace, filespace;
+  char *name, datasetname [128], chunkname [128];
+  hssize_t *chunk_origin;
+  hsize_t *chunk_dims, *file_dims;
+
+
+  ioUtilGH = (ioGH *) GH->extensions [CCTK_GHExtensionHandle ("IO")];
+  h5GH = (ioHDF5GH *) GH->extensions [CCTK_GHExtensionHandle ("IOHDF5")];
+
+  vdim = slab_geo->vdim;
+  sdim = slab_geo->sdim;
+
+  chunk_origin = (hssize_t *) malloc (vdim * sizeof (hssize_t));
+  chunk_dims   = (hsize_t *)  malloc (2*vdim * sizeof (hsize_t));
+  file_dims    = chunk_dims + vdim;
+
+  /* HDF5 needs it in reverse order */
+  for (i = 0; i < sdim; i++) {
+    chunk_origin [i] = geom [1*sdim - 1 - i];
+    chunk_dims   [i] = geom [2*sdim - 1 - i];
+    file_dims    [i] = geom [3*sdim - 1 - i];
+  }
+
+  /* build the unique dataset name: eg. WAVETOY::phi@v3s01@3@1  
+     volume (v) 3-dim ; slab (s) spanned by 0,1 (->2-dim) 
+     at(@) iteration at(@) timelevel
+  */
+  name = CCTK_FullName (index);
+  sprintf (datasetname, "%s@v%ds",name,vdim);
+  for (idim=0;idim<sdim;idim++)
+    sprintf (datasetname, "%s%d",datasetname, slab_geo->direction[idim]);
+  sprintf (datasetname, "%s@%d@%d",datasetname, GH->cctk_iteration, timelevel);
+  free (name);
+
+  /* create the memspace according to chunk dims */
+  CACTUS_IOHDF5_ERROR (memspace = H5Screate_simple (sdim, chunk_dims, NULL));
+
+  if (ioUtilGH->unchunked) {
+
+    /* create the (global) filespace and set the hyperslab for the chunk */
+    CACTUS_IOHDF5_ERROR (filespace = H5Screate_simple (sdim, file_dims, NULL));
+    CACTUS_IOHDF5_ERROR (H5Sselect_hyperslab (filespace, H5S_SELECT_SET,
+                         chunk_origin, NULL, chunk_dims, NULL));
+
+    /* the IO processor creates the dataset and adds the common attributes
+       when writing its own data, otherwise the dataset is reopened */
+    if (CCTK_MyProc (GH) == proc) {
+      /* if restart from recovery delete an already existing dataset
+         so that it can be created as anew */
+      if (h5GH->checkForExistingObjects [index]) {
+        CACTUS_IOHDF5_ERROR (H5Eset_auto (NULL, NULL));
+        H5Gunlink (iof, datasetname);
+        H5Eset_auto (h5GH->printErrorFn, h5GH->printErrorFnArg);
+      }
+      CACTUS_IOHDF5_ERROR (dataset = H5Dcreate (iof, datasetname,
+						iohdf5_type, filespace, H5P_DEFAULT));
+      IOHDF5_AddCommonAttributes_ND (GH, index, slab_geo, 
+				     &geom [2*sdim], dataset);
+    } else
+      CACTUS_IOHDF5_ERROR (dataset = H5Dopen (iof, datasetname));
+
+    /* write the data */
+    CACTUS_IOHDF5_ERROR (H5Dwrite (dataset, iohdf5_type, memspace,
+                         filespace, H5P_DEFAULT, outme));
+
+    /* and close the file dataspace */
+    CACTUS_IOHDF5_ERROR (H5Sclose (filespace));
+
+  } else {
+
+    /* the IO processor creates the chunk group and adds common attributes */
+    if (CCTK_MyProc (GH) == proc) {
+      /* if restart from recovery delete an already existing group
+         so that it can be created as anew */
+      if (h5GH->checkForExistingObjects [index]) {
+        CACTUS_IOHDF5_ERROR (H5Eset_auto (NULL, NULL));
+        H5Gunlink (iof, datasetname);
+        H5Eset_auto (h5GH->printErrorFn, h5GH->printErrorFnArg);
+      }
+      CACTUS_IOHDF5_ERROR (group = H5Gcreate (iof, datasetname, 0));
+      IOHDF5_AddCommonAttributes_ND (GH, index, slab_geo, 
+				     &geom [2*vdim], group);
+      CACTUS_IOHDF5_ERROR (H5Gclose (group));
+    }
+
+    /* now the chunk dataset for each processor is created within the group */
+    sprintf (chunkname, "%s/chunk%d", datasetname, proc - CCTK_MyProc (GH));
+    /* create the chunk dataset and dump the chunk data */
+    CACTUS_IOHDF5_ERROR (dataset = H5Dcreate (iof, chunkname,
+                         iohdf5_type, memspace, H5P_DEFAULT));
+    CACTUS_IOHDF5_ERROR (H5Dwrite (dataset, iohdf5_type, H5S_ALL,
+                         H5S_ALL, H5P_DEFAULT, outme));
+
+    /* add the "origin" attribute for the chunk */
+    WRITE_ATTRIBUTE ("chunk_origin", geom, dataset, h5GH->arrayDataspace, vdim,
+                     IOHDF5_INT);
+  }
+
+  /* close the dataset and the memspace */
+  CACTUS_IOHDF5_ERROR (H5Dclose (dataset));
+  CACTUS_IOHDF5_ERROR (H5Sclose (memspace));
+
+  free (chunk_origin);
+  free (chunk_dims);
+}
+
+
+#ifdef CCTK_MPI
+#ifdef HAVE_PARALLEL
+/*@@
+  @routine IOHDF5_collectiveDump
+  @author  Thomas Radke
+  @date    May 21 1999
+  @desc    All processors dump their data into an unchunked file.
+  @enddesc
+  @calledby   IOHDF5_DumpGA
+  @history
+  @endhistory
+  @var        GH
+  @vdesc      Pointer to CCTK grid hierarchy
+  @vtype      cGH
+  @vio        in
+  @endvar
+  @var        index
+  @vdesc      global index of the variable to be dumped
+  @vtype      int
+  @vio        in
+  @endvar
+  @var        timelevel
+  @vdesc      the timelevel to store
+  @vtype      int
+  @vio        in
+  @endvar
+  @var        outme
+  @vdesc      pointer to the chunk to dump
+  @vtype      void *
+  @vio        in
+  @endvar
+  @var        geom
+  @vdesc      bounds and sizes of the output
+  @vtype      CCTK_INT [3*dim]
+  @vio        in
+  @vcomment   geom [0*dim..1*dim-1]: lower bounds
+              geom [1*dim..2*dim-1]: (local) data sizes
+              geom [2*dim..3*dim-1]: global space
+  @endvar 
+  @var        hdf5io_type
+  @vdesc      the HDF5 datatype identifier
+  @vtype      int
+  @vio        in
+  @endvar
+  @var        iof
+  @vdesc      the HDF5 file handle
+  @vtype      hid_t
+  @vio        in
+  @endvar
+@@*/
+static void IOHDF5_collectiveDump (cGH *GH, int index, int timelevel, ioHDF5Geo_t *slab_geo,  
+				   void *outme, CCTK_INT *geom, 
+				   int hdf5io_type, hid_t iof)
+{
+  DECLARE_CCTK_PARAMETERS
+  int i, dim;
+  ioHDF5GH *h5GH;
+  hid_t dataset, memspace, filespace;
+  char *name, datasetname [128];
+  hssize_t *chunk_origin;
+  hsize_t *chunk_dims, *file_dims;
+
+
+  h5GH = (ioHDF5GH *) GH->extensions [CCTK_GHExtensionHandle ("IOHDF5")];
+
+  /* get the dimension of the variable */
+  dim = slab_geo->vdim;
+
+  chunk_origin = (hssize_t *) malloc (dim * sizeof (hssize_t));
+  chunk_dims   = (hsize_t *)  malloc (2*dim * sizeof (hsize_t));
+  file_dims    = chunk_dims + dim;
+
+  /* HDF5 needs it in reverse order */
+  for (i = 0; i < dim; i++) {
+    chunk_origin [i] = geom [1*dim - 1 - i];
+    chunk_dims   [i] = geom [2*dim - 1 - i];
+    file_dims    [i] = geom [3*dim - 1 - i];
+  }
+
+  /* build the unique dataset name */
+  name = CCTK_FullName (index);
+  sprintf (datasetname, "%s@%d@%d", name, GH->cctk_iteration, timelevel);
+  free (name);
+
+  /* create the memspace according to chunk dims */
+  CACTUS_IOHDF5_ERROR (memspace = H5Screate_simple (dim, chunk_dims, NULL));
+
+  /* create the (global) filespace and set the hyperslab for the chunk */
+  CACTUS_IOHDF5_ERROR (filespace = H5Screate_simple (dim, file_dims, NULL));
+  CACTUS_IOHDF5_ERROR (H5Sselect_hyperslab (filespace, H5S_SELECT_SET,
+                       chunk_origin, NULL, chunk_dims, NULL));
+
+  /* the IO processor creates the dataset and adds the common attributes
+     when writing its own data, otherwise the dataset is reopened */
+  /* if restart from recovery delete an already existing group
+     so that it can be created as anew */
+  if (h5GH->checkForExistingObjects [index]) {
+    CACTUS_IOHDF5_ERROR (H5Eset_auto (NULL, NULL));
+    H5Gunlink (iof, datasetname);
+    H5Eset_auto (h5GH->printErrorFn, h5GH->printErrorFnArg);
+  }
+  CACTUS_IOHDF5_ERROR (dataset = H5Dcreate (iof, datasetname,
+                       hdf5io_type, filespace, H5P_DEFAULT));
+  if (CCTK_MyProc (GH) == 0)
+    IOHDF5_AddCommonAttributes_ND (GH, index, slab_geo, 
+				   &geom [2*dim], dataset);
+
+  /* write the data */
+  CACTUS_IOHDF5_ERROR (H5Dwrite (dataset, hdf5io_type, memspace,
+                       filespace, H5P_DEFAULT, outme));
+
+  /* and close the file dataspace */
+  CACTUS_IOHDF5_ERROR (H5Sclose (filespace));
+
+  /* close the dataset and the memspace */
+  CACTUS_IOHDF5_ERROR (H5Dclose (dataset));
+  CACTUS_IOHDF5_ERROR (H5Sclose (memspace));
+
+  free (chunk_origin);
+  free (chunk_dims);
+}
+#endif /* HAVE_PARALLEL */
+#endif /* MPI */
diff --git a/src/File.c b/src/File.c
new file mode 100644
index 0000000..1f79218
--- /dev/null
+++ b/src/File.c
@@ -0,0 +1,204 @@
+
+#include "cctk.h"
+#include "cctk_Parameters.h"
+#include "StoreNamedData.h"
+#include "CactusPUGH/PUGH/src/include/pugh.h"
+#include "CactusBase/IOUtil/src/ioGH.h"
+#include "ioHDF5GH.h"
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <assert.h>
+
+/*@@
+  @routine IOHDF5_Get_filename_ND
+  @author  Paul Walker
+  @date    Feb 1997
+  @desc    Returns the filename for output
+           The names of all output files are stored in the filenames database.
+           The routine first searches in this database if there is already
+           registered a filename for alias. If so, this name is returned.
+           Otherwise it builds the filename from alias and stores it in the
+           database.
+           NOTE: Filenames are never stored if "out2D_septimefiles = true"
+                 since they are used only once.
+		 
+	   Generalized for Hyperslab extraction (GL)
+  @enddesc
+  @var     GH
+  @vdesc   Pointer to CCTK GH
+  @vtype   cGH
+  @vio     in
+  @endvar
+  @var     index
+  @vdesc   index of variable to output
+  @vtype   int
+  @vio     in
+  @endvar
+  @var     name
+  @vdesc   alias name of variable to output
+  @vtype   const char *
+  @vio     in
+  @endvar
+  @var     isNewFile
+  @vdesc   flag indicating if the file is to be created
+  @vtype   int
+  @vio     out
+  @endvar
+@@*/
+
+char *IOHDF5_Get_filename_ND (cGH *GH, int index, ioHDF5Geo_t *slab_geo,
+			      const char *varname, int *isNewFile)
+{
+  DECLARE_CCTK_PARAMETERS
+  ioGH *ioUtilGH;       /* handle for IOUtil extensions */
+  ioHDF5GH *myGH;       /* handle for IOHDF5 extensions */
+  pGH *pughGH;          /* handle for PUGH extensions */
+  int myproc;
+  char extra [256];     /* Extra stuff in fname based on mode */
+  char extradir [256];  /* Extra stuff for an output dir */
+  char *extrageo;       /* Extra stuff for geometry information */
+  char *filename;       /* the return value */
+  int idim;
+
+  char name[128];
+
+  /* get GH extensions for PUGH and IOUtil */
+  pughGH   = PUGH_pGH (GH);
+  ioUtilGH = (ioGH *) GH->extensions [CCTK_GHExtensionHandle ("IO")];
+  myGH     = (ioHDF5GH *) GH->extensions [CCTK_GHExtensionHandle ("IOHDF5")];
+
+  /* To identify the slab in a volume  (if sdim<vdim)
+     build a string like: "012" (3d volume, spanned by xyz)
+     or "12" (2d volume, spanned by yz(!) and append this to the name:
+     You get something like this: phi_s12_3d */
+  if (slab_geo->sdim<slab_geo->vdim) {
+    extrageo = (char*) malloc((slab_geo->sdim+1)*sizeof(char));
+    sprintf(extrageo,"%d",slab_geo->direction[0]);
+    for (idim=1;idim<slab_geo->sdim;idim++)
+      sprintf(extrageo,"%s%d",extrageo,slab_geo->direction[idim]);
+    sprintf(name,"%s_s%s_%dd",varname,extrageo,slab_geo->vdim);
+    free(extrageo);
+  }
+  /* If sdim==vdim, make it simple: e.g. phi_3d */
+  else if (slab_geo->sdim==slab_geo->vdim)
+  { 
+    sprintf(name,"%s_%dd",varname,slab_geo->vdim);
+  }
+  else {
+    CCTK_VWarn (1, __LINE__, __FILE__, CCTK_THORNSTRING,
+		"Inconsitent dimensions for >%s<: slab (%dD), variable (%dD)\n",
+		varname, slab_geo->sdim,slab_geo->vdim);
+    return(NULL);
+  }
+
+  filename = (char *) GetNamedData (myGH->filenameList, name);
+  if (filename != NULL)
+  {
+    /* set flag to indicate that file should be opened in append mode */
+    *isNewFile = 0;
+    return (filename);
+  }
+
+  extra[0]   = '\0';
+  extradir[0]= '\0';
+
+  myproc = CCTK_MyProc (GH);
+
+  if (out2D_septimefiles)
+  {
+    char *tmp = extra;
+
+    sprintf (extra, "%s.time_%7.3f", extra, GH->cctk_time);
+
+    /* And be sure to replace any spaces in the filename with an _ */
+    do
+      if (*tmp == ' ')
+        *tmp = '_';
+    while (*++tmp);
+  }
+
+  
+   /* OUTPUT ONE FILE FOR EACH N PROCESSORS - only 3D
+   *  ------------------------------------- 
+   *  
+   *  If only one output file, the single file for each variable is written 
+   *  in the normal output dir (that is extradir = ""). Otherwise
+   *  a directory is created for each output variable to hold the multiple
+   *  files.
+   */
+
+  if (! ioUtilGH->unchunked && ioUtilGH->ioproc_every < CCTK_nProcs (GH) && 
+      slab_geo->vdim==3)
+  {
+    /* Add the output processor number to the extra string */
+    sprintf (extra, "%s.file_%d", extra, myproc / ioUtilGH->ioproc_every);
+    
+    /* If necessary create the output directory */
+    if (myproc == 0)
+    {
+      int result;
+      char *outputdir = (char *) malloc (strlen (myGH->outdir3D) +
+                                         strlen (name) + 5);
+
+      sprintf (outputdir, "%s/%s_3d", myGH->outdir3D, name);
+      
+      result = CCTK_CreateDirectory (0755, outputdir);
+      if (result < 0)
+        CCTK_VWarn (1, __LINE__, __FILE__, CCTK_THORNSTRING,
+                    "Problem creating 3D output directory '%s'", outputdir);
+      if (result > 0)
+        CCTK_VWarn (2, __LINE__, __FILE__, CCTK_THORNSTRING,
+                    "3D output directory '%s' already exists", outputdir);
+
+      free (outputdir);
+    }
+    
+#ifdef CCTK_MPI
+    /* Wait for all processors to catch up */
+    CCTK_Barrier (GH);
+#endif
+    
+    /* extradir is the relative output directory */
+    sprintf (extradir, "%s_3d/", name);
+    
+  }
+  
+  /* CREATE THE COMPLETE OUTPUT FILENAME 
+     ----------------------------------- */
+  filename = (char *) malloc (strlen (myGH->outdir2D) +
+                              strlen (extradir) +
+                              strlen (name) +
+                              strlen (extra) +
+                              (pughGH->identity_string ? strlen (pughGH->identity_string) : 0) +
+                              8);
+  sprintf (filename, "%s/%s%s%s%s.h5", 
+	   myGH->outdir2D, extradir, name, extra, (pughGH->identity_string ? pughGH->identity_string : ""));
+
+  /* no need to store filenames if used only once */
+  if (! out2D_septimefiles)
+  {
+    if (myproc == ioUtilGH->ioproc)
+    {
+      if (ioUtilGH->recovered)
+      {
+        CACTUS_IOHDF5_ERROR (H5Eset_auto (NULL, NULL));
+        *isNewFile = H5Fis_hdf5 (filename) <= 0;
+        H5Eset_auto (myGH->printErrorFn, myGH->printErrorFnArg);
+        if (! *isNewFile)
+          myGH->checkForExistingObjects [index] = 1;
+      }
+      else
+      {
+        *isNewFile = 1;
+      }
+    }
+    StoreNamedData (&myGH->filenameList, name, filename);
+  }
+  else
+  {
+    *isNewFile = 1;
+  }
+
+  return (filename);
+}
diff --git a/src/GHExtension.c b/src/GHExtension.c
new file mode 100644
index 0000000..a3039eb
--- /dev/null
+++ b/src/GHExtension.c
@@ -0,0 +1,190 @@
+ /*@@
+   @file      GHExtension.c
+   @date      Fri May 21 1999
+   @author    Thomas Radke
+   @desc 
+   IOHDF5 GH extension stuff.
+   @enddesc 
+   @history
+   @hauthor Thomas Radke @hdate May 21 1999
+   @hdesc Just copied from thorn FlexIO.
+   @endhistory
+ @@*/
+
+
+/*#define DEBUG_IO*/
+
+#include <stdlib.h>
+#include <string.h>
+#include <stdio.h>
+
+#include "cctk.h"
+#include "cctk_Parameters.h"
+#include "CactusPUGH/PUGH/src/include/pugh.h"
+#include "CactusBase/IOUtil/src/ioGH.h"
+#include "ioHDF5GH.h"
+
+void IOHDF5_DefaultGeo(int sdim, ioHDF5Geo_t *geo);
+
+void *IOHDF5_SetupGH (tFleshConfig *config, int convergence_level, cGH *GH)
+{
+  int numvars,iv;
+  ioHDF5GH *newGH;
+ 
+  numvars = CCTK_NumVars ();
+
+  newGH = (ioHDF5GH *) malloc (sizeof (ioHDF5GH));
+  newGH->do_out1D = (char *) malloc (numvars * sizeof (char));
+  newGH->do_out2D = (char *) malloc (numvars * sizeof (char));
+  newGH->do_out3D = (char *) malloc (numvars * sizeof (char));
+  newGH->out1D_last = (int *) malloc (numvars * sizeof (int));
+  newGH->out2D_last = (int *) malloc (numvars * sizeof (int));
+  newGH->out3D_last = (int *) malloc (numvars * sizeof (int));
+  newGH->checkForExistingObjects = (char *) calloc (numvars, 1);
+  
+  /* Allocate geometry structure for each grid variable/array */
+  newGH->out_geo = (ioHDF5Geo_t**) malloc(IOHDF5_MAXDIM * sizeof (ioHDF5Geo_t));
+  for (iv=0;iv<numvars;iv++)
+    newGH->out_geo[iv] = (ioHDF5Geo_t*) malloc(IOHDF5_MAXDIM * sizeof (ioHDF5Geo_t));
+  
+  /* save the original error printing routine and its argument */
+  CACTUS_IOHDF5_ERROR (H5Eget_auto (&newGH->printErrorFn,
+                                    &newGH->printErrorFnArg));
+
+  /* predefine dataspaces for writing scalar and array attributes */
+  /* the dimension of the array dataspace is set when used */
+  CACTUS_IOHDF5_ERROR (newGH->scalarDataspace = H5Screate (H5S_SCALAR));
+  CACTUS_IOHDF5_ERROR (newGH->arrayDataspace = H5Screate (H5S_SIMPLE));
+
+  /* predefine a IOHDF5_COMPLEX datatype */
+  CACTUS_IOHDF5_ERROR (newGH->IOHDF5_COMPLEX =
+                       H5Tcreate (H5T_COMPOUND, sizeof (CCTK_COMPLEX)));
+  CACTUS_IOHDF5_ERROR (H5Tinsert (newGH->IOHDF5_COMPLEX, "real",
+                       offsetof (CCTK_COMPLEX, Re), IOHDF5_REAL));
+  CACTUS_IOHDF5_ERROR (H5Tinsert (newGH->IOHDF5_COMPLEX, "imag",
+                       offsetof (CCTK_COMPLEX, Im), IOHDF5_REAL));
+
+  /* predefine a C string datatype */
+  CACTUS_IOHDF5_ERROR (newGH->IOHDF5_STRING = H5Tcopy (H5T_C_S1));
+
+  return (newGH);
+}
+
+int IOHDF5_InitGH (cGH *GH)
+{
+  DECLARE_CCTK_PARAMETERS
+  int inum,idim,i;
+  ioHDF5GH *myGH;
+  const cParamData *paramdata;
+  ioHDF5Geo_t geo_default;
+
+
+  myGH = (ioHDF5GH *) GH->extensions [CCTK_GHExtensionHandle ("IOHDF5")];
+
+  /* How often to output */
+  myGH->out1D_every = out_every > 0 ? out_every : -1;
+  if (out1D_every > 0)
+    myGH->out1D_every = out1D_every;
+  IOUtil_ParseVarsForOutput (out1D_vars, myGH->do_out1D);
+
+  myGH->out2D_every = out_every > 0 ? out_every : -1;
+  if (out2D_every > 0)
+    myGH->out2D_every = out2D_every;
+  IOUtil_ParseVarsForOutput (out2D_vars, myGH->do_out2D);
+
+  myGH->out3D_every = out_every > 0 ? out_every : -1;
+  if (out3D_every > 0)
+    myGH->out3D_every = out3D_every;
+  IOUtil_ParseVarsForOutput (out3D_vars, myGH->do_out3D);
+
+  /* Check whether "outdir2D" and/or "outdir3D" was set.
+     If so take these dirs otherwise default to "IO::outdir" */
+  paramdata = CCTK_ParameterData ("outdir1D", CCTK_THORNSTRING);
+  if (paramdata && paramdata->n_set > 0)
+    myGH->outdir1D = strdup (outdir1D);
+  else
+    myGH->outdir1D = strdup (outdir);
+
+  paramdata = CCTK_ParameterData ("outdir2D", CCTK_THORNSTRING);
+  if (paramdata && paramdata->n_set > 0)
+    myGH->outdir2D = strdup (outdir2D);
+  else
+    myGH->outdir2D = strdup (outdir);
+
+  paramdata = CCTK_ParameterData ("outdir3D", CCTK_THORNSTRING);
+  if (paramdata && paramdata->n_set > 0)
+    myGH->outdir3D = strdup (outdir3D);
+  else
+    myGH->outdir3D = strdup (outdir);
+
+  /* Create the output directories */
+  if (CCTK_MyProc (GH) == 0) {
+    i = CCTK_CreateDirectory (0755, myGH->outdir1D);
+    if (i < 0)
+      CCTK_VWarn (1, __LINE__, __FILE__, CCTK_THORNSTRING,
+             "Problem creating HDF5 1D output directory '%s'", myGH->outdir1D);
+    else if (i > 0)
+      CCTK_VWarn (2, __LINE__, __FILE__, CCTK_THORNSTRING,
+             "HDF5 1D output directory '%s' already exists", myGH->outdir1D);
+
+    i = CCTK_CreateDirectory (0755, myGH->outdir2D);
+    if (i < 0)
+      CCTK_VWarn (1, __LINE__, __FILE__, CCTK_THORNSTRING,
+             "Problem creating HDF5 2D output directory '%s'", myGH->outdir2D);
+    else if (i > 0)
+      CCTK_VWarn (2, __LINE__, __FILE__, CCTK_THORNSTRING,
+             "HDF5 2D output directory '%s' already exists", myGH->outdir2D);
+
+    i = CCTK_CreateDirectory (0755, myGH->outdir3D);
+    if (i < 0)
+      CCTK_VWarn (1, __LINE__, __FILE__, CCTK_THORNSTRING,
+             "Problem creating HDF5 3D output directory '%s'", myGH->outdir3D);
+    else if (i > 0)
+      CCTK_VWarn (2, __LINE__, __FILE__, CCTK_THORNSTRING,
+             "HDF5 3D output directory '%s' already exists", myGH->outdir3D);
+  }
+
+  for (inum=0; inum<CCTK_NumVars(); inum++) 
+  {
+    myGH->out2D_last [inum] = myGH->out3D_last [inum] = -1;
+    myGH->out1D_last [inum] = myGH->out3D_last [inum] = -1;
+  }
+
+  /* Loop over all slab dimension */
+  for (idim=0;idim<IOHDF5_MAXDIM;idim++) {
+    IOHDF5_DefaultGeo(idim, &geo_default);
+
+    /* Set the default HDF5 slab geometry for slab with dimension -idim-*/
+    for (inum=0; inum<CCTK_NumVars(); inum++) 
+      myGH->out_geo[inum][idim] = geo_default;
+  }
+  myGH->filenameList = NULL;
+
+  /* create timers if timing info was requested */
+  myGH->print_timing_info = print_timing_info;
+  if (myGH->print_timing_info)
+  {
+    for (i = 0; i < 4; i++)
+    {
+      if ((myGH->timers[i] = CCTK_TimerCreateI ()) < 0)
+        break;
+    }
+    if (i != 4)
+    {
+      CCTK_WARN (1, "Could not create timers for checkpoint/recovery ! "
+                    "No timer information will be available.");
+      while (--i >= 0)
+      {
+        CCTK_TimerDestroyI (myGH->timers[i]);
+      }
+      myGH->print_timing_info = 0;
+    }   
+    else
+    {
+      CCTK_TimerResetI (myGH->timers[2]);
+      CCTK_TimerResetI (myGH->timers[3]);
+    }
+  }
+
+  return (0);
+}
diff --git a/src/Output1D.c b/src/Output1D.c
new file mode 100644
index 0000000..70c3b97
--- /dev/null
+++ b/src/Output1D.c
@@ -0,0 +1,275 @@
+ /*@@
+   @file      Output1D.c
+   @date      Tue Jan 9 1999
+   @author    Gabrielle Allen
+   @desc 
+   Functions to deal 1D HDF5 output of variables
+   @enddesc 
+   @history
+   @hauthor Thomas Radke @hdate 16 Mar 1999
+   @hdesc Converted to Cactus 4.0
+   @hendhistory
+ @@*/
+
+
+
+#include <stdio.h>
+#include <stdlib.h>
+
+#include "cctk.h"
+#include "cctk_Parameters.h"
+#include "CactusBase/IOUtil/src/ioGH.h"
+#include "ioHDF5GH.h"
+
+
+/* function prototypes */
+int IOHDF5_TimeFor1D (cGH *GH, int index);
+int IOHDF5_Output1DVarAs (cGH *GH, const char *var, const char *alias);
+static void CheckSteerableParameters (ioHDF5GH *myGH);
+
+
+/*@@
+   @routine    IOHDF5_Output1DGH
+   @date       Sat March 6 1999
+   @author     Gabrielle Allen
+   @desc
+   Loops over all variables and outputs them if necessary
+   @enddesc
+   @calledby   CCTK_OutputGH ("IOHDF5_1D")
+   @history
+
+   @endhistory
+   @var     GH
+   @vdesc   Pointer to CCTK GH
+   @vtype   cGH
+   @vio     in
+   @endvar
+@@*/
+
+int IOHDF5_Output1DGH (cGH *GH)
+{
+  DECLARE_CCTK_PARAMETERS
+  int i;
+  ioHDF5GH *myGH;
+  const char *name;
+  char *fullname;
+  int ierr=0;
+
+  /* Get the GH extension for IOHDF5 */
+  myGH = (ioHDF5GH *) GH->extensions [CCTK_GHExtensionHandle ("IOHDF5")];
+
+  CheckSteerableParameters (myGH);
+
+  if (myGH->out1D_every <= 0)
+    return (0);
+
+  /* Loop over all variables */
+  for (i = 0; i < CCTK_NumVars (); i++)
+  {
+
+    if (IOHDF5_TimeFor1D (GH, i))
+    {
+
+      name     = CCTK_VarName (i);
+      fullname = CCTK_FullName (i);
+
+      if (verbose)
+        CCTK_VInfo (CCTK_THORNSTRING, "IOHDF5_Output1DGH: fullname / name = "
+                    "%s / %s", fullname, name);
+
+      ierr=IOHDF5_Output1DVarAs (GH, fullname, name);
+
+      free (fullname);
+
+      /* Register variable as having 1D output this iteration */
+      myGH->out1D_last [i] = GH->cctk_iteration;
+    }
+  }  
+  if (verbose)
+    CCTK_INFO("IOHDF5_Output1DGH Done \n\n");
+  
+  return (ierr);
+}
+
+
+/*@@
+   @routine    IOHDF5_Output1DVarAs
+   @date       Sat March 6 1999
+   @author     Gabrielle Allen
+   @desc
+   unconditional output of a variable using the IOHDF5 1D output method
+   @enddesc
+   @calledby IOHDF5_Output1DGH, CCTK_OutputVarAsByMethod ("IOHDF5_1D")
+   @var     GH
+   @vdesc   Pointer to CCTK GH
+   @vtype   cGH
+   @vio     in
+   @vcomment
+   @endvar
+   @var     fullname
+   @vdesc   complete name of variable to output
+   @vtype   const char *
+   @vio     in
+   @vcomment
+   @endvar
+   @var     alias
+   @vdesc   alias name of variable to output (used to generate output filename)
+   @vtype   const char *
+   @vio     in
+   @vcomment
+   @endvar
+@@*/
+
+int IOHDF5_Output1DVarAs (cGH *GH, const char *fullname, const char *alias)
+{
+  DECLARE_CCTK_PARAMETERS
+  int index, ierr=0;
+
+  index = CCTK_VarIndex (fullname);
+
+  /* first, check if variable has storage assigned */
+  if (! CCTK_QueryGroupStorageI (GH, CCTK_GroupIndexFromVarI (index)))
+  {
+    CCTK_VWarn (2, __LINE__, __FILE__, CCTK_THORNSTRING,
+                "No IOHDF5 1D output for '%s' (no storage)", fullname);
+    return(-1);
+  }
+
+  if (verbose)
+    CCTK_VInfo (CCTK_THORNSTRING, "IOHDF5_Output1DVarAs: fullname, alias, "
+                "index = (%s, %s, %d)", fullname, alias, index); 
+
+  /* Do the 1D output */
+  ierr=IOHDF5_Write1D (GH, index, alias);
+
+  return (ierr);
+}
+
+
+/*@@
+   @routine    IOHDF5_TimeFor1D
+   @date       Sat March 6 1999
+   @author     Gabrielle Allen
+   @desc
+   Decides if it is time to output a variable using the IOHDF5 1D output
+   method
+   @enddesc
+   @calledby IOHDF5_Output1DGH
+   @history
+
+   @endhistory
+   @var     GH
+   @vdesc   Pointer to CCTK GH
+   @vtype   cGH
+   @vio     in
+   @vcomment
+   @endvar
+   @var     index
+   @vdesc   index of variable
+   @vtype   int
+   @vio     in
+   @vcomment
+   @endvar
+@@*/
+
+int IOHDF5_TimeFor1D (cGH *GH, int index)
+{
+  ioHDF5GH *myGH;
+
+
+  /* Get the GH extension for IOHDF5 */
+  myGH = (ioHDF5GH *) GH->extensions [CCTK_GHExtensionHandle ("IOHDF5")];
+
+  CheckSteerableParameters (myGH);
+
+  /* Check if any output was requested */
+  if(myGH->out1D_every <= 0)
+    return (0);
+
+  /* Check this variable should be output */
+  if (! (myGH->do_out1D [index] && GH->cctk_iteration % myGH->out1D_every == 0))
+    return (0);
+
+  /* Check variable not already output this iteration */
+  if (myGH->out1D_last [index] == GH->cctk_iteration)
+  {
+      CCTK_WARN (2, "Already done 1D output in IOHDF5");
+      return (0);
+  }
+
+  return (1);
+}
+
+
+/*@@
+   @routine    IOHDF5_TriggerOutput1D
+   @date       Sat March 6 1999
+   @author     Gabrielle Allen
+   @desc
+   Triggers the output a variable using the IOHDF5 1D output
+   method
+   @enddesc
+   @calledby   CCTK scheduler
+   @var     GH
+   @vdesc   Pointer to CCTK GH
+   @vtype   cGH
+   @vio     in
+   @vcomment
+   @endvar
+   @var     index
+   @vdesc   index of variable to output
+   @vtype   int
+   @vio     in
+   @vcomment
+   @endvar
+@@*/
+
+int IOHDF5_TriggerOutput1D (cGH *GH, int index)
+{
+  DECLARE_CCTK_PARAMETERS
+  ioHDF5GH *myGH;
+  const char *varname;
+  int ierr=0;
+
+  varname = CCTK_VarName (index);
+
+  myGH = (ioHDF5GH *) GH->extensions [CCTK_GHExtensionHandle ("IOHDF5")];
+
+  if (verbose)
+    CCTK_VInfo (CCTK_THORNSTRING, "IOHDF5_TriggerOutput1D: varname, index = "
+                "(%s, %d)", varname, index);
+
+  /* Do the 1D output */
+  ierr=IOHDF5_Write1D (GH, index, varname);
+
+  /* Register variable as having 1D output this iteration */
+  myGH->out1D_last [index] = GH->cctk_iteration;
+
+  return (ierr);
+}
+
+
+/**************************** local functions ******************************/
+static void CheckSteerableParameters (ioHDF5GH *myGH)
+{
+  DECLARE_CCTK_PARAMETERS
+  int out1D_vars_current_nset;
+  static int out1D_vars_lastset = 0;
+
+
+  /* How often to output */
+  myGH->out1D_every = out_every > 0 ? out_every : -1;
+  if (out1D_every > 0)
+    myGH->out1D_every = out1D_every;
+
+  /* re-parse the 'out1D_vars' parameter if it was changed */
+  out1D_vars_current_nset = CCTK_ParameterQueryTimesSet ("out1D_vars",
+                                                         CCTK_THORNSTRING);
+  if (out1D_vars_current_nset != out1D_vars_lastset)
+  {
+    IOUtil_ParseVarsForOutput (out1D_vars, myGH->do_out1D);
+
+    /* Save the last setting of 'out1D_vars' parameter */
+    out1D_vars_lastset = out1D_vars_current_nset;
+  }
+}
diff --git a/src/Output2D.c b/src/Output2D.c
new file mode 100644
index 0000000..b07efc8
--- /dev/null
+++ b/src/Output2D.c
@@ -0,0 +1,275 @@
+ /*@@
+   @file      Output2D.c
+   @date      Tue Jan 9 1999
+   @author    Gabrielle Allen
+   @desc 
+   Functions to deal 2D HDF5 output of variables
+   @enddesc 
+   @history
+   @hauthor Thomas Radke @hdate 16 Mar 1999
+   @hdesc Converted to Cactus 4.0
+   @hendhistory
+ @@*/
+
+
+
+#include <stdio.h>
+#include <stdlib.h>
+
+#include "cctk.h"
+#include "cctk_Parameters.h"
+#include "CactusBase/IOUtil/src/ioGH.h"
+#include "ioHDF5GH.h"
+
+
+/* function prototypes */
+int IOHDF5_TimeFor2D (cGH *GH, int index);
+int IOHDF5_Output2DVarAs (cGH *GH, const char *var, const char *alias);
+static void CheckSteerableParameters (ioHDF5GH *myGH);
+
+
+/*@@
+   @routine    IOHDF5_Output2DGH
+   @date       Sat March 6 1999
+   @author     Gabrielle Allen
+   @desc
+   Loops over all variables and outputs them if necessary
+   @enddesc
+   @calledby   CCTK_OutputGH ("IOHDF5_2D")
+   @history
+
+   @endhistory
+   @var     GH
+   @vdesc   Pointer to CCTK GH
+   @vtype   cGH
+   @vio     in
+   @endvar
+@@*/
+
+int IOHDF5_Output2DGH (cGH *GH)
+{
+  DECLARE_CCTK_PARAMETERS
+  int i;
+  ioHDF5GH *myGH;
+  const char *name;
+  char *fullname;
+  int ierr=0;
+
+  /* Get the GH extension for IOHDF5 */
+  myGH = (ioHDF5GH *) GH->extensions [CCTK_GHExtensionHandle ("IOHDF5")];
+
+  CheckSteerableParameters (myGH);
+
+  if (myGH->out2D_every <= 0)
+    return (0);
+
+  /* Loop over all variables */
+  for (i = 0; i < CCTK_NumVars (); i++)
+  {
+
+    if (IOHDF5_TimeFor2D (GH, i))
+    {
+
+      name     = CCTK_VarName (i);
+      fullname = CCTK_FullName (i);
+
+      if (verbose)
+        CCTK_VInfo (CCTK_THORNSTRING, "IOHDF5_Output2DGH: fullname / name = "
+                    "%s / %s", fullname, name);
+
+      ierr=IOHDF5_Output2DVarAs (GH, fullname, name);
+
+      free (fullname);
+
+      /* Register variable as having 2D output this iteration */
+      myGH->out2D_last [i] = GH->cctk_iteration;
+    }
+  }
+  if (verbose)
+    CCTK_INFO("IOHDF5_Output2DGH Done \n\n");
+
+  return (ierr);
+}
+
+
+/*@@
+   @routine    IOHDF5_Output2DVarAs
+   @date       Sat March 6 1999
+   @author     Gabrielle Allen
+   @desc
+   unconditional output of a variable using the IOHDF5 2D output method
+   @enddesc
+   @calledby IOHDF5_Output2DGH, CCTK_OutputVarAsByMethod ("IOHDF5_2D")
+   @var     GH
+   @vdesc   Pointer to CCTK GH
+   @vtype   cGH
+   @vio     in
+   @vcomment
+   @endvar
+   @var     fullname
+   @vdesc   complete name of variable to output
+   @vtype   const char *
+   @vio     in
+   @vcomment
+   @endvar
+   @var     alias
+   @vdesc   alias name of variable to output (used to generate output filename)
+   @vtype   const char *
+   @vio     in
+   @vcomment
+   @endvar
+@@*/
+
+int IOHDF5_Output2DVarAs (cGH *GH, const char *fullname, const char *alias)
+{
+  DECLARE_CCTK_PARAMETERS
+  int index,ierr=0;
+
+  index = CCTK_VarIndex (fullname);
+
+  /* first, check if variable has storage assigned */
+  if (! CCTK_QueryGroupStorageI (GH, CCTK_GroupIndexFromVarI (index)))
+  {
+    CCTK_VWarn (2, __LINE__, __FILE__, CCTK_THORNSTRING,
+                "No IOHDF5 2D output for '%s' (no storage)", fullname);
+    return(-1);
+  }
+
+  if (verbose)
+    CCTK_VInfo (CCTK_THORNSTRING, "IOHDF5_Output2DVarAs: fullname, alias, "
+                "index = (%s, %s, %d)", fullname, alias, index); 
+
+  /* Do the 2D output */
+  ierr=IOHDF5_Write2D (GH, index, alias);
+
+  return (ierr);
+}
+
+
+/*@@
+   @routine    IOHDF5_TimeFor2D
+   @date       Sat March 6 1999
+   @author     Gabrielle Allen
+   @desc
+   Decides if it is time to output a variable using the IOHDF5 2D output
+   method
+   @enddesc
+   @calledby IOHDF5_Output2DGH
+   @history
+
+   @endhistory
+   @var     GH
+   @vdesc   Pointer to CCTK GH
+   @vtype   cGH
+   @vio     in
+   @vcomment
+   @endvar
+   @var     index
+   @vdesc   index of variable
+   @vtype   int
+   @vio     in
+   @vcomment
+   @endvar
+@@*/
+
+int IOHDF5_TimeFor2D (cGH *GH, int index)
+{
+  ioHDF5GH *myGH;
+
+
+  /* Get the GH extension for IOHDF5 */
+  myGH = (ioHDF5GH *) GH->extensions [CCTK_GHExtensionHandle ("IOHDF5")];
+
+  CheckSteerableParameters (myGH);
+
+  /* Check if any output was requested */
+  if(myGH->out2D_every <= 0)
+    return (0);
+
+  /* Check this variable should be output */
+  if (! (myGH->do_out2D [index] && GH->cctk_iteration % myGH->out2D_every == 0))
+    return (0);
+
+  /* Check variable not already output this iteration */
+  if (myGH->out2D_last [index] == GH->cctk_iteration)
+  {
+      CCTK_WARN (2, "Already done 2D output in IOHDF5");
+      return (0);
+  }
+
+  return (1);
+}
+
+
+/*@@
+   @routine    IOHDF5_TriggerOutput2D
+   @date       Sat March 6 1999
+   @author     Gabrielle Allen
+   @desc
+   Triggers the output a variable using the IOHDF5 2D output
+   method
+   @enddesc
+   @calledby   CCTK scheduler
+   @var     GH
+   @vdesc   Pointer to CCTK GH
+   @vtype   cGH
+   @vio     in
+   @vcomment
+   @endvar
+   @var     index
+   @vdesc   index of variable to output
+   @vtype   int
+   @vio     in
+   @vcomment
+   @endvar
+@@*/
+
+int IOHDF5_TriggerOutput2D (cGH *GH, int index)
+{
+  DECLARE_CCTK_PARAMETERS
+  ioHDF5GH *myGH;
+  const char *varname;
+  int ierr=0;
+
+  varname = CCTK_VarName (index);
+
+  myGH = (ioHDF5GH *) GH->extensions [CCTK_GHExtensionHandle ("IOHDF5")];
+
+  if (verbose)
+    CCTK_VInfo (CCTK_THORNSTRING, "IOHDF5_TriggerOutput2D: varname, index = "
+                "(%s, %d)", varname, index);
+
+  /* Do the 2D output */
+  ierr=IOHDF5_Write2D (GH, index, varname);
+
+  /* Register variable as having 2D output this iteration */
+  myGH->out2D_last [index] = GH->cctk_iteration;
+
+  return (ierr);
+}
+
+
+/**************************** local functions ******************************/
+static void CheckSteerableParameters (ioHDF5GH *myGH)
+{
+  DECLARE_CCTK_PARAMETERS
+  int out2D_vars_current_nset;
+  static int out2D_vars_lastset = 0;
+
+
+  /* How often to output */
+  myGH->out2D_every = out_every > 0 ? out_every : -1;
+  if (out2D_every > 0)
+    myGH->out2D_every = out2D_every;
+
+  /* re-parse the 'out2D_vars' parameter if it was changed */
+  out2D_vars_current_nset = CCTK_ParameterQueryTimesSet ("out2D_vars",
+                                                         CCTK_THORNSTRING);
+  if (out2D_vars_current_nset != out2D_vars_lastset)
+  {
+    IOUtil_ParseVarsForOutput (out2D_vars, myGH->do_out2D);
+
+    /* Save the last setting of 'out2D_vars' parameter */
+    out2D_vars_lastset = out2D_vars_current_nset;
+  }
+}
diff --git a/src/Output3D.c b/src/Output3D.c
new file mode 100644
index 0000000..caaca4a
--- /dev/null
+++ b/src/Output3D.c
@@ -0,0 +1,266 @@
+ /*@@
+   @file      Output3D.c
+   @date      Tue Jan 9 1999
+   @author    Gabrielle Allen
+   @desc 
+   Functions to deal 3D HDF5 output of variables
+   @enddesc 
+   @history
+   @hauthor Thomas Radke @hdate 16 Mar 1999
+   @hdesc Converted to Cactus 4.0
+   @hendhistory
+ @@*/
+
+
+
+#include <stdio.h>
+#include <stdlib.h>
+
+#include "cctk.h"
+#include "cctk_Parameters.h"
+#include "CactusBase/IOUtil/src/ioGH.h"
+#include "ioHDF5GH.h"
+
+
+/* function prototypes */
+int IOHDF5_TimeFor3D (cGH *GH, int index);
+int IOHDF5_Output3DVarAs (cGH *GH, const char *var, const char *alias);
+static void CheckSteerableParameters (ioHDF5GH *myGH);
+
+
+/*@@
+   @routine    IOHDF5_Output3DGH
+   @date       Sat March 6 1999
+   @author     Gabrielle Allen
+   @desc
+   Loops over all variables and outputs them if necessary
+   @enddesc
+   @calledby   CCTK_OutputGH ("IOHDF5_3D")
+   @history
+
+   @endhistory
+   @var     GH
+   @vdesc   Pointer to CCTK GH
+   @vtype   cGH
+   @vio     in
+   @endvar
+@@*/
+
+int IOHDF5_Output3DGH (cGH *GH)
+{
+  DECLARE_CCTK_PARAMETERS
+  int i;
+  ioHDF5GH *myGH;
+  const char *name;
+  char *fullname;
+
+  /* Get the GH extension for IOHDF5 */
+  myGH = (ioHDF5GH *) GH->extensions [CCTK_GHExtensionHandle ("IOHDF5")];
+
+  CheckSteerableParameters (myGH);
+
+  if (myGH->out3D_every <= 0)
+    return (0);
+
+  /* Loop over all variables */
+  for (i = 0; i < CCTK_NumVars (); i++)
+  {
+
+    if (IOHDF5_TimeFor3D (GH, i))
+    {
+
+      name     = CCTK_VarName (i);
+      fullname = CCTK_FullName (i);
+
+      if (verbose)
+        CCTK_VInfo (CCTK_THORNSTRING, "IOHDF5_Output3DGH: fullname / name = "
+                    "%s / %s", fullname, name);
+
+      IOHDF5_Output3DVarAs (GH, fullname, name);
+
+      free (fullname);
+
+      /* Register variable as having 3D output this iteration */
+      myGH->out3D_last [i] = GH->cctk_iteration;
+    }
+  }
+  if (verbose)
+    CCTK_INFO("IOHDF5_Output3DGH Done \n\n");
+
+  return (0);
+}
+
+
+/*@@
+   @routine    IOHDF5_Output3DVarAs
+   @date       Sat March 6 1999
+   @author     Gabrielle Allen
+   @desc
+   unconditional output of a variable using the IOHDF5 3D output method
+   @enddesc
+   @calledby IOHDF5_Output3DGH, CCTK_OutputVarAsByMethod ("IOHDF5_3D")
+   @var     GH
+   @vdesc   Pointer to CCTK GH
+   @vtype   cGH
+   @vio     in
+   @vcomment
+   @endvar
+   @var     fullname
+   @vdesc   complete name of variable to output
+   @vtype   const char *
+   @vio     in
+   @vcomment
+   @endvar
+   @var     alias
+   @vdesc   alias name of variable to output (used to generate output filename)
+   @vtype   const char *
+   @vio     in
+   @vcomment
+   @endvar
+@@*/
+
+int IOHDF5_Output3DVarAs (cGH *GH, const char *fullname, const char *alias)
+{
+  DECLARE_CCTK_PARAMETERS
+  int index;
+
+
+  index = CCTK_VarIndex (fullname);
+
+  if (verbose)
+    CCTK_VInfo (CCTK_THORNSTRING, "IOHDF5_Output3DVarAs: fullname, alias, "
+                "index = (%s, %s, %d)", fullname, alias, index); 
+
+  /* Do the 3D output */
+  IOHDF5_Write3D (GH, index, alias);
+
+  return (0);
+}
+
+
+/*@@
+   @routine    IOHDF5_TimeFor3D
+   @date       Sat March 6 1999
+   @author     Gabrielle Allen
+   @desc
+   Decides if it is time to output a variable using the IOHDF5 3D output
+   method
+   @enddesc
+   @calledby IOHDF5_Output3DGH
+   @history
+
+   @endhistory
+   @var     GH
+   @vdesc   Pointer to CCTK GH
+   @vtype   cGH
+   @vio     in
+   @vcomment
+   @endvar
+   @var     index
+   @vdesc   index of variable
+   @vtype   int
+   @vio     in
+   @vcomment
+   @endvar
+@@*/
+
+int IOHDF5_TimeFor3D (cGH *GH, int index)
+{
+  ioHDF5GH *myGH;
+
+
+  /* Get the GH extension for IOHDF5 */
+  myGH = (ioHDF5GH *) GH->extensions [CCTK_GHExtensionHandle ("IOHDF5")];
+
+  CheckSteerableParameters (myGH);
+
+  /* Check if any output was requested */
+  if(myGH->out3D_every <= 0)
+    return (0);
+
+  /* Check this variable should be output */
+  if (! (myGH->do_out3D [index] && GH->cctk_iteration % myGH->out3D_every == 0))
+    return (0);
+
+  /* Check variable not already output this iteration */
+  if (myGH->out3D_last [index] == GH->cctk_iteration)
+  {
+      CCTK_WARN (2, "Already done 3D output in IOHDF5");
+      return (0);
+  }
+
+  return (1);
+}
+
+
+/*@@
+   @routine    IOHDF5_TriggerOutput3D
+   @date       Sat March 6 1999
+   @author     Gabrielle Allen
+   @desc
+   Triggers the output a variable using the IOHDF5 3D output
+   method
+   @enddesc
+   @calledby   CCTK scheduler
+   @var     GH
+   @vdesc   Pointer to CCTK GH
+   @vtype   cGH
+   @vio     in
+   @vcomment
+   @endvar
+   @var     index
+   @vdesc   index of variable to output
+   @vtype   int
+   @vio     in
+   @vcomment
+   @endvar
+@@*/
+
+int IOHDF5_TriggerOutput3D (cGH *GH, int index)
+{
+  DECLARE_CCTK_PARAMETERS
+  ioHDF5GH *myGH;
+  const char *varname;
+
+  varname = CCTK_VarName (index);
+
+  myGH = (ioHDF5GH *) GH->extensions [CCTK_GHExtensionHandle ("IOHDF5")];
+
+  if (verbose)
+    CCTK_VInfo (CCTK_THORNSTRING, "IOHDF5_TriggerOutput3D: varname, index = "
+                "(%s, %d)", varname, index);
+
+  /* Do the 3D output */
+  IOHDF5_Write3D (GH, index, varname);
+
+  /* Register variable as having 3D output this iteration */
+  myGH->out3D_last [index] = GH->cctk_iteration;
+
+  return (0);
+}
+
+
+/**************************** local functions ******************************/
+static void CheckSteerableParameters (ioHDF5GH *myGH)
+{
+  DECLARE_CCTK_PARAMETERS
+  int out3D_vars_current_nset;
+  static int out3D_vars_lastset = 0;
+
+
+  /* How often to output */
+  myGH->out3D_every = out_every > 0 ? out_every : -1;
+  if (out3D_every > 0)
+    myGH->out3D_every = out3D_every;
+
+  /* re-parse the 'out3D_vars' parameter if it was changed */
+  out3D_vars_current_nset = CCTK_ParameterQueryTimesSet ("out3D_vars",
+                                                         CCTK_THORNSTRING);
+  if (out3D_vars_current_nset != out3D_vars_lastset)
+  {
+    IOUtil_ParseVarsForOutput (out3D_vars, myGH->do_out3D);
+
+    /* Save the last setting of 'out3D_vars' parameter */
+    out3D_vars_lastset = out3D_vars_current_nset;
+  }
+}
diff --git a/src/ParseGeometry.c b/src/ParseGeometry.c
new file mode 100644
index 0000000..61297b3
--- /dev/null
+++ b/src/ParseGeometry.c
@@ -0,0 +1,147 @@
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+
+#include "cctk.h"
+#include "cctk_Arguments.h"
+#include "cctk_Parameters.h"
+
+#include "util_String.h"   
+#include "ioHDF5GH.h"
+
+
+void IOHDF5_DefaultGeo(int sdim, ioHDF5Geo_t *geo) {
+  DECLARE_CCTK_PARAMETERS
+
+  int idim;
+  const char *tmp_origin, *tmp_downs, *tmp_length;
+  const char *token;
+
+  geo->vdim = -1;
+  geo->sdim = -1;
+
+  for (idim=0;idim<IOHDF5_MAXDIM;idim++) {
+    geo->direction[idim] = idim;
+    geo->slab_start[idim]= 0;
+    geo->length[idim]    =-1;
+    geo->actlen[idim]    =-1;
+    geo->downs[idim]     = 1;
+  }
+
+  /* Parse the parameter of the requested dimension */
+  switch (sdim) {
+    case 1: 
+      tmp_origin = origin1D; 
+      tmp_downs  = downsampling1D;
+      tmp_length = length1D; 
+      break;
+    case 2: 
+      tmp_origin = origin2D; 
+      tmp_downs  = downsampling2D;   
+      tmp_length = length2D; 
+      break;
+    case 3: 
+      tmp_origin = origin3D; 
+      tmp_downs  = downsampling3D;   
+      tmp_length = length3D; 
+      break;
+    case 4: 
+      tmp_origin = origin4D; 
+      tmp_downs  = downsampling4D;   
+      tmp_length = length4D; 
+      break;    
+    default: 
+      tmp_origin = origin; 
+      tmp_downs  = downsampling;
+      tmp_length = length; 
+      break;
+  }
+
+  /* Origin */
+  idim=0;
+  while((token = Util_StrSep(&tmp_origin,","))) {
+    geo->slab_start[idim++]=atoi(token);
+  }
+  geo->slab_start[idim] = atoi(tmp_origin);
+
+  /* Downsample */
+  idim=0;
+   while((token = Util_StrSep(&tmp_downs,","))) {
+    geo->downs[idim++]=atoi(token);
+  }
+  geo->downs[idim] = atoi(tmp_downs);
+
+  /* Length */
+  idim=0;
+  while((token = Util_StrSep(&tmp_length,","))) {
+    geo->length[idim++]=atoi(token);
+  }
+  geo->length[idim] = atoi(tmp_length);
+}
+
+
+int NumDirection(int sdim, int vdim)
+{
+  int numdir=-1;
+  if (vdim==3) {
+    switch (sdim) { 
+      case 1: return(3);
+      case 2: return(3);
+      case 3: return(1);
+    }
+  } else if (vdim==2) {
+    switch (sdim) { 
+      case 1: return(2);
+      case 2: return(1);
+    }
+  }
+  else if (vdim==1) {
+    switch (sdim) { 
+      case 1: return(1);
+    }
+  }
+  else printf("Bad dimension: %d \n",vdim);
+  return(numdir);
+}
+
+int SetDirection(int vdim, int sdim, int ci, int *direction)
+{
+  int retval=0;
+
+  if (sdim>vdim) {
+    printf("SetDirection: slabdim gt vdim");
+    return(-1);
+  }
+  
+  if (vdim==3) {
+    if (sdim==2) {
+      switch (ci) { 
+        case 0: direction[0]=0; direction[1]=1;  break;
+        case 1: direction[0]=0; direction[1]=2;  break;
+        case 2: direction[0]=1; direction[1]=2;  break;
+      }
+    }
+    else if (sdim==1) {
+      switch (ci) { 
+        case 0: direction[0]=0; break;
+        case 1: direction[0]=1; break;
+        case 2: direction[0]=2; break;
+      }
+    }
+  }
+  else if (vdim==2) {
+    if (sdim==2) {
+      direction[0]=0; direction[1]=1; 
+    } else if (sdim==1) {
+      switch (ci) { 
+        case 0: direction[0]=0; break;
+        case 1: direction[0]=1; break;
+      }
+    }
+  } else retval = -1;
+  return(retval);
+}
+    
+      
+    
+      
diff --git a/src/RecoverGH.c b/src/RecoverGH.c
new file mode 100644
index 0000000..d2047e9
--- /dev/null
+++ b/src/RecoverGH.c
@@ -0,0 +1,491 @@
+ /*@@
+   @file      RecoverGH.c
+   @date      Fri Jun 19 09:14:22 1998
+   @author    Tom Goodale
+   @desc 
+   Contains the routines to do the internal checkpoint recovery.
+
+   Currently can recover from:
+     (1) One file containing recombined data
+     (2) Multiple unrecombined files, where the current
+         number of processors and IO processors
+         match those used to write the files.
+   @enddesc 
+   @history
+   @hauthor Gabrielle Allen @hdate 19 Oct 1998      
+   @hdesc Changed names ready for thorn_IO
+   @endhistory
+   @version $Id$
+ @@*/
+
+
+#include <ctype.h>
+#include <stdio.h>
+#include <stdlib.h>
+
+#include "cctk.h"
+#include "cctk_Parameters.h"
+#include "CactusPUGH/PUGH/src/include/pugh.h"
+#include "CactusBase/IOUtil/src/ioGH.h"
+#include "ioHDF5GH.h"
+
+
+/* the rcs ID and its dummy funtion to use it */
+static char *rcsid = "$Id$";
+CCTK_FILEVERSION(CactusPUGHIO_IOHDF5_RecoverGH_c)
+
+
+/* local function prototypes */
+static int IOHDF5i_RecoverParameters (fileinfo_t *file);
+static int IOHDF5i_RecoverGHextensions (cGH *GH, fileinfo_t *file);
+static int IOHDF5i_OpenFile (cGH *GH, const char *basename, int called_from,
+                             fileinfo_t *file);
+/* this one comes from RestoreFile.c */
+int IOHDF5i_RecoverVariables (cGH *GH, fileinfo_t *file);
+
+
+ /*@@
+   @routine    IOHDF5_Recover
+   @date       Fri Jun 19 09:22:52 1998
+   @author     Tom Goodale
+   @desc 
+               Recovers a GH from an HDF5 file.
+               This routine is registered with IOUtil
+               as IOHDF5's recovery routine.
+   @enddesc 
+   @calledby   IOUtil_RecoverFromFile
+   @history 
+   @endhistory 
+   @var        GH
+   @vdesc      Pointer to CCTK grid hierarchy
+   @vtype      cGH
+   @vio        in
+   @endvar
+   @var        basename
+   @vdesc      the basename of the file to recover from
+               The file suffix is appended by the routine.
+   @vtype      const char *
+   @vio        in
+   @endvar
+   @var        called_from
+   @vdesc      flag indicating where this routine was called from
+               (either RECOVER or filereader)
+   @vtype      int
+   @vio        in
+   @endvar
+@@*/
+int IOHDF5_Recover (cGH *GH, const char *basename, int called_from)
+{
+  DECLARE_CCTK_PARAMETERS
+  int result;
+  ioHDF5GH *myGH;
+  static fileinfo_t file;      /* this is static because info is passed from
+                                  CP_RECOVERY_PARAMETERS to CP_RECOVERY_DATA */
+
+  /* to make the compiler happy */
+  myGH = NULL;
+
+  /* start the total timer */
+  if (GH)
+  {
+    myGH = (ioHDF5GH *) GH->extensions [CCTK_GHExtensionHandle ("IOHDF5")];
+    if (myGH->print_timing_info)
+    {
+      CCTK_TimerStartI (myGH->timers[3]);
+    }
+  }
+
+  /* open the file if it wasn't already opened at CCTK_RECOVER_PARAMETERS */
+  /* FIXME Gab ... asymmetric levfac */
+  if (called_from == CP_RECOVER_PARAMETERS || called_from == FILEREADER_DATA ||
+      (GH && (GH->cctk_levfac[0] > 1 || GH->cctk_convlevel > 0))) {
+    if (IOHDF5i_OpenFile (GH, basename, called_from, &file) < 0)
+      return (-1);
+  } else {
+    /* This is the case for CP_RECOVER_DATA.
+       CCTK_RECOVER_PARAMETERS must have been called before 
+       and set up the file info structure. */
+    if (! file.is_HDF5_file)
+      return (-1);
+  }
+
+  /* Recover parameters (and return) */
+  if (called_from == CP_RECOVER_PARAMETERS)
+    return (IOHDF5i_RecoverParameters (&file));
+
+  /* Recover variables */
+  if (verbose)
+    CCTK_VInfo (CCTK_THORNSTRING, "Recovering %schunked data with ioproc %d, "
+                "ioproc_every %d", file.unchunked ? "un" : "", file.ioproc,
+                file.ioproc_every);
+  result = IOHDF5i_RecoverVariables (GH, &file);
+
+  /* Recover GH extensions */
+  if (result == 0 && called_from == CP_RECOVER_DATA) {
+    if (verbose)
+      CCTK_INFO ("Recovering GH extensions");
+    result = IOHDF5i_RecoverGHextensions (GH, &file);
+  }
+
+  /* Close the file */
+  if (CCTK_MyProc (GH) == file.ioproc) {
+    if (verbose) {
+      if (called_from == CP_RECOVER_DATA)
+        CCTK_VInfo (CCTK_THORNSTRING, "Closing checkpoint file '%s' after "
+                    "recovery", file.filename);
+      else
+        CCTK_VInfo (CCTK_THORNSTRING, "Closing data file '%s'", file.filename);
+    }
+    CACTUS_IOHDF5_ERROR (H5Fclose (file.fid));
+  }
+ 
+  /* stop total recovery timer and print timing info */
+  if (called_from == CP_RECOVER_DATA && myGH->print_timing_info)
+  {
+    const char *timer_description = "Time to recover:";
+
+
+    CCTK_TimerStopI (myGH->timers[3]);
+    IOUtil_PrintTimings ("Timing information for recovery in IOHDF5:", 1,
+                         &myGH->timers[3], &timer_description);
+  }
+
+  return (result);
+}
+
+
+ /*@@
+   @routine    IOHDF5_RecoverParameters
+   @date       Thu Apr 13 2000
+   @author     Thomas Radke
+   @desc
+               Recovers the parameters from an HDF5 checkpoint file.
+               This routine is scheduled at CCTK_RECOVER_PARAMETERS.
+
+               Note that it cannot be registered with IOUtil to be scheduled
+               from there (as done with the IOHDF5_Recover routine) because
+               the registration mechanism isn't activated yet
+               at CCTK_RECOVER_PARAMETERS.
+               Instead we call the generic parameter recovery routine
+               from IOUtil here, and just pass the necessary callback function
+               and its arguments.
+
+               Note also that this routine doesn't get passed any parameters,
+               not even a GH, because this doesn't exist yet at the time it is
+               being called.
+   @enddesc
+   @calledby   flesh's scheduler
+   @history
+   @endhistory
+
+@@*/
+int IOHDF5_RecoverParameters (void)
+{
+  return (IOUtil_RecoverParameters (IOHDF5_Recover, ".h5", "HDF5"));
+}
+
+
+/**************************** local routines ******************************/
+
+static int IOHDF5i_OpenFile (cGH *GH, const char *basename, int called_from,
+                             fileinfo_t *file)
+{
+  DECLARE_CCTK_PARAMETERS
+  hid_t group;
+  int nprocs, myproc;
+#ifdef CCTK_MPI
+  MPI_Comm comm;
+  CCTK_INT4 info [3];
+#endif
+
+
+#ifdef CCTK_MPI
+  /* Get the communicator for broadcasting the info structure */
+  /* NOTE: When recovering parameters thorn PUGH is not yet initialized
+           so that we have to use MPI_COMM_WORLD in this case */
+  comm = CCTK_GHExtensionHandle ("PUGH") < 0 ?
+           MPI_COMM_WORLD : PUGH_pGH (GH)->PUGH_COMM_WORLD;
+#endif
+
+  /* identify myself */
+  nprocs = CCTK_nProcs (GH);
+  myproc = CCTK_MyProc (GH);
+
+  /* Examine basefile to find out whether we are recovering from
+   * one or multiple files and whether the data are chunked or not.
+   *
+   * This is done by processor 0 only since this is always an IO processor
+   * and a corresponding file must exist in all cases.
+   */
+
+  /* Determine name of base file
+     NOTE: As we don't know whether the file is chunked or not
+           we need to try both file names. */
+  /* at first try with unchunked mode */
+  file->unchunked = 1;
+  IOUtil_PrepareFilename (GH, basename, file->filename, called_from, 0,
+                          file->unchunked);
+  strcat (file->filename, ".h5");
+
+  if (myproc == 0) {
+    H5E_auto_t printErrorFn;
+    void      *printErrorFnArg;
+
+    if (verbose)
+      CCTK_VInfo (CCTK_THORNSTRING, "Opening file '%s'", file->filename);
+
+    /* turn automatic error printing off again during check
+       but preserving the previous error handler settings */
+    CACTUS_IOHDF5_ERROR (H5Eget_auto (&printErrorFn, &printErrorFnArg));
+    CACTUS_IOHDF5_ERROR (H5Eset_auto (NULL, NULL));
+
+    /* Check the filetype */
+    if (H5Fis_hdf5 (file->filename) > 0)
+      file->is_HDF5_file = 1;
+    else {
+      CCTK_VWarn (2, __LINE__, __FILE__, CCTK_THORNSTRING,
+                  "No valid HDF5 file '%s' found", file->filename);
+
+      /* now try with chunked mode */
+      file->unchunked = 0;
+      IOUtil_PrepareFilename (GH, basename, file->filename, called_from, 0,
+                              file->unchunked);
+      strcat (file->filename, ".h5");
+
+      if (verbose)
+        CCTK_VInfo (CCTK_THORNSTRING, "Trying now file '%s'...",file->filename);
+
+      file->is_HDF5_file = H5Fis_hdf5 (file->filename) > 0;
+    }
+
+    /* turn automatic error printing on again */
+    CACTUS_IOHDF5_ERROR (H5Eset_auto (printErrorFn, printErrorFnArg));
+  }
+
+  /* Okay, we have the complete filename. Let's read the file now. */
+  if (myproc == 0 && file->is_HDF5_file) {
+
+    file->is_HDF5_file = 0;
+
+    /* Determine how the data was written by reading the GH extensions
+       from the predefined group GHEXTENSIONS_GROUP */
+    file->fid = H5Fopen (file->filename, H5F_ACC_RDONLY, H5P_DEFAULT);
+    if (file->fid < 0) {
+      CCTK_VWarn (1, __LINE__, __FILE__, CCTK_THORNSTRING,
+                  "Cannot open file '%s' !", file->filename);
+    } else if ((group = H5Gopen (file->fid, GHEXTENSIONS_GROUP)) < 0) {
+      CCTK_WARN (1, "Can't find GH extensions group. "
+                 "Is this really a Cactus HDF5 datafile ?");
+    } else {
+
+      READ_ATTRIBUTE (group, "nprocs", H5T_NATIVE_INT, &file->nprocs);
+      READ_ATTRIBUTE (group, "unchunked", H5T_NATIVE_INT, &file->unchunked);
+      READ_ATTRIBUTE (group, "ioproc_every",H5T_NATIVE_INT,&file->ioproc_every);
+
+      CACTUS_IOHDF5_ERROR (H5Gclose (group));
+
+      /* If we recover from multiple files the number of
+       * writing processors must match the number of reading 
+       * processors, and the total number of processors must match.
+       */
+ 
+      if ((file->ioproc_every == nprocs && nprocs > 1) || file->unchunked) {
+        if (verbose)
+          CCTK_VInfo (CCTK_THORNSTRING, "Recovering from one %schunked file",
+                      file->unchunked ? "un":"");
+        file->ioproc_every = nprocs;
+        file->is_HDF5_file = 1;
+      } else {
+        if (file->nprocs != nprocs) {
+          CCTK_VWarn (1, __LINE__, __FILE__, CCTK_THORNSTRING,
+                      "Must restart on %d processors with chunked files "
+                      "or recombine them", file->nprocs);
+        } else {
+          if (verbose)
+            CCTK_VInfo (CCTK_THORNSTRING, "Recovering from %d chunked files",
+                        nprocs / file->ioproc_every +
+                        (nprocs % file->ioproc_every ? 1 : 0));
+          file->is_HDF5_file = 1;
+        }
+      }
+    }
+  }
+
+  if (myproc == 0 && ! file->is_HDF5_file)
+    CCTK_VWarn (2, __LINE__, __FILE__, CCTK_THORNSTRING,
+                "No valid HDF5 file '%s' found", file->filename);
+
+#ifdef CCTK_MPI
+  /* Broadcast the file information to all processors */
+  info [0] = file->is_HDF5_file;
+  info [1] = file->unchunked;
+  info [2] = file->ioproc_every;
+  CACTUS_MPI_ERROR (MPI_Bcast (info, 3, PUGH_MPI_INT4, 0, comm));
+  file->is_HDF5_file = info [0];
+  file->unchunked = info [1];
+  file->ioproc_every = info [2];
+#endif
+
+  if (file->is_HDF5_file) {
+
+    /* Determine the IO processors for each node and the corresponding
+       checkpoint file */
+    file->ioproc = myproc - (myproc % file->ioproc_every);
+    IOUtil_PrepareFilename (GH, basename, file->filename, called_from,
+                            file->ioproc/file->ioproc_every, file->unchunked);
+    strcat (file->filename, ".h5");
+
+    /* Open chunked files on other IO processors */
+    if (myproc == file->ioproc && myproc != 0) {
+
+      if (verbose)
+        CCTK_VInfo (CCTK_THORNSTRING, "Opening chunked file '%s' on "
+                    "processor %d", file->filename, myproc);
+
+      /* Open file, make sure the file is valid */
+      file->fid = H5Fopen (file->filename, H5F_ACC_RDONLY, H5P_DEFAULT);
+      if (file->fid < 0) {
+        CCTK_VWarn (1, __LINE__, __FILE__, CCTK_THORNSTRING,
+              "Cannot open file '%s' on processor %d", file->filename, myproc);
+        file->is_HDF5_file = 0;
+      }
+    }
+#ifdef CCTK_MPI
+    /* Finally check whether all files have valid recovery files */
+    info [0] = file->is_HDF5_file;
+    CACTUS_MPI_ERROR (MPI_Allreduce (&info [0], &info [1], 1,
+                                     PUGH_MPI_INT4, MPI_LAND, comm));
+    file->is_HDF5_file = info [1];
+#endif
+  }
+
+  /* Return 0 for success otherwise negative */
+  return (file->is_HDF5_file ? 0 : -1);
+}
+
+
+/* NOTE: Although we could read the GH extensions from multiple recovery files
+         in parallel, this is done only on by processor 0 here.
+         Broadcasting the GH extensions is found faster than
+         sending it in a loop from each IO processor to all the non IOs
+         (don't have subcommunicators yet) */
+static int IOHDF5i_RecoverGHextensions (cGH *GH, fileinfo_t *file)
+{
+  hid_t group;
+  CCTK_REAL4 real4Buffer;
+  CCTK_INT4  int4Buffer [3];
+  extern int CCTK_SetMainLoopIndex (int main_loop_index);
+
+
+  if (CCTK_MyProc (GH) == 0) {
+    /* the GH extensions are stored in the GHEXTENSIONS_GROUP group */
+    group = H5Gopen (file->fid, GHEXTENSIONS_GROUP);
+    int4Buffer [0] = group >= 0;
+
+    if (int4Buffer [0]) {
+
+      READ_ATTRIBUTE (group, "cctk_iteration", IOHDF5_INT4, &int4Buffer [1]);
+      READ_ATTRIBUTE (group, "main_loop_index", IOHDF5_INT4, &int4Buffer [2]);
+      READ_ATTRIBUTE (group, "cctk_time", IOHDF5_REAL4, &real4Buffer);
+
+      CACTUS_IOHDF5_ERROR (H5Gclose (group));
+
+    } else
+      CCTK_WARN (1, "Can't find global attributes group. "
+                    "Is this really a Cactus HDF5 datafile ?");
+  }
+
+#ifdef CCTK_MPI
+  /* Broadcast the GH extensions to all processors */
+  /* NOTE: We have to use MPI_COMM_WORLD here 
+     because PUGH_COMM_WORLD is not yet set up at parameter recovery time.
+     We also assume that PUGH_MPI_INT4 is a compile-time defined datatype. */
+  CACTUS_MPI_ERROR (MPI_Bcast (int4Buffer, 3, PUGH_MPI_INT4, 0,MPI_COMM_WORLD));
+  if (int4Buffer [0])
+    CACTUS_MPI_ERROR (MPI_Bcast (&real4Buffer, 1, PUGH_MPI_REAL4, 0,
+                      MPI_COMM_WORLD));
+#endif
+
+  if (int4Buffer [0]) {
+    GH->cctk_time = real4Buffer;
+    GH->cctk_iteration = int4Buffer [1];
+    CCTK_SetMainLoopIndex ((int) int4Buffer [2]);
+  }
+
+  /* Return 0 for success otherwise negative */
+  return (int4Buffer [0] ? 0 : -1);
+}
+
+
+/* NOTE: Although we could read the parameters from multiple recovery files
+         in parallel, this is done only on by processor 0 here.
+         Broadcasting the complete parameter string is found faster than
+         sending it in a loop from each IO processor to all the non IOs
+         (don't have subcommunicators yet) */
+static int IOHDF5i_RecoverParameters (fileinfo_t *file)
+{
+  DECLARE_CCTK_PARAMETERS
+  hid_t group, attr, atype;
+  char *parameters;
+  CCTK_INT4 parameterSize=0;
+  cGH *GH = NULL;               /* There's no cGH set up yet so we pass
+                                   a NULL pointer to CCTK_MyProc() */
+
+  /* To make the compiler happy */
+  group = attr = 0;
+  parameters = NULL;
+
+  if (CCTK_MyProc (GH) == 0) {
+    if (verbose)
+      CCTK_VInfo (CCTK_THORNSTRING, "Recovering parameters from checkpoint "
+                  "file '%s'", file->filename);
+
+    /* the parameters are stored in the GLOBAL_PARAMETERS_GROUP group
+       in the attribute GLOBAL_PARAMETERS */
+    group = H5Gopen (file->fid, GLOBAL_PARAMETERS_GROUP);
+    if (group > 0)
+      attr = H5Aopen_name (group, GLOBAL_PARAMETERS);
+
+    if (group > 0 && attr > 0) {
+      CACTUS_IOHDF5_ERROR (atype = H5Aget_type (attr));
+      parameterSize = H5Tget_size (atype);
+      parameters = (char *) malloc (parameterSize + 1);
+      CACTUS_IOHDF5_ERROR (H5Aread (attr, atype, parameters));
+      parameters [parameterSize] = 0;
+      CACTUS_IOHDF5_ERROR (H5Tclose (atype));
+    } else
+      CCTK_WARN (1, "Can't find global parameters. "
+                    "Is this really a Cactus HDF5 checkpoint file ?");
+
+    if (attr > 0)
+      CACTUS_IOHDF5_ERROR (H5Aclose (attr));
+    if (group > 0)
+      CACTUS_IOHDF5_ERROR (H5Gclose (group));
+  }
+
+#ifdef CCTK_MPI
+  /* Broadcast the parameter buffer size to all processors */
+  /* NOTE: We have to use MPI_COMM_WORLD here 
+     because PUGH_COMM_WORLD is not yet set up at parameter recovery time.
+     We also assume that PUGH_MPI_INT4 is a compile-time defined datatype. */
+  CACTUS_MPI_ERROR (MPI_Bcast (&parameterSize, 1, PUGH_MPI_INT4, 0,
+                    MPI_COMM_WORLD));
+#endif
+
+  if (parameterSize > 0) {
+#ifdef CCTK_MPI
+    if (CCTK_MyProc (GH) != 0)
+      parameters = (char *) malloc (parameterSize + 1);
+
+    CACTUS_MPI_ERROR (MPI_Bcast (parameters, parameterSize + 1, PUGH_MPI_CHAR,
+                      0, MPI_COMM_WORLD));
+#endif
+
+    IOUtil_SetAllParameters (parameters);
+
+    free (parameters);
+  }
+
+  /* Return positive value for success otherwise negative */
+  return (parameterSize > 0 ? 1 : -1);
+}
diff --git a/src/RestoreFile.c b/src/RestoreFile.c
new file mode 100644
index 0000000..78eaa78
--- /dev/null
+++ b/src/RestoreFile.c
@@ -0,0 +1,580 @@
+ /*@@
+   @file      RestoreFile.c
+   @date      Thu Jun 18 16:34:59 1998
+   @author    Tom Goodale
+   @desc 
+   Routines to restore variables from a given file
+   @enddesc 
+   @history
+   @hauthor Gabrielle Allen @hdate 19 Oct 1998      
+   @hdesc Changed names ready for thorn_IO
+   @endhistory
+   @version $Id$
+ @@*/
+
+
+#include <stdio.h>
+#include <stdlib.h>
+
+#include "cctk.h"
+#include "cctk_Parameters.h"
+#include "CactusBase/IOUtil/src/ioGH.h"
+#include "CactusPUGH/PUGH/src/include/pugh.h"
+#include "ioHDF5GH.h"
+
+
+/* MPI tag base */
+#define STARTUPBASE 1001        
+
+
+typedef struct {
+  cGH *GH;
+  int ioproc;
+  int ioproc_every;
+  int unchunked;
+} iterate_info_t;
+
+typedef struct {
+  iterate_info_t *it_info;
+  int element_size;
+  int iohdf5_type;
+#ifdef CCTK_MPI
+  MPI_Datatype mpi_type;
+#endif
+} recover_info_t;
+
+
+/* local function prototypes */
+static herr_t processDataset (hid_t group, const char *datasetname, void *arg);
+
+
+ /*@@
+   @routine    IOHDF5i_RecoverVariables
+   @date       Fri Jun 19 09:19:48 1998
+   @author     Tom Goodale
+   @desc 
+   Reads in data from an open HDF5 file.
+   Code was originally in StartupReader.
+   @enddesc 
+   @calledby   IOHDF5_Recover
+   @history 
+   @endhistory 
+   @var        GH
+   @vdesc      Pointer to CCTK grid hierarchy
+   @vtype      cGH
+   @vio        in
+   @endvar
+   @var        file
+   @vdesc      HDF5 file info structure
+   @vtype      fileinfo_t *
+   @vio        in
+   @endvar
+@@*/
+int IOHDF5i_RecoverVariables (cGH *GH, fileinfo_t *file)
+{
+  DECLARE_CCTK_PARAMETERS
+#ifdef CCTK_MPI
+  pGH *pughGH;               /* PUGH extension handle */
+  int proc;                  /* looper */
+  CCTK_INT var_info [3];     /* communication buffer for MPI */
+#endif
+  iterate_info_t info;       /* info to pass down to the iterator routine */
+
+
+  info.GH = GH;
+  info.ioproc = file->ioproc;
+  info.unchunked = file->unchunked;
+  info.ioproc_every = file->ioproc_every;
+
+#ifdef CCTK_MPI
+  /* Get the handle for PUGH extensions */
+  pughGH = PUGH_pGH (GH);
+
+  /* Now process the datasets.
+     All IO processors read the datasets from their checkpoint file,
+     verify their contents and communicate them to the non-IO processors. */
+
+  /* At first the code for the IO processors.
+     This holds also for the single processor case. */
+  if (CCTK_MyProc (GH) == file->ioproc) {
+#endif
+
+    /* iterate over all datasets starting from "/" in the HDF5 file */
+    CACTUS_IOHDF5_ERROR (H5Giterate (file->fid, "/", NULL, processDataset,
+                         &info));
+
+#ifdef CCTK_MPI
+    /* To signal completion to the non-IO processors
+       an invalid variable index is broadcasted. */
+    var_info [0] = -1;
+    for (proc = 1; proc < file->ioproc_every; proc++)
+    for (proc = file->ioproc + 1;
+         proc < file->ioproc + file->ioproc_every && proc < CCTK_nProcs (GH);
+         proc++)
+      CACTUS_MPI_ERROR (MPI_Send (var_info, 3, PUGH_MPI_INT, proc,
+                        STARTUPBASE, pughGH->PUGH_COMM_WORLD));
+
+  } else {
+
+    /* And here the code for non-IO processors: */
+    MPI_Datatype mpi_type;
+    MPI_Status ms;
+    int index, timelevel, npoints;
+
+    /* They don't know how many datasets there are, because the IO processors
+       could skip some on the fly during their consistency checks.
+       The IO Processor sends the index of the variable to be processed next.
+       So, all non-IO processors execute a loop where the termination condition
+       is when an invalid index was received.
+    */
+    while (1) {
+      /* receive the next variable index from my IO processor */
+      CACTUS_MPI_ERROR (MPI_Recv (var_info, 3, PUGH_MPI_INT, file->ioproc,
+                        STARTUPBASE, pughGH->PUGH_COMM_WORLD, &ms));
+      index = var_info [0]; timelevel = var_info [1]; npoints = var_info [2];
+
+      /* check for termination condition */
+      if (index < 0)
+        break;
+
+      switch (CCTK_VarTypeI (index)) {
+        case CCTK_VARIABLE_CHAR:    mpi_type = PUGH_MPI_CHAR; break;
+        case CCTK_VARIABLE_INT:     mpi_type = PUGH_MPI_INT;  break;
+        case CCTK_VARIABLE_REAL:    mpi_type = PUGH_MPI_REAL; break;
+        case CCTK_VARIABLE_COMPLEX: mpi_type = pughGH->PUGH_mpi_complex; break;
+        default:
+          CCTK_VWarn (1, __LINE__, __FILE__, CCTK_THORNSTRING,
+                      "Unsupported datatype %d", CCTK_VarTypeI (index));
+          continue;
+      }
+
+      /* receive following data from my IO processor */
+      CACTUS_MPI_ERROR (MPI_Recv (GH->data [index][timelevel], npoints,mpi_type,
+                        file->ioproc, STARTUPBASE,pughGH->PUGH_COMM_WORLD,&ms));
+    }
+  }
+#endif /* MPI */
+
+  return (0);
+}
+
+
+/**************************** local routines ****************************/
+
+/* local routine GetCommonAttributes() reads in the next dataset's attributes
+   and verifies them:
+
+  * checks if there is a variable with the name given by the name attribute
+  * verifies that this variable still belongs to the same group
+  * checks the group data info:
+    - group type
+    - variable type
+    - ntimelevels
+    - sizes (rank, dimensions) according to chunking mode
+
+ If there is a mismatch a warning (warning level 2) is printed and value of 1
+ is returned to indicate that this dataset should be ignored.
+ If successful, the global variable index, the group type and the timelevel
+ to restore are stored in {*index, *gtype, *timelevel}, and 0 is returned.
+*/
+
+static int GetCommonAttributes (cGH *GH, hid_t dataset, const char *datasetname,
+                                int unchunked, int *index, int *grouptype,
+                                int *timelevel, int is_group)
+{
+  pGExtras *extras;
+  cGroup groupdata;
+  int i, flag, *dims;
+  hid_t datatype, dataspace;
+  hsize_t rank_stored, *dims_stored;
+  int grouptype_stored, ndims_stored, numtimelevels_stored;
+  char *groupname, groupname_stored [128], fullvarname [128];
+
+  /* decompose the datasetname, ignore the iteration number */
+  if (sscanf (datasetname, "%[^@]@%d@%d", fullvarname, &i, timelevel) != 3) {
+    CCTK_VWarn (2, __LINE__, __FILE__, CCTK_THORNSTRING,
+                "Cannot parse datasetname '%s'", datasetname);
+    return (1);
+  }
+
+  /* check if there is a matching variable */
+  *index = CCTK_VarIndex (fullvarname);
+  if (*index < 0) {
+    CCTK_VWarn (2, __LINE__, __FILE__, CCTK_THORNSTRING,
+                "No matching variable found for '%s'", fullvarname);
+    return (1);
+  }
+
+  /* read and verify the group name */
+  READ_ATTRIBUTE (dataset, "groupname", H5T_C_S1, groupname_stored);
+  groupname = CCTK_GroupNameFromVarI (*index);
+  if (! CCTK_Equals (groupname_stored, groupname)) {
+    CCTK_WARN (2, "Groupnames don't match");
+    return (1);
+  }
+  free (groupname);
+
+  /* verify group type, variable type, dims, sizes and ntimelevels */
+  READ_ATTRIBUTE (dataset, "grouptype", H5T_NATIVE_INT, &grouptype_stored);
+  READ_ATTRIBUTE (dataset, "ntimelevels", H5T_NATIVE_INT,&numtimelevels_stored);
+
+  if (CCTK_GroupData (CCTK_GroupIndex (groupname_stored), &groupdata) != 0) {
+    CCTK_VWarn (2, __LINE__, __FILE__, CCTK_THORNSTRING,
+                "Cannot get group data for '%s'", fullvarname);
+    return (1);
+  }
+  if (groupdata.grouptype != grouptype_stored) {
+    CCTK_VWarn (2, __LINE__, __FILE__, CCTK_THORNSTRING,
+                "Group types don't match for '%s'", fullvarname);
+    return (1);
+  }
+  if (groupdata.numtimelevels != numtimelevels_stored) {
+    CCTK_VWarn (2, __LINE__, __FILE__, CCTK_THORNSTRING,
+                "Number of timelevels don't match for '%s'", fullvarname);
+    return (1);
+  }
+  /* open the first chunk to determine datatype, dims and sizes
+     if the dataset is a chunk group */
+  if (is_group)
+    CACTUS_IOHDF5_ERROR (dataset = H5Dopen (dataset, "chunk0"));
+  CACTUS_IOHDF5_ERROR (datatype = H5Dget_type (dataset));
+
+  /* The CCTK variable type defines do not correlate with the HDF5 defines
+     so compare them explicitely here. */
+  flag = (H5Tget_class (datatype) == H5T_FLOAT &&
+          groupdata.vartype == CCTK_VARIABLE_REAL) ||
+         (H5Tget_class (datatype) == H5T_INTEGER &&
+          groupdata.vartype == CCTK_VARIABLE_INT) ||
+         (H5Tget_class (datatype) == H5T_INTEGER &&
+          groupdata.vartype == CCTK_VARIABLE_CHAR) ||
+         (H5Tget_class (datatype) == H5T_COMPOUND &&
+          groupdata.vartype == CCTK_VARIABLE_COMPLEX);
+
+  CACTUS_IOHDF5_ERROR (H5Tclose (datatype));
+  if (! flag) {
+    CCTK_VWarn (2, __LINE__, __FILE__, CCTK_THORNSTRING,
+                "Variable types don't match for '%s'", fullvarname);
+    return (1);
+  }
+
+  /* verify the dims and sizes */
+  CACTUS_IOHDF5_ERROR (dataspace = H5Dget_space (dataset));
+  CACTUS_IOHDF5_ERROR (ndims_stored = H5Sget_simple_extent_ndims (dataspace));
+  dims_stored = (hsize_t *) malloc (ndims_stored * sizeof (hsize_t));
+  CACTUS_IOHDF5_ERROR (rank_stored = H5Sget_simple_extent_dims (dataspace,
+                       dims_stored, NULL));
+  /* scalars are stored as rank 0 in HDF5 but have rank 1 in Cactus */
+  if (rank_stored == 0)
+    rank_stored = 1;
+  CACTUS_IOHDF5_ERROR (H5Sclose (dataspace));
+
+  flag = 0;
+  if (groupdata.dim != rank_stored)
+    flag = 1;
+  switch (groupdata.grouptype) {
+    case GROUP_SCALAR:
+      break;
+    case GROUP_ARRAY:
+    case GROUP_GF:
+      extras = ((pGA ***) PUGH_pGH (GH)->variables)[*index][*timelevel]->extras;
+      dims = unchunked ? extras->nsize : extras->lnsize;
+      for (i = 0; i < groupdata.dim; i++)
+        if (dims [groupdata.dim - i - 1] != dims_stored [i])
+          flag = 1;
+      break;
+  }
+
+  free (dims_stored);
+  if (flag) {
+    CCTK_VWarn (2, __LINE__, __FILE__, CCTK_THORNSTRING,
+                "Variable sizes don't match for '%s'", fullvarname);
+    return (1);
+  }
+
+  if (! CCTK_QueryGroupStorageI (GH, CCTK_GroupIndexFromVarI (*index))) {
+    CCTK_VWarn (2, __LINE__, __FILE__, CCTK_THORNSTRING,
+                "Can't read into variable '%s': no storage", fullvarname);
+    return (1);
+  }
+
+  /* close the first chunk if the dataset is a chunk group */
+  if (is_group)
+    CACTUS_IOHDF5_ERROR (H5Dclose (dataset));
+
+  *grouptype = groupdata.grouptype;
+
+  return (0);
+}
+
+
+static int IOHDF5_RestoreGS (hid_t dataset, int index, int timelevel,
+                             recover_info_t *rec_info)
+{
+  void *data;
+#ifdef CCTK_MPI
+  int proc;
+  CCTK_INT var_info [3];
+  pGH *pughGH;
+#endif
+
+
+  data = CCTK_VarDataPtrI (rec_info->it_info->GH, timelevel, index);
+
+  /* read the data into the local variable ... */
+  CACTUS_IOHDF5_ERROR (H5Dread (dataset, rec_info->iohdf5_type, H5S_ALL,
+                       H5S_ALL, H5P_DEFAULT, data));
+
+#ifdef CCTK_MPI
+  /* ... and communicate it for the MPI case */
+
+  /* Get the handle for PUGH extensions */
+  pughGH = PUGH_pGH (rec_info->it_info->GH);
+
+  /* set the variable's index and the timelevel */
+  var_info [0] = index; var_info [1] = timelevel; var_info [2] = 1;
+
+  /* send info and data to the non-IO processors */
+  for (proc = rec_info->it_info->ioproc + 1;
+       proc < rec_info->it_info->ioproc + rec_info->it_info->ioproc_every &&
+       proc < CCTK_nProcs (rec_info->it_info->GH);
+       proc++) {
+    CACTUS_MPI_ERROR (MPI_Send (var_info, 3, PUGH_MPI_INT, proc,
+                      STARTUPBASE, pughGH->PUGH_COMM_WORLD));
+    CACTUS_MPI_ERROR (MPI_Send (data, 1, rec_info->mpi_type, proc,
+                      STARTUPBASE, pughGH->PUGH_COMM_WORLD));
+  }
+#endif
+
+  return (0);
+}
+
+
+static int IOHDF5_RestoreGA (hid_t dataset, int index, int timelevel,
+                             recover_info_t *rec_info)
+{
+#ifdef CCTK_MPI
+  int proc, npoints;
+  CCTK_INT var_info [3];
+  pGH *pughGH;
+  void *buffer, *data;
+  hid_t filespace, memspace;
+  pGExtras *extras;
+#endif
+
+
+  /* single processor case is easy: just read the whole dataset */
+  if (CCTK_nProcs (rec_info->it_info->GH) == 1) {
+    CACTUS_IOHDF5_ERROR (H5Dread (dataset, rec_info->iohdf5_type, H5S_ALL,
+         H5S_ALL, H5P_DEFAULT, rec_info->it_info->GH->data [index][timelevel]));
+    return (0);
+  }
+
+#ifdef CCTK_MPI
+
+  /* Get the handle for PUGH extensions */
+  pughGH = PUGH_pGH (rec_info->it_info->GH);
+
+  /* Get the pGExtras pointer as a shortcut */
+  extras = ((pGA ***) pughGH->variables) [index][timelevel]->extras;
+
+  /* allocate memory for the biggest chunk */
+  npoints = extras->rnpoints [rec_info->it_info->ioproc + 1];
+  for (proc = 2; proc < rec_info->it_info->ioproc_every; proc++)
+    if (npoints < extras->rnpoints [rec_info->it_info->ioproc + proc])
+      npoints = extras->rnpoints [rec_info->it_info->ioproc + proc];
+  buffer = malloc (npoints * rec_info->element_size);
+
+  /* set the variable's index and timelevel to restore */
+  var_info [0] = index; var_info [1] = timelevel;
+
+  /* now loop over the group of processors associated to each IO processor */
+  for (proc = rec_info->it_info->ioproc;
+       proc < rec_info->it_info->ioproc + rec_info->it_info->ioproc_every &&
+       proc < CCTK_nProcs (rec_info->it_info->GH);
+       proc++) {
+
+    /* read own data directly into variable */
+    if (proc == rec_info->it_info->ioproc)
+      data = rec_info->it_info->GH->data [index][timelevel];
+    else
+      data = buffer;
+
+    if (! rec_info->it_info->unchunked) {
+      hid_t chunk;
+      char chunkname [32];
+
+      /* Chunked data is stored as separate chunk datasets within a group.
+         So open, read and close the separate chunks here. */
+      sprintf (chunkname, "chunk%d", proc - rec_info->it_info->ioproc);
+      CACTUS_IOHDF5_ERROR (chunk = H5Dopen (dataset, chunkname));
+      CACTUS_IOHDF5_ERROR (H5Dread (chunk, rec_info->iohdf5_type, H5S_ALL,
+                           H5S_ALL, H5P_DEFAULT, data));
+      CACTUS_IOHDF5_ERROR (H5Dclose (chunk));
+
+    } else {
+      int i, dim;
+      hssize_t *chunk_origin;
+      hsize_t *chunk_dims;
+
+      /* get the dimension of the variable */
+      dim = CCTK_GroupDimI (CCTK_GroupIndexFromVarI (index));
+      chunk_origin = (hssize_t *) malloc (dim * sizeof (hssize_t));
+      chunk_dims = (hsize_t *) malloc (dim * sizeof (hsize_t));
+
+      /* Unchunked data is read as one hyperslab per processor.
+         So prepare the memspace and the filespace and read the hyperslab. */
+      for (i = 0; i < dim; i++) {
+        chunk_dims [dim - 1 - i] = extras->rnsize [proc][i];
+        chunk_origin [dim - 1 - i] = extras->lb [proc][i];
+      }
+
+      CACTUS_IOHDF5_ERROR (filespace = H5Dget_space (dataset));
+      CACTUS_IOHDF5_ERROR (memspace = H5Screate_simple (dim, chunk_dims, NULL));
+      CACTUS_IOHDF5_ERROR (H5Sselect_hyperslab (filespace, H5S_SELECT_SET,
+                           chunk_origin, NULL, chunk_dims, NULL));
+
+      CACTUS_IOHDF5_ERROR (H5Dread (dataset, rec_info->iohdf5_type, memspace,
+                           filespace, H5P_DEFAULT, data));
+
+      CACTUS_IOHDF5_ERROR (H5Sclose (memspace));
+      CACTUS_IOHDF5_ERROR (H5Sclose (filespace));
+
+      free (chunk_dims);
+      free (chunk_origin);
+    }
+
+    /* send the index and the data to the non-IO processors */
+    if (proc != rec_info->it_info->ioproc) {
+      var_info [2] = extras->rnpoints [proc];
+      CACTUS_MPI_ERROR (MPI_Send (var_info, 3, PUGH_MPI_INT, proc,
+                        STARTUPBASE, pughGH->PUGH_COMM_WORLD));
+      CACTUS_MPI_ERROR (MPI_Send (data, extras->rnpoints [proc],
+               rec_info->mpi_type, proc, STARTUPBASE, pughGH->PUGH_COMM_WORLD));
+    }
+  }
+
+  free (buffer);
+#endif
+
+  return (0);
+}
+
+
+static herr_t processDataset (hid_t group, const char *datasetname, void *arg)
+{
+  DECLARE_CCTK_PARAMETERS
+  pGH *pughGH;
+  ioGH *ioUtilGH;
+  ioHDF5GH *myGH;
+  int index, gtype, timelevel;
+  iterate_info_t *it_info = (iterate_info_t *) arg;
+  recover_info_t rec_info;
+  hid_t dataset;
+  H5G_stat_t object_info;
+  int is_group;
+
+  /* skip the global attributes and GH extensions groups */
+  if (! strcmp (datasetname, GLOBAL_PARAMETERS_GROUP) ||
+      ! strcmp (datasetname, GHEXTENSIONS_GROUP))
+    return (0);
+
+  /* Get the handle for PUGH, IOUtil, and IOHDF5 extensions */
+  pughGH = PUGH_pGH (it_info->GH);
+  ioUtilGH = (ioGH *) it_info->GH->extensions [CCTK_GHExtensionHandle ("IO")];
+  myGH = (ioHDF5GH*) it_info->GH->extensions [CCTK_GHExtensionHandle("IOHDF5")];
+
+  CACTUS_IOHDF5_ERROR (H5Gget_objinfo (group, datasetname, 0, &object_info));
+  is_group = object_info.type == H5G_GROUP;
+  if (is_group)
+    CACTUS_IOHDF5_ERROR (dataset = H5Gopen (group, datasetname));
+  else
+    CACTUS_IOHDF5_ERROR (dataset = H5Dopen (group, datasetname));
+
+  /* read in the dataset's attributes and verify them */
+  if (GetCommonAttributes (it_info->GH, dataset, datasetname,
+      it_info->unchunked, &index, &gtype, &timelevel, is_group)) { 
+    CCTK_VWarn (1, __LINE__, __FILE__, CCTK_THORNSTRING,
+                "Ignoring dataset '%s'", datasetname);
+    return (0);
+  }
+
+  /* if we read in initial data via the file reader interface
+     check whether the user wants to have this variable restored */
+  if (ioUtilGH->do_inVars && ! ioUtilGH->do_inVars [index]) {
+    if (verbose) {
+      char *varname = CCTK_FullName (index);
+
+      CCTK_VInfo (CCTK_THORNSTRING, "Ignoring variable '%s' for file reader "
+                  "recovery", varname);
+      free (varname);
+    }
+    return (0);
+  }
+
+  if (verbose) {
+    char *varname = CCTK_FullName (index);
+
+    CCTK_VInfo (CCTK_THORNSTRING, "Restoring variable '%s' (timelevel %d)",
+                varname, timelevel);
+    free (varname);
+  }
+
+  rec_info.it_info = it_info;
+
+  switch (CCTK_VarTypeI (index)) {
+    case CCTK_VARIABLE_CHAR:
+      rec_info.iohdf5_type = IOHDF5_CHAR;
+      rec_info.element_size = sizeof (CCTK_CHAR);
+#ifdef CCTK_MPI
+      rec_info.mpi_type = PUGH_MPI_CHAR;
+#endif
+      break;
+    case CCTK_VARIABLE_INT:
+      rec_info.iohdf5_type = IOHDF5_INT;
+      rec_info.element_size = sizeof (CCTK_INT);
+#ifdef CCTK_MPI
+      rec_info.mpi_type = PUGH_MPI_INT;
+#endif
+      break;
+    case CCTK_VARIABLE_REAL:
+      rec_info.iohdf5_type = IOHDF5_REAL;
+      rec_info.element_size = sizeof (CCTK_REAL);
+#ifdef CCTK_MPI
+      rec_info.mpi_type = PUGH_MPI_REAL;
+#endif
+      break;
+    case CCTK_VARIABLE_COMPLEX:
+      rec_info.iohdf5_type = myGH->IOHDF5_COMPLEX;
+      rec_info.element_size = sizeof (CCTK_COMPLEX);
+#ifdef CCTK_MPI
+      rec_info.mpi_type = pughGH->PUGH_mpi_complex;
+#endif
+      break;
+    default:
+      CCTK_VWarn (1, __LINE__, __FILE__, CCTK_THORNSTRING,
+                  "Unsupported datatype %d", CCTK_VarTypeI (index));
+      return (0);
+  }
+
+  /* Read in the data */
+  switch (gtype) {
+    case GROUP_SCALAR:
+      IOHDF5_RestoreGS (dataset, index, timelevel, &rec_info);
+      break;
+    case GROUP_GF:
+    case GROUP_ARRAY:
+      IOHDF5_RestoreGA (dataset, index, timelevel, &rec_info);
+      break;
+    default:
+      CCTK_VWarn (1, __LINE__, __FILE__, CCTK_THORNSTRING,
+                  "Unsupported group type %d", gtype);
+      return (1);
+  }
+
+  if (is_group)
+    CACTUS_IOHDF5_ERROR (H5Gclose (dataset));
+  else
+    CACTUS_IOHDF5_ERROR (H5Dclose (dataset));
+
+  return (0);
+}
diff --git a/src/Startup.c b/src/Startup.c
new file mode 100644
index 0000000..2ec6c14
--- /dev/null
+++ b/src/Startup.c
@@ -0,0 +1,106 @@
+ /*@@
+   @file      Startup.c
+   @date      Fri May 21 1999
+   @author    Thomas Radke
+   @desc 
+   Startup routines for IOHDF5.
+   @enddesc 
+   @history
+   @hauthor Thomas Radke @hdate May 21 1999
+   @hdesc Just copied from thorn FlexIO.
+   @endhistory
+ @@*/
+
+#include <stdio.h>
+
+#include "cctk.h"
+#include "cctk_Parameters.h"
+#include "CactusBase/IOUtil/src/ioGH.h"
+
+
+/* prototypes of functions to be registered */
+int IOHDF5_Output3DGH(cGH *GH);
+int IOHDF5_TriggerOutput3D(cGH *GH, int);
+int IOHDF5_TimeFor3D(cGH *GH, int);
+int IOHDF5_Output3DVarAs(cGH *GH, const char *var, const char *alias);
+
+int IOHDF5_Output2DGH(cGH *GH);
+int IOHDF5_TriggerOutput2D(cGH *GH, int);
+int IOHDF5_TimeFor2D(cGH *GH, int);
+int IOHDF5_Output2DVarAs(cGH *GH, const char *var, const char *alias);
+
+int IOHDF5_Output1DGH(cGH *GH);
+int IOHDF5_TriggerOutput1D(cGH *GH, int);
+int IOHDF5_TimeFor1D(cGH *GH, int);
+int IOHDF5_Output1DVarAs(cGH *GH, const char *var, const char *alias);
+
+
+void *IOHDF5_SetupGH(tFleshConfig *config, int convergence_level, cGH *GH);
+int IOHDF5_InitGH(cGH *GH);
+int IOHDF5_Recover (cGH *GH, const char *basename, int called_from);
+
+
+ /*@@
+   @routine   IOHDF5_Startup
+   @date      Fri May 21 1999
+   @author    Thomas Radke
+   @desc 
+   The startup registration routine for IOHDF5.
+   Registers the GH extensions needed for IOHDF5 and
+   the registerable routines used for each method of IOHDF5.
+   IOHDF5 does not overload any functions.
+   @enddesc 
+   @calls     
+   @calledby   CCTK scheduler at STARTUP
+   @history 
+ 
+   @endhistory 
+@@*/
+void IOHDF5_Startup (void)
+{
+  DECLARE_CCTK_PARAMETERS
+  int IO_GHExtension;
+  int IOMethod;
+
+
+  if (CCTK_GHExtensionHandle ("IO") < 0)
+  {
+    CCTK_WARN (1, "Thorn IOUtil was not activated. "
+                  "No IOHDF5 IO methods will be enabled.");
+    return;
+  }
+  if (CCTK_GHExtensionHandle ("PUGH") < 0)
+  {
+    CCTK_WARN (1, "Thorn PUGH was not activated. "
+                  "No IOHDF5 IO methods will be enabled.");
+    return;
+  }
+
+  IO_GHExtension = CCTK_RegisterGHExtension ("IOHDF5");
+  CCTK_RegisterGHExtensionSetupGH (IO_GHExtension, IOHDF5_SetupGH);
+  CCTK_RegisterGHExtensionInitGH (IO_GHExtension, IOHDF5_InitGH);
+
+  /* Register the 2D and 3D IOHDF5 routines as output methods  */
+  IOMethod = CCTK_RegisterIOMethod ("IOHDF5_1D");
+  CCTK_RegisterIOMethodOutputGH (IOMethod, IOHDF5_Output1DGH);
+  CCTK_RegisterIOMethodOutputVarAs (IOMethod, IOHDF5_Output1DVarAs);
+  CCTK_RegisterIOMethodTimeToOutput (IOMethod, IOHDF5_TimeFor1D);
+  CCTK_RegisterIOMethodTriggerOutput (IOMethod, IOHDF5_TriggerOutput1D);
+
+  IOMethod = CCTK_RegisterIOMethod ("IOHDF5_2D");
+  CCTK_RegisterIOMethodOutputGH (IOMethod, IOHDF5_Output2DGH);
+  CCTK_RegisterIOMethodOutputVarAs (IOMethod, IOHDF5_Output2DVarAs);
+  CCTK_RegisterIOMethodTimeToOutput (IOMethod, IOHDF5_TimeFor2D);
+  CCTK_RegisterIOMethodTriggerOutput (IOMethod, IOHDF5_TriggerOutput2D);
+
+  IOMethod = CCTK_RegisterIOMethod ("IOHDF5_3D");
+  CCTK_RegisterIOMethodOutputGH (IOMethod, IOHDF5_Output3DGH);
+  CCTK_RegisterIOMethodOutputVarAs (IOMethod, IOHDF5_Output3DVarAs);
+  CCTK_RegisterIOMethodTimeToOutput (IOMethod, IOHDF5_TimeFor3D);
+  CCTK_RegisterIOMethodTriggerOutput (IOMethod, IOHDF5_TriggerOutput3D);
+
+  /* Register the IOHDF5 recovery routine to thorn IOUtil */
+  if (IOUtil_RegisterRecover ("IOHDF5 recovery", IOHDF5_Recover) < 0)
+    CCTK_WARN (1, "Failed to register IOHDF5 recovery routine");
+
+}
diff --git a/src/Write1D.c b/src/Write1D.c
new file mode 100644
index 0000000..a8b4b62
--- /dev/null
+++ b/src/Write1D.c
@@ -0,0 +1,179 @@
+/*@@
+   @file      Write1D.c
+   @date      Fri May 21 1999
+   @author    Thomas Radke, Gerd Lanfermann
+   @desc 
+   Output and trigger routines for 1D output into HDF5 files.
+   @enddesc 
+   @hendhistory
+ @@*/
+
+
+#include "cctk.h"
+#include "cctk_Parameters.h"
+#include "StoreNamedData.h"
+#include "CactusBase/IOUtil/src/ioGH.h"
+#include "ioHDF5GH.h"
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <assert.h>
+
+
+/* local function prototypes */
+void IOHDF5i_DumpParameters (cGH *GH, hid_t group);
+void IOHDF5i_DumpGHExtensions (cGH *GH, hid_t group);
+char *IOHDF5_Get_filename_ND (cGH *GH, int index,  ioHDF5Geo_t *slab_geo,
+			      const char *name, int *isNewFile);
+int NumDirection(int sdim, int vdim);
+int SetDirection(int vdim, int sdim, int ci, int *direction);
+
+/*@@
+  @routine Write1D
+  @author  Paul Walker 
+  @date    Feb 1997
+  @calledby IOHDF5_Output1DVarAs, IOHDF5_TriggerOutput1D
+  @var     GH
+  @vdesc   Pointer to CCTK GH
+  @vtype   cGH
+  @vio     in
+  @vcomment
+  @endvar
+  @var     index
+  @vdesc   index of variable to output
+  @vtype   int
+  @vio     in
+  @vcomment
+  @endvar
+  @var     alias
+  @vdesc   alias name of variable to output
+  @vtype   const char *
+  @vio     in
+  @vcomment
+  @endvar
+@@*/
+int IOHDF5_Write1D (cGH *GH, int index, const char *alias)
+{
+  DECLARE_CCTK_PARAMETERS
+  int timelevel;
+  ioGH *ioUtilGH; 
+  ioHDF5GH *h5GH;
+  ioHDF5Geo_t geo;
+
+  hid_t fid, plist;
+  int isNewFile;
+  char *filename;
+
+  int SDIM;
+  int si,max_slabs;
+
+  /* Get the handle for IO extensions */
+  ioUtilGH = (ioGH *) GH->extensions [CCTK_GHExtensionHandle ("IO")];
+  h5GH     = (ioHDF5GH *) GH->extensions [CCTK_GHExtensionHandle ("IOHDF5")];
+
+  /* The dimension of the slab - HARDCODED */
+  SDIM = 1;
+
+  /* Get the slab geometry for 1D slabs */
+  geo      = h5GH->out_geo[index][SDIM];
+  /* Set geo.sdim to announce 1D output */
+  geo.sdim = SDIM;
+  /* get the dimension of the variable */
+  geo.vdim = CCTK_GroupDimFromVarI(index);
+
+  if (verbose)
+    CCTK_VInfo (CCTK_THORNSTRING, "HDF5 %dD-output of variable '%s', "
+                "downsampling parameters (%d, %d %d)", SDIM, alias, 
+                geo.downs[0],geo.downs[1],geo.downs[2]);
+
+  /* get the current timelevel */
+  timelevel = CCTK_NumTimeLevelsFromVarI (index) - 1;
+  if (timelevel > 0)
+    timelevel--;
+
+  /* Maximal number of slabs (dimension sdim) in 
+     given volume (dimension vdim) */
+  if ((max_slabs=NumDirection(SDIM, geo.vdim))<0) 
+  {
+    CCTK_VWarn (1, __LINE__, __FILE__, CCTK_THORNSTRING,
+		"Cannot get valid number of slabs (%d) for given volume (%d).\n",
+	   SDIM, geo.vdim);
+    return(-1);
+  }
+  
+  /* Loop over all possible choices */
+  for (si=0;si<max_slabs;si++) {
+
+    /* Get the next set of slab directions */
+    if (SetDirection(geo.vdim, SDIM, si, geo.direction)<0) {
+         CCTK_VWarn (1, __LINE__, __FILE__, CCTK_THORNSTRING,
+	     "Cannot set direction for slab (#%d) in given volume (%d).\n",
+	     si, geo.vdim);
+	 continue;
+    }
+
+    /* get the filename for output */
+    filename = IOHDF5_Get_filename_ND (GH, index, &geo,
+				       alias, &isNewFile);
+
+    /* open the output file on IO procs no parallel HDF5  */
+    if (CCTK_MyProc (GH) != ioUtilGH->ioproc)
+      fid = -1;
+    else
+    {
+      
+      if (verbose)
+	CCTK_VInfo (CCTK_THORNSTRING, "%s HDF5 output file '%s'",
+		    isNewFile ? "Opening" : "Appending", filename);
+      
+      CACTUS_IOHDF5_ERROR (plist = H5Pcreate (H5P_FILE_ACCESS));
+      
+      if (isNewFile)
+	CACTUS_IOHDF5_ERROR (fid = H5Fcreate (filename, H5F_ACC_TRUNC,
+					      H5P_DEFAULT, plist));
+      else
+	CACTUS_IOHDF5_ERROR (fid = H5Fopen (filename, H5F_ACC_RDWR, plist));
+      
+      H5Pclose (plist);
+    }
+
+
+    /* output global attributes */
+    if (isNewFile && fid >= 0) {
+      hid_t group;
+      
+      /* all GH extension variables and parameter stuff which is not
+	 specific to any dataset goes into dedicated groups */
+      if (out1D_parameters)
+      {
+	CACTUS_IOHDF5_ERROR (group = H5Gcreate (fid, GLOBAL_PARAMETERS_GROUP, 0));
+	IOHDF5i_DumpParameters (GH, group);
+	CACTUS_IOHDF5_ERROR (H5Gclose (group));
+      }
+      
+      if (verbose)
+	CCTK_INFO ("Dumping GH extensions ...");
+      
+      CACTUS_IOHDF5_ERROR (group = H5Gcreate (fid, GHEXTENSIONS_GROUP, 0));
+      IOHDF5i_DumpGHExtensions (GH, group);
+      CACTUS_IOHDF5_ERROR (H5Gclose (group));
+    }
+    
+    IOHDF5_DumpVar(GH, index, timelevel, &geo, fid);
+    
+    /* close the file */
+    if (fid >= 0)
+    {
+      if (verbose)
+	CCTK_INFO ("Closing HDF5 output file from this iteration");
+      CACTUS_IOHDF5_ERROR (H5Fclose (fid));
+    }
+    
+    /* free the filename if it was not registered for re-opening
+       in IOHDF5_Get1D_filename () */
+    if (out1D_septimefiles)
+      free (filename);
+  }
+  return(1);
+}
+
diff --git a/src/Write2D.c b/src/Write2D.c
new file mode 100644
index 0000000..6b29d14
--- /dev/null
+++ b/src/Write2D.c
@@ -0,0 +1,180 @@
+/*@@
+   @file      Write2D.c
+   @date      Fri May 21 1999
+   @author    Thomas Radke, Gerd Lanfermann
+   @desc 
+   Output and trigger routines for 2D output into HDF5 files
+   @enddesc 
+   @hendhistory
+ @@*/
+
+
+#include "cctk.h"
+#include "cctk_Parameters.h"
+#include "StoreNamedData.h"
+#include "CactusBase/IOUtil/src/ioGH.h"
+#include "ioHDF5GH.h"
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <assert.h>
+
+
+/* local function prototypes */
+void IOHDF5i_DumpParameters (cGH *GH, hid_t group);
+void IOHDF5i_DumpGHExtensions (cGH *GH, hid_t group);
+char *IOHDF5_Get_filename_ND (cGH *GH, int index,  ioHDF5Geo_t *slab_geo,
+			      const char *name, int *isNewFile);
+int NumDirection(int sdim, int vdim);
+int SetDirection(int vdim, int sdim, int ci, int *direction);
+
+/*@@
+  @routine Write2D
+  @author  Paul Walker 
+  @date    Feb 1997
+  @calledby IOHDF5_Output2DVarAs, IOHDF5_TriggerOutput2D
+  @var     GH
+  @vdesc   Pointer to CCTK GH
+  @vtype   cGH
+  @vio     in
+  @vcomment
+  @endvar
+  @var     index
+  @vdesc   index of variable to output
+  @vtype   int
+  @vio     in
+  @vcomment
+  @endvar
+  @var     alias
+  @vdesc   alias name of variable to output
+  @vtype   const char *
+  @vio     in
+  @vcomment
+  @endvar
+@@*/
+int IOHDF5_Write2D (cGH *GH, int index, const char *alias)
+{
+  DECLARE_CCTK_PARAMETERS
+  int timelevel;
+  ioGH *ioUtilGH; 
+  ioHDF5GH *h5GH;
+  ioHDF5Geo_t geo;
+
+  hid_t fid, plist;
+  int isNewFile;
+  char *filename;
+
+  int SDIM;
+  int si,max_slabs;
+  int ierr;
+
+  /* Get the handle for IO extensions */
+  ioUtilGH = (ioGH *) GH->extensions [CCTK_GHExtensionHandle ("IO")];
+  h5GH     = (ioHDF5GH *) GH->extensions [CCTK_GHExtensionHandle ("IOHDF5")];
+
+  /* The dimension of the slab - HARDCODED */
+  SDIM = 2;
+
+  /* Get the slab geometry for 2D slabs */
+  geo      = h5GH->out_geo[index][SDIM];
+  /* Set geo.sdim to announce 2D output */
+  geo.sdim = SDIM;
+  /* get the dimension of the variable */
+  geo.vdim = CCTK_GroupDimFromVarI(index);
+
+  if (verbose)
+    CCTK_VInfo (CCTK_THORNSTRING, "HDF5 %dD-output of variable '%s', "
+                "downsampling parameters (%d, %d)", SDIM, alias, 
+                geo.downs[0], geo.downs [1]);
+
+  /* get the current timelevel */
+  timelevel = CCTK_NumTimeLevelsFromVarI (index) - 1;
+  if (timelevel > 0)
+    timelevel--;
+
+  /* Maximal number of slabs (dimension sdim) in 
+     given volume (dimension vdim) */
+  if ((max_slabs=NumDirection(SDIM, geo.vdim))<0) 
+  {
+    CCTK_VWarn (1, __LINE__, __FILE__, CCTK_THORNSTRING,
+		"Cannot get valid number of slabs (%d) for given volume (%d).\n",
+	   SDIM, geo.vdim);
+    return(-1);
+  }
+  
+  /* Loop over all possible choices */
+  for (si=0;si<max_slabs;si++) {
+
+    /* Get the next set of slab directions */
+    if (SetDirection(geo.vdim, SDIM, si, geo.direction)<0) {
+         CCTK_VWarn (1, __LINE__, __FILE__, CCTK_THORNSTRING,
+	     "Cannot set direction for slab (#%d) in given volume (%d).\n",
+	     si, geo.vdim);
+	 continue;
+    }
+
+    /* get the filename for output */
+    filename = IOHDF5_Get_filename_ND (GH, index, &geo,
+				       alias, &isNewFile); 
+
+    /* open the output file on IO procs no parallel HDF5  */
+    if (CCTK_MyProc (GH) != ioUtilGH->ioproc)
+      fid = -1;
+    else
+    {
+      
+      if (verbose)
+	CCTK_VInfo (CCTK_THORNSTRING, "%s HDF5 output file '%s'",
+		    isNewFile ? "Opening" : "Appending", filename);
+      
+      CACTUS_IOHDF5_ERROR (plist = H5Pcreate (H5P_FILE_ACCESS));
+      
+      if (isNewFile)
+	CACTUS_IOHDF5_ERROR (fid = H5Fcreate (filename, H5F_ACC_TRUNC,
+					      H5P_DEFAULT, plist));
+      else
+	CACTUS_IOHDF5_ERROR (fid = H5Fopen (filename, H5F_ACC_RDWR, plist));
+      
+      H5Pclose (plist);
+    }
+
+
+    /* output global attributes */
+    if (isNewFile && fid >= 0) {
+      hid_t group;
+      
+      /* all GH extension variables and parameter stuff which is not
+	 specific to any dataset goes into dedicated groups */
+      if (out2D_parameters)
+      {
+	CACTUS_IOHDF5_ERROR (group = H5Gcreate (fid, GLOBAL_PARAMETERS_GROUP, 0));
+	IOHDF5i_DumpParameters (GH, group);
+	CACTUS_IOHDF5_ERROR (H5Gclose (group));
+      }
+      
+      if (verbose)
+	CCTK_INFO ("Dumping GH extensions ...");
+      
+      CACTUS_IOHDF5_ERROR (group = H5Gcreate (fid, GHEXTENSIONS_GROUP, 0));
+      IOHDF5i_DumpGHExtensions (GH, group);
+      CACTUS_IOHDF5_ERROR (H5Gclose (group));
+    }
+    
+     ierr = IOHDF5_DumpVar(GH, index, timelevel, &geo, fid);
+    
+    /* close the file */
+    if (fid >= 0)
+    {
+      if (verbose)
+	CCTK_INFO ("Closing HDF5 output file from this iteration");
+      CACTUS_IOHDF5_ERROR (H5Fclose (fid));
+    }
+    
+    /* free the filename if it was not registered for re-opening
+       in IOHDF5_Get2D_filename () */
+    if (out2D_septimefiles)
+      free (filename);
+  }
+  return(ierr);
+}
+
diff --git a/src/Write3D.c b/src/Write3D.c
new file mode 100644
index 0000000..a4019f5
--- /dev/null
+++ b/src/Write3D.c
@@ -0,0 +1,195 @@
+/*@@
+   @file      Write3D.c
+   @date      Fri May 21 1999
+   @author    Thomas Radke, Gerd Lanfermann
+   @desc 
+   Output and trigger routines for 3D output into HDF5 files.
+   Added general hyperslab extraction
+   @enddesc 
+   @history
+   @hauthor Thomas Radke @hdate May 21 1999
+   @hdesc Just copied from thorn FlexIO.
+   @hendhistory
+ @@*/
+
+
+#include "cctk.h"
+#include "cctk_Parameters.h"
+#include "StoreNamedData.h"
+#include "CactusPUGH/PUGH/src/include/pugh.h"
+#include "CactusBase/IOUtil/src/ioGH.h"
+#include "ioHDF5GH.h"
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <assert.h>
+
+
+/* local function prototypes */
+void IOHDF5i_DumpParameters (cGH *GH, hid_t group);
+void IOHDF5i_DumpGHExtensions (cGH *GH, hid_t group);
+char *IOHDF5_Get_filename_ND (cGH *GH, int index,  ioHDF5Geo_t *slab_geo,
+                              const char *name, int *isNewFile);
+
+
+/*@@
+  @routine Write3D
+  @author  Paul Walker 
+  @date    Feb 1997
+  @calledby IOHDF5_Output3DVarAs, IOHDF5_TriggerOutput3D
+  @var     GH
+  @vdesc   Pointer to CCTK GH
+  @vtype   cGH
+  @vio     in
+  @vcomment
+  @endvar
+  @var     index
+  @vdesc   index of variable to output
+  @vtype   int
+  @vio     in
+  @vcomment
+  @endvar
+  @var     alias
+  @vdesc   alias name of variable to output
+  @vtype   const char *
+  @vio     in
+  @vcomment
+  @endvar
+@@*/
+int IOHDF5_Write3D (cGH *GH, int index, const char *alias)
+{
+  DECLARE_CCTK_PARAMETERS
+  int timelevel;
+  ioGH *ioUtilGH;
+  ioHDF5GH *h5GH;
+  hid_t fid, plist;
+  int isNewFile;
+  char *filename;
+  int ierr=0;
+
+  ioHDF5Geo_t geo;
+  int SDIM;
+
+
+  /* Get the handle for IO extensions */
+  ioUtilGH = (ioGH *) GH->extensions [CCTK_GHExtensionHandle ("IO")];
+  h5GH     = (ioHDF5GH *) GH->extensions [CCTK_GHExtensionHandle ("IOHDF5")];
+
+  /* The dimension of the slab - HARDCODED */
+  SDIM = 3;
+
+  /* Get the slab geometry for 2D slabs */
+  geo      = h5GH->out_geo[index][SDIM];
+
+  /* Set geo.sdim to announce 3d output */
+  geo.sdim = SDIM;
+
+  /* get the dimension of the variable */
+  geo.vdim = CCTK_GroupDimFromVarI(index);
+
+  /* check if variable has storage assigned */
+  if (! CCTK_QueryGroupStorageI (GH, CCTK_GroupIndexFromVarI (index)))
+  {
+    char *fullname = CCTK_FullName (index);
+    CCTK_VWarn (1, __LINE__, __FILE__, CCTK_THORNSTRING,
+                "No IOHDF5 3D output for '%s' (no storage)", fullname);
+    free (fullname);
+    return(-1);
+  }
+  if (verbose)
+    CCTK_VInfo (CCTK_THORNSTRING, "HDF5 %dD-output of variable '%s', "
+                "downsampling parameters (%d, %d, %d)", SDIM, alias, 
+                geo.downs[0], geo.downs [1], geo.downs[2]);
+
+  /* get the filename for output */
+  if (!(filename = IOHDF5_Get_filename_ND (GH, index, &geo,
+					   alias, &isNewFile))) {
+    char *fullname = CCTK_FullName (index);
+    CCTK_VWarn (1, __LINE__, __FILE__, CCTK_THORNSTRING,
+		"Unable to construct file name for '%s'", fullname);
+    free(fullname);
+    return(-1);
+  }
+
+
+  /* open the output file on IO procs
+     or - in case of parallel HDF5 - on all procs */
+#ifdef HAVE_PARALLEL
+  if (CCTK_MyProc (GH) != ioUtilGH->ioproc && ! ioUtilGH->unchunked)
+#else
+  if (CCTK_MyProc (GH) != ioUtilGH->ioproc)
+#endif
+  {
+    fid = -1;
+  }
+  else
+  {
+
+    if (verbose)
+      CCTK_VInfo (CCTK_THORNSTRING, "%s HDF5 output file '%s'",
+                  isNewFile ? "Opening" : "Appending", filename);
+
+    CACTUS_IOHDF5_ERROR (plist = H5Pcreate (H5P_FILE_ACCESS));
+#ifdef CCTK_MPI
+#ifdef HAVE_PARALLEL
+    if (ioUtilGH->unchunked)
+      CACTUS_IOHDF5_ERROR (H5Pset_mpi (plist, PUGH_pGH (GH)->PUGH_COMM_WORLD,
+                           MPI_INFO_NULL));
+#endif
+#endif
+
+    if (isNewFile)
+      CACTUS_IOHDF5_ERROR (fid = H5Fcreate (filename, H5F_ACC_TRUNC,
+                                               H5P_DEFAULT, plist));
+    else
+      CACTUS_IOHDF5_ERROR (fid = H5Fopen (filename, H5F_ACC_RDWR, plist));
+
+    H5Pclose (plist);
+  }
+
+  /* get the current timelevel */
+  timelevel = CCTK_NumTimeLevelsFromVarI (index) - 1;
+  if (timelevel > 0)
+    timelevel--;
+
+  /* output global attributes */
+  if (isNewFile && fid >= 0)
+  {
+    hid_t group;
+
+    /* all GH extension variables and parameter stuff which is not
+       specific to any dataset goes into dedicated groups */
+    if (out3D_parameters)
+    {
+      CACTUS_IOHDF5_ERROR (group = H5Gcreate (fid, GLOBAL_PARAMETERS_GROUP, 0));
+      IOHDF5i_DumpParameters (GH, group);
+      CACTUS_IOHDF5_ERROR (H5Gclose (group));
+    }
+
+    if (verbose)
+      CCTK_INFO ("Dumping GH extensions ...");
+
+    CACTUS_IOHDF5_ERROR (group = H5Gcreate (fid, GHEXTENSIONS_GROUP, 0));
+    IOHDF5i_DumpGHExtensions (GH, group);
+    CACTUS_IOHDF5_ERROR (H5Gclose (group));
+  }
+
+  /* output the data */
+  ierr = IOHDF5_DumpVar (GH, index, timelevel, &geo, fid);
+
+  /* close the file */
+  if (fid >= 0)
+  {
+    if (verbose)
+      CCTK_INFO ("Closing HDF5 output file from this iteration");
+    CACTUS_IOHDF5_ERROR (H5Fclose (fid));
+  }
+
+  /* free the filename if it was not registered for re-opening
+     in IOHDF5_net3D_filename () */
+  if (out3D_septimefiles)
+    free (filename);
+
+  return(ierr);
+}
+
diff --git a/src/WriteIsosurface.c b/src/WriteIsosurface.c
new file mode 100644
index 0000000..19c3f51
--- /dev/null
+++ b/src/WriteIsosurface.c
@@ -0,0 +1,209 @@
+/*@@
+   @file      WriteIsosurface.c
+   @date      Sat Mar 25 2000
+   @author    Thomas Radke
+   @desc 
+   HDF5 output routine for isosurfaces
+   @enddesc 
+   @history
+   @hendhistory
+ @@*/
+
+
+#include <stdlib.h>
+#include <string.h>
+
+#include "cctk.h"
+#include "ioHDF5GH.h"
+
+#define SOCKET_URL_PREFIX      "sock://"
+#define SOCKET_URL_PREFIX_LEN  (sizeof (SOCKET_URL_PREFIX) - 1)
+
+/* this struct stores the current slice info for further queries */
+typedef struct {
+  CCTK_INT iteration;
+  CCTK_INT timelevel;
+} sliceInfo_t;
+
+
+/*@@
+  @routine  IOHDF5_WriteIsosurface
+  @author   Thomas Radke
+  @date     Sat Mar 25 2000
+  @calledby thorn CactusPUGHIO/IsoSurfacer
+  @var     
+  @vdesc  
+  @vtype 
+  @vio  
+  @vcomment
+  @endvar
+@@*/
+int IOHDF5_WriteIsosurface (cGH *GH,
+                            const char *filename,
+                            const char *GVname,
+                            CCTK_INT iteration,
+                            CCTK_INT timelevel,
+                            CCTK_REAL isoval,
+                            CCTK_REAL minval,
+                            CCTK_REAL maxval,
+                            int nTriangles,
+                            CCTK_INT4 *triangles,
+                            int nVertices,
+                            CCTK_REAL4 *vertices)
+{
+  hid_t h5file, fapl;
+  hid_t aspace, dataspace, dataset;
+  hid_t group0, group1, group2, group3, group4;
+  hsize_t dim [2];
+  char sliceName [40], *isoVarname;
+  int isSocketURL;
+  sliceInfo_t *sliceInfo;
+  hobj_ref_t reference;
+  static pNamedData *isoFilesList = NULL;
+  static int first_called = 1;
+  static hid_t hdf5_string;
+
+  /* FIXME: should use the predefined type from iohdf5GH
+            but we cannot be sure that thorn IOHDF5 was activated */
+  if (first_called) {
+    first_called = 0;
+    /* predefine a C string datatype */
+    CACTUS_IOHDF5_ERROR (hdf5_string = H5Tcopy (H5T_C_S1));
+  }
+
+  CACTUS_IOHDF5_ERROR (fapl = H5Pcreate (H5P_FILE_ACCESS));
+
+  isSocketURL = strncasecmp (filename, SOCKET_URL_PREFIX, SOCKET_URL_PREFIX_LEN) == 0;
+  if (isSocketURL) {
+#ifndef H5FDstream_H
+    CCTK_WARN (1, "The configured HDF5 installation doesn't include the Stream "
+                  "VFD. No HDF5 streaming output available !");
+    CACTUS_IOHDF5_ERROR (H5Pclose (fapl));
+    return (-1);
+#else
+    CACTUS_IOHDF5_ERROR (H5Pset_fapl_stream (fapl, 0, -1));
+    sliceInfo = NULL;
+#endif
+  } else
+    sliceInfo = (sliceInfo_t *) GetNamedData (isoFilesList, filename);
+
+  /* create/open the file */
+  if (! sliceInfo)
+    CACTUS_IOHDF5_ERROR (h5file = H5Fcreate (filename, H5F_ACC_TRUNC,
+                                             H5P_DEFAULT, fapl));
+  else
+    CACTUS_IOHDF5_ERROR (h5file = H5Fopen (filename, H5F_ACC_RDWR, fapl));
+  if (h5file < 0) {
+    CCTK_VWarn (1, __LINE__, __FILE__, CCTK_THORNSTRING,
+                "Could not create/open output file '%s'", filename);
+    return (-1);
+  }
+
+  /* create a dataspace for storing scalar attributes */
+  CACTUS_IOHDF5_ERROR (aspace = H5Screate (H5S_SCALAR));
+
+  /* open the highest-level hierarchy which is group "Bundle" */
+  if (! sliceInfo)
+    CACTUS_IOHDF5_ERROR (group0 = H5Gcreate (h5file, "Bundle", 0));
+  else
+    CACTUS_IOHDF5_ERROR (group0 = H5Gopen (h5file, "Bundle"));
+
+  /* next level in hierarchy is the "Slice [<iteration>] [<timelevel>]" group */
+  sprintf (sliceName, "Slice [%d] [%d]", (int) iteration, (int) timelevel);
+  if (! sliceInfo ||
+      sliceInfo->iteration != iteration || sliceInfo->timelevel != timelevel) {
+    if (sliceInfo) {
+      sliceInfo->iteration = iteration;
+      sliceInfo->timelevel = timelevel;
+    }
+    CACTUS_IOHDF5_ERROR (group1 = H5Gcreate (group0, sliceName, 0));
+    WRITE_ATTRIBUTE ("time", &GH->cctk_time, group1, aspace, 0,IOHDF5_REAL);
+    WRITE_ATTRIBUTE ("timestep", &iteration, group1, aspace, 0, IOHDF5_INT);
+    WRITE_ATTRIBUTE ("timelevel", &timelevel, group1, aspace, 0,IOHDF5_INT);
+  } else
+    CACTUS_IOHDF5_ERROR (group1 = H5Gopen (group0, sliceName));
+
+  /* third level is a group for each grid variable with specific isovalue */
+  isoVarname = (char *) malloc (strlen (GVname) + 40);
+  sprintf (isoVarname, "Isolevel [%s] [%.10f]", GVname, (double) isoval);
+  CACTUS_IOHDF5_ERROR (group2 = H5Gcreate (group1, isoVarname, 0));
+  free (isoVarname);
+
+  /* fourth level is group "PointTopology" */
+  CACTUS_IOHDF5_ERROR (group3 = H5Gcreate (group2, "PointTopology", 0));
+
+  /* fifth level is group "Representation" */
+  CACTUS_IOHDF5_ERROR (group4 = H5Gcreate (group3, "Representation [CartesianChart]", 0));
+
+#if 0
+  if (IsoL.Ncolorinfo > 0) {
+    dim [0] = (hsize_t) IsoL.Ncolorinfo;
+    CACTUS_IOHDF5_ERROR (dataspace = H5Screate_simple (1, dim, NULL));
+    CACTUS_IOHDF5_ERROR (dataset = H5Dcreate (h5file, "ColorMap",
+                         IOHDF5_REAL4, dataspace, H5P_DEFAULT));
+    CACTUS_IOHDF5_ERROR (H5Dwrite (dataset, IOHDF5_REAL4, H5S_ALL,
+                         H5S_ALL, H5P_DEFAULT, &IsoL.colorinfo));
+    CACTUS_IOHDF5_ERROR (H5Dclose (dataset));
+    CACTUS_IOHDF5_ERROR (H5Sclose (dataspace));
+  }
+#endif
+
+  /* write vertices */
+  dim [0] = (hsize_t) nVertices;
+  dim [1] = 3;
+  CACTUS_IOHDF5_ERROR (dataspace = H5Screate_simple (2, dim, NULL));
+  CACTUS_IOHDF5_ERROR (dataset = H5Dcreate (group4, "Positions",
+                       IOHDF5_REAL4, dataspace, H5P_DEFAULT));
+  CACTUS_IOHDF5_ERROR (H5Dwrite (dataset, IOHDF5_REAL4, H5S_ALL,
+                       H5S_ALL, H5P_DEFAULT, vertices));
+  WRITE_ATTRIBUTE ("data_min", &minval, dataset, aspace, 0, IOHDF5_REAL);
+  WRITE_ATTRIBUTE ("data_max", &maxval, dataset, aspace, 0, IOHDF5_REAL);
+#if 0
+  WRITE_ATTRIBUTE ("instance", &isoindex, isogroup, aspace, 0, IOHDF5_INT);
+#endif
+  CACTUS_IOHDF5_ERROR (H5Rcreate (&reference, group4, "Positions", H5R_OBJECT, -1));
+  CACTUS_IOHDF5_ERROR (H5Dclose (dataset));
+  CACTUS_IOHDF5_ERROR (H5Sclose (dataspace));
+  WRITE_ATTRIBUTE ("Domain", "CartesianChart", group4, aspace, 0, hdf5_string);
+  CACTUS_IOHDF5_ERROR (H5Gclose (group4));
+  CACTUS_IOHDF5_ERROR (H5Gclose (group3));
+
+  /* fourth level is group "Connectivity" */
+  CACTUS_IOHDF5_ERROR (group3 = H5Gcreate (group2, "Connectivity", 0));
+
+  /* fifth level is group "Representation" */
+  CACTUS_IOHDF5_ERROR (group4 = H5Gcreate (group3, "Representation [PointTopology]", 0));
+
+  /* write polygons */
+  dim [0] = (hsize_t) nTriangles;
+  dim [1] = 3;
+  CACTUS_IOHDF5_ERROR (dataspace = H5Screate_simple (2, dim, NULL));
+  CACTUS_IOHDF5_ERROR (dataset = H5Dcreate (group4, "Positions",
+                       IOHDF5_INT4, dataspace, H5P_DEFAULT));
+  CACTUS_IOHDF5_ERROR (H5Dwrite (dataset, IOHDF5_INT4, H5S_ALL,
+                       H5S_ALL, H5P_DEFAULT, triangles));
+  CACTUS_IOHDF5_ERROR (H5Dclose (dataset));
+  CACTUS_IOHDF5_ERROR (H5Sclose (dataspace));
+  WRITE_ATTRIBUTE ("Domain", &reference, group4, aspace, 0, H5T_STD_REF_OBJ);
+  CACTUS_IOHDF5_ERROR (H5Glink(group4, H5G_LINK_SOFT,
+     "../../PointTopology/Representation [CartesianChart]/Positions", "Domain"));
+  CACTUS_IOHDF5_ERROR (H5Sclose (aspace));
+  CACTUS_IOHDF5_ERROR (H5Gclose (group4));
+  CACTUS_IOHDF5_ERROR (H5Gclose (group3));
+  CACTUS_IOHDF5_ERROR (H5Gclose (group2));
+  CACTUS_IOHDF5_ERROR (H5Gclose (group1));
+  CACTUS_IOHDF5_ERROR (H5Gclose (group0));
+
+  CACTUS_IOHDF5_ERROR (H5Fclose (h5file));
+
+  /* if this was a newly created file add it to the list of open files */
+  if (! isSocketURL && ! sliceInfo) {
+    sliceInfo_t *newInfo = (sliceInfo_t *) malloc (sizeof (sliceInfo_t));
+
+    newInfo->iteration = iteration;
+    newInfo->timelevel = timelevel;
+    StoreNamedData (&isoFilesList, filename, newInfo);
+  }
+
+  return (0);
+}
diff --git a/src/ioHDF5GH.h b/src/ioHDF5GH.h
new file mode 100644
index 0000000..c7134c0
--- /dev/null
+++ b/src/ioHDF5GH.h
@@ -0,0 +1,212 @@
+ /*@@
+   @header    ioHDF5GH.h
+   @date      Fri May 21 1999
+   @author    Thomas Radke
+   @desc 
+   The extensions to the GH structure from IOHDF5.
+   @history
+   @hauthor Thomas Radke @hdate May 21 1999
+   @hdesc Just copied from thorn FlexIO.
+   @endhistory
+ @@*/
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#include <string.h>
+#include <hdf5.h>
+
+#include "StoreNamedData.h"
+
+
+/* names of the (empty) groups that hold GH extensions or global parameters */
+#define GHEXTENSIONS_GROUP      "GHextensions_group"
+#define GLOBAL_PARAMETERS_GROUP "global_parameters_group"
+#define GLOBAL_PARAMETERS       "global_parameters"
+
+/*****************************************************************************/
+/* some useful macros                                                        */
+/*****************************************************************************/
+/* Check error flags from HDF5 calls */
+#define CACTUS_IOHDF5_ERROR(fn_call)                                          \
+    do {                                                                      \
+      /*size_t strlen(const char *str);*/                                     \
+                                                                              \
+      int error_code = fn_call;                                               \
+                                                                              \
+      if (error_code < 0)                                                     \
+        CCTK_VWarn (1, __LINE__, __FILE__, CCTK_THORNSTRING,                  \
+                    "HDF5 call '%s' returned error code %d\n",                \
+                    #fn_call, error_code);                                    \
+    } while (0)
+
+/* macro for writing an attribute */
+#define WRITE_ATTRIBUTE(name, value, dataset, dataspace, dim, datatype)       \
+          do {                                                                \
+            hid_t attr;                                                       \
+            void *val = value;                                                \
+            hsize_t arrayDim = dim;                                           \
+                                                                              \
+            if (H5Tget_class (datatype) == H5T_STRING) {                      \
+              int len = strlen ((char *) val);                                \
+                                                                              \
+              if (len == 0)      /* HDF5 doesn't like zero-len strings */     \
+                len++;                                                        \
+              CACTUS_IOHDF5_ERROR (H5Tset_size (datatype, len));              \
+            }                                                                 \
+            if (arrayDim > 0)                                                 \
+              CACTUS_IOHDF5_ERROR (H5Sset_extent_simple (dataspace, 1,        \
+                                   &arrayDim, NULL));                         \
+            CACTUS_IOHDF5_ERROR (attr = H5Acreate (dataset, name, datatype,   \
+                                 dataspace, H5P_DEFAULT));                    \
+            CACTUS_IOHDF5_ERROR (H5Awrite (attr, datatype, val));             \
+            CACTUS_IOHDF5_ERROR (H5Aclose (attr));                            \
+          } while (0);
+
+/* macro for reading an attribute */
+#define READ_ATTRIBUTE(dataset, attrname, requested_type, buffer)             \
+          { hid_t attr, attrtype;                                             \
+            hsize_t asize = 0;                                                \
+                                                                              \
+            if ((attr = H5Aopen_name (dataset, attrname)) < 0)                \
+              CCTK_WARN (1, "Can't find " attrname " attribute");             \
+            if (requested_type == H5T_C_S1) {                                 \
+              CACTUS_IOHDF5_ERROR (attrtype = H5Aget_type (attr));            \
+              CACTUS_IOHDF5_ERROR (asize = H5Tget_size (attrtype));           \
+              if (asize + 1 >= sizeof (buffer))                               \
+                CCTK_WARN (1, "Can't read " attrname " attribute (too long)");\
+            } else                                                            \
+              attrtype = requested_type;                                      \
+            if (H5Aread (attr, attrtype, buffer) < 0)                         \
+              CCTK_WARN (1, "Can't read " attrname " attribute");             \
+            if (requested_type == H5T_C_S1) {                                 \
+              ((char *) buffer) [asize] = 0;                                  \
+              CACTUS_IOHDF5_ERROR (H5Tclose (attrtype));                      \
+            }                                                                 \
+            CACTUS_IOHDF5_ERROR (H5Aclose (attr));                            \
+          }
+
+
+/* define the HDF5 datatypes according to CCTK_??? datatypes */
+#define IOHDF5_REAL4   H5T_NATIVE_FLOAT
+
+#ifdef  CCTK_REAL_PRECISION_16
+#define IOHDF5_REAL    H5T_NATIVE_LDOUBLE
+#elif   CCTK_REAL_PRECISION_8
+#define IOHDF5_REAL    H5T_NATIVE_DOUBLE
+#elif   CCTK_REAL_PRECISION_4
+#define IOHDF5_REAL    H5T_NATIVE_FLOAT
+#endif
+
+#define IOHDF5_INT     (sizeof (CCTK_INT) == sizeof (int) ?                   \
+                        H5T_NATIVE_INT : H5T_NATIVE_SHORT)
+#define IOHDF5_INT4    (sizeof (int) == 4 ?                                   \
+                        H5T_NATIVE_INT : H5T_NATIVE_SHORT)
+#define IOHDF5_CHAR    H5T_NATIVE_CHAR
+
+#define IOHDF5_MAXDIM 4
+
+/* structure holding necessary information about a recovery file */
+typedef struct {
+  int is_HDF5_file;         /* flag indicating valid file info */
+  hid_t fid;                /* HDF5 file handle */
+  char filename [1024];     /* complete file name for recovery */
+  int nprocs;               /* number of total processors */
+  int ioproc;               /* the associated IO processor */
+  int ioproc_every;         /* how many IO processors there are */
+  int unchunked;            /* whether data was written chunked or unchunked */
+} fileinfo_t;
+
+/* structure holding information on the output geometry */
+typedef struct ioHDF5Geo
+{
+  int vdim;                        /* dimension of the gridvariable (dynamic) */
+  int sdim;                        /* dimension of the slab (dynamic) */
+  int direction[IOHDF5_MAXDIM];    /* direction of slab (by parameter) */
+  int slab_start[IOHDF5_MAXDIM];   /* global start index  (by parameter) */
+  int length[IOHDF5_MAXDIM];       /* requested index slab length (by parameter) */
+  int actlen[IOHDF5_MAXDIM];       /* actual index slab length (by PUGHSlab)*/
+  int downs[IOHDF5_MAXDIM];        /* downsampling (by parameter) */
+} ioHDF5Geo_t;
+
+/* IOHDF5 GH extension structure */
+typedef struct IOHDF5GH {
+
+  /* how often to output */
+  int  out1D_every;
+  int  out2D_every;
+  int  out3D_every;
+
+  /* flags indicating output for var [i] */
+  char   *do_out1D;
+  char   *do_out2D;
+  char   *do_out3D;
+
+  /* directories in which to output */
+  char  *outdir1D;
+  char  *outdir2D;
+  char  *outdir3D;
+
+  /* the last iteration output */
+  int   *out1D_last;
+  int   *out2D_last;
+  int   *out3D_last;
+
+  /* filename database for opened files */
+  pNamedData *filenameList;
+
+  /* for restart from recovery: flags indicating whether to check for already
+     existing groups/datasets to delete before creating them anew */
+  char  *checkForExistingObjects;
+
+  /* variables holding the original error printing routine and its argument */
+  H5E_auto_t printErrorFn;
+  void      *printErrorFnArg;
+
+  /* predefined dataspaces for writing scalar and array attributes */
+  hid_t scalarDataspace, arrayDataspace;
+
+  /* predefined datatype for writing CCTK_COMPLEX types */
+  hid_t IOHDF5_COMPLEX;
+
+  /* predefined datatype for writing C string string attributes */
+  hid_t IOHDF5_STRING;
+
+  /* timer array for checkpointing/recovery */
+  int timers[4];
+
+  /* flag to indicate request for timer output */
+  int print_timing_info;
+
+  /* geometry information (downsampling, zooming, etc.) */
+  ioHDF5Geo_t **out_geo; 
+
+} ioHDF5GH;
+
+/* function prototypes */
+int IOHDF5_Write3D (cGH *GH, int index, const char *alias);
+int IOHDF5_Write2D (cGH *GH, int index, const char *alias);
+int IOHDF5_Write1D (cGH *GH, int index, const char *alias);
+
+void IOHDF5_DumpParams (cGH *GH, hid_t group);
+void IOHDF5_DumpGHExtensions (cGH *GH, hid_t group);
+int  IOHDF5_DumpVar (cGH *GH, int index, int timelevel, ioHDF5Geo_t *geo, hid_t iof);
+void IOHDF5_DumpGH (cGH *GH, int called_from);
+
+int IOHDF5_WriteIsosurface (cGH *GH,
+                            const char *filename,
+                            const char *GVname,
+                            CCTK_INT iteration,
+                            CCTK_INT timelevel,
+                            CCTK_REAL isoval,
+                            CCTK_REAL minval,
+                            CCTK_REAL maxval,
+                            int nTriangles,
+                            CCTK_INT4 *triangles,
+                            int nVertices,
+                            CCTK_REAL4 *vertices);
+
+#ifdef __cplusplus
+} // extern "C" {
+#endif
diff --git a/src/make.code.defn b/src/make.code.defn
new file mode 100644
index 0000000..42416c2
--- /dev/null
+++ b/src/make.code.defn
@@ -0,0 +1,10 @@
+# Main make.code.defn file for thorn HFD5IO
+# $Header$
+
+# Source files in this directory
+SRCS = Startup.c GHExtension.c Output3D.c Output2D.c Output1D.c Write3D.c \
+Write2D.c Write1D.c DumpVar.c DumpGH.c RestoreFile.c RecoverGH.c \
+WriteIsosurface.c File.c ParseGeometry.c
+
+# Subdirectories containing source files
+SUBDIRS = 
diff --git a/src/make.configuration.defn b/src/make.configuration.defn
new file mode 100644
index 0000000..fe26de1
--- /dev/null
+++ b/src/make.configuration.defn
@@ -0,0 +1,20 @@
+# make.configuration.defn for IOHDF5
+
+# make sure that IOHDF5 was configured with HDF5 and PUGH
+
+ifeq ($(strip $(HDF5_LIBS)), )
+$(NAME): MissingHDF5
+.pseudo: MissingHDF5
+MissingHDF5:
+	@echo "IOHDF5: requires HDF5"
+	@echo "IOHDF5: Please configure with HDF5 or remove IOHDF5 from Thornlist !"
+	exit 2
+endif
+
+ifeq ($(findstring CactusPUGH/PUGH,$(THORNS)),)
+.pseudo: MissingPUGHinIOHDF5
+MissingPUGHinIOHDF5:
+	@echo "IOHDF5: requires PUGH"
+	@echo "IOHDF5: Please add CactusPUGH/PUGH or remove IOHDF5 from Thornlist !"
+	exit 2
+endif
diff --git a/test/WavetoyID.chkpt.it_0.h5 b/test/WavetoyID.chkpt.it_0.h5
new file mode 100644
index 0000000..fd2f802
--- /dev/null
+++ b/test/WavetoyID.chkpt.it_0.h5
diff --git a/test/checkpoint.parfile b/test/checkpoint.parfile
new file mode 100644
index 0000000..d337cdd
--- /dev/null
+++ b/test/checkpoint.parfile
@@ -0,0 +1,46 @@
+!DESC "Checkpoint Wavetoy ID in HDF5 file format"
+
+ActiveThorns = "Time WavetoyF77 IDScalarWave PUGH CartGrid3D IOUtil IOBasic IOASCII IOHDF5"
+
+# Output directory must be the basename of the recovery parameter file's
+# filename
+IO::outdir = "test_recover"
+
+# Checkpoint initial data into a single unchunked checkpoint file
+# with basename 'WavetoyID' and place it into the current directory
+IO::checkpoint_ID      = "yes"
+IO::out3D_unchunked    = "yes"
+IO::out3D_mode         = "onefile"
+IO::checkpoint_ID_file = "WavetoyID.chkpt"
+IO::checkpoint_dir     = "."
+
+# Say that the checkpoint file should be created by the IOHDF5 method
+IOHDF5::checkpoint  = "yes"
+
+############################################################################
+# Down from here the original parameter settings from the
+# CactusWave/WaveToyF77/test/test_wavetoyf77.par test suite follow
+
+grid::type  = "box"
+
+time::dtfac = 0.5
+
+idscalarwave::initial_data = "gaussian"
+idscalarwave::sigma        = 0.1
+idscalarwave::radius       = 0
+
+driver::global_nx = 20
+driver::global_ny = 20
+driver::global_nz = 20
+
+cactus::cctk_itlast       = 10
+
+IOASCII::out1D_every = 2
+IOASCII::out1D_vars = "wavetoy::phi "
+IOASCII::out1D_z = "no"
+
+IOBasic::outScalar_every = 2
+IOBasic::outScalar_vars = "wavetoy::phi "
+
+IOBasic::outInfo_every = 2
+IOBasic::outInfo_vars = "wavetoy::phi "
diff --git a/test/test_recover.par b/test/test_recover.par
new file mode 100644
index 0000000..3a6958e
--- /dev/null
+++ b/test/test_recover.par
@@ -0,0 +1,17 @@
+!DESC "Recovery from WaveToy ID checkpoint file in HDF5 file format"
+
+ActiveThorns = "Time WavetoyF77 IDScalarWave PUGH CartGrid3D IOUtil IOBasic IOASCII IOHDF5"
+
+# Say that we want to recover ...
+# You can also try "auto" and omit the ".it_0" field in the recover_file.
+IO::recover       = "manual"
+
+# ... from a ID checkpoint file with basename ...
+IO::recover_file  = "WavetoyID.chkpt.it_0"
+
+# ... to be found in directory ...
+IO::recovery_dir  = "../../arrangements/CactusPUGHIO/IOHDF5/test/"
+
+# Note that this parameter is restored to "yes" during parameter recovery.
+# Since we don't need to checkpoint twice it is explicitely disabled here.
+IO::checkpoint_ID = "no"
diff --git a/test/test_recover/phi.dl b/test/test_recover/phi.dl
new file mode 100644
index 0000000..12f574d
--- /dev/null
+++ b/test/test_recover/phi.dl
@@ -0,0 +1,138 @@
+
+
+"Time = 0.000000
+-0.866025           0.0000000000000
+-0.774865           0.0000000000000
+-0.683704           0.0000000000000
+-0.592544           0.0000000000000
+-0.501383           0.0000000000121
+-0.410223           0.0000000491566
+-0.319062           0.0000379191081
+-0.227901           0.0055502860213
+-0.136741           0.1541535501874
+-0.045580           0.8124050503725
+0.045580           0.8124050503725
+0.136741           0.1541535501874
+0.227901           0.0055502860213
+0.319062           0.0000379191081
+0.410223           0.0000000491566
+0.501383           0.0000000000121
+0.592544           0.0000000000000
+0.683704           0.0000000000000
+0.774865           0.0000000000000
+0.866025           0.0000000000000
+
+
+"Time = 0.052632
+-0.866025           0.0000000000000
+-0.774865           0.0000000000000
+-0.683704           0.0000000000000
+-0.592544           0.0000000000000
+-0.501383          -0.0000000000791
+-0.410223          -0.0000002406983
+-0.319062          -0.0000993771711
+-0.227901          -0.0072242068990
+-0.136741          -0.0979217322775
+-0.045580          -0.1950571619856
+0.045580          -0.1950571619856
+0.136741          -0.0979217322775
+0.227901          -0.0072242068990
+0.319062          -0.0000993771711
+0.410223          -0.0000002406983
+0.501383          -0.0000000000791
+0.592544           0.0000000000000
+0.683704           0.0000000000000
+0.774865           0.0000000000000
+0.866025           0.0000000000000
+
+
+"Time = 0.105263
+-0.866025           0.0000000000000
+-0.774865           0.0000000000000
+-0.683704           0.0000000000000
+-0.592544          -0.0000000000000
+-0.501383          -0.0000000070778
+-0.410223          -0.0000069066610
+-0.319062          -0.0010528818637
+-0.227901          -0.0332682845732
+-0.136741          -0.2986866103570
+-0.045580          -1.1290503258507
+0.045580          -1.1290503258507
+0.136741          -0.2986866103570
+0.227901          -0.0332682845732
+0.319062          -0.0010528818637
+0.410223          -0.0000069066610
+0.501383          -0.0000000070778
+0.592544          -0.0000000000000
+0.683704           0.0000000000000
+0.774865           0.0000000000000
+0.866025           0.0000000000000
+
+
+"Time = 0.157895
+-0.866025           0.0000000000000
+-0.774865           0.0000000000000
+-0.683704           0.0000000000000
+-0.592544          -0.0000000001229
+-0.501383          -0.0000003859714
+-0.410223          -0.0001219858026
+-0.319062          -0.0070834276763
+-0.227901          -0.0973584266251
+-0.136741          -0.4576544210254
+-0.045580          -0.9019842677611
+0.045580          -0.9019842677611
+0.136741          -0.4576544210254
+0.227901          -0.0973584266251
+0.319062          -0.0070834276763
+0.410223          -0.0001219858026
+0.501383          -0.0000003859714
+0.592544          -0.0000000001229
+0.683704           0.0000000000000
+0.774865           0.0000000000000
+0.866025           0.0000000000000
+
+
+"Time = 0.210526
+-0.866025           0.0000000000000
+-0.774865           0.0000000000000
+-0.683704           0.0000000000016
+-0.592544          -0.0000000159252
+-0.501383          -0.0000113878002
+-0.410223          -0.0012768875054
+-0.319062          -0.0304556068145
+-0.227901          -0.2006651910238
+-0.136741          -0.4997593550838
+-0.045580          -0.2663944260771
+0.045580          -0.2663944260771
+0.136741          -0.4997593550838
+0.227901          -0.2006651910238
+0.319062          -0.0304556068145
+0.410223          -0.0012768875054
+0.501383          -0.0000113878002
+0.592544          -0.0000000159252
+0.683704           0.0000000000016
+0.774865           0.0000000000000
+0.866025           0.0000000000000
+
+
+"Time = 0.263158
+-0.866025           0.0000000000000
+-0.774865           0.0000000000002
+-0.683704          -0.0000000003848
+-0.592544          -0.0000008256207
+-0.501383          -0.0001851985238
+-0.410223          -0.0080724403495
+-0.319062          -0.0860002778716
+-0.227901          -0.3020549188440
+-0.136741          -0.2694342257320
+-0.045580          -0.0172399268943
+0.045580          -0.0172399268943
+0.136741          -0.2694342257320
+0.227901          -0.3020549188440
+0.319062          -0.0860002778716
+0.410223          -0.0080724403495
+0.501383          -0.0001851985238
+0.592544          -0.0000008256207
+0.683704          -0.0000000003848
+0.774865           0.0000000000002
+0.866025           0.0000000000000
diff --git a/test/test_recover/phi.xl b/test/test_recover/phi.xl
new file mode 100644
index 0000000..55a520c
--- /dev/null
+++ b/test/test_recover/phi.xl
@@ -0,0 +1,138 @@
+
+
+"Time = 0.000000
+-0.500000           0.0000000000121
+-0.447368           0.0000000017699
+-0.394737           0.0000001488657
+-0.342105           0.0000071951364
+-0.289474           0.0001998375962
+-0.236842           0.0031894000956
+-0.184211           0.0292505776823
+-0.131579           0.1541535501874
+-0.078947           0.4668380576006
+-0.026316           0.8124050503725
+0.026316           0.8124050503725
+0.078947           0.4668380576006
+0.131579           0.1541535501874
+0.184211           0.0292505776823
+0.236842           0.0031894000956
+0.289474           0.0001998375962
+0.342105           0.0000071951364
+0.394737           0.0000001488657
+0.447368           0.0000000017699
+0.500000           0.0000000000121
+
+
+"Time = 0.052632
+-0.500000           0.0000000001579
+-0.447368           0.0000002442580
+-0.394737           0.0000055185355
+-0.342105           0.0000636892398
+-0.289474           0.0002832523498
+-0.236842          -0.0010800497738
+-0.184211          -0.0182063021811
+-0.131579          -0.0835854975190
+-0.078947          -0.1767183072710
+-0.026316          -0.1950571619856
+0.026316          -0.1950571619856
+0.078947          -0.1767183072710
+0.131579          -0.0835854975190
+0.184211          -0.0182063021811
+0.236842          -0.0010800497738
+0.289474           0.0002832523498
+0.342105           0.0000636892398
+0.394737           0.0000055185355
+0.447368           0.0000002442580
+0.500000           0.0000000001579
+
+
+"Time = 0.105263
+-0.500000           0.0000000000000
+-0.447368           0.0000074500893
+-0.394737           0.0000571366787
+-0.342105           0.0000891850929
+-0.289474          -0.0021735254063
+-0.236842          -0.0208336611591
+-0.184211          -0.0974778459232
+-0.131579          -0.2995396203857
+-0.078947          -0.6860436385846
+-0.026316          -1.1290503258507
+0.026316          -1.1290503258507
+0.078947          -0.6860436385846
+0.131579          -0.2995396203857
+0.184211          -0.0974778459232
+0.236842          -0.0208336611591
+0.289474          -0.0021735254063
+0.342105           0.0000891850929
+0.394737           0.0000571366787
+0.447368           0.0000074500893
+0.500000           0.0000000000000
+
+
+"Time = 0.157895
+-0.500000           0.0000000000121
+-0.447368           0.0000404464522
+-0.394737          -0.0000934611932
+-0.342105          -0.0028408356421
+-0.289474          -0.0203343223824
+-0.236842          -0.0840586024960
+-0.184211          -0.2362253936422
+-0.131579          -0.4920251452215
+-0.078947          -0.7713978386790
+-0.026316          -0.9019842677611
+0.026316          -0.9019842677611
+0.078947          -0.7713978386790
+0.131579          -0.4920251452215
+0.184211          -0.2362253936422
+0.236842          -0.0840586024960
+0.289474          -0.0203343223824
+0.342105          -0.0028408356421
+0.394737          -0.0000934611932
+0.447368           0.0000404464522
+0.500000           0.0000000000121
+
+
+"Time = 0.210526
+-0.500000           0.0000000001579
+-0.447368          -0.0002560735181
+-0.394737          -0.0032223677598
+-0.342105          -0.0191504023972
+-0.289474          -0.0718425508170
+-0.236842          -0.1869773941516
+-0.184211          -0.3531984196036
+-0.131579          -0.4762760737732
+-0.078947          -0.4212470397782
+-0.026316          -0.2663944260771
+0.026316          -0.2663944260771
+0.078947          -0.4212470397782
+0.131579          -0.4762760737732
+0.184211          -0.3531984196036
+0.236842          -0.1869773941516
+0.289474          -0.0718425508170
+0.342105          -0.0191504023972
+0.394737          -0.0032223677598
+0.447368          -0.0002560735181
+0.500000           0.0000000001579
+
+
+"Time = 0.263158
+-0.500000           0.0000000000000
+-0.447368          -0.0034053134726
+-0.394737          -0.0177523942812
+-0.342105          -0.0616617809179
+-0.289474          -0.1512610524542
+-0.236842          -0.2676581405491
+-0.184211          -0.3314593226319
+-0.131579          -0.2530697030290
+-0.078947          -0.0930373771550
+-0.026316          -0.0172399268943
+0.026316          -0.0172399268943
+0.078947          -0.0930373771550
+0.131579          -0.2530697030290
+0.184211          -0.3314593226319
+0.236842          -0.2676581405491
+0.289474          -0.1512610524542
+0.342105          -0.0616617809179
+0.394737          -0.0177523942812
+0.447368          -0.0034053134726
+0.500000           0.0000000000000
diff --git a/test/test_recover/phi.yl b/test/test_recover/phi.yl
new file mode 100644
index 0000000..55a520c
--- /dev/null
+++ b/test/test_recover/phi.yl
@@ -0,0 +1,138 @@
+
+
+"Time = 0.000000
+-0.500000           0.0000000000121
+-0.447368           0.0000000017699
+-0.394737           0.0000001488657
+-0.342105           0.0000071951364
+-0.289474           0.0001998375962
+-0.236842           0.0031894000956
+-0.184211           0.0292505776823
+-0.131579           0.1541535501874
+-0.078947           0.4668380576006
+-0.026316           0.8124050503725
+0.026316           0.8124050503725
+0.078947           0.4668380576006
+0.131579           0.1541535501874
+0.184211           0.0292505776823
+0.236842           0.0031894000956
+0.289474           0.0001998375962
+0.342105           0.0000071951364
+0.394737           0.0000001488657
+0.447368           0.0000000017699
+0.500000           0.0000000000121
+
+
+"Time = 0.052632
+-0.500000           0.0000000001579
+-0.447368           0.0000002442580
+-0.394737           0.0000055185355
+-0.342105           0.0000636892398
+-0.289474           0.0002832523498
+-0.236842          -0.0010800497738
+-0.184211          -0.0182063021811
+-0.131579          -0.0835854975190
+-0.078947          -0.1767183072710
+-0.026316          -0.1950571619856
+0.026316          -0.1950571619856
+0.078947          -0.1767183072710
+0.131579          -0.0835854975190
+0.184211          -0.0182063021811
+0.236842          -0.0010800497738
+0.289474           0.0002832523498
+0.342105           0.0000636892398
+0.394737           0.0000055185355
+0.447368           0.0000002442580
+0.500000           0.0000000001579
+
+
+"Time = 0.105263
+-0.500000           0.0000000000000
+-0.447368           0.0000074500893
+-0.394737           0.0000571366787
+-0.342105           0.0000891850929
+-0.289474          -0.0021735254063
+-0.236842          -0.0208336611591
+-0.184211          -0.0974778459232
+-0.131579          -0.2995396203857
+-0.078947          -0.6860436385846
+-0.026316          -1.1290503258507
+0.026316          -1.1290503258507
+0.078947          -0.6860436385846
+0.131579          -0.2995396203857
+0.184211          -0.0974778459232
+0.236842          -0.0208336611591
+0.289474          -0.0021735254063
+0.342105           0.0000891850929
+0.394737           0.0000571366787
+0.447368           0.0000074500893
+0.500000           0.0000000000000
+
+
+"Time = 0.157895
+-0.500000           0.0000000000121
+-0.447368           0.0000404464522
+-0.394737          -0.0000934611932
+-0.342105          -0.0028408356421
+-0.289474          -0.0203343223824
+-0.236842          -0.0840586024960
+-0.184211          -0.2362253936422
+-0.131579          -0.4920251452215
+-0.078947          -0.7713978386790
+-0.026316          -0.9019842677611
+0.026316          -0.9019842677611
+0.078947          -0.7713978386790
+0.131579          -0.4920251452215
+0.184211          -0.2362253936422
+0.236842          -0.0840586024960
+0.289474          -0.0203343223824
+0.342105          -0.0028408356421
+0.394737          -0.0000934611932
+0.447368           0.0000404464522
+0.500000           0.0000000000121
+
+
+"Time = 0.210526
+-0.500000           0.0000000001579
+-0.447368          -0.0002560735181
+-0.394737          -0.0032223677598
+-0.342105          -0.0191504023972
+-0.289474          -0.0718425508170
+-0.236842          -0.1869773941516
+-0.184211          -0.3531984196036
+-0.131579          -0.4762760737732
+-0.078947          -0.4212470397782
+-0.026316          -0.2663944260771
+0.026316          -0.2663944260771
+0.078947          -0.4212470397782
+0.131579          -0.4762760737732
+0.184211          -0.3531984196036
+0.236842          -0.1869773941516
+0.289474          -0.0718425508170
+0.342105          -0.0191504023972
+0.394737          -0.0032223677598
+0.447368          -0.0002560735181
+0.500000           0.0000000001579
+
+
+"Time = 0.263158
+-0.500000           0.0000000000000
+-0.447368          -0.0034053134726
+-0.394737          -0.0177523942812
+-0.342105          -0.0616617809179
+-0.289474          -0.1512610524542
+-0.236842          -0.2676581405491
+-0.184211          -0.3314593226319
+-0.131579          -0.2530697030290
+-0.078947          -0.0930373771550
+-0.026316          -0.0172399268943
+0.026316          -0.0172399268943
+0.078947          -0.0930373771550
+0.131579          -0.2530697030290
+0.184211          -0.3314593226319
+0.236842          -0.2676581405491
+0.289474          -0.1512610524542
+0.342105          -0.0616617809179
+0.394737          -0.0177523942812
+0.447368          -0.0034053134726
+0.500000           0.0000000000000
diff --git a/test/test_recover/phi_max.tl b/test/test_recover/phi_max.tl
new file mode 100644
index 0000000..7d6d6cf
--- /dev/null
+++ b/test/test_recover/phi_max.tl
@@ -0,0 +1,7 @@
+"phi max v time
+0.000000           0.8124050503725
+0.052632           0.0002832523498
+0.105263           0.0000891850929
+0.157895           0.0000404464522
+0.210526           0.0000003961856
+0.263158           0.0000001100397
diff --git a/test/test_recover/phi_min.tl b/test/test_recover/phi_min.tl
new file mode 100644
index 0000000..a1d8579
--- /dev/null
+++ b/test/test_recover/phi_min.tl
@@ -0,0 +1,7 @@
+"phi min v time
+0.000000           0.0000000000000
+0.052632          -0.1950571619856
+0.105263          -1.1290503258507
+0.157895          -0.9019842677611
+0.210526          -0.4997593550838
+0.263158          -0.3594146186543
diff --git a/test/test_recover/phi_nm1.tl b/test/test_recover/phi_nm1.tl
new file mode 100644
index 0000000..895a3b0
--- /dev/null
+++ b/test/test_recover/phi_nm1.tl
@@ -0,0 +1,7 @@
+"phi norm1 v time
+0.000000           0.0047741452163
+0.052632           0.0023586563905
+0.105263           0.0094832901779
+0.157895           0.0166095490059
+0.210526           0.0237365580784
+0.263158           0.0308582835960
diff --git a/test/test_recover/phi_nm2.tl b/test/test_recover/phi_nm2.tl
new file mode 100644
index 0000000..0fea72d
--- /dev/null
+++ b/test/test_recover/phi_nm2.tl
@@ -0,0 +1,7 @@
+"phi norm2 v time
+0.000000           0.0410842432418
+0.052632           0.0158441957039
+0.105263           0.0637392141088
+0.157895           0.0805995803466
+0.210526           0.0821416890828
+0.263158           0.0821638479495