diff options
36 files changed, 6596 insertions, 0 deletions
diff --git a/COPYRIGHT b/COPYRIGHT new file mode 100644 index 0000000..57b2896 --- /dev/null +++ b/COPYRIGHT @@ -0,0 +1,352 @@ + + +This thorn is (c) Copyright the authors listed in the sources files. This +thorn is distributed under the GNU GPL with the specific exception that +the GNU GPL does not migrate to code linking to or using infrastructure +from this thorn. + +- The Cactus Team <cactus@cactuscode.org> + +************************************************************************** + + GNU GENERAL PUBLIC LICENSE + Version 2, June 1991 + + Copyright (C) 1989, 1991 Free Software Foundation, Inc. + 675 Mass Ave, Cambridge, MA 02139, USA + Everyone is permitted to copy and distribute verbatim copies + of this license document, but changing it is not allowed. + + Preamble + + The licenses for most software are designed to take away your +freedom to share and change it. By contrast, the GNU General Public +License is intended to guarantee your freedom to share and change free +software--to make sure the software is free for all its users. This +General Public License applies to most of the Free Software +Foundation's software and to any other program whose authors commit to +using it. (Some other Free Software Foundation software is covered by +the GNU Library General Public License instead.) You can apply it to +your programs, too. + + When we speak of free software, we are referring to freedom, not +price. Our General Public Licenses are designed to make sure that you +have the freedom to distribute copies of free software (and charge for +this service if you wish), that you receive source code or can get it +if you want it, that you can change the software or use pieces of it +in new free programs; and that you know you can do these things. + + To protect your rights, we need to make restrictions that forbid +anyone to deny you these rights or to ask you to surrender the rights. +These restrictions translate to certain responsibilities for you if you +distribute copies of the software, or if you modify it. + + For example, if you distribute copies of such a program, whether +gratis or for a fee, you must give the recipients all the rights that +you have. You must make sure that they, too, receive or can get the +source code. And you must show them these terms so they know their +rights. + + We protect your rights with two steps: (1) copyright the software, and +(2) offer you this license which gives you legal permission to copy, +distribute and/or modify the software. + + Also, for each author's protection and ours, we want to make certain +that everyone understands that there is no warranty for this free +software. If the software is modified by someone else and passed on, we +want its recipients to know that what they have is not the original, so +that any problems introduced by others will not reflect on the original +authors' reputations. + + Finally, any free program is threatened constantly by software +patents. We wish to avoid the danger that redistributors of a free +program will individually obtain patent licenses, in effect making the +program proprietary. To prevent this, we have made it clear that any +patent must be licensed for everyone's free use or not licensed at all. + + The precise terms and conditions for copying, distribution and +modification follow. + + GNU GENERAL PUBLIC LICENSE + TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION + + 0. This License applies to any program or other work which contains +a notice placed by the copyright holder saying it may be distributed +under the terms of this General Public License. The "Program", below, +refers to any such program or work, and a "work based on the Program" +means either the Program or any derivative work under copyright law: +that is to say, a work containing the Program or a portion of it, +either verbatim or with modifications and/or translated into another +language. (Hereinafter, translation is included without limitation in +the term "modification".) Each licensee is addressed as "you". + +Activities other than copying, distribution and modification are not +covered by this License; they are outside its scope. The act of +running the Program is not restricted, and the output from the Program +is covered only if its contents constitute a work based on the +Program (independent of having been made by running the Program). +Whether that is true depends on what the Program does. + + 1. You may copy and distribute verbatim copies of the Program's +source code as you receive it, in any medium, provided that you +conspicuously and appropriately publish on each copy an appropriate +copyright notice and disclaimer of warranty; keep intact all the +notices that refer to this License and to the absence of any warranty; +and give any other recipients of the Program a copy of this License +along with the Program. + +You may charge a fee for the physical act of transferring a copy, and +you may at your option offer warranty protection in exchange for a fee. + + 2. You may modify your copy or copies of the Program or any portion +of it, thus forming a work based on the Program, and copy and +distribute such modifications or work under the terms of Section 1 +above, provided that you also meet all of these conditions: + + a) You must cause the modified files to carry prominent notices + stating that you changed the files and the date of any change. + + b) You must cause any work that you distribute or publish, that in + whole or in part contains or is derived from the Program or any + part thereof, to be licensed as a whole at no charge to all third + parties under the terms of this License. + + c) If the modified program normally reads commands interactively + when run, you must cause it, when started running for such + interactive use in the most ordinary way, to print or display an + announcement including an appropriate copyright notice and a + notice that there is no warranty (or else, saying that you provide + a warranty) and that users may redistribute the program under + these conditions, and telling the user how to view a copy of this + License. (Exception: if the Program itself is interactive but + does not normally print such an announcement, your work based on + the Program is not required to print an announcement.) + +These requirements apply to the modified work as a whole. If +identifiable sections of that work are not derived from the Program, +and can be reasonably considered independent and separate works in +themselves, then this License, and its terms, do not apply to those +sections when you distribute them as separate works. But when you +distribute the same sections as part of a whole which is a work based +on the Program, the distribution of the whole must be on the terms of +this License, whose permissions for other licensees extend to the +entire whole, and thus to each and every part regardless of who wrote it. + +Thus, it is not the intent of this section to claim rights or contest +your rights to work written entirely by you; rather, the intent is to +exercise the right to control the distribution of derivative or +collective works based on the Program. + +In addition, mere aggregation of another work not based on the Program +with the Program (or with a work based on the Program) on a volume of +a storage or distribution medium does not bring the other work under +the scope of this License. + + 3. You may copy and distribute the Program (or a work based on it, +under Section 2) in object code or executable form under the terms of +Sections 1 and 2 above provided that you also do one of the following: + + a) Accompany it with the complete corresponding machine-readable + source code, which must be distributed under the terms of Sections + 1 and 2 above on a medium customarily used for software interchange; or, + + b) Accompany it with a written offer, valid for at least three + years, to give any third party, for a charge no more than your + cost of physically performing source distribution, a complete + machine-readable copy of the corresponding source code, to be + distributed under the terms of Sections 1 and 2 above on a medium + customarily used for software interchange; or, + + c) Accompany it with the information you received as to the offer + to distribute corresponding source code. (This alternative is + allowed only for noncommercial distribution and only if you + received the program in object code or executable form with such + an offer, in accord with Subsection b above.) + +The source code for a work means the preferred form of the work for +making modifications to it. For an executable work, complete source +code means all the source code for all modules it contains, plus any +associated interface definition files, plus the scripts used to +control compilation and installation of the executable. However, as a +special exception, the source code distributed need not include +anything that is normally distributed (in either source or binary +form) with the major components (compiler, kernel, and so on) of the +operating system on which the executable runs, unless that component +itself accompanies the executable. + +If distribution of executable or object code is made by offering +access to copy from a designated place, then offering equivalent +access to copy the source code from the same place counts as +distribution of the source code, even though third parties are not +compelled to copy the source along with the object code. + + 4. You may not copy, modify, sublicense, or distribute the Program +except as expressly provided under this License. Any attempt +otherwise to copy, modify, sublicense or distribute the Program is +void, and will automatically terminate your rights under this License. +However, parties who have received copies, or rights, from you under +this License will not have their licenses terminated so long as such +parties remain in full compliance. + + 5. You are not required to accept this License, since you have not +signed it. However, nothing else grants you permission to modify or +distribute the Program or its derivative works. These actions are +prohibited by law if you do not accept this License. Therefore, by +modifying or distributing the Program (or any work based on the +Program), you indicate your acceptance of this License to do so, and +all its terms and conditions for copying, distributing or modifying +the Program or works based on it. + + 6. Each time you redistribute the Program (or any work based on the +Program), the recipient automatically receives a license from the +original licensor to copy, distribute or modify the Program subject to +these terms and conditions. You may not impose any further +restrictions on the recipients' exercise of the rights granted herein. +You are not responsible for enforcing compliance by third parties to +this License. + + 7. If, as a consequence of a court judgment or allegation of patent +infringement or for any other reason (not limited to patent issues), +conditions are imposed on you (whether by court order, agreement or +otherwise) that contradict the conditions of this License, they do not +excuse you from the conditions of this License. If you cannot +distribute so as to satisfy simultaneously your obligations under this +License and any other pertinent obligations, then as a consequence you +may not distribute the Program at all. For example, if a patent +license would not permit royalty-free redistribution of the Program by +all those who receive copies directly or indirectly through you, then +the only way you could satisfy both it and this License would be to +refrain entirely from distribution of the Program. + +If any portion of this section is held invalid or unenforceable under +any particular circumstance, the balance of the section is intended to +apply and the section as a whole is intended to apply in other +circumstances. + +It is not the purpose of this section to induce you to infringe any +patents or other property right claims or to contest validity of any +such claims; this section has the sole purpose of protecting the +integrity of the free software distribution system, which is +implemented by public license practices. Many people have made +generous contributions to the wide range of software distributed +through that system in reliance on consistent application of that +system; it is up to the author/donor to decide if he or she is willing +to distribute software through any other system and a licensee cannot +impose that choice. + +This section is intended to make thoroughly clear what is believed to +be a consequence of the rest of this License. + + 8. If the distribution and/or use of the Program is restricted in +certain countries either by patents or by copyrighted interfaces, the +original copyright holder who places the Program under this License +may add an explicit geographical distribution limitation excluding +those countries, so that distribution is permitted only in or among +countries not thus excluded. In such case, this License incorporates +the limitation as if written in the body of this License. + + 9. The Free Software Foundation may publish revised and/or new versions +of the General Public License from time to time. Such new versions will +be similar in spirit to the present version, but may differ in detail to +address new problems or concerns. + +Each version is given a distinguishing version number. If the Program +specifies a version number of this License which applies to it and "any +later version", you have the option of following the terms and conditions +either of that version or of any later version published by the Free +Software Foundation. If the Program does not specify a version number of +this License, you may choose any version ever published by the Free Software +Foundation. + + 10. If you wish to incorporate parts of the Program into other free +programs whose distribution conditions are different, write to the author +to ask for permission. For software which is copyrighted by the Free +Software Foundation, write to the Free Software Foundation; we sometimes +make exceptions for this. Our decision will be guided by the two goals +of preserving the free status of all derivatives of our free software and +of promoting the sharing and reuse of software generally. + + NO WARRANTY + + 11. BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY +FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN +OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES +PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED +OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF +MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS +TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE +PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, +REPAIR OR CORRECTION. + + 12. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING +WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR +REDISTRIBUTE THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, +INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING +OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED +TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY +YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER +PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE +POSSIBILITY OF SUCH DAMAGES. + + END OF TERMS AND CONDITIONS + + Appendix: How to Apply These Terms to Your New Programs + + If you develop a new program, and you want it to be of the greatest +possible use to the public, the best way to achieve this is to make it +free software which everyone can redistribute and change under these terms. + + To do so, attach the following notices to the program. It is safest +to attach them to the start of each source file to most effectively +convey the exclusion of warranty; and each file should have at least +the "copyright" line and a pointer to where the full notice is found. + + <one line to give the program's name and a brief idea of what it does.> + Copyright (C) 19yy <name of author> + + This program is free software; you can redistribute it and/or modify + it under the terms of the GNU General Public License as published by + the Free Software Foundation; either version 2 of the License, or + (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program; if not, write to the Free Software + Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA. + +Also add information on how to contact you by electronic and paper mail. + +If the program is interactive, make it output a short notice like this +when it starts in an interactive mode: + + Gnomovision version 69, Copyright (C) 19yy name of author + Gnomovision comes with ABSOLUTELY NO WARRANTY; for details type `show w'. + This is free software, and you are welcome to redistribute it + under certain conditions; type `show c' for details. + +The hypothetical commands `show w' and `show c' should show the appropriate +parts of the General Public License. Of course, the commands you use may +be called something other than `show w' and `show c'; they could even be +mouse-clicks or menu items--whatever suits your program. + +You should also get your employer (if you work as a programmer) or your +school, if any, to sign a "copyright disclaimer" for the program, if +necessary. Here is a sample; alter the names: + + Yoyodyne, Inc., hereby disclaims all copyright interest in the program + `Gnomovision' (which makes passes at compilers) written by James Hacker. + + <signature of Ty Coon>, 1 April 1989 + Ty Coon, President of Vice + +This General Public License does not permit incorporating your program into +proprietary programs. If your program is a subroutine library, you may +consider it more useful to permit linking proprietary applications with the +library. If this is what you want to do, use the GNU Library General +Public License instead of this License. + + @@ -0,0 +1,22 @@ +Cactus Code Thorn IOHDF5 +Authors : Thomas Radke +CVS info : $Header$ +-------------------------------------------------------------------------- + +1. Purpose of the thorn + +This thorn does output of 2D and 3D variables in HDF5 format. + +It also provides checkpointing/recovery functionality - +you can create HDF5 checkpoint files and recover from them. + +The IO methods "IOHDF5_2D" and "IOHDF5_3D" are registered +which can be called by other thorns via CCTK_OutputVarAsByMethod() +to output a variable without specifying it in the parameter file. + + +2. Additional information + +Currently only 3D output is implemented. + +Fore more information on HDF5 see http://hdf.ncsa.uiuc.edu. diff --git a/doc/documentation.tex b/doc/documentation.tex new file mode 100644 index 0000000..1a617ab --- /dev/null +++ b/doc/documentation.tex @@ -0,0 +1,97 @@ +% Thorn documentation template +\documentclass{article} +\begin{document} + +\title{IOHDF5} +\author{Thomas Radke} +\date{1999} +\maketitle + +\abstract{Thorn IOHDF5 provides an IO method for output of variables in HDF5, +the Hierarchical Data Format version 5 ({\tt http://hdf.ncsa.uiuc.edu/whatishdf5.html}).\\ +It also implements checkpointing/recovery functionality using HDF5.} +% +\section{Purpose} +% +Thorn IOHDF5 makes HDF5 available to the Cactus user by registering an IO method +called {\tt IOHDF5\_3D} with the IO interface in the flesh.\\ +This method creates three-dimensional output of 3D grid functions as well as +plane output of scalar variables. +Data is written in HDF5 file format and goes into files named +{\tt "<varname>\_3D.h5"}. Such datafiles can be further processed by +visualization tools like Amira or AVS.\\ +% +\newline +% +You obtain output by an IO method by either +% +\begin{itemize} + \item setting the appropriate IO parameters + \item calling one the routines of the IO function interface provided by the flesh +\end{itemize} +% +% +IOHDF5 also provides checkpointing/recovery functionality by registering +% +\begin{itemize} + \item a checkpoint method with the Runtime Function Repository in the flesh + if checkpointing was requested for this thorn. + This method is then scheduled at {\em CCTK\_CPINITIAL} and/or + {\em CCTK\_CHECKPOINT} (to checkpoint initial and/or evolution data).\\ + Controled via checkpoint parameters, it decides when to save the current + state of simulation by dumping the contents of all Cactus variables and + parameters into a checkpoint file which is in HDF5 file format.\\ + To write this file the thorn's IO method {\tt IOHDF5\_3D} is used. + \item a recovery method with the generic recovery function interface of thorn IOUtil.\\ + At recovery time this method is called by IOUtil's generic recovery + routine. It gets passed a filename which is tested to identify a checkpoint + file in HDF5 format. If successful the method will then restore the + contents of all Cactus variables and parameters from the given checkpoint + file. +\end{itemize} +% +For a description of IO and checkpoint/recovery parameters and the generic +recovery function interface please see also the documentation of thorn IOUtil.\\ +For a description of IO function interface to invoke IO methods by application +thorns please see the flesh documentation. +% +% +\section{Comments} + +Since IOHDF5 uses parameters and the recovery function interface from IOUtil +it also needs this I/O skeleton thorn compiled into Cactus and activated.\\ +% +% +\newline +{\bf Building Cactus with HDF5}\\ +% +The Cactus distribution does not contain the HDF5 header files and library which +is used by thorn IOHDF5. So you need to configure it as an external software +package via: +% +\begin{verbatim} + make <configuration>-config HDF5=YES + [HDF5_DIR=<path to HDF5 package>] +\end{verbatim} +% +The configuration script will look in some default places for an installed +HDF5 package. If nothing is found this way you can explicitely specify it with +the {\tt HDF5\_DIR} option.\\ +% +Configure also checks which library version is contained within the HDF5 +package: it can be either serial or parallel. The latter version includes the parallel IO extensions of the MPI 2 standard. To make use of these extensions +you need to configure Cactus with both HDF5 and MPI. Please make also sure then +that the parallel HDF5 library was built with the same MPI version as is used +for Cactus.\\ +% +If Cactus was not configured to use HDF5 but has thorn IOHDF5 compiled in +it will give a warning message each time a thorn's routine is called +saying HDF5 I/O is not available. +% +% Automatically created from the ccl files +% Do not worry for now. +\include{interface} +\include{param} +\include{schedule} + +\end{document} diff --git a/interface.ccl b/interface.ccl new file mode 100644 index 0000000..7f68414 --- /dev/null +++ b/interface.ccl @@ -0,0 +1,5 @@ +# Interface definition for thorn IOHDF5 +# $Header$ + +implements: IOHDF5 +inherits: CACTUS diff --git a/par/hdf5.par b/par/hdf5.par new file mode 100644 index 0000000..ec67555 --- /dev/null +++ b/par/hdf5.par @@ -0,0 +1,31 @@ +# hdf5.par: HDF5 output + +ActiveThorns = "CartGrid3D IOUtil IDScalarWave WavetoyF77 PUGH IOHDF5" + +Cactus::cctk_brief_output = "yes" + +IDScalarWave::radius = 0.0 +IDScalarWave::amplitude = 1.0 +IDScalarWave::initial_data = "gaussian" + +cactus::cctk_itlast = 10 + +# PUGH parameters +driver::global_nx = 10 +driver::global_ny = 10 +driver::global_nz = 10 + + +####################### IO parameters ################################### + +# flag to enable verbose output +IO::verbose = "yes" + +# say how often to output +IO::out_every = 1 +IO::out3D_mode = "onefile" +IO::out3D_unchunked = "yes" + +# say what GF to output +IOHDF5::out3D_vars = "all" +IOHDF5::outdir3D = "./hdf5" diff --git a/param.ccl b/param.ccl new file mode 100644 index 0000000..f222e18 --- /dev/null +++ b/param.ccl @@ -0,0 +1,290 @@ +# Parameter definitions for thorn IOHDF5 + +############################################################################# +### declare IOHDF5 parameters +############################################################################# +private: + +######################## +# How often to do output +######################## +INT out1D_every "How often to do 1D output, overrides out_every" STEERABLE = ALWAYS +{ + -1:* :: +} -1 +INT out2D_every "How often to do 2D output, overrides out_every" STEERABLE = ALWAYS +{ + -1:* :: +} -1 +INT out3D_every "How often to do 3D output, overrides out_every" STEERABLE = ALWAYS +{ + -1:* :: +} -1 + +#################### +# Output directories +#################### +STRING outdir1D "Name of IO 1D output directory, overrides outdir" +{ + .* :: A regex which matches everything +} "." +STRING outdir2D "Name of IO 2D output directory, overrides outdir" +{ + .* :: A regex which matches everything +} "." +STRING outdir3D "Name of IO 3D output directory, overrides outdir" +{ + .* :: A regex which matches everything +} "." + + +##################### +# Variables to output +##################### +STRING out1D_vars "Variables to output in 1D HDF5 file format" STEERABLE = ALWAYS +{ + .* :: A regex which matches everything +} "" +STRING out2D_vars "Variables to output in 2D HDF5 file format" STEERABLE = ALWAYS +{ + .* :: A regex which matches everything +} "" +STRING out3D_vars "Variables to output in 3D HDF5 file format" STEERABLE = ALWAYS +{ + .* :: A regex which matches everything +} "" + +############################################## +# Downsampling parameters specific to IOHDF5 +############################################## +# 1D +###### +STRING origin1D "Default origin" +{ + .* :: "Comma separated list of positive integer values" +} "0,0,0" +STRING downsampling1D "Default downsampling" +{ + .* :: "Comma separated list of positive integer values" +} "1,1,1" +STRING length1D "Default length of the 1D hyperslab" +{ + .* :: "Comma separated list of integer values" +} "-1,-1,-1" + +###### +# 2D +###### +STRING origin2D "Default origin 2D" +{ + .* :: "Comma separated list of positive integer values" +} "0,0,0" +STRING downsampling2D "Default downsampling 2D" +{ + .* :: "Comma separated list of positive integer values" +} "1,1,1" +STRING length2D "Default length 2D" +{ + .* :: "Comma separated list of integer values" +} "-1,-1,-1" + +###### +# 3D +###### +STRING origin3D "Default origin 3D" +{ + .* :: "Comma separated list of positive integer values" +} "0,0,0" +STRING downsampling3D "Default downsampling 3D" +{ + .* :: "Comma separated list of positive integer values" +} "1,1,1" +STRING length3D "Default length of the 3D hyperslab" +{ + .* :: "Comma separated list of integer values 3D" +} "-1,-1,-1" + +###### +# 4D +###### +STRING origin4D "Default origin" +{ + .* :: "Comma separated list of positive integer values" +} "0,0,0" +STRING downsampling4D "Default downsampling" +{ + .* :: "Comma separated list of positive integer values" +} "1,1,1" +STRING length4D "Default length of the hyperslab to stream" +{ + .* :: "Comma separated list of integer values" +} "-1,-1,-1" + +########## +# General +########## +STRING origin "Default origin" +{ + .* :: "Comma separated list of positive integer values" +} "0,0,0" +STRING downsampling "Default downsampling" +{ + .* :: "Comma separated list of positive integer values" +} "1,1,1" +STRING length "Default length of the hyperslab to stream" +{ + .* :: "Comma separated list of integer values" +} "-1,-1,-1" + +######################### +# Checkpointing requested +######################### +BOOLEAN checkpoint "Do checkpointing with HDF5" +{ +} "no" + + +############################################################################# +### import IOUtil parameters +############################################################################# +shares: IO + +#################### +# Output directories +#################### +USES STRING outdir "" +{ +} + + +######################## +# How often to do output +######################## +USES INT out_every "" +{ + : :: +} + + +################ +# various things +################ +USES BOOLEAN verbose "" +{ +} +USES BOOLEAN print_timing_info "" +{ +} +USES BOOLEAN out3D_datestamp "" +{ +} +USES BOOLEAN out3D_parameters "" +{ +} +USES BOOLEAN out2D_datestamp "" +{ +} +USES BOOLEAN out2D_parameters "" +{ +} +USES BOOLEAN out1D_datestamp "" +{ +} +USES BOOLEAN out1D_parameters "" +{ +} + +####################### +# Specific to 1/2/3D output +####################### +USES KEYWORD out3D_mode "" +{ +} +USES INT out3D_procs "" +{ + : :: +} +USES BOOLEAN out3D_septimefiles "" +{ +} +USES BOOLEAN out3D_unchunked "" +{ +} + +USES KEYWORD out2D_mode "" +{ +} +USES INT out2D_procs "" +{ + : :: +} +USES BOOLEAN out2D_septimefiles "" +{ +} +USES BOOLEAN out2D_unchunked "" +{ +} +USES KEYWORD out1D_mode "" +{ +} +USES INT out1D_procs "" +{ + : :: +} +USES BOOLEAN out1D_septimefiles "" +{ +} +USES BOOLEAN out1D_unchunked "" +{ +} +############################################## +# Downsampling parameters based on IOUTIL +############################################## +USES INT out3D_downsample_x "" +{ + : :: +} +USES INT out3D_downsample_y "" +{ + : :: +} +USES INT out3D_downsample_z "" +{ + : :: +} + +################################### +# Checkpointing/recovery parameters +################################### +USES BOOLEAN checkpoint_ID "" +{ +} +USES BOOLEAN checkpoint_keep_all "" +{ +} +USES KEYWORD recover "" +{ +} +USES INT checkpoint_every "" +{ + : :: +} +USES INT checkpoint_keep "" +{ + : :: +} +USES STRING checkpoint_file "" +{ +} +USES STRING checkpoint_ID_file "" +{ +} +USES STRING recover_file "" +{ +} +USES STRING checkpoint_dir "" +{ +} +USES STRING recovery_dir "" +{ +} diff --git a/schedule.ccl b/schedule.ccl new file mode 100644 index 0000000..03fd80f --- /dev/null +++ b/schedule.ccl @@ -0,0 +1,46 @@ +# Schedule definitions for thorn IOHDF5 +# $Header$ + +######################################################################## +### register IOHDF5 routines +######################################################################## +schedule IOHDF5_Startup at STARTUP after (IOUtil_Startup Driver_Startup) +{ + LANG:C +} "IOHDF5 startup routine" + + +######################################################################## +### register checkpointing routines +######################################################################## +if (checkpoint && checkpoint_ID) +{ + schedule IOHDF5_InitialDataDumpGH at CCTK_CPINITIAL + { + LANG:C + } "Initial data checkpoint routine" +} + +if (checkpoint && checkpoint_every > 0) +{ + schedule IOHDF5_ConditionallyDumpGH at CCTK_CHECKPOINT + { + LANG:C + } "Regular checkpoint routine" +} + +#if (checkpoint) +#{ +# schedule IOHDF5_TerminationDumpGH at CCTK_TERMINATE BEFORE Driver_Terminate +# { +# LANG:C +# } "Termination checkpoint routine" +#} + +if (! CCTK_Equals (recover, "no") && *recover_file) +{ + schedule IOHDF5_RecoverParameters at CCTK_RECOVER_PARAMETERS + { + LANG:C + } "Parameter recovery routine" +} diff --git a/src/ChooseOutput.c b/src/ChooseOutput.c new file mode 100644 index 0000000..39ac818 --- /dev/null +++ b/src/ChooseOutput.c @@ -0,0 +1,145 @@ +/*@@ + @file ChooseOutput.c + @author Gabrielle Allen + @date July 6 2000 + @desc + Choose what data to write for different IO methods in IOHDF5 + @enddesc + + @history + @hauthor + @hdesc + @version $Header$ +@@*/ + +#include <stdlib.h> + +#include "cctk.h" +#include "cctk_Arguments.h" +#include "cctk_Parameters.h" +#include "CactusBase/IOUtil/src/ioGH.h" +#include "ioFlexGH.h" + +static char *rcsid = "$Header$"; +CCTK_FILEVERSION(CactusPUGHIO_IOHDF5_ChooseOutput_c) + + +/*@@ + @routine IOFlexIO_Choose2D + @author Gabrielle Allen + @date July 6 2000 + @desc + Use parameters to choose the 2D planes through the output data. + @enddesc + + @history + @hauthor + @hdesc + @version $Header$ +@@*/ + +void IOFlexIO_Choose2D(CCTK_ARGUMENTS) +{ + DECLARE_CCTK_PARAMETERS + + int i; + int maxdim; + flexioGH *myGH; /* FlexIO extension handle */ + int *origin_index; /* Specify output planes by indices */ + CCTK_REAL *origin_phys; /* Specify output planes by coordinates */ + + myGH = (flexioGH *) cctkGH->extensions + [CCTK_GHExtensionHandle ("IOFlexIO")]; + + /* Set up lines to be output */ + maxdim = CCTK_MaxDim(); + origin_phys = (CCTK_REAL *)calloc(maxdim,sizeof(CCTK_REAL )); + origin_index = (int *)calloc(maxdim,sizeof(int )); + + /* Set parameters using + 1. Indices from IOFlexIO + 2. Indices from IOUtil + 3. Coords from IOFlexIO + 4. Coords from IOUtil + */ + + if (CCTK_ParameterQueryTimesSet("out2D_yzplane_xi","IOFlexIO")>0) + { + origin_index[0] = out2D_yzplane_xi; + } + else if (CCTK_ParameterQueryTimesSet("out_yzplane_xi","IOUtil")>0) + { + origin_index[0] = out_yzplane_xi; + } + else + { + origin_index[0] = -1; + if (CCTK_ParameterQueryTimesSet("out2D_yzplane_x","IOFlexIO")>0) + { + origin_phys[0] = out2D_yzplane_x; + } + else + { + origin_phys[0] = out_yzplane_x; + } + } + + if (CCTK_ParameterQueryTimesSet("out2D_xzplane_yi","IOFlexIO")>0) + { + origin_index[1] = out2D_xzplane_yi; + } + else if (CCTK_ParameterQueryTimesSet("out_xzplane_yi","IOUtil")>0) + { + origin_index[1] = out_xzplane_yi; + } + else + { + origin_index[1] = -1; + if (CCTK_ParameterQueryTimesSet("out2D_xzplane_y","IOFlexIO")>0) + { + origin_phys[1] = out2D_xzplane_y; + } + else + { + origin_phys[1] = out_xzplane_y; + } + } + + if (CCTK_ParameterQueryTimesSet("out2D_xyplane_zi","IOFlexIO")>0) + { + origin_index[2] = out2D_xyplane_zi; + } + else if (CCTK_ParameterQueryTimesSet("out_xyplane_zi","IOUtil")>0) + { + origin_index[2] = out_xyplane_zi; + } + else + { + origin_index[2] = -1; + if (CCTK_ParameterQueryTimesSet("out2D_xyplane_z","IOFlexIO")>0) + { + origin_phys[2] = out2D_xyplane_z; + } + else + { + origin_phys[2] = out_xyplane_z; + } + } + + if (myGH) + { + for (i=1;i<=CCTK_MaxDim();i++) + { + IOUtil_2DPlanes (cctkGH, i, origin_index, + origin_phys, myGH->sp2xyz[i-1]); + } + } + else + { + CCTK_WARN(1,"IOFlexIO_Choose2D: FlexIO GH extension not found"); + } + + return; +} + + diff --git a/src/DumpGH.c b/src/DumpGH.c new file mode 100644 index 0000000..74d7729 --- /dev/null +++ b/src/DumpGH.c @@ -0,0 +1,494 @@ + /*@@ + @file DumpGH.c + @date Wed Jun 10 14:13:35 1998 + @author Paul Walker + @desc + DumpGH dumps an entire grid hierarchy (except for 1D arrays) to a + checkpoint file. This file also contains the different wrappers for + IOHDF5_DumpGH: in case of initial data, termination or regular checkpointing + @enddesc + @version $Id$ + @@*/ + + +#include <stdio.h> +#include <string.h> +#include <stdlib.h> + +#include "cctk.h" +#include "cctk_Parameters.h" +#include "CactusBase/IOUtil/src/ioGH.h" +#include "ioHDF5GH.h" + +/* the rcs ID and its dummy funtion to use it */ +static char *rcsid = "$Id$"; +CCTK_FILEVERSION(CactusPUGHIO_IOHDF5_DumpGH_c) + + +/* local function prototypes */ +void IOHDF5i_DumpParameters (cGH *GH, hid_t group); + + + /*@@ + @routine IOHDF5_ConditionallyDumpGH + @date Fri Aug 21 14:38:25 1998 + @author Gabrielle Allen + @desc + registered as CCTK_CHECKPOINT, checks if it's time for + checkpointing, sets the checkpoint type and calls IOHDF5_DumpGH + @enddesc + @calledby CCTK Scheduler + @history + + @endhistory + @var GH + @vdesc Pointer to CCTK grid hierarchy + @vtype cGH + @vio in + @endvar +@@*/ + +void IOHDF5_ConditionallyDumpGH (cGH *GH) +{ + DECLARE_CCTK_PARAMETERS + + if (checkpoint_every > 0 && GH->cctk_iteration % checkpoint_every == 0) { + if (verbose) { + CCTK_INFO("------------------------------------------------------------"); + CCTK_VInfo (CCTK_THORNSTRING, "Dumping periodic checkpoint file at " + "iteration %d", GH->cctk_iteration); + } + IOHDF5_DumpGH (GH, CP_EVOLUTION_DATA); + } +} + + + /*@@ + @routine IOHDF5_TerminationDumpGH + @date Fri Aug 21 14:40:21 1998 + @author Gabrielle Allen + @desc + This routine is registered as CCTK_TERMINATE, it checks if a + termination signal is raised (that is: global var cactus_terminate + set to TERMINATION_RAISED_BRDCAST) and checkpoints all available + grid hierarchies. + @enddesc + @calledby CCTK Scheduler + @history + + @endhistory + @var GH + @vdesc Pointer to CCTK grid hierarchy + @vtype cGH + @vio in + @endvar +@@*/ + +void IOHDF5_TerminationDumpGH (cGH *GH) +{ +/*** FIXME: no cactus_terminate flag anymore ? ***/ +#if 0 + if (cactus_terminate == TERMINATION_RAISED_BRDCAST) { + IOHDF5_DumpGH (GH, CP_EVOLUTION_DATA); + } +#else + CCTK_WARN (1, "IOHDF5_TerminationDumpGH() not yet implemented !"); +#endif +} + + /*@@ + @routine IOHDF5_InitialDataDumpGH + @date Fri Aug 21 14:46:28 1998 + @author Gerd Lanfermann + @desc + This routine dumps the initial data of the GHs, registered + as CCTK_CPINITIAL; sets 'called_from' flag to CP_INITIAL_DATA + and calls IOHDF5_DumpGH. + BoxInBox needs special treatment since for now. + @enddesc + @calledby CCTK Scheduler + @history + + @endhistory + @var GH + @vdesc Pointer to CCTK grid hierarchy + @vtype cGH + @vio in + @endvar +@@*/ + +void IOHDF5_InitialDataDumpGH (cGH *GH) +{ + +/*** FIXME ***/ +#if 0 +#ifdef THORN_BOXINBOX + /* This will go once we reorganize the CINitial calling structure! */ + /* or have a function to deregister a rfr routine */ + /* BoxinBox does its own: get maxlevel (once!) and do dumpGH for all GH*/ + /* when this is called by the highest box */ + if (Contains("boxinbox","yes")) { + int l; + cGH *helperGH; + static int maxlev = 0; + + if (maxlev == 0) while (GHbyLevel(0,maxlev+1,0)) maxlev++; + if (maxlev == 0) return; + + if (GH->level == maxlev) + for (l = 0; l <= maxlev; l++) { + CCTK_VInfo (CCTK_THORNSTRING, "Dumping BoxinBox level %d of %d", + l, maxlev); + IOHDF5_DumpGH (GHbyLevel (0, l, 0), CP_INITIAL_DATA); + } + return; + } +#endif +#endif + IOHDF5_DumpGH (GH, CP_INITIAL_DATA); +} + + + /*@@ + @routine IOHDF5i_DumpParameters + @date Thu Jan 20 2000 + @author Thomas Radke + @desc + Gets the parameters of all active implementations as a single string + and writes it as an attribute into a group of an already opened HDF5 file. + @enddesc + @calledby IOHDF5_Write3D, IOHDF5_DumpGH + @history + + @endhistory + @var GH + @vdesc Pointer to CCTK grid hierarchy + @vtype cGH + @vio in + @endvar + @var group + @vdesc the group where to dump the parameters to + @vtype hid_t + @vio in + @endvar +@@*/ +void IOHDF5i_DumpParameters (cGH *GH, hid_t group) +{ + ioHDF5GH *myGH; + char *parameters; + + + /* Get the parameter string buffer */ + parameters = IOUtil_GetAllParameters (GH); + + if (parameters) { + /* Get the handle for IOHDF5 extensions */ + myGH = (ioHDF5GH *) GH->extensions [CCTK_GHExtensionHandle ("IOHDF5")]; + + WRITE_ATTRIBUTE (GLOBAL_PARAMETERS, parameters, group, + myGH->scalarDataspace, 0, myGH->IOHDF5_STRING); + free (parameters); + } +} + + + /*@@ + @routine IOHDF5i_DumpGHExtensions + @date Thu Jan 20 2000 + @author Thomas Radke + @desc + Dumps the important variables from the grid hierarchy, PUGH, and IO + into a group of an already opened HDF5 file. + @enddesc + @calledby IOHDF5_Write3D, IOHDF5_DumpGH + @history + + @endhistory + @var GH + @vdesc Pointer to CCTK grid hierarchy + @vtype cGH + @vio in + @endvar + @var group + @vdesc the group where to dump the GH extensions to + @vtype hid_t + @vio in + @endvar +@@*/ +void IOHDF5i_DumpGHExtensions (cGH *GH, hid_t group) +{ + int attrValue, main_loop_index; + ioGH *ioUtilGH; + ioHDF5GH *myGH; + extern int CCTK_MainLoopIndex (void); + + /* Get the handles for IOUtil and IOHDF5 extensions */ + ioUtilGH = (ioGH *) GH->extensions [CCTK_GHExtensionHandle ("IO")]; + myGH = (ioHDF5GH *) GH->extensions [CCTK_GHExtensionHandle ("IOHDF5")]; + + main_loop_index = CCTK_MainLoopIndex (); + + attrValue = CCTK_nProcs (GH); + WRITE_ATTRIBUTE ("nprocs", &attrValue, group, + myGH->scalarDataspace, 0, H5T_NATIVE_INT); + WRITE_ATTRIBUTE ("ioproc_every", &ioUtilGH->ioproc_every, group, + myGH->scalarDataspace, 0, H5T_NATIVE_INT); + WRITE_ATTRIBUTE ("unchunked", &ioUtilGH->unchunked, group, + myGH->scalarDataspace, 0, H5T_NATIVE_INT); + WRITE_ATTRIBUTE ("cctk_time", &GH->cctk_time, group, + myGH->scalarDataspace, 0, IOHDF5_REAL); + WRITE_ATTRIBUTE ("cctk_iteration", &GH->cctk_iteration, group, + myGH->scalarDataspace, 0, H5T_NATIVE_INT); + WRITE_ATTRIBUTE ("main_loop_index", &main_loop_index, group, + myGH->scalarDataspace, 0, H5T_NATIVE_INT); + +} + + + /*@@ + @routine IOHDF5_DumpGH + @date Fri Aug 21 15:13:13 1998 + @author Paul Walker + @desc + The heart of checkpointing. Called by the different wrappers, this + routines get the appropriate filename by IOUtil_PrepareFilename() and + then dumps away using the dump routines from IO ... + @enddesc + @calledby IOHDF5_ConditionallyDumpGH, IOHDF5_TerminationDumpGH, + IOHDF5_InitialDataDumpGH + @history + @hdate Oct 4 1998 @hauthor Gabrielle Allen + @hdesc Removed code which checked and reset unchunked, assume that this + is done when the variable is first set. + @hdate Nov 4 1998 @hauthor Gabrielle Allen + @hdesc Do a forced synchronization before checkpointing + @hdate Apr 16 1999 @hauthor Thomas Radke + @hdesc Removed forced sync before checkpointing (just do a normal sync) + Introduced a ring buffer for keeping last <checkpoint_keep> files + @endhistory + @var GH + @vdesc Pointer to CCTK grid hierarchy + @vtype cGH + @vio in + @endvar + @var called_from + @vdesc flag indicating where this routine was called from + (either CHECKPOINT_ID, CHECKPOINT, or filereader) + @vtype int + @vio in + @endvar +@@*/ + +void IOHDF5_DumpGH (cGH *GH, int called_from) +{ + DECLARE_CCTK_PARAMETERS + char dumpfname [1024], tmpfname [1024]; + hid_t iof; + int index,idim; + int timelevel, current_timelevel; + int *old_downsample; + int old_out_single; + ioGH *ioUtilGH; + ioHDF5GH *myGH; + ioHDF5Geo_t geo; + static char **dumpfnames = NULL; /* dump filename ring buffer */ + static int findex = 0; /* index into ring buffer */ + + + /* Get the handles for IOUtil and IOHDF5 extensions */ + if ((index = CCTK_GHExtensionHandle ("IO")) < 0) + return; + ioUtilGH = (ioGH *) GH->extensions [index]; + if ((index = CCTK_GHExtensionHandle ("IOHDF5")) < 0) + return; + myGH = (ioHDF5GH *) GH->extensions [index]; + + /* allocate the ring buffer for filenames */ + if (! dumpfnames) + dumpfnames = (char **) calloc (checkpoint_keep, sizeof (char *)); + + /* disable downsampling after saving original downsampling params */ + old_downsample = (int *) malloc (CCTK_MaxDim () * sizeof (int)); + for (index = 0; index < CCTK_MaxDim (); index++) { + old_downsample [index] = ioUtilGH->downsample [index]; + ioUtilGH->downsample [index] = 1; + } + + /* disable output in single precision */ + old_out_single = ioUtilGH->out_single; + ioUtilGH->out_single = 0; + + /* start the total timer */ + if (myGH->print_timing_info) + CCTK_TimerStartI (myGH->timers[2]); + + /* sync all groups */ + for (index = 0; index < CCTK_NumGroups (); index++) + if (CCTK_IsImplementationActive (CCTK_ImpFromVarI ( + CCTK_FirstVarIndexI (index)))) + CCTK_SyncGroupI (GH, index); + + /* get the base filename ... */ + IOUtil_PrepareFilename (GH, NULL, dumpfname, called_from, + CCTK_MyProc (GH) / ioUtilGH->ioproc_every, + ioUtilGH->unchunked); + + /* ... and append the extension */ + sprintf (tmpfname, "%s.tmp.h5", dumpfname); + sprintf (dumpfname, "%s.h5", dumpfname); + + /* Now open the file */ + if (CCTK_MyProc (GH) == ioUtilGH->ioproc) { + if (verbose) + CCTK_VInfo (CCTK_THORNSTRING, "Creating checkpoint file '%s'", tmpfname); + + iof = H5Fcreate (tmpfname, H5F_ACC_TRUNC, H5P_DEFAULT, H5P_DEFAULT); + if (iof < 0) { + CCTK_VWarn (1, __LINE__, __FILE__, CCTK_THORNSTRING, + "Can't open checkpoint file '%s'. Checkpointing is skipped", + tmpfname); + return; + } + } else + iof = -1; + + /* Great; Now start dumping away! */ + + /* start timer for dumping parameters */ + if (myGH->print_timing_info) { + CCTK_TimerResetI (myGH->timers[0]); + CCTK_TimerStartI (myGH->timers[0]); + } + + if (iof >= 0) { + hid_t group; + + /* all GH extension variables and parameter stuff which is not + specific to any dataset goes into dedicated groups */ + if (out3D_parameters) { + if (verbose) + CCTK_INFO ("Dumping Parameters ..."); + + CACTUS_IOHDF5_ERROR (group = H5Gcreate (iof, GLOBAL_PARAMETERS_GROUP, 0)); + IOHDF5i_DumpParameters (GH, group); + CACTUS_IOHDF5_ERROR (H5Gclose (group)); + } + + if (verbose) + CCTK_INFO ("Dumping GH extensions ..."); + + CACTUS_IOHDF5_ERROR (group = H5Gcreate (iof, GHEXTENSIONS_GROUP, 0)); + IOHDF5i_DumpGHExtensions (GH, group); + CACTUS_IOHDF5_ERROR (H5Gclose (group)); + } + + /* stop parameter timer and start timer for dumping datasets */ + if (myGH->print_timing_info) { + CCTK_TimerStopI (myGH->timers[0]); + CCTK_TimerResetI (myGH->timers[1]); + CCTK_TimerStartI (myGH->timers[1]); + } + + /* Set the geometry variables manually, do not use the + geometry structures set by parameters; vdim,sdim are set below */ + for (idim=0; idim<IOHDF5_MAXDIM;idim++) { + geo.direction[idim] = idim; + geo.slab_start[idim] = 0; + geo.length[idim] = -1; + geo.downs[idim] = 1; + } + + if (verbose) + CCTK_INFO ("Dumping variables ..."); + + /* ... now the variables */ + for (index = 0; index < CCTK_NumVars (); index++) { + char oldFlag; + const char *thorn, *impl = CCTK_ImpFromVarI (index); + + /* find out the thorn implementing variable with index */ + thorn = CCTK_ImplementationThorn (impl); + if (! thorn) + thorn = impl; + + /* let only variables pass which belong to an active thorn and + have storage assigned */ + if (! CCTK_IsThornActive (thorn) || + ! CCTK_QueryGroupStorageI (GH, CCTK_GroupIndexFromVarI (index))) + continue; + + if (verbose && iof >= 0) + CCTK_VInfo (CCTK_THORNSTRING, " %s", CCTK_VarName (index)); + + /* get the current timelevel */ + current_timelevel = CCTK_NumTimeLevelsFromVarI (index) - 1; + if (current_timelevel > 0) + current_timelevel--; + + /* Set dimension information based on the variable */ + geo.vdim = CCTK_GroupDimFromVarI(index); + geo.sdim = geo.vdim; + + /* reset this flag because checkpoint files are always created anew */ + oldFlag = myGH->checkForExistingObjects [index]; + myGH->checkForExistingObjects [index] = 0; + + /* now dump all timelevels up to the current */ + for (timelevel = 0; timelevel <= current_timelevel; timelevel++) + IOHDF5_DumpVar (GH, index, timelevel, &geo, iof); + + /* restore the original value of the flag */ + myGH->checkForExistingObjects [index] = oldFlag; + + } /* end of loop over all variables */ + + /* stop timer for dumping datasets */ + if (myGH->print_timing_info) + CCTK_TimerStopI (myGH->timers[1]); + + if (iof >= 0) + CACTUS_IOHDF5_ERROR (H5Fclose (iof)); + + if (CCTK_MyProc (GH) == ioUtilGH->ioproc) { + + /* rename successfully created temporary file + and overwrite the old checkpoint file */ + if (verbose) + CCTK_VInfo (CCTK_THORNSTRING, "Closing and renaming checkpoint file " + "into '%s'", dumpfname); + + if (rename (tmpfname, dumpfname)) + CCTK_VWarn (0, __LINE__, __FILE__, CCTK_THORNSTRING, + "Could not rename temporary checkpoint file '%s' to '%s'", + tmpfname, dumpfname); + + /* delete the oldest checkpoint file if checkpoint_keep_all isn't set + and put the new filename into the ring buffer */ + if (dumpfnames [findex]) { + if (! checkpoint_keep_all) + remove (dumpfnames [findex]); + free (dumpfnames [findex]); + } + dumpfnames [findex] = strdup (dumpfname); + findex = (findex + 1) % checkpoint_keep; + } + + /* restore output precision flag */ + ioUtilGH->out_single = old_out_single; + + /* restore original downsampling params */ + memcpy (ioUtilGH->downsample, old_downsample, CCTK_MaxDim () * sizeof (int)); + free (old_downsample); + + /* stop total checkpoint timer and print timing info */ + if (myGH->print_timing_info) { + const char *timer_descriptions [3] = {"Time to dump parameters: ", + "Time to dump datasets: ", + "Total time to checkpoint:"}; + + + CCTK_TimerStopI (myGH->timers[2]); + IOUtil_PrintTimings ("Timing information for checkpointing in IOHDF5:", 3, + myGH->timers, timer_descriptions); + } +} diff --git a/src/DumpVar.c b/src/DumpVar.c new file mode 100644 index 0000000..dd616af --- /dev/null +++ b/src/DumpVar.c @@ -0,0 +1,1070 @@ +/*@@ + @file DumpVar.c + @date Fri May 21 1999 + @author Thomas Radke + @desc + Do the actual writing of a variable, for output or for checkpointing. + @enddesc + @history + @hauthor Thomas Radke @hdate May 21 1999 + @hdesc Just copied from thorn FlexIO. + @hendhistory + @@*/ + + +#include "cctk.h" +#include "cctk_Parameters.h" +#include "CactusPUGH/PUGH/src/include/pugh.h" +#include "CactusPUGH/PUGHSlab/src/PUGHSlab.h" +#include "CactusBase/IOUtil/src/ioGH.h" +#include "ioHDF5GH.h" + +#include <stdio.h> +#include <stdlib.h> +#ifdef SGI +#include <time.h> +#endif + +#define IOTAGBASE 20000 /* This may break on more than 2000 processors */ + +#define IOHDF5_DEBUG + +/* info structure describing how to dump a given CCTK variable type */ +typedef struct { + int iohdf5_type; /* the HDF5 type to use */ + int element_size; /* size of a single data element */ +#ifdef CCTK_MPI + MPI_Datatype mpi_type; /* the data type for MPI communication */ +#endif +} dumpInfo_t; + +static char *char_time_date = NULL; + +/* local function prototypes */ +static void IOHDF5_DumpGS (cGH *GH, int index, int timelevel, hid_t iof, + int iohdf5_type); + +static int IOHDF5_DumpGA (cGH *GH, int index, int timelevel, ioHDF5Geo_t *slab_geo, + hid_t iof, dumpInfo_t *info); + +static int IOHDF5_getDumpData_ND (cGH *GH, int index, int timelevel, ioHDF5Geo_t *slab_geo, + void **outme, int *free_outme, + CCTK_INT *geom, int element_size); + +static void IOHDF5_procDump (cGH *GH, int index, int timelevel, ioHDF5Geo_t *slab_geo, + void *outme, CCTK_INT *geom, int proc, + int iohdf5_type, hid_t iof); + +static void IOHDF5_AddCommonAttributes (cGH *GH, int index, int timelevel, + CCTK_INT *global_shape, + hid_t dataset); + +static void IOHDF5_AddCommonAttributes_ND (cGH *GH, int index, ioHDF5Geo_t *slab_geo, + CCTK_INT *global_shape, hid_t dataset); + +#ifdef CCTK_MPI +#ifdef HAVE_PARALLEL +static void IOHDF5_collectiveDump (cGH *GH, int index, int timelevel, ioHDF5Geo_t *slab_geo, + void *outme, CCTK_INT *geom, + int hdf5io_type, hid_t iof); +#endif /* HAVE_PARALLEL */ +#endif /* MPI */ + + + +/*@@ + @routine IOHDF5_DumpVar + @date 16 Apr 1999 + @author Thomas Radke + @desc + Generic dump routine, just calls the appropriate dump routine + @enddesc + @calledby IOHDF5_DumpGH IOHDF5_Write3D + @history + @endhistory + @var GH + @vdesc Pointer to CCTK grid hierarchy + @vtype cGH + @vio in + @endvar + @var index + @vdesc global index of the variable to be dumped + @vtype int + @vio in + @endvar + @var timelevel + @vdesc the timelevel to store + @vtype int + @vio in + @endvar + @var iof + @vdesc the HDF5 file handle + @vtype hid_t + @vio in + @endvar +@@*/ +int IOHDF5_DumpVar (cGH *GH, int index, int timelevel, ioHDF5Geo_t *geo, hid_t iof) +{ + pGH *pughGH; + ioHDF5GH *h5GH; + ioGH *ioUtilGH; + int variable_type; + dumpInfo_t info; + int retval = 0; + + /* Get the handle for PUGH, IOUtil, and IOHDF5 extensions */ + pughGH = PUGH_pGH (GH); + ioUtilGH = (ioGH *) GH->extensions [CCTK_GHExtensionHandle ("IO")]; + h5GH = (ioHDF5GH *) GH->extensions [CCTK_GHExtensionHandle ("IOHDF5")]; + + variable_type = CCTK_VarTypeI (index); + + switch (variable_type) { + case CCTK_VARIABLE_CHAR: + info.iohdf5_type = IOHDF5_CHAR; + info.element_size = sizeof (CCTK_CHAR); +#ifdef CCTK_MPI + info.mpi_type = PUGH_MPI_CHAR; +#endif + break; + case CCTK_VARIABLE_INT: + info.iohdf5_type = IOHDF5_INT; + info.element_size = sizeof (CCTK_INT); +#ifdef CCTK_MPI + info.mpi_type = PUGH_MPI_INT; +#endif + break; + case CCTK_VARIABLE_REAL: + info.iohdf5_type = ioUtilGH->out_single ? IOHDF5_REAL4 : IOHDF5_REAL; + info.element_size = ioUtilGH->out_single ? + sizeof (CCTK_REAL4) : sizeof (CCTK_REAL); +#ifdef CCTK_MPI + info.mpi_type = ioUtilGH->out_single ? PUGH_MPI_REAL4 : PUGH_MPI_REAL; +#endif + break; + case CCTK_VARIABLE_COMPLEX: + info.iohdf5_type = h5GH->IOHDF5_COMPLEX; + info.element_size = sizeof (CCTK_COMPLEX); +#ifdef CCTK_MPI + info.mpi_type = pughGH->PUGH_mpi_complex; +#endif + break; + default: + CCTK_VWarn (1, __LINE__, __FILE__, CCTK_THORNSTRING, + "Unsupported variable type %d", variable_type); + return(-1); + } + + switch (CCTK_GroupTypeFromVarI (index)) { + case GROUP_SCALAR: + IOHDF5_DumpGS (GH, index, timelevel, iof, info.iohdf5_type); + break; + case GROUP_ARRAY: + case GROUP_GF: + retval = IOHDF5_DumpGA (GH, index, timelevel, geo, iof, &info); + break; + default: + CCTK_VWarn (1, __LINE__, __FILE__, CCTK_THORNSTRING, + "Invalid group type %d in IOHDF5_DumpVar", + CCTK_GroupTypeFromVarI (index)); + retval = -1; + } + return(retval); +} + + +/*@@ + @routine IOHDF5_DumpGS + @date May 21 1999 + @author Thomas Radke + @desc Dumps a SCALAR type variable into a HDF5 file + @enddesc + @calledby IOHDF5_DumpVar + @var GH + @vdesc Pointer to CCTK grid hierarchy + @vtype cGH + @vio in + @endvar + @var index + @vdesc global index of the variable to be dumped + @vtype int + @vio in + @endvar + @var timelevel + @vdesc the timelevel to store + @vtype int + @vio in + @endvar + @var iof + @vdesc the HDF5 file to dump to + @vtype hid_t + @vio in + @endvar + @var iohdf5_type + @vdesc the HDF5 datatype + @vtype int + @vio in + @endvar + @history + @endhistory + +@@*/ +static void IOHDF5_DumpGS (cGH *GH, int index, int timelevel, hid_t iof, + int iohdf5_type) +{ + CCTK_INT global_shape [1] = {0}; + ioGH *ioUtilGH; + ioHDF5GH *h5GH; + hid_t dataset, dataspace; + char *name, *datasetname; + + /* Get the handles for IOHDF5 and IOUtil extensions */ + ioUtilGH = (ioGH *) GH->extensions [CCTK_GHExtensionHandle ("IO")]; + h5GH = (ioHDF5GH *) GH->extensions [CCTK_GHExtensionHandle ("IOHDF5")]; + + if (CCTK_MyProc (GH) != ioUtilGH->ioproc) + return; + + CACTUS_IOHDF5_ERROR (dataspace = H5Screate (H5S_SCALAR)); + name = CCTK_FullName (index); + datasetname = (char *) malloc (strlen (name) + 30); + sprintf (datasetname, "%s@%d@%d", name, GH->cctk_iteration, timelevel); + free (name); + + /* if restart from recovery delete an already existing dataset + so that it can be created as anew */ + if (h5GH->checkForExistingObjects [index]) { + CACTUS_IOHDF5_ERROR (H5Eset_auto (NULL, NULL)); + H5Gunlink (iof, datasetname); + CACTUS_IOHDF5_ERROR(H5Eset_auto(h5GH->printErrorFn, h5GH->printErrorFnArg)); + } + CACTUS_IOHDF5_ERROR (dataset = H5Dcreate (iof, datasetname, + iohdf5_type, dataspace, H5P_DEFAULT)); + free (datasetname); + CACTUS_IOHDF5_ERROR (H5Dwrite (dataset, iohdf5_type, H5S_ALL, H5S_ALL, + H5P_DEFAULT, + CCTK_VarDataPtrI (GH, timelevel, index))); + + IOHDF5_AddCommonAttributes (GH, index, timelevel, global_shape, dataset); + + CACTUS_IOHDF5_ERROR (H5Dclose (dataset)); + CACTUS_IOHDF5_ERROR (H5Sclose (dataspace)); +} + + +/*@@ + @routine IOHDF5_DumpGA + @date May 21 1999 + @author Paul Walker + @desc Dumps a GA type variable into a HDF5 file + @enddesc + @calledby IOHDF5_DumpVar + @var GH + @vdesc Pointer to CCTK grid hierarchy + @vtype cGH + @vio in + @endvar + @var index + @vdesc global index of the variable to be dumped + @vtype int + @vio in + @endvar + @var timelevel + @vdesc the timelevel to store + @vtype int + @vio in + @endvar + @var iof + @vdesc the HDF5 file to dump to + @vtype hid_t + @vio in + @endvar + @var info + @vdesc info structure describing + - the HDF5 datatype to store + - the size of an element of variable + - the MPI datatype to communicate + @vtype dumpInfo_t + @vio in + @endvar + + @history + @endhistory + +@@*/ +static int IOHDF5_DumpGA (cGH *GH, int index, int timelevel, ioHDF5Geo_t *slab_geo, + hid_t iof, dumpInfo_t *info) +{ + DECLARE_CCTK_PARAMETERS + ioGH *ioUtilGH; + pGH *pughGH; + int myproc; + int nprocs; + int dim; + CCTK_INT *geom; /* Lower bounds, sizes and global shape of data */ + void *outme; /* The data pointer to dump ... */ + int free_outme; /* and whether it needs freeing */ + int retval=0; +#ifdef CCTK_MPI + int i, j; + MPI_Status ms; +#endif + + /* Get the handles for PUGH and IOUtil extensions */ + pughGH = PUGH_pGH (GH); + ioUtilGH = (ioGH *) GH->extensions [CCTK_GHExtensionHandle ("IO")]; + + dim = slab_geo->vdim; + + /* allocate the geometry buffer */ + geom = (CCTK_INT *) malloc (3*dim * sizeof (CCTK_INT)); + + /* Get the pointer to the data we want to output (includes downsampling) */ + if (IOHDF5_getDumpData_ND(GH, index, timelevel, slab_geo, + &outme, &free_outme, geom, + info->element_size)<0) + { + return(-1); + } + + myproc = CCTK_MyProc (GH); + nprocs = CCTK_nProcs (GH); + +#ifdef CCTK_MPI +#ifdef HAVE_PARALLEL + if (ioUtilGH->unchunked) { + IOHDF5_collectiveDump (GH, index, timelevel, slab_geo, + outme, geom, + info->iohdf5_type, iof); + if (free_outme) + free (outme); + free (geom); + return; + } +#endif +#endif + + /* Dump data held on IO processor */ + if (myproc == ioUtilGH->ioproc) + IOHDF5_procDump (GH, index, timelevel, slab_geo, + outme, geom, myproc, + info->iohdf5_type,iof); + +#ifdef CCTK_MPI + if (myproc == ioUtilGH->ioproc) { + + /* Dump data from all other processors */ + for (i = myproc+1; i < myproc+ioUtilGH->ioproc_every && i < nprocs; i++) { + + /* Allocate temp to the right size (based on GH->ub [i] et..) */ + void *tmpd; + int incoming; + + /* receive geometry */ + CACTUS_MPI_ERROR (MPI_Recv (geom, 3*dim, PUGH_MPI_INT, i, 2*i+IOTAGBASE+1, + pughGH->PUGH_COMM_WORLD, &ms)); + + incoming = geom [dim]; + for (j = 1; j < dim; j++) + incoming *= geom [dim + j]; + + /* receive data */ + tmpd = malloc (incoming * info->element_size); + CACTUS_MPI_ERROR (MPI_Recv (tmpd, incoming, info->mpi_type, i, + 2*i+IOTAGBASE, pughGH->PUGH_COMM_WORLD, &ms)); + + IOHDF5_procDump (GH, index, timelevel, slab_geo, + tmpd, geom, i, + info->iohdf5_type, iof); + free (tmpd); + + } /* End loop over processors */ + + } else { + + /* Send the geometry and data from the non-IO processors + * to the IO processors + */ + + int outgoing = geom [dim]; + + for (i = 1; i < dim; i++) + outgoing *= geom [dim + i]; + + /* send geometry */ + CACTUS_MPI_ERROR (MPI_Send (geom, 3*dim, PUGH_MPI_INT, ioUtilGH->ioproc, + 2*myproc+IOTAGBASE+1, pughGH->PUGH_COMM_WORLD)); + /* send data */ + CACTUS_MPI_ERROR (MPI_Send (outme, outgoing, info->mpi_type, ioUtilGH->ioproc, + 2*myproc+IOTAGBASE, pughGH->PUGH_COMM_WORLD)); + +#ifdef IOHDF5_DEBUG + printf ("Processor %d sent %d data points\n", myproc, outgoing); +#endif + } + + CCTK_Barrier (GH); +#endif + + if (free_outme) + free (outme); + + free (geom); + return(retval); +} + + +/***************************** local functions ****************************/ + +/*@@ + @routine IOHDF5_AddCommonAttributes + @date May 21 1999 + @author Thomas Radke + @desc + Add "Common" attributes, these are: + <ul> + <li> the variable's groupname + <li> the grouptype + <li> number of timelevels + <li> the current date + Note that the datestamp should be turned of if you are byte + comparing two output files. + <li> simulation time + <li> origin + <li> bounding box + <li> gridspacings (both downsampled and evolution) + <li> global grid size + </ul> + @enddesc + @calledby IOHDF5_DumpGS, IOHDF5_collectiveDump, IOHDF5_procDump + @history + @endhistory + @var GH + @vdesc Pointer to CCTK grid hierarchy + @vtype cGH + @vio in + @endvar + @var index + @vdesc global index of the variable to be dumped + @vtype int + @vio in + @endvar + @var timelevel + @vdesc the timelevel to store + @vtype int + @vio in + @endvar + @var global_shape + @vdesc the global shape of the dataset + @vtype CCTK_INT [dim] + @vio in + @endvar + @var dataset + @vdesc the dataset handle where the attributes should be attached to + @vtype hid_t + @vio in + @endvar +@@*/ +static void IOHDF5_AddCommonAttributes (cGH *GH, int index, int timelevel, + CCTK_INT *global_shape, hid_t dataset) +{ + DECLARE_CCTK_PARAMETERS + int vdim; + CCTK_INT intAttr; + CCTK_REAL dummy; + CCTK_REAL realAttr [6]; + char *groupname; + ioGH *ioUtilGH; + ioHDF5GH *h5GH; + + /* Get the handles for IOUtil and IOHDF5 extensions */ + ioUtilGH = (ioGH *) GH->extensions [CCTK_GHExtensionHandle ("IO")]; + h5GH = (ioHDF5GH *) GH->extensions [CCTK_GHExtensionHandle ("IOHDF5")]; + + /* get the dimension of the variable */ + vdim = CCTK_GroupDimI (CCTK_GroupIndexFromVarI (index)); + + groupname = CCTK_GroupNameFromVarI (index); + WRITE_ATTRIBUTE ("groupname", groupname, + dataset, h5GH->scalarDataspace, 0, h5GH->IOHDF5_STRING); + free (groupname); + intAttr = CCTK_GroupTypeFromVarI (index); + WRITE_ATTRIBUTE ("grouptype", &intAttr, dataset, h5GH->scalarDataspace, 0, + IOHDF5_INT); + intAttr = CCTK_NumTimeLevelsFromVarI (index); + WRITE_ATTRIBUTE ("ntimelevels", &intAttr, dataset, h5GH->scalarDataspace, 0, + IOHDF5_INT); + + if (char_time_date && out3D_datestamp) + WRITE_ATTRIBUTE ("date", char_time_date, dataset, h5GH->scalarDataspace, 0, + h5GH->IOHDF5_STRING); + + WRITE_ATTRIBUTE ("time", &GH->cctk_time, dataset, h5GH->scalarDataspace, 0, + IOHDF5_REAL); + + /* NOTE: the attributes "origin", "min_ext", "max_ext", and "delta" + are always stored as 3-dimensional points */ + CCTK_CoordRange (GH, &realAttr [0], &dummy, -1, "x", "cart3d"); + CCTK_CoordRange (GH, &realAttr [1], &dummy, -1, "y", "cart3d"); + CCTK_CoordRange (GH, &realAttr [2], &dummy, -1, "z", "cart3d"); + WRITE_ATTRIBUTE ("origin", realAttr, dataset, h5GH->arrayDataspace, 3, + IOHDF5_REAL); + + CCTK_CoordRange (GH, &realAttr [0], &realAttr [3], -1, "x", "cart3d"); + CCTK_CoordRange (GH, &realAttr [1], &realAttr [4], -1, "y", "cart3d"); + CCTK_CoordRange (GH, &realAttr [2], &realAttr [5], -1, "z", "cart3d"); + WRITE_ATTRIBUTE ("min_ext", realAttr, dataset, h5GH->arrayDataspace, 3, + IOHDF5_REAL); + WRITE_ATTRIBUTE ("max_ext", realAttr + 3, dataset, h5GH->arrayDataspace, 3, + IOHDF5_REAL); + + + realAttr [0] = GH->cctk_delta_space [0] * ioUtilGH->downsample [0]; + realAttr [1] = GH->cctk_delta_space [1] * ioUtilGH->downsample [1]; + realAttr [2] = GH->cctk_delta_space [2] * ioUtilGH->downsample [2]; + WRITE_ATTRIBUTE ("delta", realAttr, dataset, h5GH->arrayDataspace, 3, + IOHDF5_REAL); + + if (ioUtilGH->downsample [0] > 1 || + ioUtilGH->downsample [1] > 1 || + ioUtilGH->downsample [2] > 1) + WRITE_ATTRIBUTE ("evolution_delta", GH->cctk_delta_space, dataset, + h5GH->arrayDataspace, 3, IOHDF5_REAL); + + WRITE_ATTRIBUTE ("global_size", global_shape, dataset, h5GH->arrayDataspace, + vdim, IOHDF5_INT); + +} + +static void IOHDF5_AddCommonAttributes_ND (cGH *GH, int index, ioHDF5Geo_t *geo, + CCTK_INT *global_shape, hid_t dataset) +{ + DECLARE_CCTK_PARAMETERS + CCTK_INT intAttr; + CCTK_REAL dummy; + CCTK_REAL realAttr [6]; + char *groupname; + + ioGH *ioUtilGH; + ioHDF5GH *h5GH; + + int idim; + + /* Get the handles for IOUtil and IOHDF5 extensions */ + ioUtilGH = (ioGH *) GH->extensions [CCTK_GHExtensionHandle ("IO")]; + h5GH = (ioHDF5GH *) GH->extensions [CCTK_GHExtensionHandle ("IOHDF5")]; + + /* VARIABLE INFO */ + groupname = CCTK_GroupNameFromVarI (index); + WRITE_ATTRIBUTE ("groupname", groupname, + dataset, h5GH->scalarDataspace, 0, h5GH->IOHDF5_STRING); + free (groupname); + intAttr = CCTK_GroupTypeFromVarI (index); + WRITE_ATTRIBUTE ("grouptype", &intAttr, dataset, h5GH->scalarDataspace, 0, + IOHDF5_INT); + + /* TIMLEVELS (grid) */ + intAttr = CCTK_NumTimeLevelsFromVarI (index); + WRITE_ATTRIBUTE ("ntimelevels", &intAttr, dataset, h5GH->scalarDataspace, 0, + IOHDF5_INT); + + /* DATE */ + if (char_time_date && out3D_datestamp) + WRITE_ATTRIBUTE ("date", char_time_date, dataset, h5GH->scalarDataspace, 0, + h5GH->IOHDF5_STRING); + + /* TIME (grid) */ + WRITE_ATTRIBUTE ("time", &GH->cctk_time, dataset, h5GH->scalarDataspace, 0, + IOHDF5_REAL); + + /* ORIGIN (grid) */ + CCTK_CoordRange (GH, &realAttr [0], &dummy, -1, "x", "cart3d"); + CCTK_CoordRange (GH, &realAttr [1], &dummy, -1, "y", "cart3d"); + CCTK_CoordRange (GH, &realAttr [2], &dummy, -1, "z", "cart3d"); + WRITE_ATTRIBUTE ("origin", realAttr, dataset, h5GH->arrayDataspace, geo->vdim, + IOHDF5_REAL); + + /* ORIGIN (slab) - slab may be offset by (slab_start) grid points */ + for (idim=0;idim<geo->vdim;idim++) + realAttr[idim]=geo->slab_start[idim]*GH->cctk_delta_space[idim]; + WRITE_ATTRIBUTE ("origin_slab", realAttr, dataset, h5GH->arrayDataspace, geo->vdim, + IOHDF5_REAL); + + /* MIN/MAX extent (grid) */ + CCTK_CoordRange (GH, &realAttr [0], &realAttr [3], -1, "x", "cart3d"); + CCTK_CoordRange (GH, &realAttr [1], &realAttr [4], -1, "y", "cart3d"); + CCTK_CoordRange (GH, &realAttr [2], &realAttr [5], -1, "z", "cart3d"); + WRITE_ATTRIBUTE ("min_ext", realAttr, dataset, h5GH->arrayDataspace, geo->vdim, + IOHDF5_REAL); + WRITE_ATTRIBUTE ("max_ext", realAttr + 3, dataset, h5GH->arrayDataspace, geo->vdim, + IOHDF5_REAL); + + /* MIN/MAX extent (slab) - slab may be smaller than grid*/ + for (idim=0;idim<geo->vdim;idim++) { + realAttr[idim ]=realAttr[idim]+(geo->slab_start[idim]*GH->cctk_delta_space[idim]); + realAttr[idim+3]=realAttr[idim]+(geo->slab_start[idim]+geo->actlen[idim]-1)* + (GH->cctk_delta_space[idim]*geo->downs[idim]); + } + WRITE_ATTRIBUTE ("min_ext_slab", realAttr, dataset, h5GH->arrayDataspace, geo->sdim, + IOHDF5_REAL); + WRITE_ATTRIBUTE ("max_ext_slab", realAttr + 3, dataset, h5GH->arrayDataspace, geo->sdim, + IOHDF5_REAL); + + /* DELTA SPACING (grid) */ + realAttr [0] = GH->cctk_delta_space [0]; + realAttr [1] = GH->cctk_delta_space [1]; + realAttr [2] = GH->cctk_delta_space [2]; + WRITE_ATTRIBUTE ("delta", realAttr, dataset, h5GH->arrayDataspace, geo->vdim, + IOHDF5_REAL); + + /* DELTA SPACING (slab) */ + for (idim=0;idim<geo->sdim;idim++) + realAttr[idim]*=geo->downs[idim]; + WRITE_ATTRIBUTE ("delta_slab", realAttr, dataset, h5GH->arrayDataspace, geo->sdim, + IOHDF5_REAL); + + /* FIXME: WHAT IS THIS ? */ + if (ioUtilGH->downsample [0] > 1 || + ioUtilGH->downsample [1] > 1 || + ioUtilGH->downsample [2] > 1) + WRITE_ATTRIBUTE ("evolution_delta", GH->cctk_delta_space, dataset, + h5GH->arrayDataspace, 3, IOHDF5_REAL); + + /* GLOBAL SIZE (slab) */ + WRITE_ATTRIBUTE ("global_size_grid", GH->cctk_gsh, dataset, h5GH->arrayDataspace, + geo->vdim, IOHDF5_INT); + + /* GLOBAL SIZE (slab) */ + WRITE_ATTRIBUTE ("global_size", global_shape, dataset, h5GH->arrayDataspace, + geo->sdim, IOHDF5_INT); + + /* GLOBAL SIZE (slab) */ + WRITE_ATTRIBUTE ("global_size_slab", geo->actlen, dataset, h5GH->arrayDataspace, + geo->sdim, IOHDF5_INT); +} + + +static int IOHDF5_getDumpData_ND (cGH *GH, int index, int timelevel, ioHDF5Geo_t *slab_geo, + void **outme, int *free_outme, + CCTK_INT *geom, int element_size) +{ + ioGH *ioUtilGH; + pGExtras *extras=NULL; + + CCTK_REAL4 *single_ptr; + + int myproc, do_downsample; + int idim, sdim, vdim; /*iteration, slab dim, variable dim */ + int *hsizes,*hsizes_global,*hsizes_offset; /* geometry information */ + int sdir[3]; /* slab direction FIXME */ + int ip; + + void *data = CCTK_VarDataPtrI (GH, timelevel, index); + + + myproc = CCTK_MyProc (GH); + + /* get GH extension for IOUtil */ + ioUtilGH = (ioGH *) GH->extensions [CCTK_GHExtensionHandle ("IO")]; + + /* get the pointer to PUGH specific structures */ + extras = ((pGA***) PUGH_pGH(GH)->variables) [index][timelevel]->extras; + + /* Shortcuts */ + vdim = slab_geo->vdim; + sdim = slab_geo->sdim; + + /* Set downsample flag */ + do_downsample = 0; + for (idim = 0; idim < sdim; idim++) + do_downsample |= slab_geo->downs[idim] > 1; + + /* Simple case if no downsampling and sdim==vdim */ + if ((! do_downsample)&& (sdim==vdim)) + { + if (ioUtilGH->out_single) { + single_ptr = (CCTK_REAL4 *) malloc (extras->npoints*sizeof (CCTK_REAL4)); + for (ip = 0; ip < extras->npoints; ip++) + single_ptr [ip] = (CCTK_REAL4) ((CCTK_REAL *) data) [ip]; + + *outme = single_ptr; + *free_outme = 1; + } + else + { + *outme = data; + *free_outme = 0; + } + + for (idim = 0; idim < sdim; idim++) { + geom [idim + 0*sdim] = extras->lb [myproc][idim]; + geom [idim + 1*sdim] = extras->lnsize [idim]; + geom [idim + 2*sdim] = extras->nsize [idim]; + } + } + else /* Call Hyperslab else */ { + + /* allocate array to hold size information of slab */ + hsizes = (int*)malloc(sdim*sizeof(int)); + hsizes_global= (int*)malloc(sdim*sizeof(int)); + hsizes_offset= (int*)malloc(sdim*sizeof(int)); + + /* TEMPORARY FIX DUE TO INCONSISTENT DIR SPECIFICATION */ + /* direction vector of 1d line in 3d volume */ + if (sdim==1) { + sdir[0] = (slab_geo->direction[0]==0) ? 1:0; + sdir[1] = (slab_geo->direction[0]==1) ? 1:0; + sdir[2] = (slab_geo->direction[0]==2) ? 1:0; + } + /* norm vector of 2d surface in 3d volume */ + else if (sdim==2) + { + sdir[0] = ((slab_geo->direction[0]==1)&&(slab_geo->direction[1]==2)) ? 1:0; + sdir[1] = ((slab_geo->direction[0]==0)&&(slab_geo->direction[1]==2)) ? 1:0; + sdir[2] = ((slab_geo->direction[0]==0)&&(slab_geo->direction[1]==1)) ? 1:0; + } + /* spanning directions for 3d */ + else if (sdim==3) + { + sdir[0] = slab_geo->direction[0]; + sdir[1] = slab_geo->direction[1]; + sdir[2] = slab_geo->direction[2]; + } + + if (Hyperslab_GetLocalHyperslab (GH, index, timelevel, sdim, slab_geo->slab_start, + sdir, slab_geo->length, slab_geo->downs, outme, + hsizes, hsizes_global, hsizes_offset)< 0) { + char *fullname = CCTK_FullName (index); + CCTK_VWarn (1, __LINE__, __FILE__, CCTK_THORNSTRING, + "Failed to extract hyperslab for variable '%s'", fullname); + free (fullname); + *free_outme = 0; + return(-1); + + } + + *free_outme = 1; + + for (idim = 0; idim < sdim; idim++) { + geom [idim + 0*sdim] = hsizes_offset[idim]; + geom [idim + 1*sdim] = hsizes[idim]; + geom [idim + 2*sdim] = hsizes_global[idim]; + slab_geo->actlen[idim]= hsizes_global[idim]; + } + } +#ifdef IOHDF5_DEBUG + printf ("Lower bound: %d", (int) geom [0]); + for (idim = 1; idim < sdim; idim++) + printf (" %d", (int) geom [idim]); + printf ("\n"); + printf ("Chunk size : %d", (int) geom [1*sdim + 0]); + for (idim = 1; idim < sdim; idim++) + printf (" %d", (int) geom [1*sdim + idim]); + printf ("\n"); + printf ("Global size: %d", (int) geom [2*sdim + 0]); + for (idim = 1; idim < sdim; idim++) + printf (" %d", (int) geom [2*sdim + idim]); + printf ("\n"); + printf ("Downsampling: "); + for (idim = 0; idim < sdim; idim++) + printf (" %d", slab_geo->downs[idim]); + printf ("\n"); +#endif + + return(1); +} + + +/*@@ + @routine IOHDF5_procDump + @author Thomas Radke + @date May 21 1999 + @desc All IO processors dump the data of their group into the file, + either as one chunk per processor or unchunked + (depending on parameter "unchunked"). + @enddesc + @calledby IOHDF5_DumpGA + @history + @endhistory + @var GH + @vdesc Pointer to CCTK grid hierarchy + @vtype cGH + @vio in + @endvar + @var index + @vdesc global index of the variable to be dumped + @vtype int + @vio in + @endvar + @var timelevel + @vdesc the timelevel to store + @vtype int + @vio in + @endvar + @var outme + @vdesc pointer to the chunk to dump + @vtype void * + @vio in + @endvar + @var geom + @vdesc bounds and sizes of the output + @vtype CCTK_INT [3*dim] + @vio in + @vcomment geom [0*dim..1*dim-1]: lower bounds + geom [1*dim..2*dim-1]: (local) data sizes + geom [2*dim..3*dim-1]: global space + @endvar + @var proc + @vdesc the processor which's chunk is to be dumped + @vtype int + @vio in + @endvar + @var hdf5io_type + @vdesc the HDF5 datatype identifier + @vtype int + @vio in + @endvar + @var iof + @vdesc the HDF5 file handle + @vtype hid_t + @vio in + @endvar +@@*/ +static void IOHDF5_procDump (cGH *GH, int index, int timelevel, ioHDF5Geo_t *slab_geo, + void *outme, CCTK_INT *geom, int proc, + int iohdf5_type, hid_t iof) +{ + DECLARE_CCTK_PARAMETERS + int i, vdim,sdim, idim; + ioGH *ioUtilGH; + ioHDF5GH *h5GH; + hid_t group, dataset, memspace, filespace; + char *name, datasetname [128], chunkname [128]; + hssize_t *chunk_origin; + hsize_t *chunk_dims, *file_dims; + + + ioUtilGH = (ioGH *) GH->extensions [CCTK_GHExtensionHandle ("IO")]; + h5GH = (ioHDF5GH *) GH->extensions [CCTK_GHExtensionHandle ("IOHDF5")]; + + vdim = slab_geo->vdim; + sdim = slab_geo->sdim; + + chunk_origin = (hssize_t *) malloc (vdim * sizeof (hssize_t)); + chunk_dims = (hsize_t *) malloc (2*vdim * sizeof (hsize_t)); + file_dims = chunk_dims + vdim; + + /* HDF5 needs it in reverse order */ + for (i = 0; i < sdim; i++) { + chunk_origin [i] = geom [1*sdim - 1 - i]; + chunk_dims [i] = geom [2*sdim - 1 - i]; + file_dims [i] = geom [3*sdim - 1 - i]; + } + + /* build the unique dataset name: eg. WAVETOY::phi@v3s01@3@1 + volume (v) 3-dim ; slab (s) spanned by 0,1 (->2-dim) + at(@) iteration at(@) timelevel + */ + name = CCTK_FullName (index); + sprintf (datasetname, "%s@v%ds",name,vdim); + for (idim=0;idim<sdim;idim++) + sprintf (datasetname, "%s%d",datasetname, slab_geo->direction[idim]); + sprintf (datasetname, "%s@%d@%d",datasetname, GH->cctk_iteration, timelevel); + free (name); + + /* create the memspace according to chunk dims */ + CACTUS_IOHDF5_ERROR (memspace = H5Screate_simple (sdim, chunk_dims, NULL)); + + if (ioUtilGH->unchunked) { + + /* create the (global) filespace and set the hyperslab for the chunk */ + CACTUS_IOHDF5_ERROR (filespace = H5Screate_simple (sdim, file_dims, NULL)); + CACTUS_IOHDF5_ERROR (H5Sselect_hyperslab (filespace, H5S_SELECT_SET, + chunk_origin, NULL, chunk_dims, NULL)); + + /* the IO processor creates the dataset and adds the common attributes + when writing its own data, otherwise the dataset is reopened */ + if (CCTK_MyProc (GH) == proc) { + /* if restart from recovery delete an already existing dataset + so that it can be created as anew */ + if (h5GH->checkForExistingObjects [index]) { + CACTUS_IOHDF5_ERROR (H5Eset_auto (NULL, NULL)); + H5Gunlink (iof, datasetname); + H5Eset_auto (h5GH->printErrorFn, h5GH->printErrorFnArg); + } + CACTUS_IOHDF5_ERROR (dataset = H5Dcreate (iof, datasetname, + iohdf5_type, filespace, H5P_DEFAULT)); + IOHDF5_AddCommonAttributes_ND (GH, index, slab_geo, + &geom [2*sdim], dataset); + } else + CACTUS_IOHDF5_ERROR (dataset = H5Dopen (iof, datasetname)); + + /* write the data */ + CACTUS_IOHDF5_ERROR (H5Dwrite (dataset, iohdf5_type, memspace, + filespace, H5P_DEFAULT, outme)); + + /* and close the file dataspace */ + CACTUS_IOHDF5_ERROR (H5Sclose (filespace)); + + } else { + + /* the IO processor creates the chunk group and adds common attributes */ + if (CCTK_MyProc (GH) == proc) { + /* if restart from recovery delete an already existing group + so that it can be created as anew */ + if (h5GH->checkForExistingObjects [index]) { + CACTUS_IOHDF5_ERROR (H5Eset_auto (NULL, NULL)); + H5Gunlink (iof, datasetname); + H5Eset_auto (h5GH->printErrorFn, h5GH->printErrorFnArg); + } + CACTUS_IOHDF5_ERROR (group = H5Gcreate (iof, datasetname, 0)); + IOHDF5_AddCommonAttributes_ND (GH, index, slab_geo, + &geom [2*vdim], group); + CACTUS_IOHDF5_ERROR (H5Gclose (group)); + } + + /* now the chunk dataset for each processor is created within the group */ + sprintf (chunkname, "%s/chunk%d", datasetname, proc - CCTK_MyProc (GH)); + /* create the chunk dataset and dump the chunk data */ + CACTUS_IOHDF5_ERROR (dataset = H5Dcreate (iof, chunkname, + iohdf5_type, memspace, H5P_DEFAULT)); + CACTUS_IOHDF5_ERROR (H5Dwrite (dataset, iohdf5_type, H5S_ALL, + H5S_ALL, H5P_DEFAULT, outme)); + + /* add the "origin" attribute for the chunk */ + WRITE_ATTRIBUTE ("chunk_origin", geom, dataset, h5GH->arrayDataspace, vdim, + IOHDF5_INT); + } + + /* close the dataset and the memspace */ + CACTUS_IOHDF5_ERROR (H5Dclose (dataset)); + CACTUS_IOHDF5_ERROR (H5Sclose (memspace)); + + free (chunk_origin); + free (chunk_dims); +} + + +#ifdef CCTK_MPI +#ifdef HAVE_PARALLEL +/*@@ + @routine IOHDF5_collectiveDump + @author Thomas Radke + @date May 21 1999 + @desc All processors dump their data into an unchunked file. + @enddesc + @calledby IOHDF5_DumpGA + @history + @endhistory + @var GH + @vdesc Pointer to CCTK grid hierarchy + @vtype cGH + @vio in + @endvar + @var index + @vdesc global index of the variable to be dumped + @vtype int + @vio in + @endvar + @var timelevel + @vdesc the timelevel to store + @vtype int + @vio in + @endvar + @var outme + @vdesc pointer to the chunk to dump + @vtype void * + @vio in + @endvar + @var geom + @vdesc bounds and sizes of the output + @vtype CCTK_INT [3*dim] + @vio in + @vcomment geom [0*dim..1*dim-1]: lower bounds + geom [1*dim..2*dim-1]: (local) data sizes + geom [2*dim..3*dim-1]: global space + @endvar + @var hdf5io_type + @vdesc the HDF5 datatype identifier + @vtype int + @vio in + @endvar + @var iof + @vdesc the HDF5 file handle + @vtype hid_t + @vio in + @endvar +@@*/ +static void IOHDF5_collectiveDump (cGH *GH, int index, int timelevel, ioHDF5Geo_t *slab_geo, + void *outme, CCTK_INT *geom, + int hdf5io_type, hid_t iof) +{ + DECLARE_CCTK_PARAMETERS + int i, dim; + ioHDF5GH *h5GH; + hid_t dataset, memspace, filespace; + char *name, datasetname [128]; + hssize_t *chunk_origin; + hsize_t *chunk_dims, *file_dims; + + + h5GH = (ioHDF5GH *) GH->extensions [CCTK_GHExtensionHandle ("IOHDF5")]; + + /* get the dimension of the variable */ + dim = slab_geo->vdim; + + chunk_origin = (hssize_t *) malloc (dim * sizeof (hssize_t)); + chunk_dims = (hsize_t *) malloc (2*dim * sizeof (hsize_t)); + file_dims = chunk_dims + dim; + + /* HDF5 needs it in reverse order */ + for (i = 0; i < dim; i++) { + chunk_origin [i] = geom [1*dim - 1 - i]; + chunk_dims [i] = geom [2*dim - 1 - i]; + file_dims [i] = geom [3*dim - 1 - i]; + } + + /* build the unique dataset name */ + name = CCTK_FullName (index); + sprintf (datasetname, "%s@%d@%d", name, GH->cctk_iteration, timelevel); + free (name); + + /* create the memspace according to chunk dims */ + CACTUS_IOHDF5_ERROR (memspace = H5Screate_simple (dim, chunk_dims, NULL)); + + /* create the (global) filespace and set the hyperslab for the chunk */ + CACTUS_IOHDF5_ERROR (filespace = H5Screate_simple (dim, file_dims, NULL)); + CACTUS_IOHDF5_ERROR (H5Sselect_hyperslab (filespace, H5S_SELECT_SET, + chunk_origin, NULL, chunk_dims, NULL)); + + /* the IO processor creates the dataset and adds the common attributes + when writing its own data, otherwise the dataset is reopened */ + /* if restart from recovery delete an already existing group + so that it can be created as anew */ + if (h5GH->checkForExistingObjects [index]) { + CACTUS_IOHDF5_ERROR (H5Eset_auto (NULL, NULL)); + H5Gunlink (iof, datasetname); + H5Eset_auto (h5GH->printErrorFn, h5GH->printErrorFnArg); + } + CACTUS_IOHDF5_ERROR (dataset = H5Dcreate (iof, datasetname, + hdf5io_type, filespace, H5P_DEFAULT)); + if (CCTK_MyProc (GH) == 0) + IOHDF5_AddCommonAttributes_ND (GH, index, slab_geo, + &geom [2*dim], dataset); + + /* write the data */ + CACTUS_IOHDF5_ERROR (H5Dwrite (dataset, hdf5io_type, memspace, + filespace, H5P_DEFAULT, outme)); + + /* and close the file dataspace */ + CACTUS_IOHDF5_ERROR (H5Sclose (filespace)); + + /* close the dataset and the memspace */ + CACTUS_IOHDF5_ERROR (H5Dclose (dataset)); + CACTUS_IOHDF5_ERROR (H5Sclose (memspace)); + + free (chunk_origin); + free (chunk_dims); +} +#endif /* HAVE_PARALLEL */ +#endif /* MPI */ diff --git a/src/File.c b/src/File.c new file mode 100644 index 0000000..1f79218 --- /dev/null +++ b/src/File.c @@ -0,0 +1,204 @@ + +#include "cctk.h" +#include "cctk_Parameters.h" +#include "StoreNamedData.h" +#include "CactusPUGH/PUGH/src/include/pugh.h" +#include "CactusBase/IOUtil/src/ioGH.h" +#include "ioHDF5GH.h" + +#include <stdio.h> +#include <stdlib.h> +#include <assert.h> + +/*@@ + @routine IOHDF5_Get_filename_ND + @author Paul Walker + @date Feb 1997 + @desc Returns the filename for output + The names of all output files are stored in the filenames database. + The routine first searches in this database if there is already + registered a filename for alias. If so, this name is returned. + Otherwise it builds the filename from alias and stores it in the + database. + NOTE: Filenames are never stored if "out2D_septimefiles = true" + since they are used only once. + + Generalized for Hyperslab extraction (GL) + @enddesc + @var GH + @vdesc Pointer to CCTK GH + @vtype cGH + @vio in + @endvar + @var index + @vdesc index of variable to output + @vtype int + @vio in + @endvar + @var name + @vdesc alias name of variable to output + @vtype const char * + @vio in + @endvar + @var isNewFile + @vdesc flag indicating if the file is to be created + @vtype int + @vio out + @endvar +@@*/ + +char *IOHDF5_Get_filename_ND (cGH *GH, int index, ioHDF5Geo_t *slab_geo, + const char *varname, int *isNewFile) +{ + DECLARE_CCTK_PARAMETERS + ioGH *ioUtilGH; /* handle for IOUtil extensions */ + ioHDF5GH *myGH; /* handle for IOHDF5 extensions */ + pGH *pughGH; /* handle for PUGH extensions */ + int myproc; + char extra [256]; /* Extra stuff in fname based on mode */ + char extradir [256]; /* Extra stuff for an output dir */ + char *extrageo; /* Extra stuff for geometry information */ + char *filename; /* the return value */ + int idim; + + char name[128]; + + /* get GH extensions for PUGH and IOUtil */ + pughGH = PUGH_pGH (GH); + ioUtilGH = (ioGH *) GH->extensions [CCTK_GHExtensionHandle ("IO")]; + myGH = (ioHDF5GH *) GH->extensions [CCTK_GHExtensionHandle ("IOHDF5")]; + + /* To identify the slab in a volume (if sdim<vdim) + build a string like: "012" (3d volume, spanned by xyz) + or "12" (2d volume, spanned by yz(!) and append this to the name: + You get something like this: phi_s12_3d */ + if (slab_geo->sdim<slab_geo->vdim) { + extrageo = (char*) malloc((slab_geo->sdim+1)*sizeof(char)); + sprintf(extrageo,"%d",slab_geo->direction[0]); + for (idim=1;idim<slab_geo->sdim;idim++) + sprintf(extrageo,"%s%d",extrageo,slab_geo->direction[idim]); + sprintf(name,"%s_s%s_%dd",varname,extrageo,slab_geo->vdim); + free(extrageo); + } + /* If sdim==vdim, make it simple: e.g. phi_3d */ + else if (slab_geo->sdim==slab_geo->vdim) + { + sprintf(name,"%s_%dd",varname,slab_geo->vdim); + } + else { + CCTK_VWarn (1, __LINE__, __FILE__, CCTK_THORNSTRING, + "Inconsitent dimensions for >%s<: slab (%dD), variable (%dD)\n", + varname, slab_geo->sdim,slab_geo->vdim); + return(NULL); + } + + filename = (char *) GetNamedData (myGH->filenameList, name); + if (filename != NULL) + { + /* set flag to indicate that file should be opened in append mode */ + *isNewFile = 0; + return (filename); + } + + extra[0] = '\0'; + extradir[0]= '\0'; + + myproc = CCTK_MyProc (GH); + + if (out2D_septimefiles) + { + char *tmp = extra; + + sprintf (extra, "%s.time_%7.3f", extra, GH->cctk_time); + + /* And be sure to replace any spaces in the filename with an _ */ + do + if (*tmp == ' ') + *tmp = '_'; + while (*++tmp); + } + + + /* OUTPUT ONE FILE FOR EACH N PROCESSORS - only 3D + * ------------------------------------- + * + * If only one output file, the single file for each variable is written + * in the normal output dir (that is extradir = ""). Otherwise + * a directory is created for each output variable to hold the multiple + * files. + */ + + if (! ioUtilGH->unchunked && ioUtilGH->ioproc_every < CCTK_nProcs (GH) && + slab_geo->vdim==3) + { + /* Add the output processor number to the extra string */ + sprintf (extra, "%s.file_%d", extra, myproc / ioUtilGH->ioproc_every); + + /* If necessary create the output directory */ + if (myproc == 0) + { + int result; + char *outputdir = (char *) malloc (strlen (myGH->outdir3D) + + strlen (name) + 5); + + sprintf (outputdir, "%s/%s_3d", myGH->outdir3D, name); + + result = CCTK_CreateDirectory (0755, outputdir); + if (result < 0) + CCTK_VWarn (1, __LINE__, __FILE__, CCTK_THORNSTRING, + "Problem creating 3D output directory '%s'", outputdir); + if (result > 0) + CCTK_VWarn (2, __LINE__, __FILE__, CCTK_THORNSTRING, + "3D output directory '%s' already exists", outputdir); + + free (outputdir); + } + +#ifdef CCTK_MPI + /* Wait for all processors to catch up */ + CCTK_Barrier (GH); +#endif + + /* extradir is the relative output directory */ + sprintf (extradir, "%s_3d/", name); + + } + + /* CREATE THE COMPLETE OUTPUT FILENAME + ----------------------------------- */ + filename = (char *) malloc (strlen (myGH->outdir2D) + + strlen (extradir) + + strlen (name) + + strlen (extra) + + (pughGH->identity_string ? strlen (pughGH->identity_string) : 0) + + 8); + sprintf (filename, "%s/%s%s%s%s.h5", + myGH->outdir2D, extradir, name, extra, (pughGH->identity_string ? pughGH->identity_string : "")); + + /* no need to store filenames if used only once */ + if (! out2D_septimefiles) + { + if (myproc == ioUtilGH->ioproc) + { + if (ioUtilGH->recovered) + { + CACTUS_IOHDF5_ERROR (H5Eset_auto (NULL, NULL)); + *isNewFile = H5Fis_hdf5 (filename) <= 0; + H5Eset_auto (myGH->printErrorFn, myGH->printErrorFnArg); + if (! *isNewFile) + myGH->checkForExistingObjects [index] = 1; + } + else + { + *isNewFile = 1; + } + } + StoreNamedData (&myGH->filenameList, name, filename); + } + else + { + *isNewFile = 1; + } + + return (filename); +} diff --git a/src/GHExtension.c b/src/GHExtension.c new file mode 100644 index 0000000..a3039eb --- /dev/null +++ b/src/GHExtension.c @@ -0,0 +1,190 @@ + /*@@ + @file GHExtension.c + @date Fri May 21 1999 + @author Thomas Radke + @desc + IOHDF5 GH extension stuff. + @enddesc + @history + @hauthor Thomas Radke @hdate May 21 1999 + @hdesc Just copied from thorn FlexIO. + @endhistory + @@*/ + + +/*#define DEBUG_IO*/ + +#include <stdlib.h> +#include <string.h> +#include <stdio.h> + +#include "cctk.h" +#include "cctk_Parameters.h" +#include "CactusPUGH/PUGH/src/include/pugh.h" +#include "CactusBase/IOUtil/src/ioGH.h" +#include "ioHDF5GH.h" + +void IOHDF5_DefaultGeo(int sdim, ioHDF5Geo_t *geo); + +void *IOHDF5_SetupGH (tFleshConfig *config, int convergence_level, cGH *GH) +{ + int numvars,iv; + ioHDF5GH *newGH; + + numvars = CCTK_NumVars (); + + newGH = (ioHDF5GH *) malloc (sizeof (ioHDF5GH)); + newGH->do_out1D = (char *) malloc (numvars * sizeof (char)); + newGH->do_out2D = (char *) malloc (numvars * sizeof (char)); + newGH->do_out3D = (char *) malloc (numvars * sizeof (char)); + newGH->out1D_last = (int *) malloc (numvars * sizeof (int)); + newGH->out2D_last = (int *) malloc (numvars * sizeof (int)); + newGH->out3D_last = (int *) malloc (numvars * sizeof (int)); + newGH->checkForExistingObjects = (char *) calloc (numvars, 1); + + /* Allocate geometry structure for each grid variable/array */ + newGH->out_geo = (ioHDF5Geo_t**) malloc(IOHDF5_MAXDIM * sizeof (ioHDF5Geo_t)); + for (iv=0;iv<numvars;iv++) + newGH->out_geo[iv] = (ioHDF5Geo_t*) malloc(IOHDF5_MAXDIM * sizeof (ioHDF5Geo_t)); + + /* save the original error printing routine and its argument */ + CACTUS_IOHDF5_ERROR (H5Eget_auto (&newGH->printErrorFn, + &newGH->printErrorFnArg)); + + /* predefine dataspaces for writing scalar and array attributes */ + /* the dimension of the array dataspace is set when used */ + CACTUS_IOHDF5_ERROR (newGH->scalarDataspace = H5Screate (H5S_SCALAR)); + CACTUS_IOHDF5_ERROR (newGH->arrayDataspace = H5Screate (H5S_SIMPLE)); + + /* predefine a IOHDF5_COMPLEX datatype */ + CACTUS_IOHDF5_ERROR (newGH->IOHDF5_COMPLEX = + H5Tcreate (H5T_COMPOUND, sizeof (CCTK_COMPLEX))); + CACTUS_IOHDF5_ERROR (H5Tinsert (newGH->IOHDF5_COMPLEX, "real", + offsetof (CCTK_COMPLEX, Re), IOHDF5_REAL)); + CACTUS_IOHDF5_ERROR (H5Tinsert (newGH->IOHDF5_COMPLEX, "imag", + offsetof (CCTK_COMPLEX, Im), IOHDF5_REAL)); + + /* predefine a C string datatype */ + CACTUS_IOHDF5_ERROR (newGH->IOHDF5_STRING = H5Tcopy (H5T_C_S1)); + + return (newGH); +} + +int IOHDF5_InitGH (cGH *GH) +{ + DECLARE_CCTK_PARAMETERS + int inum,idim,i; + ioHDF5GH *myGH; + const cParamData *paramdata; + ioHDF5Geo_t geo_default; + + + myGH = (ioHDF5GH *) GH->extensions [CCTK_GHExtensionHandle ("IOHDF5")]; + + /* How often to output */ + myGH->out1D_every = out_every > 0 ? out_every : -1; + if (out1D_every > 0) + myGH->out1D_every = out1D_every; + IOUtil_ParseVarsForOutput (out1D_vars, myGH->do_out1D); + + myGH->out2D_every = out_every > 0 ? out_every : -1; + if (out2D_every > 0) + myGH->out2D_every = out2D_every; + IOUtil_ParseVarsForOutput (out2D_vars, myGH->do_out2D); + + myGH->out3D_every = out_every > 0 ? out_every : -1; + if (out3D_every > 0) + myGH->out3D_every = out3D_every; + IOUtil_ParseVarsForOutput (out3D_vars, myGH->do_out3D); + + /* Check whether "outdir2D" and/or "outdir3D" was set. + If so take these dirs otherwise default to "IO::outdir" */ + paramdata = CCTK_ParameterData ("outdir1D", CCTK_THORNSTRING); + if (paramdata && paramdata->n_set > 0) + myGH->outdir1D = strdup (outdir1D); + else + myGH->outdir1D = strdup (outdir); + + paramdata = CCTK_ParameterData ("outdir2D", CCTK_THORNSTRING); + if (paramdata && paramdata->n_set > 0) + myGH->outdir2D = strdup (outdir2D); + else + myGH->outdir2D = strdup (outdir); + + paramdata = CCTK_ParameterData ("outdir3D", CCTK_THORNSTRING); + if (paramdata && paramdata->n_set > 0) + myGH->outdir3D = strdup (outdir3D); + else + myGH->outdir3D = strdup (outdir); + + /* Create the output directories */ + if (CCTK_MyProc (GH) == 0) { + i = CCTK_CreateDirectory (0755, myGH->outdir1D); + if (i < 0) + CCTK_VWarn (1, __LINE__, __FILE__, CCTK_THORNSTRING, + "Problem creating HDF5 1D output directory '%s'", myGH->outdir1D); + else if (i > 0) + CCTK_VWarn (2, __LINE__, __FILE__, CCTK_THORNSTRING, + "HDF5 1D output directory '%s' already exists", myGH->outdir1D); + + i = CCTK_CreateDirectory (0755, myGH->outdir2D); + if (i < 0) + CCTK_VWarn (1, __LINE__, __FILE__, CCTK_THORNSTRING, + "Problem creating HDF5 2D output directory '%s'", myGH->outdir2D); + else if (i > 0) + CCTK_VWarn (2, __LINE__, __FILE__, CCTK_THORNSTRING, + "HDF5 2D output directory '%s' already exists", myGH->outdir2D); + + i = CCTK_CreateDirectory (0755, myGH->outdir3D); + if (i < 0) + CCTK_VWarn (1, __LINE__, __FILE__, CCTK_THORNSTRING, + "Problem creating HDF5 3D output directory '%s'", myGH->outdir3D); + else if (i > 0) + CCTK_VWarn (2, __LINE__, __FILE__, CCTK_THORNSTRING, + "HDF5 3D output directory '%s' already exists", myGH->outdir3D); + } + + for (inum=0; inum<CCTK_NumVars(); inum++) + { + myGH->out2D_last [inum] = myGH->out3D_last [inum] = -1; + myGH->out1D_last [inum] = myGH->out3D_last [inum] = -1; + } + + /* Loop over all slab dimension */ + for (idim=0;idim<IOHDF5_MAXDIM;idim++) { + IOHDF5_DefaultGeo(idim, &geo_default); + + /* Set the default HDF5 slab geometry for slab with dimension -idim-*/ + for (inum=0; inum<CCTK_NumVars(); inum++) + myGH->out_geo[inum][idim] = geo_default; + } + myGH->filenameList = NULL; + + /* create timers if timing info was requested */ + myGH->print_timing_info = print_timing_info; + if (myGH->print_timing_info) + { + for (i = 0; i < 4; i++) + { + if ((myGH->timers[i] = CCTK_TimerCreateI ()) < 0) + break; + } + if (i != 4) + { + CCTK_WARN (1, "Could not create timers for checkpoint/recovery ! " + "No timer information will be available."); + while (--i >= 0) + { + CCTK_TimerDestroyI (myGH->timers[i]); + } + myGH->print_timing_info = 0; + } + else + { + CCTK_TimerResetI (myGH->timers[2]); + CCTK_TimerResetI (myGH->timers[3]); + } + } + + return (0); +} diff --git a/src/Output1D.c b/src/Output1D.c new file mode 100644 index 0000000..70c3b97 --- /dev/null +++ b/src/Output1D.c @@ -0,0 +1,275 @@ + /*@@ + @file Output1D.c + @date Tue Jan 9 1999 + @author Gabrielle Allen + @desc + Functions to deal 1D HDF5 output of variables + @enddesc + @history + @hauthor Thomas Radke @hdate 16 Mar 1999 + @hdesc Converted to Cactus 4.0 + @hendhistory + @@*/ + + + +#include <stdio.h> +#include <stdlib.h> + +#include "cctk.h" +#include "cctk_Parameters.h" +#include "CactusBase/IOUtil/src/ioGH.h" +#include "ioHDF5GH.h" + + +/* function prototypes */ +int IOHDF5_TimeFor1D (cGH *GH, int index); +int IOHDF5_Output1DVarAs (cGH *GH, const char *var, const char *alias); +static void CheckSteerableParameters (ioHDF5GH *myGH); + + +/*@@ + @routine IOHDF5_Output1DGH + @date Sat March 6 1999 + @author Gabrielle Allen + @desc + Loops over all variables and outputs them if necessary + @enddesc + @calledby CCTK_OutputGH ("IOHDF5_1D") + @history + + @endhistory + @var GH + @vdesc Pointer to CCTK GH + @vtype cGH + @vio in + @endvar +@@*/ + +int IOHDF5_Output1DGH (cGH *GH) +{ + DECLARE_CCTK_PARAMETERS + int i; + ioHDF5GH *myGH; + const char *name; + char *fullname; + int ierr=0; + + /* Get the GH extension for IOHDF5 */ + myGH = (ioHDF5GH *) GH->extensions [CCTK_GHExtensionHandle ("IOHDF5")]; + + CheckSteerableParameters (myGH); + + if (myGH->out1D_every <= 0) + return (0); + + /* Loop over all variables */ + for (i = 0; i < CCTK_NumVars (); i++) + { + + if (IOHDF5_TimeFor1D (GH, i)) + { + + name = CCTK_VarName (i); + fullname = CCTK_FullName (i); + + if (verbose) + CCTK_VInfo (CCTK_THORNSTRING, "IOHDF5_Output1DGH: fullname / name = " + "%s / %s", fullname, name); + + ierr=IOHDF5_Output1DVarAs (GH, fullname, name); + + free (fullname); + + /* Register variable as having 1D output this iteration */ + myGH->out1D_last [i] = GH->cctk_iteration; + } + } + if (verbose) + CCTK_INFO("IOHDF5_Output1DGH Done \n\n"); + + return (ierr); +} + + +/*@@ + @routine IOHDF5_Output1DVarAs + @date Sat March 6 1999 + @author Gabrielle Allen + @desc + unconditional output of a variable using the IOHDF5 1D output method + @enddesc + @calledby IOHDF5_Output1DGH, CCTK_OutputVarAsByMethod ("IOHDF5_1D") + @var GH + @vdesc Pointer to CCTK GH + @vtype cGH + @vio in + @vcomment + @endvar + @var fullname + @vdesc complete name of variable to output + @vtype const char * + @vio in + @vcomment + @endvar + @var alias + @vdesc alias name of variable to output (used to generate output filename) + @vtype const char * + @vio in + @vcomment + @endvar +@@*/ + +int IOHDF5_Output1DVarAs (cGH *GH, const char *fullname, const char *alias) +{ + DECLARE_CCTK_PARAMETERS + int index, ierr=0; + + index = CCTK_VarIndex (fullname); + + /* first, check if variable has storage assigned */ + if (! CCTK_QueryGroupStorageI (GH, CCTK_GroupIndexFromVarI (index))) + { + CCTK_VWarn (2, __LINE__, __FILE__, CCTK_THORNSTRING, + "No IOHDF5 1D output for '%s' (no storage)", fullname); + return(-1); + } + + if (verbose) + CCTK_VInfo (CCTK_THORNSTRING, "IOHDF5_Output1DVarAs: fullname, alias, " + "index = (%s, %s, %d)", fullname, alias, index); + + /* Do the 1D output */ + ierr=IOHDF5_Write1D (GH, index, alias); + + return (ierr); +} + + +/*@@ + @routine IOHDF5_TimeFor1D + @date Sat March 6 1999 + @author Gabrielle Allen + @desc + Decides if it is time to output a variable using the IOHDF5 1D output + method + @enddesc + @calledby IOHDF5_Output1DGH + @history + + @endhistory + @var GH + @vdesc Pointer to CCTK GH + @vtype cGH + @vio in + @vcomment + @endvar + @var index + @vdesc index of variable + @vtype int + @vio in + @vcomment + @endvar +@@*/ + +int IOHDF5_TimeFor1D (cGH *GH, int index) +{ + ioHDF5GH *myGH; + + + /* Get the GH extension for IOHDF5 */ + myGH = (ioHDF5GH *) GH->extensions [CCTK_GHExtensionHandle ("IOHDF5")]; + + CheckSteerableParameters (myGH); + + /* Check if any output was requested */ + if(myGH->out1D_every <= 0) + return (0); + + /* Check this variable should be output */ + if (! (myGH->do_out1D [index] && GH->cctk_iteration % myGH->out1D_every == 0)) + return (0); + + /* Check variable not already output this iteration */ + if (myGH->out1D_last [index] == GH->cctk_iteration) + { + CCTK_WARN (2, "Already done 1D output in IOHDF5"); + return (0); + } + + return (1); +} + + +/*@@ + @routine IOHDF5_TriggerOutput1D + @date Sat March 6 1999 + @author Gabrielle Allen + @desc + Triggers the output a variable using the IOHDF5 1D output + method + @enddesc + @calledby CCTK scheduler + @var GH + @vdesc Pointer to CCTK GH + @vtype cGH + @vio in + @vcomment + @endvar + @var index + @vdesc index of variable to output + @vtype int + @vio in + @vcomment + @endvar +@@*/ + +int IOHDF5_TriggerOutput1D (cGH *GH, int index) +{ + DECLARE_CCTK_PARAMETERS + ioHDF5GH *myGH; + const char *varname; + int ierr=0; + + varname = CCTK_VarName (index); + + myGH = (ioHDF5GH *) GH->extensions [CCTK_GHExtensionHandle ("IOHDF5")]; + + if (verbose) + CCTK_VInfo (CCTK_THORNSTRING, "IOHDF5_TriggerOutput1D: varname, index = " + "(%s, %d)", varname, index); + + /* Do the 1D output */ + ierr=IOHDF5_Write1D (GH, index, varname); + + /* Register variable as having 1D output this iteration */ + myGH->out1D_last [index] = GH->cctk_iteration; + + return (ierr); +} + + +/**************************** local functions ******************************/ +static void CheckSteerableParameters (ioHDF5GH *myGH) +{ + DECLARE_CCTK_PARAMETERS + int out1D_vars_current_nset; + static int out1D_vars_lastset = 0; + + + /* How often to output */ + myGH->out1D_every = out_every > 0 ? out_every : -1; + if (out1D_every > 0) + myGH->out1D_every = out1D_every; + + /* re-parse the 'out1D_vars' parameter if it was changed */ + out1D_vars_current_nset = CCTK_ParameterQueryTimesSet ("out1D_vars", + CCTK_THORNSTRING); + if (out1D_vars_current_nset != out1D_vars_lastset) + { + IOUtil_ParseVarsForOutput (out1D_vars, myGH->do_out1D); + + /* Save the last setting of 'out1D_vars' parameter */ + out1D_vars_lastset = out1D_vars_current_nset; + } +} diff --git a/src/Output2D.c b/src/Output2D.c new file mode 100644 index 0000000..b07efc8 --- /dev/null +++ b/src/Output2D.c @@ -0,0 +1,275 @@ + /*@@ + @file Output2D.c + @date Tue Jan 9 1999 + @author Gabrielle Allen + @desc + Functions to deal 2D HDF5 output of variables + @enddesc + @history + @hauthor Thomas Radke @hdate 16 Mar 1999 + @hdesc Converted to Cactus 4.0 + @hendhistory + @@*/ + + + +#include <stdio.h> +#include <stdlib.h> + +#include "cctk.h" +#include "cctk_Parameters.h" +#include "CactusBase/IOUtil/src/ioGH.h" +#include "ioHDF5GH.h" + + +/* function prototypes */ +int IOHDF5_TimeFor2D (cGH *GH, int index); +int IOHDF5_Output2DVarAs (cGH *GH, const char *var, const char *alias); +static void CheckSteerableParameters (ioHDF5GH *myGH); + + +/*@@ + @routine IOHDF5_Output2DGH + @date Sat March 6 1999 + @author Gabrielle Allen + @desc + Loops over all variables and outputs them if necessary + @enddesc + @calledby CCTK_OutputGH ("IOHDF5_2D") + @history + + @endhistory + @var GH + @vdesc Pointer to CCTK GH + @vtype cGH + @vio in + @endvar +@@*/ + +int IOHDF5_Output2DGH (cGH *GH) +{ + DECLARE_CCTK_PARAMETERS + int i; + ioHDF5GH *myGH; + const char *name; + char *fullname; + int ierr=0; + + /* Get the GH extension for IOHDF5 */ + myGH = (ioHDF5GH *) GH->extensions [CCTK_GHExtensionHandle ("IOHDF5")]; + + CheckSteerableParameters (myGH); + + if (myGH->out2D_every <= 0) + return (0); + + /* Loop over all variables */ + for (i = 0; i < CCTK_NumVars (); i++) + { + + if (IOHDF5_TimeFor2D (GH, i)) + { + + name = CCTK_VarName (i); + fullname = CCTK_FullName (i); + + if (verbose) + CCTK_VInfo (CCTK_THORNSTRING, "IOHDF5_Output2DGH: fullname / name = " + "%s / %s", fullname, name); + + ierr=IOHDF5_Output2DVarAs (GH, fullname, name); + + free (fullname); + + /* Register variable as having 2D output this iteration */ + myGH->out2D_last [i] = GH->cctk_iteration; + } + } + if (verbose) + CCTK_INFO("IOHDF5_Output2DGH Done \n\n"); + + return (ierr); +} + + +/*@@ + @routine IOHDF5_Output2DVarAs + @date Sat March 6 1999 + @author Gabrielle Allen + @desc + unconditional output of a variable using the IOHDF5 2D output method + @enddesc + @calledby IOHDF5_Output2DGH, CCTK_OutputVarAsByMethod ("IOHDF5_2D") + @var GH + @vdesc Pointer to CCTK GH + @vtype cGH + @vio in + @vcomment + @endvar + @var fullname + @vdesc complete name of variable to output + @vtype const char * + @vio in + @vcomment + @endvar + @var alias + @vdesc alias name of variable to output (used to generate output filename) + @vtype const char * + @vio in + @vcomment + @endvar +@@*/ + +int IOHDF5_Output2DVarAs (cGH *GH, const char *fullname, const char *alias) +{ + DECLARE_CCTK_PARAMETERS + int index,ierr=0; + + index = CCTK_VarIndex (fullname); + + /* first, check if variable has storage assigned */ + if (! CCTK_QueryGroupStorageI (GH, CCTK_GroupIndexFromVarI (index))) + { + CCTK_VWarn (2, __LINE__, __FILE__, CCTK_THORNSTRING, + "No IOHDF5 2D output for '%s' (no storage)", fullname); + return(-1); + } + + if (verbose) + CCTK_VInfo (CCTK_THORNSTRING, "IOHDF5_Output2DVarAs: fullname, alias, " + "index = (%s, %s, %d)", fullname, alias, index); + + /* Do the 2D output */ + ierr=IOHDF5_Write2D (GH, index, alias); + + return (ierr); +} + + +/*@@ + @routine IOHDF5_TimeFor2D + @date Sat March 6 1999 + @author Gabrielle Allen + @desc + Decides if it is time to output a variable using the IOHDF5 2D output + method + @enddesc + @calledby IOHDF5_Output2DGH + @history + + @endhistory + @var GH + @vdesc Pointer to CCTK GH + @vtype cGH + @vio in + @vcomment + @endvar + @var index + @vdesc index of variable + @vtype int + @vio in + @vcomment + @endvar +@@*/ + +int IOHDF5_TimeFor2D (cGH *GH, int index) +{ + ioHDF5GH *myGH; + + + /* Get the GH extension for IOHDF5 */ + myGH = (ioHDF5GH *) GH->extensions [CCTK_GHExtensionHandle ("IOHDF5")]; + + CheckSteerableParameters (myGH); + + /* Check if any output was requested */ + if(myGH->out2D_every <= 0) + return (0); + + /* Check this variable should be output */ + if (! (myGH->do_out2D [index] && GH->cctk_iteration % myGH->out2D_every == 0)) + return (0); + + /* Check variable not already output this iteration */ + if (myGH->out2D_last [index] == GH->cctk_iteration) + { + CCTK_WARN (2, "Already done 2D output in IOHDF5"); + return (0); + } + + return (1); +} + + +/*@@ + @routine IOHDF5_TriggerOutput2D + @date Sat March 6 1999 + @author Gabrielle Allen + @desc + Triggers the output a variable using the IOHDF5 2D output + method + @enddesc + @calledby CCTK scheduler + @var GH + @vdesc Pointer to CCTK GH + @vtype cGH + @vio in + @vcomment + @endvar + @var index + @vdesc index of variable to output + @vtype int + @vio in + @vcomment + @endvar +@@*/ + +int IOHDF5_TriggerOutput2D (cGH *GH, int index) +{ + DECLARE_CCTK_PARAMETERS + ioHDF5GH *myGH; + const char *varname; + int ierr=0; + + varname = CCTK_VarName (index); + + myGH = (ioHDF5GH *) GH->extensions [CCTK_GHExtensionHandle ("IOHDF5")]; + + if (verbose) + CCTK_VInfo (CCTK_THORNSTRING, "IOHDF5_TriggerOutput2D: varname, index = " + "(%s, %d)", varname, index); + + /* Do the 2D output */ + ierr=IOHDF5_Write2D (GH, index, varname); + + /* Register variable as having 2D output this iteration */ + myGH->out2D_last [index] = GH->cctk_iteration; + + return (ierr); +} + + +/**************************** local functions ******************************/ +static void CheckSteerableParameters (ioHDF5GH *myGH) +{ + DECLARE_CCTK_PARAMETERS + int out2D_vars_current_nset; + static int out2D_vars_lastset = 0; + + + /* How often to output */ + myGH->out2D_every = out_every > 0 ? out_every : -1; + if (out2D_every > 0) + myGH->out2D_every = out2D_every; + + /* re-parse the 'out2D_vars' parameter if it was changed */ + out2D_vars_current_nset = CCTK_ParameterQueryTimesSet ("out2D_vars", + CCTK_THORNSTRING); + if (out2D_vars_current_nset != out2D_vars_lastset) + { + IOUtil_ParseVarsForOutput (out2D_vars, myGH->do_out2D); + + /* Save the last setting of 'out2D_vars' parameter */ + out2D_vars_lastset = out2D_vars_current_nset; + } +} diff --git a/src/Output3D.c b/src/Output3D.c new file mode 100644 index 0000000..caaca4a --- /dev/null +++ b/src/Output3D.c @@ -0,0 +1,266 @@ + /*@@ + @file Output3D.c + @date Tue Jan 9 1999 + @author Gabrielle Allen + @desc + Functions to deal 3D HDF5 output of variables + @enddesc + @history + @hauthor Thomas Radke @hdate 16 Mar 1999 + @hdesc Converted to Cactus 4.0 + @hendhistory + @@*/ + + + +#include <stdio.h> +#include <stdlib.h> + +#include "cctk.h" +#include "cctk_Parameters.h" +#include "CactusBase/IOUtil/src/ioGH.h" +#include "ioHDF5GH.h" + + +/* function prototypes */ +int IOHDF5_TimeFor3D (cGH *GH, int index); +int IOHDF5_Output3DVarAs (cGH *GH, const char *var, const char *alias); +static void CheckSteerableParameters (ioHDF5GH *myGH); + + +/*@@ + @routine IOHDF5_Output3DGH + @date Sat March 6 1999 + @author Gabrielle Allen + @desc + Loops over all variables and outputs them if necessary + @enddesc + @calledby CCTK_OutputGH ("IOHDF5_3D") + @history + + @endhistory + @var GH + @vdesc Pointer to CCTK GH + @vtype cGH + @vio in + @endvar +@@*/ + +int IOHDF5_Output3DGH (cGH *GH) +{ + DECLARE_CCTK_PARAMETERS + int i; + ioHDF5GH *myGH; + const char *name; + char *fullname; + + /* Get the GH extension for IOHDF5 */ + myGH = (ioHDF5GH *) GH->extensions [CCTK_GHExtensionHandle ("IOHDF5")]; + + CheckSteerableParameters (myGH); + + if (myGH->out3D_every <= 0) + return (0); + + /* Loop over all variables */ + for (i = 0; i < CCTK_NumVars (); i++) + { + + if (IOHDF5_TimeFor3D (GH, i)) + { + + name = CCTK_VarName (i); + fullname = CCTK_FullName (i); + + if (verbose) + CCTK_VInfo (CCTK_THORNSTRING, "IOHDF5_Output3DGH: fullname / name = " + "%s / %s", fullname, name); + + IOHDF5_Output3DVarAs (GH, fullname, name); + + free (fullname); + + /* Register variable as having 3D output this iteration */ + myGH->out3D_last [i] = GH->cctk_iteration; + } + } + if (verbose) + CCTK_INFO("IOHDF5_Output3DGH Done \n\n"); + + return (0); +} + + +/*@@ + @routine IOHDF5_Output3DVarAs + @date Sat March 6 1999 + @author Gabrielle Allen + @desc + unconditional output of a variable using the IOHDF5 3D output method + @enddesc + @calledby IOHDF5_Output3DGH, CCTK_OutputVarAsByMethod ("IOHDF5_3D") + @var GH + @vdesc Pointer to CCTK GH + @vtype cGH + @vio in + @vcomment + @endvar + @var fullname + @vdesc complete name of variable to output + @vtype const char * + @vio in + @vcomment + @endvar + @var alias + @vdesc alias name of variable to output (used to generate output filename) + @vtype const char * + @vio in + @vcomment + @endvar +@@*/ + +int IOHDF5_Output3DVarAs (cGH *GH, const char *fullname, const char *alias) +{ + DECLARE_CCTK_PARAMETERS + int index; + + + index = CCTK_VarIndex (fullname); + + if (verbose) + CCTK_VInfo (CCTK_THORNSTRING, "IOHDF5_Output3DVarAs: fullname, alias, " + "index = (%s, %s, %d)", fullname, alias, index); + + /* Do the 3D output */ + IOHDF5_Write3D (GH, index, alias); + + return (0); +} + + +/*@@ + @routine IOHDF5_TimeFor3D + @date Sat March 6 1999 + @author Gabrielle Allen + @desc + Decides if it is time to output a variable using the IOHDF5 3D output + method + @enddesc + @calledby IOHDF5_Output3DGH + @history + + @endhistory + @var GH + @vdesc Pointer to CCTK GH + @vtype cGH + @vio in + @vcomment + @endvar + @var index + @vdesc index of variable + @vtype int + @vio in + @vcomment + @endvar +@@*/ + +int IOHDF5_TimeFor3D (cGH *GH, int index) +{ + ioHDF5GH *myGH; + + + /* Get the GH extension for IOHDF5 */ + myGH = (ioHDF5GH *) GH->extensions [CCTK_GHExtensionHandle ("IOHDF5")]; + + CheckSteerableParameters (myGH); + + /* Check if any output was requested */ + if(myGH->out3D_every <= 0) + return (0); + + /* Check this variable should be output */ + if (! (myGH->do_out3D [index] && GH->cctk_iteration % myGH->out3D_every == 0)) + return (0); + + /* Check variable not already output this iteration */ + if (myGH->out3D_last [index] == GH->cctk_iteration) + { + CCTK_WARN (2, "Already done 3D output in IOHDF5"); + return (0); + } + + return (1); +} + + +/*@@ + @routine IOHDF5_TriggerOutput3D + @date Sat March 6 1999 + @author Gabrielle Allen + @desc + Triggers the output a variable using the IOHDF5 3D output + method + @enddesc + @calledby CCTK scheduler + @var GH + @vdesc Pointer to CCTK GH + @vtype cGH + @vio in + @vcomment + @endvar + @var index + @vdesc index of variable to output + @vtype int + @vio in + @vcomment + @endvar +@@*/ + +int IOHDF5_TriggerOutput3D (cGH *GH, int index) +{ + DECLARE_CCTK_PARAMETERS + ioHDF5GH *myGH; + const char *varname; + + varname = CCTK_VarName (index); + + myGH = (ioHDF5GH *) GH->extensions [CCTK_GHExtensionHandle ("IOHDF5")]; + + if (verbose) + CCTK_VInfo (CCTK_THORNSTRING, "IOHDF5_TriggerOutput3D: varname, index = " + "(%s, %d)", varname, index); + + /* Do the 3D output */ + IOHDF5_Write3D (GH, index, varname); + + /* Register variable as having 3D output this iteration */ + myGH->out3D_last [index] = GH->cctk_iteration; + + return (0); +} + + +/**************************** local functions ******************************/ +static void CheckSteerableParameters (ioHDF5GH *myGH) +{ + DECLARE_CCTK_PARAMETERS + int out3D_vars_current_nset; + static int out3D_vars_lastset = 0; + + + /* How often to output */ + myGH->out3D_every = out_every > 0 ? out_every : -1; + if (out3D_every > 0) + myGH->out3D_every = out3D_every; + + /* re-parse the 'out3D_vars' parameter if it was changed */ + out3D_vars_current_nset = CCTK_ParameterQueryTimesSet ("out3D_vars", + CCTK_THORNSTRING); + if (out3D_vars_current_nset != out3D_vars_lastset) + { + IOUtil_ParseVarsForOutput (out3D_vars, myGH->do_out3D); + + /* Save the last setting of 'out3D_vars' parameter */ + out3D_vars_lastset = out3D_vars_current_nset; + } +} diff --git a/src/ParseGeometry.c b/src/ParseGeometry.c new file mode 100644 index 0000000..61297b3 --- /dev/null +++ b/src/ParseGeometry.c @@ -0,0 +1,147 @@ +#include <stdio.h> +#include <stdlib.h> +#include <string.h> + +#include "cctk.h" +#include "cctk_Arguments.h" +#include "cctk_Parameters.h" + +#include "util_String.h" +#include "ioHDF5GH.h" + + +void IOHDF5_DefaultGeo(int sdim, ioHDF5Geo_t *geo) { + DECLARE_CCTK_PARAMETERS + + int idim; + const char *tmp_origin, *tmp_downs, *tmp_length; + const char *token; + + geo->vdim = -1; + geo->sdim = -1; + + for (idim=0;idim<IOHDF5_MAXDIM;idim++) { + geo->direction[idim] = idim; + geo->slab_start[idim]= 0; + geo->length[idim] =-1; + geo->actlen[idim] =-1; + geo->downs[idim] = 1; + } + + /* Parse the parameter of the requested dimension */ + switch (sdim) { + case 1: + tmp_origin = origin1D; + tmp_downs = downsampling1D; + tmp_length = length1D; + break; + case 2: + tmp_origin = origin2D; + tmp_downs = downsampling2D; + tmp_length = length2D; + break; + case 3: + tmp_origin = origin3D; + tmp_downs = downsampling3D; + tmp_length = length3D; + break; + case 4: + tmp_origin = origin4D; + tmp_downs = downsampling4D; + tmp_length = length4D; + break; + default: + tmp_origin = origin; + tmp_downs = downsampling; + tmp_length = length; + break; + } + + /* Origin */ + idim=0; + while((token = Util_StrSep(&tmp_origin,","))) { + geo->slab_start[idim++]=atoi(token); + } + geo->slab_start[idim] = atoi(tmp_origin); + + /* Downsample */ + idim=0; + while((token = Util_StrSep(&tmp_downs,","))) { + geo->downs[idim++]=atoi(token); + } + geo->downs[idim] = atoi(tmp_downs); + + /* Length */ + idim=0; + while((token = Util_StrSep(&tmp_length,","))) { + geo->length[idim++]=atoi(token); + } + geo->length[idim] = atoi(tmp_length); +} + + +int NumDirection(int sdim, int vdim) +{ + int numdir=-1; + if (vdim==3) { + switch (sdim) { + case 1: return(3); + case 2: return(3); + case 3: return(1); + } + } else if (vdim==2) { + switch (sdim) { + case 1: return(2); + case 2: return(1); + } + } + else if (vdim==1) { + switch (sdim) { + case 1: return(1); + } + } + else printf("Bad dimension: %d \n",vdim); + return(numdir); +} + +int SetDirection(int vdim, int sdim, int ci, int *direction) +{ + int retval=0; + + if (sdim>vdim) { + printf("SetDirection: slabdim gt vdim"); + return(-1); + } + + if (vdim==3) { + if (sdim==2) { + switch (ci) { + case 0: direction[0]=0; direction[1]=1; break; + case 1: direction[0]=0; direction[1]=2; break; + case 2: direction[0]=1; direction[1]=2; break; + } + } + else if (sdim==1) { + switch (ci) { + case 0: direction[0]=0; break; + case 1: direction[0]=1; break; + case 2: direction[0]=2; break; + } + } + } + else if (vdim==2) { + if (sdim==2) { + direction[0]=0; direction[1]=1; + } else if (sdim==1) { + switch (ci) { + case 0: direction[0]=0; break; + case 1: direction[0]=1; break; + } + } + } else retval = -1; + return(retval); +} + + + + diff --git a/src/RecoverGH.c b/src/RecoverGH.c new file mode 100644 index 0000000..d2047e9 --- /dev/null +++ b/src/RecoverGH.c @@ -0,0 +1,491 @@ + /*@@ + @file RecoverGH.c + @date Fri Jun 19 09:14:22 1998 + @author Tom Goodale + @desc + Contains the routines to do the internal checkpoint recovery. + + Currently can recover from: + (1) One file containing recombined data + (2) Multiple unrecombined files, where the current + number of processors and IO processors + match those used to write the files. + @enddesc + @history + @hauthor Gabrielle Allen @hdate 19 Oct 1998 + @hdesc Changed names ready for thorn_IO + @endhistory + @version $Id$ + @@*/ + + +#include <ctype.h> +#include <stdio.h> +#include <stdlib.h> + +#include "cctk.h" +#include "cctk_Parameters.h" +#include "CactusPUGH/PUGH/src/include/pugh.h" +#include "CactusBase/IOUtil/src/ioGH.h" +#include "ioHDF5GH.h" + + +/* the rcs ID and its dummy funtion to use it */ +static char *rcsid = "$Id$"; +CCTK_FILEVERSION(CactusPUGHIO_IOHDF5_RecoverGH_c) + + +/* local function prototypes */ +static int IOHDF5i_RecoverParameters (fileinfo_t *file); +static int IOHDF5i_RecoverGHextensions (cGH *GH, fileinfo_t *file); +static int IOHDF5i_OpenFile (cGH *GH, const char *basename, int called_from, + fileinfo_t *file); +/* this one comes from RestoreFile.c */ +int IOHDF5i_RecoverVariables (cGH *GH, fileinfo_t *file); + + + /*@@ + @routine IOHDF5_Recover + @date Fri Jun 19 09:22:52 1998 + @author Tom Goodale + @desc + Recovers a GH from an HDF5 file. + This routine is registered with IOUtil + as IOHDF5's recovery routine. + @enddesc + @calledby IOUtil_RecoverFromFile + @history + @endhistory + @var GH + @vdesc Pointer to CCTK grid hierarchy + @vtype cGH + @vio in + @endvar + @var basename + @vdesc the basename of the file to recover from + The file suffix is appended by the routine. + @vtype const char * + @vio in + @endvar + @var called_from + @vdesc flag indicating where this routine was called from + (either RECOVER or filereader) + @vtype int + @vio in + @endvar +@@*/ +int IOHDF5_Recover (cGH *GH, const char *basename, int called_from) +{ + DECLARE_CCTK_PARAMETERS + int result; + ioHDF5GH *myGH; + static fileinfo_t file; /* this is static because info is passed from + CP_RECOVERY_PARAMETERS to CP_RECOVERY_DATA */ + + /* to make the compiler happy */ + myGH = NULL; + + /* start the total timer */ + if (GH) + { + myGH = (ioHDF5GH *) GH->extensions [CCTK_GHExtensionHandle ("IOHDF5")]; + if (myGH->print_timing_info) + { + CCTK_TimerStartI (myGH->timers[3]); + } + } + + /* open the file if it wasn't already opened at CCTK_RECOVER_PARAMETERS */ + /* FIXME Gab ... asymmetric levfac */ + if (called_from == CP_RECOVER_PARAMETERS || called_from == FILEREADER_DATA || + (GH && (GH->cctk_levfac[0] > 1 || GH->cctk_convlevel > 0))) { + if (IOHDF5i_OpenFile (GH, basename, called_from, &file) < 0) + return (-1); + } else { + /* This is the case for CP_RECOVER_DATA. + CCTK_RECOVER_PARAMETERS must have been called before + and set up the file info structure. */ + if (! file.is_HDF5_file) + return (-1); + } + + /* Recover parameters (and return) */ + if (called_from == CP_RECOVER_PARAMETERS) + return (IOHDF5i_RecoverParameters (&file)); + + /* Recover variables */ + if (verbose) + CCTK_VInfo (CCTK_THORNSTRING, "Recovering %schunked data with ioproc %d, " + "ioproc_every %d", file.unchunked ? "un" : "", file.ioproc, + file.ioproc_every); + result = IOHDF5i_RecoverVariables (GH, &file); + + /* Recover GH extensions */ + if (result == 0 && called_from == CP_RECOVER_DATA) { + if (verbose) + CCTK_INFO ("Recovering GH extensions"); + result = IOHDF5i_RecoverGHextensions (GH, &file); + } + + /* Close the file */ + if (CCTK_MyProc (GH) == file.ioproc) { + if (verbose) { + if (called_from == CP_RECOVER_DATA) + CCTK_VInfo (CCTK_THORNSTRING, "Closing checkpoint file '%s' after " + "recovery", file.filename); + else + CCTK_VInfo (CCTK_THORNSTRING, "Closing data file '%s'", file.filename); + } + CACTUS_IOHDF5_ERROR (H5Fclose (file.fid)); + } + + /* stop total recovery timer and print timing info */ + if (called_from == CP_RECOVER_DATA && myGH->print_timing_info) + { + const char *timer_description = "Time to recover:"; + + + CCTK_TimerStopI (myGH->timers[3]); + IOUtil_PrintTimings ("Timing information for recovery in IOHDF5:", 1, + &myGH->timers[3], &timer_description); + } + + return (result); +} + + + /*@@ + @routine IOHDF5_RecoverParameters + @date Thu Apr 13 2000 + @author Thomas Radke + @desc + Recovers the parameters from an HDF5 checkpoint file. + This routine is scheduled at CCTK_RECOVER_PARAMETERS. + + Note that it cannot be registered with IOUtil to be scheduled + from there (as done with the IOHDF5_Recover routine) because + the registration mechanism isn't activated yet + at CCTK_RECOVER_PARAMETERS. + Instead we call the generic parameter recovery routine + from IOUtil here, and just pass the necessary callback function + and its arguments. + + Note also that this routine doesn't get passed any parameters, + not even a GH, because this doesn't exist yet at the time it is + being called. + @enddesc + @calledby flesh's scheduler + @history + @endhistory + +@@*/ +int IOHDF5_RecoverParameters (void) +{ + return (IOUtil_RecoverParameters (IOHDF5_Recover, ".h5", "HDF5")); +} + + +/**************************** local routines ******************************/ + +static int IOHDF5i_OpenFile (cGH *GH, const char *basename, int called_from, + fileinfo_t *file) +{ + DECLARE_CCTK_PARAMETERS + hid_t group; + int nprocs, myproc; +#ifdef CCTK_MPI + MPI_Comm comm; + CCTK_INT4 info [3]; +#endif + + +#ifdef CCTK_MPI + /* Get the communicator for broadcasting the info structure */ + /* NOTE: When recovering parameters thorn PUGH is not yet initialized + so that we have to use MPI_COMM_WORLD in this case */ + comm = CCTK_GHExtensionHandle ("PUGH") < 0 ? + MPI_COMM_WORLD : PUGH_pGH (GH)->PUGH_COMM_WORLD; +#endif + + /* identify myself */ + nprocs = CCTK_nProcs (GH); + myproc = CCTK_MyProc (GH); + + /* Examine basefile to find out whether we are recovering from + * one or multiple files and whether the data are chunked or not. + * + * This is done by processor 0 only since this is always an IO processor + * and a corresponding file must exist in all cases. + */ + + /* Determine name of base file + NOTE: As we don't know whether the file is chunked or not + we need to try both file names. */ + /* at first try with unchunked mode */ + file->unchunked = 1; + IOUtil_PrepareFilename (GH, basename, file->filename, called_from, 0, + file->unchunked); + strcat (file->filename, ".h5"); + + if (myproc == 0) { + H5E_auto_t printErrorFn; + void *printErrorFnArg; + + if (verbose) + CCTK_VInfo (CCTK_THORNSTRING, "Opening file '%s'", file->filename); + + /* turn automatic error printing off again during check + but preserving the previous error handler settings */ + CACTUS_IOHDF5_ERROR (H5Eget_auto (&printErrorFn, &printErrorFnArg)); + CACTUS_IOHDF5_ERROR (H5Eset_auto (NULL, NULL)); + + /* Check the filetype */ + if (H5Fis_hdf5 (file->filename) > 0) + file->is_HDF5_file = 1; + else { + CCTK_VWarn (2, __LINE__, __FILE__, CCTK_THORNSTRING, + "No valid HDF5 file '%s' found", file->filename); + + /* now try with chunked mode */ + file->unchunked = 0; + IOUtil_PrepareFilename (GH, basename, file->filename, called_from, 0, + file->unchunked); + strcat (file->filename, ".h5"); + + if (verbose) + CCTK_VInfo (CCTK_THORNSTRING, "Trying now file '%s'...",file->filename); + + file->is_HDF5_file = H5Fis_hdf5 (file->filename) > 0; + } + + /* turn automatic error printing on again */ + CACTUS_IOHDF5_ERROR (H5Eset_auto (printErrorFn, printErrorFnArg)); + } + + /* Okay, we have the complete filename. Let's read the file now. */ + if (myproc == 0 && file->is_HDF5_file) { + + file->is_HDF5_file = 0; + + /* Determine how the data was written by reading the GH extensions + from the predefined group GHEXTENSIONS_GROUP */ + file->fid = H5Fopen (file->filename, H5F_ACC_RDONLY, H5P_DEFAULT); + if (file->fid < 0) { + CCTK_VWarn (1, __LINE__, __FILE__, CCTK_THORNSTRING, + "Cannot open file '%s' !", file->filename); + } else if ((group = H5Gopen (file->fid, GHEXTENSIONS_GROUP)) < 0) { + CCTK_WARN (1, "Can't find GH extensions group. " + "Is this really a Cactus HDF5 datafile ?"); + } else { + + READ_ATTRIBUTE (group, "nprocs", H5T_NATIVE_INT, &file->nprocs); + READ_ATTRIBUTE (group, "unchunked", H5T_NATIVE_INT, &file->unchunked); + READ_ATTRIBUTE (group, "ioproc_every",H5T_NATIVE_INT,&file->ioproc_every); + + CACTUS_IOHDF5_ERROR (H5Gclose (group)); + + /* If we recover from multiple files the number of + * writing processors must match the number of reading + * processors, and the total number of processors must match. + */ + + if ((file->ioproc_every == nprocs && nprocs > 1) || file->unchunked) { + if (verbose) + CCTK_VInfo (CCTK_THORNSTRING, "Recovering from one %schunked file", + file->unchunked ? "un":""); + file->ioproc_every = nprocs; + file->is_HDF5_file = 1; + } else { + if (file->nprocs != nprocs) { + CCTK_VWarn (1, __LINE__, __FILE__, CCTK_THORNSTRING, + "Must restart on %d processors with chunked files " + "or recombine them", file->nprocs); + } else { + if (verbose) + CCTK_VInfo (CCTK_THORNSTRING, "Recovering from %d chunked files", + nprocs / file->ioproc_every + + (nprocs % file->ioproc_every ? 1 : 0)); + file->is_HDF5_file = 1; + } + } + } + } + + if (myproc == 0 && ! file->is_HDF5_file) + CCTK_VWarn (2, __LINE__, __FILE__, CCTK_THORNSTRING, + "No valid HDF5 file '%s' found", file->filename); + +#ifdef CCTK_MPI + /* Broadcast the file information to all processors */ + info [0] = file->is_HDF5_file; + info [1] = file->unchunked; + info [2] = file->ioproc_every; + CACTUS_MPI_ERROR (MPI_Bcast (info, 3, PUGH_MPI_INT4, 0, comm)); + file->is_HDF5_file = info [0]; + file->unchunked = info [1]; + file->ioproc_every = info [2]; +#endif + + if (file->is_HDF5_file) { + + /* Determine the IO processors for each node and the corresponding + checkpoint file */ + file->ioproc = myproc - (myproc % file->ioproc_every); + IOUtil_PrepareFilename (GH, basename, file->filename, called_from, + file->ioproc/file->ioproc_every, file->unchunked); + strcat (file->filename, ".h5"); + + /* Open chunked files on other IO processors */ + if (myproc == file->ioproc && myproc != 0) { + + if (verbose) + CCTK_VInfo (CCTK_THORNSTRING, "Opening chunked file '%s' on " + "processor %d", file->filename, myproc); + + /* Open file, make sure the file is valid */ + file->fid = H5Fopen (file->filename, H5F_ACC_RDONLY, H5P_DEFAULT); + if (file->fid < 0) { + CCTK_VWarn (1, __LINE__, __FILE__, CCTK_THORNSTRING, + "Cannot open file '%s' on processor %d", file->filename, myproc); + file->is_HDF5_file = 0; + } + } +#ifdef CCTK_MPI + /* Finally check whether all files have valid recovery files */ + info [0] = file->is_HDF5_file; + CACTUS_MPI_ERROR (MPI_Allreduce (&info [0], &info [1], 1, + PUGH_MPI_INT4, MPI_LAND, comm)); + file->is_HDF5_file = info [1]; +#endif + } + + /* Return 0 for success otherwise negative */ + return (file->is_HDF5_file ? 0 : -1); +} + + +/* NOTE: Although we could read the GH extensions from multiple recovery files + in parallel, this is done only on by processor 0 here. + Broadcasting the GH extensions is found faster than + sending it in a loop from each IO processor to all the non IOs + (don't have subcommunicators yet) */ +static int IOHDF5i_RecoverGHextensions (cGH *GH, fileinfo_t *file) +{ + hid_t group; + CCTK_REAL4 real4Buffer; + CCTK_INT4 int4Buffer [3]; + extern int CCTK_SetMainLoopIndex (int main_loop_index); + + + if (CCTK_MyProc (GH) == 0) { + /* the GH extensions are stored in the GHEXTENSIONS_GROUP group */ + group = H5Gopen (file->fid, GHEXTENSIONS_GROUP); + int4Buffer [0] = group >= 0; + + if (int4Buffer [0]) { + + READ_ATTRIBUTE (group, "cctk_iteration", IOHDF5_INT4, &int4Buffer [1]); + READ_ATTRIBUTE (group, "main_loop_index", IOHDF5_INT4, &int4Buffer [2]); + READ_ATTRIBUTE (group, "cctk_time", IOHDF5_REAL4, &real4Buffer); + + CACTUS_IOHDF5_ERROR (H5Gclose (group)); + + } else + CCTK_WARN (1, "Can't find global attributes group. " + "Is this really a Cactus HDF5 datafile ?"); + } + +#ifdef CCTK_MPI + /* Broadcast the GH extensions to all processors */ + /* NOTE: We have to use MPI_COMM_WORLD here + because PUGH_COMM_WORLD is not yet set up at parameter recovery time. + We also assume that PUGH_MPI_INT4 is a compile-time defined datatype. */ + CACTUS_MPI_ERROR (MPI_Bcast (int4Buffer, 3, PUGH_MPI_INT4, 0,MPI_COMM_WORLD)); + if (int4Buffer [0]) + CACTUS_MPI_ERROR (MPI_Bcast (&real4Buffer, 1, PUGH_MPI_REAL4, 0, + MPI_COMM_WORLD)); +#endif + + if (int4Buffer [0]) { + GH->cctk_time = real4Buffer; + GH->cctk_iteration = int4Buffer [1]; + CCTK_SetMainLoopIndex ((int) int4Buffer [2]); + } + + /* Return 0 for success otherwise negative */ + return (int4Buffer [0] ? 0 : -1); +} + + +/* NOTE: Although we could read the parameters from multiple recovery files + in parallel, this is done only on by processor 0 here. + Broadcasting the complete parameter string is found faster than + sending it in a loop from each IO processor to all the non IOs + (don't have subcommunicators yet) */ +static int IOHDF5i_RecoverParameters (fileinfo_t *file) +{ + DECLARE_CCTK_PARAMETERS + hid_t group, attr, atype; + char *parameters; + CCTK_INT4 parameterSize=0; + cGH *GH = NULL; /* There's no cGH set up yet so we pass + a NULL pointer to CCTK_MyProc() */ + + /* To make the compiler happy */ + group = attr = 0; + parameters = NULL; + + if (CCTK_MyProc (GH) == 0) { + if (verbose) + CCTK_VInfo (CCTK_THORNSTRING, "Recovering parameters from checkpoint " + "file '%s'", file->filename); + + /* the parameters are stored in the GLOBAL_PARAMETERS_GROUP group + in the attribute GLOBAL_PARAMETERS */ + group = H5Gopen (file->fid, GLOBAL_PARAMETERS_GROUP); + if (group > 0) + attr = H5Aopen_name (group, GLOBAL_PARAMETERS); + + if (group > 0 && attr > 0) { + CACTUS_IOHDF5_ERROR (atype = H5Aget_type (attr)); + parameterSize = H5Tget_size (atype); + parameters = (char *) malloc (parameterSize + 1); + CACTUS_IOHDF5_ERROR (H5Aread (attr, atype, parameters)); + parameters [parameterSize] = 0; + CACTUS_IOHDF5_ERROR (H5Tclose (atype)); + } else + CCTK_WARN (1, "Can't find global parameters. " + "Is this really a Cactus HDF5 checkpoint file ?"); + + if (attr > 0) + CACTUS_IOHDF5_ERROR (H5Aclose (attr)); + if (group > 0) + CACTUS_IOHDF5_ERROR (H5Gclose (group)); + } + +#ifdef CCTK_MPI + /* Broadcast the parameter buffer size to all processors */ + /* NOTE: We have to use MPI_COMM_WORLD here + because PUGH_COMM_WORLD is not yet set up at parameter recovery time. + We also assume that PUGH_MPI_INT4 is a compile-time defined datatype. */ + CACTUS_MPI_ERROR (MPI_Bcast (¶meterSize, 1, PUGH_MPI_INT4, 0, + MPI_COMM_WORLD)); +#endif + + if (parameterSize > 0) { +#ifdef CCTK_MPI + if (CCTK_MyProc (GH) != 0) + parameters = (char *) malloc (parameterSize + 1); + + CACTUS_MPI_ERROR (MPI_Bcast (parameters, parameterSize + 1, PUGH_MPI_CHAR, + 0, MPI_COMM_WORLD)); +#endif + + IOUtil_SetAllParameters (parameters); + + free (parameters); + } + + /* Return positive value for success otherwise negative */ + return (parameterSize > 0 ? 1 : -1); +} diff --git a/src/RestoreFile.c b/src/RestoreFile.c new file mode 100644 index 0000000..78eaa78 --- /dev/null +++ b/src/RestoreFile.c @@ -0,0 +1,580 @@ + /*@@ + @file RestoreFile.c + @date Thu Jun 18 16:34:59 1998 + @author Tom Goodale + @desc + Routines to restore variables from a given file + @enddesc + @history + @hauthor Gabrielle Allen @hdate 19 Oct 1998 + @hdesc Changed names ready for thorn_IO + @endhistory + @version $Id$ + @@*/ + + +#include <stdio.h> +#include <stdlib.h> + +#include "cctk.h" +#include "cctk_Parameters.h" +#include "CactusBase/IOUtil/src/ioGH.h" +#include "CactusPUGH/PUGH/src/include/pugh.h" +#include "ioHDF5GH.h" + + +/* MPI tag base */ +#define STARTUPBASE 1001 + + +typedef struct { + cGH *GH; + int ioproc; + int ioproc_every; + int unchunked; +} iterate_info_t; + +typedef struct { + iterate_info_t *it_info; + int element_size; + int iohdf5_type; +#ifdef CCTK_MPI + MPI_Datatype mpi_type; +#endif +} recover_info_t; + + +/* local function prototypes */ +static herr_t processDataset (hid_t group, const char *datasetname, void *arg); + + + /*@@ + @routine IOHDF5i_RecoverVariables + @date Fri Jun 19 09:19:48 1998 + @author Tom Goodale + @desc + Reads in data from an open HDF5 file. + Code was originally in StartupReader. + @enddesc + @calledby IOHDF5_Recover + @history + @endhistory + @var GH + @vdesc Pointer to CCTK grid hierarchy + @vtype cGH + @vio in + @endvar + @var file + @vdesc HDF5 file info structure + @vtype fileinfo_t * + @vio in + @endvar +@@*/ +int IOHDF5i_RecoverVariables (cGH *GH, fileinfo_t *file) +{ + DECLARE_CCTK_PARAMETERS +#ifdef CCTK_MPI + pGH *pughGH; /* PUGH extension handle */ + int proc; /* looper */ + CCTK_INT var_info [3]; /* communication buffer for MPI */ +#endif + iterate_info_t info; /* info to pass down to the iterator routine */ + + + info.GH = GH; + info.ioproc = file->ioproc; + info.unchunked = file->unchunked; + info.ioproc_every = file->ioproc_every; + +#ifdef CCTK_MPI + /* Get the handle for PUGH extensions */ + pughGH = PUGH_pGH (GH); + + /* Now process the datasets. + All IO processors read the datasets from their checkpoint file, + verify their contents and communicate them to the non-IO processors. */ + + /* At first the code for the IO processors. + This holds also for the single processor case. */ + if (CCTK_MyProc (GH) == file->ioproc) { +#endif + + /* iterate over all datasets starting from "/" in the HDF5 file */ + CACTUS_IOHDF5_ERROR (H5Giterate (file->fid, "/", NULL, processDataset, + &info)); + +#ifdef CCTK_MPI + /* To signal completion to the non-IO processors + an invalid variable index is broadcasted. */ + var_info [0] = -1; + for (proc = 1; proc < file->ioproc_every; proc++) + for (proc = file->ioproc + 1; + proc < file->ioproc + file->ioproc_every && proc < CCTK_nProcs (GH); + proc++) + CACTUS_MPI_ERROR (MPI_Send (var_info, 3, PUGH_MPI_INT, proc, + STARTUPBASE, pughGH->PUGH_COMM_WORLD)); + + } else { + + /* And here the code for non-IO processors: */ + MPI_Datatype mpi_type; + MPI_Status ms; + int index, timelevel, npoints; + + /* They don't know how many datasets there are, because the IO processors + could skip some on the fly during their consistency checks. + The IO Processor sends the index of the variable to be processed next. + So, all non-IO processors execute a loop where the termination condition + is when an invalid index was received. + */ + while (1) { + /* receive the next variable index from my IO processor */ + CACTUS_MPI_ERROR (MPI_Recv (var_info, 3, PUGH_MPI_INT, file->ioproc, + STARTUPBASE, pughGH->PUGH_COMM_WORLD, &ms)); + index = var_info [0]; timelevel = var_info [1]; npoints = var_info [2]; + + /* check for termination condition */ + if (index < 0) + break; + + switch (CCTK_VarTypeI (index)) { + case CCTK_VARIABLE_CHAR: mpi_type = PUGH_MPI_CHAR; break; + case CCTK_VARIABLE_INT: mpi_type = PUGH_MPI_INT; break; + case CCTK_VARIABLE_REAL: mpi_type = PUGH_MPI_REAL; break; + case CCTK_VARIABLE_COMPLEX: mpi_type = pughGH->PUGH_mpi_complex; break; + default: + CCTK_VWarn (1, __LINE__, __FILE__, CCTK_THORNSTRING, + "Unsupported datatype %d", CCTK_VarTypeI (index)); + continue; + } + + /* receive following data from my IO processor */ + CACTUS_MPI_ERROR (MPI_Recv (GH->data [index][timelevel], npoints,mpi_type, + file->ioproc, STARTUPBASE,pughGH->PUGH_COMM_WORLD,&ms)); + } + } +#endif /* MPI */ + + return (0); +} + + +/**************************** local routines ****************************/ + +/* local routine GetCommonAttributes() reads in the next dataset's attributes + and verifies them: + + * checks if there is a variable with the name given by the name attribute + * verifies that this variable still belongs to the same group + * checks the group data info: + - group type + - variable type + - ntimelevels + - sizes (rank, dimensions) according to chunking mode + + If there is a mismatch a warning (warning level 2) is printed and value of 1 + is returned to indicate that this dataset should be ignored. + If successful, the global variable index, the group type and the timelevel + to restore are stored in {*index, *gtype, *timelevel}, and 0 is returned. +*/ + +static int GetCommonAttributes (cGH *GH, hid_t dataset, const char *datasetname, + int unchunked, int *index, int *grouptype, + int *timelevel, int is_group) +{ + pGExtras *extras; + cGroup groupdata; + int i, flag, *dims; + hid_t datatype, dataspace; + hsize_t rank_stored, *dims_stored; + int grouptype_stored, ndims_stored, numtimelevels_stored; + char *groupname, groupname_stored [128], fullvarname [128]; + + /* decompose the datasetname, ignore the iteration number */ + if (sscanf (datasetname, "%[^@]@%d@%d", fullvarname, &i, timelevel) != 3) { + CCTK_VWarn (2, __LINE__, __FILE__, CCTK_THORNSTRING, + "Cannot parse datasetname '%s'", datasetname); + return (1); + } + + /* check if there is a matching variable */ + *index = CCTK_VarIndex (fullvarname); + if (*index < 0) { + CCTK_VWarn (2, __LINE__, __FILE__, CCTK_THORNSTRING, + "No matching variable found for '%s'", fullvarname); + return (1); + } + + /* read and verify the group name */ + READ_ATTRIBUTE (dataset, "groupname", H5T_C_S1, groupname_stored); + groupname = CCTK_GroupNameFromVarI (*index); + if (! CCTK_Equals (groupname_stored, groupname)) { + CCTK_WARN (2, "Groupnames don't match"); + return (1); + } + free (groupname); + + /* verify group type, variable type, dims, sizes and ntimelevels */ + READ_ATTRIBUTE (dataset, "grouptype", H5T_NATIVE_INT, &grouptype_stored); + READ_ATTRIBUTE (dataset, "ntimelevels", H5T_NATIVE_INT,&numtimelevels_stored); + + if (CCTK_GroupData (CCTK_GroupIndex (groupname_stored), &groupdata) != 0) { + CCTK_VWarn (2, __LINE__, __FILE__, CCTK_THORNSTRING, + "Cannot get group data for '%s'", fullvarname); + return (1); + } + if (groupdata.grouptype != grouptype_stored) { + CCTK_VWarn (2, __LINE__, __FILE__, CCTK_THORNSTRING, + "Group types don't match for '%s'", fullvarname); + return (1); + } + if (groupdata.numtimelevels != numtimelevels_stored) { + CCTK_VWarn (2, __LINE__, __FILE__, CCTK_THORNSTRING, + "Number of timelevels don't match for '%s'", fullvarname); + return (1); + } + /* open the first chunk to determine datatype, dims and sizes + if the dataset is a chunk group */ + if (is_group) + CACTUS_IOHDF5_ERROR (dataset = H5Dopen (dataset, "chunk0")); + CACTUS_IOHDF5_ERROR (datatype = H5Dget_type (dataset)); + + /* The CCTK variable type defines do not correlate with the HDF5 defines + so compare them explicitely here. */ + flag = (H5Tget_class (datatype) == H5T_FLOAT && + groupdata.vartype == CCTK_VARIABLE_REAL) || + (H5Tget_class (datatype) == H5T_INTEGER && + groupdata.vartype == CCTK_VARIABLE_INT) || + (H5Tget_class (datatype) == H5T_INTEGER && + groupdata.vartype == CCTK_VARIABLE_CHAR) || + (H5Tget_class (datatype) == H5T_COMPOUND && + groupdata.vartype == CCTK_VARIABLE_COMPLEX); + + CACTUS_IOHDF5_ERROR (H5Tclose (datatype)); + if (! flag) { + CCTK_VWarn (2, __LINE__, __FILE__, CCTK_THORNSTRING, + "Variable types don't match for '%s'", fullvarname); + return (1); + } + + /* verify the dims and sizes */ + CACTUS_IOHDF5_ERROR (dataspace = H5Dget_space (dataset)); + CACTUS_IOHDF5_ERROR (ndims_stored = H5Sget_simple_extent_ndims (dataspace)); + dims_stored = (hsize_t *) malloc (ndims_stored * sizeof (hsize_t)); + CACTUS_IOHDF5_ERROR (rank_stored = H5Sget_simple_extent_dims (dataspace, + dims_stored, NULL)); + /* scalars are stored as rank 0 in HDF5 but have rank 1 in Cactus */ + if (rank_stored == 0) + rank_stored = 1; + CACTUS_IOHDF5_ERROR (H5Sclose (dataspace)); + + flag = 0; + if (groupdata.dim != rank_stored) + flag = 1; + switch (groupdata.grouptype) { + case GROUP_SCALAR: + break; + case GROUP_ARRAY: + case GROUP_GF: + extras = ((pGA ***) PUGH_pGH (GH)->variables)[*index][*timelevel]->extras; + dims = unchunked ? extras->nsize : extras->lnsize; + for (i = 0; i < groupdata.dim; i++) + if (dims [groupdata.dim - i - 1] != dims_stored [i]) + flag = 1; + break; + } + + free (dims_stored); + if (flag) { + CCTK_VWarn (2, __LINE__, __FILE__, CCTK_THORNSTRING, + "Variable sizes don't match for '%s'", fullvarname); + return (1); + } + + if (! CCTK_QueryGroupStorageI (GH, CCTK_GroupIndexFromVarI (*index))) { + CCTK_VWarn (2, __LINE__, __FILE__, CCTK_THORNSTRING, + "Can't read into variable '%s': no storage", fullvarname); + return (1); + } + + /* close the first chunk if the dataset is a chunk group */ + if (is_group) + CACTUS_IOHDF5_ERROR (H5Dclose (dataset)); + + *grouptype = groupdata.grouptype; + + return (0); +} + + +static int IOHDF5_RestoreGS (hid_t dataset, int index, int timelevel, + recover_info_t *rec_info) +{ + void *data; +#ifdef CCTK_MPI + int proc; + CCTK_INT var_info [3]; + pGH *pughGH; +#endif + + + data = CCTK_VarDataPtrI (rec_info->it_info->GH, timelevel, index); + + /* read the data into the local variable ... */ + CACTUS_IOHDF5_ERROR (H5Dread (dataset, rec_info->iohdf5_type, H5S_ALL, + H5S_ALL, H5P_DEFAULT, data)); + +#ifdef CCTK_MPI + /* ... and communicate it for the MPI case */ + + /* Get the handle for PUGH extensions */ + pughGH = PUGH_pGH (rec_info->it_info->GH); + + /* set the variable's index and the timelevel */ + var_info [0] = index; var_info [1] = timelevel; var_info [2] = 1; + + /* send info and data to the non-IO processors */ + for (proc = rec_info->it_info->ioproc + 1; + proc < rec_info->it_info->ioproc + rec_info->it_info->ioproc_every && + proc < CCTK_nProcs (rec_info->it_info->GH); + proc++) { + CACTUS_MPI_ERROR (MPI_Send (var_info, 3, PUGH_MPI_INT, proc, + STARTUPBASE, pughGH->PUGH_COMM_WORLD)); + CACTUS_MPI_ERROR (MPI_Send (data, 1, rec_info->mpi_type, proc, + STARTUPBASE, pughGH->PUGH_COMM_WORLD)); + } +#endif + + return (0); +} + + +static int IOHDF5_RestoreGA (hid_t dataset, int index, int timelevel, + recover_info_t *rec_info) +{ +#ifdef CCTK_MPI + int proc, npoints; + CCTK_INT var_info [3]; + pGH *pughGH; + void *buffer, *data; + hid_t filespace, memspace; + pGExtras *extras; +#endif + + + /* single processor case is easy: just read the whole dataset */ + if (CCTK_nProcs (rec_info->it_info->GH) == 1) { + CACTUS_IOHDF5_ERROR (H5Dread (dataset, rec_info->iohdf5_type, H5S_ALL, + H5S_ALL, H5P_DEFAULT, rec_info->it_info->GH->data [index][timelevel])); + return (0); + } + +#ifdef CCTK_MPI + + /* Get the handle for PUGH extensions */ + pughGH = PUGH_pGH (rec_info->it_info->GH); + + /* Get the pGExtras pointer as a shortcut */ + extras = ((pGA ***) pughGH->variables) [index][timelevel]->extras; + + /* allocate memory for the biggest chunk */ + npoints = extras->rnpoints [rec_info->it_info->ioproc + 1]; + for (proc = 2; proc < rec_info->it_info->ioproc_every; proc++) + if (npoints < extras->rnpoints [rec_info->it_info->ioproc + proc]) + npoints = extras->rnpoints [rec_info->it_info->ioproc + proc]; + buffer = malloc (npoints * rec_info->element_size); + + /* set the variable's index and timelevel to restore */ + var_info [0] = index; var_info [1] = timelevel; + + /* now loop over the group of processors associated to each IO processor */ + for (proc = rec_info->it_info->ioproc; + proc < rec_info->it_info->ioproc + rec_info->it_info->ioproc_every && + proc < CCTK_nProcs (rec_info->it_info->GH); + proc++) { + + /* read own data directly into variable */ + if (proc == rec_info->it_info->ioproc) + data = rec_info->it_info->GH->data [index][timelevel]; + else + data = buffer; + + if (! rec_info->it_info->unchunked) { + hid_t chunk; + char chunkname [32]; + + /* Chunked data is stored as separate chunk datasets within a group. + So open, read and close the separate chunks here. */ + sprintf (chunkname, "chunk%d", proc - rec_info->it_info->ioproc); + CACTUS_IOHDF5_ERROR (chunk = H5Dopen (dataset, chunkname)); + CACTUS_IOHDF5_ERROR (H5Dread (chunk, rec_info->iohdf5_type, H5S_ALL, + H5S_ALL, H5P_DEFAULT, data)); + CACTUS_IOHDF5_ERROR (H5Dclose (chunk)); + + } else { + int i, dim; + hssize_t *chunk_origin; + hsize_t *chunk_dims; + + /* get the dimension of the variable */ + dim = CCTK_GroupDimI (CCTK_GroupIndexFromVarI (index)); + chunk_origin = (hssize_t *) malloc (dim * sizeof (hssize_t)); + chunk_dims = (hsize_t *) malloc (dim * sizeof (hsize_t)); + + /* Unchunked data is read as one hyperslab per processor. + So prepare the memspace and the filespace and read the hyperslab. */ + for (i = 0; i < dim; i++) { + chunk_dims [dim - 1 - i] = extras->rnsize [proc][i]; + chunk_origin [dim - 1 - i] = extras->lb [proc][i]; + } + + CACTUS_IOHDF5_ERROR (filespace = H5Dget_space (dataset)); + CACTUS_IOHDF5_ERROR (memspace = H5Screate_simple (dim, chunk_dims, NULL)); + CACTUS_IOHDF5_ERROR (H5Sselect_hyperslab (filespace, H5S_SELECT_SET, + chunk_origin, NULL, chunk_dims, NULL)); + + CACTUS_IOHDF5_ERROR (H5Dread (dataset, rec_info->iohdf5_type, memspace, + filespace, H5P_DEFAULT, data)); + + CACTUS_IOHDF5_ERROR (H5Sclose (memspace)); + CACTUS_IOHDF5_ERROR (H5Sclose (filespace)); + + free (chunk_dims); + free (chunk_origin); + } + + /* send the index and the data to the non-IO processors */ + if (proc != rec_info->it_info->ioproc) { + var_info [2] = extras->rnpoints [proc]; + CACTUS_MPI_ERROR (MPI_Send (var_info, 3, PUGH_MPI_INT, proc, + STARTUPBASE, pughGH->PUGH_COMM_WORLD)); + CACTUS_MPI_ERROR (MPI_Send (data, extras->rnpoints [proc], + rec_info->mpi_type, proc, STARTUPBASE, pughGH->PUGH_COMM_WORLD)); + } + } + + free (buffer); +#endif + + return (0); +} + + +static herr_t processDataset (hid_t group, const char *datasetname, void *arg) +{ + DECLARE_CCTK_PARAMETERS + pGH *pughGH; + ioGH *ioUtilGH; + ioHDF5GH *myGH; + int index, gtype, timelevel; + iterate_info_t *it_info = (iterate_info_t *) arg; + recover_info_t rec_info; + hid_t dataset; + H5G_stat_t object_info; + int is_group; + + /* skip the global attributes and GH extensions groups */ + if (! strcmp (datasetname, GLOBAL_PARAMETERS_GROUP) || + ! strcmp (datasetname, GHEXTENSIONS_GROUP)) + return (0); + + /* Get the handle for PUGH, IOUtil, and IOHDF5 extensions */ + pughGH = PUGH_pGH (it_info->GH); + ioUtilGH = (ioGH *) it_info->GH->extensions [CCTK_GHExtensionHandle ("IO")]; + myGH = (ioHDF5GH*) it_info->GH->extensions [CCTK_GHExtensionHandle("IOHDF5")]; + + CACTUS_IOHDF5_ERROR (H5Gget_objinfo (group, datasetname, 0, &object_info)); + is_group = object_info.type == H5G_GROUP; + if (is_group) + CACTUS_IOHDF5_ERROR (dataset = H5Gopen (group, datasetname)); + else + CACTUS_IOHDF5_ERROR (dataset = H5Dopen (group, datasetname)); + + /* read in the dataset's attributes and verify them */ + if (GetCommonAttributes (it_info->GH, dataset, datasetname, + it_info->unchunked, &index, >ype, &timelevel, is_group)) { + CCTK_VWarn (1, __LINE__, __FILE__, CCTK_THORNSTRING, + "Ignoring dataset '%s'", datasetname); + return (0); + } + + /* if we read in initial data via the file reader interface + check whether the user wants to have this variable restored */ + if (ioUtilGH->do_inVars && ! ioUtilGH->do_inVars [index]) { + if (verbose) { + char *varname = CCTK_FullName (index); + + CCTK_VInfo (CCTK_THORNSTRING, "Ignoring variable '%s' for file reader " + "recovery", varname); + free (varname); + } + return (0); + } + + if (verbose) { + char *varname = CCTK_FullName (index); + + CCTK_VInfo (CCTK_THORNSTRING, "Restoring variable '%s' (timelevel %d)", + varname, timelevel); + free (varname); + } + + rec_info.it_info = it_info; + + switch (CCTK_VarTypeI (index)) { + case CCTK_VARIABLE_CHAR: + rec_info.iohdf5_type = IOHDF5_CHAR; + rec_info.element_size = sizeof (CCTK_CHAR); +#ifdef CCTK_MPI + rec_info.mpi_type = PUGH_MPI_CHAR; +#endif + break; + case CCTK_VARIABLE_INT: + rec_info.iohdf5_type = IOHDF5_INT; + rec_info.element_size = sizeof (CCTK_INT); +#ifdef CCTK_MPI + rec_info.mpi_type = PUGH_MPI_INT; +#endif + break; + case CCTK_VARIABLE_REAL: + rec_info.iohdf5_type = IOHDF5_REAL; + rec_info.element_size = sizeof (CCTK_REAL); +#ifdef CCTK_MPI + rec_info.mpi_type = PUGH_MPI_REAL; +#endif + break; + case CCTK_VARIABLE_COMPLEX: + rec_info.iohdf5_type = myGH->IOHDF5_COMPLEX; + rec_info.element_size = sizeof (CCTK_COMPLEX); +#ifdef CCTK_MPI + rec_info.mpi_type = pughGH->PUGH_mpi_complex; +#endif + break; + default: + CCTK_VWarn (1, __LINE__, __FILE__, CCTK_THORNSTRING, + "Unsupported datatype %d", CCTK_VarTypeI (index)); + return (0); + } + + /* Read in the data */ + switch (gtype) { + case GROUP_SCALAR: + IOHDF5_RestoreGS (dataset, index, timelevel, &rec_info); + break; + case GROUP_GF: + case GROUP_ARRAY: + IOHDF5_RestoreGA (dataset, index, timelevel, &rec_info); + break; + default: + CCTK_VWarn (1, __LINE__, __FILE__, CCTK_THORNSTRING, + "Unsupported group type %d", gtype); + return (1); + } + + if (is_group) + CACTUS_IOHDF5_ERROR (H5Gclose (dataset)); + else + CACTUS_IOHDF5_ERROR (H5Dclose (dataset)); + + return (0); +} diff --git a/src/Startup.c b/src/Startup.c new file mode 100644 index 0000000..2ec6c14 --- /dev/null +++ b/src/Startup.c @@ -0,0 +1,106 @@ + /*@@ + @file Startup.c + @date Fri May 21 1999 + @author Thomas Radke + @desc + Startup routines for IOHDF5. + @enddesc + @history + @hauthor Thomas Radke @hdate May 21 1999 + @hdesc Just copied from thorn FlexIO. + @endhistory + @@*/ + +#include <stdio.h> + +#include "cctk.h" +#include "cctk_Parameters.h" +#include "CactusBase/IOUtil/src/ioGH.h" + + +/* prototypes of functions to be registered */ +int IOHDF5_Output3DGH(cGH *GH); +int IOHDF5_TriggerOutput3D(cGH *GH, int); +int IOHDF5_TimeFor3D(cGH *GH, int); +int IOHDF5_Output3DVarAs(cGH *GH, const char *var, const char *alias); + +int IOHDF5_Output2DGH(cGH *GH); +int IOHDF5_TriggerOutput2D(cGH *GH, int); +int IOHDF5_TimeFor2D(cGH *GH, int); +int IOHDF5_Output2DVarAs(cGH *GH, const char *var, const char *alias); + +int IOHDF5_Output1DGH(cGH *GH); +int IOHDF5_TriggerOutput1D(cGH *GH, int); +int IOHDF5_TimeFor1D(cGH *GH, int); +int IOHDF5_Output1DVarAs(cGH *GH, const char *var, const char *alias); + + +void *IOHDF5_SetupGH(tFleshConfig *config, int convergence_level, cGH *GH); +int IOHDF5_InitGH(cGH *GH); +int IOHDF5_Recover (cGH *GH, const char *basename, int called_from); + + + /*@@ + @routine IOHDF5_Startup + @date Fri May 21 1999 + @author Thomas Radke + @desc + The startup registration routine for IOHDF5. + Registers the GH extensions needed for IOHDF5 and + the registerable routines used for each method of IOHDF5. + IOHDF5 does not overload any functions. + @enddesc + @calls + @calledby CCTK scheduler at STARTUP + @history + + @endhistory +@@*/ +void IOHDF5_Startup (void) +{ + DECLARE_CCTK_PARAMETERS + int IO_GHExtension; + int IOMethod; + + + if (CCTK_GHExtensionHandle ("IO") < 0) + { + CCTK_WARN (1, "Thorn IOUtil was not activated. " + "No IOHDF5 IO methods will be enabled."); + return; + } + if (CCTK_GHExtensionHandle ("PUGH") < 0) + { + CCTK_WARN (1, "Thorn PUGH was not activated. " + "No IOHDF5 IO methods will be enabled."); + return; + } + + IO_GHExtension = CCTK_RegisterGHExtension ("IOHDF5"); + CCTK_RegisterGHExtensionSetupGH (IO_GHExtension, IOHDF5_SetupGH); + CCTK_RegisterGHExtensionInitGH (IO_GHExtension, IOHDF5_InitGH); + + /* Register the 2D and 3D IOHDF5 routines as output methods */ + IOMethod = CCTK_RegisterIOMethod ("IOHDF5_1D"); + CCTK_RegisterIOMethodOutputGH (IOMethod, IOHDF5_Output1DGH); + CCTK_RegisterIOMethodOutputVarAs (IOMethod, IOHDF5_Output1DVarAs); + CCTK_RegisterIOMethodTimeToOutput (IOMethod, IOHDF5_TimeFor1D); + CCTK_RegisterIOMethodTriggerOutput (IOMethod, IOHDF5_TriggerOutput1D); + + IOMethod = CCTK_RegisterIOMethod ("IOHDF5_2D"); + CCTK_RegisterIOMethodOutputGH (IOMethod, IOHDF5_Output2DGH); + CCTK_RegisterIOMethodOutputVarAs (IOMethod, IOHDF5_Output2DVarAs); + CCTK_RegisterIOMethodTimeToOutput (IOMethod, IOHDF5_TimeFor2D); + CCTK_RegisterIOMethodTriggerOutput (IOMethod, IOHDF5_TriggerOutput2D); + + IOMethod = CCTK_RegisterIOMethod ("IOHDF5_3D"); + CCTK_RegisterIOMethodOutputGH (IOMethod, IOHDF5_Output3DGH); + CCTK_RegisterIOMethodOutputVarAs (IOMethod, IOHDF5_Output3DVarAs); + CCTK_RegisterIOMethodTimeToOutput (IOMethod, IOHDF5_TimeFor3D); + CCTK_RegisterIOMethodTriggerOutput (IOMethod, IOHDF5_TriggerOutput3D); + + /* Register the IOHDF5 recovery routine to thorn IOUtil */ + if (IOUtil_RegisterRecover ("IOHDF5 recovery", IOHDF5_Recover) < 0) + CCTK_WARN (1, "Failed to register IOHDF5 recovery routine"); + +} diff --git a/src/Write1D.c b/src/Write1D.c new file mode 100644 index 0000000..a8b4b62 --- /dev/null +++ b/src/Write1D.c @@ -0,0 +1,179 @@ +/*@@ + @file Write1D.c + @date Fri May 21 1999 + @author Thomas Radke, Gerd Lanfermann + @desc + Output and trigger routines for 1D output into HDF5 files. + @enddesc + @hendhistory + @@*/ + + +#include "cctk.h" +#include "cctk_Parameters.h" +#include "StoreNamedData.h" +#include "CactusBase/IOUtil/src/ioGH.h" +#include "ioHDF5GH.h" + +#include <stdio.h> +#include <stdlib.h> +#include <assert.h> + + +/* local function prototypes */ +void IOHDF5i_DumpParameters (cGH *GH, hid_t group); +void IOHDF5i_DumpGHExtensions (cGH *GH, hid_t group); +char *IOHDF5_Get_filename_ND (cGH *GH, int index, ioHDF5Geo_t *slab_geo, + const char *name, int *isNewFile); +int NumDirection(int sdim, int vdim); +int SetDirection(int vdim, int sdim, int ci, int *direction); + +/*@@ + @routine Write1D + @author Paul Walker + @date Feb 1997 + @calledby IOHDF5_Output1DVarAs, IOHDF5_TriggerOutput1D + @var GH + @vdesc Pointer to CCTK GH + @vtype cGH + @vio in + @vcomment + @endvar + @var index + @vdesc index of variable to output + @vtype int + @vio in + @vcomment + @endvar + @var alias + @vdesc alias name of variable to output + @vtype const char * + @vio in + @vcomment + @endvar +@@*/ +int IOHDF5_Write1D (cGH *GH, int index, const char *alias) +{ + DECLARE_CCTK_PARAMETERS + int timelevel; + ioGH *ioUtilGH; + ioHDF5GH *h5GH; + ioHDF5Geo_t geo; + + hid_t fid, plist; + int isNewFile; + char *filename; + + int SDIM; + int si,max_slabs; + + /* Get the handle for IO extensions */ + ioUtilGH = (ioGH *) GH->extensions [CCTK_GHExtensionHandle ("IO")]; + h5GH = (ioHDF5GH *) GH->extensions [CCTK_GHExtensionHandle ("IOHDF5")]; + + /* The dimension of the slab - HARDCODED */ + SDIM = 1; + + /* Get the slab geometry for 1D slabs */ + geo = h5GH->out_geo[index][SDIM]; + /* Set geo.sdim to announce 1D output */ + geo.sdim = SDIM; + /* get the dimension of the variable */ + geo.vdim = CCTK_GroupDimFromVarI(index); + + if (verbose) + CCTK_VInfo (CCTK_THORNSTRING, "HDF5 %dD-output of variable '%s', " + "downsampling parameters (%d, %d %d)", SDIM, alias, + geo.downs[0],geo.downs[1],geo.downs[2]); + + /* get the current timelevel */ + timelevel = CCTK_NumTimeLevelsFromVarI (index) - 1; + if (timelevel > 0) + timelevel--; + + /* Maximal number of slabs (dimension sdim) in + given volume (dimension vdim) */ + if ((max_slabs=NumDirection(SDIM, geo.vdim))<0) + { + CCTK_VWarn (1, __LINE__, __FILE__, CCTK_THORNSTRING, + "Cannot get valid number of slabs (%d) for given volume (%d).\n", + SDIM, geo.vdim); + return(-1); + } + + /* Loop over all possible choices */ + for (si=0;si<max_slabs;si++) { + + /* Get the next set of slab directions */ + if (SetDirection(geo.vdim, SDIM, si, geo.direction)<0) { + CCTK_VWarn (1, __LINE__, __FILE__, CCTK_THORNSTRING, + "Cannot set direction for slab (#%d) in given volume (%d).\n", + si, geo.vdim); + continue; + } + + /* get the filename for output */ + filename = IOHDF5_Get_filename_ND (GH, index, &geo, + alias, &isNewFile); + + /* open the output file on IO procs no parallel HDF5 */ + if (CCTK_MyProc (GH) != ioUtilGH->ioproc) + fid = -1; + else + { + + if (verbose) + CCTK_VInfo (CCTK_THORNSTRING, "%s HDF5 output file '%s'", + isNewFile ? "Opening" : "Appending", filename); + + CACTUS_IOHDF5_ERROR (plist = H5Pcreate (H5P_FILE_ACCESS)); + + if (isNewFile) + CACTUS_IOHDF5_ERROR (fid = H5Fcreate (filename, H5F_ACC_TRUNC, + H5P_DEFAULT, plist)); + else + CACTUS_IOHDF5_ERROR (fid = H5Fopen (filename, H5F_ACC_RDWR, plist)); + + H5Pclose (plist); + } + + + /* output global attributes */ + if (isNewFile && fid >= 0) { + hid_t group; + + /* all GH extension variables and parameter stuff which is not + specific to any dataset goes into dedicated groups */ + if (out1D_parameters) + { + CACTUS_IOHDF5_ERROR (group = H5Gcreate (fid, GLOBAL_PARAMETERS_GROUP, 0)); + IOHDF5i_DumpParameters (GH, group); + CACTUS_IOHDF5_ERROR (H5Gclose (group)); + } + + if (verbose) + CCTK_INFO ("Dumping GH extensions ..."); + + CACTUS_IOHDF5_ERROR (group = H5Gcreate (fid, GHEXTENSIONS_GROUP, 0)); + IOHDF5i_DumpGHExtensions (GH, group); + CACTUS_IOHDF5_ERROR (H5Gclose (group)); + } + + IOHDF5_DumpVar(GH, index, timelevel, &geo, fid); + + /* close the file */ + if (fid >= 0) + { + if (verbose) + CCTK_INFO ("Closing HDF5 output file from this iteration"); + CACTUS_IOHDF5_ERROR (H5Fclose (fid)); + } + + /* free the filename if it was not registered for re-opening + in IOHDF5_Get1D_filename () */ + if (out1D_septimefiles) + free (filename); + } + return(1); +} + diff --git a/src/Write2D.c b/src/Write2D.c new file mode 100644 index 0000000..6b29d14 --- /dev/null +++ b/src/Write2D.c @@ -0,0 +1,180 @@ +/*@@ + @file Write2D.c + @date Fri May 21 1999 + @author Thomas Radke, Gerd Lanfermann + @desc + Output and trigger routines for 2D output into HDF5 files + @enddesc + @hendhistory + @@*/ + + +#include "cctk.h" +#include "cctk_Parameters.h" +#include "StoreNamedData.h" +#include "CactusBase/IOUtil/src/ioGH.h" +#include "ioHDF5GH.h" + +#include <stdio.h> +#include <stdlib.h> +#include <assert.h> + + +/* local function prototypes */ +void IOHDF5i_DumpParameters (cGH *GH, hid_t group); +void IOHDF5i_DumpGHExtensions (cGH *GH, hid_t group); +char *IOHDF5_Get_filename_ND (cGH *GH, int index, ioHDF5Geo_t *slab_geo, + const char *name, int *isNewFile); +int NumDirection(int sdim, int vdim); +int SetDirection(int vdim, int sdim, int ci, int *direction); + +/*@@ + @routine Write2D + @author Paul Walker + @date Feb 1997 + @calledby IOHDF5_Output2DVarAs, IOHDF5_TriggerOutput2D + @var GH + @vdesc Pointer to CCTK GH + @vtype cGH + @vio in + @vcomment + @endvar + @var index + @vdesc index of variable to output + @vtype int + @vio in + @vcomment + @endvar + @var alias + @vdesc alias name of variable to output + @vtype const char * + @vio in + @vcomment + @endvar +@@*/ +int IOHDF5_Write2D (cGH *GH, int index, const char *alias) +{ + DECLARE_CCTK_PARAMETERS + int timelevel; + ioGH *ioUtilGH; + ioHDF5GH *h5GH; + ioHDF5Geo_t geo; + + hid_t fid, plist; + int isNewFile; + char *filename; + + int SDIM; + int si,max_slabs; + int ierr; + + /* Get the handle for IO extensions */ + ioUtilGH = (ioGH *) GH->extensions [CCTK_GHExtensionHandle ("IO")]; + h5GH = (ioHDF5GH *) GH->extensions [CCTK_GHExtensionHandle ("IOHDF5")]; + + /* The dimension of the slab - HARDCODED */ + SDIM = 2; + + /* Get the slab geometry for 2D slabs */ + geo = h5GH->out_geo[index][SDIM]; + /* Set geo.sdim to announce 2D output */ + geo.sdim = SDIM; + /* get the dimension of the variable */ + geo.vdim = CCTK_GroupDimFromVarI(index); + + if (verbose) + CCTK_VInfo (CCTK_THORNSTRING, "HDF5 %dD-output of variable '%s', " + "downsampling parameters (%d, %d)", SDIM, alias, + geo.downs[0], geo.downs [1]); + + /* get the current timelevel */ + timelevel = CCTK_NumTimeLevelsFromVarI (index) - 1; + if (timelevel > 0) + timelevel--; + + /* Maximal number of slabs (dimension sdim) in + given volume (dimension vdim) */ + if ((max_slabs=NumDirection(SDIM, geo.vdim))<0) + { + CCTK_VWarn (1, __LINE__, __FILE__, CCTK_THORNSTRING, + "Cannot get valid number of slabs (%d) for given volume (%d).\n", + SDIM, geo.vdim); + return(-1); + } + + /* Loop over all possible choices */ + for (si=0;si<max_slabs;si++) { + + /* Get the next set of slab directions */ + if (SetDirection(geo.vdim, SDIM, si, geo.direction)<0) { + CCTK_VWarn (1, __LINE__, __FILE__, CCTK_THORNSTRING, + "Cannot set direction for slab (#%d) in given volume (%d).\n", + si, geo.vdim); + continue; + } + + /* get the filename for output */ + filename = IOHDF5_Get_filename_ND (GH, index, &geo, + alias, &isNewFile); + + /* open the output file on IO procs no parallel HDF5 */ + if (CCTK_MyProc (GH) != ioUtilGH->ioproc) + fid = -1; + else + { + + if (verbose) + CCTK_VInfo (CCTK_THORNSTRING, "%s HDF5 output file '%s'", + isNewFile ? "Opening" : "Appending", filename); + + CACTUS_IOHDF5_ERROR (plist = H5Pcreate (H5P_FILE_ACCESS)); + + if (isNewFile) + CACTUS_IOHDF5_ERROR (fid = H5Fcreate (filename, H5F_ACC_TRUNC, + H5P_DEFAULT, plist)); + else + CACTUS_IOHDF5_ERROR (fid = H5Fopen (filename, H5F_ACC_RDWR, plist)); + + H5Pclose (plist); + } + + + /* output global attributes */ + if (isNewFile && fid >= 0) { + hid_t group; + + /* all GH extension variables and parameter stuff which is not + specific to any dataset goes into dedicated groups */ + if (out2D_parameters) + { + CACTUS_IOHDF5_ERROR (group = H5Gcreate (fid, GLOBAL_PARAMETERS_GROUP, 0)); + IOHDF5i_DumpParameters (GH, group); + CACTUS_IOHDF5_ERROR (H5Gclose (group)); + } + + if (verbose) + CCTK_INFO ("Dumping GH extensions ..."); + + CACTUS_IOHDF5_ERROR (group = H5Gcreate (fid, GHEXTENSIONS_GROUP, 0)); + IOHDF5i_DumpGHExtensions (GH, group); + CACTUS_IOHDF5_ERROR (H5Gclose (group)); + } + + ierr = IOHDF5_DumpVar(GH, index, timelevel, &geo, fid); + + /* close the file */ + if (fid >= 0) + { + if (verbose) + CCTK_INFO ("Closing HDF5 output file from this iteration"); + CACTUS_IOHDF5_ERROR (H5Fclose (fid)); + } + + /* free the filename if it was not registered for re-opening + in IOHDF5_Get2D_filename () */ + if (out2D_septimefiles) + free (filename); + } + return(ierr); +} + diff --git a/src/Write3D.c b/src/Write3D.c new file mode 100644 index 0000000..a4019f5 --- /dev/null +++ b/src/Write3D.c @@ -0,0 +1,195 @@ +/*@@ + @file Write3D.c + @date Fri May 21 1999 + @author Thomas Radke, Gerd Lanfermann + @desc + Output and trigger routines for 3D output into HDF5 files. + Added general hyperslab extraction + @enddesc + @history + @hauthor Thomas Radke @hdate May 21 1999 + @hdesc Just copied from thorn FlexIO. + @hendhistory + @@*/ + + +#include "cctk.h" +#include "cctk_Parameters.h" +#include "StoreNamedData.h" +#include "CactusPUGH/PUGH/src/include/pugh.h" +#include "CactusBase/IOUtil/src/ioGH.h" +#include "ioHDF5GH.h" + +#include <stdio.h> +#include <stdlib.h> +#include <assert.h> + + +/* local function prototypes */ +void IOHDF5i_DumpParameters (cGH *GH, hid_t group); +void IOHDF5i_DumpGHExtensions (cGH *GH, hid_t group); +char *IOHDF5_Get_filename_ND (cGH *GH, int index, ioHDF5Geo_t *slab_geo, + const char *name, int *isNewFile); + + +/*@@ + @routine Write3D + @author Paul Walker + @date Feb 1997 + @calledby IOHDF5_Output3DVarAs, IOHDF5_TriggerOutput3D + @var GH + @vdesc Pointer to CCTK GH + @vtype cGH + @vio in + @vcomment + @endvar + @var index + @vdesc index of variable to output + @vtype int + @vio in + @vcomment + @endvar + @var alias + @vdesc alias name of variable to output + @vtype const char * + @vio in + @vcomment + @endvar +@@*/ +int IOHDF5_Write3D (cGH *GH, int index, const char *alias) +{ + DECLARE_CCTK_PARAMETERS + int timelevel; + ioGH *ioUtilGH; + ioHDF5GH *h5GH; + hid_t fid, plist; + int isNewFile; + char *filename; + int ierr=0; + + ioHDF5Geo_t geo; + int SDIM; + + + /* Get the handle for IO extensions */ + ioUtilGH = (ioGH *) GH->extensions [CCTK_GHExtensionHandle ("IO")]; + h5GH = (ioHDF5GH *) GH->extensions [CCTK_GHExtensionHandle ("IOHDF5")]; + + /* The dimension of the slab - HARDCODED */ + SDIM = 3; + + /* Get the slab geometry for 2D slabs */ + geo = h5GH->out_geo[index][SDIM]; + + /* Set geo.sdim to announce 3d output */ + geo.sdim = SDIM; + + /* get the dimension of the variable */ + geo.vdim = CCTK_GroupDimFromVarI(index); + + /* check if variable has storage assigned */ + if (! CCTK_QueryGroupStorageI (GH, CCTK_GroupIndexFromVarI (index))) + { + char *fullname = CCTK_FullName (index); + CCTK_VWarn (1, __LINE__, __FILE__, CCTK_THORNSTRING, + "No IOHDF5 3D output for '%s' (no storage)", fullname); + free (fullname); + return(-1); + } + if (verbose) + CCTK_VInfo (CCTK_THORNSTRING, "HDF5 %dD-output of variable '%s', " + "downsampling parameters (%d, %d, %d)", SDIM, alias, + geo.downs[0], geo.downs [1], geo.downs[2]); + + /* get the filename for output */ + if (!(filename = IOHDF5_Get_filename_ND (GH, index, &geo, + alias, &isNewFile))) { + char *fullname = CCTK_FullName (index); + CCTK_VWarn (1, __LINE__, __FILE__, CCTK_THORNSTRING, + "Unable to construct file name for '%s'", fullname); + free(fullname); + return(-1); + } + + + /* open the output file on IO procs + or - in case of parallel HDF5 - on all procs */ +#ifdef HAVE_PARALLEL + if (CCTK_MyProc (GH) != ioUtilGH->ioproc && ! ioUtilGH->unchunked) +#else + if (CCTK_MyProc (GH) != ioUtilGH->ioproc) +#endif + { + fid = -1; + } + else + { + + if (verbose) + CCTK_VInfo (CCTK_THORNSTRING, "%s HDF5 output file '%s'", + isNewFile ? "Opening" : "Appending", filename); + + CACTUS_IOHDF5_ERROR (plist = H5Pcreate (H5P_FILE_ACCESS)); +#ifdef CCTK_MPI +#ifdef HAVE_PARALLEL + if (ioUtilGH->unchunked) + CACTUS_IOHDF5_ERROR (H5Pset_mpi (plist, PUGH_pGH (GH)->PUGH_COMM_WORLD, + MPI_INFO_NULL)); +#endif +#endif + + if (isNewFile) + CACTUS_IOHDF5_ERROR (fid = H5Fcreate (filename, H5F_ACC_TRUNC, + H5P_DEFAULT, plist)); + else + CACTUS_IOHDF5_ERROR (fid = H5Fopen (filename, H5F_ACC_RDWR, plist)); + + H5Pclose (plist); + } + + /* get the current timelevel */ + timelevel = CCTK_NumTimeLevelsFromVarI (index) - 1; + if (timelevel > 0) + timelevel--; + + /* output global attributes */ + if (isNewFile && fid >= 0) + { + hid_t group; + + /* all GH extension variables and parameter stuff which is not + specific to any dataset goes into dedicated groups */ + if (out3D_parameters) + { + CACTUS_IOHDF5_ERROR (group = H5Gcreate (fid, GLOBAL_PARAMETERS_GROUP, 0)); + IOHDF5i_DumpParameters (GH, group); + CACTUS_IOHDF5_ERROR (H5Gclose (group)); + } + + if (verbose) + CCTK_INFO ("Dumping GH extensions ..."); + + CACTUS_IOHDF5_ERROR (group = H5Gcreate (fid, GHEXTENSIONS_GROUP, 0)); + IOHDF5i_DumpGHExtensions (GH, group); + CACTUS_IOHDF5_ERROR (H5Gclose (group)); + } + + /* output the data */ + ierr = IOHDF5_DumpVar (GH, index, timelevel, &geo, fid); + + /* close the file */ + if (fid >= 0) + { + if (verbose) + CCTK_INFO ("Closing HDF5 output file from this iteration"); + CACTUS_IOHDF5_ERROR (H5Fclose (fid)); + } + + /* free the filename if it was not registered for re-opening + in IOHDF5_net3D_filename () */ + if (out3D_septimefiles) + free (filename); + + return(ierr); +} + diff --git a/src/WriteIsosurface.c b/src/WriteIsosurface.c new file mode 100644 index 0000000..19c3f51 --- /dev/null +++ b/src/WriteIsosurface.c @@ -0,0 +1,209 @@ +/*@@ + @file WriteIsosurface.c + @date Sat Mar 25 2000 + @author Thomas Radke + @desc + HDF5 output routine for isosurfaces + @enddesc + @history + @hendhistory + @@*/ + + +#include <stdlib.h> +#include <string.h> + +#include "cctk.h" +#include "ioHDF5GH.h" + +#define SOCKET_URL_PREFIX "sock://" +#define SOCKET_URL_PREFIX_LEN (sizeof (SOCKET_URL_PREFIX) - 1) + +/* this struct stores the current slice info for further queries */ +typedef struct { + CCTK_INT iteration; + CCTK_INT timelevel; +} sliceInfo_t; + + +/*@@ + @routine IOHDF5_WriteIsosurface + @author Thomas Radke + @date Sat Mar 25 2000 + @calledby thorn CactusPUGHIO/IsoSurfacer + @var + @vdesc + @vtype + @vio + @vcomment + @endvar +@@*/ +int IOHDF5_WriteIsosurface (cGH *GH, + const char *filename, + const char *GVname, + CCTK_INT iteration, + CCTK_INT timelevel, + CCTK_REAL isoval, + CCTK_REAL minval, + CCTK_REAL maxval, + int nTriangles, + CCTK_INT4 *triangles, + int nVertices, + CCTK_REAL4 *vertices) +{ + hid_t h5file, fapl; + hid_t aspace, dataspace, dataset; + hid_t group0, group1, group2, group3, group4; + hsize_t dim [2]; + char sliceName [40], *isoVarname; + int isSocketURL; + sliceInfo_t *sliceInfo; + hobj_ref_t reference; + static pNamedData *isoFilesList = NULL; + static int first_called = 1; + static hid_t hdf5_string; + + /* FIXME: should use the predefined type from iohdf5GH + but we cannot be sure that thorn IOHDF5 was activated */ + if (first_called) { + first_called = 0; + /* predefine a C string datatype */ + CACTUS_IOHDF5_ERROR (hdf5_string = H5Tcopy (H5T_C_S1)); + } + + CACTUS_IOHDF5_ERROR (fapl = H5Pcreate (H5P_FILE_ACCESS)); + + isSocketURL = strncasecmp (filename, SOCKET_URL_PREFIX, SOCKET_URL_PREFIX_LEN) == 0; + if (isSocketURL) { +#ifndef H5FDstream_H + CCTK_WARN (1, "The configured HDF5 installation doesn't include the Stream " + "VFD. No HDF5 streaming output available !"); + CACTUS_IOHDF5_ERROR (H5Pclose (fapl)); + return (-1); +#else + CACTUS_IOHDF5_ERROR (H5Pset_fapl_stream (fapl, 0, -1)); + sliceInfo = NULL; +#endif + } else + sliceInfo = (sliceInfo_t *) GetNamedData (isoFilesList, filename); + + /* create/open the file */ + if (! sliceInfo) + CACTUS_IOHDF5_ERROR (h5file = H5Fcreate (filename, H5F_ACC_TRUNC, + H5P_DEFAULT, fapl)); + else + CACTUS_IOHDF5_ERROR (h5file = H5Fopen (filename, H5F_ACC_RDWR, fapl)); + if (h5file < 0) { + CCTK_VWarn (1, __LINE__, __FILE__, CCTK_THORNSTRING, + "Could not create/open output file '%s'", filename); + return (-1); + } + + /* create a dataspace for storing scalar attributes */ + CACTUS_IOHDF5_ERROR (aspace = H5Screate (H5S_SCALAR)); + + /* open the highest-level hierarchy which is group "Bundle" */ + if (! sliceInfo) + CACTUS_IOHDF5_ERROR (group0 = H5Gcreate (h5file, "Bundle", 0)); + else + CACTUS_IOHDF5_ERROR (group0 = H5Gopen (h5file, "Bundle")); + + /* next level in hierarchy is the "Slice [<iteration>] [<timelevel>]" group */ + sprintf (sliceName, "Slice [%d] [%d]", (int) iteration, (int) timelevel); + if (! sliceInfo || + sliceInfo->iteration != iteration || sliceInfo->timelevel != timelevel) { + if (sliceInfo) { + sliceInfo->iteration = iteration; + sliceInfo->timelevel = timelevel; + } + CACTUS_IOHDF5_ERROR (group1 = H5Gcreate (group0, sliceName, 0)); + WRITE_ATTRIBUTE ("time", &GH->cctk_time, group1, aspace, 0,IOHDF5_REAL); + WRITE_ATTRIBUTE ("timestep", &iteration, group1, aspace, 0, IOHDF5_INT); + WRITE_ATTRIBUTE ("timelevel", &timelevel, group1, aspace, 0,IOHDF5_INT); + } else + CACTUS_IOHDF5_ERROR (group1 = H5Gopen (group0, sliceName)); + + /* third level is a group for each grid variable with specific isovalue */ + isoVarname = (char *) malloc (strlen (GVname) + 40); + sprintf (isoVarname, "Isolevel [%s] [%.10f]", GVname, (double) isoval); + CACTUS_IOHDF5_ERROR (group2 = H5Gcreate (group1, isoVarname, 0)); + free (isoVarname); + + /* fourth level is group "PointTopology" */ + CACTUS_IOHDF5_ERROR (group3 = H5Gcreate (group2, "PointTopology", 0)); + + /* fifth level is group "Representation" */ + CACTUS_IOHDF5_ERROR (group4 = H5Gcreate (group3, "Representation [CartesianChart]", 0)); + +#if 0 + if (IsoL.Ncolorinfo > 0) { + dim [0] = (hsize_t) IsoL.Ncolorinfo; + CACTUS_IOHDF5_ERROR (dataspace = H5Screate_simple (1, dim, NULL)); + CACTUS_IOHDF5_ERROR (dataset = H5Dcreate (h5file, "ColorMap", + IOHDF5_REAL4, dataspace, H5P_DEFAULT)); + CACTUS_IOHDF5_ERROR (H5Dwrite (dataset, IOHDF5_REAL4, H5S_ALL, + H5S_ALL, H5P_DEFAULT, &IsoL.colorinfo)); + CACTUS_IOHDF5_ERROR (H5Dclose (dataset)); + CACTUS_IOHDF5_ERROR (H5Sclose (dataspace)); + } +#endif + + /* write vertices */ + dim [0] = (hsize_t) nVertices; + dim [1] = 3; + CACTUS_IOHDF5_ERROR (dataspace = H5Screate_simple (2, dim, NULL)); + CACTUS_IOHDF5_ERROR (dataset = H5Dcreate (group4, "Positions", + IOHDF5_REAL4, dataspace, H5P_DEFAULT)); + CACTUS_IOHDF5_ERROR (H5Dwrite (dataset, IOHDF5_REAL4, H5S_ALL, + H5S_ALL, H5P_DEFAULT, vertices)); + WRITE_ATTRIBUTE ("data_min", &minval, dataset, aspace, 0, IOHDF5_REAL); + WRITE_ATTRIBUTE ("data_max", &maxval, dataset, aspace, 0, IOHDF5_REAL); +#if 0 + WRITE_ATTRIBUTE ("instance", &isoindex, isogroup, aspace, 0, IOHDF5_INT); +#endif + CACTUS_IOHDF5_ERROR (H5Rcreate (&reference, group4, "Positions", H5R_OBJECT, -1)); + CACTUS_IOHDF5_ERROR (H5Dclose (dataset)); + CACTUS_IOHDF5_ERROR (H5Sclose (dataspace)); + WRITE_ATTRIBUTE ("Domain", "CartesianChart", group4, aspace, 0, hdf5_string); + CACTUS_IOHDF5_ERROR (H5Gclose (group4)); + CACTUS_IOHDF5_ERROR (H5Gclose (group3)); + + /* fourth level is group "Connectivity" */ + CACTUS_IOHDF5_ERROR (group3 = H5Gcreate (group2, "Connectivity", 0)); + + /* fifth level is group "Representation" */ + CACTUS_IOHDF5_ERROR (group4 = H5Gcreate (group3, "Representation [PointTopology]", 0)); + + /* write polygons */ + dim [0] = (hsize_t) nTriangles; + dim [1] = 3; + CACTUS_IOHDF5_ERROR (dataspace = H5Screate_simple (2, dim, NULL)); + CACTUS_IOHDF5_ERROR (dataset = H5Dcreate (group4, "Positions", + IOHDF5_INT4, dataspace, H5P_DEFAULT)); + CACTUS_IOHDF5_ERROR (H5Dwrite (dataset, IOHDF5_INT4, H5S_ALL, + H5S_ALL, H5P_DEFAULT, triangles)); + CACTUS_IOHDF5_ERROR (H5Dclose (dataset)); + CACTUS_IOHDF5_ERROR (H5Sclose (dataspace)); + WRITE_ATTRIBUTE ("Domain", &reference, group4, aspace, 0, H5T_STD_REF_OBJ); + CACTUS_IOHDF5_ERROR (H5Glink(group4, H5G_LINK_SOFT, + "../../PointTopology/Representation [CartesianChart]/Positions", "Domain")); + CACTUS_IOHDF5_ERROR (H5Sclose (aspace)); + CACTUS_IOHDF5_ERROR (H5Gclose (group4)); + CACTUS_IOHDF5_ERROR (H5Gclose (group3)); + CACTUS_IOHDF5_ERROR (H5Gclose (group2)); + CACTUS_IOHDF5_ERROR (H5Gclose (group1)); + CACTUS_IOHDF5_ERROR (H5Gclose (group0)); + + CACTUS_IOHDF5_ERROR (H5Fclose (h5file)); + + /* if this was a newly created file add it to the list of open files */ + if (! isSocketURL && ! sliceInfo) { + sliceInfo_t *newInfo = (sliceInfo_t *) malloc (sizeof (sliceInfo_t)); + + newInfo->iteration = iteration; + newInfo->timelevel = timelevel; + StoreNamedData (&isoFilesList, filename, newInfo); + } + + return (0); +} diff --git a/src/ioHDF5GH.h b/src/ioHDF5GH.h new file mode 100644 index 0000000..c7134c0 --- /dev/null +++ b/src/ioHDF5GH.h @@ -0,0 +1,212 @@ + /*@@ + @header ioHDF5GH.h + @date Fri May 21 1999 + @author Thomas Radke + @desc + The extensions to the GH structure from IOHDF5. + @history + @hauthor Thomas Radke @hdate May 21 1999 + @hdesc Just copied from thorn FlexIO. + @endhistory + @@*/ + +#ifdef __cplusplus +extern "C" { +#endif + +#include <string.h> +#include <hdf5.h> + +#include "StoreNamedData.h" + + +/* names of the (empty) groups that hold GH extensions or global parameters */ +#define GHEXTENSIONS_GROUP "GHextensions_group" +#define GLOBAL_PARAMETERS_GROUP "global_parameters_group" +#define GLOBAL_PARAMETERS "global_parameters" + +/*****************************************************************************/ +/* some useful macros */ +/*****************************************************************************/ +/* Check error flags from HDF5 calls */ +#define CACTUS_IOHDF5_ERROR(fn_call) \ + do { \ + /*size_t strlen(const char *str);*/ \ + \ + int error_code = fn_call; \ + \ + if (error_code < 0) \ + CCTK_VWarn (1, __LINE__, __FILE__, CCTK_THORNSTRING, \ + "HDF5 call '%s' returned error code %d\n", \ + #fn_call, error_code); \ + } while (0) + +/* macro for writing an attribute */ +#define WRITE_ATTRIBUTE(name, value, dataset, dataspace, dim, datatype) \ + do { \ + hid_t attr; \ + void *val = value; \ + hsize_t arrayDim = dim; \ + \ + if (H5Tget_class (datatype) == H5T_STRING) { \ + int len = strlen ((char *) val); \ + \ + if (len == 0) /* HDF5 doesn't like zero-len strings */ \ + len++; \ + CACTUS_IOHDF5_ERROR (H5Tset_size (datatype, len)); \ + } \ + if (arrayDim > 0) \ + CACTUS_IOHDF5_ERROR (H5Sset_extent_simple (dataspace, 1, \ + &arrayDim, NULL)); \ + CACTUS_IOHDF5_ERROR (attr = H5Acreate (dataset, name, datatype, \ + dataspace, H5P_DEFAULT)); \ + CACTUS_IOHDF5_ERROR (H5Awrite (attr, datatype, val)); \ + CACTUS_IOHDF5_ERROR (H5Aclose (attr)); \ + } while (0); + +/* macro for reading an attribute */ +#define READ_ATTRIBUTE(dataset, attrname, requested_type, buffer) \ + { hid_t attr, attrtype; \ + hsize_t asize = 0; \ + \ + if ((attr = H5Aopen_name (dataset, attrname)) < 0) \ + CCTK_WARN (1, "Can't find " attrname " attribute"); \ + if (requested_type == H5T_C_S1) { \ + CACTUS_IOHDF5_ERROR (attrtype = H5Aget_type (attr)); \ + CACTUS_IOHDF5_ERROR (asize = H5Tget_size (attrtype)); \ + if (asize + 1 >= sizeof (buffer)) \ + CCTK_WARN (1, "Can't read " attrname " attribute (too long)");\ + } else \ + attrtype = requested_type; \ + if (H5Aread (attr, attrtype, buffer) < 0) \ + CCTK_WARN (1, "Can't read " attrname " attribute"); \ + if (requested_type == H5T_C_S1) { \ + ((char *) buffer) [asize] = 0; \ + CACTUS_IOHDF5_ERROR (H5Tclose (attrtype)); \ + } \ + CACTUS_IOHDF5_ERROR (H5Aclose (attr)); \ + } + + +/* define the HDF5 datatypes according to CCTK_??? datatypes */ +#define IOHDF5_REAL4 H5T_NATIVE_FLOAT + +#ifdef CCTK_REAL_PRECISION_16 +#define IOHDF5_REAL H5T_NATIVE_LDOUBLE +#elif CCTK_REAL_PRECISION_8 +#define IOHDF5_REAL H5T_NATIVE_DOUBLE +#elif CCTK_REAL_PRECISION_4 +#define IOHDF5_REAL H5T_NATIVE_FLOAT +#endif + +#define IOHDF5_INT (sizeof (CCTK_INT) == sizeof (int) ? \ + H5T_NATIVE_INT : H5T_NATIVE_SHORT) +#define IOHDF5_INT4 (sizeof (int) == 4 ? \ + H5T_NATIVE_INT : H5T_NATIVE_SHORT) +#define IOHDF5_CHAR H5T_NATIVE_CHAR + +#define IOHDF5_MAXDIM 4 + +/* structure holding necessary information about a recovery file */ +typedef struct { + int is_HDF5_file; /* flag indicating valid file info */ + hid_t fid; /* HDF5 file handle */ + char filename [1024]; /* complete file name for recovery */ + int nprocs; /* number of total processors */ + int ioproc; /* the associated IO processor */ + int ioproc_every; /* how many IO processors there are */ + int unchunked; /* whether data was written chunked or unchunked */ +} fileinfo_t; + +/* structure holding information on the output geometry */ +typedef struct ioHDF5Geo +{ + int vdim; /* dimension of the gridvariable (dynamic) */ + int sdim; /* dimension of the slab (dynamic) */ + int direction[IOHDF5_MAXDIM]; /* direction of slab (by parameter) */ + int slab_start[IOHDF5_MAXDIM]; /* global start index (by parameter) */ + int length[IOHDF5_MAXDIM]; /* requested index slab length (by parameter) */ + int actlen[IOHDF5_MAXDIM]; /* actual index slab length (by PUGHSlab)*/ + int downs[IOHDF5_MAXDIM]; /* downsampling (by parameter) */ +} ioHDF5Geo_t; + +/* IOHDF5 GH extension structure */ +typedef struct IOHDF5GH { + + /* how often to output */ + int out1D_every; + int out2D_every; + int out3D_every; + + /* flags indicating output for var [i] */ + char *do_out1D; + char *do_out2D; + char *do_out3D; + + /* directories in which to output */ + char *outdir1D; + char *outdir2D; + char *outdir3D; + + /* the last iteration output */ + int *out1D_last; + int *out2D_last; + int *out3D_last; + + /* filename database for opened files */ + pNamedData *filenameList; + + /* for restart from recovery: flags indicating whether to check for already + existing groups/datasets to delete before creating them anew */ + char *checkForExistingObjects; + + /* variables holding the original error printing routine and its argument */ + H5E_auto_t printErrorFn; + void *printErrorFnArg; + + /* predefined dataspaces for writing scalar and array attributes */ + hid_t scalarDataspace, arrayDataspace; + + /* predefined datatype for writing CCTK_COMPLEX types */ + hid_t IOHDF5_COMPLEX; + + /* predefined datatype for writing C string string attributes */ + hid_t IOHDF5_STRING; + + /* timer array for checkpointing/recovery */ + int timers[4]; + + /* flag to indicate request for timer output */ + int print_timing_info; + + /* geometry information (downsampling, zooming, etc.) */ + ioHDF5Geo_t **out_geo; + +} ioHDF5GH; + +/* function prototypes */ +int IOHDF5_Write3D (cGH *GH, int index, const char *alias); +int IOHDF5_Write2D (cGH *GH, int index, const char *alias); +int IOHDF5_Write1D (cGH *GH, int index, const char *alias); + +void IOHDF5_DumpParams (cGH *GH, hid_t group); +void IOHDF5_DumpGHExtensions (cGH *GH, hid_t group); +int IOHDF5_DumpVar (cGH *GH, int index, int timelevel, ioHDF5Geo_t *geo, hid_t iof); +void IOHDF5_DumpGH (cGH *GH, int called_from); + +int IOHDF5_WriteIsosurface (cGH *GH, + const char *filename, + const char *GVname, + CCTK_INT iteration, + CCTK_INT timelevel, + CCTK_REAL isoval, + CCTK_REAL minval, + CCTK_REAL maxval, + int nTriangles, + CCTK_INT4 *triangles, + int nVertices, + CCTK_REAL4 *vertices); + +#ifdef __cplusplus +} // extern "C" { +#endif diff --git a/src/make.code.defn b/src/make.code.defn new file mode 100644 index 0000000..42416c2 --- /dev/null +++ b/src/make.code.defn @@ -0,0 +1,10 @@ +# Main make.code.defn file for thorn HFD5IO +# $Header$ + +# Source files in this directory +SRCS = Startup.c GHExtension.c Output3D.c Output2D.c Output1D.c Write3D.c \ +Write2D.c Write1D.c DumpVar.c DumpGH.c RestoreFile.c RecoverGH.c \ +WriteIsosurface.c File.c ParseGeometry.c + +# Subdirectories containing source files +SUBDIRS = diff --git a/src/make.configuration.defn b/src/make.configuration.defn new file mode 100644 index 0000000..fe26de1 --- /dev/null +++ b/src/make.configuration.defn @@ -0,0 +1,20 @@ +# make.configuration.defn for IOHDF5 + +# make sure that IOHDF5 was configured with HDF5 and PUGH + +ifeq ($(strip $(HDF5_LIBS)), ) +$(NAME): MissingHDF5 +.pseudo: MissingHDF5 +MissingHDF5: + @echo "IOHDF5: requires HDF5" + @echo "IOHDF5: Please configure with HDF5 or remove IOHDF5 from Thornlist !" + exit 2 +endif + +ifeq ($(findstring CactusPUGH/PUGH,$(THORNS)),) +.pseudo: MissingPUGHinIOHDF5 +MissingPUGHinIOHDF5: + @echo "IOHDF5: requires PUGH" + @echo "IOHDF5: Please add CactusPUGH/PUGH or remove IOHDF5 from Thornlist !" + exit 2 +endif diff --git a/test/WavetoyID.chkpt.it_0.h5 b/test/WavetoyID.chkpt.it_0.h5 Binary files differnew file mode 100644 index 0000000..fd2f802 --- /dev/null +++ b/test/WavetoyID.chkpt.it_0.h5 diff --git a/test/checkpoint.parfile b/test/checkpoint.parfile new file mode 100644 index 0000000..d337cdd --- /dev/null +++ b/test/checkpoint.parfile @@ -0,0 +1,46 @@ +!DESC "Checkpoint Wavetoy ID in HDF5 file format" + +ActiveThorns = "Time WavetoyF77 IDScalarWave PUGH CartGrid3D IOUtil IOBasic IOASCII IOHDF5" + +# Output directory must be the basename of the recovery parameter file's +# filename +IO::outdir = "test_recover" + +# Checkpoint initial data into a single unchunked checkpoint file +# with basename 'WavetoyID' and place it into the current directory +IO::checkpoint_ID = "yes" +IO::out3D_unchunked = "yes" +IO::out3D_mode = "onefile" +IO::checkpoint_ID_file = "WavetoyID.chkpt" +IO::checkpoint_dir = "." + +# Say that the checkpoint file should be created by the IOHDF5 method +IOHDF5::checkpoint = "yes" + +############################################################################ +# Down from here the original parameter settings from the +# CactusWave/WaveToyF77/test/test_wavetoyf77.par test suite follow + +grid::type = "box" + +time::dtfac = 0.5 + +idscalarwave::initial_data = "gaussian" +idscalarwave::sigma = 0.1 +idscalarwave::radius = 0 + +driver::global_nx = 20 +driver::global_ny = 20 +driver::global_nz = 20 + +cactus::cctk_itlast = 10 + +IOASCII::out1D_every = 2 +IOASCII::out1D_vars = "wavetoy::phi " +IOASCII::out1D_z = "no" + +IOBasic::outScalar_every = 2 +IOBasic::outScalar_vars = "wavetoy::phi " + +IOBasic::outInfo_every = 2 +IOBasic::outInfo_vars = "wavetoy::phi " diff --git a/test/test_recover.par b/test/test_recover.par new file mode 100644 index 0000000..3a6958e --- /dev/null +++ b/test/test_recover.par @@ -0,0 +1,17 @@ +!DESC "Recovery from WaveToy ID checkpoint file in HDF5 file format" + +ActiveThorns = "Time WavetoyF77 IDScalarWave PUGH CartGrid3D IOUtil IOBasic IOASCII IOHDF5" + +# Say that we want to recover ... +# You can also try "auto" and omit the ".it_0" field in the recover_file. +IO::recover = "manual" + +# ... from a ID checkpoint file with basename ... +IO::recover_file = "WavetoyID.chkpt.it_0" + +# ... to be found in directory ... +IO::recovery_dir = "../../arrangements/CactusPUGHIO/IOHDF5/test/" + +# Note that this parameter is restored to "yes" during parameter recovery. +# Since we don't need to checkpoint twice it is explicitely disabled here. +IO::checkpoint_ID = "no" diff --git a/test/test_recover/phi.dl b/test/test_recover/phi.dl new file mode 100644 index 0000000..12f574d --- /dev/null +++ b/test/test_recover/phi.dl @@ -0,0 +1,138 @@ + + +"Time = 0.000000 +-0.866025 0.0000000000000 +-0.774865 0.0000000000000 +-0.683704 0.0000000000000 +-0.592544 0.0000000000000 +-0.501383 0.0000000000121 +-0.410223 0.0000000491566 +-0.319062 0.0000379191081 +-0.227901 0.0055502860213 +-0.136741 0.1541535501874 +-0.045580 0.8124050503725 +0.045580 0.8124050503725 +0.136741 0.1541535501874 +0.227901 0.0055502860213 +0.319062 0.0000379191081 +0.410223 0.0000000491566 +0.501383 0.0000000000121 +0.592544 0.0000000000000 +0.683704 0.0000000000000 +0.774865 0.0000000000000 +0.866025 0.0000000000000 + + +"Time = 0.052632 +-0.866025 0.0000000000000 +-0.774865 0.0000000000000 +-0.683704 0.0000000000000 +-0.592544 0.0000000000000 +-0.501383 -0.0000000000791 +-0.410223 -0.0000002406983 +-0.319062 -0.0000993771711 +-0.227901 -0.0072242068990 +-0.136741 -0.0979217322775 +-0.045580 -0.1950571619856 +0.045580 -0.1950571619856 +0.136741 -0.0979217322775 +0.227901 -0.0072242068990 +0.319062 -0.0000993771711 +0.410223 -0.0000002406983 +0.501383 -0.0000000000791 +0.592544 0.0000000000000 +0.683704 0.0000000000000 +0.774865 0.0000000000000 +0.866025 0.0000000000000 + + +"Time = 0.105263 +-0.866025 0.0000000000000 +-0.774865 0.0000000000000 +-0.683704 0.0000000000000 +-0.592544 -0.0000000000000 +-0.501383 -0.0000000070778 +-0.410223 -0.0000069066610 +-0.319062 -0.0010528818637 +-0.227901 -0.0332682845732 +-0.136741 -0.2986866103570 +-0.045580 -1.1290503258507 +0.045580 -1.1290503258507 +0.136741 -0.2986866103570 +0.227901 -0.0332682845732 +0.319062 -0.0010528818637 +0.410223 -0.0000069066610 +0.501383 -0.0000000070778 +0.592544 -0.0000000000000 +0.683704 0.0000000000000 +0.774865 0.0000000000000 +0.866025 0.0000000000000 + + +"Time = 0.157895 +-0.866025 0.0000000000000 +-0.774865 0.0000000000000 +-0.683704 0.0000000000000 +-0.592544 -0.0000000001229 +-0.501383 -0.0000003859714 +-0.410223 -0.0001219858026 +-0.319062 -0.0070834276763 +-0.227901 -0.0973584266251 +-0.136741 -0.4576544210254 +-0.045580 -0.9019842677611 +0.045580 -0.9019842677611 +0.136741 -0.4576544210254 +0.227901 -0.0973584266251 +0.319062 -0.0070834276763 +0.410223 -0.0001219858026 +0.501383 -0.0000003859714 +0.592544 -0.0000000001229 +0.683704 0.0000000000000 +0.774865 0.0000000000000 +0.866025 0.0000000000000 + + +"Time = 0.210526 +-0.866025 0.0000000000000 +-0.774865 0.0000000000000 +-0.683704 0.0000000000016 +-0.592544 -0.0000000159252 +-0.501383 -0.0000113878002 +-0.410223 -0.0012768875054 +-0.319062 -0.0304556068145 +-0.227901 -0.2006651910238 +-0.136741 -0.4997593550838 +-0.045580 -0.2663944260771 +0.045580 -0.2663944260771 +0.136741 -0.4997593550838 +0.227901 -0.2006651910238 +0.319062 -0.0304556068145 +0.410223 -0.0012768875054 +0.501383 -0.0000113878002 +0.592544 -0.0000000159252 +0.683704 0.0000000000016 +0.774865 0.0000000000000 +0.866025 0.0000000000000 + + +"Time = 0.263158 +-0.866025 0.0000000000000 +-0.774865 0.0000000000002 +-0.683704 -0.0000000003848 +-0.592544 -0.0000008256207 +-0.501383 -0.0001851985238 +-0.410223 -0.0080724403495 +-0.319062 -0.0860002778716 +-0.227901 -0.3020549188440 +-0.136741 -0.2694342257320 +-0.045580 -0.0172399268943 +0.045580 -0.0172399268943 +0.136741 -0.2694342257320 +0.227901 -0.3020549188440 +0.319062 -0.0860002778716 +0.410223 -0.0080724403495 +0.501383 -0.0001851985238 +0.592544 -0.0000008256207 +0.683704 -0.0000000003848 +0.774865 0.0000000000002 +0.866025 0.0000000000000 diff --git a/test/test_recover/phi.xl b/test/test_recover/phi.xl new file mode 100644 index 0000000..55a520c --- /dev/null +++ b/test/test_recover/phi.xl @@ -0,0 +1,138 @@ + + +"Time = 0.000000 +-0.500000 0.0000000000121 +-0.447368 0.0000000017699 +-0.394737 0.0000001488657 +-0.342105 0.0000071951364 +-0.289474 0.0001998375962 +-0.236842 0.0031894000956 +-0.184211 0.0292505776823 +-0.131579 0.1541535501874 +-0.078947 0.4668380576006 +-0.026316 0.8124050503725 +0.026316 0.8124050503725 +0.078947 0.4668380576006 +0.131579 0.1541535501874 +0.184211 0.0292505776823 +0.236842 0.0031894000956 +0.289474 0.0001998375962 +0.342105 0.0000071951364 +0.394737 0.0000001488657 +0.447368 0.0000000017699 +0.500000 0.0000000000121 + + +"Time = 0.052632 +-0.500000 0.0000000001579 +-0.447368 0.0000002442580 +-0.394737 0.0000055185355 +-0.342105 0.0000636892398 +-0.289474 0.0002832523498 +-0.236842 -0.0010800497738 +-0.184211 -0.0182063021811 +-0.131579 -0.0835854975190 +-0.078947 -0.1767183072710 +-0.026316 -0.1950571619856 +0.026316 -0.1950571619856 +0.078947 -0.1767183072710 +0.131579 -0.0835854975190 +0.184211 -0.0182063021811 +0.236842 -0.0010800497738 +0.289474 0.0002832523498 +0.342105 0.0000636892398 +0.394737 0.0000055185355 +0.447368 0.0000002442580 +0.500000 0.0000000001579 + + +"Time = 0.105263 +-0.500000 0.0000000000000 +-0.447368 0.0000074500893 +-0.394737 0.0000571366787 +-0.342105 0.0000891850929 +-0.289474 -0.0021735254063 +-0.236842 -0.0208336611591 +-0.184211 -0.0974778459232 +-0.131579 -0.2995396203857 +-0.078947 -0.6860436385846 +-0.026316 -1.1290503258507 +0.026316 -1.1290503258507 +0.078947 -0.6860436385846 +0.131579 -0.2995396203857 +0.184211 -0.0974778459232 +0.236842 -0.0208336611591 +0.289474 -0.0021735254063 +0.342105 0.0000891850929 +0.394737 0.0000571366787 +0.447368 0.0000074500893 +0.500000 0.0000000000000 + + +"Time = 0.157895 +-0.500000 0.0000000000121 +-0.447368 0.0000404464522 +-0.394737 -0.0000934611932 +-0.342105 -0.0028408356421 +-0.289474 -0.0203343223824 +-0.236842 -0.0840586024960 +-0.184211 -0.2362253936422 +-0.131579 -0.4920251452215 +-0.078947 -0.7713978386790 +-0.026316 -0.9019842677611 +0.026316 -0.9019842677611 +0.078947 -0.7713978386790 +0.131579 -0.4920251452215 +0.184211 -0.2362253936422 +0.236842 -0.0840586024960 +0.289474 -0.0203343223824 +0.342105 -0.0028408356421 +0.394737 -0.0000934611932 +0.447368 0.0000404464522 +0.500000 0.0000000000121 + + +"Time = 0.210526 +-0.500000 0.0000000001579 +-0.447368 -0.0002560735181 +-0.394737 -0.0032223677598 +-0.342105 -0.0191504023972 +-0.289474 -0.0718425508170 +-0.236842 -0.1869773941516 +-0.184211 -0.3531984196036 +-0.131579 -0.4762760737732 +-0.078947 -0.4212470397782 +-0.026316 -0.2663944260771 +0.026316 -0.2663944260771 +0.078947 -0.4212470397782 +0.131579 -0.4762760737732 +0.184211 -0.3531984196036 +0.236842 -0.1869773941516 +0.289474 -0.0718425508170 +0.342105 -0.0191504023972 +0.394737 -0.0032223677598 +0.447368 -0.0002560735181 +0.500000 0.0000000001579 + + +"Time = 0.263158 +-0.500000 0.0000000000000 +-0.447368 -0.0034053134726 +-0.394737 -0.0177523942812 +-0.342105 -0.0616617809179 +-0.289474 -0.1512610524542 +-0.236842 -0.2676581405491 +-0.184211 -0.3314593226319 +-0.131579 -0.2530697030290 +-0.078947 -0.0930373771550 +-0.026316 -0.0172399268943 +0.026316 -0.0172399268943 +0.078947 -0.0930373771550 +0.131579 -0.2530697030290 +0.184211 -0.3314593226319 +0.236842 -0.2676581405491 +0.289474 -0.1512610524542 +0.342105 -0.0616617809179 +0.394737 -0.0177523942812 +0.447368 -0.0034053134726 +0.500000 0.0000000000000 diff --git a/test/test_recover/phi.yl b/test/test_recover/phi.yl new file mode 100644 index 0000000..55a520c --- /dev/null +++ b/test/test_recover/phi.yl @@ -0,0 +1,138 @@ + + +"Time = 0.000000 +-0.500000 0.0000000000121 +-0.447368 0.0000000017699 +-0.394737 0.0000001488657 +-0.342105 0.0000071951364 +-0.289474 0.0001998375962 +-0.236842 0.0031894000956 +-0.184211 0.0292505776823 +-0.131579 0.1541535501874 +-0.078947 0.4668380576006 +-0.026316 0.8124050503725 +0.026316 0.8124050503725 +0.078947 0.4668380576006 +0.131579 0.1541535501874 +0.184211 0.0292505776823 +0.236842 0.0031894000956 +0.289474 0.0001998375962 +0.342105 0.0000071951364 +0.394737 0.0000001488657 +0.447368 0.0000000017699 +0.500000 0.0000000000121 + + +"Time = 0.052632 +-0.500000 0.0000000001579 +-0.447368 0.0000002442580 +-0.394737 0.0000055185355 +-0.342105 0.0000636892398 +-0.289474 0.0002832523498 +-0.236842 -0.0010800497738 +-0.184211 -0.0182063021811 +-0.131579 -0.0835854975190 +-0.078947 -0.1767183072710 +-0.026316 -0.1950571619856 +0.026316 -0.1950571619856 +0.078947 -0.1767183072710 +0.131579 -0.0835854975190 +0.184211 -0.0182063021811 +0.236842 -0.0010800497738 +0.289474 0.0002832523498 +0.342105 0.0000636892398 +0.394737 0.0000055185355 +0.447368 0.0000002442580 +0.500000 0.0000000001579 + + +"Time = 0.105263 +-0.500000 0.0000000000000 +-0.447368 0.0000074500893 +-0.394737 0.0000571366787 +-0.342105 0.0000891850929 +-0.289474 -0.0021735254063 +-0.236842 -0.0208336611591 +-0.184211 -0.0974778459232 +-0.131579 -0.2995396203857 +-0.078947 -0.6860436385846 +-0.026316 -1.1290503258507 +0.026316 -1.1290503258507 +0.078947 -0.6860436385846 +0.131579 -0.2995396203857 +0.184211 -0.0974778459232 +0.236842 -0.0208336611591 +0.289474 -0.0021735254063 +0.342105 0.0000891850929 +0.394737 0.0000571366787 +0.447368 0.0000074500893 +0.500000 0.0000000000000 + + +"Time = 0.157895 +-0.500000 0.0000000000121 +-0.447368 0.0000404464522 +-0.394737 -0.0000934611932 +-0.342105 -0.0028408356421 +-0.289474 -0.0203343223824 +-0.236842 -0.0840586024960 +-0.184211 -0.2362253936422 +-0.131579 -0.4920251452215 +-0.078947 -0.7713978386790 +-0.026316 -0.9019842677611 +0.026316 -0.9019842677611 +0.078947 -0.7713978386790 +0.131579 -0.4920251452215 +0.184211 -0.2362253936422 +0.236842 -0.0840586024960 +0.289474 -0.0203343223824 +0.342105 -0.0028408356421 +0.394737 -0.0000934611932 +0.447368 0.0000404464522 +0.500000 0.0000000000121 + + +"Time = 0.210526 +-0.500000 0.0000000001579 +-0.447368 -0.0002560735181 +-0.394737 -0.0032223677598 +-0.342105 -0.0191504023972 +-0.289474 -0.0718425508170 +-0.236842 -0.1869773941516 +-0.184211 -0.3531984196036 +-0.131579 -0.4762760737732 +-0.078947 -0.4212470397782 +-0.026316 -0.2663944260771 +0.026316 -0.2663944260771 +0.078947 -0.4212470397782 +0.131579 -0.4762760737732 +0.184211 -0.3531984196036 +0.236842 -0.1869773941516 +0.289474 -0.0718425508170 +0.342105 -0.0191504023972 +0.394737 -0.0032223677598 +0.447368 -0.0002560735181 +0.500000 0.0000000001579 + + +"Time = 0.263158 +-0.500000 0.0000000000000 +-0.447368 -0.0034053134726 +-0.394737 -0.0177523942812 +-0.342105 -0.0616617809179 +-0.289474 -0.1512610524542 +-0.236842 -0.2676581405491 +-0.184211 -0.3314593226319 +-0.131579 -0.2530697030290 +-0.078947 -0.0930373771550 +-0.026316 -0.0172399268943 +0.026316 -0.0172399268943 +0.078947 -0.0930373771550 +0.131579 -0.2530697030290 +0.184211 -0.3314593226319 +0.236842 -0.2676581405491 +0.289474 -0.1512610524542 +0.342105 -0.0616617809179 +0.394737 -0.0177523942812 +0.447368 -0.0034053134726 +0.500000 0.0000000000000 diff --git a/test/test_recover/phi_max.tl b/test/test_recover/phi_max.tl new file mode 100644 index 0000000..7d6d6cf --- /dev/null +++ b/test/test_recover/phi_max.tl @@ -0,0 +1,7 @@ +"phi max v time +0.000000 0.8124050503725 +0.052632 0.0002832523498 +0.105263 0.0000891850929 +0.157895 0.0000404464522 +0.210526 0.0000003961856 +0.263158 0.0000001100397 diff --git a/test/test_recover/phi_min.tl b/test/test_recover/phi_min.tl new file mode 100644 index 0000000..a1d8579 --- /dev/null +++ b/test/test_recover/phi_min.tl @@ -0,0 +1,7 @@ +"phi min v time +0.000000 0.0000000000000 +0.052632 -0.1950571619856 +0.105263 -1.1290503258507 +0.157895 -0.9019842677611 +0.210526 -0.4997593550838 +0.263158 -0.3594146186543 diff --git a/test/test_recover/phi_nm1.tl b/test/test_recover/phi_nm1.tl new file mode 100644 index 0000000..895a3b0 --- /dev/null +++ b/test/test_recover/phi_nm1.tl @@ -0,0 +1,7 @@ +"phi norm1 v time +0.000000 0.0047741452163 +0.052632 0.0023586563905 +0.105263 0.0094832901779 +0.157895 0.0166095490059 +0.210526 0.0237365580784 +0.263158 0.0308582835960 diff --git a/test/test_recover/phi_nm2.tl b/test/test_recover/phi_nm2.tl new file mode 100644 index 0000000..0fea72d --- /dev/null +++ b/test/test_recover/phi_nm2.tl @@ -0,0 +1,7 @@ +"phi norm2 v time +0.000000 0.0410842432418 +0.052632 0.0158441957039 +0.105263 0.0637392141088 +0.157895 0.0805995803466 +0.210526 0.0821416890828 +0.263158 0.0821638479495 |