Carpet Version 4

Version 4 will be the next release of Carpet, which is still under development. This page describes the current state of the development version and lists the changes from Version 3.

Main contributors to this version were, so far, Erik Schnetter (LSU), Thomas Radke (AEI), and Christian D. Ott (UA). Special thanks go to Luca Baiotti, Denis Pollney, Christian Reisswig, Jian Tao, and also to the CCT numerical relativity group, the AEI numerical relativity group, and the XiRel collaboration.

Major changes

• The data structures and algorithms storing and handling the communication schedule are much more efficient on large numbers (several hundred or more) processors. This makes Carpet scale to more than 8,000 cores.

• The interface for defining and making dynamic changes to grid hierarchies is simpler, and buffer zones are handled in a cleaner manner. This makes it easier to write user code which defines or updates the grid hierarchy, and reduces the chance of inconsistencies therein.

• During checkpointing and recovery, the grid structure is saved and restored by default. This avoids accidental changes upon recovery.

• The efficiency of I/O has been increased, especially for HDF5 based binary I/O. It is possible to combine several variables into one file to reduce the number of output files.

• A new thorn LoopControl offers iterators over grid points, implemented as C-style macros. These iterators allow additional important loop-level optimisations, such as loop tiling or OpenMP parallelisation. Efficient cache handling and hybrid communication models have a large potential for performance improvements on current and future architectures.

Changes in Detail

Initial Data

• The parameter Carpet::enable_all_storabe is off by default, reducing memory usage. This can uncover hidden errors in schedule.ccl files. You can use poisoning (Carpet::poison_new_timelevels = yes) to find these problems.

• The parameter Carpet::regrid_in_level_mode is on by default. This balances the load more efficiently between processors, especially for multi-patch simulations.

• Carpet requires thorn InitBase. This allows Carpet to check whether its handling of initial data is consistent with other thorns, assuming that the other thorns also use InitBase. A new parameter Carpet::init_fill_timelevels instructs Carpet to initialise all past time levels by making copies of the current time level, very similar to what MoL::initial_data_is_crap does.

• Buffer zones are specified differently. Buffer zones are enabled by setting use_buffer_zones, and outer buffer zones are now the default. (Outer buffer zones are added to the outside of the refined levels, instead of taking away grid points from the interior of the refined levels.) The necessary number of buffer zones is calculated automatically if thorn MoL is used for time evolution.

• There are new schedule bins CCTK_PREREGRIDINITIAL and CCTK_POSTREGRIDINITIAL, called before and after regridding during initial data setup, respectively. As a rule of thumb, routines scheduled in CCTK_BASEGRID must also be scheduled in both CCTK_POSTREGRID and CCTK_POSTREGRIDINITIAL. You can use poisoning (see above) to catch errors in your schedule.ccl files.

Grid Structure

• Thorn CarpetRegrid2 stores its state in grid variables instead of in parameters. This makes it easier to modify the grid hierarchy from other thorns, and is automatically consistent upon checkpointing and recovery. It also simplifies outputting the grid structure to files. These state variables can be modified at any time, but their values are only significant during regridding.

• There is a new parameter CarpetRegrid2::symmetry_rotating180, which forces the generated grid structure to have the correct symmetry for a 180 degree rotational symmetry about the z axis. (It is unfortunate that errors in the symmetry of the grid structure do not lead to errors but are currently silently ignored. You can use poisoning -- see above -- to catch errors in the grid structure.) A parameter CarpetRegrid2::symmetry_rotating90 is planned, but not yet implemented.

• Checkpointing saves and restores the grid structure by default.

I/O

• All Carpet I/O thorns support the new parameter one_file_per_group, which combines all variables from a variable group into a single file. This reduces the number of output files.

Communication, Mesh Refinement

• The algorithm determining the communication schedule is new and much more fussy. It will detect all (one hopes) inconsistencies in the grid structure. (Note that this does unfortunately not include errors regarding the outer or symmetry boundaries.)

• The communication schedule is stored in a compressed (sparse) form. This is necessary since the communication schedule between N processors has N² entries, which cannot be handled efficiently in full.

• The prolongation operators have been rewritten. Time interpolation is now handled separately, reducing the number of operators.

• Some unused CarpetLib parameters have been removed, some new CarpetLib parameters have been introduced. The default settings should be reasonable.

New Thorns

• Carpet uses the new thorn CactusBase/InitBase. Update from the Cactus CVS repository to obtain this thorn.

• A new thorn Carpet/CarpetMask can be used to exclude certain regions, such as e.g. the interior of black holes, from global reduction operations. This allows meaningful norms of constraints when Turduckened initial data are used.

• Carpet offers a new thorn Carpet/LoopControl providing iterators over grid points. These iterators are implemented as C-style macros which can replace for or do loops over grid functions. LoopControl provides loop tiling for increased cache efficiency, and OpenMP parallelisation to reduce parallelisation overhead on multi-core machines.

Go back to the Carpet home page.