Old News
New News...
|
March 30, 2009: We have ported Carpet to
the BlueGene/P
architecture, using
the Surveyor
system at the ALCF. The
graph to the right shows preliminary performance and scaling
results, comparing different compilers and options
(gcc, IBM's
XL compilers without OpenMP, and IBM's XL compilers
with OpenMP, which required
reducing the optimisation level). For these benchmarks, the
problem size was reduced to about one eighth of the standard
size, using 133 grid points per core. The results
show that Carpet scales fine up to the size of the total machine
(4k cores), but further work on compiler options is
required.
|
|
|
March 20, 2009: Carpet can now perform performance
experiments by artificially increasing the size or the
number of MPI messages exchanged between processes. This can
help determine whether either the communication bandwidth or the
communication latency are a bottleneck of a particular
simulation. The figure to the right shows results for the
standard McLachlan
AMR benchmark run on
the Cray XT4
Kraken, using 253 grid points per core. These
results indicate that the additional latency from increasing the
number of messages has no significant effect, and hence the
benchmark is bandwidth limited for this problem size.
|
|
|
March 16, 2009: Erik Schnetter and Steve Brandt
published a white
paper Relativistic
Astrophysics on the SiCortex Architecture. This paper
expands on a
webinar
by Erik and Steve that was hosted
by SiCortex.
The graph at the right shows Carpet's parallel scalability
using
the McLachlan
code with nine levels of AMR for a set of current HPC systems.
The results have been rescaled to the architectures' theoretical
single-core peak performance. This makes it possible to compare
Carpet's scalability on different architectures. (It is not
possible to compare the systems' absolute performance in this
figure.)
|
|
|
November 9, 2008: In the context of
the XiRel project,
we re-designed Carpet's communication layer to avoid many
operations that had a cost of O(N), growing linearly
with the number of MPI processes. Such costs are generally not
acceptable when running on several thousand cores, and have to
be reduced e.g. to O(log N). Carpet now stores the
communication schedule (mostly) in a distributed manner,
increasing performance and reducing its memory requirement.
These improvements are currently being tested; preliminary
scaling results are shown in the figure to the right.
|
|
June 25, 2008: We are happy to announce
the Simulation
Factory, a tool to help access remote HPC systems,
manage source trees, and submit and control simulations. The
Simulation Factory contains a set of abstractions of the tasks
which are necessary to set up and successfully finish numerical
simulations using the Cactus framework. These abstractions hide
tedious low-level management tasks, they capture "best
practices" of experienced users, and they create a log trail
ensuring repeatable and well-documented scientific results.
Using these abstractions, many types of potentially disastrous
user errors are avoided, and different supercomputers can be
used in a uniform manner.
|
March 29, 2008: We have benchmarked McLachlan, a new
BSSN-type vacuum Einstein code, using Carpet for unigrid and AMR
calculations. We compare several current large machines:
Franklin
(NERSC), Queen
Bee (LONI),
and Ranger
(TACC).
|
|
|
March 1, 2008: We have improved the development version
of Carpet significantly:
The data structures and algorithms storing and handling
the communication schedule are much more efficient on large
numbers (several hundred or more) processors. This makes Carpet
scale to more than 8,000 cores.
The interface for defining and making dynamic changes to
grid hierarchies is simpler, and buffer zones are handled in a
cleaner manner. This makes it easier to write user code which
defines or updates the grid hierarchy, and reduces the chance of
inconsistencies therein.
During checkpointing and recovery, the grid structure is
saved and restored by default. This avoids accidental changes
upon recovery.
The efficiency of I/O has been increased, especially for
HDF5 based binary I/O. It is possible to combine several
variables into one file to reduce the number of output
files.
A new thorn LoopControl offers iterators over grid
points, implemented as C-style macros. These iterators allow
additional important loop-level optimisations, such
as loop
tiling or
OpenMP parallelisation.
Efficient cache handling and hybrid communication models have a
large potential for performance improvements on current and
future architectures.
More details can be found here.
These improvements are largely due
to Erik Schnetter
(LSU),
Thomas Radke (AEI), and
Christian D. Ott
(UA). Special thanks go to Christian Reisswig and Luca
Baiotti.
March 1, 2008: The development version of Carpet is now
maintained using git instead
of darcs. Git offers a very
similar set of features to darcs, most importantly supporting
decentralised development. Git has a much larger user community
than darcs, and we hope that this makes it easier to use.
The download instructions contain
details on using git to obtain Carpet, and point to further
information. (The darcs repository for the development version
will not see any further changes.)
March 1, 2008: The repository for the development
version of Carpet moved today to
a new server. The
stable versions of Carpet continue to be served from the old
server for the time being. We plan to move all of carpetcode.org
to this new server in the future. The new server is a courtesy
of Christian
D. Ott.
|
January 14, 2008: Carpet's communication
infrastructure has been improved significantly, making Carpet
scale to at least 4,000 processors, including mesh refinement.
Using "friendly user time"
on Ranger,
the new 60,000
core TeraGrid
supercomputer
at TACC, we measured
the benchmark results below for a numerical relativity kernel
solving the BSSN equations. These benchmarks emply a hybrid
communication scheme
combining MPI
and
OpenMP, using the shared
memory capabilities of Ranger's nodes to reduce the memory
overhead of parallelisation. We are grateful for the help we
received from Ranger's support team.
The graph below shows weak scaling tests for both unigrid and
mesh refinement benchmarks. The problem size per core was
kept fixed, and there were 4 OpenMP threads per MPI process,
with 1 MPI process per socket. The benchmark was also run
with the PUGH driver for comparison for certain core counts.
As the graphs show, this benchmark scales near perfectly for
unigrid, and has only small variations in run time for nine
levels of mesh refinement.
|
|
October 3, 2007: Carpet's timing infrastructure has been
extended to automatically measure both time spent computing and
time spent in I/O. The performance of large simulations depends
not only on the computational efficiency and communication
latency, but also on the throughput to file servers. These new
statistics give a real-time overview and can point out
performance problems. The statistics are collected in the
existing Carpet::timing variables.
August 30, 2007: So far this year, ten of the
publications from three research groups examining the dynamics
of binary black hole systems are based on simulations performed
with Cactus and Carpet:
Astrophys. J. 661, 430-436 (2007)
(arXiv:gr-qc/0701143)
Phys. Rev. Lett. 99, 041102 (2007)
(arXiv:gr-qc/0701163)
Astrophys. J. 659, L5-L8 (2007)
(arXiv:gr-qc/0701164)
Phys. Rev. Lett. 98, 231102 (2007)
(arXiv:gr-qc/0702133)
Class. Quantum Grav. 24, 3911-3918 (2007)
(arXiv:gr-qc/0701038)
arXiv:0705.3829 [gr-qc]
arXiv:0706.2541 [gr-qc]
arXiv:0707.2559 [gr-qc]
arXiv:0708.3999 [gr-qc]
arXiv:0708.4048 [gr-qc]
These publications mainly examine the spin dynamics and the
gravitational wave recoil in BBH systems. Since not all
research groups use Cactus and Carpet, this represents only part
of the published work on this subject.
|
August 26, 2007: In experiments with hybrid
communication schemes
combining MPI
and
OpenMP, we found a 20%
speed improvement when using a single node
of Abe
at NCSA, and a
substantial scaling improvement when using 1024 and more CPUs.
(Abe has 8 CPUs per node.) These experiments included cache
optimisations when traversing the 3D arrays. The tests were
performed with a modified version of
the Cactus WaveToy
example application without using I/O or analysis methods.
|
|
August 15, 2007: We are happy to hear that our
proposal ALPACA: Cactus tools for Application Level Profiling
And Correctness Analysis will be funded by
NSF's SDCI
programme for three years.
The ALPACA
project is aiming at developing complex, collaborative
scientific applications, appropriate for highly scalable
hardware architectures, providing fault tolerance, advanced
debugging, and transparency against new developments in
communication, programming, and execution models. Such tools
are especially rare at the application level, where they are
most critically needed.
July 31, 2007: We are happy to hear that our
proposal XiRel: Cyberinfrastructure for Numerical
Relativity will be funded by
NSF's PIF
programme for three
years. XiRel is
collaborative proposal
by LSU, PSU,
and UTB
(now RIT). The central goal of
XiRel is the development of a highly scalable, efficient, and
accurate adaptive mesh refinement layer based on the current
Carpet driver, which will be fully integrated and supported in
Cactus and optimised for numerical relativity.
February 26, 2007: The thorn LSUPETSc
implements a generic elliptic solver for Carpet's multi-patch
infrastructure, based
on PETSc.
It assumes touching (not overlapping) patches, and uses
inter-patch interface conditions very similar to those developed
by Harald
Pfeiffer. LSUPETSc can solve "arbitrary" systems
of coupled, non-linear elliptic equations. It does not support
mesh refinement.
January 12, 2007: In order to be able to restructure
some of Carpet's internals without disturbing ongoing production
simulations, we have created an experimental version.
The main goals of this experimental version are to improve its
performance on many (>100) processors and to re-arrange some
internal details to simplify future development. Few new
features are planned, but some of the changes may be
incompatible.
December 15, 2006:
The AEI hosted a small
workshop to improve the performance of the AEI/LSU CCATIE code
for binary black hole simulations, which uses Carpet as AMR
driver. We examined especially the effect of various grid
structures on accuracy and speed and speeded up the wave
extraction routine. We were able to improve the overall
performance of the code by a factor of six for a certain
benchmark problem simulating a QC-0 configuration.
September 26, 2006: We are preparing a new release of
Carpet. This will be Carpet version 3. Among other things, this
version makes it easier to use dynamic grid structures, shows
better scaling behaviour than version 2, and has better support
for multiple patches. A detailed list of changes
is here. The
the downloading instructions
for Carpet explain how to access this version.
February 26, 2006: We have started to collect
a list of publications and theses
that use Carpet. Please tell us if you have written a publication
or a thesis using Carpet.
February 25,
2006: Christian Ott
has contributed code to Carpet, making the refined regions track
apparent horizon centroids, merging and un-merging refined regions
as necessary. (Movie, animated gif,
730 kB.) After Burkhard Zink's mechanism
which tracks the
density maximum in a star, this is the second implementation
of a production level adaptive mesh refinement criterion in
Carpet.
February 25, 2006: The
official Cactus
benchmarks now include benchmarks with Carpet. You can assess
Carpet's scaling and compare its performance on different machines
by generating graphs from the benchmark result database on these
pages.
July 15, 2005: We have now a page that links to all past montly status reports.
June 6, 2005: We have updated the downloading instructions for
Carpet.
June 6, 2005: Version 1.0.3 of the pre-compiled darcs
binary is now available.
April 13, 2005: Thomas Radke has implemented a new
communication scheme in Carpet. Instead of sending many small
messages in an interleaved manner, Carpet now collects all
messages into an internal buffer and sends only one big message
with MPI. This circumvents certain problems with internal
limitations of MPICH, and it also improves the performance
greatly.
March 9, 2005: We have started to move towards a new
stable version of Carpet.
December 7, 2004: Jonathan Thornburg is organising a Carpet Design Walkthrough,
which will take place December 13 to 15 at the AEI and will be
broadcast via the accessgrid and/or telephone.
Sepbember 18, 2004: There is now a new repository for
the development version of Carpet. This repository is managed by
darcs instead of CVS. Darcs has a number of
advantages, such as being able to use it while offline, or keeping
some changes to yourself while developing. This development
version is publicly available,
and we encourage you to contribute. Note that the stable
version of Carpet is still distributed via CVS.
August 24, 2004: The version of Carpet in the CVS
repository is now stable. That means that this version will see
no substantial further development. One of its main goal is to
not change, so that parameter files continue to work unchanged
with identical results. We will continue to correct errors that
we find in this version of Carpet; however, if this would
necessitate major changes, and there is a work-around, then this
might not happen in the interest of stability.
July 31, 2004: Carpet seems to have reached a point
where it is stable enough to be useful for at least some projects.
Consequently, people expressed the wish to have a version of
Carpet which is stable and sees no disrupting development. The
idea is to have two "branches" of Carpet: a stable version for
production use, and a development version which might not be as
stable. We plan to make the split in about three weeks. The
discussion about this is held on the mailing list; your input is
welcome.
April 7, 2004: Up to now, all Carpet thorns have been
living in a single arrangement for Cactus. This caused problems,
because stable thorns, development thorns, and outdated thorns
were sitting next to each other, confusing newcomers. We have moved the Carpet arrangement to a
new repository and split it into four. Access to the old Carpet
arrangement has been disabled.
March 3, 2004: We have recently had trouble with I/O
throuth the FlexIO
library. We suspect that it might have a bug that causes HDF5
output to fail under certain, random conditions. We have written
a new thorn CarpetIOHDF5 which uses the HDF5 library directly,
while remaining compatible to the FlexIO file format. Please test
this thorn, and report any problems or incompatibilities you
find.
In January 2004, Daniel
Kobras set up Bugzilla
for Carpet. Bugzilla
is a bug-tracking system that will, so we hope, help us remember
what is missing or broken in Carpet.
In October 2003, Erik Schnetter, Scott H. Hawley, and
Ian Hawke published the preprint "Evolutions in 3D numerical
relativity using fixed mesh refinement" as gr-qc/0310042. Its
main point is to present tests of Carpet with the BSSN code (AEI's
spacetime evolution code), and to show that mesh refinement does
not introduce instabilities.
In August 2003, these web pages were
created.
May 2003 has informally been termed "Carpet month". In a flurry of
activity, bugs were fixed and some features added. The BSSN code
of the numerical relativity group at the AEI now works together with
Carpet.
Erik Schnetter
Last modified: Feb 15 2011
