Import the recently announced changes:

Import the recently announced changes:
1. Carpet has now an infrastructure for multiple maps (aka "grid
patches").  Instead of a single grid hierarchy there can now be
several.  This is largely untested, because the remainder of Cactus
cannot handle multiple coordinate systems.

2. The order in which the schedule bins are called has changed.  As Ian
Hawke pointed out, the previous order during time evolution was
inconsistent.  The initial data ordering did not allow for recovering
and was not usable for progressively solving elliptic equations for
initial data.

3. Carpet now supports convergence levels.  The convergence level
specifies by how many factors of two the resolution in the parameter
file should be coarsened (or refined, if negative).  This should make
convergence tests and test runs much easier.  It is, in principle, also
possible to run several convergence levels at once.  This has not been
tested because the remainder of Cactus cannot handle multiple
resolutions.  This will be necessary for a multigrid solver, and also
for having a shadow hierarchy to determine where to refine adaptively.

4. Carpet works together with the new CoordBase domain specification
parameters.  Without these, using convergence levels will lead to very
strange results.

5. The "modes" have changed.  There are now:
        meta mode:      the whole simulation
        global mode:    one convergence level
        level mode:     one refinement level
        singlemap mode: one map on one refinement level
        local mode:     as previously
The whole mode handling has been cleaned up.

6. The regridding thorn has been cleaned up.

7. The kind of prolongation stencil is now determined in Carpet, i.e. at
a fairly hight level, instead of in CarpetLib.

8. The low-order prolongation operators have been made much more
efficient (as have previously the higher-order ones).

9. Assorted smaller changes.

For Carpet users, there should be no major incompatibilities.  The major
improvements are 3 and 4 combined.  Here is an example:

CoordBase::domainsize    = extent
CoordBase::spacing       = gridspacing
CoordBase::zero_origin_x = yes
CoordBase::zero_origin_y = yes
CoordBase::zero_origin_z = yes
CoordBase::xextent       = 20.0
CoordBase::yextent       = 20.0
CoordBase::zextent       = 20.0
CoordBase::dx            = 1.0
CoordBase::dy            = 1.0
CoordBase::dz            = 1.0

CoordBase::boundary_shiftout_x_lower = 1
CoordBase::boundary_shiftout_y_lower = 1
CoordBase::boundary_shiftout_z_lower = 1

Carpet::domain_from_coordbase   = yes
Carpet::convergence_level       = 0

grid::type         = coordbase
grid::domain       = octant
grid::avoid_origin = no

This gives you a grid that extends from the origin ("zero_origin") up to
20.0 with a grid spacing of 1.0.  Symmetry zones and boundary zones are
added automatically.  The "shiftout" says that there is no boundary
point on the origin.  The staggering parameters (not shown) default to
"no".  In order to change the resolution, only the convergence level
has to be adjusted.  Note that the old way of specifying the domain
extent still works.

For Carpet developers, one major change is the new mode handling.  As
described in 5, the looping macros (that loop over all refinement
levels, or all components) have changed.

darcs-hash:20040125135727-07bb3-51c9647c1b5080e7e180b52a1b81fa155cfd19e9.gz

6 files changed, 487 insertions, 280 deletions

diff --git a/Carpet/CarpetIOASCII/interface.ccl b/Carpet/CarpetIOASCII/interface.ccl
index f76c83a95..875c27690 100644
--- a/Carpet/CarpetIOASCII/interface.ccl
+++ b/Carpet/CarpetIOASCII/interface.ccl
@@ -1,5 +1,5 @@
 # Interface definition for thorn CarpetIOASCII
-# $Header: /home/eschnett/C/carpet/Carpet/Carpet/CarpetIOASCII/interface.ccl,v 1.7 2003/09/04 16:23:21 tradke Exp $
+# $Header: /home/eschnett/C/carpet/Carpet/Carpet/CarpetIOASCII/interface.ccl,v 1.8 2004/01/25 14:57:28 schnetter Exp $
 
 implements: IOASCII
 
@@ -14,3 +14,11 @@ uses include header: gdata.hh
 
 uses include header: gf.hh
 uses include header: ggf.hh
+
+
+
+# Vector groups don't work yet in Carpet
+#CCTK_INT next_output_iteration[3] TYPE=scalar
+#CCTK_REAL next_output_time[3] TYPE=scalar
+CCTK_INT next_output_iteration TYPE=array DISTRIB=constant DIM=1 SIZE=3
+CCTK_REAL next_output_time TYPE=array DISTRIB=constant DIM=1 SIZE=3
diff --git a/Carpet/CarpetIOASCII/param.ccl b/Carpet/CarpetIOASCII/param.ccl
index 959a56c69..1ec9d4b1c 100644
--- a/Carpet/CarpetIOASCII/param.ccl
+++ b/Carpet/CarpetIOASCII/param.ccl
@@ -1,5 +1,5 @@
 # Parameter definitions for thorn CarpetIOASCII
-# $Header: /home/eschnett/C/carpet/Carpet/Carpet/CarpetIOASCII/param.ccl,v 1.10 2004/01/13 15:26:58 cott Exp $
+# $Header: /home/eschnett/C/carpet/Carpet/Carpet/CarpetIOASCII/param.ccl,v 1.11 2004/01/25 14:57:28 schnetter Exp $
 
 
 
@@ -7,7 +7,9 @@ shares: IO
 
 USES STRING out_dir
 
+USES KEYWORD out_criterion
 USES CCTK_INT out_every
+USES CCTK_REAL out_dt
 
 
 
@@ -37,6 +39,10 @@ USES CCTK_REAL out_yzplane_x
 
 
 
+USES BOOLEAN new_filename_scheme
+
+
+
 private:
 
 
@@ -101,6 +107,29 @@ CCTK_STRING out3D_vars "Variables to output in 3D ASCII file format" STEERABLE =
 
 
 
+KEYWORD out1D_criterion "Criterion to select 1D ASCII output intervals, overrides out_every" STEERABLE = ALWAYS
+{
+  "never"     :: "Never output"
+  "iteration" :: "Output every so many iterations"
+  "time"      :: "Output every that much coordinate time"
+} "iteration"
+
+KEYWORD out2D_criterion "Criterion to select 2D ASCII output intervals, overrides out_every" STEERABLE = ALWAYS
+{
+  "never"     :: "Never output"
+  "iteration" :: "Output every so many iterations"
+  "time"      :: "Output every that much coordinate time"
+} "iteration"
+
+KEYWORD out3D_criterion "Criterion to select 3D ASCII output intervals, overrides out_every" STEERABLE = ALWAYS
+{
+  "never"     :: "Never output"
+  "iteration" :: "Output every so many iterations"
+  "time"      :: "Output every that much coordinate time"
+} "iteration"
+
+
+
 CCTK_INT out1D_every "How often to do 1D ASCII output, overrides out_every" STEERABLE = ALWAYS
 {
   -1:* :: "Values <= 0 disable 1D output"
@@ -118,6 +147,30 @@ CCTK_INT out3D_every "How often to do 3D ASCII output, overrides out_every" STEE
 
 
 
+REAL out1D_dt "How often to do 1D ASCII output, overrides out_dt" STEERABLE = ALWAYS
+{
+  (0:* :: "In intervals of that much coordinate time"
+   0   :: "As often as possible"
+  -1   :: "Disable output"
+  -2   :: "Default to IO::out_dt"
+} -2
+
+REAL out2D_dt "How often to do 2D ASCII output, overrides out_dt" STEERABLE = ALWAYS
+{
+  (0:* :: "In intervals of that much coordinate time"
+   0   :: "As often as possible"
+  -1   :: "Disable output"
+  -2   :: "Default to IO::out_dt"
+} -2
+
+REAL out3D_dt "How often to do 3D ASCII output, overrides out_dt" STEERABLE = ALWAYS
+{
+  (0:* :: "In intervals of that much coordinate time"
+   0   :: "As often as possible"
+  -1   :: "Disable output"
+  -2   :: "Default to IO::out_dt"
+} -2
+
 BOOLEAN out1D_x "Do 1D IOASCII output in the x-direction" STEERABLE = ALWAYS
 {
 } "yes"
diff --git a/Carpet/CarpetIOASCII/schedule.ccl b/Carpet/CarpetIOASCII/schedule.ccl
index 3c7f55c59..cee9dab8c 100644
--- a/Carpet/CarpetIOASCII/schedule.ccl
+++ b/Carpet/CarpetIOASCII/schedule.ccl
@@ -1,7 +1,16 @@
 # Schedule definitions for thorn CarpetIOASCII
-# $Header: /home/eschnett/C/carpet/Carpet/Carpet/CarpetIOASCII/schedule.ccl,v 1.4 2003/11/05 16:18:37 schnetter Exp $
+# $Header: /home/eschnett/C/carpet/Carpet/Carpet/CarpetIOASCII/schedule.ccl,v 1.5 2004/01/25 14:57:28 schnetter Exp $
+
+storage: next_output_iteration next_output_time
 
 schedule CarpetIOASCIIStartup at STARTUP after IOUtil_Startup
 {
   LANG: C
+  OPTIONS: global
 } "Startup routine"
+
+schedule CarpetIOASCIIInit at INITIAL
+{
+  LANG: C
+  OPTIONS: global
+} "Initialisation routine"
diff --git a/Carpet/CarpetIOASCII/src/ioascii.cc b/Carpet/CarpetIOASCII/src/ioascii.cc
index e8ef28719..d45163d96 100644
--- a/Carpet/CarpetIOASCII/src/ioascii.cc
+++ b/Carpet/CarpetIOASCII/src/ioascii.cc
@@ -30,7 +30,7 @@
 #include "ioascii.hh"
   
 extern "C" {
-  static const char* rcsid = "$Header: /home/eschnett/C/carpet/Carpet/Carpet/CarpetIOASCII/src/ioascii.cc,v 1.55 2003/11/13 16:03:38 schnetter Exp $";
+  static const char* rcsid = "$Header: /home/eschnett/C/carpet/Carpet/Carpet/CarpetIOASCII/src/ioascii.cc,v 1.56 2004/01/25 14:57:28 schnetter Exp $";
   CCTK_FILEVERSION(Carpet_CarpetIOASCII_ioascii_cc);
 }
 
@@ -48,8 +48,6 @@ namespace CarpetIOASCII {
   using namespace std;
   using namespace Carpet;
   
-  bool CheckForVariable (const cGH* const cgh,
-			 const char* const varlist, const int vindex);
   void SetFlag (int index, const char* optstring, void* arg);
   
   template<int D,int DD>
@@ -60,23 +58,34 @@ namespace CarpetIOASCII {
 		   const int time,
 		   const vect<int,D>& org,
 		   const vect<int,DD>& dirs,
-		   const int tl,
 		   const int rl,
-		   const int c,
 		   const int ml,
+		   const int m,
+		   const int c,
+		   const int tl,
 		   const CCTK_REAL coord_time,
 		   const vect<CCTK_REAL,D>& coord_lower,
 		   const vect<CCTK_REAL,D>& coord_upper);
   
   
   
-  int CarpetIOASCIIStartup()
+  void CarpetIOASCIIStartup()
   {
     IOASCII<1>::Startup();
     IOASCII<2>::Startup();
     IOASCII<3>::Startup();
+  }
+  
+  
+  
+  void CarpetIOASCIIInit (CCTK_ARGUMENTS)
+  {
+    DECLARE_CCTK_ARGUMENTS;
     
-    return 0;
+    for (int d=0; d<3; ++d) {
+      next_output_iteration[d] = 0;
+      next_output_time[d] = cctkGH->cctk_time;
+    }
   }
   
   
@@ -158,7 +167,14 @@ namespace CarpetIOASCII {
   {
     DECLARE_CCTK_PARAMETERS;
     
+    assert (is_level_mode());
+    
     const int n = CCTK_VarIndex(varname);
+    if (n<0) {
+      CCTK_VWarn (1, __LINE__, __FILE__, CCTK_THORNSTRING,
+		  "Variable \"%s\" does not exist", varname);
+      return -1;
+    }
     assert (n>=0 && n<CCTK_NumVars());
     const int group = CCTK_GroupIndexFromVarI (n);
     assert (group>=0 && group<(int)Carpet::arrdata.size());
@@ -178,21 +194,23 @@ namespace CarpetIOASCII {
     }
     
     const int grouptype = CCTK_GroupTypeI(group);
-    const int rl = grouptype==CCTK_GF ? reflevel : 0;
+    if (grouptype != CCTK_GF) assert (reflevel==0);
+    
+    const int groupdim = CCTK_GroupDimI(group);
+    assert (outdim <= groupdim);
     
     // Get grid hierarchy extentsion from IOUtil
     const ioGH * const iogh = (const ioGH *)CCTK_GHExtension (cgh, "IO");
     assert (iogh);
     
     // Create the output directory
-    const char* myoutdir = GetStringParameter("out%dD_dir", out_dir);
+    const char* const myoutdir = GetStringParameter("out%dD_dir", out_dir);
     if (CCTK_MyProc(cgh)==0) {
       CCTK_CreateDirectory (0755, myoutdir);
     }
     
     // Loop over all direction combinations
-    vect<int,outdim> dirs;
-    for (int d=0; d<outdim; ++d) dirs[d] = 0;
+    vect<int,outdim> dirs (0);
     
     bool done;
     do {
@@ -247,222 +265,254 @@ namespace CarpetIOASCII {
 	
 	// Skip output if not desired
 	if (desired) {
-	  
-	  // Invent a file name
-	  ostringstream filenamebuf;
-	  filenamebuf << myoutdir << "/" << alias << ".";
-	  for (int d=0; d<outdim; ++d) {
-	    assert (dirs[d]>=0 && dirs[d]<3);
-	    const char* const coords = "xyz";
-	    filenamebuf << coords[dirs[d]];
-	  }
-	  const char* const suffixes = "lpv";
-	  filenamebuf << suffixes[outdim-1];
-	  // we need a persistent temporary here
-	  string filenamestr = filenamebuf.str();
-	  const char* const filename = filenamestr.c_str();
-	  
-	  ofstream file;
-	  
-	  if (CCTK_MyProc(cgh)==0) {
-	    // If this is the first time, then write a nice header on
-	    // the root processor
-	    if (do_truncate[n]) {
-	      struct stat fileinfo;
-	      if (! iogh->recovered
-		  || stat(filename, &fileinfo)!=0) {
-		file.open (filename, ios::out | ios::trunc);
-                if (! file.good()) {
-                  CCTK_VWarn (0, __LINE__, __FILE__, CCTK_THORNSTRING,
-                              "Could not open output file \"%s\" for variable \"%s\"",
-                              filename, varname);
-                }
-		assert (file.good());
-		file << "# " << varname;
-		for (int d=0; d<outdim; ++d) {
-		  file << " " << "xyz"[dirs[d]];
-		}
-		file << " (" << alias << ")" << endl;
-		file << "#" << endl;
-		assert (file.good());
-	      }
-	    }
-	    if (! file.is_open()) {
-	      file.open (filename, ios::out | ios::app);
-	      assert (file.good());
-	    }
-	    file << setprecision(15);
-	    assert (file.good());
-	  }
-	  
-	  assert (outdim <= CCTK_GroupDimI(group));
-	  
-	  // Find the output offset
-	  vect<int,dim> offset(0);
-	  switch (outdim) {
-	  case 1:
-	    switch (dirs[0]) {
-	    case 0:
-	      offset[1] = GetGridOffset (cgh, 2,
-					 "out%dD_xline_yi", "out_xline_yi",
-					 "out%dD_xline_y",  "out_xline_y",
-					 out_xline_y);
-	      offset[2] = GetGridOffset (cgh, 3,
-					 "out%dD_xline_zi", "out_xline_zi",
-					 "out%dD_xline_z",  "out_xline_z",
-					 out_xline_z);
-	      break;
-	    case 1:
-	      offset[0] = GetGridOffset (cgh, 1,
-					 "out%dD_yline_xi", "out_yline_xi",
-					 "out%dD_yline_x",  "out_yline_x",
-					 out_yline_x);
-	      offset[2] = GetGridOffset (cgh, 3,
-					 "out%dD_yline_zi", "out_yline_zi",
-					 "out%dD_yline_z",  "out_yline_z",
-					 out_yline_z);
-	      break;
-	    case 2:
-	      offset[0] = GetGridOffset (cgh, 1,
-					 "out%dD_zline_xi", "out_zline_xi",
-					 "out%dD_zline_x",  "out_zline_x",
-					 out_zline_x);
-	      offset[1] = GetGridOffset (cgh, 2,
-					 "out%dD_zline_yi", "out_zline_yi",
-					 "out%dD_zline_y",  "out_zline_y",
-					 out_zline_y);
-	      break;
-	    default:
-	      assert (0);
-	    }
-	    break;
-	  case 2:
-	    if (dirs[0]==0 && dirs[1]==1) {
-	      offset[2] = GetGridOffset
-		(cgh, 3,
-		 "out%dD_xyplane_zi", "out_xyplane_zi",
-		 "out%dD_xyplane_z",  "out_xyplane_z",
-		 out_xyplane_z);
-	    } else if (dirs[0]==0 && dirs[1]==2) {
-	      offset[1] = GetGridOffset
-		(cgh, 2,
-		 "out%dD_xzplane_yi", "out_xzplane_yi",
-		 "out%dD_xzplane_y",  "out_xzplane_y",
-		 out_xzplane_y);
-	    } else if (dirs[0]==1 && dirs[1]==2) {
-	      offset[0] = GetGridOffset
-		(cgh, 1,
-		 "out%dD_yzplane_xi", "out_yzplane_xi",
-		 "out%dD_yzplane_x",  "out_yzplane_x",
-		 out_yzplane_x);
-	    } else {
-	      assert (0);
-	    }
-	    break;
-	  case 3:
-	    // The offset doesn't matter in this case
-	    break;
-	  default:
-	    assert (0);
-	  }
-	  
-	  // Traverse all components on this refinement and multigrid
-	  // level
-	  BEGIN_COMPONENT_LOOP(cgh, grouptype) {
-	    
-	    const ggf<dim>* ff = 0;
-	    
-	    assert (var < (int)arrdata[group].data.size());
-	    ff = (ggf<dim>*)arrdata[group].data[var];
-	    
-            const int mintl = output_all_timelevels ? 1-num_tl : 0;
-            const int maxtl = 0;
-            for (int tl=mintl; tl<=maxtl; ++tl) {
-              
-              const gdata<dim>* const data
-                = (*ff) (tl, rl, component, mglevel);
-              bbox<int,dim> ext = data->extent();
-              
-              vect<int,dim> lo = ext.lower();
-              vect<int,dim> hi = ext.upper();
-              vect<int,dim> str = ext.stride();
-              
-              // Ignore ghost zones if desired
-              for (int d=0; d<dim; ++d) {
-                bool output_lower_ghosts
-                  = cgh->cctk_bbox[2*d] ? out3D_outer_ghosts : out3D_ghosts;
-                bool output_upper_ghosts
-                  = cgh->cctk_bbox[2*d+1] ? out3D_outer_ghosts : out3D_ghosts;
-                
-                if (! output_lower_ghosts) {
-                  lo[d] += cgh->cctk_nghostzones[d] * str[d];
+          
+	  // Traverse all maps on this refinement and multigrid level
+	  BEGIN_MAP_LOOP(cgh, grouptype) {
+            
+            // Find the output offset
+            ivect offset(0);
+            if (grouptype == CCTK_GF) {
+              switch (outdim) {
+              case 1:
+                switch (dirs[0]) {
+                case 0:
+                  offset[1] = GetGridOffset (cgh, 2,
+                                             "out%dD_xline_yi", "out_xline_yi",
+                                             "out%dD_xline_y",  "out_xline_y",
+                                             out_xline_y);
+                  offset[2] = GetGridOffset (cgh, 3,
+                                             "out%dD_xline_zi", "out_xline_zi",
+                                             "out%dD_xline_z",  "out_xline_z",
+                                             out_xline_z);
+                  break;
+                case 1:
+                  offset[0] = GetGridOffset (cgh, 1,
+                                             "out%dD_yline_xi", "out_yline_xi",
+                                             "out%dD_yline_x",  "out_yline_x",
+                                             out_yline_x);
+                  offset[2] = GetGridOffset (cgh, 3,
+                                             "out%dD_yline_zi", "out_yline_zi",
+                                             "out%dD_yline_z",  "out_yline_z",
+                                             out_yline_z);
+                  break;
+                case 2:
+                  offset[0] = GetGridOffset (cgh, 1,
+                                             "out%dD_zline_xi", "out_zline_xi",
+                                             "out%dD_zline_x",  "out_zline_x",
+                                             out_zline_x);
+                  offset[1] = GetGridOffset (cgh, 2,
+                                             "out%dD_zline_yi", "out_zline_yi",
+                                             "out%dD_zline_y",  "out_zline_y",
+                                             out_zline_y);
+                  break;
+                default:
+                  assert (0);
                 }
-                if (! output_upper_ghosts) {
-                  hi[d] -= cgh->cctk_nghostzones[d] * str[d];
+                break;
+              case 2:
+                if (dirs[0]==0 && dirs[1]==1) {
+                  offset[2] = GetGridOffset
+                    (cgh, 3,
+                     "out%dD_xyplane_zi", "out_xyplane_zi",
+                     "out%dD_xyplane_z",  "out_xyplane_z",
+                     out_xyplane_z);
+                } else if (dirs[0]==0 && dirs[1]==2) {
+                  offset[1] = GetGridOffset
+                    (cgh, 2,
+                     "out%dD_xzplane_yi", "out_xzplane_yi",
+                     "out%dD_xzplane_y",  "out_xzplane_y",
+                     out_xzplane_y);
+                } else if (dirs[0]==1 && dirs[1]==2) {
+                  offset[0] = GetGridOffset
+                    (cgh, 1,
+                     "out%dD_yzplane_xi", "out_yzplane_xi",
+                     "out%dD_yzplane_x",  "out_yzplane_x",
+                     out_yzplane_x);
+                } else {
+                  assert (0);
                 }
+                break;
+              case 3:
+                // The offset doesn't matter in this case
+                break;
+              default:
+                assert (0);
               }
-              ext = bbox<int,dim>(lo,hi,str);
+            } // if grouptype is GF
+            
+            ofstream file;
+            if (CCTK_MyProc(cgh)==0) {
               
-              // coordinates
-              const CCTK_REAL coord_time = cgh->cctk_time;
-              vect<CCTK_REAL,dim> global_lower;
-              vect<CCTK_REAL,dim> coord_delta;
-              if (grouptype == CCTK_GF) {
-                for (int d=0; d<dim; ++d) {
-                  global_lower[d] = cgh->cctk_origin_space[d];
-                  coord_delta[d] = cgh->cctk_delta_space[d] / maxreflevelfact;
+              // Invent a file name
+              ostringstream filenamebuf;
+              if (new_filename_scheme) {
+                filenamebuf << myoutdir << "/" << alias << ".";
+                if (maps > 1) {
+                  cout << Carpet::map << "-";
+                }
+                for (int d=0; d<outdim; ++d) {
+                  const char* const coords = "xyz";
+                  filenamebuf << coords[dirs[d]];
                 }
+// The offsets differ per level
+//                 for (int dd=0; dd<groupdim; ++dd) {
+//                   bool print_dir = true;
+//                   for (int d=0; d<outdim; ++d) {
+//                     print_dir = print_dir && dirs[d] != dd;
+//                   }
+//                   if (print_dir) {
+//                     filenamebuf << "." << offset[dd];
+//                   }
+//                 }
+                filenamebuf << ".asc";
               } else {
-                for (int d=0; d<dim; ++d) {
-                  global_lower[d] = 0.0;
-                  coord_delta[d] = 1.0 / (cgh->cctk_gsh[d] - 1);
+                filenamebuf << myoutdir << "/" << alias << ".";
+                if (maps > 1) {
+                  cout << Carpet::map << "-";
                 }
+                for (int d=0; d<outdim; ++d) {
+                  assert (dirs[d]>=0 && dirs[d]<3);
+                  const char* const coords = "xyz";
+                  filenamebuf << coords[dirs[d]];
+                }
+                const char* const suffixes = "lpv";
+                filenamebuf << suffixes[outdim-1];
               }
-              // Note: don't permute the "coord_delta" and
-              // "data->extent().lower()"
-              // (it seems that for gcc 2.95 you then pick up the
-              // integer operator*)
-              const vect<CCTK_REAL,dim> coord_lower = global_lower + coord_delta * vect<CCTK_REAL,dim>(lo);
-              const vect<CCTK_REAL,dim> coord_upper = global_lower + coord_delta * vect<CCTK_REAL,dim>(hi);
+              // we need a persistent temporary here
+              string filenamestr = filenamebuf.str();
+              const char* const filename = filenamestr.c_str();
               
-              vect<int,dim> offset1;
-              for (int d=0; d<dim; ++d) {
-                assert (cgh->cctk_levoffdenom[d]==1);
-                offset1[d] = (cgh->cctk_levoff[d] + offset[d]) * ext.stride()[d];
+              // If this is the first time, then write a nice header
+              if (do_truncate[n]) {
+                struct stat fileinfo;
+                if (! iogh->recovered
+                    || stat(filename, &fileinfo)!=0) {
+                  file.open (filename, ios::out | ios::trunc);
+                  if (! file.good()) {
+                    CCTK_VWarn (0, __LINE__, __FILE__, CCTK_THORNSTRING,
+                                "Could not open output file \"%s\" for variable \"%s\"",
+                                filename, varname);
+                  }
+                  assert (file.good());
+                  file << "# " << varname;
+                  for (int d=0; d<outdim; ++d) {
+                    file << " " << "xyz"[dirs[d]];
+                  }
+                  file << " (" << alias << ")" << endl;
+                  file << "#" << endl;
+                  assert (file.good());
+                }
               }
-              for (int d=0; d<outdim; ++d) {
-                offset1[dirs[d]] = ext.lower()[dirs[d]];
+              
+              // Open the file
+              if (! file.is_open()) {
+                file.open (filename, ios::out | ios::app);
+                assert (file.good());
               }
               
-              WriteASCII (file, data, ext, n, cgh->cctk_iteration, offset1, dirs,
-                          tl, rl, component, mglevel,
-                          coord_time, coord_lower, coord_upper);
+              file << setprecision(15);
+              assert (file.good());
               
-              // Append EOL after every component
-              if (CCTK_MyProc(cgh)==0) {
-                if (separate_components) {
-                  assert (file.good());
-                  file << endl;
+            } // if on the root processor
+            
+            // Traverse and components on this multigrid and
+            // refinement level and map
+            BEGIN_COMPONENT_LOOP(cgh, grouptype) {
+              
+              const ggf<dim>* ff = 0;
+              
+              assert (var < (int)arrdata[group][Carpet::map].data.size());
+              ff = (ggf<dim>*)arrdata[group][Carpet::map].data[var];
+              
+              const int mintl = output_all_timelevels ? 1-num_tl : 0;
+              const int maxtl = 0;
+              for (int tl=mintl; tl<=maxtl; ++tl) {
+                
+                const gdata<dim>* const data
+                  = (*ff) (tl, reflevel, component, mglevel);
+                ibbox ext = data->extent();
+                
+                ivect lo = ext.lower();
+                ivect hi = ext.upper();
+                ivect str = ext.stride();
+                
+                // Ignore ghost zones if desired
+                for (int d=0; d<dim; ++d) {
+                  bool output_lower_ghosts
+                    = cgh->cctk_bbox[2*d] ? out3D_outer_ghosts : out3D_ghosts;
+                  bool output_upper_ghosts
+                    = cgh->cctk_bbox[2*d+1] ? out3D_outer_ghosts : out3D_ghosts;
+                  
+                  if (! output_lower_ghosts) {
+                    lo[d] += cgh->cctk_nghostzones[d] * str[d];
+                  }
+                  if (! output_upper_ghosts) {
+                    hi[d] -= cgh->cctk_nghostzones[d] * str[d];
+                  }
+                }
+                ext = ibbox(lo,hi,str);
+                
+                // coordinates
+                const CCTK_REAL coord_time = cgh->cctk_time;
+                rvect global_lower;
+                rvect coord_delta;
+                if (grouptype == CCTK_GF) {
+                  for (int d=0; d<dim; ++d) {
+                    global_lower[d] = cgh->cctk_origin_space[d];
+                    coord_delta[d] = cgh->cctk_delta_space[d] / maxreflevelfact;
+                  }
+                } else {
+                  for (int d=0; d<dim; ++d) {
+                    global_lower[d] = 0.0;
+                    coord_delta[d] = 1.0 / (cgh->cctk_gsh[d] - 1);
+                  }
+                }
+                const rvect coord_lower
+                  = global_lower + coord_delta * rvect(lo);
+                const rvect coord_upper
+                  = global_lower + coord_delta * rvect(hi);
+                
+                ivect offset1;
+                for (int d=0; d<dim; ++d) {
+                  assert (cgh->cctk_levoffdenom[d]==1);
+                  offset1[d] = (cgh->cctk_levoff[d] + offset[d]) * ext.stride()[d];
+                }
+                for (int d=0; d<outdim; ++d) {
+                  offset1[dirs[d]] = ext.lower()[dirs[d]];
+                }
+                
+                WriteASCII (file, data, ext, n, cgh->cctk_iteration,
+                            offset1, dirs,
+                            reflevel, mglevel, Carpet::map, component, tl,
+                            coord_time, coord_lower, coord_upper);
+                
+                // Append EOL after every component
+                if (CCTK_MyProc(cgh)==0) {
+                  if (separate_components) {
+                    assert (file.good());
+                    file << endl;
+                  }
                 }
+                assert (file.good());
+                
+              } // for tl
+              
+            } END_COMPONENT_LOOP;
+            
+            // Append EOL after every complete set of components
+            if (CCTK_MyProc(cgh)==0) {
+              if (separate_grids) {
+                assert (file.good());
+                file << endl;
               }
+              file.close();
               assert (file.good());
-              
-            } // for tl
-	    
-	  } END_COMPONENT_LOOP;
-	  
-	  // Append EOL after every complete set of components
-	  if (CCTK_MyProc(cgh)==0) {
-	    if (separate_grids) {
-	      assert (file.good());
-	      file << endl;
-	    }
-	    file.close();
-	    assert (file.good());
-	  }
-	  
-	  assert (! file.is_open());
-	  
+            }
+            
+            assert (! file.is_open());
+            
+          } END_MAP_LOOP;
+          
 	} // if (desired)
 	
       } // if (ascending)
@@ -471,7 +521,7 @@ namespace CarpetIOASCII {
       done = true;
       for (int d=0; d<outdim; ++d) {
 	++dirs[d];
-	if (dirs[d]<CCTK_GroupDimI(group)) {
+	if (dirs[d]<groupdim) {
 	  done = false;
 	  break;
 	}
@@ -490,30 +540,86 @@ namespace CarpetIOASCII {
   
   template<int outdim>
   int IOASCII<outdim>
-  ::TimeToOutput (const cGH* const cgh, const int vindex)
+  ::TimeToOutput (const cGH* const cctkGH, const int vindex)
   {
+    DECLARE_CCTK_ARGUMENTS;
     DECLARE_CCTK_PARAMETERS;
     
     assert (vindex>=0 && vindex<CCTK_NumVars());
     
-    const int myoutevery = GetIntParameter("out%dD_every", out_every);
+    static int output_iteration = -1;
+    static bool output_this_iteration;
+    static vector<bool> output_variables;
     
-    if (myoutevery <= 0) {
-      // Nothing should be output at all
-      return 0;
-    }
+    assert (cctk_iteration >= output_iteration);
+    if (cctk_iteration > output_iteration) {
+      // A new iteration; check whether output should happen
+      
+      output_iteration = cctk_iteration;
+      
+      const char * const myoutcriterion
+        = GetStringParameter("out%dD_criterion", out_criterion);
+      if (CCTK_EQUALS (myoutcriterion, "never")) {
+        
+        // Never output
+        output_this_iteration = false;
+        
+      } else if (CCTK_EQUALS (myoutcriterion, "iteration")) {
+        
+        const int myoutevery = GetIntParameter("out%dD_every", out_every);
+        if (myoutevery <= 0) {
+          output_this_iteration = false;
+        } else {
+          output_this_iteration
+            = cctk_iteration >= next_output_iteration[outdim-1];
+          if (output_this_iteration) {
+            next_output_iteration[outdim-1] += myoutevery;
+          }
+        }
+        
+      } else if (CCTK_EQUALS (myoutcriterion, "time")) {
+        
+        const CCTK_REAL myoutdt = GetRealParameter("out%dD_dt", out_dt);
+        if (myoutdt < 0) {
+          output_this_iteration = false;
+        } else if (myoutdt == 0) {
+          output_this_iteration = true;
+        } else {
+          output_this_iteration = cctk_time >= (next_output_time[outdim-1]
+                                                - 1.0e-12 * cctk_delta_time);
+          if (output_this_iteration) {
+            next_output_time[outdim-1] += myoutdt;
+          }
+        }
+        
+      } else {
+        
+        assert (0);
+        
+      } // select output criterion
+      
+      
+      
+      // check which variables to output
+      const int numvars = CCTK_NumVars();
+      assert (vindex>=0 && vindex<numvars);
+      output_variables.resize (numvars);
+      
+      const char * const varlist = GetStringParameter("out%dD_vars", "");
+      CCTK_TraverseString
+        (varlist, SetFlag, &output_variables, CCTK_GROUP_OR_VAR);
+      
+    } // check whether output should happen
+    
+    if (! output_this_iteration) return 0;
+    if (! output_variables[vindex]) return 0;
+    
+    const int grouptype = CCTK_GroupTypeFromVarI(vindex);
+    if (grouptype != CCTK_GF && reflevel > 0) return 0;
     
-    if (cgh->cctk_iteration % myoutevery != 0) {
-      // Nothing should be output during this iteration
-      return 0;
-    }
     
-    if (! CheckForVariable(cgh, GetStringParameter("out%dD_vars", ""), vindex)) {
-      // This variable should not be output
-      return 0;
-    }
     
-    if (last_output[reflevel][vindex] == cgh->cctk_iteration) {
+    if (last_output[reflevel][vindex] == cctk_iteration) {
       // Has already been output during this iteration
       char* varname = CCTK_FullName(vindex);
       CCTK_VWarn (5, __LINE__, __FILE__, CCTK_THORNSTRING,
@@ -525,7 +631,7 @@ namespace CarpetIOASCII {
       return 0;
     }
     
-    assert (last_output[reflevel][vindex] < cgh->cctk_iteration);
+    assert (last_output[reflevel][vindex] < cctk_iteration);
     
     // Should be output during this iteration
     return 1;
@@ -559,7 +665,7 @@ namespace CarpetIOASCII {
   {
     // First choice: explicit coordinate
     char cparam[1000];
-    sprintf (cparam, ctempl, outdim);
+    snprintf (cparam, sizeof cparam, ctempl, outdim);
     const int ncparam = CCTK_ParameterQueryTimesSet (cparam, CCTK_THORNSTRING);
     assert (ncparam >= 0);
     if (ncparam > 0) {
@@ -575,7 +681,7 @@ namespace CarpetIOASCII {
     
     // Second choice: explicit index
     char iparam[1000];
-    sprintf (iparam, itempl, outdim);
+    snprintf (iparam, sizeof iparam, itempl, outdim);
     const int niparam = CCTK_ParameterQueryTimesSet (iparam, CCTK_THORNSTRING);
     assert (niparam >= 0);
     if (niparam > 0) {
@@ -629,7 +735,7 @@ namespace CarpetIOASCII {
   {
     assert (dir>=1 && dir<=dim);
     
-    assert (reflevel!=-1 && mglevel!=-1);
+    assert (mglevel!=-1 && reflevel!=-1 && Carpet::map!=-1);
     
     const int npoints = cgh->cctk_gsh[dir-1];
     const CCTK_REAL delta = cgh->cctk_delta_space[dir-1] / cgh->cctk_levfac[dir-1];
@@ -640,10 +746,13 @@ namespace CarpetIOASCII {
     int cindex = (int)floor(rindex + 0.75);
     
     if (cindex<0 || cindex>=npoints) {
-      CCTK_VWarn (1, __LINE__, __FILE__, CCTK_THORNSTRING,
-		  "The specified coordinate value %g is not within the grid range [%g,%g]",
-		  coord, lower, upper);
       cindex = ifallback;
+      
+      assert (dir>=1 && dir<=3);
+      CCTK_VWarn (1, __LINE__, __FILE__, CCTK_THORNSTRING,
+                  "The specified coordinate value %g for the %c-direction is not within the grid range [%g,%g] on convergence level %d, level %d, map %d; using %g instead",
+                  coord, "xyz"[dir-1], lower, upper,
+                  mglevel, reflevel, Carpet::map, lower + delta * cindex);
     }
     
     assert (cindex>=0 && cindex<npoints);
@@ -659,7 +768,7 @@ namespace CarpetIOASCII {
 			const char* const fallback)
   {
     char parametername[1000];
-    sprintf (parametername, parametertemplate, outdim);
+    snprintf (parametername, sizeof parametername, parametertemplate, outdim);
     const int ntimes =
       CCTK_ParameterQueryTimesSet (parametername, CCTK_THORNSTRING);
     assert (ntimes >= 0);
@@ -669,7 +778,7 @@ namespace CarpetIOASCII {
 	(parametername, CCTK_THORNSTRING, &ptype);
       assert (ppval);
       const char* const pval = *ppval;
-      assert (ptype == PARAMETER_STRING);
+      assert (ptype == PARAMETER_STRING || ptype == PARAMETER_KEYWORD);
       return pval;
     }
     
@@ -679,22 +788,23 @@ namespace CarpetIOASCII {
   
   
   template<int outdim>
-  int IOASCII<outdim>
-  ::GetIntParameter (const char* const parametertemplate, const int fallback)
+  CCTK_INT IOASCII<outdim>
+  ::GetIntParameter (const char* const parametertemplate,
+                     const CCTK_INT fallback)
   {
     char parametername[1000];
-    sprintf (parametername, parametertemplate, outdim);
+    snprintf (parametername, sizeof parametername, parametertemplate, outdim);
     const int ntimes =
       CCTK_ParameterQueryTimesSet (parametername, CCTK_THORNSTRING);
     assert (ntimes >= 0);
     if (ntimes > 0) {
       int ptype;
-      const int* const ppval
-	= (const int*)CCTK_ParameterGet
+      const CCTK_INT* const ppval
+	= (const CCTK_INT*)CCTK_ParameterGet
         (parametername, CCTK_THORNSTRING, &ptype);
       assert (ppval);
-      const int pval = *ppval;
-      assert (ptype == PARAMETER_INT);
+      const CCTK_INT pval = *ppval;
+      assert (ptype == PARAMETER_INT || ptype == PARAMETER_BOOLEAN);
       return pval;
     }
     
@@ -703,19 +813,32 @@ namespace CarpetIOASCII {
   
   
   
-  bool CheckForVariable (const cGH* const cgh,
-			 const char* const varlist, const int vindex)
+  template<int outdim>
+  CCTK_REAL IOASCII<outdim>
+  ::GetRealParameter (const char* const parametertemplate,
+                      const CCTK_REAL fallback)
   {
-    const int numvars = CCTK_NumVars();
-    assert (vindex>=0 && vindex<numvars);
-    
-    vector<bool> flags(numvars);
-    
-    CCTK_TraverseString (varlist, SetFlag, &flags, CCTK_GROUP_OR_VAR);
+    char parametername[1000];
+    snprintf (parametername, sizeof parametername, parametertemplate, outdim);
+    const int ntimes =
+      CCTK_ParameterQueryTimesSet (parametername, CCTK_THORNSTRING);
+    assert (ntimes >= 0);
+    if (ntimes > 0) {
+      int ptype;
+      const CCTK_REAL* const ppval
+	= (const CCTK_REAL*)CCTK_ParameterGet
+        (parametername, CCTK_THORNSTRING, &ptype);
+      assert (ppval);
+      const CCTK_REAL pval = *ppval;
+      assert (ptype == PARAMETER_REAL);
+      return pval;
+    }
     
-    return flags[vindex];
+    return fallback;
   }
   
+  
+  
   void SetFlag (int index, const char* optstring, void* arg)
   {
     vector<bool>& flags = *(vector<bool>*)arg;
@@ -733,10 +856,11 @@ namespace CarpetIOASCII {
 		   const int time,
 		   const vect<int,D>& org,
 		   const vect<int,DD>& dirs,
-		   const int tl,
 		   const int rl,
-		   const int c,
 		   const int ml,
+                   const int m,
+		   const int c,
+		   const int tl,
 		   const CCTK_REAL coord_time,
 		   const vect<CCTK_REAL,D>& coord_lower,
 		   const vect<CCTK_REAL,D>& coord_upper)
@@ -753,8 +877,12 @@ namespace CarpetIOASCII {
 	assert (os.good());
         
 	os << "# iteration " << time << endl
-	   << "# time level " << tl << "   refinement level " << rl
-	   << "   component " << c << "   multigrid level " << ml << endl
+	   << "# refinement level " << rl
+           << "   multigrid level " << ml
+	   << "   map " << m
+	   << "   component " << c
+           << "   time level " << tl
+           << endl
 	   << "# column format: it   tl rl c ml  ";
 	assert (D>=1 && D<=3);
 	const char* const coords = "xyz";
@@ -770,25 +898,25 @@ namespace CarpetIOASCII {
 	
 	// Check whether the output origin is contained in the extent
 	// of the data that should be output
-	vect<int,dim> org1(org);
+	ivect org1(org);
 	for (int d=0; d<DD; ++d) org1[dirs[d]] = ext.lower()[d];
 	if (gfext.contains(org1)) {
 	  
 	  for (typename bbox<int,DD>::iteratorT it=ext.beginT();
 	       it!=ext.endT(); ++it) {
-	    vect<int,dim> index(org);
+	    ivect index(org);
 	    for (int d=0; d<DD; ++d) index[dirs[d]] = (*it)[d];
 	    os << time << "   " << tl << " " << rl << " " << c << " " << ml
-	       << "   ";
-	    for (int d=0; d<D; ++d) os << index[d] << " ";
-	    os << "  " << coord_time << "   ";
+               << "  ";
+	    for (int d=0; d<D; ++d) os << " " << index[d];
+	    os << "   " << coord_time << "  ";
 	    for (int d=0; d<D; ++d) {
 	      assert (gfext.upper()[d] - gfext.lower()[d] != 0);
-	      os << (coord_lower[d] + (index[d] - gfext.lower()[d])
-		     * (coord_upper[d] - coord_lower[d])
-		     / (gfext.upper()[d] - gfext.lower()[d])) << " ";
+	      os << " " << (coord_lower[d] + (index[d] - gfext.lower()[d])
+                            * (coord_upper[d] - coord_lower[d])
+                            / (gfext.upper()[d] - gfext.lower()[d]));
 	    }
-	    os << "  ";
+	    os << "   ";
 	    switch (CCTK_VarTypeI(vi)) {
 #define WANT_NO_COMPLEX
 #define TYPECASE(N,T)					\
@@ -837,7 +965,7 @@ namespace CarpetIOASCII {
       for (comm_state<dim> state; !state.done(); state.step()) {
         tmp->copy_from (state, gfdata, gfdata->extent());
       }
-      WriteASCII (os, tmp, gfext, vi, time, org, dirs, tl, rl, c, ml,
+      WriteASCII (os, tmp, gfext, vi, time, org, dirs, rl, ml, m, c, tl,
 		  coord_time, coord_lower, coord_upper);
       delete tmp;
       
@@ -861,10 +989,11 @@ namespace CarpetIOASCII {
 		   const int time,
 		   const vect<int,3>& org,
 		   const vect<int,1>& dirs,
-		   const int tl,
 		   const int rl,
-		   const int c,
 		   const int ml,
+		   const int m,
+		   const int c,
+		   const int tl,
 		   const CCTK_REAL coord_time,
 		   const vect<CCTK_REAL,3>& coord_lower,
 		   const vect<CCTK_REAL,3>& coord_upper);
@@ -877,10 +1006,11 @@ namespace CarpetIOASCII {
 		   const int time,
 		   const vect<int,3>& org,
 		   const vect<int,2>& dirs,
-		   const int tl,
 		   const int rl,
-		   const int c,
 		   const int ml,
+		   const int m,
+		   const int c,
+		   const int tl,
 		   const CCTK_REAL coord_time,
 		   const vect<CCTK_REAL,3>& coord_lower,
 		   const vect<CCTK_REAL,3>& coord_upper);
@@ -893,10 +1023,11 @@ namespace CarpetIOASCII {
 		   const int time,
 		   const vect<int,3>& org,
 		   const vect<int,3>& dirs,
-		   const int tl,
 		   const int rl,
-		   const int c,
 		   const int ml,
+		   const int m,
+		   const int c,
+		   const int tl,
 		   const CCTK_REAL coord_time,
 		   const vect<CCTK_REAL,3>& coord_lower,
 		   const vect<CCTK_REAL,3>& coord_upper);
diff --git a/Carpet/CarpetIOASCII/src/ioascii.h b/Carpet/CarpetIOASCII/src/ioascii.h
index d2e5c549b..acd3ffb80 100644
--- a/Carpet/CarpetIOASCII/src/ioascii.h
+++ b/Carpet/CarpetIOASCII/src/ioascii.h
@@ -1,15 +1,18 @@
-/* $Header: /home/eschnett/C/carpet/Carpet/Carpet/CarpetIOASCII/src/ioascii.h,v 1.5 2003/11/05 16:18:37 schnetter Exp $ */
+/* $Header: /home/eschnett/C/carpet/Carpet/Carpet/CarpetIOASCII/src/ioascii.h,v 1.6 2004/01/25 14:57:29 schnetter Exp $ */
 
 #ifndef CARPETIOASCII_H
 #define CARPETIOASCII_H
 
+#include "cctk_Arguments.h"
+
 #ifdef __cplusplus
 namespace CarpetIOASCII {
   extern "C" {
 #endif
     
     /* Scheduled functions */
-    int CarpetIOASCIIStartup (void);
+    void CarpetIOASCIIStartup (void);
+    void CarpetIOASCIIInit (CCTK_ARGUMENTS);
     
 #ifdef __cplusplus
   } /* extern "C" */
diff --git a/Carpet/CarpetIOASCII/src/ioascii.hh b/Carpet/CarpetIOASCII/src/ioascii.hh
index 504e7586f..c95b3ad71 100644
--- a/Carpet/CarpetIOASCII/src/ioascii.hh
+++ b/Carpet/CarpetIOASCII/src/ioascii.hh
@@ -1,4 +1,4 @@
-// $Header: /home/eschnett/C/carpet/Carpet/Carpet/CarpetIOASCII/src/ioascii.hh,v 1.13 2002/09/01 14:52:25 schnetter Exp $
+// $Header: /home/eschnett/C/carpet/Carpet/Carpet/CarpetIOASCII/src/ioascii.hh,v 1.14 2004/01/25 14:57:29 schnetter Exp $
 
 #ifndef CARPETIOASCII_HH
 #define CARPETIOASCII_HH
@@ -64,9 +64,12 @@ namespace CarpetIOASCII {
     
     static const char* GetStringParameter
     (const char* parametertemplate, const char* fallback);
-    static int GetIntParameter (const char* parametertemplate, int fallback);
+    static CCTK_INT GetIntParameter
+    (const char* parametertemplate, CCTK_INT fallback);
+    static CCTK_REAL GetRealParameter
+    (const char* parametertemplate, CCTK_REAL fallback);
     
-  };				// struct IOASCII
+  };                            // struct IOASCII
   
 } // namespace CarpetIOASCII