path: root/doc/FAQ

Cactus Code Frequently Asked Questions

$Header: /mnt/data2/cvs2svn/cvs-repositories/Cactus/doc/FAQ,v 1.54 2003-02-01 00:06:50 allen Exp $

Also available at http://www.cactuscode.org/Documentation/FAQ


==============================================================================
The Questions:


------------------------------------------------------------------------------
Background:

A1  How did Cactus start?

A2  Who develops and maintains Cactus?

A3  How can I get involved?


------------------------------------------------------------------------------
Distributions and CVS Checkout:

B1  When doing a cvs update I get 'not enough slashes in ...' .

B2  I want to recreate a Cactus executable from last year, I have a thorn list
    but when I try and check it out from the /cactus or /cactusdevcvs 
    repository I don't get some of the thorns.

B3  How do I checkout an earlier version of Cactus ?

B4  When running GetCactus it tells me it is refusing to update my thorn
    because it is installed from a different repository.


------------------------------------------------------------------------------
Configuring and Building:

C1  The make system says "Checking status of libXXX.a" and then just 
    sits there.  What can I do ?

C2  Compilation of Cactus fails with the error:
	 identifier "DECLARE_CCTK_PARAMETERS" is undefined?

C3 (Linux) I get 'g77: cannot specify -o with -c or -S and multiple
   compilations' when compiling with the Pacific VAST90 compiler.

C4  How do I compile with MPI? Why isn't it automatic like in Cactus 3?

C5  I can't compile because the compiler complains that a routine name has 
    been previously defined ... but I can't find it repeated?

C6  How can I make sure that one thorn is compiled before another? For example 
    to make sure that F90 module files from  ArrangementA/ThornA are available 
    for ArrangementB/ThornB.

C7  (Solaris) I start a configuration process with gmake, but on the second 
    gmake it wants to set up the configuration again and again .... 

C8  How do I compile Cactus in 32 bit mode on a 64 bit irix machine?

C9  What can I do if I don't have enough disk space to compile Cactus?

C10 When I compile Cactus I get an error with "unterminated character 
    constants", but on typing gmake again it proceeds without problem.

C11 Sometimes when I compile in parallel with TJOBS or FJOBS the compilation
    stops halfway through.

C12 I get the following error when compiling?

    Checking status of libIOJpeg.a
    gmake[3]: *** No rule to make target \
         `/u1/allen/scratch/Cactus/arrangements/
         CactusIO/IOJpeg/src/IOJpeg.h', needed by `JPEG.c.d'.  Stop.
    gmake[2]: *** [make.checked] Error 2
    gmake[1]: *** [/u1/allen/scratch/Cactus/configs/wave/lib/libIOJpeg.a] \
         Error 2
    gmake: *** [wave] Error 2

C13 Compiling files in Cactus seems to involve three steps, preprocessing,
    compiling, and then postprocessing. What is the postprocessing step for? 

C14 When trying to create the Cactus executable I get the error the 
    error <Unresolved text symbol "SetCartSymVN" -- 1st referenced 
    by /data2/Cactus/configs/origin/lib/libADMBase.a>

C15 My thorn is calling directly a function from another thorn. Both thorns 
    are in my ThornList, but on linking I get the error

    Startup.c.o(.text+0x4d): undefined reference to `OtherThornsFunction'

C16 (Linux) When trying the new gcc 3.1.1 compiler, I received a lot of 
    warnings about the compiler flag "-I/usr/local/include". 

C17 (OpenBSD) The system linker /usr/bin/ld crashes when linking C++ thorns.

C18 (Linux) When I compile Fortran I get errors because characters
    are in the first 5 columns, but I can't see anything wrong in the
    source code.
  
C19 (Linux) I get internal errors using the Pacific-Sierra Fortran 90 compiler

C20 (Linux) I get the linking error "/usr/bin/ld: cannot find -lIEPCF90" 
    with the Intel fortran compiler.

C21 (Solaris) Building PUGHReduce under Solaris 2.8 gives several errors 
    with macro recursion.

C22 (MacOS) Compiling with the Absoft F90 compiler I get error messages
    about large "fixups"

C23 How can I see the commands which Cactus is actually using to build 
    the executable?

C24 My configuration takes forever to build, are there any ways to speed 
    this up?

C25 On compiling Fortran code I get errors like "dummy argument CCTK_DIM has 
    not been given a type", but I'm not using CCTK_DIM in the routine? 

C26 I'm passing arguments to my scheduled routine using CCTK_ARGUMENTS and
    I get the compile error "identifier DECLARE_CCTK_ARGUMENTS is undefined"

C27 (Linux IA64) I get a segmentation violation compiling snprintf.c in
    the Cactus flesh?

C28 On linking I get an "unresolved text symbol" for a scheduled function 
    called from libCactusBindings.a

C29 (Linux) Using the Intel Fortran compiler I get the error "/usr/bin/ld
    cannot find -lintrins" on linking.

C30 Why can't I use different F90 and F77 compilers with Cactus? 


------------------------------------------------------------------------------
Running:

D1  I'm trying to run Cactus with MPI using "mpirun", but I just get an error
    "Unknown option -np"?

D2  I'm running Cactus on a parallel system with missing C++ libraries on some 
    nodes. Cactus compiles but it won't run?

D3  I can build a Cactus configuration using some external libraries (eg HDF5,
    PETSc) but get errors with shared libraries on running the executable.

D4  I can run my non-MPI executable using shared libraries without problems now
    but still get the same error as in FAQ D3 for MPI jobs started with mpirun.

D5  (Irix) I'm using the web server thorn CactusConnect/HTTPD, I compiled with
    PTHREADS=yes to get better response, but the executable just hangs.

D6  I get a segmentation fault running Cactus, right after the schedule
    tree is printed to screen?

D7  (Linux) I'm using the NaNChecker thorn and seeing very strange results.
 
D8  (Linux) Testsuites for CactusEinstein/AHFinder and CactusBench/BenchADM 
    are failing, but the release notices say that they pass.

D9  How do I run Cactus until a given coordinate time instead of to a 
    given iteration number? 

D10 Output to screen using the Info IO method of IOBasic doesn't work.


------------------------------------------------------------------------------
Thorn Writing:

E1  While implementing some MPI code in Cactus, I came across the problem of
    getting an MPI communicator.  I basically have the choice between using
    MPI_COMM_WORLD, assuming that no processors have been set aside by the
    driver, or tying myself to a certain driver, such as PUGH, and using the
    driver's internal data structures.  Both choices are clearly not 
    desirable.

E2  How can I communicate the value of a local scalar on one processor to all 
    processors? There doesn't seem to be an API for this in Cactus.

E3  What about if I only want to send a local scalar to a single processor?

E4  Why shouldn't I make direct calls to MPI from my thorns? Your default 
    driver uses MPI.

E5  What is the difference between using "const cGH *cctkGH" and "cGH *cctkGH"
    for argument declarations? What about "const cGH* const cctkGH"?

E6  On compiling Fortran code I get errors like "dummy argument CCTK_DIM has 
    not been given a type", but I'm not using CCTK_DIM in the routine? 

E7  If I have two grid array groups with identical sizes (and ghost zones, 
    distributions, etc.), are they guaranteed to be distributed in the same 
    way?  That is, can I count on the fact that their local shapes (lsh) 
    will be the same?  It is true empirically. What if the grid arrays have 
    different variable types or numbers of time levels?

E8  Is there anyway to only compile certain lines of source code if a 
    particular thorn is in the compiled ThornList?

E9  How can I find out in make.configuration.defn, whether MPI is being
    used or not?

E10 How can I see in source code whether MPI is being used?

E11 The functions CCTK_Exit and CCTK_Abort require cctkGH as an argument. 
    How can I call these functions deep inside my thorns where this pointer
    is not available?

------------------------------------------------------------------------------
General

F1  Why do I have to use e.g. "mpirun -np 1 ./exe/cactus_myconfig -O" to see 
    the compiled parameters, can't the parameters be output before MPI is 
    initialised?

F2  Why isn't a BOOLEAN parameter a LOGICAL in Fortran?

F3  Why is there an MPI call in the Flesh (MPI_Init)? I thought that the 
    Cactus flesh was independent of any parallel protocol, why isn't this
    call made from a driver thorn?

F4  Is there any way to tell what CVS version of individual source files
    was used to build an executable?

F5  Why is the main routine of Cactus written in C++ (src/main/flesh.cc) when 
    the rest of the flesh is standard ANSI C? More strangely, why does Cactus
    still compile when I have no C++ compiler on my system?

F6  Are there grid scalars of type STRING ?

F7  Why don't you use compilers mpicc, mpiCC etc when they exist on parallel 
    machines?

F8  How does Cactus manage to so seamlessly call Fortran routines from 
    C, and vice versa?


------------------------------------------------------------------------------
Documentation:

G1  How can I build the ThornGuide documentation for single thorns or 
    arrangements?


------------------------------------------------------------------------------
I/O:

H1  I want to run an old parameter file, but none of the I/O parameters are
    recognised!


------------------------------------------------------------------------------
Configuration Files:

I1  The size of array parameters is specified by a integer number in the
    param.ccl file. Why can't a parameter itself be used here?


==============================================================================
The Answers:


------------------------------------------------------------------------------
Background:

A1  How did Cactus start?

    Cactus originated in the Numerical Relativity community to realise
    the largescale computing needs and collaborative working practises 
    required to model objects such as black holes, neutron stars and 
    and gravitational waves using Einsteins Theory of General Relativity.

    In particular, Cactus grew out of codes, experiences, and ideas of 
    members of the NCSA and AEI numerical relativity groups together with
    a number of friends and collaborators. Since the first Beta release
    of Cactus 4.0 many suggestions and contributions have also been made 
    by the users.
 

A2  Who develops and maintains Cactus?

    Cactus is primarily developed and maintained at the Max Planck Institute
    for Gravitational Physics (Albert Einstein Institute) in Potsdam, Germany.
    The funding for this support is mainly obtained through independent 
    grants for projects in both computer science and physics which focus
    around Cactus. 


A3  How can I get involved?

    First, if you use Cactus, send bug reports, bug fixes, comments, 
    suggestions, documentation to cactusmaint@cactuscode.org, or use 
    the bug tracking/feature request system. 

    If you have written thorns which could be useful to other people, 
    make them available for others to use (if you have a web page, 
    send us the link to put on our page). 

    To find out about any existing or upcoming research programmer or
    student positions associated with Cactus development please contact
    Ed Seidel (eseidel@aei.mpg.de).


-------------------------------------------------------------------------------
Distributions and CVS Checkout:

B1  When doing a cvs update I get 'not enough slashes in ...' .

    It seems that CVS sometimes gets confused with multiple repositories.  
    Check that you are using the latest version of CVS, otherwise moving into 
    the arrangements directory and performing another 'cvs update', followed 
    by a final one in the toplevel directory should complete your update.

    There is now a Cactus make target "gmake cvsupdate" which avoids this 
    problem by updating the flesh and checked out thorns explicitly.


B2  I want to recreate a Cactus executable from last year, I have a thorn list
    but when I try and check it out from the /cactus or /cactusdevcvs 
    repository I don't get some of the thorns.

    This is probably because the thorns moved arrangement, or were rewritten. 
    To checkout the right code, look in doc/ReleaseNotes to decide which 
    release the thorn was in, and then checkout the thorn from the /cactus 
    repository, eg. if the thorn "OldThorn" was last in the Beta 7 release use:

    cvs -d :pserver:cvs_anon@cvs.cactuscode.org:/cactus co -d OldThorn \ 
                                                             beta7/OldThorn


B3  How do I checkout an earlier version of Cactus ?

    Cactus is "tagged" in CVS for each release, so to checkout for example
    the flesh for Beta 10 use:

    cvs -d :pserver:cvs_anon@cvs.cactuscode.org:/cactus checkout -r \
		Cactus_4_0_Beta_10 Cactus

    [to see a list of possible tags go to the Cactus home directory
     and type "cvs status -v Makefile"]

    To checkout from a particular date, for example the 7th July 2000, use

    cvs -d :pserver:cvs_anon@cvs.cactuscode.org:/cactus checkout -D \
                "7 July 2000"

    Also look at FAQ B2.


B4  When running GetCactus it tells me it is refusing to update my thorn
    because it is installed from a different repository.

    What GetCactus does is look in the thorn's CVS/Root to get the repository
    that it was originally installed from.  If the repository specified in
    your thornlist (using the !REPOSITORY_LOCATION directive) is not the same, 
    then GetCactus will refuse to update your thorn.

    This problem commonly occurs when people interchange repository locations
    that actually point to the same place.  For example, using cvs.aei.mpg.de
    for the initial checkout, and then using cvs.cactuscode.org for an update.

    To fix this problem, simply be consistent with repository locations.


------------------------------------------------------------------------------
Configuring and Building:

C1  The make system says "Checking status of libXXX.a" and then just 
    sits there.  What can I do ?

    The first thing to do whenever a problem arises from the make system 
    is to run with the "SILENT=no" option.

    In this case, though, first check that the clocks on the machine on 
    which you are editing files and the clock on the machine you are compiling 
    on are synchronised.  If the compilation machine's clock is slow, then the 
    newly made dependency files will still be `older' than the file you 
    edited, and the system will loop until this situation changes.


C2  Compilation of Cactus fails with the error:
	 identifier "DECLARE_CCTK_PARAMETERS" is undefined?

    You need to include the header: 
	#include "cctk_Parameters.h"


C3  (Linux) I get 'g77: cannot specify -o with -c or -S and multiple 
    compilations' when compiling with the Pacific VAST90 compiler.

    If the libvast90.a library is in the same directory as your F90 script, 
    F90 passes -lvast90 to the underlying G77 compile, which is of course only 
    a valid switch when linking.  The solution is to seperate these files, 
    e.g. into bin and lib directories.


C4  How do I compile with MPI? Why isn't it automatic like in Cactus 3?
              
    The standard MPI driver (PUGH) is in a thorn now (CactusPUGH/PUGH), so 
    there is now the possibility to add other parallel drivers using 
    alternative message passing schemes as separate thorns. To compile with 
    MPI, when you make a configuration, use

    gmake <configuration>-config MPI=<MPI_TYPE> 

    where the allowed values of MPI_TYPE are discussed in the documentation. 
    For machines where Cactus 3 used to use MPI by default, the correct option 
    is probably MPI=NATIVE 


C5  I can't compile because the compiler complains that a routine name has 
    been previously defined ... but I can't find it repeated?

    If it is a C routine, whose name is either all in capitals or all in 
    lowercase, and the routine has a Fortran wrapper, then it could be that 
    the compiler doesn't attach underscores to the Fortran name and there is a 
    conflict. To remove this possibility always use mixed case names for C 
    routines with Fortran wrappers.
   

C6  How can I make sure that one thorn is compiled before another? For example 
    to make sure that F90 module files from  ArrangementA/ThornA are available 
    for ArrangementB/ThornB.

    Add a dependency to a ThornB's make.configuration.deps file, for example

    ifneq (,$(findstring ArrangementA/ThornA,$(THORNS)))
    $(CCTK_LIBDIR)$(DIRSEP)libThornB.a : $(CCTK_LIBDIR)$(DIRSEP)libThornA.a
    endif

    we will try to modify the make system to make this more automatic in the 
    future.


C7  (Solaris) I start a configuration process with gmake, but on the second 
    gmake it wants to set up the configuration again and again .... 

    You are using a version of gmake which is too outdated (even though
    this version may work perfectly on other architectures). We found
    that GNU Make version 3.79 works fine.


C8  How do I compile Cactus in 32 bit mode on a 64 bit irix machine?

     gmake <config>-config IRIX_BITS=32


C9  What can I do if I don't have enough disk space to compile Cactus?

    If you have access to some scratch space, or temporary space, you can
    instruct Cactus to put the "configs" directory there (with all the 
    object files and intermediate files).

    For example, to use the directory /scratch/myconfigs with csh or tcsh,

    setenv CACTUS_CONFIGS_DIR /scratch/myconfigs

    or with bash

    CACTUS_CONFIGS_DIR=/scratch/myconfigs
    export CACTUS_CONFIGS_DIR


C10 When I compile Cactus I get an error like

    /home/allen/Cactus/arrangements/MineStuff/Test/src/metric.F90:3: 
                unterminated character constant
    make[2]: *** [metric.F90.d] Error 33                                

    But then when I type gmake again the compilation proceeds without problem.

    You are probably using Linux, and the C preprocessor is complaining 
    about single apostrophes in a Fortran comment line when it is creating the
    dependency file for metric.F90. Make continues the second time because 
    the dependency file now exists (although it won't necessarily contain 
    the right information).

    We could remove this problem by removing comments from Fortran files
    before using the C preprocessor, but we don't want to add this overhead
    at the moment, since we are writing a Cactus preprocessor which won't 
    have this problem (and will solve a number of other preprocessing 
    problems). 

    Your choices for getting past this for now are

     - try and remove any single apostrophes in Fortran comments

     - use gmake twice and live with possibly bad dependency information
       for these files

     - if you are using Linux, try using the option "-traditional" 
       with the GNU cpp which does ignore the apostrophes.  

    Unfortunately, cpp is broken in Red Hat's version 2.96, and in the
    official releases 3.0 up to 3.0.3.  The versions 2.95.x and 3.0.4
    do work fine.

    In order to use a different cpp and to pass the option
    "-traditional", use for example

    CPP = /home/user/gcc/bin/cpp -traditional

    when you configure your application.


C11 Sometimes when I compile in parallel with TJOBS or FJOBS the compilation
    stops halfway through.

    Compiling in parallel using TJOBS/FJOBS is not always failsafe. Usually
    simply issuing your gmake command again will complete the compilation
    and produce an executable. Occasionally you may see link errors 
    in the last stage of compilation, if this happens remove the libraries
    
    rm configs/<config>/lib/*

    and issue gmake once more.

    Problems with TJOBS/FJOBS can be avoided if your version of gmake 
    supports parallel make (this happened somewhere between 3.74 and3.78.1),
    in which case you can use instead the more robust:

    gmake <config> -j <number of processors>
    

C12 I get the following error when compiling?

    Checking status of libIOJpeg.a
    gmake[3]: *** No rule to make target \
         `/u1/allen/scratch/Cactus/arrangements/
         CactusIO/IOJpeg/src/IOJpeg.h', needed by `JPEG.c.d'.  Stop.
    gmake[2]: *** [make.checked] Error 2
    gmake[1]: *** [/u1/allen/scratch/Cactus/configs/wave/lib/libIOJpeg.a] \
         Error 2
    gmake: *** [wave] Error 2

    This is because an include file which was used in your previous compile is
    no longer there (in this case the file IOJpeg.h was renamed ioJpeg.h). 
    To solve this, delete the dependency information by issuing

    gmake <config>-cleandeps

    before compiling again.


C13 Compiling files in Cactus seems to involve three steps, preprocessing,
    compiling, and then postprocessing. What is the postprocessing step
    for? 

    This is for architecture dependent things which come up. For example,
    some machines don't allow using a "-o" flag for naming the resulting
    object file, in which case part of postprocessing would involve
    renaming and moving the file.

C14 When trying to create the Cactus executable I get the error

    Creating cactus_origin in /data2/convCactus/exe from
    CactusBase/IOASCII CactusBase/Boundary CactusPUGH/PUGHInterp 
    CactusBase/CartGrid3D CactusEinstein/StaticConformal CactusPUGH/PUGHReduce
    CactusEinstein/ADMBase CactusBase/Time CactusPUGH/PUGH CactusPUGH/PUGHSlab
    CactusBase/IOUtil CactusElliptic/EllBase CactusBase/IOBasic
    ld64: ERROR   33 : Unresolved text symbol "SetCartSymVN" -- 1st referenced 
    by /data2/Cactus/configs/origin/lib/libADMBase.a\
    (InitSymBound.c.o)

    In this case ADMBase is probably not inheriting from the thorn which 
    contains the SetCartSymVN function which it is using. When Cactus makes 
    up the link line order it then doesn't know to list the ADMBase library 
    before the library containing this function. To test this and/or to get an
    executable, find the link line used by issuing

    gmake origin SILENT=no
  
    and add the missing library to the end of the link line. 


C15 My thorn is calling directly a function from another thorn. Both thorns 
    are in my ThornList, but on linking I get the error

    Startup.c.o(.text+0x4d): undefined reference to `OtherThornsFunction'

    Make sure that your thorn inherits from the implementation provided by
    the other thorn. If you don't do this, the end link line may not list
    the libraries in the right order for linking.


C16 (Linux) When trying the new gcc 3.1.1 compiler, I received a lot of 
    warnings about the compiler flag "-I/usr/local/include". 

    This is because HDF5 is installed on this machine in what we
    would consider to be a sub-optimal place. HDF5 is preferably
    installed in own installation directory with (optionally) sym-links to 
    the system directories. Cactus configurations should use the HDF5 
    installation directory itself.

    It is a matter of office debate as to whether it would be really be a 
    good idea or not to check for and remove system directories, we can come 
    up with pathological examples both if they are left or if they are 
    removed.


C17 (OpenBSD) The system linker /usr/bin/ld (called by gcc's collect2) 
    crashes when linking C++ thorns if GNU ar is used to create the
    libraries.

    collect2: ld terminated with signal 11 [Segmentation fault]
    ld: /home/jonathan/cactus/Cactus/configs/test-interp2/lib/ \
    libTestInterp2.a(driver.cc.o): unexpected multiple definitions \ 
    of symbol `axis_list::string(void) const', type 0xf

    The workaround is to use the system's ar if you have any C++ thorns.


C18 (Linux) When I compile Fortran I get errors because characters
    are in the first 5 columns, but I can't see anything wrong in the
    source code.
  
    The C preprocessor /lib/cpp distributed with Redhat 7.x causes
    problems with Fortran code (Fortran source code is by default
    passed through a preprocessor). The C preprocessor can change the
    spacings in a file, which is fine for C but can seriously affect
    Fortran code. If this is happening, you will see errors such as

    ************************************************************************
    Preprocessing
    /Cactus_hydratest/arrangements/CactusPUGH/Interp/src/interp3.F
    Compiling
    /Cactus_hydratest/arrangements/CactusPUGH/Interp/src/interp3.F 
    Pacific-Sierra Research vf90 Personal V3.4N5 12:49:17 10/31/01 f90 to f77
     subroutine INTERP3_CCTK_REAL
    Syntax, line 140: Non-blank characters in columns 1-5 of continuation line
     subroutine INTERP3_CCTK_COMPLEX
    Syntax, line 319: Non-blank characters in columns 1-5 of continuation line
     function   LINEAR
     function   INTERP_QUAD
     function   CUBIC
    gmake[3]: *** [interp3.F.o] Error 16
    gmake[2]: *** [make.checked] Error 2
    gmake[1]: ***

    There are two suggested ways to get around this problem:

    1.  If you are using Beta 11 or higher, try using the
    architecture independent Cactus preprocessor which we are
    developing by configuring with

    make <config>-config CPP='$${PERL} $${CCTK_HOME}/lib/sbin/cpp.pl'

    2. Another way around this problem is to get hold of a copy of
    cpp from an earlier RedHat distribution (here is cpp from 6.2)
    and either replace the original cpp, or place it somewhere else
    and add the option CPP=<location of new cpp> when you configure
    Cactus.


C19 (Linux) I get internal errors using the Pacific-Sierra Fortran 90 compiler

    The free version of the Pacific Sierra compiler can generate
    incompilable code, this is a problem with the translator. The
    error messages look something like:

    Compiling
    /home/Cactus/arrangements/CactusEinstein/AHFinder/src/AHFinder_pow.F  \
    Pacific-Sierra Research vf90 Personal V3.4N5 19:44:34 11/20/01 f90 to f77
     module  F1COM
     subroutine POWELL
    Warning, line 167:
     Too large to translate -- reduce program or loop size (RPNGEN)
    Fatal, line 198: Internal error detected -- please report (EXPGET)
    Syntax, line 198: Duplicate I/O specifier.
     subroutine LINMIN
    Warning, line 497:
     Too large to translate -- reduce program or loop size (RPNGEN)
     function   F1DIM

    The fix is to either do what the message says and reduce the code
    size, or move to a different compiler.


C20 (Linux) I get the linking error "/usr/bin/ld: cannot find -lIEPCF90" 
    with the Intel fortran compiler.

    If you get the linking error

       /usr/bin/ld: cannot find -lIEPCF90

    Then you need to append ${IA32ROOT}/lib to your LD_LIBRARY_PATH
    environment variable. You can do this directly in your shell setup
    file (e.g. .cshrc) or alternatively there is a script
    $IA32ROOT/bin/iccvars.csh which does this for you.

       
C21 (Solaris) Building PUGHReduce under Solaris 2.8 gives several errors 
    with macro recursion.

    Checking status of libPUGHReduce.a
    "/home/CCTK/Cactus_beta13/Cactus/arrangements/CactusPUGH/PUGHReduce/src
     line 323: CCTK_Cmplx_: macro recursion

    This problem occurs only in the dependency generation part of the
    Cactus build. The Sun's /lib/cpp preprocessor cannot handle macro
    recursion levels of more than 3. Since we are loath to rewrite our
    macros just for Solaris, we would rather wait for a better system
    preprocessor, or for a fuller version of the Cactus cpp sometime
    in the future.

    As a solution, you could try using the Cactus preprocessor:

       make -config CPP='$${PERL} $${CCTK_HOME}/lib/sbin/cpp.pl'

    which has the advantage that it doesn't stop for errors but prints
    a warning message. Be wary though, it isn't perfect yet and may
    fail for some thorns.

    Otherwise, you can keep rerunning your make command to skip past
    the problem files (since only the dependency generation is
    failing). Of course, dependencies (for just these files), will
    then not work.


C22 (MacOS) Compiling with the Absoft F90 compiler I get error messages
    about large "fixups"

    Compiling /home/Cactus/arrangements/AEIThorns/ADM_BSSN/src/Sources.F
    Sources.f:unknown:Fixup of -44148 too large for field width of 16 bits
    Sources.f:unknown:Fixup of -42232 too large for field width of 16 bits
    Sources.f:unknown:Fixup of -33028 too large for field width of 16 bits

    Add the following text to the thorn's make.code.defn file:

    ifeq ($(shell uname), Darwin)
      F90FLAGS += -N11
    endif


C23 How can I see the commands which Cactus is actually using to build 
    the executable?

    Add SILENT=no to the end of the compile command and the actual commands
    will be echoed to standard output.


C24 My configuration takes forever to build, are there any ways to speed 
    this up?

    If you are developing and testing, you could build executables without
    using optimisation ... but remember that they will run much slower. To 
    do this configure Cactus with OPTIMISE=no.


C25 On compiling Fortran code I get errors like "dummy argument CCTK_DIM has 
    not been given a type", but I'm not using CCTK_DIM in the routine? 

    Look to see if you are passing CCTK_ARGUMENTS into your routine, if so
    you need to use DECLARE_CCTK_ARGUMENTS in the declarations for the routine.


C26 I'm passing arguments to my scheduled routine using CCTK_ARGUMENTS and
    I get the compile error "identifier DECLARE_CCTK_ARGUMENTS is undefined"

    Make sure you have the lines
  
    #include "cctk.h"
    #include "cctk_Arguments.h"

    at the top of your source file

C27 (Linux IA64) I get a segmentation violation compiling snprintf.c in
    the Cactus flesh?

    Compiling /home/Cactus/src/util//snprintf.c
    /home/Cactus/configs/admeccy/build/Cactus/util/snprintf.c
    ecc: error: Fatal error in /usr/local/intel/compiler60/ia64/bin/mcpcom, 
      terminated by segmentation violation
    compilation aborted for 
    /home/Cactus/configs/admeccy/build/Cactus/util/snprintf.c 
    (code 1)

    The Intel C++ compiler (Version 6.0, Build 20020320) has problems 
    dealing with this file, which needs to be compiled without the -ip 
    option. Either run gmake with the option SILENT=no then cut and paste 
    the compile statement to the command line and remove the -ip option 
    before executing it, and then continue to build the executable as normal.
 
    Alternatively configure with the -O1 option. (Note, so far our benchmarks
    have indicated that -O1 actually executes faster than -O3, although this
    may be just for our particular benchmarks).

C28 On linking I get an "unresolved text symbol" for a scheduled function 
    called from libCactusBindings.a

    Check the size of the object file which was created for the source 
    file containing this function in configs/<config>/build/<thorn>. 
    With some compilers, making object files for complicated source files 
    can fail because it takes too much memory, and an empty object file 
    will be created and the make process will fail. If you continue the
    make process Cactus will not realise that the object file is empty.
  
    If this is what happened, touch the relevant source file and try 
    compiling again. It may be that you will need to switch optimisations off
    for this file, or split up the troublesome code into smaller components.

C29 (Linux) Using the Intel Fortran compiler I get the error "/usr/bin/ld
    cannot find -lintrins" on linking.

    Set the environment variable IA32ROOT to point to the location of your
    Intel compiler installation (eg /opt/intel/compiler60/ia32). You will
    need to reconfigure, and then try building your executable again.


------------------------------------------------------------------------------
Running:

D1  I'm trying to run Cactus with MPI using "mpirun", but I just get an error
    "Unknown option -np", what's the problem?

    It looks like the MPI implementation on the machine you're using isn't
    removing its arguments correctly, so they are being passed to the
    Cactus run. You need to use the "-i" command line option for Cactus to 
    ignore the MPI arguments, e.g.

    mpirun ./cactus_foo foobar.par -i -np 8


D2  I'm running Cactus on a parallel system with missing C++ libraries on some 
    nodes.  Cactus compiles but it won't run.  Is there anything I can do?

    If C++ isn't being used in any thorn, you can compile Cactus without using
    C++ using the configure option CXX=none.


D3  I can build a Cactus configuration using some external libraries such as
    HDF5 or PETSc without any problems but when I run the executable I get
    the following error:

      ./cactus_WaveDemo: error while loading shared libraries: libhdf5.so.0:
      cannot load shared object file: No such file or directory

    This is a problem with using shared libraries on your system:
    the dynamic loader couldn't locate a shared library on the system
    you are trying to run on.
    You can set the LD_LIBRARY_PATH environment variable in your shell setup
    to point to your local installation of HDF5. For global installations
    located in a standard place (such as /usr/local/lib), this should ideally
    be done by your system administrator for you.

    Another possibility to get around the above problem is to use
    static libraries in favor of shared ones. For this you need to remove
    the *.so files from your installation directory (leaving the *.a files).
    Cactus would then link against the static libraries automatically.


D4  I can run my non-MPI executable using shared libraries without problems now
    but still get the same error as in FAQ D3 for MPI jobs started with mpirun.

    This indicates a misconfigured mpirun script. When it starts a shell on
    other nodes it probably doesn't read in the user's shell setup so that the
    LD_LIBRARY_PATH environment is not set properly.
    You should contact your system administrator to get this fixed.


D5  I'm using the web server thorn CactusConnect/HTTPD on an Irix machine, I 
    compiled with PTHREADS=yes to get better response, but the executable 
    just hangs (even with the httpd::use_pthreads="no" in the parameter file)

    SGI/Irix machines with native MPI cannot make use of Pthreads (the native
    MPI uses some Irix threads which are incompatible with Pthreads), and the
    executable hangs on the first call to a pthread routine.

    You could instead use MPICH on your Irix machine instead of native MPI. 
    Note that native MPI will probably be better optimised for you application
    than  MPICH though.


D6  I get a segmentation fault running Cactus, right after the schedule
    tree is printed to screen?

    Check to see if you are passing CCTK_ARGUMENTS into any routines
    registered at CCTK_STARTUP, if so these need to be removed and void 
    used instead. The STARTUP timebin is run before the driver sets up 
    any grid variables, and the segmentation fault occurs in 
    CCTK_DECLARE_ARGUMENTS when these undefined variables are accessed. 
    Parameters can however be used in STARTUP routines.


D7 (Linux) I'm using the NaNChecker thorn and seeing very strange results.
 
    There is a bug in the isnan function with the Fortran Intel
    compiler (for IA32 and IA64). This bug means that any thorn using
    isnan (for example CactusUtils/NaNChecker) will produce
    inconsistent results.


D8  (Linux) The testsuites for CactusEinstein/AHFinder and 
    CactusBench/BenchADM fail for me, but the release notices say that they
    pass.

    The Absoft fortran compiler (Version 6) produces the wrong answers
    for CactusEinstein/AHFinder and CactusBench/BenchADM if
    optimisation is not used.
    
D9  How do I run Cactus until a given coordinate time instead of to a 
    given iteration number? 

    Instead of setting cactus::cctk_itlast set cactus::cctk_final_time to 
    the required coordinate time, and also set cactus::terminate = "time".

D10 Output to screen using the Info IO method of IOBasic doesn't work.

    I want to watch min/max values for 'wavetoy::phi' during a cactus run 
    on stdout.  For that, I added

    IOBasic::outInfo_every = 10
    IOBasic::outInfo_vars  = "wavetoy::phi"
 
    to my parameterfile. Thorn IOBasic is active, but I only get the 
    iteration number and coordinate time printed to screen.

    You need to activate a thorn which implements the min/max reductions, 
    for example CactusPUGH/PUGHReduce.


------------------------------------------------------------------------------
Thorn Writing:

E1  While implementing some MPI code in Cactus, I came across the problem of
    getting an MPI communicator.  I basically have the choice between using
    MPI_COMM_WORLD, assuming that no processors have been set aside by the
    driver, or tying myself to a certain driver, such as PUGH, and using the
    driver's internal data structures.  Both choices are clearly not 
    desirable.

    We are aware of this, and in fact, it is the reason why some of the 
    I/O thorns are directly dependent on the PUGH driver.
    The solution for all this will be the Cactus Communication
    Infrastructure (CCI: http://www.cactuscode.org/Development/Specs/CCI.txt) 
    which will eventually be integrated into the flesh and will provide a 
    generic function interface for communicating data. 
    Thus when you plug in different communication drivers, I/O thorns 
    won't need to be changed if they use the CCI routines.


E2  How can I communicate the value of a local scalar on one processor to all 
    processors? There doesn't seem to be an API for this in Cactus.

    There isn't a general broadcast API in Cactus at the moment, this is 
    planned as part of a new Cactus Communication Interface (CCI) layer
    in the 4.1 release (see the Specs web page for more information about 
    this).

    Until then, the best way to communicate your variable is to use the the 
    CCTK reduction API. For example, make a call to CCTK_ReduceLocalScalar() 
    with the "sum" reduction operator, and set your local variable to zero 
    on all processors except for the one with the value you want to have 
    communicated.


E3  What about if I only want to send a local scalar to a single processor?

    This type of point-to-point communication is not possible with Cactus 
    APIs right now. Any CCTK_Reduce*() is a global communication so far, 
    and all processors must take part in that communication.

    For doing point-to-point communication you should use the appropriate API
    of an underlying communication layer like MPI.


E4  Why shouldn't I make direct calls to MPI from my thorns? Your default 
    driver uses MPI.

    Keeping all MPI calls in the driver thorn keeps the code modular, allowing
    for you to use alternate drivers when they become available, for example
    using PVM or some other communication layer. 

    Also, since not all underlying MPI implementations provide the same 
    fortran bindings it is best to use a C wrapper for making MPI calls, 
    and this is automatically done for you if you use the Cactus APIs.


E5  I am writing a thorn, what is the difference between using 
    "const cGH *cctkGH" and "cGH *cctkGH" for argument declarations? 
    And what about "const cGH* const cctkGH"?

    'cGH *' is a pointer to a cGH structure which can be modified. This is
    necessary eg. in routines which initialize a cGH.

    'const cGH *' is a pointer to a cGH structure which is marked as 
    constant (ie. read-only). The code is not allowed to change the cGH 
    using such a pointer. The compiler should refuse to compile it otherwise, 
    or at least print a warning.

    Our policy now is to use "const cGH*" for all CCTK_ calls, and it is
    recommended that this is also implemented in all thorns. (It is being
    implemented in all the released Cactus thorns, but you may still find
    some exceptions).

    [Note: August 2002: We have still to implement const cGH* in a few
           CCTK_ calls, namely   
                * calls to CCTK_Sync()
  	        * calls to CCTK_Reduce()
  	        * calls to CCTK_Interp() (the old API)
           and it is still a matter of debate whether the const qualifier
	   should be used for CCTK_Sync calls.]

    Finally, the additional const qualifier in 'const cGH * const GH' says 
    that the pointer variable itself cannot be changed (it's just a 
    one-time-only assignment). For instance, no pointer arithmetic is 
    allowed on such a pointer. In principle, one could declare all local 
    variables as const if they aren't changed afterwards. But it's doesn't 
    have any benefit in writing cleaner code. I think this is just 
    overkill and using just 'const cGH *' is fine.

E6  On compiling Fortran code I get errors like "dummy argument CCTK_DIM has 
    not been given a type", but I'm not using CCTK_DIM in the routine? 

    Check if you have passed CCTK_ARGUMENTS in the argument list for the
    routine, is do you must have a line

    DECLARE_CCTK_ARGUMENTS 

    in your declarations section.

E7  If I have two grid array groups with identical sizes (and ghost zones, 
    distributions, etc.), are they guaranteed to be distributed in the same 
    way?  That is, can I count on the fact that their local shapes (lsh) 
    will be the same?  It is true empirically. What if the grid arrays have 
    different variable types or numbers of time levels?

    Yes to all.
 
E8  Is there anyway to only compile certain lines of source code if a 
    particular thorn is in the compiled ThornList?

    Yes, but this is not a good idea for a number of reasons

    - it adds hidden dependencies between thorns, and can easily lead to 
      unexpected and hard to track down behaviour when a thorn is compiled 
      but not active

    - it breaks all kinds of modularity infrastructure in Cactus

    - the include file you need to use will cause your source code to be
      recompiled when other thorns change

    But if you insist, and accept the consequences, this is how you do it,

    Include the header file cctk_DefineThorn.h

    In your code use

    #ifdef ARRANGEMENTNAME_THORNNAME
	<your source code>
    #endif

    If appropriate, to give yourself some protection, use the function
    CCTK_IsThornActive to only execute the source code if the thorn was 
    actually activated (and not just compiled in).

    Add a comment to your README/Documentation.tex to remind yourself, and
    others of what you have done.

E9  How can I find out in make.configuration.defn, whether MPI is being
    used or not?  The thorn EllPETSc checks whether the makefile variable 
    MPI_LIBS is empty.  This doesn't work on systems where using MPI does 
    not require any explicitely specified libraries, such as when using the 
    mpicc script to compile.  Is there a better way?

    You can grep for MPI in the file

        configs/<your_config>/config-data/cctk_Extradefs.h

    see for example CactusPUGHIO/IOPanda/src/make.configuration.deps 
      
       if test -z "`grep CCTK_MPI $(CONFIG)/cctk_Extradefs.h`" ; then
          echo "IOPanda requires MPI";   \
          exit 2;                        \
       fi

E10 How can I see in source code whether MPI is being used?

    Check whether the preprocessor macro CCTK_MPI exists, for example,

    #ifdef CCTK_MPI
	/* Only included if MPI is being used */
    #else
        /* Only included if MPI is *not* being used */
    #endif

E11 The functions CCTK_Exit and CCTK_Abort require cctkGH as an argument. 
    How can I call these functions deep inside my thorns where this pointer
    is not available?

    These routines are overloaded by a driver, and the reason for passing
    in a cctkGH is to allow the driver to tidy up before exiting. If the 
    particular driver you are using allows a NULL pointer to be passed you 
    can pass this instead of the cctkGH pointer. (The worst that will happen 
    should be a segfault if the driver cannot handle NULL). Alternatively, 
    you need to include a mechanism to obtain the cctkGH deep inside your
    thorns code.
    

------------------------------------------------------------------------------
General

F1  Why do I have to use e.g. "mpirun -np 1 ./exe/cactus_myconfig -O" to see 
    the compiled parameters, can't the parameters be output before MPI is 
    initialised?

    Alas some MPIs add their own command-line options, so we need to do 
    MPI_Init before parsing the command line, or parse it twice, once before 
    the MPI_Init and once after, to detect invalid options.


F2  Why isn't a BOOLEAN parameter a LOGICAL in Fortran?

    There is no way we can guarantee the way the Fortran logical type is 
    represented at machine level, and since all Fortran variables need 
    to be mapped internally to C data structures, we use integers to ensure
    portability. 

           
F3  Why is there an MPI call in the Flesh (MPI_Init)? I thought that the 
    Cactus flesh was independent of any parallel protocol, why isn't this
    call made from a driver thorn?

    We would love to get rid of the last remaining MPI calls from the flesh,
    unfortunately in MPI 1, MPI implementations are allowed to play around 
    with the argument list, so we need to call MPI_Init before parsing the 
    command line argument, and this makes it very difficult to have it 
    called from the driver, since we don't even know what the parameter 
    file is at that stage, so we don't even know which driver to activate!

    MPI 2 allows NULL to be passed to MPI_Init, which gets around the 
    problem, but the only machines currently with MPI 2 are Japanese 
    supercomputers.

    It is the only MPI call made before the driver is activated, and it 
    sets a global variable to tell you it's been done, so it isn't a 
    major problem apart from being extremely ugly.  You can even turn 
    it off with an environment variable, but then if you run with MPICH 
    you'll find the code complaining about unknown command line arguments 
    like -p4grp. If we had a precise list of the arguments passed by each 
    MPI implementation we could guard against them, but that's an even 
    uglier option.


F4  Is there any way to tell what CVS version of individual source files
    was used to build an executable?

    Actually there is, just issue the Unix command:

    strings cactus_<config> | grep Header

    to see all the version information for all source files, and

    strings cactus_<config> | grep Header | grep <filename>

    to look for a particular file.

    (Note that this only works for files using the CCTK_FILEVERSION macro.)


F5  Why is the main routine of Cactus written in C++ (src/main/flesh.cc) when 
    the rest of the flesh is standard ANSI C? More strangely, why does Cactus
    still compile when I have no C++ compiler on my system?

    The main routine of the flesh is written in C++, since any static C++ 
    classes need to be initialised before the main routine is called. So if
    there is a C++ compiler available we use this for compiling flesh.cc.
    Since we use only ANSI C in flesh.cc, if there is no C++ compiler 
    available on your machine (and so you are not trying to compile any thorns
    containing C++), we use the C compiler instead.


F6  Are there grid scalars of type STRING ?

    No, there is only a CCTK_CHAR type. You can however define an array
    with DISTRIB=CONSTANT with a maximum length.


F7  Why don't you use compilers mpicc, mpiCC etc when they exist on parallel 
    machines?

    The compilers mpicc etc are scripts for compilation which are site
    dependent and whose contents can change. The major reason we don't
    use these is that in general it is difficult to work out what
    underlying compiler is being used, which in turn leads to problems
    adding the correct compilation flags.

    So far we have found it more convenient and transparent to manually add 
    libraries and options to the configure system than to rely on
    mpicc behaving well on all platforms. At some point we will be
    reinvestigating this.


F8  How does Cactus manage to so seamlessly call Fortran routines from 
    C, and vice versa?

    Part of the configuration step on a machine involves automatically 
    deducing the convention which the fortran compiler uses for its symbols 
    (for example, upper or lower case and the addition of underscores). 

    This information is then used to construct the macros which are used 
    at preprocessing to translate Fortran names to C-callable routines. 

    We also use macros to facilitate passing strings from Fortran into
    C. Strings must always be at the end of argument lists for this
    (which is why some of our choices of order in argument lists may
    seem strange), and the architecture details of string passing are
    maintained in the file src/include/cctk_FortranString.h. At the moment
    we have macros for one, two or three strings in an argument list.


------------------------------------------------------------------------------
Documentation:

G1  How can I build the ThornGuide documentation for single thorns or 
    arrangements?

    Give the names of the thorns or arrangements on the make line, e.g.

    gmake ThornGuide THORNS="CactusBase/Boundary"

    gmake ThornGuide ARRANGEMENTS="CactusBase"   


------------------------------------------------------------------------------
I/O:

H1  I want to run an old parameter file, but none of the I/O parameters are
    recognised!

    In Beta 13 the "old" I/O parameter names were finally fully removed in 
    favour of "new" more consister parameter names across the I/O thorns in 
    the computational toolkit. A complete list of I/O parameters which 
    changed is contained in the following mail:

    http://www.cactuscode.org/archives_html/users/msg00278.html

    Note that there is a perl script for which can automatically update
    parameter files for these changes.

    http://www.cactuscode.org/Development/Release/FixParametersForBETA13.pl


------------------------------------------------------------------------------
Configuration Files:

I1  The size of array parameters is specified by a integer number in the
    param.ccl file. Why can't a parameter itself be used here?

    The size of the array has to be fixed at compile time to accomodate
    the use of parameters in Fortran thorns.