\documentclass[11pt]{article} \usepackage{epsfig,amsmath,amssymb,amsfonts,mathrsfs} \RequirePackage{amssymb} \RequirePackage{amsfonts} \pagestyle{headings} %\bibliographystyle{chicagoa} %\bibliography{uli.bib} \setlength{\textwidth}{17cm} \setlength{\oddsidemargin}{-0.4cm} \setlength{\topmargin}{0.0cm} \setlength{\textheight}{22cm} %\setlength{\parindent}{0.5cm} \numberwithin{equation}{section} \renewcommand{\textfraction}{0} \renewcommand{\topfraction}{1} %\renewcommand{\bottomfraction}{1} \renewcommand{\floatpagefraction}{1} \begin{document} \title{A user's perspective on getting started with Carpet} \author{Ulrich Sperhake, Erik Schnetter} \date{August 5, 2004} \maketitle %============================================================================= \section{Introduction} These notes provide information on how to install and use the package Carpet as seen from a user's point of view. Carpet is a set of Thorns that provide fixed and to some extent adapted mesh refinement in the Cactus environment. As Cactus is a necessary requirement for using Carpet, these notes will inevitably contain some information about Cactus as well. The reader should regard these notes as a first draft and the information represents the author's personal experiences rather than an exhaustive recipe on getting Carpet to work on an arbitrary given platform. In this sense I am hopeful that users as well as developers will continue to add to this document to make it more useful in the future. Useful starting points for retrieving more detailed information on various issues are the project's web pages\\ \hspace{1cm}{\tt http://www.cactuscode.org}\\ \hspace{1cm}{\tt http://www.carpetcode.org}\\ %============================================================================= \section{Downloading the necessary packages} One first needs to download the Cactus version 4.0.13 (or alternatively for the more daring the development version). A more detailed description about how this is done can be found on the Cactus web page % \begin{center} {\tt http://www.cactuscode.org} \end{center} % Here we will summarize the required steps for downloading the complete Cactus-4.0.13 package. Change to a suitable directory on your system and log onto the Cactus cvs server via\\ \hspace{1cm}{\tt cvs -d :pserver:cvs\_anon@cvs.cactuscode.org:/cactus login} \\ which will prompt you for a password which is {\tt anon}. For the development version you will need to choose the directory {\tt /cactusdevcvs} instead. Next check out the Cactus flesh which will create a directory {\tt Cactus} under your current location\\ \hspace{1cm}{\tt cvs -d :pserver:cvs\_anon@cvs.cactuscode.org:/cactus checkout Cactus} \\ The rest of the cactus checkout is best done with the scripts that are shipped as part of Cactus. Change to that directory\\ \hspace{1cm}{\tt cd Cactus} \\ and enter the command\\ \hspace{1cm}{\tt make checkout} \\ That will give you various options to choose those parts of cactus you want to ckeckout. The default option {\em arrangements} is quite satisfactory for this purpose, so just hit return. You will then be given a list of (at the time of writing) 13 Cactus arrangements. Getting them all is a good idea, so choose once more the default option by pressing return. Depending on your internet connection this may take a while. Once all is downloaded you want to quit the script. This is not the default option, so type {\tt q} and hit return. \\ In order to run the WaveToy example that comes with CarpetExtra (see below) you will need to check out Erik Schnetter's package TAT. First switch the directory to\\ \hspace{1cm}{\tt cd arrangements} \\ then checkout\\ \hspace{1cm}{\tt cvs -d :pserver:cvs\_anon@cvs.cactuscode.org:/arrangements checkout TAT} \\ Again this may take a little time. Finally you will have to check out the {\tt Carpet} package. As of mid April 2004 Carpet consists of 4 arrangements. {\tt Carpet} contains all the necessary thorns you will need to run Carpet in the first place. The latest cutting edge thorns currently under development are located in {\tt CarpetDev}. Do not be too surprised, though, if you find some the tools in there not to be fully functional. Packages not required to run {\tt Carpet}, but probably useful for various purposes, such as scalar wave examples, are located in {\tt CarpetExtra}. % Detailed instructions can be found on the % web page % % \begin{center} % {\tt http://www.carpetcode.org}\\ % \end{center} % Remain in the {\tt arrangements} directory for this purpose and log into the carpet cvs-server\\ \hspace{1cm}{\tt cvs -d :pserver:cvs\_anon@cvs.carpetcode.org:/home/cvs/carpet login}\\ the password being once more {\tt anon}. Next checkout Carpet by typing\\ \hspace{1cm}{\tt cvs -d :pserver:cvs\_anon@cvs.carpetcode.org:/home/cvs/carpet checkout Carpet} \hspace{1cm}{\tt cvs -d :pserver:cvs\_anon@cvs.carpetcode.org:/home/cvs/carpet checkout CarpetExtra} \hspace{1cm}{\tt cvs -d :pserver:cvs\_anon@cvs.carpetcode.org:/home/cvs/carpet checkout CarpetDev}\\ %============================================================================= \section{Documentation} Documentation about Cactus, Carpet and their separate thorns comes in different forms. Most importantly you generate the UsersGuide and ReferenceManual for Cactus by going into the {\tt Cactus} directory and typing \hspace{1cm}{\tt make UsersGuide}\\ \hspace{1cm}{\tt make ReferenceManual}\\ \hspace{1cm}{\tt make ArrangementDoc}\\ \hspace{1cm}{\tt make ThornDoc}\\ (four separate commands). They will be created in postscript format under the directory\\ \hspace{1cm}{\tt doc}\\ relative to your current position, i.e.\,the {\rm Cactus} directory. In addition each thorn may contain a subdirectory {\tt doc} where the author (or users) may store additional documentation, typically in the form of a file {\tt documentation.tex}. %============================================================================= \section{Compilers} Before we indulge in using Cactus/Carpet, we have to address issues concerning the system you are working on. We begin with the compilers although we will not be able to deal with the subject in an exhaustive fashion. Basically these notes list our experiences with local machines (i.e. at Penn State) and may or may not be valid for your environment. Users are encouraged to add their experiences to this list. At Penn State we largely work with the Intel compilers and the success of compilations has been found to depend sensitively on which version of the Intel compilers we are using. We will discuss some error messages encountered in the process of compiler testing below. Free download (at least for Linux) of the Intel compiler (Fortran and C++) for non-commercial private or academic use is available from the web page % \begin{center} {\tt http://downloadfinder.intel.com/scripts-df/support\_intel.asp} \end{center} % (click on Software Development, check for the compilers on your system and follow their instructions). In case you haven't got root access, you may need to install the compiler locally or you will have to ask your sys-admin. Additional difficulties may arise in case you have no root access, i.e. install locally, while your sys-admin keeps some older version installed. In order to make sure that no conflict arises thereof (e.g. by linking against old versions of the library) the environment variable {\tt LD\_LIBRARY\_PATH} must point to your local new version and not to the old version in {\tt /usr/local} or wherever. You will probably end up with error messages such as {\tt undefined symbols ...} otherwise. We decided to use the Intel compiler for both Fortran and C++ code. This was mainly a result of the current version of g++ not having the complete stl libraries that are made use of extensively in Carpet. \\ An important aspect of the Intel compilers is that they come in various different versions. Even the same version number (say 7.1) comes in many different releases. You can check this by typing \\ \hspace{1cm}{\tt ifc -V}\\ and likewise for {\tt icc}. Note in particular the date of build given in the form of {\em 20030307Z}. This corresponds to the March 2003 build of version 7.1 and caused difficulties for me. I encountered an error message like\\ \hspace{1cm}{\tt /home/terminator/sperhake/src/2004\_02\_16\_cactus-FMR/configs/test01/build/CarpetLib/} \hspace{1cm}{\tt data.cc(173): error: no instance of overloaded function "dist::datatype"} \hspace{1cm}{\tt matches the argument list}\\ This can be rectified by switching to a newer release, at least the September 2003 build of version 7.1 (I'd recommend doing that for both the Fortran and the C++ compiler). Some Cactus-Carpet users have reported problems, such as segmentation faults, by using the most recent versions of the Intel compilers, namely the March 2004 release of version 7.1 and the latest version 8.0. So far we have been using the former of these without encountering any difficulties, but you should probably stick to the December or September 2003 version of 7.1 if you can. On my Gentoo Linux laptop, on the other hand, I experienced trouble with the September 2003 version of 7.1. I received error messages like\\ \hspace{1cm}{\tt struct stat stat\_bbox ...} \hspace{1cm}{\tt Incomplete components in structure not allowed}\\ at compilation (I have forgotten the exact wording, but you'll recognize it). I managed to work around this by using the Intel Fortran and C++ compilers version 8.0 (build October 2003). As I have not done extensive code development on this laptop, though, I cannot really comment on the potential issues concerning the 8.0 version mentioned above. \\ %In case you decide to use version 8.0, however, note that it does not %accept the {\tt rpath} option as specified in the {\tt LDFLAGS} entry %of the configuration file (see below). This option, for %what little I know, tells the linker about the paths for shared libraries. %The error message obtained with the {\tt rpath} option was something like\\ %\hspace{1cm}{\tt undefined reference to CCTK\_FullVersion, CCTK\_TimeInfo}\\ Trouble may also arise from preprocessing in case you are using RedHat 7.3 (possibly also with other versions). This is essentially related to the treatment of white space in Fortran files. Should you encounter rather stupid error messages which clearly indicate that proper lines of Fortran have been corrupted by introducing white space (e.g. line breaks) at preprocessing, you should check your cpp and possibly download another (probably older) version. Details about this can be found on % \begin{center} {\tt http://www.cactuscode.org/Documentation/Architectures/Linux.html} \end{center} % which also gives a link to the preprocessor of the older RedHat 6.2 distribution. I downloaded that older version and it solved the preprocessing problems I encountered prior to that. \\ %============================================================================= \section{Libraries} \label{sec: libraries} As much as the compiler issue is strongly dependent on your platform, the extent to which you will have to install new libraries will depend on what your system administrator has already done for you. Again these notes cannot be exhaustive and rather focus on our experience. Feel free, as before, to add to our list. %============================================================================= \subsection{HDF library} The HDF5 library is required for handling in/output in a particular binary data format. The use of these libraries in Cactus/Carpet is entirely optional, but in the end I found it easier to install the libraries than to convince my system that I do not want to use them. They should be useful in the long run anyway, so I recommend their installation unless they are already part of your system. Let us start with the hdf5 libraries. The binary version can be obtained from % \begin{center} {\tt ftp://ftp.ncsa.uiuc.edu/HDF/HDF5/hdf5-1.6.1/bin} \end{center} % As before I prefer compiling the source which you can get from % \begin{center} {\tt ftp://ftp.ncsa.uiuc.edu/HDF/HDF5/hdf5-1.6.1/src} \end{center} % Again the instructions in the {\tt INSTALL} file are straightforward. I included the C++ interface by setting the options\\ \hspace{1cm}{\tt ./configure --enable-cxx}\\ and used the variables {\tt CPPFLAGS} and {\tt LDFLAGS} to ensure that the szip libraries were found (see {\tt INSTALL} file). The Fortran interface did not work for me, so I did not enable that. In future versions of this document this issued may be readdressed. Finally you may need to point the environment variable {\tt LD\_LIBRARY\_PATH} in your {\tt .bashrc} or {\tt .cshrc} to the directory containing the hdf5 library.\\ %============================================================================= \subsection{Parallelization} This subsection is relevant only if you plan to do multi processor runs (which you are rather likely to do, though, since it is a key feature of Cactus/Carpet). There are various packages that take care of parallelization, such as {\tt MPICH} or {\tt lam} and your machine will probably come equipped with one of these. I have only had the need to install a message passing interface {\tt (MPI)} on my laptop. It's a single processor laptop but you can emulate multi-processor runs none the less. Furthermore it appears to me that Carpet expects {\tt MPI} at least in the form of a header file {\tt mpi.h}, so you'd better install it. I chose the {\tt lam} package for this purpose, so that is the only experience I have to report. Installation of this package was straightforward on my Gentoo Linux laptop by typing\\ \hspace{1cm}{\tt emerge lam-mpi}\\ Depending on your Linux flavor installation may be done differently, for example using {\tt rpm}. {\tt lam} is started by typing\\ \hspace{1cm}{\tt lamboot}\\ and then executables can be started via\\ \hspace{1cm}{\tt mpirun -np} $<$n$>$ $<$executable$>$\\ where $<$n$>$ is the number of processors and $<$executable$>$ the binary file (with full path) you want to run. %============================================================================= \section{Creating a configuration} %============================================================================= \subsection{The configuration file} Eventually we can start writing a configuration file for a Cactus-Carpet project. In this configuration file the paths to various files, such as libraries and compilers need to be specified. Naturally these paths will differ from machine to machine. In this subsection I will assume the installation path \hspace{1cm}{\tt /usr/local/}$<$name$>$\\ for most libraries, where $<$name$>$ is the name of the library, e.g. {\tt hdf4} or {\tt szip}. I further assume that each of these directories contains subdirectories {\tt lib} and {\tt include} which contain the libraries and header files. Similarly I presume that all compilers/preprocessors are installed in the directory\\ \hspace{1cm}{\tt /usr/local/for\_carpet/bin}\\ This is, of course, not where they reside on your machine (nor on mine), but it'll be sufficient for this document and you will merely have to replace each of these paths with the correct one on your system. We are now in the position to create the configuration file, say {\tt mycode\_carpet.cfg} (you can store that file wherever you think convenient). We will focus on the most important entries in this file only. Please refer to the Cactus documentation for a more detailed description. First we specify information about the compilers\\ \hspace{1cm}{\tt F90 \hspace{1cm} /usr/local/for\_carpet/bin/ifc} \hspace{1cm}{\tt F77 \hspace{1cm} /usr/local/for\_carpet/bin/ifc} \hspace{1cm}{\tt CC \hspace{1.2cm} /usr/local/for\_carpet/bin/icc} \hspace{1cm}{\tt CXX \hspace{1cm} /usr/local/for\_carpet/bin/icc} \hspace{1cm}{\tt CPP \hspace{1cm} /usr/local/for\_carpet/bin/cpp} \hspace{1cm}{\tt FPP \hspace{1cm} /usr/local/for\_carpet/bin/cpp}\\ (the exact amount of white space between the variables {\tt F90, F77,...} and their entries should not matter and you may even put in an $=$ sign). Note that you do not need to specify the full path if your environment variable {\tt PATH} points to the correct versions of the compilers/preprocessors already.\\ Next we need to specify information about the message passing interface. In my case that was {\tt lam}, so the next entries in my file {\tt mycode\_carpet.cfg} are\\ \hspace{1cm}{\tt MPI \hspace{2.5cm} LAM} \hspace{1cm}{\tt LAM\_INC\_DIR \hspace{1cm} /usr/include} \hspace{1cm}{\tt LAM\_LIB\_DIR \hspace{1cm} /usr/lib}\\ In case you are using a different {\tt MPI} package refer to the Cactus users guide to find the correct entry for {\tt MPI}. Make sure that you specify the correct paths for the corresponding header files and libraries (ask your sys-admin if necessary).\\ Next we specify the libraries to be included in the compilation. For the 7.1 version of the Intel compilers in combination with {\tt lam} we found the following to work fine\\ \hspace{1cm}{\tt LIBS \hspace{2cm} crypt lapack blas g2c z BINDF90 CEPCF90 F90 IEPCF90 PEPCF90} \hspace{4.2cm}{\tt POSF90 cprts cxa guide imf intrins irc ircmt ompstub svml} \hspace{4.2cm}{\tt unwind X11 ieeeio df m mpi lam pmpi}\\ (all in one line). It goes without saying that all these libraries must be installed on your machine. Most of them probably are and the installation of some that may not is described in more detail above in Sec.\,\ref{sec: libraries}. \\ The paths to some of these libraries may not be known automatically by the linker and needs to be specified separately. This is done with the variable {\tt LIBDIRS} which I had to set to\\ \hspace{1cm}{\tt LIBDIRS \hspace{1.4cm} /usr/local/intel/compiler70/ia32/lib} \hspace{4.2cm}{\tt /usr/X11R6/lib /usr/local/IEEEIO/lib /usr/local/hdf4/lib} \hspace{4.2cm}{\tt /usr/lib/gcc-lib/i386-redhat-linux/egcs-2.91.66}\\ (again on all in one line). As before you will have to adjust this line to your demands.\\ Finally I set %\hspace{1cm}{\tt LDFLAGS \hspace{1.4cm} -Wl,-rpath,/usr/nrlocal/petsc-2.1.0/lib/libO/linux}\\ \hspace{1cm}{\tt PTHREADS \hspace{1.2cm} yes}\\ though I am not sure what this is exactly doing. %As has been mentioned above the first of these lines is valid for %version 7.1 of the Intel compiler and should be omitted if you are using %version 8.0. %============================================================================= \subsection{make-config} In order to create a configuration change into the {\tt Cactus} directory and type\\ \hspace{1cm}{\tt make} $<$name$>${\tt-config options=}$<$config-file$>$\\ where you can choose an arbitrary $<$name$>$ for your configuration and $<$config-file$>$ is the file (with full path) created in the previous subsection. %============================================================================= \subsection{Creating a thornlist} Next you will need to generate a thornlist, i.e.\,a list of all those thorns you want to compile. This is done in the {\tt Cactus} directory by typing\\ \hspace{1cm}{\tt make} $<$name$>${\tt-thornlist}\\ where $<$name$>$ must be the same as in setting up the configuration. This command will search all arrangements for all thorns and eventually prompt you whether you want to modify the list. As all thorns are activated by default you do want to modify the list and type {\em yes} and hit return. This will open an editor session where you can unselect thorns by putting a hash '\#' at the beginning of the line. Unselect all thorns in this way except for the following\\ \hspace{1cm}{\tt CactusBase/Boundary} \hspace{1cm}{\tt CactusBase/CartGrid3D} \hspace{1cm}{\tt CactusBase/CoordBase} \hspace{1cm}{\tt CactusBase/IOBasic} \hspace{1cm}{\tt CactusBase/IOUtil} \hspace{1cm}{\tt CactusBase/LocalInterp} \hspace{1cm}{\tt CactusBase/SymBase} \hspace{1cm}{\tt CactusBase/Time} \hspace{1cm}{\tt Carpet/Carpet} \hspace{1cm}{\tt Carpet/CarpetIOASCII} \hspace{1cm}{\tt Carpet/CarpetIOHDF5} \hspace{1cm}{\tt Carpet/CarpetInterp} \hspace{1cm}{\tt Carpet/CarpetLib} \hspace{1cm}{\tt Carpet/CarpetReduce} \hspace{1cm}{\tt Carpet/CarpetRegrid} \hspace{1cm}{\tt Carpet/CarpetSlab} \hspace{1cm}{\tt CarpetExtra/IDScalarWave} \hspace{1cm}{\tt CarpetExtra/WaveToyF77}\\ Before you compile, you need to apply one modification to the file\\ {\tt arrangements/CarpetExtra/WaveToyF77/configuration.ccl}\\ namely remove the entry {\tt Cart3d} from the list of {\tt REQUIRED} thorns. This thorn is actually required but, for some reason unknown to me, must not be mentioned here. It gave an error message complaining that there is no thorn {\tt Cart3d}. Having applied this modification you can start compiling by typing\\ \hspace{1cm}{\tt make} $<$name$>$ There is no guarantee, but at least you have a chance of compiling through without error messages (do not be intimidated by the odd warning, though). In case you still cannot compile, please add your wisdom to this document to help future users. %============================================================================= \subsection{Running the first application: WaveToyF77} If you've gotten this far, you should be able to run your first simulation with mesh refinement. Change to some convenient directory for this purpose and copy over from relative to the main {\tt Cactus} directory the parameter file\\ \hspace{1cm}{\tt arrangements/CarpetExtra/WaveToyF77/par/wavetoyf77\_rad\_full\_rl2.par}\\ You will need to adjust this parameter file a little to get it running (I am not aware of a WaveToy-parameter file that does not require such minor modification). First add to the first line beginning with {\tt ActiveThorns} the thorns {\tt Slab CoordBase SymBase} (that is within the quotes). Finally you should be able to run this example by typing something like\\ \hspace{1cm}{\tt mpirun -np 1} $\tilde{\,\,}${\tt /Cactus/exe/cactus-}$<$name$>$ {\tt wavetoyf77\_rad\_full\_rl2.par}\\ where $<$name$>$ is again the name of the configuration above. In case you do not have your main {\tt Cactus} directory under your home directory you will need to adjust that part in the command.\\ By running this command you should obtain a directory {\tt wavetoyf77\_rad\_full\_rl2} with the resulting data in ascii format. You can check for example the file\\ \hspace{1cm}{\tt wavetoyf77\_rad\_full\_rl2/phi.x.asc}\\ (relative to the directory where you ran the code) which lists the data on the separate refinement levels. \end{document}