\documentclass[11pt, tightenlines]{revtex4}

\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{color}
\usepackage{float}
\usepackage{graphicx}
\usepackage[latin9]{inputenc}
\usepackage{url}

% Put this package last
\usepackage[bookmarks, bookmarksopen, bookmarksnumbered]{hyperref}
% Put this package after hyperref
\usepackage[all]{hypcap}
% Don't use tt font for urls
\urlstyle{rm}

% Make a comment stand out visually
\newcommand{\todo}[1]{{\color{blue}$\blacksquare$~\textsf{[TODO: #1]}}}
% Name of a code
\newcommand{\code}[1]{\texttt{#1}}

\hyphenation{Cac-tus-Ein-stein Schwarz-schild South-amp-ton}
\sloppypar

\begin{document}



\title{McLachlan}
\date{August 18, 2009}

\author{Erik Schnetter}
\address{Center for Computation \& Technology, Louisiana State
  University, USA}
\homepage{http://www.cct.lsu.edu/~eschnett/McLachlan/}
\email{schnetter@cct.lsu.edu}

\begin{abstract}
  McLachlan is a free Einstein solver that uses the Cactus framework
  and the Einstein toolkit.  This document describes the basic
  features of the code, and also how to obtain, build, and use the
  code.
\end{abstract}

\maketitle



\section{McLachlan}

McLachlan is a free Einstein solver that uses the Cactus framework and
the Einstein toolkit.  McLachlan was developed by Erik Schnetter and
Peter Diener with the help of Jian Tao and Ian Hinder.  It was first
described in \cite{ES-Brown2007b}, where (to our knowledge) the first
fully fourth order accurate black hole evolution with adaptive mesh
refinement is presented.  The McLachlan web pages are located at
\cite{ES-mclachlanweb}.



% \section{Formulation}
% 
% \todo{To be written.  BSSN equations, time evolution, constraints.}



% \section{Automated Code Generation}
% 
% \todo{To be written.  Kranc.  Modifying the equations.}



\section{Obtaining McLachlan}

McLachlan uses the Cactus Software Framework and the Einstein Toolkit.
Cactus organises applications into \emph{thorns} (modules) that can be
maintained independently of each other.  In order to use McLachlan, it
is necessary to obtain Cactus as well as a set of supporting thorns.

The subsections below describe how to obtain Cactus and other
necessary thorns.  McLachlan contains also a shell script
\code{checkout.sh} that attempts to automate this.  However, this
script is very basic and does not handle errors well.

\subsection{Tools}

Cactus thorns are ususall stored in \emph{repositories} that are
managed by version control systems such as CVS \cite{cvsweb}, SVN
(subversion) \cite{svnweb}, or git \cite{gitweb}.

Before getting the code, you will need to install the following
software on your local system:
\begin{enumerate}
\item wget (or curl)
\item CVS
\item SVN
\item git
\item Perl
\end{enumerate}
These are standard packages, and they should be easily available for
all Linux systems.

\subsection{Cactus}
\label{sec:cactus}

Cactus \cite{Goodale02a, ES-cactusweb} is a software framework that
makes it possible to maintain parts of applications independent of
each other, and combine them into an efficient code when building the
application.  Cactus is described at \url{http://www.cactuscode.org/},
and a new version of the of Cactus web site is currently being
prepared at \url{http://preview.cactuscode.org/download/}.

To obtain Cactus itself as well as a set of basic thorns, follow the
instructions at \url{http://preview.cactuscode.org/download/}.
(Additional thorns specific to numerical relativity are also located
elsewhere.)  Don't use a particular thorn list for this; instead,
download all the basic thorn that Cactus offers.

For reference, here is a brief overview over the commands to do this:
\begin{enumerate}
\item\verb+wget http://preview.cactuscode.org/download/GetCactus+
\item\verb-chmod a+x GetCactus-
\item\verb+./GetCactus+
  
  This checks out Cactus itself (the \emph{flesh}) into a new
  subdirectory \code{Cactus}.  When asked, choose the
  \emph{development version} of Cactus; use the default answer for all
  other questions.
\item\verb+cd Cactus+
\item\verb+make checkout+
  
  This checks out some arrangements with basic thorns for Cactus,
  including the important CactusEinstein arrangement.  Again, use the
  default answer for all questions, except when you are asked for the
  second time whetyer you want to quit.  In this case, quit.
\end{enumerate}

\subsection{Carpet}

Carpet \cite{ES-Schnetter2003b, ES-Schnetter2006a, ES-carpetweb} is a
\emph{driver} for Cactus.  A driver manages memory, handles
parallelism, and performs I/O on behalf of the application.  Carpet
supports adaptive mesh refinement (AMR) and multi-block methods.
Carpet is described at \url{http://www.carpetcode.org/}.

To obtain Carpet, follow the instructions at
\url{http://www.carpetcode.org/get-carpet.html}.  Please check out the
\emph{Development Version}, which is currently quite stable.  (We are
planning to release a new stable version soon.)

In particular, the commands to obtain the development version are:
\begin{enumerate}
\item\verb+cd Cactus+
\item\verb+git clone -o carpet git://carpetcode.dyndns.org/carpet.git+
\item\verb+cd arrangements+
\item\verb+ln -s ../carpet/Carpet* .+
\end{enumerate}
(Don't miss the dot after the \verb+Carpet*+ in the last line.)  Note
that Carpet should be checked out into the main Cactus directory, and
the \code{arrangements} subdirectory needs to contain symbolic links
pointing into the \code{carpet} directory.

\subsection{McLachlan}

McLachlan \cite{ES-Brown2007b, ES-mclachlanweb} in an Einstein solver.
It uses one of the BSSN formulations of the Einstein equations.
McLachlan is described at
\url{http://www.cct.lsu.edu/~eschnett/McLachlan/}, which is where you
may have obtained this documentation.

To obtain McLachlan, issue the following commands:
\begin{enumerate}
\item\verb+cd Cactus+
\item\verb+cd arrangements+
\item\verb+git clone git://carpetcode.dyndns.org/McLachlan.git+
\end{enumerate}
Note that McLachlan needs to be checked out directly into the
\code{arrangements} subdirectory.

McLachlan uses the Kranc code generation package \cite{kranc04,
  Husa:2004ip, krancweb, ES-krancweb}.  Kranc also contains some
thorns that McLachlan needs.  (However, it is not necessary to run
Kranc in order to use McLachlan.  It is only necessary to run Kranc if
McLachlan is modified.)

To obtain Kranc, issue the following commands:
\begin{enumerate}
\item\verb+cd Cactus+
\item\verb+git clone http://www.aei.mpg.de/~ianhin/kranc.git+
\item\verb+cd arrangements+
\item\verb+ln -s ../kranc/Auxiliary/Cactus/KrancNumericalTools .+
\end{enumerate}
(Don't miss the dot at the end of the last line.)  Note that Kranc
needs to be check out into the main Cactus directory, and the
\code{arrangements} subdirectory needs to contain symbolic links
pointing into the \code{kranc} directory.

\subsection{Other Thorns}

All other thorns, including the public Whisky thorns, can be obtained
via the GetCactus script that was downloaded above (see section
\ref{sec:cactus}):

\begin{enumerate}
\item\verb+cd Cactus+
\item\verb+cd ..+
\item\verb+./GetCactus Cactus/arrangements/McLachlan/doc/mclachlan-public.th+
  
  Use the default answer for all questions.
\end{enumerate}

\subsection{Consistency Check}

The \emph{thorn list} \code{mclachlan-public.th} lists the thorns
that are necessary for a simple spacetime evolution.  The thorns are
grouped into arrangements.  All thorns listed in this file must now be
present in the \code{arrangements} subdirectory of the main Cactus
directory.



\section{Building McLachlan}

Building McLachlan and the other thorns requires C, C++, and Fortran
90 compilers, MPI, as well as the BLAS, GSL, HDF5, and LAPACK
libraries.

\subsection{Documentation}

It is best to begin building Cactus with building documentation:
\begin{enumerate}
\item\verb+cd Cactus+
\item\verb+make UsersGuide+
\end{enumerate}
This creates the users' guide as \code{doc/UsersGuide.pdf}.

All Cactus commands are listed with
\begin{enumerate}
\item\verb+make help+
\end{enumerate}

\subsection{Option List}

To build a Cactus application one needs to create an \emph{options
  list}.  This is a text file containing the configuration options
that tell Cactus what compilers and compiler options to use, and where
MPI and the auxiliary libraries are installed.  This process is very
specific to each machine and may require some trial and error.

We distribute options lists for a range of machines that we are using
on the Cactus web site at
\url{http://preview.cactuscode.org/download/configfiles/}.  Option
lists are also available together with the Simulation Factory
\cite{ES-simfactoryweb} at
\url{https://svn.cct.lsu.edu/repos/numrel/simfactory/optionlists/}.

\subsection{Building}

To build a Cactus application one starts with an option list and a
thorn list.  The text below assumes that you have an option list
called \code{einstein-redshift-gcc.cfg}.

To configure an application
\begin{enumerate}
\item\verb+cd Cactus+
\item\verb+make sim-config options=redshift-gcc.cfg \+\\
  \verb+THORNLIST=arrangements/McLachlan/doc/mclachlan-public.th+
\item\verb+make sim+
\end{enumerate}
This is necessary only once, or when the configuration options change.
This will create an application called \code{sim}; of course, the name
could also be different.  Different applications, e.g.\ with different
options or different thorn lists, can exist side by side.

To build the application:
\begin{enumerate}
\item\verb+cd Cactus+
\item\verb+make sim -j4+
\end{enumerate}
The make option \code{-j4} builds 4 files at the same time.  Use this
option if you have several processors available; this will speed up
building the application.  The executable is called \code{cactus\_sim}
and is placed in the \code{exe} subdirectory.

\subsection{Cleaning}

You can also clean the application, removing all object files but
keeping the configuration options and thorn list:
\begin{enumerate}
\item\verb+cd Cactus+
\item\verb+make sim-realclean+
\end{enumerate}



\section{Running McLachlan}

To run the McLachlan code, one needs a \emph{parameter file}.
Parameter files select which thorns are activated at run time, and
what values the thorns' run-time parameters have.  They typically have
a \code{.par} suffix.  Below, we use the parameter file
\code{ks-mclachlan-public.par}.

Cactus applications are started like a regular MPI application.  The
exact mechanism depends on the particular MPI implementation.  On
Redshift, the command is
\begin{enumerate}
\item\verb+cd Cactus+
\item\verb+mkdir simulations+
\item\verb+cd simulations+
\item\verb+env OMP_NUM_THREADS=1 mpirun -np 1 ../exe/cactus_sim \+
\verb+   ../arrangements/McLachlan/doc/ks-mclachlan-public.par+
\end{enumerate}
This parameter file simulates a single, stationary, spinning black
hole in Kerr-Schild coordinates.  It requires about 4~GByte of RAM to
run.

\emph{Note:} If the options list enables OpenMP, then the Cactus
application will be multi-threaded.  Multi-threading can improve
performance and reduce memory consumption, especially when many
($>100$) cores are used.  However, it is usually a bad idea to
over-subscribe cores by having too many threads per node.  It is
usually best to choose both the number of MPI processes per node and
the number of OpenMP threads per process such that their product
equals the number of cores on a node.  The way in which these numbers
are chosen depend on the MPI implementation.



\bibliographystyle{apsrev-titles-manyauthors}
\bibliography{references,publications-schnetter}

\end{document}