splitting the UsersGuide into three parts and include them in the main tex file

git-svn-id: http://svn.cactuscode.org/flesh/trunk@200 17b73243-c579-4c4c-a9d2-2d5706c11dac

3 files changed, 801 insertions, 0 deletions

diff --git a/doc/UsersGuide/UsersGuide_Part1.tex b/doc/UsersGuide/UsersGuide_Part1.tex
new file mode 100644
index 00000000..7e5ebe28
--- /dev/null
+++ b/doc/UsersGuide/UsersGuide_Part1.tex
@@ -0,0 +1,590 @@
+\part{Introduction}
+
+\chapter{Overview of documentation}
+
+This documentation splits naturally into these parts: 
+
+\begin{itemize}
+
+\item {\bf The history of Cactus and the Concepts behind the
+    CCTK. Getting you up and running.}
+  We give an overview on the required hardware and
+  software and we willll talk you through the installation of a working
+  CCTK. You will be able to verify the correct installation by the
+  CCTK testsuite. We show how to perform some basic simulation and
+  introduce the concepts behind data output. We provide brief
+  information on how to vizualize the 1D, 2D and 3D data.
+
+\item {\bf Part2: How to write a physics thorn.} We introduce a
+  sample thorn that illustrates the implementation of simple initial
+  data and the subsequent evolution. You will learn how to use the
+  programming interface to take advantage of parallelism and modularity.
+
+\item {\bf Part3: How to writes an infrastructure thorn.} In this more
+  advanced part, we talk about user supplied infrastructure routines
+  as {\em additional output routines, boundary conditions, etc.}
+
+\item {\bf The standard toolkit.} 
+
+\end{itemize}
+
+Other topics will be discussed in separate documents:
+
+\begin{itemize}
+
+\item The numerical relativity toolkit.
+
+\item A description of the flesh, for the maintainers.
+
+\end{itemize}
+
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+
+\chapter{A Cactus history}
+
+To begin with a one-phrase summary: Cactus is a numerical
+infrastructure for solving partial differential equations (typically
+in 3-dimensional space and time), a modular code for solving the
+Einstein equations, and an international collaboration in numerical
+general relativity pooling ideas in this code.
+
+Cactus 1.0 was written in 1996 by Joan Mass\'o and Paul Walker out of
+frustration with the numerical relativity codes they had been working
+on. It has come a long way. The name is meant to suggest a modular
+structure consisting of a ``flesh'' (or backbone, or core) and
+``thorns'' (or modules, or collections of subroutines). Some essential
+features of the code have been present from the beginning:
+
+\begin{itemize}
+
+\item Parallelism (the work is split between multiple processors, or
+multiple computers) that is largely automatic and hidden from the
+user.
+
+\item A modular structure with a fixed interface between the flesh and the
+thorns, which allows users to add new parts.
+
+\item Giving the user a choice of C and Fortran for writing thorns, while
+automating the interface between the two languages.
+
+\item A spirit of open-source collaboration similar to the gnu
+project.
+
+\end{itemize}
+
+Other aspects have changed completely since Cactus 1.0. The flesh
+started out as a complete numerical relativity code, using one
+particular formulation of the Einstein equations, with the thorns
+providing only initial data, analysis, and other peripheral
+functions. In 4.0 not only the Einstein equations and general
+relativity basics, but even input/output and parallelism are provided
+by thorns. The only function of the flesh is now to define the
+interface between thorns, to hold varying collections of thorns, and
+to provide an interface between C and Fortran routines. 
+
+The aim of this radicalism is to allow for future growth. We now
+distinguish informally between physics thorns and infrastructure
+thorns. The main infrastructure project for now is adding different
+flavors of adaptive mesh refinement -- while the user writing a
+physics thorn can still large she is working a single-processor,
+unigrid code. Similarly, an infrastructure thorn could be replaced by
+one that does the same job, but simply better. Even the flesh could be
+rewritten in the future, but the interface between the flesh and
+thorns is to remain unchanged.
+
+Cactus thorns can be written in F90, F77 and C (or a mixture). The
+decision to support Fortran was made for the pragmatic reason that
+many people like it, or don't write C. This decision does make the
+flesh more complex. The flesh is written in C, and in perl that
+generates C.
+
+The change from Cactus 3.0 to Cactus 4.0 is the largest one yet. The
+new interface forces modifications on all existing thorns, but it is
+intended not to change again. {\q Can we realistically say any more?}
+
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+
+\chapter{Cactus concepts} 
+
+Cactus is designed to solve time evolution problems in three space
+dimensions by finite differencing on a Cartesian grid. This typically
+includes solving elliptic equations to obtain initial data, evolving
+them by hyperbolic or parabolic equations, and perhaps solving more
+elliptic equations at each time step.  One basic concept is therefore
+a series of initialization steps followed by a do-loop that implements
+the time evolution, along with periodic analysis and output. Problems
+that do not naturally fit into this concept are methods which are
+global in space or time (such as pseudo-spectral methods) or methods
+which do not have a grid (such as finite elements). Thorns interact
+with the flesh by, among other things, announcing when they want to be
+called in this basic schedule.
+
+Another basic concept is parallelism by distributing spatial domains
+among processors. Finite differencing is an almost local procedure,
+which means that each processor can work on its domain of space,
+needing to communicate only with neighboring processors in order to
+work on points of the boundary of its domain. The domain decomposition
+is automatic and largely hidden. A physics thorn can pretend to work
+on a single grid, which in reality is only a chunk of the complete
+numerical domain. This chunk consists of an inner region of points the
+processor ``owns'', and needs to update, surrounded by a ``ghost
+zone'' of points which it can use but doesn't need to update -- they
+are updated by neighboring processors. Clearly this concept is geared
+towards explicit finite differencing schemes for the time-evolution
+part. {\q Say something about elliptic equations here, and implicit
+schemes}
+
+The notion that we are dealing with a collection of fields that live
+on the same (typically three-dimensional) space, and that evolve
+together in time defines the third basic concept, that of ``grid
+functions'' grouped together in a ``grid hierarchy''. Multiple
+grid hierarchies arise, for example, when one wants to cover the
+surface of a sphere with two Cartesian coordinate patches, or in
+adaptive mesh refinement.  Grid functions are defined on Cartesian
+grids that are equally spaced in each of the three coordinates $x$,
+$y$ and $z$. (These could be spherical type coordinates, although that
+was not the original intention.) Grid functions can be two or
+one-dimensional as well as three dimensional.
+
+Routines are grouped in thorns, and thorns informally in toolkits,
+such as the standard toolkit, and the general relativity
+toolkit. The routines inside a thorn can interact in any way, but
+thorns interact with the flesh and with each other only through a
+well-defined interface. Confirming to this interface guarantees that
+thorns will be compatible with, and useful for, all future extensions
+of the code. The interface also helps users and developers to
+understand what a given thorn does. 
+
+The flesh and many thorns are public and can be used by anyone,
+assuming ethical behavior. Public thorns and the flesh are distributed
+by CVS (see {\q where?} below). Public thorns are checked out in
+toolkits. The flesh may change in the future but this should never
+affect users, as Cactus 4.0 is making every effort to codify and
+freeze the interface. Public thorns should of course not lose
+important functionality they once had. (Nevertheless, CVS gives the
+user access to all earlier versions.)
+
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+\chapter{Installation} 
+
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+
+\section{Required software}
+
+In general, Cactus {\em requires} the following set of software to function
+in a single processor mode. Please refer to the architecture section
+\ref{sec:architecture} for architecture specific items.
+\begin{itemize}
+\item{\tt Perl5.0} Perl is used extensively during the CCTK
+  configuration phase. {\tt Perl} is available for nearly all
+  operating systems known to man and can be obtained at {\tt
+    http://www.perl.org}.
+\item{\tt gmake} The GNU make utility needss to be installed. Errors
+ {\q wanna name them here ?} during the make process  are often caused
+ by not using {\tt gmake}. {\tt gmake} can be obtained from any
+ friendly GNU site.
+\item{\tt C/C++} A C/C++ compiler. These can be the GNU compilers as well.
+\item{\tt Fortran 90} 
+\item{\tt CVS} The {\em ``Concurrent Versioning System''} is not needed
+  to run/compile the CCTK, but you are strongly encourage to install
+  this software to take advantage of the update procedures.
+\end{itemize}
+
+\noindent
+The following software is {\em optional} and not needed to get the
+basic CCTK installation on track. Its listed in the order of importance.
+\begin{itemize}
+\item{\tt Fortran 90} CCTK requires a F90 compiler to run modules
+  written in F77 or F90. There is no
+  GNU F90 and just a very limited set of free F90 compilers for
+  various architectures.
+\item{\tt MPI} Currently the communication layer of the CCTK uses the
+  {\em Message Passing Interface (MPI)} provide inter process
+  communication. Supercomputers very often supply a native MPI
+  implementation at their site. CCTK is very likely to work with
+  them. For local area networks we suggest installing the {\tt MPICH}
+  version, which can be obtained for various architectures and
+  operating systems at {\tt http://www-unix.mcs.anl.gov/mpi/}.
+\item{\tt TAGS}: Tags enables you browse through the calling structure
+  of a program by help of function call database. Navigating the CCTK and
+  Toolkits becomes very easy. Emacs and vi both support this database. See
+  \ref{sec:tags} how to install {\tt TAGS}.
+\end{itemize}
+
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+
+\section{Supported architectures}
+Cactus runs on many machines, under a large number of Unix operating
+systems. Here we list all the machines we have compiled and verfied
+Cactus on, including some architecture specific notes. 
+\begin{itemize} 
+\item{\bf SGI Origin 2000}
+\item{\bf SGI}
+\item{\bf Cray T3E}
+\item{\bf Dec Alpha}: Dec operating system and Linux. Single processor
+  mode and MPICH.
+  supported. The Decs need to have the GNU {\tt C/C++} compilers installed.
+\item{\bf Intel Linux}: There is a
+  free Linux F90 compiler available from  {\tt http://www.sierra.com}
+  -- the only free we know of. Single processor mode and MPICH
+  supported.
+\item{\bf Windows32 using Cygwin32 extensions} {\q still true?}
+\item{\bf Windows NT}: succesful compile with the following software:
+  \begin{itemize}
+  \item{\tt DIGITAL Visual Fortran Optimizing Compiler Version: V5.0C}
+  \item{{\tt Microsoft (R) 32-bit C/C++ Optimizing Compiler Version
+        11.00.7022 for 80x86}
+  \item{Cygnus Unix suite version 1.19}
+  \end{itemize}
+\item{\bf Solaris } {\q What is the status on this ??}
+\end{itemize}
+If you did not find you architecture/operating system of
+choice, see section sec\ref{sec:how_to_port} on how to port to an
+unsupported architecture.
+
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+
+\section{Checkout procedure}
+
+Cactus is distributed, added to, and updated through the free CVS
+software. CVS ({\em ``Concurrent Versioning System''}) allows many
+people to work on a large software project together without getting
+into a tangle by keeping track of changes made by the different
+users. For the beginner, we summarize the basics in appendix
+\ref{sec:CVS}, please refer to this section for small description of
+the CVS syntax.
+
+The installation requires about {\q HOW MANY MEGS ? }. See section
+\ref{sec:downsize_CCTK} on how to downsize CCTK installtion if space is tight.
+{\q gmake small ??}
+
+\begin{itemize}
+\item{\bf Login}: Prior to any CVS operation, you need to loginto the central
+  repository. For an  anoymous checkout, type:\\
+  {\tt
+    cvs -d :pserver:cvs\_anon@hod.aei-potsdam.mpg.de:/usr/users/cactus login
+    }
+  You will be prompted for a password which is {\tt anon}. 
+\item{\bf Checkout}: To obtain a fresh CCTK checkout, type
+  {\t
+    cvs -d :pserver:cvs\_anon@hod.aei-potsdam.mpg.de:/usr/users/cactus checkout cactus
+    }
+  The CVS checkout procedure will create a directory called {\tt
+    ./CCTK} and install the CCTK inside this directory.  From now on we
+  will reference all directory names relative to {\tt ./CCTK}. 
+  
+\item{\bf Update}: To update an existing CCTK checkout (to patch in
+  possible changes, etc.), do the following {\em within} the {\tt ./CCTK} directory.
+  {\t
+    cvs -d :pserver:cvs\_anon@hod.aei-potsdam.mpg.de:/usr/users/cactus update
+    }
+  The update process will operate downwards relative to you current position
+  within the CCTK tree. To update only certain directories, change
+  into these directories and issue the update command.
+\item{\bf CVS status}: to obtain a status report on the ``age'' of your
+  CCTK or Toolkit routines (from your current directory position
+  downward), type
+  {\t
+    cvs -d :pserver:cvs\_anon@hod.aei-potsdam.mpg.de:/usr/users/cactus status
+    }
+\item{\bf non-anonymous CVS}: if you have an account at the central
+  repository and you would like to perform any of the operation above
+  {\em non-anonymously}, replace {\tt cvs\_anon} by your login name
+  and provide the appropriate password during the CVS login
+  process. Depending on your permissions, you may then make commits to the CCTK
+  or its Toolkits.
+\item{\bf Commits}: you need to perform a personalized login and have
+  proper permissions to commit code to the repository. 
+\end{itemize}
+
+For more CVS commands please refer to appendix \ref{sec:CVS}.
+
+{\q Change CCTK to cactus in the distribution}
+
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+
+\section{Directory structure}
+
+A fresh CCTK checkout creates a directory {\tt ./CCTK} with the
+following files substructure:
+\begin{itemize}
+
+\item{\tt ./CVS/} the CVS book-keeping directory, present in every subdirectory.
+
+\item{\tt ./doc/} CCTK documentation
+
+\item{\tt ./lib/} contains libraries.
+
+\item{\tt ./src/} contains the source code for the CCTK
+
+\item {\tt ./toolkits/} contains the Cactus toolkits. The toolkits
+  (the actual ``physics'') is not supplied by the CCTK. If the  toolkits
+  you want to use are part of the central repository, they can be
+  checked out in similar way the CCTK.  Each directory inside {\tt
+    toolkits/} contains thorns. {\q ??}
+\end{itemize}
+
+When Cactus is first compiled it creates a new directory {\tt
+./CCTK/build/}. Disk space may be a problem on supercomputers where
+home directories are small. A workaround is to first create a
+build directory on scratch space, say {\tt scratch/cactus\_build/} (where
+{\tt scratch/} is your scratch directory), and soft link ({\tt ln -s
+scratch/cactus\_build cactus/build/}) it to the Cactus directory.
+
+When the code has been compiled for some configuration CONF, the
+directory {\tt build/CONF/} contains the following:
+
+\begin{itemize}
+
+\item {\tt ActiveThorns} 
+
+\item {\tt datestamp.o}
+
+\item {\tt bindings/} contains all the files created by the perl
+scripts, sorted into subdirectories {\tt Parameters}, 
+{\tt Variables}, {\tt Schedule}, {\tt thorn\_***} (one for each thorn).
+
+\item {\tt build/} contains the post-processed source code, the
+dependencies and the object files, sorted into the subdirectories {\tt
+Cactus}, {\tt CactusBindings} and {\tt thorn\_***}.
+
+\item {\tt config-data} : contains the files created by the configure
+script:
+
+\begin{itemize}
+
+\item {\tt config.cache}
+
+\item {\tt config.h}
+
+\item {\tt config.log}
+
+\item {\tt config.status}
+
+\item {\tt fortran\_name.pl} (used to determine how fortran routines
+are called from C) 
+
+\item {\tt make.config.defn} (contains compilers etc. for a configuration)
+
+\item {\tt make.config.deps}
+
+\item {\tt make.config.rule} (rules to generate object files from source files)
+\end{itemize}
+
+\item {\tt lib} : contains the libraries built from the object files,
+
+\begin{itemize}
+
+\item {\tt libCactus.a} for the flesh
+
+\item{\tt llibCactusBinding.a} for the Bindings 
+
+\item {\tt llib***.a} for each thorn
+
+\end{itemize}
+
+{\q It might be useful to give a couple of commands like nm for seeing
+what is in libraries.}
+
+\item {\tt scratch}: empty at the end.
+
+\end{itemize}
+
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+
+\section{Getting help}
+
+Cactus Maint and GNATS
+
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+
+\section{Testing} 
+
+simple make and test suite
+
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+
+\chapter{Compilation} 
+
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+
+\section{Compilation of the flesh, and configurations}
+
+Cactus can be built in different configurations from the same copy of
+the source files, and these different configurations coexist in the
+{\tt cactus/} directory. Different configurations can be for different
+machines. This is useful when your {\tt cactus/} directory is mounted
+on different machines in a cluster, and you want to run cactus on all
+these machines while only keeping and updating a single copy of the
+source files. It is also useful if you want to compile cactus with
+different compilers or optimization settings on the same machine, and
+compare one against the other. As a third example, you may want to
+compile with different sets of thorns.  As a final example, you might
+want to build cactus using different multi-processor communication
+protocols, such as MPI (currently the cactus standard) or GLOBUS.
+
+Building the code should be easy. Just type
+
+{\t gmake <configuration options> <CONF>}
+
+where {\tt <...>} denotes optional arguments. {\tt gmake} is the gnu
+make program. Note that {\tt make} will not work (if you try, you'll
+get funny error messages.) Without any arguments, the make process
+picks a default configuration (looking for standard compilers and so
+on) and gives this configuration a default name (typically the name of
+your operating system). It then runs configuration scripts (in {\tt
+lib/make/}) which figure out details of your computer.
+
+You'll see that the make process has created a {\tt build/} directory,
+and in it a directory {\tt build/CONF}, where {\tt CONF} is the name
+of the configuration, in this case the default name chosen for you.
+Inside {\tt CONF}, {\tt CONF/config-data/} now contains the results of
+the configuration process. To actually compile with this
+configuration, type
+
+{\t gmake CONF}
+
+where {\tt CONF} is the name of the configuration. The argument {\tt
+CONF} is now obligatory. 
+
+gmake will ask if you want to edit the file {\tt
+CONF/ActiveThorns}. As the name says, it contains the names of all the
+thorns you want compiled in. By default it contains all the thorns you
+have checked out. If you say {\tt yes}, gmake opens the editor
+specified in your {\tt EDITOR} shell variable. gmake continues when
+you close the editor. Alternatively, you could have created {\tt
+CONF/ActiveThorns} beforehand. {\tt ActiveThorns} can contain the
+names of individual thorns or entire toolkits {\q Implement this}.
+
+{\q move setup file elsewhere.}
+
+{\q move executable elsewhere, and make one for each configuration.}
+
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+
+\section{Compiling in thorns}
+
+Cactus compiles in all thorns listed in {\tt
+build/CONF/ActiveThorns}. {\q If the thorn has not been checked
+out...} Cactus looks for source code in a thorn's {\tt src}
+directory. It first looks for a file {\tt make.code.defn} which
+should contain
+
+\begin{itemize}
+
+\item {\t SRCS = <list of all source files in this directory>}
+
+\item {\t SUBDIRS = <list of all subdirectories containing source files>}
+
+\end{itemize}
+
+and each subdirectory mentioned should also contain a {\tt
+make.code.defn} file with a {\tt SRCS} line ({\tt SUBDIRS} in a
+subdirectory {\tt make.code.defn} file will be ignored).
+
+The {\tt make.code.defn} file in a directory is included at the top of
+the makefile {\q where does it sit?}  used to build the sources. If a
+file {\tt make.code.deps} exists in the directory {\q where?} it is
+included at the bottom.
+
+The standard make system may be overridden by placing a makefile
+called {\tt Makefile} in the {\tt src/} directory.  This can do
+whatever it likes, but must create a library called
+
+{\tt \$(NAME)}
+
+(which is actually {\tt \$(CCTK\_LIBDIR)/lib<thorn>.a }). {\q We need more detail here. What does ``is
+actually'' mean?}
+
+{\q Watch for CCTK here and elsewhere. Don't remove this query before
+CCTK is gone.}
+
+A makefile is passed the following variables
+
+{\q How many makefiles are involved? Which are created at runtime etc.}
+
+\begin{itemize}
+
+\item {\tt \$(CCTK\_HOME)} - the main CCTK directory
+
+\item {\tt \$(TOP)}    - the CONF directory
+
+\item {\tt \$(SRCDIR)} - the directory in which the source files can be found
+
+\item {\tt \$(CONFIG)} - the directory containing the configuration files
+
+\item {\tt \$(THORN)}  - the thorn name
+
+\item {\tt \$(SCRATCH\_BUILD)} - the scratch directory where f90 module
+files should end up if they need to be seen by other thorns.
+
+\item {\tt \$(NAME)}.
+
+\end{itemize}
+
+
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+
+\section {gmake targets}
+
+(A target for {\tt gmake} can be naively thought of as an argument
+that tells it which of several things listed in the {\tt Makefile} it
+is to do.)
+
+\begin{itemize}
+
+\item {\tt gmake config} will create the default (uname) configuration
+
+\item {\tt gmake <config>-clean} cleans a configuration (rm -r *.o)
+
+\item {\tt gmake <config>-delete} deletes a configuration (rm -r build/<config>)
+
+\item {\tt gmake help} lists all make options
+
+\item {\tt gmake tags} creates a {\tt vi} style tags file
+
+\item {\tt gmake TAGS} creates an Emacs style TAGS file
+
+\item {\tt gmake distclean} nukes your build directory.
+
+\end{itemize}
+
+
+{\q Tom, this is for you}
+
+        b) Make methodology (dependencies,libraries, thorn make files
+        etc)
+
+      d) Configuration options
+
+      f) (New platforms??)
+
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+
+\chapter{Running Cactus}
+
+      a) Command line options
+      b) Simple parameter file syntax (will be explained fully in thorn
+         writers part)
+      c) Description of standard parameters (although maybe should be an
+         appendix)
+      d) Checkpointing and outputting
+      e) Reference to web pages with machine dependant stuff
+      f) Understanding the screen output (RFR tree, errors and warnings etc)
+      g) Convergence mode
+      h) Environment variables
+
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+
+\chapter{Looking at output}
+
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
diff --git a/doc/UsersGuide/UsersGuide_Part2.tex b/doc/UsersGuide/UsersGuide_Part2.tex
new file mode 100644
index 00000000..0ab76a3d
--- /dev/null
+++ b/doc/UsersGuide/UsersGuide_Part2.tex
@@ -0,0 +1,5 @@
+\part{Using Cactus}
+
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+
diff --git a/doc/UsersGuide/UsersGuide_Part3.tex b/doc/UsersGuide/UsersGuide_Part3.tex
new file mode 100644
index 00000000..c4354abb
--- /dev/null
+++ b/doc/UsersGuide/UsersGuide_Part3.tex
@@ -0,0 +1,206 @@
+\part{Application thorn writing}
+
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+
+\chapter{Thorn concepts} 
+
+      (This section has to contain enough explanation to make the rest of
+       the writers guide readable the first time through)
+(The hardest bugs to find are
+those arising from plausible but incorrect assumptions about the
+behavior of someone else's thorn.)
+      a) Again probably emphasize collaboration, what are thorns,
+         toolkits, how to share them.
+      b) Things to think about before you start programming:
+             Language, read all the documentation, emphasize use of
+             standard supported Cactus infrastructure
+      c) Available data types
+         i)   Scalars
+         ii)  Arrays and GFs
+         iii) Groups
+      d) Ghost zones and parallelism
+      e) Understanding the RFR concept
+      f) Understanding the GH concept
+
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+
+\chapter{Anatomy of a thorn}
+
+Each thorn must be in a toolkit. Toolkits are fairly informal. If you
+want, you can just make your own toolkit directory, and stick your new
+thorn into it.
+
+      a) Creating a thorn (gmake new thorn)
+      b) Thorn directory structure
+      c) Thorn make files
+      d) Thorn configuration
+         (i)   Parameters (param.ccl)
+         (ii)  Variables (interface.ccl)
+         (iii) Scheduler (schedule.ccl)
+
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+
+\chapter{Putting code into your thorn}
+
+      a) What the flesh basically provides (header files, macros, ...)
+      b) Argument lists and parameters
+      c) A simple example (Baloney)
+      d) Differences between F and C with the interface
+      e) A more complex example (Wavetoy)
+      f) Error handling and code stopping
+      g) Calling C from F, F from C, (C++ from F? ....)
+      h) Good Cactus Style Programming Guidelines
+      i) Adding a test suite
+      j) Sharing your thorn with the world.
+
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+
+\appendix{Appendix}
+
+\chapter{Using GNATS}
+\chapter{Using CVS}
+\chapter{Using TAGS}
+    \section{TAGS with emacs}
+    \section{TAGS with vi}
+
+
+\chapter{Configuration file syntax}
+    
+\section{General Concepts}
+
+Each thorn is configured by three compulsory files in the top level thorn
+directory:
+\begin{itemize}
+\item{} interface.ccl
+\item{} param.ccl
+\item{} schedule.ccl
+\end{itemize}
+
+\section{interface.ccl}
+
+The interface configuration file consists of a header block giving details 
+of the thorns relationship with other thorns, and then a series of blocks 
+listing the thorn's global variables.
+The header block has the form:
+{\t 
+\begin{verbatim}
+implements: <implementation>
+[inherits: <implementation>, <implementation>]
+[friend: <implentation>, <implementation>]
+\end{verbatim}
+}
+where
+\begin{itemize}
+\item{} The implentation name must be unique among all thorns, unless ????.
+\item{} Inheriting from another implentation makes all that implentations public 
+        variables and parameters available to your thorn. At least one thorn 
+        providing any inherited implentation must be present at compile time.
+        A thorn cannot inherit from itself. Inheritance is associative, but not
+        commutative.
+\item{} Being a friend of another implementation makes all that implementations 
+        protected and public (????)
+        variables and parameters available to your thorn. A friend is not required 
+        for compilation. A thorn cannot be its own friend. Friendship is 
+        associative and commutative.
+\end{itemize} 
+
+The thorn's variables are defined by:
+{\t
+\begin{verbatim}
+[<access>:]
+
+VTYPE <group name> [TYPE=<gtype>] [DIM=<dim>] 
+[{
+   <variable name>[,]<variable name>
+   <variable name>
+}]
+\end{verbatim}}
+
+\begin{itemize}
+\item{} {\t access} defines which thorns have can use the following 
+        groups, and can be either {\t public}, {\t protected} or {\t private}.
+\item{} {\t VTYPE} defines the data type of the variables in the group.
+Supported data types are {\t LOGICAL}, {\t INTEGER}, {\t CHAR} and {\t REAL}.
+\item{} {\t group name} must be a alpha-numeric name which is unique 
+within the scope of the thorn. A group name is compulsory.
+\item{} {\t TYPE} designates the kind of variables held by the group.
+The choices are {\t GF}, {\t ARRAY} or {\t SCALAR}. This field is 
+optional, with the default variable type being {\t SCALAR}.
+\item{} {\t DIM} defines the spatial dimension if the group is of type
+{\t ARRAY} or {\t GF}, and can take the value {\t 1}, {\t 2}, or {\t 3}.
+The default value is {\t DIM=3}.
+\item{} The block following the group declaration line contains a list of
+        variables contained in the group. All variables in a group have 
+        the same data type, variable type and dimension. The list can be 
+        separated by spaces, commas, or new lines. The variable names 
+        must be unique within the scope of the thorn. 
+        A variable can only be a member of 
+        one group. The block must be delimited by brackets on new lines.
+        If no block is given after a group declaration line, a 
+        variable with the same name as the group is created.
+\end{itemize} 
+
+\section{param.ccl}
+
+\section{schedule.ccl}
+
+The schedule configuration files consists of 
+\begin{itemize}
+\item{} assignments to switch on storage and communications 
+        for array variables at the start of program execution.
+\item{} {\it schedule blocks} which schedule a subroutine from the
+        thorn to be called at a specified time during program 
+        executation. Statements within the schedule block can
+        be used to switch on storage and communication for groups
+        of variables during the duration that the subroutine is called.
+\item{} Conditional statements
+\end{itemize}
+
+{\it assignments statements} have the form:
+{\t
+\begin{verbatim}
+[STORAGE: <group>, <group>]
+[COMMUNICATION: <group>, <group>]
+\end{verbatim}
+}
+
+Each {\it schedule block} in the file {\t schedule.ccl} must have
+the syntax:
+{\t
+\begin{verbatim}
+schedule [<function name>] at <time>
+{
+  [LANG: <language>]
+  [STORAGE: <group>,<group>]
+  [COMMUNICATION: <group>,<group>]
+  [TRIGGER: <grid function>,<grid function>]
+}
+\end{verbatim}
+}
+
+Any schedule block or assignment statements can be optionally 
+surrounded by conditional {\t if-elseif-else}
+constructs using the parameter data base. These can be nested,
+and have the general form:
+{\t
+\begin{verbatim}
+if (CCTK_Equals(<parameter>,<string>)) 
+{
+  [<assignments>]
+  [<schedule blocks>]
+}
+\end{verbatim}
+Conditional constructs cannot be used inside of a schedule block.
+
+
+\chapter{CCTK Parameters}
+\chapter{Thorn-Flesh Interface}
+
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+
+\end{document}