Underworld Community Computer Cluster Tips
Note - for getting the code working on the APAC AC in 2008, see UnderworldClusterAPAC.
This page provides a range of tips and links for using
UnderWorld on computer clusters.
Getting an Account on and Using VPAC Computer Clusters
Information and forms for setting up an account on, and using VPAC computer clusters, is available at
https://www.vpac.org/users.
Running Module Load UnderWorld on a Cluster
When using recent releases of
UnderWorld on the following clusters:
you do
NOT need to set up any environment variables. Instead, the module load system automatically sets up the necessary environment variables. Therefore, it is recommended that you comment out
all environment variables, including paths, in your ~/.bashrc or ~/.tcshrc file.
Note: your ~/.bashrc or ~/.tcshrc file will not be overwritten permanently by the module load system.
For lists of applications loaded onto each computer cluster facility, see:
The command to run the
UnderWorld module is
module load underworld or
module load snark, depending on which
UnderWorld module you wish to use:
- "underworld" - latest stable release Underworld-1.1.0, to be available on the VPAC computer clusters edda and tango.
- "snark" - an earlier stable release "Snark 3.0.2"
See
UnderworldReleases if you require further information about each release.
The module command will automatically update your PATH, LD_LIBRARY_PATH etc. to reference the appropriate MPI, gcc, PETSc and
UnderWorld version. All other AuScope Victorian packages such as "PICellerator" and "
gLucifer" are sub-packages of this one, and will be automatically loaded when required.
To use this command, add the following to the top of either:
- each individual pbs file, after the lines specifying the number of processors and walltime, without hashes infront of the module text (e.g. if you want to run some models with the stable build, and / or some with a hand-compiled version of the code) - recommended.
- your ~/.bashrc or ~/.tcshrc file (e.g. if you just want to use the latest module)
APAC and edda
module purge
module load underworld
tango (
Warning: the Tango cluster is still in experimental testing phase and can not yet be relied upon for production modelling)
module unload mpiexec
module load underworld
module load mpiexec
To use the Underworld executable, include the launch command in your
pbs file appropriate to the cluster you are using (see below), where InputFile.xml is the xml file that you want to run (see
Creating your own InputFile).
Note: if your own InputFile is located in a different directory to your pbs file, you will need to provide the relative path to the InputFile from the location of your pbs file.
APAC
mpirun -np X Underworld InputFile.xml
replacing X with the number of processors you want to run your model across.
edda
/usr/local/bin/mpiexec Underworld InputFile.xml
tango (
Warning: the Tango cluster is still in experimental testing phase and can not yet be relied upon for production modelling)
mpiexec Underworld InputFile.xml
Note: the full path to mpiexec is not required on APAC or edda in your pbs script if you add the following to your list of modules:
module load mpiexec
A copy of the InputFiles directory for each module's release can be downloaded or checked out from the
UnderworldReleases page.
Building the Bleeding Edge version of UnderWorld on a Computer Cluster
If you need to use a feature in the Bleeding Edge version of
UnderWorld on a cluster, that is not available in a pre-loaded
UnderWorld module, you now have two options as to how to lod the necessary environment variables, when installing your own copy of the Bleeding Edge version on your account on the cluster.
Submit jobs to be run using the PBS Queueing System
This can best be done by reading the following websites:
Additional Configuration Conditions for Computer Clusters
When compiling
UnderWorld on computer clusters, there is a set of conditions that must be applied when configuring on each cluster. These are in addition to any options you may choose from those listed under
UnderworldDownload#Step_1_Configure. Add the following to the end of your configure command that pertains to the cluster you are using:
apac, marvin, edda, tango: --without-X11 --without-SDL --without-python
ess: --without-X11
Please refer to
Compiling UnderWorld for full instructions on the compiling procedure.
Environment Variables on Computer Clusters
As of 16 May, a Machines repository comes with the latest versions of
UnderWorld, which contains bash environment settings for several computer clusters. You can copy the contents of the relevant file into your .bashrc on the cluster. However some clusters have a module load facility for the Stable Build which sets up all the necessary environment variables for you.
Click here to view files in the Machines repository.
Setting your own environment variables for your own full checkout on a cluster
It is best to have a look in the Machines directory of any checkout of the source code. Alternatively, you can look at the files on the repository website, see
UnderworldReleases for locations.
The Machines directory contains a list of recommended environment variable settings for
all cluster machines.
To use these values, either
- copy and paste the file contents to your ~/.bashrc or ~/.bash_profile file in your home directory on the cluster, or
- reference the machine file for the particular cluster machine from your ~/.bashrc or ~/.bash_profile file in your home directory on the cluster.
Then to load these variables into your terminal environment, type:
source ~/.bashrc
or
source ~/.bash_profile
Performance on Computer Clusters
Please refer to
UnderworldClusterPerformance for tips on timing & memory usage on computer clusters for the April 2006 Stable Build of
UnderWorld.
Technical Support
<< Go back to Creating and Running Models in UnderWorld
<< Go back to UnderWorld Download page
Primary Page maintainers:
to top
