Talk:Mod:Hunt Research Group/cx2

Setting up cx2 (notes by Ling)

1. PBS runscript:

PBS -lselect=N:ncpus=C:mpiprocs=M:ompthreads=T:mem=23500mb

N number of nodes, each with 23.5GB of free memory and 8 cores. N = 8, 16, 32, 64.

M sets the number of MPI processes that will be run per node, default = C (8 or 12).

T is the number of OpenMP threads per MPI rank, default = 1.

C is the number of cores, for 8 core nodes set this to 8, for 12 core nodes set this to 12.

For example, a simple MPI job that uses all the cores on 4 8-core nodes would require:

PBS -lselect=4:ncpus=8:mpiprocs=8:ompthreads=1:mem=23500mb

And for a hybrid job (means mixed OpenMP/MPI) that runs 2 MPI ranks per node, each with 4 OpenMP threads:

PBS -lselect=4:ncpus=8:mpiprocs=2:ompthreads=4:mem=23500mb

PBS allocates nodes exclusively to a job, so even if M is not 8 no other jobs will be run on the idle cores.

2. MPI / Open MP

They will run best in MPI-only form.

3. Queue

Shouldn't specify any queue when submitting.

(N=8)*(C=8) = 64 cpus => q64

(N=16)*(C=8) = 128 cpus => q256 (short queuing time)

(N=32)*(C=8) = 256 cpus => q256 (long queuing time)

(N=64)*(C=8) = 512 cpus => q256 (unbelievably long queuing time)

4. $SCRATCH

Use $SCRATCH on cx2. $SCRATCH and $TMPDIR on cx2 are using the same space so on cx2, shouldn't need to copy or move.

To make a link to the resart file instead of copying it - this will save time and disk space.