ChemWiki - User contributions [en]

Resgrp:comp-photo-hpc

2017-04-20T09:35:22Z

Ipolyak:

IC has two centrally-managed HPC computer systems: a PC cluster CX1, and a small Silicon Graphics AX1.
We mainly use CX1, as it has many more accessible nodes and processors. Also, we own two nodes, which have a dedicated batch queue (pqmb) for the group, mainly for short test calculations. The chemistry queue is pqchem.

More details here: [http://www.hpc.ic.ac.uk high performance computing]

Join the [https://mailman.ic.ac.uk/mailman/listinfo/hpc-announce mailing list]. If you have problems, ask around within the group first, otherwise contact Matt Harvey in HPC support directly (m.j.harvey@imperial.ac.uk).

== Using The Cluster ==
Before running calculations on the cluster, look at the tutorial:

[https://www.ch.ic.ac.uk/wiki/index.php/Using_the_cluster_:_tutorial%2C_examples Using the cluster : tutorial and examples]

Below is a summary / reminder.

== Connecting ==
To connect to the PC cluster and forward display information for X-windows, use '''ssh -Y myname@login.cx1.hpc.ic.ac.uk'''

Use your IC college account (as 'myname').

This connects to one of two front-end nodes. All cluster nodes (the rest are for running calculations through a queueing system) share common file systems.
(To compile Gaussian code, need to specify login-0 explicitly when connecting, as this is the node the supported Gaussian compiler is licensed for).

Once connected, using the command id should give something like the following:

[login-0 ~]$ id
uid=45751(mjbear) gid=11000(hpc-users) groups=1010(gaussian-users),11000(hpc-users),11100(gaussian-devel),11232(pgi-users)

To access the current development version of Gaussian, you will need to be in the ''gaussian-devel'' group (and sign the developer's license agreement).
To access run-time libraries to run Gaussian, you also need to be in the ''pgi-users'' groups.
Both should have been set up when your account was created.

== Queuing system ==

PBS queuing system. '''xpbs''' gives you an interactive display.
Need to supply a script for this queuing system that requests the resources to run the calculation you want.
Examples in
/home/gaussian-devel/test_h11
(This is for the development version of Gaussian that was current as of --[[User:Mjbear|Mjbear]] 11:46, 23 March 2011 (UTC))

Using files in this directory, the command '''qsub jobscript_test009''' sends the following script to the queueing system:

[login-0 test_g01]$ cat jobscript_test009
#PBS -l ncpus=2
#PBS -l mem=1700mb
#PBS -l walltime=00:09:00
#PBS -joe

module load gaussian/devel-modules
module load gdvh11

gdv < /home/mjbear/test_h11/test009.com > $WORK/test009.log

In general, it's best to request the resources you really need! Rather than try to second-guess the queueing system.

Queue suggestion: ''xdbg'' is for 4 processor calculations using up to 15600 MB for 15 mins. This is basically for running calculations on one node - but an nprocshared=2 nproclinda=2 calculation will also work if there's space.

We have two of our own nodes, which are 8 processor and 12 GB.
These have slow memory access, so are best used for short test calculations to check larger jobs will run.

To run a calculation on our private nodes you need to:

1) use qsub -q pqmb
to send the job to our private queue.
(This can be included in the job script itself, as a PBS option)

2) '''select=1:ncpus=8'''
-will give you 8 cores, 1 node (shared memory parallelism only).

'''select=2:ncpus=8'''
-will give you 16 cores, 2 nodes (shared + distributed memory parallelism). See test160.com for an example.

Output files go on $WORK filesystem. This is not backed up!

To Do:
# using '''cpsub''' and '''cpcomchk'''

== Nodes on pqmb ==

We have an own queue named "pqmb" on CX1 (see above). Jobs can be directed to pqmb using <code>#PBS -q pqmb</code>. As of April 2017, there exist three groups of nodes accessible by their microarchitecture variable through PBS:

{|class=wikitable
|-
! Group
! Nodes
! Cores/Node
! Memory /Node (GB)
! Microarchitecture
! Gaussian
|-
| 104
| 2
| 12
| 50
| westmere
| G03+G09
|-
| 5
| 8
| 16
| 132
| sandybridge
| G03+G09+G16
|-
| 100
| 8
| 24
| 264
| broadwell
| G03+G09+G16
|}

This table shows the maximum resources available for each node. A single node with 8 cores available in the Broadwell group may be selected using <code>nodes=1:broadwell:ppn=8</code>. Replacing <code>broadwell</code> with one of the microarchitecture variables above will allow you to specify which node to run on. Note that Gaussian 16 will not run on the old Westmere nodes.

Example:

<pre>
#PBS -l nodes=1:broadwell:ppn=8
#PBS -l mem=16000mb
#PBS -l walltime=2096:00:00
#PBS -q pqmb
</pre>

This script will request one node in the Broadwell group with 8 CPUs and 16000MB of RAM. Note that the current maximum walltime on the private queue is 2096 hours.

With the above notation, multiple nodes may be selected in a job. <code>ppn</code> defines the processors per node to be used.

One can also run jobs across different types of nodes, as follows:

<code>#PBS -l nodes=2:broadwell:ppn=24+sandyb:ppn=16</code>

Further, a '''specific node''' can be assigned using the same notation:

<code>#PBS -l nodes=cx1-100-4-3:ppn=4</code>

The <code>-l select</code> argument can also be used, but does not seem to work well for running across several nodes. If used, the following format can be used:

<pre>
#PBS -l select=1:ncpus=8:broadwell=true
#PBS -l mem=16000mb
#PBS -l walltime=2096:00:00
#PBS -q pqmb
</pre>

Resgrp:comp-photo-hpc

2017-04-10T17:29:53Z

Ipolyak:

IC has two centrally-managed HPC computer systems: a PC cluster CX1, and a small Silicon Graphics AX1.
We mainly use CX1, as it has many more accessible nodes and processors. Also, we own two nodes, which have a dedicated batch queue (pqmb) for the group, mainly for short test calculations. The chemistry queue is pqchem.

More details here: [http://www.hpc.ic.ac.uk high performance computing]

Join the [https://mailman.ic.ac.uk/mailman/listinfo/hpc-announce mailing list]. If you have problems, ask around within the group first, otherwise contact Matt Harvey in HPC support directly (m.j.harvey@imperial.ac.uk).

== Using The Cluster ==
Before running calculations on the cluster, look at the tutorial:

[https://www.ch.ic.ac.uk/wiki/index.php/Using_the_cluster_:_tutorial%2C_examples Using the cluster : tutorial and examples]

Below is a summary / reminder.

== Connecting ==
To connect to the PC cluster and forward display information for X-windows, use '''ssh -Y myname@login.cx1.hpc.ic.ac.uk'''

Use your IC college account (as 'myname').

This connects to one of two front-end nodes. All cluster nodes (the rest are for running calculations through a queueing system) share common file systems.
(To compile Gaussian code, need to specify login-0 explicitly when connecting, as this is the node the supported Gaussian compiler is licensed for).

Once connected, using the command id should give something like the following:

[login-0 ~]$ id
uid=45751(mjbear) gid=11000(hpc-users) groups=1010(gaussian-users),11000(hpc-users),11100(gaussian-devel),11232(pgi-users)

To access the current development version of Gaussian, you will need to be in the ''gaussian-devel'' group (and sign the developer's license agreement).
To access run-time libraries to run Gaussian, you also need to be in the ''pgi-users'' groups.
Both should have been set up when your account was created.

== Queuing system ==

PBS queuing system. '''xpbs''' gives you an interactive display.
Need to supply a script for this queuing system that requests the resources to run the calculation you want.
Examples in
/home/gaussian-devel/test_h11
(This is for the development version of Gaussian that was current as of --[[User:Mjbear|Mjbear]] 11:46, 23 March 2011 (UTC))

Using files in this directory, the command '''qsub jobscript_test009''' sends the following script to the queueing system:

[login-0 test_g01]$ cat jobscript_test009
#PBS -l ncpus=2
#PBS -l mem=1700mb
#PBS -l walltime=00:09:00
#PBS -joe

module load gaussian/devel-modules
module load gdvh11

gdv < /home/mjbear/test_h11/test009.com > $WORK/test009.log

In general, it's best to request the resources you really need! Rather than try to second-guess the queueing system.

Queue suggestion: ''xdbg'' is for 4 processor calculations using up to 15600 MB for 15 mins. This is basically for running calculations on one node - but an nprocshared=2 nproclinda=2 calculation will also work if there's space.

We have two of our own nodes, which are 8 processor and 12 GB.
These have slow memory access, so are best used for short test calculations to check larger jobs will run.

To run a calculation on our private nodes you need to:

1) use qsub -q pqmb
to send the job to our private queue.
(This can be included in the job script itself, as a PBS option)

2) '''select=1:ncpus=8'''
-will give you 8 cores, 1 node (shared memory parallelism only).

'''select=2:ncpus=8'''
-will give you 16 cores, 2 nodes (shared + distributed memory parallelism). See test160.com for an example.

Output files go on $WORK filesystem. This is not backed up!

To Do:
# using '''cpsub''' and '''cpcomchk'''

== Nodes on pqmb ==

We have an own queue named "pqmb" on CX1 (see above). Jobs can be directed to pqmb using <code>#PBS -q pqmb</code>. As of April 2017, there exist three groups of nodes accessible by their microarchitecture variable through PBS:

{|class=wikitable
|-
! Group
! Nodes
! Cores/Node
! Memory /Node (GB)
! Microarchitecture
! Gaussian
|-
| 104
| 2
| 12
| 50
| westmere
| G03+G09
|-
| 5
| 8
| 16
| 132
| sandybridge
| G03+G09+G16
|-
| 100
| 8
| 24
| 264
| broadwell
| G03+G09+G16
|}

This table shows the maximum resources available for each node. A single node with 8 cores available in the Broadwell group may be selected using <code>nodes=1:broadwell:ppn=8</code>. Replacing <code>broadwell</code> with one of the microarchitecture variables above will allow you to specify which node to run on. Note that Gaussian 16 will not run on the old Westmere nodes.

Example:

<pre>
#PBS -l nodes=1:broadwell:ppn=8
#PBS -l mem=16000mb
#PBS -l walltime=2096:00:00
#PBS -q pqmb
</pre>

This script will request one node in the Broadwell group with 8 CPUs and 16000MB of RAM. Note that the current maximum walltime on the private queue is 2096 hours.

With the above notation, multiple nodes may be selected in a job. <code>ppn</code> defines the processors per node to be used.

One can also run jobs across different types of nodes, as follows:

<code>#PBS -l nodes=2:broadwell:ppn=24+sandyb:ppn=16</code>

The <code>-l select</code> argument can also be used, but does not seem to work well for running across several nodes. If used, the following format can be used:

<pre>
#PBS -l select=1:ncpus=8:broadwell=true
#PBS -l mem=16000mb
#PBS -l walltime=2096:00:00
#PBS -q pqmb
</pre>