ChemWiki - User contributions [en]

Imperial CX1: Instructions and basic concepts of parallel computing

2024-02-13T21:06:07Z

Hz1420: /* The external coordinator: What is a batch system */

This tutorial is divided into 2 separate sections. In the fist section, introductions and available resources of CX1 are listed and classified. Since the [https://icl-rcs-user-guide.readthedocs.io/en/latest/ Research Computing Service (RCS)] team already developed great tutorials on their webpages, this part functions as a guide towards RCS webpages with necessary supplementary comments. In the second section, basic concepts of parallel computing and explanations of important terms are introduced. The main focus of this section is helping beginners to understand how high-performance computers (HPC) works on the basis of their daily practise.

''This tutorial was initially written between Feb. and Mar. 2022 to be shared within the group for induction and training proposes <ref>https://spica-vir.github.io/posts/Connect-to-the-Imperial-Cluster/</ref><ref>https://spica-vir.github.io/posts/Structure-and-usage-of-clusters/</ref>. Special thanks to Mr K. Tallat-Kelpsa, Ms A. Arber, Dr G. Mallia and Prof N. M. Harrison.''

== Introduction to CX1 ==
CX1 is the old name of the first HPC that served the whole college. New facilities (known as CX2) were gradually installed and integrated with the old system (CX3, a rather short-lived domain), while CX1 remains to be the most popular name that generally referring to the college-owned clusters. To grant a student access to CX1, the group PI can, on behave of that student, ask RCS team to add the specified account into HPC active user mailing list.
=== Connect to CX1 ===
CX1 is typically accessed via ssh (secured shell). Linux command line (Linux & MacOS) / sub-system (Windows 10,11) <ref>https://learn.microsoft.com/en-us/windows/wsl/install</ref> / SSH client (such as XShell <ref>https://www.xshell.com/en/xshell/</ref>) can be used. VPN is needed for off-campus users.

* [https://www.imperial.ac.uk/admin-services/ict/self-service/connect-communicate/remote-access/virtual-private-network-vpn/ Step-by-step guide to setup VPN]
* [https://www.imperial.ac.uk/admin-services/ict/self-service/research-support/rcs/support/getting-started/using-ssh/ How to use ssh]

In linux command line, use the following command to connect CX1:

<code>
~$ ssh -XY username@login.hpc.ic.ac.uk
</code>

P.S. <code>-XY</code> option can be omitted for most of cases, if you do not need GUI to run that program.

Alternatively, when the VPN service is unstable or even not available, it is possible to channel through the gateway of the cluster via a client, which is an 'agent'. To visit CX1, type the previous command in the client's command line.

<code>
~$ ssh username@sshgw.ic.ac.uk
</code>

Use the <code>scp</code> command to upload / download files, which is similar to <code>ssh</code> and <code>cp</code> command. For example, to upload a file:

<code>
~$ scp /local/path/file_name username@login.hpc.ic.ac.uk:/path/file_name
</code>

=== Usage ===
The [https://icl-rcs-user-guide.readthedocs.io/en/latest/hpc/ RCS Wiki Page in ReadTheDocs] contains information needed. [https://www.imperial.ac.uk/admin-services/ict/self-service/research-support/rcs/get-support/contact-us/ The support page], [https://wiki.imperial.ac.uk/display/HPC/Attend+a+clinic online clinic] and [https://wiki.imperial.ac.uk/display/HPC/Courses courses from graduate school] are available. To examine the status of CX1, use [https://status.rcs.imperial.ac.uk/d/u0zkcYQ7z/rcs-status?orgId=2&refresh=1m RCS status page].

==== Environmental Variables and Disk Space ====
Use <code>env</code> to access all the environmental variables - be careful, the output is '''HUGE'''. Some useful environmental variables:

* <code>${USER}</code> The user's college account, i.e., login credential.
* <code>${HOME}</code> '/rds/general/user/${USER}/home', or '~', which has 1 TB disk space for data backups.
* <code>${EPHEMERAL}</code> '/rds/general/user/${USER}/ephemeral' Temporal unlimited disk space lasting for 30 days. Suitable for running calculations.
* <code>${PATH}</code> Path to the executable can be attached for quick access. The Environment Modules package (see below) can automatically do that.

==== Software Management ====

The Environment Modules<ref>https://modules.readthedocs.io/en/latest/</ref> package is implemented on CX1 to manage computing software (see the following section for introductions). Basic commands are listed below:

* <code>module avail</code> List the available modules
* <code>module load mod_name</code> Load a specific module, 'mod_name'
* <code>module rm mod_name</code> Remove a specific module, 'mod_name'
* <code>module list</code> List all the loaded modules in the current environment
* <code>module help mod_name</code> Check the instructions of the module 'mod_name'

''Note: There is a CRYSTAL14 module in the list. For users in NMH's group, the latest CRYSTAL edition is available, so do not use that module.''

==== Job Partition Guide ====
A hierachy of jobs is designed for the optimial efficiency of CX1. The current job partition guide is available on [https://icl-rcs-user-guide.readthedocs.io/en/latest/hpc/queues/classes-of-jobs/ RCS Wiki Page]

==== Batch System ====

The PBS batch system <ref>https://en.wikipedia.org/wiki/Portable_Batch_System</ref> is used on CX1 (see the following section for the meaning of batch system). Basic commands of PBS are listed below:

* <code>availability</code> Check the availability of computational resources
* <code>qsub</code> filename.qsub</code> Submit the job 'filename'
* <code>qstat</code> Check the state of submitted jobs
* <code>qdel jobID</code> Kill the process with the ID number 'jobID'

To examine the queue status across the whole system, use [https://status.rcs.imperial.ac.uk/d/u0zkcYQ7z/rcs-status?orgId=2&refresh=1m RCS status page].

==== A General Job Submission Script ====
A general job submission script for CX1 is developed by the author himself. See the [https://github.com/cmsg-icl/HPC-job-submission GitHub repository] of CMSG for details. Parameterised software includes: CRYSTAL14/17/23, Quantum Espresso 7, LAMMPS, GROMACS, GULP6.

== Basic Concepts of Parallel Computing ==
A brief introduction to parallel computing is given in this section by taking CX1, a medium-sized general-propose cluster, as an example.

=== Divide a job: Nodes, Processors and Threads ===

;Node
:A bunch of CPUs and probably with GPUs / coprocessors for acceleration. Memory and input files are shared by processors in the same node, so a node can be considered as an independent computer. The communication between nodes are achieved by ultra-fast network, which is the bottleneck of modern clusters.

;Processor
:The unit to deal with a 'process', also known as 'central processing unit', or CPU. Processors in the same node communicate via shared memory.

;Thread
:Subdivision of a process. Multiple threads in the same process share the resources allocated to the CPU.

The figure on the right hand side illustrates the hierarchy of node, processor, and thread. '''Note:''' The word 'processor' is not a very accurate term. Might be better with 'process' (I am just too lazy to update that figure). Many modern CPUs supports sub-CPU threading, which means the number of logical CPUs is larger than physical CPUs, so it is possible to have multiple threads within 1 processor. However, it is also possible to use multiple processors for 1 process, or even 1 thread.

[[File:Job_Partition.png|450px|right|Job Partition]]

==== Multiple processes vs multiple threads ====

From the figure above, it is not difficult to distinguish the differences between a 'process' and a 'thread': process is the smallest unit for resource allocation; thread is part of a process. The idea of 'thread' is introduced to address the huge difference in the speed of CPU and RAM. CPU is always several orders of magnitude faster than RAM, so typically the bottleneck of a process is loading the required environment from RAM, rather than computations in CPU. By using multiple threads in the same process, various branches of the same program can be executed simultaneously. Therefore, the shared environmental requirements doesn't need to be read from RAM for multiple times, and the loading time for threads is much smaller than for processes.

However, multithreading is not always advantageous. A technical prerequisite is that the program should be developed for multithread proposes. Python, for example, is a pseudo-multithread language, while Java is a real one. Sometimes multithreading can lead to catastrophic results. Since threads share the same resource allocation (CPU, RAM, I/O, etc.), when a thread fails, the whole process fails as well. Comparatively, in multiple processes, other processes will be protected if a process fails.

In practice, users can either run each process in serial (i.e., number of threads = 1), or in parallel (i.e., number of threads > 1) on clusters. However, '''the former one is recommended''', because of more secured resource managements. The latter is not advantageous. Besides the problem mentioned above, it might lead to problems such as memory leak when running programs either: not developed for multithreading / requires improper packages (Here is [https://docs.archer2.ac.uk/known-issues/#oom-due-to-memory-leak-in-libfabric-added-2022-02-23 a famous issue] with libfabric on ARCHER2 identified in early 2022).

==== More nodes vs more CPUs ====

When the allocated memory permits, from my experience, using more CPUs/processes per node is usually a better idea, considering that all nodes have independent memory space and the inter-node communications are achieved by wired networks. It almost always takes longer to coordinate nodes than to coordinate processors within the same node.

=== The internal coordinator: What is MPI ===

Message passing interface, or MPI, is a standard for communicating and transferring data between nodes and therefore distributed memories. It is utilised via MPI libraries. The most popular implementations include:

* MPICH <ref>https://www.mpich.org/</ref> - an open-source library
* Intel MPI <ref>https://www.intel.com/content/www/us/en/developer/tools/oneapi/mpi-library.html#gs.xld8oa</ref> - a popular implementation of MPICH especially optimised for Intel CPUs
* OpenMPI <ref>https://www.open-mpi.org/</ref> - an open-source library
* OpenMP <ref>https://www.openmp.org/</ref> - Not MPI; parallelization based on shared memory, so only implemented in a single node; can be used for multithreading

In practice, a hybrid parallelization combining MPI and OpenMP to run multithread jobs on cluster is allowed, though sometimes not recommended. The first process (probably not a node or a processor) is usually allocated for I/O, and the rest is used for parallel computing.

So far, MPI only supports C/C++ and FORTRAN, which explains why all parallel computing software is based on these languages. To launch an executable in parallel, one should use: <code>mpiexec</code> or <code>mpirun</code>.

=== Secure your storage: Tmp memory, Work directory and home directory ===

Almost all the modern clusters have separate disk spaces for differently proposes, namely, temporary memory, work directory and home directory. This originates again from the famous speed difference between CPU and RAM/ROM. 2 distinctly kinds of disks are used respectively to improve the overall efficiency and secure important data:

* For temporary memory large, high-frequency disks are used. It is allocated by job requests, which is not accessible by login nodes. Everything is erased after the job is terminated.
* For work directory, large, high-frequency disks are used. Data stored in work directory is usually not backed up, and in the case of CX1, will be automatically cleaned after a fixed time length.
* For home directory, mechanical disks with slower read/write frequency but better robustness are used. Usually files in home space are backed up.

For large clusters like ARCHER2 <ref>https://www.archer2.ac.uk/</ref>, the work directory and the home directory are completely separated, i.e., directory is only viable by login nodes; work directory is viable by both job and login nodes. Job submission in home directory is prohibited. For more flexible clusters like Imperial CX1, submitting jobs in home directory and visiting of home directory by job nodes are allowed, but storing temporary files during calculation in home directory is still not recommended because of the potential influence on other files and the reduced overall efficiency. (And it is not something new for CX1 users to receive the RDS failure news email)

=== Setup your environment: What does an application need? ===

==== Executable ====
The binary executable should, theoretically, all be stored in '\usr\bin'. This never happens in practice, unless you are a fanatical fundamentalist of the early Linux releases. To guide your system to the desired executable, you can either laboriously type its absolute path every time you need it or add the path to the environmental variable:

<code>
~$ export PATH=${PATH}:path_to_bin
</code>

Running any executable in parallel requires mpi to coordinate all the processes/threads. The path to mpi executable is also required. Besides, many scientific codes require other specific environmental variables such as linear algebra packages. Read their documentations for further information.

==== .lib/.a/.o files ====

When writing a script, you might need some extra packages to do more complex jobs. Those packages are developed by experts in computer science and can be called by a line of code. The same thing happens when people were developing applications like CRYSTAL and ONETEP.

However, scientific computing codes are usually distributed in the form of source code. Source codes in FORTRAN/C/C++ need be compiled into a binary executable. There are 2 options during compiling:

# Include the whole package as long as one of its functions is called, also known as a 'static lib'.
# Only include a 'table of contents' when compiling, also known as 'dynamic lib'. The packages needed are separately stored in '.dll/.so' files, making it possible for multiple applications sharing the same lib.

Details about compilation are beyond the scope of this post. The thing is: when running a dynamically linked application, information should be given to help the code find the libs needed. This can be specified by:

<code>
~$ export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:path_to_lib`
</code>

For statically linked applications, usually you need not worry about it - but the volume of the compiled executable might make you wonder whether there is an alternative way.

==== Conflicts ====

Improper previous settings may lead to a wrong application, or a wrong version, if multiple applications with similar functions are installed in the system, such as Intel compiler and GCC, OpenMPI and MPICH - a common phenomenon for shared computing resources. To avoid this, the path to the undesired application or lib should be removed from the environmental variables.

==== Environmental Modules ====

Environmental Modules <ref>http://modules.sourceforge.net/</ref> is a popular software managing the necessary environmental setups and conflicts for each application. It can easily add or erase the environmental variables by commands (such as <code>module load</code>, <code>module rm</code>) and modulefiles written in Tool Command Language (TCL)<ref>https://www.tcl.tk/</ref>. The default directory of modulefiles is given in the environmental variable <code>${MODULEPATH}</code>, but files in other directories can also be loaded by their absolute path.

Both Imperial CX1 and ARCHER2 adopt this application, with which pre-compiled applications are offered.

=== The external coordinator: What is a batch system ===

Always bear in mind that the computational resources are limited, so you need to acquire reasonable resources for your job. Besides, the cluster also needs to calculate your budget, coordinate jobs submitted by various users, and make the best of available resources. When job is running, maybe you also want to check its status. All of this are fulfilled by batch systems.

In practice, a Linux shell script is needed. Parameters of the batch system of are set in the commented lines at the top of the file. After the user submit the script to batch system, the system will:

# Examine the parameters
# Allocate and coordinate the requested resources
# Set up the environments, such as environmental variables, package dependency, and sync the same setting to all nodes
# Launch a parallel calculation - see mpi part
# Post-process

Note that a 'walltime' is usually required for a batch job, i.e., the maximum allowed time for the running job. The job will be 'killed', or suspended, when the time exceeds the walltime, and the rest part of the script will not be executed. <code>timeout</code> command can be used to set another walltime for a specific command.

Common batch systems include PBS and Slurm <ref>https://slurm.schedmd.com/overview.html</ref>. For Imperial cluster CX1 and MMM Hub Young (managed by UCL) <ref>http://mmmhub.ac.uk/young/</ref>, PBS system is implemented; for ARCHER2 and Tianhe-2 LvLiang(天河二号-吕梁), Slurm is implemented. Tutorials of batch systems are not covered here, since they are heavily tailored according to specific machines - usually modifications are made to enhance the efficiency. Refer to the specific user documentations for more information.

=== How to run a job in parallel: Things to consider ===

Successfully setting and submitting a batch job script symbolises that you do not need this tutorial any more. Before being able to do that, some considerations might be important:

* How large is my system? Is it efficient to use the resources I requested(Note that it is not a linear-scaling problem... Refer to [https://tutorials.crystalsolutions.eu/tutorial.html?td=tuto_HPC&tf=tuto_hpc#scale this test] on CRYSTAL17)?
* To which queue should I submit my job? Is it too long/not applicable/not available?
* Is it safe to use multi-threading?
* Is it memory, GPU etc. demanding?
* Roughly how long will it take?
* What is my budget code? Do I have enough resources?
* Which MPI release version is my code compatible with? Should I load a module or set variables?
* Any other specific environmental setups does my code need?
* Do I have any post-processing script after MPI part is finished? How long does it take?

==References==
<references>

CMSG disk and Shared Software on CX1

2024-02-13T20:40:38Z

Hz1420:

General information of shared software available for CMSG group members on Imperial CX1 is summarized in this page. Software listed below, unless stated otherwise, is hosted under 'CMSG' project of Imperial Research Data Store (RDS) <ref>https://www.imperial.ac.uk/admin-services/ict/self-service/research-support/rcs/service-offering/rds/</ref> (referred as RDS-CMSG below), which is accessible for both login and computational nodes of Imperial HPC CX1. For Imperial HPC HX1, by the time this page is upated (Feb. 2024), it is not accessible there. Only login node can visit RDS-CMSG disk after the pilot phase <ref>https://icl-rcs-user-guide.readthedocs.io/en/latest/hpc/pilot/hx1/</ref>. Therefore software on RDS-CMSG are classified into 2 catagories:

# For CX1, use <code>/rds/general/project/cmsg/live/app/</code> (executables, libraries and headers) and <code>/rds/general/project/cmsg/live/etc/</code> (other files, such as pseudopotential). Executable can be called directly from the user's home directory.
# For HX1, use <code>/rds/general/project/cmsg/live/share/</code>. Executables must be downloaded and uploaded to HX1. It is user's responsibility to set up an effective running environment according to 'readme' files (if there is one) saved in the same, or upper, directory.

''Please note that accessibility to CMSG disk is granted for CMSG group members only. If you need to access it, please contact group PI [[Contributors | Prof. Nicholas Harrison]].''

== Step 0 : Environment Modules ==
Environment modules <ref>https://modules.readthedocs.io/en/latest/</ref> manages the compilation and running environment to ensure all of its dependencies are loaded when an executable is launched. For software on RDS-CMSG, typically a module file is prepared, which can be called by <code>module load</code> command. The user is strongly suggested to add the following line to their <code>~/.bashrc</code> file on CX1:

<code>
export MODULEPATH="/rds/general/project/cmsg/live/etc/modulefiles:${MODULEPATH}"
</code>

That helps the 'module' executable to find module files hosted on RDS-CMSG. Then 'module' commands do not require real path. But in case that readers skip this section (as they always do), in the following text, the read path is always used. Use the following command to update the environment variable <code>${MODULEPATH}</code>:

<code>
~$ source ~/.bashrc
</code>

Due to the settings of CX1, the <code>source</code> command should be executed everytime you login. To aviod this, try the following commands:

~$ cat << EOF >> ~/.bash_profile
if test -f ~/.bashrc; then
source ~/.bashrc
fi
EOF

== CRYSTAL23 v1 (Intel) ==
The current default for CX1. Not available for HX1.

=== CX1 version ===
* Compiling Env : EasyBuild Intel 2023a, OpenMP
* Note: MPPproperties and MP2 (CRYSCOR and CRYSTAL2) are currently not available, which involves separate code packages (DMat2 and CRYSCOR). For MPPproperties please read the GNU version below.

The general job submission script for Imperial HPC, developed by the author himself, is used here.

''Thanks to Mr. K Tallat-Kelpsa for testing''

''' Job submission script configuration '''

The source code (written in bash) is available on group [https://github.com/cmsg-icl/HPC-job-submission/tree/main/Imperial-HPC-Job-Submission GitHub repo], though it is not needed in practice. Useful for developing new features. On CX1, use the following command to configure your local settings:

<code>
~$ bash /rds/general/project/cmsg/live/share/HPC-job-submission/Imperial-HPC-Job-Submission/CRYSTAL23/config_CRYSTAL23.sh
</code>

Then follow the instructions on the screen. Typically the default values are good. To use default vaules, press enter. After configuration is finished, the information is stored in a 'settings' file which is by default saved as <code>${HOME}/etc/runCRYSTAL23/settings</code>. It functions as a dictionary for job sumbmission scripts to refer to. It is also editable according to the user's needs.

''' Job submission script commands '''

After configuration, use <code>source ~/.bashrc</code> to enable alias commands.

{| class="wikitable" style="margin:auto"
|+ List of available commands
|-
! Command !! Definition
|-
| Pcrys23 || Generate parallel OMP CRYSTAL23 job submission files
|-
| MPPcrys23 || Generate massive parallel OMP CRYSTAL23 job submission files
|-
| Pprop23 || Generate parallel OMP CRYSTAL23 properties job submission files
|-
| Scrys23 || Generate serial OMP CRYSTAL23 job submission files
|-
| Sprop23 || Generate serial OMP CRYSTAL23 properties job submission files
|-
| Xcrys23 || User-defined executables and multiple jobs (see below for code examples)
|-
| SETcrys23 || Print the local 'settings' file
|-
| HELPcrys23 || Print the instructions of commands
|}

''' Examples '''

To generate a qsub file for a parallel crystal23 job on 'mgo.d12', the following command can be used:

<code>
~$ Pcrys23 -in mgo.d12 -wt 01:00 -nd 1
</code>

That generates a 'mgo.qsub' file requesting for 1 node and the maximum running time is 1 hour. Similarly, after the job is done, using the following command can run a parallel properties calculation based on 'mgo-band.d3' and data from the previous 'mgo' SCF calculation on 12 CPUs:

<code>
~$ Pprop23 -in mgo-band.d3 -nc 12 -wt 00:30 -ref mgo
</code>

For detailed instructions and testing cases, please refer to the [https://github.com/cmsg-icl/HPC-job-submission/tree/main/Imperial-HPC-Job-Submission documentation].

=== PC version ===
A few variants are available in <code>/rds/general/project/cmsg/live/share/CRYSTAL</code>, including executables compiled for Windows Subsystem for Linux (WSL) and MacOSx. Please check the readme files in saved in individual directories for specifications.

A set of statically linked 'crystal' and 'properties' executables is available in <code>/rds/general/project/cmsg/live/share/CRYSTAL/23v1/Linux-intel2023-x86-intel2023</code>, which do not have any prerequisite and can be run in serial on either Linux or MacOSx with x86 CPU.

== CRYSTAL23 v1 (GNU) ==
Compared to Intel version above, GNU version is slightly slower to allow for compatibility with MPPproperties. Available for CX1 and HX1.

=== CX1 version ===
* Compiling Env : gcc11.2.0, aocl4.0, mpich4.0.2, OpenMP
* Note: MP2 (CRYSCOR and CRYSTAL2) is currently not available.

The same job submission script is used. However, GNU version is not the default option for CX1. To enable GNU version, if there is a 'settings' file, change the following parameters:

* '''EXEDIR''' : <code>module load /rds/general/project/cmsg/live/etc/modulefiles/CRYSTAL/23v1-gcc</code>
* '''MPIDIR''' : <code>module load /rds/general/project/cmsg/live/etc/compiler/gcc11.2.0-aocl</code>.

Alternatively, rerun the configuration file (same as above) and specify 2 values during configuration.

Note that by default, there is no command for 'MPPproperties'. The user has to modify their <code>~/.bashrc</code> file to set the alias and 'settings' file for executable flags. An example:

# in ~/.bashrc
alias MPPprop23="/rds/general/project/cmsg/live/share/HPC-job-submission/Imperial-HPC-Job-Submission/gen_sub -x mppprop -set ${HOME}/etc/runCRYSTAL23/settings"
# in settings, keep the column width
mppprop mpiexec -np ${V_TPROC} MPPproperties Massive parallel properties calculation, OMP

=== HX1 version ===
* Compiling Env : EasyBuild foss/2023a, OpenMP
* Note: MP2 (CRYSCOR and CRYSTAL2) is currently not available.

Executables are saved in <code>/rds/general/project/cmsg/live/share/CRYSTAL/23v1/Linux-EBFOSS2023a-HX1-ompi</code>. To setup the running environment on HX1, if the user has a copy of 'settings' file, they need to modify the following keywords:

* '''EXEDIR''' : No default. The directory where the user put their executables in.
* '''MPIDIR''' : No default, but please use <code>module load foss/2023a</code> unless a self-compiled version is used.

Alternatively, run the configuration file (The one with '-HX1' in the same directory of Github repo) and specify 2 values during configuration. Other options are consistent with CRYSTAL on CX1.

== CRYSTAL17 v2 (GNU)==
Not hosted on RDS-CMSG. Default for CX1. Not available for HX1.

=== CX1 version ===
* Compilor : gcc/6.2.0
* MPI : mpich/3.4.3

'''For job submission scripts'''

The configuration and usage are identical to CRYSTAL23. Please refer to the previous section and substitute '23' with '17'. Also, please be noted that CRYSCOR (MP2 calculation) is not available for CRYSTAL17 and 'MPP' as a massive parallel strategy is limited to 'crystal' calculations, i.e., only MPPcrystal is released.

== Quantum Espresso v7.2 (GNU) ==
Job submission file configured for CX1 and HX1, but only CX1 executable is shared as these executables are highly dependent on environment.

=== CX1 version ===
* Compiling Env : EasyBuild foss2022a, OpenMP
* libxc: Yes, version 5.1.2
* hdf5: No

Pseudopotential files are saved in <code>/rds/general/project/cmsg/live/etc/QE_PseudoP</code>.

''' Job submission script configuration '''

<code>
~$ bash /rds/general/project/cmsg/live/share/HPC-job-submission/Imperial-HPC-Job-Submission/QE7/config_QE7.sh
</code>

The procedure is the same as CRYSTAL. Type enter for default setups, which typically works fine. Please note that moving files between temporal and home directories are disabled as QE has built-in file management system.

''' Job submission script commands '''

{| class="wikitable" style="margin:auto"
|+ List of available commands
|-
! Command !! Definition
|-
| PWqe7 || Generate pw.x job submission files
|-
| PHqe7 || Generate ph.x job submission files
|-
| CPqe7 || Generate cp.x job submission files
|-
| PPqe7 || Generate pp.x job submission files
|-
| Xqe7 || Generate job submission files for user-defined executables
|-
| SETqe7 || Print the local 'settings' file
|-
| HELPqe7 || Print the instructions of commands
|}

== GULP v6.1.2 (GNU) ==
Default for CX1. Not available for HX1.

=== CX1 version ===
* Compiling Env : EasyBuild foss2022a
* OpenKIM: Yes, version 2.3.0
* PLUMED: Yes, version 2.9.0
* ALAMODE: Yes, git repo copied into <code>app/</code>

GULP force field libraries are saved in <code>/rds/general/project/cmsg/live/etc/GULP_Libraries</code>. Executable name: 'gulp-mpi'.

'''OpenKIM'''

KIM force field <ref>https://openkim.org/</ref> are saved in <code>/rds/general/project/cmsg/live/etc/KIM_Models</code>. To run OpenKIM executable and download new models, run:

<code>
~$ module load /rds/general/project/cmsg/live/etc/modulefiles/OpenKIM/2.3.0-foss
</code>

'''Job submission script configuration'''

<code>
~$ bash /rds/general/project/cmsg/live/share/HPC-job-submission/Imperial-HPC-Job-Submission/GULP6/config_GULP6.sh
</code>

Type enter for default setups, which typically works fine.

'''Job submission script commands'''

{| class="wikitable" style="margin:auto"
|+ List of available commands
|-
! Command !! Definition
|-
| Pglp6 || Generate job submission files for 'gulp-mpi'
|-
| Xglp6 || Generate job submission files for user defined executable (by default nothing else, so not useful).
|-
| SETglp6 || Print the local 'settings' file
|-
| HELPglp6|| Print the instructions of commands
|}

== ONETEP v6.1.9.2 (GNU) ==
Default for CX1. Not available for HX1. Job submission script not available.

=== CX1 version ===
* Compiling Env : EasyBuild foss2022a, OpenMP, FFTW, Scalapack
* Libxc: Yes, version 5.1.2

Executable path is <code>/rds/general/project/cmsg/live/app/ONETEP/6.1.2.2__foss2022a/bin</code>. Pseudopotential files are saved in <code>/rds/general/project/cmsg/live/etc/ONETEP_PseudoP</code>.

==References==
<references>

Imperial CX1: Instructions and basic concepts of parallel computing

2024-02-13T20:38:43Z

Hz1420: /* A General Job Submission Script */

This tutorial is divided into 2 separate sections. In the fist section, introductions and available resources of CX1 are listed and classified. Since the [https://icl-rcs-user-guide.readthedocs.io/en/latest/ Research Computing Service (RCS)] team already developed great tutorials on their webpages, this part functions as a guide towards RCS webpages with necessary supplementary comments. In the second section, basic concepts of parallel computing and explanations of important terms are introduced. The main focus of this section is helping beginners to understand how high-performance computers (HPC) works on the basis of their daily practise.

''This tutorial was initially written between Feb. and Mar. 2022 to be shared within the group for induction and training proposes <ref>https://spica-vir.github.io/posts/Connect-to-the-Imperial-Cluster/</ref><ref>https://spica-vir.github.io/posts/Structure-and-usage-of-clusters/</ref>. Special thanks to Mr K. Tallat-Kelpsa, Ms A. Arber, Dr G. Mallia and Prof N. M. Harrison.''

== Introduction to CX1 ==
CX1 is the old name of the first HPC that served the whole college. New facilities (known as CX2) were gradually installed and integrated with the old system (CX3, a rather short-lived domain), while CX1 remains to be the most popular name that generally referring to the college-owned clusters. To grant a student access to CX1, the group PI can, on behave of that student, ask RCS team to add the specified account into HPC active user mailing list.
=== Connect to CX1 ===
CX1 is typically accessed via ssh (secured shell). Linux command line (Linux & MacOS) / sub-system (Windows 10,11) <ref>https://learn.microsoft.com/en-us/windows/wsl/install</ref> / SSH client (such as XShell <ref>https://www.xshell.com/en/xshell/</ref>) can be used. VPN is needed for off-campus users.

* [https://www.imperial.ac.uk/admin-services/ict/self-service/connect-communicate/remote-access/virtual-private-network-vpn/ Step-by-step guide to setup VPN]
* [https://www.imperial.ac.uk/admin-services/ict/self-service/research-support/rcs/support/getting-started/using-ssh/ How to use ssh]

In linux command line, use the following command to connect CX1:

<code>
~$ ssh -XY username@login.hpc.ic.ac.uk
</code>

P.S. <code>-XY</code> option can be omitted for most of cases, if you do not need GUI to run that program.

Alternatively, when the VPN service is unstable or even not available, it is possible to channel through the gateway of the cluster via a client, which is an 'agent'. To visit CX1, type the previous command in the client's command line.

<code>
~$ ssh username@sshgw.ic.ac.uk
</code>

Use the <code>scp</code> command to upload / download files, which is similar to <code>ssh</code> and <code>cp</code> command. For example, to upload a file:

<code>
~$ scp /local/path/file_name username@login.hpc.ic.ac.uk:/path/file_name
</code>

=== Usage ===
The [https://icl-rcs-user-guide.readthedocs.io/en/latest/hpc/ RCS Wiki Page in ReadTheDocs] contains information needed. [https://www.imperial.ac.uk/admin-services/ict/self-service/research-support/rcs/get-support/contact-us/ The support page], [https://wiki.imperial.ac.uk/display/HPC/Attend+a+clinic online clinic] and [https://wiki.imperial.ac.uk/display/HPC/Courses courses from graduate school] are available. To examine the status of CX1, use [https://status.rcs.imperial.ac.uk/d/u0zkcYQ7z/rcs-status?orgId=2&refresh=1m RCS status page].

==== Environmental Variables and Disk Space ====
Use <code>env</code> to access all the environmental variables - be careful, the output is '''HUGE'''. Some useful environmental variables:

* <code>${USER}</code> The user's college account, i.e., login credential.
* <code>${HOME}</code> '/rds/general/user/${USER}/home', or '~', which has 1 TB disk space for data backups.
* <code>${EPHEMERAL}</code> '/rds/general/user/${USER}/ephemeral' Temporal unlimited disk space lasting for 30 days. Suitable for running calculations.
* <code>${PATH}</code> Path to the executable can be attached for quick access. The Environment Modules package (see below) can automatically do that.

==== Software Management ====

The Environment Modules<ref>https://modules.readthedocs.io/en/latest/</ref> package is implemented on CX1 to manage computing software (see the following section for introductions). Basic commands are listed below:

* <code>module avail</code> List the available modules
* <code>module load mod_name</code> Load a specific module, 'mod_name'
* <code>module rm mod_name</code> Remove a specific module, 'mod_name'
* <code>module list</code> List all the loaded modules in the current environment
* <code>module help mod_name</code> Check the instructions of the module 'mod_name'

''Note: There is a CRYSTAL14 module in the list. For users in NMH's group, the latest CRYSTAL edition is available, so do not use that module.''

==== Job Partition Guide ====
A hierachy of jobs is designed for the optimial efficiency of CX1. The current job partition guide is available on [https://icl-rcs-user-guide.readthedocs.io/en/latest/hpc/queues/classes-of-jobs/ RCS Wiki Page]

==== Batch System ====

The PBS batch system <ref>https://en.wikipedia.org/wiki/Portable_Batch_System</ref> is used on CX1 (see the following section for the meaning of batch system). Basic commands of PBS are listed below:

* <code>availability</code> Check the availability of computational resources
* <code>qsub</code> filename.qsub</code> Submit the job 'filename'
* <code>qstat</code> Check the state of submitted jobs
* <code>qdel jobID</code> Kill the process with the ID number 'jobID'

To examine the queue status across the whole system, use [https://status.rcs.imperial.ac.uk/d/u0zkcYQ7z/rcs-status?orgId=2&refresh=1m RCS status page].

==== A General Job Submission Script ====
A general job submission script for CX1 is developed by the author himself. See the [https://github.com/cmsg-icl/HPC-job-submission GitHub repository] of CMSG for details. Parameterised software includes: CRYSTAL14/17/23, Quantum Espresso 7, LAMMPS, GROMACS, GULP6.

== Basic Concepts of Parallel Computing ==
A brief introduction to parallel computing is given in this section by taking CX1, a medium-sized general-propose cluster, as an example.

=== Divide a job: Nodes, Processors and Threads ===

;Node
:A bunch of CPUs and probably with GPUs / coprocessors for acceleration. Memory and input files are shared by processors in the same node, so a node can be considered as an independent computer. The communication between nodes are achieved by ultra-fast network, which is the bottleneck of modern clusters.

;Processor
:The unit to deal with a 'process', also known as 'central processing unit', or CPU. Processors in the same node communicate via shared memory.

;Thread
:Subdivision of a process. Multiple threads in the same process share the resources allocated to the CPU.

The figure on the right hand side illustrates the hierarchy of node, processor, and thread. '''Note:''' The word 'processor' is not a very accurate term. Might be better with 'process' (I am just too lazy to update that figure). Many modern CPUs supports sub-CPU threading, which means the number of logical CPUs is larger than physical CPUs, so it is possible to have multiple threads within 1 processor. However, it is also possible to use multiple processors for 1 process, or even 1 thread.

[[File:Job_Partition.png|450px|right|Job Partition]]

==== Multiple processes vs multiple threads ====

From the figure above, it is not difficult to distinguish the differences between a 'process' and a 'thread': process is the smallest unit for resource allocation; thread is part of a process. The idea of 'thread' is introduced to address the huge difference in the speed of CPU and RAM. CPU is always several orders of magnitude faster than RAM, so typically the bottleneck of a process is loading the required environment from RAM, rather than computations in CPU. By using multiple threads in the same process, various branches of the same program can be executed simultaneously. Therefore, the shared environmental requirements doesn't need to be read from RAM for multiple times, and the loading time for threads is much smaller than for processes.

However, multithreading is not always advantageous. A technical prerequisite is that the program should be developed for multithread proposes. Python, for example, is a pseudo-multithread language, while Java is a real one. Sometimes multithreading can lead to catastrophic results. Since threads share the same resource allocation (CPU, RAM, I/O, etc.), when a thread fails, the whole process fails as well. Comparatively, in multiple processes, other processes will be protected if a process fails.

In practice, users can either run each process in serial (i.e., number of threads = 1), or in parallel (i.e., number of threads > 1) on clusters. However, '''the former one is recommended''', because of more secured resource managements. The latter is not advantageous. Besides the problem mentioned above, it might lead to problems such as memory leak when running programs either: not developed for multithreading / requires improper packages (Here is [https://docs.archer2.ac.uk/known-issues/#oom-due-to-memory-leak-in-libfabric-added-2022-02-23 a famous issue] with libfabric on ARCHER2 identified in early 2022).

==== More nodes vs more CPUs ====

When the allocated memory permits, from my experience, using more CPUs/processes per node is usually a better idea, considering that all nodes have independent memory space and the inter-node communications are achieved by wired networks. It almost always takes longer to coordinate nodes than to coordinate processors within the same node.

=== The internal coordinator: What is MPI ===

Message passing interface, or MPI, is a standard for communicating and transferring data between nodes and therefore distributed memories. It is utilised via MPI libraries. The most popular implementations include:

* MPICH <ref>https://www.mpich.org/</ref> - an open-source library
* Intel MPI <ref>https://www.intel.com/content/www/us/en/developer/tools/oneapi/mpi-library.html#gs.xld8oa</ref> - a popular implementation of MPICH especially optimised for Intel CPUs
* OpenMPI <ref>https://www.open-mpi.org/</ref> - an open-source library
* OpenMP <ref>https://www.openmp.org/</ref> - Not MPI; parallelization based on shared memory, so only implemented in a single node; can be used for multithreading

In practice, a hybrid parallelization combining MPI and OpenMP to run multithread jobs on cluster is allowed, though sometimes not recommended. The first process (probably not a node or a processor) is usually allocated for I/O, and the rest is used for parallel computing.

So far, MPI only supports C/C++ and FORTRAN, which explains why all parallel computing software is based on these languages. To launch an executable in parallel, one should use: <code>mpiexec</code> or <code>mpirun</code>.

=== Secure your storage: Tmp memory, Work directory and home directory ===

Almost all the modern clusters have separate disk spaces for differently proposes, namely, temporary memory, work directory and home directory. This originates again from the famous speed difference between CPU and RAM/ROM. 2 distinctly kinds of disks are used respectively to improve the overall efficiency and secure important data:

* For temporary memory large, high-frequency disks are used. It is allocated by job requests, which is not accessible by login nodes. Everything is erased after the job is terminated.
* For work directory, large, high-frequency disks are used. Data stored in work directory is usually not backed up, and in the case of CX1, will be automatically cleaned after a fixed time length.
* For home directory, mechanical disks with slower read/write frequency but better robustness are used. Usually files in home space are backed up.

For large clusters like ARCHER2 <ref>https://www.archer2.ac.uk/</ref>, the work directory and the home directory are completely separated, i.e., directory is only viable by login nodes; work directory is viable by both job and login nodes. Job submission in home directory is prohibited. For more flexible clusters like Imperial CX1, submitting jobs in home directory and visiting of home directory by job nodes are allowed, but storing temporary files during calculation in home directory is still not recommended because of the potential influence on other files and the reduced overall efficiency. (And it is not something new for CX1 users to receive the RDS failure news email)

=== Setup your environment: What does an application need? ===

==== Executable ====
The binary executable should, theoretically, all be stored in '\usr\bin'. This never happens in practice, unless you are a fanatical fundamentalist of the early Linux releases. To guide your system to the desired executable, you can either laboriously type its absolute path every time you need it or add the path to the environmental variable:

<code>
~$ export PATH=${PATH}:path_to_bin
</code>

Running any executable in parallel requires mpi to coordinate all the processes/threads. The path to mpi executable is also required. Besides, many scientific codes require other specific environmental variables such as linear algebra packages. Read their documentations for further information.

==== .lib/.a/.o files ====

When writing a script, you might need some extra packages to do more complex jobs. Those packages are developed by experts in computer science and can be called by a line of code. The same thing happens when people were developing applications like CRYSTAL and ONETEP.

However, scientific computing codes are usually distributed in the form of source code. Source codes in FORTRAN/C/C++ need be compiled into a binary executable. There are 2 options during compiling:

# Include the whole package as long as one of its functions is called, also known as a 'static lib'.
# Only include a 'table of contents' when compiling, also known as 'dynamic lib'. The packages needed are separately stored in '.dll/.so' files, making it possible for multiple applications sharing the same lib.

Details about compilation are beyond the scope of this post. The thing is: when running a dynamically linked application, information should be given to help the code find the libs needed. This can be specified by:

<code>
~$ export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:path_to_lib`
</code>

For statically linked applications, usually you need not worry about it - but the volume of the compiled executable might make you wonder whether there is an alternative way.

==== Conflicts ====

Improper previous settings may lead to a wrong application, or a wrong version, if multiple applications with similar functions are installed in the system, such as Intel compiler and GCC, OpenMPI and MPICH - a common phenomenon for shared computing resources. To avoid this, the path to the undesired application or lib should be removed from the environmental variables.

==== Environmental Modules ====

Environmental Modules <ref>http://modules.sourceforge.net/</ref> is a popular software managing the necessary environmental setups and conflicts for each application. It can easily add or erase the environmental variables by commands (such as <code>module load</code>, <code>module rm</code>) and modulefiles written in Tool Command Language (TCL)<ref>https://www.tcl.tk/</ref>. The default directory of modulefiles is given in the environmental variable <code>${MODULEPATH}</code>, but files in other directories can also be loaded by their absolute path.

Both Imperial CX1 and ARCHER2 adopt this application, with which pre-compiled applications are offered.

=== The external coordinator: What is a batch system ===

Always bear in mind that the computational resources are limited, so you need to acquire reasonable resources for your job. Besides, the cluster also needs to calculate your budget, coordinate jobs submitted by various users, and make the best of available resources. When job is running, maybe you also want to check its status. All of this are fulfilled by batch systems.

In practice, a Linux shell script is needed. Parameters of the batch system of are set in the commented lines at the top of the file. After the user submit the script to batch system, the system will:

# Examine the parameters
# Allocate and coordinate the requested resources
# Set up the environments, such as environmental variables, package dependency, and sync the same setting to all nodes
# Launch a parallel calculation - see mpi part
# Post-process

Note that a 'walltime' is usually required for a batch job, i.e., the maximum allowed time for the running job. The job will be 'killed', or suspended, when the time exceeds the walltime, and the rest part of the script will not be executed. <code>timeout</code> command can be used to set another walltime for a specific command.

Common batch systems include PBS and Slurm <ref>https://slurm.schedmd.com/overview.html</ref>. For Imperial cluster CX1 and MMM Hub Young (managed by UCL) <ref>http://mmmhub.ac.uk/young/</ref>, PBS system is implemented; for ARCHER2 and Tianhe-2 LvLiang(天河二号-吕梁), Slurm is implemented. Tutorials of batch systems are not covered here, since they are heavily tailored according to specific machines - usually modifications are made to enhance the efficiency. Refer to the specific user documentations for more information.

Successfully setting and submitting a batch job script symbolises that you do not need this tutorial any more. Before being able to do that, some considerations might be important:

* How large is my system? Is it efficient to use the resources I requested(Note that it is not a linear-scaling problem... Refer to [https://tutorials.crystalsolutions.eu/tutorial.html?td=tuto_HPC&tf=tuto_hpc#scale this test] on CRYSTAL17)?
* To which queue should I submit my job? Is it too long/not applicable/not available?
* Is it safe to use multi-threading?
* Is it memory, GPU etc. demanding?
* Roughly how long will it take?
* What is my budget code? Do I have enough resources?
* Which MPI release version is my code compatible with? Should I load a module or set variables?
* Any other specific environmental setups does my code need?
* Do I have any post-processing script after MPI part is finished? How long does it take?

==References==
<references>

Imperial CX1: Instructions and basic concepts of parallel computing

2024-02-13T20:38:21Z

Hz1420: /* A General Job Submission Script */

This tutorial is divided into 2 separate sections. In the fist section, introductions and available resources of CX1 are listed and classified. Since the [https://icl-rcs-user-guide.readthedocs.io/en/latest/ Research Computing Service (RCS)] team already developed great tutorials on their webpages, this part functions as a guide towards RCS webpages with necessary supplementary comments. In the second section, basic concepts of parallel computing and explanations of important terms are introduced. The main focus of this section is helping beginners to understand how high-performance computers (HPC) works on the basis of their daily practise.

''This tutorial was initially written between Feb. and Mar. 2022 to be shared within the group for induction and training proposes <ref>https://spica-vir.github.io/posts/Connect-to-the-Imperial-Cluster/</ref><ref>https://spica-vir.github.io/posts/Structure-and-usage-of-clusters/</ref>. Special thanks to Mr K. Tallat-Kelpsa, Ms A. Arber, Dr G. Mallia and Prof N. M. Harrison.''

== Introduction to CX1 ==
CX1 is the old name of the first HPC that served the whole college. New facilities (known as CX2) were gradually installed and integrated with the old system (CX3, a rather short-lived domain), while CX1 remains to be the most popular name that generally referring to the college-owned clusters. To grant a student access to CX1, the group PI can, on behave of that student, ask RCS team to add the specified account into HPC active user mailing list.
=== Connect to CX1 ===
CX1 is typically accessed via ssh (secured shell). Linux command line (Linux & MacOS) / sub-system (Windows 10,11) <ref>https://learn.microsoft.com/en-us/windows/wsl/install</ref> / SSH client (such as XShell <ref>https://www.xshell.com/en/xshell/</ref>) can be used. VPN is needed for off-campus users.

* [https://www.imperial.ac.uk/admin-services/ict/self-service/connect-communicate/remote-access/virtual-private-network-vpn/ Step-by-step guide to setup VPN]
* [https://www.imperial.ac.uk/admin-services/ict/self-service/research-support/rcs/support/getting-started/using-ssh/ How to use ssh]

In linux command line, use the following command to connect CX1:

<code>
~$ ssh -XY username@login.hpc.ic.ac.uk
</code>

P.S. <code>-XY</code> option can be omitted for most of cases, if you do not need GUI to run that program.

Alternatively, when the VPN service is unstable or even not available, it is possible to channel through the gateway of the cluster via a client, which is an 'agent'. To visit CX1, type the previous command in the client's command line.

<code>
~$ ssh username@sshgw.ic.ac.uk
</code>

Use the <code>scp</code> command to upload / download files, which is similar to <code>ssh</code> and <code>cp</code> command. For example, to upload a file:

<code>
~$ scp /local/path/file_name username@login.hpc.ic.ac.uk:/path/file_name
</code>

=== Usage ===
The [https://icl-rcs-user-guide.readthedocs.io/en/latest/hpc/ RCS Wiki Page in ReadTheDocs] contains information needed. [https://www.imperial.ac.uk/admin-services/ict/self-service/research-support/rcs/get-support/contact-us/ The support page], [https://wiki.imperial.ac.uk/display/HPC/Attend+a+clinic online clinic] and [https://wiki.imperial.ac.uk/display/HPC/Courses courses from graduate school] are available. To examine the status of CX1, use [https://status.rcs.imperial.ac.uk/d/u0zkcYQ7z/rcs-status?orgId=2&refresh=1m RCS status page].

==== Environmental Variables and Disk Space ====
Use <code>env</code> to access all the environmental variables - be careful, the output is '''HUGE'''. Some useful environmental variables:

* <code>${USER}</code> The user's college account, i.e., login credential.
* <code>${HOME}</code> '/rds/general/user/${USER}/home', or '~', which has 1 TB disk space for data backups.
* <code>${EPHEMERAL}</code> '/rds/general/user/${USER}/ephemeral' Temporal unlimited disk space lasting for 30 days. Suitable for running calculations.
* <code>${PATH}</code> Path to the executable can be attached for quick access. The Environment Modules package (see below) can automatically do that.

==== Software Management ====

The Environment Modules<ref>https://modules.readthedocs.io/en/latest/</ref> package is implemented on CX1 to manage computing software (see the following section for introductions). Basic commands are listed below:

* <code>module avail</code> List the available modules
* <code>module load mod_name</code> Load a specific module, 'mod_name'
* <code>module rm mod_name</code> Remove a specific module, 'mod_name'
* <code>module list</code> List all the loaded modules in the current environment
* <code>module help mod_name</code> Check the instructions of the module 'mod_name'

''Note: There is a CRYSTAL14 module in the list. For users in NMH's group, the latest CRYSTAL edition is available, so do not use that module.''

==== Job Partition Guide ====
A hierachy of jobs is designed for the optimial efficiency of CX1. The current job partition guide is available on [https://icl-rcs-user-guide.readthedocs.io/en/latest/hpc/queues/classes-of-jobs/ RCS Wiki Page]

==== Batch System ====

The PBS batch system <ref>https://en.wikipedia.org/wiki/Portable_Batch_System</ref> is used on CX1 (see the following section for the meaning of batch system). Basic commands of PBS are listed below:

* <code>availability</code> Check the availability of computational resources
* <code>qsub</code> filename.qsub</code> Submit the job 'filename'
* <code>qstat</code> Check the state of submitted jobs
* <code>qdel jobID</code> Kill the process with the ID number 'jobID'

To examine the queue status across the whole system, use [https://status.rcs.imperial.ac.uk/d/u0zkcYQ7z/rcs-status?orgId=2&refresh=1m RCS status page].

==== A General Job Submission Script ====
A general job submission script for CX1 is developed by the author himself. See the [https://github.com/cmsg-icl/HPC-job-submission GitHub repository] of CMSG for details. Parameterised software includes: CRYSTAL17/23, Quantum Espresso 7, LAMMPS, GROMACS, GULP6.

== Basic Concepts of Parallel Computing ==
A brief introduction to parallel computing is given in this section by taking CX1, a medium-sized general-propose cluster, as an example.

=== Divide a job: Nodes, Processors and Threads ===

;Node
:A bunch of CPUs and probably with GPUs / coprocessors for acceleration. Memory and input files are shared by processors in the same node, so a node can be considered as an independent computer. The communication between nodes are achieved by ultra-fast network, which is the bottleneck of modern clusters.

;Processor
:The unit to deal with a 'process', also known as 'central processing unit', or CPU. Processors in the same node communicate via shared memory.

;Thread
:Subdivision of a process. Multiple threads in the same process share the resources allocated to the CPU.

The figure on the right hand side illustrates the hierarchy of node, processor, and thread. '''Note:''' The word 'processor' is not a very accurate term. Might be better with 'process' (I am just too lazy to update that figure). Many modern CPUs supports sub-CPU threading, which means the number of logical CPUs is larger than physical CPUs, so it is possible to have multiple threads within 1 processor. However, it is also possible to use multiple processors for 1 process, or even 1 thread.

[[File:Job_Partition.png|450px|right|Job Partition]]

==== Multiple processes vs multiple threads ====

From the figure above, it is not difficult to distinguish the differences between a 'process' and a 'thread': process is the smallest unit for resource allocation; thread is part of a process. The idea of 'thread' is introduced to address the huge difference in the speed of CPU and RAM. CPU is always several orders of magnitude faster than RAM, so typically the bottleneck of a process is loading the required environment from RAM, rather than computations in CPU. By using multiple threads in the same process, various branches of the same program can be executed simultaneously. Therefore, the shared environmental requirements doesn't need to be read from RAM for multiple times, and the loading time for threads is much smaller than for processes.

However, multithreading is not always advantageous. A technical prerequisite is that the program should be developed for multithread proposes. Python, for example, is a pseudo-multithread language, while Java is a real one. Sometimes multithreading can lead to catastrophic results. Since threads share the same resource allocation (CPU, RAM, I/O, etc.), when a thread fails, the whole process fails as well. Comparatively, in multiple processes, other processes will be protected if a process fails.

In practice, users can either run each process in serial (i.e., number of threads = 1), or in parallel (i.e., number of threads > 1) on clusters. However, '''the former one is recommended''', because of more secured resource managements. The latter is not advantageous. Besides the problem mentioned above, it might lead to problems such as memory leak when running programs either: not developed for multithreading / requires improper packages (Here is [https://docs.archer2.ac.uk/known-issues/#oom-due-to-memory-leak-in-libfabric-added-2022-02-23 a famous issue] with libfabric on ARCHER2 identified in early 2022).

==== More nodes vs more CPUs ====

When the allocated memory permits, from my experience, using more CPUs/processes per node is usually a better idea, considering that all nodes have independent memory space and the inter-node communications are achieved by wired networks. It almost always takes longer to coordinate nodes than to coordinate processors within the same node.

=== The internal coordinator: What is MPI ===

Message passing interface, or MPI, is a standard for communicating and transferring data between nodes and therefore distributed memories. It is utilised via MPI libraries. The most popular implementations include:

* MPICH <ref>https://www.mpich.org/</ref> - an open-source library
* Intel MPI <ref>https://www.intel.com/content/www/us/en/developer/tools/oneapi/mpi-library.html#gs.xld8oa</ref> - a popular implementation of MPICH especially optimised for Intel CPUs
* OpenMPI <ref>https://www.open-mpi.org/</ref> - an open-source library
* OpenMP <ref>https://www.openmp.org/</ref> - Not MPI; parallelization based on shared memory, so only implemented in a single node; can be used for multithreading

In practice, a hybrid parallelization combining MPI and OpenMP to run multithread jobs on cluster is allowed, though sometimes not recommended. The first process (probably not a node or a processor) is usually allocated for I/O, and the rest is used for parallel computing.

So far, MPI only supports C/C++ and FORTRAN, which explains why all parallel computing software is based on these languages. To launch an executable in parallel, one should use: <code>mpiexec</code> or <code>mpirun</code>.

=== Secure your storage: Tmp memory, Work directory and home directory ===

Almost all the modern clusters have separate disk spaces for differently proposes, namely, temporary memory, work directory and home directory. This originates again from the famous speed difference between CPU and RAM/ROM. 2 distinctly kinds of disks are used respectively to improve the overall efficiency and secure important data:

* For temporary memory large, high-frequency disks are used. It is allocated by job requests, which is not accessible by login nodes. Everything is erased after the job is terminated.
* For work directory, large, high-frequency disks are used. Data stored in work directory is usually not backed up, and in the case of CX1, will be automatically cleaned after a fixed time length.
* For home directory, mechanical disks with slower read/write frequency but better robustness are used. Usually files in home space are backed up.

For large clusters like ARCHER2 <ref>https://www.archer2.ac.uk/</ref>, the work directory and the home directory are completely separated, i.e., directory is only viable by login nodes; work directory is viable by both job and login nodes. Job submission in home directory is prohibited. For more flexible clusters like Imperial CX1, submitting jobs in home directory and visiting of home directory by job nodes are allowed, but storing temporary files during calculation in home directory is still not recommended because of the potential influence on other files and the reduced overall efficiency. (And it is not something new for CX1 users to receive the RDS failure news email)

=== Setup your environment: What does an application need? ===

==== Executable ====
The binary executable should, theoretically, all be stored in '\usr\bin'. This never happens in practice, unless you are a fanatical fundamentalist of the early Linux releases. To guide your system to the desired executable, you can either laboriously type its absolute path every time you need it or add the path to the environmental variable:

<code>
~$ export PATH=${PATH}:path_to_bin
</code>

Running any executable in parallel requires mpi to coordinate all the processes/threads. The path to mpi executable is also required. Besides, many scientific codes require other specific environmental variables such as linear algebra packages. Read their documentations for further information.

==== .lib/.a/.o files ====

When writing a script, you might need some extra packages to do more complex jobs. Those packages are developed by experts in computer science and can be called by a line of code. The same thing happens when people were developing applications like CRYSTAL and ONETEP.

However, scientific computing codes are usually distributed in the form of source code. Source codes in FORTRAN/C/C++ need be compiled into a binary executable. There are 2 options during compiling:

# Include the whole package as long as one of its functions is called, also known as a 'static lib'.
# Only include a 'table of contents' when compiling, also known as 'dynamic lib'. The packages needed are separately stored in '.dll/.so' files, making it possible for multiple applications sharing the same lib.

Details about compilation are beyond the scope of this post. The thing is: when running a dynamically linked application, information should be given to help the code find the libs needed. This can be specified by:

<code>
~$ export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:path_to_lib`
</code>

For statically linked applications, usually you need not worry about it - but the volume of the compiled executable might make you wonder whether there is an alternative way.

==== Conflicts ====

Improper previous settings may lead to a wrong application, or a wrong version, if multiple applications with similar functions are installed in the system, such as Intel compiler and GCC, OpenMPI and MPICH - a common phenomenon for shared computing resources. To avoid this, the path to the undesired application or lib should be removed from the environmental variables.

==== Environmental Modules ====

Environmental Modules <ref>http://modules.sourceforge.net/</ref> is a popular software managing the necessary environmental setups and conflicts for each application. It can easily add or erase the environmental variables by commands (such as <code>module load</code>, <code>module rm</code>) and modulefiles written in Tool Command Language (TCL)<ref>https://www.tcl.tk/</ref>. The default directory of modulefiles is given in the environmental variable <code>${MODULEPATH}</code>, but files in other directories can also be loaded by their absolute path.

Both Imperial CX1 and ARCHER2 adopt this application, with which pre-compiled applications are offered.

=== The external coordinator: What is a batch system ===

Always bear in mind that the computational resources are limited, so you need to acquire reasonable resources for your job. Besides, the cluster also needs to calculate your budget, coordinate jobs submitted by various users, and make the best of available resources. When job is running, maybe you also want to check its status. All of this are fulfilled by batch systems.

In practice, a Linux shell script is needed. Parameters of the batch system of are set in the commented lines at the top of the file. After the user submit the script to batch system, the system will:

# Examine the parameters
# Allocate and coordinate the requested resources
# Set up the environments, such as environmental variables, package dependency, and sync the same setting to all nodes
# Launch a parallel calculation - see mpi part
# Post-process

Note that a 'walltime' is usually required for a batch job, i.e., the maximum allowed time for the running job. The job will be 'killed', or suspended, when the time exceeds the walltime, and the rest part of the script will not be executed. <code>timeout</code> command can be used to set another walltime for a specific command.

Common batch systems include PBS and Slurm <ref>https://slurm.schedmd.com/overview.html</ref>. For Imperial cluster CX1 and MMM Hub Young (managed by UCL) <ref>http://mmmhub.ac.uk/young/</ref>, PBS system is implemented; for ARCHER2 and Tianhe-2 LvLiang(天河二号-吕梁), Slurm is implemented. Tutorials of batch systems are not covered here, since they are heavily tailored according to specific machines - usually modifications are made to enhance the efficiency. Refer to the specific user documentations for more information.

Successfully setting and submitting a batch job script symbolises that you do not need this tutorial any more. Before being able to do that, some considerations might be important:

* How large is my system? Is it efficient to use the resources I requested(Note that it is not a linear-scaling problem... Refer to [https://tutorials.crystalsolutions.eu/tutorial.html?td=tuto_HPC&tf=tuto_hpc#scale this test] on CRYSTAL17)?
* To which queue should I submit my job? Is it too long/not applicable/not available?
* Is it safe to use multi-threading?
* Is it memory, GPU etc. demanding?
* Roughly how long will it take?
* What is my budget code? Do I have enough resources?
* Which MPI release version is my code compatible with? Should I load a module or set variables?
* Any other specific environmental setups does my code need?
* Do I have any post-processing script after MPI part is finished? How long does it take?

==References==
<references>

Imperial CX1: Instructions and basic concepts of parallel computing

2024-02-13T20:37:36Z

Hz1420: /* Job Partition Guide */

This tutorial is divided into 2 separate sections. In the fist section, introductions and available resources of CX1 are listed and classified. Since the [https://icl-rcs-user-guide.readthedocs.io/en/latest/ Research Computing Service (RCS)] team already developed great tutorials on their webpages, this part functions as a guide towards RCS webpages with necessary supplementary comments. In the second section, basic concepts of parallel computing and explanations of important terms are introduced. The main focus of this section is helping beginners to understand how high-performance computers (HPC) works on the basis of their daily practise.

''This tutorial was initially written between Feb. and Mar. 2022 to be shared within the group for induction and training proposes <ref>https://spica-vir.github.io/posts/Connect-to-the-Imperial-Cluster/</ref><ref>https://spica-vir.github.io/posts/Structure-and-usage-of-clusters/</ref>. Special thanks to Mr K. Tallat-Kelpsa, Ms A. Arber, Dr G. Mallia and Prof N. M. Harrison.''

== Introduction to CX1 ==
CX1 is the old name of the first HPC that served the whole college. New facilities (known as CX2) were gradually installed and integrated with the old system (CX3, a rather short-lived domain), while CX1 remains to be the most popular name that generally referring to the college-owned clusters. To grant a student access to CX1, the group PI can, on behave of that student, ask RCS team to add the specified account into HPC active user mailing list.
=== Connect to CX1 ===
CX1 is typically accessed via ssh (secured shell). Linux command line (Linux & MacOS) / sub-system (Windows 10,11) <ref>https://learn.microsoft.com/en-us/windows/wsl/install</ref> / SSH client (such as XShell <ref>https://www.xshell.com/en/xshell/</ref>) can be used. VPN is needed for off-campus users.

* [https://www.imperial.ac.uk/admin-services/ict/self-service/connect-communicate/remote-access/virtual-private-network-vpn/ Step-by-step guide to setup VPN]
* [https://www.imperial.ac.uk/admin-services/ict/self-service/research-support/rcs/support/getting-started/using-ssh/ How to use ssh]

In linux command line, use the following command to connect CX1:

<code>
~$ ssh -XY username@login.hpc.ic.ac.uk
</code>

P.S. <code>-XY</code> option can be omitted for most of cases, if you do not need GUI to run that program.

Alternatively, when the VPN service is unstable or even not available, it is possible to channel through the gateway of the cluster via a client, which is an 'agent'. To visit CX1, type the previous command in the client's command line.

<code>
~$ ssh username@sshgw.ic.ac.uk
</code>

Use the <code>scp</code> command to upload / download files, which is similar to <code>ssh</code> and <code>cp</code> command. For example, to upload a file:

<code>
~$ scp /local/path/file_name username@login.hpc.ic.ac.uk:/path/file_name
</code>

=== Usage ===
The [https://icl-rcs-user-guide.readthedocs.io/en/latest/hpc/ RCS Wiki Page in ReadTheDocs] contains information needed. [https://www.imperial.ac.uk/admin-services/ict/self-service/research-support/rcs/get-support/contact-us/ The support page], [https://wiki.imperial.ac.uk/display/HPC/Attend+a+clinic online clinic] and [https://wiki.imperial.ac.uk/display/HPC/Courses courses from graduate school] are available. To examine the status of CX1, use [https://status.rcs.imperial.ac.uk/d/u0zkcYQ7z/rcs-status?orgId=2&refresh=1m RCS status page].

==== Environmental Variables and Disk Space ====
Use <code>env</code> to access all the environmental variables - be careful, the output is '''HUGE'''. Some useful environmental variables:

* <code>${USER}</code> The user's college account, i.e., login credential.
* <code>${HOME}</code> '/rds/general/user/${USER}/home', or '~', which has 1 TB disk space for data backups.
* <code>${EPHEMERAL}</code> '/rds/general/user/${USER}/ephemeral' Temporal unlimited disk space lasting for 30 days. Suitable for running calculations.
* <code>${PATH}</code> Path to the executable can be attached for quick access. The Environment Modules package (see below) can automatically do that.

==== Software Management ====

The Environment Modules<ref>https://modules.readthedocs.io/en/latest/</ref> package is implemented on CX1 to manage computing software (see the following section for introductions). Basic commands are listed below:

* <code>module avail</code> List the available modules
* <code>module load mod_name</code> Load a specific module, 'mod_name'
* <code>module rm mod_name</code> Remove a specific module, 'mod_name'
* <code>module list</code> List all the loaded modules in the current environment
* <code>module help mod_name</code> Check the instructions of the module 'mod_name'

''Note: There is a CRYSTAL14 module in the list. For users in NMH's group, the latest CRYSTAL edition is available, so do not use that module.''

==== Job Partition Guide ====
A hierachy of jobs is designed for the optimial efficiency of CX1. The current job partition guide is available on [https://icl-rcs-user-guide.readthedocs.io/en/latest/hpc/queues/classes-of-jobs/ RCS Wiki Page]

==== Batch System ====

The PBS batch system <ref>https://en.wikipedia.org/wiki/Portable_Batch_System</ref> is used on CX1 (see the following section for the meaning of batch system). Basic commands of PBS are listed below:

* <code>availability</code> Check the availability of computational resources
* <code>qsub</code> filename.qsub</code> Submit the job 'filename'
* <code>qstat</code> Check the state of submitted jobs
* <code>qdel jobID</code> Kill the process with the ID number 'jobID'

To examine the queue status across the whole system, use [https://status.rcs.imperial.ac.uk/d/u0zkcYQ7z/rcs-status?orgId=2&refresh=1m RCS status page].

==== A General Job Submission Script ====
A general job submission script for CX1 is developed by the author himself. See the [https://github.com/cmsg-icl/crystal_shape_control/tree/main/Imperial-HPC-Job-Submission GitHub repository] of CMSG for details. Parameterised software includes: CRYSTAL17/23, Quantum Espresso 7, LAMMPS, GROMACS, GULP6.

== Basic Concepts of Parallel Computing ==
A brief introduction to parallel computing is given in this section by taking CX1, a medium-sized general-propose cluster, as an example.

=== Divide a job: Nodes, Processors and Threads ===

;Node
:A bunch of CPUs and probably with GPUs / coprocessors for acceleration. Memory and input files are shared by processors in the same node, so a node can be considered as an independent computer. The communication between nodes are achieved by ultra-fast network, which is the bottleneck of modern clusters.

;Processor
:The unit to deal with a 'process', also known as 'central processing unit', or CPU. Processors in the same node communicate via shared memory.

;Thread
:Subdivision of a process. Multiple threads in the same process share the resources allocated to the CPU.

The figure on the right hand side illustrates the hierarchy of node, processor, and thread. '''Note:''' The word 'processor' is not a very accurate term. Might be better with 'process' (I am just too lazy to update that figure). Many modern CPUs supports sub-CPU threading, which means the number of logical CPUs is larger than physical CPUs, so it is possible to have multiple threads within 1 processor. However, it is also possible to use multiple processors for 1 process, or even 1 thread.

[[File:Job_Partition.png|450px|right|Job Partition]]

==== Multiple processes vs multiple threads ====

From the figure above, it is not difficult to distinguish the differences between a 'process' and a 'thread': process is the smallest unit for resource allocation; thread is part of a process. The idea of 'thread' is introduced to address the huge difference in the speed of CPU and RAM. CPU is always several orders of magnitude faster than RAM, so typically the bottleneck of a process is loading the required environment from RAM, rather than computations in CPU. By using multiple threads in the same process, various branches of the same program can be executed simultaneously. Therefore, the shared environmental requirements doesn't need to be read from RAM for multiple times, and the loading time for threads is much smaller than for processes.

However, multithreading is not always advantageous. A technical prerequisite is that the program should be developed for multithread proposes. Python, for example, is a pseudo-multithread language, while Java is a real one. Sometimes multithreading can lead to catastrophic results. Since threads share the same resource allocation (CPU, RAM, I/O, etc.), when a thread fails, the whole process fails as well. Comparatively, in multiple processes, other processes will be protected if a process fails.

In practice, users can either run each process in serial (i.e., number of threads = 1), or in parallel (i.e., number of threads > 1) on clusters. However, '''the former one is recommended''', because of more secured resource managements. The latter is not advantageous. Besides the problem mentioned above, it might lead to problems such as memory leak when running programs either: not developed for multithreading / requires improper packages (Here is [https://docs.archer2.ac.uk/known-issues/#oom-due-to-memory-leak-in-libfabric-added-2022-02-23 a famous issue] with libfabric on ARCHER2 identified in early 2022).

==== More nodes vs more CPUs ====

When the allocated memory permits, from my experience, using more CPUs/processes per node is usually a better idea, considering that all nodes have independent memory space and the inter-node communications are achieved by wired networks. It almost always takes longer to coordinate nodes than to coordinate processors within the same node.

=== The internal coordinator: What is MPI ===

Message passing interface, or MPI, is a standard for communicating and transferring data between nodes and therefore distributed memories. It is utilised via MPI libraries. The most popular implementations include:

* MPICH <ref>https://www.mpich.org/</ref> - an open-source library
* Intel MPI <ref>https://www.intel.com/content/www/us/en/developer/tools/oneapi/mpi-library.html#gs.xld8oa</ref> - a popular implementation of MPICH especially optimised for Intel CPUs
* OpenMPI <ref>https://www.open-mpi.org/</ref> - an open-source library
* OpenMP <ref>https://www.openmp.org/</ref> - Not MPI; parallelization based on shared memory, so only implemented in a single node; can be used for multithreading

In practice, a hybrid parallelization combining MPI and OpenMP to run multithread jobs on cluster is allowed, though sometimes not recommended. The first process (probably not a node or a processor) is usually allocated for I/O, and the rest is used for parallel computing.

So far, MPI only supports C/C++ and FORTRAN, which explains why all parallel computing software is based on these languages. To launch an executable in parallel, one should use: <code>mpiexec</code> or <code>mpirun</code>.

=== Secure your storage: Tmp memory, Work directory and home directory ===

Almost all the modern clusters have separate disk spaces for differently proposes, namely, temporary memory, work directory and home directory. This originates again from the famous speed difference between CPU and RAM/ROM. 2 distinctly kinds of disks are used respectively to improve the overall efficiency and secure important data:

* For temporary memory large, high-frequency disks are used. It is allocated by job requests, which is not accessible by login nodes. Everything is erased after the job is terminated.
* For work directory, large, high-frequency disks are used. Data stored in work directory is usually not backed up, and in the case of CX1, will be automatically cleaned after a fixed time length.
* For home directory, mechanical disks with slower read/write frequency but better robustness are used. Usually files in home space are backed up.

For large clusters like ARCHER2 <ref>https://www.archer2.ac.uk/</ref>, the work directory and the home directory are completely separated, i.e., directory is only viable by login nodes; work directory is viable by both job and login nodes. Job submission in home directory is prohibited. For more flexible clusters like Imperial CX1, submitting jobs in home directory and visiting of home directory by job nodes are allowed, but storing temporary files during calculation in home directory is still not recommended because of the potential influence on other files and the reduced overall efficiency. (And it is not something new for CX1 users to receive the RDS failure news email)

=== Setup your environment: What does an application need? ===

==== Executable ====
The binary executable should, theoretically, all be stored in '\usr\bin'. This never happens in practice, unless you are a fanatical fundamentalist of the early Linux releases. To guide your system to the desired executable, you can either laboriously type its absolute path every time you need it or add the path to the environmental variable:

<code>
~$ export PATH=${PATH}:path_to_bin
</code>

Running any executable in parallel requires mpi to coordinate all the processes/threads. The path to mpi executable is also required. Besides, many scientific codes require other specific environmental variables such as linear algebra packages. Read their documentations for further information.

==== .lib/.a/.o files ====

When writing a script, you might need some extra packages to do more complex jobs. Those packages are developed by experts in computer science and can be called by a line of code. The same thing happens when people were developing applications like CRYSTAL and ONETEP.

However, scientific computing codes are usually distributed in the form of source code. Source codes in FORTRAN/C/C++ need be compiled into a binary executable. There are 2 options during compiling:

# Include the whole package as long as one of its functions is called, also known as a 'static lib'.
# Only include a 'table of contents' when compiling, also known as 'dynamic lib'. The packages needed are separately stored in '.dll/.so' files, making it possible for multiple applications sharing the same lib.

Details about compilation are beyond the scope of this post. The thing is: when running a dynamically linked application, information should be given to help the code find the libs needed. This can be specified by:

<code>
~$ export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:path_to_lib`
</code>

For statically linked applications, usually you need not worry about it - but the volume of the compiled executable might make you wonder whether there is an alternative way.

==== Conflicts ====

Improper previous settings may lead to a wrong application, or a wrong version, if multiple applications with similar functions are installed in the system, such as Intel compiler and GCC, OpenMPI and MPICH - a common phenomenon for shared computing resources. To avoid this, the path to the undesired application or lib should be removed from the environmental variables.

==== Environmental Modules ====

Environmental Modules <ref>http://modules.sourceforge.net/</ref> is a popular software managing the necessary environmental setups and conflicts for each application. It can easily add or erase the environmental variables by commands (such as <code>module load</code>, <code>module rm</code>) and modulefiles written in Tool Command Language (TCL)<ref>https://www.tcl.tk/</ref>. The default directory of modulefiles is given in the environmental variable <code>${MODULEPATH}</code>, but files in other directories can also be loaded by their absolute path.

Both Imperial CX1 and ARCHER2 adopt this application, with which pre-compiled applications are offered.

=== The external coordinator: What is a batch system ===

Always bear in mind that the computational resources are limited, so you need to acquire reasonable resources for your job. Besides, the cluster also needs to calculate your budget, coordinate jobs submitted by various users, and make the best of available resources. When job is running, maybe you also want to check its status. All of this are fulfilled by batch systems.

In practice, a Linux shell script is needed. Parameters of the batch system of are set in the commented lines at the top of the file. After the user submit the script to batch system, the system will:

# Examine the parameters
# Allocate and coordinate the requested resources
# Set up the environments, such as environmental variables, package dependency, and sync the same setting to all nodes
# Launch a parallel calculation - see mpi part
# Post-process

Note that a 'walltime' is usually required for a batch job, i.e., the maximum allowed time for the running job. The job will be 'killed', or suspended, when the time exceeds the walltime, and the rest part of the script will not be executed. <code>timeout</code> command can be used to set another walltime for a specific command.

Common batch systems include PBS and Slurm <ref>https://slurm.schedmd.com/overview.html</ref>. For Imperial cluster CX1 and MMM Hub Young (managed by UCL) <ref>http://mmmhub.ac.uk/young/</ref>, PBS system is implemented; for ARCHER2 and Tianhe-2 LvLiang(天河二号-吕梁), Slurm is implemented. Tutorials of batch systems are not covered here, since they are heavily tailored according to specific machines - usually modifications are made to enhance the efficiency. Refer to the specific user documentations for more information.

Successfully setting and submitting a batch job script symbolises that you do not need this tutorial any more. Before being able to do that, some considerations might be important:

* How large is my system? Is it efficient to use the resources I requested(Note that it is not a linear-scaling problem... Refer to [https://tutorials.crystalsolutions.eu/tutorial.html?td=tuto_HPC&tf=tuto_hpc#scale this test] on CRYSTAL17)?
* To which queue should I submit my job? Is it too long/not applicable/not available?
* Is it safe to use multi-threading?
* Is it memory, GPU etc. demanding?
* Roughly how long will it take?
* What is my budget code? Do I have enough resources?
* Which MPI release version is my code compatible with? Should I load a module or set variables?
* Any other specific environmental setups does my code need?
* Do I have any post-processing script after MPI part is finished? How long does it take?

==References==
<references>

Imperial CX1: Instructions and basic concepts of parallel computing

2024-02-13T20:36:35Z

Hz1420:

This tutorial is divided into 2 separate sections. In the fist section, introductions and available resources of CX1 are listed and classified. Since the [https://icl-rcs-user-guide.readthedocs.io/en/latest/ Research Computing Service (RCS)] team already developed great tutorials on their webpages, this part functions as a guide towards RCS webpages with necessary supplementary comments. In the second section, basic concepts of parallel computing and explanations of important terms are introduced. The main focus of this section is helping beginners to understand how high-performance computers (HPC) works on the basis of their daily practise.

''This tutorial was initially written between Feb. and Mar. 2022 to be shared within the group for induction and training proposes <ref>https://spica-vir.github.io/posts/Connect-to-the-Imperial-Cluster/</ref><ref>https://spica-vir.github.io/posts/Structure-and-usage-of-clusters/</ref>. Special thanks to Mr K. Tallat-Kelpsa, Ms A. Arber, Dr G. Mallia and Prof N. M. Harrison.''

== Introduction to CX1 ==
CX1 is the old name of the first HPC that served the whole college. New facilities (known as CX2) were gradually installed and integrated with the old system (CX3, a rather short-lived domain), while CX1 remains to be the most popular name that generally referring to the college-owned clusters. To grant a student access to CX1, the group PI can, on behave of that student, ask RCS team to add the specified account into HPC active user mailing list.
=== Connect to CX1 ===
CX1 is typically accessed via ssh (secured shell). Linux command line (Linux & MacOS) / sub-system (Windows 10,11) <ref>https://learn.microsoft.com/en-us/windows/wsl/install</ref> / SSH client (such as XShell <ref>https://www.xshell.com/en/xshell/</ref>) can be used. VPN is needed for off-campus users.

* [https://www.imperial.ac.uk/admin-services/ict/self-service/connect-communicate/remote-access/virtual-private-network-vpn/ Step-by-step guide to setup VPN]
* [https://www.imperial.ac.uk/admin-services/ict/self-service/research-support/rcs/support/getting-started/using-ssh/ How to use ssh]

In linux command line, use the following command to connect CX1:

<code>
~$ ssh -XY username@login.hpc.ic.ac.uk
</code>

P.S. <code>-XY</code> option can be omitted for most of cases, if you do not need GUI to run that program.

Alternatively, when the VPN service is unstable or even not available, it is possible to channel through the gateway of the cluster via a client, which is an 'agent'. To visit CX1, type the previous command in the client's command line.

<code>
~$ ssh username@sshgw.ic.ac.uk
</code>

Use the <code>scp</code> command to upload / download files, which is similar to <code>ssh</code> and <code>cp</code> command. For example, to upload a file:

<code>
~$ scp /local/path/file_name username@login.hpc.ic.ac.uk:/path/file_name
</code>

=== Usage ===
The [https://icl-rcs-user-guide.readthedocs.io/en/latest/hpc/ RCS Wiki Page in ReadTheDocs] contains information needed. [https://www.imperial.ac.uk/admin-services/ict/self-service/research-support/rcs/get-support/contact-us/ The support page], [https://wiki.imperial.ac.uk/display/HPC/Attend+a+clinic online clinic] and [https://wiki.imperial.ac.uk/display/HPC/Courses courses from graduate school] are available. To examine the status of CX1, use [https://status.rcs.imperial.ac.uk/d/u0zkcYQ7z/rcs-status?orgId=2&refresh=1m RCS status page].

==== Environmental Variables and Disk Space ====
Use <code>env</code> to access all the environmental variables - be careful, the output is '''HUGE'''. Some useful environmental variables:

* <code>${USER}</code> The user's college account, i.e., login credential.
* <code>${HOME}</code> '/rds/general/user/${USER}/home', or '~', which has 1 TB disk space for data backups.
* <code>${EPHEMERAL}</code> '/rds/general/user/${USER}/ephemeral' Temporal unlimited disk space lasting for 30 days. Suitable for running calculations.
* <code>${PATH}</code> Path to the executable can be attached for quick access. The Environment Modules package (see below) can automatically do that.

==== Software Management ====

The Environment Modules<ref>https://modules.readthedocs.io/en/latest/</ref> package is implemented on CX1 to manage computing software (see the following section for introductions). Basic commands are listed below:

* <code>module avail</code> List the available modules
* <code>module load mod_name</code> Load a specific module, 'mod_name'
* <code>module rm mod_name</code> Remove a specific module, 'mod_name'
* <code>module list</code> List all the loaded modules in the current environment
* <code>module help mod_name</code> Check the instructions of the module 'mod_name'

''Note: There is a CRYSTAL14 module in the list. For users in NMH's group, the latest CRYSTAL edition is available, so do not use that module.''

==== Job Partition Guide ====
A hierachy of jobs is designed for the optimial efficiency of CX1. The current job partition guide is available on [https://wiki.imperial.ac.uk/display/HPC/New+Job+sizing+guidance RCS Wiki Page]

==== Batch System ====

The PBS batch system <ref>https://en.wikipedia.org/wiki/Portable_Batch_System</ref> is used on CX1 (see the following section for the meaning of batch system). Basic commands of PBS are listed below:

* <code>availability</code> Check the availability of computational resources
* <code>qsub</code> filename.qsub</code> Submit the job 'filename'
* <code>qstat</code> Check the state of submitted jobs
* <code>qdel jobID</code> Kill the process with the ID number 'jobID'

To examine the queue status across the whole system, use [https://status.rcs.imperial.ac.uk/d/u0zkcYQ7z/rcs-status?orgId=2&refresh=1m RCS status page].

==== A General Job Submission Script ====
A general job submission script for CX1 is developed by the author himself. See the [https://github.com/cmsg-icl/crystal_shape_control/tree/main/Imperial-HPC-Job-Submission GitHub repository] of CMSG for details. Parameterised software includes: CRYSTAL17/23, Quantum Espresso 7, LAMMPS, GROMACS, GULP6.

== Basic Concepts of Parallel Computing ==
A brief introduction to parallel computing is given in this section by taking CX1, a medium-sized general-propose cluster, as an example.

=== Divide a job: Nodes, Processors and Threads ===

;Node
:A bunch of CPUs and probably with GPUs / coprocessors for acceleration. Memory and input files are shared by processors in the same node, so a node can be considered as an independent computer. The communication between nodes are achieved by ultra-fast network, which is the bottleneck of modern clusters.

;Processor
:The unit to deal with a 'process', also known as 'central processing unit', or CPU. Processors in the same node communicate via shared memory.

;Thread
:Subdivision of a process. Multiple threads in the same process share the resources allocated to the CPU.

The figure on the right hand side illustrates the hierarchy of node, processor, and thread. '''Note:''' The word 'processor' is not a very accurate term. Might be better with 'process' (I am just too lazy to update that figure). Many modern CPUs supports sub-CPU threading, which means the number of logical CPUs is larger than physical CPUs, so it is possible to have multiple threads within 1 processor. However, it is also possible to use multiple processors for 1 process, or even 1 thread.

[[File:Job_Partition.png|450px|right|Job Partition]]

==== Multiple processes vs multiple threads ====

From the figure above, it is not difficult to distinguish the differences between a 'process' and a 'thread': process is the smallest unit for resource allocation; thread is part of a process. The idea of 'thread' is introduced to address the huge difference in the speed of CPU and RAM. CPU is always several orders of magnitude faster than RAM, so typically the bottleneck of a process is loading the required environment from RAM, rather than computations in CPU. By using multiple threads in the same process, various branches of the same program can be executed simultaneously. Therefore, the shared environmental requirements doesn't need to be read from RAM for multiple times, and the loading time for threads is much smaller than for processes.

However, multithreading is not always advantageous. A technical prerequisite is that the program should be developed for multithread proposes. Python, for example, is a pseudo-multithread language, while Java is a real one. Sometimes multithreading can lead to catastrophic results. Since threads share the same resource allocation (CPU, RAM, I/O, etc.), when a thread fails, the whole process fails as well. Comparatively, in multiple processes, other processes will be protected if a process fails.

In practice, users can either run each process in serial (i.e., number of threads = 1), or in parallel (i.e., number of threads > 1) on clusters. However, '''the former one is recommended''', because of more secured resource managements. The latter is not advantageous. Besides the problem mentioned above, it might lead to problems such as memory leak when running programs either: not developed for multithreading / requires improper packages (Here is [https://docs.archer2.ac.uk/known-issues/#oom-due-to-memory-leak-in-libfabric-added-2022-02-23 a famous issue] with libfabric on ARCHER2 identified in early 2022).

==== More nodes vs more CPUs ====

When the allocated memory permits, from my experience, using more CPUs/processes per node is usually a better idea, considering that all nodes have independent memory space and the inter-node communications are achieved by wired networks. It almost always takes longer to coordinate nodes than to coordinate processors within the same node.

=== The internal coordinator: What is MPI ===

Message passing interface, or MPI, is a standard for communicating and transferring data between nodes and therefore distributed memories. It is utilised via MPI libraries. The most popular implementations include:

* MPICH <ref>https://www.mpich.org/</ref> - an open-source library
* Intel MPI <ref>https://www.intel.com/content/www/us/en/developer/tools/oneapi/mpi-library.html#gs.xld8oa</ref> - a popular implementation of MPICH especially optimised for Intel CPUs
* OpenMPI <ref>https://www.open-mpi.org/</ref> - an open-source library
* OpenMP <ref>https://www.openmp.org/</ref> - Not MPI; parallelization based on shared memory, so only implemented in a single node; can be used for multithreading

In practice, a hybrid parallelization combining MPI and OpenMP to run multithread jobs on cluster is allowed, though sometimes not recommended. The first process (probably not a node or a processor) is usually allocated for I/O, and the rest is used for parallel computing.

So far, MPI only supports C/C++ and FORTRAN, which explains why all parallel computing software is based on these languages. To launch an executable in parallel, one should use: <code>mpiexec</code> or <code>mpirun</code>.

=== Secure your storage: Tmp memory, Work directory and home directory ===

Almost all the modern clusters have separate disk spaces for differently proposes, namely, temporary memory, work directory and home directory. This originates again from the famous speed difference between CPU and RAM/ROM. 2 distinctly kinds of disks are used respectively to improve the overall efficiency and secure important data:

* For temporary memory large, high-frequency disks are used. It is allocated by job requests, which is not accessible by login nodes. Everything is erased after the job is terminated.
* For work directory, large, high-frequency disks are used. Data stored in work directory is usually not backed up, and in the case of CX1, will be automatically cleaned after a fixed time length.
* For home directory, mechanical disks with slower read/write frequency but better robustness are used. Usually files in home space are backed up.

For large clusters like ARCHER2 <ref>https://www.archer2.ac.uk/</ref>, the work directory and the home directory are completely separated, i.e., directory is only viable by login nodes; work directory is viable by both job and login nodes. Job submission in home directory is prohibited. For more flexible clusters like Imperial CX1, submitting jobs in home directory and visiting of home directory by job nodes are allowed, but storing temporary files during calculation in home directory is still not recommended because of the potential influence on other files and the reduced overall efficiency. (And it is not something new for CX1 users to receive the RDS failure news email)

=== Setup your environment: What does an application need? ===

==== Executable ====
The binary executable should, theoretically, all be stored in '\usr\bin'. This never happens in practice, unless you are a fanatical fundamentalist of the early Linux releases. To guide your system to the desired executable, you can either laboriously type its absolute path every time you need it or add the path to the environmental variable:

<code>
~$ export PATH=${PATH}:path_to_bin
</code>

Running any executable in parallel requires mpi to coordinate all the processes/threads. The path to mpi executable is also required. Besides, many scientific codes require other specific environmental variables such as linear algebra packages. Read their documentations for further information.

==== .lib/.a/.o files ====

When writing a script, you might need some extra packages to do more complex jobs. Those packages are developed by experts in computer science and can be called by a line of code. The same thing happens when people were developing applications like CRYSTAL and ONETEP.

However, scientific computing codes are usually distributed in the form of source code. Source codes in FORTRAN/C/C++ need be compiled into a binary executable. There are 2 options during compiling:

# Include the whole package as long as one of its functions is called, also known as a 'static lib'.
# Only include a 'table of contents' when compiling, also known as 'dynamic lib'. The packages needed are separately stored in '.dll/.so' files, making it possible for multiple applications sharing the same lib.

Details about compilation are beyond the scope of this post. The thing is: when running a dynamically linked application, information should be given to help the code find the libs needed. This can be specified by:

<code>
~$ export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:path_to_lib`
</code>

For statically linked applications, usually you need not worry about it - but the volume of the compiled executable might make you wonder whether there is an alternative way.

==== Conflicts ====

Improper previous settings may lead to a wrong application, or a wrong version, if multiple applications with similar functions are installed in the system, such as Intel compiler and GCC, OpenMPI and MPICH - a common phenomenon for shared computing resources. To avoid this, the path to the undesired application or lib should be removed from the environmental variables.

==== Environmental Modules ====

Environmental Modules <ref>http://modules.sourceforge.net/</ref> is a popular software managing the necessary environmental setups and conflicts for each application. It can easily add or erase the environmental variables by commands (such as <code>module load</code>, <code>module rm</code>) and modulefiles written in Tool Command Language (TCL)<ref>https://www.tcl.tk/</ref>. The default directory of modulefiles is given in the environmental variable <code>${MODULEPATH}</code>, but files in other directories can also be loaded by their absolute path.

Both Imperial CX1 and ARCHER2 adopt this application, with which pre-compiled applications are offered.

=== The external coordinator: What is a batch system ===

Always bear in mind that the computational resources are limited, so you need to acquire reasonable resources for your job. Besides, the cluster also needs to calculate your budget, coordinate jobs submitted by various users, and make the best of available resources. When job is running, maybe you also want to check its status. All of this are fulfilled by batch systems.

In practice, a Linux shell script is needed. Parameters of the batch system of are set in the commented lines at the top of the file. After the user submit the script to batch system, the system will:

# Examine the parameters
# Allocate and coordinate the requested resources
# Set up the environments, such as environmental variables, package dependency, and sync the same setting to all nodes
# Launch a parallel calculation - see mpi part
# Post-process

Note that a 'walltime' is usually required for a batch job, i.e., the maximum allowed time for the running job. The job will be 'killed', or suspended, when the time exceeds the walltime, and the rest part of the script will not be executed. <code>timeout</code> command can be used to set another walltime for a specific command.

Common batch systems include PBS and Slurm <ref>https://slurm.schedmd.com/overview.html</ref>. For Imperial cluster CX1 and MMM Hub Young (managed by UCL) <ref>http://mmmhub.ac.uk/young/</ref>, PBS system is implemented; for ARCHER2 and Tianhe-2 LvLiang(天河二号-吕梁), Slurm is implemented. Tutorials of batch systems are not covered here, since they are heavily tailored according to specific machines - usually modifications are made to enhance the efficiency. Refer to the specific user documentations for more information.

Successfully setting and submitting a batch job script symbolises that you do not need this tutorial any more. Before being able to do that, some considerations might be important:

* How large is my system? Is it efficient to use the resources I requested(Note that it is not a linear-scaling problem... Refer to [https://tutorials.crystalsolutions.eu/tutorial.html?td=tuto_HPC&tf=tuto_hpc#scale this test] on CRYSTAL17)?
* To which queue should I submit my job? Is it too long/not applicable/not available?
* Is it safe to use multi-threading?
* Is it memory, GPU etc. demanding?
* Roughly how long will it take?
* What is my budget code? Do I have enough resources?
* Which MPI release version is my code compatible with? Should I load a module or set variables?
* Any other specific environmental setups does my code need?
* Do I have any post-processing script after MPI part is finished? How long does it take?

==References==
<references>

CMSG disk and Shared Software on CX1

2023-12-01T00:03:02Z

Hz1420:

General information of shared software available for CMSG group members on Imperial CX1 is summarized in this page. Software listed below, unless stated otherwise, is hosted under 'CMSG' project of Imperial Research Data Store (RDS) <ref>https://www.imperial.ac.uk/admin-services/ict/self-service/research-support/rcs/service-offering/rds/</ref> (referred as RDS-CMSG below), which is accessible for both login and computational nodes of Imperial HPC CX1. For Imperial HPC HX1, by the time this page is upated (Nov. 2023), it is not accessible there. Only login node can visit RDS-CMSG disk after the pilot phase <ref>https://wiki.imperial.ac.uk/display/HPC/HX1+Cluster</ref>. Therefore software on RDS-CMSG are classified into 2 catagories:

# For CX1, use <code>/rds/general/project/cmsg/live/app/</code> (executables, libraries and headers) and <code>/rds/general/project/cmsg/live/etc/</code> (other files, such as pseudopotential). Executable can be called directly from the user's home directory.
# For HX1, use <code>/rds/general/project/cmsg/live/share/</code>. Executables must be downloaded and uploaded to HX1. It is user's responsibility to set up an effective running environment according to 'readme' files (if there is one) saved in the same, or upper, directory.

''Please note that accessibility to CMSG disk is granted for CMSG group members only. If you need to access it, please contact group PI [[Contributors | Prof. Nicholas Harrison]].''

== Step 0 : Environment Modules ==
Environment modules <ref>https://modules.readthedocs.io/en/latest/</ref> manages the compilation and running environment to ensure all of its dependencies are loaded when an executable is launched. For software on RDS-CMSG, typically a module file is prepared, which can be called by <code>module load</code> command. The user is strongly suggested to add the following line to their <code>~/.bashrc</code> file on CX1:

<code>
export MODULEPATH="/rds/general/project/cmsg/live/etc/modulefiles:${MODULEPATH}"
</code>

That helps the 'module' executable to find module files hosted on RDS-CMSG. Then 'module' commands do not require real path. But in case that readers skip this section (as they always do), in the following text, the read path is always used. Use the following command to update the environment variable <code>${MODULEPATH}</code>:

<code>
~$ source ~/.bashrc
</code>

Due to the settings of CX1, the <code>source</code> command should be executed everytime you login. To aviod this, try the following commands:

~$ cat << EOF >> ~/.bash_profile
if test -f ~/.bashrc; then
source ~/.bashrc
fi
EOF

== CRYSTAL23 v1 (Intel) ==
The current default for CX1. Not available for HX1.

=== CX1 version ===
* Compiling Env : EasyBuild Intel 2023a, OpenMP
* Note: MPPproperties and MP2 (CRYSCOR and CRYSTAL2) are currently not available, which involves separate code packages (DMat2 and CRYSCOR). For MPPproperties please read the GNU version below.

The general job submission script for Imperial HPC, developed by the author himself, is used here.

''Thanks to Mr. K Tallat-Kelpsa for testing''

''' Job submission script configuration '''

The source code (written in bash) is available on group [https://github.com/cmsg-icl/HPC-job-submission/tree/main/Imperial-HPC-Job-Submission GitHub repo], though it is not needed in practice. Useful for developing new features. On CX1, use the following command to configure your local settings:

<code>
~$ bash /rds/general/project/cmsg/live/share/HPC-job-submission/Imperial-HPC-Job-Submission/CRYSTAL23/config_CRYSTAL23.sh
</code>

Then follow the instructions on the screen. Typically the default values are good. To use default vaules, press enter. After configuration is finished, the information is stored in a 'settings' file which is by default saved as <code>${HOME}/etc/runCRYSTAL23/settings</code>. It functions as a dictionary for job sumbmission scripts to refer to. It is also editable according to the user's needs.

''' Job submission script commands '''

After configuration, use <code>source ~/.bashrc</code> to enable alias commands.

{| class="wikitable" style="margin:auto"
|+ List of available commands
|-
! Command !! Definition
|-
| Pcrys23 || Generate parallel OMP CRYSTAL23 job submission files
|-
| MPPcrys23 || Generate massive parallel OMP CRYSTAL23 job submission files
|-
| Pprop23 || Generate parallel OMP CRYSTAL23 properties job submission files
|-
| Scrys23 || Generate serial OMP CRYSTAL23 job submission files
|-
| Sprop23 || Generate serial OMP CRYSTAL23 properties job submission files
|-
| Xcrys23 || User-defined executables and multiple jobs (see below for code examples)
|-
| SETcrys23 || Print the local 'settings' file
|-
| HELPcrys23 || Print the instructions of commands
|}

''' Examples '''

To generate a qsub file for a parallel crystal23 job on 'mgo.d12', the following command can be used:

<code>
~$ Pcrys23 -in mgo.d12 -wt 01:00 -nd 1
</code>

That generates a 'mgo.qsub' file requesting for 1 node and the maximum running time is 1 hour. Similarly, after the job is done, using the following command can run a parallel properties calculation based on 'mgo-band.d3' and data from the previous 'mgo' SCF calculation on 12 CPUs:

<code>
~$ Pprop23 -in mgo-band.d3 -nc 12 -wt 00:30 -ref mgo
</code>

For detailed instructions and testing cases, please refer to the [https://github.com/cmsg-icl/HPC-job-submission/tree/main/Imperial-HPC-Job-Submission documentation].

=== PC version ===
A few variants are available in <code>/rds/general/project/cmsg/live/share/CRYSTAL</code>, including executables compiled for Windows Subsystem for Linux (WSL) and MacOSx. Please check the readme files in saved in individual directories for specifications.

A set of statically linked 'crystal' and 'properties' executables is available in <code>/rds/general/project/cmsg/live/share/CRYSTAL/23v1/Linux-intel2023-x86-intel2023</code>, which do not have any prerequisite and can be run in serial on either Linux or MacOSx with x86 CPU.

== CRYSTAL23 v1 (GNU) ==
Compared to Intel version above, GNU version is slightly slower to allow for compatibility with MPPproperties. Available for CX1 and HX1.

=== CX1 version ===
* Compiling Env : gcc11.2.0, aocl4.0, mpich4.0.2, OpenMP
* Note: MP2 (CRYSCOR and CRYSTAL2) is currently not available.

The same job submission script is used. However, GNU version is not the default option for CX1. To enable GNU version, if there is a 'settings' file, change the following parameters:

* '''EXEDIR''' : <code>module load /rds/general/project/cmsg/live/etc/modulefiles/CRYSTAL/23v1-gcc</code>
* '''MPIDIR''' : <code>module load /rds/general/project/cmsg/live/etc/compiler/gcc11.2.0-aocl</code>.

Alternatively, rerun the configuration file (same as above) and specify 2 values during configuration.

Note that by default, there is no command for 'MPPproperties'. The user has to modify their <code>~/.bashrc</code> file to set the alias and 'settings' file for executable flags. An example:

# in ~/.bashrc
alias MPPprop23="/rds/general/project/cmsg/live/share/HPC-job-submission/Imperial-HPC-Job-Submission/gen_sub -x mppprop -set ${HOME}/etc/runCRYSTAL23/settings"
# in settings, keep the column width
mppprop mpiexec -np ${V_TPROC} MPPproperties Massive parallel properties calculation, OMP

=== HX1 version ===
* Compiling Env : EasyBuild foss/2023a, OpenMP
* Note: MP2 (CRYSCOR and CRYSTAL2) is currently not available.

Executables are saved in <code>/rds/general/project/cmsg/live/share/CRYSTAL/23v1/Linux-EBFOSS2023a-HX1-ompi</code>. To setup the running environment on HX1, if the user has a copy of 'settings' file, they need to modify the following keywords:

* '''EXEDIR''' : No default. The directory where the user put their executables in.
* '''MPIDIR''' : No default, but please use <code>module load foss/2023a</code> unless a self-compiled version is used.

Alternatively, run the configuration file (The one with '-HX1' in the same directory of Github repo) and specify 2 values during configuration. Other options are consistent with CRYSTAL on CX1.

== CRYSTAL17 v2 (GNU)==
Not hosted on RDS-CMSG. Default for CX1. Not available for HX1.

=== CX1 version ===
* Compilor : gcc/6.2.0
* MPI : mpich/3.4.3

'''For job submission scripts'''

The configuration and usage are identical to CRYSTAL23. Please refer to the previous section and substitute '23' with '17'. Also, please be noted that CRYSCOR (MP2 calculation) is not available for CRYSTAL17 and 'MPP' as a massive parallel strategy is limited to 'crystal' calculations, i.e., only MPPcrystal is released.

== Quantum Espresso v7.2 (GNU) ==
Job submission file configured for CX1 and HX1, but only CX1 executable is shared as these executables are highly dependent on environment.

=== CX1 version ===
* Compiling Env : EasyBuild foss2022a, OpenMP
* libxc: Yes, version 5.1.2
* hdf5: No

Pseudopotential files are saved in <code>/rds/general/project/cmsg/live/etc/QE_PseudoP</code>.

''' Job submission script configuration '''

<code>
~$ bash /rds/general/project/cmsg/live/share/HPC-job-submission/Imperial-HPC-Job-Submission/QE7/config_QE7.sh
</code>

The procedure is the same as CRYSTAL. Type enter for default setups, which typically works fine. Please note that moving files between temporal and home directories are disabled as QE has built-in file management system.

''' Job submission script commands '''

{| class="wikitable" style="margin:auto"
|+ List of available commands
|-
! Command !! Definition
|-
| PWqe7 || Generate pw.x job submission files
|-
| PHqe7 || Generate ph.x job submission files
|-
| CPqe7 || Generate cp.x job submission files
|-
| PPqe7 || Generate pp.x job submission files
|-
| Xqe7 || Generate job submission files for user-defined executables
|-
| SETqe7 || Print the local 'settings' file
|-
| HELPqe7 || Print the instructions of commands
|}

== GULP v6.1.2 (GNU) ==
Default for CX1. Not available for HX1.

=== CX1 version ===
* Compiling Env : EasyBuild foss2022a
* OpenKIM: Yes, version 2.3.0
* PLUMED: Yes, version 2.9.0
* ALAMODE: Yes, git repo copied into <code>app/</code>

GULP force field libraries are saved in <code>/rds/general/project/cmsg/live/etc/GULP_Libraries</code>. Executable name: 'gulp-mpi'.

'''OpenKIM'''

KIM force field <ref>https://openkim.org/</ref> are saved in <code>/rds/general/project/cmsg/live/etc/KIM_Models</code>. To run OpenKIM executable and download new models, run:

<code>
~$ module load /rds/general/project/cmsg/live/etc/modulefiles/OpenKIM/2.3.0-foss
</code>

'''Job submission script configuration'''

<code>
~$ bash /rds/general/project/cmsg/live/share/HPC-job-submission/Imperial-HPC-Job-Submission/GULP6/config_GULP6.sh
</code>

Type enter for default setups, which typically works fine.

'''Job submission script commands'''

{| class="wikitable" style="margin:auto"
|+ List of available commands
|-
! Command !! Definition
|-
| Pglp6 || Generate job submission files for 'gulp-mpi'
|-
| Xglp6 || Generate job submission files for user defined executable (by default nothing else, so not useful).
|-
| SETglp6 || Print the local 'settings' file
|-
| HELPglp6|| Print the instructions of commands
|}

== ONETEP v6.1.9.2 (GNU) ==
Default for CX1. Not available for HX1. Job submission script not available.

=== CX1 version ===
* Compiling Env : EasyBuild foss2022a, OpenMP, FFTW, Scalapack
* Libxc: Yes, version 5.1.2

Executable path is <code>/rds/general/project/cmsg/live/app/ONETEP/6.1.2.2__foss2022a/bin</code>. Pseudopotential files are saved in <code>/rds/general/project/cmsg/live/etc/ONETEP_PseudoP</code>.

==References==
<references>

CMSG disk and Shared Software on CX1

2023-11-30T23:48:30Z

Hz1420: /* CRYSTAL23 v1 (Intel) */

General information of shared software available for CMSG group members on Imperial CX1 is summarized in this page. Software listed below, unless stated otherwise, is hosted under 'CMSG' project of Imperial Research Data Store (RDS) <ref>https://www.imperial.ac.uk/admin-services/ict/self-service/research-support/rcs/service-offering/rds/</ref> (referred as RDS-CMSG below), which is accessible for both login and computational nodes of Imperial HPC CX1. For Imperial HPC HX1, by the time this page is upated (Nov. 2023), it is not accessible there. Only login node can visit RDS-CMSG disk after the pilot phase <ref>https://wiki.imperial.ac.uk/display/HPC/HX1+Cluster</ref>. Therefore software on RDS-CMSG are classified into 2 catagories:

# For CX1, use <code>/rds/general/project/cmsg/live/app/</code> (executables, libraries and headers) and <code>/rds/general/project/cmsg/live/etc/</code> (other files, such as pseudopotential). Executable can be called directly from the user's home directory.
# For HX1, use <code>/rds/general/project/cmsg/live/share/</code>. Executables must be downloaded and uploaded to HX1. It is user's responsibility to set up an effective running environment according to 'readme' files (if there is one) saved in the same, or upper, directory.

''Please note that accessibility to CMSG disk is granted for CMSG group members only. If you need to access it, please contact group PI [[Contributors | Prof. Nicholas Harrison]].''

== CRYSTAL23 v1 (Intel) ==
The current default for CX1. Not available for HX1.

=== CX1 version ===
* Compiling Env : EasyBuild Intel 2023a, OpenMP
* Note: MPPproperties and MP2 (CRYSCOR and CRYSTAL2) are currently not available, which involves separate code packages (DMat2 and CRYSCOR). For MPPproperties please read the GNU version below.

The general job submission script for Imperial HPC, developed by the author himself, is used here.

''Thanks to Mr. K Tallat-Kelpsa for testing''

''' Job submission script configuration '''

The source code (written in bash) is available on group [https://github.com/cmsg-icl/HPC-job-submission/tree/main/Imperial-HPC-Job-Submission GitHub repo], though it is not needed in practice. Useful for developing new features. On CX1, use the following command to configure your local settings:

<code>
~$ bash /rds/general/project/cmsg/live/share/HPC-job-submission/Imperial-HPC-Job-Submission/CRYSTAL23/config_CRYSTAL23.sh
</code>

Then follow the instructions on the screen. Typically the default values are good. To use default vaules, press enter. After configuration is finished, the information is stored in a 'settings' file which is by default saved as <code>${HOME}/etc/runCRYSTAL23/settings</code>. It functions as a dictionary for job sumbmission scripts to refer to. It is also editable according to the user's needs.

Use the following command to activate alias commands:

<code>
~$ source ~/.bashrc
</code>

Due to the settings of CX1, the <code>source</code> command should be executed everytime you login. To aviod this, try the following commands:

~$ cat << EOF >> ~/.bash_profile
if test -f ~/.bashrc; then
source ~/.bashrc
fi
EOF

''' Job submission script commands '''

{| class="wikitable" style="margin:auto"
|+ List of available commands
|-
! Command !! Definition
|-
| Pcrys23 || Generate parallel OMP CRYSTAL23 job submission files
|-
| MPPcrys23 || Generate massive parallel OMP CRYSTAL23 job submission files
|-
| Pprop23 || Generate parallel OMP CRYSTAL23 properties job submission files
|-
| Scrys23 || Generate serial OMP CRYSTAL23 job submission files
|-
| Sprop23 || Generate serial OMP CRYSTAL23 properties job submission files
|-
| Xcrys23 || User-defined executables and multiple jobs (see below for code examples)
|-
| SETcrys23 || Print the local 'settings' file
|-
| HELPcrys23 || Print the instructions of commands
|}

''' Examples '''

To generate a qsub file for a parallel crystal23 job on 'mgo.d12', the following command can be used:

<code>
~$ Pcrys23 -in mgo.d12 -wt 01:00 -nd 1
</code>

That generates a 'mgo.qsub' file requesting for 1 node and the maximum running time is 1 hour. Similarly, after the job is done, using the following command can run a parallel properties calculation based on 'mgo-band.d3' and data from the previous 'mgo' SCF calculation on 12 CPUs:

<code>
~$ Pprop23 -in mgo-band.d3 -nc 12 -wt 00:30 -ref mgo
</code>

For detailed instructions and testing cases, please refer to the [https://github.com/cmsg-icl/HPC-job-submission/tree/main/Imperial-HPC-Job-Submission documentation].

=== PC version ===
A few variants are available in <code>/rds/general/project/cmsg/live/share/CRYSTAL</code>, including executables compiled for Windows Subsystem for Linux (WSL) and MacOSx. Please check the readme files in saved in individual directories for specifications.

A set of statically linked 'crystal' and 'properties' executables is available in <code>/rds/general/project/cmsg/live/share/CRYSTAL/23v1/Linux-intel2023-x86-intel2023</code>, which do not have any prerequisite and can be run in serial on either Linux or MacOSx with x86 CPU.

== CRYSTAL23 v1 (GNU) ==
Compared to Intel version above, GNU version is slightly slower to allow for compatibility with MPPproperties. Available for CX1 and HX1.

=== CX1 version ===
* Compiling Env : gcc11.2.0, aocl4.0, mpich4.0.2, OpenMP
* Note: MP2 (CRYSCOR and CRYSTAL2) is currently not available.

The same job submission script is used. However, GNU version is not the default option for CX1. To enable GNU version, if there is a 'settings' file, change the following parameters:

* '''EXEDIR''' : <code>module load /rds/general/project/cmsg/live/etc/modulefiles/CRYSTAL/23v1-gcc</code>
* '''MPIDIR''' : <code>module load /rds/general/project/cmsg/live/etc/compiler/gcc11.2.0-aocl</code>.

Alternatively, rerun the configuration file (same as above) and specify 2 values during configuration.

Note that by default, there is no command for 'MPPproperties'. The user has to modify their <code>~/.bashrc</code> file to set the alias and 'settings' file for executable flags. An example:

# in ~/.bashrc
alias MPPprop23="/rds/general/project/cmsg/live/share/HPC-job-submission/Imperial-HPC-Job-Submission/gen_sub -x mppprop -set ${HOME}/etc/runCRYSTAL23/settings"
# in settings, keep the column width
mppprop mpiexec -np ${V_TPROC} MPPproperties Massive parallel properties calculation, OMP

=== HX1 version ===
* Compiling Env : EasyBuild foss/2023a, OpenMP
* Note: MP2 (CRYSCOR and CRYSTAL2) is currently not available.

Executables are saved in <code>/rds/general/project/cmsg/live/share/CRYSTAL/23v1/Linux-EBFOSS2023a-HX1-ompi</code>. To setup the running environment on HX1, if the user has a copy of 'settings' file, they need to modify the following keywords:

* '''EXEDIR''' : No default. The directory where the user put their executables in.
* '''MPIDIR''' : No default, but please use <code>module load foss/2023a</code> unless a self-compiled version is used.

Alternatively, run the configuration file (The one with '-HX1' in the same directory of Github repo) and specify 2 values during configuration. Other options are consistent with CRYSTAL on CX1.

== CRYSTAL17 v2 (GNU)==
Not hosted on RDS-CMSG. Default for CX1. Not available for HX1.

=== CX1 version ===
* Compilor : gcc/6.2.0
* MPI : mpich/3.4.3

'''For job submission scripts'''

The configuration and usage are identical to CRYSTAL23. Please refer to the previous section and substitute '23' with '17'. Also, please be noted that CRYSCOR (MP2 calculation) is not available for CRYSTAL17 and 'MPP' as a massive parallel strategy is limited to 'crystal' calculations, i.e., only MPPcrystal is released.

== Quantum Espresso v7.2 (GNU) ==
Job submission file configured for CX1 and HX1, but only CX1 executable is shared as these executables are highly dependent on environment.

=== CX1 version ===
* Compiling Env : EasyBuild foss2022a, OpenMP
* libxc: Yes, version 5.1.2
* hdf5: No

Pseudopotential files are saved in <code>/rds/general/project/cmsg/live/etc/QE_PseudoP</code>.

''' Job submission script configuration '''

<code>
~$ bash /rds/general/project/cmsg/live/share/HPC-job-submission/Imperial-HPC-Job-Submission/QE7/config_QE7.sh
</code>

The procedure is the same as CRYSTAL. Type enter for default setups, which typically works fine. Please note that moving files between temporal and home directories are disabled as QE has built-in file management system.

''' Job submission script commands '''

{| class="wikitable" style="margin:auto"
|+ List of available commands
|-
! Command !! Definition
|-
| PWqe7 || Generate pw.x job submission files
|-
| PHqe7 || Generate ph.x job submission files
|-
| CPqe7 || Generate cp.x job submission files
|-
| PPqe7 || Generate pp.x job submission files
|-
| Xqe7 || Generate job submission files for user-defined executables
|-
| SETqe7 || Print the local 'settings' file
|-
| HELPqe7 || Print the instructions of commands
|}

== GULP v6.1.2 (GNU) ==
Default for CX1. Not available for HX1.

=== CX1 version ===
* Compiling Env : EasyBuild foss2022a
* OpenKIM: Yes, version 2.3.0
* PLUMED: Yes, version 2.9.0
* ALAMODE: Yes, git repo copied into <code>app/</code>

GULP force field libraries are saved in <code>/rds/general/project/cmsg/live/etc/GULP_Libraries</code>. Executable name: 'gulp-mpi'.

'''OpenKIM'''

KIM force field <ref>https://openkim.org/</ref> are saved in <code>/rds/general/project/cmsg/live/etc/KIM_Models</code>. To run OpenKIM executable and download new models, run:

<code>
~$ module load /rds/general/project/cmsg/live/etc/modulefiles/OpenKIM/2.3.0-foss
</code>

'''Job submission script configuration'''

<code>
~$ bash /rds/general/project/cmsg/live/share/HPC-job-submission/Imperial-HPC-Job-Submission/GULP6/config_GULP6.sh
</code>

Type enter for default setups, which typically works fine.

'''Job submission script commands'''

{| class="wikitable" style="margin:auto"
|+ List of available commands
|-
! Command !! Definition
|-
| Pglp6 || Generate job submission files for 'gulp-mpi'
|-
| Xglp6 || Generate job submission files for user defined executable (by default nothing else, so not useful).
|-
| SETglp6 || Print the local 'settings' file
|-
| HELPglp6|| Print the instructions of commands
|}

== ONETEP v6.1.9.2 (GNU) ==
Default for CX1. Not available for HX1. Job submission script not available.

=== CX1 version ===
* Compiling Env : EasyBuild foss2022a, OpenMP, FFTW, Scalapack
* Libxc: Yes, version 5.1.2

Executable path is <code>/rds/general/project/cmsg/live/app/ONETEP/6.1.2.2__foss2022a/bin</code>. Pseudopotential files are saved in <code>/rds/general/project/cmsg/live/etc/ONETEP_PseudoP</code>.

==References==
<references>

CMSG disk and Shared Software on CX1

2023-11-30T23:40:45Z

Hz1420:

General information of shared software available for CMSG group members on Imperial CX1 is summarized in this page. Software listed below, unless stated otherwise, is hosted under 'CMSG' project of Imperial Research Data Store (RDS) <ref>https://www.imperial.ac.uk/admin-services/ict/self-service/research-support/rcs/service-offering/rds/</ref> (referred as RDS-CMSG below), which is accessible for both login and computational nodes of Imperial HPC CX1. For Imperial HPC HX1, by the time this page is upated (Nov. 2023), it is not accessible there. Only login node can visit RDS-CMSG disk after the pilot phase <ref>https://wiki.imperial.ac.uk/display/HPC/HX1+Cluster</ref>. Therefore software on RDS-CMSG are classified into 2 catagories:

# For CX1, use <code>/rds/general/project/cmsg/live/app/</code> (executables, libraries and headers) and <code>/rds/general/project/cmsg/live/etc/</code> (other files, such as pseudopotential). Executable can be called directly from the user's home directory.
# For HX1, use <code>/rds/general/project/cmsg/live/share/</code>. Executables must be downloaded and uploaded to HX1. It is user's responsibility to set up an effective running environment according to 'readme' files (if there is one) saved in the same, or upper, directory.

''Please note that accessibility to CMSG disk is granted for CMSG group members only. If you need to access it, please contact group PI [[Contributors | Prof. Nicholas Harrison]].''

== CRYSTAL23 v1 (Intel) ==
The current default for CX1. Not available for HX1.

=== CX1 version ===
* Compiling Env : EasyBuild Intel 2023a, OpenMP
* Note: MPPproperties and MP2 (CRYSCOR and CRYSTAL2) are currently not available, which involves separate code packages (DMat2 and CRYSCOR). For MPPproperties please read the GNU version below.

The general job submission script for Imperial HPC, developed by the author himself, is used here.

''Thanks to Mr. K Tallat-Kelpsa for testing''

''' Job submission script configuration '''

The source code (written in bash) is available on group [https://github.com/cmsg-icl/HPC-job-submission/tree/main/Imperial-HPC-Job-Submission GitHub repo], though it is not needed in practice. Useful for developing new features. On CX1, use the following command to configure your local settings:

<code>
~$ bash /rds/general/project/cmsg/live/share/HPC-job-submission/Imperial-HPC-Job-Submission/CRYSTAL23/config_CRYSTAL23.sh
</code>

Then follow the instructions on the screen. Typically the default values are good. To use default vaules, press enter. After configuration is finished, the information is stored in a 'settings' file which is by default saved as <code>${HOME}/etc/runCRYSTAL23/settings</code>. It functions as a dictionary for job sumbmission scripts to refer to. It is also editable according to the user's needs.

Use the following command to activate alias commands:

<code>
~$ source ~/.bashrc
</code>

Due to the settings of CX1, the <code>source</code> command should be executed everytime you login. To aviod this, try the following commands:

~$ cat << EOF >> ~/.bash_profile
if test -f ~/.bashrc; then
source ~/.bashrc
fi
EOF

''' Job submission script commands '''

{| class="wikitable" style="margin:auto"
|+ List of available commands
|-
! Command !! Definition
|-
| Pcrys23 || Generate parallel OMP CRYSTAL23 job submission files
|-
| MPPcrys23 || Generate massive parallel OMP CRYSTAL23 job submission files
|-
| Pprop23 || Generate parallel OMP CRYSTAL23 properties job submission files
|-
| Scrys23 || Generate serial OMP CRYSTAL23 job submission files
|-
| Sprop23 || Generate serial OMP CRYSTAL23 properties job submission files
|-
| Xcrys23 || User-defined executables and multiple jobs (see below for code examples)
|-
| SETcrys23 || Print the local 'settings' file
|-
| HELPcrys23 || Print the instructions of commands
|}

''' Examples '''

To generate a qsub file for a parallel crystal23 job on 'mgo.d12', the following command can be used:

<code>
~$ Pcrys23 -in mgo.d12 -wt 01:00 -nd 1
</code>

That generates a 'mgo.qsub' file requesting for 1 node and the maximum running time is 1 hour. Similarly, after the job is done, using the following command can run a parallel properties calculation based on 'mgo-band.d3' and data from the previous 'mgo' SCF calculation on 12 CPUs:

<code>
~$ Pprop23 -in mgo-band.d3 -nc 12 -wt 00:30 -ref mgo
</code>

For detailed instructions and testing cases, please refer to the [https://github.com/cmsg-icl/HPC-job-submission/tree/main/Imperial-HPC-Job-Submission documentation].

== CRYSTAL23 v1 (GNU) ==
Compared to Intel version above, GNU version is slightly slower to allow for compatibility with MPPproperties. Available for CX1 and HX1.

=== CX1 version ===
* Compiling Env : gcc11.2.0, aocl4.0, mpich4.0.2, OpenMP
* Note: MP2 (CRYSCOR and CRYSTAL2) is currently not available.

The same job submission script is used. However, GNU version is not the default option for CX1. To enable GNU version, if there is a 'settings' file, change the following parameters:

* '''EXEDIR''' : <code>module load /rds/general/project/cmsg/live/etc/modulefiles/CRYSTAL/23v1-gcc</code>
* '''MPIDIR''' : <code>module load /rds/general/project/cmsg/live/etc/compiler/gcc11.2.0-aocl</code>.

Alternatively, rerun the configuration file (same as above) and specify 2 values during configuration.

Note that by default, there is no command for 'MPPproperties'. The user has to modify their <code>~/.bashrc</code> file to set the alias and 'settings' file for executable flags. An example:

# in ~/.bashrc
alias MPPprop23="/rds/general/project/cmsg/live/share/HPC-job-submission/Imperial-HPC-Job-Submission/gen_sub -x mppprop -set ${HOME}/etc/runCRYSTAL23/settings"
# in settings, keep the column width
mppprop mpiexec -np ${V_TPROC} MPPproperties Massive parallel properties calculation, OMP

=== HX1 version ===
* Compiling Env : EasyBuild foss/2023a, OpenMP
* Note: MP2 (CRYSCOR and CRYSTAL2) is currently not available.

Executables are saved in <code>/rds/general/project/cmsg/live/share/CRYSTAL/23v1/Linux-EBFOSS2023a-HX1-ompi</code>. To setup the running environment on HX1, if the user has a copy of 'settings' file, they need to modify the following keywords:

* '''EXEDIR''' : No default. The directory where the user put their executables in.
* '''MPIDIR''' : No default, but please use <code>module load foss/2023a</code> unless a self-compiled version is used.

Alternatively, run the configuration file (The one with '-HX1' in the same directory of Github repo) and specify 2 values during configuration. Other options are consistent with CRYSTAL on CX1.

== CRYSTAL17 v2 (GNU)==
Not hosted on RDS-CMSG. Default for CX1. Not available for HX1.

=== CX1 version ===
* Compilor : gcc/6.2.0
* MPI : mpich/3.4.3

'''For job submission scripts'''

The configuration and usage are identical to CRYSTAL23. Please refer to the previous section and substitute '23' with '17'. Also, please be noted that CRYSCOR (MP2 calculation) is not available for CRYSTAL17 and 'MPP' as a massive parallel strategy is limited to 'crystal' calculations, i.e., only MPPcrystal is released.

== Quantum Espresso v7.2 (GNU) ==
Job submission file configured for CX1 and HX1, but only CX1 executable is shared as these executables are highly dependent on environment.

=== CX1 version ===
* Compiling Env : EasyBuild foss2022a, OpenMP
* libxc: Yes, version 5.1.2
* hdf5: No

Pseudopotential files are saved in <code>/rds/general/project/cmsg/live/etc/QE_PseudoP</code>.

''' Job submission script configuration '''

<code>
~$ bash /rds/general/project/cmsg/live/share/HPC-job-submission/Imperial-HPC-Job-Submission/QE7/config_QE7.sh
</code>

The procedure is the same as CRYSTAL. Type enter for default setups, which typically works fine. Please note that moving files between temporal and home directories are disabled as QE has built-in file management system.

''' Job submission script commands '''

{| class="wikitable" style="margin:auto"
|+ List of available commands
|-
! Command !! Definition
|-
| PWqe7 || Generate pw.x job submission files
|-
| PHqe7 || Generate ph.x job submission files
|-
| CPqe7 || Generate cp.x job submission files
|-
| PPqe7 || Generate pp.x job submission files
|-
| Xqe7 || Generate job submission files for user-defined executables
|-
| SETqe7 || Print the local 'settings' file
|-
| HELPqe7 || Print the instructions of commands
|}

== GULP v6.1.2 (GNU) ==
Default for CX1. Not available for HX1.

=== CX1 version ===
* Compiling Env : EasyBuild foss2022a
* OpenKIM: Yes, version 2.3.0
* PLUMED: Yes, version 2.9.0
* ALAMODE: Yes, git repo copied into <code>app/</code>

GULP force field libraries are saved in <code>/rds/general/project/cmsg/live/etc/GULP_Libraries</code>. Executable name: 'gulp-mpi'.

'''OpenKIM'''

KIM force field <ref>https://openkim.org/</ref> are saved in <code>/rds/general/project/cmsg/live/etc/KIM_Models</code>. To run OpenKIM executable and download new models, run:

<code>
~$ module load /rds/general/project/cmsg/live/etc/modulefiles/OpenKIM/2.3.0-foss
</code>

'''Job submission script configuration'''

<code>
~$ bash /rds/general/project/cmsg/live/share/HPC-job-submission/Imperial-HPC-Job-Submission/GULP6/config_GULP6.sh
</code>

Type enter for default setups, which typically works fine.

'''Job submission script commands'''

{| class="wikitable" style="margin:auto"
|+ List of available commands
|-
! Command !! Definition
|-
| Pglp6 || Generate job submission files for 'gulp-mpi'
|-
| Xglp6 || Generate job submission files for user defined executable (by default nothing else, so not useful).
|-
| SETglp6 || Print the local 'settings' file
|-
| HELPglp6|| Print the instructions of commands
|}

== ONETEP v6.1.9.2 (GNU) ==
Default for CX1. Not available for HX1. Job submission script not available.

=== CX1 version ===
* Compiling Env : EasyBuild foss2022a, OpenMP, FFTW, Scalapack
* Libxc: Yes, version 5.1.2

Executable path is <code>/rds/general/project/cmsg/live/app/ONETEP/6.1.2.2__foss2022a/bin</code>. Pseudopotential files are saved in <code>/rds/general/project/cmsg/live/etc/ONETEP_PseudoP</code>.

==References==
<references>

CMSG disk and Shared Software on CX1

2023-11-30T22:52:19Z

Hz1420: /* HX1 version */

General information of shared software available for CMSG group members on Imperial CX1 is summarized in this page. Software listed below, unless stated otherwise, is hosted under 'CMSG' project of Imperial Research Data Store (RDS) <ref>https://www.imperial.ac.uk/admin-services/ict/self-service/research-support/rcs/service-offering/rds/</ref> (referred as RDS-CMSG below), which is accessible for both login and computational nodes of Imperial HPC CX1. For Imperial HPC HX1, by the time this page is upated (Nov. 2023), it is not accessible there. Only login node can visit RDS-CMSG disk after the pilot phase <ref>https://wiki.imperial.ac.uk/display/HPC/HX1+Cluster</ref>. Therefore software on RDS-CMSG are classified into 2 catagories:

# For CX1, use <code>/rds/general/project/cmsg/live/app/</code> (executables, libraries and headers) and <code>/rds/general/project/cmsg/live/etc/</code> (other files, such as pseudopotential). Executable can be called directly from the user's home directory.
# For HX1, use <code>/rds/general/project/cmsg/live/share/</code>. Executables must be downloaded and uploaded to HX1. It is user's responsibility to set up an effective running environment according to 'readme' files (if there is one) saved in the same, or upper, directory.

''Please note that accessibility to CMSG disk is granted for CMSG group members only. If you need to access it, please contact group PI [[Contributors | Prof. Nicholas Harrison]].''

== CRYSTAL23 v1 (Intel) ==
The current default for CX1. Not available for HX1.

=== CX1 version ===
* Compiling Env : EasyBuild Intel 2023a
* Note: MPPproperties and MP2 (CRYSCOR and CRYSTAL2) are currently not available, which involves separate code packages (DMat2 and CRYSCOR). For MPPproperties please read the GNU version below.

The general job submission script for Imperial HPC, developed by the author himself, is used here.

''Thanks to Mr. K Tallat-Kelpsa for testing''

''' Job submission script configuration '''

The source code (written in bash) is available on group [https://github.com/cmsg-icl/HPC-job-submission/tree/main/Imperial-HPC-Job-Submission GitHub repo], though it is not needed in practice. Useful for developing new features. On CX1, use the following command to configure your local settings:

<code>
~$ bash /rds/general/project/cmsg/live/share/HPC-job-submission/Imperial-HPC-Job-Submission/CRYSTAL23/config_CRYSTAL23.sh
</code>

Then follow the instructions on the screen. Typically the default values are good. To use default vaules, press enter. After configuration is finished, the information is stored in a 'settings' file which is by default saved as <code>${HOME}/etc/runCRYSTAL23/settings</code>. It functions as a dictionary for job sumbmission scripts to refer to. It is also editable according to the user's needs.

Use the following command to activate alias commands:

<code>
~$ source ~/.bashrc
</code>

Due to the settings of CX1, the <code>source</code> command should be executed everytime you login. To aviod this, try the following commands:

~$ cat << EOF >> ~/.bash_profile
if test -f ~/.bashrc; then
source ~/.bashrc
fi
EOF

''' Job submission script commands '''

{| class="wikitable" style="margin:auto"
|+ List of available commands
|-
! Command !! Definition
|-
| Pcrys23 || Generate parallel OMP CRYSTAL23 job submission files
|-
| MPPcrys23 || Generate massive parallel OMP CRYSTAL23 job submission files
|-
| Pprop23 || Generate parallel OMP CRYSTAL23 properties job submission files
|-
| Scrys23 || Generate serial OMP CRYSTAL23 job submission files
|-
| Sprop23 || Generate serial OMP CRYSTAL23 properties job submission files
|-
| Xcrys23 || User-defined executables and multiple jobs (see below for code examples)
|-
| SETcrys23 || Print the local 'settings' file
|-
| HELPcrys23 || Print the instructions of commands
|}

''' Examples '''

To generate a qsub file for a parallel crystal23 job on 'mgo.d12', the following command can be used:

<code>
~$ Pcrys23 -in mgo.d12 -wt 01:00 -nd 1
</code>

That generates a 'mgo.qsub' file requesting for 1 node and the maximum running time is 1 hour. Similarly, after the job is done, using the following command can run a parallel properties calculation based on 'mgo-band.d3' and data from the previous 'mgo' SCF calculation on 12 CPUs:

<code>
~$ Pprop23 -in mgo-band.d3 -nc 12 -wt 00:30 -ref mgo
</code>

For detailed instructions and testing cases, please refer to the [https://github.com/cmsg-icl/HPC-job-submission/tree/main/Imperial-HPC-Job-Submission documentation].

== CRYSTAL23 v1 (GNU) ==
Compared to Intel version above, GNU version is slightly slower to allow for compatibility with MPPproperties. Available for CX1 and HX1.

=== CX1 version ===
* Compiling Env : gcc11.2.0, aocl4.0, mpich4.0.2
* Note: MP2 (CRYSCOR and CRYSTAL2) is currently not available.

The same job submission script is used. However, GNU version is not the default option for CX1. To enable GNU version, if there is a 'settings' file, change the following parameters:

* '''EXEDIR''' : <code>module load /rds/general/project/cmsg/live/etc/modulefiles/CRYSTAL/23v1-gcc</code>
* '''MPIDIR''' : <code>module load /rds/general/project/cmsg/live/etc/compiler/gcc11.2.0-aocl</code>.

Alternatively, rerun the configuration file (same as above) and specify 2 values during configuration.

Note that by default, there is no command for 'MPPproperties'. The user has to modify their <code>~/.bashrc</code> file to set the alias and 'settings' file for executable flags. An example:

# in ~/.bashrc
alias MPPprop23="/rds/general/project/cmsg/live/share/HPC-job-submission/Imperial-HPC-Job-Submission/gen_sub -x mppprop -set ${HOME}/etc/runCRYSTAL23/settings"
# in settings, keep the column width
mppprop mpiexec -np ${V_TPROC} MPPproperties Massive parallel properties calculation, OMP

=== HX1 version ===
* Compiling Env : EasyBuild foss/2023a
* Note: MP2 (CRYSCOR and CRYSTAL2) is currently not available.

Executables are saved in <code>/rds/general/project/cmsg/live/share/CRYSTAL/23v1/Linux-EBFOSS2023a-HX1-ompi</code>. To setup the running environment on HX1, if the user has a copy of 'settings' file, they need to modify the following keywords:

* '''EXEDIR''' : No default. The directory where the user put their executables in.
* '''MPIDIR''' : No default, but please use <code>module load foss/2023a</code> unless a self-compiled version is used.

Alternatively, run the configuration file (The one with '-HX1' in the same directory of Github repo) and specify 2 values during configuration. Other options are consistent with CRYSTAL on CX1.

== CRYSTAL17 v2 (GNU)==
Not hosted on RDS-CMSG. Default for CX1. Not available for HX1.

=== CX1 version ===
* Compilor : gcc/6.2.0
* MPI : mpich/3.4.3

'''For job submission scripts'''

The configuration and usage are identical to CRYSTAL23. Please refer to the previous section and substitute '23' with '17'. Also, please be noted that CRYSCOR (MP2 calculation) is not available for CRYSTAL17 and 'MPP' as a massive parallel strategy is limited to 'crystal' calculations, i.e., only MPPcrystal is released.

== Quantum Espresso v7.2 (GNU) ==
Job submission file configured for CX1 and HX1, but only CX1 executable is shared as these executables are highly dependent on environment.

=== CX1 version ===
* Compiling Env : EasyBuild foss2022a
* libxc: Yes, version 5.1.2
* hdf5: No

Pseudopotential files are saved in <code>/rds/general/project/cmsg/live/etc/QE_PseudoP</code>.

''' Job submission script configuration '''

<code>
~$ bash /rds/general/project/cmsg/live/share/HPC-job-submission/Imperial-HPC-Job-Submission/QE7/config_QE7.sh
</code>

The procedure is the same as CRYSTAL. Type enter for default setups, which typically works fine. Please note that moving files between temporal and home directories are disabled as QE has built-in file management system.

''' Job submission script commands '''

{| class="wikitable" style="margin:auto"
|+ List of available commands
|-
! Command !! Definition
|-
| PWqe7 || Generate pw.x job submission files
|-
| PHqe7 || Generate ph.x job submission files
|-
| CPqe7 || Generate cp.x job submission files
|-
| PPqe7 || Generate pp.x job submission files
|-
| Xqe7 || Generate job submission files for user-defined executables
|-
| SETqe7 || Print the local 'settings' file
|-
| HELPqe7 || Print the instructions of commands
|}

== GULP v6.1.2 (GNU) ==
Default for CX1. Not available for HX1.

=== CX1 version ===
* Compiling Env : EasyBuild foss2022a
* OpenKIM: Yes, version 2.3.0
* PLUMED: Yes, version 2.9.0
* ALAMODE: Yes, git repo copied into <code>app/</code>

GULP force field libraries are saved in <code>/rds/general/project/cmsg/live/etc/GULP_Libraries</code>. Executable name: 'gulp-mpi'.

'''OpenKIM'''

KIM force field <ref>https://openkim.org/</ref> are saved in <code>/rds/general/project/cmsg/live/etc/KIM_Models</code>. To run OpenKIM executable and download new models, run:

<code>
~$ module load /rds/general/project/cmsg/live/etc/modulefiles/OpenKIM/2.3.0-foss
</code>

'''Job submission script configuration'''

<code>
~$ bash /rds/general/project/cmsg/live/share/HPC-job-submission/Imperial-HPC-Job-Submission/GULP6/config_GULP6.sh
</code>

Type enter for default setups, which typically works fine.

'''Job submission script commands'''

{| class="wikitable" style="margin:auto"
|+ List of available commands
|-
! Command !! Definition
|-
| Pglp6 || Generate job submission files for 'gulp-mpi'
|-
| Xglp6 || Generate job submission files for user defined executable (by default nothing else, so not useful).
|-
| SETglp6 || Print the local 'settings' file
|-
| HELPglp6|| Print the instructions of commands
|}

==References==
<references>

CMSG disk and Shared Software on CX1

2023-11-30T22:51:08Z

Hz1420: /* HX1 version */

General information of shared software available for CMSG group members on Imperial CX1 is summarized in this page. Software listed below, unless stated otherwise, is hosted under 'CMSG' project of Imperial Research Data Store (RDS) <ref>https://www.imperial.ac.uk/admin-services/ict/self-service/research-support/rcs/service-offering/rds/</ref> (referred as RDS-CMSG below), which is accessible for both login and computational nodes of Imperial HPC CX1. For Imperial HPC HX1, by the time this page is upated (Nov. 2023), it is not accessible there. Only login node can visit RDS-CMSG disk after the pilot phase <ref>https://wiki.imperial.ac.uk/display/HPC/HX1+Cluster</ref>. Therefore software on RDS-CMSG are classified into 2 catagories:

# For CX1, use <code>/rds/general/project/cmsg/live/app/</code> (executables, libraries and headers) and <code>/rds/general/project/cmsg/live/etc/</code> (other files, such as pseudopotential). Executable can be called directly from the user's home directory.
# For HX1, use <code>/rds/general/project/cmsg/live/share/</code>. Executables must be downloaded and uploaded to HX1. It is user's responsibility to set up an effective running environment according to 'readme' files (if there is one) saved in the same, or upper, directory.

''Please note that accessibility to CMSG disk is granted for CMSG group members only. If you need to access it, please contact group PI [[Contributors | Prof. Nicholas Harrison]].''

== CRYSTAL23 v1 (Intel) ==
The current default for CX1. Not available for HX1.

=== CX1 version ===
* Compiling Env : EasyBuild Intel 2023a
* Note: MPPproperties and MP2 (CRYSCOR and CRYSTAL2) are currently not available, which involves separate code packages (DMat2 and CRYSCOR). For MPPproperties please read the GNU version below.

The general job submission script for Imperial HPC, developed by the author himself, is used here.

''Thanks to Mr. K Tallat-Kelpsa for testing''

''' Job submission script configuration '''

The source code (written in bash) is available on group [https://github.com/cmsg-icl/HPC-job-submission/tree/main/Imperial-HPC-Job-Submission GitHub repo], though it is not needed in practice. Useful for developing new features. On CX1, use the following command to configure your local settings:

<code>
~$ bash /rds/general/project/cmsg/live/share/HPC-job-submission/Imperial-HPC-Job-Submission/CRYSTAL23/config_CRYSTAL23.sh
</code>

Then follow the instructions on the screen. Typically the default values are good. To use default vaules, press enter. After configuration is finished, the information is stored in a 'settings' file which is by default saved as <code>${HOME}/etc/runCRYSTAL23/settings</code>. It functions as a dictionary for job sumbmission scripts to refer to. It is also editable according to the user's needs.

Use the following command to activate alias commands:

<code>
~$ source ~/.bashrc
</code>

Due to the settings of CX1, the <code>source</code> command should be executed everytime you login. To aviod this, try the following commands:

~$ cat << EOF >> ~/.bash_profile
if test -f ~/.bashrc; then
source ~/.bashrc
fi
EOF

''' Job submission script commands '''

{| class="wikitable" style="margin:auto"
|+ List of available commands
|-
! Command !! Definition
|-
| Pcrys23 || Generate parallel OMP CRYSTAL23 job submission files
|-
| MPPcrys23 || Generate massive parallel OMP CRYSTAL23 job submission files
|-
| Pprop23 || Generate parallel OMP CRYSTAL23 properties job submission files
|-
| Scrys23 || Generate serial OMP CRYSTAL23 job submission files
|-
| Sprop23 || Generate serial OMP CRYSTAL23 properties job submission files
|-
| Xcrys23 || User-defined executables and multiple jobs (see below for code examples)
|-
| SETcrys23 || Print the local 'settings' file
|-
| HELPcrys23 || Print the instructions of commands
|}

''' Examples '''

To generate a qsub file for a parallel crystal23 job on 'mgo.d12', the following command can be used:

<code>
~$ Pcrys23 -in mgo.d12 -wt 01:00 -nd 1
</code>

That generates a 'mgo.qsub' file requesting for 1 node and the maximum running time is 1 hour. Similarly, after the job is done, using the following command can run a parallel properties calculation based on 'mgo-band.d3' and data from the previous 'mgo' SCF calculation on 12 CPUs:

<code>
~$ Pprop23 -in mgo-band.d3 -nc 12 -wt 00:30 -ref mgo
</code>

For detailed instructions and testing cases, please refer to the [https://github.com/cmsg-icl/HPC-job-submission/tree/main/Imperial-HPC-Job-Submission documentation].

== CRYSTAL23 v1 (GNU) ==
Compared to Intel version above, GNU version is slightly slower to allow for compatibility with MPPproperties. Available for CX1 and HX1.

=== CX1 version ===
* Compiling Env : gcc11.2.0, aocl4.0, mpich4.0.2
* Note: MP2 (CRYSCOR and CRYSTAL2) is currently not available.

The same job submission script is used. However, GNU version is not the default option for CX1. To enable GNU version, if there is a 'settings' file, change the following parameters:

* '''EXEDIR''' : <code>module load /rds/general/project/cmsg/live/etc/modulefiles/CRYSTAL/23v1-gcc</code>
* '''MPIDIR''' : <code>module load /rds/general/project/cmsg/live/etc/compiler/gcc11.2.0-aocl</code>.

Alternatively, rerun the configuration file (same as above) and specify 2 values during configuration.

Note that by default, there is no command for 'MPPproperties'. The user has to modify their <code>~/.bashrc</code> file to set the alias and 'settings' file for executable flags. An example:

# in ~/.bashrc
alias MPPprop23="/rds/general/project/cmsg/live/share/HPC-job-submission/Imperial-HPC-Job-Submission/gen_sub -x mppprop -set ${HOME}/etc/runCRYSTAL23/settings"
# in settings, keep the column width
mppprop mpiexec -np ${V_TPROC} MPPproperties Massive parallel properties calculation, OMP

=== HX1 version ===
* Compiling Env : EasyBuild foss/2023a
* Note: MP2 (CRYSCOR and CRYSTAL2) is currently not available.

Executables are saved in <code>/rds/general/project/cmsg/live/share/CRYSTAL/23v1/Linux-EBFOSS2023a-HX1-ompi</code>. To setup the running environment on HX1, if the user has a copy of 'settings' file, they need to modify the following keywords:

* '''EXEDIR''' : No default. The directory where the user put their executables in.
* '''MPIDIR''' : <code>module load foss/2023a</code>

Alternatively, run the configuration file (The one with '-HX1' in the same directory of Github repo) and specify 2 values during configuration. Other options are consistent with CRYSTAL on CX1.

== CRYSTAL17 v2 (GNU)==
Not hosted on RDS-CMSG. Default for CX1. Not available for HX1.

=== CX1 version ===
* Compilor : gcc/6.2.0
* MPI : mpich/3.4.3

'''For job submission scripts'''

The configuration and usage are identical to CRYSTAL23. Please refer to the previous section and substitute '23' with '17'. Also, please be noted that CRYSCOR (MP2 calculation) is not available for CRYSTAL17 and 'MPP' as a massive parallel strategy is limited to 'crystal' calculations, i.e., only MPPcrystal is released.

== Quantum Espresso v7.2 (GNU) ==
Job submission file configured for CX1 and HX1, but only CX1 executable is shared as these executables are highly dependent on environment.

=== CX1 version ===
* Compiling Env : EasyBuild foss2022a
* libxc: Yes, version 5.1.2
* hdf5: No

Pseudopotential files are saved in <code>/rds/general/project/cmsg/live/etc/QE_PseudoP</code>.

''' Job submission script configuration '''

<code>
~$ bash /rds/general/project/cmsg/live/share/HPC-job-submission/Imperial-HPC-Job-Submission/QE7/config_QE7.sh
</code>

The procedure is the same as CRYSTAL. Type enter for default setups, which typically works fine. Please note that moving files between temporal and home directories are disabled as QE has built-in file management system.

''' Job submission script commands '''

{| class="wikitable" style="margin:auto"
|+ List of available commands
|-
! Command !! Definition
|-
| PWqe7 || Generate pw.x job submission files
|-
| PHqe7 || Generate ph.x job submission files
|-
| CPqe7 || Generate cp.x job submission files
|-
| PPqe7 || Generate pp.x job submission files
|-
| Xqe7 || Generate job submission files for user-defined executables
|-
| SETqe7 || Print the local 'settings' file
|-
| HELPqe7 || Print the instructions of commands
|}

== GULP v6.1.2 (GNU) ==
Default for CX1. Not available for HX1.

=== CX1 version ===
* Compiling Env : EasyBuild foss2022a
* OpenKIM: Yes, version 2.3.0
* PLUMED: Yes, version 2.9.0
* ALAMODE: Yes, git repo copied into <code>app/</code>

GULP force field libraries are saved in <code>/rds/general/project/cmsg/live/etc/GULP_Libraries</code>. Executable name: 'gulp-mpi'.

'''OpenKIM'''

KIM force field <ref>https://openkim.org/</ref> are saved in <code>/rds/general/project/cmsg/live/etc/KIM_Models</code>. To run OpenKIM executable and download new models, run:

<code>
~$ module load /rds/general/project/cmsg/live/etc/modulefiles/OpenKIM/2.3.0-foss
</code>

'''Job submission script configuration'''

<code>
~$ bash /rds/general/project/cmsg/live/share/HPC-job-submission/Imperial-HPC-Job-Submission/GULP6/config_GULP6.sh
</code>

Type enter for default setups, which typically works fine.

'''Job submission script commands'''

{| class="wikitable" style="margin:auto"
|+ List of available commands
|-
! Command !! Definition
|-
| Pglp6 || Generate job submission files for 'gulp-mpi'
|-
| Xglp6 || Generate job submission files for user defined executable (by default nothing else, so not useful).
|-
| SETglp6 || Print the local 'settings' file
|-
| HELPglp6|| Print the instructions of commands
|}

==References==
<references>

CMSG disk and Shared Software on CX1

2023-11-30T22:50:51Z

Hz1420: /* HX1 version */

General information of shared software available for CMSG group members on Imperial CX1 is summarized in this page. Software listed below, unless stated otherwise, is hosted under 'CMSG' project of Imperial Research Data Store (RDS) <ref>https://www.imperial.ac.uk/admin-services/ict/self-service/research-support/rcs/service-offering/rds/</ref> (referred as RDS-CMSG below), which is accessible for both login and computational nodes of Imperial HPC CX1. For Imperial HPC HX1, by the time this page is upated (Nov. 2023), it is not accessible there. Only login node can visit RDS-CMSG disk after the pilot phase <ref>https://wiki.imperial.ac.uk/display/HPC/HX1+Cluster</ref>. Therefore software on RDS-CMSG are classified into 2 catagories:

# For CX1, use <code>/rds/general/project/cmsg/live/app/</code> (executables, libraries and headers) and <code>/rds/general/project/cmsg/live/etc/</code> (other files, such as pseudopotential). Executable can be called directly from the user's home directory.
# For HX1, use <code>/rds/general/project/cmsg/live/share/</code>. Executables must be downloaded and uploaded to HX1. It is user's responsibility to set up an effective running environment according to 'readme' files (if there is one) saved in the same, or upper, directory.

''Please note that accessibility to CMSG disk is granted for CMSG group members only. If you need to access it, please contact group PI [[Contributors | Prof. Nicholas Harrison]].''

== CRYSTAL23 v1 (Intel) ==
The current default for CX1. Not available for HX1.

=== CX1 version ===
* Compiling Env : EasyBuild Intel 2023a
* Note: MPPproperties and MP2 (CRYSCOR and CRYSTAL2) are currently not available, which involves separate code packages (DMat2 and CRYSCOR). For MPPproperties please read the GNU version below.

The general job submission script for Imperial HPC, developed by the author himself, is used here.

''Thanks to Mr. K Tallat-Kelpsa for testing''

''' Job submission script configuration '''

The source code (written in bash) is available on group [https://github.com/cmsg-icl/HPC-job-submission/tree/main/Imperial-HPC-Job-Submission GitHub repo], though it is not needed in practice. Useful for developing new features. On CX1, use the following command to configure your local settings:

<code>
~$ bash /rds/general/project/cmsg/live/share/HPC-job-submission/Imperial-HPC-Job-Submission/CRYSTAL23/config_CRYSTAL23.sh
</code>

Then follow the instructions on the screen. Typically the default values are good. To use default vaules, press enter. After configuration is finished, the information is stored in a 'settings' file which is by default saved as <code>${HOME}/etc/runCRYSTAL23/settings</code>. It functions as a dictionary for job sumbmission scripts to refer to. It is also editable according to the user's needs.

Use the following command to activate alias commands:

<code>
~$ source ~/.bashrc
</code>

Due to the settings of CX1, the <code>source</code> command should be executed everytime you login. To aviod this, try the following commands:

~$ cat << EOF >> ~/.bash_profile
if test -f ~/.bashrc; then
source ~/.bashrc
fi
EOF

''' Job submission script commands '''

{| class="wikitable" style="margin:auto"
|+ List of available commands
|-
! Command !! Definition
|-
| Pcrys23 || Generate parallel OMP CRYSTAL23 job submission files
|-
| MPPcrys23 || Generate massive parallel OMP CRYSTAL23 job submission files
|-
| Pprop23 || Generate parallel OMP CRYSTAL23 properties job submission files
|-
| Scrys23 || Generate serial OMP CRYSTAL23 job submission files
|-
| Sprop23 || Generate serial OMP CRYSTAL23 properties job submission files
|-
| Xcrys23 || User-defined executables and multiple jobs (see below for code examples)
|-
| SETcrys23 || Print the local 'settings' file
|-
| HELPcrys23 || Print the instructions of commands
|}

''' Examples '''

To generate a qsub file for a parallel crystal23 job on 'mgo.d12', the following command can be used:

<code>
~$ Pcrys23 -in mgo.d12 -wt 01:00 -nd 1
</code>

That generates a 'mgo.qsub' file requesting for 1 node and the maximum running time is 1 hour. Similarly, after the job is done, using the following command can run a parallel properties calculation based on 'mgo-band.d3' and data from the previous 'mgo' SCF calculation on 12 CPUs:

<code>
~$ Pprop23 -in mgo-band.d3 -nc 12 -wt 00:30 -ref mgo
</code>

For detailed instructions and testing cases, please refer to the [https://github.com/cmsg-icl/HPC-job-submission/tree/main/Imperial-HPC-Job-Submission documentation].

== CRYSTAL23 v1 (GNU) ==
Compared to Intel version above, GNU version is slightly slower to allow for compatibility with MPPproperties. Available for CX1 and HX1.

=== CX1 version ===
* Compiling Env : gcc11.2.0, aocl4.0, mpich4.0.2
* Note: MP2 (CRYSCOR and CRYSTAL2) is currently not available.

The same job submission script is used. However, GNU version is not the default option for CX1. To enable GNU version, if there is a 'settings' file, change the following parameters:

* '''EXEDIR''' : <code>module load /rds/general/project/cmsg/live/etc/modulefiles/CRYSTAL/23v1-gcc</code>
* '''MPIDIR''' : <code>module load /rds/general/project/cmsg/live/etc/compiler/gcc11.2.0-aocl</code>.

Alternatively, rerun the configuration file (same as above) and specify 2 values during configuration.

Note that by default, there is no command for 'MPPproperties'. The user has to modify their <code>~/.bashrc</code> file to set the alias and 'settings' file for executable flags. An example:

# in ~/.bashrc
alias MPPprop23="/rds/general/project/cmsg/live/share/HPC-job-submission/Imperial-HPC-Job-Submission/gen_sub -x mppprop -set ${HOME}/etc/runCRYSTAL23/settings"
# in settings, keep the column width
mppprop mpiexec -np ${V_TPROC} MPPproperties Massive parallel properties calculation, OMP

=== HX1 version ===
* Compiling Env : EasyBuild foss/2023a
* Note: MP2 (CRYSCOR and CRYSTAL2) is currently not available.

Saved in <code>/rds/general/project/cmsg/live/share/CRYSTAL/23v1/Linux-EBFOSS2023a-HX1-ompi</code>. To setup the running environment on HX1, if the user has a copy of 'settings' file, they need to modify the following keywords:

* '''EXEDIR''' : No default. The directory where the user put their executables in.
* '''MPIDIR''' : <code>module load foss/2023a</code>

Alternatively, run the configuration file (The one with '-HX1' in the same directory of Github repo) and specify 2 values during configuration. Other options are consistent with CRYSTAL on CX1.

== CRYSTAL17 v2 (GNU)==
Not hosted on RDS-CMSG. Default for CX1. Not available for HX1.

=== CX1 version ===
* Compilor : gcc/6.2.0
* MPI : mpich/3.4.3

'''For job submission scripts'''

The configuration and usage are identical to CRYSTAL23. Please refer to the previous section and substitute '23' with '17'. Also, please be noted that CRYSCOR (MP2 calculation) is not available for CRYSTAL17 and 'MPP' as a massive parallel strategy is limited to 'crystal' calculations, i.e., only MPPcrystal is released.

== Quantum Espresso v7.2 (GNU) ==
Job submission file configured for CX1 and HX1, but only CX1 executable is shared as these executables are highly dependent on environment.

=== CX1 version ===
* Compiling Env : EasyBuild foss2022a
* libxc: Yes, version 5.1.2
* hdf5: No

Pseudopotential files are saved in <code>/rds/general/project/cmsg/live/etc/QE_PseudoP</code>.

''' Job submission script configuration '''

<code>
~$ bash /rds/general/project/cmsg/live/share/HPC-job-submission/Imperial-HPC-Job-Submission/QE7/config_QE7.sh
</code>

The procedure is the same as CRYSTAL. Type enter for default setups, which typically works fine. Please note that moving files between temporal and home directories are disabled as QE has built-in file management system.

''' Job submission script commands '''

{| class="wikitable" style="margin:auto"
|+ List of available commands
|-
! Command !! Definition
|-
| PWqe7 || Generate pw.x job submission files
|-
| PHqe7 || Generate ph.x job submission files
|-
| CPqe7 || Generate cp.x job submission files
|-
| PPqe7 || Generate pp.x job submission files
|-
| Xqe7 || Generate job submission files for user-defined executables
|-
| SETqe7 || Print the local 'settings' file
|-
| HELPqe7 || Print the instructions of commands
|}

== GULP v6.1.2 (GNU) ==
Default for CX1. Not available for HX1.

=== CX1 version ===
* Compiling Env : EasyBuild foss2022a
* OpenKIM: Yes, version 2.3.0
* PLUMED: Yes, version 2.9.0
* ALAMODE: Yes, git repo copied into <code>app/</code>

GULP force field libraries are saved in <code>/rds/general/project/cmsg/live/etc/GULP_Libraries</code>. Executable name: 'gulp-mpi'.

'''OpenKIM'''

KIM force field <ref>https://openkim.org/</ref> are saved in <code>/rds/general/project/cmsg/live/etc/KIM_Models</code>. To run OpenKIM executable and download new models, run:

<code>
~$ module load /rds/general/project/cmsg/live/etc/modulefiles/OpenKIM/2.3.0-foss
</code>

'''Job submission script configuration'''

<code>
~$ bash /rds/general/project/cmsg/live/share/HPC-job-submission/Imperial-HPC-Job-Submission/GULP6/config_GULP6.sh
</code>

Type enter for default setups, which typically works fine.

'''Job submission script commands'''

{| class="wikitable" style="margin:auto"
|+ List of available commands
|-
! Command !! Definition
|-
| Pglp6 || Generate job submission files for 'gulp-mpi'
|-
| Xglp6 || Generate job submission files for user defined executable (by default nothing else, so not useful).
|-
| SETglp6 || Print the local 'settings' file
|-
| HELPglp6|| Print the instructions of commands
|}

==References==
<references>

CMSG disk and Shared Software on CX1

2023-11-30T22:29:38Z

Hz1420: /* CRYSTAL23 v1 (Intel) */

General information of shared software available for CMSG group members on Imperial CX1 is summarized in this page. Software listed below, unless stated otherwise, is hosted under 'CMSG' project of Imperial Research Data Store (RDS) <ref>https://www.imperial.ac.uk/admin-services/ict/self-service/research-support/rcs/service-offering/rds/</ref> (referred as RDS-CMSG below), which is accessible for both login and computational nodes of Imperial HPC CX1. For Imperial HPC HX1, by the time this page is upated (Nov. 2023), it is not accessible there. Only login node can visit RDS-CMSG disk after the pilot phase <ref>https://wiki.imperial.ac.uk/display/HPC/HX1+Cluster</ref>. Therefore software on RDS-CMSG are classified into 2 catagories:

# For CX1, use <code>/rds/general/project/cmsg/live/app/</code> (executables, libraries and headers) and <code>/rds/general/project/cmsg/live/etc/</code> (other files, such as pseudopotential). Executable can be called directly from the user's home directory.
# For HX1, use <code>/rds/general/project/cmsg/live/share/</code>. Executables must be downloaded and uploaded to HX1. It is user's responsibility to set up an effective running environment according to 'readme' files (if there is one) saved in the same, or upper, directory.

''Please note that accessibility to CMSG disk is granted for CMSG group members only. If you need to access it, please contact group PI [[Contributors | Prof. Nicholas Harrison]].''

== CRYSTAL23 v1 (Intel) ==
The current default for CX1. Not available for HX1.

=== CX1 version ===
* Compiling Env : EasyBuild Intel 2023a
* Note: MPPproperties and MP2 (CRYSCOR and CRYSTAL2) are currently not available, which involves separate code packages (DMat2 and CRYSCOR). For MPPproperties please read the GNU version below.

The general job submission script for Imperial HPC, developed by the author himself, is used here.

''Thanks to Mr. K Tallat-Kelpsa for testing''

''' Job submission script configuration '''

The source code (written in bash) is available on group [https://github.com/cmsg-icl/HPC-job-submission/tree/main/Imperial-HPC-Job-Submission GitHub repo], though it is not needed in practice. Useful for developing new features. On CX1, use the following command to configure your local settings:

<code>
~$ bash /rds/general/project/cmsg/live/share/HPC-job-submission/Imperial-HPC-Job-Submission/CRYSTAL23/config_CRYSTAL23.sh
</code>

Then follow the instructions on the screen. Typically the default values are good. To use default vaules, press enter. After configuration is finished, the information is stored in a 'settings' file which is by default saved as <code>${HOME}/etc/runCRYSTAL23/settings</code>. It functions as a dictionary for job sumbmission scripts to refer to. It is also editable according to the user's needs.

Use the following command to activate alias commands:

<code>
~$ source ~/.bashrc
</code>

Due to the settings of CX1, the <code>source</code> command should be executed everytime you login. To aviod this, try the following commands:

~$ cat << EOF >> ~/.bash_profile
if test -f ~/.bashrc; then
source ~/.bashrc
fi
EOF

''' Job submission script commands '''

{| class="wikitable" style="margin:auto"
|+ List of available commands
|-
! Command !! Definition
|-
| Pcrys23 || Generate parallel OMP CRYSTAL23 job submission files
|-
| MPPcrys23 || Generate massive parallel OMP CRYSTAL23 job submission files
|-
| Pprop23 || Generate parallel OMP CRYSTAL23 properties job submission files
|-
| Scrys23 || Generate serial OMP CRYSTAL23 job submission files
|-
| Sprop23 || Generate serial OMP CRYSTAL23 properties job submission files
|-
| Xcrys23 || User-defined executables and multiple jobs (see below for code examples)
|-
| SETcrys23 || Print the local 'settings' file
|-
| HELPcrys23 || Print the instructions of commands
|}

''' Examples '''

To generate a qsub file for a parallel crystal23 job on 'mgo.d12', the following command can be used:

<code>
~$ Pcrys23 -in mgo.d12 -wt 01:00 -nd 1
</code>

That generates a 'mgo.qsub' file requesting for 1 node and the maximum running time is 1 hour. Similarly, after the job is done, using the following command can run a parallel properties calculation based on 'mgo-band.d3' and data from the previous 'mgo' SCF calculation on 12 CPUs:

<code>
~$ Pprop23 -in mgo-band.d3 -nc 12 -wt 00:30 -ref mgo
</code>

For detailed instructions and testing cases, please refer to the [https://github.com/cmsg-icl/HPC-job-submission/tree/main/Imperial-HPC-Job-Submission documentation].

== CRYSTAL23 v1 (GNU) ==
Compared to Intel version above, GNU version is slightly slower to allow for compatibility with MPPproperties. Available for CX1 and HX1.

=== CX1 version ===
* Compiling Env : gcc11.2.0, aocl4.0, mpich4.0.2
* Note: MP2 (CRYSCOR and CRYSTAL2) is currently not available.

The same job submission script is used. However, GNU version is not the default option for CX1. To enable GNU version, if there is a 'settings' file, change the following parameters:

* '''EXEDIR''' : <code>module load /rds/general/project/cmsg/live/etc/modulefiles/CRYSTAL/23v1-gcc</code>
* '''MPIDIR''' : <code>module load /rds/general/project/cmsg/live/etc/compiler/gcc11.2.0-aocl</code>.

Alternatively, rerun the configuration file (same as above) and specify 2 values during configuration.

Note that by default, there is no command for 'MPPproperties'. The user has to modify their <code>~/.bashrc</code> file to set the alias and 'settings' file for executable flags. An example:

# in ~/.bashrc
alias MPPprop23="/rds/general/project/cmsg/live/share/HPC-job-submission/Imperial-HPC-Job-Submission/gen_sub -x mppprop -set ${HOME}/etc/runCRYSTAL23/settings"
# in settings, keep the column width
mppprop mpiexec -np ${V_TPROC} MPPproperties Massive parallel properties calculation, OMP

=== HX1 version ===
* Compiling Env : EasyBuild foss/2023a
* Note: MP2 (CRYSCOR and CRYSTAL2) is currently not available.

* '''EXEDIR''' : No default. The directory where the user put their executables in.
* '''MPIDIR''' : <code>module load foss/2023a</code>

Alternatively, run the configuration file (The one with '-HX1' in the same directory of Github repo) and specify 2 values during configuration. Other options are consistent with CRYSTAL on CX1.

== CRYSTAL17 v2 (GNU)==
Not hosted on RDS-CMSG. Default for CX1. Not available for HX1.

=== CX1 version ===
* Compilor : gcc/6.2.0
* MPI : mpich/3.4.3

'''For job submission scripts'''

The configuration and usage are identical to CRYSTAL23. Please refer to the previous section and substitute '23' with '17'. Also, please be noted that CRYSCOR (MP2 calculation) is not available for CRYSTAL17 and 'MPP' as a massive parallel strategy is limited to 'crystal' calculations, i.e., only MPPcrystal is released.

== Quantum Espresso v7.2 (GNU) ==
Job submission file configured for CX1 and HX1, but only CX1 executable is shared as these executables are highly dependent on environment.

=== CX1 version ===
* Compiling Env : EasyBuild foss2022a
* libxc: Yes, version 5.1.2
* hdf5: No

Pseudopotential files are saved in <code>/rds/general/project/cmsg/live/etc/QE_PseudoP</code>.

''' Job submission script configuration '''

<code>
~$ bash /rds/general/project/cmsg/live/share/HPC-job-submission/Imperial-HPC-Job-Submission/QE7/config_QE7.sh
</code>

The procedure is the same as CRYSTAL. Type enter for default setups, which typically works fine. Please note that moving files between temporal and home directories are disabled as QE has built-in file management system.

''' Job submission script commands '''

{| class="wikitable" style="margin:auto"
|+ List of available commands
|-
! Command !! Definition
|-
| PWqe7 || Generate pw.x job submission files
|-
| PHqe7 || Generate ph.x job submission files
|-
| CPqe7 || Generate cp.x job submission files
|-
| PPqe7 || Generate pp.x job submission files
|-
| Xqe7 || Generate job submission files for user-defined executables
|-
| SETqe7 || Print the local 'settings' file
|-
| HELPqe7 || Print the instructions of commands
|}

== GULP v6.1.2 (GNU) ==
Default for CX1. Not available for HX1.

=== CX1 version ===
* Compiling Env : EasyBuild foss2022a
* OpenKIM: Yes, version 2.3.0
* PLUMED: Yes, version 2.9.0
* ALAMODE: Yes, git repo copied into <code>app/</code>

GULP force field libraries are saved in <code>/rds/general/project/cmsg/live/etc/GULP_Libraries</code>. Executable name: 'gulp-mpi'.

'''OpenKIM'''

KIM force field <ref>https://openkim.org/</ref> are saved in <code>/rds/general/project/cmsg/live/etc/KIM_Models</code>. To run OpenKIM executable and download new models, run:

<code>
~$ module load /rds/general/project/cmsg/live/etc/modulefiles/OpenKIM/2.3.0-foss
</code>

'''Job submission script configuration'''

<code>
~$ bash /rds/general/project/cmsg/live/share/HPC-job-submission/Imperial-HPC-Job-Submission/GULP6/config_GULP6.sh
</code>

Type enter for default setups, which typically works fine.

'''Job submission script commands'''

{| class="wikitable" style="margin:auto"
|+ List of available commands
|-
! Command !! Definition
|-
| Pglp6 || Generate job submission files for 'gulp-mpi'
|-
| Xglp6 || Generate job submission files for user defined executable (by default nothing else, so not useful).
|-
| SETglp6 || Print the local 'settings' file
|-
| HELPglp6|| Print the instructions of commands
|}

==References==
<references>

CMSG disk and Shared Software on CX1

2023-11-30T22:29:05Z

Hz1420: /* CRYSTAL23 v1 (Intel) */

General information of shared software available for CMSG group members on Imperial CX1 is summarized in this page. Software listed below, unless stated otherwise, is hosted under 'CMSG' project of Imperial Research Data Store (RDS) <ref>https://www.imperial.ac.uk/admin-services/ict/self-service/research-support/rcs/service-offering/rds/</ref> (referred as RDS-CMSG below), which is accessible for both login and computational nodes of Imperial HPC CX1. For Imperial HPC HX1, by the time this page is upated (Nov. 2023), it is not accessible there. Only login node can visit RDS-CMSG disk after the pilot phase <ref>https://wiki.imperial.ac.uk/display/HPC/HX1+Cluster</ref>. Therefore software on RDS-CMSG are classified into 2 catagories:

# For CX1, use <code>/rds/general/project/cmsg/live/app/</code> (executables, libraries and headers) and <code>/rds/general/project/cmsg/live/etc/</code> (other files, such as pseudopotential). Executable can be called directly from the user's home directory.
# For HX1, use <code>/rds/general/project/cmsg/live/share/</code>. Executables must be downloaded and uploaded to HX1. It is user's responsibility to set up an effective running environment according to 'readme' files (if there is one) saved in the same, or upper, directory.

''Please note that accessibility to CMSG disk is granted for CMSG group members only. If you need to access it, please contact group PI [[Contributors | Prof. Nicholas Harrison]].''

== CRYSTAL23 v1 (Intel) ==
The current default for CX1. Not available for HX1.

=== CX1 version ===
* Compiling Env : EasyBuild Intel 2023a
* Note: MPPproperties and MP2 (CRYSCOR and CRYSTAL2) are currently not available, which involves separate code packages (DMat2 and CRYSCOR). For MPPproperties please read the GNU version below.

The general job submission script for Imperial HPC, developed by the author himself, is used here.

''Thanks to Mr. K Tallat-Kelpsa for testing''

''' Configuration '''

The source code (written in bash) is available on group [https://github.com/cmsg-icl/HPC-job-submission/tree/main/Imperial-HPC-Job-Submission GitHub repo], though it is not needed in practice. Useful for developing new features. On CX1, use the following command to configure your local settings:

<code>
~$ bash /rds/general/project/cmsg/live/share/HPC-job-submission/Imperial-HPC-Job-Submission/CRYSTAL23/config_CRYSTAL23.sh
</code>

Then follow the instructions on the screen. Typically the default values are good. To use default vaules, press enter. After configuration is finished, the information is stored in a 'settings' file which is by default saved as <code>${HOME}/etc/runCRYSTAL23/settings</code>. It functions as a dictionary for job sumbmission scripts to refer to. It is also editable according to the user's needs.

Use the following command to activate alias commands:

<code>
~$ source ~/.bashrc
</code>

Due to the settings of CX1, the <code>source</code> command should be executed everytime you login. To aviod this, try the following commands:

~$ cat << EOF >> ~/.bash_profile
if test -f ~/.bashrc; then
source ~/.bashrc
fi
EOF

''' Job submission script commands '''

{| class="wikitable" style="margin:auto"
|+ List of available commands
|-
! Command !! Definition
|-
| Pcrys23 || Generate parallel OMP CRYSTAL23 job submission files
|-
| MPPcrys23 || Generate massive parallel OMP CRYSTAL23 job submission files
|-
| Pprop23 || Generate parallel OMP CRYSTAL23 properties job submission files
|-
| Scrys23 || Generate serial OMP CRYSTAL23 job submission files
|-
| Sprop23 || Generate serial OMP CRYSTAL23 properties job submission files
|-
| Xcrys23 || User-defined executables and multiple jobs (see below for code examples)
|-
| SETcrys23 || Print the local 'settings' file
|-
| HELPcrys23 || Print the instructions of commands
|}

''' Examples '''

To generate a qsub file for a parallel crystal23 job on 'mgo.d12', the following command can be used:

<code>
~$ Pcrys23 -in mgo.d12 -wt 01:00 -nd 1
</code>

That generates a 'mgo.qsub' file requesting for 1 node and the maximum running time is 1 hour. Similarly, after the job is done, using the following command can run a parallel properties calculation based on 'mgo-band.d3' and data from the previous 'mgo' SCF calculation on 12 CPUs:

<code>
~$ Pprop23 -in mgo-band.d3 -nc 12 -wt 00:30 -ref mgo
</code>

For detailed instructions and testing cases, please refer to the [https://github.com/cmsg-icl/HPC-job-submission/tree/main/Imperial-HPC-Job-Submission documentation].

== CRYSTAL23 v1 (GNU) ==
Compared to Intel version above, GNU version is slightly slower to allow for compatibility with MPPproperties. Available for CX1 and HX1.

=== CX1 version ===
* Compiling Env : gcc11.2.0, aocl4.0, mpich4.0.2
* Note: MP2 (CRYSCOR and CRYSTAL2) is currently not available.

The same job submission script is used. However, GNU version is not the default option for CX1. To enable GNU version, if there is a 'settings' file, change the following parameters:

* '''EXEDIR''' : <code>module load /rds/general/project/cmsg/live/etc/modulefiles/CRYSTAL/23v1-gcc</code>
* '''MPIDIR''' : <code>module load /rds/general/project/cmsg/live/etc/compiler/gcc11.2.0-aocl</code>.

Alternatively, rerun the configuration file (same as above) and specify 2 values during configuration.

Note that by default, there is no command for 'MPPproperties'. The user has to modify their <code>~/.bashrc</code> file to set the alias and 'settings' file for executable flags. An example:

# in ~/.bashrc
alias MPPprop23="/rds/general/project/cmsg/live/share/HPC-job-submission/Imperial-HPC-Job-Submission/gen_sub -x mppprop -set ${HOME}/etc/runCRYSTAL23/settings"
# in settings, keep the column width
mppprop mpiexec -np ${V_TPROC} MPPproperties Massive parallel properties calculation, OMP

=== HX1 version ===
* Compiling Env : EasyBuild foss/2023a
* Note: MP2 (CRYSCOR and CRYSTAL2) is currently not available.

* '''EXEDIR''' : No default. The directory where the user put their executables in.
* '''MPIDIR''' : <code>module load foss/2023a</code>

Alternatively, run the configuration file (The one with '-HX1' in the same directory of Github repo) and specify 2 values during configuration. Other options are consistent with CRYSTAL on CX1.

== CRYSTAL17 v2 (GNU)==
Not hosted on RDS-CMSG. Default for CX1. Not available for HX1.

=== CX1 version ===
* Compilor : gcc/6.2.0
* MPI : mpich/3.4.3

'''For job submission scripts'''

The configuration and usage are identical to CRYSTAL23. Please refer to the previous section and substitute '23' with '17'. Also, please be noted that CRYSCOR (MP2 calculation) is not available for CRYSTAL17 and 'MPP' as a massive parallel strategy is limited to 'crystal' calculations, i.e., only MPPcrystal is released.

== Quantum Espresso v7.2 (GNU) ==
Job submission file configured for CX1 and HX1, but only CX1 executable is shared as these executables are highly dependent on environment.

=== CX1 version ===
* Compiling Env : EasyBuild foss2022a
* libxc: Yes, version 5.1.2
* hdf5: No

Pseudopotential files are saved in <code>/rds/general/project/cmsg/live/etc/QE_PseudoP</code>.

''' Job submission script configuration '''

<code>
~$ bash /rds/general/project/cmsg/live/share/HPC-job-submission/Imperial-HPC-Job-Submission/QE7/config_QE7.sh
</code>

The procedure is the same as CRYSTAL. Type enter for default setups, which typically works fine. Please note that moving files between temporal and home directories are disabled as QE has built-in file management system.

''' Job submission script commands '''

{| class="wikitable" style="margin:auto"
|+ List of available commands
|-
! Command !! Definition
|-
| PWqe7 || Generate pw.x job submission files
|-
| PHqe7 || Generate ph.x job submission files
|-
| CPqe7 || Generate cp.x job submission files
|-
| PPqe7 || Generate pp.x job submission files
|-
| Xqe7 || Generate job submission files for user-defined executables
|-
| SETqe7 || Print the local 'settings' file
|-
| HELPqe7 || Print the instructions of commands
|}

== GULP v6.1.2 (GNU) ==
Default for CX1. Not available for HX1.

=== CX1 version ===
* Compiling Env : EasyBuild foss2022a
* OpenKIM: Yes, version 2.3.0
* PLUMED: Yes, version 2.9.0
* ALAMODE: Yes, git repo copied into <code>app/</code>

GULP force field libraries are saved in <code>/rds/general/project/cmsg/live/etc/GULP_Libraries</code>. Executable name: 'gulp-mpi'.

'''OpenKIM'''

KIM force field <ref>https://openkim.org/</ref> are saved in <code>/rds/general/project/cmsg/live/etc/KIM_Models</code>. To run OpenKIM executable and download new models, run:

<code>
~$ module load /rds/general/project/cmsg/live/etc/modulefiles/OpenKIM/2.3.0-foss
</code>

'''Job submission script configuration'''

<code>
~$ bash /rds/general/project/cmsg/live/share/HPC-job-submission/Imperial-HPC-Job-Submission/GULP6/config_GULP6.sh
</code>

Type enter for default setups, which typically works fine.

'''Job submission script commands'''

{| class="wikitable" style="margin:auto"
|+ List of available commands
|-
! Command !! Definition
|-
| Pglp6 || Generate job submission files for 'gulp-mpi'
|-
| Xglp6 || Generate job submission files for user defined executable (by default nothing else, so not useful).
|-
| SETglp6 || Print the local 'settings' file
|-
| HELPglp6|| Print the instructions of commands
|}

==References==
<references>

CMSG disk and Shared Software on CX1

2023-11-30T22:24:46Z

Hz1420: /* CRYSTAL23 v1 (Intel) */

General information of shared software available for CMSG group members on Imperial CX1 is summarized in this page. Software listed below, unless stated otherwise, is hosted under 'CMSG' project of Imperial Research Data Store (RDS) <ref>https://www.imperial.ac.uk/admin-services/ict/self-service/research-support/rcs/service-offering/rds/</ref> (referred as RDS-CMSG below), which is accessible for both login and computational nodes of Imperial HPC CX1. For Imperial HPC HX1, by the time this page is upated (Nov. 2023), it is not accessible there. Only login node can visit RDS-CMSG disk after the pilot phase <ref>https://wiki.imperial.ac.uk/display/HPC/HX1+Cluster</ref>. Therefore software on RDS-CMSG are classified into 2 catagories:

# For CX1, use <code>/rds/general/project/cmsg/live/app/</code> (executables, libraries and headers) and <code>/rds/general/project/cmsg/live/etc/</code> (other files, such as pseudopotential). Executable can be called directly from the user's home directory.
# For HX1, use <code>/rds/general/project/cmsg/live/share/</code>. Executables must be downloaded and uploaded to HX1. It is user's responsibility to set up an effective running environment according to 'readme' files (if there is one) saved in the same, or upper, directory.

''Please note that accessibility to CMSG disk is granted for CMSG group members only. If you need to access it, please contact group PI [[Contributors | Prof. Nicholas Harrison]].''

== CRYSTAL23 v1 (Intel) ==
The current default for CX1. Not available for HX1.

=== CX1 version ===
* Compiling Env : EasyBuild Intel 2023a
* Note: MPPproperties and MP2 (CRYSCOR and CRYSTAL2) are currently not available, which involves separate code packages (DMat2 and CRYSCOR). For MPPproperties please read the GNU version below.

The general job submission script for Imperial HPC, developed by the author himself, is used here.

''Thanks to Mr. K Tallat-Kelpsa for testing''

''' Configuration '''

The source code (written in bash) is available on group [https://github.com/cmsg-icl/HPC-job-submission/tree/main/Imperial-HPC-Job-Submission GitHub repo], though it is not needed in practice. Useful for developing new features. On CX1, use the following command to configure your local settings:

<code>
~$ bash /rds/general/project/cmsg/live/share/HPC-job-submission/Imperial-HPC-Job-Submission/CRYSTAL23/config_CRYSTAL23.sh
</code>

Then follow the instructions on the screen. Typically the default values are good. To use default vaules, press enter. After configuration is finished, the information is stored in a 'settings' file which is by default saved as <code>${HOME}/etc/runCRYSTAL23/settings</code>. It functions as a dictionary for job sumbmission scripts to refer to. It is also editable according to the user's needs.

Use the following command to activate alias commands:

<code>
~$ source ~/.bashrc
</code>

Due to the settings of CX1, the <code>source</code> command should be executed everytime you login. To aviod this, try the following commands:

~$ cat << EOF >> ~/.bash_profile
if test -f ~/.bashrc; then
source ~/.bashrc
fi
EOF

''' Quick References '''

{| class="wikitable" style="margin:auto"
|+ List of available commands
|-
! Command !! Definition
|-
| Pcrys23 || Generate parallel OMP CRYSTAL23 job submission files
|-
| MPPcrys23 || Generate massive parallel OMP CRYSTAL23 job submission files
|-
| Pprop23 || Generate parallel OMP CRYSTAL23 properties job submission files
|-
| Scrys23 || Generate serial OMP CRYSTAL23 job submission files
|-
| Sprop23 || Generate serial OMP CRYSTAL23 properties job submission files
|-
| Xcrys23 || User-defined executables and multiple jobs (see below for code examples)
|-
| SETcrys23 || Print the local 'settings' file
|-
| HELPcrys23 || Print the instructions of commands
|}

{| class="wikitable" style="margin:auto"
|+ List of in-line flags
|-
! Command !! Definition
|-
| -in || Input .d12 or .d3 files
|-
| -nd || Number of nodes requested. Number of CPUs per node will be read from the settings file if '-nc' is not set
|-
| -nc || Number of CPUs requested. Number of nodes will be automatically decided by Number of CPUs per node from settings file if '-nd' is not set
|-
| -nt || Number of threads per process. If not set the default value (1) is used.
|-
| -nt || Memory requested per node. If not set the default value (512GB) is used.
|-
| -wt || Walltime for the job. Or walltime for each job if multiple jobs
|-
| -ref || Optional, name of the reference file(s). No extension
|-
| -x || Xcrys23 only. The alias, i.e., lable, of MPI+executable in-line command pair
|-
| -name || Xcrys23 only. Set a common name for multiple jobs
|-
| -set || Developer only. The path to local 'settings' file
|}

{| class="wikitable" style="margin:auto"
|+ List of available labels for '-x' flag
|-
! Label !! Acutal in-line command
|-
| pcrys || mpiexec Pcrystal
|-
| mppcrys || mpiexec MPPcrystal
|-
| pprop || mpiexec Pproperties
|-
| scrys || Scrystal < INPUT
|-
| sprop || Sproperties < INPUT
|}

Note: New labels can be defined in local 'settings' file if a new MPI+executable in-line command is needed

''' Code Examples '''

To generate a qsub file for a parallel crystal23 job on 'mgo.d12', the following command can be used:

<code>
~$ Pcrys23 -in mgo.d12 -wt 01:00 -nd 1
</code>

That generates a 'mgo.qsub' file requesting for 1 node and the maximum running time is 1 hour. Similarly, after the job is done, using the following command can run a parallel properties calculation based on 'mgo-band.d3' and data from the previous 'mgo' SCF calculation on 12 CPUs:

<code>
~$ Pprop23 -in mgo-band.d3 -nc 12 -wt 00:30 -ref mgo
</code>

Alternatively, the user can also integrate 2 jobs into the same qsub file using <code>Xcrys23</code>.

<code>
~$ Xcrys23 -name mgo-band -nd 1 -x pcrys -in mgo.d12 -wt 01:00 -ref no -x pprop -in mgo-band.d3 -wt 00:30 -ref mgo
</code>

Note that -x, -in, -wt should have the same lengths while -ref should be either 0 or the same length as -x. In the case above, 'no' is a reserved keyword for no reference. It should be noted that specifying multiple -in, -wt and -ref flags in <code>Pcrys</code>, <code>MPPcrys</code>, <code>Pprop</code>, <code>Scrys</code> and <code>Sprop</code> will lead to error.

Besides, <code>Xcrys23</code> also supports user-defined MPI+executable command pairs in 'settings' file:

<code>
~$ Xcrys23 -x pcrys_other -in mgo.d12 -wt 01:00
</code>

For more detailed instructions, test cases and keyword list of settings file, please refer to the [https://github.com/cmsg-icl/HPC-job-submission/tree/main/Imperial-HPC-Job-Submission documentation].

== CRYSTAL23 v1 (GNU) ==
Compared to Intel version above, GNU version is slightly slower to allow for compatibility with MPPproperties. Available for CX1 and HX1.

=== CX1 version ===
* Compiling Env : gcc11.2.0, aocl4.0, mpich4.0.2
* Note: MP2 (CRYSCOR and CRYSTAL2) is currently not available.

The same job submission script is used. However, GNU version is not the default option for CX1. To enable GNU version, if there is a 'settings' file, change the following parameters:

* '''EXEDIR''' : <code>module load /rds/general/project/cmsg/live/etc/modulefiles/CRYSTAL/23v1-gcc</code>
* '''MPIDIR''' : <code>module load /rds/general/project/cmsg/live/etc/compiler/gcc11.2.0-aocl</code>.

Alternatively, rerun the configuration file (same as above) and specify 2 values during configuration.

Note that by default, there is no command for 'MPPproperties'. The user has to modify their <code>~/.bashrc</code> file to set the alias and 'settings' file for executable flags. An example:

# in ~/.bashrc
alias MPPprop23="/rds/general/project/cmsg/live/share/HPC-job-submission/Imperial-HPC-Job-Submission/gen_sub -x mppprop -set ${HOME}/etc/runCRYSTAL23/settings"
# in settings, keep the column width
mppprop mpiexec -np ${V_TPROC} MPPproperties Massive parallel properties calculation, OMP

=== HX1 version ===
* Compiling Env : EasyBuild foss/2023a
* Note: MP2 (CRYSCOR and CRYSTAL2) is currently not available.

* '''EXEDIR''' : No default. The directory where the user put their executables in.
* '''MPIDIR''' : <code>module load foss/2023a</code>

Alternatively, run the configuration file (The one with '-HX1' in the same directory of Github repo) and specify 2 values during configuration. Other options are consistent with CRYSTAL on CX1.

== CRYSTAL17 v2 (GNU)==
Not hosted on RDS-CMSG. Default for CX1. Not available for HX1.

=== CX1 version ===
* Compilor : gcc/6.2.0
* MPI : mpich/3.4.3

'''For job submission scripts'''

The configuration and usage are identical to CRYSTAL23. Please refer to the previous section and substitute '23' with '17'. Also, please be noted that CRYSCOR (MP2 calculation) is not available for CRYSTAL17 and 'MPP' as a massive parallel strategy is limited to 'crystal' calculations, i.e., only MPPcrystal is released.

== Quantum Espresso v7.2 (GNU) ==
Job submission file configured for CX1 and HX1, but only CX1 executable is shared as these executables are highly dependent on environment.

=== CX1 version ===
* Compiling Env : EasyBuild foss2022a
* libxc: Yes, version 5.1.2
* hdf5: No

Pseudopotential files are saved in <code>/rds/general/project/cmsg/live/etc/QE_PseudoP</code>.

''' Job submission script configuration '''

<code>
~$ bash /rds/general/project/cmsg/live/share/HPC-job-submission/Imperial-HPC-Job-Submission/QE7/config_QE7.sh
</code>

The procedure is the same as CRYSTAL. Type enter for default setups, which typically works fine. Please note that moving files between temporal and home directories are disabled as QE has built-in file management system.

''' Job submission script commands '''

{| class="wikitable" style="margin:auto"
|+ List of available commands
|-
! Command !! Definition
|-
| PWqe7 || Generate pw.x job submission files
|-
| PHqe7 || Generate ph.x job submission files
|-
| CPqe7 || Generate cp.x job submission files
|-
| PPqe7 || Generate pp.x job submission files
|-
| Xqe7 || Generate job submission files for user-defined executables
|-
| SETqe7 || Print the local 'settings' file
|-
| HELPqe7 || Print the instructions of commands
|}

== GULP v6.1.2 (GNU) ==
Default for CX1. Not available for HX1.

=== CX1 version ===
* Compiling Env : EasyBuild foss2022a
* OpenKIM: Yes, version 2.3.0
* PLUMED: Yes, version 2.9.0
* ALAMODE: Yes, git repo copied into <code>app/</code>

GULP force field libraries are saved in <code>/rds/general/project/cmsg/live/etc/GULP_Libraries</code>. Executable name: 'gulp-mpi'.

'''OpenKIM'''

KIM force field <ref>https://openkim.org/</ref> are saved in <code>/rds/general/project/cmsg/live/etc/KIM_Models</code>. To run OpenKIM executable and download new models, run:

<code>
~$ module load /rds/general/project/cmsg/live/etc/modulefiles/OpenKIM/2.3.0-foss
</code>

'''Job submission script configuration'''

<code>
~$ bash /rds/general/project/cmsg/live/share/HPC-job-submission/Imperial-HPC-Job-Submission/GULP6/config_GULP6.sh
</code>

Type enter for default setups, which typically works fine.

'''Job submission script commands'''

{| class="wikitable" style="margin:auto"
|+ List of available commands
|-
! Command !! Definition
|-
| Pglp6 || Generate job submission files for 'gulp-mpi'
|-
| Xglp6 || Generate job submission files for user defined executable (by default nothing else, so not useful).
|-
| SETglp6 || Print the local 'settings' file
|-
| HELPglp6|| Print the instructions of commands
|}

==References==
<references>

CMSG disk and Shared Software on CX1

2023-11-30T22:24:16Z

Hz1420: /* CRYSTAL23 v1 (GNU) */

General information of shared software available for CMSG group members on Imperial CX1 is summarized in this page. Software listed below, unless stated otherwise, is hosted under 'CMSG' project of Imperial Research Data Store (RDS) <ref>https://www.imperial.ac.uk/admin-services/ict/self-service/research-support/rcs/service-offering/rds/</ref> (referred as RDS-CMSG below), which is accessible for both login and computational nodes of Imperial HPC CX1. For Imperial HPC HX1, by the time this page is upated (Nov. 2023), it is not accessible there. Only login node can visit RDS-CMSG disk after the pilot phase <ref>https://wiki.imperial.ac.uk/display/HPC/HX1+Cluster</ref>. Therefore software on RDS-CMSG are classified into 2 catagories:

# For CX1, use <code>/rds/general/project/cmsg/live/app/</code> (executables, libraries and headers) and <code>/rds/general/project/cmsg/live/etc/</code> (other files, such as pseudopotential). Executable can be called directly from the user's home directory.
# For HX1, use <code>/rds/general/project/cmsg/live/share/</code>. Executables must be downloaded and uploaded to HX1. It is user's responsibility to set up an effective running environment according to 'readme' files (if there is one) saved in the same, or upper, directory.

''Please note that accessibility to CMSG disk is granted for CMSG group members only. If you need to access it, please contact group PI [[Contributors | Prof. Nicholas Harrison]].''

== CRYSTAL23 v1 (Intel) ==
The current default for CX1. Not available for HX1.

=== CX1 version ===
* Compiling Env : EasyBuild Intel 2023a
* Note: MPPproperties and MP2 (CRYSCOR and CRYSTAL2) are currently not available, which involves separate code packages (DMat2 and CRYSCOR). For MPPproperties please read the GNU version below.

The general job submission script for Imperial HPC, developed by the author himself, is used here.

''Thanks to Mr. K Tallat-Kelpsa for testing''

''' Configuration '''

The source code (written in bash) is available on group [https://github.com/cmsg-icl/HPC-job-submission/tree/main/Imperial-HPC-Job-Submission GitHub repo], though it is not needed in practice. Useful for developing new features. On CX1, use the following command to configure your local settings:

<code>
~$ bash /rds/general/project/cmsg/live/share/HPC-job-submission/Imperial-HPC-Job-Submission/CRYSTAL23/config_CRYSTAL23.sh
</code>

Then follow the instructions on the screen. Typically the default values are good. To use default vaules, press enter. After configuration is finished, the information is stored in a 'settings' file which is by default saved as <code>${HOME}/etc/runCRYSTAL23/settings</code>. It functions as a dictionary for job sumbmission scripts to refer to. It is also editable according to the user's needs.

Use the following command to activate alias commands:

<code>
~$ source ~/.bashrc
</code>

Due to the settings of CX1, the <code>source</code> command should be executed everytime you login. To aviod this, try the following commands:

~$ cat << EOF >> ~/.bash_profile
if test -f ~/.bashrc; then
source ~/.bashrc
fi
EOF

''' Quick References '''

{| class="wikitable" style="margin:auto"
|+ List of available commands
|-
! Command !! Definition
|-
| Pcrys23 || Generate parallel OMP CRYSTAL23 job submission files
|-
| MPPcrys23 || Generate massive parallel OMP CRYSTAL23 job submission files
|-
| Pprop23 || Generate parallel OMP CRYSTAL23 properties job submission files
|-
| Scrys23 || Generate serial OMP CRYSTAL23 job submission files
|-
| Sprop23 || Generate serial OMP CRYSTAL23 properties job submission files
|-
| Xcrys23 || User-defined executables and multiple jobs (see below for code examples)
|-
| SETcrys23 || Print the local 'settings' file
|-
| HELPcrys23 || Print the instructions of commands
|}

{| class="wikitable" style="margin:auto"
|+ List of in-line flags
|-
! Command !! Definition
|-
| -in || Input .d12 or .d3 files
|-
| -nd || Number of nodes requested. Number of CPUs per node will be read from the settings file if '-nc' is not set
|-
| -nc || Number of CPUs requested. Number of nodes will be automatically decided by Number of CPUs per node from settings file if '-nd' is not set
|-
| -nt || Number of threads per process. If not set the default value (1) is used.
|-
| -nt || Memory requested per node. If not set the default value (512GB) is used.
|-
| -wt || Walltime for the job. Or walltime for each job if multiple jobs
|-
| -ref || Optional, name of the reference file(s). No extension
|-
| -x || Xcrys23 only. The alias, i.e., lable, of MPI+executable in-line command pair
|-
| -name || Xcrys23 only. Set a common name for multiple jobs
|-
| -set || Developer only. The path to local 'settings' file
|}

{| class="wikitable" style="margin:auto"
|+ List of available labels for '-x' flag
|-
! Label !! Acutal in-line command
|-
| pcrys || mpiexec Pcrystal
|-
| mppcrys || mpiexec MPPcrystal
|-
| pprop || mpiexec Pproperties
|-
| scrys || Scrystal < INPUT
|-
| sprop || Sproperties < INPUT
|}

Note: New labels can be defined in local 'settings' file if a new MPI+executable in-line command is needed

''' Code Examples '''

To generate a qsub file for a parallel crystal23 job on 'mgo.d12', the following command can be used:

<code>
~$ Pcrys23 -in mgo.d12 -wt 01:00 -nd 1
</code>

That generates a 'mgo.qsub' file requesting for 1 node and the maximum running time is 1 hour. Similarly, after the job is done, using the following command can run a parallel properties calculation based on 'mgo-band.d3' and data from the previous 'mgo' SCF calculation on 12 CPUs:

<code>
~$ Pprop23 -in mgo-band.d3 -nc 12 -wt 00:30 -ref mgo
</code>

Alternatively, the user can also integrate 2 jobs into the same qsub file using <code>Xcrys23</code>.

<code>
~$ Xcrys23 -name mgo-band -nd 1 -x pcrys -in mgo.d12 -wt 01:00 -ref no -x pprop -in mgo-band.d3 -wt 00:30 -ref mgo
</code>

Note that -x, -in, -wt should have the same lengths while -ref should be either 0 or the same length as -x. In the case above, 'no' is a reserved keyword for no reference. It should be noted that specifying multiple -in, -wt and -ref flags in <code>Pcrys</code>, <code>MPPcrys</code>, <code>Pprop</code>, <code>Scrys</code> and <code>Sprop</code> will lead to error.

Besides, <code>Xcrys23</code> also supports user-defined MPI+executable command pairs in 'settings' file:

<code>
~$ Xcrys23 -x pcrys_other -in mgo.d12 -wt 01:00
</code>

For more detailed instructions, test cases and keyword list of settings file, please refer to the [https://github.com/cmsg-icl/HPC-job-submission/tree/main/Imperial-HPC-Job-Submission documentation].

== CRYSTAL23 v1 (GNU) ==
Compared to Intel version above, GNU version is slightly slower to allow for compatibility with MPPproperties. Available for CX1 and HX1.

=== CX1 version ===
* Compiling Env : gcc11.2.0, aocl4.0, mpich4.0.2
* Note: MP2 (CRYSCOR and CRYSTAL2) is currently not available.

The same job submission script is used. However, GNU version is not the default option for CX1. To enable GNU version, if there is a 'settings' file, change the following parameters:

* '''EXEDIR''' : <code>module load /rds/general/project/cmsg/live/etc/modulefiles/CRYSTAL/23v1-gcc</code>
* '''MPIDIR''' : <code>module load /rds/general/project/cmsg/live/etc/compiler/gcc11.2.0-aocl</code>.

Alternatively, rerun the configuration file (same as above) and specify 2 values during configuration.

Note that by default, there is no command for 'MPPproperties'. The user has to modify their <code>~/.bashrc</code> file to set the alias and 'settings' file for executable flags. An example:

# in ~/.bashrc
alias MPPprop23="/rds/general/project/cmsg/live/share/HPC-job-submission/Imperial-HPC-Job-Submission/gen_sub -x mppprop -set ${HOME}/etc/runCRYSTAL23/settings"
# in settings, keep the column width
mppprop mpiexec -np ${V_TPROC} MPPproperties Massive parallel properties calculation, OMP

=== HX1 version ===
* Compiling Env : EasyBuild foss/2023a
* Note: MP2 (CRYSCOR and CRYSTAL2) is currently not available.

* '''EXEDIR''' : No default. The directory where the user put their executables in.
* '''MPIDIR''' : <code>module load foss/2023a</code>

Alternatively, run the configuration file (The one with '-HX1' in the same directory of Github repo) and specify 2 values during configuration. Other options are consistent with CRYSTAL on CX1.

== CRYSTAL17 v2 (GNU)==
Not hosted on RDS-CMSG. Default for CX1. Not available for HX1.

=== CX1 version ===
* Compilor : gcc/6.2.0
* MPI : mpich/3.4.3

'''For job submission scripts'''

The configuration and usage are identical to CRYSTAL23. Please refer to the previous section and substitute '23' with '17'. Also, please be noted that CRYSCOR (MP2 calculation) is not available for CRYSTAL17 and 'MPP' as a massive parallel strategy is limited to 'crystal' calculations, i.e., only MPPcrystal is released.

== Quantum Espresso v7.2 (GNU) ==
Job submission file configured for CX1 and HX1, but only CX1 executable is shared as these executables are highly dependent on environment.

=== CX1 version ===
* Compiling Env : EasyBuild foss2022a
* libxc: Yes, version 5.1.2
* hdf5: No

Pseudopotential files are saved in <code>/rds/general/project/cmsg/live/etc/QE_PseudoP</code>.

''' Job submission script configuration '''

<code>
~$ bash /rds/general/project/cmsg/live/share/HPC-job-submission/Imperial-HPC-Job-Submission/QE7/config_QE7.sh
</code>

The procedure is the same as CRYSTAL. Type enter for default setups, which typically works fine. Please note that moving files between temporal and home directories are disabled as QE has built-in file management system.

''' Job submission script commands '''

{| class="wikitable" style="margin:auto"
|+ List of available commands
|-
! Command !! Definition
|-
| PWqe7 || Generate pw.x job submission files
|-
| PHqe7 || Generate ph.x job submission files
|-
| CPqe7 || Generate cp.x job submission files
|-
| PPqe7 || Generate pp.x job submission files
|-
| Xqe7 || Generate job submission files for user-defined executables
|-
| SETqe7 || Print the local 'settings' file
|-
| HELPqe7 || Print the instructions of commands
|}

== GULP v6.1.2 (GNU) ==
Default for CX1. Not available for HX1.

=== CX1 version ===
* Compiling Env : EasyBuild foss2022a
* OpenKIM: Yes, version 2.3.0
* PLUMED: Yes, version 2.9.0
* ALAMODE: Yes, git repo copied into <code>app/</code>

GULP force field libraries are saved in <code>/rds/general/project/cmsg/live/etc/GULP_Libraries</code>. Executable name: 'gulp-mpi'.

'''OpenKIM'''

KIM force field <ref>https://openkim.org/</ref> are saved in <code>/rds/general/project/cmsg/live/etc/KIM_Models</code>. To run OpenKIM executable and download new models, run:

<code>
~$ module load /rds/general/project/cmsg/live/etc/modulefiles/OpenKIM/2.3.0-foss
</code>

'''Job submission script configuration'''

<code>
~$ bash /rds/general/project/cmsg/live/share/HPC-job-submission/Imperial-HPC-Job-Submission/GULP6/config_GULP6.sh
</code>

Type enter for default setups, which typically works fine.

'''Job submission script commands'''

{| class="wikitable" style="margin:auto"
|+ List of available commands
|-
! Command !! Definition
|-
| Pglp6 || Generate job submission files for 'gulp-mpi'
|-
| Xglp6 || Generate job submission files for user defined executable (by default nothing else, so not useful).
|-
| SETglp6 || Print the local 'settings' file
|-
| HELPglp6|| Print the instructions of commands
|}

==References==
<references>

CMSG disk and Shared Software on CX1

2023-11-30T22:21:29Z

Hz1420:

General information of shared software available for CMSG group members on Imperial CX1 is summarized in this page. Software listed below, unless stated otherwise, is hosted under 'CMSG' project of Imperial Research Data Store (RDS) <ref>https://www.imperial.ac.uk/admin-services/ict/self-service/research-support/rcs/service-offering/rds/</ref> (referred as RDS-CMSG below), which is accessible for both login and computational nodes of Imperial HPC CX1. For Imperial HPC HX1, by the time this page is upated (Nov. 2023), it is not accessible there. Only login node can visit RDS-CMSG disk after the pilot phase <ref>https://wiki.imperial.ac.uk/display/HPC/HX1+Cluster</ref>. Therefore software on RDS-CMSG are classified into 2 catagories:

# For CX1, use <code>/rds/general/project/cmsg/live/app/</code> (executables, libraries and headers) and <code>/rds/general/project/cmsg/live/etc/</code> (other files, such as pseudopotential). Executable can be called directly from the user's home directory.
# For HX1, use <code>/rds/general/project/cmsg/live/share/</code>. Executables must be downloaded and uploaded to HX1. It is user's responsibility to set up an effective running environment according to 'readme' files (if there is one) saved in the same, or upper, directory.

''Please note that accessibility to CMSG disk is granted for CMSG group members only. If you need to access it, please contact group PI [[Contributors | Prof. Nicholas Harrison]].''

== CRYSTAL23 v1 (Intel) ==
The current default for CX1. Not available for HX1.

=== CX1 version ===
* Compiling Env : EasyBuild Intel 2023a
* Note: MPPproperties and MP2 (CRYSCOR and CRYSTAL2) are currently not available, which involves separate code packages (DMat2 and CRYSCOR). For MPPproperties please read the GNU version below.

The general job submission script for Imperial HPC, developed by the author himself, is used here.

''Thanks to Mr. K Tallat-Kelpsa for testing''

''' Configuration '''

The source code (written in bash) is available on group [https://github.com/cmsg-icl/HPC-job-submission/tree/main/Imperial-HPC-Job-Submission GitHub repo], though it is not needed in practice. Useful for developing new features. On CX1, use the following command to configure your local settings:

<code>
~$ bash /rds/general/project/cmsg/live/share/HPC-job-submission/Imperial-HPC-Job-Submission/CRYSTAL23/config_CRYSTAL23.sh
</code>

Then follow the instructions on the screen. Typically the default values are good. To use default vaules, press enter. After configuration is finished, the information is stored in a 'settings' file which is by default saved as <code>${HOME}/etc/runCRYSTAL23/settings</code>. It functions as a dictionary for job sumbmission scripts to refer to. It is also editable according to the user's needs.

Use the following command to activate alias commands:

<code>
~$ source ~/.bashrc
</code>

Due to the settings of CX1, the <code>source</code> command should be executed everytime you login. To aviod this, try the following commands:

~$ cat << EOF >> ~/.bash_profile
if test -f ~/.bashrc; then
source ~/.bashrc
fi
EOF

''' Quick References '''

{| class="wikitable" style="margin:auto"
|+ List of available commands
|-
! Command !! Definition
|-
| Pcrys23 || Generate parallel OMP CRYSTAL23 job submission files
|-
| MPPcrys23 || Generate massive parallel OMP CRYSTAL23 job submission files
|-
| Pprop23 || Generate parallel OMP CRYSTAL23 properties job submission files
|-
| Scrys23 || Generate serial OMP CRYSTAL23 job submission files
|-
| Sprop23 || Generate serial OMP CRYSTAL23 properties job submission files
|-
| Xcrys23 || User-defined executables and multiple jobs (see below for code examples)
|-
| SETcrys23 || Print the local 'settings' file
|-
| HELPcrys23 || Print the instructions of commands
|}

{| class="wikitable" style="margin:auto"
|+ List of in-line flags
|-
! Command !! Definition
|-
| -in || Input .d12 or .d3 files
|-
| -nd || Number of nodes requested. Number of CPUs per node will be read from the settings file if '-nc' is not set
|-
| -nc || Number of CPUs requested. Number of nodes will be automatically decided by Number of CPUs per node from settings file if '-nd' is not set
|-
| -nt || Number of threads per process. If not set the default value (1) is used.
|-
| -nt || Memory requested per node. If not set the default value (512GB) is used.
|-
| -wt || Walltime for the job. Or walltime for each job if multiple jobs
|-
| -ref || Optional, name of the reference file(s). No extension
|-
| -x || Xcrys23 only. The alias, i.e., lable, of MPI+executable in-line command pair
|-
| -name || Xcrys23 only. Set a common name for multiple jobs
|-
| -set || Developer only. The path to local 'settings' file
|}

{| class="wikitable" style="margin:auto"
|+ List of available labels for '-x' flag
|-
! Label !! Acutal in-line command
|-
| pcrys || mpiexec Pcrystal
|-
| mppcrys || mpiexec MPPcrystal
|-
| pprop || mpiexec Pproperties
|-
| scrys || Scrystal < INPUT
|-
| sprop || Sproperties < INPUT
|}

Note: New labels can be defined in local 'settings' file if a new MPI+executable in-line command is needed

''' Code Examples '''

To generate a qsub file for a parallel crystal23 job on 'mgo.d12', the following command can be used:

<code>
~$ Pcrys23 -in mgo.d12 -wt 01:00 -nd 1
</code>

That generates a 'mgo.qsub' file requesting for 1 node and the maximum running time is 1 hour. Similarly, after the job is done, using the following command can run a parallel properties calculation based on 'mgo-band.d3' and data from the previous 'mgo' SCF calculation on 12 CPUs:

<code>
~$ Pprop23 -in mgo-band.d3 -nc 12 -wt 00:30 -ref mgo
</code>

Alternatively, the user can also integrate 2 jobs into the same qsub file using <code>Xcrys23</code>.

<code>
~$ Xcrys23 -name mgo-band -nd 1 -x pcrys -in mgo.d12 -wt 01:00 -ref no -x pprop -in mgo-band.d3 -wt 00:30 -ref mgo
</code>

Note that -x, -in, -wt should have the same lengths while -ref should be either 0 or the same length as -x. In the case above, 'no' is a reserved keyword for no reference. It should be noted that specifying multiple -in, -wt and -ref flags in <code>Pcrys</code>, <code>MPPcrys</code>, <code>Pprop</code>, <code>Scrys</code> and <code>Sprop</code> will lead to error.

Besides, <code>Xcrys23</code> also supports user-defined MPI+executable command pairs in 'settings' file:

<code>
~$ Xcrys23 -x pcrys_other -in mgo.d12 -wt 01:00
</code>

For more detailed instructions, test cases and keyword list of settings file, please refer to the [https://github.com/cmsg-icl/HPC-job-submission/tree/main/Imperial-HPC-Job-Submission documentation].

== CRYSTAL23 v1 (GNU) ==
Compared to Intel version above, GNU version is slightly slower to allow for compatibility with MPPproperties. Available for CX1 and HX1.

=== CX1 version ===
* Compiling Env : gcc11.2.0, aocl4.0, mpich4.0.2
* Note: MP2 (CRYSCOR and CRYSTAL2) is currently not available.

The same job submission script is used. However, GNU version is not the default option for CX1. To enable GNU version, if there is a 'settings' file, change the following parameters:

* '''EXEDIR''' : <code>module load /rds/general/project/cmsg/live/etc/modulefiles/CRYSTAL/23v1-gcc</code>
* '''MPIDIR''' : <code>module load /rds/general/project/cmsg/live/etc/compiler/gcc11.2.0-aocl</code>.

Alternatively, rerun the configuration file (same as above) and specify 2 values during configuration.

Note that by default, there is no command for 'MPPproperties'. The user has to modify their <code>~/.bashrc</code> file to set the alias and 'settings' file for executable flags. An example:

# in ~/.bashrc
alias MPPprop23="/rds/general/project/cmsg/live/share/HPC-job-submission/Imperial-HPC-Job-Submission/gen_sub -x mppprop -set ${HOME}/etc/runCRYSTAL23/settings"
# in settings, keep the column width
mppprop mpiexec -np ${V_TPROC} MPPproperties Massive parallel properties calculation, OMP

=== HX1 version ===
* Compiling Env : EasyBuild foss/2023a
* Note: MP2 (CRYSCOR and CRYSTAL2) is currently not available.

* '''EXEDIR''' : No default. The directory where the user put their executables in.
* '''MPIDIR''' : <code>module load foss/2023a</code>

Alternatively, run the configuration file (The one with '-HX1' in the same directory of Github repo) and specify 2 values during configuration. Other options are consistent with CRYSTAL on CX1.

== CRYSTAL17 v2 (GNU)==
Not hosted on RDS-CMSG. Default for CX1. Not available for HX1.

=== CX1 version ===
* Compilor : gcc/6.2.0
* MPI : mpich/3.4.3

'''For job submission scripts'''

The configuration and usage are identical to CRYSTAL23. Please refer to the previous section and substitute '23' with '17'. Also, please be noted that CRYSCOR (MP2 calculation) is not available for CRYSTAL17 and 'MPP' as a massive parallel strategy is limited to 'crystal' calculations, i.e., only MPPcrystal is released.

== Quantum Espresso v7.2 (GNU) ==
Job submission file configured for CX1 and HX1, but only CX1 executable is shared as these executables are highly dependent on environment.

=== CX1 version ===
* Compiling Env : EasyBuild foss2022a
* libxc: Yes, version 5.1.2
* hdf5: No

Pseudopotential files are saved in <code>/rds/general/project/cmsg/live/etc/QE_PseudoP</code>.

''' Job submission script configuration '''

<code>
~$ bash /rds/general/project/cmsg/live/share/HPC-job-submission/Imperial-HPC-Job-Submission/QE7/config_QE7.sh
</code>

The procedure is the same as CRYSTAL. Type enter for default setups, which typically works fine. Please note that moving files between temporal and home directories are disabled as QE has built-in file management system.

''' Job submission script commands '''

{| class="wikitable" style="margin:auto"
|+ List of available commands
|-
! Command !! Definition
|-
| PWqe7 || Generate pw.x job submission files
|-
| PHqe7 || Generate ph.x job submission files
|-
| CPqe7 || Generate cp.x job submission files
|-
| PPqe7 || Generate pp.x job submission files
|-
| Xqe7 || Generate job submission files for user-defined executables
|-
| SETqe7 || Print the local 'settings' file
|-
| HELPqe7 || Print the instructions of commands
|}

== GULP v6.1.2 (GNU) ==
Default for CX1. Not available for HX1.

=== CX1 version ===
* Compiling Env : EasyBuild foss2022a
* OpenKIM: Yes, version 2.3.0
* PLUMED: Yes, version 2.9.0
* ALAMODE: Yes, git repo copied into <code>app/</code>

GULP force field libraries are saved in <code>/rds/general/project/cmsg/live/etc/GULP_Libraries</code>. Executable name: 'gulp-mpi'.

'''OpenKIM'''

KIM force field <ref>https://openkim.org/</ref> are saved in <code>/rds/general/project/cmsg/live/etc/KIM_Models</code>. To run OpenKIM executable and download new models, run:

<code>
~$ module load /rds/general/project/cmsg/live/etc/modulefiles/OpenKIM/2.3.0-foss
</code>

'''Job submission script configuration'''

<code>
~$ bash /rds/general/project/cmsg/live/share/HPC-job-submission/Imperial-HPC-Job-Submission/GULP6/config_GULP6.sh
</code>

Type enter for default setups, which typically works fine.

'''Job submission script commands'''

{| class="wikitable" style="margin:auto"
|+ List of available commands
|-
! Command !! Definition
|-
| Pglp6 || Generate job submission files for 'gulp-mpi'
|-
| Xglp6 || Generate job submission files for user defined executable (by default nothing else, so not useful).
|-
| SETglp6 || Print the local 'settings' file
|-
| HELPglp6|| Print the instructions of commands
|}

==References==
<references>

CMSG disk and Shared Software on CX1

2023-11-30T22:20:00Z

Hz1420:

General information of shared software available for CMSG group members on Imperial CX1 is summarized in this page. Software listed below, unless stated otherwise, is hosted under 'CMSG' project of Imperial Research Data Store (RDS) <ref>https://www.imperial.ac.uk/admin-services/ict/self-service/research-support/rcs/service-offering/rds/</ref> (referred as RDS-CMSG below), which is accessible for both login and computational nodes of Imperial HPC CX1. For Imperial HPC HX1, by the time this page is upated (Nov. 2023), it is not accessible there. Only login node can visit RDS-CMSG disk after the pilot phase <ref>https://wiki.imperial.ac.uk/display/HPC/HX1+Cluster</ref>. Therefore software on RDS-CMSG are classified into 2 catagories:

# For CX1, use <code>/rds/general/project/cmsg/live/app/</code> (executables, libraries and headers) and <code>/rds/general/project/cmsg/live/etc/</code> (other files, such as pseudopotential). Executable can be called directly from the user's home directory.
# For HX1, use <code>/rds/general/project/cmsg/live/share/</code>. Executables must be downloaded and uploaded to HX1. It is user's responsibility to set up a effective running environment according to 'readme' files (if there is one) saved in the same, or upper, directory.

''Please note that accessibility to CMSG disk is granted for CMSG group members only. If you need to access it, please contact group PI [[Contributors | Prof. Nicholas Harrison]].''

== CRYSTAL23 v1 (Intel) ==
The current default for CX1. Not available for HX1.

=== CX1 version ===
* Compiling Env : EasyBuild Intel 2023a
* Note: MPPproperties and MP2 (CRYSCOR and CRYSTAL2) are currently not available, which involves separate code packages (DMat2 and CRYSCOR). For MPPproperties please read the GNU version below.

The general job submission script for Imperial HPC, developed by the author himself, is used here.

''Thanks to Mr. K Tallat-Kelpsa for testing''

''' Configuration '''

The source code (written in bash) is available on group [https://github.com/cmsg-icl/HPC-job-submission/tree/main/Imperial-HPC-Job-Submission GitHub repo], though it is not needed in practice. Useful for developing new features. On CX1, use the following command to configure your local settings:

<code>
~$ bash /rds/general/project/cmsg/live/share/HPC-job-submission/Imperial-HPC-Job-Submission/CRYSTAL23/config_CRYSTAL23.sh
</code>

Then follow the instructions on the screen. Typically the default values are good. To use default vaules, press enter. After configuration is finished, the information is stored in a 'settings' file which is by default saved as <code>${HOME}/etc/runCRYSTAL23/settings</code>. It functions as a dictionary for job sumbmission scripts to refer to. It is also editable according to the user's needs.

Use the following command to activate alias commands:

<code>
~$ source ~/.bashrc
</code>

Due to the settings of CX1, the <code>source</code> command should be executed everytime you login. To aviod this, try the following commands:

~$ cat << EOF >> ~/.bash_profile
if test -f ~/.bashrc; then
source ~/.bashrc
fi
EOF

''' Quick References '''

{| class="wikitable" style="margin:auto"
|+ List of available commands
|-
! Command !! Definition
|-
| Pcrys23 || Generate parallel OMP CRYSTAL23 job submission files
|-
| MPPcrys23 || Generate massive parallel OMP CRYSTAL23 job submission files
|-
| Pprop23 || Generate parallel OMP CRYSTAL23 properties job submission files
|-
| Scrys23 || Generate serial OMP CRYSTAL23 job submission files
|-
| Sprop23 || Generate serial OMP CRYSTAL23 properties job submission files
|-
| Xcrys23 || User-defined executables and multiple jobs (see below for code examples)
|-
| SETcrys23 || Print the local 'settings' file
|-
| HELPcrys23 || Print the instructions of commands
|}

{| class="wikitable" style="margin:auto"
|+ List of in-line flags
|-
! Command !! Definition
|-
| -in || Input .d12 or .d3 files
|-
| -nd || Number of nodes requested. Number of CPUs per node will be read from the settings file if '-nc' is not set
|-
| -nc || Number of CPUs requested. Number of nodes will be automatically decided by Number of CPUs per node from settings file if '-nd' is not set
|-
| -nt || Number of threads per process. If not set the default value (1) is used.
|-
| -nt || Memory requested per node. If not set the default value (512GB) is used.
|-
| -wt || Walltime for the job. Or walltime for each job if multiple jobs
|-
| -ref || Optional, name of the reference file(s). No extension
|-
| -x || Xcrys23 only. The alias, i.e., lable, of MPI+executable in-line command pair
|-
| -name || Xcrys23 only. Set a common name for multiple jobs
|-
| -set || Developer only. The path to local 'settings' file
|}

{| class="wikitable" style="margin:auto"
|+ List of available labels for '-x' flag
|-
! Label !! Acutal in-line command
|-
| pcrys || mpiexec Pcrystal
|-
| mppcrys || mpiexec MPPcrystal
|-
| pprop || mpiexec Pproperties
|-
| scrys || Scrystal < INPUT
|-
| sprop || Sproperties < INPUT
|}

Note: New labels can be defined in local 'settings' file if a new MPI+executable in-line command is needed

''' Code Examples '''

To generate a qsub file for a parallel crystal23 job on 'mgo.d12', the following command can be used:

<code>
~$ Pcrys23 -in mgo.d12 -wt 01:00 -nd 1
</code>

That generates a 'mgo.qsub' file requesting for 1 node and the maximum running time is 1 hour. Similarly, after the job is done, using the following command can run a parallel properties calculation based on 'mgo-band.d3' and data from the previous 'mgo' SCF calculation on 12 CPUs:

<code>
~$ Pprop23 -in mgo-band.d3 -nc 12 -wt 00:30 -ref mgo
</code>

Alternatively, the user can also integrate 2 jobs into the same qsub file using <code>Xcrys23</code>.

<code>
~$ Xcrys23 -name mgo-band -nd 1 -x pcrys -in mgo.d12 -wt 01:00 -ref no -x pprop -in mgo-band.d3 -wt 00:30 -ref mgo
</code>

Note that -x, -in, -wt should have the same lengths while -ref should be either 0 or the same length as -x. In the case above, 'no' is a reserved keyword for no reference. It should be noted that specifying multiple -in, -wt and -ref flags in <code>Pcrys</code>, <code>MPPcrys</code>, <code>Pprop</code>, <code>Scrys</code> and <code>Sprop</code> will lead to error.

Besides, <code>Xcrys23</code> also supports user-defined MPI+executable command pairs in 'settings' file:

<code>
~$ Xcrys23 -x pcrys_other -in mgo.d12 -wt 01:00
</code>

For more detailed instructions, test cases and keyword list of settings file, please refer to the [https://github.com/cmsg-icl/HPC-job-submission/tree/main/Imperial-HPC-Job-Submission documentation].

== CRYSTAL23 v1 (GNU) ==
Compared to Intel version above, GNU version is slightly slower to allow for compatibility with MPPproperties. Available for CX1 and HX1.

=== CX1 version ===
* Compiling Env : gcc11.2.0, aocl4.0, mpich4.0.2
* Note: MP2 (CRYSCOR and CRYSTAL2) is currently not available.

The same job submission script is used. However, GNU version is not the default option for CX1. To enable GNU version, if there is a 'settings' file, change the following parameters:

* '''EXEDIR''' : <code>module load /rds/general/project/cmsg/live/etc/modulefiles/CRYSTAL/23v1-gcc</code>
* '''MPIDIR''' : <code>module load /rds/general/project/cmsg/live/etc/compiler/gcc11.2.0-aocl</code>.

Alternatively, rerun the configuration file (same as above) and specify 2 values during configuration.

Note that by default, there is no command for 'MPPproperties'. The user has to modify their <code>~/.bashrc</code> file to set the alias and 'settings' file for executable flags. An example:

# in ~/.bashrc
alias MPPprop23="/rds/general/project/cmsg/live/share/HPC-job-submission/Imperial-HPC-Job-Submission/gen_sub -x mppprop -set ${HOME}/etc/runCRYSTAL23/settings"
# in settings, keep the column width
mppprop mpiexec -np ${V_TPROC} MPPproperties Massive parallel properties calculation, OMP

=== HX1 version ===
* Compiling Env : EasyBuild foss/2023a
* Note: MP2 (CRYSCOR and CRYSTAL2) is currently not available.

* '''EXEDIR''' : No default. The directory where the user put their executables in.
* '''MPIDIR''' : <code>module load foss/2023a</code>

Alternatively, run the configuration file (The one with '-HX1' in the same directory of Github repo) and specify 2 values during configuration. Other options are consistent with CRYSTAL on CX1.

== CRYSTAL17 v2 (GNU)==
Not hosted on RDS-CMSG. Default for CX1. Not available for HX1.

=== CX1 version ===
* Compilor : gcc/6.2.0
* MPI : mpich/3.4.3

'''For job submission scripts'''

The configuration and usage are identical to CRYSTAL23. Please refer to the previous section and substitute '23' with '17'. Also, please be noted that CRYSCOR (MP2 calculation) is not available for CRYSTAL17 and 'MPP' as a massive parallel strategy is limited to 'crystal' calculations, i.e., only MPPcrystal is released.

== Quantum Espresso v7.2 (GNU) ==
Job submission file configured for CX1 and HX1, but only CX1 executable is shared as these executables are highly dependent on environment.

=== CX1 version ===
* Compiling Env : EasyBuild foss2022a
* libxc: Yes, version 5.1.2
* hdf5: No

Pseudopotential files are saved in <code>/rds/general/project/cmsg/live/etc/QE_PseudoP</code>.

''' Job submission script configuration '''

<code>
~$ bash /rds/general/project/cmsg/live/share/HPC-job-submission/Imperial-HPC-Job-Submission/QE7/config_QE7.sh
</code>

The procedure is the same as CRYSTAL. Type enter for default setups, which typically works fine. Please note that moving files between temporal and home directories are disabled as QE has built-in file management system.

''' Job submission script commands '''

{| class="wikitable" style="margin:auto"
|+ List of available commands
|-
! Command !! Definition
|-
| PWqe7 || Generate pw.x job submission files
|-
| PHqe7 || Generate ph.x job submission files
|-
| CPqe7 || Generate cp.x job submission files
|-
| PPqe7 || Generate pp.x job submission files
|-
| Xqe7 || Generate job submission files for user-defined executables
|-
| SETqe7 || Print the local 'settings' file
|-
| HELPqe7 || Print the instructions of commands
|}

== GULP v6.1.2 (GNU) ==
Default for CX1. Not available for HX1.

=== CX1 version ===
* Compiling Env : EasyBuild foss2022a
* OpenKIM: Yes, version 2.3.0
* PLUMED: Yes, version 2.9.0
* ALAMODE: Yes, git repo copied into <code>app/</code>

GULP force field libraries are saved in <code>/rds/general/project/cmsg/live/etc/GULP_Libraries</code>. Executable name: 'gulp-mpi'.

'''OpenKIM'''

KIM force field <ref>https://openkim.org/</ref> are saved in <code>/rds/general/project/cmsg/live/etc/KIM_Models</code>. To run OpenKIM executable and download new models, run:

<code>
~$ module load /rds/general/project/cmsg/live/etc/modulefiles/OpenKIM/2.3.0-foss
</code>

'''Job submission script configuration'''

<code>
~$ bash /rds/general/project/cmsg/live/share/HPC-job-submission/Imperial-HPC-Job-Submission/GULP6/config_GULP6.sh
</code>

Type enter for default setups, which typically works fine.

'''Job submission script commands'''

{| class="wikitable" style="margin:auto"
|+ List of available commands
|-
! Command !! Definition
|-
| Pglp6 || Generate job submission files for 'gulp-mpi'
|-
| Xglp6 || Generate job submission files for user defined executable (by default nothing else, so not useful).
|-
| SETglp6 || Print the local 'settings' file
|-
| HELPglp6|| Print the instructions of commands
|}

==References==
<references>

CMSG disk and Shared Software on CX1

2023-11-30T22:18:44Z

Hz1420: /* CX1 version */

General information of shared software available for CMSG group members on Imperial CX1 is summarized in this page. Software listed below, unless stated otherwise, is hosted under 'CMSG' project of Imperial Research Data Store (RDS) <ref>https://www.imperial.ac.uk/admin-services/ict/self-service/research-support/rcs/service-offering/rds/</ref> (referred as RDS-CMSG below), which is accessible for both login and computational nodes of Imperial HPC CX1. For Imperial HPC HX1, by the time this page is upated (Nov. 2023), it is not accessible there. Only login node can visit RDS-CMSG disk after the pilot phase <ref>https://wiki.imperial.ac.uk/display/HPC/HX1+Cluster</ref>. Therefore software on RDS-CMSG are classified into 2 catagories:

# For CX1, use <code>/rds/general/project/cmsg/live/app/</code> (executables, libraries and headers) and <code>/rds/general/project/cmsg/live/etc/</code> (other files, such as pseudopotential). Executable can be called directly from the user's home directory.
# For HX1, use <code>/rds/general/project/cmsg/live/share/</code>. Executables must be downloaded and uploaded to HX1. It is user's responsibility to set up a effective running environment according to 'readme' files (if there is one) saved in the same, or upper, directory.

''Please note that accessibility to CMSG disk is granted for CMSG group members only. If you need to access it, please contact group PI [[Contributors | Prof. Nicholas Harrison]].''

== CRYSTAL23 v1 (Intel) ==
The current default for CX1. Not available for HX1.

=== CX1 version ===
* Compiling Env : EasyBuild Intel 2023a
* Note: MPPproperties and MP2 (CRYSCOR and CRYSTAL2) are currently not available, which involves separate code packages (DMat2 and CRYSCOR). For MPPproperties please read the GNU version below.

The general job submission script for Imperial HPC, developed by the author himself, is used here.

''Thanks to Mr. K Tallat-Kelpsa for testing''

''' Configuration '''

The source code (written in bash) is available on group [https://github.com/cmsg-icl/HPC-job-submission/tree/main/Imperial-HPC-Job-Submission GitHub repo], though it is not needed in practice. Useful for developing new features. On CX1, use the following command to configure your local settings:

<code>
~$ bash /rds/general/project/cmsg/live/share/HPC-job-submission/Imperial-HPC-Job-Submission/CRYSTAL23/config_CRYSTAL23.sh
</code>

Then follow the instructions on the screen. Typically the default values are good. To use default vaules, press enter. After configuration is finished, the information is stored in a 'settings' file which is by default saved as <code>${HOME}/etc/runCRYSTAL23/settings</code>. It functions as a dictionary for job sumbmission scripts to refer to. It is also editable according to the user's needs.

Use the following command to activate alias commands:

<code>
~$ source ~/.bashrc
</code>

Due to the settings of CX1, the <code>source</code> command should be executed everytime you login. To aviod this, try the following commands:

~$ cat << EOF >> ~/.bash_profile
if test -f ~/.bashrc; then
source ~/.bashrc
fi
EOF

''' Quick References '''

{| class="wikitable" style="margin:auto"
|+ List of available commands
|-
! Command !! Definition
|-
| Pcrys23 || Generate parallel OMP CRYSTAL23 job submission files
|-
| MPPcrys23 || Generate massive parallel OMP CRYSTAL23 job submission files
|-
| Pprop23 || Generate parallel OMP CRYSTAL23 properties job submission files
|-
| Scrys23 || Generate serial OMP CRYSTAL23 job submission files
|-
| Sprop23 || Generate serial OMP CRYSTAL23 properties job submission files
|-
| Xcrys23 || User-defined executables and multiple jobs (see below for code examples)
|-
| SETcrys23 || Print the local 'settings' file
|-
| HELPcrys23 || Print the instructions of commands
|}

{| class="wikitable" style="margin:auto"
|+ List of in-line flags
|-
! Command !! Definition
|-
| -in || Input .d12 or .d3 files
|-
| -nd || Number of nodes requested. Number of CPUs per node will be read from the settings file if '-nc' is not set
|-
| -nc || Number of CPUs requested. Number of nodes will be automatically decided by Number of CPUs per node from settings file if '-nd' is not set
|-
| -nt || Number of threads per process. If not set the default value (1) is used.
|-
| -nt || Memory requested per node. If not set the default value (512GB) is used.
|-
| -wt || Walltime for the job. Or walltime for each job if multiple jobs
|-
| -ref || Optional, name of the reference file(s). No extension
|-
| -x || Xcrys23 only. The alias, i.e., lable, of MPI+executable in-line command pair
|-
| -name || Xcrys23 only. Set a common name for multiple jobs
|-
| -set || Developer only. The path to local 'settings' file
|}

{| class="wikitable" style="margin:auto"
|+ List of available labels for '-x' flag
|-
! Label !! Acutal in-line command
|-
| pcrys || mpiexec Pcrystal
|-
| mppcrys || mpiexec MPPcrystal
|-
| pprop || mpiexec Pproperties
|-
| scrys || Scrystal < INPUT
|-
| sprop || Sproperties < INPUT
|}

Note: New labels can be defined in local 'settings' file if a new MPI+executable in-line command is needed

''' Code Examples '''

To generate a qsub file for a parallel crystal23 job on 'mgo.d12', the following command can be used:

<code>
~$ Pcrys23 -in mgo.d12 -wt 01:00 -nd 1
</code>

That generates a 'mgo.qsub' file requesting for 1 node and the maximum running time is 1 hour. Similarly, after the job is done, using the following command can run a parallel properties calculation based on 'mgo-band.d3' and data from the previous 'mgo' SCF calculation on 12 CPUs:

<code>
~$ Pprop23 -in mgo-band.d3 -nc 12 -wt 00:30 -ref mgo
</code>

Alternatively, the user can also integrate 2 jobs into the same qsub file using <code>Xcrys23</code>.

<code>
~$ Xcrys23 -name mgo-band -nd 1 -x pcrys -in mgo.d12 -wt 01:00 -ref no -x pprop -in mgo-band.d3 -wt 00:30 -ref mgo
</code>

Note that -x, -in, -wt should have the same lengths while -ref should be either 0 or the same length as -x. In the case above, 'no' is a reserved keyword for no reference. It should be noted that specifying multiple -in, -wt and -ref flags in <code>Pcrys</code>, <code>MPPcrys</code>, <code>Pprop</code>, <code>Scrys</code> and <code>Sprop</code> will lead to error.

Besides, <code>Xcrys23</code> also supports user-defined MPI+executable command pairs in 'settings' file:

<code>
~$ Xcrys23 -x pcrys_other -in mgo.d12 -wt 01:00
</code>

For more detailed instructions, test cases and keyword list of settings file, please refer to the [https://github.com/cmsg-icl/HPC-job-submission/tree/main/Imperial-HPC-Job-Submission documentation].

== CRYSTAL23 v1 (GNU) ==
Compared to Intel version above, GNU version is slightly slower to allow for compatibility with MPPproperties. Available for CX1 and HX1.

=== CX1 version ===
* Compiling Env : gcc11.2.0, aocl4.0, mpich4.0.2
* Note: MP2 (CRYSCOR and CRYSTAL2) is currently not available.

The same job submission script is used. However, GNU version is not the default option for CX1. To enable GNU version, if there is a 'settings' file, change the following parameters:

* '''EXEDIR''' : <code>module load /rds/general/project/cmsg/live/etc/modulefiles/CRYSTAL/23v1-gcc</code>
* '''MPIDIR''' : <code>module load /rds/general/project/cmsg/live/etc/compiler/gcc11.2.0-aocl</code>.

Alternatively, rerun the configuration file (same as above) and specify 2 values during configuration.

Note that by default, there is no command for 'MPPproperties'. The user has to modify their <code>~/.bashrc</code> file to set the alias and 'settings' file for executable flags. An example:

# in ~/.bashrc
alias MPPprop23="/rds/general/project/cmsg/live/share/HPC-job-submission/Imperial-HPC-Job-Submission/gen_sub -x mppprop -set ${HOME}/etc/runCRYSTAL23/settings"
# in settings, keep the column width
mppprop mpiexec -np ${V_TPROC} MPPproperties Massive parallel properties calculation, OMP

=== HX1 version ===
* Compiling Env : EasyBuild foss/2023a
* Note: MP2 (CRYSCOR and CRYSTAL2) is currently not available.

* '''EXEDIR''' : No default. The directory where the user put their executables in.
* '''MPIDIR''' : <code>module load foss/2023a</code>

Alternatively, run the configuration file (The one with '-HX1' in the same directory of Github repo) and specify 2 values during configuration. Other options are consistent with CRYSTAL on CX1.

== CRYSTAL17 v2 (GNU)==
Not hosted on RDS-CMSG. Default for CX1. Not available for HX1.

=== CX1 version ===
* Compilor : gcc/6.2.0
* MPI : mpich/3.4.3

'''For job submission scripts'''

The configuration and usage are identical to CRYSTAL23. Please refer to the previous section and substitute '23' with '17'. Also, please be noted that CRYSCOR (MP2 calculation) is not available for CRYSTAL17 and 'MPP' as a massive parallel strategy is limited to 'crystal' calculations, i.e., only MPPcrystal is released.

== Quantum Espresso v7.2 (GNU) ==
Job submission file configured for CX1 and HX1, but only CX1 executable is shared as these executables are highly dependent on environment.

=== CX1 version ===
* Compiling Env : EasyBuild foss2022a
* libxc: Yes, version 5.1.2
* hdf5: No

Pseudopotential files are saved in <code>/rds/general/project/cmsg/live/etc/QE_PseudoP</code>.

''' Job submission script configuration '''

<code>
~$ bash /rds/general/project/cmsg/live/share/HPC-job-submission/Imperial-HPC-Job-Submission/QE7/config_QE7.sh
</code>

The procedure is the same as CRYSTAL. Type enter for default setups, which typically works fine. Please note that moving files between temporal and home directories are disabled as QE has built-in file management system.

''' Job submission script commands '''

{| class="wikitable" style="margin:auto"
|+ List of available commands
|-
! Command !! Definition
|-
| PWqe7 || Generate pw.x job submission files
|-
| PHqe7 || Generate ph.x job submission files
|-
| CPqe7 || Generate cp.x job submission files
|-
| PPqe7 || Generate pp.x job submission files
|-
| Xqe7 || Generate job submission files for user-defined executables
|-
| SETqe7 || Print the local 'settings' file
|-
| HELPqe7 || Print the instructions of commands
|}

== GULP v6.1.2 (GNU) ==
Default for CX1. Not available for HX1.

=== CX1 version ===
* Compiling Env : EasyBuild foss2022a
* OpenKIM: Yes, version 2.3.0
* PLUMED: Yes, version 2.9.0
* ALAMODE: Yes, git repo copied into <code>app/</code>

GULP force field libraries are saved in <code>/rds/general/project/cmsg/live/etc/GULP_Libraries</code>. Executable name: 'gulp-mpi'.

'''OpenKIM'''

KIM force field <ref>https://openkim.org/</ref> are saved in <code>/rds/general/project/cmsg/live/etc/KIM_Models</code>. To run OpenKIM executable and download new models, run:

<code>
~$ module load /rds/general/project/cmsg/live/etc/modulefiles/OpenKIM/2.3.0-foss
</code>

'''Job submission script configuration'''

<code>
~$ bash /rds/general/project/cmsg/live/share/HPC-job-submission/Imperial-HPC-Job-Submission/GULP6/config_GULP6.sh
</code>

Type enter for default setups, which typically works fine.

'''Job submission script commands'''

{| class="wikitable" style="margin:auto"
|+ List of available commands
|-
! Command !! Definition
|-
| Pglp6 || Generate job submission files for 'gulp-mpi'
|-
| Xglp6 || Generate job submission files for user defined executable (by default nothing else, so not useful).
|-
| SETglp6 || Print the local 'settings' file
|-
| HELPglp6|| Print the instructions of commands
|}

CMSG disk and Shared Software on CX1

2023-11-30T22:18:20Z

Hz1420: /* GULP v6.1.2 (GNU) */

General information of shared software available for CMSG group members on Imperial CX1 is summarized in this page. Software listed below, unless stated otherwise, is hosted under 'CMSG' project of Imperial Research Data Store (RDS) <ref>https://www.imperial.ac.uk/admin-services/ict/self-service/research-support/rcs/service-offering/rds/</ref> (referred as RDS-CMSG below), which is accessible for both login and computational nodes of Imperial HPC CX1. For Imperial HPC HX1, by the time this page is upated (Nov. 2023), it is not accessible there. Only login node can visit RDS-CMSG disk after the pilot phase <ref>https://wiki.imperial.ac.uk/display/HPC/HX1+Cluster</ref>. Therefore software on RDS-CMSG are classified into 2 catagories:

# For CX1, use <code>/rds/general/project/cmsg/live/app/</code> (executables, libraries and headers) and <code>/rds/general/project/cmsg/live/etc/</code> (other files, such as pseudopotential). Executable can be called directly from the user's home directory.
# For HX1, use <code>/rds/general/project/cmsg/live/share/</code>. Executables must be downloaded and uploaded to HX1. It is user's responsibility to set up a effective running environment according to 'readme' files (if there is one) saved in the same, or upper, directory.

''Please note that accessibility to CMSG disk is granted for CMSG group members only. If you need to access it, please contact group PI [[Contributors | Prof. Nicholas Harrison]].''

== CRYSTAL23 v1 (Intel) ==
The current default for CX1. Not available for HX1.

=== CX1 version ===
* Compiling Env : EasyBuild Intel 2023a
* Note: MPPproperties and MP2 (CRYSCOR and CRYSTAL2) are currently not available, which involves separate code packages (DMat2 and CRYSCOR). For MPPproperties please read the GNU version below.

The general job submission script for Imperial HPC, developed by the author himself, is used here.

''Thanks to Mr. K Tallat-Kelpsa for testing''

''' Configuration '''

The source code (written in bash) is available on group [https://github.com/cmsg-icl/HPC-job-submission/tree/main/Imperial-HPC-Job-Submission GitHub repo], though it is not needed in practice. Useful for developing new features. On CX1, use the following command to configure your local settings:

<code>
~$ bash /rds/general/project/cmsg/live/share/HPC-job-submission/Imperial-HPC-Job-Submission/CRYSTAL23/config_CRYSTAL23.sh
</code>

Then follow the instructions on the screen. Typically the default values are good. To use default vaules, press enter. After configuration is finished, the information is stored in a 'settings' file which is by default saved as <code>${HOME}/etc/runCRYSTAL23/settings</code>. It functions as a dictionary for job sumbmission scripts to refer to. It is also editable according to the user's needs.

Use the following command to activate alias commands:

<code>
~$ source ~/.bashrc
</code>

Due to the settings of CX1, the <code>source</code> command should be executed everytime you login. To aviod this, try the following commands:

~$ cat << EOF >> ~/.bash_profile
if test -f ~/.bashrc; then
source ~/.bashrc
fi
EOF

''' Quick References '''

{| class="wikitable" style="margin:auto"
|+ List of available commands
|-
! Command !! Definition
|-
| Pcrys23 || Generate parallel OMP CRYSTAL23 job submission files
|-
| MPPcrys23 || Generate massive parallel OMP CRYSTAL23 job submission files
|-
| Pprop23 || Generate parallel OMP CRYSTAL23 properties job submission files
|-
| Scrys23 || Generate serial OMP CRYSTAL23 job submission files
|-
| Sprop23 || Generate serial OMP CRYSTAL23 properties job submission files
|-
| Xcrys23 || User-defined executables and multiple jobs (see below for code examples)
|-
| SETcrys23 || Print the local 'settings' file
|-
| HELPcrys23 || Print the instructions of commands
|}

{| class="wikitable" style="margin:auto"
|+ List of in-line flags
|-
! Command !! Definition
|-
| -in || Input .d12 or .d3 files
|-
| -nd || Number of nodes requested. Number of CPUs per node will be read from the settings file if '-nc' is not set
|-
| -nc || Number of CPUs requested. Number of nodes will be automatically decided by Number of CPUs per node from settings file if '-nd' is not set
|-
| -nt || Number of threads per process. If not set the default value (1) is used.
|-
| -nt || Memory requested per node. If not set the default value (512GB) is used.
|-
| -wt || Walltime for the job. Or walltime for each job if multiple jobs
|-
| -ref || Optional, name of the reference file(s). No extension
|-
| -x || Xcrys23 only. The alias, i.e., lable, of MPI+executable in-line command pair
|-
| -name || Xcrys23 only. Set a common name for multiple jobs
|-
| -set || Developer only. The path to local 'settings' file
|}

{| class="wikitable" style="margin:auto"
|+ List of available labels for '-x' flag
|-
! Label !! Acutal in-line command
|-
| pcrys || mpiexec Pcrystal
|-
| mppcrys || mpiexec MPPcrystal
|-
| pprop || mpiexec Pproperties
|-
| scrys || Scrystal < INPUT
|-
| sprop || Sproperties < INPUT
|}

Note: New labels can be defined in local 'settings' file if a new MPI+executable in-line command is needed

''' Code Examples '''

To generate a qsub file for a parallel crystal23 job on 'mgo.d12', the following command can be used:

<code>
~$ Pcrys23 -in mgo.d12 -wt 01:00 -nd 1
</code>

That generates a 'mgo.qsub' file requesting for 1 node and the maximum running time is 1 hour. Similarly, after the job is done, using the following command can run a parallel properties calculation based on 'mgo-band.d3' and data from the previous 'mgo' SCF calculation on 12 CPUs:

<code>
~$ Pprop23 -in mgo-band.d3 -nc 12 -wt 00:30 -ref mgo
</code>

Alternatively, the user can also integrate 2 jobs into the same qsub file using <code>Xcrys23</code>.

<code>
~$ Xcrys23 -name mgo-band -nd 1 -x pcrys -in mgo.d12 -wt 01:00 -ref no -x pprop -in mgo-band.d3 -wt 00:30 -ref mgo
</code>

Note that -x, -in, -wt should have the same lengths while -ref should be either 0 or the same length as -x. In the case above, 'no' is a reserved keyword for no reference. It should be noted that specifying multiple -in, -wt and -ref flags in <code>Pcrys</code>, <code>MPPcrys</code>, <code>Pprop</code>, <code>Scrys</code> and <code>Sprop</code> will lead to error.

Besides, <code>Xcrys23</code> also supports user-defined MPI+executable command pairs in 'settings' file:

<code>
~$ Xcrys23 -x pcrys_other -in mgo.d12 -wt 01:00
</code>

For more detailed instructions, test cases and keyword list of settings file, please refer to the [https://github.com/cmsg-icl/HPC-job-submission/tree/main/Imperial-HPC-Job-Submission documentation].

== CRYSTAL23 v1 (GNU) ==
Compared to Intel version above, GNU version is slightly slower to allow for compatibility with MPPproperties. Available for CX1 and HX1.

=== CX1 version ===
* Compiling Env : gcc11.2.0, aocl4.0, mpich4.0.2
* Note: MP2 (CRYSCOR and CRYSTAL2) is currently not available.

The same job submission script is used. However, GNU version is not the default option for CX1. To enable GNU version, if there is a 'settings' file, change the following parameters:

* '''EXEDIR''' : <code>module load /rds/general/project/cmsg/live/etc/modulefiles/CRYSTAL/23v1-gcc</code>
* '''MPIDIR''' : <code>module load /rds/general/project/cmsg/live/etc/compiler/gcc11.2.0-aocl</code>.

Alternatively, rerun the configuration file (same as above) and specify 2 values during configuration.

Note that by default, there is no command for 'MPPproperties'. The user has to modify their <code>~/.bashrc</code> file to set the alias and 'settings' file for executable flags. An example:

# in ~/.bashrc
alias MPPprop23="/rds/general/project/cmsg/live/share/HPC-job-submission/Imperial-HPC-Job-Submission/gen_sub -x mppprop -set ${HOME}/etc/runCRYSTAL23/settings"
# in settings, keep the column width
mppprop mpiexec -np ${V_TPROC} MPPproperties Massive parallel properties calculation, OMP

=== HX1 version ===
* Compiling Env : EasyBuild foss/2023a
* Note: MP2 (CRYSCOR and CRYSTAL2) is currently not available.

* '''EXEDIR''' : No default. The directory where the user put their executables in.
* '''MPIDIR''' : <code>module load foss/2023a</code>

Alternatively, run the configuration file (The one with '-HX1' in the same directory of Github repo) and specify 2 values during configuration. Other options are consistent with CRYSTAL on CX1.

== CRYSTAL17 v2 (GNU)==
Not hosted on RDS-CMSG. Default for CX1. Not available for HX1.

=== CX1 version ===
* Compilor : gcc/6.2.0
* MPI : mpich/3.4.3

'''For job submission scripts'''

The configuration and usage are identical to CRYSTAL23. Please refer to the previous section and substitute '23' with '17'. Also, please be noted that CRYSCOR (MP2 calculation) is not available for CRYSTAL17 and 'MPP' as a massive parallel strategy is limited to 'crystal' calculations, i.e., only MPPcrystal is released.

== Quantum Espresso v7.2 (GNU) ==
Job submission file configured for CX1 and HX1, but only CX1 executable is shared as these executables are highly dependent on environment.

=== CX1 version ===
* Compiling Env : EasyBuild foss2022a
* libxc: Yes, version 5.1.2
* hdf5: No

Pseudopotential files are saved in <code>/rds/general/project/cmsg/live/etc/QE_PseudoP</code>.

''' Job submission script configuration '''

<code>
~$ bash /rds/general/project/cmsg/live/share/HPC-job-submission/Imperial-HPC-Job-Submission/QE7/config_QE7.sh
</code>

The procedure is the same as CRYSTAL. Please note that moving files between temporal and home directories are disabled as QE has built-in file management system.

''' Job submission script commands '''

{| class="wikitable" style="margin:auto"
|+ List of available commands
|-
! Command !! Definition
|-
| PWqe7 || Generate pw.x job submission files
|-
| PHqe7 || Generate ph.x job submission files
|-
| CPqe7 || Generate cp.x job submission files
|-
| PPqe7 || Generate pp.x job submission files
|-
| Xqe7 || Generate job submission files for user-defined executables
|-
| SETqe7 || Print the local 'settings' file
|-
| HELPqe7 || Print the instructions of commands
|}

== GULP v6.1.2 (GNU) ==
Default for CX1. Not available for HX1.

=== CX1 version ===
* Compiling Env : EasyBuild foss2022a
* OpenKIM: Yes, version 2.3.0
* PLUMED: Yes, version 2.9.0
* ALAMODE: Yes, git repo copied into <code>app/</code>

GULP force field libraries are saved in <code>/rds/general/project/cmsg/live/etc/GULP_Libraries</code>. Executable name: 'gulp-mpi'.

'''OpenKIM'''

KIM force field <ref>https://openkim.org/</ref> are saved in <code>/rds/general/project/cmsg/live/etc/KIM_Models</code>. To run OpenKIM executable and download new models, run:

<code>
~$ module load /rds/general/project/cmsg/live/etc/modulefiles/OpenKIM/2.3.0-foss
</code>

'''Job submission script configuration'''

<code>
~$ bash /rds/general/project/cmsg/live/share/HPC-job-submission/Imperial-HPC-Job-Submission/GULP6/config_GULP6.sh
</code>

Type enter for default setups, which typically works fine.

'''Job submission script commands'''

{| class="wikitable" style="margin:auto"
|+ List of available commands
|-
! Command !! Definition
|-
| Pglp6 || Generate job submission files for 'gulp-mpi'
|-
| Xglp6 || Generate job submission files for user defined executable (by default nothing else, so not useful).
|-
| SETglp6 || Print the local 'settings' file
|-
| HELPglp6|| Print the instructions of commands
|}

CMSG disk and Shared Software on CX1

2023-11-30T22:17:42Z

Hz1420: /* GULP v6.1.2 (GNU) */

General information of shared software available for CMSG group members on Imperial CX1 is summarized in this page. Software listed below, unless stated otherwise, is hosted under 'CMSG' project of Imperial Research Data Store (RDS) <ref>https://www.imperial.ac.uk/admin-services/ict/self-service/research-support/rcs/service-offering/rds/</ref> (referred as RDS-CMSG below), which is accessible for both login and computational nodes of Imperial HPC CX1. For Imperial HPC HX1, by the time this page is upated (Nov. 2023), it is not accessible there. Only login node can visit RDS-CMSG disk after the pilot phase <ref>https://wiki.imperial.ac.uk/display/HPC/HX1+Cluster</ref>. Therefore software on RDS-CMSG are classified into 2 catagories:

# For CX1, use <code>/rds/general/project/cmsg/live/app/</code> (executables, libraries and headers) and <code>/rds/general/project/cmsg/live/etc/</code> (other files, such as pseudopotential). Executable can be called directly from the user's home directory.
# For HX1, use <code>/rds/general/project/cmsg/live/share/</code>. Executables must be downloaded and uploaded to HX1. It is user's responsibility to set up a effective running environment according to 'readme' files (if there is one) saved in the same, or upper, directory.

''Please note that accessibility to CMSG disk is granted for CMSG group members only. If you need to access it, please contact group PI [[Contributors | Prof. Nicholas Harrison]].''

== CRYSTAL23 v1 (Intel) ==
The current default for CX1. Not available for HX1.

=== CX1 version ===
* Compiling Env : EasyBuild Intel 2023a
* Note: MPPproperties and MP2 (CRYSCOR and CRYSTAL2) are currently not available, which involves separate code packages (DMat2 and CRYSCOR). For MPPproperties please read the GNU version below.

The general job submission script for Imperial HPC, developed by the author himself, is used here.

''Thanks to Mr. K Tallat-Kelpsa for testing''

''' Configuration '''

The source code (written in bash) is available on group [https://github.com/cmsg-icl/HPC-job-submission/tree/main/Imperial-HPC-Job-Submission GitHub repo], though it is not needed in practice. Useful for developing new features. On CX1, use the following command to configure your local settings:

<code>
~$ bash /rds/general/project/cmsg/live/share/HPC-job-submission/Imperial-HPC-Job-Submission/CRYSTAL23/config_CRYSTAL23.sh
</code>

Then follow the instructions on the screen. Typically the default values are good. To use default vaules, press enter. After configuration is finished, the information is stored in a 'settings' file which is by default saved as <code>${HOME}/etc/runCRYSTAL23/settings</code>. It functions as a dictionary for job sumbmission scripts to refer to. It is also editable according to the user's needs.

Use the following command to activate alias commands:

<code>
~$ source ~/.bashrc
</code>

Due to the settings of CX1, the <code>source</code> command should be executed everytime you login. To aviod this, try the following commands:

~$ cat << EOF >> ~/.bash_profile
if test -f ~/.bashrc; then
source ~/.bashrc
fi
EOF

''' Quick References '''

{| class="wikitable" style="margin:auto"
|+ List of available commands
|-
! Command !! Definition
|-
| Pcrys23 || Generate parallel OMP CRYSTAL23 job submission files
|-
| MPPcrys23 || Generate massive parallel OMP CRYSTAL23 job submission files
|-
| Pprop23 || Generate parallel OMP CRYSTAL23 properties job submission files
|-
| Scrys23 || Generate serial OMP CRYSTAL23 job submission files
|-
| Sprop23 || Generate serial OMP CRYSTAL23 properties job submission files
|-
| Xcrys23 || User-defined executables and multiple jobs (see below for code examples)
|-
| SETcrys23 || Print the local 'settings' file
|-
| HELPcrys23 || Print the instructions of commands
|}

{| class="wikitable" style="margin:auto"
|+ List of in-line flags
|-
! Command !! Definition
|-
| -in || Input .d12 or .d3 files
|-
| -nd || Number of nodes requested. Number of CPUs per node will be read from the settings file if '-nc' is not set
|-
| -nc || Number of CPUs requested. Number of nodes will be automatically decided by Number of CPUs per node from settings file if '-nd' is not set
|-
| -nt || Number of threads per process. If not set the default value (1) is used.
|-
| -nt || Memory requested per node. If not set the default value (512GB) is used.
|-
| -wt || Walltime for the job. Or walltime for each job if multiple jobs
|-
| -ref || Optional, name of the reference file(s). No extension
|-
| -x || Xcrys23 only. The alias, i.e., lable, of MPI+executable in-line command pair
|-
| -name || Xcrys23 only. Set a common name for multiple jobs
|-
| -set || Developer only. The path to local 'settings' file
|}

{| class="wikitable" style="margin:auto"
|+ List of available labels for '-x' flag
|-
! Label !! Acutal in-line command
|-
| pcrys || mpiexec Pcrystal
|-
| mppcrys || mpiexec MPPcrystal
|-
| pprop || mpiexec Pproperties
|-
| scrys || Scrystal < INPUT
|-
| sprop || Sproperties < INPUT
|}

Note: New labels can be defined in local 'settings' file if a new MPI+executable in-line command is needed

''' Code Examples '''

To generate a qsub file for a parallel crystal23 job on 'mgo.d12', the following command can be used:

<code>
~$ Pcrys23 -in mgo.d12 -wt 01:00 -nd 1
</code>

That generates a 'mgo.qsub' file requesting for 1 node and the maximum running time is 1 hour. Similarly, after the job is done, using the following command can run a parallel properties calculation based on 'mgo-band.d3' and data from the previous 'mgo' SCF calculation on 12 CPUs:

<code>
~$ Pprop23 -in mgo-band.d3 -nc 12 -wt 00:30 -ref mgo
</code>

Alternatively, the user can also integrate 2 jobs into the same qsub file using <code>Xcrys23</code>.

<code>
~$ Xcrys23 -name mgo-band -nd 1 -x pcrys -in mgo.d12 -wt 01:00 -ref no -x pprop -in mgo-band.d3 -wt 00:30 -ref mgo
</code>

Note that -x, -in, -wt should have the same lengths while -ref should be either 0 or the same length as -x. In the case above, 'no' is a reserved keyword for no reference. It should be noted that specifying multiple -in, -wt and -ref flags in <code>Pcrys</code>, <code>MPPcrys</code>, <code>Pprop</code>, <code>Scrys</code> and <code>Sprop</code> will lead to error.

Besides, <code>Xcrys23</code> also supports user-defined MPI+executable command pairs in 'settings' file:

<code>
~$ Xcrys23 -x pcrys_other -in mgo.d12 -wt 01:00
</code>

For more detailed instructions, test cases and keyword list of settings file, please refer to the [https://github.com/cmsg-icl/HPC-job-submission/tree/main/Imperial-HPC-Job-Submission documentation].

== CRYSTAL23 v1 (GNU) ==
Compared to Intel version above, GNU version is slightly slower to allow for compatibility with MPPproperties. Available for CX1 and HX1.

=== CX1 version ===
* Compiling Env : gcc11.2.0, aocl4.0, mpich4.0.2
* Note: MP2 (CRYSCOR and CRYSTAL2) is currently not available.

The same job submission script is used. However, GNU version is not the default option for CX1. To enable GNU version, if there is a 'settings' file, change the following parameters:

* '''EXEDIR''' : <code>module load /rds/general/project/cmsg/live/etc/modulefiles/CRYSTAL/23v1-gcc</code>
* '''MPIDIR''' : <code>module load /rds/general/project/cmsg/live/etc/compiler/gcc11.2.0-aocl</code>.

Alternatively, rerun the configuration file (same as above) and specify 2 values during configuration.

Note that by default, there is no command for 'MPPproperties'. The user has to modify their <code>~/.bashrc</code> file to set the alias and 'settings' file for executable flags. An example:

# in ~/.bashrc
alias MPPprop23="/rds/general/project/cmsg/live/share/HPC-job-submission/Imperial-HPC-Job-Submission/gen_sub -x mppprop -set ${HOME}/etc/runCRYSTAL23/settings"
# in settings, keep the column width
mppprop mpiexec -np ${V_TPROC} MPPproperties Massive parallel properties calculation, OMP

=== HX1 version ===
* Compiling Env : EasyBuild foss/2023a
* Note: MP2 (CRYSCOR and CRYSTAL2) is currently not available.

* '''EXEDIR''' : No default. The directory where the user put their executables in.
* '''MPIDIR''' : <code>module load foss/2023a</code>

Alternatively, run the configuration file (The one with '-HX1' in the same directory of Github repo) and specify 2 values during configuration. Other options are consistent with CRYSTAL on CX1.

== CRYSTAL17 v2 (GNU)==
Not hosted on RDS-CMSG. Default for CX1. Not available for HX1.

=== CX1 version ===
* Compilor : gcc/6.2.0
* MPI : mpich/3.4.3

'''For job submission scripts'''

The configuration and usage are identical to CRYSTAL23. Please refer to the previous section and substitute '23' with '17'. Also, please be noted that CRYSCOR (MP2 calculation) is not available for CRYSTAL17 and 'MPP' as a massive parallel strategy is limited to 'crystal' calculations, i.e., only MPPcrystal is released.

== Quantum Espresso v7.2 (GNU) ==
Job submission file configured for CX1 and HX1, but only CX1 executable is shared as these executables are highly dependent on environment.

=== CX1 version ===
* Compiling Env : EasyBuild foss2022a
* libxc: Yes, version 5.1.2
* hdf5: No

Pseudopotential files are saved in <code>/rds/general/project/cmsg/live/etc/QE_PseudoP</code>.

''' Job submission script configuration '''

<code>
~$ bash /rds/general/project/cmsg/live/share/HPC-job-submission/Imperial-HPC-Job-Submission/QE7/config_QE7.sh
</code>

The procedure is the same as CRYSTAL. Please note that moving files between temporal and home directories are disabled as QE has built-in file management system.

''' Job submission script commands '''

{| class="wikitable" style="margin:auto"
|+ List of available commands
|-
! Command !! Definition
|-
| PWqe7 || Generate pw.x job submission files
|-
| PHqe7 || Generate ph.x job submission files
|-
| CPqe7 || Generate cp.x job submission files
|-
| PPqe7 || Generate pp.x job submission files
|-
| Xqe7 || Generate job submission files for user-defined executables
|-
| SETqe7 || Print the local 'settings' file
|-
| HELPqe7 || Print the instructions of commands
|}

== GULP v6.1.2 (GNU) ==
Default for CX1. Not available for HX1.

=== CX1 version ===
* Compiling Env : EasyBuild foss2022a
* OpenKIM: Yes, version 2.3.0
* PLUMED: Yes, version 2.9.0
* ALAMODE: Yes, git repo copied into <code>app/</code>

GULP force field libraries are saved in <code>/rds/general/project/cmsg/live/etc/GULP_Libraries</code>. Executable name: 'gulp-mpi'.

'''OpenKIM'''

KIM force field <ref>https://openkim.org/</ref> are saved in <code>/rds/general/project/cmsg/live/etc/KIM_Models</code>. To run OpenKIM executable and download new models, run:

<code>
~$ module load /rds/general/project/cmsg/live/etc/modulefiles/OpenKIM/2.3.0-foss
</code>

'''Job submission script configuration'''

<code>
~$ bash /rds/general/project/cmsg/live/share/HPC-job-submission/Imperial-HPC-Job-Submission/GULP6/config_GULP6.sh
</code>

Type enter for default setups, which typically works fine.

'''Job submission script commands'''

{| class="wikitable" style="margin:auto"
|+ List of available commands
|-
! Command !! Definition
|-
| Pglp6 || Generate job submission files for 'gulp-mpi'
|-
| Xglp6 || Generate job submission files for user defined executable (by default nothing else).
|-
| SETglp6 || Print the local 'settings' file
|-
| HELPglp6|| Print the instructions of commands
|}

CMSG disk and Shared Software on CX1

2023-11-30T22:05:36Z

Hz1420:

General information of shared software available for CMSG group members on Imperial CX1 is summarized in this page. Software listed below, unless stated otherwise, is hosted under 'CMSG' project of Imperial Research Data Store (RDS) <ref>https://www.imperial.ac.uk/admin-services/ict/self-service/research-support/rcs/service-offering/rds/</ref> (referred as RDS-CMSG below), which is accessible for both login and computational nodes of Imperial HPC CX1. For Imperial HPC HX1, by the time this page is upated (Nov. 2023), it is not accessible there. Only login node can visit RDS-CMSG disk after the pilot phase <ref>https://wiki.imperial.ac.uk/display/HPC/HX1+Cluster</ref>. Therefore software on RDS-CMSG are classified into 2 catagories:

# For CX1, use <code>/rds/general/project/cmsg/live/app/</code> (executables, libraries and headers) and <code>/rds/general/project/cmsg/live/etc/</code> (other files, such as pseudopotential). Executable can be called directly from the user's home directory.
# For HX1, use <code>/rds/general/project/cmsg/live/share/</code>. Executables must be downloaded and uploaded to HX1. It is user's responsibility to set up a effective running environment according to 'readme' files (if there is one) saved in the same, or upper, directory.

''Please note that accessibility to CMSG disk is granted for CMSG group members only. If you need to access it, please contact group PI [[Contributors | Prof. Nicholas Harrison]].''

== CRYSTAL23 v1 (Intel) ==
The current default for CX1. Not available for HX1.

=== CX1 version ===
* Compiling Env : EasyBuild Intel 2023a
* Note: MPPproperties and MP2 (CRYSCOR and CRYSTAL2) are currently not available, which involves separate code packages (DMat2 and CRYSCOR). For MPPproperties please read the GNU version below.

The general job submission script for Imperial HPC, developed by the author himself, is used here.

''Thanks to Mr. K Tallat-Kelpsa for testing''

''' Configuration '''

The source code (written in bash) is available on group [https://github.com/cmsg-icl/HPC-job-submission/tree/main/Imperial-HPC-Job-Submission GitHub repo], though it is not needed in practice. Useful for developing new features. On CX1, use the following command to configure your local settings:

<code>
~$ bash /rds/general/project/cmsg/live/share/HPC-job-submission/Imperial-HPC-Job-Submission/CRYSTAL23/config_CRYSTAL23.sh
</code>

Then follow the instructions on the screen. Typically the default values are good. To use default vaules, press enter. After configuration is finished, the information is stored in a 'settings' file which is by default saved as <code>${HOME}/etc/runCRYSTAL23/settings</code>. It functions as a dictionary for job sumbmission scripts to refer to. It is also editable according to the user's needs.

Use the following command to activate alias commands:

<code>
~$ source ~/.bashrc
</code>

Due to the settings of CX1, the <code>source</code> command should be executed everytime you login. To aviod this, try the following commands:

~$ cat << EOF >> ~/.bash_profile
if test -f ~/.bashrc; then
source ~/.bashrc
fi
EOF

''' Quick References '''

{| class="wikitable" style="margin:auto"
|+ List of available commands
|-
! Command !! Definition
|-
| Pcrys23 || Generate parallel OMP CRYSTAL23 job submission files
|-
| MPPcrys23 || Generate massive parallel OMP CRYSTAL23 job submission files
|-
| Pprop23 || Generate parallel OMP CRYSTAL23 properties job submission files
|-
| Scrys23 || Generate serial OMP CRYSTAL23 job submission files
|-
| Sprop23 || Generate serial OMP CRYSTAL23 properties job submission files
|-
| Xcrys23 || User-defined executables and multiple jobs (see below for code examples)
|-
| SETcrys23 || Print the local 'settings' file
|-
| HELPcrys23 || Print the instructions of commands
|}

{| class="wikitable" style="margin:auto"
|+ List of in-line flags
|-
! Command !! Definition
|-
| -in || Input .d12 or .d3 files
|-
| -nd || Number of nodes requested. Number of CPUs per node will be read from the settings file if '-nc' is not set
|-
| -nc || Number of CPUs requested. Number of nodes will be automatically decided by Number of CPUs per node from settings file if '-nd' is not set
|-
| -nt || Number of threads per process. If not set the default value (1) is used.
|-
| -nt || Memory requested per node. If not set the default value (512GB) is used.
|-
| -wt || Walltime for the job. Or walltime for each job if multiple jobs
|-
| -ref || Optional, name of the reference file(s). No extension
|-
| -x || Xcrys23 only. The alias, i.e., lable, of MPI+executable in-line command pair
|-
| -name || Xcrys23 only. Set a common name for multiple jobs
|-
| -set || Developer only. The path to local 'settings' file
|}

{| class="wikitable" style="margin:auto"
|+ List of available labels for '-x' flag
|-
! Label !! Acutal in-line command
|-
| pcrys || mpiexec Pcrystal
|-
| mppcrys || mpiexec MPPcrystal
|-
| pprop || mpiexec Pproperties
|-
| scrys || Scrystal < INPUT
|-
| sprop || Sproperties < INPUT
|}

Note: New labels can be defined in local 'settings' file if a new MPI+executable in-line command is needed

''' Code Examples '''

To generate a qsub file for a parallel crystal23 job on 'mgo.d12', the following command can be used:

<code>
~$ Pcrys23 -in mgo.d12 -wt 01:00 -nd 1
</code>

That generates a 'mgo.qsub' file requesting for 1 node and the maximum running time is 1 hour. Similarly, after the job is done, using the following command can run a parallel properties calculation based on 'mgo-band.d3' and data from the previous 'mgo' SCF calculation on 12 CPUs:

<code>
~$ Pprop23 -in mgo-band.d3 -nc 12 -wt 00:30 -ref mgo
</code>

Alternatively, the user can also integrate 2 jobs into the same qsub file using <code>Xcrys23</code>.

<code>
~$ Xcrys23 -name mgo-band -nd 1 -x pcrys -in mgo.d12 -wt 01:00 -ref no -x pprop -in mgo-band.d3 -wt 00:30 -ref mgo
</code>

Note that -x, -in, -wt should have the same lengths while -ref should be either 0 or the same length as -x. In the case above, 'no' is a reserved keyword for no reference. It should be noted that specifying multiple -in, -wt and -ref flags in <code>Pcrys</code>, <code>MPPcrys</code>, <code>Pprop</code>, <code>Scrys</code> and <code>Sprop</code> will lead to error.

Besides, <code>Xcrys23</code> also supports user-defined MPI+executable command pairs in 'settings' file:

<code>
~$ Xcrys23 -x pcrys_other -in mgo.d12 -wt 01:00
</code>

For more detailed instructions, test cases and keyword list of settings file, please refer to the [https://github.com/cmsg-icl/HPC-job-submission/tree/main/Imperial-HPC-Job-Submission documentation].

== CRYSTAL23 v1 (GNU) ==
Compared to Intel version above, GNU version is slightly slower to allow for compatibility with MPPproperties. Available for CX1 and HX1.

=== CX1 version ===
* Compiling Env : gcc11.2.0, aocl4.0, mpich4.0.2
* Note: MP2 (CRYSCOR and CRYSTAL2) is currently not available.

The same job submission script is used. However, GNU version is not the default option for CX1. To enable GNU version, if there is a 'settings' file, change the following parameters:

* '''EXEDIR''' : <code>module load /rds/general/project/cmsg/live/etc/modulefiles/CRYSTAL/23v1-gcc</code>
* '''MPIDIR''' : <code>module load /rds/general/project/cmsg/live/etc/compiler/gcc11.2.0-aocl</code>.

Alternatively, rerun the configuration file (same as above) and specify 2 values during configuration.

Note that by default, there is no command for 'MPPproperties'. The user has to modify their <code>~/.bashrc</code> file to set the alias and 'settings' file for executable flags. An example:

# in ~/.bashrc
alias MPPprop23="/rds/general/project/cmsg/live/share/HPC-job-submission/Imperial-HPC-Job-Submission/gen_sub -x mppprop -set ${HOME}/etc/runCRYSTAL23/settings"
# in settings, keep the column width
mppprop mpiexec -np ${V_TPROC} MPPproperties Massive parallel properties calculation, OMP

=== HX1 version ===
* Compiling Env : EasyBuild foss/2023a
* Note: MP2 (CRYSCOR and CRYSTAL2) is currently not available.

* '''EXEDIR''' : No default. The directory where the user put their executables in.
* '''MPIDIR''' : <code>module load foss/2023a</code>

Alternatively, run the configuration file (The one with '-HX1' in the same directory of Github repo) and specify 2 values during configuration. Other options are consistent with CRYSTAL on CX1.

== CRYSTAL17 v2 (GNU)==
Not hosted on RDS-CMSG. Default for CX1. Not available for HX1.

=== CX1 version ===
* Compilor : gcc/6.2.0
* MPI : mpich/3.4.3

'''For job submission scripts'''

The configuration and usage are identical to CRYSTAL23. Please refer to the previous section and substitute '23' with '17'. Also, please be noted that CRYSCOR (MP2 calculation) is not available for CRYSTAL17 and 'MPP' as a massive parallel strategy is limited to 'crystal' calculations, i.e., only MPPcrystal is released.

== Quantum Espresso v7.2 (GNU) ==
Job submission file configured for CX1 and HX1, but only CX1 executable is shared as these executables are highly dependent on environment.

=== CX1 version ===
* Compiling Env : EasyBuild foss2022a
* libxc: Yes, version 5.1.2
* hdf5: No

Pseudopotential files are saved in <code>/rds/general/project/cmsg/live/etc/QE_PseudoP</code>.

''' Job submission script configuration '''

<code>
~$ bash /rds/general/project/cmsg/live/share/HPC-job-submission/Imperial-HPC-Job-Submission/QE7/config_QE7.sh
</code>

The procedure is the same as CRYSTAL. Please note that moving files between temporal and home directories are disabled as QE has built-in file management system.

''' Job submission script commands '''

{| class="wikitable" style="margin:auto"
|+ List of available commands
|-
! Command !! Definition
|-
| PWqe7 || Generate pw.x job submission files
|-
| PHqe7 || Generate ph.x job submission files
|-
| CPqe7 || Generate cp.x job submission files
|-
| PPqe7 || Generate pp.x job submission files
|-
| Xqe7 || Generate job submission files for user-defined executables
|-
| SETqe7 || Print the local 'settings' file
|-
| HELPqe7 || Print the instructions of commands
|}

== GULP v6.1.2 (GNU) ==
Default for CX1. Not available for HX1.

=== CX1 version ===
* Compiling Env : EasyBuild foss2022a
* OpenKIM: Yes, version 5.1.2
* hdf5: No

CMSG disk and Shared Software on CX1

2023-11-30T22:00:08Z

Hz1420: /* Quantum Espresso v7.2 (GNU) */

General information of shared software available for CMSG group members on Imperial CX1 is summarized in this page. Software listed below, unless stated otherwise, is hosted under 'CMSG' project of Imperial Research Data Store (RDS) <ref>https://www.imperial.ac.uk/admin-services/ict/self-service/research-support/rcs/service-offering/rds/</ref> (referred as RDS-CMSG below), which is accessible for both login and computational nodes of Imperial HPC CX1. For Imperial HPC HX1, by the time this page is upated (Nov. 2023), it is not accessible there. Only login node can visit RDS-CMSG disk after the pilot phase <ref>https://wiki.imperial.ac.uk/display/HPC/HX1+Cluster</ref>. Therefore software on RDS-CMSG are classified into 2 catagories:

# For CX1, use <code>/rds/general/project/cmsg/live/app/</code> (executables, libraries and headers) and <code>/rds/general/project/cmsg/live/etc/</code> (other files, such as pseudopotential). Executable can be called directly from the user's home directory.
# For HX1, use <code>/rds/general/project/cmsg/live/share/</code>. Executables must be downloaded and uploaded to HX1. It is user's responsibility to set up a effective running environment according to 'readme' files (if there is one) saved in the same, or upper, directory.

''Please note that accessibility to CMSG disk is granted for CMSG group members only. If you need to access it, please contact group PI [[Contributors | Prof. Nicholas Harrison]].''

== CRYSTAL23 v1 (Intel) ==
The current default for CX1.

=== CX1 version ===
* Compiling Env : EasyBuild Intel 2023a
* Note: MPPproperties and MP2 (CRYSCOR and CRYSTAL2) are currently not available, which involves separate code packages (DMat2 and CRYSCOR). For MPPproperties please read the GNU version below.

The general job submission script for Imperial HPC, developed by the author himself, is used here.

''Thanks to Mr. K Tallat-Kelpsa for testing''

''' Configuration '''

The source code (written in bash) is available on group [https://github.com/cmsg-icl/HPC-job-submission/tree/main/Imperial-HPC-Job-Submission GitHub repo], though it is not needed in practice. Useful for developing new features. On CX1, use the following command to configure your local settings:

<code>
~$ bash /rds/general/project/cmsg/live/share/HPC-job-submission/Imperial-HPC-Job-Submission/CRYSTAL23/config_CRYSTAL23.sh
</code>

Then follow the instructions on the screen. Typically the default values are good. To use default vaules, press enter. After configuration is finished, the information is stored in a 'settings' file which is by default saved as <code>${HOME}/etc/runCRYSTAL23/settings</code>. It functions as a dictionary for job sumbmission scripts to refer to. It is also editable according to the user's needs.

Use the following command to activate alias commands:

<code>
~$ source ~/.bashrc
</code>

Due to the settings of CX1, the <code>source</code> command should be executed everytime you login. To aviod this, try the following commands:

~$ cat << EOF >> ~/.bash_profile
if test -f ~/.bashrc; then
source ~/.bashrc
fi
EOF

''' Quick References '''

{| class="wikitable" style="margin:auto"
|+ List of available commands
|-
! Command !! Definition
|-
| Pcrys23 || Generate parallel OMP CRYSTAL23 job submission files
|-
| MPPcrys23 || Generate massive parallel OMP CRYSTAL23 job submission files
|-
| Pprop23 || Generate parallel OMP CRYSTAL23 properties job submission files
|-
| Scrys23 || Generate serial OMP CRYSTAL23 job submission files
|-
| Sprop23 || Generate serial OMP CRYSTAL23 properties job submission files
|-
| Xcrys23 || User-defined executables and multiple jobs (see below for code examples)
|-
| SETcrys23 || Print the local 'settings' file
|-
| HELPcrys23 || Print the instructions of commands
|}

{| class="wikitable" style="margin:auto"
|+ List of in-line flags
|-
! Command !! Definition
|-
| -in || Input .d12 or .d3 files
|-
| -nd || Number of nodes requested. Number of CPUs per node will be read from the settings file if '-nc' is not set
|-
| -nc || Number of CPUs requested. Number of nodes will be automatically decided by Number of CPUs per node from settings file if '-nd' is not set
|-
| -nt || Number of threads per process. If not set the default value (1) is used.
|-
| -nt || Memory requested per node. If not set the default value (512GB) is used.
|-
| -wt || Walltime for the job. Or walltime for each job if multiple jobs
|-
| -ref || Optional, name of the reference file(s). No extension
|-
| -x || Xcrys23 only. The alias, i.e., lable, of MPI+executable in-line command pair
|-
| -name || Xcrys23 only. Set a common name for multiple jobs
|-
| -set || Developer only. The path to local 'settings' file
|}

{| class="wikitable" style="margin:auto"
|+ List of available labels for '-x' flag
|-
! Label !! Acutal in-line command
|-
| pcrys || mpiexec Pcrystal
|-
| mppcrys || mpiexec MPPcrystal
|-
| pprop || mpiexec Pproperties
|-
| scrys || Scrystal < INPUT
|-
| sprop || Sproperties < INPUT
|}

Note: New labels can be defined in local 'settings' file if a new MPI+executable in-line command is needed

''' Code Examples '''

To generate a qsub file for a parallel crystal23 job on 'mgo.d12', the following command can be used:

<code>
~$ Pcrys23 -in mgo.d12 -wt 01:00 -nd 1
</code>

That generates a 'mgo.qsub' file requesting for 1 node and the maximum running time is 1 hour. Similarly, after the job is done, using the following command can run a parallel properties calculation based on 'mgo-band.d3' and data from the previous 'mgo' SCF calculation on 12 CPUs:

<code>
~$ Pprop23 -in mgo-band.d3 -nc 12 -wt 00:30 -ref mgo
</code>

Alternatively, the user can also integrate 2 jobs into the same qsub file using <code>Xcrys23</code>.

<code>
~$ Xcrys23 -name mgo-band -nd 1 -x pcrys -in mgo.d12 -wt 01:00 -ref no -x pprop -in mgo-band.d3 -wt 00:30 -ref mgo
</code>

Note that -x, -in, -wt should have the same lengths while -ref should be either 0 or the same length as -x. In the case above, 'no' is a reserved keyword for no reference. It should be noted that specifying multiple -in, -wt and -ref flags in <code>Pcrys</code>, <code>MPPcrys</code>, <code>Pprop</code>, <code>Scrys</code> and <code>Sprop</code> will lead to error.

Besides, <code>Xcrys23</code> also supports user-defined MPI+executable command pairs in 'settings' file:

<code>
~$ Xcrys23 -x pcrys_other -in mgo.d12 -wt 01:00
</code>

For more detailed instructions, test cases and keyword list of settings file, please refer to the [https://github.com/cmsg-icl/HPC-job-submission/tree/main/Imperial-HPC-Job-Submission documentation].

== CRYSTAL23 v1 (GNU) ==
Compared to Intel version above, GNU version is slightly slower to allow for compatibility with MPPproperties.

=== CX1 version ===
* Compiling Env : gcc11.2.0, aocl4.0, mpich4.0.2
* Note: MP2 (CRYSCOR and CRYSTAL2) is currently not available.

The same job submission script is used. However, GNU version is not the default option for CX1. To enable GNU version, if there is a 'settings' file, change the following parameters:

* '''EXEDIR''' : <code>module load /rds/general/project/cmsg/live/etc/modulefiles/CRYSTAL/23v1-gcc</code>
* '''MPIDIR''' : <code>module load /rds/general/project/cmsg/live/etc/compiler/gcc11.2.0-aocl</code>.

Alternatively, rerun the configuration file (same as above) and specify 2 values during configuration.

Note that by default, there is no command for 'MPPproperties'. The user has to modify their <code>~/.bashrc</code> file to set the alias and 'settings' file for executable flags. An example:

# in ~/.bashrc
alias MPPprop23="/rds/general/project/cmsg/live/share/HPC-job-submission/Imperial-HPC-Job-Submission/gen_sub -x mppprop -set ${HOME}/etc/runCRYSTAL23/settings"
# in settings, keep the column width
mppprop mpiexec -np ${V_TPROC} MPPproperties Massive parallel properties calculation, OMP

=== HX1 version ===
* Compiling Env : EasyBuild foss/2023a
* Note: MP2 (CRYSCOR and CRYSTAL2) is currently not available.

* '''EXEDIR''' : No default. The directory where the user put their executables in.
* '''MPIDIR''' : <code>module load foss/2023a</code>

Alternatively, run the configuration file (The one with '-HX1' in the same directory of Github repo) and specify 2 values during configuration. Other options are consistent with CRYSTAL on CX1.

== CRYSTAL17 v2 (GNU)==
Not hosted on RDS-CMSG. Default for CX1.

=== CX1 version ===
* Compilor : gcc/6.2.0
* MPI : mpich/3.4.3

'''For job submission scripts'''

The configuration and usage are identical to CRYSTAL23. Please refer to the previous section and substitute '23' with '17'. Also, please be noted that CRYSCOR (MP2 calculation) is not available for CRYSTAL17 and 'MPP' as a massive parallel strategy is limited to 'crystal' calculations, i.e., only MPPcrystal is released.

== Quantum Espresso v7.2 (GNU) ==
Default for CX1.

=== CX1 version ===
* Compiling Env : EasyBuild foss2022a
* libxc: Yes, version 5.1.2
* hdf5: No

''' Job submission script configuration '''

<code>
~$ bash /rds/general/project/cmsg/live/share/HPC-job-submission/Imperial-HPC-Job-Submission/QE7/config_QE7.sh
</code>

The procedure is the same as CRYSTAL. Please note that moving files between temporal and home directories are disabled as QE has built-in file management system.

''' Job submission script commands '''

{| class="wikitable" style="margin:auto"
|+ List of available commands
|-
! Command !! Definition
|-
| PWqe7 || Generate pw.x job submission files
|-
| PHqe7 || Generate ph.x job submission files
|-
| CPqe7 || Generate cp.x job submission files
|-
| PPqe7 || Generate pp.x job submission files
|-
| Xqe7 || Generate job submission files for user-defined executables
|-
| SETqe7 || Print the local 'settings' file
|-
| HELPqe7 || Print the instructions of commands
|}

CMSG disk and Shared Software on CX1

2023-11-30T21:59:51Z

Hz1420: /* CX1 version */

General information of shared software available for CMSG group members on Imperial CX1 is summarized in this page. Software listed below, unless stated otherwise, is hosted under 'CMSG' project of Imperial Research Data Store (RDS) <ref>https://www.imperial.ac.uk/admin-services/ict/self-service/research-support/rcs/service-offering/rds/</ref> (referred as RDS-CMSG below), which is accessible for both login and computational nodes of Imperial HPC CX1. For Imperial HPC HX1, by the time this page is upated (Nov. 2023), it is not accessible there. Only login node can visit RDS-CMSG disk after the pilot phase <ref>https://wiki.imperial.ac.uk/display/HPC/HX1+Cluster</ref>. Therefore software on RDS-CMSG are classified into 2 catagories:

# For CX1, use <code>/rds/general/project/cmsg/live/app/</code> (executables, libraries and headers) and <code>/rds/general/project/cmsg/live/etc/</code> (other files, such as pseudopotential). Executable can be called directly from the user's home directory.
# For HX1, use <code>/rds/general/project/cmsg/live/share/</code>. Executables must be downloaded and uploaded to HX1. It is user's responsibility to set up a effective running environment according to 'readme' files (if there is one) saved in the same, or upper, directory.

''Please note that accessibility to CMSG disk is granted for CMSG group members only. If you need to access it, please contact group PI [[Contributors | Prof. Nicholas Harrison]].''

== CRYSTAL23 v1 (Intel) ==
The current default for CX1.

=== CX1 version ===
* Compiling Env : EasyBuild Intel 2023a
* Note: MPPproperties and MP2 (CRYSCOR and CRYSTAL2) are currently not available, which involves separate code packages (DMat2 and CRYSCOR). For MPPproperties please read the GNU version below.

The general job submission script for Imperial HPC, developed by the author himself, is used here.

''Thanks to Mr. K Tallat-Kelpsa for testing''

''' Configuration '''

The source code (written in bash) is available on group [https://github.com/cmsg-icl/HPC-job-submission/tree/main/Imperial-HPC-Job-Submission GitHub repo], though it is not needed in practice. Useful for developing new features. On CX1, use the following command to configure your local settings:

<code>
~$ bash /rds/general/project/cmsg/live/share/HPC-job-submission/Imperial-HPC-Job-Submission/CRYSTAL23/config_CRYSTAL23.sh
</code>

Then follow the instructions on the screen. Typically the default values are good. To use default vaules, press enter. After configuration is finished, the information is stored in a 'settings' file which is by default saved as <code>${HOME}/etc/runCRYSTAL23/settings</code>. It functions as a dictionary for job sumbmission scripts to refer to. It is also editable according to the user's needs.

Use the following command to activate alias commands:

<code>
~$ source ~/.bashrc
</code>

Due to the settings of CX1, the <code>source</code> command should be executed everytime you login. To aviod this, try the following commands:

~$ cat << EOF >> ~/.bash_profile
if test -f ~/.bashrc; then
source ~/.bashrc
fi
EOF

''' Quick References '''

{| class="wikitable" style="margin:auto"
|+ List of available commands
|-
! Command !! Definition
|-
| Pcrys23 || Generate parallel OMP CRYSTAL23 job submission files
|-
| MPPcrys23 || Generate massive parallel OMP CRYSTAL23 job submission files
|-
| Pprop23 || Generate parallel OMP CRYSTAL23 properties job submission files
|-
| Scrys23 || Generate serial OMP CRYSTAL23 job submission files
|-
| Sprop23 || Generate serial OMP CRYSTAL23 properties job submission files
|-
| Xcrys23 || User-defined executables and multiple jobs (see below for code examples)
|-
| SETcrys23 || Print the local 'settings' file
|-
| HELPcrys23 || Print the instructions of commands
|}

{| class="wikitable" style="margin:auto"
|+ List of in-line flags
|-
! Command !! Definition
|-
| -in || Input .d12 or .d3 files
|-
| -nd || Number of nodes requested. Number of CPUs per node will be read from the settings file if '-nc' is not set
|-
| -nc || Number of CPUs requested. Number of nodes will be automatically decided by Number of CPUs per node from settings file if '-nd' is not set
|-
| -nt || Number of threads per process. If not set the default value (1) is used.
|-
| -nt || Memory requested per node. If not set the default value (512GB) is used.
|-
| -wt || Walltime for the job. Or walltime for each job if multiple jobs
|-
| -ref || Optional, name of the reference file(s). No extension
|-
| -x || Xcrys23 only. The alias, i.e., lable, of MPI+executable in-line command pair
|-
| -name || Xcrys23 only. Set a common name for multiple jobs
|-
| -set || Developer only. The path to local 'settings' file
|}

{| class="wikitable" style="margin:auto"
|+ List of available labels for '-x' flag
|-
! Label !! Acutal in-line command
|-
| pcrys || mpiexec Pcrystal
|-
| mppcrys || mpiexec MPPcrystal
|-
| pprop || mpiexec Pproperties
|-
| scrys || Scrystal < INPUT
|-
| sprop || Sproperties < INPUT
|}

Note: New labels can be defined in local 'settings' file if a new MPI+executable in-line command is needed

''' Code Examples '''

To generate a qsub file for a parallel crystal23 job on 'mgo.d12', the following command can be used:

<code>
~$ Pcrys23 -in mgo.d12 -wt 01:00 -nd 1
</code>

That generates a 'mgo.qsub' file requesting for 1 node and the maximum running time is 1 hour. Similarly, after the job is done, using the following command can run a parallel properties calculation based on 'mgo-band.d3' and data from the previous 'mgo' SCF calculation on 12 CPUs:

<code>
~$ Pprop23 -in mgo-band.d3 -nc 12 -wt 00:30 -ref mgo
</code>

Alternatively, the user can also integrate 2 jobs into the same qsub file using <code>Xcrys23</code>.

<code>
~$ Xcrys23 -name mgo-band -nd 1 -x pcrys -in mgo.d12 -wt 01:00 -ref no -x pprop -in mgo-band.d3 -wt 00:30 -ref mgo
</code>

Note that -x, -in, -wt should have the same lengths while -ref should be either 0 or the same length as -x. In the case above, 'no' is a reserved keyword for no reference. It should be noted that specifying multiple -in, -wt and -ref flags in <code>Pcrys</code>, <code>MPPcrys</code>, <code>Pprop</code>, <code>Scrys</code> and <code>Sprop</code> will lead to error.

Besides, <code>Xcrys23</code> also supports user-defined MPI+executable command pairs in 'settings' file:

<code>
~$ Xcrys23 -x pcrys_other -in mgo.d12 -wt 01:00
</code>

For more detailed instructions, test cases and keyword list of settings file, please refer to the [https://github.com/cmsg-icl/HPC-job-submission/tree/main/Imperial-HPC-Job-Submission documentation].

== CRYSTAL23 v1 (GNU) ==
Compared to Intel version above, GNU version is slightly slower to allow for compatibility with MPPproperties.

=== CX1 version ===
* Compiling Env : gcc11.2.0, aocl4.0, mpich4.0.2
* Note: MP2 (CRYSCOR and CRYSTAL2) is currently not available.

The same job submission script is used. However, GNU version is not the default option for CX1. To enable GNU version, if there is a 'settings' file, change the following parameters:

* '''EXEDIR''' : <code>module load /rds/general/project/cmsg/live/etc/modulefiles/CRYSTAL/23v1-gcc</code>
* '''MPIDIR''' : <code>module load /rds/general/project/cmsg/live/etc/compiler/gcc11.2.0-aocl</code>.

Alternatively, rerun the configuration file (same as above) and specify 2 values during configuration.

Note that by default, there is no command for 'MPPproperties'. The user has to modify their <code>~/.bashrc</code> file to set the alias and 'settings' file for executable flags. An example:

# in ~/.bashrc
alias MPPprop23="/rds/general/project/cmsg/live/share/HPC-job-submission/Imperial-HPC-Job-Submission/gen_sub -x mppprop -set ${HOME}/etc/runCRYSTAL23/settings"
# in settings, keep the column width
mppprop mpiexec -np ${V_TPROC} MPPproperties Massive parallel properties calculation, OMP

=== HX1 version ===
* Compiling Env : EasyBuild foss/2023a
* Note: MP2 (CRYSCOR and CRYSTAL2) is currently not available.

* '''EXEDIR''' : No default. The directory where the user put their executables in.
* '''MPIDIR''' : <code>module load foss/2023a</code>

Alternatively, run the configuration file (The one with '-HX1' in the same directory of Github repo) and specify 2 values during configuration. Other options are consistent with CRYSTAL on CX1.

== CRYSTAL17 v2 (GNU)==
Not hosted on RDS-CMSG. Default for CX1.

=== CX1 version ===
* Compilor : gcc/6.2.0
* MPI : mpich/3.4.3

'''For job submission scripts'''

The configuration and usage are identical to CRYSTAL23. Please refer to the previous section and substitute '23' with '17'. Also, please be noted that CRYSCOR (MP2 calculation) is not available for CRYSTAL17 and 'MPP' as a massive parallel strategy is limited to 'crystal' calculations, i.e., only MPPcrystal is released.

== Quantum Espresso v7.2 (GNU) ==
Default for CX1.

=== CX1 version ===
* Compiling Env : EasyBuild foss2022a
* libxc: Yes, version 5.1.2
* hdf5: No

''' Job submission script configuration '''
<code>
~$ bash /rds/general/project/cmsg/live/share/HPC-job-submission/Imperial-HPC-Job-Submission/QE7/config_QE7.sh
</code>

The procedure is the same as CRYSTAL. Please note that moving files between temporal and home directories are disabled as QE has built-in file management system.

''' Job submission script commands '''

{| class="wikitable" style="margin:auto"
|+ List of available commands
|-
! Command !! Definition
|-
| PWqe7 || Generate pw.x job submission files
|-
| PHqe7 || Generate ph.x job submission files
|-
| CPqe7 || Generate cp.x job submission files
|-
| PPqe7 || Generate pp.x job submission files
|-
| Xqe7 || Generate job submission files for user-defined executables
|-
| SETqe7 || Print the local 'settings' file
|-
| HELPqe7 || Print the instructions of commands
|}

CMSG disk and Shared Software on CX1

2023-11-30T21:59:14Z

Hz1420:

General information of shared software available for CMSG group members on Imperial CX1 is summarized in this page. Software listed below, unless stated otherwise, is hosted under 'CMSG' project of Imperial Research Data Store (RDS) <ref>https://www.imperial.ac.uk/admin-services/ict/self-service/research-support/rcs/service-offering/rds/</ref> (referred as RDS-CMSG below), which is accessible for both login and computational nodes of Imperial HPC CX1. For Imperial HPC HX1, by the time this page is upated (Nov. 2023), it is not accessible there. Only login node can visit RDS-CMSG disk after the pilot phase <ref>https://wiki.imperial.ac.uk/display/HPC/HX1+Cluster</ref>. Therefore software on RDS-CMSG are classified into 2 catagories:

# For CX1, use <code>/rds/general/project/cmsg/live/app/</code> (executables, libraries and headers) and <code>/rds/general/project/cmsg/live/etc/</code> (other files, such as pseudopotential). Executable can be called directly from the user's home directory.
# For HX1, use <code>/rds/general/project/cmsg/live/share/</code>. Executables must be downloaded and uploaded to HX1. It is user's responsibility to set up a effective running environment according to 'readme' files (if there is one) saved in the same, or upper, directory.

''Please note that accessibility to CMSG disk is granted for CMSG group members only. If you need to access it, please contact group PI [[Contributors | Prof. Nicholas Harrison]].''

== CRYSTAL23 v1 (Intel) ==
The current default for CX1.

=== CX1 version ===
* Compiling Env : EasyBuild Intel 2023a
* Note: MPPproperties and MP2 (CRYSCOR and CRYSTAL2) are currently not available, which involves separate code packages (DMat2 and CRYSCOR). For MPPproperties please read the GNU version below.

The general job submission script for Imperial HPC, developed by the author himself, is used here.

''Thanks to Mr. K Tallat-Kelpsa for testing''

''' Configuration '''

The source code (written in bash) is available on group [https://github.com/cmsg-icl/HPC-job-submission/tree/main/Imperial-HPC-Job-Submission GitHub repo], though it is not needed in practice. Useful for developing new features. On CX1, use the following command to configure your local settings:

<code>
~$ bash /rds/general/project/cmsg/live/share/HPC-job-submission/Imperial-HPC-Job-Submission/CRYSTAL23/config_CRYSTAL23.sh
</code>

Then follow the instructions on the screen. Typically the default values are good. To use default vaules, press enter. After configuration is finished, the information is stored in a 'settings' file which is by default saved as <code>${HOME}/etc/runCRYSTAL23/settings</code>. It functions as a dictionary for job sumbmission scripts to refer to. It is also editable according to the user's needs.

Use the following command to activate alias commands:

<code>
~$ source ~/.bashrc
</code>

Due to the settings of CX1, the <code>source</code> command should be executed everytime you login. To aviod this, try the following commands:

~$ cat << EOF >> ~/.bash_profile
if test -f ~/.bashrc; then
source ~/.bashrc
fi
EOF

''' Quick References '''

{| class="wikitable" style="margin:auto"
|+ List of available commands
|-
! Command !! Definition
|-
| Pcrys23 || Generate parallel OMP CRYSTAL23 job submission files
|-
| MPPcrys23 || Generate massive parallel OMP CRYSTAL23 job submission files
|-
| Pprop23 || Generate parallel OMP CRYSTAL23 properties job submission files
|-
| Scrys23 || Generate serial OMP CRYSTAL23 job submission files
|-
| Sprop23 || Generate serial OMP CRYSTAL23 properties job submission files
|-
| Xcrys23 || User-defined executables and multiple jobs (see below for code examples)
|-
| SETcrys23 || Print the local 'settings' file
|-
| HELPcrys23 || Print the instructions of commands
|}

{| class="wikitable" style="margin:auto"
|+ List of in-line flags
|-
! Command !! Definition
|-
| -in || Input .d12 or .d3 files
|-
| -nd || Number of nodes requested. Number of CPUs per node will be read from the settings file if '-nc' is not set
|-
| -nc || Number of CPUs requested. Number of nodes will be automatically decided by Number of CPUs per node from settings file if '-nd' is not set
|-
| -nt || Number of threads per process. If not set the default value (1) is used.
|-
| -nt || Memory requested per node. If not set the default value (512GB) is used.
|-
| -wt || Walltime for the job. Or walltime for each job if multiple jobs
|-
| -ref || Optional, name of the reference file(s). No extension
|-
| -x || Xcrys23 only. The alias, i.e., lable, of MPI+executable in-line command pair
|-
| -name || Xcrys23 only. Set a common name for multiple jobs
|-
| -set || Developer only. The path to local 'settings' file
|}

{| class="wikitable" style="margin:auto"
|+ List of available labels for '-x' flag
|-
! Label !! Acutal in-line command
|-
| pcrys || mpiexec Pcrystal
|-
| mppcrys || mpiexec MPPcrystal
|-
| pprop || mpiexec Pproperties
|-
| scrys || Scrystal < INPUT
|-
| sprop || Sproperties < INPUT
|}

Note: New labels can be defined in local 'settings' file if a new MPI+executable in-line command is needed

''' Code Examples '''

To generate a qsub file for a parallel crystal23 job on 'mgo.d12', the following command can be used:

<code>
~$ Pcrys23 -in mgo.d12 -wt 01:00 -nd 1
</code>

That generates a 'mgo.qsub' file requesting for 1 node and the maximum running time is 1 hour. Similarly, after the job is done, using the following command can run a parallel properties calculation based on 'mgo-band.d3' and data from the previous 'mgo' SCF calculation on 12 CPUs:

<code>
~$ Pprop23 -in mgo-band.d3 -nc 12 -wt 00:30 -ref mgo
</code>

Alternatively, the user can also integrate 2 jobs into the same qsub file using <code>Xcrys23</code>.

<code>
~$ Xcrys23 -name mgo-band -nd 1 -x pcrys -in mgo.d12 -wt 01:00 -ref no -x pprop -in mgo-band.d3 -wt 00:30 -ref mgo
</code>

Note that -x, -in, -wt should have the same lengths while -ref should be either 0 or the same length as -x. In the case above, 'no' is a reserved keyword for no reference. It should be noted that specifying multiple -in, -wt and -ref flags in <code>Pcrys</code>, <code>MPPcrys</code>, <code>Pprop</code>, <code>Scrys</code> and <code>Sprop</code> will lead to error.

Besides, <code>Xcrys23</code> also supports user-defined MPI+executable command pairs in 'settings' file:

<code>
~$ Xcrys23 -x pcrys_other -in mgo.d12 -wt 01:00
</code>

For more detailed instructions, test cases and keyword list of settings file, please refer to the [https://github.com/cmsg-icl/HPC-job-submission/tree/main/Imperial-HPC-Job-Submission documentation].

== CRYSTAL23 v1 (GNU) ==
Compared to Intel version above, GNU version is slightly slower to allow for compatibility with MPPproperties.

=== CX1 version ===
* Compiling Env : gcc11.2.0, aocl4.0, mpich4.0.2
* Note: MP2 (CRYSCOR and CRYSTAL2) is currently not available.

The same job submission script is used. However, GNU version is not the default option for CX1. To enable GNU version, if there is a 'settings' file, change the following parameters:

* '''EXEDIR''' : <code>module load /rds/general/project/cmsg/live/etc/modulefiles/CRYSTAL/23v1-gcc</code>
* '''MPIDIR''' : <code>module load /rds/general/project/cmsg/live/etc/compiler/gcc11.2.0-aocl</code>.

Alternatively, rerun the configuration file (same as above) and specify 2 values during configuration.

Note that by default, there is no command for 'MPPproperties'. The user has to modify their <code>~/.bashrc</code> file to set the alias and 'settings' file for executable flags. An example:

# in ~/.bashrc
alias MPPprop23="/rds/general/project/cmsg/live/share/HPC-job-submission/Imperial-HPC-Job-Submission/gen_sub -x mppprop -set ${HOME}/etc/runCRYSTAL23/settings"
# in settings, keep the column width
mppprop mpiexec -np ${V_TPROC} MPPproperties Massive parallel properties calculation, OMP

=== HX1 version ===
* Compiling Env : EasyBuild foss/2023a
* Note: MP2 (CRYSCOR and CRYSTAL2) is currently not available.

* '''EXEDIR''' : No default. The directory where the user put their executables in.
* '''MPIDIR''' : <code>module load foss/2023a</code>

Alternatively, run the configuration file (The one with '-HX1' in the same directory of Github repo) and specify 2 values during configuration. Other options are consistent with CRYSTAL on CX1.

== CRYSTAL17 v2 (GNU)==
Not hosted on RDS-CMSG. Default for CX1.

=== CX1 version ===
* Compilor : gcc/6.2.0
* MPI : mpich/3.4.3

'''For job submission scripts'''

The configuration and usage are identical to CRYSTAL23. Please refer to the previous section and substitute '23' with '17'. Also, please be noted that CRYSCOR (MP2 calculation) is not available for CRYSTAL17 and 'MPP' as a massive parallel strategy is limited to 'crystal' calculations, i.e., only MPPcrystal is released.

== Quantum Espresso v7.2 (GNU) ==
Default for CX1.

=== CX1 version ===
* Compiling Env : EasyBuild foss2022a
* libxc: Yes, version 5.1.2
* hdf5: No

''' Job submission script configuration '''
<code>
~$ bash /rds/general/project/cmsg/live/share/HPC-job-submission/Imperial-HPC-Job-Submission/QE7/config_QE7.sh
</code>

The procedure is the same as CRYSTAL. Please note that moving files between temporal and home directories are disabled as QE has built-in file management system.

''' Job submission script commands '''

{| class="wikitable" style="margin:auto"
|+ List of available commands
|-
! Command !! Definition
|-
| PWqe7 || Generate pw.x job submission files
|-
| PHqe7 || Generate ph.x job submission files
|-
| CPqe7 || Generate cp.x job submission files
|-
| PPqe7 || Generate pp.x job submission files
|-
| Xqe7 || Generate job submission files for user-defined executables
|-
| SETqe7 || Print the local 'settings' file
|-
| HELPqe7 || Print the instructions of commands
|}

CMSG disk and Shared Software on CX1

2023-11-30T21:44:52Z

Hz1420: /* CRYSTAL17 */

General information of shared software available for CMSG group members on Imperial CX1 is summarized in this page. Software listed below, unless stated otherwise, is hosted under 'CMSG' project of Imperial Research Data Store (RDS) <ref>https://www.imperial.ac.uk/admin-services/ict/self-service/research-support/rcs/service-offering/rds/</ref> (referred as RDS-CMSG below), which is accessible for both login and computational nodes of Imperial HPC CX1. For Imperial HPC HX1, by the time this page is upated (Nov. 2023), it is not accessible there. Only login node can visit RDS-CMSG disk after the pilot phase <ref>https://wiki.imperial.ac.uk/display/HPC/HX1+Cluster</ref>. Therefore software on RDS-CMSG are classified into 2 catagories:

# For CX1, use <code>/rds/general/project/cmsg/live/app/</code> (executables, libraries and headers) and <code>/rds/general/project/cmsg/live/etc/</code> (other files, such as pseudopotential). Executable can be called directly from the user's home directory.
# For HX1, use <code>/rds/general/project/cmsg/live/share/</code>. Executables must be downloaded and uploaded to HX1. It is user's responsibility to set up a effective running environment according to 'readme' files (if there is one) saved in the same, or upper, directory.

''Please note that accessibility to CMSG disk is granted for CMSG group members only. If you need to access it, please contact group PI [[Contributors | Prof. Nicholas Harrison]].''

== CRYSTAL23 v1 (Intel) ==
The current default for CX1.

=== CX1 version ===
* Compiling Env : EasyBuild Intel 2023a
* Note: MPPproperties and MP2 (CRYSCOR and CRYSTAL2) are currently not available, which involves separate code packages (DMat2 and CRYSCOR). For MPPproperties please read the GNU version below.

The general job submission script for Imperial HPC, developed by the author himself, is used here.

''Thanks to Mr. K Tallat-Kelpsa for testing''

==== Configuration ====

The source code (written in bash) is available on group [https://github.com/cmsg-icl/HPC-job-submission/tree/main/Imperial-HPC-Job-Submission GitHub repo], though it is not needed in practice. Useful for developing new features. On CX1, use the following command to configure your local settings:

<code>
~$ bash /rds/general/project/cmsg/live/share/HPC-job-submission/Imperial-HPC-Job-Submission/CRYSTAL23/config_CRYSTAL23.sh
</code>

Then follow the instructions on the screen. Typically the default values are good. To use default vaules, press enter. After configuration is finished, the information is stored in a 'settings' file which is by default saved as <code>${HOME}/etc/runCRYSTAL23/settings</code>. It functions as a dictionary for job sumbmission scripts to refer to. It is also editable according to the user's needs.

Use the following command to activate alias commands:

<code>
~$ source ~/.bashrc
</code>

Due to the settings of CX1, the <code>source</code> command should be executed everytime you login. To aviod this, try the following commands:

~$ cat << EOF >> ~/.bash_profile
if test -f ~/.bashrc; then
source ~/.bashrc
fi
EOF

==== Quick References ====

{| class="wikitable" style="margin:auto"
|+ List of available commands
|-
! Command !! Definition
|-
| Pcrys23 || Generate parallel OMP CRYSTAL23 job submission files
|-
| MPPcrys23 || Generate massive parallel OMP CRYSTAL23 job submission files
|-
| Pprop23 || Generate parallel OMP CRYSTAL23 properties job submission files
|-
| Scrys23 || Generate serial OMP CRYSTAL23 job submission files
|-
| Sprop23 || Generate serial OMP CRYSTAL23 properties job submission files
|-
| Xcrys23 || User-defined executables and multiple jobs (see below for code examples)
|-
| SETcrys23 || Print the local 'settings' file
|-
| HELPcrys23 || Print the instructions of commands
|}

{| class="wikitable" style="margin:auto"
|+ List of in-line flags
|-
! Command !! Definition
|-
| -in || Input .d12 or .d3 files
|-
| -nd || Number of nodes requested. Number of CPUs per node will be read from the settings file if '-nc' is not set
|-
| -nc || Number of CPUs requested. Number of nodes will be automatically decided by Number of CPUs per node from settings file if '-nd' is not set
|-
| -nt || Number of threads per process. If not set the default value (1) is used.
|-
| -nt || Memory requested per node. If not set the default value (512GB) is used.
|-
| -wt || Walltime for the job. Or walltime for each job if multiple jobs
|-
| -ref || Optional, name of the reference file(s). No extension
|-
| -x || Xcrys23 only. The alias, i.e., lable, of MPI+executable in-line command pair
|-
| -name || Xcrys23 only. Set a common name for multiple jobs
|-
| -set || Developer only. The path to local 'settings' file
|}

{| class="wikitable" style="margin:auto"
|+ List of available labels for '-x' flag
|-
! Label !! Acutal in-line command
|-
| pcrys || mpiexec Pcrystal
|-
| mppcrys || mpiexec MPPcrystal
|-
| pprop || mpiexec Pproperties
|-
| scrys || Scrystal < INPUT
|-
| sprop || Sproperties < INPUT
|}

Note: New labels can be defined in local 'settings' file if a new MPI+executable in-line command is needed

==== Code Examples ====

To generate a qsub file for a parallel crystal23 job on 'mgo.d12', the following command can be used:

<code>
~$ Pcrys23 -in mgo.d12 -wt 01:00 -nd 1
</code>

That generates a 'mgo.qsub' file requesting for 1 node and the maximum running time is 1 hour. Similarly, after the job is done, using the following command can run a parallel properties calculation based on 'mgo-band.d3' and data from the previous 'mgo' SCF calculation on 12 CPUs:

<code>
~$ Pprop23 -in mgo-band.d3 -nc 12 -wt 00:30 -ref mgo
</code>

Alternatively, the user can also integrate 2 jobs into the same qsub file using <code>Xcrys23</code>.

<code>
~$ Xcrys23 -name mgo-band -nd 1 -x pcrys -in mgo.d12 -wt 01:00 -ref no -x pprop -in mgo-band.d3 -wt 00:30 -ref mgo
</code>

Note that -x, -in, -wt should have the same lengths while -ref should be either 0 or the same length as -x. In the case above, 'no' is a reserved keyword for no reference. It should be noted that specifying multiple -in, -wt and -ref flags in <code>Pcrys</code>, <code>MPPcrys</code>, <code>Pprop</code>, <code>Scrys</code> and <code>Sprop</code> will lead to error.

Besides, <code>Xcrys23</code> also supports user-defined MPI+executable command pairs in 'settings' file:

<code>
~$ Xcrys23 -x pcrys_other -in mgo.d12 -wt 01:00
</code>

For more detailed instructions, test cases and keyword list of settings file, please refer to the [https://github.com/cmsg-icl/HPC-job-submission/tree/main/Imperial-HPC-Job-Submission documentation].

== CRYSTAL23 v1 (GNU) ==
Compared to Intel version above, GNU version is slightly slower to allow for compatibility with MPPproperties.

=== CX1 version ===
* Compiling Env : gcc11.2.0, aocl4.0, mpich4.0.2
* Note: MP2 (CRYSCOR and CRYSTAL2) is currently not available.

The same job submission script is used. However, GNU version is not the default option for CX1. To enable GNU version, if there is a 'settings' file, change the following parameters:

* '''EXEDIR''' : <code>module load /rds/general/project/cmsg/live/etc/modulefiles/CRYSTAL/23v1-gcc</code>
* '''MPIDIR''' : <code>module load /rds/general/project/cmsg/live/etc/compiler/gcc11.2.0-aocl</code>.

Alternatively, rerun the configuration file (same as above) and specify 2 values during configuration.

Note that by default, there is no command for 'MPPproperties'. The user has to modify their <code>~/.bashrc</code> file to set the alias and 'settings' file for executable flags. An example:

# in ~/.bashrc
alias MPPprop23="/rds/general/project/cmsg/live/share/HPC-job-submission/Imperial-HPC-Job-Submission/gen_sub -x mppprop -set ${HOME}/etc/runCRYSTAL23/settings"
# in settings, keep the column width
mppprop mpiexec -np ${V_TPROC} MPPproperties Massive parallel properties calculation, OMP

=== HX1 version ===
* Compiling Env : EasyBuild foss/2023a
* Note: MP2 (CRYSCOR and CRYSTAL2) is currently not available.

* '''EXEDIR''' : No default. The directory where the user put their executables in.
* '''MPIDIR''' : <code>module load foss/2023a</code>

Alternatively, run the configuration file (The one with '-HX1' in the same directory of Github repo) and specify 2 values during configuration. Other options are consistent with CRYSTAL on CX1.

== CRYSTAL17 v2 (GNU)==
Not hosted on RDS-CMSG.

=== CX1 version ===
* Compilor : gcc/6.2.0
* MPI : mpich/3.4.3

'''For job submission scripts'''

The configuration and usage are identical to CRYSTAL23. Please refer to the previous section and substitute '23' with '17'. Also, please be noted that CRYSCOR (MP2 calculation) is not available for CRYSTAL17 and 'MPP' as a massive parallel strategy is limited to 'crystal' calculations, i.e., only MPPcrystal is released.

CMSG disk and Shared Software on CX1

2023-11-30T21:42:57Z

Hz1420:

General information of shared software available for CMSG group members on Imperial CX1 is summarized in this page. Software listed below, unless stated otherwise, is hosted under 'CMSG' project of Imperial Research Data Store (RDS) <ref>https://www.imperial.ac.uk/admin-services/ict/self-service/research-support/rcs/service-offering/rds/</ref> (referred as RDS-CMSG below), which is accessible for both login and computational nodes of Imperial HPC CX1. For Imperial HPC HX1, by the time this page is upated (Nov. 2023), it is not accessible there. Only login node can visit RDS-CMSG disk after the pilot phase <ref>https://wiki.imperial.ac.uk/display/HPC/HX1+Cluster</ref>. Therefore software on RDS-CMSG are classified into 2 catagories:

# For CX1, use <code>/rds/general/project/cmsg/live/app/</code> (executables, libraries and headers) and <code>/rds/general/project/cmsg/live/etc/</code> (other files, such as pseudopotential). Executable can be called directly from the user's home directory.
# For HX1, use <code>/rds/general/project/cmsg/live/share/</code>. Executables must be downloaded and uploaded to HX1. It is user's responsibility to set up a effective running environment according to 'readme' files (if there is one) saved in the same, or upper, directory.

''Please note that accessibility to CMSG disk is granted for CMSG group members only. If you need to access it, please contact group PI [[Contributors | Prof. Nicholas Harrison]].''

== CRYSTAL23 v1 (Intel) ==
The current default for CX1.

=== CX1 version ===
* Compiling Env : EasyBuild Intel 2023a
* Note: MPPproperties and MP2 (CRYSCOR and CRYSTAL2) are currently not available, which involves separate code packages (DMat2 and CRYSCOR). For MPPproperties please read the GNU version below.

The general job submission script for Imperial HPC, developed by the author himself, is used here.

''Thanks to Mr. K Tallat-Kelpsa for testing''

==== Configuration ====

The source code (written in bash) is available on group [https://github.com/cmsg-icl/HPC-job-submission/tree/main/Imperial-HPC-Job-Submission GitHub repo], though it is not needed in practice. Useful for developing new features. On CX1, use the following command to configure your local settings:

<code>
~$ bash /rds/general/project/cmsg/live/share/HPC-job-submission/Imperial-HPC-Job-Submission/CRYSTAL23/config_CRYSTAL23.sh
</code>

Then follow the instructions on the screen. Typically the default values are good. To use default vaules, press enter. After configuration is finished, the information is stored in a 'settings' file which is by default saved as <code>${HOME}/etc/runCRYSTAL23/settings</code>. It functions as a dictionary for job sumbmission scripts to refer to. It is also editable according to the user's needs.

Use the following command to activate alias commands:

<code>
~$ source ~/.bashrc
</code>

Due to the settings of CX1, the <code>source</code> command should be executed everytime you login. To aviod this, try the following commands:

~$ cat << EOF >> ~/.bash_profile
if test -f ~/.bashrc; then
source ~/.bashrc
fi
EOF

==== Quick References ====

{| class="wikitable" style="margin:auto"
|+ List of available commands
|-
! Command !! Definition
|-
| Pcrys23 || Generate parallel OMP CRYSTAL23 job submission files
|-
| MPPcrys23 || Generate massive parallel OMP CRYSTAL23 job submission files
|-
| Pprop23 || Generate parallel OMP CRYSTAL23 properties job submission files
|-
| Scrys23 || Generate serial OMP CRYSTAL23 job submission files
|-
| Sprop23 || Generate serial OMP CRYSTAL23 properties job submission files
|-
| Xcrys23 || User-defined executables and multiple jobs (see below for code examples)
|-
| SETcrys23 || Print the local 'settings' file
|-
| HELPcrys23 || Print the instructions of commands
|}

{| class="wikitable" style="margin:auto"
|+ List of in-line flags
|-
! Command !! Definition
|-
| -in || Input .d12 or .d3 files
|-
| -nd || Number of nodes requested. Number of CPUs per node will be read from the settings file if '-nc' is not set
|-
| -nc || Number of CPUs requested. Number of nodes will be automatically decided by Number of CPUs per node from settings file if '-nd' is not set
|-
| -nt || Number of threads per process. If not set the default value (1) is used.
|-
| -nt || Memory requested per node. If not set the default value (512GB) is used.
|-
| -wt || Walltime for the job. Or walltime for each job if multiple jobs
|-
| -ref || Optional, name of the reference file(s). No extension
|-
| -x || Xcrys23 only. The alias, i.e., lable, of MPI+executable in-line command pair
|-
| -name || Xcrys23 only. Set a common name for multiple jobs
|-
| -set || Developer only. The path to local 'settings' file
|}

{| class="wikitable" style="margin:auto"
|+ List of available labels for '-x' flag
|-
! Label !! Acutal in-line command
|-
| pcrys || mpiexec Pcrystal
|-
| mppcrys || mpiexec MPPcrystal
|-
| pprop || mpiexec Pproperties
|-
| scrys || Scrystal < INPUT
|-
| sprop || Sproperties < INPUT
|}

Note: New labels can be defined in local 'settings' file if a new MPI+executable in-line command is needed

==== Code Examples ====

To generate a qsub file for a parallel crystal23 job on 'mgo.d12', the following command can be used:

<code>
~$ Pcrys23 -in mgo.d12 -wt 01:00 -nd 1
</code>

That generates a 'mgo.qsub' file requesting for 1 node and the maximum running time is 1 hour. Similarly, after the job is done, using the following command can run a parallel properties calculation based on 'mgo-band.d3' and data from the previous 'mgo' SCF calculation on 12 CPUs:

<code>
~$ Pprop23 -in mgo-band.d3 -nc 12 -wt 00:30 -ref mgo
</code>

Alternatively, the user can also integrate 2 jobs into the same qsub file using <code>Xcrys23</code>.

<code>
~$ Xcrys23 -name mgo-band -nd 1 -x pcrys -in mgo.d12 -wt 01:00 -ref no -x pprop -in mgo-band.d3 -wt 00:30 -ref mgo
</code>

Note that -x, -in, -wt should have the same lengths while -ref should be either 0 or the same length as -x. In the case above, 'no' is a reserved keyword for no reference. It should be noted that specifying multiple -in, -wt and -ref flags in <code>Pcrys</code>, <code>MPPcrys</code>, <code>Pprop</code>, <code>Scrys</code> and <code>Sprop</code> will lead to error.

Besides, <code>Xcrys23</code> also supports user-defined MPI+executable command pairs in 'settings' file:

<code>
~$ Xcrys23 -x pcrys_other -in mgo.d12 -wt 01:00
</code>

For more detailed instructions, test cases and keyword list of settings file, please refer to the [https://github.com/cmsg-icl/HPC-job-submission/tree/main/Imperial-HPC-Job-Submission documentation].

== CRYSTAL23 v1 (GNU) ==
Compared to Intel version above, GNU version is slightly slower to allow for compatibility with MPPproperties.

=== CX1 version ===
* Compiling Env : gcc11.2.0, aocl4.0, mpich4.0.2
* Note: MP2 (CRYSCOR and CRYSTAL2) is currently not available.

The same job submission script is used. However, GNU version is not the default option for CX1. To enable GNU version, if there is a 'settings' file, change the following parameters:

* '''EXEDIR''' : <code>module load /rds/general/project/cmsg/live/etc/modulefiles/CRYSTAL/23v1-gcc</code>
* '''MPIDIR''' : <code>module load /rds/general/project/cmsg/live/etc/compiler/gcc11.2.0-aocl</code>.

Alternatively, rerun the configuration file (same as above) and specify 2 values during configuration.

Note that by default, there is no command for 'MPPproperties'. The user has to modify their <code>~/.bashrc</code> file to set the alias and 'settings' file for executable flags. An example:

# in ~/.bashrc
alias MPPprop23="/rds/general/project/cmsg/live/share/HPC-job-submission/Imperial-HPC-Job-Submission/gen_sub -x mppprop -set ${HOME}/etc/runCRYSTAL23/settings"
# in settings, keep the column width
mppprop mpiexec -np ${V_TPROC} MPPproperties Massive parallel properties calculation, OMP

=== HX1 version ===
* Compiling Env : EasyBuild foss/2023a
* Note: MP2 (CRYSCOR and CRYSTAL2) is currently not available.

* '''EXEDIR''' : No default. The directory where the user put their executables in.
* '''MPIDIR''' : <code>module load foss/2023a</code>

Alternatively, run the configuration file (The one with '-HX1' in the same directory of Github repo) and specify 2 values during configuration. Other options are consistent with CRYSTAL on CX1.

== CRYSTAL17 ==
* Version : 1.0.2
* Compilor : gcc/6.2.0
* MPI : mpich/3.4.3
* CPU Architecture : Intel Xeon

=== Launch Job with CX1 General Job Submission Script ===
The configuration and usage are identical to CRYSTAL23. Please refer to the previous section and substitute '23' with '17'. Also, please be noted that CRYSCOR (MP2 calculation) is not available for CRYSTAL17 and 'MPP' as a massive parallel strategy is limited to 'crystal' calculations, i.e., only MPPcrystal is released.

Nano Electrochemistry Group

2023-11-30T20:45:02Z

Hz1420: /* CMSG disk and Shared Software on CX1 */

<div style="padding: 20px; background: #87adde; border: 1px solid #FFAA99; font-family: Trebuchet MS, sans-serif; font-size: 105%;">
<div style="border:1px solid #90C0FF; background:#ffffff; width:98%; padding:20px; margin:auto">

This page provides a series of tutorials designed to help with the computational modelling of electrochemical system; their aim is to provide general workflows and useful tip to model fundamental components and properties of electrochemical systems. The tutorials have been designed by the researchers of the Computational NanoElectrochemistry Group led by Dr Clotilde Cucinotta [link to group page] and collaborators.

Several simulation packages (CP2K, LAMMPS, QuantumEspresso, etc.), as well as other tools, such as molecular visualisers or programming languages, are described in these tutorials; links to the relevant manuals are provided at the bottom of the page.

Script and programs written by the components of the research group are also described in each tutorial; these tools have been devised to help with running calculations and with data analysis and can be found in the linked GitLub repository [https://gitlab.doc.ic.ac.uk/rgc].

__TOC__

</div>
<div style="border:1px solid #90C0FF; background:#ffffff; width:98%; padding:20px; margin:auto">

=Compiling Codes and Running Calculations on a HPC cluster=

===[[How to run on ARCHER 2]]===
: <small>''by [[Contributors | Songyuan]]''</small>

===[[Imperial CX1: Instructions and basic concepts of parallel computing]]===
: <small>''by [[Contributors | Huanyu]]''</small>
: A collection of useful resources and brief introductions to the basic concepts of parallel computing for beginners to use the high-performance computing service at Imperial.

===[[CMSG disk and Shared Software on CX1]]===
: <small>''by [[Contributors | Huanyu]]''</small>
: About shared software hosted on RDS CMSG (''For CMSG group members'').

===[[Compile CP2Kv9.1 on Imperial CX1]]===
: <small>''by [[Contributors | Margherita]]''</small>

</div>
<div style="border:1px solid #90C0FF; background:#ffffff; width:98%; padding:20px; margin:auto">

=Modelling and Visualising Materials=

==Modelling of Interfaces and Adsorption processes==

===[[Building structures with Pymatgen]]===
: <small>''by [[Contributors | Fei]]''</small>
: Tutorial for generating crystal structure and surface with Python.

===[[ASE and materials modelling]]===
: Currently left blank

===[[Adsorption of molecule on surfaces|Adsorption of molecule on surfaces]]===
: <small>''by [[Contributors | Paolo]]''</small>
: Tutorial for calculating the adsorption energy of a molecule (or, more in general, any particle) over a specific surface.

== Error Evaluation during Simulations==

===[[Optimization of metallic surfaces parameters | CP2K: Optimizing parameters for metallic surfaces]]===
: <small>''by [[Contributors | Margherita]]''</small>
: Contents:
:: Tutorials on how to define the appropriate set of parameters needed to model a metallic system: Basis set, CUTOFF and '''k'''-points grid;
:: Tutorials on how to calculate relevant quantities of metallic surfaces: work function, equilibrium lattice parameter and electronic structure;
: System: metallic surfaces (Platinum slab used as example);
: Computational package: CP2K.

===[[Hard_carbon | CP2K: Simulation of Hard Carbons]]===
: <small>''by [[Contributors | Luke]]''</small>
: Tutorial for the simulation of hard carbon?

===[[Convergence test of critical parameters by CRYSTAL | CRYSTAL: Convergence tests of critical parameters]]===
: <small>''by [[Contributors | Huanyu]]''</small>
: Tutorial for optimising simulation parameters using the DFT code CRYSTAL (LCAO-GTO basis set).

===[[Memristors | Quantum Espresso: Simulation of Memristors]]===
: <small>''by [[Contributors | Felix]]''</small>
: Tutorial for optimising simulation parameters using the DFT code QuantumEspresso (plane waves basis set). The simulated system is a ZnO surface.

</div>
<div style="border:1px solid #90C0FF; background:#ffffff; width:98%; padding:20px; margin:auto">

==Postprocessing==

===[[Analysing AIMD runs with MATLAB in-house suit|Analysing AIMD runs with MATLAB in-house suit]]===
: <small>''by [[Contributors | Rashid]]''</small>

===Surface analysis===
: <small>''by [[Contributors| Songyuan]]''</small>

===[[Calculation of radial average|Calculation of radial average]]===
: <small>''[[Contributors| Kalman]]''</small>
: Tutorial for calculating the radial average ?.

==Machine Learning Potentials==

===[[Building ML potentials with AML|Building ML potentials with AML]]===
: <small>''by [[Contributors | Anthony]]''</small>
: Tutorial for building ML potentials with AML.

==Activation Barriers==

===[[NEB Calculation]]===
: Currently left blank

===[[Lammps and plumed | Metadynamics with Lammps and plumed]]===
: <small>''by [[Contributors| Frederik]]''</small>
: Tutorial on how to use the PLUMED software package to perform biased molecular dynamics simulations in LAMMPS.

==Methodologic developments==

===[[Potential control and current flow using CP2K+SMEAGOL]]===
: <small>''by [[Contributors| Chris]]''</small>
: Contents:
:: How to run CP2K+SMEAGOL and SIESTA+SMEAGOL calculations
:: How to exploit SMEAGOL parallelism
: System: Au nanojunctions
: Computational package: CP2K, SIESTA, SMEAGOL.

===[[Converging magnetic systems in CP2K]]===
: <small>''by [[Contributors| Chris]]''</small>
: Contents:
:: MULTIPLICITY keyword to calculate magnetic systems
:: &BS section and MAGNETIZATION keyword to improve convergence
: System: Metallic bulk Ni and slab in vacuum
: Computational package: CP2K.

===[[Running a HP-DFT calculation with CP2K]]===
: <small>''by [[Contributors| Margherita ]]''</small>
: A tutorial to run a HP-DFT calculation using CP2K

===[[Solving 1D Poisson equation |Solving 1D Poisson equation]]===
: <small>''by [[Contributors| Remi Khatib ]]''</small>
: Tutorial for the solution of the 1D Poisson equations given a distribution of point charges

</div>
<div style="border:1px solid #90C0FF; background:#ffffff; width:98%; padding:20px; margin:auto">

=Tutorials =

===[[Dimers in gas phase|Dimers in gas phase]]===
: <small>''by [[Contributors | Fredrik]]''</small>
: Tutorial for optimising dimers in the gas phase using Gaussian.

===[[TrendsCatalyticActivity | Trends in catalytic Activity]]===
: <small>''by [[Contributors | Clotilde Cucinotta]]''</small>
: Tutorial for a computational experiment about trends in catalytic activity for hydrogen evolution. This experiment is part of the third year computational chemistry lab.

=Others=

== Becoming an Efficient Research Scientist ==

===[[Writing a Project Proposal]]===
: <small>''by [[Contributors| Nicholas Harrison ]]''</small>

==Computational Tools==

===[https://www.cp2k.org/about CP2K]===
* [[CP2K_Tutorial|CP2K TUTORIAL]];
* [https://github.com/cp2k/cp2k/blob/master/INSTALL.md Download and install CP2K ];
* [https://manual.cp2k.org/#gsc.tab=0 Manual];
* [https://www.cp2k.org/howto Useful HOWTOs];
* Reading inputs and outputs (commented files and examples);

===[https://www.quantum-espresso.org/ QUANTUM ESPRESSO]===
* [https://www.quantum-espresso.org/download Download and install QUANTUM ESPRESSO];
* [https://www.quantum-espresso.org/resources/tutorials Useful Tutorials];
* Reading inputs and outputs (commented files and examples);

===[https://www.lammps.org/ LAMMPS]===
* [https://www.lammps.org/download.html Download LAMMPS];
* [https://docs.lammps.org/Manual.html Manual];
* [https://www.lammps.org/tutorials.html Tutorials];

===[https://www.crystal.unito.it/index.html CRYSTAL]===
* [https://tutorials.crystalsolutions.eu/ CRYSTAL Tutorial Project]
* [https://www.crystal.unito.it/basis_sets.html CRYSTAL basis set database] - Paramaterised and tested for solid state calculations
* [https://www.basissetexchange.org/ Basis Set Exchange] - Note that this site usually contains very diffuse basis sets for quantum chemmistry, which might cause problems for solid state calculations.
* [https://vallico.net/mike_towler/crystal.html Mike Towler's basis set] - Parameterised around early 2000s
* [https://crysplot.crystalsolutions.eu/ CRYSPLOT] - A web-based visualisation tool
* [https://crystal-code-tools.github.io/CRYSTALpytools/ CRYSTALpytools] - A python-based toolbox for CRYSTAL inputs and outputs.
More information is available in [https://www.crystal.unito.it/documentation.html CRYSTAL23 official site].

===[https://www.tcd.ie/Physics/Smeagol/SmeagolAbout.htm Smeagol]===

==Molecular visualizers==
* [http://www.ks.uiuc.edu/Research/vmd/ VMD]
* [http://www.xcrysden.org/ Xcrysden]
* [https://jp-minerals.org/vesta/en/ VESTA]
* [https://gitlab.com/bmgcsc/dl-visualize-v3 DLV3]

==Useful programming languages and environments==
* [http://www-eio.upc.edu/lceio/manuals/Fortran95-manual.pdf Fortran]
* [https://docs.python.org/3/ Python]
* [https://www.anaconda.com/ Anaconda]
* [https://wiki.fysik.dtu.dk/ase/ ASE]
* [https://pymatgen.org/ Pymatgen]
* [https://phonopy.github.io/phonopy/ Phonopy]

==Crystallography==
* [https://it.iucr.org/ International Crystallography Table]
* [https://www.cryst.ehu.es/#retrievaltop Bilbao Crystallographic Server]
* [https://www.ccdc.cam.ac.uk/structures/ Cambridge Database]
* [https://stokes.byu.edu/iso/findsym.php Find Symmetry Web Service]

</div>

[https://wiki.ch.ic.ac.uk/wiki/index.php?title=Main_Page info]

</div>

Run CRYSTALs on Imperial CX1

2023-11-30T20:44:34Z

Hz1420: Hz1420 moved page Run CRYSTALs on Imperial CX1 to CMSG disk and Shared Software on CX1: Rename the page to allow for more general proposes

#REDIRECT [[CMSG disk and Shared Software on CX1]]

CMSG disk and Shared Software on CX1

2023-11-30T20:44:34Z

Hz1420: Hz1420 moved page Run CRYSTALs on Imperial CX1 to CMSG disk and Shared Software on CX1: Rename the page to allow for more general proposes

Basic information of [https://www.crystal.unito.it/index.html CRYSTAL] DFT code previously and currently used within the group on Imperial CX1 are collected in this page. Instructions on submitting jobs are included.

== CRYSTAL23 ==
* Version : 1.0.1
* Compiling Env : EasyBuild Intel 2023a
* Note: MPPproperties and MP2 (CRYSCOR and CRYSTAL2) are currently not available, which involves separate code packages (DMat2 and CRYSCOR)

=== Launch Job with CX1 General Job Submission Script ===
''Thanks to Mr. K Tallat-Kelpsa for testing''

==== Configuration ====

The source code (written in bash) of CX1 general job submission script is available on group [https://github.com/cmsg-icl/HPC-job-submission/tree/main/Imperial-HPC-Job-Submission GitHub repo], though it is not needed in practice. Useful for developing new features. On CX1, use the following command to configure your local settings:

<code>
~$ bash /rds/general/project/cmsg/live/share/HPC-job-submission/Imperial-HPC-Job-Submission/CRYSTAL23/config_CRYSTAL23.sh
</code>

Then follow the instructions on the screen. Typically the default values are good. To use default vaules, press enter. After configuration is finished, the information is stored in a 'settings' file which is by default saved as <code>${HOME}/etc/runCRYSTAL23/settings</code>. It functions as a dictionary for job sumbmission scripts to refer to. It is also editable according to the user's needs.

Use the following command to activate alias commands:

<code>
~$ source ~/.bashrc
</code>

Due to the settings of CX1, the <code>source</code> command should be executed everytime you login. To aviod this, try the following commands:

~$ cat << EOF >> ~/.bash_profile
if test -f ~/.bashrc; then
source ~/.bashrc
fi
EOF

==== Quick References ====

{| class="wikitable" style="margin:auto"
|+ List of available commands
|-
! Command !! Definition
|-
| Pcrys23 || Generate parallel OMP CRYSTAL23 job submission files
|-
| MPPcrys23 || Generate massive parallel OMP CRYSTAL23 job submission files
|-
| Pprop23 || Generate parallel OMP CRYSTAL23 properties job submission files
|-
| Scrys23 || Generate serial OMP CRYSTAL23 job submission files
|-
| Sprop23 || Generate serial OMP CRYSTAL23 properties job submission files
|-
| Xcrys23 || User-defined executables and multiple jobs (see below for code examples)
|-
| SETcrys23 || Print the local 'settings' file
|-
| HELPcrys23 || Print the instructions of commands
|}

{| class="wikitable" style="margin:auto"
|+ List of in-line flags
|-
! Command !! Definition
|-
| -in || Input .d12 or .d3 files
|-
| -nd || Number of nodes requested. Number of CPUs per node will be read from the settings file if '-nc' is not set
|-
| -nc || Number of CPUs requested. Number of nodes will be automatically decided by Number of CPUs per node from settings file if '-nd' is not set
|-
| -nt || Number of threads per process. If not set the default value (1) is used.
|-
| -nt || Memory requested per node. If not set the default value (512GB) is used.
|-
| -wt || Walltime for the job. Or walltime for each job if multiple jobs
|-
| -ref || Optional, name of the reference file(s). No extension
|-
| -x || Xcrys23 only. The alias, i.e., lable, of MPI+executable in-line command pair
|-
| -name || Xcrys23 only. Set a common name for multiple jobs
|-
| -set || Developer only. The path to local 'settings' file
|}

{| class="wikitable" style="margin:auto"
|+ List of available labels for '-x' flag
|-
! Label !! Acutal in-line command
|-
| pcrys || mpiexec Pcrystal
|-
| mppcrys || mpiexec MPPcrystal
|-
| pprop || mpiexec Pproperties
|-
| scrys || Scrystal < INPUT
|-
| sprop || Sproperties < INPUT
|}

Note: New labels can be defined in local 'settings' file if a new MPI+executable in-line command is needed

==== Code Examples ====

To generate a qsub file for a parallel crystal23 job on 'mgo.d12', the following command can be used:

<code>
~$ Pcrys23 -in mgo.d12 -wt 01:00 -nd 1
</code>

That generates a 'mgo.qsub' file requesting for 1 node and the maximum running time is 1 hour. Similarly, after the job is done, using the following command can run a parallel properties calculation based on 'mgo-band.d3' and data from the previous 'mgo' SCF calculation on 12 CPUs:

<code>
~$ Pprop23 -in mgo-band.d3 -nc 12 -wt 00:30 -ref mgo
</code>

Alternatively, the user can also integrate 2 jobs into the same qsub file using <code>Xcrys23</code>.

<code>
~$ Xcrys23 -name mgo-band -nd 1 -x pcrys -in mgo.d12 -wt 01:00 -ref no -x pprop -in mgo-band.d3 -wt 00:30 -ref mgo
</code>

Note that -x, -in, -wt should have the same lengths while -ref should be either 0 or the same length as -x. In the case above, 'no' is a reserved keyword for no reference. It should be noted that specifying multiple -in, -wt and -ref flags in <code>Pcrys</code>, <code>MPPcrys</code>, <code>Pprop</code>, <code>Scrys</code> and <code>Sprop</code> will lead to error.

Besides, <code>Xcrys23</code> also supports user-defined MPI+executable command pairs in 'settings' file:

<code>
~$ Xcrys23 -x pcrys_other -in mgo.d12 -wt 01:00
</code>

For more detailed instructions, test cases and keyword list of settings file, please refer to the [https://github.com/cmsg-icl/HPC-job-submission/tree/main/Imperial-HPC-Job-Submission documentation].

== CRYSTAL17 ==
* Version : 1.0.2
* Compilor : gcc/6.2.0
* MPI : mpich/3.4.3
* CPU Architecture : Intel Xeon

=== Launch Job with CX1 General Job Submission Script ===
The configuration and usage are identical to CRYSTAL23. Please refer to the previous section and substitute '23' with '17'. Also, please be noted that CRYSCOR (MP2 calculation) is not available for CRYSTAL17 and 'MPP' as a massive parallel strategy is limited to 'crystal' calculations, i.e., only MPPcrystal is released.

Nano Electrochemistry Group

2023-11-30T20:42:35Z

Hz1420: /* Run CRYSTALs on Imperial CX1 */

<div style="padding: 20px; background: #87adde; border: 1px solid #FFAA99; font-family: Trebuchet MS, sans-serif; font-size: 105%;">
<div style="border:1px solid #90C0FF; background:#ffffff; width:98%; padding:20px; margin:auto">

This page provides a series of tutorials designed to help with the computational modelling of electrochemical system; their aim is to provide general workflows and useful tip to model fundamental components and properties of electrochemical systems. The tutorials have been designed by the researchers of the Computational NanoElectrochemistry Group led by Dr Clotilde Cucinotta [link to group page] and collaborators.

Several simulation packages (CP2K, LAMMPS, QuantumEspresso, etc.), as well as other tools, such as molecular visualisers or programming languages, are described in these tutorials; links to the relevant manuals are provided at the bottom of the page.

Script and programs written by the components of the research group are also described in each tutorial; these tools have been devised to help with running calculations and with data analysis and can be found in the linked GitLub repository [https://gitlab.doc.ic.ac.uk/rgc].

__TOC__

</div>
<div style="border:1px solid #90C0FF; background:#ffffff; width:98%; padding:20px; margin:auto">

=Compiling Codes and Running Calculations on a HPC cluster=

===[[How to run on ARCHER 2]]===
: <small>''by [[Contributors | Songyuan]]''</small>

===[[Imperial CX1: Instructions and basic concepts of parallel computing]]===
: <small>''by [[Contributors | Huanyu]]''</small>
: A collection of useful resources and brief introductions to the basic concepts of parallel computing for beginners to use the high-performance computing service at Imperial.

===[[Run CRYSTALs on Imperial CX1 | CMSG disk and Shared Software on CX1]]===
: <small>''by [[Contributors | Huanyu]]''</small>
: About shared software hosted on RDS CMSG (''For CMSG group members'').

===[[Compile CP2Kv9.1 on Imperial CX1]]===
: <small>''by [[Contributors | Margherita]]''</small>

</div>
<div style="border:1px solid #90C0FF; background:#ffffff; width:98%; padding:20px; margin:auto">

=Modelling and Visualising Materials=

==Modelling of Interfaces and Adsorption processes==

===[[Building structures with Pymatgen]]===
: <small>''by [[Contributors | Fei]]''</small>
: Tutorial for generating crystal structure and surface with Python.

===[[ASE and materials modelling]]===
: Currently left blank

===[[Adsorption of molecule on surfaces|Adsorption of molecule on surfaces]]===
: <small>''by [[Contributors | Paolo]]''</small>
: Tutorial for calculating the adsorption energy of a molecule (or, more in general, any particle) over a specific surface.

== Error Evaluation during Simulations==

===[[Optimization of metallic surfaces parameters | CP2K: Optimizing parameters for metallic surfaces]]===
: <small>''by [[Contributors | Margherita]]''</small>
: Contents:
:: Tutorials on how to define the appropriate set of parameters needed to model a metallic system: Basis set, CUTOFF and '''k'''-points grid;
:: Tutorials on how to calculate relevant quantities of metallic surfaces: work function, equilibrium lattice parameter and electronic structure;
: System: metallic surfaces (Platinum slab used as example);
: Computational package: CP2K.

===[[Hard_carbon | CP2K: Simulation of Hard Carbons]]===
: <small>''by [[Contributors | Luke]]''</small>
: Tutorial for the simulation of hard carbon?

===[[Convergence test of critical parameters by CRYSTAL | CRYSTAL: Convergence tests of critical parameters]]===
: <small>''by [[Contributors | Huanyu]]''</small>
: Tutorial for optimising simulation parameters using the DFT code CRYSTAL (LCAO-GTO basis set).

===[[Memristors | Quantum Espresso: Simulation of Memristors]]===
: <small>''by [[Contributors | Felix]]''</small>
: Tutorial for optimising simulation parameters using the DFT code QuantumEspresso (plane waves basis set). The simulated system is a ZnO surface.

</div>
<div style="border:1px solid #90C0FF; background:#ffffff; width:98%; padding:20px; margin:auto">

==Postprocessing==

===[[Analysing AIMD runs with MATLAB in-house suit|Analysing AIMD runs with MATLAB in-house suit]]===
: <small>''by [[Contributors | Rashid]]''</small>

===Surface analysis===
: <small>''by [[Contributors| Songyuan]]''</small>

===[[Calculation of radial average|Calculation of radial average]]===
: <small>''[[Contributors| Kalman]]''</small>
: Tutorial for calculating the radial average ?.

==Machine Learning Potentials==

===[[Building ML potentials with AML|Building ML potentials with AML]]===
: <small>''by [[Contributors | Anthony]]''</small>
: Tutorial for building ML potentials with AML.

==Activation Barriers==

===[[NEB Calculation]]===
: Currently left blank

===[[Lammps and plumed | Metadynamics with Lammps and plumed]]===
: <small>''by [[Contributors| Frederik]]''</small>
: Tutorial on how to use the PLUMED software package to perform biased molecular dynamics simulations in LAMMPS.

==Methodologic developments==

===[[Potential control and current flow using CP2K+SMEAGOL]]===
: <small>''by [[Contributors| Chris]]''</small>
: Contents:
:: How to run CP2K+SMEAGOL and SIESTA+SMEAGOL calculations
:: How to exploit SMEAGOL parallelism
: System: Au nanojunctions
: Computational package: CP2K, SIESTA, SMEAGOL.

===[[Converging magnetic systems in CP2K]]===
: <small>''by [[Contributors| Chris]]''</small>
: Contents:
:: MULTIPLICITY keyword to calculate magnetic systems
:: &BS section and MAGNETIZATION keyword to improve convergence
: System: Metallic bulk Ni and slab in vacuum
: Computational package: CP2K.

===[[Running a HP-DFT calculation with CP2K]]===
: <small>''by [[Contributors| Margherita ]]''</small>
: A tutorial to run a HP-DFT calculation using CP2K

===[[Solving 1D Poisson equation |Solving 1D Poisson equation]]===
: <small>''by [[Contributors| Remi Khatib ]]''</small>
: Tutorial for the solution of the 1D Poisson equations given a distribution of point charges

</div>
<div style="border:1px solid #90C0FF; background:#ffffff; width:98%; padding:20px; margin:auto">

=Tutorials =

===[[Dimers in gas phase|Dimers in gas phase]]===
: <small>''by [[Contributors | Fredrik]]''</small>
: Tutorial for optimising dimers in the gas phase using Gaussian.

===[[TrendsCatalyticActivity | Trends in catalytic Activity]]===
: <small>''by [[Contributors | Clotilde Cucinotta]]''</small>
: Tutorial for a computational experiment about trends in catalytic activity for hydrogen evolution. This experiment is part of the third year computational chemistry lab.

=Others=

== Becoming an Efficient Research Scientist ==

===[[Writing a Project Proposal]]===
: <small>''by [[Contributors| Nicholas Harrison ]]''</small>

==Computational Tools==

===[https://www.cp2k.org/about CP2K]===
* [[CP2K_Tutorial|CP2K TUTORIAL]];
* [https://github.com/cp2k/cp2k/blob/master/INSTALL.md Download and install CP2K ];
* [https://manual.cp2k.org/#gsc.tab=0 Manual];
* [https://www.cp2k.org/howto Useful HOWTOs];
* Reading inputs and outputs (commented files and examples);

===[https://www.quantum-espresso.org/ QUANTUM ESPRESSO]===
* [https://www.quantum-espresso.org/download Download and install QUANTUM ESPRESSO];
* [https://www.quantum-espresso.org/resources/tutorials Useful Tutorials];
* Reading inputs and outputs (commented files and examples);

===[https://www.lammps.org/ LAMMPS]===
* [https://www.lammps.org/download.html Download LAMMPS];
* [https://docs.lammps.org/Manual.html Manual];
* [https://www.lammps.org/tutorials.html Tutorials];

===[https://www.crystal.unito.it/index.html CRYSTAL]===
* [https://tutorials.crystalsolutions.eu/ CRYSTAL Tutorial Project]
* [https://www.crystal.unito.it/basis_sets.html CRYSTAL basis set database] - Paramaterised and tested for solid state calculations
* [https://www.basissetexchange.org/ Basis Set Exchange] - Note that this site usually contains very diffuse basis sets for quantum chemmistry, which might cause problems for solid state calculations.
* [https://vallico.net/mike_towler/crystal.html Mike Towler's basis set] - Parameterised around early 2000s
* [https://crysplot.crystalsolutions.eu/ CRYSPLOT] - A web-based visualisation tool
* [https://crystal-code-tools.github.io/CRYSTALpytools/ CRYSTALpytools] - A python-based toolbox for CRYSTAL inputs and outputs.
More information is available in [https://www.crystal.unito.it/documentation.html CRYSTAL23 official site].

===[https://www.tcd.ie/Physics/Smeagol/SmeagolAbout.htm Smeagol]===

==Molecular visualizers==
* [http://www.ks.uiuc.edu/Research/vmd/ VMD]
* [http://www.xcrysden.org/ Xcrysden]
* [https://jp-minerals.org/vesta/en/ VESTA]
* [https://gitlab.com/bmgcsc/dl-visualize-v3 DLV3]

==Useful programming languages and environments==
* [http://www-eio.upc.edu/lceio/manuals/Fortran95-manual.pdf Fortran]
* [https://docs.python.org/3/ Python]
* [https://www.anaconda.com/ Anaconda]
* [https://wiki.fysik.dtu.dk/ase/ ASE]
* [https://pymatgen.org/ Pymatgen]
* [https://phonopy.github.io/phonopy/ Phonopy]

==Crystallography==
* [https://it.iucr.org/ International Crystallography Table]
* [https://www.cryst.ehu.es/#retrievaltop Bilbao Crystallographic Server]
* [https://www.ccdc.cam.ac.uk/structures/ Cambridge Database]
* [https://stokes.byu.edu/iso/findsym.php Find Symmetry Web Service]

</div>

[https://wiki.ch.ic.ac.uk/wiki/index.php?title=Main_Page info]

</div>

Nano Electrochemistry Group

2023-11-30T20:39:50Z

Hz1420: Undo revision 814600 by Hz1420 (talk)

<div style="padding: 20px; background: #87adde; border: 1px solid #FFAA99; font-family: Trebuchet MS, sans-serif; font-size: 105%;">
<div style="border:1px solid #90C0FF; background:#ffffff; width:98%; padding:20px; margin:auto">

This page provides a series of tutorials designed to help with the computational modelling of electrochemical system; their aim is to provide general workflows and useful tip to model fundamental components and properties of electrochemical systems. The tutorials have been designed by the researchers of the Computational NanoElectrochemistry Group led by Dr Clotilde Cucinotta [link to group page] and collaborators.

Several simulation packages (CP2K, LAMMPS, QuantumEspresso, etc.), as well as other tools, such as molecular visualisers or programming languages, are described in these tutorials; links to the relevant manuals are provided at the bottom of the page.

Script and programs written by the components of the research group are also described in each tutorial; these tools have been devised to help with running calculations and with data analysis and can be found in the linked GitLub repository [https://gitlab.doc.ic.ac.uk/rgc].

__TOC__

</div>
<div style="border:1px solid #90C0FF; background:#ffffff; width:98%; padding:20px; margin:auto">

=Compiling Codes and Running Calculations on a HPC cluster=

===[[How to run on ARCHER 2]]===
: <small>''by [[Contributors | Songyuan]]''</small>

===[[Imperial CX1: Instructions and basic concepts of parallel computing]]===
: <small>''by [[Contributors | Huanyu]]''</small>
: A collection of useful resources and brief introductions to the basic concepts of parallel computing for beginners to use the high-performance computing service at Imperial.

===[[Run CRYSTALs on Imperial CX1]]===
: <small>''by [[Contributors | Huanyu]]''</small>
: Intructions on how to submit a CRYSTAL job on CX1

===[[Compile CP2Kv9.1 on Imperial CX1]]===
: <small>''by [[Contributors | Margherita]]''</small>

</div>
<div style="border:1px solid #90C0FF; background:#ffffff; width:98%; padding:20px; margin:auto">

=Modelling and Visualising Materials=

==Modelling of Interfaces and Adsorption processes==

===[[Building structures with Pymatgen]]===
: <small>''by [[Contributors | Fei]]''</small>
: Tutorial for generating crystal structure and surface with Python.

===[[ASE and materials modelling]]===
: Currently left blank

===[[Adsorption of molecule on surfaces|Adsorption of molecule on surfaces]]===
: <small>''by [[Contributors | Paolo]]''</small>
: Tutorial for calculating the adsorption energy of a molecule (or, more in general, any particle) over a specific surface.

== Error Evaluation during Simulations==

===[[Optimization of metallic surfaces parameters | CP2K: Optimizing parameters for metallic surfaces]]===
: <small>''by [[Contributors | Margherita]]''</small>
: Contents:
:: Tutorials on how to define the appropriate set of parameters needed to model a metallic system: Basis set, CUTOFF and '''k'''-points grid;
:: Tutorials on how to calculate relevant quantities of metallic surfaces: work function, equilibrium lattice parameter and electronic structure;
: System: metallic surfaces (Platinum slab used as example);
: Computational package: CP2K.

===[[Hard_carbon | CP2K: Simulation of Hard Carbons]]===
: <small>''by [[Contributors | Luke]]''</small>
: Tutorial for the simulation of hard carbon?

===[[Convergence test of critical parameters by CRYSTAL | CRYSTAL: Convergence tests of critical parameters]]===
: <small>''by [[Contributors | Huanyu]]''</small>
: Tutorial for optimising simulation parameters using the DFT code CRYSTAL (LCAO-GTO basis set).

===[[Memristors | Quantum Espresso: Simulation of Memristors]]===
: <small>''by [[Contributors | Felix]]''</small>
: Tutorial for optimising simulation parameters using the DFT code QuantumEspresso (plane waves basis set). The simulated system is a ZnO surface.

</div>
<div style="border:1px solid #90C0FF; background:#ffffff; width:98%; padding:20px; margin:auto">

==Postprocessing==

===[[Analysing AIMD runs with MATLAB in-house suit|Analysing AIMD runs with MATLAB in-house suit]]===
: <small>''by [[Contributors | Rashid]]''</small>

===Surface analysis===
: <small>''by [[Contributors| Songyuan]]''</small>

===[[Calculation of radial average|Calculation of radial average]]===
: <small>''[[Contributors| Kalman]]''</small>
: Tutorial for calculating the radial average ?.

==Machine Learning Potentials==

===[[Building ML potentials with AML|Building ML potentials with AML]]===
: <small>''by [[Contributors | Anthony]]''</small>
: Tutorial for building ML potentials with AML.

==Activation Barriers==

===[[NEB Calculation]]===
: Currently left blank

===[[Lammps and plumed | Metadynamics with Lammps and plumed]]===
: <small>''by [[Contributors| Frederik]]''</small>
: Tutorial on how to use the PLUMED software package to perform biased molecular dynamics simulations in LAMMPS.

==Methodologic developments==

===[[Potential control and current flow using CP2K+SMEAGOL]]===
: <small>''by [[Contributors| Chris]]''</small>
: Contents:
:: How to run CP2K+SMEAGOL and SIESTA+SMEAGOL calculations
:: How to exploit SMEAGOL parallelism
: System: Au nanojunctions
: Computational package: CP2K, SIESTA, SMEAGOL.

===[[Converging magnetic systems in CP2K]]===
: <small>''by [[Contributors| Chris]]''</small>
: Contents:
:: MULTIPLICITY keyword to calculate magnetic systems
:: &BS section and MAGNETIZATION keyword to improve convergence
: System: Metallic bulk Ni and slab in vacuum
: Computational package: CP2K.

===[[Running a HP-DFT calculation with CP2K]]===
: <small>''by [[Contributors| Margherita ]]''</small>
: A tutorial to run a HP-DFT calculation using CP2K

===[[Solving 1D Poisson equation |Solving 1D Poisson equation]]===
: <small>''by [[Contributors| Remi Khatib ]]''</small>
: Tutorial for the solution of the 1D Poisson equations given a distribution of point charges

</div>
<div style="border:1px solid #90C0FF; background:#ffffff; width:98%; padding:20px; margin:auto">

=Tutorials =

===[[Dimers in gas phase|Dimers in gas phase]]===
: <small>''by [[Contributors | Fredrik]]''</small>
: Tutorial for optimising dimers in the gas phase using Gaussian.

===[[TrendsCatalyticActivity | Trends in catalytic Activity]]===
: <small>''by [[Contributors | Clotilde Cucinotta]]''</small>
: Tutorial for a computational experiment about trends in catalytic activity for hydrogen evolution. This experiment is part of the third year computational chemistry lab.

=Others=

== Becoming an Efficient Research Scientist ==

===[[Writing a Project Proposal]]===
: <small>''by [[Contributors| Nicholas Harrison ]]''</small>

==Computational Tools==

===[https://www.cp2k.org/about CP2K]===
* [[CP2K_Tutorial|CP2K TUTORIAL]];
* [https://github.com/cp2k/cp2k/blob/master/INSTALL.md Download and install CP2K ];
* [https://manual.cp2k.org/#gsc.tab=0 Manual];
* [https://www.cp2k.org/howto Useful HOWTOs];
* Reading inputs and outputs (commented files and examples);

===[https://www.quantum-espresso.org/ QUANTUM ESPRESSO]===
* [https://www.quantum-espresso.org/download Download and install QUANTUM ESPRESSO];
* [https://www.quantum-espresso.org/resources/tutorials Useful Tutorials];
* Reading inputs and outputs (commented files and examples);

===[https://www.lammps.org/ LAMMPS]===
* [https://www.lammps.org/download.html Download LAMMPS];
* [https://docs.lammps.org/Manual.html Manual];
* [https://www.lammps.org/tutorials.html Tutorials];

===[https://www.crystal.unito.it/index.html CRYSTAL]===
* [https://tutorials.crystalsolutions.eu/ CRYSTAL Tutorial Project]
* [https://www.crystal.unito.it/basis_sets.html CRYSTAL basis set database] - Paramaterised and tested for solid state calculations
* [https://www.basissetexchange.org/ Basis Set Exchange] - Note that this site usually contains very diffuse basis sets for quantum chemmistry, which might cause problems for solid state calculations.
* [https://vallico.net/mike_towler/crystal.html Mike Towler's basis set] - Parameterised around early 2000s
* [https://crysplot.crystalsolutions.eu/ CRYSPLOT] - A web-based visualisation tool
* [https://crystal-code-tools.github.io/CRYSTALpytools/ CRYSTALpytools] - A python-based toolbox for CRYSTAL inputs and outputs.
More information is available in [https://www.crystal.unito.it/documentation.html CRYSTAL23 official site].

===[https://www.tcd.ie/Physics/Smeagol/SmeagolAbout.htm Smeagol]===

==Molecular visualizers==
* [http://www.ks.uiuc.edu/Research/vmd/ VMD]
* [http://www.xcrysden.org/ Xcrysden]
* [https://jp-minerals.org/vesta/en/ VESTA]
* [https://gitlab.com/bmgcsc/dl-visualize-v3 DLV3]

==Useful programming languages and environments==
* [http://www-eio.upc.edu/lceio/manuals/Fortran95-manual.pdf Fortran]
* [https://docs.python.org/3/ Python]
* [https://www.anaconda.com/ Anaconda]
* [https://wiki.fysik.dtu.dk/ase/ ASE]
* [https://pymatgen.org/ Pymatgen]
* [https://phonopy.github.io/phonopy/ Phonopy]

==Crystallography==
* [https://it.iucr.org/ International Crystallography Table]
* [https://www.cryst.ehu.es/#retrievaltop Bilbao Crystallographic Server]
* [https://www.ccdc.cam.ac.uk/structures/ Cambridge Database]
* [https://stokes.byu.edu/iso/findsym.php Find Symmetry Web Service]

</div>

[https://wiki.ch.ic.ac.uk/wiki/index.php?title=Main_Page info]

</div>

Nano Electrochemistry Group

2023-11-30T20:39:08Z

Hz1420: /* Run CRYSTALs on Imperial CX1 */

<div style="padding: 20px; background: #87adde; border: 1px solid #FFAA99; font-family: Trebuchet MS, sans-serif; font-size: 105%;">
<div style="border:1px solid #90C0FF; background:#ffffff; width:98%; padding:20px; margin:auto">

This page provides a series of tutorials designed to help with the computational modelling of electrochemical system; their aim is to provide general workflows and useful tip to model fundamental components and properties of electrochemical systems. The tutorials have been designed by the researchers of the Computational NanoElectrochemistry Group led by Dr Clotilde Cucinotta [link to group page] and collaborators.

Several simulation packages (CP2K, LAMMPS, QuantumEspresso, etc.), as well as other tools, such as molecular visualisers or programming languages, are described in these tutorials; links to the relevant manuals are provided at the bottom of the page.

Script and programs written by the components of the research group are also described in each tutorial; these tools have been devised to help with running calculations and with data analysis and can be found in the linked GitLub repository [https://gitlab.doc.ic.ac.uk/rgc].

__TOC__

</div>
<div style="border:1px solid #90C0FF; background:#ffffff; width:98%; padding:20px; margin:auto">

=Compiling Codes and Running Calculations on a HPC cluster=

===[[How to run on ARCHER 2]]===
: <small>''by [[Contributors | Songyuan]]''</small>

===[[Imperial CX1: Instructions and basic concepts of parallel computing]]===
: <small>''by [[Contributors | Huanyu]]''</small>
: A collection of useful resources and brief introductions to the basic concepts of parallel computing for beginners to use the high-performance computing service at Imperial.

===[[CMSG disk and Shared Software on Imperial CX1]]===
: <small>''by [[Contributors | Huanyu]]''</small>
: Intructions on shared software hosted on CMSG disk, available for Imperial CX1

===[[Compile CP2Kv9.1 on Imperial CX1]]===
: <small>''by [[Contributors | Margherita]]''</small>

</div>
<div style="border:1px solid #90C0FF; background:#ffffff; width:98%; padding:20px; margin:auto">

=Modelling and Visualising Materials=

==Modelling of Interfaces and Adsorption processes==

===[[Building structures with Pymatgen]]===
: <small>''by [[Contributors | Fei]]''</small>
: Tutorial for generating crystal structure and surface with Python.

===[[ASE and materials modelling]]===
: Currently left blank

===[[Adsorption of molecule on surfaces|Adsorption of molecule on surfaces]]===
: <small>''by [[Contributors | Paolo]]''</small>
: Tutorial for calculating the adsorption energy of a molecule (or, more in general, any particle) over a specific surface.

== Error Evaluation during Simulations==

===[[Optimization of metallic surfaces parameters | CP2K: Optimizing parameters for metallic surfaces]]===
: <small>''by [[Contributors | Margherita]]''</small>
: Contents:
:: Tutorials on how to define the appropriate set of parameters needed to model a metallic system: Basis set, CUTOFF and '''k'''-points grid;
:: Tutorials on how to calculate relevant quantities of metallic surfaces: work function, equilibrium lattice parameter and electronic structure;
: System: metallic surfaces (Platinum slab used as example);
: Computational package: CP2K.

===[[Hard_carbon | CP2K: Simulation of Hard Carbons]]===
: <small>''by [[Contributors | Luke]]''</small>
: Tutorial for the simulation of hard carbon?

===[[Convergence test of critical parameters by CRYSTAL | CRYSTAL: Convergence tests of critical parameters]]===
: <small>''by [[Contributors | Huanyu]]''</small>
: Tutorial for optimising simulation parameters using the DFT code CRYSTAL (LCAO-GTO basis set).

===[[Memristors | Quantum Espresso: Simulation of Memristors]]===
: <small>''by [[Contributors | Felix]]''</small>
: Tutorial for optimising simulation parameters using the DFT code QuantumEspresso (plane waves basis set). The simulated system is a ZnO surface.

</div>
<div style="border:1px solid #90C0FF; background:#ffffff; width:98%; padding:20px; margin:auto">

==Postprocessing==

===[[Analysing AIMD runs with MATLAB in-house suit|Analysing AIMD runs with MATLAB in-house suit]]===
: <small>''by [[Contributors | Rashid]]''</small>

===Surface analysis===
: <small>''by [[Contributors| Songyuan]]''</small>

===[[Calculation of radial average|Calculation of radial average]]===
: <small>''[[Contributors| Kalman]]''</small>
: Tutorial for calculating the radial average ?.

==Machine Learning Potentials==

===[[Building ML potentials with AML|Building ML potentials with AML]]===
: <small>''by [[Contributors | Anthony]]''</small>
: Tutorial for building ML potentials with AML.

==Activation Barriers==

===[[NEB Calculation]]===
: Currently left blank

===[[Lammps and plumed | Metadynamics with Lammps and plumed]]===
: <small>''by [[Contributors| Frederik]]''</small>
: Tutorial on how to use the PLUMED software package to perform biased molecular dynamics simulations in LAMMPS.

==Methodologic developments==

===[[Potential control and current flow using CP2K+SMEAGOL]]===
: <small>''by [[Contributors| Chris]]''</small>
: Contents:
:: How to run CP2K+SMEAGOL and SIESTA+SMEAGOL calculations
:: How to exploit SMEAGOL parallelism
: System: Au nanojunctions
: Computational package: CP2K, SIESTA, SMEAGOL.

===[[Converging magnetic systems in CP2K]]===
: <small>''by [[Contributors| Chris]]''</small>
: Contents:
:: MULTIPLICITY keyword to calculate magnetic systems
:: &BS section and MAGNETIZATION keyword to improve convergence
: System: Metallic bulk Ni and slab in vacuum
: Computational package: CP2K.

===[[Running a HP-DFT calculation with CP2K]]===
: <small>''by [[Contributors| Margherita ]]''</small>
: A tutorial to run a HP-DFT calculation using CP2K

===[[Solving 1D Poisson equation |Solving 1D Poisson equation]]===
: <small>''by [[Contributors| Remi Khatib ]]''</small>
: Tutorial for the solution of the 1D Poisson equations given a distribution of point charges

</div>
<div style="border:1px solid #90C0FF; background:#ffffff; width:98%; padding:20px; margin:auto">

=Tutorials =

===[[Dimers in gas phase|Dimers in gas phase]]===
: <small>''by [[Contributors | Fredrik]]''</small>
: Tutorial for optimising dimers in the gas phase using Gaussian.

===[[TrendsCatalyticActivity | Trends in catalytic Activity]]===
: <small>''by [[Contributors | Clotilde Cucinotta]]''</small>
: Tutorial for a computational experiment about trends in catalytic activity for hydrogen evolution. This experiment is part of the third year computational chemistry lab.

=Others=

== Becoming an Efficient Research Scientist ==

===[[Writing a Project Proposal]]===
: <small>''by [[Contributors| Nicholas Harrison ]]''</small>

==Computational Tools==

===[https://www.cp2k.org/about CP2K]===
* [[CP2K_Tutorial|CP2K TUTORIAL]];
* [https://github.com/cp2k/cp2k/blob/master/INSTALL.md Download and install CP2K ];
* [https://manual.cp2k.org/#gsc.tab=0 Manual];
* [https://www.cp2k.org/howto Useful HOWTOs];
* Reading inputs and outputs (commented files and examples);

===[https://www.quantum-espresso.org/ QUANTUM ESPRESSO]===
* [https://www.quantum-espresso.org/download Download and install QUANTUM ESPRESSO];
* [https://www.quantum-espresso.org/resources/tutorials Useful Tutorials];
* Reading inputs and outputs (commented files and examples);

===[https://www.lammps.org/ LAMMPS]===
* [https://www.lammps.org/download.html Download LAMMPS];
* [https://docs.lammps.org/Manual.html Manual];
* [https://www.lammps.org/tutorials.html Tutorials];

===[https://www.crystal.unito.it/index.html CRYSTAL]===
* [https://tutorials.crystalsolutions.eu/ CRYSTAL Tutorial Project]
* [https://www.crystal.unito.it/basis_sets.html CRYSTAL basis set database] - Paramaterised and tested for solid state calculations
* [https://www.basissetexchange.org/ Basis Set Exchange] - Note that this site usually contains very diffuse basis sets for quantum chemmistry, which might cause problems for solid state calculations.
* [https://vallico.net/mike_towler/crystal.html Mike Towler's basis set] - Parameterised around early 2000s
* [https://crysplot.crystalsolutions.eu/ CRYSPLOT] - A web-based visualisation tool
* [https://crystal-code-tools.github.io/CRYSTALpytools/ CRYSTALpytools] - A python-based toolbox for CRYSTAL inputs and outputs.
More information is available in [https://www.crystal.unito.it/documentation.html CRYSTAL23 official site].

===[https://www.tcd.ie/Physics/Smeagol/SmeagolAbout.htm Smeagol]===

==Molecular visualizers==
* [http://www.ks.uiuc.edu/Research/vmd/ VMD]
* [http://www.xcrysden.org/ Xcrysden]
* [https://jp-minerals.org/vesta/en/ VESTA]
* [https://gitlab.com/bmgcsc/dl-visualize-v3 DLV3]

==Useful programming languages and environments==
* [http://www-eio.upc.edu/lceio/manuals/Fortran95-manual.pdf Fortran]
* [https://docs.python.org/3/ Python]
* [https://www.anaconda.com/ Anaconda]
* [https://wiki.fysik.dtu.dk/ase/ ASE]
* [https://pymatgen.org/ Pymatgen]
* [https://phonopy.github.io/phonopy/ Phonopy]

==Crystallography==
* [https://it.iucr.org/ International Crystallography Table]
* [https://www.cryst.ehu.es/#retrievaltop Bilbao Crystallographic Server]
* [https://www.ccdc.cam.ac.uk/structures/ Cambridge Database]
* [https://stokes.byu.edu/iso/findsym.php Find Symmetry Web Service]

</div>

[https://wiki.ch.ic.ac.uk/wiki/index.php?title=Main_Page info]

</div>

2023-06-09T10:42:28Z

Hz1420: /* Usage */

This tutorial is divided into 2 separate sections. In the fist section, introductions and available resources of CX1 are listed and classified. Since the [https://www.imperial.ac.uk/admin-services/ict/self-service/research-support/rcs/ Research Computing Service (RCS)] team already developed great tutorials on their webpages, this part functions as a guide towards RCS webpages with necessary supplementary comments. In the second section, basic concepts of parallel computing and explanations of important terms are introduced. The main focus of this section is helping beginners to understand how high-performance computers (HPC) works on the basis of their daily practise.

''This tutorial was initially written between Feb. and Mar. 2022 to be shared within the group for induction and training proposes <ref>https://spica-vir.github.io/posts/Connect-to-the-Imperial-Cluster/</ref><ref>https://spica-vir.github.io/posts/Structure-and-usage-of-clusters/</ref>. Special thanks to Mr K. Tallat-Kelpsa, Ms A. Arber, Dr G. Mallia and Prof N. M. Harrison.''

== Introduction to CX1 ==
CX1 is the old name of the first HPC that served the whole college. New facilities (known as CX2) were gradually installed and integrated with the old system (CX3, a rather short-lived domain), while CX1 remains to be the most popular name that generally referring to the college-owned clusters. To grant a student access to CX1, the group PI can, on behave of that student, ask RCS team to add the specified account into HPC active user mailing list.
=== Connect to CX1 ===
CX1 is typically accessed via ssh (secured shell). Linux command line (Linux & MacOS) / sub-system (Windows 10,11) <ref>https://learn.microsoft.com/en-us/windows/wsl/install</ref> / SSH client (such as XShell <ref>https://www.xshell.com/en/xshell/</ref>) can be used. VPN is needed for off-campus users.

* [https://www.imperial.ac.uk/admin-services/ict/self-service/connect-communicate/remote-access/virtual-private-network-vpn/ Step-by-step guide to setup VPN]
* [https://www.imperial.ac.uk/admin-services/ict/self-service/research-support/rcs/support/getting-started/using-ssh/ How to use ssh]

In linux command line, use the following command to connect CX1:

<code>
~$ ssh -XY username@login.hpc.ic.ac.uk
</code>

P.S. <code>-XY</code> option can be omitted for most of cases, if you do not need GUI to run that program.

Alternatively, when the VPN service is unstable or even not available, it is possible to channel through the gateway of the cluster via a client, which is an 'agent'. To visit CX1, type the previous command in the client's command line.

<code>
~$ ssh username@sshgw.ic.ac.uk
</code>

Use the <code>scp</code> command to upload / download files, which is similar to <code>ssh</code> and <code>cp</code> command. For example, to upload a file:

<code>
~$ scp /local/path/file_name username@login.hpc.ic.ac.uk:/path/file_name
</code>

=== Usage ===
The [https://wiki.imperial.ac.uk/pages/viewpage.action?spaceKey=HPC&title=High+Performance+Computing RCS Wiki Page] contains information needed. [https://www.imperial.ac.uk/admin-services/ict/self-service/research-support/rcs/get-support/contact-us/ The support page], [https://wiki.imperial.ac.uk/display/HPC/Attend+a+clinic online clinic] and [https://wiki.imperial.ac.uk/display/HPC/Courses courses from graduate school] are available. To examine the status of CX1, use [https://status.rcs.imperial.ac.uk/d/u0zkcYQ7z/rcs-status?orgId=2&refresh=1m RCS status page].

==== Environmental Variables and Disk Space ====
Use <code>env</code> to access all the environmental variables - be careful, the output is '''HUGE'''. Some useful environmental variables:

* <code>${USER}</code> The user's college account, i.e., login credential.
* <code>${HOME}</code> '/rds/general/user/${USER}/home', or '~', which has 1 TB disk space for data backups.
* <code>${EPHEMERAL}</code> '/rds/general/user/${USER}/ephemeral' Temporal unlimited disk space lasting for 30 days. Suitable for running calculations.
* <code>${PATH}</code> Path to the executable can be attached for quick access. The Environment Modules package (see below) can automatically do that.

==== Software Management ====

The Environment Modules<ref>https://modules.readthedocs.io/en/latest/</ref> package is implemented on CX1 to manage computing software (see the following section for introductions). Basic commands are listed below:

* <code>module avail</code> List the available modules
* <code>module load mod_name</code> Load a specific module, 'mod_name'
* <code>module rm mod_name</code> Remove a specific module, 'mod_name'
* <code>module list</code> List all the loaded modules in the current environment
* <code>module help mod_name</code> Check the instructions of the module 'mod_name'

''Note: There is a CRYSTAL14 module in the list. For users in NMH's group, the latest CRYSTAL edition is available, so do not use that module.''

==== Job Partition Guide ====
A hierachy of jobs is designed for the optimial efficiency of CX1. The current job partition guide is available on [https://wiki.imperial.ac.uk/display/HPC/New+Job+sizing+guidance RCS Wiki Page]

==== Batch System ====

The PBS batch system <ref>https://en.wikipedia.org/wiki/Portable_Batch_System</ref> is used on CX1 (see the following section for the meaning of batch system). Basic commands of PBS are listed below:

* <code>availability</code> Check the availability of computational resources
* <code>qsub</code> filename.qsub</code> Submit the job 'filename'
* <code>qstat</code> Check the state of submitted jobs
* <code>qdel jobID</code> Kill the process with the ID number 'jobID'

To examine the queue status across the whole system, use [https://status.rcs.imperial.ac.uk/d/u0zkcYQ7z/rcs-status?orgId=2&refresh=1m RCS status page].

==== A General Job Submission Script ====
A general job submission script for CX1 is developed by the author himself. See the [https://github.com/cmsg-icl/crystal_shape_control/tree/main/Imperial-HPC-Job-Submission GitHub repository] of CMSG for details. Parameterised software includes: CRYSTAL17/23, Quantum Espresso 7, LAMMPS, GROMACS, GULP6.

== Basic Concepts of Parallel Computing ==
A brief introduction to parallel computing is given in this section by taking CX1, a medium-sized general-propose cluster, as an example.

=== Divide a job: Nodes, Processors and Threads ===

;Node
:A bunch of CPUs and probably with GPUs / coprocessors for acceleration. Memory and input files are shared by processors in the same node, so a node can be considered as an independent computer. The communication between nodes are achieved by ultra-fast network, which is the bottleneck of modern clusters.

;Processor
:The unit to deal with a 'process', also known as 'central processing unit', or CPU. Processors in the same node communicate via shared memory.

;Thread
:Subdivision of a process. Multiple threads in the same process share the resources allocated to the CPU.

The figure on the right hand side illustrates the hierarchy of node, processor, and thread:

[[File:Job_Partition.png|450px|right|Job Partition]]

==== Multiple processes vs multiple threads ====

From the figure above, it is not difficult to distinguish the differences between a 'process' and a 'thread': process is the smallest unit for resource allocation; thread is part of a process. The idea of 'thread' is introduced to address the huge difference in the speed of CPU and RAM. CPU is always several orders of magnitude faster than RAM, so typically the bottleneck of a process is loading the required environment from RAM, rather than computations in CPU. By using multiple threads in the same process, various branches of the same program can be executed simultaneously. Therefore, the shared environmental requirements doesn't need to be read from RAM for multiple times, and the loading time for threads is much smaller than for processes.

However, multithreading is not always advantageous. A technical prerequisite is that the program should be developed for multithread proposes. Python, for example, is a pseudo-multithread language, while Java is a real one. Sometimes multithreading can lead to catastrophic results. Since threads share the same resource allocation (CPU, RAM, I/O, etc.), when a thread fails, the whole process fails as well. Comparatively, in multiple processes, other processes will be protected if a process fails.

In practice, users can either run each process in serial (i.e., number of threads = 1), or in parallel (i.e., number of threads > 1) on clusters. However, '''the former one is recommended''', because of more secured resource managements. The latter is not advantageous. Besides the problem mentioned above, it might lead to problems such as memory leak when running programs either: not developed for multithreading / requires improper packages (Here is [https://docs.archer2.ac.uk/known-issues/#oom-due-to-memory-leak-in-libfabric-added-2022-02-23 a famous issue] with libfabric on ARCHER2 identified in early 2022).

==== More nodes vs more CPUs ====

When the allocated memory permits, from my experience, using more CPUs/processes per node is usually a better idea, considering that all nodes have independent memory space and the inter-node communications are achieved by wired networks. It almost always takes longer to coordinate nodes than to coordinate processors within the same node.

=== The internal coordinator: What is MPI ===

Message passing interface, or MPI, is a standard for communicating and transferring data between nodes and therefore distributed memories. It is utilised via MPI libraries. The most popular implementations include:

* MPICH <ref>https://www.mpich.org/</ref> - an open-source library
* Intel MPI <ref>https://www.intel.com/content/www/us/en/developer/tools/oneapi/mpi-library.html#gs.xld8oa</ref> - a popular implementation of MPICH especially optimised for Intel CPUs
* OpenMPI <ref>https://www.open-mpi.org/</ref> - an open-source library
* OpenMP <ref>https://www.openmp.org/</ref> - Not MPI; parallelization based on shared memory, so only implemented in a single node; can be used for multithreading

In practice, a hybrid parallelization combining MPI and OpenMP to run multithread jobs on cluster is allowed, though sometimes not recommended. The first process (probably not a node or a processor) is usually allocated for I/O, and the rest is used for parallel computing.

So far, MPI only supports C/C++ and FORTRAN, which explains why all parallel computing software is based on these languages. To launch an executable in parallel, one should use: <code>mpiexec</code> or <code>mpirun</code>.

=== Secure your storage: Work directory and home directory ===

Almost all the modern clusters have separate disk spaces for differently proposes, namely, work directory and home directory. This originates again from the famous speed difference between CPU and RAM/ROM. 2 distinctly kinds of disks are used respectively to improve the overall efficiency and secure important data:

* For work directory, large, high-frequency disks are used. Data stored in work directory is usually not backed up, and in the case of CX1, will be automatically cleaned after a fixed time length.
* For home directory, mechanical disks with slower read/write frequency but better robustness are used. Usually files in home space are backed up.

For large clusters like ARCHER2 <ref>https://www.archer2.ac.uk/</ref>, the work directory and the home directory are completely separated, i.e., directory is only viable by login nodes; work directory is viable by both job and login nodes. Job submission in home directory is prohibited. For more flexible clusters like Imperial CX1, submitting jobs in home directory and visiting of home directory by job nodes are allowed, but storing temporary files during calculation in home directory is still not recommended because of the potential influence on other files and the reduced overall efficiency.

=== Setup your environment: What does an application need? ===

==== Executable ====
The binary executable should, theoretically, all be stored in '\usr\bin'. This never happens in practice, unless you are a fanatical fundamentalist of the early Linux releases. To guide your system to the desired executable, you can either laboriously type its absolute path every time you need it or add the path to the environmental variable:

<code>
~$ export PATH=${PATH}:path_to_bin
</code>

Running any executable in parallel requires mpi to coordinate all the processes/threads. The path to mpi executable is also required. Besides, many scientific codes require other specific environmental variables such as linear algebra packages. Read their documentations for further information.

==== .lib/.a/.o files ====

When writing a script, you might need some extra packages to do more complex jobs. Those packages are developed by experts in computer science and can be called by a line of code. The same thing happens when people were developing applications like CRYSTAL and ONETEP.

However, scientific computing codes are usually distributed in the form of source code. Source codes in FORTRAN/C/C++ need be compiled into a binary executable. There are 2 options during compiling:

# Include the whole package as long as one of its functions is called, also known as a 'static lib'.
# Only include a 'table of contents' when compiling, also known as 'dynamic lib'. The packages needed are separately stored in '.dll/.so' files, making it possible for multiple applications sharing the same lib.

Details about compilation are beyond the scope of this post. The thing is: when running a dynamically linked application, information should be given to help the code find the libs needed. This can be specified by:

<code>
~$ export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:path_to_lib`
</code>

For statically linked applications, usually you need not worry about it - but the volume of the compiled executable might make you wonder whether there is an alternative way.

==== Conflicts ====

Improper previous settings may lead to a wrong application, or a wrong version, if multiple applications with similar functions are installed in the system, such as Intel compiler and GCC, OpenMPI and MPICH - a common phenomenon for shared computing resources. To avoid this, the path to the undesired application or lib should be removed from the environmental variables.

==== Environmental Modules ====

Environmental Modules <ref>http://modules.sourceforge.net/</ref> is a popular software managing the necessary environmental setups and conflicts for each application. It can easily add or erase the environmental variables by commands (such as <code>module load</code>, <code>module rm</code>) and modulefiles written in Tool Command Language (TCL)<ref>https://www.tcl.tk/</ref>. The default directory of modulefiles is given in the environmental variable <code>${MODULEPATH}</code>, but files in other directories can also be loaded by their absolute path.

Both Imperial CX1 and ARCHER2 adopt this application, with which pre-compiled applications are offered.

=== The external coordinator: What is a batch system ===

Always bear in mind that the computational resources are limited, so you need to acquire reasonable resources for your job. Besides, the cluster also needs to calculate your budget, coordinate jobs submitted by various users, and make the best of available resources. When job is running, maybe you also want to check its status. All of this are fulfilled by batch systems.

In practice, a Linux shell script is needed. Parameters of the batch system of are set in the commented lines at the top of the file. After the user submit the script to batch system, the system will:

# Examine the parameters
# Allocate and coordinate the requested resources
# Set up the environments, such as environmental variables, package dependency, and sync the same setting to all nodes
# Launch a parallel calculation - see mpi part
# Post-process

Note that a 'walltime' is usually required for a batch job, i.e., the maximum allowed time for the running job. The job will be 'killed', or suspended, when the time exceeds the walltime, and the rest part of the script will not be executed. <code>timeout</code> command can be used to set another walltime for a specific command.

Common batch systems include PBS and Slurm <ref>https://slurm.schedmd.com/overview.html</ref>. For Imperial cluster CX1 and MMM Hub Young (managed by UCL) <ref>http://mmmhub.ac.uk/young/</ref>, PBS system is implemented; for ARCHER2 and Tianhe-2 LvLiang(天河二号-吕梁), Slurm is implemented. Tutorials of batch systems are not covered here, since they are heavily tailored according to specific machines - usually modifications are made to enhance the efficiency. Refer to the specific user documentations for more information.

Successfully setting and submitting a batch job script symbolises that you do not need this tutorial any more. Before being able to do that, some considerations might be important:

* How large is my system? Is it efficient to use the resources I requested(Note that it is not a linear-scaling problem... Refer to [https://tutorials.crystalsolutions.eu/tutorial.html?td=tuto_HPC&tf=tuto_hpc#scale this test] on CRYSTAL17)?
* To which queue should I submit my job? Is it too long/not applicable/not available?
* Is it safe to use multi-threading?
* Is it memory, GPU etc. demanding?
* Roughly how long will it take?
* What is my budget code? Do I have enough resources?
* Which MPI release version is my code compatible with? Should I load a module or set variables?
* Any other specific environmental setups does my code need?
* Do I have any post-processing script after MPI part is finished? How long does it take?

==References==
<references>

Imperial CX1: Instructions and basic concepts of parallel computing

2023-06-08T15:59:09Z

Hz1420: /* Usage */

This tutorial is divided into 2 separate sections. In the fist section, introductions and available resources of CX1 are listed and classified. Since the [https://www.imperial.ac.uk/admin-services/ict/self-service/research-support/rcs/ Research Computing Service (RCS)] team already developed great tutorials on their webpages, this part functions as a guide towards RCS webpages with necessary supplementary comments. In the second section, basic concepts of parallel computing and explanations of important terms are introduced. The main focus of this section is helping beginners to understand how high-performance computers (HPC) works on the basis of their daily practise.

''This tutorial was initially written between Feb. and Mar. 2022 to be shared within the group for induction and training proposes <ref>https://spica-vir.github.io/posts/Connect-to-the-Imperial-Cluster/</ref><ref>https://spica-vir.github.io/posts/Structure-and-usage-of-clusters/</ref>. Special thanks to Mr K. Tallat-Kelpsa, Ms A. Arber, Dr G. Mallia and Prof N. M. Harrison.''

== Introduction to CX1 ==
CX1 is the old name of the first HPC that served the whole college. New facilities (known as CX2) were gradually installed and integrated with the old system (CX3, a rather short-lived domain), while CX1 remains to be the most popular name that generally referring to the college-owned clusters. To grant a student access to CX1, the group PI can, on behave of that student, ask RCS team to add the specified account into HPC active user mailing list.
=== Connect to CX1 ===
CX1 is typically accessed via ssh (secured shell). Linux command line (Linux & MacOS) / sub-system (Windows 10,11) <ref>https://learn.microsoft.com/en-us/windows/wsl/install</ref> / SSH client (such as XShell <ref>https://www.xshell.com/en/xshell/</ref>) can be used. VPN is needed for off-campus users.

* [https://www.imperial.ac.uk/admin-services/ict/self-service/connect-communicate/remote-access/virtual-private-network-vpn/ Step-by-step guide to setup VPN]
* [https://www.imperial.ac.uk/admin-services/ict/self-service/research-support/rcs/support/getting-started/using-ssh/ How to use ssh]

In linux command line, use the following command to connect CX1:

<code>
~$ ssh -XY username@login.hpc.ic.ac.uk
</code>

P.S. <code>-XY</code> option can be omitted for most of cases, if you do not need GUI to run that program.

Alternatively, when the VPN service is unstable or even not available, it is possible to channel through the gateway of the cluster via a client, which is an 'agent'. To visit CX1, type the previous command in the client's command line.

<code>
~$ ssh username@sshgw.ic.ac.uk
</code>

Use the <code>scp</code> command to upload / download files, which is similar to <code>ssh</code> and <code>cp</code> command. For example, to upload a file:

<code>
~$ scp /local/path/file_name username@login.hpc.ic.ac.uk:/path/file_name
</code>

=== Usage ===
The [https://wiki.imperial.ac.uk/pages/viewpage.action?spaceKey=HPC&title=High+Performance+Computing RCS Wiki Page] contains information needed. [https://www.imperial.ac.uk/admin-services/ict/self-service/research-support/rcs/get-support/contact-us/ The support page], [https://wiki.imperial.ac.uk/display/HPC/Attend+a+clinic online clinic] and [https://wiki.imperial.ac.uk/display/HPC/Courses courses from graduate school] are available. To examine the status of CX1, use [https://github.com/cmsg-icl/crystal_shape_control/tree/main/Imperial-HPC-Job-Submission RCS status page].

==== Environmental Variables and Disk Space ====
Use <code>env</code> to access all the environmental variables - be careful, the output is '''HUGE'''. Some useful environmental variables:

* <code>${USER}</code> The user's college account, i.e., login credential.
* <code>${HOME}</code> '/rds/general/user/${USER}/home', or '~', which has 1 TB disk space for data backups.
* <code>${EPHEMERAL}</code> '/rds/general/user/${USER}/ephemeral' Temporal unlimited disk space lasting for 30 days. Suitable for running calculations.
* <code>${PATH}</code> Path to the executable can be attached for quick access. The Environment Modules package (see below) can automatically do that.

==== Software Management ====

The Environment Modules<ref>https://modules.readthedocs.io/en/latest/</ref> package is implemented on CX1 to manage computing software (see the following section for introductions). Basic commands are listed below:

* <code>module avail</code> List the available modules
* <code>module load mod_name</code> Load a specific module, 'mod_name'
* <code>module rm mod_name</code> Remove a specific module, 'mod_name'
* <code>module list</code> List all the loaded modules in the current environment
* <code>module help mod_name</code> Check the instructions of the module 'mod_name'

''Note: There is a CRYSTAL14 module in the list. For users in NMH's group, the latest CRYSTAL edition is available, so do not use that module.''

==== Job Partition Guide ====
A hierachy of jobs is designed for the optimial efficiency of CX1. The current job partition guide is available on [https://wiki.imperial.ac.uk/display/HPC/New+Job+sizing+guidance RCS Wiki Page]

==== Batch System ====

The PBS batch system <ref>https://en.wikipedia.org/wiki/Portable_Batch_System</ref> is used on CX1 (see the following section for the meaning of batch system). Basic commands of PBS are listed below:

* <code>availability</code> Check the availability of computational resources
* <code>qsub</code> filename.qsub</code> Submit the job 'filename'
* <code>qstat</code> Check the state of submitted jobs
* <code>qdel jobID</code> Kill the process with the ID number 'jobID'

To examine the queue status across the whole system, use [https://status.rcs.imperial.ac.uk/d/u0zkcYQ7z/rcs-status?orgId=2&refresh=1m RCS status page].

==== A General Job Submission Script ====
A general job submission script for CX1 is developed by the author himself. See the [https://github.com/cmsg-icl/crystal_shape_control/tree/main/Imperial-HPC-Job-Submission GitHub repository] of CMSG for details. Parameterised software includes: CRYSTAL17/23, Quantum Espresso 7, LAMMPS, GROMACS, GULP6.

== Basic Concepts of Parallel Computing ==
A brief introduction to parallel computing is given in this section by taking CX1, a medium-sized general-propose cluster, as an example.

=== Divide a job: Nodes, Processors and Threads ===

;Node
:A bunch of CPUs and probably with GPUs / coprocessors for acceleration. Memory and input files are shared by processors in the same node, so a node can be considered as an independent computer. The communication between nodes are achieved by ultra-fast network, which is the bottleneck of modern clusters.

;Processor
:The unit to deal with a 'process', also known as 'central processing unit', or CPU. Processors in the same node communicate via shared memory.

;Thread
:Subdivision of a process. Multiple threads in the same process share the resources allocated to the CPU.

The figure on the right hand side illustrates the hierarchy of node, processor, and thread:

[[File:Job_Partition.png|450px|right|Job Partition]]

==== Multiple processes vs multiple threads ====

From the figure above, it is not difficult to distinguish the differences between a 'process' and a 'thread': process is the smallest unit for resource allocation; thread is part of a process. The idea of 'thread' is introduced to address the huge difference in the speed of CPU and RAM. CPU is always several orders of magnitude faster than RAM, so typically the bottleneck of a process is loading the required environment from RAM, rather than computations in CPU. By using multiple threads in the same process, various branches of the same program can be executed simultaneously. Therefore, the shared environmental requirements doesn't need to be read from RAM for multiple times, and the loading time for threads is much smaller than for processes.

However, multithreading is not always advantageous. A technical prerequisite is that the program should be developed for multithread proposes. Python, for example, is a pseudo-multithread language, while Java is a real one. Sometimes multithreading can lead to catastrophic results. Since threads share the same resource allocation (CPU, RAM, I/O, etc.), when a thread fails, the whole process fails as well. Comparatively, in multiple processes, other processes will be protected if a process fails.

In practice, users can either run each process in serial (i.e., number of threads = 1), or in parallel (i.e., number of threads > 1) on clusters. However, '''the former one is recommended''', because of more secured resource managements. The latter is not advantageous. Besides the problem mentioned above, it might lead to problems such as memory leak when running programs either: not developed for multithreading / requires improper packages (Here is [https://docs.archer2.ac.uk/known-issues/#oom-due-to-memory-leak-in-libfabric-added-2022-02-23 a famous issue] with libfabric on ARCHER2 identified in early 2022).

==== More nodes vs more CPUs ====

When the allocated memory permits, from my experience, using more CPUs/processes per node is usually a better idea, considering that all nodes have independent memory space and the inter-node communications are achieved by wired networks. It almost always takes longer to coordinate nodes than to coordinate processors within the same node.

=== The internal coordinator: What is MPI ===

Message passing interface, or MPI, is a standard for communicating and transferring data between nodes and therefore distributed memories. It is utilised via MPI libraries. The most popular implementations include:

* MPICH <ref>https://www.mpich.org/</ref> - an open-source library
* Intel MPI <ref>https://www.intel.com/content/www/us/en/developer/tools/oneapi/mpi-library.html#gs.xld8oa</ref> - a popular implementation of MPICH especially optimised for Intel CPUs
* OpenMPI <ref>https://www.open-mpi.org/</ref> - an open-source library
* OpenMP <ref>https://www.openmp.org/</ref> - Not MPI; parallelization based on shared memory, so only implemented in a single node; can be used for multithreading

In practice, a hybrid parallelization combining MPI and OpenMP to run multithread jobs on cluster is allowed, though sometimes not recommended. The first process (probably not a node or a processor) is usually allocated for I/O, and the rest is used for parallel computing.

So far, MPI only supports C/C++ and FORTRAN, which explains why all parallel computing software is based on these languages. To launch an executable in parallel, one should use: <code>mpiexec</code> or <code>mpirun</code>.

=== Secure your storage: Work directory and home directory ===

Almost all the modern clusters have separate disk spaces for differently proposes, namely, work directory and home directory. This originates again from the famous speed difference between CPU and RAM/ROM. 2 distinctly kinds of disks are used respectively to improve the overall efficiency and secure important data:

* For work directory, large, high-frequency disks are used. Data stored in work directory is usually not backed up, and in the case of CX1, will be automatically cleaned after a fixed time length.
* For home directory, mechanical disks with slower read/write frequency but better robustness are used. Usually files in home space are backed up.

For large clusters like ARCHER2 <ref>https://www.archer2.ac.uk/</ref>, the work directory and the home directory are completely separated, i.e., directory is only viable by login nodes; work directory is viable by both job and login nodes. Job submission in home directory is prohibited. For more flexible clusters like Imperial CX1, submitting jobs in home directory and visiting of home directory by job nodes are allowed, but storing temporary files during calculation in home directory is still not recommended because of the potential influence on other files and the reduced overall efficiency.

=== Setup your environment: What does an application need? ===

==== Executable ====
The binary executable should, theoretically, all be stored in '\usr\bin'. This never happens in practice, unless you are a fanatical fundamentalist of the early Linux releases. To guide your system to the desired executable, you can either laboriously type its absolute path every time you need it or add the path to the environmental variable:

<code>
~$ export PATH=${PATH}:path_to_bin
</code>

Running any executable in parallel requires mpi to coordinate all the processes/threads. The path to mpi executable is also required. Besides, many scientific codes require other specific environmental variables such as linear algebra packages. Read their documentations for further information.

==== .lib/.a/.o files ====

When writing a script, you might need some extra packages to do more complex jobs. Those packages are developed by experts in computer science and can be called by a line of code. The same thing happens when people were developing applications like CRYSTAL and ONETEP.

However, scientific computing codes are usually distributed in the form of source code. Source codes in FORTRAN/C/C++ need be compiled into a binary executable. There are 2 options during compiling:

# Include the whole package as long as one of its functions is called, also known as a 'static lib'.
# Only include a 'table of contents' when compiling, also known as 'dynamic lib'. The packages needed are separately stored in '.dll/.so' files, making it possible for multiple applications sharing the same lib.

Details about compilation are beyond the scope of this post. The thing is: when running a dynamically linked application, information should be given to help the code find the libs needed. This can be specified by:

<code>
~$ export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:path_to_lib`
</code>

For statically linked applications, usually you need not worry about it - but the volume of the compiled executable might make you wonder whether there is an alternative way.

==== Conflicts ====

Improper previous settings may lead to a wrong application, or a wrong version, if multiple applications with similar functions are installed in the system, such as Intel compiler and GCC, OpenMPI and MPICH - a common phenomenon for shared computing resources. To avoid this, the path to the undesired application or lib should be removed from the environmental variables.

==== Environmental Modules ====

Environmental Modules <ref>http://modules.sourceforge.net/</ref> is a popular software managing the necessary environmental setups and conflicts for each application. It can easily add or erase the environmental variables by commands (such as <code>module load</code>, <code>module rm</code>) and modulefiles written in Tool Command Language (TCL)<ref>https://www.tcl.tk/</ref>. The default directory of modulefiles is given in the environmental variable <code>${MODULEPATH}</code>, but files in other directories can also be loaded by their absolute path.

Both Imperial CX1 and ARCHER2 adopt this application, with which pre-compiled applications are offered.

=== The external coordinator: What is a batch system ===

Always bear in mind that the computational resources are limited, so you need to acquire reasonable resources for your job. Besides, the cluster also needs to calculate your budget, coordinate jobs submitted by various users, and make the best of available resources. When job is running, maybe you also want to check its status. All of this are fulfilled by batch systems.

In practice, a Linux shell script is needed. Parameters of the batch system of are set in the commented lines at the top of the file. After the user submit the script to batch system, the system will:

# Examine the parameters
# Allocate and coordinate the requested resources
# Set up the environments, such as environmental variables, package dependency, and sync the same setting to all nodes
# Launch a parallel calculation - see mpi part
# Post-process

Note that a 'walltime' is usually required for a batch job, i.e., the maximum allowed time for the running job. The job will be 'killed', or suspended, when the time exceeds the walltime, and the rest part of the script will not be executed. <code>timeout</code> command can be used to set another walltime for a specific command.

Common batch systems include PBS and Slurm <ref>https://slurm.schedmd.com/overview.html</ref>. For Imperial cluster CX1 and MMM Hub Young (managed by UCL) <ref>http://mmmhub.ac.uk/young/</ref>, PBS system is implemented; for ARCHER2 and Tianhe-2 LvLiang(天河二号-吕梁), Slurm is implemented. Tutorials of batch systems are not covered here, since they are heavily tailored according to specific machines - usually modifications are made to enhance the efficiency. Refer to the specific user documentations for more information.

Successfully setting and submitting a batch job script symbolises that you do not need this tutorial any more. Before being able to do that, some considerations might be important:

* How large is my system? Is it efficient to use the resources I requested(Note that it is not a linear-scaling problem... Refer to [https://tutorials.crystalsolutions.eu/tutorial.html?td=tuto_HPC&tf=tuto_hpc#scale this test] on CRYSTAL17)?
* To which queue should I submit my job? Is it too long/not applicable/not available?
* Is it safe to use multi-threading?
* Is it memory, GPU etc. demanding?
* Roughly how long will it take?
* What is my budget code? Do I have enough resources?
* Which MPI release version is my code compatible with? Should I load a module or set variables?
* Any other specific environmental setups does my code need?
* Do I have any post-processing script after MPI part is finished? How long does it take?

==References==
<references>