How to run on ARCHER 2

This is a document for Research Group Cucinotta and Research Group Harrison guiding you through the setups for EPCC ARCHER2 and perform CP2K calculations there:

If you wish to perform calculation on ARCHER2 in the group, typically it means you have a fairly large system to compute, and your calculation may not proceed or proceed very slowly on CX1. You then may consult your group leader, Dr. Cucinotta, for permission to switch to run on ARCHER2;
A budget will be applied as an annual basis. Currently, the budget we (RGC) are using is: e05-surfin-clo (not exclusive);
You will need to apply for an ARCHER2 account at SAFE: https://safe.epcc.ed.ac.uk/. The details for registering on SAFE could be found here: https://epcced.github.io/safe-docs/safe-for-users/#register. For accessing ARCHER2, you will need to generate a SSH keypair;
If you are an UNIX (Mac or so) user, the SSH keypair generation scheme is:
1. Open your terminal window and type: ssh-keygen -t rsa;
2. And the SSH keypair will be generated;
3. You will be asked to setup a password for your SSH keypair. A kind reminder, a keypair with a password is definitely more secure; however, if your SSH keypair has a password, then every time when you login to ARCHER2, you will need to type in two passwords, one for your ARCHER2 account and one for your keypair.
4. If you want to visualize your keypair, you will find them in a hidden directory. To view the hidden directory, press command+shift+.;
Copy your public SSH key to SAFE and do not share your private key to anyone;
Sample script 2.
Then you should be able to login to ARCHER2. You will need to use the initial password for ARCHER2 provided by SAFE and then the first time you login, you will setup a new password.
ARCHER2 is different from CX1. In CX1, you will be performing your calculation and saving your results in the same directory, the home directory. However, on ARCHER2, you will be submitting your calculation on a different directory, called the work directory, and then after you finish your calculations, move the data to home for permanent (not really, if anyone promise you anything you have is permanent, slap the person’s face) storage. So, the first wise thing to do is to check the access of the two directories. For example, your home directory will have the path: /home/e05/e05/[your username] and your work directory will have a path: /work/e05/e05/[your username].
Job scripts:
1. You will be using CP2K/8.1 version, the latest version of CP2K. Check the CP2K website for some keywords’ changes (Sample script 1).
2. If your calculation will involve hybrid functional calculations, things turns to be tricker. Luckily, someone has already resolved the problem for you. This is what you need to do:
  1. Unfortunately, unlike ARCHER, ARCHER2 has not opened the “large memory” queue yet. So you will need to find a way of tuning the nodes and cores to balance the memory.
  2. The first thing you will need to do is to increase the number of nodes. For example, a given system, if you perform GGA calculation, you may only need 4 nodes; however, a hybrid calculation will at least demand a multiplication of 4 to 6 of the nodes.
  3. Switch cp2k.popt to cp2k.psmp;
  4. If you still have memory issues, then, decrease the number of tasks per node and increase the cpus per task. This should solve your problem (Sample script 2).
Finally, if you are a loyal user of PBS instead of slurm, and you find using the slurm command for submitting jobs and checking status cumbersome; you may setup something to allow you use “qstat”, “qsub”; most importantly, if you do not want to cd into your work directory every time, it is also helpful to setup an alias for change directory into work directory. However, I will leave this up to you and will not iterate it here. Best of luck and always have fun.