IV - Specialized Jobs

Environment Variables

An environment variable describes something about your working environment. Some of them you can set or modify; others are set by the system. To see what is currently set in your terminal, run

$printenv

To set an environment variable yourself, use the export command.

$export VAR=value

When your job starts, SLURM will initialize several environment variables. Many of them correspond to options you have set in your SBATCH preamble. Do not attempt to assign to any variable beginning with SLURM_

Some variables that you may wish to examine or use in your scripts:

Interactive Jobs

Most HPC sites, including UVa’s, restrict the memory and time allowed to processes on the login nodes. Most jobs can be submitted through the batch system we have been discussing, but sometimes more interactive work is required. For example

1. Jobs that must be or are best run through a graphical interface,
2. Short development jobs,
3. “Computational steering” in which a program runs for an interval, then the output is examined and parameters may be adjusted.

For most of these cases, we strongly recommend the use of the Open OnDemand Interactive Applications. Jupyterlab is available to run notebooks. Rstudio and Matlab Desktop are also available to run through this interface. For more general work, including command-line options, the Desktop is usually the best option. It provides a basic terminal, but also access to other applications should they be needed.

For general-purpose interactive work with graphics, please use the Open OnDemand Desktop. The X11 service that Linux uses for graphics is very slow over a network. Even with a fast connection between two systems, the Desktop will perform better since the X11 server process and the programs that use it are running on the same computer.

If you must use a basic terminal for an interactive job, you must first use the command salloc. This is the general Slurm command to request resources. This would be followed by srun to launch the processes. However, this is complex and requires knowledge of the options, so we have provided a local “wrapper” script called ijob.

ijob takes options similar to those used with SBATCH, most of which are actually arguments to salloc.

$ijob –c 1 –A myalloc -t <time> --mem <memory in MB> -p <partition> -J <jobname>

When the job starts you will be logged in to a bash shell in a terminal on the compute node.

Multicore and Multinode Jobs

One of the advantages of using a high-performance cluster is the ability to use many cores and/or nodes at once. This is called parallelism. There are three main types of parallelism.

You should understand whether your program can make use of more than one core or node before you request multiple cores and/or nodes. Special programming is required to enable these capabilities. Asking for multiple cores or nodes that your program cannot use will result in idle cores and wasted SUs, since you are charged for each core-hour. The seff command can help with this.

High Throughput Serial Parallelism

High throughput parallelism is when many identical jobs are run at once, each on a single core. Examples can include Monte-Carlo methods, parameter searches, image processing on many related images, some areas of bioinformatics, and many others. For most cases of this type of parallelism, the best Slurm option is a job array.

When planning a high-throughput project, it is important to keep in mind that if the individual jobs are very short, less than roughly 15-30 minutes each, it is very inefficient to run each one separately, whether you do this manually or through an array. In this case you should group your jobs and run multiple instances within the same job script. Please contact us if you would like assistance setting this up.

Multicore (Threaded)

Shared-memory programs can use multiple cores but they must be physically located on the same node. The appropriate Slurm option in this case is -c (equivalent to cpus-per-task). Shared memory programs use threading of one form or another.

Example Slurm script for a threaded program:

#SBATCH -n 1
#SBATCH -c 25
#SBATCH -p interactive
#SBATCH -A hpc_training
#SBATCH -t 05:00:00
#SBATCH --mail-user=mst3k@virginia.edu
#SBATCH --mail-type=END

module load gcc
export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK

./ompheatedplate .0000001 plate

Multinode (MPI)

In this type of parallelism, each process runs independently and communicates with others through a library, the most widely-used of which is MPI. Distributed memory programs can run on single or multiple nodes and often can run on hundreds or even thousands of cores. For distributed-memory programs you can use the -N option to request a number of nodes, along with ntasks-per-node to schedule a number of processes on each of those nodes.

#!/bin/bash
#SBATCH --nodes=3
#SBATCH --ntasks-per-node=40
#SBATCH --account=hpc_training
#SBATCH -p parallel
#SBATCH -t 10:00:00
#SBATCH --mail-user=mst3k@virginia.edu
#SBATCH --mail-type=END

module load gcc openmpi

srun ./mpiheatedplate .0000001 plate${SLURM_NTASKS_PER_NODE}

Hybrid MPI plus Threading

Some codes can run with distributed-memory processes, each of which can run in threaded mode. For this, request --ntasks-per-node=NT and cpus-per-task=NC, keeping in mind that the total number of cores requested on each node is then $NT \times NC$.

#!/bin/bash
#SBATCH -N 3
#SBATCH --ntasks-per-node=4
#SBATCH -c 10
#SBATCH -p parallel
#SBATCH -A hpc_training
#SBATCH -t 05:00:00
#SBATCH --mail-user=mst3k@virginia.edu
#SBATCH --mail-type=END

module load gcc
export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK

srun ./heatedplate .0000001 plate${SLURM_NTASKS_PER_NODE}

Job Arrays

Many similar jobs can be submitted simultaneously through job arrays. There are some restrictions:

• It must be a batch job.
• Job arrays should be explicitly named with -J
• It is generally prudent to separate stdout and stderror with -o and -e

A job array is submitted with sbatch --array=<range>, where range is two digits separated by a hyphen.

$sbatch --array=0-30 myjobs.sh

An increment can be provided

$sbatch --array=1-7:2 myjobs.sh

This will number them 1, 3, 5, 7

It is also possible to provide a list

$sbatch --array=1,3,4,5,7,9 myjobs.sh

Each job will be provided an environment variable SLURM_ARRAY_JOB_ID and each task will be assigned a SLURM_ARRAY_TASK_ID. The ARRAY_JOB_ID is the overall jobid, whereas the ARRAY_TASK_ID will take on the values of the numbers in the specified range or list.

Slurm also provides two variables %A (global array ID) and %a (array task ID) which can be used in the -o and -e options. If they are not used, then the different tasks will attempt to write to the same file, which can result in garbled output or file corruption, so please use them if you wish to redirect streams with those options.

To prepare a job array, set up any input files using appropriate names that will correspond to the numbers in your range or list, e.g.

myinput.0.in
myinput.1.in
...
myinput.30.in

You would submit a job for the above files with

$sbatch --array=0-30

In your Slurm script you would use a command such as

python myscript.py myinput.${SLURM_ARRAY_TASK_ID}.in

The script should be prepared to request resources for one instance of your program.

Complete example array job script:

#!/bin/bash
#SBATCH --ntasks=1
#SBATCH --partition=interactive
#SBATCH -A hpc_training
#SBATCH --time=3:00:00
#SBATCH -o out%A.%a
#SBATCH -e err%A.%a

python myscript.py myinput.${SLURM_ARRAY_TASK_ID}.in

To cancel an entire array, cancel the global ID

scancel 1283839

You can also cancel individual tasks

scancel 1283839_11

Useful Commands

When you submit a job and it doesn’t start or fails for an unknown reason it could be due to restraints in your account. This could include running out of storage space or SUs on your allocation. Additionally, it’s useful to see how busy the queue is. The following subsections highlight how to identify these problems.

Allocations

Sometimes it’s useful to check how many SUs are still available on your allocation. The allocations command displays information on your allocations and how many SUs are associated with them:

$ allocations
Account                      Balance        Reserved       Available                
-----------------          ---------       ---------       ---------
hpc_training                 1000000               0          999882 

running allocations -a <allocation_name> provides even more detail on when the allocation was last renewed and its members. E.g.

$ allocations -a hpc_training
Description StartTime           EndTime    Allocated   Remaining  PercentUsed Active 
----------- ------------------- ---------- ----------- ---------- ----------- ------ 
new         2024-05-29 17:33:13 2025-05-29 1000000.000 999881.524        0.01 True   

Name   Active CommonName                     EmailAddress        DefaultAccount         
------ ------ ------------------------------ ------------------- ----------------------
.
.
.

Storage Quota

One way to check your storage utilization is with the hdquota command. This command will show you how much of your home, scratch, and leased (if applicable) storage are being utilized. Below is the sample output for hdquota:

$ hdquota
Type             Location         Name                                        Size Used Avail Use%
====================================================================================================
home             /home            mst3k                                       50G   16G   35G  32%
Scratch          /scratch         mst3k                                       12T  2.0T   11T  17%

This is a useful command to check whether you’re running out of storage space or to see where files need to be cleaned up. For more detailed information on disk utilization you may also use the du command to investigate specific directories.

Queue limits and Usage

To gain information on the different queues you can use the qlist command. This will show the list of partitions, their usage, and the SU charge rate. You can use qlimits for information on each queue’s limits.

The sinfo command will provide some more detailed information on the health of each queue and the number of active nodes available. These commands can be useful in diagnosing why a job may not be running, or to better understand the queue usage for more efficient job throughput. More information on hardware specifications and queue information can be found here on our website.

Need Help

Research Computing is ready to help you learn to use our systems efficiently. You can submit a ticket. For in-person help, please attend one of our weekly sessions of office hours.