CVL Slurm cluster

The Computer Vision Lab (CVL) owns a Slurm cluster with restricted access. The following information is an addendum to the main Slurm article in this wiki specific for usage of the CVL cluster. Furthermore, CVL maintains it's own wiki article to help you getting started and listing frequently asked questions. Consult these two articles if the information you're looking for isn't available here:

• Computing wiki main Slurm article

• CVL wiki slurm article

Setting environment

The environment variable SLURM_CONF needs to be adjusted to point to the configuration of the CVL cluster:

export SLURM_CONF=/home/sladmcvl/slurm/slurm.conf

Hardware

The following tables summarizes node specific information:

Detailed information about all nodes can be seen by issuing the command

scontrol show nodes

An overview of utilization of individual node's resources can be shown with:

sinfo --Format nodehost:14,statecompact:7,cpusstate:16,cpusload:11,memory:8,allocmem:10,gres:55,gresused:62,reason:10

(Adapt the field length for gres and gresused to your needs)

Automatic resource assignment

As the hardware outfit of nodes is heterogenous, resource allocation is controlled automatically to maximise utilization and simplify job submission for most use cases:

• Jobs not explicitely specifying resource allocations receive defaults
• Upper limits on resource allocations are imposed on all jobs

These defaults and limits differ by partition. For details, see the job submit script /home/sladmcvl/slurm/job_submit.lua which is interpreted for each job by the Slurm scheduler to set defaults and enforce limits.

Don't use the --mem and/or --cpus-per-task options for GPU jobs outside of the defaults and limits. This can create conditions which are impossible to satisfy by the Slurm scheduler and obfuscate the reason why a job cannot be scheduled. Such conditions will results in the following error message:

srun: error: Unable to allocate resources: Requested node configuration is not available

To properly warn about impossible conditions, the job submit script would have to duplicate information about partitions and node outfits, which leads to maintenance overhead and introduces more sources for errors. This would defeat its purpose of simplifying job submission.

Partitions

Partitions group nodes with similar a hardware outfit together. Their defaults and limits are shown in the following table:

Def: Default, Max: Maximum, MPG: Memory Per GPU, CPG: CPUs Per GPU

gpu.debug

This partition is reserved to run interactive jobs for debugging purposes. A typical use case is to start an interactive shell for a short time in this partitions to debug a job script. In the following example, the time limit is set to 10 minutes:

srun --time 10 --gres=gpu:1 --partition=gpu.debug --pty bash -i

gpu.bmic

Access to this partition is restricted to members of the BMIC group.

Access to gpu.bmic

Access to the partition gpu.bmic is available for members of the Slurm account bmic. You can check your Slurm account membership with the following command:

sacctmgr show users WithAssoc Format=User%-15,DefaultAccount%-15,Account%-15 ${USER}

If you're a member of the Slurm account staff and have also been added to bmic, your default account is the latter and all your jobs will by default be sent to partition gpu.bmic.
If you want to have your jobs sent to other partitions, you have to specify the account staff as in the following example:

sbatch --account=staff job_script.sh

If you already have a PENDING job in the wrong partition you can correct it to partition <partition name> by issuing the following command:

scontrol update jobid=<job id> partition=<partition name> account=staff

If you want to send your jobs to nodes in other partitions, make sure to always specify --account=staff. Job quotas are calculated per account, by setting the account to staff you will make sure not to use up your quota from account bmic on nodes in partitions outside of gpu.bmic.

*.long

The *.long partitions are only accessible to members of the account "long". Membership is temporary and granted on demand by CVL administration.

Display specific information

The following is a collection of command sequences to quickly extract specific summaries.

GPU availability

Information about the GPU nodes and current availability of the installed GPUs is updated every 5 minutes to the file /home/sladmcvl/smon.txt. Here are some convenient aliases to display the file with highlighting of either free GPUs or those running the current user's jobs:

alias smon_free="grep --color=always --extended-regexp 'free|$' /home/sladmcvl/smon.txt"
alias smon_mine="grep --color=always --extended-regexp '${USER}|$' /home/sladmcvl/smon.txt"

For monitoring its content the following aliases can be used:

alias watch_smon_free="watch --interval 300 --no-title --differences --color \"grep --color=always --extended-regexp 'free|$' /home/sladmcvl/smon.txt\""
alias watch_smon_mine="watch --interval 300 --no-title --differences --color \"grep --color=always --extended-regexp '${USER}|$' /home/sladmcvl/smon.txt\""

GPU quota

A slurm user is member of a so-called slurm account. Accounts are associated with so-called quality of service (qos) rules. The amount of GPUs an account member's jobs can use at the same time, a.k.a a quota is defined in a qos by the same name as the account. These qos can be shown with the following command:

sacctmgr show qos format=name%8,maxtrespu%12

GPUs per user

Show a sorted list of users, their account an QOS and a summary of the GPU's used by their running jobs:

(
    echo 'User;Account;QOS;GPUs' \
    && echo '----;-------;---;----' \
    && scontrol -a show jobs \
    |grep -E '(UserId|Account|JobState|TRES)=' \
    |paste - - - - \
    |grep -E 'JobState=RUNNING.*gres/gpu' \
    |sed -E 's:^\s+UserId=([^\(]+).*Account=(\S+)\s+QOS=(\S+).*gres/gpu=([0-9]+)$:\1_\2_\3;\4:' \
    |awk -F ';' -v OFS=';' '{a[$1]+=$2}END{for(i in a) print i,a[i]}' \
    |sort \
    |tr '_' ';'
) \
|column -s ';' -t