CVL Slurm cluster

The Computer Vision Lab (CVL) owns a Slurm cluster with restricted access. The following information is an addendum to the main Slurm article in this wiki specific for usage of the CVL cluster. Furthermore, CVL maintains it's own wiki article to help you getting started and listing frequently asked questions. Consult these two articles if the information you're looking for isn't available here:

• Computing wiki main Slurm article

• CVL wiki slurm article

Setting environment

The environment variable SLURM_CONF needs to be adjusted to point to the configuration of the CVL cluster:

export SLURM_CONF=/home/sladmcvl/slurm/slurm.conf

Hardware

The following tables summarizes node specific information:

Detailed information about all nodes can be seen by issuing the command

scontrol show nodes

An overview of utilization of individual node's resources can be shown with:

sinfo --Format nodehost:14,statecompact:7,cpusstate:16,cpusload:11,memory:8,allocmem:10,gres:55,gresused:62,reason:10

(Adapt the field length for gres and gresused to your needs)

Automatic resource assignment

As the hardware outfit of nodes is heterogenous, resource allocation is controlled automatically to maximise utilization and simplify job submission for most use cases:

• Jobs not explicitely specifying resource allocations receive defaults
• Upper limits on resource allocations are imposed on all jobs

These defaults and limits differ by partition. For details, see the job submit script /home/sladmcvl/slurm/job_submit.lua which is interpreted for each job by the Slurm scheduler to set defaults and enforce limits.

Don't use the --mem and/or --cpus-per-task options for GPU jobs outside of the defaults and limits. This can create conditions which are impossible to satisfy by the Slurm scheduler and obfuscate the reason why a job cannot be scheduled. Such conditions will results in the following error message:

srun: error: Unable to allocate resources: Requested node configuration is not available

To properly warn about impossible conditions, the job submit script would have to duplicate information about partitions and node outfits, which leads to maintenance overhead and introduces more sources for errors. This would defeat its purpose of simplifying job submission.

Partitions

Partitions group nodes with similar a hardware outfit together. Their defaults and limits are shown in the following table:

Def: Default, Max: Maximum, MPG: Memory Per GPU, CPG: CPUs Per GPU

gpu.debug

This partition is reserved to run interactive jobs for debugging purposes. If a job doesn't run a process on an allocated GPU after 20 minutes it will be killed.

*.long

The *.long partitions are only accessible to members of the account "long". Membership is temporary and granted on demand by <contact to be filled in>.

Display specific information

The following is a collection of command sequences to quickly extract specific summaries.

GPU usage

Information about the GPU nodes and usage of the installed GPUs is updated every 5 minutes to the file /home/sladmcvl2/smon.txt. Here are some convenient aliases to display the file with highlighting of either free GPUs or those running the current user's jobs:

alias smon_free="grep --color=always --extended-regexp 'free|$' /home/sladmcvl2/smon.txt"
alias smon_mine="grep --color=always --extended-regexp '${USER}|$' /home/sladmcvl2/smon.txt"

For monitoring its content the following aliases can be used:

alias watch_smon_free="watch --interval 300 --no-title --differences --color \"grep --color=always --extended-regexp 'free|$' /home/sladmcvl2/smon.txt\""
alias watch_smon_mine="watch --interval 300 --no-title --differences --color \"grep --color=always --extended-regexp '${USER}|$' /home/sladmcvl2/smon.txt\""

GPU quota

A slurm user is member of a so-called slurm account. Accounts are associated with so-called quality of service (qos) rules. The amount of GPUs an account member's jobs can use at the same time, a.k.a a quota is defined in a qos by the same name as the account. These qos can be shown with the following command:

sacctmgr show qos format=name%8,maxtrespu%12

GPUs per user

Show a sorted list of users, their account an QOS and a summary of the GPU's used by their running jobs:

(
    echo 'User;Account;QOS;GPUs' \
    && echo '----;-------;---;----' \
    && scontrol -a show jobs \
    |grep -E '(UserId|Account|JobState|TRES)=' \
    |paste - - - - \
    |grep -E 'JobState=RUNNING.*gres/gpu' \
    |sed -E 's:^\s+UserId=([^\(]+).*Account=(\S+)\s+QOS=(\S+).*gres/gpu=([0-9]+)$:\1_\2_\3;\4:' \
    |awk -F ';' -v OFS=';' '{a[$1]+=$2}END{for(i in a) print i,a[i]}' \
    |sort \
    |tr '_' ';'
) \
|column -s ';' -t