CVL Slurm cluster

The Computer Vision Lab (CVL) owns a Slurm cluster with restricted access. The following information is an addendum to the main Slurm article in this wiki specific for usage of the CVL cluster. Furthermore, CVL maintains it's own wiki article to help you getting started and listing frequently asked questions. Consult these two articles if the information you're looking for isn't available here:

Setting environment

The environment variable SLURM_CONF needs to be adjusted to point to the configuration of the CVL cluster:

export SLURM_CONF=/home/sladmcvl/slurm/slurm.conf

Hardware

The following tables summarizes node specific information:

Server

CPU

Frequency

Physical cores

Logical processors

Memory

/scratch SSD

GPUs

Operating System

bender[01]

Intel Xeon E5-2670 v2

2.50 GHz

20

40

125 GB

-

-

Debian 9

bender[02]

Intel Xeon E5-2670 v2

2.50 GHz

20

20

125 GB

-

-

Debian 9

bender[03-06]

Intel Xeon E5-2670 v2

2.50 GHz

20

40

125 GB

-

-

Debian 9

bender[39-52]

Intel Xeon X5650

2.67 GHz

24

48

94 GB

-

-

Debian 9

bender[53-70]

Intel Xeon E5-2665 0

2.40 GHz

32

64

125 GB

-

-

Debian 9

bmiccomp01

Intel Xeon E5-2697 v4

2.30 GHz

36

36

251 GB

-

-

Debian 9

biwirender03

Intel Xeon E5-2650 v2

2.60 GHz

16

32

125 GB

-

6 Tesla K40c (11 GB)

Debian 9

biwirender04

Intel Xeon E5-2637 v2

3.50 GHz

8

8

125 GB

5 Tesla K40c (11 GB)

Debian 9

biwirender[05,06]

Intel Xeon E5-2637 v2

3.50 GHz

8

16

251 GB

5 GeForce GTX TITAN X (12 GB)

Debian 9

biwirender[07,09]

Intel Xeon E5-2640 v3

2.60 GHz

16

16

251 GB

5 GeForce GTX TITAN X (12 GB)

Debian 9

biwirender[08]

Intel Xeon E5-2640 v3

2.60 GHz

16

32

251 GB

5 GeForce GTX TITAN X (12 GB)

Debian 9

biwirender10

Intel Xeon E5-2650 v4

2.20 GHz

24

24

251 GB

5 GeForce GTX TITAN X (12 GB)

Debian 9

biwirender11

Intel Xeon E5-2640 v3

2.60 GHz

16

16

251 GB

5 GeForce GTX TITAN X (12 GB)

Debian 9

biwirender12

Intel Xeon E5-2640 v3

2.60 GHz

16

32

251 GB

6 GeForce RTX 2080 Ti (10 GB)

Debian 9

biwirender13

Intel Xeon E5-2680 v3

2.50 GHz

24

24

503 GB

6 TITAN Xp (12 GB)

Debian 9

biwirender14

Intel Xeon E5-2680 v4

2.40 GHz

28

28

503 GB

7 TITAN Xp (12 GB)

Debian 9

biwirender15

Intel Xeon E5-2680 v4

2.40 GHz

28

28

503 GB

7 TITAN Xp (12 GB)

Debian 9

biwirender17

Intel Xeon E5-2620 v4

2.10 GHz

16

32

503 GB

8 GeForce GTX 1080 Ti (11 GB)

Debian 9

biwirender20

Intel Xeon E5-2620 v4

2.10 GHz

16

32

377 GB

8 GeForce GTX 1080 Ti (11 GB)

Debian 9

bmicgpu01

Intel Xeon E5-2680 v3

2.50 GHz

24

24

251 GB

6 TITAN X (12 GB)

Debian 9

bmicgpu02

Intel Xeon E5-2640 v3

2.60 GHz

16

16

251 GB

5 TITAN Xp (12 GB)

Debian 9

bmicgpu03

Intel Xeon E5-2630 v4

2.20 GHz

20

20

251 GB

6 TITAN Xp (12 GB)

Debian 9

bmicgpu[04,05]

Intel Xeon E5-2630 v4

2.20 GHz

20

20

251 GB

5 TITAN Xp (12 GB)

Debian 9

Detailled information about all nodes can be seen by issuing the command

scontrol show nodes

An overview of utilization of individual node's resources can be shown with:

sinfo --Format nodehost:14,statecompact:7,cpusstate:16,cpusload:11,memory:8,allocmem:10,gres:55,gresused:62,reason:10

(Adapt the field length for gres and gresused to your needs)

Partitions

Partitions including their limits are shown in the following table:

Partition

DefMPG

MaxMPG

DefCPG

MaxCPG

Time limit

cpu.medium.normal

-

-

-

-

2 d

gpu.low.normal

20 GB

25 GB

3

3

2 d

gpu.medium.normal

40 GB

50 GB

3

5

2 d

gpu.medium.long

40 GB

50 GB

3

5

5 d

gpu.high.normal

70 GB

70 GB

4

4

2 d

gpu.high.long

70 GB

70 GB

4

4

5 d

gpu.debug

20 GB

25 GB

3

3

8 h

Def: Default, Max: Maximum, MPG: Memory Per GPU, CPG: CPUs Per GPU

gpu.debug

This partition is reserved to run interactive jobs for debugging purposes. If a job doesn't run a process on an allocated GPU after 20 minutes it will be killed.

*.long

The *.long partitions are only accessible to members of the account "long". Membership is temporary and granted on demand by <contact to be filled in>.

Display specific information

The following is a collection of command sequences to quickly extract specific summaries.

GPU quota

A slurm user is member of a so-called slurm account. Accounts are associated with so-called quality of service (qos) rules. The amount of GPUs an account member's jobs can use at the same time, a.k.a a quota is defined in a qos by the same name as the account. These qos can be shown with the following command:

sacctmgr show qos format=name%8,maxtrespu%12

GPUs per user

Show a sorted list of users, their account an QOS and a summary of the GPU's used by their running jobs:

(
    echo 'User;Account;QOS;GPUs' \
    && echo '----;-------;---;----' \
    && scontrol -a show jobs \
    |grep -E '(UserId|Account|JobState|TRES)=' \
    |paste - - - - \
    |grep -E 'JobState=RUNNING.*gres/gpu' \
    |sed -E 's:^\s+UserId=([^\(]+).*Account=(\S+)\s+QOS=(\S+).*gres/gpu=([0-9]+)$:\1_\2_\3;\4:' \
    |awk -F ';' -v OFS=';' '{a[$1]+=$2}END{for(i in a) print i,a[i]}' \
    |sort \
    |tr '_' ';'
) \
|column -s ';' -t

Services/SLURM-Biwi (last edited 2020-09-15 07:26:01 by stroth)