#rev 2020-09-10 stroth <> = CVL Slurm cluster = The [[https://vision.ee.ethz.ch/|Computer Vision Lab]] (CVL) owns a Slurm cluster with restricted access. The following information is an addendum to the main Slurm article in this wiki specific for usage of the CVL cluster. Furthermore, CVL maintains it's own wiki article to help you getting started and listing frequently asked questions. Consult these two articles if the information you're looking for isn't available here: * [[Services/SLURM|Computing wiki main Slurm article]] * [[https://wiki.vision.ee.ethz.ch/itet/gpuclusterslurm|CVL wiki slurm article]] == Setting environment == The environment variable SLURM_CONF needs to be adjusted to point to the configuration of the CVL cluster: {{{#!highlight bash numbers=disable export SLURM_CONF=/home/sladmcvl/slurm/slurm.conf }}} == Hardware == The following tables summarizes node specific information: ||'''Server''' ||'''CPU''' ||'''Frequency'''||'''Physical cores'''||'''Logical processors'''||'''Memory'''||'''/scratch SSD'''||'''/scratch Size'''||'''GPUs'''||'''Operating System'''|| ||biwirender04 ||Intel Xeon E5-2637 v2||3.50 GHz || 8 || 8 ||125 GB||✓||6.1 TB||5 Tesla K40c (11 GB)||Debian 10|| ||biwirender05 ||Intel Xeon E5-2637 v2||3.50 GHz || 8 ||16 ||251 GB||✓||6.1 TB||4 !GeForce GTX TITAN X (12 GB)||Debian 10|| ||biwirender06 ||Intel Xeon E5-2637 v2||3.50 GHz || 8 ||16 ||251 GB||✓||6.1 TB||5 !GeForce GTX TITAN X (12 GB)||Debian 10|| ||biwirender08 ||Intel Xeon E5-2640 v3||2.60 GHz ||16 ||32 ||251 GB||✓||701 GB||5 !GeForce GTX TITAN X (12 GB)||Debian 10|| ||biwirender09 ||Intel Xeon E5-2640 v3||2.60 GHz ||16 ||16 ||251 GB||✓||701 GB||3 !GeForce GTX TITAN X (12 GB)||Debian 10|| ||biwirender10 ||Intel Xeon E5-2650 v4||2.20 GHz ||24 ||24 ||251 GB||✓||701 GB||5 !GeForce GTX TITAN X (12 GB)||Debian 10|| ||biwirender11 ||Intel Xeon E5-2640 v3||2.60 GHz ||16 ||16 ||251 GB||✓||701 GB||5 !GeForce GTX TITAN X (12 GB)||Debian 10|| ||biwirender12 ||Intel Xeon E5-2640 v3||2.60 GHz ||16 ||32 ||251 GB||✓||701 GB||4 !GeForce RTX 2080 Ti (10 GB)||Debian 10|| ||biwirender13 ||Intel Xeon E5-2680 v3||2.50 GHz ||24 ||24 ||503 GB||✓||701 GB||5 TITAN Xp (12 GB)||Debian 10|| ||biwirender14 ||Intel Xeon E5-2680 v4||2.40 GHz ||28 ||28 ||503 GB||✓||701 GB||7 TITAN Xp (12 GB)||Debian 10|| ||biwirender15 ||Intel Xeon E5-2680 v4||2.40 GHz ||28 ||28 ||503 GB||✓||1.1 TB||7 TITAN Xp (12 GB)||Debian 10|| ||biwirender17 ||Intel Xeon E5-2620 v4||2.10 GHz ||16 ||32 ||503 GB||✓||403 GB||6 !GeForce GTX 1080 Ti (11 GB)||Debian 10|| ||biwirender20 ||Intel Xeon E5-2620 v4||2.10 GHz ||16 ||32 ||376 GB||✓||403 GB||6 !GeForce GTX 1080 Ti (11 GB)||Debian 10|| ||bmicgpu01 ||Intel Xeon E5-2680 v3||2.50 GHz ||24 ||24 ||251 GB||✓||1.1 TB||6 TITAN X (12 GB)||Debian 10|| ||bmicgpu02 ||Intel Xeon E5-2640 v3||2.60 GHz ||16 ||16 ||251 GB||✓||692 GB||5 TITAN Xp (12 GB)||Debian 10|| ||bmicgpu03 ||Intel Xeon E5-2630 v4||2.20 GHz ||20 ||40 ||251 GB||✓||1.1 TB||5 TITAN Xp (12 GB)||Debian 10|| ||bmicgpu05 ||Intel Xeon E5-2630 v4||2.20 GHz ||20 ||20 ||251 GB||✓||1.1 TB||5 TITAN Xp (12 GB)||Debian 10|| ||bmicgpu05 ||Intel Xeon E5-2630 v4||2.20 GHz ||20 ||20 ||251 GB||✓||1.1 TB||4 TITAN Xp (12 GB)||Debian 10|| ||bmicgpu06 ||AMD EPYC 7742 ||1.50 GHz ||128 ||128 ||503 GB||✓||1.8 TB||4 A100 (40 GB)<
>1 A100 (80GB) ||Debian 10|| ||bmicgpu07 ||AMD EPYC 7763 ||1.50 GHz ||128 ||128 ||755 GB||✓||6.9 TB||8 A6000 (48 GB)||Debian 11|| ||bmicgpu08 ||AMD EPYC 7763 ||1.50 GHz ||128 ||128 ||755 GB||✓||6.9 TB||8 A6000 (48 GB)||Debian 11|| ||bmicgpu09 ||AMD EPYC 7763 ||1.50 GHz ||128 ||128 ||755 GB||✓||6.9 TB||8 A6000 (48 GB)||Debian 11|| Detailed information about all nodes can be seen by issuing the command {{{#!highlight bash numbers=disable scontrol show nodes }}} An overview of utilization of individual node's resources can be shown with: {{{#!highlight bash numbers=disable sinfo --Format nodehost:14,statecompact:7,cpusstate:16,cpusload:11,memory:8,allocmem:10,gres:55,gresused:62,reason:10 }}} (Adapt the field length for gres and gresused to your needs) == Automatic resource assignment == As the [[#Hardware|hardware outfit]] of nodes is heterogenous, resource allocation is controlled automatically to maximise utilization and simplify job submission for most use cases: * Jobs not explicitely specifying resource allocations receive defaults * Upper limits on resource allocations are imposed on all jobs * Run time for interactive jobs is limited to 2 hours (Exceptions: See [[#gpu.debug|gpu.debug]]) These defaults and limits differ by [[#Partitions|partition]]. For details, see the ''job submit script'' `/home/sladmcvl/slurm/job_submit.lua` which is interpreted for each job by the Slurm scheduler to set defaults and enforce limits. /!\ Don't use the '''--mem''' and/or '''--cpus-per-task''' options for GPU jobs outside of the defaults and limits. This can create conditions which are impossible to satisfy by the Slurm scheduler and obfuscate the reason why a job cannot be scheduled. Such conditions will results in the following error message: {{{ srun: error: Unable to allocate resources: Requested node configuration is not available }}} To properly warn about impossible conditions, the ''job submit script'' would have to duplicate information about partitions and node outfits, which leads to maintenance overhead and introduces more sources for errors. This would defeat its purpose of simplifying job submission. == Partitions == Partitions group nodes with similar a [[#Hardware|hardware outfit]] together. Their defaults and limits are shown in the following table: ||'''Partition'''||'''DefMPG'''||'''MaxMPG'''||'''DefCPG'''||'''MaxCPG'''||'''Time limit'''|| ||gpu.low.normal ||30 GB||30 GB||3 ||3||2 d|| ||gpu.medium.normal||30 GB||50 GB||3 ||5||2 d|| ||gpu.medium.long ||30 GB||50 GB||3 ||5||5 d|| ||gpu.high.normal ||50 GB||70 GB||4 ||4||2 d|| ||gpu.high.long ||50 GB||70 GB||4 ||4||5 d|| ||gpu.debug ||30 GB||70 GB||3 ||5||8 h|| ||gpu.bmic ||64 GB||- ||16||-||2 d|| '''Def''': Default, '''Max''': Maximum, '''MPG''': Memory Per GPU, '''CPG''': CPUs Per GPU === gpu.debug === This partition is reserved to run interactive jobs for debugging purposes. A typical use case is to start an interactive shell for a short time in this partitions to debug a job script. In the following example, the time limit is set to 10 minutes: {{{#!highlight bash numbers=disable srun --time 10 --gres=gpu:1 --partition=gpu.debug --pty bash -i }}} === cpu.bmic, gpu.bmic === Access to these partitions is restricted to members of the [[https://bmic.ee.ethz.ch/the-group.html|BMIC group]]. ==== Access to cpu.bmic, gpu.bmic ==== Access to the partitions `cpu.bmic` and `gpu.bmic` are available for members of the Slurm account `bmic` only. You can check your Slurm account membership with the following command: {{{#!highlight bash numbers=disable sacctmgr show users WithAssoc Format=User%-15,DefaultAccount%-15,Account%-15 ${USER} }}} If you're a member of the Slurm account `staff` and have also been added to `bmic`, your default account is the latter and all your jobs will by default be sent to partition `gpu.bmic`.<
> If you do not ask for GPU resources your jobs will be sent to partition `cpu.bmic`.<
> If you want to have your jobs sent to other partitions, you have to specify the account `staff` as in the following example: {{{#!highlight bash numbers=disable sbatch --account=staff job_script.sh }}} If you already have a PENDING job in the wrong partition you can correct it to partition `` by issuing the following command: {{{#!highlight bash numbers=disable scontrol update jobid= partition= account=staff }}} If you want to send your jobs to nodes in other partitions, make sure to always specify `--account=staff`. Job quotas are calculated per account, by setting the account to `staff` you will make sure not to use up your quota from account `bmic` on nodes in partitions outside of `gpu.bmic`. ==== Time limit for interactive jobs ==== In `gpu.bmic` the time limit for interactive jobs is 8 h. === *.long === The *.long partitions are only accessible to members of the account "long". Membership is temporary and granted on demand by [[https://wiki.vision.ee.ethz.ch/itet/gpuclusterslurm|CVL administration|]]. == Display specific information == The following is a collection of command sequences to quickly extract specific summaries. === GPU availability === Information about the GPU nodes and current availability of the installed GPUs is updated every 5 minutes to the file `/home/sladmcvl/smon.txt`. Here are some convenient aliases to display the file with highlighting of either free GPUs or those running the current user's jobs: {{{#!highlight bash numbers=disable alias smon_free="grep --color=always --extended-regexp 'free|$' /home/sladmcvl/smon.txt" alias smon_mine="grep --color=always --extended-regexp '${USER}|$' /home/sladmcvl/smon.txt" }}} For monitoring its content the following aliases can be used: {{{#!highlight bash numbers=disable alias watch_smon_free="watch --interval 300 --no-title --differences --color \"grep --color=always --extended-regexp 'free|$' /home/sladmcvl/smon.txt\"" alias watch_smon_mine="watch --interval 300 --no-title --differences --color \"grep --color=always --extended-regexp '${USER}|$' /home/sladmcvl/smon.txt\"" }}} === GPU quota === A slurm user is member of a so-called slurm account. Accounts are associated with so-called quality of service (qos) rules. The amount of GPUs an account member's jobs can use at the same time, a.k.a a quota is defined in a qos by the same name as the account. These qos can be shown with the following command: {{{#!highlight bash numbers=disable sacctmgr show qos format=name%8,maxtrespu%12 }}} === GPUs per user === Show a sorted list of users, their account an QOS and a summary of the GPU's used by their running jobs: {{{#!highlight bash numbers=disable ( echo 'User;Account;QOS;GPUs' \ && echo '----;-------;---;----' \ && scontrol -a show jobs \ |grep -E '(UserId|Account|JobState|TRES)=' \ |paste - - - - \ |grep -E 'JobState=RUNNING.*gres/gpu' \ |sed -E 's:^\s+UserId=([^\(]+).*Account=(\S+)\s+QOS=(\S+).*gres/gpu=([0-9]+)$:\1_\2_\3;\4:' \ |awk -F ';' -v OFS=';' '{a[$1]+=$2}END{for(i in a) print i,a[i]}' \ |sort \ |tr '_' ';' ) \ |column -s ';' -t }}}