#rev 2020-09-10 stroth <> == Introduction == At D-ITET the Slurm job scheduling sytem can be used for running compute-intensive jobs. It consists of a master host, where the scheduler resides and the compute nodes, where batch jobs are executed. The compute nodes are powerful servers located in server rooms, they are exclusively reserved for batch processing. Interactive logins are disabled. === Access === Access to the Slurm cluster is reserved for staff of the contributing institutes '''APS, MINS, TIK''' and '''PBL'''.<
> Access is granted on request: * Members of an institute supported by [[https://www.s4d.id.ethz.ch/|ID Services for Departments]] (S4D), use the email address listed for your institute instead * Contact [[mailto:support@ee.ethz.ch|ISG D-ITET support]] if your institute is supported by us If your circumstances differ and you'd still like to use the cluster, please contact [[mailto:support@ee.ethz.ch|ISG D-ITET support]] as well and ask for an offer. Time-limited test accounts for up to 2 weeks are also available on request. === Additional information for institutes === Some institutes have additional setup and configuration, if you are a member of such an institute, make sure to read the information linked below after reading this article: * '''CVL''' uses it's own Slurm cluster, please read it's [[Services/SLURM-Biwi|documentation]] for access and specific additional information to this article. * '''TIK''' owns nodes in the Slurm cluster, please read the [[Services/SLURM-tik|additional information]] about those nodes and access. * '''PBL''' student supervisors can apply for access for their students. === Additional information for courses === D-ITET runs an additional GPU cluster reserved for official courses. Information specific to this cluster are available in this article: * '''[[Services/SLURM-Snowflake|Snowflake Cluster]]''' == Slurm == Slurm ('''S'''imple '''L'''inux '''U'''tility for '''R'''esource '''M'''anagement) is a free and open-source job scheduler for Linux and Unix-like kernels, used by many of the world's supercomputers and compute clusters. Slurm's design is very modular with about 100 optional plugins. In 2010, the developers of Slurm founded [[https://www.schedmd.com|SchedMD]], which maintains the canonical source, provides development, level 3 commercial support and training services and also provide a very good online documentation to [[https://slurm.schedmd.com|Slurm]]. == Slurm Cluster == === Hardware === At the moment the computing power of the Slurm cluster is based on the following 11 cpu compute nodes and 1 gpu compute node: ||'''Server'''||'''CPU'''||'''Frequency'''||'''Cores'''||'''Memory'''||'''/scratch SSD'''||'''/scratch Size'''||'''GPUs'''||'''GPU Memory'''||'''Operating System'''|| ||arton[01-03]||Dual Octa-Core Intel Xeon E5-2690||2.90 GHz||16||125 GB||-||895 GB||-||-||Debian 10|| ||arton[04-08]||Dual Deca-Core Intel Xeon E5-2690 v2||3.00 GHz||20||125 GB||-||895 GB||-||-||Debian 10|| ||arton[09-10]||Dual Deca-Core Intel Xeon E5-2690 v2||3.00 GHz||20||251 GB||✓||1.7 TB||-||-||Debian 10|| ||arton11||Dual Deca-Core Intel Xeon E5-2690 v2||3.00 GHz||20||535 GB||✓||1.7 TB||-||-||Debian 10|| ||artongpu01||Dual Octa-Core Intel Xeon Silver 4208||2.10 GHz||16||125 GB||✓||1.1 TB||4 RTX 2080 Ti||11 GB||Debian 11|| * '''Memory''' shows the amount available to Slurm The nodes are "weighted", which gives the scheduler an additional selection criteria between nodes which fulfill criterias to run a job, like resources and membership in certain partitions. The idea is to prefer nodes with faster CPUs and of those, prefer those with lower RAM. For details, see `/home/sladmitet/slurm/nodes.conf`. The Slurm job scheduler runs on the linux server `itetmaster01`. === Software === The nodes offer the same software environment as all D-ITET managed Linux clients, gpu nodes have a restricted software (no desktops installed, minimal dependencies needed for driver support). === Using Slurm === At a basic level, Slurm is very easy to use. The following sections will describe the commands you need to run and manage your batch jobs. The commands that will be most useful to you are as follows: * `sbatch` - submit a job to the batch scheduler * `squeue` - examine running and waiting jobs * `sinfo` - status compute nodes * `scancel` - delete a running job ==== Setting environment ==== The above commands only work if the environment variables for Slurm are set. Please issue the following command in your bash shell to start working with the cluster immediately or add them to your `~/.bashrc` to reference the Slurm cluster for new instances of bash:<
> {{{#!highlight bash numbers=disable export SLURM_CONF=/home/sladmitet/slurm/slurm.conf }}} ==== sbatch → Submitting a job ==== `sbatch` doesn't allow to submit a binary program directly, wrap the program to run into a surrounding bash script. The `sbatch` command has the following syntax:<
> {{{#!highlight console numbers=disable > sbatch [temporary_options] job_script [job_script arguments] }}} The `job_script` is a standard UNIX shell script. The fixed options for the Slurm Scheduler are placed in the `job_script` in lines starting with '''#SBATCH'''. The UNIX shell interprets these lines as comments and ignores them. * Put options into the `job_script` for easier reference. Place only temporary options outside the `job_script` as options to the `sbatch` command. * Make sure to create the directories you intend to store logfiles in before submitting the `job_script` * Use absolute paths in your scripts to ensure your log files and commands are found * Make sure the paths you use in your scripts [[Services/StorageOverview|are available]] on cluster nodes To test your `job-script` simply run it interactively on your host.<
><
> Assume there is a c program [[attachment:primes.c]] which is compiled to an executable binary with "gcc -o primes primes.c" and stored as `/absolute/path/to/primes`. The program runs 5 seconds and calculates prime numbers. The found prime numbers and a final summary are sent to standard output. A sample `job_script` placed in the same location `/absolute/path/to/primes.sh` to perform a batch run of the binary primes on the Arton cluster looks like this: {{{#!highlight bash numbers=disable #!/bin/bash #SBATCH --mail-type=ALL # mail configuration: NONE, BEGIN, END, FAIL, REQUEUE, ALL #SBATCH --output=/absolute/path/to/log/%j.out # where to store the output (%j is the JOBID), subdirectory "log" must exist #SBATCH --error=/absolute/path/to/log/log/%j.err # where to store error messages # Exit on errors set -o errexit # Set a directory for temporary files unique to the job with automatic removal at job termination TMPDIR=$(mktemp -d) if [[ ! -d ${TMPDIR} ]]; then echo 'Failed to create temp directory' >&2 exit 1 fi trap "exit 1" HUP INT TERM trap 'rm -rf "${TMPDIR}"' EXIT export TMPDIR # Change the current directory to the location where you want to store temporary files, exit if changing didn't succeed. # Adapt this to your personal preference cd "${TMPDIR}" || exit 1 # Send some noteworthy information to the output log echo "Running on node: $(hostname)" echo "In directory: $(pwd)" echo "Starting on: $(date)" echo "SLURM_JOB_ID: ${SLURM_JOB_ID}" # Binary or script to execute /absolute/path/to/primes # Send more noteworthy information to the output log echo "Finished at: $(date)" # End the script with exit code 0 exit 0 }}} You can test the script by running it interactively in a terminal: {{{#!highlight console numbers=disable $ /absolute/path/to/primes.sh }}} If the script runs successfully you can now submit it as a batch job to the Slurm arton cluster: {{{#!highlight console numbers=disable $ sbatch /absolute/path/to/primes.sh sbatch: Start executing function slurm_job_submit...... sbatch: Job partition set to : cpu.normal Submitted batch job 931 }}} After the job has finished, you will find the output file of the job in the file `/absolute/path/to/log/.out`. If there were errors they are stored in the file `/absolute/path/to/log/.err`.<
> ⚠ Remember: The directory for the job output has to exist before submitting the job, it is not created automatically! <
><
> You can only submit jobs to Slurm if your account is configured in the Slurm user database. If it isn't, you'll receive this [[#Batch_job_submission_failed:_Invalid_account|error message]] ==== sbatch → Submitting an array job ==== It is also possible to start an array job. The above job would run 10 times if you added the option '''#SBATCH --array=0-9''' to the job-script. A repeated execution only makes sense if the executed program adapts its behaviour according to the changing array task count number. The array count number can be referenced through the variable '''$SLURM_ARRAY_TASK_ID'''. You can pass the value of `$SLURM_ARRAY_TASK_ID` or some derived parameters to the executable.<
> Here is a simple example of passing an input filename parameter changing with `$SLURM_ARRAY_TASK_ID` to the executable: {{{#!highlight bash numbers=disable . #SBATCH --array=0-9 # # binary to execute data$SLURM_ARRAY_TASK_ID.dat }}} Every run of the program in the array job with a different task-id will produce a separate output file.<
> The option expects a '''range of task-ids''' expressed in the form `--array=n[,k[,...]][-m[:s]]%l`<
> where `n`, `k`, `m` are discreet task-ids, `s` is a step applied to a range `n`-`m` and `l` applies a limit to the number of simultaneously running tasks. See `man sbatch` for examples.<
> Specifying '''one''' task-id instead of a range as in `--array=10` results in an array job with a single task with task-id 10.<
> The following variables will be available in the job context and reflect the option arguments given: '''$SLURM_ARRAY_TASK_MAX''', '''$SLURM_ARRAY_TASK_MIN''', '''$SLURM_ARRAY_TASK_STEP'''. ==== sbatch → Common options ==== The following table shows the most common options available for '''sbatch''' to be used in the `job_script` in lines starting with `#SBATCH`<
> ||'''option'''||'''description'''|| ||--mail-type=...||Possible Values: NONE, BEGIN, END, FAIL, REQUEUE, ALL|| ||--mem=G||the job needs a maximum of GByte ( if omitted the default of 6G is used )|| ||--cpus-per-task=||number of cores to be used for the job|| ||--gres=gpu:1||number of GPUs needed for the job|| ||--nodes=||number of compute nodes to be used for the job|| ||--hint=||Bind tasks to CPU cores according to application hints (See `man --pager='less +/--hint' srun` and [[https://slurm.schedmd.com/mc_support.html#srun_hints|multi-core support]]|| ||--constraint=||Request one or more [[#sinfo_.2BIZI_Show_available_features|features]], optionally combined by operators (See `man --pager='less +/--constraint' sbatch`)|| * ⚠ The `--nodes` option should only be used for MPI jobs ! * The operators to combine `--constraint` lists are: . '''AND (&)''': `#SBATCH --constraint='geforce_rtx_2080_ti&titan_rtx` . '''OR (|)''': `#SBATCH --constraint='titan_rtx|titan_xp'` ==== squeue → Show running/waiting jobs ==== The squeue command shows the actual list of running and pending jobs in the system. As you can see in the following sample output the default format is quite minimalistic: {{{#!highlight console numbers=disable $ squeue JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) 951 cpu.norma primes.s gfreudig R 0:11 1 arton02 950 cpu.norma primes_4 gfreudig R 0:36 1 arton02 949 cpu.norma primes.s fgtest01 R 1:22 1 arton02 948 gpu.norma primes.s fgtest01 R 1:39 1 artongpu01 }}} More detailed information can be obtained by issuing the following command: {{{#!highlight console numbers=disable $ squeue --Format=jobarrayid:10,state:10,partition:16,reasonlist:18,username:10,tres-alloc:45,timeused:8,command:50 JOBID STATE PARTITION NODELIST(REASON) USER TRES_ALLOC TIME COMMAND 951 RUNNING cpu.normal arton02 gfreudig cpu=1,mem=32G,node=1,billing=1 1:20 /home/gfreudig/BTCH/Slurm/jobs/single/primes.sh 600 950 RUNNING cpu.normal arton02 gfreudig cpu=4,mem=8G,node=1,billing=4 1:45 /home/gfreudig/BTCH/Slurm/jobs/multi/primes_4.sh 600 949 RUNNING cpu.normal arton02 fgtest01 cpu=1,mem=8G,node=1,billing=1 2:31 /home/fgtest01/BTCH/Slurm/jobs/single/primes.sh 600 948 RUNNING gpu.normal artongpu01 fgtest01 cpu=1,mem=8G,node=1,billing=1,gres/gpu=1 2:48 /home/fgtest01/BTCH/Slurm/jobs/single/primes.sh 600 }}} * `STATE` is explained in the squeue man page in section `JOB STATE CODES`, see `man --pager='less +/^JOB\ STATE\ CODES' squeue` for details * `REASON` is explained there as well in section `JOB REASON CODE`, see `man --pager='less +/^JOB\ REASON\ CODES' squeue` Defining an alias in your `.bashrc` with {{{#!highlight bash numbers=disable alias sq1='squeue --Format=jobarrayid:10,state:10,partition:16,reasonlist:18,username:10,tres-alloc:45,timeused:8,command:50' }}} puts the command `sq1` at your fingertips. ⚠ '''Never call squeue from any kind of loop''', i.e. never do `watch squeue`. See `man --pager='less +/^PERFORMANCE' squeue` for an explanation.<
> To monitor your jobs, set the `sbatch` option `--mail-type` to send you notifications. If you absolutely have to see a live display of your jobs, use the `--iterate` option with a value of several seconds: {{{#!highlight bash numbers=disable squeue --user=$USER --iterate=30 }}} ==== squeue → Show job steps ==== Individual [[#srun_.2BIZI_Launch_a_command_as_a_job_step|job steps]] are listed with a specific option: {{{#!highlight bash numbers=disable squeue -s }}} ==== scancel → Deleting a job ==== With `scancel` you can remove your waiting and running jobs from the scheduler queue by their associated `JOBID`. The command `squeue` lists your jobs including their `JOBID`s. A job can then be deleted with {{{#!highlight console numbers=disable > scancel }}} To operate on an array job you can use the following commands {{{#!highlight console numbers=disable > scancel # all jobs (waiting or running) of the array job are deleted > scancel _n # the job with task-ID n is deleted > scancel _[n1-n2] # the jobs with task-ID in the range n1-n2 are deleted }}} ==== sinfo → Show partition configuration ==== The partition status can be obtained by using the `sinfo` command. An example listing is shown below. {{{ PARTITION AVAIL TIMELIMIT NODES STATE NODELIST cpu.normal* up 7-00:00:00 10 idle arton[01-11] gpu.normal up 2-00:00:00 1 idle artongpu01 tikgpu.all up 2-00:00:00 7 idle tikgpu[01-07] tikgpu.medium up 2-00:00:00 3 idle tikgpu[01-03] }}} The partition is chosen by the scheduler according to your resource request and memberships in Slurm accounts. The logic can be seen in `/home/sladmitet/slurm/jobsumit.lua`. ==== sinfo → Show resources and utilization ==== Adding selected format parameters to the `sinfo` command shows the resources available on every node and their utilization: {{{#!highlight bash numbers=disable sinfo -Node --Format nodelist:12,statecompact:7,memory:7,allocmem:10,freemem:10,cpusstate:15,cpusload:10,gresused:100 |(sed -u 1q; sort -u) }}} Restricting the command to a selected partition allows to show only GPU nodes: {{{#!highlight bash numbers=disable sinfo -Node --partition=gpu.normal --Format nodelist:12,statecompact:7,memory:7,allocmem:10,freemem:10,cpusstate:15,cpusload:10,gresused:100 }}} ==== sinfo → Show available features ==== So-called ''features'' are used to constrain jobs to nodes with different hardware capabilities, typically GPU types. To show currently active features issue the following command sequence: {{{#!highlight bash numbers=disable sinfo --Format nodehost:20,features_act:80 |grep -v '(null)' |awk 'NR == 1; NR > 1 {print $0 | "sort -n"}' }}} An example of feature use can be seen in section [[#Specifying_GPUs_based_on_compute_capability|Specifying GPUs based on compute capability]]. ==== srun → Start an interactive shell ==== An interactive session on a compute node is possible for short tests, checking your environment or transferring data to the local scratch of a node available under `/scratch_net/arton[0-11]`. Such sessions are limited to a maximum run time of 720 minutes (12 hours) regardless of the partition they are sent to. An interactive session lasting for 10 minutes on a GPU node can be started with: {{{#!highlight bash numbers=disable srun --time 10 --gres=gpu:1 --pty bash -i }}} The ouptut will look similar to the following: {{{ srun: Start executing function slurm_job_submit...... srun: Your job is a gpu job. srun: Setting partition to gpu.normal srun: job 11526 queued and waiting for resources }}} Omitting the parameter `--gres=gpu:1` opens an interactive session on a CPU-only node.<
> Do not use an interactive login to run compute jobs, use this only briefly as outlined above. Restrict job time to the necessary minimum with the `--time` option as shown above. For details see the related section in the `srun` man page by issuing the command `man --pager='less +/--time' srun` in your shell. ==== srun → Attaching an interactive shell to a running job ==== An interactive shell can be opened inside a running job by specifying its job id: {{{#!highlight bash numbers=disable srun --time 10 --jobid=123456 --overlap --pty bash -i }}} A typical use case of the above is interactive live-monitoring of a running job. ==== srun → Launch a command as a job step ==== When `srun` is used inside a `sbatch` script it spawns the given command inside a job step. This allows resource monitoring with the `sstat` command (see `man sstat`. Spawning several single-threaded commands and putting them in the background allows to schedule these commands inside the job allocation.<
> Here's an example how to run overall GPU logging and per-process logging in job steps before starting the actual computing commands. {{{#!highlight bash numbers=disable ... set -o errexit srun --ntasks=1 --cpus-per-task=1 nvidia-smi dmon -i ${CUDA_VISIBLE_DEVICES} -d 5 -s ucm -o DT > "${SLURM_JOB_ID}.gpulog" & srun --ntasks=1 --cpus-per-task=1 nvidia-smi pmon -i ${CUDA_VISIBLE_DEVICES} -d 5 -s um -o DT > "${SLURM_JOB_ID}.processlog" & ... echo finished at: `date` exit 0; }}} ==== sstat → Display status information of a running job ==== The status information shows your job's resource usage while it is running: {{{#!highlight bash numbers=disable sstat --jobs= --allsteps --format=JobID,AveVMSize%15,MaxRSS%15,AveCPU%15 }}} * `AveVMSize`: Average virtual memory of all tasks in the job * `MaxRSS`: Peak memory usage of all tasks in the job * `AveCPU`: Average CPU time of all tasks in the job All your currently running job's resource usages can be shown with: {{{#!highlight bash numbers=disable sstat --jobs=$(squeue --noheader --me --format=%A |paste -s -d ',') --allsteps --format=JobID,AveVMSize%15,MaxRSS%15,AveCPU%15 }}} ==== sacct → Display accounting information of past jobs ==== Accounting information for past jobs can be displayed with various details (see man page).<
> The following example lists all jobs of the logged in user since the beginning of the year 2020: {{{#!highlight bash numbers=disable sacct --user ${USER} --starttime=2020-01-01 --format=JobID,Start%20,Partition%20,ReqTRES%50,AveVMSize%15,MaxRSS%15,AveCPU%15,Elapsed%15,State%20 }}} ==== sprio → Show priorities of pending jobs ==== Pending jobs are prioritized by the scheduler by accounting for fair sharing of resources and age of a pending job. Priorities of pending jobs and the factors comprising them can be shown with {{{#!highlight bash numbers=disable sprio --long }}} On job submission, a nice value can be added to influence priorities of your own jobs: {{{#!highlight bash numbers=disable sbatch --nice=10 job_script.sh }}} The nice value of an already pending job can be incremented with positive values: {{{#!highlight bash numbers=disable scontrol update job nice=5 }}} Only incrementation is possible. The value can be reset to `nice=0`. The offical Slurm manual contains a detailled explanation of [[https://slurm.schedmd.com/priority_multifactor.htm|job priorisation]]. ==== smon → GPU / CPU availability ==== Information about the GPU nodes and current availability of the installed GPUs as well as CPU availability of CPU-only nodes is updated every 5 minutes to the file `/home/sladmitet/smon.txt`. Here are some convenient aliases to display the file with highlighting of either free GPUs or those running the current user's jobs: {{{#!highlight bash numbers=disable alias smon_free="grep --color=always --extended-regexp 'free|$' /home/sladmitet/smon.txt" alias smon_mine="grep --color=always --extended-regexp '${USER}|$' /home/sladmitet/smon.txt" }}} For monitoring its content the following aliases can be used: {{{#!highlight bash numbers=disable alias watch_smon_free="watch --interval 300 --no-title --differences --color \"grep --color=always --extended-regexp 'free|$' /home/sladmitet/smon.txt\"" alias watch_smon_mine="watch --interval 300 --no-title --differences --color \"grep --color=always --extended-regexp '${USER}|$' /home/sladmitet/smon.txt\"" }}} ⚠ Never use `watch` directly on `smon`, as this places considerable load on the Slurm controller! === GPU jobs === ==== Selecting the allocated GPUs ==== To select the GPU allocated by the scheduler, Slurm sets the environment variable `CUDA_VISIBLE_DEVICES` in the context of a job to the GPUs allocated to the job. The numbering always starts at 0 and is consecutively numbered up to the requested amount of GPUs - 1.<
> It is '''imperative''' to work with this variable exactly as it is set by Slurm, anything else leads to unexpected errors.<
> For details see the section [[https://slurm.schedmd.com/gres.html#GPU_Management|GPU Management]] in the official Slurm documentation. ==== Specifying a GPU type ==== It's possible to specify a GPU type by inserting the type description in the gres allocation: {{{#!highlight bash numbers=disable --gres=gpu:titan_rtx:1 }}} Available GPU type descriptions can be filtered from an appropriate `sinfo` command: {{{#!highlight bash numbers=disable sinfo --noheader --Format gres:200 |tr ':' '\n' |sort -u |grep -vE '^(gpu|[0-9,\(]+)' }}} Multiple GPU types can be requested by using the [[#sbatch_.2BIZI_Common_options|--constraint]] option. ==== Specifying GPUs based on compute capability ==== CUDA code compiled with `nvcc` can be optimized for ranges of so-called [[https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#compute-capability|Compute Capabilities]] defining `generation.version` of a NVIDIA GPU.<
> More information about compute capabilites can be read here: * [[https://docs.nvidia.com/cuda/cuda-compiler-driver-nvcc/index.html#gpu-feature-list||GPU feature list]] * [[https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#compute-capabilities|Detailled features contained in each capability]] The following table shows an abbreviated list of the compute capabilities of available GPU types selectable by [[#sinfo_.2BIZI_Show_available_features|features]]: ||'''Compute capability'''||'''Features'''|| ||3.5||tesla_k40c|| ||5.2||geforce_gtx_titan_x|| ||6.1||geforce_gtx_1080_ti, titan_x, titan_xp|| ||7.0||tesla_v100|| ||7.5||geforce_rtx_2080_ti, titan_rtx|| ||8.0||a100|| ||8.6||geforce_rtx_3090, a6000|| For more information see the [[https://developer.nvidia.com/cuda-gpus|full list of NVIDIA CUDA GPUs]].<
> As GPU nodes may house different generations of GPUs, compiled CUDA code might not run on all of them and errors similar to the following can appear: {{{ RuntimeError: CUDA error: no kernel image is available for execution on the device }}} If you see a similar error: * Note the type of GPU on which your job failed * In the table above, note the features with a higher compute capability than the GPU type you noted before * Check the list of [[https://computing.ee.ethz.ch/Services/SLURM#sinfo_.2BIZI_Show_available_features|available features]] to exclude non-existing GPU types * Build a [[#sbatch_.2BIZI_Common_options|constraint]] to include only GPUs with supported compute capabilites Example: Add the constraint `--constraint='tesla_v100|geforce_rtx_2080_ti|titan_rtx|geforce_rtx_3090'` to your job submission to run a job only on nodes with GPUs of compute capability `7.0` or higher. === Multicore jobs/ job to core binding === A modern linux kernel is able to bind a process and all its children to a fixed number of cores. By default a job submitted to the Slurm arton cluster is bound to to the numbers of requested cores/cpus. The default number of requested cpus is 1, if you have an application which is able to run multithreaded on several cores you must use the '''--cpus-per-task''' option in the sbatch command to get a binding to more than one core. To check for processes with core bindings, use the command `hwloc-ps -c`: {{{ $ ssh arton02 hwloc-ps -c 43369 0x00010001 slurmstepd: [984.batch] 43374 0x00010001 /bin/sh 43385 0x00010001 codebin/primes }}} === Job input/output data storage === Temporary data storage of a job used only while the job is running, should be placed in the `/scratch` directory of the compute nodes. Set the environment variables of the tools you use accordingly. The Matlab `MCR_ROOT_CACHE` variable is set automatically by the Slurm scheduler.<
> The file system protection of the `/scratch` directory allows everybody to create files and directories in it. A cron job runs periodically on the execution hosts to prevent the `/scratch` directory from filling up and cleans it governed by pre-set policies. Therefore data you place in the `/scratch` directory of a compute node cannot be assumed to stay there forever. * Small sized input and output data for the jobs is best placed in your home directory. It is available on every compute node through the `/home` automounter. * Larger amounts of data should be placed in your personal [[Services/NetScratch|netscratch]] folder and can be accessed on all compute nodes. * If you have problems with the quota limit in your home directory you could transfer data from your home or the `/scratch` directory of your submit host to the `/scratch` directories of the arton compute nodes and vice versa. All `/scratch` directories of the compute nodes are available through the `/scratch_net` automount system. You can access the `/scratch` directory of `arton` under `/scratch_net/arton`. This allows you to transfer data between the `/scratch_net` directories and your home with normal linux file copy and to the `/scratch` of your submission host with scp, for example from an [[#srun_.2BIZI_Start_an_interactive_shell|interactive session]] on any node. Other data storage concepts for the arton cluster are possible and will be investigated, if the above solution proves not to be sufficient. === Matlab Distributed Computing Environment (MDCE) === The Matlab '''P'''arallel '''C'''omputing '''T'''oolbox (PCT) can be configured with an interface to the Slurm cluster. To work with MDCE please import [[attachment:Slurm.mlsettings]] in Matlab GUI (Parallel → Create and manage Clusters → Import ). Adjust the setting ''!JobStorageLocation'' to your requirements. The cluster profile Slurm will now appear besides the standard local(default) profile in the profile list. With the local profile, you can use as many workers on one computer as there are physical cores while the Slurm profile allows to initiate up to 32 worker processes distributed over all Slurm compute nodes.<
> ⚠ Please temporay reduce the number of workers to 4 in the Slurm profile when performing the profile "Validation" function in the Matlab Cluster Manager.<
> ⚠ Don't forget to set the Slurm environment variables before starting Matlab!<
> The Slurm cluster profile can be used with Matlab programs running as Slurm batch jobs but it's also possible to use the profile in an interactive Matlab session on your client. When you open a Slurm parpool, the workers are started automatically as jobs in the cluster.<
> ⚠ In interactive mode please always close your parpool if you aren't performing any calculations on the workers.<
> Sample code for the 3 Matlab PCT methods parfor, spmd, tasks using the local or Slurm cluster profile is provided in [[attachment:PCTRefJobs.tar.gz]]. === Reservations === Nodes may be reserved at certain times for courses or maintenance. If your job is pending with the reason `ReqNodeNotAvail,_May_be_reserved_for_other_job`, check reservations and adjust the `--time` parameter of your job accordingly. ==== Showing current reservations ==== Current reservations can be shown by issuing {{{#!highlight bash numbers=disable scontrol show reservation }}} ==== Using a reservation ==== If you are entitled to use a reservation, specify the reservation in your job submission by appending the parameter `--reservation=`. ==== Requesting a reservation ==== Reservations are managed by Slurm administrators. Please contact [[mailto:support@ee.ethz.ch|ISG D-ITET support]] if you're in need of a reservation. === Frequently Asked Questions === If your question isn't listed below, an answer might be listed in the [[https://slurm.schedmd.com/faq.htm|official Slurm FAQ]]. ==== Batch job submission failed: Invalid account ==== If you receive one of the following error messages after submitting a job with `sbatch` or using `srun` {{{ sbatch: error: Batch job submission failed: Invalid account or account/partition combination specified }}} {{{ srun: error: Unable to allocate resources: Invalid account or account/partition combination specified }}} your account hasn't been registered with Slurm yet. Please contact [[mailto:support@ee.ethz.ch|support]] and ask to be registered. ==== Code runs with srun but fails with sbatch ==== This points to a difference in the environment of the interactive job and the batch job. By default both `srun` and `sbatch` forward the complete current submission environment to a job (See `man --pager='less +/--export=' srun`, `man --pager='less +/--export=' sbatch`) and also change path to the current path of the submission environment (See `man --pager='less +/--chdir=' srun`, `man --pager='less +/--chdir=' sbatch`) Compare the output of `printenv` from jobs started with `srun` and `sbatch` to figure out the differences: * Environment variable starting with `SLURM_` are set by Slurm * `HOSTNAME` reflects the node a job runs on * `ENVIRONMENT=BATCH` is set by `sbatch`. Differences have to come from: * Anything done interactively in the submission session after submitting the job with `srun` and before submitting it with `sbatch` * Or the job submit script used with `sbatch`. ==== Invalid user for SlurmUser slurm ==== After executing one of the Slurm executables like `sbatch` or `sinfo` the following error appears: {{{ error: Invalid user for SlurmUser slurm, ignored }}} The user `slurm` doesn't exist on the host you're running your Slurm executable. If this happens on a host managed by ISG D-ITET, please contact [[mailto:support@ee.ethz.ch|support]], tell us the name of your host and ask us to configure it as a Slurm submission host. ==== Node(s) in drain state ==== If `sinfo` shows one or more nodes in ''drain'' state, the reason can be shown with {{{ sinfo -R }}} or in case the reason is cut off with {{{ sinfo -o '%60E %9u %19H %N' }}} Nodes are set to drain by ISG D-ITET to empty them of jobs in time for scheduled maintenance or by the scheduler itself in case a problem is detected on a node. ==== : fatal: Could not establish a configuration source ==== If you receive the following error messages after using a Slurm command like `srun`, `sbatch`, `squeue` or `sinfo` (replace `squeue` with the name of the command you used to end up with the error message): {{{ squeue: fatal: Could not establish a configuration source }}} Make sure you [[#Setting_environment|set your environment]]. ==== My job was terminated by the OOM killer ==== If your job got terminated and you see a line similar to the following in your job log: {{{ slurmstepd: error: Detected 1 oom-kill event(s) in StepId=.batch cgroup. ... }}} this means a process in your job attempted to use more memory than you requested for the job, so it was killed by the OOM (__O__ut __O__f __M__emory) killer. This in turn resulted in termination of your job by the Slurm scheduler.<
> Check the value of `MaxRSS` in the output of [[#sacct_.2BIZI_Display_accounting_information_of_past_jobs|sacct]] for your job to verify the maximum memory usage of your job. Run tests by adjusting your memory allocation with [[#sbatch_.2BIZI_Common_options|--mem]] until you figure out how much memory your job needs.<
> Slurm's accounting samples jobs every 30 seconds, so there is no useful data if a job was killed within the first 30 seconds after it started. Also sudden spikes in memory consumption may not be recorded, but can still trigger the OOM killer.<
> ⚠ This is error pertains to the onboard memory allocated to a job, a GPU allocation always contains its full GPU memory.