26034
Comment:
|
26955
|
Deletions are marked like this. | Additions are marked like this. |
Line 9: | Line 9: |
Access to the Slurm cluster is reserved for staff of the contributing institutes '''APS, IBT, IFA, MINS, NARI, TIK'''. Access is granted on request, please contact [[mailto:support@ee.ethz.ch|ISG.EE support]].<<BR>> | Access to the Slurm cluster is reserved for staff of the contributing institutes '''APS, IBT, IFA, MINS, NARI, TIK''' and '''PBL'''. Access is granted on request, please contact [[mailto:support@ee.ethz.ch|ISG.EE support]].<<BR>> |
Line 13: | Line 13: |
* '''PBL''' student supervisors can apply for access for their students. | |
Line 35: | Line 36: |
At a basic level, Slurm is very easy to use. The following sections will describe the commands you need to run and manage your batch jobs. The commands that will be most useful to you are as follows<<BR>> | At a basic level, Slurm is very easy to use. The following sections will describe the commands you need to run and manage your batch jobs. The commands that will be most useful to you are as follows: |
Line 50: | Line 51: |
> sbatch [options] job_script [job_script arguments] }}} The `job_script` is a standard UNIX shell script. The fixed options for the Slurm Scheduler are placed in the `job_script` in lines starting with '''#SBATCH'''. The UNIX shell interprets these lines as comments and ignores them. Only temporary options should be placed outside the `job_script`. To test your `job-script` you can simply run it interactively.<<BR>><<BR>> Assume there is a c program [[attachment:primes.c]] which is compiled to an executable binary named `primes` with "gcc -o primes primes.c". The program runs 5 seconds and calculates prime numbers. The found prime numbers and a final summary are sent to standard output. A sample `job_script` `primes.sh` to perform a batch run of the binary primes on the Arton cluster looks like this: |
> sbatch [temporary_options] job_script [job_script arguments] }}} The `job_script` is a standard UNIX shell script. The fixed options for the Slurm Scheduler are placed in the `job_script` in lines starting with '''#SBATCH'''. The UNIX shell interprets these lines as comments and ignores them. * Put options into the `job_script` for easier reference. Place only temporary options outside the `job_script` as options to the `sbatch` command. * Make sure to create the directories you intend to store logfiles in before submitting the `job_script` * Use absolute paths in your scripts to simplify debugging To test your `job-script` simply run it interactively on your host.<<BR>><<BR>> Assume there is a c program [[attachment:primes.c]] which is compiled to an executable binary with "gcc -o primes primes.c" and stored as `/absolute/path/to/primes`. The program runs 5 seconds and calculates prime numbers. The found prime numbers and a final summary are sent to standard output. A sample `job_script` placed in the same location `/absolute/path/to/primes.sh` to perform a batch run of the binary primes on the Arton cluster looks like this: |
Line 57: | Line 62: |
#SBATCH --mail-type=ALL # mail configuration: NONE, BEGIN, END, FAIL, REQUEUE, ALL #SBATCH --output=log/%j.out # where to store the output (%j is the JOBID), subdirectory must exist #SBATCH --error=log/%j.err # where to store error messages echo "Running on host: $(hostname)" echo "In directory: $(pwd)" echo "Starting on: $(date)" echo "SLURM_JOB_ID: ${SLURM_JOB_ID}" |
#SBATCH --mail-type=ALL # mail configuration: NONE, BEGIN, END, FAIL, REQUEUE, ALL #SBATCH --output=/absolute/path/to/log/%j.out # where to store the output (%j is the JOBID), subdirectory "log" must exist #SBATCH --error=/absolute/path/to/log/log/%j.err # where to store error messages |
Line 79: | Line 79: |
# Change the current directory to the location where you want to store intermediate files, exit if changing didn't succeed. # Adapt this to your personal preference cd "${TMPDIR}" || exit 1 # Send some noteworthy information to the output log echo "Running on node: $(hostname)" echo "In directory: $(pwd)" echo "Starting on: $(date)" echo "SLURM_JOB_ID: ${SLURM_JOB_ID}" |
|
Line 80: | Line 90: |
./primes | /absolute/path/to/primes # Send more noteworthy information to the output log |
Line 86: | Line 98: |
$ ./primes.sh | $ /absolute/path/to/primes.sh |
Line 90: | Line 102: |
$ sbatch primes.sh | $ sbatch /absolute/path/to/primes.sh |
Line 95: | Line 107: |
After the job has finished, you will find the output file of the job in the log subdirectory with a name of `<JOBID>.out`.<<BR>> /!\ The directory for the job output must exist, it is not created automatically! |
After the job has finished, you will find the output file of the job in the file `/absolute/path/to/log/<JOBID>.out`. If there were errors they are stored in the file `/absolute/path/to/log/<JOBID>.err`.<<BR>> /!\ Remember: The directory for the job output has to exist before submitting the job, it is not created automatically! |
Contents
- Introduction
- Slurm
-
Slurm Cluster
- Hardware
- Software
-
Using Slurm
- Setting environment
- sbatch -> Submitting a job
- sbatch -> Submitting an array job
- sbatch -> Common options
- squeue -> Show running/waiting jobs
- squeue -> Show job steps
- scancel -> Deleting a job
- sinfo -> Show partition configuration
- sinfo -> Show resources and utilization
- sinfo -> Show available features
- srun -> Start an interactive shell
- srun -> Attaching an interactive shell to a running job
- srun -> Launch a command as a job step
- sstat -> Display status information of a running job
- sacct -> Display accounting information of past jobs
- GPU jobs
- Multicore jobs/ job to core binding
- Job input/output data storage
- Matlab Distributed Computing Environment (MDCE)
- Reservations
- Frequently Asked Questions
Introduction
At ITET the Condor Batch Queueing System has been used for a long time and is still used for running compute-intensive jobs. It uses the free resources on the tardis-PCs of the student rooms and on numerous PCs and compute servers at ITET institutes. Interactive work is privileged over batch computing, so running jobs could be killed by new interactive load or by shutdown/restart of a PC.
The Slurm system installed on the powerful ITET arton compute servers is an alternative to the Condor batch computing system. It consists of a master host, where the scheduler resides and the compute nodes, where batch jobs are executed. The compute nodes are powerful servers located in server rooms, they are exclusively reserved for batch processing. Interactive logins are disabled.
Access
Access to the Slurm cluster is reserved for staff of the contributing institutes APS, IBT, IFA, MINS, NARI, TIK and PBL. Access is granted on request, please contact ISG.EE support.
If your circumstances differ and you'd still like to use the cluster, please contact ISG.EE support as well and ask for an offer. Time-limited test accounts for up to 2 weeks are also available on request.
CVL uses it's own Slurm cluster, please read it's documentation for access and specific additional information to this article.
TIK owns nodes in the Slurm cluster, please read the additional information about those nodes and access.
PBL student supervisors can apply for access for their students.
Slurm
Slurm (Simple Linux Utility for Resource Management) is a free and open-source job scheduler for Linux and Unix-like kernels, used by many of the world's supercomputers and compute clusters. Slurm's design is very modular with about 100 optional plugins. In 2010, the developers of Slurm founded SchedMD, which maintains the canonical source, provides development, level 3 commercial support and training services and also provide a very good online documentation to Slurm.
Slurm Cluster
Hardware
At the moment the computing power of the Slurm cluster is based on the following 11 cpu compute nodes and 1 gpu compute node:
Server |
CPU |
Frequency |
Cores |
Memory |
/scratch SSD |
/scratch Size |
GPUs |
GPU Memory |
Operating System |
arton[01-03] |
Dual Octa-Core Intel Xeon E5-2690 |
2.90 GHz |
16 |
125 GB |
- |
895 GB |
- |
- |
Debian 10 |
arton[04-08] |
Dual Deca-Core Intel Xeon E5-2690 v2 |
3.00 GHz |
20 |
125 GB |
- |
895 GB |
- |
- |
Debian 10 |
arton[09-10] |
Dual Deca-Core Intel Xeon E5-2690 v2 |
3.00 GHz |
20 |
251 GB |
✓ |
1.7 TB |
- |
- |
Debian 10 |
arton11 |
Dual Deca-Core Intel Xeon E5-2690 v2 |
3.00 GHz |
20 |
724 GB |
✓ |
1.7 TB |
- |
- |
Debian 10 |
artongpu01 |
Dual Octa-Core Intel Xeon Silver 4208 |
2.10 GHz |
16 |
125 GB |
✓ |
1.1 TB |
4 RTX 2080 Ti |
11 GB |
Debian 10 |
Memory shows the amount available to Slurm
The Slurm job scheduler runs on the linux server itetmaster01.
Software
The nodes offer the same software environment as all D-ITET managed Linux clients, gpu nodes have a restricted software (no desktops installed, minimal dependencies needed for driver support).
Using Slurm
At a basic level, Slurm is very easy to use. The following sections will describe the commands you need to run and manage your batch jobs. The commands that will be most useful to you are as follows:
sbatch - submit a job to the batch scheduler
squeue - examine running and waiting jobs
sinfo - status compute nodes
scancel - delete a running job
Setting environment
The above commands only work if the environment variables for Slurm are set. Please issue the following command in your bash shell to start working with the cluster immediately or add them to your ~/.bashrc to reference the Slurm cluster for new instances of bash:
export SLURM_CONF=/home/sladmitet/slurm/slurm.conf
sbatch -> Submitting a job
sbatch doesn't allow to submit a binary program directly, please wrap the program to run into a surrounding bash script. The sbatch command has the following syntax:
> sbatch [temporary_options] job_script [job_script arguments]
The job_script is a standard UNIX shell script. The fixed options for the Slurm Scheduler are placed in the job_script in lines starting with #SBATCH. The UNIX shell interprets these lines as comments and ignores them.
Put options into the job_script for easier reference. Place only temporary options outside the job_script as options to the sbatch command.
Make sure to create the directories you intend to store logfiles in before submitting the job_script
- Use absolute paths in your scripts to simplify debugging
To test your job-script simply run it interactively on your host.
Assume there is a c program primes.c which is compiled to an executable binary with "gcc -o primes primes.c" and stored as /absolute/path/to/primes. The program runs 5 seconds and calculates prime numbers. The found prime numbers and a final summary are sent to standard output. A sample job_script placed in the same location /absolute/path/to/primes.sh to perform a batch run of the binary primes on the Arton cluster looks like this:
#!/bin/bash
#SBATCH --mail-type=ALL # mail configuration: NONE, BEGIN, END, FAIL, REQUEUE, ALL
#SBATCH --output=/absolute/path/to/log/%j.out # where to store the output (%j is the JOBID), subdirectory "log" must exist
#SBATCH --error=/absolute/path/to/log/log/%j.err # where to store error messages
# Exit on errors
set -o errexit
# Set a directory for temporary files unique to the job with automatic removal at job termination
TMPDIR=$(mktemp -d)
if [[ ! -d ${TMPDIR} ]]; then
echo 'Failed to create temp directory' >&2
exit 1
fi
trap "exit 1" HUP INT TERM
trap 'rm -rf "${TMPDIR}"' EXIT
export TMPDIR
# Change the current directory to the location where you want to store intermediate files, exit if changing didn't succeed.
# Adapt this to your personal preference
cd "${TMPDIR}" || exit 1
# Send some noteworthy information to the output log
echo "Running on node: $(hostname)"
echo "In directory: $(pwd)"
echo "Starting on: $(date)"
echo "SLURM_JOB_ID: ${SLURM_JOB_ID}"
# Binary or script to execute
/absolute/path/to/primes
# Send more noteworthy information to the output log
echo "Finished at: $(date)"
exit 0
You can test the script by running it interactively in a terminal:
$ /absolute/path/to/primes.sh
If the script runs successfully you can now submit it as a batch job to the Slurm arton cluster:
$ sbatch /absolute/path/to/primes.sh sbatch: Start executing function slurm_job_submit...... sbatch: Job partition set to : cpu.normal.32 (normal memory) Submitted batch job 931
After the job has finished, you will find the output file of the job in the file /absolute/path/to/log/<JOBID>.out. If there were errors they are stored in the file /absolute/path/to/log/<JOBID>.err.
Remember: The directory for the job output has to exist before submitting the job, it is not created automatically!
You can only submit jobs to Slurm if your account is configured in the Slurm user database. If it isn't, you'll receive this error message
sbatch -> Submitting an array job
Similar to condor it is also possible to start an array job. The above job would run 10 times if you added the option #SBATCH --array=0-9 to the job-script. A repeated execution only makes sense if the executed program adapts its behaviour according to the changing array task count number. The array count number can be referenced through the variable $SLURM_ARRAY_TASK_ID. You can pass the value of $SLURM_ARRAY_TASK_ID or some derived parameters to the executable.
Here is a simple example of passing an input filename parameter changing with $SLURM_ARRAY_TASK_ID to the executable:
.
#SBATCH --array=0-9
#
# binary to execute
<path-to-executable> data$SLURM_ARRAY_TASK_ID.dat
Every run of the program in the array job with a different task-id will produce a separate output file.
The option expects a range of task-ids expressed in the form --array=n[,k[,...]][-m[:s]]%l
where n, k, m are discreet task-ids, s is a step applied to a range n-m and l applies a limit to the number of simultaneously running tasks. See man sbatch for examples.
Specifying one task-id instead of a range as in --array=10 results in an array job with a single task with task-id 10.
The following variables will be available in the job context and reflect the option arguments given: $SLURM_ARRAY_TASK_MAX, $SLURM_ARRAY_TASK_MIN, $SLURM_ARRAY_TASK_STEP.
sbatch -> Common options
The following table shows the most common options available for sbatch to be used in the job_script in lines starting with #SBATCH
option |
description |
--mail-type=... |
Possible Values: NONE, BEGIN, END, FAIL, REQUEUE, ALL |
--mem=<n>G |
the job needs a maximum of <n> GByte ( if omitted the default of 6G is used ) |
--cpus-per-task=<n> |
number of cores to be used for the job |
--gres=gpu:1 |
number of GPUs needed for the job |
--nodes=<n> |
number of compute nodes to be used for the job |
--hint=<type> |
Bind tasks to CPU cores according to application hints (See man --pager='less +/--hint' srun and multi-core support |
--constraint=<feature_name> |
Request one or more features, optionally combined by operators |
The --nodes option should only be used for MPI jobs !
The operators to combine --constraint lists are:
AND (&): #SBATCH --constraint='geforce_rtx_2080_ti&titan_rtx
OR (|): #SBATCH --constraint='titan_rtx|titan_xp'
squeue -> Show running/waiting jobs
The squeue command shows the actual list of running and pending jobs in the system. As you can see in the following sample output the default format is quite minimalistic:
$ squeue JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) 951 cpu.norma primes.s gfreudig R 0:11 1 arton02 950 cpu.norma primes_4 gfreudig R 0:36 1 arton02 949 cpu.norma primes.s fgtest01 R 1:22 1 arton02 948 gpu.norma primes.s fgtest01 R 1:39 1 artongpu01
More detailed information can be obtained by issuing the following command:
$ squeue --Format=jobarrayid:10,state:10,partition:16,reasonlist:18,username:10,tres-alloc:45,timeused:8,command:50 JOBID STATE PARTITION NODELIST(REASON) USER TRES_ALLOC TIME COMMAND 951 RUNNING cpu.normal.32 arton02 gfreudig cpu=1,mem=32G,node=1,billing=1 1:20 /home/gfreudig/BTCH/Slurm/jobs/single/primes.sh 600 950 RUNNING cpu.normal.32 arton02 gfreudig cpu=4,mem=8G,node=1,billing=4 1:45 /home/gfreudig/BTCH/Slurm/jobs/multi/primes_4.sh 600 949 RUNNING cpu.normal.32 arton02 fgtest01 cpu=1,mem=8G,node=1,billing=1 2:31 /home/fgtest01/BTCH/Slurm/jobs/single/primes.sh 600 948 RUNNING gpu.normal artongpu01 fgtest01 cpu=1,mem=8G,node=1,billing=1,gres/gpu=1 2:48 /home/fgtest01/BTCH/Slurm/jobs/single/primes.sh 600
STATE is explained in the squeue man page in section JOB STATE CODES, see man --pager='less +/^JOB\ STATE\ CODES' squeue for details
REASON is explained there as well in section JOB REASON CODE, see man --pager='less +/^JOB\ REASON\ CODES' squeue
Defining an alias in your .bashrc with
alias sq1='squeue --Format=jobarrayid:10,state:10,partition:16,reasonlist:18,username:10,tres-alloc:45,timeused:8,command:50'
puts the command sq1 at your fingertips.
Never call squeue from any kind of loop, i.e. never do watch squeue. See man --pager='less +/^PERFORMANCE' squeue for an explanation.
To monitor your jobs, set the sbatch option --mail-type to send you notifications. If you absolutely have to see a live display of your jobs, use the --iterate option with a value of several seconds:
squeue --user=$USER --iterate=30
squeue -> Show job steps
Individual job steps are listed with a specific option:
squeue -s
scancel -> Deleting a job
With scancel you can remove your waiting and running jobs from the scheduler queue by their associated JOBID. The command squeue lists your jobs including their JOBIDs. A job can then be deleted with
> scancel <JOBID>
To operate on an array job you can use the following commands
> scancel <JOBID> # all jobs (waiting or running) of the array job are deleted > scancel <JOBID>_n # the job with task-ID n is deleted > scancel <JOBID>_[n1-n2] # the jobs with task-ID in the range n1-n2 are deleted
sinfo -> Show partition configuration
The partition status can be obtained by using the sinfo command. An example listing is shown below.
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST cpu.normal.32* up 2-00:00:00 11 idle arton[01-11] cpu.normal.64 up 2-00:00:00 3 idle arton[09-11] cpu.normal.256 up 2-00:00:00 1 idle arton11 array.normal up 2-00:00:00 10 idle arton[01-10] gpu.normal up 2-00:00:00 1 idle artongpu01
For normal jobs (single, multicore) you can not choose the partition for the job to run in the sbatch command, the partition is selected by the scheduler according to your memory request. Array jobs are put in the array.normal partition, gpu jobs in the gpu.normal partition. The following table shows the job memory limits in different partitions:
PARTITION |
Requested Memory |
cpu.normal.32 |
< 32 GB |
cpu.normal.64 |
32 - 64 GB |
cpu.normal.256 |
> 64 GB |
array.normal |
< 32 GB |
gpu.normal |
< 64 GB |
Only a job with a --mem request of a maximum of 32 GByte can run in the cpu.normal.32 partition, which contains all 11 artons.
sinfo -> Show resources and utilization
Adding selected format parameters to the sinfo command shows the resources available on every node and their utilization:
sinfo -Node --Format nodelist:12,statecompact:7,memory:7,allocmem:10,freemem:10,cpusstate:15,cpusload:10,gresused:100 |(sed -u 1q; sort -u)
Restricting the command to a selected partition allows to show only GPU nodes:
sinfo -Node --partition=gpu.normal --Format nodelist:12,statecompact:7,memory:7,allocmem:10,freemem:10,cpusstate:15,cpusload:10,gresused:100
sinfo -> Show available features
So-called features are used to constrain jobs to nodes with different hardware capabilities, typically GPU types. To show currently active features issue the following command sequence:
sinfo --Format nodehost:20,features_act:80 |grep -v '(null)' |awk 'NR == 1; NR > 1 {print $0 | "sort -n"}'
srun -> Start an interactive shell
An interactive session on a compute node is possible for short tests, checking your environment or transferring data to the local scratch of a node available under /scratch_net/arton[0-11]. An interactive session lasting for 10 minutes on a GPU node can be started with:
srun --time 10 --gres=gpu:1 --pty bash -i
The ouptut will look similar to the following:
srun: Start executing function slurm_job_submit...... srun: Your job is a gpu job. srun: Setting partition to gpu.normal srun: job 11526 queued and waiting for resources
Omitting the parameter --gres=gpu:1 opens an interactive session on a CPU-only node.
Do not use an interactive login to run compute jobs, use this only briefly as outlined above. Restrict job time to the necessary minimum with the --time option as shown above. For details see the related section in the srun man page by issuing the command man --pager='less +/--time' srun in your shell.
srun -> Attaching an interactive shell to a running job
An interactive shell can be opened inside a running job by specifying its job id:
srun --time 10 --jobid=123456 --pty bash -i
A typical use case of the above is interactive live-monitoring of a running job. Another use case is running a graphical application to live-debug a running job. In the latter case X11 forwarding has to be enabled as follows:
srun --time 10 --jobid=123456 --x11 --pty bash -i
srun -> Launch a command as a job step
When srun is used inside a sbatch script it spawns the given command inside a job step. This allows resource monitoring with the sstat command (see man sstat. Spawning several single-threaded commands and putting them in the background allows to schedule these commands inside the job allocation.
Here's an example how to run overall GPU logging and per-process logging in job steps before starting the actual computing commands.
...
set -o errexit
srun --exclusive --ntasks=1 --cpus-per-task=1 nvidia-smi dmon -i ${CUDA_VISIBLE_DEVICES} -d 5 -s ucm -o DT > "${SLURM_JOB_ID}.gpulog" &
srun --exclusive --ntasks=1 --cpus-per-task=1 nvidia-smi pmon -i ${CUDA_VISIBLE_DEVICES} -d 5 -s um -o DT > "${SLURM_JOB_ID}.processlog" &
...
echo finished at: `date`
exit 0;
sstat -> Display status information of a running job
The status information shows your job's resource usage while it is running:
sstat --jobs=<JOBID> --format=JobID,AveVMSize%15,MaxRSS%15,AveCPU%15
AveVMSize: Average virtual memory of all tasks in the job
MaxRSS: Peak memory usage of all tasks in the job
AveCPU: Average CPU time of all tasks in the job
sacct -> Display accounting information of past jobs
Accounting information for past jobs can be displayed with various details (see man page).
The following example lists all jobs of the logged in user since the beginning of the year 2020:
sacct --user ${USER} --starttime=2020-01-01 --format=JobID,Start%20,Partition%20,ReqTRES%50,AveVMSize%15,MaxRSS%15,AveCPU%15,Elapsed%15,State%20
GPU jobs
Selecting the correct GPUs
To select the GPU allocated by the scheduler, Slurm sets the environment variable CUDA_VISIBLE_DEVICES in the context of a job to the GPUs allocated to the job. The numbering always starts at 0 and is consecutively numbered up to the requested amount of GPUs - 1.
It is imperative to work with this variable exactly as it is set by Slurm, anything else leads to unexpected errors.
For details see the section GPU Management in the official Slurm documentation.
Specifying a GPU type
It's possible to specify a GPU type by inserting the type description in the gres allocation:
--gres=gpu:titan_rtx:1
Available GPU type descriptions can be filtered from an appropriate sinfo command:
sinfo --noheader --Format gres:200 |tr ':' '\n' |sort -u |grep -vE '^(gpu|[0-9,\(]+)'
Multiple GPU types can be requested by using the --constraint option.
Multicore jobs/ job to core binding
A modern linux kernel is able to bind a process and all its children to a fixed number of cores. By default a job submitted to the Slurm arton cluster is bound to to the numbers of requested cores/cpus. The default number of requested cpus is 1, if you have an application which is able to run multithreaded on several cores you must use the --cpus-per-task option in the sbatch command to get a binding to more than one core. To check for processes with core bindings, use the command hwloc-ps -c:
$ ssh arton02 hwloc-ps -c 43369 0x00010001 slurmstepd: [984.batch] 43374 0x00010001 /bin/sh 43385 0x00010001 codebin/primes
Job input/output data storage
Temporary data storage of a job used only while the job is running, should be placed in the /scratch directory of the compute nodes. Set the environment variables of the tools you use accordingly. The Matlab MCR_ROOT_CACHE variable is set automatically by the Slurm scheduler.
The file system protection of the /scratch directory allows everybody to create files and directories in it. A cron job runs periodically on the execution hosts to prevent the /scratch directory from filling up and cleans it governed by pre-set policies. Therefore data you place in the /scratch directory of a compute node cannot be assumed to stay there forever.
Small sized input and output data for the jobs is best placed in your home directory. It is available on every compute node through the /home automounter.
Larger amounts of data should be placed in your personal netscratch folder and can be accessed on all compute nodes.
If you have problems with the quota limit in your home directory you could transfer data from your home or the /scratch directory of your submit host to the /scratch directories of the arton compute nodes and vice versa. All /scratch directories of the compute nodes are available through the /scratch_net automount system. You can access the /scratch directory of arton<nn> under /scratch_net/arton<nn>. This allows you to transfer data between the /scratch_net directories and your home with normal linux file copy and to the /scratch of your submission host with scp, for example from an interactive session on any node.
Other data storage concepts for the arton cluster are possible and will be investigated, if the above solution proves not to be sufficient.
Matlab Distributed Computing Environment (MDCE)
The Matlab Parallel Computing Toolbox (PCT) can be configured with an interface to the Slurm cluster. To work with MDCE please import Slurm.mlsettings in Matlab GUI (Parallel -> Create and manage Clusters -> Import ). Adjust the setting "JobStorageLocation" to your requirements. The cluster profile Slurm will now appear besides the standard local(default) profile in the profile list. With the local profile, you can use as many workers on one computer as there are physical cores while the Slurm profile allows to initiate up to 32 worker processes distributed over all Slurm compute nodes.
Please temporay reduce the number of workers to 4 in the Slurm profile when performing the profile "Validation" function in the Matlab Cluster Manager.
Don't forget to set the Slurm environment variables before starting Matlab!
The Slurm cluster profile can be used with Matlab programs running as Slurm batch jobs but it's also possible to use the profile in an interactive Matlab session on your client. When you open a Slurm parpool, the workers are started automatically as jobs in the cluster.
In interactive mode please always close your parpool if you aren't performing any calculations on the workers.
Sample code for the 3 Matlab PCT methods parfor, spmd, tasks using the local or Slurm cluster profile is provided in PCTRefJobs.tar.gz.
Reservations
Nodes may be reserved at certain times for courses or maintenance. If your job is pending with the reason ReqNodeNotAvail,_May_be_reserved_for_other_job, check reservations and adjust the --time parameter of your job accordingly.
Showing current reservations
Current reservations can be shown by issuing
scontrol show reservation
Using a reservation
If you are entitled to use a reservation, specify the reservation in your job submission by appending the parameter --reservation=<ReservationName>.
Requesting a reservation
Reservations are managed by Slurm administrators. Please contact ISG.EE support if you're in need of a reservation.
Frequently Asked Questions
If your question isn't listed below, an answer might be listed in the official Slurm FAQ.
Batch job submission failed: Invalid account
If you receive one of the following error messages after submitting a job with sbatch or using srun
sbatch: error: Batch job submission failed: Invalid account or account/partition combination specified
srun: error: Unable to allocate resources: Invalid account or account/partition combination specified
your account hasn't been registered with Slurm yet. Please contact support and ask to be registered.
Invalid user for SlurmUser slurm
After executing one of the Slurm executables like sbatch or sinfo the following error appears:
error: Invalid user for SlurmUser slurm, ignored
The user slurm doesn't exist on the host you're running your Slurm executable. If this happens on a host managed by ISG.EE, please contact support, tell us the name of your host and ask us to configure it as a Slurm submission host.
Node(s) in drain state
If sinfo shows one or more nodes in drain state, the reason can be shown with
sinfo -R
or in case the reason is cut off with
sinfo -o '%60E %9u %19H %N'
Nodes are set to drain by ISG.EE to empty them of jobs in time for scheduled maintenance or by the scheduler itself in case a problem is detected on a node.