Slurm heterogeneous job for multiple tests - cluster-computing

Question

I have to perform some tests on an HPC cluster and I am using Slurm as workload manager.
Since I have to perform similar tests on different allocations, I decided to exploit the heterogenous job support for Slurm.
Here is my Slurm script:
# begin of slurm_script.sh
#!/bin/bash
#SBATCH -p my_partition
#SBATCH --exclusive
#SBATCH --time 16:00:00 # format: HH:MM:SS
#SBATCH -N 1 # 1 node
#SBATCH --ntasks-per-node=32 # tasks out of 128
#SBATCH --gres=gpu:4 # gpus per node out of 4
#SBATCH --mem=246000 # memory per node out of 246000MB
#SBATCH hetjob
#SBATCH -p my_partition
#SBATCH --exclusive
#SBATCH --time 16:00:00 # format: HH:MM:SS
#SBATCH -N 2 # 2 nodes
#SBATCH --ntasks-per-node=32 # tasks out of 128
#SBATCH --gres=gpu:4 # gpus per node out of 4
#SBATCH --mem=246000 # memory per node out of 246000MB
#SBATCH hetjob
#SBATCH -p my_partition
#SBATCH --exclusive
#SBATCH --time 16:00:00 # format: HH:MM:SS
#SBATCH -N 4 # 4 nodes
#SBATCH --ntasks-per-node=32 # tasks out of 128
#SBATCH --gres=gpu:4 # gpus per node out of 4
#SBATCH --mem=246000 # memory per node out of 246000MB
#SBATCH hetjob
#SBATCH -p my_partition
#SBATCH --exclusive
#SBATCH --time 16:00:00 # format: HH:MM:SS
#SBATCH -N 8 # 8 nodes
#SBATCH --ntasks-per-node=32 # tasks out of 128
#SBATCH --gres=gpu:4 # gpus per node out of 4
#SBATCH --mem=246000 # memory per node out of 246000MB
#SBATCH hetjob
#SBATCH -p my_partition
#SBATCH --exclusive
#SBATCH --time 16:00:00 # format: HH:MM:SS
#SBATCH -N 16 # 16 nodes
#SBATCH --ntasks-per-node=32 # tasks out of 128
#SBATCH --gres=gpu:4 # gpus per node out of 4
#SBATCH --mem=246000 # memory per node out of 246000MB
srun --job-name=job1 --output=4cpu_%N_%j.out --het-group=0 script.sh 4
srun --job-name=job2 --output=8cpu_%N_%j.out --het-group=0 script.sh 8
srun --job-name=job3 --output=16cpu_%N_%j.out --het-group=0 script.sh 16
srun --job-name=job4 --output=32cpu_%N_%j.out --het-group=0 script.sh 32
srun --job-name=job5 --output=64cpu_%N_%j.out --het-group=1 script.sh 64
srun --job-name=job6 --output=128cpu_%N_%j.out --het-group=2 script.sh 128
srun --job-name=job7 --output=256cpu_%N_%j.out --het-group=3 script.sh 256
srun --job-name=job8 --output=512cpu_%N_%j.out --het-group=4 script.sh 512
Here script.sh takes as argument the number of processors and it is of the form
make cpp_program_I_need_to_run
mkdir -p my_results
mpirun -n $1 cpp_program_I_need_to_run
# other tasks
When I perform, on my cluster sbatch slurm_script.slurm jobs launched crash with exit code 8 and the following error(s):
cat slurm-8482798.out
srun: error: r242n13: tasks 0-31: Exited with exit code 8
srun: launch/slurm: _step_signal: Terminating StepId=8482798.0
srun: error: r242n13: tasks 0-31: Exited with exit code 8
srun: launch/slurm: _step_signal: Terminating StepId=8482798.1
srun: error: r242n13: tasks 0-31: Exited with exit code 8
srun: launch/slurm: _step_signal: Terminating StepId=8482798.2
srun: error: r242n13: tasks 0-31: Exited with exit code 8
srun: launch/slurm: _step_signal: Terminating StepId=8482798.3
...
also
slurmstepd: error: Unable to create TMPDIR [/scratch_local/slurm_job.8482798]: Permission denied
slurmstepd: error: Setting TMPDIR to /tmp
slurmstepd: error: execve(): /cluster/home/userexternal/username/myfolder/script.sh: Exec format error
slurmstepd: error: execve(): /cluster/home/userexternal/username/myfolder/script.sh: Exec format error
...
and so on for many lines.
Is there a way to make it work? The only thing I can think is that the mpirun call in my script.sh is redundant, but then I don't have many ideas.
Thank you in advance

Answer 1

Indeed the mpirun command is redundant. Could you clarify what script.sh is expected to perform?
My approach would be to execute the make in advance, place mkdir -p my_results just after the #SBATCH directives (I assume the directory should be shared among all the job elements, otherwise you should use environment variables to point to node-local storage) and remove mpirun in favour of srun ... cpp_program_I_need_to_run.