Problem: Inspired by this thread, I'm trying to write a wrapper script that submits SLURM array jobs with bash variables. However, I'm running into issues with SLURM environment variables like $SLURM_ARRAY_TASK_ID as it acts as an empty variable.
I suspect it has something to do with how the test_wrapper.sh is parsing the yet undefined SLURM variable, but I can't seem to find a solution.
Below I provide a working example with a simple python script that should take an array ID as an input variable, but when it is called by the bash wrapper script, the python script crashes as it receives an empty variable.
test_wrapper.sh :
#!/bin/bash
for argument in "$#"
do
key=$(echo $argument | cut -f 1 -d'=')
value=$(echo $argument | cut -f 2 -d'=')
case "$key" in
"job_name") job_name="$value" ;;
"cpus") cpus="$value" ;;
"memory") memory="$value" ;;
"time") time="$value" ;;
"array") array="$value" ;;
*)
esac
done
sbatch <<EOT
#!/bin/bash
#SBATCH --account=foobar
#SBATCH --cpus-per-task=${cpus:-1}
#SBATCH --mem-per-cpu=${memory:-1}GB
#SBATCH --time=${time:-00:01:00}
#SBATCH --array=${array:-1-2}
#SBATCH --job-name=${job_name:-Default_Job_Name}
if [ -z "$SLURM_ARRAY_TASK_ID" ]
then
echo "The array ID \$SLURM_ARRAY_TASK_ID is empty"
else
echo "The array ID \$SLURM_ARRAY_TASK_ID is NOT empty"
fi
srun python foo.py -a $SLURM_ARRAY_TASK_ID
echo "Job finished with exit code $?"
EOT
where foo.py is:
import argparse
def main(args):
print('array number is : {}'.format(args.array_number))
if __name__ == "__main__":
parser = argparse.ArgumentParser()
parser.add_argument("-a", "--array_number",
help="the value passed from SLURM_ARRAY_TASK_ID"
)
args = parser.parse_args()
main(args)
$cat slurm-123456789_1.out yields :
The array ID 1 is empty
usage: foo.py [-h] [-a ARRAY_NUMBER]
foo.py: error: argument -a/--array_number: expected one argument
srun: error: nc10931: task 0: Exited with exit code 2
Job finished with exit code 0
I find it strange, that "The array ID 1 is empty" is correctly printing the $SLURM_ARRAY_TASK_ID (??)
So according to this page:
Job arrays will have two additional environment variable set. SLURM_ARRAY_JOB_ID will be set to the first job ID of the array. SLURM_ARRAY_TASK_ID will be set to the job array index value.
That suggests to me that sbatch is supposed to set these for you. In that case, you need to escape all instances of $SLURM_ARRAY_TASK_ID in the script you pass via the heredoc so that they don't get prematurely substituted before sbatch can set the relevant environment variable.
The two options for this are:
If you don't want any expansions to occur at all, quote the heredoc delimiter.
sbatch <<"EOT"
<your script here>
EOT
If you need some expansions to occur but want to disable others, then escape the ones that should not be expanded by putting a \ in front of them like you have done in your existing script.
Thanks to the feedback posted in the comments I was able to fix the issue. Posting a "fixed" version of the wrapper script below.
In short, the solution is to escape $SLURM_ARRAY_TASK_ID.
#!/bin/bash
for argument in "$#"
do
key=$(echo $argument | cut -f 1 -d'=')
value=$(echo $argument | cut -f 2 -d'=')
case "$key" in
"job_name") job_name="$value" ;;
"cpus") cpus="$value" ;;
"memory") memory="$value" ;;
"time") time="$value" ;;
"array") array="$value" ;;
*)
esac
done
{ tee /dev/stderr | sbatch; } <<EOT
#!/bin/bash
#SBATCH --account=foobar
#SBATCH --cpus-per-task=${cpus:-1}
#SBATCH --mem-per-cpu=${memory:-1}GB
#SBATCH --time=${time:-00:01:00}
#SBATCH --array=${array:-1-2}
#SBATCH --job-name=${job_name:-Default_Job_Name}
if [ -z "\$SLURM_ARRAY_TASK_ID" ]
then
echo "The array ID \$SLURM_ARRAY_TASK_ID is empty"
else
echo "The array ID \$SLURM_ARRAY_TASK_ID is NOT empty"
fi
python foo.py -a \$SLURM_ARRAY_TASK_ID
EOT
cat slurm-123456789_1.out yields :
The array ID 1 is NOT empty
array number is : 1
Note: the { tee /dev/stderr | sbatch; } is not necessary, but is very useful for debugging (thanks Charles Duffy)
I submit a job to cluster using qsub SubmitJob.sh. It works well but takes a long time to finish. Inside of SubmitJob.sh there is for loop which runs sequentially. I would like to convert my for loop for parallel job submission, such that each of them submits a single job (SubmitJob.sh).
#!/bin/bash
#$ -S /bin/bash
#$ -V -cwd
#$ -e ./error.$JOB_NAME.$JOB_ID
#$ -o ./outpt.$JOB_NAME.$JOB_ID
#$ -l h_vmem=256g
##$ -q long
##$ -pe smp 4
#$ -l h_rt=24:00:00
cd /mydirectroy/
for ID in $(cat FilID.txt) ; do
Do_Somthing -n $ID -o /OutputDirectory/$ID
done
I had to do something like this once or twice. The generic idea is that you supply parts of a array as reference to a function and execute it as child processes. I choose to use the square root as divider, because the work load will grow linear to the amount of items to process.
#! /bin/bash
FILE="FilID.txt"
DATA=($(cat ${FILE}))
AMOUNT=${#DATA[#]}
RANGE=$(echo "sqrt(${AMOUNT})" | bc)
echo ${amount}
echo $range
function _child {
local -n numbers=$1
echo "From ${numbers[0]} to ${numbers[-1]}"
for n in ${numbers[#]}; do echo -n "$n, "; done
echo
}
for ((i=0; i<AMOUNT; i+=RANGE)) {
part=(${DATA[#]:$i:$RANGE})
_child part &
# wait
}
wait
exit 0
You can test the script by populating FilID.txt as follows. Uncomment the wait in the for loop for readable output.
$ seq 0 98 > FilID.txt
You might want to wait until every N child processes are finished before you start the next batch. Back when I executed the script, the load became too high and Linux choose to kill our virtual development environment :p
P.S. if FilID.txt contain spaces with filenames you have to set IFS=$'\n' or something.
I'm trying to edit my working bash script to an SGE script in order to submit it as a job to the cluster.
Currently I have:
#!/bin/bash
# Perform fastqc on files in a specified directory.
for ((j=1; j <=17; j++))
do
directory=/data4/una/batch"$j"/
files=$""$directory"/*.fastq.gz"
batch=$"batch_"$j""
outfile=$""$batch"_submit_script.sh"
echo "#!/bin/bash">>$outfile;
echo "# Your job name">>$outfile;
echo "# -N $batch">>$outfile;
echo "# The job should be placed into the queue 'all.q'">>$outfile;
echo "#$ -q all.q">>$outfile;
echo "# Running in the current working directory">>$outfile;
echo "#$ -cwd">>$outfile;
echo "">>$outfile;
echo "# Export some necessary environment variables">>$outfile;
echo "#$ -S /bin/bash">>$outfile;
echo "#$ -v PATH">>$outfile;
echo "#$ -v LD_LIBRARY_PATH">>$outfile;
echo "#$ -v PYTHONPATH">>$outfile;
echo "# Finally, put your command here">>$outfile;
echo "">>$outfile;
echo "#$ for i in $files;">>$outfile;
echo "#$ do;">>$outfile;
echo "#$ fastqc -f fastq -o /data4/una/test/fastq/$i;">>$outfile;
echo "#$done">>$outfile;
echo "">>$outfile;
qsub $outfile;
done
But I'm getting an error:
Unable to read script file because of error: ERROR! invalid option argument "-f"
But
fastqc -f fastq -o /data4/una/test/fastq/$i
is a totally valid line in my bash script.
Thoughts?
Thanks!
It actually was poor formatting for my loop that was causing this error. I didn't need to start those lines with #$ at all, so those lines become:
echo "for i in $files;">>$outfile;
echo "do">>$outfile;
echo " fastqc -f fastq -o /data4/una/test/fastqc $i">>$outfile;
echo "done">>$outfile;
echo "">>$outfile;
qsub $outfile;
This question already has answers here:
Command not found error in Bash variable assignment
(5 answers)
Closed 6 years ago.
I'm not certain what the problem is with my slurm script - the error messages that I'm receiving are ambiguous redirect for my $input and command not found for when I'm trying to define my variables.
#!/bin/bash
#SBATCH --job-name=gim
#SBATCH --time=24:00:00
#SBATCH --ntasks=20
#SBATCH --ntasks-per-node=2
#SBATCH --cpus-per-task=1
#SBATCH -o output_%A_%a.out #Standard Output
#SBATCH -e output_%A_%a.err #Standard Error
module load program
input= gim${SLURM_ARRAY_TASK_ID}.gjf
output= gim${SLURM_ARRAY_TASK_ID}.log
program $input > $output
The way I run it is:
sbatch --array=1-500 ./slurm.job
Whitespace matters:
#!/bin/bash
# ...etc...
input=gim${SLURM_ARRAY_TASK_ID}.gjf
output=gim${SLURM_ARRAY_TASK_ID}.log
program "$input" > "$output"
Note the lack of spaces surrounding the = sign for the assignments. Whitespace matters:
foo = bar # this runs "foo" with "=" as the first argument and "bar" as the second
foo =bar # this runs "foo" with "=bar" as its first argument
foo= bar # this runs "bar" as a command with "foo" as an empty string in its environment
foo=bar # this assigns the value "bar" to the shell variable "foo"
I have more than 10 tasks to execute, and the system restrict that there at most 4 tasks can run at the same time.
My task can be started like:
myprog taskname
How can I write a bash shell script to run these task. The most important thing is that when one task finish, the script can start another immediately, making the running tasks count remain 4 all the time.
Use xargs:
xargs -P <maximum-number-of-process-at-a-time> -n <arguments-per-process> <command>
Details here.
I chanced upon this thread while looking into writing my own process pool and particularly liked Brandon Horsley's solution, though I couldn't get the signals working right, so I took inspiration from Apache and decided to try a pre-fork model with a fifo as my job queue.
The following function is the function that the worker processes run when forked.
# \brief the worker function that is called when we fork off worker processes
# \param[in] id the worker ID
# \param[in] job_queue the fifo to read jobs from
# \param[in] result_log the temporary log file to write exit codes to
function _job_pool_worker()
{
local id=$1
local job_queue=$2
local result_log=$3
local line=
exec 7<> ${job_queue}
while [[ "${line}" != "${job_pool_end_of_jobs}" && -e "${job_queue}" ]]; do
# workers block on the exclusive lock to read the job queue
flock --exclusive 7
read line <${job_queue}
flock --unlock 7
# the worker should exit if it sees the end-of-job marker or run the
# job otherwise and save its exit code to the result log.
if [[ "${line}" == "${job_pool_end_of_jobs}" ]]; then
# write it one more time for the next sibling so that everyone
# will know we are exiting.
echo "${line}" >&7
else
_job_pool_echo "### _job_pool_worker-${id}: ${line}"
# run the job
{ ${line} ; }
# now check the exit code and prepend "ERROR" to the result log entry
# which we will use to count errors and then strip out later.
local result=$?
local status=
if [[ "${result}" != "0" ]]; then
status=ERROR
fi
# now write the error to the log, making sure multiple processes
# don't trample over each other.
exec 8<> ${result_log}
flock --exclusive 8
echo "${status}job_pool: exited ${result}: ${line}" >> ${result_log}
flock --unlock 8
exec 8>&-
_job_pool_echo "### _job_pool_worker-${id}: exited ${result}: ${line}"
fi
done
exec 7>&-
}
You can get a copy of my solution at Github. Here's a sample program using my implementation.
#!/bin/bash
. job_pool.sh
function foobar()
{
# do something
true
}
# initialize the job pool to allow 3 parallel jobs and echo commands
job_pool_init 3 0
# run jobs
job_pool_run sleep 1
job_pool_run sleep 2
job_pool_run sleep 3
job_pool_run foobar
job_pool_run foobar
job_pool_run /bin/false
# wait until all jobs complete before continuing
job_pool_wait
# more jobs
job_pool_run /bin/false
job_pool_run sleep 1
job_pool_run sleep 2
job_pool_run foobar
# don't forget to shut down the job pool
job_pool_shutdown
# check the $job_pool_nerrors for the number of jobs that exited non-zero
echo "job_pool_nerrors: ${job_pool_nerrors}"
Hope this helps!
Using GNU Parallel you can do:
cat tasks | parallel -j4 myprog
If you have 4 cores, you can even just do:
cat tasks | parallel myprog
From http://git.savannah.gnu.org/cgit/parallel.git/tree/README:
Full installation
Full installation of GNU Parallel is as simple as:
./configure && make && make install
Personal installation
If you are not root you can add ~/bin to your path and install in
~/bin and ~/share:
./configure --prefix=$HOME && make && make install
Or if your system lacks 'make' you can simply copy src/parallel
src/sem src/niceload src/sql to a dir in your path.
Minimal installation
If you just need parallel and do not have 'make' installed (maybe the
system is old or Microsoft Windows):
wget http://git.savannah.gnu.org/cgit/parallel.git/plain/src/parallel
chmod 755 parallel
cp parallel sem
mv parallel sem dir-in-your-$PATH/bin/
Test the installation
After this you should be able to do:
parallel -j0 ping -nc 3 ::: foss.org.my gnu.org freenetproject.org
This will send 3 ping packets to 3 different hosts in parallel and print
the output when they complete.
Watch the intro video for a quick introduction:
https://www.youtube.com/playlist?list=PL284C9FF2488BC6D1
I would suggest writing four scripts, each one of which executes a certain number of tasks in series. Then write another script that starts the four scripts in parallel. For instance, if you have scripts, script1.sh, script2.sh, script3.sh, and script4.sh, you could have a script called headscript.sh like so.
#!/bin/sh
./script1.sh &
./script2.sh &
./script3.sh &
./script4.sh &
I found the best solution proposed in A Foo Walks into a Bar... blog using build-in functionality of well know xargs tool
First create a file commands.txt with list of commands you want to execute
myprog taskname1
myprog taskname2
myprog taskname3
myprog taskname4
...
myprog taskname123
and then pipe it to xargs like this to execute in 4 processes pool:
cat commands.txt | xargs -I CMD --max-procs=4 bash -c CMD
you can modify no of process
Following #Parag Sardas' answer and the documentation linked here's a quick script you might want to add on your .bash_aliases.
Relinking the doc link because it's worth a read
#!/bin/bash
# https://stackoverflow.com/a/19618159
# https://stackoverflow.com/a/51861820
#
# Example file contents:
# touch /tmp/a.txt
# touch /tmp/b.txt
if [ "$#" -eq 0 ]; then
echo "$0 <file> [max-procs=0]"
exit 1
fi
FILE=${1}
MAX_PROCS=${2:-0}
cat $FILE | while read line; do printf "%q\n" "$line"; done | xargs --max-procs=$MAX_PROCS -I CMD bash -c CMD
I.e.
./xargs-parallel.sh jobs.txt 4 maximum of 4 processes read from jobs.txt
You could probably do something clever with signals.
Note this is only to illustrate the concept, and thus not thoroughly tested.
#!/usr/local/bin/bash
this_pid="$$"
jobs_running=0
sleep_pid=
# Catch alarm signals to adjust the number of running jobs
trap 'decrement_jobs' SIGALRM
# When a job finishes, decrement the total and kill the sleep process
decrement_jobs()
{
jobs_running=$(($jobs_running - 1))
if [ -n "${sleep_pid}" ]
then
kill -s SIGKILL "${sleep_pid}"
sleep_pid=
fi
}
# Check to see if the max jobs are running, if so sleep until woken
launch_task()
{
if [ ${jobs_running} -gt 3 ]
then
(
while true
do
sleep 999
done
) &
sleep_pid=$!
wait ${sleep_pid}
fi
# Launch the requested task, signalling the parent upon completion
(
"$#"
kill -s SIGALRM "${this_pid}"
) &
jobs_running=$((${jobs_running} + 1))
}
# Launch all of the tasks, this can be in a loop, etc.
launch_task task1
launch_task tast2
...
launch_task task99
This tested script runs 5 jobs at a time and will restart a new job as soon as it does (due to the kill of the sleep 10.9 when we get a SIGCHLD. A simpler version of this could use direct polling (change the sleep 10.9 to sleep 1 and get rid of the trap).
#!/usr/bin/bash
set -o monitor
trap "pkill -P $$ -f 'sleep 10\.9' >&/dev/null" SIGCHLD
totaljobs=15
numjobs=5
worktime=10
curjobs=0
declare -A pidlist
dojob()
{
slot=$1
time=$(echo "$RANDOM * 10 / 32768" | bc -l)
echo Starting job $slot with args $time
sleep $time &
pidlist[$slot]=`jobs -p %%`
curjobs=$(($curjobs + 1))
totaljobs=$(($totaljobs - 1))
}
# start
while [ $curjobs -lt $numjobs -a $totaljobs -gt 0 ]
do
dojob $curjobs
done
# Poll for jobs to die, restarting while we have them
while [ $totaljobs -gt 0 ]
do
for ((i=0;$i < $curjobs;i++))
do
if ! kill -0 ${pidlist[$i]} >&/dev/null
then
dojob $i
break
fi
done
sleep 10.9 >&/dev/null
done
wait
Other answer about 4 shell scripts does not fully satisfies me as it assumes that all tasks take approximatelu the same time and because it requires manual set up. But here is how I would improve it.
Main script will create symbolic links to executables following certain namimg convention. For example,
ln -s executable1 ./01-task.01
first prefix is for sorting and suffix identifies batch (01-04).
Now we spawn 4 shell scripts that would take batch number as input and do something like this
for t in $(ls ./*-task.$batch | sort ; do
t
rm t
done
Look at my implementation of job pool in bash: https://github.com/spektom/shell-utils/blob/master/jp.sh
For example, to run at most 3 processes of cURL when downloading from a lot of URLs, you can wrap your cURL commands as follows:
./jp.sh "My Download Pool" 3 curl http://site1/...
./jp.sh "My Download Pool" 3 curl http://site2/...
./jp.sh "My Download Pool" 3 curl http://site3/...
...
Here is my solution. The idea is quite simple. I create a fifo as a semaphore, where each line stands for an available resource. When reading the queue, the main process blocks if there is nothing left. And, we return the resource after the task is done by simply echoing anything to the queue.
function task() {
local task_no="$1"
# doing the actual task...
echo "Executing Task ${task_no}"
# which takes a long time
sleep 1
}
function execute_concurrently() {
local tasks="$1"
local ps_pool_size="$2"
# create an anonymous fifo as a Semaphore
local sema_fifo
sema_fifo="$(mktemp -u)"
mkfifo "${sema_fifo}"
exec 3<>"${sema_fifo}"
rm -f "${sema_fifo}"
# every 'x' stands for an available resource
for i in $(seq 1 "${ps_pool_size}"); do
echo 'x' >&3
done
for task_no in $(seq 1 "${tasks}"); do
read dummy <&3 # blocks util a resource is available
(
trap 'echo x >&3' EXIT # returns the resource on exit
task "${task_no}"
)&
done
wait # wait util all forked tasks have finished
}
execute_concurrently 10 4
The script above will run 10 tasks and 4 each time concurrently. You can change the $(seq 1 "${tasks}") sequence to the actual task queue you want to run.
I made my modifications based on methods introduced in this Writing a process pool in Bash.
#!/bin/bash
#set -e # this doesn't work here for some reason
POOL_SIZE=4 # number of workers running in parallel
#######################################################################
# populate jobs #
#######################################################################
declare -a jobs
for (( i = 1988; i < 2019; i++ )); do
jobs+=($i)
done
echo '################################################'
echo ' Launching jobs'
echo '################################################'
parallel() {
local proc procs jobs cur
jobs=("$#") # input jobs array
declare -a procs=() # processes array
cur=0 # current job idx
morework=true
while $morework; do
# if process array size < pool size, try forking a new proc
if [[ "${#procs[#]}" -lt "$POOL_SIZE" ]]; then
if [[ $cur -lt "${#jobs[#]}" ]]; then
proc=${jobs[$cur]}
echo "JOB ID = $cur; JOB = $proc."
###############
# do job here #
###############
sleep 3 &
# add to current running processes
procs+=("$!")
# move to the next job
((cur++))
else
morework=false
continue
fi
fi
for n in "${!procs[#]}"; do
kill -0 "${procs[n]}" 2>/dev/null && continue
# if process is not running anymore, remove from array
unset procs[n]
done
done
wait
}
parallel "${jobs[#]}"
xargs with -P and -L options does the job.
You can extract the idea from the example below:
#!/usr/bin/env bash
workers_pool_size=10
set -e
function doit {
cmds=""
for e in 4 8 16; do
for m in 1 2 3 4 5 6; do
cmd="python3 ./doit.py --m $m -e $e -m $m"
cmds="$cmd\n$cmds"
done
done
echo -e "All commands:\n$cmds"
echo "Workers pool size = $workers_pool_size"
echo -e "$cmds" | xargs -t -P $workers_pool_size -L 1 time > /dev/null
}
doit
#! /bin/bash
doSomething() {
<...>
}
getCompletedThreads() {
_runningThreads=("$#")
removableThreads=()
for pid in "${_runningThreads[#]}"; do
if ! ps -p $pid > /dev/null; then
removableThreads+=($pid)
fi
done
echo "$removableThreads"
}
releasePool() {
while [[ ${#runningThreads[#]} -eq $MAX_THREAD_NO ]]; do
echo "releasing"
removableThreads=( $(getCompletedThreads "${runningThreads[#]}") )
if [ ${#removableThreads[#]} -eq 0 ]; then
sleep 0.2
else
for removableThread in "${removableThreads[#]}"; do
runningThreads=( ${runningThreads[#]/$removableThread} )
done
echo "released"
fi
done
}
waitAllThreadComplete() {
while [[ ${#runningThreads[#]} -ne 0 ]]; do
removableThreads=( $(getCompletedThreads "${runningThreads[#]}") )
for removableThread in "${removableThreads[#]}"; do
runningThreads=( ${runningThreads[#]/$removableThread} )
done
if [ ${#removableThreads[#]} -eq 0 ]; then
sleep 0.2
fi
done
}
MAX_THREAD_NO=10
runningThreads=()
sequenceNo=0
for i in {1..36}; do
releasePool
((sequenceNo++))
echo "added $sequenceNo"
doSomething &
pid=$!
runningThreads+=($pid)
done
waitAllThreadComplete