limiting number of concurrent processes in a bash script - bash

Question

I am trying to use a bash script to do several jobs in parallel. The jobs are memory intensive so I need to control the number that launch at a time. What I have is below, and it broadly works, but sometimes the delay loop is unaware of a job that has just been started, so several extra jobs get launched, causing the system to run out of memory.
Adding a sleep before the while statement in the delay loop reduces this problem, but does not completely eliminate it. Anyone know of a way to cure this. I'm running on Solaris if that's relevant.
#!/bin/bash
delay(){
while [ 8 -le $(ps -ef |grep myjob |wc -l) ]
do
sleep 1
done
}
./myjob -params1 &
delay
./myjob -params2 &
delay
./myjob -params3 &
delay
./myjob -params4 &
delay
.
.
.

Answer 1

GNU parallel utility http://www.gnu.org/software/parallel/ might be the right tool as it can be said more easy to use than xargs

Answer 2

First, I'll give you a stripped down example of something I do in a couple of my linux scripts. This should work on solaris, but I don't have any systems currently to test on. I modified a couple of things that used /proc, so if anything doesn't work let me know.
#!/bin/bash
# set the max # of threads
max_threads=4
# set the max system load
max_load=4
print_jobs(){
# flush finished jobs messages
jobs > /dev/null
for x in $(jobs -p) ; do
# print all jobs
echo "$x"
done
}
job_count(){
cnt=$(print_jobs $1)
if [ -n "$cnt" ]; then
wc -l <<< "$cnt"
else
echo 0
fi
}
cur_load(){
# get the 1 minute load average integer
uptime |sed 's/.*load average[s]*:[[:space:]]*\([^.]*\)\..*/\1/g'
}
main_function(){
# get current job count and load
jcnow=$(job_count)
loadnow=$(cur_load)
# first, enter a loop waiting for load/threads to be below thresholds
while [ $loadnow -ge $max_load ] || [ $jcnow -ge $max_threads ]; do
if ! [ $firstout ]; then
echo "entering sleep loop. load: $loadnow, threads: $jcnow"
st=$(date +%s)
local firstout=true
else
now=$(date +%s)
# if it's been 5 minutes, echo again:
if [ $(($now - $st)) -ge 300 ]; then
echo "still sleeping. load: $loadnow, threads: $jcnow"
st=$(date +%s)
fi
fi
sleep 5s
# refresh these variables for loop
loadnow=$(cur_load)
jcnow=$(job_count)
unset firstout
done
( ./myjob $# ) &
}
# do some actual work
for jobparams in "params1" "params2" "params3" "params4" "params5" "params6" "params7" ; do
main_function $jobparams
done
wait
A couple of caveats:
you should trap signals so you can kill child processes. I do not know how to do this in solaris, but this works for on linux: trap 'echo "exiting" ; rm -f $lockfile ; kill 0 ; exit' INT TERM EXIT
if load climbs while jobs are already running there's no facility to throttle down
If you're not concerned about load at all, this can be a bit simpler:
#!/bin/bash
# set the max # of threads
max_threads=4
print_jobs(){
# flush finished jobs messages
jobs > /dev/null
for x in $(jobs -p) ; do
# print all jobs
echo "$x"
done
}
job_count(){
cnt=$(print_jobs $1)
if [ -n "$cnt" ]; then
wc -l <<< "$cnt"
else
echo 0
fi
}
main_function(){
# get current job count
jcnow=$(job_count)
# first, enter a loop waiting for threads to be below thresholds
while [ $jcnow -ge $max_threads ]; do
if ! [ $firstout ]; then
echo "entering sleep loop. threads: $jcnow"
st=$(date +%s)
local firstout=true
else
now=$(date +%s)
# if it's been 5 minutes, echo again:
if [ $(($now - $st)) -ge 300 ]; then
echo "still sleeping. threads: $jcnow"
st=$(date +%s)
fi
fi
sleep 5s
# refresh these variables for loop
jcnow=$(job_count)
unset firstout
done
( ./myjob $# ) &
}
# do some actual work
for jobparams in "params1" "params2" "params3" "params4" "params5" "params6" "params7" ; do
main_function $jobparams
done
wait

Answer 3

Use xargs to do this. Pass it -n 1 to indicate one parameter per job and use the --max-jobs parameter to specify the number of concurrent processes.

Answer 4

Formulate your script in terms of a makefile and let make -j N sort it out through Parallel Execution.