Related
I have a command CMD called from my main bourne shell script that takes forever.
I want to modify the script as follows:
Run the command CMD in parallel as a background process (CMD &).
In the main script, have a loop to monitor the spawned command every few seconds. The loop also echoes some messages to stdout indicating progress of the script.
Exit the loop when the spawned command terminates.
Capture and report the exit code of the spawned process.
Can someone give me pointers to accomplish this?
1: In bash, $! holds the PID of the last background process that was executed. That will tell you what process to monitor, anyway.
4: wait <n> waits until the process with PID <n> is complete (it will block until the process completes, so you might not want to call this until you are sure the process is done), and then returns the exit code of the completed process.
2, 3: ps or ps | grep " $! " can tell you whether the process is still running. It is up to you how to understand the output and decide how close it is to finishing. (ps | grep isn't idiot-proof. If you have time you can come up with a more robust way to tell whether the process is still running).
Here's a skeleton script:
# simulate a long process that will have an identifiable exit code
(sleep 15 ; /bin/false) &
my_pid=$!
while ps | grep " $my_pid " # might also need | grep -v grep here
do
echo $my_pid is still in the ps output. Must still be running.
sleep 3
done
echo Oh, it looks like the process is done.
wait $my_pid
# The variable $? always holds the exit code of the last command to finish.
# Here it holds the exit code of $my_pid, since wait exits with that code.
my_status=$?
echo The exit status of the process was $my_status
This is how I solved it when I had a similar need:
# Some function that takes a long time to process
longprocess() {
# Sleep up to 14 seconds
sleep $((RANDOM % 15))
# Randomly exit with 0 or 1
exit $((RANDOM % 2))
}
pids=""
# Run five concurrent processes
for i in {1..5}; do
( longprocess ) &
# store PID of process
pids+=" $!"
done
# Wait for all processes to finish, will take max 14s
# as it waits in order of launch, not order of finishing
for p in $pids; do
if wait $p; then
echo "Process $p success"
else
echo "Process $p fail"
fi
done
The pid of a backgrounded child process is stored in $!.
You can store all child processes' pids into an array, e.g. PIDS[].
wait [-n] [jobspec or pid …]
Wait until the child process specified by each process ID pid or job specification jobspec exits and return the exit status of the last command waited for. If a job spec is given, all processes in the job are waited for. If no arguments are given, all currently active child processes are waited for, and the return status is zero. If the -n option is supplied, wait waits for any job to terminate and returns its exit status. If neither jobspec nor pid specifies an active child process of the shell, the return status is 127.
Use wait command you can wait for all child processes finish, meanwhile you can get exit status of each child processes via $? and store status into STATUS[]. Then you can do something depending by status.
I have tried the following 2 solutions and they run well. solution01 is
more concise, while solution02 is a little complicated.
solution01
#!/bin/bash
# start 3 child processes concurrently, and store each pid into array PIDS[].
process=(a.sh b.sh c.sh)
for app in ${process[#]}; do
./${app} &
PIDS+=($!)
done
# wait for all processes to finish, and store each process's exit code into array STATUS[].
for pid in ${PIDS[#]}; do
echo "pid=${pid}"
wait ${pid}
STATUS+=($?)
done
# after all processed finish, check their exit codes in STATUS[].
i=0
for st in ${STATUS[#]}; do
if [[ ${st} -ne 0 ]]; then
echo "$i failed"
else
echo "$i finish"
fi
((i+=1))
done
solution02
#!/bin/bash
# start 3 child processes concurrently, and store each pid into array PIDS[].
i=0
process=(a.sh b.sh c.sh)
for app in ${process[#]}; do
./${app} &
pid=$!
PIDS[$i]=${pid}
((i+=1))
done
# wait for all processes to finish, and store each process's exit code into array STATUS[].
i=0
for pid in ${PIDS[#]}; do
echo "pid=${pid}"
wait ${pid}
STATUS[$i]=$?
((i+=1))
done
# after all processed finish, check their exit codes in STATUS[].
i=0
for st in ${STATUS[#]}; do
if [[ ${st} -ne 0 ]]; then
echo "$i failed"
else
echo "$i finish"
fi
((i+=1))
done
As I see almost all answers use external utilities (mostly ps) to poll the state of the background process. There is a more unixesh solution, catching the SIGCHLD signal. In the signal handler it has to be checked which child process was stopped. It can be done by kill -0 <PID> built-in (universal) or checking the existence of /proc/<PID> directory (Linux specific) or using the jobs built-in (bash specific. jobs -l also reports the pid. In this case the 3rd field of the output can be Stopped|Running|Done|Exit . ).
Here is my example.
The launched process is called loop.sh. It accepts -x or a number as an argument. For -x is exits with exit code 1. For a number it waits num*5 seconds. In every 5 seconds it prints its PID.
The launcher process is called launch.sh:
#!/bin/bash
handle_chld() {
local tmp=()
for((i=0;i<${#pids[#]};++i)); do
if [ ! -d /proc/${pids[i]} ]; then
wait ${pids[i]}
echo "Stopped ${pids[i]}; exit code: $?"
else tmp+=(${pids[i]})
fi
done
pids=(${tmp[#]})
}
set -o monitor
trap "handle_chld" CHLD
# Start background processes
./loop.sh 3 &
pids+=($!)
./loop.sh 2 &
pids+=($!)
./loop.sh -x &
pids+=($!)
# Wait until all background processes are stopped
while [ ${#pids[#]} -gt 0 ]; do echo "WAITING FOR: ${pids[#]}"; sleep 2; done
echo STOPPED
For more explanation see: Starting a process from bash script failed
#/bin/bash
#pgm to monitor
tail -f /var/log/messages >> /tmp/log&
# background cmd pid
pid=$!
# loop to monitor running background cmd
while :
do
ps ax | grep $pid | grep -v grep
ret=$?
if test "$ret" != "0"
then
echo "Monitored pid ended"
break
fi
sleep 5
done
wait $pid
echo $?
I would change your approach slightly. Rather than checking every few seconds if the command is still alive and reporting a message, have another process that reports every few seconds that the command is still running and then kill that process when the command finishes. For example:
#!/bin/sh
cmd() { sleep 5; exit 24; }
cmd & # Run the long running process
pid=$! # Record the pid
# Spawn a process that coninually reports that the command is still running
while echo "$(date): $pid is still running"; do sleep 1; done &
echoer=$!
# Set a trap to kill the reporter when the process finishes
trap 'kill $echoer' 0
# Wait for the process to finish
if wait $pid; then
echo "cmd succeeded"
else
echo "cmd FAILED!! (returned $?)"
fi
Our team had the same need with a remote SSH-executed script which was timing out after 25 minutes of inactivity. Here is a solution with the monitoring loop checking the background process every second, but printing only every 10 minutes to suppress an inactivity timeout.
long_running.sh &
pid=$!
# Wait on a background job completion. Query status every 10 minutes.
declare -i elapsed=0
# `ps -p ${pid}` works on macOS and CentOS. On both OSes `ps ${pid}` works as well.
while ps -p ${pid} >/dev/null; do
sleep 1
if ((++elapsed % 600 == 0)); then
echo "Waiting for the completion of the main script. $((elapsed / 60))m and counting ..."
fi
done
# Return the exit code of the terminated background process. This works in Bash 4.4 despite what Bash docs say:
# "If neither jobspec nor pid specifies an active child process of the shell, the return status is 127."
wait ${pid}
A simple example, similar to the solutions above. This doesn't require monitoring any process output. The next example uses tail to follow output.
$ echo '#!/bin/bash' > tmp.sh
$ echo 'sleep 30; exit 5' >> tmp.sh
$ chmod +x tmp.sh
$ ./tmp.sh &
[1] 7454
$ pid=$!
$ wait $pid
[1]+ Exit 5 ./tmp.sh
$ echo $?
5
Use tail to follow process output and quit when the process is complete.
$ echo '#!/bin/bash' > tmp.sh
$ echo 'i=0; while let "$i < 10"; do sleep 5; echo "$i"; let i=$i+1; done; exit 5;' >> tmp.sh
$ chmod +x tmp.sh
$ ./tmp.sh
0
1
2
^C
$ ./tmp.sh > /tmp/tmp.log 2>&1 &
[1] 7673
$ pid=$!
$ tail -f --pid $pid /tmp/tmp.log
0
1
2
3
4
5
6
7
8
9
[1]+ Exit 5 ./tmp.sh > /tmp/tmp.log 2>&1
$ wait $pid
$ echo $?
5
Another solution is to monitor processes via the proc filesystem (safer than ps/grep combo); when you start a process it has a corresponding folder in /proc/$pid, so the solution could be
#!/bin/bash
....
doSomething &
local pid=$!
while [ -d /proc/$pid ]; do # While directory exists, the process is running
doSomethingElse
....
else # when directory is removed from /proc, process has ended
wait $pid
local exit_status=$?
done
....
Now you can use the $exit_status variable however you like.
With this method, your script doesnt have to wait for the background process, you will only have to monitor a temporary file for the exit status.
FUNCmyCmd() { sleep 3;return 6; };
export retFile=$(mktemp);
FUNCexecAndWait() { FUNCmyCmd;echo $? >$retFile; };
FUNCexecAndWait&
now, your script can do anything else while you just have to keep monitoring the contents of retFile (it can also contain any other information you want like the exit time).
PS.: btw, I coded thinking in bash
My solution was to use an anonymous pipe to pass the status to a monitoring loop. There are no temporary files used to exchange status so nothing to cleanup. If you were uncertain about the number of background jobs the break condition could be [ -z "$(jobs -p)" ].
#!/bin/bash
exec 3<> <(:)
{ sleep 15 ; echo "sleep/exit $?" >&3 ; } &
while read -u 3 -t 1 -r STAT CODE || STAT="timeout" ; do
echo "stat: ${STAT}; code: ${CODE}"
if [ "${STAT}" = "sleep/exit" ] ; then
break
fi
done
how about ...
# run your stuff
unset PID
for process in one two three four
do
( sleep $((RANDOM%20)); echo hello from process $process; exit $((RANDOM%3)); ) & 2>&1
PID+=($!)
done
# (optional) report on the status of that stuff as it exits
for pid in "${PID[#]}"
do
( wait "$pid"; echo "process $pid complemted with exit status $?") &
done
# (optional) while we wait, monitor that stuff
while ps --pid "${PID[*]}" --ppid "${PID[*]}" --format pid,ppid,command,pcpu
do
sleep 5
done | xargs -i date '+%x %X {}'
# return non-zero if any are non zero
SUCCESS=0
for pid in "${PID[#]}"
do
wait "$pid" && ((SUCCESS++)) && echo "$pid OK" || echo "$pid returned $?"
done
echo "success for $SUCCESS out of ${#PID} jobs"
exit $(( ${#PID} - SUCCESS ))
This may be extending beyond your question, however if you're concerned about the length of time processes are running for, you may be interested in checking the status of running background processes after an interval of time. It's easy enough to check which child PIDs are still running using pgrep -P $$, however I came up with the following solution to check the exit status of those PIDs that have already expired:
cmd1() { sleep 5; exit 24; }
cmd2() { sleep 10; exit 0; }
pids=()
cmd1 & pids+=("$!")
cmd2 & pids+=("$!")
lasttimeout=0
for timeout in 2 7 11; do
echo -n "interval-$timeout: "
sleep $((timeout-lasttimeout))
# you can only wait on a pid once
remainingpids=()
for pid in ${pids[*]}; do
if ! ps -p $pid >/dev/null ; then
wait $pid
echo -n "pid-$pid:exited($?); "
else
echo -n "pid-$pid:running; "
remainingpids+=("$pid")
fi
done
pids=( ${remainingpids[*]} )
lasttimeout=$timeout
echo
done
which outputs:
interval-2: pid-28083:running; pid-28084:running;
interval-7: pid-28083:exited(24); pid-28084:running;
interval-11: pid-28084:exited(0);
Note: You could change $pids to a string variable rather than array to simplify things if you like.
How to wait in a bash script for several subprocesses spawned from that script to finish, and then return exit code !=0 when any of the subprocesses ends with code !=0?
Simple script:
#!/bin/bash
for i in `seq 0 9`; do
doCalculations $i &
done
wait
The above script will wait for all 10 spawned subprocesses, but it will always give exit status 0 (see help wait). How can I modify this script so it will discover exit statuses of spawned subprocesses and return exit code 1 when any of subprocesses ends with code !=0?
Is there any better solution for that than collecting PIDs of the subprocesses, wait for them in order and sum exit statuses?
wait also (optionally) takes the PID of the process to wait for, and with $! you get the PID of the last command launched in the background.
Modify the loop to store the PID of each spawned sub-process into an array, and then loop again waiting on each PID.
# run processes and store pids in array
for i in $n_procs; do
./procs[${i}] &
pids[${i}]=$!
done
# wait for all pids
for pid in ${pids[*]}; do
wait $pid
done
http://jeremy.zawodny.com/blog/archives/010717.html :
#!/bin/bash
FAIL=0
echo "starting"
./sleeper 2 0 &
./sleeper 2 1 &
./sleeper 3 0 &
./sleeper 2 0 &
for job in `jobs -p`
do
echo $job
wait $job || let "FAIL+=1"
done
echo $FAIL
if [ "$FAIL" == "0" ];
then
echo "YAY!"
else
echo "FAIL! ($FAIL)"
fi
Here is simple example using wait.
Run some processes:
$ sleep 10 &
$ sleep 10 &
$ sleep 20 &
$ sleep 20 &
Then wait for them with wait command:
$ wait < <(jobs -p)
Or just wait (without arguments) for all.
This will wait for all jobs in the background are completed.
If the -n option is supplied, waits for the next job to terminate and returns its exit status.
See: help wait and help jobs for syntax.
However the downside is that this will return on only the status of the last ID, so you need to check the status for each subprocess and store it in the variable.
Or make your calculation function to create some file on failure (empty or with fail log), then check of that file if exists, e.g.
$ sleep 20 && true || tee fail &
$ sleep 20 && false || tee fail &
$ wait < <(jobs -p)
$ test -f fail && echo Calculation failed.
How about simply:
#!/bin/bash
pids=""
for i in `seq 0 9`; do
doCalculations $i &
pids="$pids $!"
done
wait $pids
...code continued here ...
Update:
As pointed by multiple commenters, the above waits for all processes to be completed before continuing, but does not exit and fail if one of them fails, it can be made to do with the following modification suggested by #Bryan, #SamBrightman, and others:
#!/bin/bash
pids=""
RESULT=0
for i in `seq 0 9`; do
doCalculations $i &
pids="$pids $!"
done
for pid in $pids; do
wait $pid || let "RESULT=1"
done
if [ "$RESULT" == "1" ];
then
exit 1
fi
...code continued here ...
If you have GNU Parallel installed you can do:
# If doCalculations is a function
export -f doCalculations
seq 0 9 | parallel doCalculations {}
GNU Parallel will give you exit code:
0 - All jobs ran without error.
1-253 - Some of the jobs failed. The exit status gives the number of failed jobs
254 - More than 253 jobs failed.
255 - Other error.
Watch the intro videos to learn more: http://pi.dk/1
Here's what I've come up with so far. I would like to see how to interrupt the sleep command if a child terminates, so that one would not have to tune WAITALL_DELAY to one's usage.
waitall() { # PID...
## Wait for children to exit and indicate whether all exited with 0 status.
local errors=0
while :; do
debug "Processes remaining: $*"
for pid in "$#"; do
shift
if kill -0 "$pid" 2>/dev/null; then
debug "$pid is still alive."
set -- "$#" "$pid"
elif wait "$pid"; then
debug "$pid exited with zero exit status."
else
debug "$pid exited with non-zero exit status."
((++errors))
fi
done
(("$#" > 0)) || break
# TODO: how to interrupt this sleep when a child terminates?
sleep ${WAITALL_DELAY:-1}
done
((errors == 0))
}
debug() { echo "DEBUG: $*" >&2; }
pids=""
for t in 3 5 4; do
sleep "$t" &
pids="$pids $!"
done
waitall $pids
To parallelize this...
for i in $(whatever_list) ; do
do_something $i
done
Translate it to this...
for i in $(whatever_list) ; do echo $i ; done | ## execute in parallel...
(
export -f do_something ## export functions (if needed)
export PATH ## export any variables that are required
xargs -I{} --max-procs 0 bash -c ' ## process in batches...
{
echo "processing {}" ## optional
do_something {}
}'
)
If an error occurs in one process, it won't interrupt the other processes, but it will result in a non-zero exit code from the sequence as a whole.
Exporting functions and variables may or may not be necessary, in any particular case.
You can set --max-procs based on how much parallelism you want (0 means "all at once").
GNU Parallel offers some additional features when used in place of xargs -- but it isn't always installed by default.
The for loop isn't strictly necessary in this example since echo $i is basically just regenerating the output of $(whatever_list). I just think the use of the for keyword makes it a little easier to see what is going on.
Bash string handling can be confusing -- I have found that using single quotes works best for wrapping non-trivial scripts.
You can easily interrupt the entire operation (using ^C or similar), unlike the the more direct approach to Bash parallelism.
Here's a simplified working example...
for i in {0..5} ; do echo $i ; done |xargs -I{} --max-procs 2 bash -c '
{
echo sleep {}
sleep 2s
}'
This is something that I use:
#wait for jobs
for job in `jobs -p`; do wait ${job}; done
This is an expansion on the most-upvoted answer, by #Luca Tettamanti, to make a fully-runnable example.
That answer left me wondering:
What type of variable is n_procs, and what does it contain? What type of variable is procs, and what does it contain? Can someone please update this answer to make it runnable by adding definitions for those variables? I don't understand how.
...and also:
How do you get the return code from the subprocess when it has completed (which is the whole crux of this question)?
Anyway, I figured it out, so here is a fully-runnable example.
Notes:
$! is how to obtain the PID (Process ID) of the last-executed sub-process.
Running any command with the & after it, like cmd &, for example, causes it to run in the background as a parallel suprocess with the main process.
myarray=() is how to create an array in bash.
To learn a tiny bit more about the wait built-in command, see help wait. See also, and especially, the official Bash user manual on Job Control built-ins, such as wait and jobs, here: https://www.gnu.org/software/bash/manual/html_node/Job-Control-Builtins.html#index-wait.
Full, runnable program: wait for all processes to end
multi_process_program.sh (from my eRCaGuy_hello_world repo):
#!/usr/bin/env bash
# This is a special sleep function which returns the number of seconds slept as
# the "error code" or return code" so that we can easily see that we are in
# fact actually obtaining the return code of each process as it finishes.
my_sleep() {
seconds_to_sleep="$1"
sleep "$seconds_to_sleep"
return "$seconds_to_sleep"
}
# Create an array of whatever commands you want to run as subprocesses
procs=() # bash array
procs+=("my_sleep 5")
procs+=("my_sleep 2")
procs+=("my_sleep 3")
procs+=("my_sleep 4")
num_procs=${#procs[#]} # number of processes
echo "num_procs = $num_procs"
# run commands as subprocesses and store pids in an array
pids=() # bash array
for (( i=0; i<"$num_procs"; i++ )); do
echo "cmd = ${procs[$i]}"
${procs[$i]} & # run the cmd as a subprocess
# store pid of last subprocess started; see:
# https://unix.stackexchange.com/a/30371/114401
pids+=("$!")
echo " pid = ${pids[$i]}"
done
# OPTION 1 (comment this option out if using Option 2 below): wait for all pids
for pid in "${pids[#]}"; do
wait "$pid"
return_code="$?"
echo "PID = $pid; return_code = $return_code"
done
echo "All $num_procs processes have ended."
Change the file above to be executable by running chmod +x multi_process_program.sh, then run it like this:
time ./multi_process_program.sh
Sample output. See how the output of the time command in the call shows it took 5.084sec to run. We were also able to successfully retrieve the return code from each subprocess.
eRCaGuy_hello_world/bash$ time ./multi_process_program.sh
num_procs = 4
cmd = my_sleep 5
pid = 21694
cmd = my_sleep 2
pid = 21695
cmd = my_sleep 3
pid = 21697
cmd = my_sleep 4
pid = 21699
PID = 21694; return_code = 5
PID = 21695; return_code = 2
PID = 21697; return_code = 3
PID = 21699; return_code = 4
All 4 processes have ended.
PID 21694 is done; return_code = 5; 3 PIDs remaining.
PID 21695 is done; return_code = 2; 2 PIDs remaining.
PID 21697 is done; return_code = 3; 1 PIDs remaining.
PID 21699 is done; return_code = 4; 0 PIDs remaining.
real 0m5.084s
user 0m0.025s
sys 0m0.061s
Going further: determine live when each individual process ends
If you'd like to do some action as each process finishes, and you don't know when they will finish, you can poll in an infinite while loop to see when each process terminates, then do whatever action you want.
Simply comment out the "OPTION 1" block of code above, and replace it with this "OPTION 2" block instead:
# OR OPTION 2 (comment out Option 1 above if using Option 2): poll to detect
# when each process terminates, and print out when each process finishes!
while true; do
for i in "${!pids[#]}"; do
pid="${pids[$i]}"
# echo "pid = $pid" # debugging
# See if PID is still running; see my answer here:
# https://stackoverflow.com/a/71134379/4561887
ps --pid "$pid" > /dev/null
if [ "$?" -ne 0 ]; then
# PID doesn't exist anymore, meaning it terminated
# 1st, read its return code
wait "$pid"
return_code="$?"
# 2nd, remove this PID from the `pids` array by `unset`ting the
# element at this index; NB: due to how bash arrays work, this does
# NOT actually remove this element from the array. Rather, it
# removes its index from the `"${!pids[#]}"` list of indices,
# adjusts the array count(`"${#pids[#]}"`) accordingly, and it sets
# the value at this index to either a null value of some sort, or
# an empty string (I'm not exactly sure).
unset "pids[$i]"
num_pids="${#pids[#]}"
echo "PID $pid is done; return_code = $return_code;" \
"$num_pids PIDs remaining."
fi
done
# exit the while loop if the `pids` array is empty
if [ "${#pids[#]}" -eq 0 ]; then
break
fi
# Do some small sleep here to keep your polling loop from sucking up
# 100% of one of your CPUs unnecessarily. Sleeping allows other processes
# to run during this time.
sleep 0.1
done
Sample run and output of the full program with Option 1 commented out and Option 2 in-use:
eRCaGuy_hello_world/bash$ ./multi_process_program.sh
num_procs = 4
cmd = my_sleep 5
pid = 22275
cmd = my_sleep 2
pid = 22276
cmd = my_sleep 3
pid = 22277
cmd = my_sleep 4
pid = 22280
PID 22276 is done; return_code = 2; 3 PIDs remaining.
PID 22277 is done; return_code = 3; 2 PIDs remaining.
PID 22280 is done; return_code = 4; 1 PIDs remaining.
PID 22275 is done; return_code = 5; 0 PIDs remaining.
Each of those PID XXXXX is done lines prints out live right after that process has terminated! Notice that even though the process for sleep 5 (PID 22275 in this case) was run first, it finished last, and we successfully detected each process right after it terminated. We also successfully detected each return code, just like in Option 1.
Other References:
*****+ [VERY HELPFUL] Get exit code of a background process - this answer taught me the key principle that (emphasis added):
wait <n> waits until the process with PID is complete (it will block until the process completes, so you might not want to call this until you are sure the process is done), and then returns the exit code of the completed process.
In other words, it helped me know that even after the process is complete, you can still call wait on it to get its return code!
How to check if a process id (PID) exists
my answer
Remove an element from a Bash array - note that elements in a bash array aren't actually deleted, they are just "unset". See my comments in the code above for what that means.
How to use the command-line executable true to make an infinite while loop in bash: https://www.cyberciti.biz/faq/bash-infinite-loop/
I see lots of good examples listed on here, wanted to throw mine in as well.
#! /bin/bash
items="1 2 3 4 5 6"
pids=""
for item in $items; do
sleep $item &
pids+="$! "
done
for pid in $pids; do
wait $pid
if [ $? -eq 0 ]; then
echo "SUCCESS - Job $pid exited with a status of $?"
else
echo "FAILED - Job $pid exited with a status of $?"
fi
done
I use something very similar to start/stop servers/services in parallel and check each exit status. Works great for me. Hope this helps someone out!
I don't believe it's possible with Bash's builtin functionality.
You can get notification when a child exits:
#!/bin/sh
set -o monitor # enable script job control
trap 'echo "child died"' CHLD
However there's no apparent way to get the child's exit status in the signal handler.
Getting that child status is usually the job of the wait family of functions in the lower level POSIX APIs. Unfortunately Bash's support for that is limited - you can wait for one specific child process (and get its exit status) or you can wait for all of them, and always get a 0 result.
What it appears impossible to do is the equivalent of waitpid(-1), which blocks until any child process returns.
The following code will wait for completion of all calculations and return exit status 1 if any of doCalculations fails.
#!/bin/bash
for i in $(seq 0 9); do
(doCalculations $i >&2 & wait %1; echo $?) &
done | grep -qv 0 && exit 1
Here's my version that works for multiple pids, logs warnings if execution takes too long, and stops the subprocesses if execution takes longer than a given value.
[EDIT] I have uploaded my newer implementation of WaitForTaskCompletion, called ExecTasks at https://github.com/deajan/ofunctions.
There's also a compat layer for WaitForTaskCompletion
[/EDIT]
function WaitForTaskCompletion {
local pids="${1}" # pids to wait for, separated by semi-colon
local soft_max_time="${2}" # If execution takes longer than $soft_max_time seconds, will log a warning, unless $soft_max_time equals 0.
local hard_max_time="${3}" # If execution takes longer than $hard_max_time seconds, will stop execution, unless $hard_max_time equals 0.
local caller_name="${4}" # Who called this function
local exit_on_error="${5:-false}" # Should the function exit program on subprocess errors
Logger "${FUNCNAME[0]} called by [$caller_name]."
local soft_alert=0 # Does a soft alert need to be triggered, if yes, send an alert once
local log_ttime=0 # local time instance for comparaison
local seconds_begin=$SECONDS # Seconds since the beginning of the script
local exec_time=0 # Seconds since the beginning of this function
local retval=0 # return value of monitored pid process
local errorcount=0 # Number of pids that finished with errors
local pidCount # number of given pids
IFS=';' read -a pidsArray <<< "$pids"
pidCount=${#pidsArray[#]}
while [ ${#pidsArray[#]} -gt 0 ]; do
newPidsArray=()
for pid in "${pidsArray[#]}"; do
if kill -0 $pid > /dev/null 2>&1; then
newPidsArray+=($pid)
else
wait $pid
result=$?
if [ $result -ne 0 ]; then
errorcount=$((errorcount+1))
Logger "${FUNCNAME[0]} called by [$caller_name] finished monitoring [$pid] with exitcode [$result]."
fi
fi
done
## Log a standby message every hour
exec_time=$(($SECONDS - $seconds_begin))
if [ $((($exec_time + 1) % 3600)) -eq 0 ]; then
if [ $log_ttime -ne $exec_time ]; then
log_ttime=$exec_time
Logger "Current tasks still running with pids [${pidsArray[#]}]."
fi
fi
if [ $exec_time -gt $soft_max_time ]; then
if [ $soft_alert -eq 0 ] && [ $soft_max_time -ne 0 ]; then
Logger "Max soft execution time exceeded for task [$caller_name] with pids [${pidsArray[#]}]."
soft_alert=1
SendAlert
fi
if [ $exec_time -gt $hard_max_time ] && [ $hard_max_time -ne 0 ]; then
Logger "Max hard execution time exceeded for task [$caller_name] with pids [${pidsArray[#]}]. Stopping task execution."
kill -SIGTERM $pid
if [ $? == 0 ]; then
Logger "Task stopped successfully"
else
errrorcount=$((errorcount+1))
fi
fi
fi
pidsArray=("${newPidsArray[#]}")
sleep 1
done
Logger "${FUNCNAME[0]} ended for [$caller_name] using [$pidCount] subprocesses with [$errorcount] errors."
if [ $exit_on_error == true ] && [ $errorcount -gt 0 ]; then
Logger "Stopping execution."
exit 1337
else
return $errorcount
fi
}
# Just a plain stupid logging function to be replaced by yours
function Logger {
local value="${1}"
echo $value
}
Example, wait for all three processes to finish, log a warning if execution takes loger than 5 seconds, stop all processes if execution takes longer than 120 seconds. Don't exit program on failures.
function something {
sleep 10 &
pids="$!"
sleep 12 &
pids="$pids;$!"
sleep 9 &
pids="$pids;$!"
WaitForTaskCompletion $pids 5 120 ${FUNCNAME[0]} false
}
# Launch the function
someting
If you have bash 4.2 or later available the following might be useful to you. It uses associative arrays to store task names and their "code" as well as task names and their pids. I have also built in a simple rate-limiting method which might come handy if your tasks consume a lot of CPU or I/O time and you want to limit the number of concurrent tasks.
The script launches all tasks in the first loop and consumes the results in the second one.
This is a bit overkill for simple cases but it allows for pretty neat stuff. For example one can store error messages for each task in another associative array and print them after everything has settled down.
#! /bin/bash
main () {
local -A pids=()
local -A tasks=([task1]="echo 1"
[task2]="echo 2"
[task3]="echo 3"
[task4]="false"
[task5]="echo 5"
[task6]="false")
local max_concurrent_tasks=2
for key in "${!tasks[#]}"; do
while [ $(jobs 2>&1 | grep -c Running) -ge "$max_concurrent_tasks" ]; do
sleep 1 # gnu sleep allows floating point here...
done
${tasks[$key]} &
pids+=(["$key"]="$!")
done
errors=0
for key in "${!tasks[#]}"; do
pid=${pids[$key]}
local cur_ret=0
if [ -z "$pid" ]; then
echo "No Job ID known for the $key process" # should never happen
cur_ret=1
else
wait $pid
cur_ret=$?
fi
if [ "$cur_ret" -ne 0 ]; then
errors=$(($errors + 1))
echo "$key (${tasks[$key]}) failed."
fi
done
return $errors
}
main
I've had a go at this and combined all the best parts from the other examples here. This script will execute the checkpids function when any background process exits, and output the exit status without resorting to polling.
#!/bin/bash
set -o monitor
sleep 2 &
sleep 4 && exit 1 &
sleep 6 &
pids=`jobs -p`
checkpids() {
for pid in $pids; do
if kill -0 $pid 2>/dev/null; then
echo $pid is still alive.
elif wait $pid; then
echo $pid exited with zero exit status.
else
echo $pid exited with non-zero exit status.
fi
done
echo
}
trap checkpids CHLD
wait
#!/bin/bash
set -m
for i in `seq 0 9`; do
doCalculations $i &
done
while fg; do true; done
set -m allows you to use fg & bg in a script
fg, in addition to putting the last process in the foreground, has the same exit status as the process it foregrounds
while fg will stop looping when any fg exits with a non-zero exit status
unfortunately this won't handle the case when a process in the background exits with a non-zero exit status. (the loop won't terminate immediately. it will wait for the previous processes to complete.)
Wait for all jobs and return the exit code of the last failing job. Unlike solutions above, this does not require pid saving, or modifying inner loops of scripts. Just bg away, and wait.
function wait_ex {
# this waits for all jobs and returns the exit code of the last failing job
ecode=0
while true; do
[ -z "$(jobs)" ] && break
wait -n
err="$?"
[ "$err" != "0" ] && ecode="$err"
done
return $ecode
}
EDIT: Fixed the bug where this could be fooled by a script that ran a command that didn't exist.
Just store the results out of the shell, e.g. in a file.
#!/bin/bash
tmp=/tmp/results
: > $tmp #clean the file
for i in `seq 0 9`; do
(doCalculations $i; echo $i:$?>>$tmp)&
done #iterate
wait #wait until all ready
sort $tmp | grep -v ':0' #... handle as required
I've just been modifying a script to background and parallelise a process.
I did some experimenting (on Solaris with both bash and ksh) and discovered that 'wait' outputs the exit status if it's not zero , or a list of jobs that return non-zero exit when no PID argument is provided. E.g.
Bash:
$ sleep 20 && exit 1 &
$ sleep 10 && exit 2 &
$ wait
[1]- Exit 2 sleep 20 && exit 2
[2]+ Exit 1 sleep 10 && exit 1
Ksh:
$ sleep 20 && exit 1 &
$ sleep 10 && exit 2 &
$ wait
[1]+ Done(2) sleep 20 && exit 2
[2]+ Done(1) sleep 10 && exit 1
This output is written to stderr, so a simple solution to the OPs example could be:
#!/bin/bash
trap "rm -f /tmp/x.$$" EXIT
for i in `seq 0 9`; do
doCalculations $i &
done
wait 2> /tmp/x.$$
if [ `wc -l /tmp/x.$$` -gt 0 ] ; then
exit 1
fi
While this:
wait 2> >(wc -l)
will also return a count but without the tmp file. This might also be used this way, for example:
wait 2> >(if [ `wc -l` -gt 0 ] ; then echo "ERROR"; fi)
But this isn't very much more useful than the tmp file IMO. I couldn't find a useful way to avoid the tmp file whilst also avoiding running the "wait" in a subshell, which wont work at all.
I needed this, but the target process wasn't a child of current shell, in which case wait $PID doesn't work. I did find the following alternative instead:
while [ -e /proc/$PID ]; do sleep 0.1 ; done
That relies on the presence of procfs, which may not be available (Mac doesn't provide it for example). So for portability, you could use this instead:
while ps -p $PID >/dev/null ; do sleep 0.1 ; done
There are already a lot of answers here, but I am surprised no one seems to have suggested using arrays... So here's what I did - this might be useful to some in the future.
n=10 # run 10 jobs
c=0
PIDS=()
while true
my_function_or_command &
PID=$!
echo "Launched job as PID=$PID"
PIDS+=($PID)
(( c+=1 ))
# required to prevent any exit due to error
# caused by additional commands run which you
# may add when modifying this example
true
do
if (( c < n ))
then
continue
else
break
fi
done
# collect launched jobs
for pid in "${PIDS[#]}"
do
wait $pid || echo "failed job PID=$pid"
done
This works, should be just as a good if not better than #HoverHell's answer!
#!/usr/bin/env bash
set -m # allow for job control
EXIT_CODE=0; # exit code of overall script
function foo() {
echo "CHLD exit code is $1"
echo "CHLD pid is $2"
echo $(jobs -l)
for job in `jobs -p`; do
echo "PID => ${job}"
wait ${job} || echo "At least one test failed with exit code => $?" ; EXIT_CODE=1
done
}
trap 'foo $? $$' CHLD
DIRN=$(dirname "$0");
commands=(
"{ echo "foo" && exit 4; }"
"{ echo "bar" && exit 3; }"
"{ echo "baz" && exit 5; }"
)
clen=`expr "${#commands[#]}" - 1` # get length of commands - 1
for i in `seq 0 "$clen"`; do
(echo "${commands[$i]}" | bash) & # run the command via bash in subshell
echo "$i ith command has been issued as a background job"
done
# wait for all to finish
wait;
echo "EXIT_CODE => $EXIT_CODE"
exit "$EXIT_CODE"
# end
and of course, I have immortalized this script, in an NPM project which allows you to run bash commands in parallel, useful for testing:
https://github.com/ORESoftware/generic-subshell
Exactly for this purpose I wrote a bash function called :for.
Note: :for not only preserves and returns the exit code of the failing function, but also terminates all parallel running instance. Which might not be needed in this case.
#!/usr/bin/env bash
# Wait for pids to terminate. If one pid exits with
# a non zero exit code, send the TERM signal to all
# processes and retain that exit code
#
# usage:
# :wait 123 32
function :wait(){
local pids=("$#")
[ ${#pids} -eq 0 ] && return $?
trap 'kill -INT "${pids[#]}" &>/dev/null || true; trap - INT' INT
trap 'kill -TERM "${pids[#]}" &>/dev/null || true; trap - RETURN TERM' RETURN TERM
for pid in "${pids[#]}"; do
wait "${pid}" || return $?
done
trap - INT RETURN TERM
}
# Run a function in parallel for each argument.
# Stop all instances if one exits with a non zero
# exit code
#
# usage:
# :for func 1 2 3
#
# env:
# FOR_PARALLEL: Max functions running in parallel
function :for(){
local f="${1}" && shift
local i=0
local pids=()
for arg in "$#"; do
( ${f} "${arg}" ) &
pids+=("$!")
if [ ! -z ${FOR_PARALLEL+x} ]; then
(( i=(i+1)%${FOR_PARALLEL} ))
if (( i==0 )) ;then
:wait "${pids[#]}" || return $?
pids=()
fi
fi
done && [ ${#pids} -eq 0 ] || :wait "${pids[#]}" || return $?
}
usage
for.sh:
#!/usr/bin/env bash
set -e
# import :for from gist: https://gist.github.com/Enteee/c8c11d46a95568be4d331ba58a702b62#file-for
# if you don't like curl imports, source the actual file here.
source <(curl -Ls https://gist.githubusercontent.com/Enteee/c8c11d46a95568be4d331ba58a702b62/raw/)
msg="You should see this three times"
:(){
i="${1}" && shift
echo "${msg}"
sleep 1
if [ "$i" == "1" ]; then sleep 1
elif [ "$i" == "2" ]; then false
elif [ "$i" == "3" ]; then
sleep 3
echo "You should never see this"
fi
} && :for : 1 2 3 || exit $?
echo "You should never see this"
$ ./for.sh; echo $?
You should see this three times
You should see this three times
You should see this three times
1
References
[1]: blog
[2]: gist
set -e
fail () {
touch .failure
}
expect () {
wait
if [ -f .failure ]; then
rm -f .failure
exit 1
fi
}
sleep 2 || fail &
sleep 2 && false || fail &
sleep 2 || fail
expect
The set -e at top makes your script stop on failure.
expect will return 1 if any subjob failed.
There can be a case where the process is complete before waiting for the process. If we trigger wait for a process that is already finished, it will trigger an error like pid is not a child of this shell. To avoid such cases, the following function can be used to find whether the process is complete or not:
isProcessComplete(){
PID=$1
while [ -e /proc/$PID ]
do
echo "Process: $PID is still running"
sleep 5
done
echo "Process $PID has finished"
}
Starting with Bash 5.1, there is a nice new way of waiting for and handling the results of multiple background jobs thanks to the introduction of wait -p:
#!/usr/bin/env bash
# Spawn background jobs
for ((i=0; i < 10; i++)); do
secs=$((RANDOM % 10)); code=$((RANDOM % 256))
(sleep ${secs}; exit ${code}) &
echo "Started background job (pid: $!, sleep: ${secs}, code: ${code})"
done
# Wait for background jobs, print individual results, determine overall result
result=0
while true; do
wait -n -p pid; code=$?
[[ -z "${pid}" ]] && break
echo "Background job ${pid} finished with code ${code}"
(( ${code} != 0 )) && result=1
done
# Return overall result
exit ${result}
I used this recently (thanks to Alnitak):
#!/bin/bash
# activate child monitoring
set -o monitor
# locking subprocess
(while true; do sleep 0.001; done) &
pid=$!
# count, and kill when all done
c=0
function kill_on_count() {
# you could kill on whatever criterion you wish for
# I just counted to simulate bash's wait with no args
[ $c -eq 9 ] && kill $pid
c=$((c+1))
echo -n '.' # async feedback (but you don't know which one)
}
trap "kill_on_count" CHLD
function save_status() {
local i=$1;
local rc=$2;
# do whatever, and here you know which one stopped
# but remember, you're called from a subshell
# so vars have their values at fork time
}
# care must be taken not to spawn more than one child per loop
# e.g don't use `seq 0 9` here!
for i in {0..9}; do
(doCalculations $i; save_status $i $?) &
done
# wait for locking subprocess to be killed
wait $pid
echo
From there one can easily extrapolate, and have a trigger (touch a file, send a signal) and change the counting criteria (count files touched, or whatever) to respond to that trigger. Or if you just want 'any' non zero rc, just kill the lock from save_status.
Trapping CHLD signal may not work because you can lose some signals if they arrived simultaneously.
#!/bin/bash
trap 'rm -f $tmpfile' EXIT
tmpfile=$(mktemp)
doCalculations() {
echo start job $i...
sleep $((RANDOM % 5))
echo ...end job $i
exit $((RANDOM % 10))
}
number_of_jobs=10
for i in $( seq 1 $number_of_jobs )
do
( trap "echo job$i : exit value : \$? >> $tmpfile" EXIT; doCalculations ) &
done
wait
i=0
while read res; do
echo "$res"
let i++
done < "$tmpfile"
echo $i jobs done !!!
solution to wait for several subprocesses and to exit when any one of them exits with non-zero status code is by using 'wait -n'
#!/bin/bash
wait_for_pids()
{
for (( i = 1; i <= $#; i++ )) do
wait -n $#
status=$?
echo "received status: "$status
if [ $status -ne 0 ] && [ $status -ne 127 ]; then
exit 1
fi
done
}
sleep_for_10()
{
sleep 10
exit 10
}
sleep_for_20()
{
sleep 20
}
sleep_for_10 &
pid1=$!
sleep_for_20 &
pid2=$!
wait_for_pids $pid2 $pid1
status code '127' is for non-existing process which means the child might have exited.
I almost fell into the trap of using jobs -p to collect PIDs, which does not work if the child has already exited, as shown in the script below. The solution I picked was simply calling wait -n N times, where N is the number of children I have, which I happen to know deterministically.
#!/usr/bin/env bash
sleeper() {
echo "Sleeper $1"
sleep $2
echo "Exiting $1"
return $3
}
start_sleepers() {
sleeper 1 1 0 &
sleeper 2 2 $1 &
sleeper 3 5 0 &
sleeper 4 6 0 &
sleep 4
}
echo "Using jobs"
start_sleepers 1
pids=( $(jobs -p) )
echo "PIDS: ${pids[*]}"
for pid in "${pids[#]}"; do
wait "$pid"
echo "Exit code $?"
done
echo "Clearing other children"
wait -n; echo "Exit code $?"
wait -n; echo "Exit code $?"
echo "Waiting for N processes"
start_sleepers 2
for ignored in $(seq 1 4); do
wait -n
echo "Exit code $?"
done
Output:
Using jobs
Sleeper 1
Sleeper 2
Sleeper 3
Sleeper 4
Exiting 1
Exiting 2
PIDS: 56496 56497
Exiting 3
Exit code 0
Exiting 4
Exit code 0
Clearing other children
Exit code 0
Exit code 1
Waiting for N processes
Sleeper 1
Sleeper 2
Sleeper 3
Sleeper 4
Exiting 1
Exiting 2
Exit code 0
Exit code 2
Exiting 3
Exit code 0
Exiting 4
Exit code 0
How to wait in a bash script for several subprocesses spawned from that script to finish, and then return exit code !=0 when any of the subprocesses ends with code !=0?
Simple script:
#!/bin/bash
for i in `seq 0 9`; do
doCalculations $i &
done
wait
The above script will wait for all 10 spawned subprocesses, but it will always give exit status 0 (see help wait). How can I modify this script so it will discover exit statuses of spawned subprocesses and return exit code 1 when any of subprocesses ends with code !=0?
Is there any better solution for that than collecting PIDs of the subprocesses, wait for them in order and sum exit statuses?
wait also (optionally) takes the PID of the process to wait for, and with $! you get the PID of the last command launched in the background.
Modify the loop to store the PID of each spawned sub-process into an array, and then loop again waiting on each PID.
# run processes and store pids in array
for i in $n_procs; do
./procs[${i}] &
pids[${i}]=$!
done
# wait for all pids
for pid in ${pids[*]}; do
wait $pid
done
http://jeremy.zawodny.com/blog/archives/010717.html :
#!/bin/bash
FAIL=0
echo "starting"
./sleeper 2 0 &
./sleeper 2 1 &
./sleeper 3 0 &
./sleeper 2 0 &
for job in `jobs -p`
do
echo $job
wait $job || let "FAIL+=1"
done
echo $FAIL
if [ "$FAIL" == "0" ];
then
echo "YAY!"
else
echo "FAIL! ($FAIL)"
fi
Here is simple example using wait.
Run some processes:
$ sleep 10 &
$ sleep 10 &
$ sleep 20 &
$ sleep 20 &
Then wait for them with wait command:
$ wait < <(jobs -p)
Or just wait (without arguments) for all.
This will wait for all jobs in the background are completed.
If the -n option is supplied, waits for the next job to terminate and returns its exit status.
See: help wait and help jobs for syntax.
However the downside is that this will return on only the status of the last ID, so you need to check the status for each subprocess and store it in the variable.
Or make your calculation function to create some file on failure (empty or with fail log), then check of that file if exists, e.g.
$ sleep 20 && true || tee fail &
$ sleep 20 && false || tee fail &
$ wait < <(jobs -p)
$ test -f fail && echo Calculation failed.
How about simply:
#!/bin/bash
pids=""
for i in `seq 0 9`; do
doCalculations $i &
pids="$pids $!"
done
wait $pids
...code continued here ...
Update:
As pointed by multiple commenters, the above waits for all processes to be completed before continuing, but does not exit and fail if one of them fails, it can be made to do with the following modification suggested by #Bryan, #SamBrightman, and others:
#!/bin/bash
pids=""
RESULT=0
for i in `seq 0 9`; do
doCalculations $i &
pids="$pids $!"
done
for pid in $pids; do
wait $pid || let "RESULT=1"
done
if [ "$RESULT" == "1" ];
then
exit 1
fi
...code continued here ...
If you have GNU Parallel installed you can do:
# If doCalculations is a function
export -f doCalculations
seq 0 9 | parallel doCalculations {}
GNU Parallel will give you exit code:
0 - All jobs ran without error.
1-253 - Some of the jobs failed. The exit status gives the number of failed jobs
254 - More than 253 jobs failed.
255 - Other error.
Watch the intro videos to learn more: http://pi.dk/1
Here's what I've come up with so far. I would like to see how to interrupt the sleep command if a child terminates, so that one would not have to tune WAITALL_DELAY to one's usage.
waitall() { # PID...
## Wait for children to exit and indicate whether all exited with 0 status.
local errors=0
while :; do
debug "Processes remaining: $*"
for pid in "$#"; do
shift
if kill -0 "$pid" 2>/dev/null; then
debug "$pid is still alive."
set -- "$#" "$pid"
elif wait "$pid"; then
debug "$pid exited with zero exit status."
else
debug "$pid exited with non-zero exit status."
((++errors))
fi
done
(("$#" > 0)) || break
# TODO: how to interrupt this sleep when a child terminates?
sleep ${WAITALL_DELAY:-1}
done
((errors == 0))
}
debug() { echo "DEBUG: $*" >&2; }
pids=""
for t in 3 5 4; do
sleep "$t" &
pids="$pids $!"
done
waitall $pids
To parallelize this...
for i in $(whatever_list) ; do
do_something $i
done
Translate it to this...
for i in $(whatever_list) ; do echo $i ; done | ## execute in parallel...
(
export -f do_something ## export functions (if needed)
export PATH ## export any variables that are required
xargs -I{} --max-procs 0 bash -c ' ## process in batches...
{
echo "processing {}" ## optional
do_something {}
}'
)
If an error occurs in one process, it won't interrupt the other processes, but it will result in a non-zero exit code from the sequence as a whole.
Exporting functions and variables may or may not be necessary, in any particular case.
You can set --max-procs based on how much parallelism you want (0 means "all at once").
GNU Parallel offers some additional features when used in place of xargs -- but it isn't always installed by default.
The for loop isn't strictly necessary in this example since echo $i is basically just regenerating the output of $(whatever_list). I just think the use of the for keyword makes it a little easier to see what is going on.
Bash string handling can be confusing -- I have found that using single quotes works best for wrapping non-trivial scripts.
You can easily interrupt the entire operation (using ^C or similar), unlike the the more direct approach to Bash parallelism.
Here's a simplified working example...
for i in {0..5} ; do echo $i ; done |xargs -I{} --max-procs 2 bash -c '
{
echo sleep {}
sleep 2s
}'
This is something that I use:
#wait for jobs
for job in `jobs -p`; do wait ${job}; done
This is an expansion on the most-upvoted answer, by #Luca Tettamanti, to make a fully-runnable example.
That answer left me wondering:
What type of variable is n_procs, and what does it contain? What type of variable is procs, and what does it contain? Can someone please update this answer to make it runnable by adding definitions for those variables? I don't understand how.
...and also:
How do you get the return code from the subprocess when it has completed (which is the whole crux of this question)?
Anyway, I figured it out, so here is a fully-runnable example.
Notes:
$! is how to obtain the PID (Process ID) of the last-executed sub-process.
Running any command with the & after it, like cmd &, for example, causes it to run in the background as a parallel suprocess with the main process.
myarray=() is how to create an array in bash.
To learn a tiny bit more about the wait built-in command, see help wait. See also, and especially, the official Bash user manual on Job Control built-ins, such as wait and jobs, here: https://www.gnu.org/software/bash/manual/html_node/Job-Control-Builtins.html#index-wait.
Full, runnable program: wait for all processes to end
multi_process_program.sh (from my eRCaGuy_hello_world repo):
#!/usr/bin/env bash
# This is a special sleep function which returns the number of seconds slept as
# the "error code" or return code" so that we can easily see that we are in
# fact actually obtaining the return code of each process as it finishes.
my_sleep() {
seconds_to_sleep="$1"
sleep "$seconds_to_sleep"
return "$seconds_to_sleep"
}
# Create an array of whatever commands you want to run as subprocesses
procs=() # bash array
procs+=("my_sleep 5")
procs+=("my_sleep 2")
procs+=("my_sleep 3")
procs+=("my_sleep 4")
num_procs=${#procs[#]} # number of processes
echo "num_procs = $num_procs"
# run commands as subprocesses and store pids in an array
pids=() # bash array
for (( i=0; i<"$num_procs"; i++ )); do
echo "cmd = ${procs[$i]}"
${procs[$i]} & # run the cmd as a subprocess
# store pid of last subprocess started; see:
# https://unix.stackexchange.com/a/30371/114401
pids+=("$!")
echo " pid = ${pids[$i]}"
done
# OPTION 1 (comment this option out if using Option 2 below): wait for all pids
for pid in "${pids[#]}"; do
wait "$pid"
return_code="$?"
echo "PID = $pid; return_code = $return_code"
done
echo "All $num_procs processes have ended."
Change the file above to be executable by running chmod +x multi_process_program.sh, then run it like this:
time ./multi_process_program.sh
Sample output. See how the output of the time command in the call shows it took 5.084sec to run. We were also able to successfully retrieve the return code from each subprocess.
eRCaGuy_hello_world/bash$ time ./multi_process_program.sh
num_procs = 4
cmd = my_sleep 5
pid = 21694
cmd = my_sleep 2
pid = 21695
cmd = my_sleep 3
pid = 21697
cmd = my_sleep 4
pid = 21699
PID = 21694; return_code = 5
PID = 21695; return_code = 2
PID = 21697; return_code = 3
PID = 21699; return_code = 4
All 4 processes have ended.
PID 21694 is done; return_code = 5; 3 PIDs remaining.
PID 21695 is done; return_code = 2; 2 PIDs remaining.
PID 21697 is done; return_code = 3; 1 PIDs remaining.
PID 21699 is done; return_code = 4; 0 PIDs remaining.
real 0m5.084s
user 0m0.025s
sys 0m0.061s
Going further: determine live when each individual process ends
If you'd like to do some action as each process finishes, and you don't know when they will finish, you can poll in an infinite while loop to see when each process terminates, then do whatever action you want.
Simply comment out the "OPTION 1" block of code above, and replace it with this "OPTION 2" block instead:
# OR OPTION 2 (comment out Option 1 above if using Option 2): poll to detect
# when each process terminates, and print out when each process finishes!
while true; do
for i in "${!pids[#]}"; do
pid="${pids[$i]}"
# echo "pid = $pid" # debugging
# See if PID is still running; see my answer here:
# https://stackoverflow.com/a/71134379/4561887
ps --pid "$pid" > /dev/null
if [ "$?" -ne 0 ]; then
# PID doesn't exist anymore, meaning it terminated
# 1st, read its return code
wait "$pid"
return_code="$?"
# 2nd, remove this PID from the `pids` array by `unset`ting the
# element at this index; NB: due to how bash arrays work, this does
# NOT actually remove this element from the array. Rather, it
# removes its index from the `"${!pids[#]}"` list of indices,
# adjusts the array count(`"${#pids[#]}"`) accordingly, and it sets
# the value at this index to either a null value of some sort, or
# an empty string (I'm not exactly sure).
unset "pids[$i]"
num_pids="${#pids[#]}"
echo "PID $pid is done; return_code = $return_code;" \
"$num_pids PIDs remaining."
fi
done
# exit the while loop if the `pids` array is empty
if [ "${#pids[#]}" -eq 0 ]; then
break
fi
# Do some small sleep here to keep your polling loop from sucking up
# 100% of one of your CPUs unnecessarily. Sleeping allows other processes
# to run during this time.
sleep 0.1
done
Sample run and output of the full program with Option 1 commented out and Option 2 in-use:
eRCaGuy_hello_world/bash$ ./multi_process_program.sh
num_procs = 4
cmd = my_sleep 5
pid = 22275
cmd = my_sleep 2
pid = 22276
cmd = my_sleep 3
pid = 22277
cmd = my_sleep 4
pid = 22280
PID 22276 is done; return_code = 2; 3 PIDs remaining.
PID 22277 is done; return_code = 3; 2 PIDs remaining.
PID 22280 is done; return_code = 4; 1 PIDs remaining.
PID 22275 is done; return_code = 5; 0 PIDs remaining.
Each of those PID XXXXX is done lines prints out live right after that process has terminated! Notice that even though the process for sleep 5 (PID 22275 in this case) was run first, it finished last, and we successfully detected each process right after it terminated. We also successfully detected each return code, just like in Option 1.
Other References:
*****+ [VERY HELPFUL] Get exit code of a background process - this answer taught me the key principle that (emphasis added):
wait <n> waits until the process with PID is complete (it will block until the process completes, so you might not want to call this until you are sure the process is done), and then returns the exit code of the completed process.
In other words, it helped me know that even after the process is complete, you can still call wait on it to get its return code!
How to check if a process id (PID) exists
my answer
Remove an element from a Bash array - note that elements in a bash array aren't actually deleted, they are just "unset". See my comments in the code above for what that means.
How to use the command-line executable true to make an infinite while loop in bash: https://www.cyberciti.biz/faq/bash-infinite-loop/
I see lots of good examples listed on here, wanted to throw mine in as well.
#! /bin/bash
items="1 2 3 4 5 6"
pids=""
for item in $items; do
sleep $item &
pids+="$! "
done
for pid in $pids; do
wait $pid
if [ $? -eq 0 ]; then
echo "SUCCESS - Job $pid exited with a status of $?"
else
echo "FAILED - Job $pid exited with a status of $?"
fi
done
I use something very similar to start/stop servers/services in parallel and check each exit status. Works great for me. Hope this helps someone out!
I don't believe it's possible with Bash's builtin functionality.
You can get notification when a child exits:
#!/bin/sh
set -o monitor # enable script job control
trap 'echo "child died"' CHLD
However there's no apparent way to get the child's exit status in the signal handler.
Getting that child status is usually the job of the wait family of functions in the lower level POSIX APIs. Unfortunately Bash's support for that is limited - you can wait for one specific child process (and get its exit status) or you can wait for all of them, and always get a 0 result.
What it appears impossible to do is the equivalent of waitpid(-1), which blocks until any child process returns.
The following code will wait for completion of all calculations and return exit status 1 if any of doCalculations fails.
#!/bin/bash
for i in $(seq 0 9); do
(doCalculations $i >&2 & wait %1; echo $?) &
done | grep -qv 0 && exit 1
Here's my version that works for multiple pids, logs warnings if execution takes too long, and stops the subprocesses if execution takes longer than a given value.
[EDIT] I have uploaded my newer implementation of WaitForTaskCompletion, called ExecTasks at https://github.com/deajan/ofunctions.
There's also a compat layer for WaitForTaskCompletion
[/EDIT]
function WaitForTaskCompletion {
local pids="${1}" # pids to wait for, separated by semi-colon
local soft_max_time="${2}" # If execution takes longer than $soft_max_time seconds, will log a warning, unless $soft_max_time equals 0.
local hard_max_time="${3}" # If execution takes longer than $hard_max_time seconds, will stop execution, unless $hard_max_time equals 0.
local caller_name="${4}" # Who called this function
local exit_on_error="${5:-false}" # Should the function exit program on subprocess errors
Logger "${FUNCNAME[0]} called by [$caller_name]."
local soft_alert=0 # Does a soft alert need to be triggered, if yes, send an alert once
local log_ttime=0 # local time instance for comparaison
local seconds_begin=$SECONDS # Seconds since the beginning of the script
local exec_time=0 # Seconds since the beginning of this function
local retval=0 # return value of monitored pid process
local errorcount=0 # Number of pids that finished with errors
local pidCount # number of given pids
IFS=';' read -a pidsArray <<< "$pids"
pidCount=${#pidsArray[#]}
while [ ${#pidsArray[#]} -gt 0 ]; do
newPidsArray=()
for pid in "${pidsArray[#]}"; do
if kill -0 $pid > /dev/null 2>&1; then
newPidsArray+=($pid)
else
wait $pid
result=$?
if [ $result -ne 0 ]; then
errorcount=$((errorcount+1))
Logger "${FUNCNAME[0]} called by [$caller_name] finished monitoring [$pid] with exitcode [$result]."
fi
fi
done
## Log a standby message every hour
exec_time=$(($SECONDS - $seconds_begin))
if [ $((($exec_time + 1) % 3600)) -eq 0 ]; then
if [ $log_ttime -ne $exec_time ]; then
log_ttime=$exec_time
Logger "Current tasks still running with pids [${pidsArray[#]}]."
fi
fi
if [ $exec_time -gt $soft_max_time ]; then
if [ $soft_alert -eq 0 ] && [ $soft_max_time -ne 0 ]; then
Logger "Max soft execution time exceeded for task [$caller_name] with pids [${pidsArray[#]}]."
soft_alert=1
SendAlert
fi
if [ $exec_time -gt $hard_max_time ] && [ $hard_max_time -ne 0 ]; then
Logger "Max hard execution time exceeded for task [$caller_name] with pids [${pidsArray[#]}]. Stopping task execution."
kill -SIGTERM $pid
if [ $? == 0 ]; then
Logger "Task stopped successfully"
else
errrorcount=$((errorcount+1))
fi
fi
fi
pidsArray=("${newPidsArray[#]}")
sleep 1
done
Logger "${FUNCNAME[0]} ended for [$caller_name] using [$pidCount] subprocesses with [$errorcount] errors."
if [ $exit_on_error == true ] && [ $errorcount -gt 0 ]; then
Logger "Stopping execution."
exit 1337
else
return $errorcount
fi
}
# Just a plain stupid logging function to be replaced by yours
function Logger {
local value="${1}"
echo $value
}
Example, wait for all three processes to finish, log a warning if execution takes loger than 5 seconds, stop all processes if execution takes longer than 120 seconds. Don't exit program on failures.
function something {
sleep 10 &
pids="$!"
sleep 12 &
pids="$pids;$!"
sleep 9 &
pids="$pids;$!"
WaitForTaskCompletion $pids 5 120 ${FUNCNAME[0]} false
}
# Launch the function
someting
If you have bash 4.2 or later available the following might be useful to you. It uses associative arrays to store task names and their "code" as well as task names and their pids. I have also built in a simple rate-limiting method which might come handy if your tasks consume a lot of CPU or I/O time and you want to limit the number of concurrent tasks.
The script launches all tasks in the first loop and consumes the results in the second one.
This is a bit overkill for simple cases but it allows for pretty neat stuff. For example one can store error messages for each task in another associative array and print them after everything has settled down.
#! /bin/bash
main () {
local -A pids=()
local -A tasks=([task1]="echo 1"
[task2]="echo 2"
[task3]="echo 3"
[task4]="false"
[task5]="echo 5"
[task6]="false")
local max_concurrent_tasks=2
for key in "${!tasks[#]}"; do
while [ $(jobs 2>&1 | grep -c Running) -ge "$max_concurrent_tasks" ]; do
sleep 1 # gnu sleep allows floating point here...
done
${tasks[$key]} &
pids+=(["$key"]="$!")
done
errors=0
for key in "${!tasks[#]}"; do
pid=${pids[$key]}
local cur_ret=0
if [ -z "$pid" ]; then
echo "No Job ID known for the $key process" # should never happen
cur_ret=1
else
wait $pid
cur_ret=$?
fi
if [ "$cur_ret" -ne 0 ]; then
errors=$(($errors + 1))
echo "$key (${tasks[$key]}) failed."
fi
done
return $errors
}
main
I've had a go at this and combined all the best parts from the other examples here. This script will execute the checkpids function when any background process exits, and output the exit status without resorting to polling.
#!/bin/bash
set -o monitor
sleep 2 &
sleep 4 && exit 1 &
sleep 6 &
pids=`jobs -p`
checkpids() {
for pid in $pids; do
if kill -0 $pid 2>/dev/null; then
echo $pid is still alive.
elif wait $pid; then
echo $pid exited with zero exit status.
else
echo $pid exited with non-zero exit status.
fi
done
echo
}
trap checkpids CHLD
wait
#!/bin/bash
set -m
for i in `seq 0 9`; do
doCalculations $i &
done
while fg; do true; done
set -m allows you to use fg & bg in a script
fg, in addition to putting the last process in the foreground, has the same exit status as the process it foregrounds
while fg will stop looping when any fg exits with a non-zero exit status
unfortunately this won't handle the case when a process in the background exits with a non-zero exit status. (the loop won't terminate immediately. it will wait for the previous processes to complete.)
Wait for all jobs and return the exit code of the last failing job. Unlike solutions above, this does not require pid saving, or modifying inner loops of scripts. Just bg away, and wait.
function wait_ex {
# this waits for all jobs and returns the exit code of the last failing job
ecode=0
while true; do
[ -z "$(jobs)" ] && break
wait -n
err="$?"
[ "$err" != "0" ] && ecode="$err"
done
return $ecode
}
EDIT: Fixed the bug where this could be fooled by a script that ran a command that didn't exist.
Just store the results out of the shell, e.g. in a file.
#!/bin/bash
tmp=/tmp/results
: > $tmp #clean the file
for i in `seq 0 9`; do
(doCalculations $i; echo $i:$?>>$tmp)&
done #iterate
wait #wait until all ready
sort $tmp | grep -v ':0' #... handle as required
I've just been modifying a script to background and parallelise a process.
I did some experimenting (on Solaris with both bash and ksh) and discovered that 'wait' outputs the exit status if it's not zero , or a list of jobs that return non-zero exit when no PID argument is provided. E.g.
Bash:
$ sleep 20 && exit 1 &
$ sleep 10 && exit 2 &
$ wait
[1]- Exit 2 sleep 20 && exit 2
[2]+ Exit 1 sleep 10 && exit 1
Ksh:
$ sleep 20 && exit 1 &
$ sleep 10 && exit 2 &
$ wait
[1]+ Done(2) sleep 20 && exit 2
[2]+ Done(1) sleep 10 && exit 1
This output is written to stderr, so a simple solution to the OPs example could be:
#!/bin/bash
trap "rm -f /tmp/x.$$" EXIT
for i in `seq 0 9`; do
doCalculations $i &
done
wait 2> /tmp/x.$$
if [ `wc -l /tmp/x.$$` -gt 0 ] ; then
exit 1
fi
While this:
wait 2> >(wc -l)
will also return a count but without the tmp file. This might also be used this way, for example:
wait 2> >(if [ `wc -l` -gt 0 ] ; then echo "ERROR"; fi)
But this isn't very much more useful than the tmp file IMO. I couldn't find a useful way to avoid the tmp file whilst also avoiding running the "wait" in a subshell, which wont work at all.
I needed this, but the target process wasn't a child of current shell, in which case wait $PID doesn't work. I did find the following alternative instead:
while [ -e /proc/$PID ]; do sleep 0.1 ; done
That relies on the presence of procfs, which may not be available (Mac doesn't provide it for example). So for portability, you could use this instead:
while ps -p $PID >/dev/null ; do sleep 0.1 ; done
There are already a lot of answers here, but I am surprised no one seems to have suggested using arrays... So here's what I did - this might be useful to some in the future.
n=10 # run 10 jobs
c=0
PIDS=()
while true
my_function_or_command &
PID=$!
echo "Launched job as PID=$PID"
PIDS+=($PID)
(( c+=1 ))
# required to prevent any exit due to error
# caused by additional commands run which you
# may add when modifying this example
true
do
if (( c < n ))
then
continue
else
break
fi
done
# collect launched jobs
for pid in "${PIDS[#]}"
do
wait $pid || echo "failed job PID=$pid"
done
This works, should be just as a good if not better than #HoverHell's answer!
#!/usr/bin/env bash
set -m # allow for job control
EXIT_CODE=0; # exit code of overall script
function foo() {
echo "CHLD exit code is $1"
echo "CHLD pid is $2"
echo $(jobs -l)
for job in `jobs -p`; do
echo "PID => ${job}"
wait ${job} || echo "At least one test failed with exit code => $?" ; EXIT_CODE=1
done
}
trap 'foo $? $$' CHLD
DIRN=$(dirname "$0");
commands=(
"{ echo "foo" && exit 4; }"
"{ echo "bar" && exit 3; }"
"{ echo "baz" && exit 5; }"
)
clen=`expr "${#commands[#]}" - 1` # get length of commands - 1
for i in `seq 0 "$clen"`; do
(echo "${commands[$i]}" | bash) & # run the command via bash in subshell
echo "$i ith command has been issued as a background job"
done
# wait for all to finish
wait;
echo "EXIT_CODE => $EXIT_CODE"
exit "$EXIT_CODE"
# end
and of course, I have immortalized this script, in an NPM project which allows you to run bash commands in parallel, useful for testing:
https://github.com/ORESoftware/generic-subshell
Exactly for this purpose I wrote a bash function called :for.
Note: :for not only preserves and returns the exit code of the failing function, but also terminates all parallel running instance. Which might not be needed in this case.
#!/usr/bin/env bash
# Wait for pids to terminate. If one pid exits with
# a non zero exit code, send the TERM signal to all
# processes and retain that exit code
#
# usage:
# :wait 123 32
function :wait(){
local pids=("$#")
[ ${#pids} -eq 0 ] && return $?
trap 'kill -INT "${pids[#]}" &>/dev/null || true; trap - INT' INT
trap 'kill -TERM "${pids[#]}" &>/dev/null || true; trap - RETURN TERM' RETURN TERM
for pid in "${pids[#]}"; do
wait "${pid}" || return $?
done
trap - INT RETURN TERM
}
# Run a function in parallel for each argument.
# Stop all instances if one exits with a non zero
# exit code
#
# usage:
# :for func 1 2 3
#
# env:
# FOR_PARALLEL: Max functions running in parallel
function :for(){
local f="${1}" && shift
local i=0
local pids=()
for arg in "$#"; do
( ${f} "${arg}" ) &
pids+=("$!")
if [ ! -z ${FOR_PARALLEL+x} ]; then
(( i=(i+1)%${FOR_PARALLEL} ))
if (( i==0 )) ;then
:wait "${pids[#]}" || return $?
pids=()
fi
fi
done && [ ${#pids} -eq 0 ] || :wait "${pids[#]}" || return $?
}
usage
for.sh:
#!/usr/bin/env bash
set -e
# import :for from gist: https://gist.github.com/Enteee/c8c11d46a95568be4d331ba58a702b62#file-for
# if you don't like curl imports, source the actual file here.
source <(curl -Ls https://gist.githubusercontent.com/Enteee/c8c11d46a95568be4d331ba58a702b62/raw/)
msg="You should see this three times"
:(){
i="${1}" && shift
echo "${msg}"
sleep 1
if [ "$i" == "1" ]; then sleep 1
elif [ "$i" == "2" ]; then false
elif [ "$i" == "3" ]; then
sleep 3
echo "You should never see this"
fi
} && :for : 1 2 3 || exit $?
echo "You should never see this"
$ ./for.sh; echo $?
You should see this three times
You should see this three times
You should see this three times
1
References
[1]: blog
[2]: gist
set -e
fail () {
touch .failure
}
expect () {
wait
if [ -f .failure ]; then
rm -f .failure
exit 1
fi
}
sleep 2 || fail &
sleep 2 && false || fail &
sleep 2 || fail
expect
The set -e at top makes your script stop on failure.
expect will return 1 if any subjob failed.
There can be a case where the process is complete before waiting for the process. If we trigger wait for a process that is already finished, it will trigger an error like pid is not a child of this shell. To avoid such cases, the following function can be used to find whether the process is complete or not:
isProcessComplete(){
PID=$1
while [ -e /proc/$PID ]
do
echo "Process: $PID is still running"
sleep 5
done
echo "Process $PID has finished"
}
Starting with Bash 5.1, there is a nice new way of waiting for and handling the results of multiple background jobs thanks to the introduction of wait -p:
#!/usr/bin/env bash
# Spawn background jobs
for ((i=0; i < 10; i++)); do
secs=$((RANDOM % 10)); code=$((RANDOM % 256))
(sleep ${secs}; exit ${code}) &
echo "Started background job (pid: $!, sleep: ${secs}, code: ${code})"
done
# Wait for background jobs, print individual results, determine overall result
result=0
while true; do
wait -n -p pid; code=$?
[[ -z "${pid}" ]] && break
echo "Background job ${pid} finished with code ${code}"
(( ${code} != 0 )) && result=1
done
# Return overall result
exit ${result}
I used this recently (thanks to Alnitak):
#!/bin/bash
# activate child monitoring
set -o monitor
# locking subprocess
(while true; do sleep 0.001; done) &
pid=$!
# count, and kill when all done
c=0
function kill_on_count() {
# you could kill on whatever criterion you wish for
# I just counted to simulate bash's wait with no args
[ $c -eq 9 ] && kill $pid
c=$((c+1))
echo -n '.' # async feedback (but you don't know which one)
}
trap "kill_on_count" CHLD
function save_status() {
local i=$1;
local rc=$2;
# do whatever, and here you know which one stopped
# but remember, you're called from a subshell
# so vars have their values at fork time
}
# care must be taken not to spawn more than one child per loop
# e.g don't use `seq 0 9` here!
for i in {0..9}; do
(doCalculations $i; save_status $i $?) &
done
# wait for locking subprocess to be killed
wait $pid
echo
From there one can easily extrapolate, and have a trigger (touch a file, send a signal) and change the counting criteria (count files touched, or whatever) to respond to that trigger. Or if you just want 'any' non zero rc, just kill the lock from save_status.
Trapping CHLD signal may not work because you can lose some signals if they arrived simultaneously.
#!/bin/bash
trap 'rm -f $tmpfile' EXIT
tmpfile=$(mktemp)
doCalculations() {
echo start job $i...
sleep $((RANDOM % 5))
echo ...end job $i
exit $((RANDOM % 10))
}
number_of_jobs=10
for i in $( seq 1 $number_of_jobs )
do
( trap "echo job$i : exit value : \$? >> $tmpfile" EXIT; doCalculations ) &
done
wait
i=0
while read res; do
echo "$res"
let i++
done < "$tmpfile"
echo $i jobs done !!!
solution to wait for several subprocesses and to exit when any one of them exits with non-zero status code is by using 'wait -n'
#!/bin/bash
wait_for_pids()
{
for (( i = 1; i <= $#; i++ )) do
wait -n $#
status=$?
echo "received status: "$status
if [ $status -ne 0 ] && [ $status -ne 127 ]; then
exit 1
fi
done
}
sleep_for_10()
{
sleep 10
exit 10
}
sleep_for_20()
{
sleep 20
}
sleep_for_10 &
pid1=$!
sleep_for_20 &
pid2=$!
wait_for_pids $pid2 $pid1
status code '127' is for non-existing process which means the child might have exited.
I almost fell into the trap of using jobs -p to collect PIDs, which does not work if the child has already exited, as shown in the script below. The solution I picked was simply calling wait -n N times, where N is the number of children I have, which I happen to know deterministically.
#!/usr/bin/env bash
sleeper() {
echo "Sleeper $1"
sleep $2
echo "Exiting $1"
return $3
}
start_sleepers() {
sleeper 1 1 0 &
sleeper 2 2 $1 &
sleeper 3 5 0 &
sleeper 4 6 0 &
sleep 4
}
echo "Using jobs"
start_sleepers 1
pids=( $(jobs -p) )
echo "PIDS: ${pids[*]}"
for pid in "${pids[#]}"; do
wait "$pid"
echo "Exit code $?"
done
echo "Clearing other children"
wait -n; echo "Exit code $?"
wait -n; echo "Exit code $?"
echo "Waiting for N processes"
start_sleepers 2
for ignored in $(seq 1 4); do
wait -n
echo "Exit code $?"
done
Output:
Using jobs
Sleeper 1
Sleeper 2
Sleeper 3
Sleeper 4
Exiting 1
Exiting 2
PIDS: 56496 56497
Exiting 3
Exit code 0
Exiting 4
Exit code 0
Clearing other children
Exit code 0
Exit code 1
Waiting for N processes
Sleeper 1
Sleeper 2
Sleeper 3
Sleeper 4
Exiting 1
Exiting 2
Exit code 0
Exit code 2
Exiting 3
Exit code 0
Exiting 4
Exit code 0
I have a command CMD called from my main bourne shell script that takes forever.
I want to modify the script as follows:
Run the command CMD in parallel as a background process (CMD &).
In the main script, have a loop to monitor the spawned command every few seconds. The loop also echoes some messages to stdout indicating progress of the script.
Exit the loop when the spawned command terminates.
Capture and report the exit code of the spawned process.
Can someone give me pointers to accomplish this?
1: In bash, $! holds the PID of the last background process that was executed. That will tell you what process to monitor, anyway.
4: wait <n> waits until the process with PID <n> is complete (it will block until the process completes, so you might not want to call this until you are sure the process is done), and then returns the exit code of the completed process.
2, 3: ps or ps | grep " $! " can tell you whether the process is still running. It is up to you how to understand the output and decide how close it is to finishing. (ps | grep isn't idiot-proof. If you have time you can come up with a more robust way to tell whether the process is still running).
Here's a skeleton script:
# simulate a long process that will have an identifiable exit code
(sleep 15 ; /bin/false) &
my_pid=$!
while ps | grep " $my_pid " # might also need | grep -v grep here
do
echo $my_pid is still in the ps output. Must still be running.
sleep 3
done
echo Oh, it looks like the process is done.
wait $my_pid
# The variable $? always holds the exit code of the last command to finish.
# Here it holds the exit code of $my_pid, since wait exits with that code.
my_status=$?
echo The exit status of the process was $my_status
This is how I solved it when I had a similar need:
# Some function that takes a long time to process
longprocess() {
# Sleep up to 14 seconds
sleep $((RANDOM % 15))
# Randomly exit with 0 or 1
exit $((RANDOM % 2))
}
pids=""
# Run five concurrent processes
for i in {1..5}; do
( longprocess ) &
# store PID of process
pids+=" $!"
done
# Wait for all processes to finish, will take max 14s
# as it waits in order of launch, not order of finishing
for p in $pids; do
if wait $p; then
echo "Process $p success"
else
echo "Process $p fail"
fi
done
The pid of a backgrounded child process is stored in $!.
You can store all child processes' pids into an array, e.g. PIDS[].
wait [-n] [jobspec or pid …]
Wait until the child process specified by each process ID pid or job specification jobspec exits and return the exit status of the last command waited for. If a job spec is given, all processes in the job are waited for. If no arguments are given, all currently active child processes are waited for, and the return status is zero. If the -n option is supplied, wait waits for any job to terminate and returns its exit status. If neither jobspec nor pid specifies an active child process of the shell, the return status is 127.
Use wait command you can wait for all child processes finish, meanwhile you can get exit status of each child processes via $? and store status into STATUS[]. Then you can do something depending by status.
I have tried the following 2 solutions and they run well. solution01 is
more concise, while solution02 is a little complicated.
solution01
#!/bin/bash
# start 3 child processes concurrently, and store each pid into array PIDS[].
process=(a.sh b.sh c.sh)
for app in ${process[#]}; do
./${app} &
PIDS+=($!)
done
# wait for all processes to finish, and store each process's exit code into array STATUS[].
for pid in ${PIDS[#]}; do
echo "pid=${pid}"
wait ${pid}
STATUS+=($?)
done
# after all processed finish, check their exit codes in STATUS[].
i=0
for st in ${STATUS[#]}; do
if [[ ${st} -ne 0 ]]; then
echo "$i failed"
else
echo "$i finish"
fi
((i+=1))
done
solution02
#!/bin/bash
# start 3 child processes concurrently, and store each pid into array PIDS[].
i=0
process=(a.sh b.sh c.sh)
for app in ${process[#]}; do
./${app} &
pid=$!
PIDS[$i]=${pid}
((i+=1))
done
# wait for all processes to finish, and store each process's exit code into array STATUS[].
i=0
for pid in ${PIDS[#]}; do
echo "pid=${pid}"
wait ${pid}
STATUS[$i]=$?
((i+=1))
done
# after all processed finish, check their exit codes in STATUS[].
i=0
for st in ${STATUS[#]}; do
if [[ ${st} -ne 0 ]]; then
echo "$i failed"
else
echo "$i finish"
fi
((i+=1))
done
As I see almost all answers use external utilities (mostly ps) to poll the state of the background process. There is a more unixesh solution, catching the SIGCHLD signal. In the signal handler it has to be checked which child process was stopped. It can be done by kill -0 <PID> built-in (universal) or checking the existence of /proc/<PID> directory (Linux specific) or using the jobs built-in (bash specific. jobs -l also reports the pid. In this case the 3rd field of the output can be Stopped|Running|Done|Exit . ).
Here is my example.
The launched process is called loop.sh. It accepts -x or a number as an argument. For -x is exits with exit code 1. For a number it waits num*5 seconds. In every 5 seconds it prints its PID.
The launcher process is called launch.sh:
#!/bin/bash
handle_chld() {
local tmp=()
for((i=0;i<${#pids[#]};++i)); do
if [ ! -d /proc/${pids[i]} ]; then
wait ${pids[i]}
echo "Stopped ${pids[i]}; exit code: $?"
else tmp+=(${pids[i]})
fi
done
pids=(${tmp[#]})
}
set -o monitor
trap "handle_chld" CHLD
# Start background processes
./loop.sh 3 &
pids+=($!)
./loop.sh 2 &
pids+=($!)
./loop.sh -x &
pids+=($!)
# Wait until all background processes are stopped
while [ ${#pids[#]} -gt 0 ]; do echo "WAITING FOR: ${pids[#]}"; sleep 2; done
echo STOPPED
For more explanation see: Starting a process from bash script failed
#/bin/bash
#pgm to monitor
tail -f /var/log/messages >> /tmp/log&
# background cmd pid
pid=$!
# loop to monitor running background cmd
while :
do
ps ax | grep $pid | grep -v grep
ret=$?
if test "$ret" != "0"
then
echo "Monitored pid ended"
break
fi
sleep 5
done
wait $pid
echo $?
I would change your approach slightly. Rather than checking every few seconds if the command is still alive and reporting a message, have another process that reports every few seconds that the command is still running and then kill that process when the command finishes. For example:
#!/bin/sh
cmd() { sleep 5; exit 24; }
cmd & # Run the long running process
pid=$! # Record the pid
# Spawn a process that coninually reports that the command is still running
while echo "$(date): $pid is still running"; do sleep 1; done &
echoer=$!
# Set a trap to kill the reporter when the process finishes
trap 'kill $echoer' 0
# Wait for the process to finish
if wait $pid; then
echo "cmd succeeded"
else
echo "cmd FAILED!! (returned $?)"
fi
Our team had the same need with a remote SSH-executed script which was timing out after 25 minutes of inactivity. Here is a solution with the monitoring loop checking the background process every second, but printing only every 10 minutes to suppress an inactivity timeout.
long_running.sh &
pid=$!
# Wait on a background job completion. Query status every 10 minutes.
declare -i elapsed=0
# `ps -p ${pid}` works on macOS and CentOS. On both OSes `ps ${pid}` works as well.
while ps -p ${pid} >/dev/null; do
sleep 1
if ((++elapsed % 600 == 0)); then
echo "Waiting for the completion of the main script. $((elapsed / 60))m and counting ..."
fi
done
# Return the exit code of the terminated background process. This works in Bash 4.4 despite what Bash docs say:
# "If neither jobspec nor pid specifies an active child process of the shell, the return status is 127."
wait ${pid}
A simple example, similar to the solutions above. This doesn't require monitoring any process output. The next example uses tail to follow output.
$ echo '#!/bin/bash' > tmp.sh
$ echo 'sleep 30; exit 5' >> tmp.sh
$ chmod +x tmp.sh
$ ./tmp.sh &
[1] 7454
$ pid=$!
$ wait $pid
[1]+ Exit 5 ./tmp.sh
$ echo $?
5
Use tail to follow process output and quit when the process is complete.
$ echo '#!/bin/bash' > tmp.sh
$ echo 'i=0; while let "$i < 10"; do sleep 5; echo "$i"; let i=$i+1; done; exit 5;' >> tmp.sh
$ chmod +x tmp.sh
$ ./tmp.sh
0
1
2
^C
$ ./tmp.sh > /tmp/tmp.log 2>&1 &
[1] 7673
$ pid=$!
$ tail -f --pid $pid /tmp/tmp.log
0
1
2
3
4
5
6
7
8
9
[1]+ Exit 5 ./tmp.sh > /tmp/tmp.log 2>&1
$ wait $pid
$ echo $?
5
Another solution is to monitor processes via the proc filesystem (safer than ps/grep combo); when you start a process it has a corresponding folder in /proc/$pid, so the solution could be
#!/bin/bash
....
doSomething &
local pid=$!
while [ -d /proc/$pid ]; do # While directory exists, the process is running
doSomethingElse
....
else # when directory is removed from /proc, process has ended
wait $pid
local exit_status=$?
done
....
Now you can use the $exit_status variable however you like.
With this method, your script doesnt have to wait for the background process, you will only have to monitor a temporary file for the exit status.
FUNCmyCmd() { sleep 3;return 6; };
export retFile=$(mktemp);
FUNCexecAndWait() { FUNCmyCmd;echo $? >$retFile; };
FUNCexecAndWait&
now, your script can do anything else while you just have to keep monitoring the contents of retFile (it can also contain any other information you want like the exit time).
PS.: btw, I coded thinking in bash
My solution was to use an anonymous pipe to pass the status to a monitoring loop. There are no temporary files used to exchange status so nothing to cleanup. If you were uncertain about the number of background jobs the break condition could be [ -z "$(jobs -p)" ].
#!/bin/bash
exec 3<> <(:)
{ sleep 15 ; echo "sleep/exit $?" >&3 ; } &
while read -u 3 -t 1 -r STAT CODE || STAT="timeout" ; do
echo "stat: ${STAT}; code: ${CODE}"
if [ "${STAT}" = "sleep/exit" ] ; then
break
fi
done
how about ...
# run your stuff
unset PID
for process in one two three four
do
( sleep $((RANDOM%20)); echo hello from process $process; exit $((RANDOM%3)); ) & 2>&1
PID+=($!)
done
# (optional) report on the status of that stuff as it exits
for pid in "${PID[#]}"
do
( wait "$pid"; echo "process $pid complemted with exit status $?") &
done
# (optional) while we wait, monitor that stuff
while ps --pid "${PID[*]}" --ppid "${PID[*]}" --format pid,ppid,command,pcpu
do
sleep 5
done | xargs -i date '+%x %X {}'
# return non-zero if any are non zero
SUCCESS=0
for pid in "${PID[#]}"
do
wait "$pid" && ((SUCCESS++)) && echo "$pid OK" || echo "$pid returned $?"
done
echo "success for $SUCCESS out of ${#PID} jobs"
exit $(( ${#PID} - SUCCESS ))
This may be extending beyond your question, however if you're concerned about the length of time processes are running for, you may be interested in checking the status of running background processes after an interval of time. It's easy enough to check which child PIDs are still running using pgrep -P $$, however I came up with the following solution to check the exit status of those PIDs that have already expired:
cmd1() { sleep 5; exit 24; }
cmd2() { sleep 10; exit 0; }
pids=()
cmd1 & pids+=("$!")
cmd2 & pids+=("$!")
lasttimeout=0
for timeout in 2 7 11; do
echo -n "interval-$timeout: "
sleep $((timeout-lasttimeout))
# you can only wait on a pid once
remainingpids=()
for pid in ${pids[*]}; do
if ! ps -p $pid >/dev/null ; then
wait $pid
echo -n "pid-$pid:exited($?); "
else
echo -n "pid-$pid:running; "
remainingpids+=("$pid")
fi
done
pids=( ${remainingpids[*]} )
lasttimeout=$timeout
echo
done
which outputs:
interval-2: pid-28083:running; pid-28084:running;
interval-7: pid-28083:exited(24); pid-28084:running;
interval-11: pid-28084:exited(0);
Note: You could change $pids to a string variable rather than array to simplify things if you like.
I was wondering how, if possible, I can create a simple job management in BASH to process several commands in parallel. That is, I have a big list of commands to run, and I'd like to have two of them running at any given time.
I know quite a bit about bash, so here are the requirements that make it tricky:
The commands have variable running time so I can't just spawn 2, wait, and then continue with the next two. As soon as one command is done a next command must be run.
The controlling process needs to know the exit code of each command so that it can keep a total of how many failed
I'm thinking somehow I can use trap but I don't see an easy way to get the exit value of a child inside the handler.
So, any ideas on how this can be done?
Well, here is some proof of concept code that should probably work, but it breaks bash: invalid command lines generated, hanging, and sometimes a core dump.
# need monitor mode for trap CHLD to work
set -m
# store the PIDs of the children being watched
declare -a child_pids
function child_done
{
echo "Child $1 result = $2"
}
function check_pid
{
# check if running
kill -s 0 $1
if [ $? == 0 ]; then
child_pids=("${child_pids[#]}" "$1")
else
wait $1
ret=$?
child_done $1 $ret
fi
}
# check by copying pids, clearing list and then checking each, check_pid
# will add back to the list if it is still running
function check_done
{
to_check=("${child_pids[#]}")
child_pids=()
for ((i=0;$i<${#to_check};i++)); do
check_pid ${to_check[$i]}
done
}
function run_command
{
"$#" &
pid=$!
# check this pid now (this will add to the child_pids list if still running)
check_pid $pid
}
# run check on all pids anytime some child exits
trap 'check_done' CHLD
# test
for ((tl=0;tl<10;tl++)); do
run_command bash -c "echo FAIL; sleep 1; exit 1;"
run_command bash -c "echo OKAY;"
done
# wait for all children to be done
wait
Note that this isn't what I ultimately want, but would be groundwork to getting what I want.
Followup: I've implemented a system to do this in Python. So anybody using Python for scripting can have the above functionality. Refer to shelljob
GNU Parallel is awesomesauce:
$ parallel -j2 < commands.txt
$ echo $?
It will set the exit status to the number of commands that failed. If you have more than 253 commands, check out --joblog. If you don't know all the commands up front, check out --bg.
Can I persuade you to use make? This has the advantage that you can tell it how many commands to run in parallel (modify the -j number)
echo -e ".PHONY: c1 c2 c3 c4\nall: c1 c2 c3 c4\nc1:\n\tsleep 2; echo c1\nc2:\n\tsleep 2; echo c2\nc3:\n\tsleep 2; echo c3\nc4:\n\tsleep 2; echo c4" | make -f - -j2
Stick it in a Makefile and it will be much more readable
.PHONY: c1 c2 c3 c4
all: c1 c2 c3 c4
c1:
sleep 2; echo c1
c2:
sleep 2; echo c2
c3:
sleep 2; echo c3
c4:
sleep 2; echo c4
Beware, those are not spaces at the beginning of the lines, they're a TAB, so a cut and paste won't work here.
Put an "#" infront of each command if you don't the command echoed. e.g.:
#sleep 2; echo c1
This would stop on the first command that failed. If you need a count of the failures you'd need to engineer that in the makefile somehow. Perhaps something like
command || echo F >> failed
Then check the length of failed.
The problem you have is that you cannot wait for one of multiple background processes to complete. If you observe job status (using jobs) then finished background jobs are removed from the job list. You need another mechanism to determine whether a background job has finished.
The following example uses starts to background processes (sleeps). It then loops using ps to see if they are still running. If not it uses wait to gather the exit code and starts a new background process.
#!/bin/bash
sleep 3 &
pid1=$!
sleep 6 &
pid2=$!
while ( true ) do
running1=`ps -p $pid1 --no-headers | wc -l`
if [ $running1 == 0 ]
then
wait $pid1
echo process 1 finished with exit code $?
sleep 3 &
pid1=$!
else
echo process 1 running
fi
running2=`ps -p $pid2 --no-headers | wc -l`
if [ $running2 == 0 ]
then
wait $pid2
echo process 2 finished with exit code $?
sleep 6 &
pid2=$!
else
echo process 2 running
fi
sleep 1
done
Edit: Using SIGCHLD (without polling):
#!/bin/bash
set -bm
trap 'ChildFinished' SIGCHLD
function ChildFinished() {
running1=`ps -p $pid1 --no-headers | wc -l`
if [ $running1 == 0 ]
then
wait $pid1
echo process 1 finished with exit code $?
sleep 3 &
pid1=$!
else
echo process 1 running
fi
running2=`ps -p $pid2 --no-headers | wc -l`
if [ $running2 == 0 ]
then
wait $pid2
echo process 2 finished with exit code $?
sleep 6 &
pid2=$!
else
echo process 2 running
fi
sleep 1
}
sleep 3 &
pid1=$!
sleep 6 &
pid2=$!
sleep 1000d
I think the following example answers some of your questions, I am looking into the rest of question
(cat list1 list2 list3 | sort | uniq > list123) &
(cat list4 list5 list6 | sort | uniq > list456) &
from:
Running parallel processes in subshells
There is another package for debian systems named xjobs.
You might want to check it out:
http://packages.debian.org/wheezy/xjobs
If you cannot install parallel for some reason this will work in plain shell or bash
# String to detect failure in subprocess
FAIL_STR=failed_cmd
result=$(
(false || echo ${FAIL_STR}1) &
(true || echo ${FAIL_STR}2) &
(false || echo ${FAIL_STR}3)
)
wait
if [[ ${result} == *"$FAIL_STR"* ]]; then
failure=`echo ${result} | grep -E -o "$FAIL_STR[^[:space:]]+"`
echo The following commands failed:
echo "${failure}"
echo See above output of these commands for details.
exit 1
fi
Where true & false are placeholders for your commands. You can also echo $? along with the FAIL_STR to get the command status.
Yet another bash only example for your interest. Of course, prefer the use of GNU parallel, which will offer much more features out of the box.
This solution involve tmp file output creation for collecting of job status.
We use /tmp/${$}_ as temporary file prefix $$ is the actual parent process number and it is the same for all the script execution.
First, the loop for starting parallel job by batch. The batch size is set using max_parrallel_connection. try_connect_DB() is a slow bash function in the same file. Here we collect stdout + stderr 2>&1 for failure diagnostic.
nb_project=$(echo "$projects" | wc -w)
i=0
parrallel_connection=0
max_parrallel_connection=10
for p in $projects
do
i=$((i+1))
parrallel_connection=$((parrallel_connection+1))
try_connect_DB $p "$USERNAME" "$pass" > /tmp/${$}_${p}.out 2>&1 &
if [[ $parrallel_connection -ge $max_parrallel_connection ]]
then
echo -n " ... ($i/$nb_project)"
wait
parrallel_connection=0
fi
done
if [[ $nb_project -gt $max_parrallel_connection ]]
then
# final new line
echo
fi
# wait for all remaining jobs
wait
After run all jobs is finished review all results:
SQL_connection_failed is our convention of error, outputed by try_connect_DB() you may filter job success or failure the way that most suite your need.
Here we decided to only output failed results in order to reduce the amount of output on large sized jobs. Especially if most of them, or all, passed successfully.
# displaying result that failed
file_with_failure=$(grep -l SQL_connection_failed /tmp/${$}_*.out)
if [[ -n $file_with_failure ]]
then
nb_failed=$(wc -l <<< "$file_with_failure")
# we will collect DB name from our output file naming convention, for post treatment
db_names=""
echo "=========== failed connections : $nb_failed/$nb_project"
for failure in $file_with_failure
do
echo "============ $failure"
cat $failure
db_names+=" $(basename $failure | sed -e 's/^[0-9]\+_\([^.]\+\)\.out/\1/')"
done
echo "$db_names"
ret=1
else
echo "all tests passed"
ret=0
fi
# temporary files cleanup, could be kept is case of error, adapt to suit your needs.
rm /tmp/${$}_*.out
exit $ret