Below I have a script that is collecting the process ids of individual commands, and appending them to an array in bash. For some reason as you can see stdout below, the end resulting array just contains one item, the latest id. How can the resulting PROCESS_IDS array at the end of this script contain all four process ids?
PROCESS_IDS=()
function append {
echo $1
PROCESS_IDS=("${PROCESS_IDS[#]}" $1)
}
sleep 1 && echo 'one' & append $! &
sleep 5 && echo 'two' & append $! &
sleep 1 && echo 'three' & append $! &
sleep 5 && echo 'four' & append $!
wait
echo "${PROCESS_IDS[#]}"
Here is the stdout:
83873
83875
83879
83882
three
one
four
two
83882
Don't send the append operation itself to the background. Putting an & after the content you want to background but before the append suffices: The sleep and echo are still backgrounded, but the append is not.
process_ids=( )
append() { process_ids+=( "$1" ); } # POSIX-standard function declaration syntax
{ sleep 1 && echo 'one'; } & append "$!"
{ sleep 5 && echo 'two'; } & append "$!"
{ sleep 1 && echo 'three'; } & append "$!"
{ sleep 5 && echo 'four'; } & append "$!"
echo "Background processes:" # Demonstrate that our array was populated
printf ' - %s\n' "${process_ids[#]}"
wait
My guess is that whenever you send a function call to the background, it has a copy of the global variable on its own, so they're appending the PID to four independent copies of PROCESS_IDS. That's why every function call finds that it is empty and stores a single PID in it.
http://www.gnu.org/software/bash/manual/bashref.html#Lists
If a command is terminated by the control operator ‘&’, the shell executes the command asynchronously in a subshell. This is known as executing the command in the background.
If you want to collect the outputs from all four function calls, let them write to disk and read the output at the end:
function append {
echo $1 | tee /tmp/file.$1
}
sleep 1 && echo 'one' & append $! &
sleep 5 && echo 'two' & append $! &
sleep 1 && echo 'three' & append $! &
sleep 5 && echo 'four' & append $!
wait
cat /tmp/file.*
Edit: this is just a proof of concept - don't do it with file system anyway (as William pointed out, this is going to be error prone unless you take care of uniqueness and synchronization). I only wanted to illustrate that you need to find another way of getting the information out of the subshells.
Related
I would like to run two commands in bash at the same time, wait for the success of both, and then run a third command. In addition, I would like to include this into a time command. Schematically:
time { { cmd_1 & cmd_2 & } && cmd 3; } &> time.log
If, for example, I run:
time { { echo "hi" & sleep 5 & } && echo "ok"; } &> time.log
I get "ok" printed out immediately, without waiting for the success of the commands between curly brackets. Moreover, the time.log file contains the standard error and standard output of the two commands in brackets but the output of time is printed in the terminal only, not in the file.
I am sure I am missing something clear and easy!
Something like this:
time (
( sleep 5; exit 1 ) & pid1=$! # process returns 1 and causes nok
( sleep 6; exit 0 ) & pid2=$!
wait $pid1 && wait $pid2 && echo aok || echo nok # wait for both and act accordingly
) 2> time_out # time outputs to std_err
outputs
nok
as the first bg process returns 1.
wait returns when all child processes have finished, ignoring their exit status. wait -n returns when the next one finishes and returns its exit status.
Since you have two child processes you can chain two wait -ns.
time { echo "hi" & sleep 5 & wait -n && wait -n && echo "ok"; } &> time.log
I have a CI script that I want to speed up by running several things in the background. I want the script wait for all processes and check each one to see if it failed.
Here a a simplification:
#!/bin/bash
set -e
bg()
{
sleep .$[ ( $RANDOM % 10 ) + 1 ]s
}
bg2()
{
sleep .$[ ( $RANDOM % 10 ) + 1 ]s
exit 1
}
bg & # will pass after a random delay
bg2 & # will fail after a random delay
# I want the output of the program to be a failure since bg2 fails
Yes.
You can use the wait command in bash to wait for completion on one or more sub-processes to terminate in which case we provide the PID to wait on it. Also wait can optionally take no arguments in which case it waits for all background process to terminate.
Example:-
#!/bin/bash
sleep 3 &
wait "$!" # Feeding the non-zero process-id as argument to wait command.
# Can also be stored in a variable as pid=$(echo $!)
# Waits until the process 'sleep 3' is completed. Here the wait
# on a single process is done by capturing its process id
echo "I am waking up"
sleep 4 &
sleep 5 &
wait # Without specifying the id, just 'wait' waits until all jobs
# started on the background is complete.
# (or) simply
# wait < <(jobs -p) # To wait on all background jobs started with (job &)
echo "I woke up again"
Update:-
To identify the jobs when the fail, it is best to loop over the list of background jobs and log their exit-code for visibility. Thanks to wonderful suggestion by chepner. It goes like
#!/bin/bash
for p in $(jobs -p)
do
wait "$p" || { echo "job $p failed" >&2; exit; }
done
#!/bin/bash
set -e
bg()
{
sleep .$[ ( $RANDOM % 10 ) + 1 ]s
}
bg2()
{
sleep .$[ ( $RANDOM % 10 ) + 1 ]s
exit 1
}
export -f bg
export -f bg2
parallel ::: bg bg2 || echo $? of the jobs failed
I have multiple scripts that I launch in background.
What I want to do is to launch another script when any of previous has finished execution (theese scripts have unpredictable execution time, so I don't know which of them finish first).
Something like this
exec ./MyScript1.sh &
exec ./MyScript2.sh &
exec ./MyScript3.sh &
#Now wait until any of them finishes and launch ./MyScript4.sh
#What do I have to do?
I've read about wait shell builtin but it waits for all jobs to finish, and I need when only one of them does this.
Any ideas?
Start a long-running process which each of the three background processes will try to kill once they've completed, then wait for that process to die before running MyScript4.sh. This does require that you not use exec to execute MyScript{1,2,3}.sh, so hopefully that is not a hard requirement.
# Assuming that 100,000 seconds is long enough
# for at least one bg job to complete
sleep 100000 & sleep_pid=$!
{ MyScript1.sh; kill $sleep_pid 2>/dev/null; } &
{ MyScript2.sh; kill $sleep_pid 2>/dev/null; } &
{ MyScript3.sh; kill $sleep_pid 2>/dev/null; } &
wait $sleep_pid
MyScript4.sh
A similar option is to use a blocking read on a named pipe. (As presented, this has the additional drawback that only one of the background jobs can finish, as the other two will block trying to write to starter until somebody reads two more lines from it.)
mkfifo starter
{ MyScript1.sh; echo foo > starter; } &
{ MyScript2.sh; echo foo > starter; } &
{ MyScript3.sh; echo foo > starter; } &
read < starter && MyScript4.sh
I would recommend you to start a counter that increment a 1 before the execution of each script and decrement 1 after it finishes. This way you know how many are running simultaneously. Then you just need to watch where this value is higher or lower than a certain threshold (lets say 3) and run an additional process.
let me give an example:
run_scripts.sh:
#!/bin/bash
#allowed simultaneous scripts
num_scripts=3
#initialize counter
script_counter=0
echo $script_counter
echo $script_counter > ./script_counter
# for loop to run the scripts
for i in `seq 1 10`
do
./script1.sh &
sleep 0.1
script_counter=`head -n 1 ./script_counter`
echo $script_counter
# wait until the number of running scripts is lower than 4
while [ $script_counter -gt $num_scripts ]
do
sleep 0.5
script_counter=`head -n 1 ./script_counter`
done
done
script1.sh :
#!/bin/bash
# increment counter
script_counter=`head -n 1 ./script_counter`
script_counter=$(($script_counter + 1))
echo $script_counter > ./script_counter
#echo $script_counter
# your code goes here
rand_numb=$(($RANDOM%10+1))
sleep $rand_numb
######
# decrement counter
script_counter=`head -n 1 ./script_counter`
script_counter=$(($script_counter - 1))
echo $script_counter > ./script_counter
#echo $script_counter
Add these lines at the end of your MyScript1-3.sh
if mkdir /tmp/mylock ; then
./MyScript4.sh
fi
This use /tmp/mylock as a lock to synchronize process.
Only the first process that run mkdir command is success and get into if block.
The others will all fail.
I wrote a script:
Old script:
var="$(sleep 5 && echo "Linux is...")" &
sleep 5
echo $var
New script:
var="$(cat file | grep Succeeded && kilall cat)" & killer1=$!
(sleep 60; kill $killer1) & killer2=$!
fg 1
kill $killer2
echo $var
Cat file works all the time. Should return "... \n Succeeded \n ...". Echo empty always returns. Is there a solution? I want to necessarily result in a variable.
When you terminate a command by &, the shell executes the command in a subshell. var is set in the subshell, not the original process.
If you run it on the background using &, it is in separate process, so you no more share variables. You need to use IPC (interprocess comunication) to assure this. Easiest IPC to use is a pipe:
{ sleep 2 && echo 'Linux is ...' ; } |
{
echo 'doing something here in the meantime...'
sleep 1
read var
echo $var
}
Remove the & from the assignment of var (line 1 in your script).
I was wondering how, if possible, I can create a simple job management in BASH to process several commands in parallel. That is, I have a big list of commands to run, and I'd like to have two of them running at any given time.
I know quite a bit about bash, so here are the requirements that make it tricky:
The commands have variable running time so I can't just spawn 2, wait, and then continue with the next two. As soon as one command is done a next command must be run.
The controlling process needs to know the exit code of each command so that it can keep a total of how many failed
I'm thinking somehow I can use trap but I don't see an easy way to get the exit value of a child inside the handler.
So, any ideas on how this can be done?
Well, here is some proof of concept code that should probably work, but it breaks bash: invalid command lines generated, hanging, and sometimes a core dump.
# need monitor mode for trap CHLD to work
set -m
# store the PIDs of the children being watched
declare -a child_pids
function child_done
{
echo "Child $1 result = $2"
}
function check_pid
{
# check if running
kill -s 0 $1
if [ $? == 0 ]; then
child_pids=("${child_pids[#]}" "$1")
else
wait $1
ret=$?
child_done $1 $ret
fi
}
# check by copying pids, clearing list and then checking each, check_pid
# will add back to the list if it is still running
function check_done
{
to_check=("${child_pids[#]}")
child_pids=()
for ((i=0;$i<${#to_check};i++)); do
check_pid ${to_check[$i]}
done
}
function run_command
{
"$#" &
pid=$!
# check this pid now (this will add to the child_pids list if still running)
check_pid $pid
}
# run check on all pids anytime some child exits
trap 'check_done' CHLD
# test
for ((tl=0;tl<10;tl++)); do
run_command bash -c "echo FAIL; sleep 1; exit 1;"
run_command bash -c "echo OKAY;"
done
# wait for all children to be done
wait
Note that this isn't what I ultimately want, but would be groundwork to getting what I want.
Followup: I've implemented a system to do this in Python. So anybody using Python for scripting can have the above functionality. Refer to shelljob
GNU Parallel is awesomesauce:
$ parallel -j2 < commands.txt
$ echo $?
It will set the exit status to the number of commands that failed. If you have more than 253 commands, check out --joblog. If you don't know all the commands up front, check out --bg.
Can I persuade you to use make? This has the advantage that you can tell it how many commands to run in parallel (modify the -j number)
echo -e ".PHONY: c1 c2 c3 c4\nall: c1 c2 c3 c4\nc1:\n\tsleep 2; echo c1\nc2:\n\tsleep 2; echo c2\nc3:\n\tsleep 2; echo c3\nc4:\n\tsleep 2; echo c4" | make -f - -j2
Stick it in a Makefile and it will be much more readable
.PHONY: c1 c2 c3 c4
all: c1 c2 c3 c4
c1:
sleep 2; echo c1
c2:
sleep 2; echo c2
c3:
sleep 2; echo c3
c4:
sleep 2; echo c4
Beware, those are not spaces at the beginning of the lines, they're a TAB, so a cut and paste won't work here.
Put an "#" infront of each command if you don't the command echoed. e.g.:
#sleep 2; echo c1
This would stop on the first command that failed. If you need a count of the failures you'd need to engineer that in the makefile somehow. Perhaps something like
command || echo F >> failed
Then check the length of failed.
The problem you have is that you cannot wait for one of multiple background processes to complete. If you observe job status (using jobs) then finished background jobs are removed from the job list. You need another mechanism to determine whether a background job has finished.
The following example uses starts to background processes (sleeps). It then loops using ps to see if they are still running. If not it uses wait to gather the exit code and starts a new background process.
#!/bin/bash
sleep 3 &
pid1=$!
sleep 6 &
pid2=$!
while ( true ) do
running1=`ps -p $pid1 --no-headers | wc -l`
if [ $running1 == 0 ]
then
wait $pid1
echo process 1 finished with exit code $?
sleep 3 &
pid1=$!
else
echo process 1 running
fi
running2=`ps -p $pid2 --no-headers | wc -l`
if [ $running2 == 0 ]
then
wait $pid2
echo process 2 finished with exit code $?
sleep 6 &
pid2=$!
else
echo process 2 running
fi
sleep 1
done
Edit: Using SIGCHLD (without polling):
#!/bin/bash
set -bm
trap 'ChildFinished' SIGCHLD
function ChildFinished() {
running1=`ps -p $pid1 --no-headers | wc -l`
if [ $running1 == 0 ]
then
wait $pid1
echo process 1 finished with exit code $?
sleep 3 &
pid1=$!
else
echo process 1 running
fi
running2=`ps -p $pid2 --no-headers | wc -l`
if [ $running2 == 0 ]
then
wait $pid2
echo process 2 finished with exit code $?
sleep 6 &
pid2=$!
else
echo process 2 running
fi
sleep 1
}
sleep 3 &
pid1=$!
sleep 6 &
pid2=$!
sleep 1000d
I think the following example answers some of your questions, I am looking into the rest of question
(cat list1 list2 list3 | sort | uniq > list123) &
(cat list4 list5 list6 | sort | uniq > list456) &
from:
Running parallel processes in subshells
There is another package for debian systems named xjobs.
You might want to check it out:
http://packages.debian.org/wheezy/xjobs
If you cannot install parallel for some reason this will work in plain shell or bash
# String to detect failure in subprocess
FAIL_STR=failed_cmd
result=$(
(false || echo ${FAIL_STR}1) &
(true || echo ${FAIL_STR}2) &
(false || echo ${FAIL_STR}3)
)
wait
if [[ ${result} == *"$FAIL_STR"* ]]; then
failure=`echo ${result} | grep -E -o "$FAIL_STR[^[:space:]]+"`
echo The following commands failed:
echo "${failure}"
echo See above output of these commands for details.
exit 1
fi
Where true & false are placeholders for your commands. You can also echo $? along with the FAIL_STR to get the command status.
Yet another bash only example for your interest. Of course, prefer the use of GNU parallel, which will offer much more features out of the box.
This solution involve tmp file output creation for collecting of job status.
We use /tmp/${$}_ as temporary file prefix $$ is the actual parent process number and it is the same for all the script execution.
First, the loop for starting parallel job by batch. The batch size is set using max_parrallel_connection. try_connect_DB() is a slow bash function in the same file. Here we collect stdout + stderr 2>&1 for failure diagnostic.
nb_project=$(echo "$projects" | wc -w)
i=0
parrallel_connection=0
max_parrallel_connection=10
for p in $projects
do
i=$((i+1))
parrallel_connection=$((parrallel_connection+1))
try_connect_DB $p "$USERNAME" "$pass" > /tmp/${$}_${p}.out 2>&1 &
if [[ $parrallel_connection -ge $max_parrallel_connection ]]
then
echo -n " ... ($i/$nb_project)"
wait
parrallel_connection=0
fi
done
if [[ $nb_project -gt $max_parrallel_connection ]]
then
# final new line
echo
fi
# wait for all remaining jobs
wait
After run all jobs is finished review all results:
SQL_connection_failed is our convention of error, outputed by try_connect_DB() you may filter job success or failure the way that most suite your need.
Here we decided to only output failed results in order to reduce the amount of output on large sized jobs. Especially if most of them, or all, passed successfully.
# displaying result that failed
file_with_failure=$(grep -l SQL_connection_failed /tmp/${$}_*.out)
if [[ -n $file_with_failure ]]
then
nb_failed=$(wc -l <<< "$file_with_failure")
# we will collect DB name from our output file naming convention, for post treatment
db_names=""
echo "=========== failed connections : $nb_failed/$nb_project"
for failure in $file_with_failure
do
echo "============ $failure"
cat $failure
db_names+=" $(basename $failure | sed -e 's/^[0-9]\+_\([^.]\+\)\.out/\1/')"
done
echo "$db_names"
ret=1
else
echo "all tests passed"
ret=0
fi
# temporary files cleanup, could be kept is case of error, adapt to suit your needs.
rm /tmp/${$}_*.out
exit $ret