Trying to capture stdout of background process in bash script - bash

I wrote the following to be used with Jenkins to kick off my selenium tests. Now since they are being executed as background processes, when they fail Jenkins believes they haven't. As you can see my futile attempt on making this work. Any ideas? Thought about piping the output into a file and grepping for keywords. Then I realized I don't know how to reflect an error if a grep for string returns true.
#!/bin/bash
FAIL=0
echo "starting"
ruby /root/selenium-tests/test/test1.rb &
ruby /root/selenium-tests/test/test2.rb &
ruby /root/selenium-tests/test/test3.rb &
#wait
for job in `jobs -p`
do
echo $job
wait $job || let "FAIL+=1"
done
echo $FAIL
if [ "$FAIL" == "0" ];
then
echo "PASS"
else
echo "FAIL! ($FAIL)"
fi

Jenkins most likely looks at the process exit code to determine whether tests fails. This is what all Unix tools do.
There are multiple ways of doing this. If your test files output something like "FAIL" instead of properly returning an exit code, you can do:
#!/bin/bash
(
ruby /root/selenium-tests/test/test1.rb &
ruby /root/selenium-tests/test/test2.rb &
ruby /root/selenium-tests/test/test3.rb &
wait
) > log
! grep "FAIL" log
exit $? # <- happens implicitly at the end of the script, and can be left out
In this case, grep finding "FAIL" will cause the script to fail, and Jenkins to detect the failure.
The more correct way, if your scripts return proper exit codes, is your method but without relying on job control (which by default is turned off in non-interactive shells), and returning correct exit codes:
for test in /root/selenium-tests/test/test*.rb
do
ruby "$test" &
pids+=( $! )
done
for pid in "${pids[#]}"
do
if wait $pid
then
echo "$pid succeeded"
else
echo "$pid failed"
(( failures++ ))
fi
done
if [[ $failures -gt 0 ]]
then
echo "FAIL: $failures failed tests"
exit 1 # return failure
else
echo "PASS!"
exit 0 # return success
fi

Related

How to write process ID and get exit code of the next to last command

I want to run a command, write the process id instantly to a file when the command started and afterwards get the exit status of the command. This means, while the process id has to be written instantly, I want the exit status only when the initial command has finished.
The following statement will unfortunately run the command, write the process id instantly but it won't wait for the command to be finished. Furthermore I will only get the exit status of the echo command, not of the initial command
command in my case is rdiff-backup.
How do I need to modify the statement?
<command> & echo $! > "/pid_file"
RESULT=$?
if [ "$RESULT" -ne "0" ]; then
echo "Finished with errors"
fi
You need to wait on the background process to get its exit status:
_command_for_background_ & echo $! > pid_file
: ... do other things, if any ...
#
# it is better to grab $? on the same line to prevent any
# future modifications inadvertently breaking the strict sequence
#
wait $(< pid_file); child_status=$?
if [[ $child_status != 0 ]]; then
echo "Finished with errors"
fi

Is it logical to use the killall command to exit a script?

I am working around with a pin generator and I have come across a small issue.
I know of a few different methods to exiting a script but I have been playing around with calling the same script that is running as a child process, however when the child process is not called, the script exits perfectly. When called, the parent script does not exit properly after the child has completed and exited and the parent script loops back to the user input. I cannot think of anything other than possibly using the "wait" command though I don't know if this command would be proper with this code. Any thoughts on using the "killall" command to exit the script? I have tested it out, as you may see it in the code below, but I am left with the message, "Terminated" and if I can use killall how would I prevent that message from printing to standard out? Here is my code:
#!/bin/bash
clear
echo ""
echo "Now generating a random pin."
sleep 3
echo ""
echo "----------------------------------------------"
echo ""
# Generates a random 8-digit number
gen_num=$(tr -dc '0-9' </dev/urandom | head -c 8)
echo " Pin = $gen_num "
echo ""
echo "Pin has been generated!"
sleep 3
echo ""
clear
PS3="Would you like to generate another pin?: "
select CHOICE in "YES" "NO"
do
if [ "$CHOICE" == "YES" ]
then
bash "/home/yokai/Modules/Wps-options.sh"
elif [ "$CHOICE" == "NO" ]
then
clear
echo ""
echo "Okay bye!"
sleep 3
clear
killall "Wps-options.sh"
break
exit 0
fi
done
exit 0
You don't need to call the same script recursively (and then kill all its instances). The following script performs the task without forking:
#!/bin/bash
gen_pin () {
echo 'Now generating a random pin.'
# Generates a random 8-digit number
gen_num="$(tr -dc '0-9' </dev/urandom | head -c 8)"
echo "Pin = ${gen_num}"
PS3='Would you like to generate another pin?:'
select CHOICE in 'NO' 'YES'
do
case ${CHOICE} in
'NO')
echo 'OK'
exit 0;;
*)
break;;
esac
done
}
while true
do
gen_pin
done
You can find a lot of information about how to program in bash here.
First of all, when you execut
bash "/home/yokai/Modules/Wps-options.sh"
The script forks and crates a child process, then, it waits for the child termination, and it does not continue with execution, unless, your script Wps-options.sh executes something else in background (forking again) without reaping its child. But i can not tell you more because i dont know what is in your script Wps-options.sh
To prevent messages to be printed to stdout when you execute killall:
killall "Wps-options.sh" 1> /dev/null 2> /dev/null
1> stands for stdout redirection to file /dev/null and 2> stands for stderr redirection to file /dev/null

Stop Bash Script if Hive Fails

I have a bash script that loops through a folder and processes all *.hql files. Sometimes one of the hive script fails (syntax, resource constraint, etc), instead of the script failing it will continue onto the next .hql file.
Anyway I can stop the bash from processing the remaining? Below is my sample bash:
for i in `ls ${layer}/*.hql`; do
echo "Processing $i ..."
hive ${hiveconf_all} -hiveconf DATE=${date} -f ${i} &
if [ $j -le 5 ]; then
j=$(( j+1 ))
else
wait
j=0
fi
done
I would check the process completion state of the previous command and invoke the exit command to come out the loop
(( $? == 0 )) && exit 1
Introduce the above line after the hive command and should do the trick.
add
set -e
to the top of your script
Use this template for running parallel processes and wait for their completion. Add your date, layer, hiveconf_all and other variables:
#!/bin/bash
set -e
#Run parallel processes and write their logs
log_dir=/tmp/my_script_logs
for i in `ls ${layer}/*.hql`; do
echo "Processing $i ..."
#Run hive in parallel and redirect to the log file
hive ${hiveconf_all} -hiveconf DATE=${date} -f ${i} 2>&1 | tee "log_dir/${i}".log &
done
#Now wait for all processes to complete
FAILED=0
for job in `jobs -p`
do
echo "job=$job"
wait $job || let "FAILED+=1"
done
if [ "$FAILED" != "0" ]; then
echo "Execution FAILED! ($FAILED)"
#Do something here, log or send message, etc
exit 1
fi
#All processes are completed successfully!
#Do something here
echo "Done successfully"
Then you will be able to inspect each process log individually.

Bash script: spawning multiple processes issues

So i am writing a script to call a process 365 times and they should run in 10 batches, so this is something i wrote but there are multiple issues -
1. the log message is not getting written to the log file, i see the error message in err file
2. there is this "Command not found" error I keep getting from the script for the line process.
3. even if the command doesnt succeed, still it doesn't print FAIL but prints success
#!/bin/bash
set -m
FAIL=0
for i in {1..10}
do
waitPIDS=()
j=$i
while [ $j -lt 366 ]; do
exec 1>logfile
exec 2>errorfile
`process $j &`
waitPIDS[${#waitPIDS[#]}]=$!
j=$[$j+1]
done
for jpid in "${waitPIDS[#]}"
do
echo $jpid
wait $jpid
if [[ $? != 0 ]] ; then
echo "fail"
else
echo "success"
fi
done
done
What is wrong with it ?
thanks!
At the very least, this line:
`process $j &`
Shouldn't have any backticks in it. You probably just want:
process $j &
Besides that, you're overwriting your log files instead of appending to them; is that intended?

Calling script from script

How can I have my shell script echo to me that the script that it calls has failed?
#!/bin/sh
test="/Applications/test.sh"
sh $test
exit 0
exit 1
#!/bin/sh
if sh /Applications/test.sh; then
echo "Well done $USER"
exit 0
else
echo "script failed with code [$?]" >&2
exit 1
fi
The /Applications/test.sh script should be well coded to exit with conventional status. 0 if it's ok and > 0 if it fails.
Like you can see, no need to test the special variable $?, we use boolean expression directly.
I usually take the following approach:
#!/usr/bin/env bash
test="/Applications/test.sh"
sh "${test}"
exit_status=$?
if [[ ${exit_status} ]] ; then
echo "Error: ${test} failed with status ${exit_status}." >&2
else
echo "Success!"
fi
In terms of best practice, you should not. If a script fails, it should emit an error message before it terminates so that its parent doesn't have to. The main reason for this is that the process knows why it is failing, while the parent can only guess. In other words, you should just write:
#!/bin/sh
test="/Applications/test.sh"
sh $test
Although really, it would be more typical to just write:
#!/bin/sh
/Applications/test.sh
test.sh will emit the necessary error message, and your script will return the same value as did test.sh. Also, in its current form your script will always be successful, even if test.sh actually failed because exit 0; exit 1 is pretty pointless: the exit 1 will never be called.

Resources