I am executing:
Command1 | tee >(grep sth) || Command2
I want Command2 to be executed based on the exit status of grep, while in the current configuration it is being executed based on the result of tee.
As far as I know pipefail and pipestatus are not working here (please correct me if I am wrong).
Modification to the Origian question based on Alexej Answer
I also tried Command1 | tee >(grep sth || Command2), which works for my original question, but as I am trying to set the status of my test in the subshell; ex, Command 1 | tee>(grep sth || Result="PASSED") and later have access to the Result in other chunks of my code. So I still have problem.
Thanks
Change your script to:
Command1 | tee >(grep sth || Command2)
to achieve the desired result.
A word about Subshells
>(....) is a subshell. Anything and everything that you do within that subshell (except for the exit status of said subshell) is completely isolated from the outside world: (a=1); echo $a will never echo the number 1, because a only has meaning within the subshell where it is defined.
I don't entirely understand why, but when you do redirection to a subshell, it seems to invert the exit status of that subshell, so that a failure will return true and a success will return false.
echo 'a' >(grep 'b') && echo false
# false
(exit 1) || echo false
# false
So if my first suggestion isn't working out for you, then try re-writing your script thusly:
Command1 | tee >(grep sth) && Command2
An example
a=1 # `a` now equals `1`
# if I run `exit`, $a will go out of scope and the terminal I'm in might exit
(exit) # $a doesn't go out of scope because `exit` was run from within a subshell.
echo $a # $a still equals `1`
Where you can go to learn more about subshells
Set a parent shell's variable from a subshell
Pass variable from a child to parent in KSH
Variables value gets lost in subshell
http://www.tldp.org/LDP/abs/html/subshells.html
http://mywiki.wooledge.org/SubShell
http://wiki.bash-hackers.org/syntax/expansion/proc_subst
Related
The following script calls another program reading its output in a while loop (see Bash - How to pipe input to while loop and preserve variables after loop ends):
while read -r col0 col1; do
# [...]
done < <(other_program [args ...])
How can I check for the exit code of other_program to see if the loop was executed properly?
Note: ls -d / /nosuch is used as an example command below, because it fails (exit code 1) while still producing stdout output (/) (in addition to stderr output).
Bash v4.2+ solution:
ccarton's helpful answer works well in principle, but by default the while loop runs in a subshell, which means that any variables created or modified in the loop will not be visible to the current shell.
In Bash v4.2+, you can change this by turning the lastpipe option on, which makes the last segment of a pipeline run in the current shell;
as in ccarton's answer, the pipefail option must be set to have $? reflect the exit code of the first failing command in the pipeline:
shopt -s lastpipe # run the last segment of a pipeline in the current shell
shopt -so pipefail # reflect a pipeline's first failing command's exit code in $?
ls -d / /nosuch | while read -r line; do
result=$line
done
echo "result: [$result]; exit code: $?"
The above yields (stderr output omitted):
result: [/]; exit code: 1
As you can see, the $result variable, set in the while loop, is available, and the ls command's (nonzero) exit code is reflected in $?.
Bash v3+ solution:
ikkachu's helpful answer works well and shows advanced techniques, but it is a bit cumbersome.
Here is a simpler alternative:
while read -r line || { ec=$line && break; }; do # Note the `|| { ...; }` part.
result=$line
done < <(ls -d / /nosuch; printf $?) # Note the `; printf $?` part.
echo "result: [$result]; exit code: $ec"
By appending the value of $?, the ls command's exit code, to the output without a trailing \n (printf $?), read reads it in the last loop operation, but indicates failure (exit code 1), which would normally exit the loop.
We can detect this case with ||, and assign the exit code (that was still read into $line) to variable $ec and exit the loop then.
On the off chance that the command's output doesn't have a trailing \n, more work is needed:
while read -r line ||
{ [[ $line =~ ^(.*)/([0-9]+)$ ]] && ec=${BASH_REMATCH[2]} && line=${BASH_REMATCH[1]};
[[ -n $line ]]; }
do
result=$line
done < <(printf 'no trailing newline'; ls /nosuch; printf "/$?")
echo "result: [$result]; exit code: $ec"
The above yields (stderr output omitted):
result: [no trailing newline]; exit code: 1
At least one way would be to redirect the output of the background process through a named pipe. This would allow to pick up its PID and then get the exit status through waiting on the PID.
#!/bin/bash
mkfifo pipe || exit 1
(echo foo ; exit 19) > pipe &
pid=$!
while read x ; do echo "read: $x" ; done < pipe
wait $pid
echo "exit status of bg process: $?"
rm pipe
If you can use a direct pipe (i.e. don't mind the loop being run in a subshell), you could use Bash's PIPESTATUS, which contains the exit codes of all commands in the pipeline:
(echo foo ; exit 19) | while read x ; do
echo "read: $x" ; done;
echo "status: ${PIPESTATUS[0]}"
A simple way is to use the bash pipefail option to propagate the first error code from a pipeline.
set -o pipefail
other_program | while read x; do
echo "Read: $x"
done || echo "Error: $?"
Another way is to use coproc (requires 4.0+).
coproc other_program [args ...]
while read -r -u ${COPROC[0]} col0 col1; do
# [...]
done
wait $COPROC_PID || echo "Error exit status: $?"
coproc frees you from having to setup asynchronicity and stdin/stdout redirection that you'd otherwise need to do in an equivalent mkfifo.
Let's imagine we have this code (script.sh):
#!/bin/bash
set -e
f() {
echo "[f] Start" >&2
echo "f:before-false1"
echo "f:before-false2"
false
echo "f:after-false"
echo "[f] Fail! I don't want this executed" >&2
}
out=$(f)
The output:
$ bash myscript.sh
[f] Start
[f] Fail! I don't want this executed
I understand that $(...) starts a sub-shell where set -e is not propagated, so my question is: what's the idiomatic way to make this run as expected without too much clutter? I can see 3 solutions, none of which I like (nor I am actually sure they indeed work): 1) Add set -e to the start of f (and every other function in the app). 2) Run $(set -e && f). 3) Add ... || return 1 to every command that may fail.
It's not the prettiest solution, but it does allow you to emulate set -e for the current shell as well as any functions and subshells:
#!/bin/bash
# Set up an ERR trap that unconditionally exits with a nonzero exit code.
# Similar to -e, this trap is invoked when a command reports a nonzero
# exit code (outside of a conditional / test).
# Note: This trap may be called *multiple* times.
trap 'exit 1' ERR
# Now ensure that the ERR trap is called not only by the current shell,
# but by any subshells too:
# set -E (set -o errtrace) causes functions and subshells to inherit
# ERR traps.
set -E
f() {
echo "[f] Start" >&2
echo "f:before-false1"
echo "f:before-false2"
false
echo "f:after-false"
echo "[f] Fail! I don't want this executed" >&2
}
out=$(f)
Output (to stderr) if you call this script (exit code afterward will be 1) - note how the 2nd echo to stderr (>&2) is not printed, proving that the execution of false aborted the command substitution:
[f] Start
Note:
By design, set -e / trap ERR only respond to failures that aren't part of conditionals (see man bash, under the description of set (search for literal " set ["), for the exact rules, which changed slightly between Bash 3.x and 4.x).
Thus, for instance, f does NOT trigger the trap in the following commands: if ! f; then ..., f && echo ok; the following triggers the trap in the subshell (command substitution $(...), but not in the enclosing conditional ([[ ... ]]): [[ $(f) == 'foo' ]] && echo ok, so the script as a whole doesn't abort.
To exit a function / subshell explicitly in such cases, use something like || return 1 / || exit 1, or call the function / subshell separately, outside of a conditional first; e.g., in the [[ $(f) == 'foo' ]] case: res=$(f); [[ $res == 'foo' ]] - res=$(f) will then trigger the trap for the current shell too.
As for why the trap code may be invoked multiple times: In the case at hand, false inside f() first triggers the trap, and then, because the trap's exit 1 exits the subshell ($(f)), the trap is triggered again for the current shell (the one running the script).
Instead of command substitution, you should use process substitution to call your function so that set -e remains in effect:
mapfile arr < <(f) # call function f using process substitution
out="${arr[*]}" # convert array content into a string
declare -p out # check output
Output:
[f] Start
declare -- out="f:before-false1
f:before-false2
"
Consider I have following commandline: do-things arg1 arg2 | progress-meter "Doing things...";, where progress-meter is bash function I want to implement. It should print Doing things... before running do-things arg1 arg2 or in parallel (so, it will be printed anyway at the very beginning), and record stdout+stderr of do-things command, and check it's exit status. If exit status is 0, it should print [ OK ], otherwise it should print [FAIL] and dump recorded output.
Currently I have things done using progress-meter "Doing things..." "do-things arg1 arg2";, and evaluating second argument inside, which is clumsy and I don't like that and believe there is better solution.
The problem with pipe syntax is that I don't know how can I get do-things' exit status from inside the pipeline? $PIPESTATUS seems to be useful only after all commands in pipeline finished.
Maybe process substitution like progress-meter "Doing things..." <(do-things arg1 arg2); will be fine, but in this case I also don't know how can I get exit status of do-things.
I'll be happy to hear if there is some other neat syntax possible to achieve same task without escaping command to be executed like in my example.
I greatly hope for the help of community.
UPD1: As question seems not to be clear enough, I paraphrase it:
I want bash function that can be fed with command, that will execute in parallel to function, and bash function will receive it's stdout+stderr, wait for completion and get its exit status.
Example implementation using evals:
progress_meter() {
local output;
local errcode;
echo -n -e $1;
output=$( {
eval "${cmd}";
} 2>&1; );
errcode=$?;
if (( errcode )); then {
echo '[FAIL]';
echo "Output was: ${output}"
} else {
echo '[ OK ]';
}; fi;
}
So this can be used as progress_meter "Do things..." "do-things arg1 arg2". I want the same without eval.
Why eval things? Assuming you have one fixed argument to progress-meter, you can do something like:
#!/bin/bash
# progress meter
prompt="$1"
shift
echo "$prompt"
"$#" # this just executes a command made up of
# arguments 2, 3, ... of the script
# the real script should actually read its input,
# display progress meter etc.
and call it
$ progress-meter "Doing stuff" do-things arg1 arg2
If you insist on putting progress-meter in a pipeline, I'm afraid your best bet is something like
(do-things arg1 arg2 ; echo $?) | progress-meter "Doing stuff"
I'm not sure I understand what exactly you're trying to achieve,
but you could check the pipefail option:
pipefail
If set, the return value of a pipeline is the
value of the last (rightmost) command to exit
with a non-zero status, or zero if all commands
in the pipeline exit successfully. This option
is disabled by default.
For example:
bash-4.1 $ ls no_such_a_file 2>&- | : && echo ok: $? || echo ko: $?
ok: 0
bash-4.1 $ set -o pipefail
bash-4.1 $ ls no_such_a_file 2>&- | : && echo ok: $? || echo ko: $?
ko: 2
Edit: I just read your comment on the other post. Why don't you just handle the error?
bash-4.1 $ ls -d /tmp 2>&- || echo failed | while read; do [[ $REPLY == failed ]] && echo failed || echo "$REPLY"; done
/tmp
bash-4.1 $ ls -d /tmpp 2>&- || echo failed | while read; do [[ $REPLY == failed ]] && echo failed || echo "$REPLY"; done
failed
Have your scrips in the pipeline communicate by proxy (much like the Blackboard Pattern: some guy writes on the blackboard, another guy reads it):
Modify your do-things script so that it reports its exit status to a file somewhere.
Modify your progress-meter script to read that file, using command line switches if you like so as not to hardcode the name of the blackboard file, for reporting the exit status of the program that it is reporting the progress for.
I'd like to send the result of a series of commands to a variable:
variable=$(a | few | commands)
However, the command substitution resets PIPESTATUS, so I can't inspect where it went wrong after the fact. One solution would be to use mktemp and put the result there temporarily:
variable_file=$(mktemp) || exit 1
a | few | commands > $variable_file
exit_codes="${PIPESTATUS[*]}"
variable=$(<$variable_file)
Is there a more elegant solution?
Kinda hacky but I think you could fudge it like this.
variable=$(a | few | commands; echo ": ${PIPESTATUS[*]}")
PIPESTATUS=(${variable##*: })
variable=${variable%:*}
variable=${variable%$'\n'}
Building on ephemient's answer, if we need the output of the piped commands stored without them being mixed in with the pipestatus codes, but we don't really care what the exit codes themselves are (just that they all succeeded), we can do:
variable=$(a | few | commands; [[ ${PIPESTATUS[*]} == "0 0 0" ]])
This will check on the status of all the piped command status in the above example and if its exit code is not 0, will set $? to 1 (false)
If you want to exit with a different code instead of 1, you could capture the contents of PIPESTATUS[#], e.g. r_code=${PIPESTATUS[2]}, and then use (exit ${r_code[2]}) instead of false.
Below captures all the codes of PIPESTATUS, ensures they're all 0, and if not, sets the exit code to be the $? value of commands:
declare -a r_code
variable=$(a | few | commands
r_code=(${PIPESTATUS[#]})
[[ ${r_code[*]} == "0 0 0" ]] || (exit ${r_code[2]})
)
echo ${?} # echoes the exit code of `commands`
echo ${variable} # echoes only the output of a | few | commands
This question already has answers here:
Aborting a shell script if any command returns a non-zero value
(10 answers)
Closed 1 year ago.
I have a shell script that executes a number of commands. How do I make the shell script exit if any of the commands exit with a non-zero exit code?
After each command, the exit code can be found in the $? variable so you would have something like:
ls -al file.ext
rc=$?; if [[ $rc != 0 ]]; then exit $rc; fi
You need to be careful of piped commands since the $? only gives you the return code of the last element in the pipe so, in the code:
ls -al file.ext | sed 's/^/xx: /"
will not return an error code if the file doesn't exist (since the sed part of the pipeline actually works, returning 0).
The bash shell actually provides an array which can assist in that case, that being PIPESTATUS. This array has one element for each of the pipeline components, that you can access individually like ${PIPESTATUS[0]}:
pax> false | true ; echo ${PIPESTATUS[0]}
1
Note that this is getting you the result of the false command, not the entire pipeline. You can also get the entire list to process as you see fit:
pax> false | true | false; echo ${PIPESTATUS[*]}
1 0 1
If you wanted to get the largest error code from a pipeline, you could use something like:
true | true | false | true | false
rcs=${PIPESTATUS[*]}; rc=0; for i in ${rcs}; do rc=$(($i > $rc ? $i : $rc)); done
echo $rc
This goes through each of the PIPESTATUS elements in turn, storing it in rc if it was greater than the previous rc value.
If you want to work with $?, you'll need to check it after each command, since $? is updated after each command exits. This means that if you execute a pipeline, you'll only get the exit code of the last process in the pipeline.
Another approach is to do this:
set -e
set -o pipefail
If you put this at the top of the shell script, it looks like Bash will take care of this for you. As a previous poster noted, "set -e" will cause Bash to exit with an error on any simple command. "set -o pipefail" will cause Bash to exit with an error on any command in a pipeline as well.
See here or here for a little more discussion on this problem. Here is the Bash manual section on the set builtin.
"set -e" is probably the easiest way to do this. Just put that before any commands in your program.
If you just call exit in Bash without any parameters, it will return the exit code of the last command. Combined with OR, Bash should only invoke exit, if the previous command fails. But I haven't tested this.
command1 || exit;
command2 || exit;
Bash will also store the exit code of the last command in the variable $?.
[ $? -eq 0 ] || exit $?; # Exit for nonzero return code
http://cfaj.freeshell.org/shell/cus-faq-2.html#11
How do I get the exit code of cmd1 in cmd1|cmd2
First, note that cmd1 exit code could be non-zero and still don't mean an error. This happens for instance in
cmd | head -1
You might observe a 141 (or 269 with ksh93) exit status of cmd1, but it's because cmd was interrupted by a SIGPIPE signal when head -1 terminated after having read one line.
To know the exit status of the elements of a pipeline
cmd1 | cmd2 | cmd3
a. with Z shell (zsh):
The exit codes are provided in the pipestatus special array.
cmd1 exit code is in $pipestatus[1], cmd3 exit code in
$pipestatus[3], so that $? is always the same as
$pipestatus[-1].
b. with Bash:
The exit codes are provided in the PIPESTATUS special array.
cmd1 exit code is in ${PIPESTATUS[0]}, cmd3 exit code in
${PIPESTATUS[2]}, so that $? is always the same as
${PIPESTATUS: -1}.
...
For more details see Z shell.
For Bash:
# This will trap any errors or commands with non-zero exit status
# by calling function catch_errors()
trap catch_errors ERR;
#
# ... the rest of the script goes here
#
function catch_errors() {
# Do whatever on errors
#
#
echo "script aborted, because of errors";
exit 0;
}
In Bash this is easy. Just tie them together with &&:
command1 && command2 && command3
You can also use the nested if construct:
if command1
then
if command2
then
do_something
else
exit
fi
else
exit
fi
#
#------------------------------------------------------------------------------
# purpose: to run a command, log cmd output, exit on error
# usage:
# set -e; do_run_cmd_or_exit "$cmd" ; set +e
#------------------------------------------------------------------------------
do_run_cmd_or_exit(){
cmd="$#" ;
do_log "DEBUG running cmd or exit: \"$cmd\""
msg=$($cmd 2>&1)
export exit_code=$?
# If occurred during the execution, exit with error
error_msg="Failed to run the command:
\"$cmd\" with the output:
\"$msg\" !!!"
if [ $exit_code -ne 0 ] ; then
do_log "ERROR $msg"
do_log "FATAL $msg"
do_exit "$exit_code" "$error_msg"
else
# If no errors occurred, just log the message
do_log "DEBUG : cmdoutput : \"$msg\""
fi
}