I'd like to send the result of a series of commands to a variable:
variable=$(a | few | commands)
However, the command substitution resets PIPESTATUS, so I can't inspect where it went wrong after the fact. One solution would be to use mktemp and put the result there temporarily:
variable_file=$(mktemp) || exit 1
a | few | commands > $variable_file
exit_codes="${PIPESTATUS[*]}"
variable=$(<$variable_file)
Is there a more elegant solution?
Kinda hacky but I think you could fudge it like this.
variable=$(a | few | commands; echo ": ${PIPESTATUS[*]}")
PIPESTATUS=(${variable##*: })
variable=${variable%:*}
variable=${variable%$'\n'}
Building on ephemient's answer, if we need the output of the piped commands stored without them being mixed in with the pipestatus codes, but we don't really care what the exit codes themselves are (just that they all succeeded), we can do:
variable=$(a | few | commands; [[ ${PIPESTATUS[*]} == "0 0 0" ]])
This will check on the status of all the piped command status in the above example and if its exit code is not 0, will set $? to 1 (false)
If you want to exit with a different code instead of 1, you could capture the contents of PIPESTATUS[#], e.g. r_code=${PIPESTATUS[2]}, and then use (exit ${r_code[2]}) instead of false.
Below captures all the codes of PIPESTATUS, ensures they're all 0, and if not, sets the exit code to be the $? value of commands:
declare -a r_code
variable=$(a | few | commands
r_code=(${PIPESTATUS[#]})
[[ ${r_code[*]} == "0 0 0" ]] || (exit ${r_code[2]})
)
echo ${?} # echoes the exit code of `commands`
echo ${variable} # echoes only the output of a | few | commands
Related
I am interested on which is better in the sense of either being more flexible/general/extensible or faster or less error prone or uses less memory.
Snippet1:
grep '^+[^+]' /tmp/p0rn.lst | while read x;do
wget $x
done
Snippet2:
while read x;do
wget $x
done < <(grep '^+[^+]' /tmp/p0rn.lst)
The first uses a pipeline, the second process substitution. Process substitution is usually better.
Piping input to a loop means the loop runs in a subshell. Variable changes inside a subshell are lost when the subshell finishes, i.e. when the loop ends.
For example, if you wanted to count the number of URLs processed by incrementing a variable each iteration you'd have to use process substitution:
count=0
while read x; do
wget "$x"
((++count))
done < <(grep '^+[^+]' /tmp/p0rn.lst)
echo "processed $count urls"
If you used a pipeline it would always print "processed 0 urls":
count=0
grep '^+[^+]' /tmp/p0rn.lst | while read x; do
wget "$x"
((++count))
done
# Doesn't work!
echo "processed $count urls"
In addition to #john-kugelman 's answer I would like to point out that the bash holds a variable called PIPESTATUS which is an array of all the exit codes of the last pipe:
(exit 1) | (exit 2) | (exit 3) | (exit 4); echo "${PIPESTATUS[#]}"
This will print
1 2 3 4
There is no similar mechanism if this is rewritten using process substitutions:
(exit 4) < <(
(exit 3) < <(
(exit 2) < <(
(exit 1)
)
)
)
In this case, the inner exit codes are just lost, unless they are explicitly stored somehow (e. g. dumped in a file or similar).
The overall exit code of a pipe is the exit code of the last pipe element, on default, so true | false is like false. In many cases this is desirable, e. g.
if curl "$url" | grep -q "searchterm"
then
# do something in case the searchterm was found
In this usage the exit value of the grep clearly is what is wanted.
There are other uses, however, in which you would rather react on the exit value of the left element:
curl "$url" | sed 's/password.*/###/g' > x || { # incorrect usage!
echo "Curl failed." 1>&2
}
This will not work because the pipe's exit value is the exit value of sed (which in this case is not the relevant part). Moving the check deeper inside can solve the issue:
(
curl "$url" || echo "Curl failed." 1>&2
) | sed 's/password.*/###/g' > x
Or also:
sed 's/password.*/###/g' < <(
curl "$url" || echo "Curl failed." 1>&2
) > x
But neither allows e. g. exiting the surrounding shell properly (because the check is done in a nested scope).
You can, however, use the pipefail option of the shell in which case a pipe's exit value is the exit value of the rightmost non-zero exit value:
set -o pipefail
(exit 1) | (exit 2) | (exit 3) | (exit 0); echo $?
This will print 3 instead of 0. Applied to our last example:
set -o pipefail
if ! curl "$url" | sed 's/password.*/###/g' > x
then
echo "Curl failed (or sed)." 1>&2
# Now we can even exit this shell here if we like.
fi
(You might want to scope the pipefail option to avoid influencing the rest of the script.)
You can, however, achieve a similar thing using process substitutions for the output:
if ! curl "$url" > >(sed 's/password.*/###/g' > x)
then
echo "Curl failed." 1>&2
# Now we can even exit this shell here if we like.
fi
But this will ignore failures in the right commands of the pipe (sed in this case). Using pipefail you can react on failures in any of the piped commands. There is no (simple and recommendable) way of achieving this using process substitutions.
I'm trying to assign the exit command of the last command in a pipeline to a variable but it is not giving the expected result. Essentially I'm grepping a variable to see if it ends with '-SNAPSHOT' or not. So if I try this:
export PROJECT_VERSION=1.0.0
echo ${PROJECT_VERSION} | grep \\-SNAPSHOT$
And then do echo $? the result is 1 as expected (no match found).
If I then add the echo $? to the end of the pipe:
echo ${PROJECT_VERSION} | grep \\-SNAPSHOT$ | echo $?
The result then becomes 0.
How can I get the exit result of the grep \\-SNAPSHOT so that I can assign it to a variable?
The exit status is in $?.
echo ${PROJECT_VERSION} | grep \\-SNAPSHOT$
variable="$?"
I would like to be able to create a bash function that can read the exit code of the command before the pipe. I'm not sure it is possible to have access to that.
echo "1" | grep 2 returns a 1 status code
echo "1" | grep 1 returns a 0 status code
Now I would like to add a third command to read the status, with a pipe:
echo "1" | grep 2 | echo $? will echo "0", even if the status code is 1.
I know I can use the echo "1" | grep 2 && echo "0" || echo "1", but I would prefer to write it using a pipe.
Is they anyway to do that (it would be even better if it was working on most shells, like bash, sh, and zsh)
You're going to have to get the exit status before the next stage of the pipeline. Something like
exec 3> debug.txt
{ echo "1"; echo "$?" >&3; } | long | command | here
You can't (easily) encapsulate this in a function, since it would require passing a properly quoted string and executing it via eval:
debug () {
eval "$#"
echo $? >&3
}
# It looks easy in this example, but it won't take long to find
# an example that breaks it.
debug echo 1 | long | command | here
You have to write the exit status to a different file descriptor, otherwise it will interfere with the output sent to the next command in the pipeline.
In bash you can do this with the PIPESTATUS variable
echo "1" | grep 1
echo ${PIPESTATUS[0]} # returns 0
echo "1" | grep 2
echo ${PIPESTATUS[0]} # returns 0
echo "1" | grep 2
echo ${PIPESTATUS[1]} # returns 1
First, hello to stackoverflow community. I have learnt a lot thank to very clear questions and professional replies. I never needed to ask question because there is always someone who has asked the same question before.
But not today. I haven't found any solution to my problem, and I beg for your help.
I need to process the output of a function line by line in order to update a log online. I am working with bash.
The following block works pretty well:
convertor some parameters | while read line
do
if [ "${line:0:14}" != "[informations]" ]
then
update_online_log "${line}"
fi
done
But convertor may exit with different status. And I need to know what was the exit status. The code below doesn't work as it gives me the exit status of the last executed command (update_online_log).
convertor some parameters | while read line
do
if [ "${line:0:14}" != "[informations]" ]
then
update_online_log "${line}"
fi
done
exit_status=$?
The code below should work (I haven't tried it yet):
convertor some parameters > out.txt
exit_status=$?
while read line
do
if [ "${line:0:14}" != "[informations]" ]
then
update_online_log "${line}"
fi
done < out.txt
rm out.txt
But if I use this, the online log will be updated at the end of the conversion. Conversion may be a very long process, and I want to keep users updated while the conversion is in progress.
Thank you in advance for your help.
The PIPESTATUS array may be helpful for you: it saves the exit statuses of each component of the previous pipeline:
$ (echo a; exit 42) | (cat; echo b; exit 21) | (cat; echo c; exit 3) | { cat; echo hello; }
a
b
c
hello
$ echo "${PIPESTATUS[*]}"
42 21 3 0
That array is pretty fragile, so if you want to do stuff with it, immediately save it to another array:
$ (echo a; exit 42) | ... as above
$ ps=( "${PIPESTATUS[#]}" )
$ for i in "${!ps[#]}"; do echo "$i ${ps[$i]}"; done
0 42
1 21
2 3
3 0
Add set -o pipefail at the top of the script. From the documentation:
the return value of a pipeline is the status of
the last command to exit with a non-zero status,
or zero if no command exited with a non-zero status
That means it's zero if all commands succeed and non-zero if any command fails.
Just check the exit status after the done, that should work. Test it with
convertor some parameters | false | while read line
...
No line should be processed and the exit code should be 1.
As a POSIX-compliant solution, and to show that your approach was close to working, you can use a named pipe instead of a regular file.
mkfifo out.txt
while read line; do
if [[ $line != \[informations\]* ]]; then
update_online_log "$line"
fi
done < out.txt &
convertor some parameters > out.txt
rm out.txt
This code creates the named pipe, then runs the loop that consumes it in the background. It will block waiting for data from converter. Once converter exits (and you can get its exit code at that time), out.txt will be closed, the while loop process will exit, and you can remove the named pipe.
This question already has answers here:
Aborting a shell script if any command returns a non-zero value
(10 answers)
Closed 1 year ago.
I have a shell script that executes a number of commands. How do I make the shell script exit if any of the commands exit with a non-zero exit code?
After each command, the exit code can be found in the $? variable so you would have something like:
ls -al file.ext
rc=$?; if [[ $rc != 0 ]]; then exit $rc; fi
You need to be careful of piped commands since the $? only gives you the return code of the last element in the pipe so, in the code:
ls -al file.ext | sed 's/^/xx: /"
will not return an error code if the file doesn't exist (since the sed part of the pipeline actually works, returning 0).
The bash shell actually provides an array which can assist in that case, that being PIPESTATUS. This array has one element for each of the pipeline components, that you can access individually like ${PIPESTATUS[0]}:
pax> false | true ; echo ${PIPESTATUS[0]}
1
Note that this is getting you the result of the false command, not the entire pipeline. You can also get the entire list to process as you see fit:
pax> false | true | false; echo ${PIPESTATUS[*]}
1 0 1
If you wanted to get the largest error code from a pipeline, you could use something like:
true | true | false | true | false
rcs=${PIPESTATUS[*]}; rc=0; for i in ${rcs}; do rc=$(($i > $rc ? $i : $rc)); done
echo $rc
This goes through each of the PIPESTATUS elements in turn, storing it in rc if it was greater than the previous rc value.
If you want to work with $?, you'll need to check it after each command, since $? is updated after each command exits. This means that if you execute a pipeline, you'll only get the exit code of the last process in the pipeline.
Another approach is to do this:
set -e
set -o pipefail
If you put this at the top of the shell script, it looks like Bash will take care of this for you. As a previous poster noted, "set -e" will cause Bash to exit with an error on any simple command. "set -o pipefail" will cause Bash to exit with an error on any command in a pipeline as well.
See here or here for a little more discussion on this problem. Here is the Bash manual section on the set builtin.
"set -e" is probably the easiest way to do this. Just put that before any commands in your program.
If you just call exit in Bash without any parameters, it will return the exit code of the last command. Combined with OR, Bash should only invoke exit, if the previous command fails. But I haven't tested this.
command1 || exit;
command2 || exit;
Bash will also store the exit code of the last command in the variable $?.
[ $? -eq 0 ] || exit $?; # Exit for nonzero return code
http://cfaj.freeshell.org/shell/cus-faq-2.html#11
How do I get the exit code of cmd1 in cmd1|cmd2
First, note that cmd1 exit code could be non-zero and still don't mean an error. This happens for instance in
cmd | head -1
You might observe a 141 (or 269 with ksh93) exit status of cmd1, but it's because cmd was interrupted by a SIGPIPE signal when head -1 terminated after having read one line.
To know the exit status of the elements of a pipeline
cmd1 | cmd2 | cmd3
a. with Z shell (zsh):
The exit codes are provided in the pipestatus special array.
cmd1 exit code is in $pipestatus[1], cmd3 exit code in
$pipestatus[3], so that $? is always the same as
$pipestatus[-1].
b. with Bash:
The exit codes are provided in the PIPESTATUS special array.
cmd1 exit code is in ${PIPESTATUS[0]}, cmd3 exit code in
${PIPESTATUS[2]}, so that $? is always the same as
${PIPESTATUS: -1}.
...
For more details see Z shell.
For Bash:
# This will trap any errors or commands with non-zero exit status
# by calling function catch_errors()
trap catch_errors ERR;
#
# ... the rest of the script goes here
#
function catch_errors() {
# Do whatever on errors
#
#
echo "script aborted, because of errors";
exit 0;
}
In Bash this is easy. Just tie them together with &&:
command1 && command2 && command3
You can also use the nested if construct:
if command1
then
if command2
then
do_something
else
exit
fi
else
exit
fi
#
#------------------------------------------------------------------------------
# purpose: to run a command, log cmd output, exit on error
# usage:
# set -e; do_run_cmd_or_exit "$cmd" ; set +e
#------------------------------------------------------------------------------
do_run_cmd_or_exit(){
cmd="$#" ;
do_log "DEBUG running cmd or exit: \"$cmd\""
msg=$($cmd 2>&1)
export exit_code=$?
# If occurred during the execution, exit with error
error_msg="Failed to run the command:
\"$cmd\" with the output:
\"$msg\" !!!"
if [ $exit_code -ne 0 ] ; then
do_log "ERROR $msg"
do_log "FATAL $msg"
do_exit "$exit_code" "$error_msg"
else
# If no errors occurred, just log the message
do_log "DEBUG : cmdoutput : \"$msg\""
fi
}