#!/bin/bash
set -exuo pipefail
# Run delorean to update the namespaces folder
main() {
if [ !$(yq -r '.random' file_that_doesnt_exist.yaml) = "true" ]; then
echo "yes"
else
echo "no"
fi
}
# shellcheck disable=SC2068
main $#
set -e pipefail based on my understanding should exit the bash script on the first occurence of error. However, I get "no" in stdout even though `echo "no" occurs after the error. How does that happen?
set -e stops the script on error, but not always:
The shell does not exit if the command that fails is part of the command list immediately following a while or until keyword, part of the test following the if or elif reserved words, part of any command executed in a && or || list except the command following the final && or ||, any command in a pipeline but the last, or if the command's return value is being inverted with !.
(Quote from man bash).
Basically, don't use -e (or at least don't only use that.)
Use a trap, and carefully design your tests with it in mind.
c.f. this whole page for good discussions on alternatives and explanations, and this one for some other examples and elaborations that should give you good inspiration for a better solution.
Related
To check the validity of lines in a file I'm using a condition which is met when egrep -v does NOT return an empty result. When there are invalid lines, then this works fine (i.e. the conditional block is executed), but when every line is valid then the script ends without further processing.
Script:
INVALID_HOSTS=$(egrep -v ${IP_REGEX} hosts)
if [[ ! -z "${INVALID_HOSTS}" ]]; then
echo "Invalid hosts:"
for entry in ${INVALID_HOSTS}
do echo ${entry}
done
exit_with_error_msg "hosts file contains invalid hosts (Pattern must be: \"\d+.\d+.\d+.\d+:\d+\"), exiting"
else
echo "all cool"
fi
echo "after if-else"
So when there are no invalid lines then neither the echo "all cool" nor echo "after if-else" get executed. The script just stops and returns to the shell.
When set -x is enabled, then it prints:
++ egrep -v '^(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?):([1-9]|[1-5]?[0-9]{2,4}|6[1-4][0-9]{3}|65[1-4][0-9]{2}|655[1-2][0-9]|6553[1-5])$' hosts
+ INVALID_HOSTS=
Playing around with it I'm sure that it's about the if [[ ! -z "${INVALID_HOSTS}" ]]; then, but my bash wizardry is not strong enough to overcome this magical barrier.
Thanks for any help!
This is a bit long for a comment. I'll start it as an answer and we can work our way through further details or I can scrap it entirely if not helpful. I'll make some assumptions and let us see if it hits the spot.
For starters, you do indeed use the value further, so command expansion into a variable is not entirely useless, but otherwise it's much easier to determine match (or lack thereof) of grep through it's return value. If anything matched (output would be non-empty), it returns (shell true) value of 0, otherwise it returns false (in this case 1). Not to mention the ! -z test notation should really be -n if used at all.
And this is where I'd start assuming a bit. I suspect this is not your entire script and you have errexit option turned on in that shell session (or through rc file in general). Either by means of set -o errexit or set -e or running bash with -e option. Since grep not matching anything returns as failed, your shell (script execution) would terminate after having encountered a failing command.
Observe the difference between:
$ bash -ec 'grep "BOGUS" /etc/fstab ; echo "$?"'
$ bash -c 'grep "BOGUS" /etc/fstab ; echo "$?"'
1
With errexit, bash terminates after grep has "failed" and we never even reach the echo.
Since the assumption has proven to be correct, small extension. If errexit is what you want, you'd need to either change the option value before/after a command you want to be able to fail (return non-zero value) without affecting your script:
set +o errexit
grep THIS_COULD_NOT_MATCH...
set -o errexit
Or you can ignore return value of individual commands by ensuring their success:
grep THIS_COULD_NOT_MATCH... || true
You can also still use potentially "failing" commands safely in conditionals (such as if) without terminating your shell.
I am currently using something like this:
(
false
true
) && echo "OK" || echo "FAILED";
And it doesn't work. I would like the subshell to exit with an error if something fails (false in this case). Currently it only fails if the last command fails.
It should only exit out of the current subshell and not the whole script.
I am giving this script to people and I don't want them to see all the output but still give them some kind of response if the script was successful or not.
Edit: The commands inside the subshell above are only an example. I would like to run multiple commands inside a subshell without checking the return value after each command. Something like set -e for subshells.
Edit2: I tried adding set -e inside a subshell. Maybe I did something wrong but it didn't change the behavior of my script. It didn't stop execution or exit out of the subshell with a non-0 code.
(
set -e
false
echo "test"
) && echo "OK" || echo "FAILED";
First prints test and then OK. It should print FAILED because of false.
This effect of bash set -e inside a conditional expression like foo || true, is known and considered a problem. I think it is good reason to hate both set -e and shell scripting in general.
http://david.rothlis.net/shell-set-e/
http://fvue.nl/wiki/Bash:_Error_handling
The first link makes the following suggestion. It looks good in small examples, but maybe less clear in the real world.
Do your own error checking by stringing together a series of commands
with “&&” like this:
mkdir abc &&
cd abc &&
do_something_else &&
last_thing ||
{ echo error >&2; exit 1; }
A few important notes: You don’t need a trailing backslash. You don’t
indent the following line. You must stick to 80 chars per line so that
everyone can see the “&&” or “||” at the end. If you need to mix ANDs
and ORs, group with { these braces } which don’t spawn a sub-shell.
To your edited question:
Something like set -e for subshells.
Well, you can just do set -e for the subshell.
( set -e
my
commands
)
You can't implicitly make just your subshells have the errexit option. You can do some trickery using eval, or use a subprocess as a shell (even though a subprocess is not the same as a subshell), like
errexit_shell() {
bash -e
}
but those options are both inadvisable for various reasons, not the least of which being readability. Your best bet in that case would just be to adapt your entire script to use set -e, and your subshells will come along for the ride.
To your original question:
Just capture the status of the part that indicates success or failure:
(
cat teisatrt
status=$?
echo "true"
exit "$status"
) && echo passed || echo failed
(Of course, if all you want to know is if that file is readable, don't cat it, just use test -r.)
As you have it, you are redirecting output from the whole subshell to /dev/null, so you will never see your "true" echo. You should move the redirect inside the subshell to the command you really want it on. In order to exit the subshell when cat fails, you will need to check for failure after cat runs. If you don't, then as you have noted, its return code is wiped out by the following statement. So something like this:
echo "Installing App"
(
cat teisatrt &> /dev/null || exit 1
echo "true"
) && echo "OK" || echo "FAILED";
I can't find any documentation indicating that this is correct behavior. I've verified this behavior on bash 4.2.25 and 4.1.2. Those are the latest to which I have simple access.
Much appreciated if anybody has an explanation for why the conditional changes the behavior of the preceding subshell.
# ( set -e; false ; echo bye )
# ( set -e; false ; echo bye ) || echo "failed"
bye
#
thanks,
Brian
See the the following link
The -e setting shall be ignored when executing the compound
list following the while, until, if, or elif reserved word,
a pipeline beginning with the ! reserved word, or any
command of an AND-OR list other than the last.
I believe the emphasized portion of the description of the -e option in the bash 4.2 man page applies:
The shell does not exit if
the command that fails is [...] part of any command executed in
a && or || list [...]
The false command is part of the subshell which makes up the LHS of the || operator, so although it fails, the -e is ignored. Arguably, this is a bug, since false is not part of an || in the shell where set -e is actually set.
set -e (or a script starting with #!/bin/sh -e) is extremely useful to automatically bomb out if there is a problem. It saves me having to error check every single command that might fail.
How do I get the equivalent of this inside a function?
For example, I have the following script that exits immediately on error with an error exit status:
#!/bin/sh -e
echo "the following command could fail:"
false
echo "this is after the command that fails"
The output is as expected:
the following command could fail:
Now I'd like to wrap this into a function:
#!/bin/sh -e
my_function() {
echo "the following command could fail:"
false
echo "this is after the command that fails"
}
if ! my_function; then
echo "dealing with the problem"
fi
echo "run this all the time regardless of the success of my_function"
Expected output:
the following command could fail:
dealing with the problem
run this all the time regardless of the success of my_function
Actual output:
the following output could fail:
this is after the command that fails
run this all the time regardless of the success of my_function
(ie. the function is ignoring set -e)
This presumably is expected behaviour. My question is: how do I get the effect and usefulness of set -e inside a shell function? I'd like to be able to set something up such that I don't have to individually error check every call, but the script will stop on encountering an error. It should unwind the stack as far as is needed until I do check the result, or exit the script itself if I haven't checked it. This is what set -e does already, except it doesn't nest.
I've found the same question asked outside Stack Overflow but no suitable answer.
I eventually went with this, which apparently works. I tried the export method at first, but then found that I needed to export every global (constant) variable the script uses.
Disable set -e, then run the function call inside a subshell that has set -e enabled. Save the exit status of the subshell in a variable, re-enable set -e, then test the var.
f() { echo "a"; false; echo "Should NOT get HERE"; }
# Don't pipe the subshell into anything or we won't be able to see its exit status
set +e ; ( set -e; f ) ; err_status=$?
set -e
## cleaner syntax which POSIX sh doesn't support. Use bash/zsh/ksh/other fancy shells
if ((err_status)) ; then
echo "f returned false: $err_status"
fi
## POSIX-sh features only (e.g. dash, /bin/sh)
if test "$err_status" -ne 0 ; then
echo "f returned false: $err_status"
fi
echo "always print this"
You can't run f as part of a pipeline, or as part of a && of || command list (except as the last command in the pipe or list), or as the condition in an if or while, or other contexts that ignore set -e. This code also can't be in any of those contexts, so if you use this in a function, callers have to use the same subshell / save-exit-status trickery. This use of set -e for semantics similar to throwing/catching exceptions is not really suitable for general use, given the limitations and hard-to-read syntax.
trap err_handler_function ERR has the same limitations as set -e, in that it won't fire for errors in contexts where set -e won't exit on failed commands.
You might think the following would work, but it doesn't:
if ! ( set -e; f );then ##### doesn't work, f runs ignoring -e
echo "f returned false: $?"
fi
set -e doesn't take effect inside the subshell because it remembers that it's inside the condition of an if. I thought being a subshell would change that, but only being in a separate file and running a whole separate shell on it would work.
From documentation of set -e:
When this option is on, if a simple command fails for any of the
reasons listed in Consequences of
Shell Errors or returns an exit status
value > 0, and is not part of the
compound list following a while,
until, or if keyword, and is not a
part of an AND or OR list, and is not
a pipeline preceded by the ! reserved
word, then the shell shall immediately
exit.
In your case, false is a part of a pipeline preceded by ! and a part of if. So the solution is to rewrite your code so that it isn't.
In other words, there's nothing special about functions here. Try:
set -e
! { false; echo hi; }
You may directly use a subshell as your function definition and set it to exit immediately with set -e. This would limit the scope of set -e to the function subshell only and would later avoid switching between set +e and set -e.
In addition, you can use a variable assignment in the if test and then echo the result in an additional else statement.
# use subshell for function definition
f() (
set -exo pipefail
echo a
false
echo Should NOT get HERE
exit 0
)
# next line also works for non-subshell function given by agsamek above
#if ret="$( set -e && f )" ; then
if ret="$( f )" ; then
true
else
echo "$ret"
fi
# prints
# ++ echo a
# ++ false
# a
This is a bit of a kludge, but you can do:
export -f f
if sh -ec f; then
...
This will work if your shell supports export -f (bash does).
Note that this will not terminate the script. The echo
after the false in f will not execute, nor will the body
of the if, but statements after the if will be executed.
If you are using a shell that does not support export -f, you can
get the semantics you want by running sh in the function:
f() { sh -ec '
echo This will execute
false
echo This will not
'
}
Note/Edit: As a commenter pointed out, this answer uses bash, and not sh like the OP used in his question. I missed that detail when I originaly posted an answer. I will leave this answer up anyway since it might be interested to some passerby.
Y'aaaaaaaaaaaaaaaaaaallll ready for this?
Here's a way to do it with leveraging the DEBUG trap, which runs before each command, and sort of makes errors like the whole exception/try/catch idioms from other languages. Take a look. I've made your example one more 'call' deep.
#!/bin/bash
# Get rid of that disgusting set -e. We don't need it anymore!
# functrace allows RETURN and DEBUG traps to be inherited by each
# subshell and function. Plus, it doesn't suffer from that horrible
# erasure problem that -e and -E suffer from when the command
# is used in a conditional expression.
set -o functrace
# A trap to bubble up the error unless our magic command is encountered
# ('catch=$?' in this case) at which point it stops. Also don't try to
# bubble the error if were not in a function.
trap '{
code=$?
if [[ $code != 0 ]] && [[ $BASH_COMMAND != '\''catch=$?'\'' ]]; then
# If were in a function, return, else exit.
[[ $FUNCNAME ]] && return $code || exit $code
fi
}' DEBUG
my_function() {
my_function2
}
my_function2() {
echo "the following command could fail:"
false
echo "this is after the command that fails"
}
# the || isn't necessary, but the 'catch=$?' is.
my_function || catch=$?
echo "Dealing with the problem with errcode=$catch (⌐■_■)"
echo "run this all the time regardless of the success of my_function"
and the output:
the following command could fail:
Dealing with the problem with errcode=1 (⌐■_■)
run this all the time regardless of the success of my_function
I haven't tested this in the wild, but off the top of my head, there are a bunch of pros:
It's actually not that slow. I've ran the script in a tight loop with and without the functrace option, and times are very close to each other under 10 000 iterations.
You could expand on this DEBUG trap to print a stack trace without doing that whole looping over $FUNCNAME and $BASH_LINENO nonsense. You kinda get it for free (besides actually doing an echo line).
Don't have to worry about that shopt -s inherit_errexit gotcha.
Join all commands in your function with the && operator. It's not too much trouble and will give the result you want.
This is by design and POSIX specification. We can read in man bash:
If a compound command or shell function executes in a context where -e is being ignored, none of the commands executed within the compound command or function body will be affected by the -e setting, even if -e is set and a command returns a failure status. If a compound command or shell function sets -e while executing in a context where -e is ignored, that setting will not have any effect until the compound command or the command containing the function call completes.
therefore you should avoid relying on set -e within functions.
Given the following exampleAustin Group:
set -e
start() {
some_server
echo some_server started successfully
}
start || echo >&2 some_server failed
the set -e is ignored within the function, because the function is a command in an AND-OR list other than the last.
The above behaviour is specified and required by POSIX (see: Desired Action):
The -e setting shall be ignored when executing the compound list following the while, until, if, or elif reserved word, a pipeline beginning with the ! reserved word, or any command of an AND-OR list other than the last.
I know this isn't what you asked, but you may or may not be aware that the behavior you seek is built into "make". Any part of a "make" process that fails aborts the run. It's a wholly different way of "programming", though, than shell scripting.
You will need to call your function in a sub shell (inside brackets ()) to achieve this.
I think you want to write your script like this:
#!/bin/sh -e
my_function() {
echo "the following command could fail:"
false
echo "this is after the command that fails"
}
(my_function)
if [ $? -ne 0 ] ; then
echo "dealing with the problem"
fi
echo "run this all the time regardless of the success of my_function"
Then the output is (as desired):
the following command could fail:
dealing with the problem
run this all the time regardless of the success of my_function
If a subshell isn't an option (say you need to do something crazy like set a variable) then you can just check every single command that might fail and deal with it by appending || return $?. This causes the function to return the error code on failure.
It's ugly, but it works
#!/bin/sh
set -e
my_function() {
echo "the following command could fail:"
false || return $?
echo "this is after the command that fails"
}
if ! my_function; then
echo "dealing with the problem"
fi
echo "run this all the time regardless of the success of my_function"
gives
the following command could fail:
dealing with the problem
run this all the time regardless of the success of my_function
set -e (or a script starting with #!/bin/sh -e) is extremely useful to automatically bomb out if there is a problem. It saves me having to error check every single command that might fail.
How do I get the equivalent of this inside a function?
For example, I have the following script that exits immediately on error with an error exit status:
#!/bin/sh -e
echo "the following command could fail:"
false
echo "this is after the command that fails"
The output is as expected:
the following command could fail:
Now I'd like to wrap this into a function:
#!/bin/sh -e
my_function() {
echo "the following command could fail:"
false
echo "this is after the command that fails"
}
if ! my_function; then
echo "dealing with the problem"
fi
echo "run this all the time regardless of the success of my_function"
Expected output:
the following command could fail:
dealing with the problem
run this all the time regardless of the success of my_function
Actual output:
the following output could fail:
this is after the command that fails
run this all the time regardless of the success of my_function
(ie. the function is ignoring set -e)
This presumably is expected behaviour. My question is: how do I get the effect and usefulness of set -e inside a shell function? I'd like to be able to set something up such that I don't have to individually error check every call, but the script will stop on encountering an error. It should unwind the stack as far as is needed until I do check the result, or exit the script itself if I haven't checked it. This is what set -e does already, except it doesn't nest.
I've found the same question asked outside Stack Overflow but no suitable answer.
I eventually went with this, which apparently works. I tried the export method at first, but then found that I needed to export every global (constant) variable the script uses.
Disable set -e, then run the function call inside a subshell that has set -e enabled. Save the exit status of the subshell in a variable, re-enable set -e, then test the var.
f() { echo "a"; false; echo "Should NOT get HERE"; }
# Don't pipe the subshell into anything or we won't be able to see its exit status
set +e ; ( set -e; f ) ; err_status=$?
set -e
## cleaner syntax which POSIX sh doesn't support. Use bash/zsh/ksh/other fancy shells
if ((err_status)) ; then
echo "f returned false: $err_status"
fi
## POSIX-sh features only (e.g. dash, /bin/sh)
if test "$err_status" -ne 0 ; then
echo "f returned false: $err_status"
fi
echo "always print this"
You can't run f as part of a pipeline, or as part of a && of || command list (except as the last command in the pipe or list), or as the condition in an if or while, or other contexts that ignore set -e. This code also can't be in any of those contexts, so if you use this in a function, callers have to use the same subshell / save-exit-status trickery. This use of set -e for semantics similar to throwing/catching exceptions is not really suitable for general use, given the limitations and hard-to-read syntax.
trap err_handler_function ERR has the same limitations as set -e, in that it won't fire for errors in contexts where set -e won't exit on failed commands.
You might think the following would work, but it doesn't:
if ! ( set -e; f );then ##### doesn't work, f runs ignoring -e
echo "f returned false: $?"
fi
set -e doesn't take effect inside the subshell because it remembers that it's inside the condition of an if. I thought being a subshell would change that, but only being in a separate file and running a whole separate shell on it would work.
From documentation of set -e:
When this option is on, if a simple command fails for any of the
reasons listed in Consequences of
Shell Errors or returns an exit status
value > 0, and is not part of the
compound list following a while,
until, or if keyword, and is not a
part of an AND or OR list, and is not
a pipeline preceded by the ! reserved
word, then the shell shall immediately
exit.
In your case, false is a part of a pipeline preceded by ! and a part of if. So the solution is to rewrite your code so that it isn't.
In other words, there's nothing special about functions here. Try:
set -e
! { false; echo hi; }
You may directly use a subshell as your function definition and set it to exit immediately with set -e. This would limit the scope of set -e to the function subshell only and would later avoid switching between set +e and set -e.
In addition, you can use a variable assignment in the if test and then echo the result in an additional else statement.
# use subshell for function definition
f() (
set -exo pipefail
echo a
false
echo Should NOT get HERE
exit 0
)
# next line also works for non-subshell function given by agsamek above
#if ret="$( set -e && f )" ; then
if ret="$( f )" ; then
true
else
echo "$ret"
fi
# prints
# ++ echo a
# ++ false
# a
This is a bit of a kludge, but you can do:
export -f f
if sh -ec f; then
...
This will work if your shell supports export -f (bash does).
Note that this will not terminate the script. The echo
after the false in f will not execute, nor will the body
of the if, but statements after the if will be executed.
If you are using a shell that does not support export -f, you can
get the semantics you want by running sh in the function:
f() { sh -ec '
echo This will execute
false
echo This will not
'
}
Note/Edit: As a commenter pointed out, this answer uses bash, and not sh like the OP used in his question. I missed that detail when I originaly posted an answer. I will leave this answer up anyway since it might be interested to some passerby.
Y'aaaaaaaaaaaaaaaaaaallll ready for this?
Here's a way to do it with leveraging the DEBUG trap, which runs before each command, and sort of makes errors like the whole exception/try/catch idioms from other languages. Take a look. I've made your example one more 'call' deep.
#!/bin/bash
# Get rid of that disgusting set -e. We don't need it anymore!
# functrace allows RETURN and DEBUG traps to be inherited by each
# subshell and function. Plus, it doesn't suffer from that horrible
# erasure problem that -e and -E suffer from when the command
# is used in a conditional expression.
set -o functrace
# A trap to bubble up the error unless our magic command is encountered
# ('catch=$?' in this case) at which point it stops. Also don't try to
# bubble the error if were not in a function.
trap '{
code=$?
if [[ $code != 0 ]] && [[ $BASH_COMMAND != '\''catch=$?'\'' ]]; then
# If were in a function, return, else exit.
[[ $FUNCNAME ]] && return $code || exit $code
fi
}' DEBUG
my_function() {
my_function2
}
my_function2() {
echo "the following command could fail:"
false
echo "this is after the command that fails"
}
# the || isn't necessary, but the 'catch=$?' is.
my_function || catch=$?
echo "Dealing with the problem with errcode=$catch (⌐■_■)"
echo "run this all the time regardless of the success of my_function"
and the output:
the following command could fail:
Dealing with the problem with errcode=1 (⌐■_■)
run this all the time regardless of the success of my_function
I haven't tested this in the wild, but off the top of my head, there are a bunch of pros:
It's actually not that slow. I've ran the script in a tight loop with and without the functrace option, and times are very close to each other under 10 000 iterations.
You could expand on this DEBUG trap to print a stack trace without doing that whole looping over $FUNCNAME and $BASH_LINENO nonsense. You kinda get it for free (besides actually doing an echo line).
Don't have to worry about that shopt -s inherit_errexit gotcha.
Join all commands in your function with the && operator. It's not too much trouble and will give the result you want.
This is by design and POSIX specification. We can read in man bash:
If a compound command or shell function executes in a context where -e is being ignored, none of the commands executed within the compound command or function body will be affected by the -e setting, even if -e is set and a command returns a failure status. If a compound command or shell function sets -e while executing in a context where -e is ignored, that setting will not have any effect until the compound command or the command containing the function call completes.
therefore you should avoid relying on set -e within functions.
Given the following exampleAustin Group:
set -e
start() {
some_server
echo some_server started successfully
}
start || echo >&2 some_server failed
the set -e is ignored within the function, because the function is a command in an AND-OR list other than the last.
The above behaviour is specified and required by POSIX (see: Desired Action):
The -e setting shall be ignored when executing the compound list following the while, until, if, or elif reserved word, a pipeline beginning with the ! reserved word, or any command of an AND-OR list other than the last.
I know this isn't what you asked, but you may or may not be aware that the behavior you seek is built into "make". Any part of a "make" process that fails aborts the run. It's a wholly different way of "programming", though, than shell scripting.
You will need to call your function in a sub shell (inside brackets ()) to achieve this.
I think you want to write your script like this:
#!/bin/sh -e
my_function() {
echo "the following command could fail:"
false
echo "this is after the command that fails"
}
(my_function)
if [ $? -ne 0 ] ; then
echo "dealing with the problem"
fi
echo "run this all the time regardless of the success of my_function"
Then the output is (as desired):
the following command could fail:
dealing with the problem
run this all the time regardless of the success of my_function
If a subshell isn't an option (say you need to do something crazy like set a variable) then you can just check every single command that might fail and deal with it by appending || return $?. This causes the function to return the error code on failure.
It's ugly, but it works
#!/bin/sh
set -e
my_function() {
echo "the following command could fail:"
false || return $?
echo "this is after the command that fails"
}
if ! my_function; then
echo "dealing with the problem"
fi
echo "run this all the time regardless of the success of my_function"
gives
the following command could fail:
dealing with the problem
run this all the time regardless of the success of my_function