I couldn't explain what happens in my scripts, could anyone shed some light please?
I am doing some pretty standard stuff, set errexit, sourcing one script from another, catching errors and eventually bailing out if any.
s1.sh
#!/bin/bash
num=1
if [ $num -eq 1 ]; then
FOO="$(set -o | grep -e "errexit" -e "nounset" | grep off >&2)"
VAR="SOME/TEXT/$(basename "$UNBOUND_VARIABLE")"
RET="$(echo $?)"
#ERR="$UNBOUND_VARIABLE" # this will be trapped and source will exit at this line
BAR="LAST_IS_GOOD"
fi
s2.sh
function source_all
{
local __f
set -exu
for __f in ${#}; do
case "$__f" in
"s1.sh" ) set -o posix; (source "$(pwd)/$__f") || return 1; echo "$$ $?" >&2 ;;
esac
done
set +eux +o posix
}
function main
{
source_all s1.sh || return 1
}
main
output
+ for __f in ${#}
+ case "$__f" in
+ set -o posix
++ pwd
+ source (blah/blah)/s1.sh
++ num=1
++ '[' 1 -eq 1 ']'
+++ set -o
+++ grep -e errexit -e nounset
+++ grep off
++ FOO=
(blah/blah)/s1.sh: line 6: UNBOUND_VARIABLE: unbound variable # should exit
++ VAR=SOME/TEXT/
+++ echo 1
++ RET=1
++ BAR=LAST_IS_GOOD
+ echo '9568 0'
9568 0
+ set +eux +o posix
source --help
Exit Status:
Returns the status of the last command executed in FILENAME; fails if
FILENAME cannot be read.
question is: why source invoked in s2.sh doesn't return 1? Why does it keep processing s1.sh after UNBOUND_VARIABLE?
thanks for your inputs
The UNBOUND VARIABLE error comes because you are using set -u are referencing with $UNBOUND_VARIABLE a variable named _UNBOUND_VARIABLE_ which has not been assigned to, in the statement
VAR="SOME/TEXT/$(basename "$UNBOUND_VARIABLE")"
. The set -e does have an effect, in that the subshell this is executed, i.e.
(source "$(pwd)/$__f")
is aborted. While the subshell due this abort indeed returns with non-zero exit code, but this does not trigger an exit of the parent process, because you have a || return to the right. For the same reason, the command
false || echo x
would not terminate the execution, even though a single
false
would.
I'm new to bash so assume that I don't understand everything in this simple script as I've been putting this together as of today with no prior experience with bash.
I get this error when I run test.sh:
command substitution: line 29: syntax error near unexpected token `$1,'
./f.sh: command substitution: line 29: `index_of($1, $urls))'
FILE: f.sh
#!/bin/bash
urls=( "example.com" "example2.com")
error_exit()
{
echo "$1" 1>&2
exit 1
}
index_of(){
needle=$1
haystack=$2
for i in "${!haystack[#]}"; do
if [[ "${haystack[$i]}" = "${needle}" ]]; then
echo "${i}"
fi
done
echo -1
}
validate_url_param(){
index=-2 #-2 as flag
if [ $# -eq 0 ]; then
error_exit "No url provided. Exiting"
else
index=$(index_of($1, $urls)) #ERROR POINTS TO THIS LINE
if [ $index -eq -1 ]; then
error_exit "Provided url not found in list. Exiting"
fi
fi
echo $index
}
FILE: test.sh
#!/bin/bash
. ./f.sh
index=$(validate_url_param "example.com")
echo $index
echo "${urls[0]}"
I've lost track of all of the tweaks I tried but google is failing me and I'm sure this is some basic stuff so... thanks in advance.
The immediate error, just like the error message tells you, is that shell functions (just like shell scripts) do not require or accept commas between their arguments or parentheses around the argument list. But there are several changes you could make to improve this code.
Here's a refactored version, with inlined comments.
#!/bin/bash
urls=("example.com" "example2.com")
error_exit()
{
# Include script name in error message; echo all parameters
echo "$0: $#" 1>&2
exit 1
}
# A function can't really accept an array. But it's easy to fix:
# make the first argument the needle, and the rest, the haystack.
# Also, mark variables as local
index_of(){
local needle=$1
shift
local i
for ((i=1; i<=$#; ++i)); do
if [[ "${!i}" = "${needle}" ]]; then
echo "${i}"
# Return when you found it
return 0
fi
done
# Don't echo anything on failure; just return false
return 1
}
validate_url_param(){
# global ${urls[#]} is still a bit of a wart
if [ $# -eq 0 ]; then
error_exit "No url provided. Exiting"
else
if ! index_of "$1" "${urls[#]}"; then
error_exit "Provided url not found in list. Exiting"
fi
fi
}
# Just run the function from within the script itself
validate_url_param "example.com"
echo "${urls[0]}"
Notice how the validate_url_param function doesn't capture the output from the function it is calling. index_of simply prints the result to standard output and that's fine, just let that happen and don't intervene. The exit code tells us whether it succeeded or not.
However, reading the URLs into memory is often not useful or necessary. Perhaps you are simply looking for
grep -Fx example.com urls.txt
I am running bash 4.4.19 on MacOS. I found the signal function doesn't work when if "set -e" is set. is it expected behavior? Here is my sample code:
#!/usr/local/bin/bash
set -e
declare -A Array1
Array1=([index1]="abc" [index2]="def" [index3]="dfkdjkfjdkdjfdk")
trap ReactSignal USR1
fun() {
PPid=$1
NUM=0
Array1[index4]="insidefunction"
while [ $NUM -le 5 ]
do
((NUM++))
echo "inside number is $NUM"
sleep 1
done
kill -USR1 $PPid
}
ReactSignal() {
IFS= read -r -d '' -u 3 checkOutput
echo "function output is ${checkOutput}"
}
Ppid="$$"
echo "start...."
coproc funfd { fun $Ppid; }
exec 3>&${funfd[0]}
echo "end...."
sleep 7
echo array value is ${Array1[#]}
It's because ((expression)) actually has a return code, as per the bash documentation:
((expression))
The expression is evaluated according to the rules described below under ARITHMETIC EVALUATION. If the value of the expression is non-zero, the return status is 0; otherwise the return status is 1.
So, of course, if NUM is zero going in to the expression, the return code will be one and set -e will pick this up as an error that needs termination.
There are any number of ways to solve this, from using the (rather ugly, in my opinion):
set +e ; ((NUM++)); set -e
to not using ((expression)) at all:
for i in {0..4} ; do
doSomethingWith $i
done
I have an if statement which is used many times in my script:
if [[ $? -ne 0 ]]; then
echo [$JOB_NAME] failed.
exit 1
fi
is it possible to define a variable and assign this statement to it, and then call it each time I need ti?
Example of use:
echo [$JOB_NAME] extracting manifests...
unzip -o ZIP_FILE "*.yml"
# Push the app to CF
cf push -f $MANIFEST_FILE -p ZIP_FILE $NEW_APP
if [[ $? -ne 0 ]]; then
echo [$JOB_NAME] failed.
exit 1
fi
As mentioned in the comments, this should be a function, which could be written like this:
ensure_success () {
if [[ $# -eq 0 ]]; then
echo "no command passed to ensure_success."
elif ! "$#"; then
echo "[$JOB_NAME] failed."
exit 1
fi
}
"$#" expands to the full list of arguments passed to the function. I added a check based on Inian's suggestion in the comments, to ensure that at least one argument is passed to the function.
This combines running the command and checking the error code, so you can use it like:
ensure_success command arg1 arg2 arg3
So, based on the example in your question it would be:
ensure_success cf push -f "$MANIFEST_FILE" -p ZIP_FILE "$NEW_APP"
The quotes are free.
I'm not sure where $JOB_NAME is defined but presumably it is a global.
When I write bash scripts I usually get the verbose mode this way (simplified):
_V=0
while getopts "v" OPTION
do
case $OPTION in
v) _V=1
;;
esac
done
and then every time I want a "verbose output" I type this:
[ $_V -eq 1 ] && echo "verbose mode on" || echo "verbose mode off"
or for example this:
[ $_V -eq 1 ] && command -v || command
Is there a way to do it more elegant? I was thinking about defining a function named "verbose" and type it instead of [ $_V -eq 1 ], but this would only be a tiny improvement.
I'm sure, there is more common way to do it…
As you noticed, you can define some log functions like log, log_debug, log_error, etc.
function log () {
if [[ $_V -eq 1 ]]; then
echo "$#"
fi
}
It can help increasing your main code readability and hide show\nonshow logic into logging function.
log "some text"
If _V(global variable) is equal 1 "some text" will be printed, in other case it will not.
After reading all other posts I came up with this
# set verbose level to info
__VERBOSE=6
declare -A LOG_LEVELS
# https://en.wikipedia.org/wiki/Syslog#Severity_level
LOG_LEVELS=([0]="emerg" [1]="alert" [2]="crit" [3]="err" [4]="warning" [5]="notice" [6]="info" [7]="debug")
function .log () {
local LEVEL=${1}
shift
if [ ${__VERBOSE} -ge ${LEVEL} ]; then
echo "[${LOG_LEVELS[$LEVEL]}]" "$#"
fi
}
Then you can simply use it like this
# verbose error
.log 3 "Something is wrong here"
Which will output
[error] Something is wrong here
#!/bin/bash
# A flexible verbosity redirection function
# John C. Petrucci (http://johncpetrucci.com)
# 2013-10-19
# Allows your script to accept varying levels of verbosity flags and give appropriate feedback via file descriptors.
# Example usage: ./this [-v[v[v]]]
verbosity=2 #Start counting at 2 so that any increase to this will result in a minimum of file descriptor 3. You should leave this alone.
maxverbosity=5 #The highest verbosity we use / allow to be displayed. Feel free to adjust.
while getopts ":v" opt; do
case $opt in
v) (( verbosity=verbosity+1 ))
;;
esac
done
printf "%s %d\n" "Verbosity level set to:" "$verbosity"
for v in $(seq 3 $verbosity) #Start counting from 3 since 1 and 2 are standards (stdout/stderr).
do
(( "$v" <= "$maxverbosity" )) && echo This would display $v
(( "$v" <= "$maxverbosity" )) && eval exec "$v>&2" #Don't change anything higher than the maximum verbosity allowed.
done
for v in $(seq $(( verbosity+1 )) $maxverbosity ) #From the verbosity level one higher than requested, through the maximum;
do
(( "$v" > "2" )) && echo This would not display $v
(( "$v" > "2" )) && eval exec "$v>/dev/null" #Redirect these to bitbucket, provided that they don't match stdout and stderr.
done
# Some confirmations:
printf "%s\n" "This message is seen at verbosity level 3 and above." >&3
printf "%s\n" "This message is seen at verbosity level 4 and above." >&4
printf "%s\n" "This message is seen at verbosity level 5 and above." >&5
I also came up with this function to do a quick ifelse:
function verbose () {
[[ $_V -eq 1 ]] && return 0 || return 1
}
This executes a command if $_V is set to 1. Use it like this:
verbose && command #command will be executed if $_V == 1
or
verbose && command -v || command # execute 'command -v' if $_V==1, else execute 'command'
If you want to avoid doing an "if" statement every single time you want to log something, you can try this approach (which is how I do it).
The idea is that instead of calling log, you call $echoLog instead. So, if you are in verbose mode, $echoLog will just be echo, but in non-verbose mode, it is a function that prints nothing and just ignores the arguments.
Here's some code you can copy.
# Use `$echoLog` everywhere you print verbose logging messages to console
# By default, it is disabled and will be enabled with the `-v` or `--verbose` flags
declare echoLog='silentEcho'
function silentEcho() {
:
}
# Somewhere else in your script's setup, do something like this
while [[ $# > 0 ]]; do
case "$1" in
-v|--verbose) echoLog='echo'; ;;
esac
shift;
done
Now, you can just drop lines like $echoLog "Doing something verbose log worthy" anywhere you want.
A first try at a more flexible system with verbosity levels (Bash 4):
# CONFIG SECTION
# verbosity level definitions
config[verb_levels]='debug info status warning error critical fatal'
# verbosity levels that are to be user-selectable (0-this value)
config[verb_override]=3
# user-selected verbosity levels (0=none, 1=warnings, 2=warnings+info, 3=warning+info+debug)
config[verbosity]=2
# FUNCTION DEFINITIONS SECTION
_messages() {
# shortcut functions for messages
# non overridable levels exit with errlevel
# safe eval, it only uses two (namespaced) values, and a few builtins
local verbosity macro level=0
for verbosity in ${config[verb_levels]}; do
IFS="" read -rd'' macro <<MACRO
_$verbosity() {
$( (( $level <= ${config[verb_override]} )) && echo "(( \${config[verbosity]} + $level > ${config[verb_override]} )) &&" ) echo "${verbosity}: \$#";
$( (( $level > ${config[verb_override]} )) && echo "exit $(( level - ${config[verb_override]} ));" )
}
MACRO
eval "$macro"
(( level++ ))
done
}
# INITIALIZATION SECTION
_messages
After initialization, anywhere in your code you can use things like:
! (( $# )) && _error "parameter expected"
[[ -f somefile ]] && _warning "file $somefile already exists"
_info "some info"
_status "running command"
if (( ${config[verbosity]} <= 1 )); then
command
else
command -v
fi
# explicitly changing verbosity at run time
old_verbosity=${config[verbosity]}
config[verbosity]=1
etc.
verbose=false
while getopts "v" OPTION
do
case $OPTION in
v) verbose=true
;;
esac
done
Then
$verbose && echo "Verbose mode on" || echo "Verbose mode off"
This will execute /bin/true or /bin/false, returning 0 or 1 respectively.
To avoid using multiple if statements or using a variable to hold a function name how about declaring different functions based on the verbosity!
This works for ALL bourne shell derivatives not just bash!
#verbose=verbose_true # uncomment to make script verbose
if [ "$verbose" ]; then
log() { echo "$#"; }
else
log() { :; }
fi
log This Script is Verbose
NOTE: the use of "verbose=verbose_true" makes script tracing a lot nicer
but you can make that one if however you like.
I would propose a modified version of #fentas's answer:
# set verbose level to info
__VERBOSE=6
declare -A LOG_LEVELS
# https://en.wikipedia.org/wiki/Syslog#Severity_level
LOG_LEVELS=([0]="emerg" [1]="alert" [2]="crit" [3]="err" [4]="warning" [5]="notice" [6]="info" [7]="debug")
function .log () {
local LEVEL=${1}
shift
if [ ${__VERBOSE} -ge ${LEVEL} ]; then
if [ -t 0 ]; then
# seems we are in an interactive shell
echo "[${LOG_LEVELS[$LEVEL]}]" "$#" >&2
else
# seems we are in a cron job
logger -p "${LOG_LEVELS[$LEVEL]}" -t "$0[$$]" -- "$*"
fi
fi
}