A way to ignore exit status in gitlab job pipeline [duplicate] - bash

In our project we have a shell script which is to be sourced to set up environment variables for the subsequent build process or to run the built applications.
It contains a block which checks the already set variables and does some adjustment.
# part of setup.sh
for LIBRARY in "${LIBRARIES_WE_NEED[#]}"
do
echo $LD_LIBRARY_PATH | \grep $LIBRARY > /dev/null
if [ $? -ne 0 ]
then
echo Adding $LIBRARY
LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$LIBRARY
else
echo Not adding $LIBRARY
fi
done
i.e. it checks if a path to a library is already in $LD_LIBRARY_PATH and if not, adds it.
(To be fair, this could be written differently (like here), but assume the script is supposed to achieve something which is very hard to do without calling a program, checking $? and then either doing one thing or doing another thing).
The .gitlab-ci.yml then contains
before_script:
- yum install -y <various packages>
- source setup.sh
but the runner decides to stop the before script the very moment $? is non-zero, i.e. when the if-statement decides to add a path to $LD_LIBRARY_PATH.
Now it is nice that the gitlab runner checks $? after each line of my script, but here it'd be great if the lines in .gitlab-ci.yml were considered atomic.
Is there a way to avoid the intermediate checks of $? in a script that's sourced in .gitlab-ci.yml?

Use command_that_might_fail || true to mask the exit status of said command.
Also note that you can use grep -q to prevent output:
echo "$LD_LIBRARY_PATH" | grep -q "$LIBRARY" || true
This will however also mask $? which you might not want. If you want to check if the command exits correct you might use:
if echo "$LD_LIBRARY_PATH" | grep -q "$LIBRARY"; then
echo "Adding $LIBRARY"
else
...
fi
I suspect that gitlab-ci sets -e which you can disabled with set +e:
set +e # Disable exit on error
for library in "${LIBRARIES_WE_NEED[#]}"; do
...
done
set -e # Enable exit on error
Future reading: Why double quotes matter and Pitfalls with set -e

Another trick that I am using is a special kind of "|| true", combined with having access to previous exit code.
- exit_code=0
- ./myScript.sh || exit_code=$?
- if [ ${exit_code} -ne 0 ]; then echo "It failed!" ; else echo "It worked!"; fi
The $exit_code=$? always evaluates to "true" so you get a non failing command but you also receive exit_code and you can do whatever you want with it.
Note please, that you shouldn't skip the first line or exit_code will be uninitialized (since on successful run of script, the or'ed part is never executed and the if ends up being)
if [ -ne 0 ];
instead of
if [ 0 -ne 0 ];
Which causes syntax error.

Related

How to check if file exists in Google Cloud Storage with the gcloud bash command?

I need to check if a file exists in a gitlab deployment pipeline. How to do it efficiently and reliably?
Use gsutil ls gs://bucket/object-name and check the return value for 0.
If the object does not exist, the return value is 1.
You can add the following Shell script in a Gitlab job :
#!/usr/bin/env bash
set -o pipefail
set -u
gsutil -q stat gs://your_bucket/folder/your_file.csv
PATH_EXIST=$?
if [ ${PATH_EXIST} -eq 0 ]; then
echo "Exist"
else
echo "Not Exist"
fi
I used gcloud cli and gsutil with stat command with -q option.
In this case, if the file exists the command returns 0 otherwise 1.
This answer evolved from the answer of Mazlum Tosun. Because I think it is a substantial improvement with less lines and no global settings switching it needs to be a separate answer.
Ideally the answer would be something like this
- gsutil stat $BUCKET_PATH
- if [ $? -eq 0 ]; then
... # do if file exists
else
... # do if file does not exists
fi
$? stores the exit_status of the previous command. 0 if success. This works fine in a local console. The problem with Gitlab will be that if the file does not exists, then "gsutil stat $BUCKET_PATH" will produce a non-zero exit code and the whole pipeline will stop at that line with an error. We need to catch the error, while still storing the exit code.
We will use the or operator || to suppress the error. FILE_EXISTS=false will only be executed if gsutil stat fails.
- gsutil stat $BUCKET_PATH || FILE_EXISTS=false
- if [ "$FILE_EXISTS" = false ]; then
... # do stuff if file does not exist
else
... # do stuff if file exists
fi
Also we can use the -q flag to let the command stats be silent if that is desired.
- gsutil -q stat $BUCKET_PATH || FILE_EXISTS=false

How to trigger a fail in Travis CI?

One of my test is a simple bash command with an if condition. I want Travis CI to consider a build as failed if the condition is positive.
I try to do it this way (a part of the .travis.yml file):
# ...
script:
- npm run build
- if [[ `git status --porcelain` ]]; then >&2 echo "Fail"; fi
# ...
But when the condition is positive, the message is just printed and the build is considered as successful.
What should I do to make a build failed when the condition is positive?
Just add exit 1; after the echo. More info.
If you just want to assert a condition but continue with testing, the following worked for me:
bash -c 'if [[ `git status --porcelain` ]]; then >&2 echo "Fail"; exit 1; fi'
This will make the build results fail but not terminate it.
Add return 1;
Travis compiles the different commands into a single bash script so exit 1 or travis_terminate 1 will abruptly interrupt the workflow and skip the after_script phase.
For complex commands that you want to make more readable and don't want to move to their own script, you can take advantage of YAML's literal scalar indicator:
script:
- |
if [[ `git status --porcelain` ]]; then
>&2 echo "Fail"
return 1
fi

Writing successful bash scripts

I have the following bash script
aws s3 sync s3://test/ s3://test-li/
if [[ ! $? -eq 0 ]]; then
echo "Unable to copy from test bucket"
exit 1
fi
Is that the right way to run a command and check its return value?
Your code will work, but any good shell script declares what shell should interpret it. You should add a first line of your script (no spaces at the left margin)
#!/bin/bash
(Or whatever is the correct path to bash in your environment). 99% that it is /bin/bash.
Your test is a slightly baroque, there is an -ne operator that you could use, i.e.
if [[ $? -ne 0 ]] ; then
. . .
Or you can go advanced, and let if directly test the return code from you aws command, i.e.
if ! aws sync s3://test/ s3://test-li/ ; then
echo "Unable to copy from test bucket"
exit 1
fi
In this case, you need the ! to have block execute.
You could even capture any output so you can review error msgs etc with
if ! aws sync s3://test/ s3://test-li/ > /tmp/aws_launchlog.txt 2>&1 ; then
echo "Unable to copy from test bucket"
exit 1
fi
IHTH
Unlike most other languages, in shells like bash the if statement is followed by a command, the brackets are fooling you. [[ is actually a shell keyword (as is !), a development of [ which is a shell built-in, also known as the test command.
Use [[ when you wish to do pattern matching, use (( when you wish to do arithmetic comparisons.
if (( some_variable > 0 ))
then
...
fi
If you just want to test if a command worked (returned zero) then any form of brackets are superfluous.
if ! aws s3 sync s3://test/ s3://test-li/
then
# Always send error messages to stderr
echo "Unable to copy from test bucket" >&2
exit 1
fi
Having said that, there are thousands of scripts out there in the wild that do what you have done, and they still work. Unfortunately.
Your way will work. A simpler way is to put the command directly in the if:
if ! aws s3 sync s3://test/ s3://test-li/
then
echo "Unable to copy from test bucket"
exit 1
fi
bash's set -e is quite useful in these cases.
#!/bin/bash
set -e
aws s3 sync s3://test/ s3://test-li/
# if `aws` returns non-zero, the following code will not be executed
echo 'it succeeded!'

use of ? in sh script

When I went through some shell scripts I came across the following line of codes
FILENAME=/home/user/test.tar.gz
tar -zxvf $FILENAME
RES=$?FILENAME
if [ $RES -eq 0 ]; then
echo "TAR extract success
fi
I want to know
What is the use of '?' mark in front of the variable(RES=$?FILENAME).
How to check whether tar extracted successfully
In standard (POSIX-ish) shells, $? is a special parameter. Even Bash's parameter expansion doesn't document an alternative meaning.
In the context, $?FILENAME might expand to 0FILENAME if the previous command succeeded, and perhaps 1FILENAME if it failed.
Since there's a numeric comparison requested (-eq) the value 0FILENAME might convert to 0 and then compare OK. However, on my system (Mac OS X 10.10.5, Bash 3.2.57) attempting:
if [ 0FILE -eq 0 ]; then echo equal; fi
yields the error -bash: [: 0FILE: integer expression expected.
So, adding the FILENAME after the $? is unorthodox at best (or confusing, or even, ultimately, wrong).
By default, the exit status of a function is the exit status returned by the last command in the function. After the function executes, you use the standard $? variable to determine the exit status of the function:
#!/bin/bash
# testing the exit status of a function
my_function() {
echo "trying to display a non-existent file"
ls -l no_file
}
echo "calling the function: "
my_function
echo "The exit status is: $?"
$
$ ./test4
testing the function:
trying to display a non-existent file
ls: badfile: No such file or directory
The exit status is: 1
To check if tar successfully executed or not use
tar xvf "$tar" || exit 1

How can I check if a program exists from a Bash script?

How would I validate that a program exists, in a way that will either return an error and exit, or continue with the script?
It seems like it should be easy, but it's been stumping me.
Answer
POSIX compatible:
command -v <the_command>
Example use:
if ! command -v <the_command> &> /dev/null
then
echo "<the_command> could not be found"
exit
fi
For Bash specific environments:
hash <the_command> # For regular commands. Or...
type <the_command> # To check built-ins and keywords
Explanation
Avoid which. Not only is it an external process you're launching for doing very little (meaning builtins like hash, type or command are way cheaper), you can also rely on the builtins to actually do what you want, while the effects of external commands can easily vary from system to system.
Why care?
Many operating systems have a which that doesn't even set an exit status, meaning the if which foo won't even work there and will always report that foo exists, even if it doesn't (note that some POSIX shells appear to do this for hash too).
Many operating systems make which do custom and evil stuff like change the output or even hook into the package manager.
So, don't use which. Instead use one of these:
command -v foo >/dev/null 2>&1 || { echo >&2 "I require foo but it's not installed. Aborting."; exit 1; }
type foo >/dev/null 2>&1 || { echo >&2 "I require foo but it's not installed. Aborting."; exit 1; }
hash foo 2>/dev/null || { echo >&2 "I require foo but it's not installed. Aborting."; exit 1; }
(Minor side-note: some will suggest 2>&- is the same 2>/dev/null but shorter – this is untrue. 2>&- closes FD 2 which causes an error in the program when it tries to write to stderr, which is very different from successfully writing to it and discarding the output (and dangerous!))
If your hash bang is /bin/sh then you should care about what POSIX says. type and hash's exit codes aren't terribly well defined by POSIX, and hash is seen to exit successfully when the command doesn't exist (haven't seen this with type yet). command's exit status is well defined by POSIX, so that one is probably the safest to use.
If your script uses bash though, POSIX rules don't really matter anymore and both type and hash become perfectly safe to use. type now has a -P to search just the PATH and hash has the side-effect that the command's location will be hashed (for faster lookup next time you use it), which is usually a good thing since you probably check for its existence in order to actually use it.
As a simple example, here's a function that runs gdate if it exists, otherwise date:
gnudate() {
if hash gdate 2>/dev/null; then
gdate "$#"
else
date "$#"
fi
}
Alternative with a complete feature set
You can use scripts-common to reach your need.
To check if something is installed, you can do:
checkBin <the_command> || errorMessage "This tool requires <the_command>. Install it please, and then run this tool again."
The following is a portable way to check whether a command exists in $PATH and is executable:
[ -x "$(command -v foo)" ]
Example:
if ! [ -x "$(command -v git)" ]; then
echo 'Error: git is not installed.' >&2
exit 1
fi
The executable check is needed because bash returns a non-executable file if no executable file with that name is found in $PATH.
Also note that if a non-executable file with the same name as the executable exists earlier in $PATH, dash returns the former, even though the latter would be executed. This is a bug and is in violation of the POSIX standard. [Bug report] [Standard]
Edit: This seems to be fixed as of dash 0.5.11 (Debian 11).
In addition, this will fail if the command you are looking for has been defined as an alias.
I agree with lhunath to discourage use of which, and his solution is perfectly valid for Bash users. However, to be more portable, command -v shall be used instead:
$ command -v foo >/dev/null 2>&1 || { echo "I require foo but it's not installed. Aborting." >&2; exit 1; }
Command command is POSIX compliant. See here for its specification: command - execute a simple command
Note: type is POSIX compliant, but type -P is not.
It depends on whether you want to know whether it exists in one of the directories in the $PATH variable or whether you know the absolute location of it. If you want to know if it is in the $PATH variable, use
if which programname >/dev/null; then
echo exists
else
echo does not exist
fi
otherwise use
if [ -x /path/to/programname ]; then
echo exists
else
echo does not exist
fi
The redirection to /dev/null/ in the first example suppresses the output of the which program.
I have a function defined in my .bashrc that makes this easier.
command_exists () {
type "$1" &> /dev/null ;
}
Here's an example of how it's used (from my .bash_profile.)
if command_exists mvim ; then
export VISUAL="mvim --nofork"
fi
Expanding on #lhunath's and #GregV's answers, here's the code for the people who want to easily put that check inside an if statement:
exists()
{
command -v "$1" >/dev/null 2>&1
}
Here's how to use it:
if exists bash; then
echo 'Bash exists!'
else
echo 'Your system does not have Bash'
fi
Try using:
test -x filename
or
[ -x filename ]
From the Bash manpage under Conditional Expressions:
-x file
True if file exists and is executable.
To use hash, as #lhunath suggests, in a Bash script:
hash foo &> /dev/null
if [ $? -eq 1 ]; then
echo >&2 "foo not found."
fi
This script runs hash and then checks if the exit code of the most recent command, the value stored in $?, is equal to 1. If hash doesn't find foo, the exit code will be 1. If foo is present, the exit code will be 0.
&> /dev/null redirects standard error and standard output from hash so that it doesn't appear onscreen and echo >&2 writes the message to standard error.
Command -v works fine if the POSIX_BUILTINS option is set for the <command> to test for, but it can fail if not. (It has worked for me for years, but I recently ran into one where it didn't work.)
I find the following to be more failproof:
test -x "$(which <command>)"
Since it tests for three things: path, existence and execution permission.
There are a ton of options here, but I was surprised no quick one-liners. This is what I used at the beginning of my scripts:
[[ "$(command -v mvn)" ]] || { echo "mvn is not installed" 1>&2 ; exit 1; }
[[ "$(command -v java)" ]] || { echo "java is not installed" 1>&2 ; exit 1; }
This is based on the selected answer here and another source.
If you check for program existence, you are probably going to run it later anyway. Why not try to run it in the first place?
if foo --version >/dev/null 2>&1; then
echo Found
else
echo Not found
fi
It's a more trustworthy check that the program runs than merely looking at PATH directories and file permissions.
Plus you can get some useful result from your program, such as its version.
Of course the drawbacks are that some programs can be heavy to start and some don't have a --version option to immediately (and successfully) exit.
Check for multiple dependencies and inform status to end users
for cmd in latex pandoc; do
printf '%-10s' "$cmd"
if hash "$cmd" 2>/dev/null; then
echo OK
else
echo missing
fi
done
Sample output:
latex OK
pandoc missing
Adjust the 10 to the maximum command length. It is not automatic, because I don't see a non-verbose POSIX way to do it:
How can I align the columns of a space separated table in Bash?
Check if some apt packages are installed with dpkg -s and install them otherwise.
See: Check if an apt-get package is installed and then install it if it's not on Linux
It was previously mentioned at: How can I check if a program exists from a Bash script?
I never did get the previous answers to work on the box I have access to. For one, type has been installed (doing what more does). So the builtin directive is needed. This command works for me:
if [ `builtin type -p vim` ]; then echo "TRUE"; else echo "FALSE"; fi
I wanted the same question answered but to run within a Makefile.
install:
#if [[ ! -x "$(shell command -v ghead)" ]]; then \
echo 'ghead does not exist. Please install it.'; \
exit -1; \
fi
It could be simpler, just:
#!/usr/bin/env bash
set -x
# if local program 'foo' returns 1 (doesn't exist) then...
if ! type -P foo; then
echo 'crap, no foo'
else
echo 'sweet, we have foo!'
fi
Change foo to vi to get the other condition to fire.
hash foo 2>/dev/null: works with Z shell (Zsh), Bash, Dash and ash.
type -p foo: it appears to work with Z shell, Bash and ash (BusyBox), but not Dash (it interprets -p as an argument).
command -v foo: works with Z shell, Bash, Dash, but not ash (BusyBox) (-ash: command: not found).
Also note that builtin is not available with ash and Dash.
zsh only, but very useful for zsh scripting (e.g. when writing completion scripts):
The zsh/parameter module gives access to, among other things, the internal commands hash table. From man zshmodules:
THE ZSH/PARAMETER MODULE
The zsh/parameter module gives access to some of the internal hash ta‐
bles used by the shell by defining some special parameters.
[...]
commands
This array gives access to the command hash table. The keys are
the names of external commands, the values are the pathnames of
the files that would be executed when the command would be in‐
voked. Setting a key in this array defines a new entry in this
table in the same way as with the hash builtin. Unsetting a key
as in `unset "commands[foo]"' removes the entry for the given
key from the command hash table.
Although it is a loadable module, it seems to be loaded by default, as long as zsh is not used with --emulate.
example:
martin#martin ~ % echo $commands[zsh]
/usr/bin/zsh
To quickly check whether a certain command is available, just check if the key exists in the hash:
if (( ${+commands[zsh]} ))
then
echo "zsh is available"
fi
Note though that the hash will contain any files in $PATH folders, regardless of whether they are executable or not. To be absolutely sure, you have to spend a stat call on that:
if (( ${+commands[zsh]} )) && [[ -x $commands[zsh] ]]
then
echo "zsh is available"
fi
The which command might be useful. man which
It returns 0 if the executable is found and returns 1 if it's not found or not executable:
NAME
which - locate a command
SYNOPSIS
which [-a] filename ...
DESCRIPTION
which returns the pathnames of the files which would
be executed in the current environment, had its
arguments been given as commands in a strictly
POSIX-conformant shell. It does this by searching
the PATH for executable files matching the names
of the arguments.
OPTIONS
-a print all matching pathnames of each argument
EXIT STATUS
0 if all specified commands are
found and executable
1 if one or more specified commands is nonexistent
or not executable
2 if an invalid option is specified
The nice thing about which is that it figures out if the executable is available in the environment that which is run in - it saves a few problems...
Use Bash builtins if you can:
which programname
...
type -P programname
For those interested, none of the methodologies in previous answers work if you wish to detect an installed library. I imagine you are left either with physically checking the path (potentially for header files and such), or something like this (if you are on a Debian-based distribution):
dpkg --status libdb-dev | grep -q not-installed
if [ $? -eq 0 ]; then
apt-get install libdb-dev
fi
As you can see from the above, a "0" answer from the query means the package is not installed. This is a function of "grep" - a "0" means a match was found, a "1" means no match was found.
This will tell according to the location if the program exist or not:
if [ -x /usr/bin/yum ]; then
echo "This is Centos"
fi
I'd say there isn't any portable and 100% reliable way due to dangling aliases. For example:
alias john='ls --color'
alias paul='george -F'
alias george='ls -h'
alias ringo=/
Of course, only the last one is problematic (no offence to Ringo!). But all of them are valid aliases from the point of view of command -v.
In order to reject dangling ones like ringo, we have to parse the output of the shell built-in alias command and recurse into them (command -v isn't a superior to alias here.) There isn't any portable solution for it, and even a Bash-specific solution is rather tedious.
Note that a solution like this will unconditionally reject alias ls='ls -F':
test() { command -v $1 | grep -qv alias }
If you guys/gals can't get the things in answers here to work and are pulling hair out of your back, try to run the same command using bash -c. Just look at this somnambular delirium. This is what really happening when you run $(sub-command):
First. It can give you completely different output.
$ command -v ls
alias ls='ls --color=auto'
$ bash -c "command -v ls"
/bin/ls
Second. It can give you no output at all.
$ command -v nvm
nvm
$ bash -c "command -v nvm"
$ bash -c "nvm --help"
bash: nvm: command not found
#!/bin/bash
a=${apt-cache show program}
if [[ $a == 0 ]]
then
echo "the program doesn't exist"
else
echo "the program exists"
fi
#program is not literal, you can change it to the program's name you want to check
The hash-variant has one pitfall: On the command line you can for example type in
one_folder/process
to have process executed. For this the parent folder of one_folder must be in $PATH. But when you try to hash this command, it will always succeed:
hash one_folder/process; echo $? # will always output '0'
I second the use of "command -v". E.g. like this:
md=$(command -v mkdirhier) ; alias md=${md:=mkdir} # bash
emacs="$(command -v emacs) -nw" || emacs=nano
alias e=$emacs
[[ -z $(command -v jed) ]] && alias jed=$emacs
I had to check if Git was installed as part of deploying our CI server. My final Bash script was as follows (Ubuntu server):
if ! builtin type -p git &>/dev/null; then
sudo apt-get -y install git-core
fi
To mimic Bash's type -P cmd, we can use the POSIX compliant env -i type cmd 1>/dev/null 2>&1.
man env
# "The option '-i' causes env to completely ignore the environment it inherits."
# In other words, there are no aliases or functions to be looked up by the type command.
ls() { echo 'Hello, world!'; }
ls
type ls
env -i type ls
cmd=ls
cmd=lsx
env -i type $cmd 1>/dev/null 2>&1 || { echo "$cmd not found"; exit 1; }
If there isn't any external type command available (as taken for granted here), we can use POSIX compliant env -i sh -c 'type cmd 1>/dev/null 2>&1':
# Portable version of Bash's type -P cmd (without output on stdout)
typep() {
command -p env -i PATH="$PATH" sh -c '
export LC_ALL=C LANG=C
cmd="$1"
cmd="`type "$cmd" 2>/dev/null || { echo "error: command $cmd not found; exiting ..." 1>&2; exit 1; }`"
[ $? != 0 ] && exit 1
case "$cmd" in
*\ /*) exit 0;;
*) printf "%s\n" "error: $cmd" 1>&2; exit 1;;
esac
' _ "$1" || exit 1
}
# Get your standard $PATH value
#PATH="$(command -p getconf PATH)"
typep ls
typep builtin
typep ls-temp
At least on Mac OS X v10.6.8 (Snow Leopard) using Bash 4.2.24(2) command -v ls does not match a moved /bin/ls-temp.
My setup for a Debian server:
I had the problem when multiple packages contained the same name.
For example apache2. So this was my solution:
function _apt_install() {
apt-get install -y $1 > /dev/null
}
function _apt_install_norecommends() {
apt-get install -y --no-install-recommends $1 > /dev/null
}
function _apt_available() {
if [ `apt-cache search $1 | grep -o "$1" | uniq | wc -l` = "1" ]; then
echo "Package is available : $1"
PACKAGE_INSTALL="1"
else
echo "Package $1 is NOT available for install"
echo "We can not continue without this package..."
echo "Exitting now.."
exit 0
fi
}
function _package_install {
_apt_available $1
if [ "${PACKAGE_INSTALL}" = "1" ]; then
if [ "$(dpkg-query -l $1 | tail -n1 | cut -c1-2)" = "ii" ]; then
echo "package is already_installed: $1"
else
echo "installing package : $1, please wait.."
_apt_install $1
sleep 0.5
fi
fi
}
function _package_install_no_recommends {
_apt_available $1
if [ "${PACKAGE_INSTALL}" = "1" ]; then
if [ "$(dpkg-query -l $1 | tail -n1 | cut -c1-2)" = "ii" ]; then
echo "package is already_installed: $1"
else
echo "installing package : $1, please wait.."
_apt_install_norecommends $1
sleep 0.5
fi
fi
}

Resources