Bash Script - Will not completely execute - bash

I am writing a script that will take in 3 outputs and then search all files within a predefined path. However, my grep command seems to be breaking the script with error code 123. I have been staring at it for a while and cannot really seem the error so I was hoping someone could point out my error. Here is the code:
#! /bin/bash -e
#Check if path exists
if [ -z $ARCHIVE ]; then
echo "ARCHIVE NOT SET, PLEASE SET TO PROCEED."
echo "EXITING...."
exit 1
elif [ $# -ne 3 ]; then
echo "Illegal number of arguments"
echo "Please enter the date in yyyy mm dd"
echo "EXITING..."
exit 1
fi
filename=output.txt
#Simple signal handler
signal_handler()
{
echo ""
echo "Process killed or interrupted"
echo "Cleaning up files..."
rm -f out
echo "Finsihed"
exit 1
}
trap 'signal_handler' KILL
trap 'signal_handler' TERM
trap 'signal_handler' INT
echo "line 32"
echo $1 $2 $3
#Search for the TimeStamp field and replace the / and : characters
find $ARCHIVE | xargs grep -l "TimeStamp: $2/$3/$1"
echo "line 35"
fileSize=`wc -c out.txt | cut -f 1 -d ' '`
echo $fileSize
if [ $fileSize -ge 1 ]; then
echo "no"
xargs -n1 basename < $filename
else
echo "NO FILES EXIST"
fi
I added the echo's to know where it was breaking. My program prints out line 32 and the args but never line 35. When I check the exit code I get 123.
Thanks!
Notes:
ARCHIVE is set to a test directory, i.e. /home/'uname'/testDir
$1 $2 $3 == yyyy mm dd (ie a date)
In testDir there are N number of directories. Inside these directories there are data files that have contain data as well as a time tag. The time tag is of the following format: TimeStamp: 02/02/2004 at 20:38:01
The scripts goal is to find all files that have the date tag you are searching for.

Here's a simpler test case that demonstrates your problem:
#!/bin/bash -e
echo "This prints"
true | xargs false
echo "This does not"
The snippet exits with code 123.
The problem is that xargs exits with code 123 if any command fails. When xargs exits with non-zero status, -e causes the script to exit.
The quickest fix is to use || true to effectively ignore xargs' status:
#!/bin/bash -e
echo "This prints"
true | xargs false || true
echo "This now prints too"
The better fix is to not rely on -e, since this option is misleading and unpredictable.

xargs makes the error code 123 when grep returns a nonzero code even just once. Since you're using -e (#!/bin/bash -e), bash would exit the script when one of its commands return a nonzero exit code. Not using -e would allow your code to continue. Just disabling it on that part can be a solution too:
set +e ## Disable
find "$ARCHIVE" | xargs grep -l "TimeStamp: $2/$1/$3" ## If one of the files doesn't match the pattern, `grep` would return a nonzero code.
set -e ## Enable again.
Consider placing your variables around quotes to prevent word splitting as well like "$ARCHIVE".
-d '\n' may also be required if one of your files' filename contain spaces.
find "$ARCHIVE" | xargs -d '\n' grep -l "TimeStamp: $2/$1/$3"

Related

Why same command fails in GitLab CI?

The following command works perfectly on the terminal but the same command fails in GitLab CI.
echo Hello >> foo.txt; cat foo.txt | grep "test"; [[ $? -eq 0 ]] && echo fail || echo success
return is success
but the same command in GitLab CI
$ echo Hello >> foo.txt; cat foo.txt | grep "test"; [[ $? -eq 0 ]] && echo fail || echo success
Cleaning up file based variables
ERROR: Job failed: command terminated with exit code 1
is simply failing. I have no idea why.
echo $SHELL return /bin/bash in both.
Source of the issue
The behavior you observe is pretty standard given the "implied" set -e in a CI context.
To be more precise, your code consists in three compound commands:
echo Hello >> foo.txt
cat foo.txt | grep "test"
[[ $? -eq 0 ]] && echo fail || echo success
And the grep "test" command returns a non-zero exit code (namely, 1). As a result, the script immediately exits and the last line is not executed.
Note that this feature is typical in a CI context, because if some intermediate command fails in a complex script, we'd typically want to get a failure, and avoid running the next commands (which could potentially be "off-topic" given the error).
You can reproduce this locally as well, by writing for example:
bash -e -c "
echo Hello >> foo.txt
cat foo.txt | grep "test"
[[ $? -eq 0 ]] && echo fail || echo success
"
which is mostly equivalent to:
bash -c "
set -e
echo Hello >> foo.txt
cat foo.txt | grep "test"
[[ $? -eq 0 ]] && echo fail || echo success
"
Relevant manual page
For more insight:
on set -e, see man 1 set
on bash -e, see man 1 bash
How to fix the issue?
You should just adopt another phrasing, avoiding [[ $? -eq 0 ]] a-posteriori tests. So the commands that may return a non-zero exit code without meaning failure should be "protected" by some if:
echo Hello >> foo.txt
if cat foo.txt | grep "test"; then
echo fail
false # if ever you want to "trigger a failure manually" at some point.
else
echo success
fi
Also, note that grep "test" foo.txt would be more idiomatic than cat foo.txt | grep "test" − which is precisely an instance of UUOC (useless use of cat).
I have no idea why.
Gitlab executes each command one at a time and checks the exit status of each command. When the exit status is not zero, the job is failed.
There is no test string inside foo.txt, so the command cat foo.txt | grep "test" exits with nonzero exit status. Thus the job is failed.

Bash: How can I check the return value of a command

I am new to bash scripting and want to write a short script, that checks if a certain program is running. If it runs, the script should bring the window to the foreground, if it does not run, the script should start it.
#!/bin/bash
if [ "$(wmctrl -l | grep Wunderlist)" = ""]; then
/opt/google/chrome/google-chrome --profile-directory=Default --app-id=ojcflmmmcfpacggndoaaflkmcoblhnbh
else
wmctrl -a Wunderlist
fi
My comparison is wrong, but I am not even sure what I should google to find a solution. My idea is, that the "$(wmctrl -l | grep Wunderlist)" will return an empty string, if the window does not exist. I get this error when I run the script:
~/bin » sh handle_wunderlist.sh
handle_wunderlist.sh: 3: [: =: argument expected
You need a space before the closing argument, ], of the [ (test) command:
if [ "$(wmctrl -l | grep Wunderlist)" = "" ]; then
....
else
....
fi
As a side note, you have used the shebang as bash but running the script using sh (presumably dash, from the error message).
Replace:
if [ "$(wmctrl -l | grep Wunderlist)" = ""]; then
With:
if ! wmctrl -l | grep -q Wunderlist; then
grep sets its exit condition to true (0) is a match was found and false (1) if it wasn't. Because you want the inverse of that, we placed ! at the beginning of the command to invert the exit code.
Normally, grep will send the matching text to standard out. We don't want that text, we just want to know if there was a match or not. Consequently, we added the -q option to make grep quiet.
Example
To illustrate the use of grep -q in an if statement:
$ if ! echo Wunderlist | grep -q Wunderlist; then echo Not found; else echo Found; fi
Found
$ if ! echo Wunderabcd | grep -q Wunderlist; then echo Not found; else echo Found; fi
Not found

How do I redirect errors to /dev/null in bash?

We run daily Selenium tests to test our website and extensions. I wrote a script (according to this question) to count the number of passed and failed tests. Here is the script:
#!/bin/bash
today=`TZ='Asia/Tel_Aviv' date +"%Y-%m-%d"`
yesterday=`TZ='Asia/Tel_Aviv' date +"%Y-%m-%d" -d "yesterday"`
...
print_test_results()
{
declare -i passed_tests=0
declare -i failed_tests=0
declare -i total_tests=0
log_suffix="_${file_name}.log"
yesterday_logs="${log_prefix}${yesterday}_[1,2]*${log_suffix}"
today_logs="${log_prefix}${today}_0*${log_suffix}"
for temp_file_name in $yesterday_logs $today_logs ; do
total_tests+=1
if grep -q FAILED "$temp_file_name" ; then
failed_tests+=1
elif grep -q OK "$temp_file_name" ; then
passed_tests+=1
else
failed_tests+=1
fi
done
echo "<tr>"
echo "<td>$test_name - $today</td>"
if [ $passed_tests = "0" ]; then
echo "<td>$passed_tests passed</td>"
echo "<td><span style=\"color: red;\">$failed_tests failed</span></td>"
else
echo "<td><span style=\"color: green;\">$passed_tests passed</span></td>"
echo "<td>$failed_tests failed</td>"
fi
echo "<td>$total_tests tests total</td>"
echo "</tr>"
}
file_name="chrome_gmail_1_with_extension_test"
test_name="Chrome Gmail 1 With Extension Test"
print_test_results
...
But the problem is, if the files are not there (in $yesterday_logs $today_logs), I get error messages. How do I redirect these error messages to /dev/null? I want to redirect them to /dev/null from the script, and not from the line calling the script - I want this script to never show error messages about files which don't exist.
Just for the record:
In general, to suppress error messages in bash, use command 2>/dev/null. So in your case you should use grep -q OK 2>/dev/null.
But as your case also shows (from what I read in the comments) this is a risky thing to do, as it cloaks errors you might have in your code. "I want this script to never print error messages" should only be said when one knows all possible error cases which could possibly occur.
Inside your script you can place this line at start:
shopt -s nullglob
This will not match anything if your glob pattern doesn't find any matching file. Otherwise whole glob pattern is returned when you use something like:
for temp_file_name in $yesterday_logs $today_logs; do ... done
Eventually I changed this line to:
for temp_file_name in `ls $yesterday_logs $today_logs 2>/dev/null` ; do
total_tests+=1
if grep -q FAILED "$temp_file_name" ; then
failed_tests+=1
elif grep -q OK "$temp_file_name" ; then
passed_tests+=1
else
failed_tests+=1
fi
done
Then only the ls errors are directed to /dev/null.

trying to test zero length output from command in shell script

I'm sort of a newbie when it comes to shell scripting. What am I doing wrong?
I'm trying to grep a running log file and take action if the grep returns data.
# grep for "success" in the log which will tell us if we were successful
tail -f file.log | grep success > named_pipe &
# send signal to my server to do something
/bin/kill -10 $PID
timeout=0;
while : ; do
OUTPUT=$(cat < named_pipe)
if test [-n] $OUTPUT
then
echo "output is '" $OUTPUT "'"
echo_success
break;
else
timeout=$((timeout+1))
sleep 1
if [ $timeout -ge $SHUTDOWN_TIMEOUT ]; then
echo_failure
break
fi
fi
done
I'm finding that even when "success" is not in the log, test [-n] $OUTPUT returns true. This is because apparently OUTPUT is equal to " ". Why is OUTPUT a single space rather than empty?
How can I fix this?
Here's a smaller test case for your problem:
output=""
if test [-n] $output
then
echo "Why does this happen?"
fi
This happens because when $output is empty or whitespace, it expands to nothing, and you just run test [-n].
test foo is true when foo is non-empty. It doesn't matter that your foo is a flag wrapped in square brackets.
The correct way to do this is without the brackets, and with quotes:
if test -n "$output"
then
...
fi
As for why $OUTPUT is a single space, that's simple: it isn't. echo just writes out its arguments separated as spaces, and you specified multiple arguments. The correct code is echo "output is '$OUTPUT'"

If grep finds what it is looking for do X else Y [duplicate]

This question already has answers here:
How do I use a file grep comparison inside a bash if/else statement?
(5 answers)
Closed 2 years ago.
In this statement I am trying to match if a version ($var2) exist in the path /app/$var1 (application name)
if
find /app/$var1 -maxdepth 1 -type l -o -type d | grep $var2 #results in a nice list where i can manually get a true match.
# if match is found then execute command.
$realcmd "$#"
rc=$?
exit $rc
else
echo "no match, the version you are looking for does not exist"
fi
current code:
this include all my code (not cleaned).
command I run: "./xmodule load firefox/3.6.12"
this version does exit
#!/bin/bash
# hook for some commands
#echo $# #value of sting that is entered after "xmodule"
cmd=$(basename "$0")
#echo "called as $cmd"
if [[ $cmd = "xmodule" ]]
then
realcmd='/app/modules/0/bin/modulecmd tcsh'
# verify parameters
fi
# check if $# contains value "/" to determine if a specific version is requested.
case "$#" in
*/*)
echo "has slash"
var1=$(echo "$#" | grep -Eio '\s\w*') # Gets the aplication name and put it into var1
echo $var1 # is not printed should be "firefox"
var2=$(echo "$#" | grep -o '[^/]*$') # Gets version name and put it into var2
echo $var2
# Checking if version number exist in /app/appname/
if find /app/$var1 -noleaf -maxdepth 1 -type l -o -type d | grep $var2; then
$realcmd "$#"
exit $?
else
echo "no match, the version you are looking for does not exist"
# Should there be an exit here?
fi
;;
*)
echo "doesn't have a slash"
;;
esac
output:
mycomputer [9:55am] [user/Desktop/script] -> ./xmodule load firefox/3.6.12
'has slash
3.6.12
no match, the version you are looking for does not exist
Where there is a blank (above 3.6.1) there should be the application name. I am now realizing that this must be my problem sins the path that it uses i likely just /app.
But I do not think I changed anything in that part of the code.
You can use the entire grep pipeline as the condition of the if statement. Use grep -q to keep it from printing the match it finds (unless you want that printed). I also simplified the exit (there's no need to store $? in a variable if you're just going to use it immediately). Here's the result:
if find "/app/$var1" -maxdepth 1 -type l -o -type d | grep -q "$var2"; then
$realcmd "$#"
exit $?
else
echo "no match, the version you are looking for does not exist"
# Should there be an exit here?
fi
BTW, since you're going to exit immediately after $realcmd, you could use exec $realcmd "$#" to replace the shell with $realcmd instead of running $realcmd as a subprocess.
From the grep manpage:
The exit status is 0 if selected lines are found, and 1 if not found. If an error occurred the exit status is 2.
In other words, immediately following your blah blah | grep $var2, simply check the return value.
Since the exit code for a pipeline is the exit code for the last process in that pipeline, you can use something like:
find /app/$var1 -maxdepth 1 -type l -o -type d | grep $var2 ; greprc=$?
if [[ $greprc -eq 0 ]] ; then
echo Found
else
if [[ $greprc -eq 1 ]] ; then
echo Not found
else
echo Some sort of error
fi
fi

Resources