I want to extract a matched regex from this file:
abc
de
{my_pattern.global} # want to extract my_pattern.global
# without curly brackets
123
and assign it to a variable in a shell script:
#!/bin/bash
l_config_file="my_file.cfg"
l_extracted_pattern=""
l_match_pattern="(?<={).+\.global(?=})"
l_my_dir=$(pwd)
echo "grep -oP '$l_match_pattern' $l_my_dir/$l_config_file"
echo "debug 1 - exit code: $?"
grep -oP '$l_match_pattern' $l_my_dir/$l_config_file
echo "debug 2 - exit code: $?"
sh -c "grep -oP '$l_match_pattern' $l_my_dir/$l_config_file"
echo "debug 3 - exit code: $?"
$l_extracted_pattern = "$(sh -c "grep -oP '$l_match_pattern' $l_my_dir/$l_config_file")"
echo "debug 4 - exit code: $?"
echo $l_extracted_pattern
Output:
grep -oP '(?<={).+\.global(?=})' /tmp/my_file.cfg
debug 1 - exit code: 0
debug 2 - exit code: 1
my_pattern.global
debug 3 - exit code: 0
./sto.sh: line 14: =: command not found.
debug 4 - exit code: 127
As you can see, the grep command works well (when executed via sh -c) but fails when trying to assign the output to the variable $l_extracted_pattern with exit code 127. That means the shell doesn't recognise the command. I suspect the regex is cause of the trouble here, but couldn't figure out what in particular. What's going wrong?
Even though I assigned it before already :
l_extracted_pattern=""
and tried to overwrite it later:
$l_extracted_pattern = "$(sh -c "grep -oP '$l_match_pattern' $l_my_dir/$l_config_file")"
That was a mistake. Apparently no variable assignment in bash may contain the $ before the variable name - even not when instantiated already before. Changed it to:
l_extracted_pattern = "$(sh -c "grep -oP '$l_match_pattern' $l_my_dir/$l_config_file")"
Related
The following command works perfectly on the terminal but the same command fails in GitLab CI.
echo Hello >> foo.txt; cat foo.txt | grep "test"; [[ $? -eq 0 ]] && echo fail || echo success
return is success
but the same command in GitLab CI
$ echo Hello >> foo.txt; cat foo.txt | grep "test"; [[ $? -eq 0 ]] && echo fail || echo success
Cleaning up file based variables
ERROR: Job failed: command terminated with exit code 1
is simply failing. I have no idea why.
echo $SHELL return /bin/bash in both.
Source of the issue
The behavior you observe is pretty standard given the "implied" set -e in a CI context.
To be more precise, your code consists in three compound commands:
echo Hello >> foo.txt
cat foo.txt | grep "test"
[[ $? -eq 0 ]] && echo fail || echo success
And the grep "test" command returns a non-zero exit code (namely, 1). As a result, the script immediately exits and the last line is not executed.
Note that this feature is typical in a CI context, because if some intermediate command fails in a complex script, we'd typically want to get a failure, and avoid running the next commands (which could potentially be "off-topic" given the error).
You can reproduce this locally as well, by writing for example:
bash -e -c "
echo Hello >> foo.txt
cat foo.txt | grep "test"
[[ $? -eq 0 ]] && echo fail || echo success
"
which is mostly equivalent to:
bash -c "
set -e
echo Hello >> foo.txt
cat foo.txt | grep "test"
[[ $? -eq 0 ]] && echo fail || echo success
"
Relevant manual page
For more insight:
on set -e, see man 1 set
on bash -e, see man 1 bash
How to fix the issue?
You should just adopt another phrasing, avoiding [[ $? -eq 0 ]] a-posteriori tests. So the commands that may return a non-zero exit code without meaning failure should be "protected" by some if:
echo Hello >> foo.txt
if cat foo.txt | grep "test"; then
echo fail
false # if ever you want to "trigger a failure manually" at some point.
else
echo success
fi
Also, note that grep "test" foo.txt would be more idiomatic than cat foo.txt | grep "test" − which is precisely an instance of UUOC (useless use of cat).
I have no idea why.
Gitlab executes each command one at a time and checks the exit status of each command. When the exit status is not zero, the job is failed.
There is no test string inside foo.txt, so the command cat foo.txt | grep "test" exits with nonzero exit status. Thus the job is failed.
I'm trying to assign the exit command of the last command in a pipeline to a variable but it is not giving the expected result. Essentially I'm grepping a variable to see if it ends with '-SNAPSHOT' or not. So if I try this:
export PROJECT_VERSION=1.0.0
echo ${PROJECT_VERSION} | grep \\-SNAPSHOT$
And then do echo $? the result is 1 as expected (no match found).
If I then add the echo $? to the end of the pipe:
echo ${PROJECT_VERSION} | grep \\-SNAPSHOT$ | echo $?
The result then becomes 0.
How can I get the exit result of the grep \\-SNAPSHOT so that I can assign it to a variable?
The exit status is in $?.
echo ${PROJECT_VERSION} | grep \\-SNAPSHOT$
variable="$?"
I would like to be able to create a bash function that can read the exit code of the command before the pipe. I'm not sure it is possible to have access to that.
echo "1" | grep 2 returns a 1 status code
echo "1" | grep 1 returns a 0 status code
Now I would like to add a third command to read the status, with a pipe:
echo "1" | grep 2 | echo $? will echo "0", even if the status code is 1.
I know I can use the echo "1" | grep 2 && echo "0" || echo "1", but I would prefer to write it using a pipe.
Is they anyway to do that (it would be even better if it was working on most shells, like bash, sh, and zsh)
You're going to have to get the exit status before the next stage of the pipeline. Something like
exec 3> debug.txt
{ echo "1"; echo "$?" >&3; } | long | command | here
You can't (easily) encapsulate this in a function, since it would require passing a properly quoted string and executing it via eval:
debug () {
eval "$#"
echo $? >&3
}
# It looks easy in this example, but it won't take long to find
# an example that breaks it.
debug echo 1 | long | command | here
You have to write the exit status to a different file descriptor, otherwise it will interfere with the output sent to the next command in the pipeline.
In bash you can do this with the PIPESTATUS variable
echo "1" | grep 1
echo ${PIPESTATUS[0]} # returns 0
echo "1" | grep 2
echo ${PIPESTATUS[0]} # returns 0
echo "1" | grep 2
echo ${PIPESTATUS[1]} # returns 1
I am writing a script that will take in 3 outputs and then search all files within a predefined path. However, my grep command seems to be breaking the script with error code 123. I have been staring at it for a while and cannot really seem the error so I was hoping someone could point out my error. Here is the code:
#! /bin/bash -e
#Check if path exists
if [ -z $ARCHIVE ]; then
echo "ARCHIVE NOT SET, PLEASE SET TO PROCEED."
echo "EXITING...."
exit 1
elif [ $# -ne 3 ]; then
echo "Illegal number of arguments"
echo "Please enter the date in yyyy mm dd"
echo "EXITING..."
exit 1
fi
filename=output.txt
#Simple signal handler
signal_handler()
{
echo ""
echo "Process killed or interrupted"
echo "Cleaning up files..."
rm -f out
echo "Finsihed"
exit 1
}
trap 'signal_handler' KILL
trap 'signal_handler' TERM
trap 'signal_handler' INT
echo "line 32"
echo $1 $2 $3
#Search for the TimeStamp field and replace the / and : characters
find $ARCHIVE | xargs grep -l "TimeStamp: $2/$3/$1"
echo "line 35"
fileSize=`wc -c out.txt | cut -f 1 -d ' '`
echo $fileSize
if [ $fileSize -ge 1 ]; then
echo "no"
xargs -n1 basename < $filename
else
echo "NO FILES EXIST"
fi
I added the echo's to know where it was breaking. My program prints out line 32 and the args but never line 35. When I check the exit code I get 123.
Thanks!
Notes:
ARCHIVE is set to a test directory, i.e. /home/'uname'/testDir
$1 $2 $3 == yyyy mm dd (ie a date)
In testDir there are N number of directories. Inside these directories there are data files that have contain data as well as a time tag. The time tag is of the following format: TimeStamp: 02/02/2004 at 20:38:01
The scripts goal is to find all files that have the date tag you are searching for.
Here's a simpler test case that demonstrates your problem:
#!/bin/bash -e
echo "This prints"
true | xargs false
echo "This does not"
The snippet exits with code 123.
The problem is that xargs exits with code 123 if any command fails. When xargs exits with non-zero status, -e causes the script to exit.
The quickest fix is to use || true to effectively ignore xargs' status:
#!/bin/bash -e
echo "This prints"
true | xargs false || true
echo "This now prints too"
The better fix is to not rely on -e, since this option is misleading and unpredictable.
xargs makes the error code 123 when grep returns a nonzero code even just once. Since you're using -e (#!/bin/bash -e), bash would exit the script when one of its commands return a nonzero exit code. Not using -e would allow your code to continue. Just disabling it on that part can be a solution too:
set +e ## Disable
find "$ARCHIVE" | xargs grep -l "TimeStamp: $2/$1/$3" ## If one of the files doesn't match the pattern, `grep` would return a nonzero code.
set -e ## Enable again.
Consider placing your variables around quotes to prevent word splitting as well like "$ARCHIVE".
-d '\n' may also be required if one of your files' filename contain spaces.
find "$ARCHIVE" | xargs -d '\n' grep -l "TimeStamp: $2/$1/$3"
I have this code:
error(){
time=$( date +"%T %F" )
echo "Start : ${time} : ${1}" 1>&2
result=$( eval "${1}" )
if [ `echo "${PIPESTATUS[#]}" | tr -s ' ' + | bc` -ne 0 ]; then
echo "command ${1} return ERROR" 1>&2
exit
else
if [ "${2}" != "silent" ]; then
echo "${result}"
fi
fi
}
I start testing command:
error "ifconfig | wc -l" "silent"
Start : 14:41:53 2014-02-19 : ifconfig | wc -l
error "ifconfiggg | wc -l" "silent"
Start : 14:43:13 2014-02-19 : ifconfiggg | wc -l
./install-streamserver.sh: line 42: ifconfiggg: command not found
But, I expect a different result. Example:
error "ifconfig" "silent"
Start : 14:44:52 2014-02-19 : ifconfig
Start : 14:45:40 2014-02-19 : ifconfiggg
./install-streamserver.sh: line 42: ifconfiggg: command not found
command ifconfiggg return ERROR (<<<<<<<<<<<< This message)
I don't have it, because when bash runs a command with eval, as in
eval "ifconfiggg | wc -l"
the $PIPESTATUS[#] array just contains "0" instead of the expected "1 0".
How can I fix this?
The eval starts a new shell context which has a separate PIPESTATUS[] array. The lifetime of this context ends when the eval ends. You can communicate this array to the parent context through assigning to a variable, say, PIPE as follows:
$ eval 'ifconfiggg | wc -l; PIPE=${PIPESTATUS[#]}'
bash: ifconfiggg: command not found
0
$ echo $PIPE
127 0
Note the single quotes to prevent ${PIPESTATUS[#]} from expanding too early.
Wrapping this in yet another result=$(...) does not work, since this creates yet another shell context. I suggest instead something along the lines of
eval "${1}; "'PIPE=${PIPESTATUS[#]}' >result.out 2>result.err
# do something with $PIPE here
# do something with result.out or result.err here
Note the use of both double quotes followed by single quotes.
Thanks #Jens for posting this information. I noticed for
eval "${1}; "'PIPE=${PIPESTATUS[#]}' >result.out 2>result.err
that it's better to use parentheses around array PIPESTATUS. Otherwise it seems to be interpreted as string and the complete result is in ${PIPESTATUS[0]} only. So
eval "${1}; "'PIPE=(${PIPESTATUS[#]})' >result.out 2>result.err
is working as expected.