How to check the stdout of a command in a Bash while conditionality check? - bash

I am trying to check for the output of a command as a breakout condition for while loop in Bash, but it keeps skipping past the block, even while the last echo confirms the value as "DELETE_IN_PROGRESS".
This is what I have:
stackname=cf_test
while [[ $(aws cloudformation describe-stacks --stack-name ${stackname} | jq '.Stacks | .[0] | .StackStatus') == "DELETE_IN_PROGRESS" ]]; do
echo -e " $(aws cloudformation describe-stacks --stack-name ${stackname} | jq '.Stacks | .[0] | .StackStatus'): waiting for current stack to delete before re-deploying..."
sleep 30
done
echo -e $(aws cloudformation describe-stacks --stack-name ${stackname} | jq '.Stacks | .[0] | .StackStatus')
What should I change?
edit:
Adding -x debug flag, gives:
▶ bash -x ~/Downloads/test_script.sh
+ stackname=cf_test
++ aws cloudformation describe-stacks --stack-name cf_test
++ jq '.Stacks | .[0] | .StackStatus'
+ [[ "DELETE_IN_PROGRESS" == \D\E\L\E\T\E\_\I\N\_\P\R\O\G\R\E\S\S ]]
++ aws cloudformation describe-stacks --stack-name cf_test
++ jq '.Stacks | .[0] | .StackStatus'
+ echo -e '"DELETE_IN_PROGRESS"'
"DELETE_IN_PROGRESS"

I would move the comparison inside jq. By default, jq will succeed, regardless of the logical status of the comparison you make.
% jq -n '3 == 3'; echo $?
true
0
% jq -n '3 != 3'; echo $?
false
0
To change this, use the -e option.
% jq -en '3 == 3'; echo $?
true
0
% jq -en '3 != 3'; echo $?
false
1
Doing this eliminates the need for the [[ ... ]] command.
while x=$(aws cloudformation describe-stacks --stack-name ${stackname});
jq -ne --arg x "$x" '$x.Stacks.[0].StackStatus == "DELETE_IN_PROGRESS"'; do
printf '%s: waiting for current stack to delete before re-deploying...\n' "$x"
sleep 30
done
printf '%s\n' "$x"

Your debug log shows the problem (even though the [[ line itself weirdly obscures it):
+ echo -e '"DELETE_IN_PROGRESS"'
The value you are comparing contains literal double quotes. That's why the match fails. Your right-hand side of == contains syntactical double quotes that are not treated as part of the string.
To fix it, use jq -r to output the string without JSON formatting and escaping:
json='{ "foo": "bar" }'
jq '.foo' <<< "$json" # Shows bad 5 character value: "bar"
jq -r '.foo' <<< "$json" # Shows good 3 character value: bar
In your case:
while [[ $(aws cloudformation describe-stacks --stack-name ${stackname} | jq -r '.Stacks | .[0] | .StackStatus') == "DELETE_IN_PROGRESS" ]]; doq -r '.Stacks | .[0] | .StackStatus') == "DELETE_IN_PROGRESS" ]]; do
echo "Still deleting"
sleep 30
done

Related

Writing a comparison BATCH file to verify sha256sum to released code

Trying to write a script that takes 2 arguments ($1 and $2) one to represent the $hash and the $file_name.
I am trying to utilize jq to parse the required data to download and compare PASS or FAIL.
I see to be stuck trying to think this out.
Here is my code
#!/usr/bin/env sh
#
# Sifchain shasum check (revised).
#
# $1
hash_url=$( curl -R -s https://api.github.com/repos/Sifchain/sifnode/releases | jq '.[] | select(.name=="v0.10.0-rc.4")' | jq '.assets[]' | jq 'select(.name=="sifnoded-v0.10.0-rc.4-linux-amd64.zip.sha256")' | jq '.browser_download_url' | xargs $1 $2 )
echo $hash_url
# $2
hash=$( curl -s -L $hash_url | jq'.$2')
file_name=$(curl -R -s https://api.github.com/repos/Sifchain/sifnode/releases | jq '.[] | .name')
#
#
echo $hash | sha256sum
echo $file_name | sha256sum #null why?
echo "\n"
## version of the release $1, and the hash $2
## sha256 <expected_sha_256_sum> <name_of_the_file>
sha256() {
if echo "$1 $2" #| sha256sum -c --quiet
then
echo pass $1 $2
exit 0
else
echo FAIL $1 $2
exit 1
fi
}
# Invoke sha256
sha256 $hash_url $file_name
Ideally this should work for any comparison of hash with correct file, pulling the 2 parameters when the BASH script is invoked.
I can suggest the following corrections/modifications:
#!/bin/bash
#sha file
SHA_URL=$(curl -R -s https://api.github.com/repos/Sifchain/sifnode/releases | \
jq --arg VERSION v0.10.0-rc.4 -r \
'.[] | select(.name==$VERSION) | .assets[] | select(.name |test("\\.sha256$")) | .browser_download_url')
SHA_VALUE=$(curl -s -L $SHA_URL| tr 1 2)
FILENAME=$(curl -R -s https://api.github.com/repos/Sifchain/sifnode/releases | \
jq --arg VERSION v0.10.0-rc.4 -r \
'.[] | select(.name==$VERSION) | .assets[] | select(.content_type =="application/zip") | .name')
#added just for testing, I'm assuming you have the files locally allready
FILEURL=$(curl -R -s https://api.github.com/repos/Sifchain/sifnode/releases | \
jq --arg VERSION v0.10.0-rc.4 -r \
'.[] | select(.name==$VERSION) | .assets[] | select(.content_type =="application/zip") | .browser_download_url')
wget --quiet $FILEURL -O $FILENAME
echo $SHA_VALUE $FILENAME | sha256sum -c --quiet >/dev/null 2>&1
RESULT=$?
if [ $RESULT -eq 0 ]; then
echo -n "PASS "
else
echo -n "FAIL "
fi
echo $SHA_VALUE $FILENAME
exit $RESULT
Notes:
jq
--arg VERSION v0.10.0-rc.4 creates a "variable" to be used in the script
-r - raw output, strings are not quoted
test("\\.sha256$") - regular expresion, used to search for a generic sha256, so you don't have to hardcode the full name
select(.content_type =="application/zip") - I'm assuming that's the file you are searching for
wget is used just for demo purpose, to download the file, I'm assuming you already have the file on your machine
sha256sum -c --quiet >/dev/null 2>&1 - redirecting to /dev/null is necessary because in case of error sha256sum is not quiet

Gitlab CI/CD create and use custom user functions

I tried create like this function in my gitlab config file:
deploy:
stage: dev
services:
- docker:dind
script:
- myFunction () { api_pl_tmp=$(curl -s --header "PRIVATE-TOKEN: $TOKEN_VAR" "https://git.example.ru/api/v4/projects/1/pipelines/latest" | jq .) }
- while myFunction; do
- if [ $(echo $api_pl_tmp | jq -r .status) = "success" ]
- then
- export PROJECT_CURRENT=$($api_pl_tmp | jq -r '{id:.id,sha:.sha[0:8]}' | base64)
- break
- fi
- if [ $(echo $api_pl_tmp | jq -r .status) = "failed" ]
- then
- echo "Error: Frontend can't be deployed!"
- exit 1
- fi
- if [ $(echo $api_pl_tmp | jq -r .status) = "running" ]
- then
- echo "Wait 5 sec... Frontend deploying!"
- sleep 5
- else
- echo Unknow status $(echo $api_pl_tmp | jq -r .status)
- exit 1
- fi
- done
But it's doesn't work and gitlab return me error with message:
This GitLab CI configuration is invalid: jobs:deploy-to-dev:script
config should be a string or a nested array of strings up to 10 levels
deep
How I can fix this problem or maybe I have an error in my custom function?
- separates commands with commands in between. Put your commands as one command, not multiple. Remember they are joined with spaces.
The problem with your script is the colon - see https://gitlab.com/gitlab-org/gitlab-foss/-/issues/30097 .
deploy:
stage: dev
services:
- docker:dind
script:
- "colon=:"
- myFunction () {
api_pl_tmp=$(curl -s --header "PRIVATE-TOKEN$colon $TOKEN_VAR" \
"https$colon//git.example.ru/api/v4/projects/1/pipelines/latest" | jq .);
}
- while myFunction; do
if [ $(echo $api_pl_tmp | jq -r .status) = "success" ]; then
export PROJECT_CURRENT=$($api_pl_tmp | jq -r '{id:.id,sha:.sha[0:8]}' | base64);
break;
fi;
if [ $(echo $api_pl_tmp | jq -r .status) = "failed" ]; then
echo "Error$colon Frontend cant be deployed";
exit 1;
fi;
if [ $(echo $api_pl_tmp | jq -r .status) = "running" ]; then
echo "Wait 5 sec... Frontend deploying!";
sleep 5;
else
echo Unknow status $(echo $api_pl_tmp | jq -r .status);
exit 1;
fi;
done
Also jq .) } is missing a ;. Check your scripts in your own shell one at a time first. Check your scripts with https://shellcheck.net .
Also $($api_pl_tmp is missing an echo and there are a lot of problem with quoting. Use consistent indentation and try to write readable code to minimize typos.

Remove "/usr/bin/sensors" from output of Bash script

So I have this piece of code:
if type -P sensors 2>/dev/null; then
returnString=`sensors`
#amd
if [[ "${returnString/"k10"}" != "${returnString}" ]] ; then
$SENSORS | grep Tdie | $CUT -d ' ' -f 10 | { echo "{"; cat; echo "}"; } | tr -d '\n'
#intel
elif [[ "${returnString/"core"}" != "${returnString}" ]] ; then
fromcore=${returnString##*"coretemp"}
$ECHO ${fromcore##*Physical} | $CUT -d ' ' -f 3 | $CUT -c 2-5 | _parseAndPrint
fi
else
$ECHO "[]" | _parseAndPrint
fi
Its output is:
/usr/bin/sensors
{+31.9°C}
But the desired output is:
{+31.9°C}
Sensors output:
/usr/bin/sensors
nvme-pci-0100
Adapter: PCI adapter
Composite: +33.9°C (low = -0.1°C, high = +74.8°C)
(crit = +79.8°C)
k10temp-pci-00c3
Adapter: PCI adapter
Vcore: 1.12 V
Vsoc: 888.00 mV
Tctl: +31.1°C
Tdie: +31.1°C
Icore: 8.00 A
Isoc: 5.00 A
I would really prefer to leave if type -P sensors 2>/dev/null; then in place, so it can still detect if it is Intel or AMD.
The line is run in a script file, which can be found here.
The problem is the type command. You only need the exit status of this command, but it writes in your case the path of the sensor command to stdout. You should redirect its stdout to /dev/null.
I am just not the smartest right now, should've looked at the rest of the Code and should sleep xD. I could just remove the check,that checks if "sensors" is accessible (at least i think that is what it was used for).
so from
if type -P sensors 2>/dev/null; then
returnString=`sensors`
#amd
if [[ "${returnString/"k10"}" != "${returnString}" ]] ; then
sensors | grep Tdie | $CUT -d ' ' -f 10 | { echo "{"; cat; echo "}"; } | tr -d '\n'
#intel
elif [[ "${returnString/"core"}" != "${returnString}" ]] ; then
fromcore=${returnString##*"coretemp"}
$ECHO ${fromcore##*Physical} | $CUT -d ' ' -f 3 | $CUT -c 2-5 | _parseAndPrint
fi
else
$ECHO "[]" | _parseAndPrint
fi
I made
returnString=`sensors`
#amd
if [[ "${returnString/"k10"}" != "${returnString}" ]] ; then
sensors | sed -nE 's/^Tdie: *([^ ]*)/{\1}/p'
#intel
elif [[ "${returnString/"core"}" != "${returnString}" ]] ; then
fromcore=${returnString##*"coretemp"}
$ECHO ${fromcore##*Physical} | $CUT -d ' ' -f 3 | $CUT -c 2-5 | _parseAndPrint
fi

bash script loop to check if variable contains string - not working

i have a script which copy files from one s3 bucket to local server, do some stuff and upload it to another s3 bucket.
in the original bucket i have few folders, one of them called "OTHER"
i dot want my script to work on this folder
i tried to define a loop to check if the path string does not contains the string "OTHER" only then to continue to other commands but for some reason it is not working.
what am i doing wrong ?
#!/bin/bash
shopt -s extglob
gcs3='s3://gc-reporting-pud-production/splunk_printer_log_files/'
gcs3ls=$((aws s3 ls 's3://gc-reporting-pud-production/splunk_printer_log_files/' --recursive) | sed 's/^.*\(splunk_printer.*\)/\1/g'| tr -s ' ' | tr ' ' '_')
ssyss3=s3://ssyssplunk
tokenFile=/splunkData/GCLogs/tokenFile.txt
nextToken=$((aws s3api list-objects-v2 --bucket "gc-reporting-pud-production" --prefix splunk_printer_log_files/ --max-items 5) |grep -o 'NEXTTOKEN.*' |awk -F " " '{print $2}')
newToken=$( tail -n 1 /splunkData/GCLogs/tokenFile.txt )
waterMark=$(aws s3api list-objects-v2 --bucket "gc-reporting-pud-production" --prefix splunk_printer_log_files/ --max-items 5 --starting-token
$newToken|sed 's/^.*\(splunk_printer.*zip\).*$/\1/'|sed '1d'|sed '$d')
while true; do
for j in $waterMark ; do
echo $j
if [ "$j" != *"OTHER"* ]; then
gcRegion=$(echo $j | awk -F'/' '{print $2}')
echo "gcRegion:"$gcRegion
if [ "$gcRegion" != "OTHER" ]; then
gcTech=$(echo $j | awk -F'/' '{print $3}')
echo "GCTech:"$gcTech
gcPrinterFamily=$(echo $j | awk -F'/' '{print $4}')
echo "gcPrinterFamily:" $gcPrinterFamily
gcPrinterType=$(echo $j | awk -F'/' '{print $5}')
echo "gcPrinterType:" $gcPrinterType
gcPrinterName=$(echo $j| awk -F'/' '{print $6}')
echo "gcPrinterName:" $gcPrinterName
gcFileName=$(echo $j| awk -F'/' '{print $7}'| awk -F'.zip' '{print $1}')
echo "gcFileName:" $gcFileName
cd /splunkData/GCLogs
dir="/splunkData/GCLogs/$gcRegion/$gcTech/$gcPrinterFamily/$gcPrinterType/$gcPrinterName"
echo "dir:"$dir
mkdir -p $dir
aws s3 sync $gcs3$gcRegion/$gcTech/$gcPrinterFamily/$gcPrinterType/$gcPrinterName/ $dir
find $dir -name '*.zip' -exec sh -c 'unzip -o -d "${0%.*}" "$0"' '{}' ';'
aws s3 cp $dir $ssyss3/$gcRegion/$gcTech/$gcPrinterFamily/$gcPrinterType/$gcPrinterName/ --recursive --exclude "*.zip"
newToken=$( tail -n 1 /splunkData/GCLogs/tokenFile.txt )
nextToken=$(aws s3api list-objects-v2 --bucket "gc-reporting-pud-production" --prefix splunk_printer_log_files/ --max-items 5 --starting-token $newToken |grep -o 'NEXTTOKEN.*' |awk -F " " '{print $2}')
waterMark=$(aws s3api list-objects-v2 --bucket "gc-reporting-pud-production" --prefix splunk_printer_log_files/ --max-items 5 --starting-token $newToken|sed 's/^.*\(splunk_printer.*zip\).*$/\1/'|sed '1d'|sed '$d')
echo "$nextToken" > "$tokenFile"
fi
fi
done
done
You need to use the double-bracket conditional command to turn == and != into pattern matching operators:
if [[ "$j" != *"OTHER"* ]]; then
# ^^ ^^
Or use case
case "$j" in
*OTHER*) ... ;;
*) echo "this is like an `else` block" ;;
esac
Paste your code into https://www.shellcheck.net/ for other things to fix.
I think glenn jackman was on the right path. Try this:
if [[ "$j" != *OTHER* ]]; then
The [[ ]] is required for pattern string matching (and you have to remove the " ). The case statement is also a good idea. You can abandon the shell test altogether and use grep as follows:
if
grep -q '.*OTHER.*' <<< "$j" 2>/dev/null
then
...
fi
Here's a check of the [[ ]]:
$ echo $j
abOTHERc
$ [[ "$j" == *OTHER* ]]
$ echo $?
0
As per BenjaminW., the quotes around $j in [[ ]] are unnecessary. However, the quotes around *OTHER* do make a big difference. See below:
$ j="OTHER THINGS"
$ [[ $j == "*OTHER*" ]] ; echo "$j" matches '"*OTHER*"': $?
OTHER THINGS matches "*OTHER*": 1
$ [[ $j == *OTHER* ]] ; echo "$j" matches '*OTHER*': $?
OTHER THINGS matches *OTHER*: 0

How to replace or escape <tab> characters with \t in bash script and being able to output single quotes?

In the goal to create a file from a one line (bash) command, the goal is to output the contents of any text file - in this example a bash script - and wrap each line inside a command that is able to output that same line when pasted in a Terminal window.
Example source input file:
Line 1
Line 2
Line 3
Example desired output:
echo 'Line 1';echo 'Line 2';echo 'Line 3';
Note: whether printf, echo or another command is used to create the output, doesn't matter as long as the source is human readable.
One hurdle were the single quotes, that would not be recreated. Therefore use the form $'string', which are treated specially. The word expands to string, with backslash-escaped characters replaced as specified by the ANSI C standard.
Another requirement is to re-create tab characters from the old file in the new file. Therefore the wish is to replace <\tab> characters with \t.
Our tries to do this with sed or tr fail. How to replace tabs with their escape \t counterpart and still being able to output lines with original quotes?
Input file /Library/Scripts/BootRepairMount.sh contains:
$ cat /Library/Scripts/BootRepairMount.sh
#!/bin/bash
sleep 18
for OUTPUT in $(diskutil list | grep ': Apple_HFS' | awk '{ print $NF }')
do
if [[ -z $(df -lnh | grep /dev/$OUTPUT) ]]; then
echo "$OUTPUT is not mounted, repair and mount"
diskutil repairVolume $OUTPUT
diskutil mount $OUTPUT
fi
done
The best shell one line command we could create is:
$ oldifs=$IFS;printf '\n';printf '{';while IFS= read -r p;do [[ "$p" == *"'"* ]] && echo -n "echo $'$p';" || echo -n "echo '$p';"; done < /Library/Scripts/BootRepairMount.sh | tr '\t' '\134\164';printf '}';printf '\n\n';IFS=$oldifs
Which returns this faulty output:
{echo '#!/bin/bash';echo 'sleep 18';echo $'for OUTPUT in $(diskutil list | grep ': Apple_HFS' | awk '{ print $NF }')';echo 'do';echo '\if [[ -z $(df -lnh | grep /dev/$OUTPUT) ]]; then';echo '\\echo "$OUTPUT is not mounted, repair and mount"';echo '\\diskutil repairVolume $OUTPUT';echo '\\diskutil mount $OUTPUT';echo '\fi';echo 'done';}
Desired output is:
{echo '#!/bin/bash';echo 'sleep 18';echo $'for OUTPUT in $(diskutil list | grep ': Apple_HFS' | awk '{ print $NF }')';echo 'do';echo '\tif [[ -z $(df -lnh | grep /dev/$OUTPUT) ]]; then';echo '\t\techo "$OUTPUT is not mounted, repair and mount"';echo '\t\tdiskutil repairVolume $OUTPUT';echo '\t\tdiskutil mount $OUTPUT';echo '\tfi';echo 'done';}
Bash one line command version 2
$ oldifs=$IFS;printf '\n';printf '{';while IFS= read -r p;do [[ "$p" == *"'"* ]] && printf 'printf $'\''%q'\'';' "$p" || printf 'printf '\''%q'\'';' "$p"; done < /Library/Scripts/BootRepairMount.sh;printf '}';printf '\n\n';IFS=$oldifs
returns output that is heavy escaped:
{printf '\#\!/bin/bash';printf 'sleep\ 18';printf $'for\ OUTPUT\ in\ \$\(diskutil\ list\ \|\ grep\ \':\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ Apple_HFS\'\ \|\ awk\ \'\{\ print\ \$NF\ \}\'\)';printf 'do';printf '$'\tif [[ -z $(df -lnh | grep /dev/$OUTPUT) ]]; then'';printf '$'\t\techo "$OUTPUT is not mounted, repair and mount"'';printf '$'\t\tdiskutil repairVolume $OUTPUT'';printf '$'\t\tdiskutil mount $OUTPUT'';printf '$'\tfi'';printf 'done';}
that never gets unescaped back to its original values in Mac OS X 10.7.5.
printf '\#\!/bin/bash';
outputs:
\#\!/bin/bash
As well as:
echo -e '\#\!/bin/bash'
does output the unescaped value
\#\!/bin/bash
-e is not a valid command switch for the Mac OS X 10.7.5 echo command, according to its man page.
bash's builtin command printf has %q format code that handles this:
printf '\n{ '; while IFS= read -r p; do printf "echo %q; " "$p"; done < /Library/Scripts/BootRepairMount.sh; printf '}\n\n'
Unfortunately, it doesn't always choose quoting/escaping modes that're easy to read. Specifically, it tends to prefer escaping individual metacharacters (e.g. spaces) rather than enclosing them in quotes:
{ echo \#\!/bin/bash; echo sleep\ 18; echo for\ OUTPUT\ in\ \$(diskutil\ list\ \|\ grep\ \':\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ Apple_HFS\'\ \|\ awk\ \'{\ print\ \$NF\ }\'); echo do; echo $'\tif [[ -z $(df -lnh | grep /dev/$OUTPUT) ]]; then'; echo $'\t\techo "$OUTPUT is not mounted, repair and mount"'; echo $'\t\tdiskutil repairVolume $OUTPUT'; echo $'\t\tdiskutil mount $OUTPUT'; echo $'\tfi'; echo done; }
If I understand right you want paste one long line to the Terminal.app and want get the "source code" of original script. So, need a script what will generate the one-line script.
Maybe a bit unusual solution, but it is easy and simple.
here is the test script called test.sh (instead of your BootReapirMount.sh)
for i in {1..10}
do
date
done
Here is the generator script mkecho.sh
#!/bin/bash
[[ ! -f "$1" ]] && echo "Need filename" && exit 1
asc=$(gzip < "$1" | base64)
echo "base64 -D <<<'$asc'| gzip -d"
Now, run:
bash mkecho.sh test.sh
you will get the next:
base64 -D <<<'H4sIAASwqFEAA0vLL1LIVMjMU6g21NMzNKjlSsnn4kxJLEkFMvJSuQBZFmY0HwAAAA=='| gzip -d
If you copy and paste the above into the terminal, it will will display the original test.sh
Variant2
If you want directly execute the script, you should modify the mkecho.sh to the next mkeval.sh
#!/bin/bash
[[ ! -f "$1" ]] && echo "Need filename" && exit 1
asc=$(gzip < "$1" | base64)
echo -n 'eval "$(base64 -D <<<"'
echo -n $asc
echo -n '" | gzip -d)"'
echo
When run
bash mkeval.sh test.sh
will get
eval "$(base64 -D <<<"H4sIAASwqFEAA0vLL1LIVMjMU6g21NMzNKjlSsnn4kxJLEkFMvJSuQBZFmY0HwAAAA==" | gzip -d)"
and finally when you copy and paste it into the terminal, you run the test.sh and will get:
Fri May 31 16:25:08 CEST 2013
... 8 lined deleted...
Fri May 31 16:25:08 CEST 2013
Warning: because the script is NOT TESTED for every possible conditions, nor for redirects and so on - I really don't recommending using the eval verision.
sed 's/\\t/\\/g'
$ echo 'ffsd \tif [[ -z $' | sed 's/\\t/\\/g'
ffsd \if [[ -z $

Resources