Here is the piece of code I use in my Jenkinsfile.
The issue is located on the sha256sum check.
It does not work, however the two sha256 are equal.
stage('Validate') {
docker.image('alpine:latest').inside("-u root") {
sh '''
apk add make bash curl && \
export terraform_ver=0.12.7 && \
export terraform_url=https://releases.hashicorp.com/terraform/${terraform_ver}/terraform_${terraform_ver}_linux_amd64.zip && \
export terraform_sha256=$(curl https://releases.hashicorp.com/terraform/\${terraform_ver}/terraform_\${terraform_ver}_SHA256SUMS | grep linux_amd64 | awk \'{print \$1}\') && \
curl -Ls --fail -o /tmp/terraform.zip ${terraform_url} && \
sha256sum /tmp/terraform.zip
echo "${terraform_sha256} /tmp/terraform.zip" | sha256sum -c && \
unzip /tmp/terraform.zip -d /usr/local/bin && \
make test-validate
'''
}
}
The result in Jenkins is:
+ sha256sum /tmp/terraform.zip
a0fa11217325f76bf1b4f53b0f7a6efb1be1826826ef8024f2f45e60187925e7 /tmp/terraform.zip
+ echo 'a0fa11217325f76bf1b4f53b0f7a6efb1be1826826ef8024f2f45e60187925e7 /tmp/terraform.zip'
+ sha256sum -c
sha256sum: WARNING: 1 of 1 computed checksums did NOT match
I tried to replace
echo "${terraform_sha256} /tmp/terraform.zip" | sha256sum -c
by
echo \"${terraform_sha256} /tmp/terraform.zip\" | sha256sum -c
but it does not work.
I assume the issue is located on this double quote...
The sha256 are equal so I expect the sha256sum check does not fail.
sha256sum expects two spaces between the checksum and the file name. You have to echo this exactly in order for sha256sum -c to succeed.
The double quotes should not be escapesd. Putting quotes around the string ensures that e.g. spaces between the arguments are preserved. See also When to wrap quotes around a shell variable?
(The output from sh -x which you see in the debug output puts single quotes around strings so you can see exactly how they are demarcated.)
Related
I have the following command as a makefile command:
update-env:
echo "{ \"Variables\":${ENV_VALUE}}" > ./new_env.json && \
UPDATE_ENVVARS=$$(jq -s '.[0] as $$a |.[1] as $$b | $$a * $$b' old_env.json new_env.json | jq -c . ) && \
echo "${UPDATE_ENVVARS}"
the ${ENV_VALUE} is taken when
make update-env ENV_VALUE="{\\\"HOST_URL\\\": \\\"https:\\\/\\\/test.com\\\"}"
the file new_env.json is generated properly
when executing jq -s '.[0] as $a |.[1] as $b | $a * $b' old_env.json new_env.json | jq -c . it generates the appropriate compact json result desired.
When running everything in sequence (the assignment and echo for validation), I get an empty result.
My goal for the command is to merge two json output and assign it to the UPDATE_ENVVARS for it to be reused as an input for another command that will accept the json. Per testing, it came out empty when echo, when I execute the jq solo, the merge output is functional.
Only a minor bit of editing was needed:
update-env:
echo '{ "Variables":${ENV_VALUE}}' > ./new_env.json && \
UPDATE_ENVVARS=$$(jq -s '.[0] as $$a |.[1] as $$b | $$a * $$b' old_env.json new_env.json | jq -c . ) && \
echo "$${UPDATE_ENVVARS}"
Note:
We're using single quotes in the first-line echo -- the substitution is performed by make, not by the shell, so single quotes don't suppress it.
We're doubling up the $$ in the last line, so we're expanding the shell variable set in the second line of the recipe, not a make variable that nothing ever set at all.
See this working at https://replit.com/#CharlesDuffy2/RoyalIdolizedProfile
By contrast, if you want to assign to a make variable instead of a shell variable, this question is duplicative of Makefile command substitution problem, and the answer by Ignacio Vazquez-Abrams is appropriate.
Example here:
gitrepo=$(jq -r '.gitrepo' 0.json)
releasetag=$(curl --silent ""https://api.github.com/repos/\"$gitrepo\""/releases/latest" | grep '"tag_name":' | sed -E 's/.*"([^"]+)".*/\1/')
echo "$releasetag"
Used \" to escape characters.
0.json:
{
"type": "github-releases",
"gitrepo": "ipfs/go-ipfs"
}
How to put $gitrepo to work inside $releasetag?
Thanks in advance!
Bash variables expand inside quoted " strings.
gitrepo="$(jq -r '.gitrepo' 0.json)"
releasetag="$(
curl --silent "https://api.github.com/repos/$gitrepo/releases/latest" \
| grep '"tag_name":' | sed -E 's/.*"([^"]+)".*/\1/'
)"
echo "$releasetag"
Btw, as you are using jq to extract .gitrepo from 0.json, you could also use it in the exact same way to extract .tag_name from curl's output (instead of using grep and sed) like so:
gitrepo="$(jq -r '.gitrepo' 0.json)"
releasetag="$(
curl --silent "https://api.github.com/repos/$gitrepo/releases/latest" \
| jq -r '.tag_name'
)"
echo "$releasetag"
And to simplify it even further (depending on your use case), just write:
curl --silent "https://api.github.com/repos/$(jq -r '.gitrepo' 0.json)/releases/latest" \
| jq -r '.tag_name'
I need to escape single quotes in a variable.
ssh_command 'file=$(hostname)_server-setup_$(date +%Y-%m-%d).tar.gz && cd /var && tar -zcvf $file ini | wc -l | xargs printf \'Num files: %d, File: $file\''
I can surround the variable with double quotes but then the internal variables will be evaluated when the variable is declared, and that's not what I want.
ssh_command "file=$(hostname)_server-setup_$(date +%Y-%m-%d).tar.gz && cd /var && tar -zcvf $file ini | wc -l | xargs printf 'Num files: %d, File: $file'"
update
Have now come up with this, but then $file is just printed as $file
ssh_command (){
ssh root#host $1
}
ssh_command 'file=$(hostname)_server-setup_$(date +%Y-%m-%d).tar.gz && cd /var && tar -zcvf $file ini | wc -l | xargs printf '"'"'Num files: %d, File: $file'"'"
output
Num files: 61, File: $file
When sending over a complicated command over SSH (using quotes, dollar signs, semi-colons), then I prefer to base64 encode/decode it. Here I've made base64_ssh.bash:
#!/bin/bash
script64=$(cat script.txt | base64 -w 0)
ssh 127.0.0.1 "echo $script64 | base64 -d | bash"
In the example above, simply put the command you would like to run on the remote server in script.txt, and then run the bash script.
This does require one extra file, but not having to escape quotes or other special characters makes this a better solution in my opinion.
This will also work with creating functions too.
The way it works, is it converts the command into a base64 encoded string which has a simpler character set (Wikipedia base64), and these characters will never need to be escaped. Then once it's on the other side, it is decoded and then piped through to the bash interpreter.
To make this work with your example above, put the following into script.txt:
file=$(hostname)_server-setup_$(date +%Y-%m-%d).tar.gz && cd /var && tar -zcvf $file ini | wc -l | xargs printf "Num files: %d, File: $file"
For this not too big command I would not overcomplicate it and stick to the original double quotes, and escape just the $ which should not be expanded locally.
ssh_command "file=\$(hostname)_server-setup_$(date +%Y-%m-%d).tar.gz && cd /var && tar -zcvf \$file ini | wc -l | xargs printf 'Num files: %d, File: \$file'"
This is my target, where i want to run a loop for each element in the list variable.
The problem is the loop runs but the test variable value is passed as empty
list = mlflow emr
common=$(someDir)/common
.PHONY:build
build:
for var in $(list); do \
cd ${common}; \
test=$(git diff --name-only --diff-filter=AM master | grep ^$(var)/); \
if [ "$(test)" != "" ]; then \
echo "condition met"; \
else \
echo "It is Not Setup"; \
fi \
done
Error:
bash-5.0# sudo make build n=1
for var in mlflow emr; do \
cd /mak/epa-toolchain/common; \
test=; \
if [ "" != "" ]; then \
echo "condition met"; \
else \
echo "It is Not Setup"; \
fi \
done
It is Not Setup
It is Not Setup
The $ is a special character to make: it introduces a make variable reference. So this:
$(git diff --name-only --diff-filter=AM master | grep ^$(var)/)
is not a shell $(...) command, it's a make variable with a very strange name. Wherever you want the shell to see $ you have to escape it as $$:
$$(git diff --name-only --diff-filter=AM master | grep ^$$var/)
(note you have to change $(var) to $$var because the former is a reference to a make variable var, but you are looping in the shell which sets the shell variable var).
Ditto this:
if [ "$(test)" != "" ]; then \
has to be:
if [ "$$test" != "" ]; then \
because test is a shell variable you just assigned, not a make variable.
In the goal to create a file from a one line (bash) command, the goal is to output the contents of any text file - in this example a bash script - and wrap each line inside a command that is able to output that same line when pasted in a Terminal window.
Example source input file:
Line 1
Line 2
Line 3
Example desired output:
echo 'Line 1';echo 'Line 2';echo 'Line 3';
Note: whether printf, echo or another command is used to create the output, doesn't matter as long as the source is human readable.
One hurdle were the single quotes, that would not be recreated. Therefore use the form $'string', which are treated specially. The word expands to string, with backslash-escaped characters replaced as specified by the ANSI C standard.
Another requirement is to re-create tab characters from the old file in the new file. Therefore the wish is to replace <\tab> characters with \t.
Our tries to do this with sed or tr fail. How to replace tabs with their escape \t counterpart and still being able to output lines with original quotes?
Input file /Library/Scripts/BootRepairMount.sh contains:
$ cat /Library/Scripts/BootRepairMount.sh
#!/bin/bash
sleep 18
for OUTPUT in $(diskutil list | grep ': Apple_HFS' | awk '{ print $NF }')
do
if [[ -z $(df -lnh | grep /dev/$OUTPUT) ]]; then
echo "$OUTPUT is not mounted, repair and mount"
diskutil repairVolume $OUTPUT
diskutil mount $OUTPUT
fi
done
The best shell one line command we could create is:
$ oldifs=$IFS;printf '\n';printf '{';while IFS= read -r p;do [[ "$p" == *"'"* ]] && echo -n "echo $'$p';" || echo -n "echo '$p';"; done < /Library/Scripts/BootRepairMount.sh | tr '\t' '\134\164';printf '}';printf '\n\n';IFS=$oldifs
Which returns this faulty output:
{echo '#!/bin/bash';echo 'sleep 18';echo $'for OUTPUT in $(diskutil list | grep ': Apple_HFS' | awk '{ print $NF }')';echo 'do';echo '\if [[ -z $(df -lnh | grep /dev/$OUTPUT) ]]; then';echo '\\echo "$OUTPUT is not mounted, repair and mount"';echo '\\diskutil repairVolume $OUTPUT';echo '\\diskutil mount $OUTPUT';echo '\fi';echo 'done';}
Desired output is:
{echo '#!/bin/bash';echo 'sleep 18';echo $'for OUTPUT in $(diskutil list | grep ': Apple_HFS' | awk '{ print $NF }')';echo 'do';echo '\tif [[ -z $(df -lnh | grep /dev/$OUTPUT) ]]; then';echo '\t\techo "$OUTPUT is not mounted, repair and mount"';echo '\t\tdiskutil repairVolume $OUTPUT';echo '\t\tdiskutil mount $OUTPUT';echo '\tfi';echo 'done';}
Bash one line command version 2
$ oldifs=$IFS;printf '\n';printf '{';while IFS= read -r p;do [[ "$p" == *"'"* ]] && printf 'printf $'\''%q'\'';' "$p" || printf 'printf '\''%q'\'';' "$p"; done < /Library/Scripts/BootRepairMount.sh;printf '}';printf '\n\n';IFS=$oldifs
returns output that is heavy escaped:
{printf '\#\!/bin/bash';printf 'sleep\ 18';printf $'for\ OUTPUT\ in\ \$\(diskutil\ list\ \|\ grep\ \':\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ Apple_HFS\'\ \|\ awk\ \'\{\ print\ \$NF\ \}\'\)';printf 'do';printf '$'\tif [[ -z $(df -lnh | grep /dev/$OUTPUT) ]]; then'';printf '$'\t\techo "$OUTPUT is not mounted, repair and mount"'';printf '$'\t\tdiskutil repairVolume $OUTPUT'';printf '$'\t\tdiskutil mount $OUTPUT'';printf '$'\tfi'';printf 'done';}
that never gets unescaped back to its original values in Mac OS X 10.7.5.
printf '\#\!/bin/bash';
outputs:
\#\!/bin/bash
As well as:
echo -e '\#\!/bin/bash'
does output the unescaped value
\#\!/bin/bash
-e is not a valid command switch for the Mac OS X 10.7.5 echo command, according to its man page.
bash's builtin command printf has %q format code that handles this:
printf '\n{ '; while IFS= read -r p; do printf "echo %q; " "$p"; done < /Library/Scripts/BootRepairMount.sh; printf '}\n\n'
Unfortunately, it doesn't always choose quoting/escaping modes that're easy to read. Specifically, it tends to prefer escaping individual metacharacters (e.g. spaces) rather than enclosing them in quotes:
{ echo \#\!/bin/bash; echo sleep\ 18; echo for\ OUTPUT\ in\ \$(diskutil\ list\ \|\ grep\ \':\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ Apple_HFS\'\ \|\ awk\ \'{\ print\ \$NF\ }\'); echo do; echo $'\tif [[ -z $(df -lnh | grep /dev/$OUTPUT) ]]; then'; echo $'\t\techo "$OUTPUT is not mounted, repair and mount"'; echo $'\t\tdiskutil repairVolume $OUTPUT'; echo $'\t\tdiskutil mount $OUTPUT'; echo $'\tfi'; echo done; }
If I understand right you want paste one long line to the Terminal.app and want get the "source code" of original script. So, need a script what will generate the one-line script.
Maybe a bit unusual solution, but it is easy and simple.
here is the test script called test.sh (instead of your BootReapirMount.sh)
for i in {1..10}
do
date
done
Here is the generator script mkecho.sh
#!/bin/bash
[[ ! -f "$1" ]] && echo "Need filename" && exit 1
asc=$(gzip < "$1" | base64)
echo "base64 -D <<<'$asc'| gzip -d"
Now, run:
bash mkecho.sh test.sh
you will get the next:
base64 -D <<<'H4sIAASwqFEAA0vLL1LIVMjMU6g21NMzNKjlSsnn4kxJLEkFMvJSuQBZFmY0HwAAAA=='| gzip -d
If you copy and paste the above into the terminal, it will will display the original test.sh
Variant2
If you want directly execute the script, you should modify the mkecho.sh to the next mkeval.sh
#!/bin/bash
[[ ! -f "$1" ]] && echo "Need filename" && exit 1
asc=$(gzip < "$1" | base64)
echo -n 'eval "$(base64 -D <<<"'
echo -n $asc
echo -n '" | gzip -d)"'
echo
When run
bash mkeval.sh test.sh
will get
eval "$(base64 -D <<<"H4sIAASwqFEAA0vLL1LIVMjMU6g21NMzNKjlSsnn4kxJLEkFMvJSuQBZFmY0HwAAAA==" | gzip -d)"
and finally when you copy and paste it into the terminal, you run the test.sh and will get:
Fri May 31 16:25:08 CEST 2013
... 8 lined deleted...
Fri May 31 16:25:08 CEST 2013
Warning: because the script is NOT TESTED for every possible conditions, nor for redirects and so on - I really don't recommending using the eval verision.
sed 's/\\t/\\/g'
$ echo 'ffsd \tif [[ -z $' | sed 's/\\t/\\/g'
ffsd \if [[ -z $