assigning jq result to makefile variable - bash

I have the following command as a makefile command:
update-env:
echo "{ \"Variables\":${ENV_VALUE}}" > ./new_env.json && \
UPDATE_ENVVARS=$$(jq -s '.[0] as $$a |.[1] as $$b | $$a * $$b' old_env.json new_env.json | jq -c . ) && \
echo "${UPDATE_ENVVARS}"
the ${ENV_VALUE} is taken when
make update-env ENV_VALUE="{\\\"HOST_URL\\\": \\\"https:\\\/\\\/test.com\\\"}"
the file new_env.json is generated properly
when executing jq -s '.[0] as $a |.[1] as $b | $a * $b' old_env.json new_env.json | jq -c . it generates the appropriate compact json result desired.
When running everything in sequence (the assignment and echo for validation), I get an empty result.
My goal for the command is to merge two json output and assign it to the UPDATE_ENVVARS for it to be reused as an input for another command that will accept the json. Per testing, it came out empty when echo, when I execute the jq solo, the merge output is functional.

Only a minor bit of editing was needed:
update-env:
echo '{ "Variables":${ENV_VALUE}}' > ./new_env.json && \
UPDATE_ENVVARS=$$(jq -s '.[0] as $$a |.[1] as $$b | $$a * $$b' old_env.json new_env.json | jq -c . ) && \
echo "$${UPDATE_ENVVARS}"
Note:
We're using single quotes in the first-line echo -- the substitution is performed by make, not by the shell, so single quotes don't suppress it.
We're doubling up the $$ in the last line, so we're expanding the shell variable set in the second line of the recipe, not a make variable that nothing ever set at all.
See this working at https://replit.com/#CharlesDuffy2/RoyalIdolizedProfile
By contrast, if you want to assign to a make variable instead of a shell variable, this question is duplicative of Makefile command substitution problem, and the answer by Ignacio Vazquez-Abrams is appropriate.

Related

Makefile: cannot create a variable with random

I need to generate a random string and save in a variable, with a Makefile.
I wrote
install:
LOCALSTORAGE_ACCESS_TOKEN = echo $RANDOM | md5sum | head -c 20; echo;
echo $(LOCALSTORAGE_ACCESS_TOKEN)
When I launch it with make install, I get
LOCALSTORAGE_ACCESS_TOKEN = echo ANDOM | md5sum | head -c 20; echo;
/bin/sh: 1: LOCALSTORAGE_ACCESS_TOKEN: not found
d41d8cd98f00b204e980
echo
So
missing assigned value to the variable LOCALSTORAGE_ACCESS_TOKEN.
getting every time same result for RANDOM
I tried also
LOCALSTORAGE_ACCESS_TOKEN = echo $$RANDOM | md5sum | head -c 20; echo;
echo $(LOCALSTORAGE_ACCESS_TOKEN)
with same result
LOCALSTORAGE_ACCESS_TOKEN = echo $RANDOM | md5sum | head -c 20; echo;
/bin/sh: 1: LOCALSTORAGE_ACCESS_TOKEN: not found
d41d8cd98f00b204e980
echo
Each line of a recipe is a shell script, run by a different shell. Your first line is syntactically incorrect. You are trying to assign a make variable in a recipe, this is not how make works. Try:
install:
LOCALSTORAGE_ACCESS_TOKEN=$$(echo $$RANDOM | md5sum | head -c 20); \
echo $$LOCALSTORAGE_ACCESS_TOKEN
LOCALSTORAGE_ACCESS_TOKEN is now a shell variable, it is assigned using shell syntax in the first line of the recipe but is still defined in the second line because of the line continuation (the trailing \). Note the use of $$ instead of $ to escape the first expansion by make.
If you would like LOCALSTORAGE_ACCESS_TOKEN to be a make variable available in all lines of all recipes then simply assign it as a real make variable and outside any recipe :
LOCALSTORAGE_ACCESS_TOKEN := $(shell echo $$RANDOM | md5sum | head -c 20)
install:
echo $(LOCALSTORAGE_ACCESS_TOKEN)
This should work:
install:
$(eval LOCALSTORAGE_ACCESS_TOKEN := $(shell /bin/bash -c 'echo $$RANDOM' | md5sum | head -c 20; echo))
echo $(LOCALSTORAGE_ACCESS_TOKEN)
As you also noticed, you need $$RANDOM in order to mask the $ sign in the Makefile
You need to run the command in a bash, because $RANDOM may not work in all shells (did not work for me, so I had to use bash)
You need the $(shell ... so that the variable gets assigned the output of the command (instead of the command text itself)
You need $(eval ... because the code is inside a target, and you can not define variables in targets otherwise

How to use variable with jq cmd in shell

I am facing issue with below commands. I need to use variable but it is returning me null whereas when I hardcode its value, it return me correct response.
Can anybody help me whats the correct way of writing this command?
My intension is to pull value of corresponding key passed as a variable?
temp1="{ \"SSM_DEV_SECRET_KEY\": \"Smkfnkhnb48dh\", \"SSM_DEV_GRAPH_DB\": \"Prod=bolt://neo4j:Grt56#atc.preprod.test.com:7687\", \"SSM_DEV_RDS_DB\": \"sqlite:////var/local/ecosystem_dashboard/config.db\", \"SSM_DEV_SUPPERUSER_USERNAME\": \"admin\", \"SSM_DEV_SUPPERUSER_PASSWORD\": \"9dW6JE8#KH9qiO006\" }"
var_name=SSM_DEV_SECRET_KEY
echo $temp1 | jq -r '.SSM_DEV_SECRET_KEY' <----- return Smkfnkhnb48dh // output
echo $temp1 | jq -r '."$var_name"' <---- return null
echo $temp1 | jq -r --arg var_name "$var_name" '."$var_name"' <---- return null , alternative way
Update: I am adding actual piece of where I am trying to use above fix. My intension is to first read all values which start with SSM_DEV_... and then get there original values from aws than replace it in. one key pair look like this --> SECRET_KEY=$SSM_DEV_SECRET_KEY
temp0="dev"
temp1="DEV"
result1=$(aws secretsmanager get-secret-value --secret-id "xxx-secret-$temp0" | jq '.SecretString')
while IFS= read -r line; do
if [[ "$line" == *"=\$SSM_$temp1"* ]]; then
before=${line%%"="*}
after=${line#*"="}
var_name="${after:1}"
jq -r --arg var_name "$var_name" '.[$var_name]' <<< "$result1"
fi
done < sample_file.txt
Fix: I have solved my issue which was of carriage return character.
Below cmd help me:
var_name=`echo ${after:1} | tr -d '\r'`
jq -r --arg var_name "$var_name" '.[$var_name]' <<< "$result1"
You'll need to use Generic Object Index (.[$var_name]) to let jq know the variable should be seen as a key
The command should look like:
jq -r --arg var_name "$var_name" '.[$var_name]' <<< "$temp1"
Wich will output:
Smkfnkhnb48dh
Note: <<< "$temp1" instead off the echo
Let's look at the following statement:
echo $temp1 | jq -r '."$var_name"' <---- return null
Your problem is actually with the shell quoting and not jq. The single quotes tell the shell not to interpolate (do variable substitution) among other things (like escaping white space and preventing globing). Thus, jq is receiving literally ."$var_name" as it's script - which is not what you want. You simply need to remove the single quotes and you'll be good:
echo $temp1 | jq -r ."$var_name" <---- Does this work?
That said, I would never write my script that way. I would definitely want to include the '.' in the quoted string like this:
echo $temp1 | jq -r ".$var_name" <---- Does this work?
Some would also suggest that you quote "$temp1" as well (typically all variable references should be quoted to protect against white space, but this is not a problem with echo):
echo "$temp1" | jq -r ".$var_name" <---- Does this work?

Set a command to a variable in bash script problem

Trying to run a command as a variable but I am getting strange results
Expected result "1" :
grep -i nosuid /etc/fstab | grep -iq nfs
echo $?
1
Unexpected result as a variable command:
cmd="grep -i nosuid /etc/fstab | grep -iq nfs"
$cmd
echo $?
0
It seems it returns 0 as the command was correct not actual outcome. How to do this better ?
You can only execute exactly one command stored in a variable. The pipe is passed as an argument to the first grep.
Example
$ printArgs() { printf %s\\n "$#"; }
# Two commands. The 1st command has parameters "a" and "b".
# The 2nd command prints stdin from the first command.
$ printArgs a b | cat
a
b
$ cmd='printArgs a b | cat'
# Only one command with parameters "a", "b", "|", and "cat".
$ $cmd
a
b
|
cat
How to do this better?
Don't execute the command using variables.
Use a function.
$ cmd() { grep -i nosuid /etc/fstab | grep -iq nfs; }
$ cmd
$ echo $?
1
Solution to the actual problem
I see three options to your actual problem:
Use a DEBUG trap and the BASH_COMMAND variable inside the trap.
Enable bash's history feature for your script and use the hist command.
Use a function which takes a command string and executes it using eval.
Regarding your comment on the last approach: You only need one function. Something like
execAndLog() {
description="$1"
shift
if eval "$*"; then
info="PASSED: $description: $*"
passed+=("${FUNCNAME[1]}")
else
info="FAILED: $description: $*"
failed+=("${FUNCNAME[1]}")
done
}
You can use this function as follows
execAndLog 'Scanned system' 'grep -i nfs /etc/fstab | grep -iq noexec'
The first argument is the description for the log, the remaining arguments are the command to be executed.
using bash -x or set -x will allow you to see what bash executes:
> cmd="grep -i nosuid /etc/fstab | grep -iq nfs"
> set -x
> $cmd
+ grep -i nosuid /etc/fstab '|' grep -iq nfs
as you can see your pipe | is passed as an argument to the first grep command.

printf command within Bash throwing error

I am using the following command in bash to subtract 2 numbers and print the result. Using bc tool as well
printf '\n"runtime": %s' "$(($a - $b) | bc -l)"
But getting an error
1517359690.775414500: command not found
How should i rewrite my printf command?
If your shell is bash, then this could be:
printf '\n"runtime": %s' "$(bc -l <<<"($a - $b)")"
If instead your shell is sh, then this could be:
printf '\n"runtime": %s' "$(echo "($a - $b)" | bc -l)"
Note that we're invoking a separate command -- echo -- whose output is piped into bc, rather than trying to run the numbers as a command themselves.
However, you shouldn't be using printf to create JSON documents in the first place.
Instead, use jq:
start=5.5; stop=6.10
other_value='this is an example string
it crosses multiple lines, and has "literal quotes" within it'
jq -nc \
--argjson start "$start" \
--argjson stop "$stop" \
--arg other_value "$other_value" \
'{"runtime": ($stop - $start), "other key": $other_value}'
You'll note that the string here is correctly escaped to be included in JSON: " is changed to \", the literal newline is changed to \n, and so forth.

Remove a substring from a bash variable

A bash variable contents are command line arguments, like this:
args="file-1.txt file-2.txt -k file-3.txt -k --some-argument-1 --some-argument-2"
the string -k can appear anywhere in the above string, there are some other arguments that are not -k.
Is it possible to extract all the strings (i.e. file names with all other arguments) except -k from the argument, and assign it to a bash variable?
Using sed
Is is possible to extract all the strings (i.e. file names with all other arguments) except -k from the argument, and assign it to a bash variable?
I am taking that to mean that you want to remove -k while keeping everything else. If that is the case:
$ new=$(echo " $args " | sed -e 's/[[:space:]]-k[[:space:]]/ /g')
$ echo $new
file-1.txt file-2.txt file-3.txt --some-argument-1 --some-argument-2
Using only bash
This question is tagged with bash. Under bash, the use of sed is unnecessary:
$ new=" $args "
$ new=${new// -k / }
$ echo $new
file-1.txt file-2.txt file-3.txt --some-argument-1 --some-argument-2
Piping it to sed should work:
echo $args | sed -e 's/[[:space:]]\-[[:alnum:]\-]*//g'
file-1.txt file-2.txt file-3.txt
and you can assign it to a variable with:
newvar=`echo $args | sed -e 's/[[:space:]]\-[[:alnum:]\-]*//g'`
Command-line arguments in bash should be stored in an array, to allow for arguments that contain characters that need to be quoted.
args=(file-1.txt file-2.txt -k file-3.txt -k --some-argument-1 --some-argument-2)
To extract strings other than -k, just use a for loop to filter them.
newargs=()
for arg in "${args[#]}"; do
[[ $arg = "-k" ]] && newargs+=("$arg")
done

Resources