So, I'm trying to export an environment variable that comes from an api that returns json values. Would like to use jq to just do a one liner, but if the values have spaces I cannot get it working
Trying without surrounding the value in quotes
/app/src $ $(echo '{"params":[{ "Name":"KEY","Value":"value with space"}]}' | jq
-r '.params[] | "export " + .Name + "=" + .Value')
/app/src $ printenv KEY
value
/app/src $
Next, I try wrapping the value in quotes
/app/src $ $(echo '{"params":[{ "Name":"KEY","Value":"value with space"}]}' | jq
-r '.params[] | "export " + .Name + "=\"" + .Value + "\""')
sh: export: space": bad variable name
/app/src $
For all of the below, I'm assuming that:
json='{"params":[{ "Name":"KEY","Value":"value with space"}]}'
It can be done, but ONLY IF YOU TRUST YOUR INPUT.
A solution that uses eval might look like:
eval "$(jq -r '.params[] | "export \(.Name | #sh)=\(.Value | #sh)"' <<<"$json")"
The #sh builtin in jq escapes content to be eval-safe in bash, and the eval invocation then ensures that the content goes through all parsing stages (so literal quotes in the data emitted by jq become syntactic).
However, all solutions that allow arbitrary shell variables to be assigned have innate security problems, as the ability to set variables like PATH, LD_LIBRARY_PATH, LD_PRELOAD and the like can be leveraged into arbitrary code execution.
Better form is to generate a NUL-delimited key/value list...
build_kv_nsv() {
jq -j '.params[] |
((.Name | gsub("\u0000"; "")),
"\u0000",
(.Value | gsub("\u0000"; "")),
"\u0000")'
}
...and either populate an associative array...
declare -A content_received=( )
while IFS= read -r -d '' name && IFS= read -r -d '' value; do
content_received[$name]=$value
done < <(build_kv_nsv <<<"$json")
# print the value of the populated associative array
declare -p content_received
...or to use a namespace that's prefixed to guarantee safety.
while IFS= read -r -d '' name && IFS= read -r -d '' value; do
printf -v "received_$name" %s "$value" && export "received_$name"
done < <(build_kv_nsv <<<"$json")
# print names and values of our variables that start with received_
declare -p "${!received_#}" >&2
If the values are known not to contain (raw) newlines, and if you have access to mapfile, it may be worthwhile considering using it, e.g.
$ json='{"params":[{ "Name":"KEY","Value":"value with space"}]}'
$ mapfile -t KEY < <( jq -r '.params[] | .Value' <<< "$json" )
$ echo N=${#KEY[#]}
N=1
If the values might contain (raw) newlines, then you'd need a version of mapfile with the -d option, which could be used as illustrated below:
$ json='{"params":[{ "Name":"KEY1","Value":"value with space"}, { "Name":"KEY2","Value":"value with \n newline"}]}'
$ mapfile -d $'\0' KEY < <( jq -r -j '.params[] | .Value + "\u0000"' <<< "$json" )
$ echo N=${#KEY[#]}
N=2
Related
I want to read into bash associative array the content of one yaml file, which is a simple key value mapping.
Example map.yaml
---
a: "2"
b: "3"
api_key: "somekey:thatcancontainany#chara$$ter"
the key can contain any characters excluding space
the value can contain any characters without limitations $!:=#etc
What will always be constant is the separator between key and value is :
the script proceses.sh
#!/usr/bin/env bash
declare -A map
# how to read here into map variable, from map.yml file
#map=populatesomehowfrommap.yaml
for key in "${!map[#]}"
do
echo "key : $key"
echo "value: ${map[$key]}"
done
I tried to play around with yq tool, similar to json tool jq but did not have success yet.
With the following limitations:
simple YAML key: "value" in single lines
keys cannot contain :
values are always wrapped in "
#!/usr/bin/env bash
declare -A map
regex='^([^:]+):[[:space:]]+"(.*)"[[:space:]]*$'
while IFS='' read -r line
do
if [[ $line =~ $regex ]]
then
printf -v map["${BASH_REMATCH[1]}"] '%b' "${BASH_REMATCH[2]}"
else
echo "skipping: $line" 1>&2
fi
done < map.yaml
Update
Here's a robust solution using yq, which would be simpler if the builtin #tsv filter implemented the lossless TSV escaping rules instead of the CSV ones.
#!/usr/bin/env bash
declare -A map
while IFS=$'\t' read key value
do
printf -v map["$key"] '%b' "$value"
done < <(
yq e '
to_entries | .[] |
[
(.key | sub("\\","\\") | sub("\n","\n") | sub("\r","\r") | sub("\t","\t")),
(.value | sub("\\","\\") | sub("\n","\n") | sub("\r","\r") | sub("\t","\t"))
] |
join(" ")
' map.yaml
)
note: the join needs a literal Tab
One way, is by letting yq output each key/value pair on a single line, in the following syntax:
key#value
Then we can use bash's IFS to split those values.
The # is just an example and can be replaced with any single char
This works, but please note the following limitations:
It does not expect nested values, only a flat list`
The field seperator (# in the example) does not exist in the YAML key/value's
#!/bin/bash
declare -A arr
while IFS="#" read -r key value
do
arr[$key]="$value"
done < <(yq e 'to_entries | .[] | (.key + "#" + .value)' input.yaml)
for key in "${!arr[#]}"
do
echo "key : $key"
echo "value: ${arr[$key]}"
done
$ cat input.yaml
---
a: "bar"
b: "foo"
$
$
$ ./script.sh
key : a
value: bar
key : b
value: foo
$
I used #Fravadona s answer so will mark it as answer
After some modification to my use case, what worked for me looks like:
DEFS_PATH="definitions"
declare -A ssmMap
for file in ${DEFS_PATH}/*
do
filename=$(basename -- "$file")
projectName="${filename%.*}"
regex='^([^:]+):[[:space:]]*"(.*)"[[:space:]]*$'
while IFS='' read -r line
do
if [[ $line =~ $regex ]]
then
value="${BASH_REMATCH[2]}"
value=${value//"{{ ssm_env }}"/$INFRA_ENV}
value=${value//"{{ ssm_reg }}"/$SSM_REGION}
value=${value//"{{ projectName }}"/$projectName}
printf -v ssmMap["${BASH_REMATCH[1]}"] '%b' "$value"
else
echo "skipping: $line" 1>&2
fi
done < "$file"
done
Basically in real use case I have one folder where yaml definitions are located. I iterate over all of them to form the associative array ssmMap
I am facing issue with below commands. I need to use variable but it is returning me null whereas when I hardcode its value, it return me correct response.
Can anybody help me whats the correct way of writing this command?
My intension is to pull value of corresponding key passed as a variable?
temp1="{ \"SSM_DEV_SECRET_KEY\": \"Smkfnkhnb48dh\", \"SSM_DEV_GRAPH_DB\": \"Prod=bolt://neo4j:Grt56#atc.preprod.test.com:7687\", \"SSM_DEV_RDS_DB\": \"sqlite:////var/local/ecosystem_dashboard/config.db\", \"SSM_DEV_SUPPERUSER_USERNAME\": \"admin\", \"SSM_DEV_SUPPERUSER_PASSWORD\": \"9dW6JE8#KH9qiO006\" }"
var_name=SSM_DEV_SECRET_KEY
echo $temp1 | jq -r '.SSM_DEV_SECRET_KEY' <----- return Smkfnkhnb48dh // output
echo $temp1 | jq -r '."$var_name"' <---- return null
echo $temp1 | jq -r --arg var_name "$var_name" '."$var_name"' <---- return null , alternative way
Update: I am adding actual piece of where I am trying to use above fix. My intension is to first read all values which start with SSM_DEV_... and then get there original values from aws than replace it in. one key pair look like this --> SECRET_KEY=$SSM_DEV_SECRET_KEY
temp0="dev"
temp1="DEV"
result1=$(aws secretsmanager get-secret-value --secret-id "xxx-secret-$temp0" | jq '.SecretString')
while IFS= read -r line; do
if [[ "$line" == *"=\$SSM_$temp1"* ]]; then
before=${line%%"="*}
after=${line#*"="}
var_name="${after:1}"
jq -r --arg var_name "$var_name" '.[$var_name]' <<< "$result1"
fi
done < sample_file.txt
Fix: I have solved my issue which was of carriage return character.
Below cmd help me:
var_name=`echo ${after:1} | tr -d '\r'`
jq -r --arg var_name "$var_name" '.[$var_name]' <<< "$result1"
You'll need to use Generic Object Index (.[$var_name]) to let jq know the variable should be seen as a key
The command should look like:
jq -r --arg var_name "$var_name" '.[$var_name]' <<< "$temp1"
Wich will output:
Smkfnkhnb48dh
Note: <<< "$temp1" instead off the echo
Let's look at the following statement:
echo $temp1 | jq -r '."$var_name"' <---- return null
Your problem is actually with the shell quoting and not jq. The single quotes tell the shell not to interpolate (do variable substitution) among other things (like escaping white space and preventing globing). Thus, jq is receiving literally ."$var_name" as it's script - which is not what you want. You simply need to remove the single quotes and you'll be good:
echo $temp1 | jq -r ."$var_name" <---- Does this work?
That said, I would never write my script that way. I would definitely want to include the '.' in the quoted string like this:
echo $temp1 | jq -r ".$var_name" <---- Does this work?
Some would also suggest that you quote "$temp1" as well (typically all variable references should be quoted to protect against white space, but this is not a problem with echo):
echo "$temp1" | jq -r ".$var_name" <---- Does this work?
Basically I have a bash script that at one point makes an API call and a cert and key are generated and returned in json. I pipe it to jq and can select either the cert or the key and store it in a variable.
Something like this:
CERT=$(API call | jq -r '.certificate')
or
KEY=$(API call | jq -r '.key')
I want to store each in its own variable but I can't make the call twice because it will generate a new cert/key.
I know that I can just store both in a file and then manipulate after to accomplish my task but I am curious if jq offers a direct way to selectively store each value in its own variable?
You could store the original JSON in a variable (instead of a file), and then extract the key and cert from that:
apiResult=$(API call)
cert=$(jq -r '.certificate' <<<"$apiResult")
key=$(jq -r '.key' <<<"$apiResult")
Notes: I recommend using lower- or mixed-case variables, to avoid accidental conflicts with the many all-caps names with special meanings. Also, <<< is a bashism, and won't work in all other shells; if you need this be portable to e.g. dash, use something like key=$(printf '%s\n' "$apiResult" | jq -r '.key').
Unless there are NUL characters in the strings, there should be no need to call jq more than once, or to serialize the data.
In the simplest case, you could proceed along the following lines:
{ IFS= read -r certificate
IFS= read -r key
echo "certificate=$certificate"
echo "key=$key"
} < <(API call | jq -r '.certificate, .key')
If the values do not contain NUL but might contain newline characters,
then you could use NUL as the delimiter. For the sake of variety,
we could also use a while loop:
while IFS= read -r -d $'\0' certificate
do
IFS= read -r -d $'\0' key
echo "certificate=$certificate"
echo "key=$key"
done < <(API call | jq -rj '[.certificate, .key] | join("\u0000")')
Conversely ...
If the values of interest might contain literal NUL values ("\u0000"), then the question seems to be problematic, as bash variables in effect cannot contain literal NULs.
If any of the values of interest might contain literal NUL values, then here are two strategies for extracting the "raw" string equivalents into separate files:
Save the JSON output in a (temporary) file, and invoke jq -r once per value of interest in the obvious way.
Set up a bash pipeline starting with:
API call | jq -r '.certificate, .key | #base64`
and continuing with a loop in which each line is decoded, e.g. using base64 --decode or jq's #base64d.
The second strategy might make sense if the API call produces a very large JSON document.
If neither of the values contain a newline, you can easily use read:
$ read cert key < <( echo '{"certificate": "foo", "key": "bar"}' | jq -r .certificate,.key | tr \\n ' ')
$ echo $cert
foo
$ echo $key
bar
or:
$ { read cert; read key; } << EOF
> $( echo '{"certificate": "foo", "key": "bar"}' | jq -r .certificate,.key )
> EOF
$ echo $cert:$key
foo:bar
The certificate almost certainly will contain newlines, so you many want to serialize that data (eg, base64 encode it). But you're probably better off using Gordon Davisson's approach.
Say you created a program that output shell commands.
$ data_source | jq -r '#sh "certificate=\( .certificate )\nkey=\( .key )"'
certificate='the certificate'
key='the key'
Then, all you would need to do is evaluate them.
eval "$( data_source | jq -r '#sh "certificate=\( .certificate )\nkey=\( .key )"' )"
If you were dealing with many variables, you could use the following:
eval "$(
data_source |
jq -r '
{ "certificate": "certificate", "key": "key" } as $map |
( $map | keys_unsorted[] ) as $shell_var |
$shell_var + #sh "=\( .[$map[$shell_var]] )"
'
)"
You could even use paths.
eval "$(
data_source |
jq -r '
{ "certificate": [ "certificate" ], "key": [ "key" ] } as $map |
( $map | keys_unsorted[] ) as $shell_var |
$shell_var + #sh "=\( getpath($map[$shell_var]) )"
'
)"
My shell-fu is at a below-beginner level. I have a file that contains some lines that happen to be the names of environment variables.
e.g.
ENV_VAR_A
ENV_VAR_B
...
What I want to do is use this file to generate a JSON string containing the names and current values of the named variables using jq like this:
jq -n --arg arg1 "$ENV_VAR_A" --arg arg2 "$ENV_VAR_B" '{ENV_VAR_A:$arg1,ENV_VAR_B:$arg2}'
# if ENV_VAR_A=one and ENV_VAR_B=two then the preceding command would output
# {"ENV_VAR_A":"one","ENV_VAR_B":"two"}
I'm trying to create the jq command through a shell script and I have no idea what I'm doing :(
Short and sweet (if you have jq 1.5 or higher):
jq -Rn '[inputs | {(.): env[.]}] | add' tmp.txt
What you want here is an indirect reference. Those can be done with ${!varname}. As a trivial example limited to exactly two lines:
# read arg1_varname and arg2_varname from the first two lines of file.txt
{ read -r arg1_varname; read -r arg2_varname; } <file.txt
# pass the variable named by the contents of arg1_varname as $arg1 in jq
# and the variable named by the contents of arg2_varname as $arg2 in jq
jq -n --arg arg1_name "$arg1_varname" --arg arg1_value "${!arg1_varname}" \
--arg arg2_name "$arg2_varname" --arg arg2_value "${!arg2_varname}" \
'{($arg1_name):$arg1_value, ($arg2_name):$arg2_value}'
To support an arbitrary number of key/value pairs, consider instead something like:
# Transform into NUL-separate key=value pairs (same format as /proc/*/environ)
while IFS= read -r name; do # for each variable named in file.txt
printf '%s=%s\0' "$name" "${!name}" # print its name and value, and a NUL
done \
<file.txt \
| jq -Rs 'split("\u0000") # split on those NULs
| [.[] | select(.) # ignore any empty strings
| capture("^(?<name>[^=]+)=(?<val>.*)$") # break into k/v pairs
| {(.name): .val}] # make each a JSON map
| add # combine those maps
'
jq can look up the values from the environment itself.
$ export A=1
$ export B=2
$ cat tmp.txt
A
B
$ jq -Rn '[inputs] | map({key: ., value: $ENV[.]}) | from_entries' tmp.txt
{
"A": "1",
"B": "2"
}
A few notes on how this works:
-R reads raw text, rather than trying to parse the input as JSON
-n prevents jq from reading input itself.
inputs reads all the input explicitly, allowing an array of names to be built.
map creates an array of objects with key and value as the keys; . is the current array input (a variable name), and $ENV[.] is the value of the environment variable whose name is the current array input.
from_entries finally coalesces all those {"key": ..., "value": ...} objects into a single object.
Try something along the following script in bash:
# array of arguments to pass to jq
jqarg=()
# the script to pass to jq
jqscript=""
# just a number for the arg$num for indexing
# suggestion: just index using variable names...
num=1
# for each variable name from the input
while IFS= read -r varname; do
# just an assertion - check if the variable is not empty
# the syntax ${!var} is indirect reference
# you could do more here, ex. see if such variable exists
# or if $varname is a valid variable name
if [[ -z "${!varname}" ]]; then
echo "ERROR: variable $varname has empty value!" >&2
exit 50
fi
# add the arguments to jqarg array
jqarg+=(--arg "arg$num" "${!varname}")
# update jqscript
# if jqscript is not empty, add a comma on the end
if [[ -n "$jqscript" ]]; then
jqscript+=","
fi
# add the ENV_VAR_A:$arg<number>
jqscript+="$varname:\$arg$num"
# update number - one up!
num=$((num + 1))
# the syntax of while read loop is that input file is on the end
done < input_file_with_variable_names.txt
# finally execute jq
# note the `{` and `}` in `{$jqscript}` are concious
jq -n "${jqarg[#]}" "{$jqscript}"
Just something that hopefully will give you a easier start with your journey in bash.
I guess I would do something unreadable with xargs like:
< input_file_with_variable_names.txt xargs -d$'\n' -n1 bash -c '
printf %s\\0%s\\0%s\\0 --arg "$1" "${!1}"
' -- |
xargs -0 sh -c 'jq -n "$#" "$0"' "{$(
sed 's/\(.*\)/\1: $\1 /' input_file_with_variable_names.txt |
paste -sd,
)}"
I have two strings of same number of substrings divided by a delimiter.
I need to create key-value pairs from substrings.
Short example:
Input:
firstString='00011010:00011101:00100001'
secondString='H:K:O'
delimiter=':'
Desired result:
${translateMap['00011010']} -> 'H'
${translateMap['00011101']} -> 'K'
${translateMap['00100001']} -> 'O'
So, I wrote:
IFS="$delimiter" read -ra fromArray <<< "$firstString"
IFS="$delimiter" read -ra toArray <<< "$secondString"
declare -A translateMap
curIndex=0
for from in "${fromArray[#]}"; do
translateMap["$from"]="${toArray[curIndex]}"
((curIndex++))
done
Is there any way to create the associative array directly from 2 strings without the unneeded arrays and loop? Something like:
IFS="$delimiter" read -rA translateMap["$(read -ra <<< "$firstString")"] <<< "$secondString"
Is it possible?
A (somewhat convoluted) variation on #accdias's answer of assigning the values via the declare -A command, but will need a bit of explanation for each step ...
First we need to break the 2 variables into separate lines for each item:
$ echo "${firstString}" | tr "${delimiter}" '\n'
00011010
00011101
00100001
$ echo "${secondString}" | tr "${delimiter}" '\n'
H
K
O
What's nice about this is that we can now process these 2 sets of key/value pairs as separate files.
NOTE: For the rest off this discussion I'm going to replace "${delimiter}" with ':' to make this a tad bit (but not much) less convoluted.
Next we make use of the paste command to merge our 2 'files' into a single file; we'll also designate ']' as the delimiter between key/value mappings:
$ paste -d ']' <(echo "${firstString}" | tr ':' '\n') <(echo "${secondString}" | tr ':' '\n')
00011010]H
00011101]K
00100001]O
We'll now run these results through a couple sed patterns to build our array assignments:
$ paste -d ']' <(echo "${firstString}" | tr ':' '\n') <(echo "${secondString}" | tr ':' '\n') | sed 's/^/[/g;s/]/]=/g'
[00011010]=H
[00011101]=K
[00100001]=O
What we'd like to do now is use this output in the typeset -A command but unfortunately we need to build the entire command and then eval it:
$ evalstring="typeset -A kv=( "$(paste -d ']' <(echo "${firstString}" | tr ':' '\n') <(echo "${secondString}" | tr ':' '\n') | sed 's/^/[/g;s/]/]=/g')" )"
$ echo "$evalstring"
typeset -A kv=( [00011010]=H
[00011101]=K
[00100001]=O )
If we want to remove the carriage returns and put on a single line we append another tr at the output from the sed command:
$ evalstring="typeset -A kv=( "$(paste -d ']' <(echo "${firstString}" | tr ':' '\n') <(echo "${secondString}" | tr ':' '\n') | sed 's/^/[/g;s/]/]=/g' | tr '\n' ' ')" )"
$ cat "${evalstring}"
typeset -A kv=( [00011010]=H [00011101]=K [00100001]=O )
At this point we can eval our auto-generated typeset -A command:
$ eval "${evalstring}"
And now loop through our array displaying the key/value pairs:
$ for i in ${!kv[#]}; do echo "kv[${i}] = ${kv[${i}]}"; done
kv[00011010] = H
kv[00100001] = O
kv[00011101] = K
Hey, I did say this would be a bit convoluted! :-)
It is probably not what you expect, but this works:
key_string="A:B:C:D"
val_string="1:2:3:4"
declare -A map
while [ -n "$key_string" ] && [ -n "$val_string" ]; do
IFS=: read -r key key_string <<<"$key_string"
IFS=: read -r val val_string <<<"$val_string"
map[$key]="$val"
done
for key in "${!map[#]}"; do echo "$key => ${map[$key]}"; done
It uses recursion in the read function to reassign the string value.
The downside of this method is that it destroys the original strings. The while-loop checks constantly if both strings have a non-zero length.
Next to the above in pure bash, you could any command to generate the associative array. See How do I populate a bash associative array with command output?
This generally looks like:
declare -A map="( $( magic_command ) )"
where the magic_command generates an output like
[key1]=val1
[key2]=val2
[key3]=val3
In this case we use the command:
paste -d "" <(echo "[${key_string//:/]=$'\n'[}]=") \
<(echo "${val_string//:/$'\n'}")
where we use bash substitution to replace the delimiter with a newline. However, any other magic_command might do. For completion:
key_string="A:B:C:D"
val_string="1:2:3:4"
declare -A map="( $(paste -d "" <(echo "[${key_string//:/]=$'\n'[}]=") \
<(echo "${val_string//:/$'\n'}")) )"
for key in "${!map[#]}"; do echo "$key => ${map[$key]}"; done
Both examples generate the following output
D => 4
C => 3
B => 2
A => 1
Not exactly the answer for what you asked but at least it is shorter:
key='00011010:00011101:00100001'
value='H:K:O'
ifs=':'
IFS="$ifs" read -ra keys <<< "$key"
IFS="$ifs" read -ra values <<< "$value"
declare -A kv
for ((i=0; i<${#keys[*]}; i++)); do
kv[${keys[i]}]=${values[i]}
done
As a side note, you can initialize an associative array in one step with:
declare -A kv=([key1]=value1 [key2]=value2 [keyn]=valuen)
But I don't know how to use that in your case.
If values in your strings won't use spaces i would suggest this approach
firstString='00011010:00011101:00100001'
secondString='H:K:O'
delimiter=':'
declare -A translateMap
firstArray=( ${firstString//$delimiter/' '} )
secondArray=( ${secondString//$delimiter/' '} )
for i in ${!firstArray[#]}; {
translateMap[firstArray[$i]}]=${secondArray[$i]}
}