a curl command gives me a JSON output as follows:- [{"Description":"","Value":"bc","Key":"a"},{"Description":"","Value":"ef","Key":"d"},...]
Now I want to capture the values of only "Key" & "Value" in such a way that it if I perform echo $a it gives "bc", echo $d it gives "ef" etc.
Note: I am absolutely new in shell scripting/bash, thus this question
Thank you in advance
You can use jq to parse the array and obtain the necessary value.
echo '[{"Description":"","Value":"bc","Key":"a"},{"Description":"","Value":"ef","Key":"d"}]' | jq '.[] | select(.Key == "d").Value'
This Should give you "ef" and,
echo '[{"Description":"","Value":"bc","Key":"a"},{"Description":"","Value":"ef","Key":"d"}]' | jq '.[] | select(.Key == "a").Value'
Will give you "bc"
You can replace the echo with your curl statement.
More about jq : https://stedolan.github.io/jq/
Something like this may do the job :
dataString='[{"Description":"","Value":"bc","Key":"a"},{"Description":"","Value":"ef","Key":"d"},...]'
while read value key; do
echo "key: '$key', value : '$value'"
done < <(echo "$dataString" | sed 's/},{/\n/g' | tr -d '[]{}"' | tr ':' ',' | cut -d',' -f 4,6 --output-delimiter=' ')
Details :
dataString is the output of curl
the whole (echo "$dataString"....) simply turns $dataString into :
bc a
ef d
the <(...) construct is a process substitution . It allows reading the output of a command with the same method you'd be reading from a file
Related
I am facing issue with below commands. I need to use variable but it is returning me null whereas when I hardcode its value, it return me correct response.
Can anybody help me whats the correct way of writing this command?
My intension is to pull value of corresponding key passed as a variable?
temp1="{ \"SSM_DEV_SECRET_KEY\": \"Smkfnkhnb48dh\", \"SSM_DEV_GRAPH_DB\": \"Prod=bolt://neo4j:Grt56#atc.preprod.test.com:7687\", \"SSM_DEV_RDS_DB\": \"sqlite:////var/local/ecosystem_dashboard/config.db\", \"SSM_DEV_SUPPERUSER_USERNAME\": \"admin\", \"SSM_DEV_SUPPERUSER_PASSWORD\": \"9dW6JE8#KH9qiO006\" }"
var_name=SSM_DEV_SECRET_KEY
echo $temp1 | jq -r '.SSM_DEV_SECRET_KEY' <----- return Smkfnkhnb48dh // output
echo $temp1 | jq -r '."$var_name"' <---- return null
echo $temp1 | jq -r --arg var_name "$var_name" '."$var_name"' <---- return null , alternative way
Update: I am adding actual piece of where I am trying to use above fix. My intension is to first read all values which start with SSM_DEV_... and then get there original values from aws than replace it in. one key pair look like this --> SECRET_KEY=$SSM_DEV_SECRET_KEY
temp0="dev"
temp1="DEV"
result1=$(aws secretsmanager get-secret-value --secret-id "xxx-secret-$temp0" | jq '.SecretString')
while IFS= read -r line; do
if [[ "$line" == *"=\$SSM_$temp1"* ]]; then
before=${line%%"="*}
after=${line#*"="}
var_name="${after:1}"
jq -r --arg var_name "$var_name" '.[$var_name]' <<< "$result1"
fi
done < sample_file.txt
Fix: I have solved my issue which was of carriage return character.
Below cmd help me:
var_name=`echo ${after:1} | tr -d '\r'`
jq -r --arg var_name "$var_name" '.[$var_name]' <<< "$result1"
You'll need to use Generic Object Index (.[$var_name]) to let jq know the variable should be seen as a key
The command should look like:
jq -r --arg var_name "$var_name" '.[$var_name]' <<< "$temp1"
Wich will output:
Smkfnkhnb48dh
Note: <<< "$temp1" instead off the echo
Let's look at the following statement:
echo $temp1 | jq -r '."$var_name"' <---- return null
Your problem is actually with the shell quoting and not jq. The single quotes tell the shell not to interpolate (do variable substitution) among other things (like escaping white space and preventing globing). Thus, jq is receiving literally ."$var_name" as it's script - which is not what you want. You simply need to remove the single quotes and you'll be good:
echo $temp1 | jq -r ."$var_name" <---- Does this work?
That said, I would never write my script that way. I would definitely want to include the '.' in the quoted string like this:
echo $temp1 | jq -r ".$var_name" <---- Does this work?
Some would also suggest that you quote "$temp1" as well (typically all variable references should be quoted to protect against white space, but this is not a problem with echo):
echo "$temp1" | jq -r ".$var_name" <---- Does this work?
Basically I have a bash script that at one point makes an API call and a cert and key are generated and returned in json. I pipe it to jq and can select either the cert or the key and store it in a variable.
Something like this:
CERT=$(API call | jq -r '.certificate')
or
KEY=$(API call | jq -r '.key')
I want to store each in its own variable but I can't make the call twice because it will generate a new cert/key.
I know that I can just store both in a file and then manipulate after to accomplish my task but I am curious if jq offers a direct way to selectively store each value in its own variable?
You could store the original JSON in a variable (instead of a file), and then extract the key and cert from that:
apiResult=$(API call)
cert=$(jq -r '.certificate' <<<"$apiResult")
key=$(jq -r '.key' <<<"$apiResult")
Notes: I recommend using lower- or mixed-case variables, to avoid accidental conflicts with the many all-caps names with special meanings. Also, <<< is a bashism, and won't work in all other shells; if you need this be portable to e.g. dash, use something like key=$(printf '%s\n' "$apiResult" | jq -r '.key').
Unless there are NUL characters in the strings, there should be no need to call jq more than once, or to serialize the data.
In the simplest case, you could proceed along the following lines:
{ IFS= read -r certificate
IFS= read -r key
echo "certificate=$certificate"
echo "key=$key"
} < <(API call | jq -r '.certificate, .key')
If the values do not contain NUL but might contain newline characters,
then you could use NUL as the delimiter. For the sake of variety,
we could also use a while loop:
while IFS= read -r -d $'\0' certificate
do
IFS= read -r -d $'\0' key
echo "certificate=$certificate"
echo "key=$key"
done < <(API call | jq -rj '[.certificate, .key] | join("\u0000")')
Conversely ...
If the values of interest might contain literal NUL values ("\u0000"), then the question seems to be problematic, as bash variables in effect cannot contain literal NULs.
If any of the values of interest might contain literal NUL values, then here are two strategies for extracting the "raw" string equivalents into separate files:
Save the JSON output in a (temporary) file, and invoke jq -r once per value of interest in the obvious way.
Set up a bash pipeline starting with:
API call | jq -r '.certificate, .key | #base64`
and continuing with a loop in which each line is decoded, e.g. using base64 --decode or jq's #base64d.
The second strategy might make sense if the API call produces a very large JSON document.
If neither of the values contain a newline, you can easily use read:
$ read cert key < <( echo '{"certificate": "foo", "key": "bar"}' | jq -r .certificate,.key | tr \\n ' ')
$ echo $cert
foo
$ echo $key
bar
or:
$ { read cert; read key; } << EOF
> $( echo '{"certificate": "foo", "key": "bar"}' | jq -r .certificate,.key )
> EOF
$ echo $cert:$key
foo:bar
The certificate almost certainly will contain newlines, so you many want to serialize that data (eg, base64 encode it). But you're probably better off using Gordon Davisson's approach.
Say you created a program that output shell commands.
$ data_source | jq -r '#sh "certificate=\( .certificate )\nkey=\( .key )"'
certificate='the certificate'
key='the key'
Then, all you would need to do is evaluate them.
eval "$( data_source | jq -r '#sh "certificate=\( .certificate )\nkey=\( .key )"' )"
If you were dealing with many variables, you could use the following:
eval "$(
data_source |
jq -r '
{ "certificate": "certificate", "key": "key" } as $map |
( $map | keys_unsorted[] ) as $shell_var |
$shell_var + #sh "=\( .[$map[$shell_var]] )"
'
)"
You could even use paths.
eval "$(
data_source |
jq -r '
{ "certificate": [ "certificate" ], "key": [ "key" ] } as $map |
( $map | keys_unsorted[] ) as $shell_var |
$shell_var + #sh "=\( getpath($map[$shell_var]) )"
'
)"
I have a json file like this:
[
{
"classname": "Test endpoint",
"name": "expect failure",
"failure_system_out": "expected 404 Not Found\nError in test endpoint\n\tat Test._assertStatus"
},
{
"classname": "Test inner functions",
"name": "expect failure",
"failure_system_out": "Example fo test\n\tExpect 4 and got 5"
}
]
As you see the value in "failure_system_out" is a string containing newline chars (\n) and tab chars (\t).
I am trying to read the file, loop around the objects and print them with this code:
jq -c '.[]' myfile.json | while read i; do
test_name=$(echo "$i" | jq -r .name)
system_error=$(echo "$i" | jq -r .failure_system_out)
printf "${system_error}"
done
The problem is that using this approach, printf doesn't print the script according the the new line & tab chars, but It prints something like this expected 404 Not FoundnError in test endpointntat Test._assertStatus
Basically, I think that jq -c removes the \ char and therefore the printf doesn't work properly.
How can I iterate over an array of object stored in a file and keep the chars using to format the string?
Desired output for the first item:
expected 404 Not Found
Error in test endpoint
at Test._assertStatus
Desired output for the second item:
Example fo test
Expect 4 and got 5
Just use jq it's a scripting language on it's own.
$ jq -r '.[0].failure_system_out' /tmp/1
expected 404 Not Found
Error in test endpoint
at Test._assertStatus
$ jq -r '.[1].failure_system_out' /tmp/1
Example fo test
Expect 4 and got 5
$ jq -r '.[] | .name as $test_name | .failure_system_out as $system_error | $system_error' /tmp/1
expected 404 Not Found
Error in test endpoint
at Test._assertStatus
Example fo test
Expect 4 and got 5
As for using bash, first read https://mywiki.wooledge.org/BashFAQ/001 . I like using base64 to properly transfer context from jq to bash and handle all corner cases.
jq -r '.[] | #base64' /tmp/1 |
while IFS= read -r line; do
line=$(<<<"$line" base64 -d);
test_name=$(<<<"$line" jq -r .name);
system_error=$(<<<"$line" jq -r .failure_system_out);
printf "%s\n" "$system_error";
done
but it's not needed here, just a proper while read loop should be enough:
jq -c '.[]' /tmp/1 |
while IFS= read -r line; do
test_name=$(<<<"$line" jq -r .name);
system_error=$(<<<"$line" jq -r .failure_system_out);
printf "%s\n" "$system_error";
done
The question seems to weave amongst several goals, but in any case:
there is no need for jq to be called more than once, and
there should be no need to use base64 conversions, except possibly if the values corresponding to the keys of interest contain NULs.
If the goal is simply to emit the values of .failure_system_out then:
jq -r '.[].failure_system_out' test.json
would do it.
If the values of both .name and .failure_system_out must be made available separately as bash variables, then consider:
while IFS= read -d $'\0' system_error ; do
IFS= read -d $'\0' test_name
printf "%s\n" name="$test_name"
printf "%s\n" fso="$system_error"
echo ""
done < <(jq -rj '.[] | [.name, .failure_system_out, ""] | join("\u0000")' test.json)
readarray could also be used -- see e.g.
Storing JQ NULL-delimited output in bash array
#KamilCuk's answer works great and gives quite some more control.
Thought I'd still share this jq only solution:
printf "%s\n" "$(jq -r -c '.[] | .failure_system_out' test.json)"
This will produce:
expected 404 Not Found
Error in test endpoint
at Test._assertStatus
Example fo test
Expect 4 and got 5
I parsed a json file with jq like this :
# cat test.json | jq '.logs' | jq '.[]' | jq '._id' | jq -s
It returns an array like this : [34,235,436,546,.....]
Using bash script i described an array :
# declare -a msgIds = ...
This array uses () instead of [] so when I pass the array given above to this array it won't work.
([324,32,45..]) this causes problem. If i remove the jq -s, an array forms with only 1 member in it.
Is there a way to solve this issue?
We can solve this problem by two ways. They are:
Input string:
// test.json
{
"keys": ["key1","key2","key3"]
}
Approach 1:
1) Use jq -r (output raw strings, not JSON texts) .
KEYS=$(jq -r '.keys' test.json)
echo $KEYS
# Output: [ "key1", "key2", "key3" ]
2) Use #sh (Converts input string to a series of space-separated strings). It removes square brackets[], comma(,) from the string.
KEYS=$(<test.json jq -r '.keys | #sh')
echo $KEYS
# Output: 'key1' 'key2' 'key3'
3) Using tr to remove single quotes from the string output. To delete specific characters use the -d option in tr.
KEYS=$((<test.json jq -r '.keys | #sh')| tr -d \')
echo $KEYS
# Output: key1 key2 key3
4) We can convert the comma-separated string to the array by placing our string output in a round bracket().
It also called compound Assignment, where we declare the array with a bunch of values.
ARRAYNAME=(value1 value2 .... valueN)
#!/bin/bash
KEYS=($((<test.json jq -r '.keys | #sh') | tr -d \'\"))
echo "Array size: " ${#KEYS[#]}
echo "Array elements: "${KEYS[#]}
# Output:
# Array size: 3
# Array elements: key1 key2 key3
Approach 2:
1) Use jq -r to get the string output, then use tr to delete characters like square brackets, double quotes and comma.
#!/bin/bash
KEYS=$(jq -r '.keys' test.json | tr -d '[],"')
echo $KEYS
# Output: key1 key2 key3
2) Then we can convert the comma-separated string to the array by placing our string output in a round bracket().
#!/bin/bash
KEYS=($(jq -r '.keys' test.json | tr -d '[]," '))
echo "Array size: " ${#KEYS[#]}
echo "Array elements: "${KEYS[#]}
# Output:
# Array size: 3
# Array elements: key1 key2 key3
To correctly parse values that have spaces, newlines (or any other arbitrary characters) just use jq's #sh filter and bash's declare -a. (No need for a while read loop or any other pre-processing)
// foo.json
{"data": ["A B", "C'D", ""]}
str=$(jq -r '.data | #sh' foo.json)
declare -a arr="($str)" # must be quoted like this
$ declare -p arr
declare -a arr=([0]="A B" [1]="C'D" [2]="")
The reason that this works correctly is that #sh will produce a space-separated list of shell-quoted words:
$ echo "$str"
'A B' 'C'\''D' ''
and this is exactly the format that declare expects for an array definition.
Use jq -r to output a string "raw", without JSON formatting, and use the #sh formatter to format your results as a string for shell consumption. Per the jq docs:
#sh:
The input is escaped suitable for use in a command-line for a POSIX shell. If the input is an array, the output will be a series of space-separated strings.
So can do e.g.
msgids=($(<test.json jq -r '.logs[]._id | #sh'))
and get the result you want.
From the jq FAQ (https://github.com/stedolan/jq/wiki/FAQ):
𝑸: How can a stream of JSON texts produced by jq be converted into a bash array of corresponding values?
A: One option would be to use mapfile (aka readarray), for example:
mapfile -t array <<< $(jq -c '.[]' input.json)
An alternative that might be indicative of what to do in other shells is to use read -r within a while loop. The following bash script populates an array, x, with JSON texts. The key points are the use of the -c option, and the use of the bash idiom while read -r value; do ... done < <(jq .......):
#!/bin/bash
x=()
while read -r value
do
x+=("$value")
done < <(jq -c '.[]' input.json)
++ To resolve this, we can use a very simple approach:
++ Since I am not aware of you input file, I am creating a file input.json with the following contents:
input.json:
{
"keys": ["key1","key2","key3"]
}
++ Use jq to get the value from the above file input.json:
Command: cat input.json | jq -r '.keys | #sh'
Output: 'key1' 'key2' 'key3'
Explanation: | #sh removes [ and "
++ To remove ' ' as well we use tr
command: cat input.json | jq -r '.keys | #sh' | tr -d \'
Explanation: use tr delete -d to remove '
++ To store this in a bash array we use () with `` and print it:
command:
KEYS=(`cat input.json | jq -r '.keys | #sh' | tr -d \'`)
To print all the entries of the array: echo "${KEYS[*]}"
I am getting a list of quoted strings back from AWS,
and I would like to isolate one of them into a parameter.
the command I am using is:
allelb=$(aws elb describe-load-balancers --query 'LoadBalancerDescriptions[].LoadBalancerName') && echo $allelb
And this is the output I am getting:
[ "elb-app-mprest-dev", "elb-core-mprest-dev", "api-vector-k8s-local-0j8ccl", "a2e6a899d111011e897b0067693cf815", "api-clusters-sydney7-mpre-rqae1h" ]
What I want to do is, get only the string with the word 'sydney7' inside a parameter. So I will have a new parameter with this content inside :
ELB=api-clusters-sydney7-mpre-rqae1h
This is waht I got so far:
allelb=$(aws elb describe-load-balancers --query 'LoadBalancerDescriptions[].LoadBalancerName' | tr -d '"' | tr -d ',' | tr -d ']' | tr -d '[') && echo $allelb | grep -o sydney7
but the output I get is not good enough-
sydney7
How can I achieve that?
Since the output from your command is a JSON array, you should probably use a program for parsing and filtering JSON, like jq:
$ jq '.[] | match(".*sydney.*").string' <<< "$allelb"
"api-clusters-sydney7-mpre-rqae1h"
If you want it raw, use the -r flag:
$ jq -r '.[] | match(".*sydney.*").string' <<< "$allelb"
api-clusters-sydney7-mpre-rqae1h
With cut
cut -d '"' -f10
with grep
grep -o '[^"]*sydney7[^"]*'