I'm using jq to parse a JSON file as shown here. However, the results for string values contain the "double-quotes" as expected, as shown below:
$ cat json.txt | jq '.name'
"Google"
How can I pipe this into another command to remove the ""? so I get
$ cat json.txt | jq '.name' | some_other_command
Google
What some_other_command can I use?
Use the -r (or --raw-output) option to emit raw strings as output:
jq -r '.name' <json.txt
So for a file containing just {"name": "Google"} then yes
sample='{"name":"Google"}'
echo $sample| jq '.name'
"Google"
using --raw-input helps
echo $sample| jq --raw-output '.name'
Google
But I stumbled upon this question because I was using --raw-output on a json array like this
sample='[{"name":"Yahoo"},{"name":"Google"}]'
echo $sample | jq --raw-output 'map(.name)'
[
"Yahoo",
"Google"
]
And I didn't understand why the quotes remained. I came across this post, and now I know adding | .[] does the trick!
echo $sample | jq --raw-output 'map(.name)| .[]'
Yahoo
Google
Related
I have a json file like this:
[
{
"classname": "Test endpoint",
"name": "expect failure",
"failure_system_out": "expected 404 Not Found\nError in test endpoint\n\tat Test._assertStatus"
},
{
"classname": "Test inner functions",
"name": "expect failure",
"failure_system_out": "Example fo test\n\tExpect 4 and got 5"
}
]
As you see the value in "failure_system_out" is a string containing newline chars (\n) and tab chars (\t).
I am trying to read the file, loop around the objects and print them with this code:
jq -c '.[]' myfile.json | while read i; do
test_name=$(echo "$i" | jq -r .name)
system_error=$(echo "$i" | jq -r .failure_system_out)
printf "${system_error}"
done
The problem is that using this approach, printf doesn't print the script according the the new line & tab chars, but It prints something like this expected 404 Not FoundnError in test endpointntat Test._assertStatus
Basically, I think that jq -c removes the \ char and therefore the printf doesn't work properly.
How can I iterate over an array of object stored in a file and keep the chars using to format the string?
Desired output for the first item:
expected 404 Not Found
Error in test endpoint
at Test._assertStatus
Desired output for the second item:
Example fo test
Expect 4 and got 5
Just use jq it's a scripting language on it's own.
$ jq -r '.[0].failure_system_out' /tmp/1
expected 404 Not Found
Error in test endpoint
at Test._assertStatus
$ jq -r '.[1].failure_system_out' /tmp/1
Example fo test
Expect 4 and got 5
$ jq -r '.[] | .name as $test_name | .failure_system_out as $system_error | $system_error' /tmp/1
expected 404 Not Found
Error in test endpoint
at Test._assertStatus
Example fo test
Expect 4 and got 5
As for using bash, first read https://mywiki.wooledge.org/BashFAQ/001 . I like using base64 to properly transfer context from jq to bash and handle all corner cases.
jq -r '.[] | #base64' /tmp/1 |
while IFS= read -r line; do
line=$(<<<"$line" base64 -d);
test_name=$(<<<"$line" jq -r .name);
system_error=$(<<<"$line" jq -r .failure_system_out);
printf "%s\n" "$system_error";
done
but it's not needed here, just a proper while read loop should be enough:
jq -c '.[]' /tmp/1 |
while IFS= read -r line; do
test_name=$(<<<"$line" jq -r .name);
system_error=$(<<<"$line" jq -r .failure_system_out);
printf "%s\n" "$system_error";
done
The question seems to weave amongst several goals, but in any case:
there is no need for jq to be called more than once, and
there should be no need to use base64 conversions, except possibly if the values corresponding to the keys of interest contain NULs.
If the goal is simply to emit the values of .failure_system_out then:
jq -r '.[].failure_system_out' test.json
would do it.
If the values of both .name and .failure_system_out must be made available separately as bash variables, then consider:
while IFS= read -d $'\0' system_error ; do
IFS= read -d $'\0' test_name
printf "%s\n" name="$test_name"
printf "%s\n" fso="$system_error"
echo ""
done < <(jq -rj '.[] | [.name, .failure_system_out, ""] | join("\u0000")' test.json)
readarray could also be used -- see e.g.
Storing JQ NULL-delimited output in bash array
#KamilCuk's answer works great and gives quite some more control.
Thought I'd still share this jq only solution:
printf "%s\n" "$(jq -r -c '.[] | .failure_system_out' test.json)"
This will produce:
expected 404 Not Found
Error in test endpoint
at Test._assertStatus
Example fo test
Expect 4 and got 5
I have written a script to retrieve certain value from file.json. It works if I provide the value to jq select, but the variable doesn't seem to work (or I don't know how to use it).
#!/bin/sh
#this works ***
projectID=$(cat file.json | jq -r '.resource[] | select(.username=="myemail#hotmail.com") | .id')
echo "$projectID"
EMAILID=myemail#hotmail.com
#this does not work *** no value is printed
projectID=$(cat file.json | jq -r '.resource[] | select(.username=="$EMAILID") | .id')
echo "$projectID"
Consider also passing in the shell variable (EMAILID) as a jq variable (here also EMAILID, for the sake of illustration):
projectID=$(jq -r --arg EMAILID "$EMAILID" '
.resource[]
| select(.username==$EMAILID)
| .id' file.json)
Postscript
For the record, another possibility would be to use jq's env function for accessing environment variables. For example, consider this sequence of bash commands:
EMAILID=foo#bar.com # not exported
EMAILID="$EMAILID" jq -n 'env.EMAILID'
The output is a JSON string:
"foo#bar.com"
I resolved this issue by escaping the inner double quotes
projectID=$(cat file.json | jq -r ".resource[] | select(.username==\"$EMAILID\") | .id")
Little unrelated but I will still put it here,
For other practical purposes shell variables can be used as -
value=10
jq '."key" = "'"$value"'"' file.json
Posting it here as it might help others. In string it might be necessary to pass the quotes to jq. To do the following with jq:
.items[] | select(.name=="string")
in bash you could do
EMAILID=$1
projectID=$(cat file.json | jq -r '.resource[] | select(.username=='\"$EMAILID\"') | .id')
essentially escaping the quotes and passing it on to jq
It's a quote issue, you need :
projectID=$(
cat file.json | jq -r ".resource[] | select(.username=='$EMAILID') | .id"
)
If you put single quotes to delimit the main string, the shell takes $EMAILID literally.
"Double quote" every literal that contains spaces/metacharacters and every expansion: "$var", "$(command "$var")", "${array[#]}", "a & b". Use 'single quotes' for code or literal $'s: 'Costs $5 US', ssh host 'echo "$HOSTNAME"'. See
http://mywiki.wooledge.org/Quotes
http://mywiki.wooledge.org/Arguments
http://wiki.bash-hackers.org/syntax/words
Jq now have better way to access environment variables, you can use env.EMAILID:
projectID=$(cat file.json | jq -r ".resource[] | select(.username==env.EMAILID) | .id")
Another way to accomplish this is with the jq "--arg" flag.
Using the original example:
#!/bin/sh
#this works ***
projectID=$(cat file.json | jq -r '.resource[] |
select(.username=="myemail#hotmail.com") | .id')
echo "$projectID"
EMAILID=myemail#hotmail.com
# Use --arg to pass the variable to jq. This should work:
projectID=$(cat file.json | jq --arg EMAILID $EMAILID -r '.resource[]
| select(.username=="$EMAILID") | .id')
echo "$projectID"
See here, which is where I found this solution:
https://github.com/stedolan/jq/issues/626
I know is a bit later to reply, sorry. But that works for me.
export K8S_public_load_balancer_url="$(kubectl get services -n ${TENANT}-production -o wide | grep "ingress-nginx-internal$" | awk '{print $4}')"
And now I am able to fetch and pass the content of the variable to jq
export TF_VAR_public_load_balancer_url="$(aws elbv2 describe-load-balancers --region eu-west-1 | jq -r '.LoadBalancers[] | select (.DNSName == "'$K8S_public_load_balancer_url'") | .LoadBalancerArn')"
In my case I needed to use double quote and quote to access the variable value.
Cheers.
I also faced same issue of variable substitution with jq. I found that --arg is the option which must be used with square bracket [] otherwise it won't work.. I am giving you sample example below:
RUNNER_TOKEN=$(aws secretsmanager get-secret-value --secret-id $SECRET_ID | jq '.SecretString|fromjson' | jq --arg kt $SECRET_KEY -r '.[$kt]' | tr -d '"')
In case where we want to append some string to the variable value and we are using the escaped double quotes, for example appending .crt to a variable CERT_TYPE; the following should work:
$ CERT_TYPE=client.reader
$ cat certs.json | jq -r ".\"${CERT_TYPE}\".crt" #### This will *not* work #####
$ cat certs.json | jq -r ".\"${CERT_TYPE}.crt\""
I have written a script to retrieve certain value from file.json. It works if I provide the value to jq select, but the variable doesn't seem to work (or I don't know how to use it).
#!/bin/sh
#this works ***
projectID=$(cat file.json | jq -r '.resource[] | select(.username=="myemail#hotmail.com") | .id')
echo "$projectID"
EMAILID=myemail#hotmail.com
#this does not work *** no value is printed
projectID=$(cat file.json | jq -r '.resource[] | select(.username=="$EMAILID") | .id')
echo "$projectID"
Consider also passing in the shell variable (EMAILID) as a jq variable (here also EMAILID, for the sake of illustration):
projectID=$(jq -r --arg EMAILID "$EMAILID" '
.resource[]
| select(.username==$EMAILID)
| .id' file.json)
Postscript
For the record, another possibility would be to use jq's env function for accessing environment variables. For example, consider this sequence of bash commands:
EMAILID=foo#bar.com # not exported
EMAILID="$EMAILID" jq -n 'env.EMAILID'
The output is a JSON string:
"foo#bar.com"
I resolved this issue by escaping the inner double quotes
projectID=$(cat file.json | jq -r ".resource[] | select(.username==\"$EMAILID\") | .id")
Little unrelated but I will still put it here,
For other practical purposes shell variables can be used as -
value=10
jq '."key" = "'"$value"'"' file.json
Posting it here as it might help others. In string it might be necessary to pass the quotes to jq. To do the following with jq:
.items[] | select(.name=="string")
in bash you could do
EMAILID=$1
projectID=$(cat file.json | jq -r '.resource[] | select(.username=='\"$EMAILID\"') | .id')
essentially escaping the quotes and passing it on to jq
It's a quote issue, you need :
projectID=$(
cat file.json | jq -r ".resource[] | select(.username=='$EMAILID') | .id"
)
If you put single quotes to delimit the main string, the shell takes $EMAILID literally.
"Double quote" every literal that contains spaces/metacharacters and every expansion: "$var", "$(command "$var")", "${array[#]}", "a & b". Use 'single quotes' for code or literal $'s: 'Costs $5 US', ssh host 'echo "$HOSTNAME"'. See
http://mywiki.wooledge.org/Quotes
http://mywiki.wooledge.org/Arguments
http://wiki.bash-hackers.org/syntax/words
Jq now have better way to access environment variables, you can use env.EMAILID:
projectID=$(cat file.json | jq -r ".resource[] | select(.username==env.EMAILID) | .id")
Another way to accomplish this is with the jq "--arg" flag.
Using the original example:
#!/bin/sh
#this works ***
projectID=$(cat file.json | jq -r '.resource[] |
select(.username=="myemail#hotmail.com") | .id')
echo "$projectID"
EMAILID=myemail#hotmail.com
# Use --arg to pass the variable to jq. This should work:
projectID=$(cat file.json | jq --arg EMAILID $EMAILID -r '.resource[]
| select(.username=="$EMAILID") | .id')
echo "$projectID"
See here, which is where I found this solution:
https://github.com/stedolan/jq/issues/626
I know is a bit later to reply, sorry. But that works for me.
export K8S_public_load_balancer_url="$(kubectl get services -n ${TENANT}-production -o wide | grep "ingress-nginx-internal$" | awk '{print $4}')"
And now I am able to fetch and pass the content of the variable to jq
export TF_VAR_public_load_balancer_url="$(aws elbv2 describe-load-balancers --region eu-west-1 | jq -r '.LoadBalancers[] | select (.DNSName == "'$K8S_public_load_balancer_url'") | .LoadBalancerArn')"
In my case I needed to use double quote and quote to access the variable value.
Cheers.
I also faced same issue of variable substitution with jq. I found that --arg is the option which must be used with square bracket [] otherwise it won't work.. I am giving you sample example below:
RUNNER_TOKEN=$(aws secretsmanager get-secret-value --secret-id $SECRET_ID | jq '.SecretString|fromjson' | jq --arg kt $SECRET_KEY -r '.[$kt]' | tr -d '"')
In case where we want to append some string to the variable value and we are using the escaped double quotes, for example appending .crt to a variable CERT_TYPE; the following should work:
$ CERT_TYPE=client.reader
$ cat certs.json | jq -r ".\"${CERT_TYPE}\".crt" #### This will *not* work #####
$ cat certs.json | jq -r ".\"${CERT_TYPE}.crt\""
I am getting a list of quoted strings back from AWS,
and I would like to isolate one of them into a parameter.
the command I am using is:
allelb=$(aws elb describe-load-balancers --query 'LoadBalancerDescriptions[].LoadBalancerName') && echo $allelb
And this is the output I am getting:
[ "elb-app-mprest-dev", "elb-core-mprest-dev", "api-vector-k8s-local-0j8ccl", "a2e6a899d111011e897b0067693cf815", "api-clusters-sydney7-mpre-rqae1h" ]
What I want to do is, get only the string with the word 'sydney7' inside a parameter. So I will have a new parameter with this content inside :
ELB=api-clusters-sydney7-mpre-rqae1h
This is waht I got so far:
allelb=$(aws elb describe-load-balancers --query 'LoadBalancerDescriptions[].LoadBalancerName' | tr -d '"' | tr -d ',' | tr -d ']' | tr -d '[') && echo $allelb | grep -o sydney7
but the output I get is not good enough-
sydney7
How can I achieve that?
Since the output from your command is a JSON array, you should probably use a program for parsing and filtering JSON, like jq:
$ jq '.[] | match(".*sydney.*").string' <<< "$allelb"
"api-clusters-sydney7-mpre-rqae1h"
If you want it raw, use the -r flag:
$ jq -r '.[] | match(".*sydney.*").string' <<< "$allelb"
api-clusters-sydney7-mpre-rqae1h
With cut
cut -d '"' -f10
with grep
grep -o '[^"]*sydney7[^"]*'
I'm using jq to parse a JSON file as shown here. However, the results for string values contain the "double-quotes" as expected, as shown below:
$ cat json.txt | jq '.name'
"Google"
How can I pipe this into another command to remove the ""? so I get
$ cat json.txt | jq '.name' | some_other_command
Google
What some_other_command can I use?
Use the -r (or --raw-output) option to emit raw strings as output:
jq -r '.name' <json.txt
So for a file containing just {"name": "Google"} then yes
sample='{"name":"Google"}'
echo $sample| jq '.name'
"Google"
using --raw-input helps
echo $sample| jq --raw-output '.name'
Google
But I stumbled upon this question because I was using --raw-output on a json array like this
sample='[{"name":"Yahoo"},{"name":"Google"}]'
echo $sample | jq --raw-output 'map(.name)'
[
"Yahoo",
"Google"
]
And I didn't understand why the quotes remained. I came across this post, and now I know adding | .[] does the trick!
echo $sample | jq --raw-output 'map(.name)| .[]'
Yahoo
Google