I am struggling to understand what the cause of the following bug is and how I can fix it.
I have this code:
set_filters() {
json=$1
filters='"Name=instance-state-name,Values=running,stopped"'
echo $json | jq -r '. | keys[]' | \
while read tag ; do
value=$(echo "$json" | jq -r ".[\"$tag\"]")
filters="$filters \"Name=tag:${tag},Values=${value}\""
done
echo $filters
}
set_filters '{"Name": "*FOO*", "Cost Center": "XX111"}'
The output I am expecting:
"Name=instance-state-name,Values=running,stopped" "Name=tag:Cost Center,Values=XX111" "Name=tag:Name,Values=*FOO*"
The output I am getting:
"Name=instance-state-name,Values=running,stopped"
If I insert echo statements to assist with debugging:
set_filters() {
json=$1
filters='"Name=instance-state-name,Values=running,stopped"'
echo $json | jq -r '. | keys[]' | \
while read tag ; do
value=$(echo "$json" | jq -r ".[\"$tag\"]")
filters="$filters \"Name=tag:${tag},Values=${value}\""
echo "FILTERS INSIDE LOOP: $filters"
done
echo "FILTERS OUTSIDE LOOP: $filters"
}
The output I then get is:
FILTERS INSIDE LOOP: "Name=instance-state-name,Values=running,stopped" "Name=tag:Cost Center,Values=XX111"
FILTERS INSIDE LOOP: "Name=instance-state-name,Values=running,stopped" "Name=tag:Cost Center,Values=XX111" "Name=tag:Name,Values=*FOO*"
FILTERS OUTSIDE LOOP: "Name=instance-state-name,Values=running,stopped"
I can't explain the behaviour. In a language other than Bash I would assume a variable scope issue for the variable $filters, but I thought the scope would basically be global.
I am using JQ version 1.3 and Bash version 4.1.2 on Red Hat Enterprise Linux 6.8.
Bash executes loops in a subshell if they are part of a pipeline. See for example BashFAQ/024 and "Bash Script: While-Loop Subshell Dilemma".
A possible workaround is to use process substitution:
while read tag; do
...
done < <(jq -r '. | keys[]' <<< "$1")
Related
I am facing issue with below commands. I need to use variable but it is returning me null whereas when I hardcode its value, it return me correct response.
Can anybody help me whats the correct way of writing this command?
My intension is to pull value of corresponding key passed as a variable?
temp1="{ \"SSM_DEV_SECRET_KEY\": \"Smkfnkhnb48dh\", \"SSM_DEV_GRAPH_DB\": \"Prod=bolt://neo4j:Grt56#atc.preprod.test.com:7687\", \"SSM_DEV_RDS_DB\": \"sqlite:////var/local/ecosystem_dashboard/config.db\", \"SSM_DEV_SUPPERUSER_USERNAME\": \"admin\", \"SSM_DEV_SUPPERUSER_PASSWORD\": \"9dW6JE8#KH9qiO006\" }"
var_name=SSM_DEV_SECRET_KEY
echo $temp1 | jq -r '.SSM_DEV_SECRET_KEY' <----- return Smkfnkhnb48dh // output
echo $temp1 | jq -r '."$var_name"' <---- return null
echo $temp1 | jq -r --arg var_name "$var_name" '."$var_name"' <---- return null , alternative way
Update: I am adding actual piece of where I am trying to use above fix. My intension is to first read all values which start with SSM_DEV_... and then get there original values from aws than replace it in. one key pair look like this --> SECRET_KEY=$SSM_DEV_SECRET_KEY
temp0="dev"
temp1="DEV"
result1=$(aws secretsmanager get-secret-value --secret-id "xxx-secret-$temp0" | jq '.SecretString')
while IFS= read -r line; do
if [[ "$line" == *"=\$SSM_$temp1"* ]]; then
before=${line%%"="*}
after=${line#*"="}
var_name="${after:1}"
jq -r --arg var_name "$var_name" '.[$var_name]' <<< "$result1"
fi
done < sample_file.txt
Fix: I have solved my issue which was of carriage return character.
Below cmd help me:
var_name=`echo ${after:1} | tr -d '\r'`
jq -r --arg var_name "$var_name" '.[$var_name]' <<< "$result1"
You'll need to use Generic Object Index (.[$var_name]) to let jq know the variable should be seen as a key
The command should look like:
jq -r --arg var_name "$var_name" '.[$var_name]' <<< "$temp1"
Wich will output:
Smkfnkhnb48dh
Note: <<< "$temp1" instead off the echo
Let's look at the following statement:
echo $temp1 | jq -r '."$var_name"' <---- return null
Your problem is actually with the shell quoting and not jq. The single quotes tell the shell not to interpolate (do variable substitution) among other things (like escaping white space and preventing globing). Thus, jq is receiving literally ."$var_name" as it's script - which is not what you want. You simply need to remove the single quotes and you'll be good:
echo $temp1 | jq -r ."$var_name" <---- Does this work?
That said, I would never write my script that way. I would definitely want to include the '.' in the quoted string like this:
echo $temp1 | jq -r ".$var_name" <---- Does this work?
Some would also suggest that you quote "$temp1" as well (typically all variable references should be quoted to protect against white space, but this is not a problem with echo):
echo "$temp1" | jq -r ".$var_name" <---- Does this work?
I have a json file like this:
[
{
"classname": "Test endpoint",
"name": "expect failure",
"failure_system_out": "expected 404 Not Found\nError in test endpoint\n\tat Test._assertStatus"
},
{
"classname": "Test inner functions",
"name": "expect failure",
"failure_system_out": "Example fo test\n\tExpect 4 and got 5"
}
]
As you see the value in "failure_system_out" is a string containing newline chars (\n) and tab chars (\t).
I am trying to read the file, loop around the objects and print them with this code:
jq -c '.[]' myfile.json | while read i; do
test_name=$(echo "$i" | jq -r .name)
system_error=$(echo "$i" | jq -r .failure_system_out)
printf "${system_error}"
done
The problem is that using this approach, printf doesn't print the script according the the new line & tab chars, but It prints something like this expected 404 Not FoundnError in test endpointntat Test._assertStatus
Basically, I think that jq -c removes the \ char and therefore the printf doesn't work properly.
How can I iterate over an array of object stored in a file and keep the chars using to format the string?
Desired output for the first item:
expected 404 Not Found
Error in test endpoint
at Test._assertStatus
Desired output for the second item:
Example fo test
Expect 4 and got 5
Just use jq it's a scripting language on it's own.
$ jq -r '.[0].failure_system_out' /tmp/1
expected 404 Not Found
Error in test endpoint
at Test._assertStatus
$ jq -r '.[1].failure_system_out' /tmp/1
Example fo test
Expect 4 and got 5
$ jq -r '.[] | .name as $test_name | .failure_system_out as $system_error | $system_error' /tmp/1
expected 404 Not Found
Error in test endpoint
at Test._assertStatus
Example fo test
Expect 4 and got 5
As for using bash, first read https://mywiki.wooledge.org/BashFAQ/001 . I like using base64 to properly transfer context from jq to bash and handle all corner cases.
jq -r '.[] | #base64' /tmp/1 |
while IFS= read -r line; do
line=$(<<<"$line" base64 -d);
test_name=$(<<<"$line" jq -r .name);
system_error=$(<<<"$line" jq -r .failure_system_out);
printf "%s\n" "$system_error";
done
but it's not needed here, just a proper while read loop should be enough:
jq -c '.[]' /tmp/1 |
while IFS= read -r line; do
test_name=$(<<<"$line" jq -r .name);
system_error=$(<<<"$line" jq -r .failure_system_out);
printf "%s\n" "$system_error";
done
The question seems to weave amongst several goals, but in any case:
there is no need for jq to be called more than once, and
there should be no need to use base64 conversions, except possibly if the values corresponding to the keys of interest contain NULs.
If the goal is simply to emit the values of .failure_system_out then:
jq -r '.[].failure_system_out' test.json
would do it.
If the values of both .name and .failure_system_out must be made available separately as bash variables, then consider:
while IFS= read -d $'\0' system_error ; do
IFS= read -d $'\0' test_name
printf "%s\n" name="$test_name"
printf "%s\n" fso="$system_error"
echo ""
done < <(jq -rj '.[] | [.name, .failure_system_out, ""] | join("\u0000")' test.json)
readarray could also be used -- see e.g.
Storing JQ NULL-delimited output in bash array
#KamilCuk's answer works great and gives quite some more control.
Thought I'd still share this jq only solution:
printf "%s\n" "$(jq -r -c '.[] | .failure_system_out' test.json)"
This will produce:
expected 404 Not Found
Error in test endpoint
at Test._assertStatus
Example fo test
Expect 4 and got 5
I have a yaml array like below,
identitymappings:
- arn: "arn:aws:iam::12345567:role/AdmRole"
group: "system:masters"
user: "user1"
- arn: "arn:aws:iam::12345567:role/TestRole"
group: "system:masters"
user: "user2"
I am trying to parse this yaml in a bash script using for loop and yq.
for identityMapping in $(yq read test.yaml "identitymappings[*]"); do
roleArn=$identityMapping["arn"]
group=$identityMapping.group
user=$identityMapping.user
done
But I am not getting the expected results like not able to fetch the values of roleArn,group,user.
Please let me know how to fix this.
The way I would do it is:
# load array into a bash array
# need to output each entry as a single line
readarray identityMappings < <(yq e -o=j -I=0 '.identitymappings[]' test.yml )
for identityMapping in "${identityMappings[#]}"; do
# identity mapping is a yaml snippet representing a single entry
roleArn=$(echo "$identityMapping" | yq e '.arn' -)
echo "roleArn: $roleArn"
done
output:
roleArn: arn:aws:iam::12345567:role/AdmRole
roleArn: arn:aws:iam::12345567:role/TestRole
Disclaimer: I wrote yq
I wasn't able to comment on Charles Duffy's proper answer, but this works for yq v4 without the use of jq...
while IFS=$'\t' read -r roleArn group user _; do
echo "Role: $roleArn"
echo "Group: $group"
echo "User: $user"
done < <(yq e '.identitymappings[] | [.arn, .group, .user] | #tsv' test.yaml)
The easiest way to read from jq or yq into bash is to use a BashFAQ #1 while read loop to handle line-oriented data; in the below, we use #tsv to generate line-oriented output:
while IFS=$'\t' read -r roleArn group user _; do
echo "Role: $roleArn"
echo "Group: $group"
echo "User: $user"
done < <(yq -j read test.yaml \
| jq -r '.identitymappings[] | [.arn, .group, .user] | #tsv')
Note that if you were using the Python yq rather than the Go one, you could remove the yq -j read and just use yq -r '...' in place of jq -r '...'.
There is an improvement of #Rad4's answer that worked for me.
You can neatly loop through using latest yq and jq via:
for im in $(yq eval -o=j test.yaml | jq -cr '.identitymappings[]'); do
arn=$(echo $im | jq -r '.arn' -)
group=$(echo $im | jq -r '.group' -)
user=$(echo $im | jq -r '.user' -)
echo $arn $group $user
done
This loops through valid online jsons, which makes jq still work inside the loop.
The answer by #mike.f is a good one. However, it does not work on OSX machines, because readarray is not an available command.
You can read more about this here.
Here is the equivalent that would work on a mac:
# load array into a bash array
# need to output each entry as a single line
identitymappings=( $(yq e -o=j -I=0 '.identitymappings[]' test.yml ) )
for identityMapping in "${identityMappings[#]}"; do
# identity mapping is a yaml snippet representing a single entry
roleArn=$(echo "$identityMapping" | yq e '.arn' -)
echo "roleArn: $roleArn"
done
Get identitymappings length
Using index to access element of identitymappings
array_length=`yq e ". identitymappings | length - 1" test.yaml`
if [ $array_length -le 0 ] ; then
exit
fi
for element_index in `seq 0 $array_length`;do
arn=`yq e ".identitymappings[$element_index]. arn" test.yml`
group=`yq e ".identitymappings[$element_index]. group" test.yml`
user=`yq e ".identitymappings[$element_index]. user" test.yml`
done
How to get length of array in yq?
https://mikefarah.gitbook.io/yq/operators/length
I figured out..
for identityMapping in $(yq read test.yaml -j "identitymappings[*]"); do
echo $identityMapping
roleArn= echo $identityMapping | jq -r '.arn'
echo $roleArn
group= echo $identityMapping | jq -r '.group'
echo $group
user= echo $identityMapping | jq -r '.user'
echo $user
I have written a script to retrieve certain value from file.json. It works if I provide the value to jq select, but the variable doesn't seem to work (or I don't know how to use it).
#!/bin/sh
#this works ***
projectID=$(cat file.json | jq -r '.resource[] | select(.username=="myemail#hotmail.com") | .id')
echo "$projectID"
EMAILID=myemail#hotmail.com
#this does not work *** no value is printed
projectID=$(cat file.json | jq -r '.resource[] | select(.username=="$EMAILID") | .id')
echo "$projectID"
Consider also passing in the shell variable (EMAILID) as a jq variable (here also EMAILID, for the sake of illustration):
projectID=$(jq -r --arg EMAILID "$EMAILID" '
.resource[]
| select(.username==$EMAILID)
| .id' file.json)
Postscript
For the record, another possibility would be to use jq's env function for accessing environment variables. For example, consider this sequence of bash commands:
EMAILID=foo#bar.com # not exported
EMAILID="$EMAILID" jq -n 'env.EMAILID'
The output is a JSON string:
"foo#bar.com"
I resolved this issue by escaping the inner double quotes
projectID=$(cat file.json | jq -r ".resource[] | select(.username==\"$EMAILID\") | .id")
Little unrelated but I will still put it here,
For other practical purposes shell variables can be used as -
value=10
jq '."key" = "'"$value"'"' file.json
Posting it here as it might help others. In string it might be necessary to pass the quotes to jq. To do the following with jq:
.items[] | select(.name=="string")
in bash you could do
EMAILID=$1
projectID=$(cat file.json | jq -r '.resource[] | select(.username=='\"$EMAILID\"') | .id')
essentially escaping the quotes and passing it on to jq
It's a quote issue, you need :
projectID=$(
cat file.json | jq -r ".resource[] | select(.username=='$EMAILID') | .id"
)
If you put single quotes to delimit the main string, the shell takes $EMAILID literally.
"Double quote" every literal that contains spaces/metacharacters and every expansion: "$var", "$(command "$var")", "${array[#]}", "a & b". Use 'single quotes' for code or literal $'s: 'Costs $5 US', ssh host 'echo "$HOSTNAME"'. See
http://mywiki.wooledge.org/Quotes
http://mywiki.wooledge.org/Arguments
http://wiki.bash-hackers.org/syntax/words
Jq now have better way to access environment variables, you can use env.EMAILID:
projectID=$(cat file.json | jq -r ".resource[] | select(.username==env.EMAILID) | .id")
Another way to accomplish this is with the jq "--arg" flag.
Using the original example:
#!/bin/sh
#this works ***
projectID=$(cat file.json | jq -r '.resource[] |
select(.username=="myemail#hotmail.com") | .id')
echo "$projectID"
EMAILID=myemail#hotmail.com
# Use --arg to pass the variable to jq. This should work:
projectID=$(cat file.json | jq --arg EMAILID $EMAILID -r '.resource[]
| select(.username=="$EMAILID") | .id')
echo "$projectID"
See here, which is where I found this solution:
https://github.com/stedolan/jq/issues/626
I know is a bit later to reply, sorry. But that works for me.
export K8S_public_load_balancer_url="$(kubectl get services -n ${TENANT}-production -o wide | grep "ingress-nginx-internal$" | awk '{print $4}')"
And now I am able to fetch and pass the content of the variable to jq
export TF_VAR_public_load_balancer_url="$(aws elbv2 describe-load-balancers --region eu-west-1 | jq -r '.LoadBalancers[] | select (.DNSName == "'$K8S_public_load_balancer_url'") | .LoadBalancerArn')"
In my case I needed to use double quote and quote to access the variable value.
Cheers.
I also faced same issue of variable substitution with jq. I found that --arg is the option which must be used with square bracket [] otherwise it won't work.. I am giving you sample example below:
RUNNER_TOKEN=$(aws secretsmanager get-secret-value --secret-id $SECRET_ID | jq '.SecretString|fromjson' | jq --arg kt $SECRET_KEY -r '.[$kt]' | tr -d '"')
In case where we want to append some string to the variable value and we are using the escaped double quotes, for example appending .crt to a variable CERT_TYPE; the following should work:
$ CERT_TYPE=client.reader
$ cat certs.json | jq -r ".\"${CERT_TYPE}\".crt" #### This will *not* work #####
$ cat certs.json | jq -r ".\"${CERT_TYPE}.crt\""
I have written a script to retrieve certain value from file.json. It works if I provide the value to jq select, but the variable doesn't seem to work (or I don't know how to use it).
#!/bin/sh
#this works ***
projectID=$(cat file.json | jq -r '.resource[] | select(.username=="myemail#hotmail.com") | .id')
echo "$projectID"
EMAILID=myemail#hotmail.com
#this does not work *** no value is printed
projectID=$(cat file.json | jq -r '.resource[] | select(.username=="$EMAILID") | .id')
echo "$projectID"
Consider also passing in the shell variable (EMAILID) as a jq variable (here also EMAILID, for the sake of illustration):
projectID=$(jq -r --arg EMAILID "$EMAILID" '
.resource[]
| select(.username==$EMAILID)
| .id' file.json)
Postscript
For the record, another possibility would be to use jq's env function for accessing environment variables. For example, consider this sequence of bash commands:
EMAILID=foo#bar.com # not exported
EMAILID="$EMAILID" jq -n 'env.EMAILID'
The output is a JSON string:
"foo#bar.com"
I resolved this issue by escaping the inner double quotes
projectID=$(cat file.json | jq -r ".resource[] | select(.username==\"$EMAILID\") | .id")
Little unrelated but I will still put it here,
For other practical purposes shell variables can be used as -
value=10
jq '."key" = "'"$value"'"' file.json
Posting it here as it might help others. In string it might be necessary to pass the quotes to jq. To do the following with jq:
.items[] | select(.name=="string")
in bash you could do
EMAILID=$1
projectID=$(cat file.json | jq -r '.resource[] | select(.username=='\"$EMAILID\"') | .id')
essentially escaping the quotes and passing it on to jq
It's a quote issue, you need :
projectID=$(
cat file.json | jq -r ".resource[] | select(.username=='$EMAILID') | .id"
)
If you put single quotes to delimit the main string, the shell takes $EMAILID literally.
"Double quote" every literal that contains spaces/metacharacters and every expansion: "$var", "$(command "$var")", "${array[#]}", "a & b". Use 'single quotes' for code or literal $'s: 'Costs $5 US', ssh host 'echo "$HOSTNAME"'. See
http://mywiki.wooledge.org/Quotes
http://mywiki.wooledge.org/Arguments
http://wiki.bash-hackers.org/syntax/words
Jq now have better way to access environment variables, you can use env.EMAILID:
projectID=$(cat file.json | jq -r ".resource[] | select(.username==env.EMAILID) | .id")
Another way to accomplish this is with the jq "--arg" flag.
Using the original example:
#!/bin/sh
#this works ***
projectID=$(cat file.json | jq -r '.resource[] |
select(.username=="myemail#hotmail.com") | .id')
echo "$projectID"
EMAILID=myemail#hotmail.com
# Use --arg to pass the variable to jq. This should work:
projectID=$(cat file.json | jq --arg EMAILID $EMAILID -r '.resource[]
| select(.username=="$EMAILID") | .id')
echo "$projectID"
See here, which is where I found this solution:
https://github.com/stedolan/jq/issues/626
I know is a bit later to reply, sorry. But that works for me.
export K8S_public_load_balancer_url="$(kubectl get services -n ${TENANT}-production -o wide | grep "ingress-nginx-internal$" | awk '{print $4}')"
And now I am able to fetch and pass the content of the variable to jq
export TF_VAR_public_load_balancer_url="$(aws elbv2 describe-load-balancers --region eu-west-1 | jq -r '.LoadBalancers[] | select (.DNSName == "'$K8S_public_load_balancer_url'") | .LoadBalancerArn')"
In my case I needed to use double quote and quote to access the variable value.
Cheers.
I also faced same issue of variable substitution with jq. I found that --arg is the option which must be used with square bracket [] otherwise it won't work.. I am giving you sample example below:
RUNNER_TOKEN=$(aws secretsmanager get-secret-value --secret-id $SECRET_ID | jq '.SecretString|fromjson' | jq --arg kt $SECRET_KEY -r '.[$kt]' | tr -d '"')
In case where we want to append some string to the variable value and we are using the escaped double quotes, for example appending .crt to a variable CERT_TYPE; the following should work:
$ CERT_TYPE=client.reader
$ cat certs.json | jq -r ".\"${CERT_TYPE}\".crt" #### This will *not* work #####
$ cat certs.json | jq -r ".\"${CERT_TYPE}.crt\""