jq bash issue with filtering with CONTAINS - bash

I have an issue where I am trying to filter records with a CONTAINS, but it won't accept a variable that has spaces in it. I am including the JSON and the calls. I explain what works and the last one that does not work. I have looked High and Low but I can't make it work. I have seen and tried many (hundreds of ways taking into account the double quotes, escaped, not escaped, with, without, but no luck) can someone take a look and point me to something that might help.
JSON used to test
_metadatadashjson='{ "meta": { "provisionedExternalId": "" }, "dashboard": { "liveNow": false, "panels": [ { "collapsed": false, "title": "Gyrex Thread Count Gauges", "type": "row", "targets": [ { "expr": "jvm_threads_current{instance=\"192.1.50.22:8055\",job=\"prometheus_gyrex\"}", "refId": "B" } ] }, { "datasource": "Prometheus_16_Docker", "targets": [ { "exemplar": true, "expr": "jvm_threads_current{instance=\"10.32.0.4:8055\",job=\"prometheus_gyrex\"}" } ], "title": ".16 : 3279", "type": "gauge" }, { "description": "", "targets": [ { "expr": "jvm_threads_current{instance=\"10.32.0.7:8055\",job=\"prometheus_gyrex\"}", "refId": "B" } ], "title": ".16 : 3288", "type": "graph" }, { "description": "", "targets": [ { "expr": "jvm_threads_current{instance=\"192.168.2.16:3288\",job=\"prometheus_gyrex\"}", "refId": "C" } ], "title": ".16 : 3288", "type": "graph" } ], "version": 55 }}'
Set the string to search for in key "expr"
exprStrSearch="10.32.0.4:8055"
This works returns one record
echo "${_metadatadashjson}" | jq -r --arg EXPRSTRSEARCH "$exprStrSearch" '.dashboard.panels[] | select(.targets[].expr | contains($EXPRSTRSEARCH)) | .targets[].expr'
This works no problem returns two records.
echo "${_metadatadashjson}" | jq -r --arg EXPRSTRSEARCH "$exprStrSearch" '.dashboard.panels[] | select(.targets[].expr | contains("10.32.0.4:8055", "10.32.0.7:8055")) | .targets[].expr'
Change the value to include a space and another string
exprStrSearch="10.32.0.4:8055 10.32.0.7:8055"
Does not work.
echo "${_metadatadashjson}" | jq -r --arg EXPRSTRSEARCH "$exprStrSearch" '.dashboard.panels[] | select(.targets[].expr | contains($EXPRSTRSEARCH)) | .targets[].expr'

None of your data contains "10.32.0.4:8055 10.32.0.7:8055".
You could pass multiple strings to contains(), using a bash array:
strings=("10.32.0.4:8055" "10.32.0.7:8055")
echo "${_metadatadashjson}" |
jq -r --args '.dashboard.panels[] | select(.targets[].expr | contains($ARGS.positional[])) | .targets[].expr' "${strings[#]}"
But contains will evaluate to true for each match. Ie. if one expr contained both strings, it would be selected (and printed) twice.
With test, that won't happen. Here's how you can add the |s between multiple strings, and pass them in a single jq variable (as well as escape all the dots):
strings=("10.32.0.4:8055" "10.32.0.7:8055")
IFS=\|
echo "${_metadatadashjson}" |
jq -r --arg str "${strings[*]//./\\.}" '.dashboard.panels[] | select(.targets[].expr | test($str)) | .targets[].expr'
Both examples print this:
jvm_threads_current{instance="10.32.0.4:8055",job="prometheus_gyrex"}
jvm_threads_current{instance="10.32.0.7:8055",job="prometheus_gyrex"}
Update: I forgot to escape the dots for test. I edited the test example so that all the dots get escaped (with a single backslash). It's regex, so (unescaped) dots will match any character. The contains example matches the strings literally (not regex).

The problem is that the string with the space in it does not in fact occur in the given JSON. It's not too clear what you are trying to do but please note that contains is not symmetric:
"a" | contains("a b")
evaluates to false.
If you intended to write a boolean search criterion, you could use a boolean expression, or use jq's regular expression machinery, e.g.
test("10.32.0.4:8055|10.32.0.7:8055")
or probably even better:
test("\"(10[.]32[.]0[.]4:8055|10[.]32[.]0[.]7:8055)\"")

Related

How I can edit my response to 1 row + add condition to jq function

Hey I am using the next command
aws ec2 describe-volumes --filters Name=status,Values=available | jq '.Volumes[] | { State: .State, VolumeId: .VolumeId, Tags: .Tags}'
I get the next response:
{
"State": "available",
"VolumeId": "vol-03449dadd29f2067f",
"Tags": [
{
"Key": "sre",
"Value": "test"
}
]}
1.I want my response will be in 1 row regarding each volume.
2. I want to check if the volume has the tag "Name" if so I want the value, otherwise the tags are not interest me. How can I manipulate it ?
Use the -c option.
Use a query of the form select(any(Tags[]; CONDITION))
e.g. select(any(Tags[]; .Key == "Name"))
You can abbreviate expressions of the form {foo: .foo} to just {foo}
So, depending on your exact requirements, your invocation of jq could look like:
jq -c '.Volumes[] | {State, VolumeId, Tags} | select(any(.Tags[]; .Key == "Name"))'

looping through json object using jq

I have a json file which i have obtained using curl command and it looks as below.
{
"Storages": [
{
"Creation": "2020-04-21T14:01:54",
"Modified": "2020-04-21T14:01:54",
"Volume": "/dev/null",
"id": 10000,
"version": "20190925-230722"
},
{
"Creation": "2020-04-22T14:01:54",
"Modified": "2020-04-22T14:01:54",
"Volume": "/opt/home",
"id": 10001,
"version": "22a-20190925-230722"
},
{
"Creation": "2020-04-23T14:01:54",
"Modified": "2020-04-23T14:01:54",
"Volume": "/home/abcd",
"id": 10003,
"version": "21c-20190925-230722"
}
]
}
Now I need to loop thorough array and get id and volume values into 2 variables if version startswith 21a. No need to form another json
For educational purposes, here's a jq command that does both the things you want, but in 2 separate steps:
jq -r 'del(.Storages[] | select(.version | startswith("21a") | not))
.Storages[] | {id, version}'
The first part (del(.Storages[] | select(.version | startswith("21a") | not))) filters out the array elements that don't have a version starting with 21a. The second part (.Storages[] | {id, version}) drills and extracts the specific information you need.
You can use startswith builtin function such as
jq -r '.Storages[] | select(.version | startswith("21a")) | {id, Volume}'
Demo
Edit : Assuming the JSON embedded into a file(Storages.json), then you can assign the results into shell variables such as
$ readarray -t vars < <( jq -r '.Storages[] | select(.version|startswith("21a"))| .id, .Volume' Storages.json )
and display those variables as
$ declare -p vars
declare -a vars='([0]="10003" [1]="/home/abcd")'

use bash string as jq filter

I don't understand what I'm doing wrong or why this does not work.
test.json file:
[
{
"Header": {
"Region": "US",
"Tenant": "Tenant1",
"Stage": "testing",
"ProductType": "old"
},
"Body": []
},
{
"Header": {
"Region": "EU",
"Tenant": "Tenant2",
"Stage": "development",
"ProductType": "new"
},
"Body": []
}
]
I want to display the values of the .Header.Tenant key. So the simple jq call does its job:
$ jq '[.[].Header.Tenant]' test.json
[
"Tenant1",
"Tenant2"
]
Now I want to assign that jq filter to a bash variable and use it with jq's --arg variable.
And I am getting this:
$ a=".[].Header.Tenant"; jq --arg xx "$a" '[$xx]' test.json
[
".[].Header.Tenant"
]
What is wrong?
jq does not have an eval function for evaluating arbitrary jq expressions, but it does provide functions that can be used to achieve much the same effect, the key idea being that certain JSON values can be used to specify query operations.
In your case, you would have to translate the jq query into a suitable jq operation, such as:
jq --argjson a '["Header","Tenant"]' '
getpath(paths|select( .[- ($a|length) :]== $a))
' test.json
Extending jq's JSON-based query language
More interestingly, you could write your own eval, e.g.
jq --argjson a '[[], "Header","Tenant"]' '
def eval($expr):
if $expr == [] then .
else $expr[0] as $op
| if $op == [] then .[] | eval($expr[1:])
else getpath([$op]) | eval($expr[1:])
end
end;
eval($a)
' test.json
With eval.jq as a module
If the above def of eval were put in a file, say ~/jq/eval.jq, then you could simply write:
jq -L ~/jq --argjson a '[[], "Header","Tenant"]' '
include "eval";
eval($a)' test.json
Or you could specify the search path in the jq program:
jq --argjson a '[[], "Header","Tenant"]' '
include "eval" { "search": "~/jq" };
eval($a)' input.json
Or you could use import ...
TLDR; The following code does the job:
$ a=".[].Header.Tenant"; jq -f <(echo "[$a]") test.json
[
"Tenant1",
"Tenant2"
]
One as well can add/modify the filter in the jq call, if needed:
$ a=".[].Header.Tenant"; jq -f <(echo "[$a]|length") test.json
2
Longer explanation
My ultimate goal was to figure out how I can define the lowest common denominator jq filter in a variable and use it when calling jq, plus add additional parameters if necessary. If you have a really complex jq filter spanning multiple lines that you call frequently, you probably want to template it somehow and use that template when calling jq.
While peak demonstrated how it can be done, I think it is overengineering the simple task.
However, using process substitution combined with the jq's -f option to read a filter from the file does solve my problem.

Handling json object with special characters in jq

I have a json object with below element,
rsrecords="{
"ResourceRecords": [
{
"Value": "\"heritage=external-dns,external-dns/owner=us-east-1:sandbox,external-dns/resource=service/api""
}
],
"Type": "TXT",
"Name": "\\052.apiconsumer.alpha.sandbox.test.net.",
"TTL": 300
}"
And in my bash script,I have below code snippet,
jq -r '.[] | .Name ,.ResourceRecords[0].Value' <<< "$rsrecords" | \
while read -r name; read -r value; do
echo $name
Output is printed as,
\052.apiconsumer.alpha.sandbox.test.net.
But I am expecting it to print as \\052.apiconsumer.alpha.sandbox.test.net., which is , as it is "Name" from the json object..
How can this be done?
Before getting to the heart of the matter, please note that
the sample data as given is a bit of a mishmash, so I'll assume you meant something like:
rsrecords='
{
"ResourceRecords": [
{
"Value": "heritage=external-dns,external-dns/owner=us-east-1:sandbox,external-dns/resource=service/api"
}
],
"Type": "TXT",
"Name": "\\052.apiconsumer.alpha.sandbox.test.net.",
"TTL": 300
}
'
Your jq query does not match the above JSON, so I'll assume you intended the query to be simply:
.Name, .ResourceRecords[0].Value
In any case, with the above JSON, the bash commands:
jq -r '.Name, .ResourceRecords[0].Value' <<< "$rsrecords" |
while read -r name; read -r value; do
echo "$name"
done
yields:
\052.apiconsumer.alpha.sandbox.test.net.
This is correct, because the JSON string "\\X" is an encoding of the raw string: \X
If you want to see the JSON string, then invoke jq without the -r option. If you want to invoke jq with the -r option and want to see two backslashes, you will have to encode them as four backslashes in your JSON.

jq how to get the last entry in an object

I'm working with jq 1.6 to get the last entry in an object. It should work like this:
data='{ "1": { "a": "1" }, "2": { "a": "2" }, "3": { "a": "3" } }'
result=`echo $data | jq 'myfilter'`
echo $result
{ "3": { "a": "3" } }
I tried these filters:
jq '. | last' # error: Cannot index object with number
How can I tell jq to quote the number?
jq '. | to_entries | last' # { "key": "3", "value": { "a": "3" } }
I guess I could munge this up by concatenating the key and value entries. Is there a simpler way?
The tutorial and the manual didn't help. No joy on SO either.
You can use the following :
jq 'to_entries | [last] | from_entries'
Try it here.
We can't use with_entries(last) because last returns a single element and from_entries requires an array, hence the [...] construct above.

Resources