use bash string as jq filter - bash

I don't understand what I'm doing wrong or why this does not work.
test.json file:
[
{
"Header": {
"Region": "US",
"Tenant": "Tenant1",
"Stage": "testing",
"ProductType": "old"
},
"Body": []
},
{
"Header": {
"Region": "EU",
"Tenant": "Tenant2",
"Stage": "development",
"ProductType": "new"
},
"Body": []
}
]
I want to display the values of the .Header.Tenant key. So the simple jq call does its job:
$ jq '[.[].Header.Tenant]' test.json
[
"Tenant1",
"Tenant2"
]
Now I want to assign that jq filter to a bash variable and use it with jq's --arg variable.
And I am getting this:
$ a=".[].Header.Tenant"; jq --arg xx "$a" '[$xx]' test.json
[
".[].Header.Tenant"
]
What is wrong?

jq does not have an eval function for evaluating arbitrary jq expressions, but it does provide functions that can be used to achieve much the same effect, the key idea being that certain JSON values can be used to specify query operations.
In your case, you would have to translate the jq query into a suitable jq operation, such as:
jq --argjson a '["Header","Tenant"]' '
getpath(paths|select( .[- ($a|length) :]== $a))
' test.json
Extending jq's JSON-based query language
More interestingly, you could write your own eval, e.g.
jq --argjson a '[[], "Header","Tenant"]' '
def eval($expr):
if $expr == [] then .
else $expr[0] as $op
| if $op == [] then .[] | eval($expr[1:])
else getpath([$op]) | eval($expr[1:])
end
end;
eval($a)
' test.json
With eval.jq as a module
If the above def of eval were put in a file, say ~/jq/eval.jq, then you could simply write:
jq -L ~/jq --argjson a '[[], "Header","Tenant"]' '
include "eval";
eval($a)' test.json
Or you could specify the search path in the jq program:
jq --argjson a '[[], "Header","Tenant"]' '
include "eval" { "search": "~/jq" };
eval($a)' input.json
Or you could use import ...

TLDR; The following code does the job:
$ a=".[].Header.Tenant"; jq -f <(echo "[$a]") test.json
[
"Tenant1",
"Tenant2"
]
One as well can add/modify the filter in the jq call, if needed:
$ a=".[].Header.Tenant"; jq -f <(echo "[$a]|length") test.json
2
Longer explanation
My ultimate goal was to figure out how I can define the lowest common denominator jq filter in a variable and use it when calling jq, plus add additional parameters if necessary. If you have a really complex jq filter spanning multiple lines that you call frequently, you probably want to template it somehow and use that template when calling jq.
While peak demonstrated how it can be done, I think it is overengineering the simple task.
However, using process substitution combined with the jq's -f option to read a filter from the file does solve my problem.

Related

How to specify jq output fields from variable in bash?

given the following (simplified) json file:
{
"data": [
{
"datum": "2023-01-11 00:00:00",
"prijs": "0.005000",
"prijsZP": "0.161550",
"prijsEE": "0.181484",
"prijsTI": "0.160970",
},
{
"datum": "2023-01-11 01:00:00",
"prijs": "0.000000",
"prijsZP": "0.155500",
"prijsEE": "0.175434",
"prijsTI": "0.154920",
}
]
}
I want to specify in my jq command which fields to retreive, i.e. only "datum" and "prijsTI". But on another moment this selection will be different.
I use the following command to gather all the fields, but would like to set the field selection via a variable:
cat data.json |jq -r '.data[]|[.datum, .prijsTI]|#csv'
I already tried using arguments, but this did not work :-(
myJQselect=".datum, .prijsTI"
cat data.json |jq -r --arg myJQselect "$myJQselect" '.data[$myHour |tonumber]|[$myJQselect]|#csv'
gives the following result: ".datum, .prijs" instead of the correct values.
Would this be possible?
Thanks,
Jeroen
You can use the --args option to provide a variable number of fields to query, then use the $ARGS.positional array to retrieve them:
jq -r '.data[] | [.[$ARGS.positional[]]] | #csv' data.json --args datum prijsTI
"2023-01-11 00:00:00","0.160970"
"2023-01-11 01:00:00","0.154920"

jq format when running from a bash script with variable expansion

I've got a jq command that works when running directly from the shell or from within a shell script, but when I try to add variable expansion, I get jq errors for unexpected format or invalid characters. My goal is to have a quick and easy way to update some json configuration.
Here's a simplified example.
The format of the json I'm modifying:
{
"pets": {
"some-new-pet": {
"PetInfo": {
"name": "my-brand-new-pet",
"toys": [
"toy1-postfix",
"toy2-postfix",
"toy3-postfix"
]
}
}
}
}
The jq without variable expansion:
cat myfile.json | jq '.pets."some-new-pet" += {PetInfo: {name: "my-brand-new-pet"}, toys: ["toy1", "toy2", "toy3"]}}'
The above runs fine, and adds the new pets.some-new-pet entry to my json.
Below is what I'm trying to do with variable expansion that fails.
jq_args = "'.pets.\"${PET}\" += {PetInfo: {name: \"${NAME}\"}, toys: [\"${toy1}-postfix\", \"${toy2}-postfix\", \"${toy3}-postfix\"]}}'"
cat myfile.json | jq $jq_args
The error message I get with the above:
jq: error: syntax error, unexpected INVALID_CHARACTER, expecting $end (Unix shell quoting issues?) at <top-level>, line 1: '.pets."some-new-pet"
My file is formatted as utf-8 and uses LF line endings.
I do not recommend constructing a jq filter using variable expansion or printf. It will work for simple cases but will fail if the string contains double quotes, backslashes or control-codes, as they have special meanings inside a JSON string. As an alternative to using printf, jq has a way to pass in variables directly via the command-line, avoiding all these issues.
pet='some-second-pet'
name='my-even-newer'
toy1=toy1
toy2=toy2
toy3=toy3
jq \
--arg pet "$pet" \
--arg name "$name" \
--arg toy1 "$toy1" \
--arg toy2 "$toy2" \
--arg toy3 "$toy3" \
'.pets[$pet] += {
PetInfo: {name: $name},
toys: ["\($toy1)-postfix", "\($toy2)-postfix", "\($toy3)-postfix"]
}' \
myfile.json
Output:
{
"pets": {
"some-new-pet": {
"PetInfo": {
"name": "my-brand-new-pet",
"toys": [
"toy1-postfix",
"toy2-postfix",
"toy3-postfix"
]
}
},
"some-second-pet": {
"PetInfo": {
"name": "my-even-newer-pet"
},
"toys": [
"toy1-postfix",
"toy2-postfix",
"toy3-postfix"
]
}
}
}
It would be cleaner and less error prone to format the string using printf
PET='dog'
NAME='sam'
toy1="t1"
toy2="t2"
toy3="t3"
jq_args=$(printf '.pets."%s" += {PetInfo: {name: "%s"}, toys: ["%s-postfix", "%s-postfix", "%s-postfix"]}}' "${PET}" "${NAME}" "${toy1}" "${toy2}" "${toy3}")
echo "$jq_args"
Result:
.pets."dog" += {PetInfo: {name: "sam"}, toys: ["t1-postfix", "t2-postfix", "t3-postfix"]}
Additionally, redundant quoting could be avoided by quoting the arg on this command
cat myfile.json | jq "$jq_args"
Fix your jq code by removing extra } at end
Fix bash jq call:
Add cotes "..." around your $jq_args
so don't use singles '...' in your jq_args definition
Use printf with -v option to define jq_args:
printf -v jq_args "...format..." value1 value2 ...
So your code became:
PET="some-new-pet"
NAME="my-brand-new-pet"
toy1="toy1"
toy2="toy2"
toy3="toy3"
format='.pets."%s" += {PetInfo: {name: "%s"}, toys: ["%s", "%s", "%s"]}'
printf -v jq_args "${format}" "${PET}" "${NAME}" "${toy1}" "${toy2}" "${toy3}"
cat myfile.json | jq "$jq_args"
Output:
{
"pets": {
"some-new-pet": {
"PetInfo": {
"name": "my-brand-new-pet"
},
"toys": [
"toy1",
"toy2",
"toy3"
]
}
}
}
Notes:
When you define your format, you put it into simple cotes '...'. It's really better to format JSON (or XML) without back-slashes (\\) before each double cotes (")
Use printf -v variable_name. It's more readable than var_name=$(printf ...)
By constructing the jq filter ("code") using outer bash variables ("data") you may run into escaping issues, which could eventually break or even divert your filter. (see https://en.wikipedia.org/wiki/Code_injection)
Instead, use mechanisms by jq to introduce external data through variables (parameter --arg):
jq --arg pet "${PET}" \
--arg name "${NAME}" \
--arg toy1 "${toy1}-postfix" \
--arg toy2 "${toy2}-postfix" \
--arg toy3 "${toy3}-postfix" \
'
.pets[$pet] += {PetInfo: {$name, toys: [$toy1,$toy2,$toy3]}}
' myfile.json
If you have an unknown number of variables to include, check out jq's --args parameter (note the additional s)

Handling json object with special characters in jq

I have a json object with below element,
rsrecords="{
"ResourceRecords": [
{
"Value": "\"heritage=external-dns,external-dns/owner=us-east-1:sandbox,external-dns/resource=service/api""
}
],
"Type": "TXT",
"Name": "\\052.apiconsumer.alpha.sandbox.test.net.",
"TTL": 300
}"
And in my bash script,I have below code snippet,
jq -r '.[] | .Name ,.ResourceRecords[0].Value' <<< "$rsrecords" | \
while read -r name; read -r value; do
echo $name
Output is printed as,
\052.apiconsumer.alpha.sandbox.test.net.
But I am expecting it to print as \\052.apiconsumer.alpha.sandbox.test.net., which is , as it is "Name" from the json object..
How can this be done?
Before getting to the heart of the matter, please note that
the sample data as given is a bit of a mishmash, so I'll assume you meant something like:
rsrecords='
{
"ResourceRecords": [
{
"Value": "heritage=external-dns,external-dns/owner=us-east-1:sandbox,external-dns/resource=service/api"
}
],
"Type": "TXT",
"Name": "\\052.apiconsumer.alpha.sandbox.test.net.",
"TTL": 300
}
'
Your jq query does not match the above JSON, so I'll assume you intended the query to be simply:
.Name, .ResourceRecords[0].Value
In any case, with the above JSON, the bash commands:
jq -r '.Name, .ResourceRecords[0].Value' <<< "$rsrecords" |
while read -r name; read -r value; do
echo "$name"
done
yields:
\052.apiconsumer.alpha.sandbox.test.net.
This is correct, because the JSON string "\\X" is an encoding of the raw string: \X
If you want to see the JSON string, then invoke jq without the -r option. If you want to invoke jq with the -r option and want to see two backslashes, you will have to encode them as four backslashes in your JSON.

cannot call bash environment variable inside jq

In the below script, I am not able to successfully call the "repovar" variable in the jq command.
cat quayrepo.txt | while read line
do
export repovar="$line"
jq -r --arg repovar "$repovar" '.data.Layer| .Features[] | "\(.Name), \(.Version), $repovar"' severity.json > volume.csv
done
The script uses a text file to loop through the repo names
quayrepo.txt---> file has the list of names in this case the file has a value of "Reponame1"
sample input severity.json file:
{
"status": "scanned",
"data": {
"Layer": {
"IndexedByVersion": 3,
"Features": [
{
"Name": "elfutils",
"Version": "0.168-1",
"Vulnerabilities": [
{
"NamespaceName": "debian:9",
"Severity": "Medium",
"Name": "CVE-2016-2779"
}
]
}
]
}
}
}
desired output:
elfutils, 0.168-1, Medium, Reponame1
Required output: I need to retrieve the value of my environment variable as the last column in my output csv file
You need to surround $repovar with parenthesis, as the other values
repovar='qweqe'; jq -r --arg repovar "$repovar" '.data.Layer| .Features[] | "\(.Name), \(.Version), \($repovar)"' tmp.json
Result:
elfutils, 0.168-1, qweqe
There's no need for the export.
#!/usr/bin/env bash
while read line
do
jq -r --arg repovar "$line" '.data.Layer.Features[] | .Name + ", " + .Version + ", " + $repovar' severity.json
done < quayrepo.txt > volume.csv
with quayrepo.txt as
Reponame1
and severity.json as
{
"status": "scanned",
"data": {
"Layer": {
"IndexedByVersion": 3,
"Features": [
{
"Name": "elfutils",
"Version": "0.168-1",
"Vulnerabilities": [
{
"NamespaceName": "debian:9",
"Severity": "Medium",
"Name": "CVE-2016-2779"
}
]
}
]
}
}
}
produces volume.csv containing
elfutils, 0.168-1, Reponame1
To #peak's point, changing > to >> in ...severity.json >> volume.csv will create a multi-line csv instead of just overwriting until the last line
You don't need a while read loop in bash at all; jq itself can loop over your input lines, even when they aren't JSON, letting you run jq only once, not once per line in quayrepo.txt.
jq -rR --slurpfile inJson severity.json <quayrepo.txt >volume.csv '
($inJson[0].data.Layer | .Features[]) as $features |
[$features.Name, $features.Version, .] |
#csv
'
jq -R specifies raw input, letting jq directly read lines from quayrepo.txt into .
jq --slurpfile varname filename.json reads filename.json into an array of JSON objects parsed from that file. If the file contains only one object, one needs to refer to $varname[0] to refer to it.
#csv converts an array to a CSV output line, correctly handling data with embedded quotes or other oddities that require special processing.

How to fetch next URL from JSON using Bash

I have created a simple script to store response from a third party API
The request is like this..
https://externalservice.io/orders?key=password&records=50&offset=0
The response is as follows:
{
"results": [
{
"engagement": {
"id": 29090716,
"portalId": 62515,
"active": true,
"createdAt": 1444223400781,
"lastUpdated": 1444223400781,
"createdBy": 215482,
"modifiedBy": 215482,
"ownerId": 70,
"type": "NOTE",
"timestamp": 1444223400781
},
},
],
"hasMore": true,
"offset": 4623406
}
If there is a hasMore attribute, I need to read the offset value to get the next set of records.
Right now I've created a script that simply loops over the estimated number of records (I believed there is) and thought incrementing the offset would work but this is not the case as the offset is not incremental.
#!/bin/bash
for i in {1..100}; do
curl -s "https://externalservice.io/orders?key=password&records=50&offset=$i" >>outfile.txt 2>&1
done
Can someone explain how I can read continue the script reading the offset value until hasMore=false?
You can read a value from a json using the jq utility:
$ jq -r ".hasMore" outfile
true
Here is what you could use:
more="true"
offset=0
while [ $more = "true" ]; do
echo $offset
response=$(curl -s "https://example.com/orders?offset=$offset")
more=$(echo $response | tr '\r\n' ' ' | jq -r ".hasMore")
offset=$(echo $response | tr '\r\n' ' ' | jq -r ".offset")
done
You can use jq utility to extract specific attribute from you json response:
$ jq -r ".hasMore" outfile
true
jq expects perfectly valid json input, otherwise it will print error.
Most common mistake is to echo json stored in variable. A mistake, because echo will interpret all escaped newlines in your json and will send unescaped newlines within values causing jq to throw parse error: Invalid string: control characters from U+0000 through U+001F must be escaped at line message.
To avoid modification to json (and avoiding an error) we need to use printf instead of echo.
Here is the solution without breaking json content:
more="true"
offset=0
while [ $more = "true" ]; do
echo $offset
response=$(curl -s "https://example.com/orders?offset=$offset")
more=$(printf "%s\n" "$response" | jq -r ".hasMore")
offset=$(printf "%s\n" "$response" | jq -r ".offset")
done

Resources