How can I use jq 'try-catch' inside of an object construction - bash

The general problem statement is that I have some json data with fields that may or may not exist. Patterns like try .foo catch "bar" or .foo? // "bar" work just fine on their own, but not inside of an object construction cat myfile.json | jq -r '{foo: try .foo catch "bar"}'. Using an object construction is important for my use case, where I need to pass the same input through many filters and format the data nicely for later functions.
The data I'm working with is the output of kubectl get pods xxxxx -o json, and the part I'm trying to parse looks like this
{
"items": [
{
"status": {
"conditions": [
{
"type": "Initialized",
"message": "foo"
},
{
"type": "Ready"
}
]
}
},
{
"status": {
}
}
]
}
For each item, if it has .status.conditions, I want the message from the first element in .status.conditions that has a message. The filter I used looks like this
jq -r '.items[] | {message: .status.conditions?|map(select(has("message")).message)[0]?}'
The problem is that when it gets to an item that doesn't have a .status.conditions, It returns the error Cannot iterate over null (null). This makes sense given that .status.conditions? passes null to the next filter, map which needs to iterate over a list. My attempted solution was to use various ways of try-catch to pass an empty list to map instead of null
jq -r '.items[] | {message: .status | try .conditions catch [] |map(select(has("message")).message)[0]?}'
jq -r '.items[] | {message: .status.conditions? // []|map(select(has("message")).message)[0]?}'
or to use a ? that includes the map call
jq -r '.items[] | {message: (.status.conditions?|map(select(has("message")).message)[0])?}'
jq -r '.items[] | {message: .status.conditions?|(map(select(has("message")).message)[0])?}'
All of these attempts return 1 compile error when written inside of an object constructor as shown, and work as expected when written on their own without an object constructor (e.g. '.items[] | .status.conditions?|(map(select(has("message")).message)[0])?')
The jq man page doesn't give any warnings (that I noticed) about how object construction changes the syntax requirements of what's inside, nor how try-catch can be affected by being inside an object construction. Any ideas?

Thanks to #Fravadona, it was just a matter of parentheses. I had tried it at some point and made some mistake or other, but a working solution for my case is
jq -r '{message: .status | (.conditions? // []) | map(select(has("message")).message)[0]?}'

Related

How to specify jq output fields from variable in bash?

given the following (simplified) json file:
{
"data": [
{
"datum": "2023-01-11 00:00:00",
"prijs": "0.005000",
"prijsZP": "0.161550",
"prijsEE": "0.181484",
"prijsTI": "0.160970",
},
{
"datum": "2023-01-11 01:00:00",
"prijs": "0.000000",
"prijsZP": "0.155500",
"prijsEE": "0.175434",
"prijsTI": "0.154920",
}
]
}
I want to specify in my jq command which fields to retreive, i.e. only "datum" and "prijsTI". But on another moment this selection will be different.
I use the following command to gather all the fields, but would like to set the field selection via a variable:
cat data.json |jq -r '.data[]|[.datum, .prijsTI]|#csv'
I already tried using arguments, but this did not work :-(
myJQselect=".datum, .prijsTI"
cat data.json |jq -r --arg myJQselect "$myJQselect" '.data[$myHour |tonumber]|[$myJQselect]|#csv'
gives the following result: ".datum, .prijs" instead of the correct values.
Would this be possible?
Thanks,
Jeroen
You can use the --args option to provide a variable number of fields to query, then use the $ARGS.positional array to retrieve them:
jq -r '.data[] | [.[$ARGS.positional[]]] | #csv' data.json --args datum prijsTI
"2023-01-11 00:00:00","0.160970"
"2023-01-11 01:00:00","0.154920"

jq How to pass key starting with numeral as argument [duplicate]

I am new to jq and facing an issue while parsing my json
I have a json stored in a variable like this
temp='{ "1": { "my_name": "one" }, "2": { "my_name": "two" } }'
Now I need to get the value of my_name for both other entries
I have tried something like this
echo $temp | jq '.1' //out put 0.1
I was assuming to get { "my_name": "one" }
And similarly to get my_name value I did
echo $temp | jq '.1.my_name' // Its output is giving me error
Can any one please help to identify what is wrong with my syntax and how can I correct it.
Just the number gets interpreted as a float. You need to use it in a context where it is unambiguously a key string.
echo "$temp" | jq '.["1"]["my_name"]'
and
echo "$temp" | jq '.["1"]'
to get the containing dict.
With a sufficiently new jq (I think >= 1.4) you can also say
echo "$temp" | jq '."1".my_name'
Whenever you are trying to reference a key that is not a valid identifier, you have to quote it. See the manual for more details.
To select the item under the key 1, you'd do this:
."1"
For your other question on how to obtain the my_name values, you could do this:
to_entries | map(.value.my_name)

use bash string as jq filter

I don't understand what I'm doing wrong or why this does not work.
test.json file:
[
{
"Header": {
"Region": "US",
"Tenant": "Tenant1",
"Stage": "testing",
"ProductType": "old"
},
"Body": []
},
{
"Header": {
"Region": "EU",
"Tenant": "Tenant2",
"Stage": "development",
"ProductType": "new"
},
"Body": []
}
]
I want to display the values of the .Header.Tenant key. So the simple jq call does its job:
$ jq '[.[].Header.Tenant]' test.json
[
"Tenant1",
"Tenant2"
]
Now I want to assign that jq filter to a bash variable and use it with jq's --arg variable.
And I am getting this:
$ a=".[].Header.Tenant"; jq --arg xx "$a" '[$xx]' test.json
[
".[].Header.Tenant"
]
What is wrong?
jq does not have an eval function for evaluating arbitrary jq expressions, but it does provide functions that can be used to achieve much the same effect, the key idea being that certain JSON values can be used to specify query operations.
In your case, you would have to translate the jq query into a suitable jq operation, such as:
jq --argjson a '["Header","Tenant"]' '
getpath(paths|select( .[- ($a|length) :]== $a))
' test.json
Extending jq's JSON-based query language
More interestingly, you could write your own eval, e.g.
jq --argjson a '[[], "Header","Tenant"]' '
def eval($expr):
if $expr == [] then .
else $expr[0] as $op
| if $op == [] then .[] | eval($expr[1:])
else getpath([$op]) | eval($expr[1:])
end
end;
eval($a)
' test.json
With eval.jq as a module
If the above def of eval were put in a file, say ~/jq/eval.jq, then you could simply write:
jq -L ~/jq --argjson a '[[], "Header","Tenant"]' '
include "eval";
eval($a)' test.json
Or you could specify the search path in the jq program:
jq --argjson a '[[], "Header","Tenant"]' '
include "eval" { "search": "~/jq" };
eval($a)' input.json
Or you could use import ...
TLDR; The following code does the job:
$ a=".[].Header.Tenant"; jq -f <(echo "[$a]") test.json
[
"Tenant1",
"Tenant2"
]
One as well can add/modify the filter in the jq call, if needed:
$ a=".[].Header.Tenant"; jq -f <(echo "[$a]|length") test.json
2
Longer explanation
My ultimate goal was to figure out how I can define the lowest common denominator jq filter in a variable and use it when calling jq, plus add additional parameters if necessary. If you have a really complex jq filter spanning multiple lines that you call frequently, you probably want to template it somehow and use that template when calling jq.
While peak demonstrated how it can be done, I think it is overengineering the simple task.
However, using process substitution combined with the jq's -f option to read a filter from the file does solve my problem.

How do I concatenate dummy values in JQ based on field value, and then CSV-aggregate these concatenations?

In my bash script, when I run the following jq against my curl result:
curl -u someKey:someSecret someURL 2>/dev/null | jq -r '.schema' | jq -r -c '.fields'
I get back a JSON array as follows:
[{"name":"id","type":"int","doc":"Documentation for the id field."},{"name":"test_string","type":"string","doc":"Documentation for the test_string field"}]
My goal is to do a call with jq applied to return the following (given the example above):
{"id":1234567890,"test_string":"xxxxxxxxxx"}
NB: I am trying to automatically generate templated values that match the "schema" JSON shown above.
So just to clarify, that is:
all array objects (there could be more than 2 shown above) returned in a single comma-delimited row
doc fields are ignored
the values for "name" (including their surrounding double-quotes) are concatenated with either:
:1234567890 ...when the "type" for that object is "int"
":xxxxxxxxxx" ...when the "type" for that object is "string"
NB: these will be the only types we ever get for now
Can someone show me how I can expand upon my initial jq to return this?
NB: I tried working down the following path but am failing beyond this...
curl -u someKey:someSecret someURL 2>/dev/null | jq -r '.schema' | jq -r -c '.fields' | "\(.name):xxxxxxxxxxx"'
If it's not possible in pure JQ (my preference) I'm also happy for a solution that mixes in a bit of sed/awk magic :)
Cheers,
Stan
Given the JSON shown, you could add the following to your pipeline:
jq -c 'map({(.name): (if .type == "int" then 1234567890 else "xxxxxxxxxx" end)})|add'
With that JSON, the output would be:
{"id":1234567890,"test_string":"xxxxxxxxxx"}
However, it would be far better if you combined the three calls to jq into one.

jq command throws error "cannot iterate over string" when

I am writing a bash script utilizing jq to filter out JSON entries given some bash variables and return some of the key values from each entry into a tab-delimited file. I think the first few lines of this command are okay, but the 4th line I think is causing the problem. I have tried piping each entry in line 4 to tostring but to no avail.
info=`cat $FILE | jq -r \
' .[] \
| map(select(.host| contains(env.A))) \
| [."ip.A",."ts",."ip.B"] \
| #tsv'`
JSON example entry:
{
"ts": "2019-06-19T00:00:00.000000Z",
"ip.A": "0.0.0.0",
"ip.B": "0.0.0.0",
"host": "www.google.com",
}
In these files, there are no brackets surrounding the entire text within the file.
Error Given:
jq: error (at <stdin>:0): Cannot iterate over string ("2019-06-18...)
Do I need to handle ".ts" in some special way?
This code is broken long before the 3rd line.
If there isn't an outer array or object, you can't use .[].
If your data type is an object and not a list, using map() on it throws away data (specifically, it discards the keys, leaving only the values).
...so, .[] iterates over the values in your object, and then map() tries to iterate over each of those values as if it was an iterable object itself, which it's not... hence your error.
A version cut down to remove the broken parts might look like:
a="google.com" jq -r '
if (.host | contains(env.a)) then
[."ip.A",."ts",."ip.B"] | #tsv
else
empty
end
' <<'EOF'
{
"ts": "2019-06-19T00:00:00.000000Z",
"ip.A": "0.0.0.0",
"ip.B": "0.0.0.0",
"host": "www.google.com"
}
EOF
...which works fine.

Resources