I have the following JSON named as my.json.
[
{
"action": "copy",
"artifact_location": "one foo one"
},
{
"action": "copy",
"artifact_location": "one bar one"
},
{
"action": "remove",
"artifact_location": "two foo two"
}
]
My goal is to delete all the objects in the root JSON array if the artifact_location property of the a object contains the string value "foo".
I'm using jq command line utility to accomplish this task.
Following is my jq command. It is working perfectly when I'm running it on my local machine (macOS and jq version is 1.6).
jq 'del(.[] | select(.artifact_location | test("foo")))' my.json
However, the above commands gives the following error when I try to run it as a shell script in a Jenkins job(Ubuntu and jq version is 1.3).
error: test is not defined
del(.[] | select(.artifact_location | test("foo")))
^^^^
1 compile error
What am I possibly doing wrong here?
test/0 is for testing if a string matches a particular regular expression, which is not available in jq 1.3 (as mentioned in the comments). contains/1 could be used in this case.
del(.[] | select(.artifact_location | contains("foo")))
I would rather approach this as filtering out the objects, rather than deleting them. Select objects that does not contain "foo".
map(select(.artifact_location | contains("foo") | not))
Related
given the following (simplified) json file:
{
"data": [
{
"datum": "2023-01-11 00:00:00",
"prijs": "0.005000",
"prijsZP": "0.161550",
"prijsEE": "0.181484",
"prijsTI": "0.160970",
},
{
"datum": "2023-01-11 01:00:00",
"prijs": "0.000000",
"prijsZP": "0.155500",
"prijsEE": "0.175434",
"prijsTI": "0.154920",
}
]
}
I want to specify in my jq command which fields to retreive, i.e. only "datum" and "prijsTI". But on another moment this selection will be different.
I use the following command to gather all the fields, but would like to set the field selection via a variable:
cat data.json |jq -r '.data[]|[.datum, .prijsTI]|#csv'
I already tried using arguments, but this did not work :-(
myJQselect=".datum, .prijsTI"
cat data.json |jq -r --arg myJQselect "$myJQselect" '.data[$myHour |tonumber]|[$myJQselect]|#csv'
gives the following result: ".datum, .prijs" instead of the correct values.
Would this be possible?
Thanks,
Jeroen
You can use the --args option to provide a variable number of fields to query, then use the $ARGS.positional array to retrieve them:
jq -r '.data[] | [.[$ARGS.positional[]]] | #csv' data.json --args datum prijsTI
"2023-01-11 00:00:00","0.160970"
"2023-01-11 01:00:00","0.154920"
The general problem statement is that I have some json data with fields that may or may not exist. Patterns like try .foo catch "bar" or .foo? // "bar" work just fine on their own, but not inside of an object construction cat myfile.json | jq -r '{foo: try .foo catch "bar"}'. Using an object construction is important for my use case, where I need to pass the same input through many filters and format the data nicely for later functions.
The data I'm working with is the output of kubectl get pods xxxxx -o json, and the part I'm trying to parse looks like this
{
"items": [
{
"status": {
"conditions": [
{
"type": "Initialized",
"message": "foo"
},
{
"type": "Ready"
}
]
}
},
{
"status": {
}
}
]
}
For each item, if it has .status.conditions, I want the message from the first element in .status.conditions that has a message. The filter I used looks like this
jq -r '.items[] | {message: .status.conditions?|map(select(has("message")).message)[0]?}'
The problem is that when it gets to an item that doesn't have a .status.conditions, It returns the error Cannot iterate over null (null). This makes sense given that .status.conditions? passes null to the next filter, map which needs to iterate over a list. My attempted solution was to use various ways of try-catch to pass an empty list to map instead of null
jq -r '.items[] | {message: .status | try .conditions catch [] |map(select(has("message")).message)[0]?}'
jq -r '.items[] | {message: .status.conditions? // []|map(select(has("message")).message)[0]?}'
or to use a ? that includes the map call
jq -r '.items[] | {message: (.status.conditions?|map(select(has("message")).message)[0])?}'
jq -r '.items[] | {message: .status.conditions?|(map(select(has("message")).message)[0])?}'
All of these attempts return 1 compile error when written inside of an object constructor as shown, and work as expected when written on their own without an object constructor (e.g. '.items[] | .status.conditions?|(map(select(has("message")).message)[0])?')
The jq man page doesn't give any warnings (that I noticed) about how object construction changes the syntax requirements of what's inside, nor how try-catch can be affected by being inside an object construction. Any ideas?
Thanks to #Fravadona, it was just a matter of parentheses. I had tried it at some point and made some mistake or other, but a working solution for my case is
jq -r '{message: .status | (.conditions? // []) | map(select(has("message")).message)[0]?}'
I have an issue where I am trying to filter records with a CONTAINS, but it won't accept a variable that has spaces in it. I am including the JSON and the calls. I explain what works and the last one that does not work. I have looked High and Low but I can't make it work. I have seen and tried many (hundreds of ways taking into account the double quotes, escaped, not escaped, with, without, but no luck) can someone take a look and point me to something that might help.
JSON used to test
_metadatadashjson='{ "meta": { "provisionedExternalId": "" }, "dashboard": { "liveNow": false, "panels": [ { "collapsed": false, "title": "Gyrex Thread Count Gauges", "type": "row", "targets": [ { "expr": "jvm_threads_current{instance=\"192.1.50.22:8055\",job=\"prometheus_gyrex\"}", "refId": "B" } ] }, { "datasource": "Prometheus_16_Docker", "targets": [ { "exemplar": true, "expr": "jvm_threads_current{instance=\"10.32.0.4:8055\",job=\"prometheus_gyrex\"}" } ], "title": ".16 : 3279", "type": "gauge" }, { "description": "", "targets": [ { "expr": "jvm_threads_current{instance=\"10.32.0.7:8055\",job=\"prometheus_gyrex\"}", "refId": "B" } ], "title": ".16 : 3288", "type": "graph" }, { "description": "", "targets": [ { "expr": "jvm_threads_current{instance=\"192.168.2.16:3288\",job=\"prometheus_gyrex\"}", "refId": "C" } ], "title": ".16 : 3288", "type": "graph" } ], "version": 55 }}'
Set the string to search for in key "expr"
exprStrSearch="10.32.0.4:8055"
This works returns one record
echo "${_metadatadashjson}" | jq -r --arg EXPRSTRSEARCH "$exprStrSearch" '.dashboard.panels[] | select(.targets[].expr | contains($EXPRSTRSEARCH)) | .targets[].expr'
This works no problem returns two records.
echo "${_metadatadashjson}" | jq -r --arg EXPRSTRSEARCH "$exprStrSearch" '.dashboard.panels[] | select(.targets[].expr | contains("10.32.0.4:8055", "10.32.0.7:8055")) | .targets[].expr'
Change the value to include a space and another string
exprStrSearch="10.32.0.4:8055 10.32.0.7:8055"
Does not work.
echo "${_metadatadashjson}" | jq -r --arg EXPRSTRSEARCH "$exprStrSearch" '.dashboard.panels[] | select(.targets[].expr | contains($EXPRSTRSEARCH)) | .targets[].expr'
None of your data contains "10.32.0.4:8055 10.32.0.7:8055".
You could pass multiple strings to contains(), using a bash array:
strings=("10.32.0.4:8055" "10.32.0.7:8055")
echo "${_metadatadashjson}" |
jq -r --args '.dashboard.panels[] | select(.targets[].expr | contains($ARGS.positional[])) | .targets[].expr' "${strings[#]}"
But contains will evaluate to true for each match. Ie. if one expr contained both strings, it would be selected (and printed) twice.
With test, that won't happen. Here's how you can add the |s between multiple strings, and pass them in a single jq variable (as well as escape all the dots):
strings=("10.32.0.4:8055" "10.32.0.7:8055")
IFS=\|
echo "${_metadatadashjson}" |
jq -r --arg str "${strings[*]//./\\.}" '.dashboard.panels[] | select(.targets[].expr | test($str)) | .targets[].expr'
Both examples print this:
jvm_threads_current{instance="10.32.0.4:8055",job="prometheus_gyrex"}
jvm_threads_current{instance="10.32.0.7:8055",job="prometheus_gyrex"}
Update: I forgot to escape the dots for test. I edited the test example so that all the dots get escaped (with a single backslash). It's regex, so (unescaped) dots will match any character. The contains example matches the strings literally (not regex).
The problem is that the string with the space in it does not in fact occur in the given JSON. It's not too clear what you are trying to do but please note that contains is not symmetric:
"a" | contains("a b")
evaluates to false.
If you intended to write a boolean search criterion, you could use a boolean expression, or use jq's regular expression machinery, e.g.
test("10.32.0.4:8055|10.32.0.7:8055")
or probably even better:
test("\"(10[.]32[.]0[.]4:8055|10[.]32[.]0[.]7:8055)\"")
In my bash script, when I run the following jq against my curl result:
curl -u someKey:someSecret someURL 2>/dev/null | jq -r '.schema' | jq -r -c '.fields'
I get back a JSON array as follows:
[{"name":"id","type":"int","doc":"Documentation for the id field."},{"name":"test_string","type":"string","doc":"Documentation for the test_string field"}]
My goal is to do a call with jq applied to return the following (given the example above):
{"id":1234567890,"test_string":"xxxxxxxxxx"}
NB: I am trying to automatically generate templated values that match the "schema" JSON shown above.
So just to clarify, that is:
all array objects (there could be more than 2 shown above) returned in a single comma-delimited row
doc fields are ignored
the values for "name" (including their surrounding double-quotes) are concatenated with either:
:1234567890 ...when the "type" for that object is "int"
":xxxxxxxxxx" ...when the "type" for that object is "string"
NB: these will be the only types we ever get for now
Can someone show me how I can expand upon my initial jq to return this?
NB: I tried working down the following path but am failing beyond this...
curl -u someKey:someSecret someURL 2>/dev/null | jq -r '.schema' | jq -r -c '.fields' | "\(.name):xxxxxxxxxxx"'
If it's not possible in pure JQ (my preference) I'm also happy for a solution that mixes in a bit of sed/awk magic :)
Cheers,
Stan
Given the JSON shown, you could add the following to your pipeline:
jq -c 'map({(.name): (if .type == "int" then 1234567890 else "xxxxxxxxxx" end)})|add'
With that JSON, the output would be:
{"id":1234567890,"test_string":"xxxxxxxxxx"}
However, it would be far better if you combined the three calls to jq into one.
I am writing a bash script utilizing jq to filter out JSON entries given some bash variables and return some of the key values from each entry into a tab-delimited file. I think the first few lines of this command are okay, but the 4th line I think is causing the problem. I have tried piping each entry in line 4 to tostring but to no avail.
info=`cat $FILE | jq -r \
' .[] \
| map(select(.host| contains(env.A))) \
| [."ip.A",."ts",."ip.B"] \
| #tsv'`
JSON example entry:
{
"ts": "2019-06-19T00:00:00.000000Z",
"ip.A": "0.0.0.0",
"ip.B": "0.0.0.0",
"host": "www.google.com",
}
In these files, there are no brackets surrounding the entire text within the file.
Error Given:
jq: error (at <stdin>:0): Cannot iterate over string ("2019-06-18...)
Do I need to handle ".ts" in some special way?
This code is broken long before the 3rd line.
If there isn't an outer array or object, you can't use .[].
If your data type is an object and not a list, using map() on it throws away data (specifically, it discards the keys, leaving only the values).
...so, .[] iterates over the values in your object, and then map() tries to iterate over each of those values as if it was an iterable object itself, which it's not... hence your error.
A version cut down to remove the broken parts might look like:
a="google.com" jq -r '
if (.host | contains(env.a)) then
[."ip.A",."ts",."ip.B"] | #tsv
else
empty
end
' <<'EOF'
{
"ts": "2019-06-19T00:00:00.000000Z",
"ip.A": "0.0.0.0",
"ip.B": "0.0.0.0",
"host": "www.google.com"
}
EOF
...which works fine.