How to select on existing (or non existing) key/value pair in sub structure - filter

I have the following (simplified) JSON object:
{
"data": {
"items": [
{
"id": 2584,
"customProperties": [
{
"name": "jira.assetid",
"value": "1"
},
{
"name": "system.categories",
"value": "collector"
}
]
},
{
"id": 2603,
"customProperties": [
{
"name": "system.categories",
"value": "snmp"
}
]
},
{
"id": 2703,
"customProperties": [
{
"name": "jira.assetid",
"value": "5"
},
{
"name": "system.categories",
"value": "snmpTCPUDP,Netsnmp,snmpHR,snmp"
}
]
}
]
}
}
How can I filter all devices which have (or don't have) a "jira.assetid"?
My goal is to get 2 outputs:
No jira.assetid present: id, jira.assetid
Yes jira.assetid present: only id
The following gives me the id's of the devices with a jira.assetid, but how to get the assetid also on the output:
cat devices.txt | jq -r '.data.items[]
| select(.customProperties[].name == "jira.assetid")
|.id'

Here are at least two possible solutions, one first building a list of all id + assetid combinations, the other converts the custom properties to an object first.
Streaming solution:
.data.items[]
| {
id,
assetId: (.customProperties[] | select(.name == "jira.assetid").value)
}
Object solution:
.data.items[]
| {
id,
assetId: (.customProperties | from_entries."jira.assetid")
}
| select(.assetId)
Or using the special empty filter instead of selecting a second time:
.data.items[]
| {
id,
assetId: (.customProperties | from_entries."jira.assetid" // empty)
}
Output:
{
"id": 2584,
"assetId": "1"
}
{
"id": 2703,
"assetId": "5"
}
To get only the elements without an assetid:
.data.items[]
| select(any(.customProperties[]; .name == "jira.assetid") | not)
| { id }
or by building an object first, which I find slightly easier to grok:
.data.items[]
| select(.customProperties | from_entries | has("jira.assetid") | not)
| { id }
Output:
{
"id": 2603
}
To get only the ids for items without asset id:
.data.items[]
| select(.customProperties | from_entries | has("jira.assetid") | not)
| .id
Output:
2603

Related

jq - generate list of dictionaries instead of list of strings

The following command:
cat foo | jq -r '[.items[] | select(.metadata.random_field["foo/foo"] == "true") | .metadata.name]'
results in
[
"test_data"
]
I would like the following output to be:
[
{"name":"test", "data":"test_data"}
]
foo
{
"apiVersion": "v1",
"items":[
{
"metadata": {
"random_field": {
"foo/foo": "true"
},
"creationTimestamp": "2022-03-09T21:54:08Z",
"name": "test_data"
}
}
]
}
UPDATE: added test data so the point can be illustrated properly.
It's not entirely clear where the value for name in the output comes from in your question, but let's assume for a moment that it is the first part until the underscore of the name from the input. The the following produces your required output:
.items | map(
.metadata
| select(.random_field."foo/foo" == "true")
| { name: .name|split("_")|first, data: .name }
)
Output:
[
{
"name": "test",
"data": "test_data"
}
]

Nested Filtering json file with jq statement

I have a json file with the following structure of each object inside
{
"id": 2400321267,
"data": {
"q": "quinoa black bean and shrimp r",
"r": "quinoa black bean and shrimps r",
"s": "3"
},
"job_id": 1413792,
"results": {
"judgments": [
{
"id": 5022700047,
"unit_state": "good",
"data": {
"rewrite_quality": "1"
},
}
],
}
},
{
"id": 2400321267,
"data": {
"q": "quinoa black bean and shrimp r",
"r": "quinoa black bean and shrimps r",
"s": "3"
},
"job_id": 1413792,
"results": {
"judgments": [
{
"id": 5022700047,
"unit_state": "good",
"data": {
"rewrite_quality": "2"
},
}
],
}
}
and I was trying to use the command jq '.[] | select(any(.Tags[]; .rewrite_quality == "1"))' | less to try to see if the output is correct but I don't see any output.
I want the output to have only entries with rewrite_quality == '1', in this case only the first entry.
Reading between the lines, it would appear that the following filter should achieve the stated goals:
.[]
| select( .results | any(.judgments[]; .data.rewrite_quality == "1"))
"Tags"
If the intent in using ".Tags" was to indicate that it does not matter what path leads to .rewrite_quality, then the filter to use would be:
.[]
| select( any(.. | objects | .rewrite_quality == "1"))
Alternative to using less
If you want a brief indication of whether there are any matches, you could use this filter, which has the added value of revealing how many objects satisfy the criterion:
map(select(any(.. | objects | .rewrite_quality == "1"))) | length

unable to parse json into csv using jq

I have a JSON file that I want to convert into a CSV file using the jq in a shell script. I want to create a single row from this entire JSON file. I have to extract value from values. The row output should be something like
null,642,642,412,0,null,null
Here is my JSON file
{
"data": [
{
"name": "exits",
"period": "lifetime",
"values": [
{
"value": {}
}
],
"title": "Exits",
"description": "Number of times someone exited the carousel"
},
{
"name": "impressions",
"period": "lifetime",
"values": [
{
"value": 642
}
],
"title": "Impressions",
"description": "Total number of times the media object has been seen"
},
{
"name": "reach",
"period": "lifetime",
"values": [
{
"value": 412
}
],
"title": "Reach",
"description": "Total number of unique accounts that have seen the media object"
},
{
"name": "replies",
"period": "lifetime",
"values": [
{
"value": 0
}
],
"title": "Replies",
"description": "Total number of replies to the carousel"
},
{
"name": "taps_forward",
"period": "lifetime",
"values": [
{
"value": {}
}
],
"title": "Taps Forward",
"description": "Total number of taps to see this story's next photo or video"
},
{
"name": "taps_back",
"period": "lifetime",
"values": [
{
"value": {}
}
],
"title": "Taps Back",
"description": "Total number of taps to see this story's previous photo or video"
}
]
}
Hi tried using this jq command :
.data | map(.values[].value) | #csv
This is giving the following output:
jq: error (at :70): object ({}) is not valid in a csv row
exit status 5
So when I am getting this empty JSON object it is reflecting an error.
Please Help!!
The row output should be something like
null,642,642,412,0,null,null
Using length==0 here is dubious at best. To check for {} one could write:
jq '.data | map(.values[].value | if . == {} then "null" else . end) | #csv'
Similarly for [].
If you run the command without the #csv part you will see that the output is:
[
{},
642,
412,
0,
{},
{}
]
By replacing the empty objects with "null": (length == 0)
jq '.data | map(.values[].value) | map(if (type == "object" and length == 0 ) then "null" else . end) | #csv'
Output:
"\"null\",642,412,0,\"null\",\"null\""
Per suggestion from #aaron (see comment). The following can produce the requested output without extra post-processing. Disclaimer: this is not working with my jq 1.5, but working on jqplay with jq 1.6.
jq --raw-output '.data | map(.values[].value) | map(if (type == "object" and length == 0 ) then "null" else . end) | join(",")'
Output:
null,642,412,0,null,null

Analogue for sql join in jq

I have the following json:
[
{"id": "1", "type": "folder", "title": "folder-1"},
{"id": "2", "type": "folder", "title": "folder-2"},
{"id": "3", "type": "item", "title": "item-1", "folder": "1"},
{"id": "4", "type": "item", "title": "item-2", "folder": "2"},
{"id": "5", "type": "item", "title": "item-3"}
]
Basically, I need to produce this output using jq, which is similar to the result of sql join:
[
{"type": "item", "title": "item-1", "folder": "folder-1"},
{"type": "item", "title": "item-2", "folder": "folder-2"},
{"type": "item", "title": "item-3"}
]
Any ideas?
Try this filter:
map(
(select(.type=="item") | { key: .folder, value: { type, title } }),
(select(.type=="folder") | { key: .id, value: { folder: .title } })
)
| group_by(.key)
| map(
(map(select(.key != null) | .value) | add)
// map(.value)[]
)
You'll have to break this out into steps.
Get the items and folders and for each, take the values you're interested in and assign it a key to associate with.
map(
(select(.type=="item") | { key: .folder, value: { type, title } }),
(select(.type=="folder") | { key: .id, value: { folder: .title } })
)
Group all by the key
| group_by(.key)
Then combine the values that have keys (folders) and the value otherwise
| map(
(map(select(.key != null) | .value) | add)
// map(.value)[]
)
jq 'map(select(has("folder") or (.["title"] | startswith("item"))) | del(.id))' sample_file
Output:
[
{
"type": "item",
"title": "item-1",
"folder": "1"
},
{
"type": "item",
"title": "item-2",
"folder": "2"
},
{
"type": "item",
"title": "item-3"
}
]
Here is another solution which works by separating the data into two objects $folders and $items and then constructing the desired result.
(
reduce map(select(.type == "folder"))[] as $f (
{}
; .[$f.id] = $f
)
) as $folders
| (
reduce map(select(.type == "item"))[] as $i (
{}
; .[$i.id] = $i
)
) as $items
| [
$items[]
| {type, title, folder}
| if .folder == null then del(.folder) else .folder = $folders[.folder].title end
]
If your version of jq has INDEX/2
def INDEX(stream; idx_expr):
reduce stream as $row ({};
.[$row|idx_expr|
if type != "string" then tojson
else .
end] |= $row);
this can be simplified to
INDEX(.[] | select(.type == "folder"); .id) as $folders
| INDEX(.[] | select(.type == "item"); .id) as $items
| [
$items[]
| {type, title, folder}
| if .folder == null then del(.folder) else .folder = $folders[.folder].title end
]

Multi-level array of structured objects in elasticsearch using jdbc-river

I'm using elasticsearch(elasticsearch-1.1.1) with jdbc river plugin(1.1.0.2) I’m trying to achieve multi-level array of structured object.
Following is my data.
_id | Column1 | Column2 | Column3
------------+------------+------------+------------
4702560747 | 4702560747 | 4706318209 | 4816766466
---------------------------------------------------
4702560747 | 4702560747 | 4706318209 | 4816862011
---------------------------------------------------
4702560747 | 4702560747 | 4707978645 | 4816862008
---------------------------------------------------
4702560747 | 4702560747 | 4707978645 | 4816862010
I want to json to be generated as follows.
{
"deal": {
"id": 4702560747,
"quotes": [
{
"quote": {
"id": 4706318209,
"lines": [
{
"line": {
"id": 4816766466
}
},
{
"line": {
"id": 4816862011
}
}
]
}
},
{
"quote": {
"id": 4707978645,
"lines": [
{
"line": {
"id": 4816862008
}
},
{
"line": {
"id": 4816862010
}
}
]
}
}
]
}
}

Resources