Related
I have two datasets:
data1='[
{ "bookings": 2984, "timestamp": 1675854900 },
{ "bookings": 2967, "timestamp": 1675855200 }
]'
data2='[
{ "errors": 51, "timestamp": 1675854900 },
{ "errors": 90, "timestamp": 1675855200 }
]'
I want the output to be:
combined='[
{ "errors": 51, bookings: 2984, "timestamp": 1675854900 },
{ "errors": 90, bookings: 2967, "timestamp": 1675855200 }
]'
Can this be achieved by shell scripting and jq command?
Assume that timestamp will always be present and will always have a common value across two datasets. Even the order is same.
This last paragraph just caught my attention:
Assume that timestamp will always be present and will always have a common value across two datasets. Even the order is same.
If this is truly the case then it is reasonable to assume that both arrays have the same length and their items are aligned respectively. Thus, there's no need to build up a hash-based INDEX as accessing the items by their numeric keys (positions within the arrays) can already be achieved in constant time.
jq -n --argjson data1 "$data1" --argjson data2 "$data2" '
$data1 | [keys[] | $data2[.] + $data1[.]]
'
[
{
"errors": 51,
"timestamp": 1675854900,
"bookings": 2984
},
{
"errors": 90,
"timestamp": 1675855200,
"bookings": 2967
}
]
A simple JOIN operation could do:
jq -n --argjson data1 "$data1" --argjson data2 "$data2" '
[JOIN(INDEX($data1[]; .timestamp); $data2[]; .timestamp | #text; add)]
'
[
{
"errors": 51,
"timestamp": 1675854900,
"bookings": 2984
},
{
"errors": 90,
"timestamp": 1675855200,
"bookings": 2967
}
]
I'm getting this error: jq: error: JOIN/4 is not defined at <top-level>, line 2: [JOIN(INDEX($data1[]; .timestamp); $data2[]; .timestamp | #text; add)] jq: 1 compile error
You are probably using an older version of jq. JOIN and INDEX were introduced in jq 1.6. Either define them yourself by taking their definitions from source, or take those definitions and modify them to fit your very use case (both work well with jq 1.5).
Definitions from source:
jq -n --argjson data1 "$data1" --argjson data2 "$data2" '
def INDEX(stream; idx_expr):
reduce stream as $row ({}; .[$row | idx_expr | tostring] = $row);
def JOIN($idx; stream; idx_expr; join_expr):
stream | [., $idx[idx_expr]] | join_expr;
[JOIN(INDEX($data1[]; .timestamp); $data2[]; .timestamp | #text; add)]
'
Adapted to your use case:
jq -n --argjson data1 "$data1" --argjson data2 "$data2" '
($data1 | with_entries(.key = (.value.timestamp | #text))) as $ix
| $data2 | map(. + $ix[.timestamp | #text])
'
In general, if you find JOIN a bit tricky to understand or use, then consider using INDEX for this type of problem. In the present case, you could get away with a trivially simple approach, e.g.:
jq -n --argjson data1 "$data1" --argjson data2 "$data2" '
INDEX($data1[]; .timestamp) as $dict
| $data2 | map( . + $dict[.timestamp|tostring])
Another way to do this is to build a map from timestamps to error counts, and perform a lookup in it.
jq -n '
input as $data1
| input as $data2
| ($data2
| map({ "key": (.timestamp | tostring), "value": .errors })
| from_entries
) as $errors_by_timestamp
| $data1 | map(.errors = $errors_by_timestamp[(.timestamp | tostring)])
' <<<"$data1 $data2"
By the way, I have trying to this answer from AI since morning and finally it also gave me correct solution this time
#!/bin/bash
data1='[
{ "bookings": 2984, "timestamp": 1675854900 },
{ "bookings": 2967, "timestamp": 1675855200 }
]'
data2='[
{ "errors": 51, "timestamp": 1675854900 },
{ "errors": 90, "timestamp": 1675855200 }
]'
combined=$(jq -n --argjson d1 "$data1" --argjson d2 "$data2" '
[ $d1, $d2 ] | transpose[] | group_by(.timestamp) | map(
reduce .[] as $i ({}; . * $i)
)
')
echo "$combined"
Just pasting it here for you guys in case you didn't think of this method
I am new to Bash and I am currently trying to get the content of a JSON without showing the names of the key values.
This is how the JSON looks like (part of it):
[
{
"V1": 65,
"V2": "Female",
"V3": 0.7,
"V4": 0.1,
"V5": 187,
"V6": 16,
"V7": 18,
"V8": 6.8,
"V9": 3.3,
"V10": 0.9,
"Class": 1
},
{
"V1": 62,
"V2": "Male",
"V3": 10.9,
"V4": 5.5,
"V5": 699,
"V6": 64,
"V7": 100,
"V8": 7.5,
"V9": 3.2,
"V10": 0.74,
"Class": 1
},
{
"V1": 62,
"V2": "Male",
"V3": 7.3,
"V4": 4.1,
"V5": 490,
"V6": 60,
"V7": 68,
"V8": 7,
"V9": 3.3,
"V10": 0.89,
"Class": 1
}
]
This is my script
#!/bin/bash
echo "Albumin =3";
echo "Age Sex Albumin Proteins";
echo "******";
echo " "
echo "Women";
echo "--------------";
cat csvjson.json | jq -c '.[] | {V1, V2, V8, V9} | select(.V9 ==3) | select(.V2 =="Female")';
echo " "
echo "Men";
echo "-------------";
cat csvjson.json | jq -c '.[] | {V1, V2, V8, V9} | select(.V9 ==3) | select(.V2 =="Male")';
This is what the script shows
Women
--------------
{"V1":38,"V2":"Female","V8":5.6,"V9":3}
{"V1":38,"V2":"Female","V8":5.6,"V9":3}
{"V1":32,"V2":"Female","V8":6,"V9":3}
{"V1":31,"V2":"Female","V8":6,"V9":3}
{"V1":19,"V2":"Female","V8":5.5,"V9":3}
{"V1":38,"V2":"Female","V8":7,"V9":3}
{"V1":20,"V2":"Female","V8":6.1,"V9":3}
{"V1":32,"V2":"Female","V8":7,"V9":3}
{"V1":42,"V2":"Female","V8":6.7,"V9":3}
Men
-------------
{"V1":72,"V2":"Male","V8":7.4,"V9":3}
{"V1":60,"V2":"Male","V8":6.3,"V9":3}
{"V1":33,"V2":"Male","V8":5.4,"V9":3}
{"V1":60,"V2":"Male","V8":6.8,"V9":3}
{"V1":60,"V2":"Male","V8":7.4,"V9":3}
{"V1":60,"V2":"Male","V8":7,"V9":3}
{"V1":72,"V2":"Male","V8":6.2,"V9":3}
And this is what I want to show
Women
--------------
38,Female,3, 5.6
38,Female,3, 5.6
32,Female,3, 6
31,Female,3, 6
19,Female,3, 5.5
38,Female,3, 7
20,Female,3, 6.1
32,Female,3, 7
42,Female,3, 6.7
Men
--------------
72,Male,3, 7.4
60,Male,3, 6.3
33,Male,3, 5.4
60,Male,3, 6.8
60,Male,3, 7.4
60,Male,3, 7
72,Male,3, 6.2
So, how can I hide the key values and only show the content of the JSON after doing the filters I did?
This can be accomplished entirely within jq (although some constraints are not all clear, so please comment and I will update the code):
jq --raw-output '
group_by(.V2)[]
| if first.V2 == "Male" then "Men" else "Women" end,
"--------------",
(
.[]
| select(.V9 == 3.3) # this filters to matching records
| [.V1, .V2, .V9, .V8]
| join(",")
)
' csvjson.json
Demo
Demo stand-alone jq script and code bloc language highlight for use here in stack sites using pmf's answer.
#!/usr/bin/env -S jq --raw-output --from-file
group_by(.V2)[]
| if first.V2 == "Male" then "Men" else "Women" end,
"--------------",
(
.[]
| select(.V9 == 3.3) # this filters to matching records
| [.V1, .V2, .V9, .V8]
| join(",")
)
This question already has answers here:
jq not working on tag name with dashes and numbers
(2 answers)
Closed 4 years ago.
Whole file:https://1drv.ms/u/s!AizscpxS0QM4hJpEkp12VPHiKO_gBg
Using this command i get part bellow (get latest job)
jq '.|[ .executions[] | select(.job.name != null) | select(.job.name) ]
| sort_by(.id)
| reverse
| .[0] ' 1.json
{
"argstring": null,
"date-ended": {
"date": "2018-04-03T17:43:38Z",
"unixtime": 1522777418397
},
"date-started": {
"date": "2018-04-03T17:43:34Z",
"unixtime": 1522777414646
},
"description": "",
"executionType": "user",
"failedNodes": [
"172.30.61.88"
],
"href": "http://172.30.61.88:4440/api/21/execution/126",
"id": 126,
"job": {
"averageDuration": 4197,
"description": "",
"group": "",
"href": "http://172.30.61.88:4440/api/21/job/271cbcec-5042-4d52-b794-ede2056b2ab8",
"id": "271cbcec-5042-4d52-b794-ede2056b2ab8",
"name": "aa",
"permalink": "http://172.30.61.88:4440/project/demo/job/show/271cbcec-5042-4d52-b794-ede2056b2ab8",
"project": "demo"
},
"permalink": "http://172.30.61.88:4440/project/demo/execution/show/126",
"project": "demo",
"status": "failed",
"user": "administrator"
I managed to extract job name and status, now want to get date-ended.date ?
jq '.|[ .executions[] |select(.job.name != null) | select(.job.name) ]
| sort_by(.id)
| reverse
| .[0]
| "\(.status), \(.job.name)"' 1.json
With the "-r" command-line option, the following filter:
[.executions[] | select(.job.name != null)]
| sort_by(.id)
| reverse
| .[0]
| [.status, .job.name, ."date-ended".date]
| #csv
produces:
"failed","aa","2018-04-03T17:43:38Z"
An important point that you might have missed is that "-" is a "special" character in that it can signify negation or subtraction.
If your jq does not support the syntax ."date-ended".date, then you could fall back to the basic syntax: (.["date-ended"] | .date)
I guess you have troubles extracting .date-ended.date because the name contains a dash that is interpreted by jq as subtraction.
The solution is listed in the documentation:
If the key contains special characters, you need to surround it with double quotes like this: ."foo$", or else .["foo$"].
This means the last filter of your jq program should be:
"\(.status), \(.job.name), \(."date-ended".date)"
I'm indexing my PostgreSQL data for Elasticsearch using the Logstash JDBC Input Plugin. I have two tables called REQUEST and ASSIGNMENT listed below.
How can I use Logstash to index the two tables into one Elasticsearch document of type REQUEST with a nested arrays for all child ASSIGNMENT records?
Table: REQUEST
REQUEST_ID | POC
---------- | ----------------
1234 | Jon Snow
1256 | Tyrion Lannister
Table: ASSIGNMENT
ASSIGN_ID | REQUEST_ID | STATUS | CREATED
--------- | ---------- | ------- | ----------
2345 | 1234 | New | 2017-01-06
2364 | 1234 | Working | 2017-03-12
2399 | 1234 | Working | 2017-05-20
5736 | 1256 | New | 2017-06-28
This is what I want my Elasticsearch document to look like. It is a sample of the _source value of the search result:
"_source": {
"request_id": "1234",
"poc": "Jon Snow",
"assignments": [
{
"assign_id": "2345",
"status": "New",
"created": "2017-01-06"
},
{
"assign_id": "2364",
"status": "Working",
"created": "2017-03-12"
},
{
"assign_id": "2399",
"status": "Working",
"created": "2017-05-20"
}
]
}
Need to read data from a json log file and get analytics data from that using shell script.
log file contains json as below:
{
info: 'label1',
description: 'some desc',
timestamp: '2016-07-27T06:24:50.335Z'
}
{
info: 'label2',
description: 'some desc',
timestamp: '2016-07-27T06:24:50.335Z'
}
{
info: 'label2',
description: 'some desc',
timestamp: '2016-07-27T06:24:50.335Z'
}
{
info: 'label2',
description: 'some desc',
timestamp: '2016-07-29T06:24:50.335Z'
}
{
info: 'label3',
description: 'some desc',
timestamp: '2016-07-29T06:24:50.335Z'
}
I need the result as below (using shell script):
Labels Date Count
label1 2016-07-27 1
label2 2016-07-27 2
label2 2016-07-29 1
label3 2016-07-29 1
This is as far I could go, need some suggestion on how to approach.\
#!/bin/bash
my_dir=`dirname $0`
file="out.log"
#keysFile="$my_dir/keys.txt"
for log in $(cat $file | jq '{id: .info,time: .timestamp}'); do
#This is as far as I could get. I was able to read the data in the form of {id: 'label1', time: '2016-07-27T06:24:50.335Z' }
#Now I need to somehow create a key value thing in shell and store timestamp / label as key and increment the count
echo $log
done
Given your input, you could create that as csv data using this:
$ jq -rs '
def to_csv($headers):
def _object_to_csv:
($headers | #csv),
(.[] | [.[$headers[]]] | #csv);
def _array_to_csv:
($headers | #csv),
(.[][:$headers|length] | #csv);
if .[0]|type == "object" then
_object_to_csv
else
_array_to_csv
end;
map({ label: .info, date: .timestamp[:10] })
| group_by(.)
| map(.[0] + { count: length })
| to_csv(["label", "date", "count"])
' input.json
This yields:
"label","date","count"
"label1","2016-07-27",1
"label2","2016-07-27",2
"label2","2016-07-29",1
"label3","2016-07-29",1
Here is an approach which uses reduce instead of group_by.
Assuming your data is in out.log and the following filter in filter.jq
["Labels", "Date", "Count"],
["", "", ""],
(
reduce .[] as $r (
{}
; [$r.info, $r.timestamp[0:10]] as $p
| setpath($p;getpath($p)+1)
)
| tostream
| select(length==2)
| flatten
)
| #tsv
you can run
jq -M -s -r -f filter.jq out.log
to produce tab-separated output
Labels Date Count
label1 2016-07-27 1
label2 2016-07-27 2
label2 2016-07-29 1
label3 2016-07-29 1