I have a json data which is in the form of key and all values in a array but I need to transform it into a array of key value pairs, here is the data
Source data
"2022-08-30T06:58:56.573730Z": [
{ "tag": "AC 3 Phase/7957", "value": 161.37313113545272 },
{ "tag": "AC 3 Phase/7956", "value": 285.46869739695853 }
]
}
Transformation looking for
[
{ "tag": "AC 3 Phase/7957",
"ts": 2022-08-30T06:58:56.573730Z,
"value": 161.37313113545272
},
{ "tag": "AC 3 Phase/7956",
"ts": 2022-08-30T06:58:56.573730Z,
"value": 285.46869739695853
}
]
I would do it like this:
$each($$, function($entries, $ts) {
$entries.{
"tag": tag,
"ts": $ts,
"value": value
}
}) ~> $reduce($append, [])
Feel free to play with this example on the playground: https://stedi.link/g6qJGcP
Related
I'm trying to use JSONata to convert arrays of "key/value" objects into properties of the parent object. My input looks like this:
[
{
"city": "Ottawa",
"properties": [
{
"name": "population",
"value": 37
},
{
"name": "postalCode",
"value": 10001
},
{
"name": "founded",
"value": 1826
}
]
},
{
"city": "Toronto",
"properties": [
{
"name": "population",
"value": 54
},
{
"name": "postalCode",
"value": 10002
}
]
}
]
I'm struggling to generate the output I need, I've seen examples that reference explicit elements, like in this answer, but I need the properties to be converted "dynamically" since I don't know them in advance. I think I need something like this, but I'm missing some particular function:
$[].{
"city": city,
properties.name: properties.value
}
This is the output I need to generate:
[
{
"city": "Ottawa",
"population": 37,
"postalCode": 10001,
"founded": 1826
},
{
"city": "Toronto",
"population": 54,
"postalCode": 10002
}
]
The properties arrays don't always contain the same keys, but the city attributes are always present.
You can use the reduce operator, as described in the Grouping docs here:
$[].(
$city := city;
properties{ "city": $city, name: value }
)
You can play with it live: https://stedi.link/uUANwtE
Please try this expression.
$[].{
"city": $.city,
$.properties[0].name: $.properties[0].value,
$.properties[1].name: $.properties[1].value,
$.properties[2].name: $.properties[2].value,
$.properties[3].name: $.properties[3].value
}
https://try.jsonata.org/s1Ea4kUvo
I'm using JSONpath to try and find data with an array of JSON objects but I'm struggling to get to the information I want. The array contains many objects similar to below where there are values for RecID throughout. If I use $..RecID I get them all when I only want the first Key.RecID of each object (with a value 1338438 in this example). Is there a way to only extract the top level Key.RecID value?
BTW I'm trying to do this in jMeter and I'm assuming JSONpath is the best way to do what I want but if there is a better way I'd be happy to hear about it.
Thanks in advance
[{
"Key": {
"RecID": 1338438
},
"Users": [{
"FullName": "Miss Burns",
"Users": {
"Key": {
"Name": "Burns",
"RecID": 1317474
}
}
},
{
"FullName": "Mrs Fisher",
"Users": {
"Key": {
"Name": "Fisher",
"RecID": 1317904
}
}
}
],
"User": {
"FullName": "Mrs Fisher",
"Key": {
"Name": "Fisher",
"RecID": 1317904
}
},
"Organisation": {
"Key": {
"RecID": 1313881
}
}
}]
I have the following input in Nifi Jolt Specification processor:
[
{
"values": [
{
"id": "paramA",
"value": 1
}
]
},
{
"values": [
{
"id": "paramB",
"value": 3
}
]
}
]
Expected output:
[
{
"id": "paramA",
"value": 1
},
{
"id": "paramB",
"value": 2
}
]
Can you explain how I have to do?
thanks in advance
You want to reach the objects of the values array which are nested within seperate object signs ({}). A "*" notation is needed in order to cross them over per each individual values array, and then use another "*" notation for indexes of those arrays while choosing "" as the counterpart values in order to grab nothing but the sub-objects such as
[
{
"operation": "shift",
"spec": {
"*": {
"values": {
"*": ""
}
}
}
}
]
My NiFi application receives two kinda different types of JSON's.
First of them looks like:
[
{
"campaign": {
"resourceName": "customers/8952771329/campaigns/11381694617",
"status": "ENABLED",
"name": "Saint_Spring_Active Minerals_oct-nov_2020_trueview_skip_5766500views",
"id": "11381694617"
},
"metrics": {
"interactionEventTypes": [
"VIDEO_VIEW"
],
"clicks": "6",
"videoQuartileP100Rate": 0.44493171079034244,
"videoQuartileP25Rate": 0.9747718298919024,
"videoQuartileP50Rate": 0.7339309987701469,
"videoQuartileP75Rate": 0.5337562301767105,
"videoViewRate": 0.4471109114825628,
"videoViews": "27872",
"viewThroughConversions": "0",
"contentBudgetLostImpressionShare": 0.0000013066088274492382,
"contentImpressionShare": 0.0999,
"contentRankLostImpressionShare": 0.9001,
"conversionsValue": 0,
"conversions": 0,
"costMicros": "9338700950",
"ctr": 0.00009624947864865732,
"currentModelAttributedConversions": 0,
"currentModelAttributedConversionsValue": 0,
"engagementRate": 0,
"engagements": "0",
},
"segments": {
"device": "CONNECTED_TV",
"date": "2020-12-20"
}
}
]
And second:
[
{
"adGroup": {
"resourceName": "customers/5404177717/adGroups/110501283582",
"campaign": "customers/5404177717/campaigns/11628802542"
},
"metrics": {
"interactionEventTypes": [
"CLICK"
],
"clicks": "1",
"averageCpm": 95497428.02172929,
"gmailForwards": "0",
"gmailSaves": "0",
"gmailSecondaryClicks": "0",
"impressions": "4418",
"interactionRate": 0.00022634676324128565,
"interactions": "1"
},
"adGroupAd": {
"resourceName": "customers/5404177717/adGroupAds/110501283582~480227690139",
"status": "ENABLED",
"ad": {
"resourceName": "customers/5404177717/ads/480227690139",
"id": "480227690139",
"name": "20 sec perek"
},
"adGroup": "customers/5404177717/adGroups/110501283582"
},
"segments": {
"device": "DESKTOP",
"date": "2020-11-21"
}
}
]
I already have 2 tables in my database to save this data. I have an attribute table.name just to not create same block where's only table name is different.
My next block is FlattenJson. After this i'm using ReplaceText with search value (replacement value is empty string): (customers\\\/${client.customer.id}\\\/campaigns\\\/|customers\\\/${client.customer.id}\\\/adGroups\\\/).
Why this? From this line: "adGroup": "customers/5404177717/adGroups/110501283582" i only need last value 110501283582 as ad_group_id. And from this line: "campaign": "customers/5404177717/campaigns/11628802542" i only need 11628802542. ${client.customer.id} can be different, so i'm using EL features.
Also i need to change json value name adGroup to ad.group.id, for this i'm also using ReplaceText.
Can i do it faster without two ReplaceText processors?
Look at the following processors...I think using them can be an alternative:
JoltTransformJSON:
https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.5.0/org.apache.nifi.processors.standard.JoltTransformJSON/
UpdateRecord:
https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.5.0/org.apache.nifi.processors.standard.UpdateRecord/index.html
Is there away in elasticsearch to get the results back in a sort of flattend form (multiple child/sub aggs?
For instance currently i am trying to get back all product types and their status (online / offline).
This is what i end up with:
aggs
[
{ key: SuperProduct, doc_count:3, subagg:[
{status:online, doc_count:1},
{status:offline, doc_count:2}
]
},
{ key: SuperProduct2, doc_count:10, subagg:[
{status:online, doc_count:7},
{status:offline, doc_count:3}
]
Charting libraries tend to like it flattened so i was wondering if elasticsearch could probide it in this sort of manner:
[
{ products_key: 'SuperProduct', status_key:'online', doc_count:1},
{ products_key: 'SuperProduct', status_key:'offline', doc_count:2},
{ products_key: 'SuperProduct2', status_key:'online', doc_count:7},
{ products_key: 'SuperProduct2', status_key:'offline', doc_count:3}
]
Thanks
It is possible with composite aggregation which you can use to link two terms aggregations:
// POST /i/_search
{
"size": 0,
"aggregations": {
"distribution": {
"composite": {
"sources": [
{"product": {"terms": {"field": "product.keyword"}}},
{"status": {"terms": {"field": "status.keyword"}}}
]
}
}
}
}
This results in following structure:
{
"aggregations": {
"distribution": {
"after_key": {
"product": "B",
"status": "online"
},
"buckets": [
{
"key": {
"product": "A",
"status": "offline"
},
"doc_count": 3
},
{
"key": {
"product": "A",
"status": "online"
},
"doc_count": 2
},
{
"key": {
"product": "B",
"status": "offline"
},
"doc_count": 1
},
{
"key": {
"product": "B",
"status": "online"
},
"doc_count": 4
}
]
}
}
}
If for any reason composite aggregation doesn't fulfill your needs, you can create (via copy_to or by concatenation) or simulate (via scripted fields) field that would uniquely identify bucket. In our project we went with concatenation (partially for the necessity to collapse on this field), e.g. {"bucket": "SuperProductA:online"}, which results in dirtier output (you'll have to decode that field back or use top hits to get original values) but still does the job.