JOLT - Join arrays in nested array - apache-nifi

I am trying to achieve the following transformation. However, my solution adds undesired null values into the final array.
The transformation needs to shift names in child array for all root elements. I have created 3 cases to illustrate the problem.
Case 1
Input
{
"root": [
{
"child": [
{
"name": "John"
},
{
"name": "Frazer"
}
]
},
{
"child": [
{
"name": "Brandon"
},
{
"name": "Josef"
}
]
}
]
}
Desired Output
{
"NAMES": ["John,Frazer","Brandon,Josef"]
}
Case 2: One child is empty
Input
{
"root": [
{
"child": []
},
{
"child": [
{
"name": "Brandon"
},
{
"name": "Josef"
}
]
}
]
}
Desired Output
{
"NAMES": ["","Brandon,Josef"]
}
Case 3: All childs are empty
Input
{
"root": [
{
"child": []
},
{
"child": []
}
]
}
Desired Output
{
"NAMES": ["",""]
}
EDIT: root array will always have at least 1 element.
Current JOLT spec works fine except for cases where child is an empty array. It generates null values and I'm trying to specify an empty string instead (or any hardcoded string value such as "NO_NAMES")
[
{
"operation": "shift",
"spec": {
"root": {
"*": {
"child": {
"*": {
"name": "NAMES[&3]"
}
}
}
}
}
},
{
"operation": "modify-overwrite-beta",
"spec": {
"NAMES": {
"*": "=trim"
}
}
},
{
"operation": "cardinality",
"spec": {
"NAMES": "MANY"
}
},
{
"operation": "default",
"spec": {
"NAMES": []
}
},
{
"operation": "modify-overwrite-beta",
"spec": {
"NAMES": {
"*": "=join(',',#0)"
}
}
}
]

You can apply consecutive transformations modify-overwrite-beta and then shift in order to determine comma-seperated elements of the list(unless they have zero size,this case only double quotes will appear), and then concatenate them within a list such as
[
{
"operation": "modify-overwrite-beta",
"spec": {
"root": {
"*": {
"child": { "*": "#(0,name)" },
"NAMES": "=join(',',#(1,child))"
}
}
}
},
{
"operation": "shift",
"spec": {
"root": {
"*": {
"NAMES": "&"
}
}
}
}
]

Related

How to add a keyname to every element in an array in Nifi (Json)

I am quite new to Nifi and I already run into an issue:
I have a list of items in an array:
{
"locations": [
1,
2,
3
]
}
I want to transform this to:
{
"locations": [
{
"locationid": 1
},
{
"locationid": 2
},
{
"locationid": 3
}
]
}
Any ideas?
(After this I also want to add another element from a nifi attribute, but I think I will be able to manage that myself.)
Never mind:
[
{
"operation": "shift",
"spec": {
"locations": {
"*": {
"#": "[#2].locationID"
}
}
}
},
{
"operation": "modify-default-beta",
"spec": {
"*": {
"ReportTimestamp": "${ReportTimestamp:toDate('yyyy-MM-dd HH:mm:ss'):toNumber()}"
}
}
}
]

How to add a object to a Json in a Jolt Transform spec

In nifi, I am trying to transform a JSON with a variable amount of keys, but will always have a "date" key. I would like to transform the Json and change the string value of the date into a json object. However I am not getting what I need. Which operation/spec can I use in order to accomplish the expected output.
Input:
{
"name": "val1",
"date": "2021-05-19T00:53:20+00:00"
}
Spec:
[
{
"operation": "shift",
"spec": {
"#0": "wrapper"
}
}, {
"operation": "default",
"spec": {
"wrapper": {
"date": { "$date": "${date_attr}" }
}
}
}
]
expected output
{
"wrapper": {
"name": "val1",
"date": {"$date": "2021-05-19T00:53:20+00:00"}
}
}
what I am getting
{
"wrapper": {
"name": "val1",
"date": "2021-05-19T00:53:20+00:00"
}
}
Only single step of shift transformation along with the escaping characters (\\) for $ operator would suffice to use such as
[
{
"operation": "shift",
"spec": {
"name": "wrapper.name",
"date": { "#(1,date)": "wrapper.&.\\$date" }
}
}
]
where we're going one level up by using the first argument 1 in #(1,date) as been staying in the nested object
Edit : Considering that you only need to override the attribute date, and leave the others as they're without individually adding, use the following which again has only single step of shift transformation
[
{
"operation": "shift",
"spec": {
"*": "wrapper.&",
"date": { "#(1,date)": "wrapper.&.\\$date" }
}
}
]
In case of default, if the key mentioned in the spec is missing in the input json, then its added or else no changes may happen.
Here you are trying to push a node to a more level, which can be achieved by creating and assigning an temporary node.
\\ is an escape character.
[
{
"operation": "shift",
"spec": {
"date": "date1",
"#0": "."
}
},
{
"operation": "remove",
"spec": {
"date": ""
}
}, {
"operation": "shift",
"spec": {
"date1": "wrapper.date.\\$date",
"#0": "wrapper"
}
}, {
"operation": "remove",
"spec": {
"wrapper": {
"date1": ""
}
}
}
]

JOLT - Nested arrays sum

I am not able to find a solution for this transformation. My goal is to sum all parts and storing it into an array with the same length as operations.
Input data:
{
"operations": [
{
"partOne": 10,
"partTwo": 12.5,
"partThree": 30.5
},
{
"partOne": 25.5,
"partTwo": 2,
"partThree": 7.5
}
]
}
Output data:
{
"costs": [
53, // operations[0].partOne + operations[0].partTwo + operations[0].partThree
35 // operations[1].partOne + operations[1].partTwo + operations[1].partThree
]
}
EDIT: I was able to find this intermediate step, still missing the sum part.
[
{
"operation": "shift",
"spec": {
"operations": {
"*": {
"partOne": "costs[&1].[]",
"partTwo": "costs[&1].[]",
"partThree": "costs[&1].[]"
}
}
}
},
{
"operation": "modify-overwrite-beta",
"spec": {
"*": {
"costs": "=doubleSum(#(1,costs[&1]))"
}
}
}
]
Also, if you could share some documentation about JOLT specification I would highly appreciate that. Could not find anything to get me started properly.
You can add one more shift operation between the current ones as rearranging by getting rid of square brackets like this :
[
{
"operation": "shift",
"spec": {
"operations": {
"*": {
"partOne": "costs&1",
"partTwo": "costs&1",
"partThree": "costs&1"
}
}
}
},
{
"operation": "shift",
"spec": {
"*": "&"
}
},
{
"operation": "modify-overwrite-beta",
"spec": {
"tot_costs0": "=doubleSum(#(1,costs0))",
"tot_costs1": "=doubleSum(#(1,costs1))"
}
}
]
P.S. this source might help.
I was able to find the solution.
Spec:
[
{
"operation": "shift",
"spec": {
"operations": {
"*": {
"partOne": "costs[&1].[]",
"partTwo": "costs[&1].[]",
"partThree": "costs[&1].[]"
}
}
}
},
{
"operation": "modify-overwrite-beta",
"spec": {
"costs": {
"*": "=doubleSum"
}
}
}
]

JOLT use value of variable to create imbrications in json

I am trying to give as input a JSON that contains a fixed variable (in my example group) that has as value an imbrication of objects and my goal is to transform it to sub objects.
My Input :
[
{
"key": "name",
"value": "marc",
"group": "office.people"
}
]
My Spec :
[
{
"operation": "shift",
"spec": {
"*": {
"value": "#(1,key).#(1,group)"
}
}
}
]
Expected:
{
"name": {
"office": {
"people": "marc"
}
}
}
Actual:
{
"name" : {
"office.people" : "marc"
}
}
You could split the group first:
[
{
"operation": "modify-overwrite-beta",
"spec": {
"*": {
"group": "=split('\\.',#(1,group))"
}
}
},
{
"operation": "shift",
"spec": {
"*": {
"value": "#(1,key).#(1,group[0]).#(1,group[1])"
}
}
}
]

In Jolt How to select based on jsonarray value

In JOLT Based on the jsonarray values the output key and value should be added.
In the output CO2_VAL,CO_VAL,O3_VAL shoould come based on param value in env_values. So, how to apply the above filter.
The input payload is:
{
"id":"abcd",
"env_values":[
{
"param":"CO2",
"values":"20.0"
},
{
"param":"CO",
"values":"21.0"
},
{
"param":"O3",
"values":"22.0"
}
]
}
The output is :
{
"sl":"abcd",
"CO2_VAL":"20.0",
"CO_VAL":"21.0",
"O3_VAL":"22.0"
}
Hope it is what you ment. First of all, we are adding '_VAL' to the key. In the second spec we are putting keys to the values. And at the end we are pairing each key with value.
[
{
"operation": "shift",
"spec": {
"id": "s1",
"env_values": {
"*": {
"param": {
"*": "param[&2].t.&_VAL"
},
"values": "param[&1].values"
}
}
}
},
{
"operation": "shift",
"spec": {
"s1": "s1",
"param": {
"*": {
"t": {
"*": {
"$": "param[&3].key"
}
},
"values": "param[&1].value"
}
}
}
},
{
"operation": "shift",
"spec": {
"s1": "s1",
"param": {
"*": {
"value": "#(1,key)"
}
}
}
}
]

Resources