I have json input of form as shown below. It can also be empty like []
[
{
"NAME": "Aron",
"CITY": "NewYork",
"PROV": "NY",
"POSTALCODE": "12345",
"DATE": "2021-08-19",
"TIME": "14:25:25"
},
{
"NAME": "Paul",
"CITY": "Chicago",
"PROV": "BI",
"POSTALCODE": null,
"DATE": "2021-08-19",
"TIME": ""
}
]
I am writing NiFi JoltTransformJSON spec so my output will contain the JSON as
[
{
"NAME" : "Aron",
"Address" : "NewYork, NY, Pin: 12345",
"DATE" : "2021-08-19",
"TIME" : "14:25:25"
},
{
"NAME" : "Paul",
"Address" : "Chicago, BI",
"DATE" : "2021-08-19",
"TIME" :""
}
]
Basically it merges City, Prov, Postalcode and puts in new key Address. They are concatenated using , comma between them and also if Postalcode is not null it is concatenated in Address as Pin:. If everything (City, Prov, Postalcode) is null then Address will be empty like "Address":""
I am trying with shift followed by remove
[
{
"operation": "shift",
"spec": {
"#(1,CITY)": "",
"*": "&"
}
},
{
"operation": "remove",
"spec": {
"CITY": "",
"PROV": "",
"POSTALCODE" : "",
}
}
]
I have achieved this using SplitJSON, EvaluateJSONPath and UpdateAttribute. But I am trying this using JoltTransformJSON now. Is this achievable using JoltTransformJSON ?
You can consecutively use modify-overwrite-beta along with concat and size functions and shift transformations such that
[
{
"operation": "modify-overwrite-beta",
"spec": {
"*":{
"Address1": "=concat(#(1,CITY),',',#(1,PROV))",
"Address2": "=concat(',Pin:',#(1,POSTALCODE))",
"Addr2Size": "=size(#(1,Address2))",
"Address": "=concat(#(1,Address1),#(1,Address2))"
}
}
},
{
"operation": "shift",
"spec": {
"*":{
"NAME": "[&1].&",
"Addr2Size": {
"5": {
"#(2,Address1)": "[&3].Address"
},
"*": {
"#(2,Address)": "[&3].Address"
}
},
"DATE": "[&1].&",
"TIME": "[&1].&"
}
}
}
]
where size function is used to determine whether POSTALCODE value is null or not.
Related
I'm receving mistake with #timestamp parameter in my jolt transformation:
[
{
"operation": "shift",
"spec": {
"*": {
"#(1,timestamp)": "values.[#2].timestamp",
"$": [
"values.[#2].id",
"values.[#2].doc_id"
],
"#": "values.[#2].value"
}
}
},
{
"operation": "modify-overwrite-beta",
"spec": {
"values": {
"*": {
"doc_id": "=concat(#(1,doc_id),#(1,timestamp))",
"#timestamp": "=concat(#(1,timestamp),'000')",
"value": "=concat(#(1,value),'')"
}
}
}
}
]
Input ex:
{
"timestamp": 1559350080,
"param": 12
}
Is it possible to use the character # in Jolt Transformation?
# is a special character. Need to prefix with escape characters(\\) in order to make it a literal by replacing "#timestamp" key with "\\#timestamp".
This way, you'll have "#timestamp" : "1559350080000" as key-value pair in the output.
In nifi, I am trying to transform a JSON with a variable amount of keys, but will always have a "date" key. I would like to transform the Json and change the string value of the date into a json object. However I am not getting what I need. Which operation/spec can I use in order to accomplish the expected output.
Input:
{
"name": "val1",
"date": "2021-05-19T00:53:20+00:00"
}
Spec:
[
{
"operation": "shift",
"spec": {
"#0": "wrapper"
}
}, {
"operation": "default",
"spec": {
"wrapper": {
"date": { "$date": "${date_attr}" }
}
}
}
]
expected output
{
"wrapper": {
"name": "val1",
"date": {"$date": "2021-05-19T00:53:20+00:00"}
}
}
what I am getting
{
"wrapper": {
"name": "val1",
"date": "2021-05-19T00:53:20+00:00"
}
}
Only single step of shift transformation along with the escaping characters (\\) for $ operator would suffice to use such as
[
{
"operation": "shift",
"spec": {
"name": "wrapper.name",
"date": { "#(1,date)": "wrapper.&.\\$date" }
}
}
]
where we're going one level up by using the first argument 1 in #(1,date) as been staying in the nested object
Edit : Considering that you only need to override the attribute date, and leave the others as they're without individually adding, use the following which again has only single step of shift transformation
[
{
"operation": "shift",
"spec": {
"*": "wrapper.&",
"date": { "#(1,date)": "wrapper.&.\\$date" }
}
}
]
In case of default, if the key mentioned in the spec is missing in the input json, then its added or else no changes may happen.
Here you are trying to push a node to a more level, which can be achieved by creating and assigning an temporary node.
\\ is an escape character.
[
{
"operation": "shift",
"spec": {
"date": "date1",
"#0": "."
}
},
{
"operation": "remove",
"spec": {
"date": ""
}
}, {
"operation": "shift",
"spec": {
"date1": "wrapper.date.\\$date",
"#0": "wrapper"
}
}, {
"operation": "remove",
"spec": {
"wrapper": {
"date1": ""
}
}
}
]
I am trying to give as input a JSON that contains a fixed variable (in my example group) that has as value an imbrication of objects and my goal is to transform it to sub objects.
My Input :
[
{
"key": "name",
"value": "marc",
"group": "office.people"
}
]
My Spec :
[
{
"operation": "shift",
"spec": {
"*": {
"value": "#(1,key).#(1,group)"
}
}
}
]
Expected:
{
"name": {
"office": {
"people": "marc"
}
}
}
Actual:
{
"name" : {
"office.people" : "marc"
}
}
You could split the group first:
[
{
"operation": "modify-overwrite-beta",
"spec": {
"*": {
"group": "=split('\\.',#(1,group))"
}
}
},
{
"operation": "shift",
"spec": {
"*": {
"value": "#(1,key).#(1,group[0]).#(1,group[1])"
}
}
}
]
I am trying to write a jolt in Apache Nifi that converts the json form. I managed to convert the jolt but one attribute is missing underscore:
Output:
{
"Source": { //source is missing underscore. It should look like _Source
"userName": "Lulu"
}
}
My input is as following:
{
"user_name": "Lulu"
}
Currently my jolt expression looks like following:
[
{
"operation": "shift",
"spec": {
"user_name":"userName"
}
},
{
"operation": "modify-overwrite-beta",
"spec": {
"_Source":{
"userName":"#(2,userName)"
}
}
},
{ "operation": "remove",
"spec": {
"userName": "" }
}
]
How do I keep the underscore sign in attribute "Source"?
I am stuck at figuring out this part. I'm wondering what am I missing in the jolt expression. Thanks in advance, guys
I think the underscore might be some kind of special character in that operator, try double-backslashes in front of _Source:
[
{
"operation": "shift",
"spec": {
"user_name": "userName"
}
},
{
"operation": "modify-overwrite-beta",
"spec": {
"\\_Source": {
"userName": "#(2,userName)"
}
}
},
{
"operation": "remove",
"spec": {
"userName": ""
}
}
]
Can do it just with a single "shift".
Spec
[
{
"operation": "shift",
"spec": {
"user_name": "_Source.userName"
}
}
]
I am new to nifi. I was trying to create a jolt spec but not getting it. Could anyone please help me.
Details as below:
The Attributes in flow file : details, id, name , address, status
Flow file looks like :
[{"to": "xxx1"},{"to": "xxx2"},{"to": "xxx3"},{"to": "xxxn"}]
Expecting below output:
{ "details": "personal",
"home":[
{"mobileno": "xxx1",
"id": "1",
"name" :"bbb",
"address": "Address1" },
{ "mobileno": "xxx2",
"id": "2",
"name": "aaa",
"address": "address2" }
],
"status": "enabled" }
Am able to develop till this. But am not getting how to get "details" field
[{
"operation": "shift",
"spec": {
"*": "home",
"mobileno": "home[0].mobileno"
}
}, {
"operation": "default",
"spec": {
"status": "${status}",
"home[]": {
"*": {
"name": "${name}",
"id" : "${id},
"address": "${address}"
}
}
}
}]
In addition to 7632695's answer, your shift spec won't match "mobileno" on the input, try the following:
[{
"operation": "shift",
"spec": {
"*": {
"to": "home[&1].mobileno"
}
}
}, {
"operation": "default",
"spec": {
"status": "${status}",
"details": "${details}",
"home[]": {
"*": {
"name": "${name}",
"id": "${id}",
"address": "${address}"
}
}
}
}]
Also, note that for a single flow file, the attributes are constant, so for each entry in the home array, each id, name, and address field will be the same. From your attributes, how would JOLT know to use id=1 for the first element and id=2 for the second, and so on?
If you want to use the index of the input array as the id, you can add this spec to your chain:
{
"operation": "shift",
"spec": {
"home": {
"*": {
"$": "home[&1].id",
"*": "home[&1].&"
}
},
"*": "&"
}
}
And if you want them to start at 1 instead of 0, you can add 1 to each of them by adding the following spec to your chain:
{
"operation": "modify-overwrite-beta",
"spec": {
"home": {
"*": {
"id": "=intSum(#0, 1)"
}
}
}
}
In default operation you need to add details attribute.
try with below jolt spec
[{
"operation": "shift",
"spec": {
"*": "home",
"mobileno": "home[0].mobileno"
}
}, {
"operation": "default",
"spec": {
"status": "${status}",
"details":"${details}",
"home[]": {
"*": {
"name": "${name}",
"id": "${id}",
"address": "${address}"
}
}
}
}]