I'm receving mistake with #timestamp parameter in my jolt transformation:
[
{
"operation": "shift",
"spec": {
"*": {
"#(1,timestamp)": "values.[#2].timestamp",
"$": [
"values.[#2].id",
"values.[#2].doc_id"
],
"#": "values.[#2].value"
}
}
},
{
"operation": "modify-overwrite-beta",
"spec": {
"values": {
"*": {
"doc_id": "=concat(#(1,doc_id),#(1,timestamp))",
"#timestamp": "=concat(#(1,timestamp),'000')",
"value": "=concat(#(1,value),'')"
}
}
}
}
]
Input ex:
{
"timestamp": 1559350080,
"param": 12
}
Is it possible to use the character # in Jolt Transformation?
# is a special character. Need to prefix with escape characters(\\) in order to make it a literal by replacing "#timestamp" key with "\\#timestamp".
This way, you'll have "#timestamp" : "1559350080000" as key-value pair in the output.
Related
In nifi, I am trying to transform a JSON with a variable amount of keys, but will always have a "date" key. I would like to transform the Json and change the string value of the date into a json object. However I am not getting what I need. Which operation/spec can I use in order to accomplish the expected output.
Input:
{
"name": "val1",
"date": "2021-05-19T00:53:20+00:00"
}
Spec:
[
{
"operation": "shift",
"spec": {
"#0": "wrapper"
}
}, {
"operation": "default",
"spec": {
"wrapper": {
"date": { "$date": "${date_attr}" }
}
}
}
]
expected output
{
"wrapper": {
"name": "val1",
"date": {"$date": "2021-05-19T00:53:20+00:00"}
}
}
what I am getting
{
"wrapper": {
"name": "val1",
"date": "2021-05-19T00:53:20+00:00"
}
}
Only single step of shift transformation along with the escaping characters (\\) for $ operator would suffice to use such as
[
{
"operation": "shift",
"spec": {
"name": "wrapper.name",
"date": { "#(1,date)": "wrapper.&.\\$date" }
}
}
]
where we're going one level up by using the first argument 1 in #(1,date) as been staying in the nested object
Edit : Considering that you only need to override the attribute date, and leave the others as they're without individually adding, use the following which again has only single step of shift transformation
[
{
"operation": "shift",
"spec": {
"*": "wrapper.&",
"date": { "#(1,date)": "wrapper.&.\\$date" }
}
}
]
In case of default, if the key mentioned in the spec is missing in the input json, then its added or else no changes may happen.
Here you are trying to push a node to a more level, which can be achieved by creating and assigning an temporary node.
\\ is an escape character.
[
{
"operation": "shift",
"spec": {
"date": "date1",
"#0": "."
}
},
{
"operation": "remove",
"spec": {
"date": ""
}
}, {
"operation": "shift",
"spec": {
"date1": "wrapper.date.\\$date",
"#0": "wrapper"
}
}, {
"operation": "remove",
"spec": {
"wrapper": {
"date1": ""
}
}
}
]
I am trying to give as input a JSON that contains a fixed variable (in my example group) that has as value an imbrication of objects and my goal is to transform it to sub objects.
My Input :
[
{
"key": "name",
"value": "marc",
"group": "office.people"
}
]
My Spec :
[
{
"operation": "shift",
"spec": {
"*": {
"value": "#(1,key).#(1,group)"
}
}
}
]
Expected:
{
"name": {
"office": {
"people": "marc"
}
}
}
Actual:
{
"name" : {
"office.people" : "marc"
}
}
You could split the group first:
[
{
"operation": "modify-overwrite-beta",
"spec": {
"*": {
"group": "=split('\\.',#(1,group))"
}
}
},
{
"operation": "shift",
"spec": {
"*": {
"value": "#(1,key).#(1,group[0]).#(1,group[1])"
}
}
}
]
In JOLT Based on the jsonarray values the output key and value should be added.
In the output CO2_VAL,CO_VAL,O3_VAL shoould come based on param value in env_values. So, how to apply the above filter.
The input payload is:
{
"id":"abcd",
"env_values":[
{
"param":"CO2",
"values":"20.0"
},
{
"param":"CO",
"values":"21.0"
},
{
"param":"O3",
"values":"22.0"
}
]
}
The output is :
{
"sl":"abcd",
"CO2_VAL":"20.0",
"CO_VAL":"21.0",
"O3_VAL":"22.0"
}
Hope it is what you ment. First of all, we are adding '_VAL' to the key. In the second spec we are putting keys to the values. And at the end we are pairing each key with value.
[
{
"operation": "shift",
"spec": {
"id": "s1",
"env_values": {
"*": {
"param": {
"*": "param[&2].t.&_VAL"
},
"values": "param[&1].values"
}
}
}
},
{
"operation": "shift",
"spec": {
"s1": "s1",
"param": {
"*": {
"t": {
"*": {
"$": "param[&3].key"
}
},
"values": "param[&1].value"
}
}
}
},
{
"operation": "shift",
"spec": {
"s1": "s1",
"param": {
"*": {
"value": "#(1,key)"
}
}
}
}
]
I am trying to write a jolt in Apache Nifi that converts the json form. I managed to convert the jolt but one attribute is missing underscore:
Output:
{
"Source": { //source is missing underscore. It should look like _Source
"userName": "Lulu"
}
}
My input is as following:
{
"user_name": "Lulu"
}
Currently my jolt expression looks like following:
[
{
"operation": "shift",
"spec": {
"user_name":"userName"
}
},
{
"operation": "modify-overwrite-beta",
"spec": {
"_Source":{
"userName":"#(2,userName)"
}
}
},
{ "operation": "remove",
"spec": {
"userName": "" }
}
]
How do I keep the underscore sign in attribute "Source"?
I am stuck at figuring out this part. I'm wondering what am I missing in the jolt expression. Thanks in advance, guys
I think the underscore might be some kind of special character in that operator, try double-backslashes in front of _Source:
[
{
"operation": "shift",
"spec": {
"user_name": "userName"
}
},
{
"operation": "modify-overwrite-beta",
"spec": {
"\\_Source": {
"userName": "#(2,userName)"
}
}
},
{
"operation": "remove",
"spec": {
"userName": ""
}
}
]
Can do it just with a single "shift".
Spec
[
{
"operation": "shift",
"spec": {
"user_name": "_Source.userName"
}
}
]
I am new to nifi. I was trying to create a jolt spec but not getting it. Could anyone please help me.
Details as below:
The Attributes in flow file : details, id, name , address, status
Flow file looks like :
[{"to": "xxx1"},{"to": "xxx2"},{"to": "xxx3"},{"to": "xxxn"}]
Expecting below output:
{ "details": "personal",
"home":[
{"mobileno": "xxx1",
"id": "1",
"name" :"bbb",
"address": "Address1" },
{ "mobileno": "xxx2",
"id": "2",
"name": "aaa",
"address": "address2" }
],
"status": "enabled" }
Am able to develop till this. But am not getting how to get "details" field
[{
"operation": "shift",
"spec": {
"*": "home",
"mobileno": "home[0].mobileno"
}
}, {
"operation": "default",
"spec": {
"status": "${status}",
"home[]": {
"*": {
"name": "${name}",
"id" : "${id},
"address": "${address}"
}
}
}
}]
In addition to 7632695's answer, your shift spec won't match "mobileno" on the input, try the following:
[{
"operation": "shift",
"spec": {
"*": {
"to": "home[&1].mobileno"
}
}
}, {
"operation": "default",
"spec": {
"status": "${status}",
"details": "${details}",
"home[]": {
"*": {
"name": "${name}",
"id": "${id}",
"address": "${address}"
}
}
}
}]
Also, note that for a single flow file, the attributes are constant, so for each entry in the home array, each id, name, and address field will be the same. From your attributes, how would JOLT know to use id=1 for the first element and id=2 for the second, and so on?
If you want to use the index of the input array as the id, you can add this spec to your chain:
{
"operation": "shift",
"spec": {
"home": {
"*": {
"$": "home[&1].id",
"*": "home[&1].&"
}
},
"*": "&"
}
}
And if you want them to start at 1 instead of 0, you can add 1 to each of them by adding the following spec to your chain:
{
"operation": "modify-overwrite-beta",
"spec": {
"home": {
"*": {
"id": "=intSum(#0, 1)"
}
}
}
}
In default operation you need to add details attribute.
try with below jolt spec
[{
"operation": "shift",
"spec": {
"*": "home",
"mobileno": "home[0].mobileno"
}
}, {
"operation": "default",
"spec": {
"status": "${status}",
"details":"${details}",
"home[]": {
"*": {
"name": "${name}",
"id": "${id}",
"address": "${address}"
}
}
}
}]