In an Apache NIFI dataflow I try to transform a date from MM-dd-yyyy to yyyy-MM-dd (the default format which wil be accepted by MySQL as a date).
In the NIFI advanced editor of a JoltTransformJSON 1.15.0 Processor with DSL "Chain" I have entered:
Input Json:
{
"Name": "Jan",
"Birthday": "12-31-1994"
}
Transformation Jolt script:
[{
"operation": "modify-overwrite-beta",
"spec": {
"Birthday": "=${Birthday:toDate('MM-dd-yyyy'):format('yyyy-MM-dd')}"
}
}]
Result:
{
"Name": "Jan",
"Birthday": "12-31-1994"
}
I do not get any syntax errors. It seems the Jolt transformation does not change anything.
Why is Birthday not transformed?
You can use the following consecutive specs within a JoltTransformJSON processor's specification part
[
{
"operation": "modify-overwrite-beta",
"spec": {
"b": "=split('-', #(1,Birthday))",
"Birthday": "=concat(#(1,b[2]),'-',#(1,b[0]),'-',#(1,b[1]))"
}
},
{
"operation": "remove",
"spec": {
"b": ""
}
}
]
where split the current date value by dashes and reorder them by use of concat function in the first step. Then remove the auxiliary b attribute in the last step
Related
Probably a simple beginners question:
Using NIFI, I want to split an array (represented flowfile-content) in the form of
["x1","x2", ..]
and format it to a JSON object of the form
{"key1":"x1", "key2":"x2", ..}
(also as flowfile-content)
What processor is the most efficent to be used, how would the expression script look like ?
Thanks in advance,
Marc
Applying Jolt transformations through use of a JoltTransformJSON processor might be a good choice. In which, you can add such a specification below by clicking on the ADVANCED button that's stated in the SETTINGS tab of Configure Processor dialog box :
[
//Determine respective lists to be used to calculate their sizes within the next step
//in order to get ordinals for the key values
{
"operation": "shift",
"spec": {
"*": {
"*": "&(0,0).[#2]"
}
}
},
{
"operation": "modify-overwrite-beta",
"spec": {
"*": "=size(#(1,&))"
}
},
//exchange key and values
{
"operation": "shift",
"spec": {
"*": {
"$": "#(0)"
}
}
},
//add the desired word such as "key"
{
"operation": "shift",
"spec": {
"*": "key&(0,0)"
}
}
]
I am a newbie to Nifi so lack still in learning this too. I have a task to get data from executeSQL and the record return like this:
[{
"PKG_HA_01_ON":2.0,
"PKG_HA_03_ON":28.0,
"PKG_HA_04_ON":7.0,
"PKG_HA_05_ON":0.0,
"PKG_HA_06_ON":1.0,
"PKG_HA_09_ON":5.0
},
{
"PKG_HA_01_ON":8.02905,
"PKG_HA_03_ON":57.29038333333333,
"PKG_HA_04_ON":2.2858666666666663,
"PKG_HA_05_ON":60.0,
"PKG_HA_06_ON":12.291533333333332,
"PKG_HA_09_ON":12.3363
}]
This is the result of a union query. Now, I would like to convert into this:
[{
"machine":"PKG_HA_01_ON",
"counter":2.0,
"duration":8.02905
},
{
"machine":"PKG_HA_03_ON",
"counter":28.0,
"duration":57.29038333333333
}]
I have researched on JoltTransformJSON but still stuck with it. So what is the best way to achieve desired JSON?
Thanks in advance!!!
You can consecutively apply shift transformation as
[
{
"operation": "shift",
"spec": {
"*": {
"#PKG_HA_01_ON": "PKG_HA_01_ON.&",
"#PKG_HA_03_ON": "PKG_HA_03_ON.&"
}
}
},
{
"operation": "shift",
"spec": {
"*": {
"$": "&.machine",
"#(1,&.0)": "&.counter",
"#(1,&.1)": "&.duration"
}
}
},
{
"operation": "shift",
"spec": {
"*": ""
}
}
]
Restrict the result set to two keys(PKG_HA_01/3_ON) in the first step while determining the values for the keys 0 and 1 respectively, then convert those numbered keys to yours (counter and duration respectively) while adding the machine element, and remove the key names at the last step.
I am very new to Nifi. In Nifi using JoltTransformJson I need to convert following input to below output: Can someone please provide the Jolt specification. I have tried several ways to do this but could not get the same output.
Input
{
"myOps": {
"Ops1": {
"type": "software",
"url": "url-software"
},
"Ops2": {
"type": "hardware",
"url": "url-hardware"
}
}
}
Output
{
"type": "software",
"url": "url-software"
},
{
"type": "hardware",
"url": "url-hardware"
}
You don't need Jolt for this. You can use the EvaluateJsonPath processor with a JsonPath expression of $.myOps.* which will return a valid JSON array containing the two objects you're looking for. JsonPath.com is a good resource for evaluating & debugging JsonPath expressions.
If you have a requirement to avoid the wrapping [] (although that's valid JSON), you can use ReplaceText to remove them.
Check this spec,
[
{
"operation": "shift",
"spec": {
"myOps": {
"Ops*": {
"#": "[]"
}
}
}
}
]
I need to transform a JSON response from one time-series DB and output it to as a response in a new format.
Having an input JSON array, I need to transform it with JoltTransformJson (NiFi) and add a key-value for each item from a FlowFile attribute.
This is my input JSON:
{
"Items": [
{
"Timestamp": "2020-04-29T07:46:20.558731Z",
"Value": 66.0303
},
{
"Timestamp": "2020-04-29T07:46:35.558731Z",
"Value": 69.11584
}
]
}
The desired output should be:
[{
"sensor_id": "xyz",
"sample_time": "2020-04-29T07:46:20.558731Z",
"sample_value": 66.0303
}, {
"sensor_id": "xyz",
"sample_time": "2020-04-29T07:46:35.558731Z",
"sample_value": 69.11584
}]
where sensor_id is a FlowFile attribute...
I came across to this spec:
[{
"operation": "shift",
"spec": {
"Items": {
"*": {
"#Timestamp": "[#2].sample_time",
"#Value": "[#2].sample_value",
"${sensor_id}": "[#2].sensor_id"
}
}
}
}]
But I cannot get the sensor_id in the output json.... instead, this is what I get:
[{
"sample_time": "2020-04-29T07:46:20.558731Z",
"sample_value": 66.0303
}, {
"sample_time": "2020-04-29T07:46:35.558731Z",
"sample_value": 69.11584
}]
Finally I've managed to project the ${sensor_id} into the output json, what was missing is #, so the final spec will be:
[{
"operation": "shift",
"spec": {
"Items": {
"*": {
"#Timestamp": "[#2].sample_time",
"#Value": "[#2].sample_value",
"#${sensor_id}": "[#2].sensor_id"
}
}
}
}]
Now, I've found that the Timestamp returned from my time-series DB is not exact what I need it to be. Instead I would like to convert it to integer timestamp (epoch sinse 1970)... Any idea how this can be achieved?
I have JSON payload like this;
{
"id": "",
"name": "",
"A": {...},
"B": {...},
"C": {...}
}
And I want to extract A, B and C fields with id and name field as different record. Like this;
{
"id": "",
"name": "",
"A": {...}
}
{
"id": "",
"name": "",
"B": {...}
}
{
"id": "",
"name": "",
"C": {...}
}
I'm using record based processors. But I don't know that how can I do this in Nifi using record based processors.
The "EvaluateJsonPath" is probably what you're looking for. You can add JSONPath expressions, that will be converted to attributes, or written to the flowfile.
http://jsonpath.com/ is a handy web tool to test your expressions.
If you want to use record based processors, then JoltTransformRecord would do the trick. Just set Jolt Transformation DSL as Chain and Jolt Specification as:
[
{
"operation": "shift",
"spec": {
"id": "id",
"name": "name",
"*": {
"#": "array.&"
}
}
},
{
"operation": "shift",
"spec": {
"array": {
"*": {
"#(2,id)": "[#2].id",
"#(2,name)": "[#2].name",
"#": "[#2].&"
}
}
}
}
]
This will first put your unique elements in an array and separate the common keys from them, then it will put the common keys in all of the elements while extracting the array to a top array.
Then, if you want them as different FlowFiles too, you can SplitRecord the array and you got it!