Can someone help me on defining jolt specification to transform input json to Output json for below json ..I have input json in a file as few json objects like {"filename":"test1","value":"10"} {"filename":"test2","value":"10"} . Transformed Output json should be like below : {"filename":"test","overallvalue":"20", "filename":[{"filename":"test1","value":"10"},{"filename":"test2","value":"10"}]}
Filename and Overall value bith I am getting some other fields so can pass as parameter to the jolt specification
Jolt uses \\ as escape character, since the data has $ which is reserved character for jolt.
[
{
"operation": "shift",
"spec": {
"#\\${main_file}": "filename",
"#\\${Overall_status}": "ex_status",
"#\\${final_message}": "error",
"*": "fl_status"
}
}
]
Related
I have stored Jolt Spec which has expression language
[
{
"operation": "modify-overwrite-beta",
"spec": {
"id": "${UUID()}"
}
}
]
I have stored this in DB and fetching and putting it into distributed map cache and then fetching from cache and putting as attribute of the flow file attribute and using JoltTransformJson with nifi version 1.13 but it is not evaluating the EL and the result is coming as same
{
"id" : "${UUID()}"
}
The processor mentioned that it supports Attribute. Am I missing something?
I'm fairly new to NiFi, my question might be basic.
I would like to rename the JSON key in the flowfile. For example:
{"path":"/home/a/a", "size":"12345"}
and I would like to convert to
{"filename":"/home/a/a", "size":"12345"}
Tried using UpdateAttribute, adding a filename attribute with the value ${path} but either I'm doing something wrong or it's not meant to be used for this kind of operation.
How could I rename the attribute in a JSON ?
This is the content of your FlowFile, not an Attribute, so UpdateAttribute is not the right way to go.
The easiest way with JSON content of FlowFiles is going to be via a JOLTTransform.
Give this spec a try:
[
{
"operation": "shift",
"spec": {
"path": "filename",
"*": {
"#": "&"
}
}
}
]
You can test JOLT transforms here with input data and see what the output will be.
I'm trying to access values of an array in Json using FreeFormTextRecordSetWriter
Input data:
{"data":[["1580860800000","67.2"]],"itemid":5917,"label":"xxx","type":"stacked_element"}
Desired output
{"data1":"1580860800000", "data2":"67.2","itemid":5917,"label":"xxx","type":"stacked_element"}
Can this be done using Nifi Expression language ?
I don't believe FreeFormTextRecordSetWriter currently allows access to nested fields, please feel free to write a Jira to add this capability or perhaps a FreeFormTextRecordPathWriter to enable the use of RecordPath expressions.
I assume if you're trying FreeFormTextRecordSetWriter, then you know there will always be two entries in the data array. If that's the case, since the input/output is valid JSON, if there's one object in the flowfile you can use JoltTransformJSON with the following spec:
[
{
"operation": "shift",
"spec": {
"data": {
"*": {
"0": "data1",
"1": "data2"
}
},
"*": "&"
}
}
]
If there is more than one JSON object in the file, you can use JoltTransformRecord with a JsonTreeReader and JsonRecordSetWriter and the above spec.
If you don't know how many elements are in the array, you can still split them up with the following spec, but note that the first element has an index of 0 not 1 (so data0 instead of data1):
[
{
"operation": "shift",
"spec": {
"data": {
"*": {
"*": "data&"
}
},
"*": "&"
}
}
]
UpdateRecord is another option, but I believe you'd still have to know how many elements are in the array.
For example,there are 8 FFs,and then i’ve convert json to attribute for each FF,as follows:
I've add 5 Properties and Value with EvaluateJsonPath in pic.
If i need to convert 1000 multi-attribute,to set 1000 P/V with EvaluateJsonPath is too trouble.
What can i do to this easily?
Any help is appreciate!TIA
You don't have to (and shouldn't) split the individual JSON objects out of the array if you intend to keep them as a group (i.e. merge them back in). In most cases the split-transform-merge pattern has been replaced by record-based processors such as UpdateRecord or JoltTransformRecord. In your case since the data is JSON you can use JoltTransformJSON with the following spec to change the ID field to ID2 without splitting up the array:
[
{
"operation": "shift",
"spec": {
"*": {
"ID": "[#2].ID2",
"*": "[#2].&"
}
}
}
]
Note that you can also do this (especially for non-JSON input/output) with JoltTransformRecord, the only difference being that the spec is applied to each object in the array rather than JoltTransformJSON which applies the spec to the entire array. The JoltTransformRecord spec would look like this:
[
{
"operation": "shift",
"spec": {
"ID": "ID2",
"*": "&"
}
}
]
I need to change the input date to a SQL friendly format in order to insert it into DB. I get errors on both imported_at and processed_at when trying to insert into DB.
My flow: JoltTransformJSON -> ConvertJsonToSql -> PutSql
Input:
{
"transactionDate": "2018-01-01T18:06:00",
}
My Spec:
[
{
"operation": "shift",
"spec": {
"transactionDate": "processed_at"
}
},
{
"operation": "modify-overwrite-beta",
"spec": {
"processed_at": "=${processed_at.replaceAll('T',' '):toDate('yyyy-MM-dd HH:mm:ss'):format('yyyy-MM-dd HH:mm:ss')}"
}
},
{
"operation": "default",
"spec": {
"processed_at": null,
"imported_at": "${now():format('yyyy-MM-dd HH:mm:ss')}"
}
}
]
My idea was this:
1. shift transactionDate into processed_at
2. override processed_at and transform it into a date via toDate function
3. format it into my desired format via format function
This doesn't work, in the best case, I either get an empty processed_at or the initial value.
I tried
${processed_at.replaceAll('T',' '):toDate('yyyy-MM-dd HH:mm:ss'):format('yyyy-MM-dd HH:mm:ss')}
${processed_at:toDate('yyyy-MM-ddTHH:mm:ss'):format('yyyy-MM-dd HH:mm:ss')}
Apparently, I cannot access JSON properties with expression language in the jolt spec in the JoltTransformJSON processor.
The way I made it to work was:
I added before JoltTransformJSON an EvaluateJSONPath processor and extracted processed_at as a Flowfile attribute.
My flow would look like this: EvaluateJSONPath -> JoltTransformJSON -> ConvertJsonToSql -> PutSql
In the JoltTransformJSON I now have access to the Flowfile attribute processed_at extracted earlier. In the Jolt spec, I updated the default operation:
{
"operation": "default",
"spec": {
"processed_at": null,
"processed_at": "${processed_at:replace('T', ''):toDate('yyyy-MM-ddHH:mm:ss'):format('yyyy-MM-dd HH:mm:ss.SSS')}"
}
}
The correct SQL date field format in expression language is: yyyy-MM-dd HH:mm:ss.SSS
Now the flow inserts rows into the database.