I'm fairly new to NiFi, my question might be basic.
I would like to rename the JSON key in the flowfile. For example:
{"path":"/home/a/a", "size":"12345"}
and I would like to convert to
{"filename":"/home/a/a", "size":"12345"}
Tried using UpdateAttribute, adding a filename attribute with the value ${path} but either I'm doing something wrong or it's not meant to be used for this kind of operation.
How could I rename the attribute in a JSON ?
This is the content of your FlowFile, not an Attribute, so UpdateAttribute is not the right way to go.
The easiest way with JSON content of FlowFiles is going to be via a JOLTTransform.
Give this spec a try:
[
{
"operation": "shift",
"spec": {
"path": "filename",
"*": {
"#": "&"
}
}
}
]
You can test JOLT transforms here with input data and see what the output will be.
Related
I'm working in Apache NiFi and I've a question: how to handle nested arrays in JSON with QueryRecord processor? For example I've a JSON:
{
"offerName":"Viatti Strada Asimmetrico V-130 205/55 R16 91V",
"detailedStats":[
{
"type":"mobile",
"clicks":4,
"spending":"2.95"
}
]
}
How can I extract array to get the following result:
{
"offerName": "Viatti Strada Asimmetrico V-130 205/55 R16 91V",
"type": "mobile",
"clicks": 4,
"spending": "2.95"
}
I read about RPATH, but didnt find good examples.
Tried with:
SELECT RPATH(detailedStats, '/detailedStats[1]')
FROM FLOWFILE
But it throws error. How can i get expected result with RPATH?
You can select like below via QueryRecord . However it seems you are having an issue while writing. I used JsonRecordSetWriter with Inherent Record Schema. this is a good tutorial If you prefer avro schema
SELECT offerName,
RPATH_STRING(detailedStats, '/type') type,
RPATH_INT(detailedStats, '/clicks') clicks,
RPATH_STRING(detailedStats, '/spending') spending
FROM FLOWFILE
result is an array, so you should split it with $.* at the downstream
An alternative method might be adding a JoltTransformJSON processor with (shift type) specification, which's reached from the Advanced button of Settings tab, with the following code
[
{
"operation": "shift",
"spec": {
"detailedStats": {
"*": {
"#(2,offerName)": "offerName",
"*": "&"
}
}
}
}
]
in order to extract your desired result.
Can someone help me on defining jolt specification to transform input json to Output json for below json ..I have input json in a file as few json objects like {"filename":"test1","value":"10"} {"filename":"test2","value":"10"} . Transformed Output json should be like below : {"filename":"test","overallvalue":"20", "filename":[{"filename":"test1","value":"10"},{"filename":"test2","value":"10"}]}
Filename and Overall value bith I am getting some other fields so can pass as parameter to the jolt specification
Jolt uses \\ as escape character, since the data has $ which is reserved character for jolt.
[
{
"operation": "shift",
"spec": {
"#\\${main_file}": "filename",
"#\\${Overall_status}": "ex_status",
"#\\${final_message}": "error",
"*": "fl_status"
}
}
]
I'm trying to access values of an array in Json using FreeFormTextRecordSetWriter
Input data:
{"data":[["1580860800000","67.2"]],"itemid":5917,"label":"xxx","type":"stacked_element"}
Desired output
{"data1":"1580860800000", "data2":"67.2","itemid":5917,"label":"xxx","type":"stacked_element"}
Can this be done using Nifi Expression language ?
I don't believe FreeFormTextRecordSetWriter currently allows access to nested fields, please feel free to write a Jira to add this capability or perhaps a FreeFormTextRecordPathWriter to enable the use of RecordPath expressions.
I assume if you're trying FreeFormTextRecordSetWriter, then you know there will always be two entries in the data array. If that's the case, since the input/output is valid JSON, if there's one object in the flowfile you can use JoltTransformJSON with the following spec:
[
{
"operation": "shift",
"spec": {
"data": {
"*": {
"0": "data1",
"1": "data2"
}
},
"*": "&"
}
}
]
If there is more than one JSON object in the file, you can use JoltTransformRecord with a JsonTreeReader and JsonRecordSetWriter and the above spec.
If you don't know how many elements are in the array, you can still split them up with the following spec, but note that the first element has an index of 0 not 1 (so data0 instead of data1):
[
{
"operation": "shift",
"spec": {
"data": {
"*": {
"*": "data&"
}
},
"*": "&"
}
}
]
UpdateRecord is another option, but I believe you'd still have to know how many elements are in the array.
For example,there are 8 FFs,and then i’ve convert json to attribute for each FF,as follows:
I've add 5 Properties and Value with EvaluateJsonPath in pic.
If i need to convert 1000 multi-attribute,to set 1000 P/V with EvaluateJsonPath is too trouble.
What can i do to this easily?
Any help is appreciate!TIA
You don't have to (and shouldn't) split the individual JSON objects out of the array if you intend to keep them as a group (i.e. merge them back in). In most cases the split-transform-merge pattern has been replaced by record-based processors such as UpdateRecord or JoltTransformRecord. In your case since the data is JSON you can use JoltTransformJSON with the following spec to change the ID field to ID2 without splitting up the array:
[
{
"operation": "shift",
"spec": {
"*": {
"ID": "[#2].ID2",
"*": "[#2].&"
}
}
}
]
Note that you can also do this (especially for non-JSON input/output) with JoltTransformRecord, the only difference being that the spec is applied to each object in the array rather than JoltTransformJSON which applies the spec to the entire array. The JoltTransformRecord spec would look like this:
[
{
"operation": "shift",
"spec": {
"ID": "ID2",
"*": "&"
}
}
]
If input is something like this:
{
"Vendor":[{"a":"..."},{"b":"..."}]
}
and attribute is something like this:
{
"Vendor":[{"c":"..."},{"d":"..."}]
}
how can I generate this output:
{
"Vendor":[{"a":"..."},{"b":"..."},{"c":"..."},{"d":"..."}]
}
I think joltTransformJson is best option, but I couldn't generate desired output.
If the attribute is named json then this should work:
[
{
"operation": "default",
"spec": {
// extract vendor array from json attribute and put it in a temporary array
"tempArray": ${json:jsonPath('$.Vendor')}
}
},
{
"operation": "shift",
"spec": {
"Vendor": "Vendor", // keep Vendor array as is
"tempArray": { // put temp array elements inside Vendor
"*": "Vendor"
},
"*": "&" // keep all of the other elements of the json
}
}
]
I provided explanation in comments, don't forget to remove them though the json will be valid!
How about putting the attribute to content by using AttributesToJson processor and then merging the two flow files by using MergeContent processor?