Hi I've the following json in flowfile: {"destination.port":"0000","network.packets":"1","event.end":"1563361839","source.address":"1.2.2.1","message":"OK","server.address":"ip-1-2-2-3.ec2.internal","event.action":"ACCEPT","event.module":"s3bucket","source.port":"478","network.protocol":"6","cloud.account.id":"123456","event.type":"data","organization.id":"Fusion","destination.address":"1.2.2.2","network.bytes":"60","event.start":"1563361837","event.kind":"2","host.id":"eni-06f72","timestamp":"2019-07-17T11:16:39.792Z"} with event.end and event.start as epoch.
I'm using nifi-1.9.2.
Help me convert the epoch to date in UTC and replace the same. I have tried using updateAttribute processor with /event.start=${event.start:format("yyyy-MM-dd HH:mm:ss.SSS")}, didn't go right. I have done some reading for a processor but couldn't. Thanks
The source to your problem is how you try to extract event.start from the JSON. The expression language is used for manipulating FlowFile's attributes, and not content of your FlowFile.
If this JSON is in the content of your FlowFile, then use EvaluateJsonPath, with Destination property set to flowfile-attribute and new custom property named event.start set to $['event.start'].
And now that you have it as attribute, transfer it to JoltTransformJSON with Jolt Transformation DSL set to Modify - Overwrite and Jolt Specification set to:
{
"event.start": "${event.start:append('000'):format('yyyy-MM-dd HH:mm:ss.SSS')}"
}
The appended 3 zeros are since epoch in NiFi is in milliseconds.
Related
I have defined a header in my camelcontext file which reads the incoming date and sets into another body. But the problem is the incoming date format is dd-mm-yyyy and I have to pass as MM/DD/YYYY.
/* In this header I am setting the incoming date */
<setHeader headerName="fitnessUpto">
<simple>${exchangeProperty.fitnessUpto}</simple>
</setHeader>
/*But in body I have to pass as MM/DD/YYYY format */
<tm:TMRCValidityDate>${header.fitnessUpto}</tm:TMRCValidityDate>
Can you please suggest how to convert the incoming date format into desired Date format using apache camel, also please attach any dependancy required.
Incoming date format, "fitnessUpto":"11-11-2036" and I have set this in myheader. But want to convert this date format into MM/DD/YYYY and set this new format into Header
You can use this
from("timer:hello?repeatCount=1")
.setProperty("someDateField",constant(new Date()))
.log("${date:exchangeProperty.someDateField:yyyy/MM/dd}");
I have a scenario where list of files are coming from previous processor, where for each file, I have to create json file with attributes of the flowfile. In AttributesToJSON processor configuration there is option to extract pipeline attributes and can create json files/object, if we set Include Core Attributes to true, it will read some of the file properties and forms the json file
the out for the above case in my scenario is …
{"fragment.size":"125"
file.group:"root",
file.lastModifiedTime:"2020-12-22T15:09:13+0000",
fragment.identifier:"ee5770ea-8406-400a-a2fd-2362bd706fe0",
fragment.index:"1",
file.creationTime:"2020-12-22T15:09:13+0000",
file.lastAccessTime:"2020-12-22T17:34:22+0000",
segment.original.filename:"Sample-Spreadsheet-10000-rows.csv",
file.owner:"root",
fragment.count:"2",
file.permissions:"rw-r--r--",
text.line.count:"1"}
}
But the files has other properties, like absolute.path, filename, uuid are missing in the above json file.
My requirement is, get the absolute.path, filename and uuid and concatenate absolute.path+/+filename, assign this to custom attribute say filepath:absolute.path+/+filename and also add uuid to json object.
so my json file should like
{ uuid:"file uuid value", filepath:"absolute.path+/+filename" }
any inputs to get above form of json file
If you look at the docs for AttributesToJSON you can see that you can specificy attributes in the Attributes List property. So you could try listing the properties you want there.
Alternatively. Sounds like you have 1 FlowFile for each File? You could use UpdateRecord to insert fields. You can use the Literal Value for the Replacement Value Strategy which will let you use Expression Language to insert values - for example, you could add a Property called filename with value ${filename} to insert the value of the filename attribute to a field in the JSON called filename.
To concat the two fields you could do ${allAttributes("absolute.path", "filename"):join('/')} or use append().
I want to create Jolt Specification in Update Attribute processor and use in JoltTransformer processor as an attribute like below.
I get this error in JoltTransformer:
. "is invalid because of the specification not valid for the selected transformation.
Reason being Jolt Specification will set during runtime.
Thanks for the help!
Ani
In NiFi-1.6.0 JoltTransformJson processor not accepting attributes in jolt specification property value.
We still need to keep literal jolt specification in property value.
Example:
But Starting from NiFi-1.7.0 JoltTransformJson processor will be evaluated using flow file attributes and variable registry
we can use ${mapping} starting from NiFi-1.7.0
I am trying to insert the whole content of a row of an XML file into a JSON attribute (I am a newbie).
I am doing it this way (tell me if there is an easier way, it's good to now):
I have configured Extract text this way:
And to finish, I configure the Replace Text, giving a JSON format:
But he result appears to be wrong (doesn't work like a normal JSON file, for example if I a try to do a httpPost):
How can I fix this problem?
cheers
If you are concern regards to new lines and json key/values then use NiFi expression language functions on the extracted attribute(data).
ReplaceText Configs:
Replacement value:
{"name" : "user1","time" : "${now()}","data" : "${data:replaceAll('\s',''):escapeJson()}"}
Use escapeJson and replaceAll function to replace all spaces,newlines with ''
Replacement Strategy as Always Replace
(or)
Another way of preparing json message is by using AttributesToJson processor.
if we are using this processor then we need to prepare attributes/values before AttributesToJson processor by using UpdateAttribute processor
Flow:
1.SplitXml
2.ExtractText //add data property to extract content to flowfile attribute
3.UpdateAttribute //add name property -> user1
add time property -> ${now()}
add data property -> ${data:replaceAll('\s',''):escapeJson()}}
4.AttributeToJson //Attributes List -> name,time,data
Destination -> flowfile content
include core attributes -> false
Please guide me the right component for converting string to json
using appropriate Nifi processor component.
Input is a string of content type text/plain
{ productName : "tv", locationName: " chennai"}
Output of EvaluateJsonPath is still the same as I am unable to evaluate json property based on json path due to wrong content type sent as input.
{
productName : "tv",
locationName: " chennai"
}
Note: Tried SplitText, AttirtubesToJson processors not able to achieve desired conversion.
This is because the input data is not valid JSON. I recreated this flow locally and the error from EvaluateJsonPath is
2017-08-22 10:20:21,079 ERROR [Timer-Driven Process Thread-5] o.a.n.p.standard.EvaluateJsonPath EvaluateJsonPath[id=0aec27af-015e-1000-fac5-4e0f455a10fe] FlowFile StandardFlowFileRecord[uuid=b903eeb0-8985-4517-910f-5e3bbbccb8dc,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1503421928125-1, container=default, section=1], offset=376, length=47],offset=0,name=91708717370829,size=47] did not have valid JSON content.
Which condenses to [flowfile] did not have valid JSON content. The processor uses a strict validator and the input you're providing is not valid JSON. You'll need to use text manipulation or regexes to update this content to the following:
{"productName":"tv", "locationName":"chennai"}
Once you have accomplished this (via ReplaceText, etc.), the EvaluateJsonPath processor will work properly.
Also, to be clear, EvaluateJsonPath is designed to execute JSONPath expressions to extract JSON values to flowfile attributes. It is not designed to manipulate arbitrary text into JSON format.
Update
There is no universal process to convert arbitrary data to JSON. Given the specific input you provided, the following values for ReplaceText will convert this to valid JSON:
Search Value: (?<!\")(\w+)(?=[\s:])
Replacement Value: "$1"
Replacement Strategy: Regex Replace
Evaluation Mode: Entire text
If you get incoming data that is invalid in some other way, you'll have to modify this process. You may be interested in something like JSONLint to validate and format your incoming data.