Get JSON data through Kafka broker.
The data are in the following format and the image data is encoded into Base64.
e.g){"filename":"test.jpg","filedata":"/9j/4AAQSkZJRgABAQEAYABgA....."}
I want to save image data that I received through Kafka as a file.
However, it is not working properly.
Below is the order in which I wrote the flow and describes only the key settings.
ConsumeKafka_2_0 processor
EvaluateJsonPath Processor
Destination flowfile-content
rawbytes $.filedata
EvaluateJsonPath Processor (error : did not have valid JSON Content)
Destination flowfile-attribute
filename $.filename
Base64EncodeContent processor
PutFile processor
When the flow is executed, the image file is saved normally, but the file name cannot be set. What should I do?
Do you have any site or examples to refer to?
The site of reference is https://community.hortonworks.com/articles/218015/ingesting-binary-files-like-pdf-jpg-png-to-hbase-w.html
according to PutFile documentation:
Reads Attributes filename: The filename to use when writing the FlowFile to disk.
you just need to use UpdateAttribute processor to set value for filename attribute
From question I understood that there is a kafka topic which has filename and base64 encoded file content in json format; You want to consume the kafka topic, decode file content with Base64 to construct an image, and store the image in the filename using PutFile.
I came up with a flow that will achieve this requirement and is self explanatory.
ConsumeKafkaRecord_2_0 (Consumes {"filename":"test.jpg","filedata":"/9j/4AAQSkZ.."})
EvaluateJsonPath
Destination: flowfile-attribute
rawtypes: $.filedata
filename: $.filename
ReplaceText (Changing flow file content to encoded image content for next processor)
Base64EncodeContent (rawtypes is decoded to image by this processor)
UpdateAttribute (File name to store the image is updated here)
PutFile
Unable to upload flow template here. Posting key processors screenshot
EvaluateJSONPath
ReplaceText - (Note replacement value)
UpdateAttribute
In step #2 you have replaced the flow file content with the value of $.filedata which is no longer JSON, so you can't use EvaluateJsonPath in step 3 since there is no more JSON.
If you reverse the steps 2 and 3, then you can extract the filename to an attribute and still have the original JSON in the flow file content, then extract the filedata to the content.
Related
while using RouteOnAttribute nifi processor , i have input of json data
[{"dev":"xyz","detail":"abc"}] which i got from convertRecord processor
Routing Strategy :Route to Property name
ifmatch: ${dev:equals( "xyz" )}
I tried ${dev:matches( "xyz")} in both single quotes and double quotes still am not getting flowfile redirecting towards "ifmatch" . its redirecting to unmatched
is there anyway to resolve this i tried many other option
The flowfile content is different from attributes. Content is arbitrary -- could be empty, text, KB of XML, GB of video or binary. Each flowfile also has attributes which are key/value pairs of Strings kept in memory.
If you want to route on this piece of data, you have multiple options:
Use RouteOnText or RouteOnContent to use the actual flowfile content directly.
Extract it to an attribute using EvaluateJsonPath and then route on that attribute.
The Apache NiFi User Guide and In-Depth provide more information around this distinction.
I have following business need. Can anybody please suggest me NiFi WorkFlow I should create? Thanks
1) Through Kakfa I get metadata as JSON Object. This JSON Object has an image or video which is in binary format. This binary file is pretty huge.
2) I need to extract binary data and send it to HTTP rest (POST).
In my mind I have following workflow:
ConsumeKakfa==>EvaluateJsonPath==>UpdateAttributes=>InvokeHTTP
Explanation:
1) ConsumeKakfa will receives metadata as json object.
2) EvaluateJsonPath will extract content json attribute which has image or video data stored as base64.
3) UpdateAttribute will update the flowfile to insert POST payload.
4) InvokeHTTP will invoke POST HTTP rest call.
I am not sure whether huge data will be handle by InvokeHTTP.
your flow should be like this:
ConsumeKafka
EvaluateJsonPath (destination=content) stores evaluated base64 binary into flowfile content
Base64EncodeContent (decode) decodes base64 content into a binary
InvokeHTTP sends everything in content as a body
I am trying to insert the whole content of a row of an XML file into a JSON attribute (I am a newbie).
I am doing it this way (tell me if there is an easier way, it's good to now):
I have configured Extract text this way:
And to finish, I configure the Replace Text, giving a JSON format:
But he result appears to be wrong (doesn't work like a normal JSON file, for example if I a try to do a httpPost):
How can I fix this problem?
cheers
If you are concern regards to new lines and json key/values then use NiFi expression language functions on the extracted attribute(data).
ReplaceText Configs:
Replacement value:
{"name" : "user1","time" : "${now()}","data" : "${data:replaceAll('\s',''):escapeJson()}"}
Use escapeJson and replaceAll function to replace all spaces,newlines with ''
Replacement Strategy as Always Replace
(or)
Another way of preparing json message is by using AttributesToJson processor.
if we are using this processor then we need to prepare attributes/values before AttributesToJson processor by using UpdateAttribute processor
Flow:
1.SplitXml
2.ExtractText //add data property to extract content to flowfile attribute
3.UpdateAttribute //add name property -> user1
add time property -> ${now()}
add data property -> ${data:replaceAll('\s',''):escapeJson()}}
4.AttributeToJson //Attributes List -> name,time,data
Destination -> flowfile content
include core attributes -> false
Hi,
I am using GetSNMP processor to connect a radio. As per the NiFi documentation, this information is written to flow file attributes not to flow file contents. So, I used AttributesToJSON processor. After that I used PutFile processor to write these attributes to a file. Files are generated but there are not attributes written there. Only "{}" is written in each of the file. Using LogAttribute processor , I can see all attributes in the log file but I want them in a separate file.
Please guide.
Thanks,
SGaur,
If incoming flow file content is empty before putFile processor then it will writes empty content in local directory.
So you have to write attributes into flowfile content using ReplaceText.
For an example, You having this attributes like
${filename}-->input.1,
${input.content.1}-->content.1,
${input.content.2}-->content.2
comes before putFile.
Now you have to write those attributes into flow file content like below.,
In ReplaceText, Just mention replacement value to be like this-->
${filename},${input.content.1},${input.content.2}
It will replace content like below.,
input.1,content.1,content.2
Now it will write into local file using put file processor.
Hope this helpful for you.
Please guide me the right component for converting string to json
using appropriate Nifi processor component.
Input is a string of content type text/plain
{ productName : "tv", locationName: " chennai"}
Output of EvaluateJsonPath is still the same as I am unable to evaluate json property based on json path due to wrong content type sent as input.
{
productName : "tv",
locationName: " chennai"
}
Note: Tried SplitText, AttirtubesToJson processors not able to achieve desired conversion.
This is because the input data is not valid JSON. I recreated this flow locally and the error from EvaluateJsonPath is
2017-08-22 10:20:21,079 ERROR [Timer-Driven Process Thread-5] o.a.n.p.standard.EvaluateJsonPath EvaluateJsonPath[id=0aec27af-015e-1000-fac5-4e0f455a10fe] FlowFile StandardFlowFileRecord[uuid=b903eeb0-8985-4517-910f-5e3bbbccb8dc,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1503421928125-1, container=default, section=1], offset=376, length=47],offset=0,name=91708717370829,size=47] did not have valid JSON content.
Which condenses to [flowfile] did not have valid JSON content. The processor uses a strict validator and the input you're providing is not valid JSON. You'll need to use text manipulation or regexes to update this content to the following:
{"productName":"tv", "locationName":"chennai"}
Once you have accomplished this (via ReplaceText, etc.), the EvaluateJsonPath processor will work properly.
Also, to be clear, EvaluateJsonPath is designed to execute JSONPath expressions to extract JSON values to flowfile attributes. It is not designed to manipulate arbitrary text into JSON format.
Update
There is no universal process to convert arbitrary data to JSON. Given the specific input you provided, the following values for ReplaceText will convert this to valid JSON:
Search Value: (?<!\")(\w+)(?=[\s:])
Replacement Value: "$1"
Replacement Strategy: Regex Replace
Evaluation Mode: Entire text
If you get incoming data that is invalid in some other way, you'll have to modify this process. You may be interested in something like JSONLint to validate and format your incoming data.