How to take Entire flowfile content in nifi processor - apache-nifi

Question

I am using nifi to develop the data drifting. In my flow using SelectHiveQL processor. The output(flowFile) of the selectHiveQL need to take into next processor.
what is the suitable processor to take the flowFile content and store into userdefined variable have to use the same variable in Executescript to manipulate the data.

Answer 1

The ExecuteScript processor has direct access to the content of the incoming flowfile via the standard API. Here is an example:
def flowFile = session.get();
if (flowFile == null) {
return;
}
// This uses a closure acting as a StreamCallback to do the writing of the new content to the flowfile
flowFile = session.write(flowFile,
{ inputStream, outputStream ->
String line
// This code creates a buffered reader over the existing flowfile input
final BufferedReader inReader = new BufferedReader(new InputStreamReader(inputStream, 'UTF-8'))
// For each line, write the reversed line to the output
while (line = inReader.readLine()) {
outputStream.write("${line.reverse()}\n".getBytes('UTF-8'))
}
} as StreamCallback)
flowFile = session?.putAttribute(flowFile, "reversed_lines", "true")
session.transfer(flowFile, /*ExecuteScript.*/ REL_SUCCESS)
It is dangerous to move the flowfile content to an attribute because attributes and content memory are managed differently in NiFi. There is a more detailed explanation of the differences in the Apache NiFi In Depth guide.

Answer 2

You could use ExtractText to extract the content of your flowfile to an attribute.
In the ExtractText processor, you would create a property(the name you give this property will be a new attribute in your flowfile), and the value of the property will be the regular expression (\A.+\Z). In my experience, this regex is enough to capture the entire content of the flowfile, though I suppose mileage could vary depending on the type of content within your flowfile.