I am using NiFi version 1.8.0.3.3.0.0-165, and not getting an idea for converting an attribute value data type (String to byte).
Is it possible to convert the data type of NiFi flow file attribute.
for attributes you can use this guide
Apache NiFi Expression Language Guide
if you don't find the solution you can use a groovy script to load your attribute and do whatever you want
def flowFile = session.get()
if(!flowFile) return
def val = flowFile.getAttribute('yourattribue')
//mod your val
flowFile = session.putAttribute(flowFile, 'yourattributeout', yourattributeout)
session.transfer(flowFile, REL_SUCCESS)
Related
The following flowfile is the response of an "InvokeHttp":
[
{"data1":"[{....},{...},{....}]","info":"data-from_site"},
{"data2":"[{....},{...},{....}]","info":"data-from_site"},
{"data3":"[{....},{...},{....}]","info":"data-from_site"}
]
I did a "SplitJson", i got each json record as a single flowfile
flowfile 1:
{"data1":"[{....},{...},{....}]","info":"data-from_site"}
flowfile 2:
{"data2":"[{....},{...},{....}]","info":"data-from_site"}
flowfile 3:
{"data3":"[{....},{...},{....}]","info":"data-from_site"}
I want to store each json record in each flowfile in a variable like that:
variable1 = "{"data1":"[{....},{...},{....}]","info":"data-from_site"}"
variable2 = "{"data2":"[{....},{...},{....}]","info":"data-from_site"}"
variable3 = "{"data3":"[{....},{...},{....}]","info":"data-from_site"}"
can someone show me how to store the json record in a variable !
If I understand correctly what you want to do (by "variable", do you mean what is called "attribute" in NiFi?), you can use the EvaluateJsonPath processor configured with:
flowfile-attribute as Destination
json as Return type
Processors I'm referring to
Is it possible that the processor "InvokeHTTP" takes the information "id" from the previous processor(in this case SELECT_FROM_SNOWFLAKE)?
Where i want to change
I would like the "Remote URL" to be something like:
http://${hostname()}:8080/nifi-api/processors/${previousProcessorId()}
No, you can't. But you can get name, id or other properties for current processor group using ExecuteScript or ExecuteGroovy processors somewhere in this flow to find these informations with script:
def flowFile = session.get()
if(!flowFile) return
processGroupId = context.procNode?.processGroupIdentifier ?: 'unknown'
processGroupName = context.procNode?.getProcessGroup().getName() ?: 'unknown'
flowFile = session.putAttribute(flowFile, 'processGroupId', processGroupId)
flowFile = session.putAttribute(flowFile, 'processGroupName', processGroupName)
session.transfer(flowFile, REL_SUCCESS)
After that, you can find get the id of this snow_flake processor in this processor group for example in rest api.
the Remote URL property in InvokeHTTP processor supports nifi expression language.
So, if previous processor sets attribute hostname then you can use it as http://${hostname}:8080/...
However SelectSQL returns result in Avro format.
Probably before InvokeHTTP you need to convert avro to json and then evaluatejsonpath to extract required values into attributes.
I'm using Django rest framework mongoengine after created few documents, if i want add a new field with default value. Is there any way to do that orelse i need to update with few custom function.
Note: I want to fetch the data with filter having a new field name. That time the field is not there. So i'm getting empty.
From what I understand, you are modifying a MongoEngine model (adding a field with a default value) after documents were inserted. And you are having issue when filtering your collection on that new field.
Basically you have the following confusing situation:
from mongoengine import *
conn = connect()
conn.test.test_person.insert({'age': 5}) # Simulate an old object
class TestPerson(Document):
name = StringField(default='John') # the new field
age = IntField()
person = TestPerson.objects().first()
assert person.name == "John"
assert Test.objects(name='John').count() == 0
In fact, MongoEngine dynamically applies the default value when the field of the underlying pymongo document is empty but it doesn't account for that when filtering.
The only reliable way to guarantee that filtering will work is to migrate your existing documents.
If its only adding a field with a default value, you could do this with MongoEngine: TestPerson.objects().update(name='John')
If you did more important/complicated changes to your document structure, then the best option is to get down to pymongo.
coll = TestPerson._get_collection()
coll.update({}, {'$set': {'name': 'John'}})
I am using nifi to develop the data drifting. In my flow using SelectHiveQL processor. The output(flowFile) of the selectHiveQL need to take into next processor.
what is the suitable processor to take the flowFile content and store into userdefined variable have to use the same variable in Executescript to manipulate the data.
The ExecuteScript processor has direct access to the content of the incoming flowfile via the standard API. Here is an example:
def flowFile = session.get();
if (flowFile == null) {
return;
}
// This uses a closure acting as a StreamCallback to do the writing of the new content to the flowfile
flowFile = session.write(flowFile,
{ inputStream, outputStream ->
String line
// This code creates a buffered reader over the existing flowfile input
final BufferedReader inReader = new BufferedReader(new InputStreamReader(inputStream, 'UTF-8'))
// For each line, write the reversed line to the output
while (line = inReader.readLine()) {
outputStream.write("${line.reverse()}\n".getBytes('UTF-8'))
}
} as StreamCallback)
flowFile = session?.putAttribute(flowFile, "reversed_lines", "true")
session.transfer(flowFile, /*ExecuteScript.*/ REL_SUCCESS)
It is dangerous to move the flowfile content to an attribute because attributes and content memory are managed differently in NiFi. There is a more detailed explanation of the differences in the Apache NiFi In Depth guide.
You could use ExtractText to extract the content of your flowfile to an attribute.
In the ExtractText processor, you would create a property(the name you give this property will be a new attribute in your flowfile), and the value of the property will be the regular expression (\A.+\Z). In my experience, this regex is enough to capture the entire content of the flowfile, though I suppose mileage could vary depending on the type of content within your flowfile.
What's the best practice with NIFI to extract an attribute in a flowfile and transform it in a Text Format Example :
{ "data" : "ex" } ===> My data is ex
How can I do this with NIFI wihtout using a executeScript Processor
You could use ExtractText to extract the values into attributes. If you added a property in ExtractText like foo = {"(.+)" : "(.+)"} then your flow file would get two attributes for each of the capture groups in the regex:
foo.1 = data
foo.2 = ex
Then you can use ReplaceText with a Replacement Value of:
My ${foo.1} is ${foo.2}