How to route a flow on the basis of URL(if it has file path route to one processor and if it contains username and password it routes to some different processor ). Please specify the flow of processor to be used. Thanks.
With the help of NiFi Expression Language and RouteOnAttribute processor, you can route the incoming FlowFiles to different processors.
For ex, lets assume that the FlowFiles generated by the source processor has an attribute assigned to it named url and this attribute can have either of the following value formats:
file:///path/to/some/file
http://somesite.xyz/path/to/some/resource
For the above case, you can add two dynamic properties to RouteOnAttribute as follows:
file : ${url:startsWith("file:///")}
websource : ${url:startsWith("http://")}
Then connect file and websource relationships to the different processors. The following links point to a detailed usage guide(s) and examples:
https://community.hortonworks.com/questions/54811/redirecting-flow-based-on-certain-condition-nifi.html
https://docs.hortonworks.com/HDPDocuments/HDF3/HDF-3.1.1/bk_getting-started-with-apache-nifi/content/routing-on-attributes.html
Related
I need some help in UpdateAttribute processor:
I have a CSV file which contains hostnames. I need to separate each hostname in the FlowFile and pass it as a variable to the REST API.
My REST API part is working fine when passing data manually. However, I didn't get how to pass a variable value as hostname in it.
Sharing sample file:
SRHAPP001,SRHWEBAPP002,SRHDB006,SRHUATAPP4,ARHUATDB98
I don't quite understand your goal, but I assume that you try to pass the hostname to your REST API module by using FlowFile variables.
You can achieve this by using the ExtractText-Processor. You simply use RegEx for separating your hostnames from the CSV file.
For more information, see
https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.12.1/org.apache.nifi.processors.standard.ExtractText/
How can I extract a substring from a flowfile data in Nifi?
If needed, you can split incoming FlowFiles on every hostname by using SplitContent-Processor
while using RouteOnAttribute nifi processor , i have input of json data
[{"dev":"xyz","detail":"abc"}] which i got from convertRecord processor
Routing Strategy :Route to Property name
ifmatch: ${dev:equals( "xyz" )}
I tried ${dev:matches( "xyz")} in both single quotes and double quotes still am not getting flowfile redirecting towards "ifmatch" . its redirecting to unmatched
is there anyway to resolve this i tried many other option
The flowfile content is different from attributes. Content is arbitrary -- could be empty, text, KB of XML, GB of video or binary. Each flowfile also has attributes which are key/value pairs of Strings kept in memory.
If you want to route on this piece of data, you have multiple options:
Use RouteOnText or RouteOnContent to use the actual flowfile content directly.
Extract it to an attribute using EvaluateJsonPath and then route on that attribute.
The Apache NiFi User Guide and In-Depth provide more information around this distinction.
I am new to nifi(using version 1.8.0). I have the requirement of consuming kafka messages which contain vehicle position in the form of lat,lon per message. Since each message will arrive as a flow file, I need to merge all these flow files and make a json file containing the complete path followed by the vehicle. I am using consume kafka processor to subscribe to messages, update attribute processor(properties added are filename:${getStateValue("seq")},seq:${getStateValue("seq"):plus(1)}) to add a sequence number as filename (eg. filename is 1,2, 3 etc) and put file processor to write these files in the specified directory. I have configured FIFO priority queue on all the success relationship between the above mentioned processors.Once, I have received all the messages I want to merge all the flow files. For this I know I have to use get file, enforce order, merge content(merge strategy:bin packing algorithm, merge format:binary concatenation) and put file processor, respectively. Is my approach correct? How should I establish that merging of files takes place in the sequence of their names as filename is a seq number. What should I put in order attribute in enforce order processor?What should in put in group identifier? Are there more custom fields to be added in enforce order processor?
EnforceOrder processor documentation
1.Group Identifier
This property evaluate on each flowfile for your case use UpdateAttribute Processor, add group_name attribute and use the same ${group_name} attribute in Group Identifier property value.
2.Order Attribute
Expression language is not supported.
You can use filename (or) create new attribute in
UpdateAttribute processor and use same attribute name in your
Order Attribute property value.
For reference/usage of Enforce order processor use this template and upload to your NiFi instance.
I am testing a flow in NIFI that checks for specific value of a counter using REST API. I am able to extract it correct value from REST response. However, when I check the condition in Route on Attribute processor, the expected matched condition's results are routing to unmatched processor.
Attached is the :
Flow and configuration
I have already checked my response to be "1". But its routing to unmatched branch.
Is there something wrong with NIFI expression language I am using?
Jasim,
Initial setup check counter attribute in which value is 1.
And modify the expression language like ${counter:equals('1')} or ${counter:matches('1')} instead of contains.
because contains not suitable for your scanerio.
Hope this helpful for you.
Our requirement is split the flow data based on condition.
We thought to use "ExecuteStreamCommand" processor for that (intern it will use java class) but it is giving single flow data file only. We would like to have two flow data files, one is for matched and another is for unmatched criteria.
I looked at "RouteText" processor but it has no feature to use java class as part of it.
Let me know if anyone has any suggestion.
I think you could use GetMongo to read those definition values and store them in a map accessed by DistributedMapCacheClientService, then use RouteOnContent to route the incoming flowfiles based on the absence/presence of the retrieved values.
If that doesn't work, you could instead route the query result from GetMongo to PutFile and then use ScanContent, which reads from a dictionary file on the file system and routes flowfiles based on the absence/presence of those keywords in the content.
Finally, if all else fails, you can use ExecuteScript to combine those steps into a single processor and route to matched/unmatched relationships. It processes Groovy code easily, so you can directly invoke your existing Java class if necessary.