I have following two scenario and for each one I need recommendation as to which NiFi processor to use:
I have Restful web services running outside NiFi. NiFi would like to get/post/delete/update some data by calling specific restful API. Once the Restful API receives request from NiFi it sends back the response to NiFi. Which NiFi processor to use here?
In 2nd scenario, I have an application running outside NiFi. This application has its own GUI. The user need some information so he want to send request to NiFi. In NiFi, is there any processor which accepts request from application, process the request, and sends response back?
I actually read all the question with getHTTP and invokeHTTP.
I have initially tried with invokeHTTP processor. I tried both get and post call using invokeHTTP. But I don't see any response from Restful API running outside NiFi.
I did not try getHTTP.
I am using NiFi. NiFi do not have code.
I expect NiFi should be able to call Restful API running outside. I expect NiFi should accept request coming from application running outside and process that request.
Yep, NiFi comes bundled with processors that satisfy both of your requirements.
For scenario #1, you can use either a combination of GetHTTP/PostHTTP which as their name implies are HTTP clients that make GET and POST calls respectively. However, later the community came up with InvokeHTTP that offers more features like support for NiFi Expression Language, support for incoming flowfiles, etc.,
For scenario #2, you can either use ListenHTTP or the combination of HandleHttpRequest/HandleHttpResponse. The later literally offers you have a more robust web-service implementation while the former is a simple web-hook kind. I haven't worked much with ListenHTTP so probably can't comment more on that.
Having said that, for your second scenario, if your objective is to consume NiFi statistics, you can directly hit NiFi's rest api, rather than having a separate NiFi flow with web service capability.
Useful Links
https://pierrevillard.com/2016/03/13/get-data-from-dropbox-using-apache-nifi/
https://dzone.com/articles/using-websockets-with-apache-nifi
https://ddewaele.github.io/http-communication-with-apache-nifi/
Related
I want to develop an application where I want to push the messages (or data) to UI from backend Spring boot application.
I have the following requirement -
Consider there is a REST service that accepts the data from other applications using the POST method.
This data will be pushed to UI.
OR
Consider that there is a background process running which generate events and we want to push these events to UI.
For this, I came across about the WebSocket component that we can use in the Spring Boot application.
However, is there any other settings required to make it possible to push the incoming data to the UI?
Any help is appreciated.
Thanks,
Avinash Deshmukh
The backend cannot magically push updates to a client UI. The backend will have no way of knowing where the UI exists (i.e. what the UI's ip address is) and even if it did, it may not have access to establish a connection (due to firewalls or a NAT).
For this reason a client UI has to request updates. One way this could be done would be to have a timer in the UI application that polls for updates via REST. But this is essentially what websockets do - with much less overhead.
This is how common applications that you use everyday work all the time. So I'm not sure why you do not want to go down the websockets route.
...
Starting with Spring 5.0.5.RELEASE, it isn’t necessary to do any customization because of the improvement of #SendToUser annotation, that allows us to send a message to a user destination via “/user/{sessionId}/…” rather than “/user/{user}/…“.
That means the annotation works relying on the session id of the input message, effectively sending a reply to destination private to the session:
...
There is a good example over here:
https://www.baeldung.com/spring-websockets-sendtouser
We have multiple team nifi applications running in same nifi machine... Is there any way to log the logs specific to my application? Also by default nifi-app.log file is difficult to track the issues and bulletin board shows the error msg for only 5 mins... How to get the errors captured and send an mail alert in Nifi?
Please help me to get through this. Thanks in advance!
There are a couple ways to approach this. One is to route failure relationships from processors to a PutEmail processor which can send an alert on errors. Another is to use a custom reporting task to alert a monitoring service when a certain number of flowfiles are in an error queue.
Finally, we have heard that in multitenant environments, log parsing is difficult. While NiFi aims to reduce or completely eliminate the need to visually inspect logs by providing the data provenance feature, in the event you do need to inspect the logs, we recommend searching the log by processor ID to isolate relevant messages. You can also use NiFi itself to ingest those same logs and perform parsing and filtering activities if desired. Future versions may improve this experience.
By parsing the nifi log, you can separate the logs which is specific to your team applications, by using the processor group id and using Nifi Rest API. Check the below link for the nifi template and python codes to solve this issue:
https://link.medium.com/L6IY1wTimV
You can send all the errors in a Processor Group to the same processor, it could be a regular UpdateAttribute or a custom processor, this processor is going to add the path and all the relevant information and then send this to a general error/logs flow that is going to check the information inside the flowfile regarding to the error, and will make the decision of send or not an email, to whom and this kind of things.
Using this approach, the system keeps simple and inside NiFi, so you don't add more layers of complexity, and you are going to have only one processor to manage the errors per Process Group.
This is the way we are managing errors in my company.
Objective:- Looking for an option to use SNMP trap receiving mechanism as a custom processor in Apache Nifi.
Looked at references like how to create a custom processor in Nifi (Apache docs and youtube videos),
Nifi Source code of GetSNMP (including AbstractSNMPProcessor), ListenSysLog, GetFile etc.
Have got a simple (java code) SNMPTrap receiver using SNMP4J library and can listen to specific address with UDP or TCP based port. And then on occurence of a Trap (can be simulated by simple Java code of SNMP4J library) can print PDU details.
Now while trying to write this code within Nifi as a custom processor, not sure where to put my listen mechanism and then processing the actual PDU part.
Kind of struck here.
More details :-
GetSNMP processor talks about a Specific OID and using that we have options of GET or Walk strategies to get info. Here (for my objective) looking for an option where Nifi server is running and want to get SNMP traps picked from specific system where Nifi is running. And for my objective, didn't get context to extend GetSNMP code.
In ListenSyslogProcessor, there was a blocking queue mechanism. And from it I could not derive to go for listening at an ip address for traps and how exactly to use ProcessContext, ProcessSession option of onTrigger method for a Nifi custom processor.
Any inputs are welcome...
One Possible Solution :-
Could not create a Custom processor in Apache Nifi to Receive SNMP traps. Something else was possible, I will illustrate that here. It may be of some help to others.
No change in the Java code
Used ExecuteProcess Processor of Nifi to run the above Java code.
On success result is pushed to further components in the Nifi Flow.
Actually tried to catch SNMP traps using linux commands like net-snmp, snmptrapd. These options were not actually catching the SNMP traps, tested thru Java code trap sender and then Traps generated using commands from online sources.
If you have better ideas, please share.
I am currently using the ListenHTTP processor to accept flow files from a different NiFi instance. This works fine but for some reason GET requests do not work. Does ListenHTTP only allow POST requests ?
This is the error I receive:
HTTP method GET is not supported by this URL
P. S I am aware of a more advanced HandleHTTPRequest processor.
Yes, ListenHTTP only accepts POST and HEAD requests. GET, PUT, and DELETE are not accepted by the processor and will return a 405 HTTP Status Code. The documentation of the processor could be improved to document this.
You are correct that to handle GET requests, you should use the HandleHTTPRequest processor.
However, if your use case is transmitting flowfiles between two NiFi instances, you will get much better behavior and performance by using the Site to Site capability. This can be routed over HTTP(S) or raw sockets, provides security, integrity, load balancing, and many additional features.
I would like to just HTTP POST events into a spout. Do I need to set up a web server myself, or would that be redundant? All of the tutorials that I have seen so far assume that an application will be fetching (or even just generating) the data itself and passing it to emit-spout!.
Storm used a pull based model in Spouts.nextTuple(). Thus, it might be best to have a buffer in between -- a WebServer takes HTTP POST requests and writes into that buffer. A Spout can pull the date from the buffer.