I am new to Nifi and I am trying to do a PostHttp to send XML/JSON data to an nodejs app (it's on Localhost:3000)
I am using this processors:
But I get the next error: PostHTTP Cannot send data to localhost:3000 because the destination does not accept FlowFiles and this processor is configureto deliver flowfiles.
Do I need to use other type of processor?
The App side, works properly (I have tried doing Posts with Insomnia)
Try changing the property "Send as FlowFile" to "false" in PostHTTP.
Related
I would like to use Apache NiFi to subscribe to a REST endpoint that continuously sends streaming data, and process this data using NiFi.
In short, the REST endpoint is provided by an external organization. I use Basic authentication to login. I do not have control over the configuration of this endpoint. I can only consume its data.
When connecting to the endpoint, the server never ends this request. Instead, the connection is kept alive. When the server processes updates to its database, a copy of this data is transmitted to the consumer via this endpoint. This allows me to replicate changes to the database.
From time to time, the server sends empty responses to keep the connection alive. I will filter these "heartbeat" responses.
How can I model this dataflow in Apache NiFi? I was thinking about using the InvokeHTTP processor, but apparently it does not support data streams. Also, how would I have to configure the processors to deal with the streaming nature of data. In other words, the processor should only once connect to the HTTP endpoint, never end this connection, and never reconnect using a pre-defined interval.
Thank you.
I want to send messages from kafka to clients through a websocket,
I have kafka topic of with json records like: {...fields...., type:1}.
A client initiates the connection to websocket with a json message that includes an array field of the types he intrested in like: {...fields..., types:[1,3,4]}
I tried to do it with Nifi but I dont have a way to consume from kafka and at the same time filter the messages that I get by the type.
I would be happy to get recommandations on a way of doing this in Nifi or any other way (Streamsets, implementing costumized websocket that consumes from kafka etc...)
(My code is written in java)
Thanks :)
with Nifi but I dont have a way to consume from kafka and at the same time filter the messages that I get by the type
ConsumeKafka + RecordPath filter should work fine...
Alternatively, you have not mentioned what type of system your webserver is running...
You would write it in Spring Boot... or Django, ... or Flask+Faust, or NodeJS then forward Kafka messages from a built-in consumer to a web-socket.
tl;dr Search "your language + websocket + kafka" ... I'm sure there will be results
Or you could use the Kafka REST Proxy, then just use AJAX instead of a WebSocket.
You could also use Kafka Connect HTTP Sink to just send events to an open webhook (such as Nifi's ListenHTTP), for example.
I want to send a request to a server to download a file. However, when the file is sent by the server, I don't want to actually receive the file. I'm trying to see if it's possible for me to overload a server's bandwidth without having to actually download files.
I'm guessing this should be possible using some sort of a UDP protocol where no acknowledgements are required.
If the download is available via HTTP GET, you can try sending an HTTP HEAD request with the same headers to try to receive a response without the server actually sending the file.
I am trying to send a file through postman with a POST request and that file needs to be saved in the local FS. I tried all the 4 options GetHttp,InvokeHttp,ListenHttp,HandleHttpRequest. Seems like GetHttp,InvokeHttp requires a webservice to be running at the end who is sending the POST request, that would not fit our case use case. I tried using ListenHttp but I am not able to change the default hostname option which it is pointing to localhost. Is there a way to change that?
And I tried using HandleHttpRequest as well, and it seems like the client who is receiving the request should send an acknowledgement to end the connection at the sender side, nifi user guide doesn't provide an example on how to use or configure that. Can someone provide me an example for the use case which I mentioned.
ListenHttp starts an embedded Jetty server and it does not specify a hostname when creating the connector which according to Jetty documentation says that it should be binding to all available network interfaces:
"The network interface this connector binds to as an IP address or a
hostname. If null or 0.0.0.0, then bind to all interfaces."
To use HandleHttpRequest you also need to use HandleHttpResponse... you basically receive a request from a client which turns into a flow file, pass the flow file through whatever processing you want to do, and then send it to HandleHttpResponse to reply back to the client. There is a template here that shows how to use it:
https://cwiki.apache.org/confluence/download/attachments/57904847/Hello_NiFi_Web_Service.xml?version=1&modificationDate=1449369797000&api=v2
http://msdn.microsoft.com/en-us/library/fx6588te%28v=vs.110%29.aspx
This example sets up a server that listens to a port and echoes the received data.
I want to be able to pass data to it via URL in a manner such as typing ip:port;data.
Is that thing possible in any way?
I don't want to have any user interface at all. Just type in the address, port and data in the URL and the server receives the data.