Is there a way to modify NiFi flow dynamically using Java API? The use case is to add a processor to an active data flow (data is flowing through it). The new processor should be added at the beginning of the flow without application disruption or downtime.
In case Java API is not available, please feel free to suggest alternatives. I have already looked at change-nifi-flow-using-rest-api-part-1. Thanks.
Any action you can perform from the UI can also be performed from REST API, the UI is just making calls to the REST API behind the scenes.
I would suggest opening Chrome's Dev Tools and performing the action you are interested in and then seeing what requests were made to perform the action. You can then script these operations however you need.
In addition, if you are trying to deploy flows then you should be taking advantage of NiFi Registry which allows you to place a flow under version control. You can then make changes from your local instance or dev instance, and upgrade the flow in production in-place without stopping your whole NiFi instance.
Related
I have the following scenario: 2 Services (client) running in one location that also make calls to one central backend not in the same location. The latency to this backend is not very good since the physical distance is very large.
I have now instumented everything using Open Telemerty. Focus relies on traces.
But now I am struggeling to decide how to correctly setup the infrastructure i.e. Storage Backend, OTel Collector. Ideally I would find a solution that collects locally and then on a pull basis gets the data to a central location.
my idea
I am new to this topic and so I need some input please if this makes sense at all.
I also dont want too much overhead for the tracing. Only what is necessary in order for it to work properly.
I understand that the official documentation recommends using NiFi with HTTPS, but it nonetheless contains a word for using NiFi under HTTP, like the nifi.web.http.port property.
Also, I'd like to incrementally incorporate and evolve the NiFi instance into our's current data infrastructure, starting with non-critical data pipelines. So, the TLS layer right now is not necessary and could add friction during the deployment phase. So, I decide to go on the HTTP path.
After changing some settings, I am able to access NiFi's GUI at http://localhost:8080/nifi but I find out that I cannot make any change to the Flow. Write operations, i.e POST / PUT / DELETE requests, are rejected by HTTP 403.
NiFi doc says:
And by monitoring the API traffic between the GUI and NiFi instance, I can confirm that the PermissionsEntity has both canRead:true and canWrite:true.
I used a containerized NiFi instance.
Has anyone also encounter similar problems?
The root canvas may have been set for the default single-user that NiFi 1.14 generates if it starts up without security configuration.
First thing to try is right-clicking on the canvas and granting yourself access if you can.
The second option: try (re)moving the flow.xml.gz, users.xml and authorizations.xml and then restarting Nifi. New files will be generated that may work better with anonymous access.
Either way, setting up security now will probably mean less friction down the road, not more. I strongly advise you to bite the bullet and get it set up securely.
Good afternoon. I'm able to change the global policies for NiFi through REST API, however, I'm trying to edit the access policies for an ARBITRARY processor. I have no idea how to do so. Everything in the NiFi REST API website calls everything else a component (or maybe I'm misinterpreting...)
Anyway, I appreciate all the help/guidance!
The NiFi UI uses the API behind the scenes to perform every action. You can set policies on process groups, remote process groups, processors, funnels, input & output ports, queues, controller services, and reporting tasks. Collectively, these resources are called "components".
If a policy is not set on a specific component, it inherits the policies set on the parent object (i.e. the process group containing it). You can override these policies directly at a granular level.
To set the policy for a specific component, use the POST /policies API. The easiest way to observe the explicit API invocation necessary is to use your browser's developer tools to record the calls made by the UI client while you manually perform the action and then use those API calls.
There are also other tools which make this process easier, such as the official NiFi CLI Toolkit and the (unofficial but very good) NiPyAPI.
Here are some commonly suggested approaches for using NiFi without web UI, along with their respective limitations. Is there a better way to use NiFi in Production without using web UI while still being able to makes changes to data flow design dynamically?
REST API approach: The REST APIs can be used only with previous knowledge of the ID of the components and do not work with NAME of the components.
MiNiFi approach: The MiNiFi is more focused on collection of data at the source. Additionally, the MiNiFi configuration too is tied to the previous knowledge of ID vs NAME of the components.
A typical NiFi dataflow goes through the following environment lifecycle.
You build your flow in a development NiFi setup. You run it, test it, debug it, fix it.
Once you are sure that the flow runs as expected, promote it to the QA setup and perform similar actions.
Finally when your flow passes QA, promote it to the production setup. Have stringent policies set so that no one expect the support team or the admin have access to make the changes to the flow(s).
In other words, you don't have to rely on the REST API (event the UI changes are done through internal REST API calls) or disable Web UI, if you follow the proper dev-qa-prod promotion.
On a side note, you can leverage NiFi Registry to do the dev-qa-prod lifecycle.
Since the Nifi GUI is really making api calls under the hood, is there anyway to capture those requests or logs? I've been using chrome dev tools. Just wondering if there is a way to capture this within nifi for governance purposes.
Chrome Dev tools is the best bet to get the actual API calls.
For auditing purposes there is something a little bit different... from the menu in the top-right there is "Flow Configuration History" which shows every change that has been made to the flow, and who made it (when in a secure instance).
The flow configuration history is also available through the ReportingTask API if you wanted to implement a custom reporting task to push these events somewhere.