Nifi flow has stopped working. Today, on starting the flow all the processors icon turns green, but there is no input or output as GenerateFlowFile shows active threads and terminated threads. This count keeps on increasing or decreasing if I terminate manually or start flow. I also tried to remove the GenerateFlowFile processor and create once again, but same result.
On disabling all processors are disabled but not the GenerateFlowFile.
What is the reason behind this active thread and how to run the flow once again.
Opening the nifi-app.log file and clearing it resolved this issue for me.
Related
I have a process (it is a windows service). It throws bad_alloc exception and stops. Later it is being started by another monitoring tool. I want to see the memory related details specific to that process just before it stops.
The tools like Process explorer, VMmap can be used for running processes. But, as my process stops we loose the data here. Is there any way to log the data of this process till it stops/ till some time period?
I tried 2 options in VMmap for the same.
(a) View Running process option works fine, but it needs regular 'Refresh' from user and During refresh if the process is stopped/restarted (now it is with new PID) the previous data are lost.
(b) Launch and trace a new process(here I have option of auto refresh after each second) -But it is not able to initiate my windows service.
Could you please suggest if there are any other ways for it?
I referred multiple articles for this , but none of them helped in my case.
The reason to capture logs is- these services are in production system on customer machines, so cannot analyse at the time of issue.
I am using Performance Monitor (PerfMon) to capture data specific to my process for every 10 minutes. It gives me both historic data as well as the current data.
It works fine these days. Suddenly Processors still have task running even been stopped, and the running task need to terminate manually.
Any thoughts?
update 1
I use nipyapi to manipulate some processors to start and stop over and over again. There are the APIs I used
nipyapi.canvas.get_processor(identifier=p_id, identifier_type='id')
nipyapi.canvas.get_process_group(identifier=pg_id, identifier_type='id')
nipyapi.canvas.schedule_processor(processor=p_id, scheduled=True, refresh=True)
I restart NiFi and problem solved, but after executing those APIs many times (about 10000 times, grep processor id | wc -l) problem occurred.
I reckon those APIs create a lot web connections and not being stopped.
Stopping a processor is really just telling the scheduler not to trigger any more executions. It is often the case that an already-triggered thread is still executing after the processor has been stopped, which is why the Terminate option was added.
I'm trying to pull some data from HDFS. I'm running the listHDFS and fetchHDFS processor for this.
When I stopped the fetchHDFS processor, there were a number of active threads even after stopping the processor. To kill these threads, I used the "terminate" option.
Just wanted to know the working of the terminate option.
Does it gracefully shut all the connections with the FS?
Since all the threads are killed, do I lose out on the data that was consumed by these threads?
Is it advised the terminate option only when the threads are stuck or the flow enters a frozen state?
When you stop a processor it tells the NiFi framework to no longer schedule/execute the processor, but there may already be threads executing which need to finish what they were doing. Usually these threads should complete and you will see the active threads go away, but sometimes a thread is blocked (typically when trying to make a network connection somewhere without having proper timeouts set) and this thread may never complete, and therefore needs to be terminated.
The terminate option will issue an interrupt to the thread and then quarantine it, which takes it out of the pool for further execution. The thread may then complete in the background, or if it did not respond to the interrupt and is blocked then it may stay stuck in the background until the next restart of NiFi.
In the FetchHDFS case, assuming it was successfully fetching data, it was most likely in the middle of reading a file from HDFS and just needs a few minutes to complete and shouldn't need to use terminate. If it was never fetching data and was stuck connecting to HDFS then you would use terminate.
We're running a NiFi 1.7.1 server that picks up files via 3 GetFile processors and uploads them to the cloud.
When the server starts, it runs fine and chugs through any files that it finds. However, after running for a few days it seems to grind to a halt:
The GetFile processors all show 1 thread running, but they don't seem to be doing anything, even though there are files present in their source directories.
Nothing is waiting in any queues.
No messages appear in the logs.
The "top" command shows java using about 3% CPU and 21% mem. This is a 4-CPU server with 8GB of memory.
If I try to stop any processors via the web interface, it will become unresponsive. Upon reloading the page, I get the login screen, but after login it hangs on the loading animation, without showing the flow schema.
If I restart the NiFi service, it suddenly runs fine: it will pick up all the waiting files and not leave any threads hanging (according to the web interface). This will last another few days...
What is going on here? How can I resolve this?
Edit: The three GetFile processors each read from a different folder, but they all send their files to the same place. They are configured to pick up all files (filepattern .*), poll every 10 seconds, minimum file age 1 minute, don't keep the source file. I didn't touch the scheduling tab so it's just defaults.
I can't start or stop 2 processors: PutHiveQL and ListHDFS after I had to restart NiFi because of memory overload. Other processors react fine.
Even after 4 hours the processors don't react. I tried stopping the processors by stopping the Process Group but it failed.
Check the nifi-app.log file, you can also make a tail -f to the file to see what's going on in real time.
Also check this values in the processors:
Concurrent tasks.
Run schedule.
Run duration.
The default configuration is not so good and can give problems.