I am new to Nifi. I have define a processor group in Nifi and I have started the dataflow.
From the backend, how can I check the status of the processor group whether it is running or not?.
I tried
/bin/nifi.sh status
But it only gives the overall nifi status whether it is running or not
you can't see status of a process group because process groups do not have status
. Nifi just adds a added log like this
2021-04-09 13:26:44,766 INFO [main] o.a.nifi.groups.StandardProcessGroup StandardProcessGroup[identifier=feffff20-c806-305a-5d38-2b8def09bebe] added to StandardProcessGroup[identifier=1be26a7f-0175-1000-6d70-e5784c0dde33]
you can see the IDs in 'Operate' table on the right side of canvas.
On the other hand processor based logs can be seen
2021-04-09 13:40:59,290 INFO [Timer-Driven Process Thread-2] o.a.n.c.s.TimerDrivenSchedulingAgent Scheduled QuickFixInitiator[id=31ee54ea-5043- 3415-6f6e-4b8df429188f] to run with 3 threads
2021-04-09 13:45:31,164 INFO [Timer-Driven Process Thread-2] o.a.n.c.s.TimerDrivenSchedulingAgent Stopped scheduling QuickFixInitiator[id=31ee54ea-5043-3415-6f6e-4b8df429188f] to run
also via rest api
http://localhost:8080/nifi-api/processors/31ee54ea-5043-3415-6f6e-4b8df429188f
It is hard to follow nifi rest api doc All the UI requests go through rest api so the best way to learn it watching UI requests in developer console -> network tab
Related
Allocation status Screenshot
Hi team,
we can see about the allocation status of a job in nomad UI / command line.
but how nomad finds whether a job is running or it failed or it completed ?
Basically want to know how nomad figure out about allocation status.
Most probably the agent running on each node in the cluster will report it back to the nomad servers.
How the agent knows?
Most certainly there's a watch loop in agent's code for monitoring each allocation running on the node.
I'm implementing a service that would reject job requests from being processed if an existing job is running. Unfortunately, I'm not sure if there is a way to tell the difference between a job that is actively running and a job that ended due to an unexpected shutdown like turning Tomcat off. The statuses in the tables are the same with status = STARTED and exit_code = UNKNOWN.
Set<JobExecution> jobExecutions = jobExplorer.findRunningJobExecutions("MY_JOB");
Is there a way to tell the two apart or implementation that would change active job statuses to maybe ABANDONED?
There is indeed no way, by just looking at the database, to distinguish between a job that is effectively running and a job that has been abruptly killed (in both cases, the status is STARTED).
What you need to do is, in addition to checking status in the database, find a way to see if a job is effectively running. This really depends on how you run your jobs. For example, if you run your jobs in separate JVMs, you can write some code that checks if there is a JVM currently running your job. If you deploy your jobs to Kubernetes, you could ask Kubernetes if there is a pod currently running your job, etc.
However, if you can identify the execution that has been abruptly stopped for which the status has been stuck at STARTED (because Spring Batch did not have a chance to update its status to FAILED with a graceful shutdown), then you can update the status manually to ABONDONED and set its END_TIME to a non null value. This way, JobExplorer#findRunningExecutions will not return it anymore as a running execution.
We have Spring Cloud Data Flow local setup and the task is running the Spring Batch Job which reads from a Database and writes to AWS S3, all of this works fine.
When it comes to stopping the JOB, the task stops but resuming the job is not possible since the status is in "STARTED", this I think we can handle in code, by setting the batch status to 'STOPPED' when the stop is triggered, correct me if this can't be handled?
Also when trying to stop an individual slave task, there's an error:
2020-03-27 10:48:48.140 INFO 11258 --- [nio-9393-exec-7]
.s.c.d.s.s.i.DefaultTaskExecutionService : Task execution stop request
for id 192 for platform default has been submitted 2020-03-27
10:48:48.144 ERROR 11258 --- [nio-9393-exec-7]
o.s.c.d.s.c.RestControllerAdvice : Caught exception while
handling a request
java.lang.NullPointerException: null at
org.springframework.cloud.dataflow.server.service.impl.DefaultTaskExecutionService.cancelTaskExecution(DefaultTaskExecutionService.java:669)
~[spring-cloud-dataflow-server-core-2.3.0.RELEASE.jar!/:2.3.0.RELEASE]
at
org.springframework.cloud.dataflow.server.service.impl.DefaultTaskExecutionService.lambda$stopTaskExecution$0(DefaultTaskExecutionService.java:583)
~[spring-cloud-dataflow-server-core-2.3.0.RELEASE.jar!/:2.3.0.RELEASE]
How do we implement this is in distributed environment where we have a master server which can start the master on the master server and start the workers on respective slave servers?
1) You are correct you will need to change your status from STARTED to FAILED.
2) Since remote partitioning uses Spring Cloud Deployer (not Spring Cloud Data Flow) to launch the worker tasks, SCDF does not have a way to determine platform information to properly stop the the worker task. I've added GH Issue spring-cloud/spring-cloud-dataflow#3857 to resolve this problem.
3) The current implementation prevents a user from launching on multiple servers, rather lets the platform (Kubernetes, Cloud Foundry) distribute the worker tasks. You can implement your own deployer to add this feature.
I am new to Kylo.
I manually deployed Kylo on a test cluster of Hortonworks HDP 2.5 and have reused my Nifi instance prior to kylo.
I made a sample feed by following like ingestion tutorial (User Signups) and was successful.
However, when I drop sample data file in /var/dropzone/ the file is removed (assuming it is fetched and read by Nifi) but the operational dashboard does not show any any job running. No status against the feed job is populated.
I looked at the generated nifi process flow and there are two red processes and both are ReleaseHighWaterMark processes.
Also, Upon checking nifi-app.log I found following exception
2017-05-25 16:42:51,939 ERROR [Timer-Driven Process Thread-1] c.t.n.p.ProvenanceEventCollector ERROR PROCESSING EVENT! ProvenanceEventRecordDTO{eventId=759716, processorName=null, componentId=01a84157-0b38-14f2-d63d-c41fbd9c38a3, flowFile=ab93de46-e659-4c41-9812-94bbe2f90cfc, previous=null, eventType=CREATE, eventDetails=null, isEndOfJob=false, isBatch=true, isStream=false, feed=null}. ERROR: null
java.lang.NullPointerException: null
It seems there is a configuration issue and there is hardly any good troubleshooting guide available.
Any idea?
Please check that the KyloPrevenanceEventReportingTask is running in NiFi: http://kylo.readthedocs.io/en/latest/how-to-guides/NiFiKyloProvenanceReportingTask.html
If that doesn't resolve the issue, please post the stack trace that accompanies the error message.
Hi am working with OozieClient API.
I need to retrieve the task tracker logs for a particular workflow job using the OozieClient API. If not with OozieClient API any other way using a program is also fine. As of now with the OozieClient i am able to get the job log using client.getJobLog(), but i need task tracker logs and not job logs. Kindly help.
Try retrieving the yarn application Id from oozie using OozieClient API.
Once you have this ID you can make a call to history server using its rest api/or history server's client library, to fetch the Log dir path using "jobAttempts" api.
Now you can browse this directory using hadoop client.