I am new to Kylo.
I manually deployed Kylo on a test cluster of Hortonworks HDP 2.5 and have reused my Nifi instance prior to kylo.
I made a sample feed by following like ingestion tutorial (User Signups) and was successful.
However, when I drop sample data file in /var/dropzone/ the file is removed (assuming it is fetched and read by Nifi) but the operational dashboard does not show any any job running. No status against the feed job is populated.
I looked at the generated nifi process flow and there are two red processes and both are ReleaseHighWaterMark processes.
Also, Upon checking nifi-app.log I found following exception
2017-05-25 16:42:51,939 ERROR [Timer-Driven Process Thread-1] c.t.n.p.ProvenanceEventCollector ERROR PROCESSING EVENT! ProvenanceEventRecordDTO{eventId=759716, processorName=null, componentId=01a84157-0b38-14f2-d63d-c41fbd9c38a3, flowFile=ab93de46-e659-4c41-9812-94bbe2f90cfc, previous=null, eventType=CREATE, eventDetails=null, isEndOfJob=false, isBatch=true, isStream=false, feed=null}. ERROR: null
java.lang.NullPointerException: null
It seems there is a configuration issue and there is hardly any good troubleshooting guide available.
Any idea?
Please check that the KyloPrevenanceEventReportingTask is running in NiFi: http://kylo.readthedocs.io/en/latest/how-to-guides/NiFiKyloProvenanceReportingTask.html
If that doesn't resolve the issue, please post the stack trace that accompanies the error message.
Related
I have a problem on nifi at each step of my flow I have this error:
11:44:07 CESTERROR
Failed to index Provenance Events. See logs for more information.
i find in nifi-app.log
however there is no error and my nifi flow is working well
I have no error on the processor but the error is displayed at the top right
the nifi.properties looks good,
I am new to Nifi. I have define a processor group in Nifi and I have started the dataflow.
From the backend, how can I check the status of the processor group whether it is running or not?.
I tried
/bin/nifi.sh status
But it only gives the overall nifi status whether it is running or not
you can't see status of a process group because process groups do not have status
. Nifi just adds a added log like this
2021-04-09 13:26:44,766 INFO [main] o.a.nifi.groups.StandardProcessGroup StandardProcessGroup[identifier=feffff20-c806-305a-5d38-2b8def09bebe] added to StandardProcessGroup[identifier=1be26a7f-0175-1000-6d70-e5784c0dde33]
you can see the IDs in 'Operate' table on the right side of canvas.
On the other hand processor based logs can be seen
2021-04-09 13:40:59,290 INFO [Timer-Driven Process Thread-2] o.a.n.c.s.TimerDrivenSchedulingAgent Scheduled QuickFixInitiator[id=31ee54ea-5043- 3415-6f6e-4b8df429188f] to run with 3 threads
2021-04-09 13:45:31,164 INFO [Timer-Driven Process Thread-2] o.a.n.c.s.TimerDrivenSchedulingAgent Stopped scheduling QuickFixInitiator[id=31ee54ea-5043-3415-6f6e-4b8df429188f] to run
also via rest api
http://localhost:8080/nifi-api/processors/31ee54ea-5043-3415-6f6e-4b8df429188f
It is hard to follow nifi rest api doc All the UI requests go through rest api so the best way to learn it watching UI requests in developer console -> network tab
I am stuck with a strange problem. Pentaho Data integration provides sample Job "word Count Job" in order to understand MapReduce Jobs.
I am learning MapReduce and I am really lost with one strange error.
Error is :
"Caused by: java.io.IOException: Cannot initialize Cluster.
Please check your configuration for mapreduce.framework.name
and the correspond server addresses."
I have tried everything in my repertoire to resolve from chaging "plugin-properties" file in Pentaho data integration to re-installing Pentaho SHIM but to no avail.
As per the job's flow, file is correctly getting transferred to HDFS server from my local(where pentaho data integration is running) but the moment MapReduce job starts it throws error.
Finally cracked it. The error was because in core-site.xml file, "IP-Address" of cluster was mentioned by me where as "hostname" was recognized by the cluster. Hence, because of this ambiguity this error was happening.
Hurray!!
I am running a spark job via pyspark, which consistently returns an error:
Diagnostics: org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block: BP-908041201-10.122.103.38-1485808002236:blk_1073741831_1007 file=/hdp/apps/2.5.3.0-37/spark/spark-hdp-assembly.jar
The error is always on the same block, namely BP-908041201-10.122.103.38-1485808002236:blk_1073741831_1007.
When i look in the hadoop tracking url, the message reads:
Application application_1505726128034_2371 failed 2 times due to AM Container
for appattempt_1505726128034_2371_000002 exited with exitCode: -1000
I can only assume from this that there is some corrupted data? How can i view the data / block via hadoop command line and see exactly which data is on this potentially corrupted block.
Unfortunately there doesnt appear to be more detailed logs on the specific nodes of failure when looking in the web based logs.
Also - is there a way in pyspark to ignore any 'corrupted' blocks and simply ignore any files/blocks it cannot fully read?
Thanks
I have a topology running on a Storm cluster with 3 supervisor nodes(32GRAM each node). In the first several days, the topology goes well, everything is ok. But the following error always occurred and the topology gone down after several days running:
java.lang.RuntimeException: java.lang.RuntimeException: java.lang.RuntimeException
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /brokers/topics/TOPICNAME/partitions at storm.kafka.ZkCoordinator.refresh
The topology uses a spout to consume messages from a remote Kafka service which sits on an remote server and this server is also the zookeeper service on.
I guess the reason for this exception is that the zookeeper server is instability, OR the network connection is unstable.
I have no permission to do anything with the remote kafka/zookeeper server, So I need a solution by my side to keep the topology running stably. Is there anyway to let the topology runs stably OR anyway to skip the exception while it comes out?
Or is there anyway to resubmit topology automatically?
Thank you very much!
The first thing you should have done is to google for what causes the connection loss error.
Then go to storm's log files and view which line of code is causing the error.
The right way to do things is to find out what is causing the error.
However, if you want the quicker temporary solution, then use Storm's REST API to kill the topology. Then you can use a normal Java program or a script in any language to re-launch the topology from the commandline.