I'm trying to create a site to site connection between 2 Nifi servers.
Nifi1 (Hostname wef2 - running on windows) need to deliver traffic to nifi2 (IP 10.168.233.168 - running on linux) - I defined the connection and all is green but on Nifi1 I'm getting the following error messages
*port ID 0176100c-8d25-196b-1f72-6befa5cab12a is the input port in NIFI2
**2020-12-01 01:54:13,508 INFO [pool-12-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile Repository with 2 records in 0 milliseconds
2020-12-01 01:54:20,677 INFO [Http Site-to-Site PeerSelector] o.apache.nifi.remote.client.PeerSelector Successfully refreshed peer status cache; remote group consists of 2 peers
2020-12-01 01:54:23,203 WARN [Timer-Driven Process Thread-3] o.a.nifi.remote.client.http.HttpClient Penalizing a peer Peer[url=http://localhost:8080/nifi-api] due to java.io.IOException: Unexpected response code: 404 errCode:Abort errMessage:Unable to find port with id '0176100c-8d25-196b-1f72-6befa5cab12a'.
2020-12-01 01:54:23,208 WARN [Timer-Driven Process Thread-3] o.a.nifi.remote.client.http.HttpClient Penalizing a peer Peer[url=http://wef2:8080/nifi-api] due to java.io.IOException: Unexpected response code: 404 errCode:Abort errMessage:Unable to find port with id '0176100c-8d25-196b-1f72-6befa5cab12a'.
2020-12-01 01:54:23,208 INFO [Timer-Driven Process Thread-3] o.a.nifi.remote.client.http.HttpClient Couldn't find a valid peer to communicate with.
2020-12-01 01:54:33,214 WARN [Timer-Driven Process Thread-2] o.a.nifi.remote.client.http.HttpClient Penalizing a peer Peer[url=http://wef2:8080/nifi-api] due to java.io.IOException: Unexpected response code: 404 errCode:Abort errMessage:Unable to find port with id '0176100c-8d25-196b-1f72-6befa5cab12a'.
2020-12-01 01:54:33,219 WARN [Timer-Driven Process Thread-2] o.a.nifi.remote.client.http.HttpClient Penalizing a peer Peer[url=http://localhost:8080/nifi-api] due to java.io.IOException: Unexpected response code: 404 errCode:Abort errMessage:Unable to find port with id '0176100c-8d25-196b-1f72-6befa5cab12a'.
2020-12-01 01:54:33,219 INFO [Timer-Driven Process Thread-2] o.a.nifi.remote.client.http.HttpClient Couldn't find a valid peer to communicate with.
2020-12-01 01:54:33,510 INFO [pool-12-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of FlowFile Repository
2020-12-01 01:54:33,510 INFO [pool-12-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile Repository with 2 records in 0 milliseconds**
Nifi 1
Nifi 2
remote process
remote process configuration
I have updated the nifi.proporties file with the following parameters
Under # Site to Site properties section update the following lines
nifi.remote.input.host =
nifi.remote.input.socket.port= 10000
Related
Im trying to configure Oracle cloud streaming kafka from topic with sasl_plaintext security using in nifi. While im gettting issue and stacktrace mentioned below; Kindly have a look and jass configuration also. Kindly anyone help me out from this issue;
And i have configured in bootstrap.conf file also;
I have tried nifi processor itself also; Here is the configuration img;
kafka version 2.0 and kafka nifi kakfa consumer_processor also consume_kafka_record_2.0 processor also. NiFi-1.9,2
java.arg.16=-Djava.security.auth.login.config=/home/opc/nifi-1.9.2/kafka_client_jaas.conf
kafka_client_jaas.conf
KafkaClient {
org.apache.kafka.common.security.plain.PlainLoginModule required
username="aaabbb"
password="cccdddde";
};
2020-10-30 05:41:18,638 WARN [Timer-Driven Process Thread-6] org.apache.kafka.clients.NetworkClient [Consumer clientId=consumer-2, groupId=devtes_grp] Connection to node -1 terminated during authentication. This may indicate that authentication failed due to invalid credentials.
2020-10-30 05:41:18,691 WARN [Timer-Driven Process Thread-6] org.apache.kafka.clients.NetworkClient [Consumer clientId=consumer-2, groupId=devtes_grp] Connection to node -1 terminated during authentication. This may indicate that authentication failed due to invalid credentials.
2020-10-30 05:41:18,719 INFO [Flow Service Tasks Thread-1] o.a.nifi.controller.StandardFlowService Saved flow controller org.apache.nifi.controller.FlowController#b4bca55 // Another save pending = false
2020-10-30 05:41:18,794 WARN [Timer-Driven Process Thread-6] org.apache.kafka.clients.NetworkClient [Consumer clientId=consumer-2, groupId=devtes_grp] Connection to node -1 terminated during authentication. This may indicate that authentication failed due to invalid credentials.
2020-10-30 05:41:19,047 WARN [Timer-Driven Process Thread-6] org.apache.kafka.clients.NetworkClient [Consumer clientId=consumer-2, groupId=devtes_grp] Connection to node -1 terminated during authentication. This may indicate that authentication failed due to invalid credentials.
2020-10-30 05:41:19,450 WARN [Timer-Driven Process Thread-6] org.apache.kafka.clients.NetworkClient [Consumer clientId=consumer-2, groupId=devtes_grp] Connection to node -1 terminated during authentication. This may indicate that authentication failed due to invalid credentials.
2020-10-30 05:41:20,255 WARN [Timer-Driven Process Thread-6] org.apache.kafka.clients.NetworkClient [Consumer clientId=consumer-2, groupId=devtes_grp] Connection to node -1 terminated during authentication. This may indicate that authentication failed due to invalid credentials.
2020-10-30 05:41:21,114 WARN [Timer-Driven Process Thread-6] org.apache.kafka.clients.NetworkClient [Consumer clientId=consumer-2, groupId=devtes_grp] Connection to node -1 terminated during authentication. This may indicate that authentication failed due to invalid credentials.
2020-10-30 05:41:21,920 WARN [Timer-Driven Process Thread-6] org.apache.kafka.clients.NetworkClient [Consumer clientId=consumer-2, groupId=devtes_grp] Connection to node -1 terminated during authentication. This may indicate that authentication failed due to invalid credentials.
2020-10-30 05:41:22,875 WARN [Timer-Driven Process Thread-6] org.apache.kafka.clients.NetworkClient [Consumer clientId=consumer-2, groupId=devtes_grp] Connection to node -1 terminated during authentication. This may indicate that authentication failed due to invalid credentials.
2020-10-30 05:41:23,189 INFO [NiFi Web Server-19] o.a.n.c.s.StandardProcessScheduler Stopping ConsumeKafkaRecord_2_0[id=7479eb76-0175-1000-bee8-c2013de52f23]
I can able to resolve this issue. I added standardsslsecurityservice with default java caerts and password and jks type and changed securityprotocol - SASL, security mechanishm - PLAIN
I got an error like this. The health test result for HDFS_CANARY_HEALTH has become bad: Canary test failed to write file in directory /tmp/.cloudera_health_monitoring_canary_files.
2019-07-14 00:02:13,312 INFO hive.metastore: Trying to connect to metastore with URI xxxx:abc
2019-07-14 00:02:13,322 INFO hive.metastore: Opened a connection to metastore, current connections: 1
2019-07-14 00:02:13,322 INFO hive.metastore: Connected to metastore.
2019-07-14 00:02:14,255 INFO hive.metastore: Closed a connection to metastore, current connections: 0
2019-07-14 00:02:21,444 INFO com.cloudera.cmon.tstore.leveldb.LDBTimeSeriesRollupManager: Finished rollup: duration=PT21.280S, numStreamsChecked=607071, numStreamsRolle
dUp=199173
2019-07-14 00:03:04,172 INFO org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x2659d409 connecting to ZooKeeper ensemble=___
2019-07-14 00:03:04,182 INFO org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Process identifier=ReplicationAdmin connecting to ZooKeeper ensemble=____
2019-07-14 00:03:04,198 INFO org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x16b95f7c324bcf3
2019-07-14 00:03:08,346 INFO org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x2fc2fce5 connecting to ZooKeeper ensemble=___
2019-07-14 00:03:08,354 INFO org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Process identifier=ReplicationAdmin connecting to ZooKeeper ensemble=___
2019-07-14 00:03:08,365 INFO org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x26b8ea7e7c6cfde
2019-07-14 00:03:11,897 INFO org.apache.hadoop.hdfs.DFSClient: Could not complete /tmp/.cloudera_health_monitoring_canary_files/.canary_file_2019_07_14-00_03_03 retrying...
2019-07-14 00:03:18,388 INFO org.apache.hadoop.hdfs.DFSClient: Could not complete /tmp/.cloudera_health_monitoring_canary_files/.canary_file_2019_07_14-00_03_03 retrying...
2019-07-14 00:03:18,562 ERROR com.cloudera.cmon.firehose.polling.hdfs.HdfsCanary: com.cloudera.cmon.firehose.polling.hdfs.HdfsCanary#5b4e2d83 for hdfs://hdfs-cluster_name Failed to write to /tmp/.cloudera_health_monitoring_canary_files/.canary_file_2019_07_14-00_03_03. Error: java.io.IOException: Unable to close file because the last block
BP-972893351-10.14.208.100-1438206436439:blk_4349006744_3275530539 does not have enough number of replicas.
My question is which "file" it referring to in ERROR: Unable to close file because the last block does not have enough number of replicas.
we are using spark application version 2.1 in out ambari cluster
ambari thrift servers isn't stable and restarted all times
from the log we can see that:
ERROR SparkContext: Error initializing SparkContext.
org.apache.spark.SparkException: Yarn application has already ended! It might have been killed or unable to launch application master.
we found the following link that described solution for this problem
https://markobigdata.com/2016/08/11/yarn-application-has-already-ended-it-might-have-been-killed-or-unable-to-launch-application-master/
but after we set the parameters as described in the article , the problem still exsist
please advice what is the solution for this?
full log:
tail -f spark-hive-org.apache.spark.sql.hive.thriftserver.HiveThriftServer2-1-master01.sys873dns.com.out
Spark Command: /usr/jdk64/jdk1.8.0_112/bin/java -Dhdp.version=2.6.0.3-8 -cp /usr/hdp/current/spark2-thriftserver/conf/:/usr/hdp/current/spark2-thriftserver/jars/*:/usr/hdp/current/hadoop-client/conf/ -Xmx10000m org.apache.spark.deploy.SparkSubmit --conf spark.driver.memory=50g --properties-file /usr/hdp/current/spark2-thriftserver/conf/spark-thrift-sparkconf.conf --class org.apache.spark.sql.hive.thriftserver.HiveThriftServer2 --name Thrift JDBC/ODBC Server --executor-cores 7 spark-internal
========================================
Warning: Master yarn-client is deprecated since 2.0. Please use master "yarn" with specified deploy mode instead.
18/02/08 09:38:07 ERROR SparkContext: Error initializing SparkContext.
org.apache.spark.SparkException: Yarn application has already ended! It might have been killed or unable to launch application master.
at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.waitForApplication(YarnClientSchedulerBackend.scala:85)
at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:62)
at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:156)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:509)
at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2320)
at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:868)
at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:860)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:860)
at org.apache.spark.sql.hive.thriftserver.SparkSQLEnv$.init(SparkSQLEnv.scala:47)
at org.apache.spark.sql.hive.thriftserver.HiveThriftServer2$.main(HiveThriftServer2.scala:81)
at org.apache.spark.sql.hive.thriftserver.HiveThriftServer2.main(HiveThriftServer2.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:745)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
18/02/08 09:38:07 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Attempted to request executors before the AM has registered!
18/02/08 09:38:07 ERROR Utils: Uncaught exception in thread main
java.lang.NullPointerException
I give also the yarn logs:
grep -i erro yarn-yarn-resourcemanager-master01.sys873dns.com.log
018-02-08 11:19:00,993 INFO zookeeper.ClientCnxn (ClientCnxn.java:logStartConnect(1019)) - Opening socket connection to server master01.sys873dns.com/23.1.29.61:2181. Will not attempt to authenticate using SASL (unknown error)
2018-02-08 11:19:15,767 ERROR resourcemanager.ResourceManager (LogAdapter.java:error(69)) - RECEIVED SIGNAL 15: SIGTERM
2018-02-08 11:19:27,281 INFO zookeeper.ClientCnxn (ClientCnxn.java:logStartConnect(1019)) - Opening socket connection to server master01.sys873dns.com/23.1.29.61:2181. Will not attempt to authenticate using SASL (unknown error)
2018-02-08 11:29:00,064 INFO zookeeper.ClientCnxn (ClientCnxn.java:logStartConnect(1019)) - Opening socket connection to server master01.sys873dns.com/23.1.29.61:2181. Will not attempt to authenticate using SASL (unknown error)
2018-02-08 11:29:01,839 INFO zookeeper.ClientCnxn (ClientCnxn.java:logStartConnect(1019)) - Opening socket connection to server master01.sys873dns.com/23.1.29.61:2181. Will not attempt to authenticate using SASL (unknown error)
2018-02-08 11:29:15,725 ERROR resourcemanager.ResourceManager (LogAdapter.java:error(69)) - RECEIVED SIGNAL 15: SIGTERM
2018-02-08 11:29:27,033 INFO zookeeper.ClientCnxn (ClientCnxn.java:logStartConnect(1019)) - Opening socket connection to server master03.sys873dns.com/23.1.29.63:2181. Will not attempt to authenticate using SASL (unknown error)
ons.YarnException: Unauthorized request to start container.
2018-02-08 12:56:11,144 INFO amlauncher.AMLauncher (AMLauncher.java:run(273)) - Error launching appattempt_1518089370033_0028_000008. Got exception: org.apache.hadoop.yarn.exceptions.YarnException: Unauthorized request to start container.
2018-02-08 12:59:39,822 INFO amlauncher.AMLauncher (AMLauncher.java:run(273)) - Error launching appattempt_1518089370033_0029_000002. Got exception: org.apache.hadoop.yarn.exceptions.YarnException: Unauthorized request to start container.
2018-02-08 13:00:01,671 INFO amlauncher.AMLauncher (AMLauncher.java:run(273)) - Error launching appattempt_1518089370033_0029_000004. Got exception: org.apache.hadoop.yarn.exceptions.YarnException: Unauthorized request to start container.
2018-02-08 13:00:18,062 INFO amlauncher.AMLauncher (AMLauncher.java:run(273)) - Error launching appattempt_1518089370033_0029_000006. Got exception: org.apache.hadoop.yarn.exceptions.YarnException: Unauthorized request to start container.
2018-02-08 13:00:20,245 INFO amlauncher.AMLauncher (AMLauncher.java:run(273)) - Error launching appattempt_1518089370033_0030_000003. Got exception: org.apache.hadoop.yarn.exceptions.YarnException: Unauthorized request to start container.
2018-02-08 13:00:42,100 INFO amlauncher.AMLauncher (AMLauncher.java:run(273)) - Error launching appattempt_1518089370033_0030_000006. Got exception: org.apache.hadoop.yarn.exceptions.YarnException: Unauthorized request to start container.
2018-02-08 13:00:56,310 INFO amlauncher.AMLauncher (AMLauncher.java:run(273)) - Error launching appattempt_1518089370033_0030_000008. Got exception: org.apache.hadoop.yarn.exceptions.YarnException: Unauthorized request to start container.
2018-02-08 13:00:58,511 INFO amlauncher.AMLauncher (AMLauncher.java:run(273)) - Error launching appattempt_1518089370033_0030_000010. Got exception: org.apache.hadoop.yarn.exceptions.YarnException: Unauthorized request to start container.
2018-02-08 13:00:58,537 INFO rmapp.RMAppImpl (RMAppImpl.java:transition(1063)) - Application application_1518089370033_0030 failed 10 times due to Error launching appattempt_1518089370033_0030_000010. Got exception: org.apache.hadoop.yarn.exceptions.YarnException: Unauthorized request to start container.
last log
2018-02-08 14:14:54,410 INFO rmapp.RMAppImpl (RMAppImpl.java:handle(778)) - application_1518089370033_0050 State change from FINAL_SAVING to FAILED
2018-02-08 14:14:54,410 INFO capacity.ParentQueue (ParentQueue.java:removeApplication(385)) - Application removed - appId: application_1518089370033_0050 user: hive leaf-queue of parent: root #applications: 1
2018-02-08 14:14:54,412 INFO integration.RMRegistryOperationsService (RMRegistryOperationsService.java:onApplicationCompleted(119)) - Application application_1518089370033_0050 completed, purging application-level records
2018-02-08 14:14:54,412 INFO integration.RMRegistryOperationsService (RMRegistryOperationsService.java:purgeRecordsAsync(198)) - records under / with ID application_1518089370033_0050 and policy application: {}
2018-02-08 14:14:55,393 INFO rmcontainer.RMContainerImpl (RMContainerImpl.java:handle(422)) - container_e09_1518089370033_0049_10_000001 Container Transitioned from RUNNING to COMPLETED
2018-02-08 14:14:55,393 INFO scheduler.SchedulerNode (SchedulerNode.java:releaseContainer(220)) - Released container container_e09_1518089370033_0049_10_000001 of capacity <memory:10240, vCores:1> on host worker02.sys768.com:45454, which currently has 0 containers, <memory:0, vCores:0> used and <memory:30720, vCores:6> available, release resources=true
2018-02-08 14:14:55,393 INFO attempt.RMAppAttemptImpl (RMAppAttemptImpl.java:rememberTargetTransitionsAndStoreState(1209)) - Updating application attempt appattempt_1518089370033_0049_000010 with final state: FAILED, and exit status: -1000
2018-02-08 14:14:55,398 INFO attempt.RMAppAttemptImpl (RMAppAttemptImpl.java:handle(809)) - appattempt_1518089370033_0049_000010 State change from LAUNCHED to FINAL_SAVING
2018-02-08 14:14:55,399 INFO integration.RMRegistryOperationsService (RMRegistryOperationsService.java:onContainerFinished(144)) - Container container_e09_1518089370033_0049_10_000001 finished, purging container-level records
2018-02-08 14:14:55,400 INFO integration.RMRegistryOperationsService (RMRegistryOperationsService.java:purgeRecordsAsync(198)) - records under / with ID container_e09_1518089370033_0049_10_000001 and policy container: {}
2018-02-08 14:14:55,408 INFO resourcemanager.ApplicationMasterService (ApplicationMasterService.java:unregisterAttempt(685)) - Unregistering app attempt : appattempt_1518089370033_0049_000010
2018-02-08 14:14:55,408 INFO security.AMRMTokenSecretManager (AMRMTokenSecretManager.java:applicationMasterFinished(124)) - Application finished, removing password for appattempt_1518089370033_0049_000010
2018-02-08 14:14:55,408 INFO attempt.RMAppAttemptImpl (RMAppAttemptImpl.java:handle(809)) - appattempt_1518089370033_0049_000010 State change from FINAL_SAVING to FAILED
2018-02-08 14:14:55,408 INFO rmapp.RMAppImpl (RMAppImpl.java:transition(1330)) - The number of failed attempts is 10. The max attempts is 10
2018-02-08 14:14:55,409 INFO rmapp.RMAppImpl (RMAppImpl.java:rememberTargetTransitionsAndStoreState(1123)) - Updating application application_1518089370033_0049 with final state: FAILED
2018-02-08 14:14:55,409 INFO rmapp.RMAppImpl (RMAppImpl.java:handle(778)) - application_1518089370033_0049 State change from ACCEPTED to FINAL_SAVING
2018-02-08 14:14:55,409 INFO recovery.RMStateStore (RMStateStore.java:transition(228)) - Updating info for app: application_1518089370033_0049
2018-02-08 14:14:55,409 INFO capacity.CapacityScheduler (CapacityScheduler.java:doneApplicationAttempt(811)) - Application Attempt appattempt_1518089370033_0049_000010 is done. finalState=FAILED
2018-02-08 14:14:55,409 INFO scheduler.AppSchedulingInfo (AppSchedulingInfo.java:clearRequests(124)) - Application application_1518089370033_0049 requests cleared
2018-02-08 14:14:55,410 INFO capacity.LeafQueue (LeafQueue.java:removeApplicationAttempt(795)) - Application removed - appId: application_1518089370033_0049 user: hive queue: default #user-pending-applications: 0 #user-active-applications: 0 #queue-pending-applications: 0 #queue-active-applications: 0
2018-02-08 14:14:55,417 INFO rmapp.RMAppImpl (RMAppImpl.java:transition(1063)) - Application application_1518089370033_0049 failed 10 times due to AM Container for appattempt_1518089370033_0049_000010 exited with exitCode: -1000
For more detailed output, check the application tracking page: http://master02.sys768.com:8088/cluster/app/application_1518089370033_0049 Then click on links to logs of each attempt.
Diagnostics: org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block: BP-1212891131-25.1.53.61-1518077044052:blk_1073741833_1009 file=/hdp/apps/2.6.0.3-8/spark2/spark2-hdp-yarn-archive.tar.gz
Failing this attempt. Failing the application.
2018-02-08 14:14:55,418 INFO rmapp.RMAppImpl (RMAppImpl.java:handle(778)) - application_1518089370033_0049 State change from FINAL_SAVING to FAILED
2018-02-08 14:14:55,418 INFO capacity.ParentQueue (ParentQueue.java:removeApplication(385)) - Application removed - appId: application_1518089370033_0049 user: hive leaf-queue of parent: root #applications: 0
2018-02-08 14:14:55,419 INFO integration.RMRegistryOperationsService (RMRegistryOperationsService.java:onApplicationCompleted(119)) - Application application_1518089370033_0049 completed, purging application-level records
2018-02-08 14:14:55,419 INFO integration.RMRegistryOperationsService (RMRegistryOperationsService.java:purgeRecordsAsync(198)) - records under / with ID application_1518089370033_0049 and policy application: {}
[root#master02 yarn]#
I am facing an issue when i try to ingest json data through data ingest and standard ingest template.
After initialize feed it is going into failure feed. I am able to see hive tables with json attributes created but not the folder structure in hdfs.
Please find the logs and screen shots of kylo attached.enter image description here
2017-09-03 08:21:33,205 INFO [Timer-Driven Process Thread-1] 2017-09-07 00:00:01,724 ERROR [Timer-Driven Process Thread-8] c.t.n.p.r.KyloProvenanceEventReportingTask KyloProvenanceEventReportingTask[id=17ff8df2-015d-1000-7741-d474a8a3404d] Error checking to see if Kylo is available. Please ensure Kylo is up and running.
2017-09-07 00:00:01,725 INFO [Timer-Driven Process Thread-8] c.t.n.p.r.KyloProvenanceEventReportingTask KyloProvenanceEventReportingTask[id=17ff8df2-015d-1000-7741-d474a8a3404d] Kylo is not available to process requests yet. This task will exit and wait for its next schedule interval.
2017-09-07 00:00:01,725 INFO [Timer-Driven Process Thread-8] c.t.n.p.r.KyloProvenanceEventReportingTask KyloProvenanceEventReportingTask[id=17ff8df2-015d-1000-7741-d474a8a3404d] Reporting Task Aborted. Last Event Recorded was: 6269977
2017-09-07 00:00:06,730 ERROR [Timer-Driven Process Thread-3] c.t.n.p.r.KyloProvenanceEventReportingTask KyloProvenanceEventReportingTask[id=17ff8df2-015d-1000-7741-d474a8a3404d] Error checking to see if Kylo is available. Please ensure Kylo is up and running.
2017-09-07 00:00:06,731 INFO [Timer-Driven Process Thread-3] c.t.n.p.r.KyloProvenanceEventReportingTask KyloProvenanceEventReportingTask[id=17ff8df2-015d-1000-7741-d474a8a3404d] Kylo is not available to process requests yet. This task will exit and wait for its next schedule interval.
2017-09-07 00:00:06,731 INFO [Timer-Driven Process Thread-3] c.t.n.p.r.KyloProvenanceEventReportingTask KyloProvenanceEventReportingTask[id=17ff8df2-015d-1000-7741-d474a8a3404d] Reporting Task Aborted. Last Event Recorded was: 6269977
2017-09-07 00:00:07,004 INFO [Write-Ahead Local State Provider Maintenance] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog#8f5565c checkpointed with 14020 Records and 0 Swap Files in 207 milliseconds (Stop-the-world time = 2 milliseconds, Clear Edit Logs time = 0 millis), max Transaction ID 76071
2017-09-07 00:00:11,735 ERROR [Timer-Driven Process Thread-6] c.t.n.p.r.KyloProvenanceEventReportingTask KyloProvenanceEventReportingTask[id=17ff8df2-015d-1000-7741-d474a8a3404d] Error checking to see if Kylo is available. Please ensure Kylo is up and running.
2017-09-07 00:00:11,736 INFO [Timer-Driven Process Thread-6] c.t.n.p.r.KyloProvenanceEventReportingTask KyloProvenanceEventReportingTask[id=17ff8df2-015d-1000-7741-d474a8a3404d] Kylo is not available to process requests yet. This task will exit and wait for its next schedule interval.
2017-09-07 00:00:11,736 INFO [Timer-Driven Process Thread-6] c.t.n.p.r.KyloProvenanceEventReportingTask KyloProvenanceEventReportingTask[id=17ff8df2-015d-1000-7741-d474a8a3404d] Reporting Task Aborted. Last Event Recorded was: 6269977
2017-09-07 00:00:16,742 ERROR [Timer-Driven Process Thread-1] c.t.n.p.r.KyloProvenanceEventReportingTask KyloProvenanceEventReportingTask[id=17ff8df2-015d-1000-7741-d474a8a3404d] Error checking to see if Kylo is available. Please ensure Kylo is up and running.
2017-09-07 00:00:16,742 INFO [Timer-Driven Process Thread-1] c.t.n.p.r.KyloProvenanceEventReportingTask KyloProvenanceEventReportingTask[id=17ff8df2-015d-1000-7741-d474a8a3404d] Kylo is not available to process requests yet. This task will exit and wait for its next schedule interval.
2017-09-07 00:00:16,742 INFO [Timer-Driven Process Thread-1] c.t.n.p.r.KyloProvenanceEventReportingTask KyloProvenanceEventReportingTask[id=17ff8df2-015d-1000-7741-d474a8a3404d] Reporting Task Aborted. Last Event Recorded was: 6269977
2017-09-07 00:00:21,749 ERROR [Timer-Driven Process Thread-5] c.t.n.p.r.KyloProvenanceEventReportingTask KyloProvenanceEventReportingTask[id=17ff8df2-015d-1000-7741-d474a8a3404d] Error checking to see if Kylo is available. Please ensure Kylo is up and running.
2017-09-07 00:00:21,749 INFO [Timer-Driven Process Thread-5] c.t.n.p.r.KyloProvenanceEventReportingTask KyloProvenanceEventReportingTask[id=17ff8df2-015d-1000-7741-d474a8a3404d] Kylo is not available to process requests yet. This task will exit and wait for its next schedule interval.
2017-09-07 00:00:21,749 INFO [Timer-Driven Process Thread-5] c.t.n.p.r.KyloProvenanceEventReportingTask KyloProvenanceEventReportingTask[id=17ff8df2-015d-1000-7741-d474a8a3404d] Reporting Task Aborted. Last Event Recorded was: 6269977
2017-09-07 00:00:25,600 ERROR [Timer-Driven Process Thread-7] c.t.nifi.v2.init.InitializeFeed InitializeFeed[id=2b1f1ef0-4e16-1bd6-f57d-9191210e2a13] InitializeFeed[id=2b1f1ef0-4e16-1bd6-f57d-9191210e2a13] failed to process due to org.springframework.web.client.HttpServerErrorException: 500 Internal Server Error; rolling back session: org.springframework.web.client.HttpServerErrorException: 500 Internal Server Error
2017-09-07 00:00:25,602 ERROR [Timer-Driven Process Thread-7] c.t.nifi.v2.init.InitializeFeed
org.springframework.web.client.HttpServerErrorException: 500 Internal Server Error
at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:94) ~[spring-web-4.2.4.RELEASE.jar:4.2.4.RELEASE]
at org.springframework.web.client.RestTemplate.handleResponse(RestTemplate.java:641) ~[spring-web-4.2.4.RELEASE.jar:4.2.4.RELEASE]
at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:597) ~[spring-web-4.2.4.RELEASE.jar:4.2.4.RELEASE]
at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:572) ~[spring-web-4.2.4.RELEASE.jar:4.2.4.RELEASE]
at org.springframework.web.client.RestTemplate.getForObject(RestTemplate.java:280) ~[spring-web-4.2.4.RELEASE.jar:4.2.4.RELEASE]
at com.thinkbiganalytics.metadata.rest.client.MetadataClient.get(MetadataClient.java:700) ~[na:na]
at com.thinkbiganalytics.metadata.rest.client.MetadataClient.get(MetadataClient.java:692) ~[na:na]
at com.thinkbiganalytics.metadata.rest.client.MetadataClient.lambda$getCurrentInitStatus$1(MetadataClient.java:263) ~[na:na]
at com.thinkbiganalytics.metadata.rest.client.MetadataClient.nullable(MetadataClient.java:577) ~[na:na]
at com.thinkbiganalytics.metadata.rest.client.MetadataClient.getCurrentInitStatus(MetadataClient.java:263) ~[na:na]
at com.thinkbiganalytics.nifi.v2.core.metadata.MetadataClientRecorder.getInitializationStatus(MetadataClientRecorder.java:217) ~[na:na]
at com.thinkbiganalytics.nifi.v2.init.InitializeFeed.onTrigger(InitializeFeed.java:116) ~[kylo-nifi-core-processors-0.8.1.jar:na]
at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) ~[nifi-api-1.0.0.jar:1.0.0]
at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1064) [nifi-framework-core-1.0.0.jar:1.0.0]
While running a topology in storm we are getting error like this,
8983 [Thread-6] INFO com.netflix.curator.framework.imps.CuratorFrameworkImpl -
Starting
9144 [main] INFO **backtype.storm.daemon.nimbus** - Shutting down master
9199 [Thread-6-EventThread] INFO backtype.storm.zookeeper - Zookeeper state upd
ate: :connected:none
9241 [main] INFO backtype.storm.daemon.nimbus - Shut down master
9273 [Thread-6] INFO com.netflix.curator.framework.imps.CuratorFrameworkImpl -
Starting
9306 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2000] WARN org.apache.zookeeper.serv
er.NIOServerCnxn - EndOfStreamException: Unable to read additional data from cli
ent sessionid 0x143af55728d0003, likely client has closed socket
9354 [main] INFO backtype.storm.daemon.supervisor - Shutting down c094c3b1-a378
-4c4f-af35-9278647c217a:4beddc09-4675-4fb9-8bdc-9cf5013ce9ca
9358 [main] INFO backtype.storm.daemon.supervisor - Shut down c094c3b1-a378-4c4
f-af35-9278647c217a:4beddc09-4675-4fb9-8bdc-9cf5013ce9ca
9361 [main] INFO **backtype.storm.daemon.superviso**r - Shutting down supervisor c0
94c3b1-a378-4c4f-af35-9278647c217a
9364 [Thread-5] INFO **backtype.storm.event** - Event manager interrupted
9369 [Thread-6] INFO backtype.storm.event - Event manager interrupted
9425 [main] INFO **backtype.storm.daemon.supervisor** - Shutting down supervisor 38
6d8d71-c9b5-4b51-bd6e-f9f605034ea0
9428 [Thread-8] INFO backtype.storm.event - Event manager interrupted
9429 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2000] WARN org.apache.zookeeper.serv
er.NIOServerCnxn - EndOfStreamException: Unable to read additional data from cli
ent sessionid 0x143af55728d0007, likely client has closed socket
9429 [Thread-9] INFO backtype.storm.event - Event manager interrupted
9473 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2000] WARN org.apache.zookeeper.serv
er.NIOServerCnxn - EndOfStreamException: Unable to read additional data from cli
ent sessionid 0x143af55728d0009, likely client has closed socket
9476 [main] INFO backtype.storm.testing - Shutting down in process zookeeper
9503 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2000] WARN org.apache.zookeeper.serv
er.NIOServerCnxn - Ignoring exception
**java.nio.channels.ClosedChannelException**: null
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.jav
a:211) ~[na:1.7.0_03]
at org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.j
ava:242) ~[zookeeper-3.3.3.jar:3.3.3-1073969]
9510 [main] INFO **backtype.storm.testing** - Done shutting down in process zookeep
er
9513 [main] INFO backtype.storm.testing - Deleting temporary path C:\Users\sowm
iya\AppData\Local\Temp\c9b1bc1a-a950-4098-af77-f81a4d2b112f
9520 [main] INFO backtype.storm.testing - Deleting temporary path C:\Users\sowm
iya\AppData\Local\Temp\7e75c468-18ea-4787-a4ac-496fb108db71
9527 [main] INFO backtype.storm.testing - Unable to delete file: C:\Users\sowmi
ya\AppData\Local\Temp\7e75c468-18ea-4787-a4ac-496fb108db71\version-2\log.1
9529 [main] INFO backtype.storm.testing - Deleting temporary path C:\Users\sowm
iya\AppData\Local\Temp\fa7b3c9b-ac93-4090-b9e2-63f10019e61f
9543 [main] INFO backtype.storm.testing - Deleting temporary path C:\Users\sowm
iya\AppData\Local\Temp\55f1fd11-508e-43bb-b340-0d9b79f3af33
9579 [Thread-6-EventThread] INFO com.netflix.curator.framework.state.Connection
StateManager - State change: SUSPENDED
9580 [ConnectionStateManager-0] WARN com.netflix.curator.framework.state.Connec
tionStateManager - There are no ConnectionStateListeners registered.
9583 [Thread-6-EventThread] WARN backtype.storm.cluster - Received event :disco
nnected::none: with disconnected Zookeeper.
11232 [Thread-6-SendThread(localhost:2000)] WARN org.apache.zookeeper.ClientCnx
n - Session 0x143af55728d000b for server null, unexpected error, closing socket
connection and attempting reconnect
**java.net.ConnectException: Connection refused: no further information**
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[na:1.7.0_0
3]
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:701
) ~[na:1.7.0_03]
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1119)
~[zookeeper-3.3.3.jar:3.3.3-1073969]
13992 [Thread-6-SendThread(localhost:2000)] WARN org.apache.zookeeper.ClientCnx
n - Session 0x143af55728d000b for server null, unexpected error, closing socket
connection and attempting reconnect
java.net.ConnectException: Connection refused: no further information
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[na:1.7.0_0
3]
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:701
) ~[na:1.7.0_03]
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1119)
Whwn we are trying to run the topology jar file all the operation like nimbus,zookeeper and supervisor process going to dead.please help us to know why this is happened.
Please help us to rectify this error and help to proceed further.
Thank you,
Sowmiya
Priya
This looks like a zookeeper issue. It looks like your processes are not being able to connect to zookeeper. Can't say more without more information.