Storm Cluster Mode Error - cluster-computing
Firstly I am a beginner in Storm and so i want your tolerance for my incomplete report of my question. I have completed the project in local mode and it runs smoothly, without any problems.
I tried to run it in the setup-ed cluster of my university. And i see in the log at cluster's UI that never start running cause of an error. The same error in all the bolts and spouts of my topology. I attach the log with the error of one of a spouts.
I know that my description is inadequate but if you tell my what else is useful to inform you about i will add it to the post.
Thank you
2015-07-15 15:34:48 o.a.s.z.ZooKeeper [INFO] Client environment:zookeeper.version=3.4.6-2--1, built on 03/31/2015 19:31 GMT
2015-07-15 15:34:48 o.a.s.z.ZooKeeper [INFO] Client environment:host.name=clu18.softnet.tuc.gr
2015-07-15 15:34:48 o.a.s.z.ZooKeeper [INFO] Client environment:java.version=1.7.0_80
2015-07-15 15:34:48 o.a.s.z.ZooKeeper [INFO] Client environment:java.vendor=Oracle Corporation
2015-07-15 15:34:48 o.a.s.z.ZooKeeper [INFO] Client environment:java.home=/usr/lib/jvm/java-7-oracle/jre
2015-07-15 15:34:48 o.a.s.z.ZooKeeper [INFO] Client environment:java.class.path=/usr/hdp/2.2.4.2-2/storm/lib/ranger-plugins-common-0.4.0.2.2.4.2-2.jar:/usr/hdp/2.2.4.2-2/storm/lib/clj-stacktrace-0.2.4.jar:/usr/hdp/2.2.4.2-2/storm/lib/oncrpc-1.0.7.jar:/usr/hdp/2.2.4.2-2/storm/lib/chill-java-0.3.5.jar:/usr/hdp/2.2.4.2-2/storm/lib/reflectasm-1.07-shaded.jar:/usr/hdp/2.2.4.2-2/storm/lib/logback-classic-1.0.6.jar:/usr/hdp/2.2.4.2-2/storm/lib/jetty-http-7.6.13.v20130916.jar:/usr/hdp/2.2.4.2-2/storm/lib/snakeyaml-1.11.jar:/usr/hdp/2.2.4.2-2/storm/lib/hadoop-common-2.6.0.2.2.4.2-2.jar:/usr/hdp/2.2.4.2-2/storm/lib/eclipselink-2.5.2-M1.jar:/usr/hdp/2.2.4.2-2/storm/lib/slf4j-api-1.6.5.jar:/usr/hdp/2.2.4.2-2/storm/lib/servlet-api-2.5.jar:/usr/hdp/2.2.4.2-2/storm/lib/tools.cli-0.2.4.jar:/usr/hdp/2.2.4.2-2/storm/lib/joda-time-2.0.jar:/usr/hdp/2.2.4.2-2/storm/lib/java.classpath-0.2.2.jar:/usr/hdp/2.2.4.2-2/storm/lib/commons-codec-1.6.jar:/usr/hdp/2.2.4.2-2/storm/lib/objenesis-1.2.jar:/usr/hdp/2.2.4.2-2/storm/lib/jetty-io-7.6.13.v20130916.jar:/usr/hdp/2.2.4.2-2/storm/lib/hadoop-auth-2.4.0.jar:/usr/hdp/2.2.4.2-2/storm/lib/compojure-1.1.3.jar:/usr/hdp/2.2.4.2-2/storm/lib/ring-jetty-adapter-1.3.0.jar:/usr/hdp/2.2.4.2-2/storm/lib/jetty-security-7.6.13.v20130916.jar:/usr/hdp/2.2.4.2-2/storm/lib/ranger-plugins-impl-0.4.0.2.2.4.2-2.jar:/usr/hdp/2.2.4.2-2/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/hdp/2.2.4.2-2/storm/lib/jetty-util-7.6.13.v20130916.jar:/usr/hdp/2.2.4.2-2/storm/lib/clojure-1.5.1.jar:/usr/hdp/2.2.4.2-2/storm/lib/minlog-1.2.jar:/usr/hdp/2.2.4.2-2/storm/lib/ns-tracker-0.2.2.jar:/usr/hdp/2.2.4.2-2/storm/lib/jersey-bundle-1.17.1.jar:/usr/hdp/2.2.4.2-2/storm/lib/clout-1.0.1.jar:/usr/hdp/2.2.4.2-2/storm/lib/disruptor-2.10.1.jar:/usr/hdp/2.2.4.2-2/storm/lib/tools.logging-0.2.3.jar:/usr/hdp/2.2.4.2-2/storm/lib/javax.persistence-2.1.0.jar:/usr/hdp/2.2.4.2-2/storm/lib/jetty-continuation-7.6.13.v20130916.jar:/usr/hdp/2.2.4.2-2/storm/lib/jetty-servlets-7.6.13.v20130916.jar:/usr/hdp/2.2.4.2-2/storm/lib/ring-anti-forgery-1.0.0.jar:/usr/hdp/2.2.4.2-2/storm/lib/hiccup-0.3.6.jar:/usr/hdp/2.2.4.2-2/storm/lib/commons-lang-2.5.jar:/usr/hdp/2.2.4.2-2/storm/lib/crypto-equality-1.0.0.jar:/usr/hdp/2.2.4.2-2/storm/lib/jetty-server-7.6.13.v20130916.jar:/usr/hdp/2.2.4.2-2/storm/lib/gmetric4j-1.0.7.jar:/usr/hdp/2.2.4.2-2/storm/lib/storm-core-0.9.3.2.2.4.2-2.jar:/usr/hdp/2.2.4.2-2/storm/lib/ring-core-1.1.5.jar:/usr/hdp/2.2.4.2-2/storm/lib/commons-exec-1.1.jar:/usr/hdp/2.2.4.2-2/storm/lib/logback-core-1.0.6.jar:/usr/hdp/2.2.4.2-2/storm/lib/carbonite-1.4.0.jar:/usr/hdp/2.2.4.2-2/storm/lib/math.numeric-tower-0.0.1.jar:/usr/hdp/2.2.4.2-2/storm/lib/commons-fileupload-1.2.1.jar:/usr/hdp/2.2.4.2-2/storm/lib/ring-servlet-1.3.0.jar:/usr/hdp/2.2.4.2-2/storm/lib/ranger-plugins-cred-0.4.0.2.2.4.2-2.jar:/usr/hdp/2.2.4.2-2/storm/lib/commons-io-2.4.jar:/usr/hdp/2.2.4.2-2/storm/lib/ranger-storm-plugin-0.4.0.2.2.4.2-2.jar:/usr/hdp/2.2.4.2-2/storm/lib/gson-2.2.4.jar:/usr/hdp/2.2.4.2-2/storm/lib/tools.macro-0.1.0.jar:/usr/hdp/2.2.4.2-2/storm/lib/jetty-servlet-7.6.13.v20130916.jar:/usr/hdp/2.2.4.2-2/storm/lib/kryo-2.21.jar:/usr/hdp/2.2.4.2-2/storm/lib/commons-logging-1.2.jar:/usr/hdp/2.2.4.2-2/storm/lib/asm-4.0.jar:/usr/hdp/2.2.4.2-2/storm/lib/ranger-plugins-audit-0.4.0.2.2.4.2-2.jar:/usr/hdp/2.2.4.2-2/storm/lib/jgrapht-core-0.9.0.jar:/usr/hdp/2.2.4.2-2/storm/lib/tools.namespace-0.2.4.jar:/usr/hdp/2.2.4.2-2/storm/lib/commons-configuration-1.10.jar:/usr/hdp/2.2.4.2-2/storm/lib/core.incubator-0.1.0.jar:/usr/hdp/2.2.4.2-2/storm/lib/crypto-random-1.2.0.jar:/usr/hdp/2.2.4.2-2/storm/lib/commons-collections-3.2.1.jar:/usr/hdp/2.2.4.2-2/storm/lib/jetty-client-7.6.13.v20130916.jar:/usr/hdp/2.2.4.2-2/storm/lib/guava-11.0.2.jar:/usr/hdp/2.2.4.2-2/storm/lib/javax.servlet-2.5.0.v201103041518.jar:/usr/hdp/2.2.4.2-2/storm/lib/json-simple-1.1.jar:/usr/hdp/2.2.4.2-2/storm/lib/clj-time-0.4.1.jar:/usr/hdp/2.2.4.2-2/storm/lib/ring-devel-1.3.0.jar:/usr/hdp/2.2.4.2-2/storm/conf:/usr/hdp/current/storm-supervisor/contrib/storm-jmxetric/lib/jmxetric-1.0.4.jar:/hadoop/storm/supervisor/stormdist/aek-16-1436963685/stormjar.jar:/usr/hdp/current/storm-client/contrib/storm-jmxetric/lib/jmxetric-1.0.4.jar
2015-07-15 15:34:48 o.a.s.z.ZooKeeper [INFO] Client environment:java.library.path=/hadoop/storm/supervisor/stormdist/aek-16-1436963685/resources/Linux-amd64:/hadoop/storm/supervisor/stormdist/aek-16-1436963685/resources:/usr/local/lib:/opt/local/lib:/usr/lib:/usr/hdp/current/storm-client/lib
2015-07-15 15:34:48 o.a.s.z.ZooKeeper [INFO] Client environment:java.io.tmpdir=/tmp
2015-07-15 15:34:48 o.a.s.z.ZooKeeper [INFO] Client environment:java.compiler=<NA>
2015-07-15 15:34:48 o.a.s.z.ZooKeeper [INFO] Client environment:os.name=Linux
2015-07-15 15:34:48 o.a.s.z.ZooKeeper [INFO] Client environment:os.arch=amd64
2015-07-15 15:34:48 o.a.s.z.ZooKeeper [INFO] Client environment:os.version=3.2.0-70-generic
2015-07-15 15:34:48 o.a.s.z.ZooKeeper [INFO] Client environment:user.name=storm
2015-07-15 15:34:48 o.a.s.z.ZooKeeper [INFO] Client environment:user.home=/home/storm
2015-07-15 15:34:48 o.a.s.z.ZooKeeper [INFO] Client environment:user.dir=/home/storm
2015-07-15 15:34:48 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:zookeeper.version=3.4.6-2--1, built on 03/31/2015 19:31 GMT
2015-07-15 15:34:48 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:host.name=clu18.softnet.tuc.gr
2015-07-15 15:34:48 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:java.version=1.7.0_80
2015-07-15 15:34:48 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:java.vendor=Oracle Corporation
2015-07-15 15:34:48 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:java.home=/usr/lib/jvm/java-7-oracle/jre
2015-07-15 15:34:48 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:java.class.path=/usr/hdp/2.2.4.2-2/storm/lib/ranger-plugins-common-0.4.0.2.2.4.2-2.jar:/usr/hdp/2.2.4.2-2/storm/lib/clj-stacktrace-0.2.4.jar:/usr/hdp/2.2.4.2-2/storm/lib/oncrpc-1.0.7.jar:/usr/hdp/2.2.4.2-2/storm/lib/chill-java-0.3.5.jar:/usr/hdp/2.2.4.2-2/storm/lib/reflectasm-1.07-shaded.jar:/usr/hdp/2.2.4.2-2/storm/lib/logback-classic-1.0.6.jar:/usr/hdp/2.2.4.2-2/storm/lib/jetty-http-7.6.13.v20130916.jar:/usr/hdp/2.2.4.2-2/storm/lib/snakeyaml-1.11.jar:/usr/hdp/2.2.4.2-2/storm/lib/hadoop-common-2.6.0.2.2.4.2-2.jar:/usr/hdp/2.2.4.2-2/storm/lib/eclipselink-2.5.2-M1.jar:/usr/hdp/2.2.4.2-2/storm/lib/slf4j-api-1.6.5.jar:/usr/hdp/2.2.4.2-2/storm/lib/servlet-api-2.5.jar:/usr/hdp/2.2.4.2-2/storm/lib/tools.cli-0.2.4.jar:/usr/hdp/2.2.4.2-2/storm/lib/joda-time-2.0.jar:/usr/hdp/2.2.4.2-2/storm/lib/java.classpath-0.2.2.jar:/usr/hdp/2.2.4.2-2/storm/lib/commons-codec-1.6.jar:/usr/hdp/2.2.4.2-2/storm/lib/objenesis-1.2.jar:/usr/hdp/2.2.4.2-2/storm/lib/jetty-io-7.6.13.v20130916.jar:/usr/hdp/2.2.4.2-2/storm/lib/hadoop-auth-2.4.0.jar:/usr/hdp/2.2.4.2-2/storm/lib/compojure-1.1.3.jar:/usr/hdp/2.2.4.2-2/storm/lib/ring-jetty-adapter-1.3.0.jar:/usr/hdp/2.2.4.2-2/storm/lib/jetty-security-7.6.13.v20130916.jar:/usr/hdp/2.2.4.2-2/storm/lib/ranger-plugins-impl-0.4.0.2.2.4.2-2.jar:/usr/hdp/2.2.4.2-2/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/hdp/2.2.4.2-2/storm/lib/jetty-util-7.6.13.v20130916.jar:/usr/hdp/2.2.4.2-2/storm/lib/clojure-1.5.1.jar:/usr/hdp/2.2.4.2-2/storm/lib/minlog-1.2.jar:/usr/hdp/2.2.4.2-2/storm/lib/ns-tracker-0.2.2.jar:/usr/hdp/2.2.4.2-2/storm/lib/jersey-bundle-1.17.1.jar:/usr/hdp/2.2.4.2-2/storm/lib/clout-1.0.1.jar:/usr/hdp/2.2.4.2-2/storm/lib/disruptor-2.10.1.jar:/usr/hdp/2.2.4.2-2/storm/lib/tools.logging-0.2.3.jar:/usr/hdp/2.2.4.2-2/storm/lib/javax.persistence-2.1.0.jar:/usr/hdp/2.2.4.2-2/storm/lib/jetty-continuation-7.6.13.v20130916.jar:/usr/hdp/2.2.4.2-2/storm/lib/jetty-servlets-7.6.13.v20130916.jar:/usr/hdp/2.2.4.2-2/storm/lib/ring-anti-forgery-1.0.0.jar:/usr/hdp/2.2.4.2-2/storm/lib/hiccup-0.3.6.jar:/usr/hdp/2.2.4.2-2/storm/lib/commons-lang-2.5.jar:/usr/hdp/2.2.4.2-2/storm/lib/crypto-equality-1.0.0.jar:/usr/hdp/2.2.4.2-2/storm/lib/jetty-server-7.6.13.v20130916.jar:/usr/hdp/2.2.4.2-2/storm/lib/gmetric4j-1.0.7.jar:/usr/hdp/2.2.4.2-2/storm/lib/storm-core-0.9.3.2.2.4.2-2.jar:/usr/hdp/2.2.4.2-2/storm/lib/ring-core-1.1.5.jar:/usr/hdp/2.2.4.2-2/storm/lib/commons-exec-1.1.jar:/usr/hdp/2.2.4.2-2/storm/lib/logback-core-1.0.6.jar:/usr/hdp/2.2.4.2-2/storm/lib/carbonite-1.4.0.jar:/usr/hdp/2.2.4.2-2/storm/lib/math.numeric-tower-0.0.1.jar:/usr/hdp/2.2.4.2-2/storm/lib/commons-fileupload-1.2.1.jar:/usr/hdp/2.2.4.2-2/storm/lib/ring-servlet-1.3.0.jar:/usr/hdp/2.2.4.2-2/storm/lib/ranger-plugins-cred-0.4.0.2.2.4.2-2.jar:/usr/hdp/2.2.4.2-2/storm/lib/commons-io-2.4.jar:/usr/hdp/2.2.4.2-2/storm/lib/ranger-storm-plugin-0.4.0.2.2.4.2-2.jar:/usr/hdp/2.2.4.2-2/storm/lib/gson-2.2.4.jar:/usr/hdp/2.2.4.2-2/storm/lib/tools.macro-0.1.0.jar:/usr/hdp/2.2.4.2-2/storm/lib/jetty-servlet-7.6.13.v20130916.jar:/usr/hdp/2.2.4.2-2/storm/lib/kryo-2.21.jar:/usr/hdp/2.2.4.2-2/storm/lib/commons-logging-1.2.jar:/usr/hdp/2.2.4.2-2/storm/lib/asm-4.0.jar:/usr/hdp/2.2.4.2-2/storm/lib/ranger-plugins-audit-0.4.0.2.2.4.2-2.jar:/usr/hdp/2.2.4.2-2/storm/lib/jgrapht-core-0.9.0.jar:/usr/hdp/2.2.4.2-2/storm/lib/tools.namespace-0.2.4.jar:/usr/hdp/2.2.4.2-2/storm/lib/commons-configuration-1.10.jar:/usr/hdp/2.2.4.2-2/storm/lib/core.incubator-0.1.0.jar:/usr/hdp/2.2.4.2-2/storm/lib/crypto-random-1.2.0.jar:/usr/hdp/2.2.4.2-2/storm/lib/commons-collections-3.2.1.jar:/usr/hdp/2.2.4.2-2/storm/lib/jetty-client-7.6.13.v20130916.jar:/usr/hdp/2.2.4.2-2/storm/lib/guava-11.0.2.jar:/usr/hdp/2.2.4.2-2/storm/lib/javax.servlet-2.5.0.v201103041518.jar:/usr/hdp/2.2.4.2-2/storm/lib/json-simple-1.1.jar:/usr/hdp/2.2.4.2-2/storm/lib/clj-time-0.4.1.jar:/usr/hdp/2.2.4.2-2/storm/lib/ring-devel-1.3.0.jar:/usr/hdp/2.2.4.2-2/storm/conf:/usr/hdp/current/storm-supervisor/contrib/storm-jmxetric/lib/jmxetric-1.0.4.jar:/hadoop/storm/supervisor/stormdist/aek-16-1436963685/stormjar.jar:/usr/hdp/current/storm-client/contrib/storm-jmxetric/lib/jmxetric-1.0.4.jar
2015-07-15 15:34:48 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:java.library.path=/hadoop/storm/supervisor/stormdist/aek-16-1436963685/resources/Linux-amd64:/hadoop/storm/supervisor/stormdist/aek-16-1436963685/resources:/usr/local/lib:/opt/local/lib:/usr/lib:/usr/hdp/current/storm-client/lib
2015-07-15 15:34:48 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:java.io.tmpdir=/tmp
2015-07-15 15:34:48 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:java.compiler=<NA>
2015-07-15 15:34:48 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:os.name=Linux
2015-07-15 15:34:48 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:os.arch=amd64
2015-07-15 15:34:48 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:os.version=3.2.0-70-generic
2015-07-15 15:34:48 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:user.name=storm
2015-07-15 15:34:48 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:user.home=/home/storm
2015-07-15 15:34:48 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:user.dir=/home/storm
2015-07-15 15:34:49 b.s.d.worker [INFO] Launching worker for aek-16-1436963685 on 3a7d0fdf-91c7-461c-bc24-2c912a622f34:6701 with id 3229d690-cb75-45a3-bab4-e3d0dad1c9a3 and conf {"dev.zookeeper.path" "/tmp/dev-storm-zookeeper", "topology.tick.tuple.freq.secs" nil, "topology.builtin.metrics.bucket.size.secs" 60, "topology.fall.back.on.java.serialization" true, "supervisor.run.worker.as.user" false, "topology.max.error.report.per.interval" 5, "storm.group.mapping.service" "backtype.storm.security.auth.ShellBasedGroupsMapping", "zmq.linger.millis" 5000, "topology.skip.missing.kryo.registrations" false, "storm.messaging.netty.client_worker_threads" 1, "ui.childopts" "-Xmx768m ", "storm.zookeeper.session.timeout" 20000, "ui.filter.params" nil, "nimbus.reassign" true, "storm.auth.simple-acl.admins" [], "storm.group.mapping.service.cache.duration.secs" 120, "topology.trident.batch.emit.interval.millis" 500, "drpc.authorizer.acl.filename" "drpc-auth-acl.yaml", "storm.messaging.netty.flush.check.interval.ms" 10, "ui.header.buffer.bytes" 4096, "nimbus.monitor.freq.secs" 10, "logviewer.childopts" "-Xmx128m ", "java.library.path" "/usr/local/lib:/opt/local/lib:/usr/lib:/usr/hdp/current/storm-client/lib", "supervisor.supervisors" [], "topology.executor.send.buffer.size" 1024, "metrics.reporter.register" "org.apache.hadoop.metrics2.sink.storm.StormTimelineMetricsReporter", "storm.local.dir" "/hadoop/storm", "storm.messaging.netty.buffer_size" 5242880, "supervisor.worker.start.timeout.secs" 120, "drpc.authorizer.acl.strict" false, "storm.nimbus.retry.times" 5, "topology.enable.message.timeouts" true, "nimbus.cleanup.inbox.freq.secs" 600, "nimbus.inbox.jar.expiration.secs" 3600, "drpc.worker.threads" 64, "storm.meta.serialization.delegate" "backtype.storm.serialization.DefaultSerializationDelegate", "topology.worker.shared.thread.pool.size" 4, "nimbus.host" "clu01.softnet.tuc.gr", "storm.messaging.netty.min_wait_ms" 100, "storm.zookeeper.port" 2181, "transactional.zookeeper.port" nil, "ui.http.creds.plugin" "backtype.storm.security.auth.DefaultHttpCredentialsPlugin", "topology.executor.receive.buffer.size" 1024, "logs.users" nil, "transactional.zookeeper.servers" nil, "storm.zookeeper.root" "/storm", "storm.zookeeper.retry.intervalceiling.millis" 30000, "supervisor.enable" true, "storm.messaging.netty.server_worker_threads" 1, "storm.zookeeper.servers" ["clu02.softnet.tuc.gr" "clu01.softnet.tuc.gr" "clu03.softnet.tuc.gr"], "transactional.zookeeper.root" "/transactional", "topology.acker.executors" nil, "storm.auth.simple-acl.users" [], "storm.zookeeper.auth.user" nil, "topology.testing.always.try.serialize" false, "topology.transfer.buffer.size" 1024, "storm.principal.tolocal" "backtype.storm.security.auth.DefaultPrincipalToLocal", "topology.worker.childopts" nil, "drpc.queue.size" 128, "worker.childopts" "-Xmx768m -javaagent:/usr/hdp/current/storm-client/contrib/storm-jmxetric/lib/jmxetric-1.0.4.jar=host=localhost,port=8650,wireformat31x=true,mode=multicast,config=/usr/hdp/current/storm-client/contrib/storm-jmxetric/conf/jmxetric-conf.xml,process=Worker_%ID%_JVM", "storm.auth.simple-acl.users.commands" [], "supervisor.heartbeat.frequency.secs" 5, "topology.error.throttle.interval.secs" 10, "storm.nimbus.retry.interval.millis" 2000, "ui.users" nil, "zmq.hwm" 0, "drpc.port" 3772, "supervisor.monitor.frequency.secs" 3, "drpc.childopts" "-Xmx768m ", "topology.receiver.buffer.size" 8, "task.heartbeat.frequency.secs" 3, "topology.tasks" nil, "storm.messaging.netty.max_retries" 30, "topology.spout.wait.strategy" "backtype.storm.spout.SleepSpoutWaitStrategy", "nimbus.thrift.max_buffer_size" 1048576, "drpc.invocations.threads" 64, "drpc.https.port" -1, "supervisor.supervisors.commands" [], "topology.metrics.consumer.register" [{"class" "org.apache.hadoop.metrics2.sink.storm.StormTimelineMetricsSink", "parallelism.hint" 1}], "topology.max.spout.pending" nil, "ui.filter" nil, "logviewer.cleanup.age.mins" 10080, "storm.zookeeper.retry.interval" 1000, "topology.sleep.spout.wait.strategy.time.ms" 1, "nimbus.topology.validator" "backtype.storm.nimbus.DefaultTopologyValidator", "supervisor.slots.ports" [6700 6701], "storm.messaging.netty.authentication" false, "topology.environment" nil, "topology.debug" false, "nimbus.thrift.threads" 64, "nimbus.task.launch.secs" 120, "nimbus.supervisor.timeout.secs" 60, "drpc.http.creds.plugin" "backtype.storm.security.auth.DefaultHttpCredentialsPlugin", "topology.message.timeout.secs" 30, "task.refresh.poll.secs" 10, "topology.workers" 1, "supervisor.childopts" "-Xmx256m -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.port=56431 -javaagent:/usr/hdp/current/storm-supervisor/contrib/storm-jmxetric/lib/jmxetric-1.0.4.jar=host=localhost,port=8650,wireformat31x=true,mode=multicast,config=/usr/hdp/current/storm-supervisor/contrib/storm-jmxetric/conf/jmxetric-conf.xml,process=Supervisor_JVM", "storm.auth.simple-white-list.users" [], "nimbus.thrift.port" 6627, "drpc.https.keystore.type" "JKS", "topology.stats.sample.rate" 0.05, "task.credentials.poll.secs" 30, "worker.heartbeat.frequency.secs" 1, "ui.actions.enabled" true, "topology.tuple.serializer" "backtype.storm.serialization.types.ListDelegateSerializer", "drpc.https.keystore.password" "", "topology.disruptor.wait.strategy" "com.lmax.disruptor.BlockingWaitStrategy", "topology.multilang.serializer" "backtype.storm.multilang.JsonSerializer", "drpc.max_buffer_size" 1048576, "nimbus.task.timeout.secs" 30, "storm.zookeeper.connection.timeout" 15000, "topology.kryo.factory" "backtype.storm.serialization.DefaultKryoFactory", "drpc.invocations.port" 3773, "logviewer.port" 8000, "zmq.threads" 1, "storm.zookeeper.retry.times" 5, "topology.worker.receiver.thread.count" 1, "storm.thrift.transport" "backtype.storm.security.auth.SimpleTransportPlugin", "topology.state.synchronization.timeout.secs" 60, "supervisor.worker.timeout.secs" 30, "nimbus.file.copy.expiration.secs" 600, "nimbus.credential.renewers.freq.secs" 600, "storm.messaging.transport" "backtype.storm.messaging.netty.Context", "worker.gc.childopts" "", "logviewer.appender.name" "A1", "storm.messaging.netty.max_wait_ms" 1000, "storm.zookeeper.auth.password" nil, "drpc.http.port" 3774, "drpc.request.timeout.secs" 600, "storm.local.mode.zmq" false, "ui.port" 8745, "nimbus.childopts" "-Xmx1024m -javaagent:/usr/hdp/current/storm-nimbus/contrib/storm-jmxetric/lib/jmxetric-1.0.4.jar=host=localhost,port=8649,wireformat31x=true,mode=multicast,config=/usr/hdp/current/storm-nimbus/contrib/storm-jmxetric/conf/jmxetric-conf.xml,process=Nimbus_JVM", "storm.cluster.mode" "distributed", "topology.optimize" true, "topology.max.task.parallelism" nil, "storm.messaging.netty.transfer.batch.size" 262144, "storm.nimbus.retry.intervalceiling.millis" 60000, "topology.classpath" nil, "storm.log.dir" "/var/log/storm"}
2015-07-15 15:34:49 b.s.u.StormBoundedExponentialBackoffRetry [INFO] The baseSleepTimeMs [1000] the maxSleepTimeMs [30000] the maxRetries [5]
2015-07-15 15:34:49 o.a.s.c.f.i.CuratorFrameworkImpl [INFO] Starting
2015-07-15 15:34:49 o.a.s.z.ZooKeeper [INFO] Initiating client connection, connectString=clu02.softnet.tuc.gr:2181,clu01.softnet.tuc.gr:2181,clu03.softnet.tuc.gr:2181 sessionTimeout=20000 watcher=org.apache.storm.curator.ConnectionState#1ce9f29c
2015-07-15 15:34:49 o.a.s.z.ClientCnxn [INFO] Opening socket connection to server clu02.softnet.tuc.gr/147.27.14.202:2181. Will not attempt to authenticate using SASL (unknown error)
2015-07-15 15:34:49 o.a.s.z.ClientCnxn [INFO] Socket connection established to clu02.softnet.tuc.gr/147.27.14.202:2181, initiating session
2015-07-15 15:34:49 o.a.s.z.ClientCnxn [INFO] Session establishment complete on server clu02.softnet.tuc.gr/147.27.14.202:2181, sessionid = 0x24d6c5b265b5e1a, negotiated timeout = 20000
2015-07-15 15:34:49 o.a.s.c.f.s.ConnectionStateManager [INFO] State change: CONNECTED
2015-07-15 15:34:49 b.s.zookeeper [INFO] Zookeeper state update: :connected:none
2015-07-15 15:34:50 o.a.s.z.ZooKeeper [INFO] Session: 0x24d6c5b265b5e1a closed
2015-07-15 15:34:50 o.a.s.z.ClientCnxn [INFO] EventThread shut down
2015-07-15 15:34:50 b.s.u.StormBoundedExponentialBackoffRetry [INFO] The baseSleepTimeMs [1000] the maxSleepTimeMs [30000] the maxRetries [5]
2015-07-15 15:34:50 o.a.s.c.f.i.CuratorFrameworkImpl [INFO] Starting
2015-07-15 15:34:50 o.a.s.z.ZooKeeper [INFO] Initiating client connection, connectString=clu02.softnet.tuc.gr:2181,clu01.softnet.tuc.gr:2181,clu03.softnet.tuc.gr:2181/storm sessionTimeout=20000 watcher=org.apache.storm.curator.ConnectionState#10c3dd25
2015-07-15 15:34:50 o.a.s.z.ClientCnxn [INFO] Opening socket connection to server clu02.softnet.tuc.gr/147.27.14.202:2181. Will not attempt to authenticate using SASL (unknown error)
2015-07-15 15:34:50 o.a.s.z.ClientCnxn [INFO] Socket connection established to clu02.softnet.tuc.gr/147.27.14.202:2181, initiating session
2015-07-15 15:34:50 o.a.s.z.ClientCnxn [INFO] Session establishment complete on server clu02.softnet.tuc.gr/147.27.14.202:2181, sessionid = 0x24d6c5b265b5e1b, negotiated timeout = 20000
2015-07-15 15:34:50 o.a.s.c.f.s.ConnectionStateManager [INFO] State change: CONNECTED
2015-07-15 15:34:50 b.s.s.a.AuthUtils [INFO] Got AutoCreds []
2015-07-15 15:34:50 b.s.d.worker [INFO] Reading Assignments.
2015-07-15 15:34:50 b.s.m.TransportFactory [INFO] Storm peer transport plugin:backtype.storm.messaging.netty.Context
2015-07-15 15:34:51 b.s.d.executor [INFO] Loading executor __metricsorg.apache.hadoop.metrics2.sink.storm.StormTimelineMetricsSink:[2 2]
2015-07-15 15:34:51 b.s.d.task [INFO] Emitting: __metricsorg.apache.hadoop.metrics2.sink.storm.StormTimelineMetricsSink __system ["startup"]
2015-07-15 15:34:51 b.s.d.executor [INFO] Loaded executor tasks __metricsorg.apache.hadoop.metrics2.sink.storm.StormTimelineMetricsSink:[2 2]
2015-07-15 15:34:51 b.s.d.executor [INFO] Finished loading executor __metricsorg.apache.hadoop.metrics2.sink.storm.StormTimelineMetricsSink:[2 2]
2015-07-15 15:34:51 b.s.d.executor [INFO] Preparing bolt __metricsorg.apache.hadoop.metrics2.sink.storm.StormTimelineMetricsSink:(2)
2015-07-15 15:34:51 b.s.d.executor [INFO] Loading executor distributeeventbolt:[3 3]
2015-07-15 15:34:51 b.s.d.task [INFO] Emitting: distributeeventbolt __system ["startup"]
2015-07-15 15:34:51 b.s.d.executor [INFO] Loaded executor tasks distributeeventbolt:[3 3]
2015-07-15 15:34:51 b.s.util [ERROR] Async loop died!
java.lang.RuntimeException: Could not instantiate a class listed in config under section topology.metrics.consumer.register with fully qualified name org.apache.hadoop.metrics2.sink.storm.StormTimelineMetricsSink
at backtype.storm.metric.MetricsConsumerBolt.prepare(MetricsConsumerBolt.java:46) ~[storm-core-0.9.3.2.2.4.2-2.jar:0.9.3.2.2.4.2-2]
at backtype.storm.daemon.executor$fn__4641$fn__4654.invoke(executor.clj:732) ~[storm-core-0.9.3.2.2.4.2-2.jar:0.9.3.2.2.4.2-2]
at backtype.storm.util$async_loop$fn__551.invoke(util.clj:463) ~[storm-core-0.9.3.2.2.4.2-2.jar:0.9.3.2.2.4.2-2]
at clojure.lang.AFn.run(AFn.java:24) [clojure-1.5.1.jar:na]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_80]
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.metrics2.sink.storm.StormTimelineMetricsSink
at java.net.URLClassLoader$1.run(URLClassLoader.java:366) ~[na:1.7.0_80]
at java.net.URLClassLoader$1.run(URLClassLoader.java:355) ~[na:1.7.0_80]
at java.security.AccessController.doPrivileged(Native Method) ~[na:1.7.0_80]
at java.net.URLClassLoader.findClass(URLClassLoader.java:354) ~[na:1.7.0_80]
at java.lang.ClassLoader.loadClass(ClassLoader.java:425) ~[na:1.7.0_80]
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) ~[na:1.7.0_80]
at java.lang.ClassLoader.loadClass(ClassLoader.java:358) ~[na:1.7.0_80]
at java.lang.Class.forName0(Native Method) ~[na:1.7.0_80]
at java.lang.Class.forName(Class.java:195) ~[na:1.7.0_80]
at backtype.storm.metric.MetricsConsumerBolt.prepare(MetricsConsumerBolt.java:44) ~[storm-core-0.9.3.2.2.4.2-2.jar:0.9.3.2.2.4.2-2]
... 4 common frames omitted
2015-07-15 15:34:51 b.s.d.executor [ERROR]
java.lang.RuntimeException: Could not instantiate a class listed in config under section topology.metrics.consumer.register with fully qualified name org.apache.hadoop.metrics2.sink.storm.StormTimelineMetricsSink
at backtype.storm.metric.MetricsConsumerBolt.prepare(MetricsConsumerBolt.java:46) ~[storm-core-0.9.3.2.2.4.2-2.jar:0.9.3.2.2.4.2-2]
at backtype.storm.daemon.executor$fn__4641$fn__4654.invoke(executor.clj:732) ~[storm-core-0.9.3.2.2.4.2-2.jar:0.9.3.2.2.4.2-2]
at backtype.storm.util$async_loop$fn__551.invoke(util.clj:463) ~[storm-core-0.9.3.2.2.4.2-2.jar:0.9.3.2.2.4.2-2]
at clojure.lang.AFn.run(AFn.java:24) [clojure-1.5.1.jar:na]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_80]
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.metrics2.sink.storm.StormTimelineMetricsSink
at java.net.URLClassLoader$1.run(URLClassLoader.java:366) ~[na:1.7.0_80]
at java.net.URLClassLoader$1.run(URLClassLoader.java:355) ~[na:1.7.0_80]
at java.security.AccessController.doPrivileged(Native Method) ~[na:1.7.0_80]
at java.net.URLClassLoader.findClass(URLClassLoader.java:354) ~[na:1.7.0_80]
at java.lang.ClassLoader.loadClass(ClassLoader.java:425) ~[na:1.7.0_80]
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) ~[na:1.7.0_80]
at java.lang.ClassLoader.loadClass(ClassLoader.java:358) ~[na:1.7.0_80]
at java.lang.Class.forName0(Native Method) ~[na:1.7.0_80]
at java.lang.Class.forName(Class.java:195) ~[na:1.7.0_80]
at backtype.storm.metric.MetricsConsumerBolt.prepare(MetricsConsumerBolt.java:44) ~[storm-core-0.9.3.2.2.4.2-2.jar:0.9.3.2.2.4.2-2]
... 4 common frames omitted
2015-07-15 15:34:51 b.s.d.executor [INFO] Finished loading executor distributeeventbolt:[3 3]
2015-07-15 15:34:51 b.s.d.executor [INFO] Preparing bolt distributeeventbolt:(3)
2015-07-15 15:34:51 b.s.d.executor [INFO] Prepared bolt distributeeventbolt:(3)
2015-07-15 15:34:51 b.s.d.executor [INFO] Loading executor distributeeventbolt:[4 4]
2015-07-15 15:34:51 b.s.d.task [INFO] Emitting: distributeeventbolt __system ["startup"]
2015-07-15 15:34:51 b.s.d.executor [INFO] Loaded executor tasks distributeeventbolt:[4 4]
2015-07-15 15:34:51 b.s.d.executor [INFO] Finished loading executor distributeeventbolt:[4 4]
2015-07-15 15:34:51 b.s.d.executor [INFO] Preparing bolt distributeeventbolt:(4)
2015-07-15 15:34:51 b.s.d.executor [INFO] Prepared bolt distributeeventbolt:(4)
2015-07-15 15:34:51 b.s.d.executor [INFO] Loading executor distributeeventbolt:[5 5]
2015-07-15 15:34:51 b.s.d.task [INFO] Emitting: distributeeventbolt __system ["startup"]
2015-07-15 15:34:51 b.s.d.executor [INFO] Loaded executor tasks distributeeventbolt:[5 5]
2015-07-15 15:34:51 b.s.d.executor [INFO] Finished loading executor distributeeventbolt:[5 5]
2015-07-15 15:34:51 b.s.d.executor [INFO] Preparing bolt distributeeventbolt:(5)
2015-07-15 15:34:51 b.s.d.executor [INFO] Prepared bolt distributeeventbolt:(5)
2015-07-15 15:34:51 b.s.d.executor [INFO] Loading executor eventspout:[6 6]
2015-07-15 15:34:51 b.s.util [ERROR] Halting process: ("Worker died")
java.lang.RuntimeException: ("Worker died")
at backtype.storm.util$exit_process_BANG_.doInvoke(util.clj:322) [storm-core-0.9.3.2.2.4.2-2.jar:0.9.3.2.2.4.2-2]
at clojure.lang.RestFn.invoke(RestFn.java:423) [clojure-1.5.1.jar:na]
at backtype.storm.daemon.worker$fn__5053$fn__5054.invoke(worker.clj:495) [storm-core-0.9.3.2.2.4.2-2.jar:0.9.3.2.2.4.2-2]
at backtype.storm.daemon.executor$mk_executor_data$fn__4474$fn__4475.invoke(executor.clj:245) [storm-core-0.9.3.2.2.4.2-2.jar:0.9.3.2.2.4.2-2]
at backtype.storm.util$async_loop$fn__551.invoke(util.clj:475) [storm-core-0.9.3.2.2.4.2-2.jar:0.9.3.2.2.4.2-2]
at clojure.lang.AFn.run(AFn.java:24) [clojure-1.5.1.jar:na]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_80]
EDIT_1:
I made an export of my topology as Runnable Jar via Eclipse but this came along in my terminal.
Exception in thread "main" java.lang.ExceptionInInitializerError
at backtype.storm.topology.TopologyBuilder$BoltGetter.customGrouping(TopologyBuilder.java:340)
at backtype.storm.topology.TopologyBuilder$BoltGetter.customGrouping(TopologyBuilder.java:264)
at main.java.storm.Main.main(Main.java:47)
Caused by: java.lang.RuntimeException: Found multiple defaults.yaml resources. You're probably bundling the Storm jars with your topology jar. [jar:file:/home/gdidymiotis/teliko_1.0.0_runnable.jar!/defaults.yaml, jar:file:/usr/hdp/2.2.4.2-2/storm/lib/storm-core-0.9.3.2.2.4.2-2.jar!/defaults.yaml]
at backtype.storm.utils.Utils.findAndReadConfigFile(Utils.java:139)
at backtype.storm.utils.Utils.readDefaultConfig(Utils.java:166)
at backtype.storm.utils.Utils.readStormConfig(Utils.java:190)
at backtype.storm.utils.Utils.<clinit>(Utils.java:77)
... 3 more
The log shows the problem clearly: Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.metrics2.sink.storm.StormTimelineMetricsSink
I guess, this file in not included in the jar you submit to Storm.
Related
Spark-submit job fails on yarn nodemanager with error Client cannot authenticate via:[TOKEN, KERBEROS]
I am running spark-submit in yarn client mode. Yarn has been setup with HDP sandbox with kerberos enabled. HDP Sandbox is running on docker container on Mac host.
When spark submit is run from within the docker container of the sandbox, it’s runs successfully but when spark submit is run from the host machine it fails immediately after ACCEPTED state with error:
19/07/28 00:41:21 INFO yarn.Client: Application report for application_1564298049378_0008 (state: ACCEPTED)
19/07/28 00:41:22 INFO yarn.Client: Application report for application_1564298049378_0008 (state: ACCEPTED)
19/07/28 00:41:23 INFO yarn.Client: Application report for application_1564298049378_0008 (state: FAILED)
19/07/28 00:41:23 INFO yarn.Client:
client token: N/A
diagnostics: Application application_1564298049378_0008 failed 2 times due to AM Container for appattempt_1564298049378_0008_000002 exited with exitCode: -1000
Failing this attempt.Diagnostics: (Client.java:1558)
... 37 more
Caused by: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
I could not find any more information about the failure. Any help will be greatly appreciated.
Here is the resourcemanager log:
2019-07-28 22:39:04,654 INFO resourcemanager.ClientRMService (ClientRMService.java:getNewApplicationId(341)) - Allocated new applicationId: 20
2019-07-28 22:39:10,982 INFO capacity.CapacityScheduler (CapacityScheduler.java:checkAndGetApplicationPriority(2526)) - Application 'application_1564332457320_0020' is submitted without priority hence considering default queue/cluster priority: 0
2019-07-28 22:39:10,982 INFO capacity.CapacityScheduler (CapacityScheduler.java:checkAndGetApplicationPriority(2547)) - Priority '0' is acceptable in queue : santosh for application: application_1564332457320_0020
2019-07-28 22:39:10,983 WARN rmapp.RMAppImpl (RMAppImpl.java:(473)) - The specific max attempts: 0 for application: 20 is invalid, because it is out of the range [1, 2]. Use the global max attempts instead.
2019-07-28 22:39:10,983 INFO collector.TimelineCollectorManager (TimelineCollectorManager.java:putIfAbsent(142)) - the collector for application_1564332457320_0020 was added
2019-07-28 22:39:10,984 INFO resourcemanager.ClientRMService (ClientRMService.java:submitApplication(648)) - Application with id 20 submitted by user santosh
2019-07-28 22:39:10,984 INFO security.DelegationTokenRenewer (DelegationTokenRenewer.java:handleAppSubmitEvent(458)) - application_1564332457320_0020 found existing hdfs token Kind: HDFS_DELEGATION_TOKEN, Service: 192.168.50.1:8020, Ident: (token for santosh: HDFS_DELEGATION_TOKEN owner=santosh#XXX.XX, renewer=yarn, realUser=, issueDate=1564353550169, maxDate=1564958350169, sequenceNumber=125, masterKeyId=20)
2019-07-28 22:39:11,011 INFO security.DelegationTokenRenewer (DelegationTokenRenewer.java:renewToken(635)) - Renewed delegation-token= [Kind: HDFS_DELEGATION_TOKEN, Service: 192.168.50.1:8020, Ident: (token for santosh: HDFS_DELEGATION_TOKEN owner=santosh#XXX.XX, renewer=yarn, realUser=, issueDate=1564353550169, maxDate=1564958350169, sequenceNumber=125, masterKeyId=20);exp=1564439951007; apps=[application_1564332457320_0020]]
2019-07-28 22:39:11,011 INFO security.DelegationTokenRenewer (DelegationTokenRenewer.java:setTimerForTokenRenewal(613)) - Renew Kind: HDFS_DELEGATION_TOKEN, Service: 192.168.50.1:8020, Ident: (token for santosh: HDFS_DELEGATION_TOKEN owner=santosh#XXX.XX, renewer=yarn, realUser=, issueDate=1564353550169, maxDate=1564958350169, sequenceNumber=125, masterKeyId=20);exp=1564439951007; apps=[application_1564332457320_0020] in 86399996 ms, appId = [application_1564332457320_0020]
2019-07-28 22:39:11,011 INFO rmapp.RMAppImpl (RMAppImpl.java:transition(1259)) - Storing application with id application_1564332457320_0020
2019-07-28 22:39:11,012 INFO rmapp.RMAppImpl (RMAppImpl.java:handle(912)) - application_1564332457320_0020 State change from NEW to NEW_SAVING on event = START
2019-07-28 22:39:11,012 INFO recovery.RMStateStore (RMStateStore.java:transition(222)) - Storing info for app: application_1564332457320_0020
2019-07-28 22:39:11,022 INFO rmapp.RMAppImpl (RMAppImpl.java:handle(912)) - application_1564332457320_0020 State change from NEW_SAVING to SUBMITTED on event = APP_NEW_SAVED
2019-07-28 22:39:11,022 INFO capacity.ParentQueue (ParentQueue.java:addApplication(494)) - Application added - appId: application_1564332457320_0020 user: santosh leaf-queue of parent: root #applications: 1
2019-07-28 22:39:11,023 INFO capacity.CapacityScheduler (CapacityScheduler.java:addApplication(990)) - Accepted application application_1564332457320_0020 from user: santosh, in queue: santosh
2019-07-28 22:39:11,023 INFO rmapp.RMAppImpl (RMAppImpl.java:handle(912)) - application_1564332457320_0020 State change from SUBMITTED to ACCEPTED on event = APP_ACCEPTED
2019-07-28 22:39:11,023 INFO resourcemanager.ApplicationMasterService (ApplicationMasterService.java:registerAppAttempt(479)) - Registering app attempt : appattempt_1564332457320_0020_000001
2019-07-28 22:39:11,024 INFO attempt.RMAppAttemptImpl (RMAppAttemptImpl.java:handle(925)) - appattempt_1564332457320_0020_000001 State change from NEW to SUBMITTED on event = START
2019-07-28 22:39:11,024 INFO capacity.LeafQueue (LeafQueue.java:activateApplications(911)) - Application application_1564332457320_0020 from user: santosh activated in queue: santosh
2019-07-28 22:39:11,025 INFO capacity.LeafQueue (LeafQueue.java:addApplicationAttempt(941)) - Application added - appId: application_1564332457320_0020 user: santosh, leaf-queue: santosh #user-pending-applications: 0 #user-active-applications: 1 #queue-pending-applications: 0 #queue-active-applications: 1
2019-07-28 22:39:11,025 INFO capacity.CapacityScheduler (CapacityScheduler.java:addApplicationAttempt(1036)) - Added Application Attempt appattempt_1564332457320_0020_000001 to scheduler from user santosh in queue santosh
2019-07-28 22:39:11,028 INFO attempt.RMAppAttemptImpl (RMAppAttemptImpl.java:handle(925)) - appattempt_1564332457320_0020_000001 State change from SUBMITTED to SCHEDULED on event = ATTEMPT_ADDED
2019-07-28 22:39:11,033 INFO allocator.AbstractContainerAllocator (AbstractContainerAllocator.java:getCSAssignmentFromAllocateResult(129)) - assignedContainer application attempt=appattempt_1564332457320_0020_000001 container=null queue=santosh clusterResource= type=OFF_SWITCH requestedPartition=
2019-07-28 22:39:11,034 INFO rmcontainer.RMContainerImpl (RMContainerImpl.java:handle(490)) - container_e20_1564332457320_0020_01_000001 Container Transitioned from NEW to ALLOCATED
2019-07-28 22:39:11,035 INFO fica.FiCaSchedulerNode (FiCaSchedulerNode.java:allocateContainer(169)) - Assigned container container_e20_1564332457320_0020_01_000001 of capacity on host sandbox-hdp.hortonworks.com:45454, which has 1 containers, used and available after allocation
2019-07-28 22:39:11,038 INFO security.NMTokenSecretManagerInRM (NMTokenSecretManagerInRM.java:createAndGetNMToken(200)) - Sending NMToken for nodeId : sandbox-hdp.hortonworks.com:45454 for container : container_e20_1564332457320_0020_01_000001
2019-07-28 22:39:11,043 INFO rmcontainer.RMContainerImpl (RMContainerImpl.java:handle(490)) - container_e20_1564332457320_0020_01_000001 Container Transitioned from ALLOCATED to ACQUIRED
2019-07-28 22:39:11,043 INFO security.NMTokenSecretManagerInRM (NMTokenSecretManagerInRM.java:clearNodeSetForAttempt(146)) - Clear node set for appattempt_1564332457320_0020_000001
2019-07-28 22:39:11,044 INFO capacity.ParentQueue (ParentQueue.java:apply(1332)) - assignedContainer queue=root usedCapacity=0.25 absoluteUsedCapacity=0.25 used= cluster=
2019-07-28 22:39:11,044 INFO capacity.CapacityScheduler (CapacityScheduler.java:tryCommit(2890)) - Allocation proposal accepted
2019-07-28 22:39:11,044 INFO attempt.RMAppAttemptImpl (RMAppAttemptImpl.java:storeAttempt(2213)) - Storing attempt: AppId: application_1564332457320_0020 AttemptId: appattempt_1564332457320_0020_000001 MasterContainer: Container: [ContainerId: container_e20_1564332457320_0020_01_000001, AllocationRequestId: -1, Version: 0, NodeId: sandbox-hdp.hortonworks.com:45454, NodeHttpAddress: sandbox-hdp.hortonworks.com:8042, Resource: , Priority: 0, Token: Token { kind: ContainerToken, service: 172.18.0.3:45454 }, ExecutionType: GUARANTEED, ]
2019-07-28 22:39:11,051 INFO attempt.RMAppAttemptImpl (RMAppAttemptImpl.java:handle(925)) - appattempt_1564332457320_0020_000001 State change from SCHEDULED to ALLOCATED_SAVING on event = CONTAINER_ALLOCATED
2019-07-28 22:39:11,057 INFO attempt.RMAppAttemptImpl (RMAppAttemptImpl.java:handle(925)) - appattempt_1564332457320_0020_000001 State change from ALLOCATED_SAVING to ALLOCATED on event = ATTEMPT_NEW_SAVED
2019-07-28 22:39:11,060 INFO amlauncher.AMLauncher (AMLauncher.java:run(307)) - Launching masterappattempt_1564332457320_0020_000001
2019-07-28 22:39:11,068 INFO amlauncher.AMLauncher (AMLauncher.java:launch(109)) - Setting up container Container: [ContainerId: container_e20_1564332457320_0020_01_000001, AllocationRequestId: -1, Version: 0, NodeId: sandbox-hdp.hortonworks.com:45454, NodeHttpAddress: sandbox-hdp.hortonworks.com:8042, Resource: , Priority: 0, Token: Token { kind: ContainerToken, service: 172.18.0.3:45454 }, ExecutionType: GUARANTEED, ] for AM appattempt_1564332457320_0020_000001
2019-07-28 22:39:11,069 INFO security.AMRMTokenSecretManager (AMRMTokenSecretManager.java:createAndGetAMRMToken(195)) - Create AMRMToken for ApplicationAttempt: appattempt_1564332457320_0020_000001
2019-07-28 22:39:11,069 INFO security.AMRMTokenSecretManager (AMRMTokenSecretManager.java:createPassword(307)) - Creating password for appattempt_1564332457320_0020_000001
2019-07-28 22:39:11,265 INFO amlauncher.AMLauncher (AMLauncher.java:launch(130)) - Done launching container Container: [ContainerId: container_e20_1564332457320_0020_01_000001, AllocationRequestId: -1, Version: 0, NodeId: sandbox-hdp.hortonworks.com:45454, NodeHttpAddress: sandbox-hdp.hortonworks.com:8042, Resource: , Priority: 0, Token: Token { kind: ContainerToken, service: 172.18.0.3:45454 }, ExecutionType: GUARANTEED, ] for AM appattempt_1564332457320_0020_000001
2019-07-28 22:39:11,265 INFO attempt.RMAppAttemptImpl (RMAppAttemptImpl.java:handle(925)) - appattempt_1564332457320_0020_000001 State change from ALLOCATED to LAUNCHED on event = LAUNCHED
2019-07-28 22:39:11,852 INFO resourcemanager.ResourceTrackerService (ResourceTrackerService.java:updateAppCollectorsMap(713)) - Update collector information for application application_1564332457320_0020 with new address: sandbox-hdp.hortonworks.com:35197 timestamp: 1564332457320, 36
2019-07-28 22:39:11,854 INFO rmcontainer.RMContainerImpl (RMContainerImpl.java:handle(490)) - container_e20_1564332457320_0020_01_000001 Container Transitioned from ACQUIRED to RUNNING
2019-07-28 22:39:12,833 INFO provider.BaseAuditHandler (BaseAuditHandler.java:logStatus(312)) - Audit Status Log: name=yarn.async.batch.hdfs, interval=01:11.979 minutes, events=162, succcessCount=162, totalEvents=17347, totalSuccessCount=17347
2019-07-28 22:39:12,834 INFO destination.HDFSAuditDestination (HDFSAuditDestination.java:logJSON(179)) - Flushing HDFS audit. Event Size:1
2019-07-28 22:39:12,857 INFO resourcemanager.ResourceTrackerService (ResourceTrackerService.java:updateAppCollectorsMap(713)) - Update collector information for application application_1564332457320_0020 with new address: sandbox-hdp.hortonworks.com:35197 timestamp: 1564332457320, 37
2019-07-28 22:39:14,054 INFO rmcontainer.RMContainerImpl (RMContainerImpl.java:handle(490)) - container_e20_1564332457320_0020_01_000001 Container Transitioned from RUNNING to COMPLETED
2019-07-28 22:39:14,055 INFO attempt.RMAppAttemptImpl (RMAppAttemptImpl.java:rememberTargetTransitionsAndStoreState(1412)) - Updating application attempt appattempt_1564332457320_0020_000001 with final state: FAILED, and exit status: -1000
2019-07-28 22:39:14,055 INFO attempt.RMAppAttemptImpl (RMAppAttemptImpl.java:handle(925)) - appattempt_1564332457320_0020_000001 State change from LAUNCHED to FINAL_SAVING on event = CONTAINER_FINISHED
2019-07-28 22:39:14,066 INFO resourcemanager.ApplicationMasterService (ApplicationMasterService.java:unregisterAttempt(496)) - Unregistering app attempt : appattempt_1564332457320_0020_000001
2019-07-28 22:39:14,066 INFO security.AMRMTokenSecretManager (AMRMTokenSecretManager.java:applicationMasterFinished(124)) - Application finished, removing password for appattempt_1564332457320_0020_000001
2019-07-28 22:39:14,066 INFO attempt.RMAppAttemptImpl (RMAppAttemptImpl.java:handle(925)) - appattempt_1564332457320_0020_000001 State change from FINAL_SAVING to FAILED on event = ATTEMPT_UPDATE_SAVED
2019-07-28 22:39:14,067 INFO rmapp.RMAppImpl (RMAppImpl.java:transition(1538)) - The number of failed attempts is 1. The max attempts is 2
2019-07-28 22:39:14,067 INFO resourcemanager.ApplicationMasterService (ApplicationMasterService.java:registerAppAttempt(479)) - Registering app attempt : appattempt_1564332457320_0020_000002
2019-07-28 22:39:14,067 INFO attempt.RMAppAttemptImpl (RMAppAttemptImpl.java:handle(925)) - appattempt_1564332457320_0020_000002 State change from NEW to SUBMITTED on event = START
2019-07-28 22:39:14,067 INFO capacity.CapacityScheduler (CapacityScheduler.java:doneApplicationAttempt(1085)) - Application Attempt appattempt_1564332457320_0020_000001 is done. finalState=FAILED
2019-07-28 22:39:14,067 INFO scheduler.AppSchedulingInfo (AppSchedulingInfo.java:clearRequests(159)) - Application application_1564332457320_0020 requests cleared
2019-07-28 22:39:14,067 INFO capacity.LeafQueue (LeafQueue.java:removeApplicationAttempt(1003)) - Application removed - appId: application_1564332457320_0020 user: santosh queue: santosh #user-pending-applications: 0 #user-active-applications: 0 #queue-pending-applications: 0 #queue-active-applications: 0
2019-07-28 22:39:14,068 INFO capacity.LeafQueue (LeafQueue.java:activateApplications(911)) - Application application_1564332457320_0020 from user: santosh activated in queue: santosh
2019-07-28 22:39:14,068 INFO capacity.LeafQueue (LeafQueue.java:addApplicationAttempt(941)) - Application added - appId: application_1564332457320_0020 user: santosh, leaf-queue: santosh #user-pending-applications: 0 #user-active-applications: 1 #queue-pending-applications: 0 #queue-active-applications: 1
2019-07-28 22:39:14,068 INFO capacity.CapacityScheduler (CapacityScheduler.java:addApplicationAttempt(1036)) - Added Application Attempt appattempt_1564332457320_0020_000002 to scheduler from user santosh in queue santosh
2019-07-28 22:39:14,068 INFO attempt.RMAppAttemptImpl (RMAppAttemptImpl.java:handle(925)) - appattempt_1564332457320_0020_000002 State change from SUBMITTED to SCHEDULED on event = ATTEMPT_ADDED
2019-07-28 22:39:14,074 INFO allocator.AbstractContainerAllocator (AbstractContainerAllocator.java:getCSAssignmentFromAllocateResult(129)) - assignedContainer application attempt=appattempt_1564332457320_0020_000002 container=null queue=santosh clusterResource= type=OFF_SWITCH requestedPartition=
2019-07-28 22:39:14,074 INFO rmcontainer.RMContainerImpl (RMContainerImpl.java:handle(490)) - container_e20_1564332457320_0020_02_000001 Container Transitioned from NEW to ALLOCATED
2019-07-28 22:39:14,075 INFO fica.FiCaSchedulerNode (FiCaSchedulerNode.java:allocateContainer(169)) - Assigned container container_e20_1564332457320_0020_02_000001 of capacity on host sandbox-hdp.hortonworks.com:45454, which has 1 containers, used and available after allocation
2019-07-28 22:39:14,075 INFO security.NMTokenSecretManagerInRM (NMTokenSecretManagerInRM.java:createAndGetNMToken(200)) - Sending NMToken for nodeId : sandbox-hdp.hortonworks.com:45454 for container : container_e20_1564332457320_0020_02_000001
2019-07-28 22:39:14,076 INFO rmcontainer.RMContainerImpl (RMContainerImpl.java:handle(490)) - container_e20_1564332457320_0020_02_000001 Container Transitioned from ALLOCATED to ACQUIRED
2019-07-28 22:39:14,076 INFO security.NMTokenSecretManagerInRM (NMTokenSecretManagerInRM.java:clearNodeSetForAttempt(146)) - Clear node set for appattempt_1564332457320_0020_000002
2019-07-28 22:39:14,076 INFO capacity.ParentQueue (ParentQueue.java:apply(1332)) - assignedContainer queue=root usedCapacity=0.25 absoluteUsedCapacity=0.25 used= cluster=
2019-07-28 22:39:14,076 INFO capacity.CapacityScheduler (CapacityScheduler.java:tryCommit(2890)) - Allocation proposal accepted
2019-07-28 22:39:14,076 INFO attempt.RMAppAttemptImpl (RMAppAttemptImpl.java:storeAttempt(2213)) - Storing attempt: AppId: application_1564332457320_0020 AttemptId: appattempt_1564332457320_0020_000002 MasterContainer: Container: [ContainerId: container_e20_1564332457320_0020_02_000001, AllocationRequestId: -1, Version: 0, NodeId: sandbox-hdp.hortonworks.com:45454, NodeHttpAddress: sandbox-hdp.hortonworks.com:8042, Resource: , Priority: 0, Token: Token { kind: ContainerToken, service: 172.18.0.3:45454 }, ExecutionType: GUARANTEED, ]
2019-07-28 22:39:14,077 INFO attempt.RMAppAttemptImpl (RMAppAttemptImpl.java:handle(925)) - appattempt_1564332457320_0020_000002 State change from SCHEDULED to ALLOCATED_SAVING on event = CONTAINER_ALLOCATED
2019-07-28 22:39:14,088 INFO attempt.RMAppAttemptImpl (RMAppAttemptImpl.java:handle(925)) - appattempt_1564332457320_0020_000002 State change from ALLOCATED_SAVING to ALLOCATED on event = ATTEMPT_NEW_SAVED
2019-07-28 22:39:14,089 INFO amlauncher.AMLauncher (AMLauncher.java:run(307)) - Launching masterappattempt_1564332457320_0020_000002
2019-07-28 22:39:14,091 INFO amlauncher.AMLauncher (AMLauncher.java:launch(109)) - Setting up container Container: [ContainerId: container_e20_1564332457320_0020_02_000001, AllocationRequestId: -1, Version: 0, NodeId: sandbox-hdp.hortonworks.com:45454, NodeHttpAddress: sandbox-hdp.hortonworks.com:8042, Resource: , Priority: 0, Token: Token { kind: ContainerToken, service: 172.18.0.3:45454 }, ExecutionType: GUARANTEED, ] for AM appattempt_1564332457320_0020_000002
2019-07-28 22:39:14,092 INFO security.AMRMTokenSecretManager (AMRMTokenSecretManager.java:createAndGetAMRMToken(195)) - Create AMRMToken for ApplicationAttempt: appattempt_1564332457320_0020_000002
2019-07-28 22:39:14,092 INFO security.AMRMTokenSecretManager (AMRMTokenSecretManager.java:createPassword(307)) - Creating password for appattempt_1564332457320_0020_000002
2019-07-28 22:39:14,110 INFO amlauncher.AMLauncher (AMLauncher.java:launch(130)) - Done launching container Container: [ContainerId: container_e20_1564332457320_0020_02_000001, AllocationRequestId: -1, Version: 0, NodeId: sandbox-hdp.hortonworks.com:45454, NodeHttpAddress: sandbox-hdp.hortonworks.com:8042, Resource: , Priority: 0, Token: Token { kind: ContainerToken, service: 172.18.0.3:45454 }, ExecutionType: GUARANTEED, ] for AM appattempt_1564332457320_0020_000002
2019-07-28 22:39:14,110 INFO attempt.RMAppAttemptImpl (RMAppAttemptImpl.java:handle(925)) - appattempt_1564332457320_0020_000002 State change from ALLOCATED to LAUNCHED on event = LAUNCHED
2019-07-28 22:39:15,056 INFO rmcontainer.RMContainerImpl (RMContainerImpl.java:handle(490)) - container_e20_1564332457320_0020_02_000001 Container Transitioned from ACQUIRED to RUNNING
2019-07-28 22:39:16,752 INFO rmcontainer.RMContainerImpl (RMContainerImpl.java:handle(490)) - container_e20_1564332457320_0020_02_000001 Container Transitioned from RUNNING to COMPLETED
2019-07-28 22:39:16,755 INFO attempt.RMAppAttemptImpl (RMAppAttemptImpl.java:rememberTargetTransitionsAndStoreState(1412)) - Updating application attempt appattempt_1564332457320_0020_000002 with final state: FAILED, and exit status: -1000
2019-07-28 22:39:16,755 INFO attempt.RMAppAttemptImpl (RMAppAttemptImpl.java:handle(925)) - appattempt_1564332457320_0020_000002 State change from LAUNCHED to FINAL_SAVING on event = CONTAINER_FINISHED
2019-07-28 22:39:16,899 INFO resourcemanager.ApplicationMasterService (ApplicationMasterService.java:unregisterAttempt(496)) - Unregistering app attempt : appattempt_1564332457320_0020_000002
2019-07-28 22:39:16,900 INFO security.AMRMTokenSecretManager (AMRMTokenSecretManager.java:applicationMasterFinished(124)) - Application finished, removing password for appattempt_1564332457320_0020_000002
2019-07-28 22:39:16,900 INFO attempt.RMAppAttemptImpl (RMAppAttemptImpl.java:handle(925)) - appattempt_1564332457320_0020_000002 State change from FINAL_SAVING to FAILED on event = ATTEMPT_UPDATE_SAVED
2019-07-28 22:39:16,900 INFO rmapp.RMAppImpl (RMAppImpl.java:transition(1538)) - The number of failed attempts is 2. The max attempts is 2
2019-07-28 22:39:16,900 INFO rmapp.RMAppImpl (RMAppImpl.java:rememberTargetTransitionsAndStoreState(1278)) - Updating application application_1564332457320_0020 with final state: FAILED
2019-07-28 22:39:16,900 INFO rmapp.RMAppImpl (RMAppImpl.java:handle(912)) - application_1564332457320_0020 State change from ACCEPTED to FINAL_SAVING on event = ATTEMPT_FAILED
2019-07-28 22:39:16,900 INFO recovery.RMStateStore (RMStateStore.java:transition(260)) - Updating info for app: application_1564332457320_0020
2019-07-28 22:39:16,900 INFO capacity.CapacityScheduler (CapacityScheduler.java:doneApplicationAttempt(1085)) - Application Attempt appattempt_1564332457320_0020_000002 is done. finalState=FAILED
2019-07-28 22:39:16,901 INFO scheduler.AppSchedulingInfo (AppSchedulingInfo.java:clearRequests(159)) - Application application_1564332457320_0020 requests cleared
2019-07-28 22:39:16,901 INFO capacity.LeafQueue (LeafQueue.java:removeApplicationAttempt(1003)) - Application removed - appId: application_1564332457320_0020 user: santosh queue: santosh #user-pending-applications: 0 #user-active-applications: 0 #queue-pending-applications: 0 #queue-active-applications: 0
2019-07-28 22:39:16,916 INFO rmapp.RMAppImpl (RMAppImpl.java:transition(1197)) - Application application_1564332457320_0020 failed 2 times due to AM Container for appattempt_1564332457320_0020_000002 exited with exitCode: -1000
Failing this attempt.Diagnostics: (Client.java:1558)
at org.apache.hadoop.ipc.Client.call(Client.java:1389)
... 37 more
Caused by: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
at org.apache.hadoop.security.SaslRpcClient.selectSaslClient(SaslRpcClient.java:173)
at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:390)
at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:614)
at org.apache.hadoop.ipc.Client$Connection.access$2300(Client.java:410)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:800)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:796)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:796)
... 40 more
Caused by: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
Configure GremlinServer to JanusGraph with HBase and Elasticsearch
Can't create instance of GremlinServer with HBase and Elasticsearch.
When i run shell script: bin/gremlin-server.sh config/gremlin.yaml. I get exception:
Exception in thread "main" java.lang.IllegalStateException: java.lang.NoSuchMethodException: org.janusgraph.graphdb.tinkerpop.plugin.JanusGraphGremlinPlugin.build()
Gremlin-server logs
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/user/janusgraph/lib/slf4j-log4j12-1.7.12.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/user/janusgraph/lib/logback-classic-1.1.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
0 [main] INFO org.apache.tinkerpop.gremlin.server.GremlinServer -
\,,,/
(o o)
-----oOOo-(3)-oOOo-----
135 [main] INFO org.apache.tinkerpop.gremlin.server.GremlinServer - Configuring Gremlin Server from config/gremlin.yaml
211 [main] INFO org.apache.tinkerpop.gremlin.server.util.MetricManager - Configured Metrics Slf4jReporter configured with interval=180000ms and loggerName=org.apache.tinkerpop.gremlin.server.Settings$Slf4jReporterMetrics
557 [main] INFO org.janusgraph.diskstorage.hbase.HBaseCompatLoader - Instantiated HBase compatibility layer supporting runtime HBase version 1.2.6: org.janusgraph.diskstorage.hbase.HBaseCompat1_0
835 [main] INFO org.janusgraph.diskstorage.hbase.HBaseStoreManager - HBase configuration: setting zookeeper.znode.parent=/hbase-unsecure
836 [main] INFO org.janusgraph.diskstorage.hbase.HBaseStoreManager - Copied host list from root.storage.hostname to hbase.zookeeper.quorum: main.local,data1.local,data2.local
836 [main] INFO org.janusgraph.diskstorage.hbase.HBaseStoreManager - Copied Zookeeper Port from root.storage.port to hbase.zookeeper.property.clientPort: 2181
866 [main] WARN org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
1214 [main] INFO org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper - Process identifier=hconnection-0x1e44b638 connecting to ZooKeeper ensemble=main.local:2181,data1.local:2181,data2.local:2181
1220 [main] INFO org.apache.hadoop.hbase.shaded.org.apache.zookeeper.ZooKeeper - Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
1220 [main] INFO org.apache.hadoop.hbase.shaded.org.apache.zookeeper.ZooKeeper - Client environment:host.name=main.local
1220 [main] INFO org.apache.hadoop.hbase.shaded.org.apache.zookeeper.ZooKeeper - Client environment:java.version=1.8.0_212
1220 [main] INFO org.apache.hadoop.hbase.shaded.org.apache.zookeeper.ZooKeeper - Client environment:java.vendor=Oracle Corporation
1220 [main] INFO org.apache.hadoop.hbase.shaded.org.apache.zookeeper.ZooKeeper - Client environment:java.home=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.212.b04-0.el7_6.x86_64/jre
1221 [main] INFO org.apache.hadoop.hbase.shaded.org.apache.zookeeper.ZooKeeper - Client environment:java.class.path=/home/user/janusgraph/conf/gremlin-server:/home/user/janusgraph/lib/slf4j-log4j12-
// Here hanusgraph download very many dependencies
1256 [main] INFO org.apache.hadoop.hbase.shaded.org.apache.zookeeper.ZooKeeper - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
1256 [main] INFO org.apache.hadoop.hbase.shaded.org.apache.zookeeper.ZooKeeper - Client environment:java.io.tmpdir=/tmp
1256 [main] INFO org.apache.hadoop.hbase.shaded.org.apache.zookeeper.ZooKeeper - Client environment:java.compiler=<NA>
1256 [main] INFO org.apache.hadoop.hbase.shaded.org.apache.zookeeper.ZooKeeper - Client environment:os.name=Linux
1256 [main] INFO org.apache.hadoop.hbase.shaded.org.apache.zookeeper.ZooKeeper - Client environment:os.arch=amd64
1256 [main] INFO org.apache.hadoop.hbase.shaded.org.apache.zookeeper.ZooKeeper - Client environment:os.version=3.10.0-862.el7.x86_64
1256 [main] INFO org.apache.hadoop.hbase.shaded.org.apache.zookeeper.ZooKeeper - Client environment:user.name=user
1257 [main] INFO org.apache.hadoop.hbase.shaded.org.apache.zookeeper.ZooKeeper - Client environment:user.home=/home/user
1257 [main] INFO org.apache.hadoop.hbase.shaded.org.apache.zookeeper.ZooKeeper - Client environment:user.dir=/home/user/janusgraph
1257 [main] INFO org.apache.hadoop.hbase.shaded.org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=main.local:2181,data1.local:2181,data2.local:2181 sessionTimeout=90000 watcher=hconnection-0x1e44b6380x0, quorum=main.local:2181,data1.local:2181,data2.local:2181, baseZNode=/hbase-unsecure
1274 [main-SendThread(data2.local:2181)] INFO org.apache.hadoop.hbase.shaded.org.apache.zookeeper.ClientCnxn - Opening socket connection to server data2.local/xxx.xxx.xxx.xxx:2181. Will not attempt to authenticate using SASL (unknown error)
1394 [main-SendThread(data2.local:2181)] INFO org.apache.hadoop.hbase.shaded.org.apache.zookeeper.ClientCnxn - Socket connection established to data2.local/xxx.xxx.xxx.xxx, initiating session
1537 [main-SendThread(data2.local:2181)] INFO org.apache.hadoop.hbase.shaded.org.apache.zookeeper.ClientCnxn - Session establishment complete on server data2.local/xxx.xxx.xxx.xxx:2181, sessionid = 0x26b266353e50014, negotiated timeout = 60000
3996 [main] INFO org.janusgraph.core.util.ReflectiveConfigOptionLoader - Loaded and initialized config classes: 13 OK out of 13 attempts in PT0.631S
4103 [main] INFO org.reflections.Reflections - Reflections took 60 ms to scan 2 urls, producing 0 keys and 0 values
4400 [main] WARN org.janusgraph.graphdb.configuration.GraphDatabaseConfiguration - Local setting cache.db-cache-time=180000 (Type: GLOBAL_OFFLINE) is overridden by globally managed value (10000). Use the ManagementSystem interface instead of the local configuration to control this setting.
4453 [main] WARN org.janusgraph.graphdb.configuration.GraphDatabaseConfiguration - Local setting cache.db-cache-clean-wait=20 (Type: GLOBAL_OFFLINE) is overridden by globally managed value (50). Use the ManagementSystem interface instead of the local configuration to control this setting.
4473 [main] INFO org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation - Closing master protocol: MasterService
4474 [main] INFO org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation - Closing zookeeper sessionid=0x26b266353e50014
4485 [main] INFO org.apache.hadoop.hbase.shaded.org.apache.zookeeper.ZooKeeper - Session: 0x26b266353e50014 closed
4485 [main-EventThread] INFO org.apache.hadoop.hbase.shaded.org.apache.zookeeper.ClientCnxn - EventThread shut down
4500 [main] INFO org.janusgraph.graphdb.configuration.GraphDatabaseConfiguration - Generated unique-instance-id=c0a8873843641-main-local1
4530 [main] INFO org.janusgraph.diskstorage.hbase.HBaseStoreManager - HBase configuration: setting zookeeper.znode.parent=/hbase-unsecure
4530 [main] INFO org.janusgraph.diskstorage.hbase.HBaseStoreManager - Copied host list from root.storage.hostname to hbase.zookeeper.quorum: main.local,data1.local,data2.local
4531 [main] INFO org.janusgraph.diskstorage.hbase.HBaseStoreManager - Copied Zookeeper Port from root.storage.port to hbase.zookeeper.property.clientPort: 2181
4532 [main] INFO org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper - Process identifier=hconnection-0x5bb3d42d connecting to ZooKeeper ensemble=main.local:2181,data1.local:2181,data2.local:2181
4532 [main] INFO org.apache.hadoop.hbase.shaded.org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=main.local:2181,data1.local:2181,data2.local:2181 sessionTimeout=90000 watcher=hconnection-0x5bb3d42d0x0, quorum=main.local:2181,data1.local:2181,data2.local:2181, baseZNode=/hbase-unsecure
4534 [main-SendThread(main.local:2181)] INFO org.apache.hadoop.hbase.shaded.org.apache.zookeeper.ClientCnxn - Opening socket connection to server main.local/xxx.xxx.xxx.xxx:2181. Will not attempt to authenticate using SASL (unknown error)
4534 [main-SendThread(main.local:2181)] INFO org.apache.hadoop.hbase.shaded.org.apache.zookeeper.ClientCnxn - Socket connection established to main.local/xxx.xxx.xxx.xxx:2181, initiating session
4611 [main-SendThread(main.local:2181)] INFO org.apache.hadoop.hbase.shaded.org.apache.zookeeper.ClientCnxn - Session establishment complete on server main.local/xxx.xxx.xxx.xxx:2181, sessionid = 0x36b266353fd0021, negotiated timeout = 60000
4616 [main] INFO org.janusgraph.diskstorage.Backend - Configuring index [search]
5781 [main] INFO org.janusgraph.diskstorage.Backend - Initiated backend operations thread pool of size 16
6322 [main] INFO org.janusgraph.diskstorage.Backend - Configuring total store cache size: 186687592
7555 [main] INFO org.janusgraph.graphdb.database.IndexSerializer - Hashing index keys
7925 [main] INFO org.janusgraph.diskstorage.log.kcvs.KCVSLog - Loaded unidentified ReadMarker start time 2019-06-13T09:54:08.929Z into org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller#656d10a4
7927 [main] INFO org.apache.tinkerpop.gremlin.server.GremlinServer - Graph [graph] was successfully configured via [config/db.properties].
7927 [main] INFO org.apache.tinkerpop.gremlin.server.util.ServerGremlinExecutor - Initialized Gremlin thread pool. Threads in pool named with pattern gremlin-*
Exception in thread "main" java.lang.IllegalStateException: java.lang.NoSuchMethodException: org.janusgraph.graphdb.tinkerpop.plugin.JanusGraphGremlinPlugin.build()
at org.apache.tinkerpop.gremlin.groovy.engine.GremlinExecutor.initializeGremlinScriptEngineManager(GremlinExecutor.java:522)
at org.apache.tinkerpop.gremlin.groovy.engine.GremlinExecutor.<init>(GremlinExecutor.java:126)
at org.apache.tinkerpop.gremlin.groovy.engine.GremlinExecutor.<init>(GremlinExecutor.java:83)
at org.apache.tinkerpop.gremlin.groovy.engine.GremlinExecutor$Builder.create(GremlinExecutor.java:813)
at org.apache.tinkerpop.gremlin.server.util.ServerGremlinExecutor.<init>(ServerGremlinExecutor.java:169)
at org.apache.tinkerpop.gremlin.server.util.ServerGremlinExecutor.<init>(ServerGremlinExecutor.java:89)
at org.apache.tinkerpop.gremlin.server.GremlinServer.<init>(GremlinServer.java:110)
at org.apache.tinkerpop.gremlin.server.GremlinServer.main(GremlinServer.java:363)
Caused by: java.lang.NoSuchMethodException: org.janusgraph.graphdb.tinkerpop.plugin.JanusGraphGremlinPlugin.build()
at java.lang.Class.getMethod(Class.java:1786)
at org.apache.tinkerpop.gremlin.groovy.engine.GremlinExecutor.initializeGremlinScriptEngineManager(GremlinExecutor.java:492)
... 7 more
Graph configuration:
storage.backend=hbase
storage.hostname=main.local,data1.local,data2.local
storage.port=2181
storage.hbase.ext.zookeeper.znode.parent=/hbase-unsecure
cache.db-cache=true
cache.db-cache-clean-wait=20
cache.db-cache-time=180000
cache.db-cache-size=0.5
index.search.backend=elasticsearch
index.search.hostname=xxx.xxx.xxx.xxx
index.search.port=9200
index.search.elasticsearch.client-only=false
gremlin.graph=org.janusgraph.core.JanusGraphFactory
host=0.0.0.0
Gremlin-server configuration
host: localhost
port: 8182
channelizer: org.apache.tinkerpop.gremlin.server.channel.HttpChannelizer
graphs: { graph: config/db.properties }
scriptEngines: {
gremlin-groovy: {
plugins: {
org.janusgraph.graphdb.tinkerpop.plugin.JanusGraphGremlinPlugin: {},
org.apache.tinkerpop.gremlin.server.jsr223.GremlinServerGremlinPlugin: {},
org.apache.tinkerpop.gremlin.tinkergraph.jsr223.TinkerGraphGremlinPlugin: {},
org.apache.tinkerpop.gremlin.jsr223.ImportGremlinPlugin: { classImports: [java.lang.Math], methodImports: [java.lang.Math#*] },
org.apache.tinkerpop.gremlin.jsr223.ScriptFileGremlinPlugin: { files: [scripts/janusgraph.groovy] }
}
}
}
serializers:
- { className: org.apache.tinkerpop.gremlin.driver.ser.GryoMessageSerializerV3d0, config: { ioRegistries: [org.janusgraph.graphdb.tinkerpop.JanusGraphIoRegistry] } }
- { className: org.apache.tinkerpop.gremlin.driver.ser.GryoMessageSerializerV3d0, config: { serializeResultToString: true } }
- { className: org.apache.tinkerpop.gremlin.driver.ser.GraphSONMessageSerializerV3d0, config: { ioRegistries: [org.janusgraph.graphdb.tinkerpop.JanusGraphIoRegistry] } }
metrics: {
slf4jReporter: {enabled: true, interval: 180000}
}
What do I need to do to server start without error?
Changing default consul http port
I need to change the default http port because another application is using 8500 already.
This command works:
consul info -http-addr=http://127.0.0.1:18500
I can't figure out what config setting this equals to in a config file.
Here are my current settings:
datacenter = "test_test"
data_dir = "/opt/consul"
encrypt = "**********"
performance {
raft_multiplier = 1
}
ports {
http = 18500
dns = 18600
server = 18300
}
addresses {
http = "127.0.0.1"
}
retry_join = ["10.60.0.5"]`
Error message when I run the join or info command:
Error querying agent: Get http://127.0.0.1:8500/v1/agent/self: dial tcp 127.0.0.1:8500: connect: connection refused
If you use:
{
"ports": {
"http": 18500
}
}
Then consul by default will bind to localhost:
==> Log data will now stream in as it occurs:
2019/02/19 17:28:23 [INFO] raft: Initial configuration (index=1): [{Suffrage:Voter ID:4887467c-c84b-15b4-66f7-ad3f822631e0 Address:172.17.0.2:8300}]
2019/02/19 17:28:23 [INFO] raft: Node at 172.17.0.2:8300 [Follower] entering Follower state (Leader: "")
2019/02/19 17:28:23 [INFO] serf: EventMemberJoin: b884fe85d115.dc1 172.17.0.2
2019/02/19 17:28:23 [INFO] serf: EventMemberJoin: b884fe85d115 172.17.0.2
2019/02/19 17:28:24 [INFO] consul: Adding LAN server b884fe85d115 (Addr: tcp/172.17.0.2:8300) (DC: dc1)
2019/02/19 17:28:24 [INFO] consul: Handled member-join event for server "b884fe85d115.dc1" in area "wan"
2019/02/19 17:28:24 [WARN] agent/proxy: running as root, will not start managed proxies
2019/02/19 17:28:24 [INFO] agent: Started DNS server 127.0.0.1:8600 (tcp)
2019/02/19 17:28:24 [INFO] agent: Started DNS server 127.0.0.1:8600 (udp)
2019/02/19 17:28:24 [INFO] agent: Started HTTP server on 127.0.0.1:18500 (tcp)
Obviously, anyone from other nodes can't connect to your bootstrap server.
You should configure address and port:
{
"addresses": {
"http": "0.0.0.0"
},
"ports": {
"http": 18500
}
}
Now you can see binding for any IP 0.0.0.0/0
==> Log data will now stream in as it occurs:
2019/02/19 17:35:11 [INFO] raft: Initial configuration (index=1): [{Suffrage:Voter ID:ef42f35f-7505-d1fc-3f91-16f144d91fc6 Address:172.17.0.2:8300}]
2019/02/19 17:35:11 [INFO] raft: Node at 172.17.0.2:8300 [Follower] entering Follower state (Leader: "")
2019/02/19 17:35:11 [INFO] serf: EventMemberJoin: ac34230483e0.dc1 172.17.0.2
2019/02/19 17:35:11 [INFO] serf: EventMemberJoin: ac34230483e0 172.17.0.2
2019/02/19 17:35:11 [INFO] agent: Started DNS server 127.0.0.1:8600 (udp)
2019/02/19 17:35:11 [WARN] agent/proxy: running as root, will not start managed proxies
2019/02/19 17:35:11 [INFO] consul: Adding LAN server ac34230483e0 (Addr: tcp/172.17.0.2:8300) (DC: dc1)
2019/02/19 17:35:11 [INFO] agent: Started DNS server 127.0.0.1:8600 (tcp)
2019/02/19 17:35:11 [INFO] consul: Handled member-join event for server "ac34230483e0.dc1" in area "wan"
2019/02/19 17:35:11 [INFO] agent: Started HTTP server on [::]:18500 (tcp)
Storm drpc server does not accept spout request
I am using apache storm 0.9.3 in Ubuntu 14.04. I put zookeeper, nimubus, drpc,supervisor, ui, worker in same box. From ui, it lookes fine:
I have storm.yaml configureation as follows:
storm.zookeeper.servers:
- "localhost"
storm.zookeeper.port: 2181
nimbus.host: "localhost"
storm.local.dir: "/var/stormtmp"
java.library.path: "/usr/local/lib"
supervisor.slots.ports:
-6700
-6701
-6702
worker.childopts: "-Xmx768m"
nimbus.childopts: "-Xmx512m"
supervisor.childopts: "-Xmx256m"
drpc.servers:
- "localhost"
Then, my java client DRPC call as follows: "callstatio" is the topology name in storm UI.
public static void main(String[] args) throws TException, DRPCExecutionException {
System.out.println("Entering main in TestSpout");
String host = "127.0.0.1";
DRPCClient client = new DRPCClient(host, 3772);
System.out.println("host is:"+host);
String result = client.execute("callstatio","hello world");
System.out.println("result is:"+result);
}
When I run the Client:
I could not see any request happens in drpc.log, neither there is exception.
Any hints why I could not get drpc server working?
The following is from tail -f drpc.log
2015-03-25T03:50:56.842-0400 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:user.home=/root
2015-03-25T03:50:56.842-0400 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:user.dir=/home/juhani/storm/apache-storm-0.9.3/bin
2015-03-25T03:50:57.293-0400 b.s.d.drpc [INFO] Starting Distributed RPC servers...
2015-03-25T04:09:27.331-0400 b.s.d.drpc [WARN] Timeout DRPC request id: 1 start at 1427270366
2015-03-25T04:11:22.337-0400 b.s.d.drpc [WARN] Timeout DRPC request id: 2 start at 1427270477
2015-03-25T04:13:42.342-0400 b.s.d.drpc [WARN] Timeout DRPC request id: 3 start at 1427270620
2015-03-25T04:16:32.349-0400 b.s.d.drpc [WARN] Timeout DRPC request id: 4 start at 1427270791
2015-03-25T04:20:52.358-0400 b.s.d.drpc [WARN] Timeout DRPC request id: 5 start at 1427271047
2015-03-25T04:23:07.373-0400 b.s.d.drpc [WARN] Timeout DRPC request id: 6 start at 1427271183
2015-03-25T04:25:27.377-0400 b.s.d.drpc [WARN] Timeout DRPC request id: 7 start at 1427271325
TestDFSIO fails with exitcode -1000
I setup a two node hadoop cluster. After having started the cluster it looks like this:
machine namenode:
hadoop#namenode:~$ jps
5691 Jps
3531 DataNode
3424 NameNode
3669 SecondaryNameNode
3822 ResourceManager
3908 NodeManager
second machine datanode:
hadoop#datanode:~$ jps
3716 Jps
2137 DataNode
2231 NodeManager
So, after having started the cluster I tried to perform a standard benchmark:
hadoop jar /opt/hadoop-2.2.0/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.2.0-tests.jar TestDFSIO -write -nrFiles 20 -fileSize 10
However the job fails and the config file contain the following messages:
On the datanode:
hadoop#datanode:~$ cat /opt/hadoop-2.2.0/logs/yarn-hadoop-nodemanager-datanode.log
...
2014-02-18 16:37:41,567 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 3547 for container-id container_1392741263071_0001_02_000001: 26.2 MB of 2 GB physical memory used; 1.2 GB of 4.2 GB virtual memory used
2014-02-18 16:37:42,158 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Sending out status for container: container_id { app_attempt_id { application_id { id: 1 cluster_timestamp: 1392741263071 } attemptId: 2 } id: 1 } state: C_RUNNING diagnostics: "" exit_status: -1000
2014-02-18 16:37:43,166 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Sending out status for container: container_id { app_attempt_id { application_id { id: 1 cluster_timestamp: 1392741263071 } attemptId: 2 } id: 1 } state: C_RUNNING diagnostics: "" exit_status: -1000
2014-02-18 16:37:44,171 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Sending out status for container: container_id { app_attempt_id { application_id { id: 1 cluster_timestamp: 1392741263071 } attemptId: 2 } id: 1 } state: C_RUNNING diagnostics: "" exit_status: -1000
2014-02-18 16:37:44,579 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 3547 for container-id container_1392741263071_0001_02_000001: 95.3 MB of 2 GB physical memory used; 1.3 GB of 4.2 GB virtual memory used
2014-02-18 16:37:45,180 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Sending out status for container: container_id { app_attempt_id { application_id { id: 1 cluster_timestamp: 1392741263071 } attemptId: 2 } id: 1 } state: C_RUNNING diagnostics: "" exit_status: -1000
2014-02-18 16:37:46,183 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Sending out status for container: container_id { app_attempt_id { application_id { id: 1 cluster_timestamp: 1392741263071 } attemptId: 2 } id: 1 } state: C_RUNNING diagnostics: "" exit_status: -1000
2014-02-18 16:37:47,189 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Sending out status for container: container_id { app_attempt_id { application_id { id: 1 cluster_timestamp: 1392741263071 } attemptId: 2 } id: 1 } state: C_RUNNING diagnostics: "" exit_status: -1000
2014-02-18 16:37:47,584 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 3547 for container-id container_1392741263071_0001_02_000001: 108.1 MB of 2 GB physical memory used; 1.3 GB of 4.2 GB virtual memory used
2014-02-18 16:37:48,196 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Sending out status for container: container_id { app_attempt_id { application_id { id: 1 cluster_timestamp: 1392741263071 } attemptId: 2 } id: 1 } state: C_RUNNING diagnostics: "" exit_status: -1000
2014-02-18 16:37:49,157 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exit code from container container_1392741263071_0001_02_000001 is : 1
2014-02-18 16:37:49,157 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exception from container-launch with container ID: container_1392741263071_0001_02_000001 and exit code: 1
org.apache.hadoop.util.Shell$ExitCodeException:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:464)
at org.apache.hadoop.util.Shell.run(Shell.java:379)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
2014-02-18 16:37:49,159 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor:
2014-02-18 16:37:49,159 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Container exited with a non-zero exit code 1
2014-02-18 16:37:49,160 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: Container container_1392741263071_0001_02_000001 transitioned from RUNNING to EXITED_WITH_FAILURE
2014-02-18 16:37:49,160 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Cleaning up container container_1392741263071_0001_02_000001
2014-02-18 16:37:49,172 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /home/hadoop/hadoop/yarn-data/usercache/hadoop/appcache/application_1392741263071_0001/container_1392741263071_0001_02_000001
2014-02-18 16:37:49,173 WARN org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hadoop OPERATION=Container Finished - Failed TARGET=ContainerImpl RESULT=FAILURE DESCRIPTION=Container failed with state: EXITED_WITH_FAILURE APPID=application_1392741263071_0001 CONTAINERID=container_1392741263071_0001_02_000001
...
On the namenode:
hadoop#namenode:/opt/hadoop-2.2.0/logs$ cat yarn-hadoop-*.log
2014-02-18 16:34:25,054 INFO org.apache.hadoop.yarn.server.nodemanager.NodeManager: STARTUP_MSG:
...
2014-02-18 16:37:37,441 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 4493 for container-id container_1392741263071_0001_01_000001: 131.1 MB of 2 GB physical memory used; 1.4 GB of 4.2 GB virtual memory used
2014-02-18 16:37:38,367 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Sending out status for container: container_id { app_attempt_id { application_id { id: 1 cluster_timestamp: 1392741263071 } attemptId: 1 } id: 1 } state: C_RUNNING diagnostics: "" exit_status: -1000
2014-02-18 16:37:39,369 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Sending out status for container: container_id { app_attempt_id { application_id { id: 1 cluster_timestamp: 1392741263071 } attemptId: 1 } id: 1 } state: C_RUNNING diagnostics: "" exit_status: -1000
...
2014-02-18 16:34:23,131 INFO org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: STARTUP_MSG:
...
2014-02-18 16:37:49,186 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerNode: Released container container_1392741263071_0001_02_000001 of capacity <memory:2048, vCores:1> on host datanode.c.forward-camera-473.internal:43994, which currently has 0 containers, <memory:0, vCores:0> used and <memory:8192, vCores:8> available, release resources=true
2014-02-18 16:37:49,186 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: default used=<memory:0, vCores:0> numContainers=0 user=hadoop user-resources=<memory:0, vCores:0>
2014-02-18 16:37:49,186 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: completedContainer container=Container: [ContainerId: container_1392741263071_0001_02_000001, NodeId: datanode.c.forward-camera-473.internal:43994, NodeHttpAddress: datanode.c.forward-camera-473.internal:8042, Resource: <memory:2048, vCores:1>, Priority: 0, Token: Token { kind: ContainerToken, service: 10.240.110.76:43994 }, ] resource=<memory:2048, vCores:1> queue=default: capacity=1.0, absoluteCapacity=1.0, usedResources=<memory:0, vCores:0>usedCapacity=0.0, absoluteUsedCapacity=0.0, numApps=1, numContainers=0 usedCapacity=0.0 absoluteUsedCapacity=0.0 used=<memory:0, vCores:0> cluster=<memory:16384, vCores:16>
2014-02-18 16:37:49,186 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: completedContainer queue=root usedCapacity=0.0 absoluteUsedCapacity=0.0 used=<memory:0, vCores:0> cluster=<memory:16384, vCores:16>
2014-02-18 16:37:49,186 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: Re-sorting completed queue: root.default stats: default: capacity=1.0, absoluteCapacity=1.0, usedResources=<memory:0, vCores:0>usedCapacity=0.0, absoluteUsedCapacity=0.0, numApps=1, numContainers=0
2014-02-18 16:37:49,186 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Application appattempt_1392741263071_0001_000002 released container container_1392741263071_0001_02_000001 on node: host: datanode.c.forward-camera-473.internal:43994 #containers=0 available=8192 used=0 with event: FINISHED
2014-02-18 16:37:49,187 INFO org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService: Unregistering app attempt : appattempt_1392741263071_0001_000002
2014-02-18 16:37:49,187 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: appattempt_1392741263071_0001_000002 State change from RUNNING to FAILED
2014-02-18 16:37:49,187 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: Application application_1392741263071_0001 failed 2 times due to AM Container for appattempt_1392741263071_0001_000002 exited with exitCode: 1 due to: Exception from container-launch:
org.apache.hadoop.util.Shell$ExitCodeException:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:464)
at org.apache.hadoop.util.Shell.run(Shell.java:379)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
.Failing this attempt.. Failing the application.
2014-02-18 16:37:49,189 INFO org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore: Removing info for app: application_1392741263071_0001
2014-02-18 16:37:49,194 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: application_1392741263071_0001 State change from RUNNING to FAILED
2014-02-18 16:37:49,194 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Application appattempt_1392741263071_0001_000002 is done. finalState=FAILED
2014-02-18 16:37:49,194 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo: Application application_1392741263071_0001 requests cleared
2014-02-18 16:37:49,194 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: Application removed - appId: application_1392741263071_0001 user: hadoop queue: default #user-pending-applications: 0 #user-active-applications: 0 #queue-pending-applications: 0 #queue-active-applications: 0
2014-02-18 16:37:49,194 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: Application removed - appId: application_1392741263071_0001 user: hadoop leaf-queue of parent: root #applications: 0
2014-02-18 16:37:49,204 WARN org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=hadoop OPERATION=Application Finished - Failed TARGET=RMAppManager RESULT=FAILURE DESCRIPTION=App failed with state: FAILED PERMISSIONS=Application application_1392741263071_0001 failed 2 times due to AM Container for appattempt_1392741263071_0001_000002 exited with exitCode: 1 due to: Exception from container-launch:
org.apache.hadoop.util.Shell$ExitCodeException:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:464)
at org.apache.hadoop.util.Shell.run(Shell.java:379)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
.Failing this attempt.. Failing the application. APPID=application_1392741263071_0001
2014-02-18 16:37:49,205 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAppManager$ApplicationSummary: appId=application_1392741263071_0001,name=hadoop-mapreduce-client-jobclient-2.2.0-tests.jar,user=hadoop,queue=default,state=FAILED,trackingUrl=namenode:8088/cluster/app/application_1392741263071_0001,appMasterHost=,startTime=1392741381131,finishTime=1392741469188,finalStatus=FAILED
2014-02-18 16:37:49,205 INFO org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher: Cleaning master appattempt_1392741263071_0001_000002
What is happening?
Look like it can't spawn new java process. Probably your .profile or .bashrc do not setup JAVA_HOME or PATH correctly, and thus the java executable is not accessible.