I am trying to run a script to install some features using puppet but I am getting a weird error (exactly a NullPointerException). However, if I run the script from my user in the machine everything runs correctly. I am using Apache Karaf 3.0.0 and Puppet 2.7.
My script is pretty simple:
#!/bin/bash
####
#### A init script for Karaf
####
# Repos
for REPO in <%= repos.sort.join(' ') %>
do
/opt/karaf/bin/client -a 8101 -h 127.0.0.1 -v feature:repo-add $REPO
done
# Features
for FEATURE in <%= features.sort.join(' ') %>
do
/opt/karaf/bin/client -a 8101 -h 127.0.0.1 -v feature:install $FEATURE
done
For running it with Puppet I am just using the following:
exec {
'/opt/karaf/bin/init.sh' :
user => 'root',
path => ["/usr/bin", "/usr/sbin"],
logoutput => true,
require => [Service['karaf'],File['/opt/karaf/bin/init.sh']];
}
This is the output I get when running Puppet in the machine:
info: /Stage[main]/Karaf/File[/opt/karaf/bin/init.sh]: Filebucketed /opt/karaf/bin/init.sh to puppet with sum 7687821cc06d43754a4f44aa9b9a69ff
notice: /Stage[main]/Karaf/File[/opt/karaf/bin/init.sh]/content: content changed '{md5}7687821cc06d43754a4f44aa9b9a69ff' to '{md5}6bb98d2e1079c66a7b9e16fff5ec1860'
notice: /Stage[main]/Karaf/Exec[/opt/karaf/bin/init.sh]/returns: 30 [main] INFO org.apache.sshd.common.util.SecurityUtils - BouncyCastle not registered, using the default JCE provider
notice: /Stage[main]/Karaf/Exec[/opt/karaf/bin/init.sh]/returns: 519 [pool-2-thread-1] INFO org.apache.sshd.client.session.ClientSessionImpl - Session created...
notice: /Stage[main]/Karaf/Exec[/opt/karaf/bin/init.sh]/returns: java.lang.NullPointerException
notice: /Stage[main]/Karaf/Exec[/opt/karaf/bin/init.sh]/returns: at org.apache.karaf.client.Main.main(Main.java:83)
notice: /Stage[main]/Karaf/Exec[/opt/karaf/bin/init.sh]/returns: 20 [main] INFO org.apache.sshd.common.util.SecurityUtils - BouncyCastle not registered, using the default JCE provider
notice: /Stage[main]/Karaf/Exec[/opt/karaf/bin/init.sh]/returns: 386 [pool-2-thread-1] INFO org.apache.sshd.client.session.ClientSessionImpl - Session created...
notice: /Stage[main]/Karaf/Exec[/opt/karaf/bin/init.sh]/returns: java.lang.NullPointerException
notice: /Stage[main]/Karaf/Exec[/opt/karaf/bin/init.sh]/returns: at org.apache.karaf.client.Main.main(Main.java:83)
notice: /Stage[main]/Karaf/Exec[/opt/karaf/bin/init.sh]/returns: 27 [main] INFO org.apache.sshd.common.util.SecurityUtils - BouncyCastle not registered, using the default JCE provider
notice: /Stage[main]/Karaf/Exec[/opt/karaf/bin/init.sh]/returns: 497 [pool-2-thread-1] INFO org.apache.sshd.client.session.ClientSessionImpl - Session created...
notice: /Stage[main]/Karaf/Exec[/opt/karaf/bin/init.sh]/returns: java.lang.NullPointerException
However, if I run it under my user or root:
30 [main] INFO org.apache.sshd.common.util.SecurityUtils - BouncyCastle not registered, using the default JCE provider
550 [pool-2-thread-1] INFO org.apache.sshd.client.session.ClientSessionImpl - Session created...
561 [pool-2-thread-1] INFO org.apache.sshd.client.session.ClientSessionImpl - Server version string: SSH-2.0-SSHD-CORE-0.9.0
Logging in as karaf
564 [pool-2-thread-1] INFO org.apache.sshd.client.session.ClientSessionImpl - Received SSH_MSG_KEXINIT
592 [pool-2-thread-1] INFO org.apache.sshd.client.kex.DHG1 - Send SSH_MSG_KEXDH_INIT
602 [pool-2-thread-2] INFO org.apache.sshd.client.kex.DHG1 - Received SSH_MSG_KEXDH_REPLY
619 [pool-2-thread-2] WARN org.apache.sshd.client.keyverifier.AcceptAllServerKeyVerifier - Server at /127.0.0.1:8101 presented unverified key:
620 [pool-2-thread-2] INFO org.apache.sshd.client.session.ClientSessionImpl - Received SSH_MSG_NEWKEYS
625 [pool-2-thread-2] INFO org.apache.sshd.client.session.ClientSessionImpl - Send SSH_MSG_SERVICE_REQUEST for ssh-userauth
631 [main] INFO org.apache.sshd.client.auth.UserAuthAgent - Send SSH_MSG_USERAUTH_REQUEST for publickey
643 [pool-2-thread-5] INFO org.apache.sshd.client.auth.UserAuthAgent - Received SSH_MSG_USERAUTH_SUCCESS
675 [main] INFO org.apache.sshd.client.channel.ChannelExec - Send SSH_MSG_CHANNEL_OPEN on channel 101
678 [pool-2-thread-2] INFO org.apache.sshd.client.channel.ChannelExec - Send SSH_MSG_CHANNEL_REQUEST exec
Adding feature url mvn:io.hawt/hawtio-karaf/1.2.3/xml/features
936 [pool-2-thread-5] INFO org.apache.sshd.client.channel.ChannelExec - Received SSH_MSG_CHANNEL_REQUEST on channel 101
29 [main] INFO org.apache.sshd.common.util.SecurityUtils - BouncyCastle not registered, using the default JCE provider
539 [pool-2-thread-1] INFO org.apache.sshd.client.session.ClientSessionImpl - Session created...
Logging in as karaf
552 [pool-2-thread-1] INFO org.apache.sshd.client.session.ClientSessionImpl - Server version string: SSH-2.0-SSHD-CORE-0.9.0
554 [pool-2-thread-1] INFO org.apache.sshd.client.session.ClientSessionImpl - Received SSH_MSG_KEXINIT
579 [pool-2-thread-1] INFO org.apache.sshd.client.kex.DHG1 - Send SSH_MSG_KEXDH_INIT
592 [pool-2-thread-2] INFO org.apache.sshd.client.kex.DHG1 - Received SSH_MSG_KEXDH_REPLY
609 [pool-2-thread-2] WARN org.apache.sshd.client.keyverifier.AcceptAllServerKeyVerifier - Server at /127.0.0.1:8101 presented unverified key:
609 [pool-2-thread-2] INFO org.apache.sshd.client.session.ClientSessionImpl - Received SSH_MSG_NEWKEYS
615 [pool-2-thread-2] INFO org.apache.sshd.client.session.ClientSessionImpl - Send SSH_MSG_SERVICE_REQUEST for ssh-userauth
620 [main] INFO org.apache.sshd.client.auth.UserAuthAgent - Send SSH_MSG_USERAUTH_REQUEST for publickey
631 [pool-2-thread-5] INFO org.apache.sshd.client.auth.UserAuthAgent - Received SSH_MSG_USERAUTH_SUCCESS
668 [main] INFO org.apache.sshd.client.channel.ChannelExec - Send SSH_MSG_CHANNEL_OPEN on channel 101
670 [pool-2-thread-2] INFO org.apache.sshd.client.channel.ChannelExec - Send SSH_MSG_CHANNEL_REQUEST exec
Adding feature url mvn:org.apache.camel.karaf/apache-camel/2.13.0/xml/features
985 [pool-2-thread-5] INFO org.apache.sshd.client.channel.ChannelExec - Received SSH_MSG_CHANNEL_REQUEST on channel 101
27 [main] INFO org.apache.sshd.common.util.SecurityUtils - BouncyCastle not registered, using the default JCE provider
515 [pool-2-thread-1] INFO org.apache.sshd.client.session.ClientSessionImpl - Session created...
526 [pool-2-thread-1] INFO org.apache.sshd.client.session.ClientSessionImpl - Server version string: SSH-2.0-SSHD-CORE-0.9.0
Logging in as karaf
528 [pool-2-thread-1] INFO org.apache.sshd.client.session.ClientSessionImpl - Received SSH_MSG_KEXINIT
553 [pool-2-thread-1] INFO org.apache.sshd.client.kex.DHG1 - Send SSH_MSG_KEXDH_INIT
562 [pool-2-thread-2] INFO org.apache.sshd.client.kex.DHG1 - Received SSH_MSG_KEXDH_REPLY
578 [pool-2-thread-2] WARN org.apache.sshd.client.keyverifier.AcceptAllServerKeyVerifier - Server at /127.0.0.1:8101 presented unverified key:
579 [pool-2-thread-2] INFO org.apache.sshd.client.session.ClientSessionImpl - Received SSH_MSG_NEWKEYS
584 [pool-2-thread-2] INFO org.apache.sshd.client.session.ClientSessionImpl - Send SSH_MSG_SERVICE_REQUEST for ssh-userauth
590 [main] INFO org.apache.sshd.client.auth.UserAuthAgent - Send SSH_MSG_USERAUTH_REQUEST for publickey
602 [pool-2-thread-5] INFO org.apache.sshd.client.auth.UserAuthAgent - Received SSH_MSG_USERAUTH_SUCCESS
630 [main] INFO org.apache.sshd.client.channel.ChannelExec - Send SSH_MSG_CHANNEL_OPEN on channel 101
632 [pool-2-thread-2] INFO org.apache.sshd.client.channel.ChannelExec - Send SSH_MSG_CHANNEL_REQUEST exec
666 [pool-2-thread-4] INFO org.apache.sshd.client.channel.ChannelExec - Received SSH_MSG_CHANNEL_REQUEST on channel 101
Is it Puppet doing something I am missing? Is there a bug in client for Karaf 3.0.0? Thanks and sorry for the long post!
It seems that this got fixed in 3.0.1. See the folllowing Jira report for details:
client fails with NullPointerException
I don't know much about puppet. Is puppet trying to grab what karaf client spits out in the console. If so then a null pointer exception is very likely. I haven't figured out a way to capture output from karaf client. Here's something I posted on SO related to this.
It is a bug in Karaf 3.0.0 and it has been fixed in Karaf 3.0.1
Related
Able to establish APM connection
2021-03-09 17:45:05,741 [Attach Listener] INFO co.elastic.apm.agent.configuration.StartupInfo - VM Arguments: [-XX:TieredStopAtLevel=1, -Xmx6g, -Dfile.encoding=UTF-8, -Duser.country=IN, -Duser.language=en, -Duser.variant]
2021-03-09 17:45:08,192 [Attach Listener] INFO co.elastic.apm.agent.impl.ElasticApmTracer - Tracer switched to RUNNING state
2021-03-09 17:45:08,734 [elastic-apm-server-healthcheck] INFO co.elastic.apm.agent.report.ApmServerHealthChecker - Elastic APM server is available: { "build_date": "2021-02-15T12:37:48Z", "build_sha": "e77061bb3aaedae5ae8dd0ca193eb662513aedde", "version": "7.11.0"}
But post connection, it still throws this error. What could be wrong here, appreciate any inputs on this
2021-03-09 17:45:53,484 [elastic-apm-server-reporter] INFO co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Backing off for 0 seconds (+/-10%)
2021-03-09 17:45:53,489 [elastic-apm-server-reporter] ERROR co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Error sending data to APM server: Read timed out, response code is -1
2021-03-09 17:45:53,489 [elastic-apm-server-reporter] WARN co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - null
2021-03-09 17:46:08,890 [elastic-apm-server-reporter] INFO co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Backing off for 1 seconds (+/-10%)
2021-03-09 17:46:09,922 [elastic-apm-server-reporter] ERROR co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Error sending data to APM server: Read timed out, response code is -1
2021-03-09 17:46:09,922 [elastic-apm-server-reporter] WARN co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - null
Check your kibana.yml file URLs, when I setup APM I my machine some of my URLs (elasticsearch.hosts, xpack.fleet.outputs) were defaulted to my current IP address (instead of localhost), which changed after a reboot.
I'm trying to setup JanusGraph for development on my local machine. My goal is to have a setup similar to the Cassandra remote server mode. As storage backend, I want to use Cassandra and as index backend I planned to use Elasticsearch.
For both, I'm using Docker containers (Cassandra, Elasticsearch).
My janusgraph-server.properties file looks like this:
gremlin.graph=org.janusgraph.core.JanusGraphFactory
storage.backend=cassandra
storage.hostname=127.0.0.1
storage.cassandra.astyanax.cluster-name=cassandra_test_cluster
index.search.backend=elasticsearch
index.search.hostname=127.0.0.1
index.search.port=9300
index.search.elasticsearch.cluster-name=elasticsearch_test_cluster
Starting the gremlin-server leads to this failures:
0 [main] INFO org.apache.tinkerpop.gremlin.server.GremlinServer -
\,,,/
(o o)
-----oOOo-(3)-oOOo-----
162 [main] INFO org.apache.tinkerpop.gremlin.server.GremlinServer - Configuring Gremlin Server from conf/gremlin-server/gremlin-server.yaml
256 [main] INFO org.apache.tinkerpop.gremlin.server.util.MetricManager - Configured Metrics ConsoleReporter configured with report interval=180000ms
263 [main] INFO org.apache.tinkerpop.gremlin.server.util.MetricManager - Configured Metrics CsvReporter configured with report interval=180000ms to fileName=/tmp/gremlin-server-metrics.csv
343 [main] INFO org.apache.tinkerpop.gremlin.server.util.MetricManager - Configured Metrics JmxReporter configured with domain= and agentId=
345 [main] INFO org.apache.tinkerpop.gremlin.server.util.MetricManager - Configured Metrics Slf4jReporter configured with interval=180000ms and loggerName=org.apache.tinkerpop.gremlin.server.Settings$Slf4jReporterMetrics
800 [main] INFO com.netflix.astyanax.connectionpool.impl.ConnectionPoolMBeanManager - Registering mbean: com.netflix.MonitoredResources:type=ASTYANAX,name=ClusterJanusGraphConnectionPool,ServiceType=connectionpool
807 [main] INFO com.netflix.astyanax.connectionpool.impl.CountingConnectionPoolMonitor - AddHost: 127.0.0.1
884 [main] INFO com.netflix.astyanax.connectionpool.impl.ConnectionPoolMBeanManager - Registering mbean: com.netflix.MonitoredResources:type=ASTYANAX,name=KeyspaceJanusGraphConnectionPool,ServiceType=connectionpool
884 [main] INFO com.netflix.astyanax.connectionpool.impl.CountingConnectionPoolMonitor - AddHost: 127.0.0.1
1070 [main] INFO org.janusgraph.graphdb.configuration.GraphDatabaseConfiguration - Generated unique-instance-id=c0a8000424833-XXX-MacBook-Pro-local1
1078 [main] INFO com.netflix.astyanax.connectionpool.impl.ConnectionPoolMBeanManager - Registering mbean: com.netflix.MonitoredResources:type=ASTYANAX,name=ClusterJanusGraphConnectionPool,ServiceType=connectionpool
1079 [main] INFO com.netflix.astyanax.connectionpool.impl.CountingConnectionPoolMonitor - AddHost: 127.0.0.1
1082 [main] INFO com.netflix.astyanax.connectionpool.impl.ConnectionPoolMBeanManager - Registering mbean: com.netflix.MonitoredResources:type=ASTYANAX,name=KeyspaceJanusGraphConnectionPool,ServiceType=connectionpool
1082 [main] INFO com.netflix.astyanax.connectionpool.impl.CountingConnectionPoolMonitor - AddHost: 127.0.0.1
1099 [main] INFO org.janusgraph.diskstorage.Backend - Configuring index [search]
1179 [main] INFO org.elasticsearch.plugins - [General Orwell Taylor] loaded [], sites []
1655 [main] INFO org.janusgraph.diskstorage.es.ElasticSearchIndex - Configured remote host: 127.0.0.1 : 9300
1738 [elasticsearch[General Orwell Taylor][generic][T#2]] INFO org.elasticsearch.client.transport - [General Orwell Taylor] failed to get local cluster state for [#transport#-1][XXX-MacBook-Pro.local][inet[/127.0.0.1:9300]], disconnecting...
org.elasticsearch.transport.NodeDisconnectedException: [][inet[/127.0.0.1:9300]][cluster:monitor/state] disconnected
1743 [main] WARN org.apache.tinkerpop.gremlin.server.GremlinServer - Graph [graph] configured at [conf/gremlin-server/janusgraph-server.properties] could not be instantiated and will not be available in Gremlin Server. GraphFactory message: GraphFactory could not instantiate this Graph implementation [class org.janusgraph.core.JanusGraphFactory]
java.lang.RuntimeException: GraphFactory could not instantiate this Graph implementation [class org.janusgraph.core.JanusGraphFactory]
at org.apache.tinkerpop.gremlin.structure.util.GraphFactory.open(GraphFactory.java:82)
at org.apache.tinkerpop.gremlin.structure.util.GraphFactory.open(GraphFactory.java:70)
at org.apache.tinkerpop.gremlin.structure.util.GraphFactory.open(GraphFactory.java:104)
at org.apache.tinkerpop.gremlin.server.GraphManager.lambda$new$0(GraphManager.java:55)
at java.util.LinkedHashMap$LinkedEntrySet.forEach(LinkedHashMap.java:671)
at org.apache.tinkerpop.gremlin.server.GraphManager.<init>(GraphManager.java:53)
at org.apache.tinkerpop.gremlin.server.util.ServerGremlinExecutor.<init>(ServerGremlinExecutor.java:83)
at org.apache.tinkerpop.gremlin.server.GremlinServer.<init>(GremlinServer.java:110)
at org.apache.tinkerpop.gremlin.server.GremlinServer.main(GremlinServer.java:344)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.tinkerpop.gremlin.structure.util.GraphFactory.open(GraphFactory.java:78)
... 8 more
Caused by: java.lang.IllegalArgumentException: Could not instantiate implementation: org.janusgraph.diskstorage.es.ElasticSearchIndex
at org.janusgraph.util.system.ConfigurationUtil.instantiate(ConfigurationUtil.java:69)
at org.janusgraph.diskstorage.Backend.getImplementationClass(Backend.java:477)
at org.janusgraph.diskstorage.Backend.getIndexes(Backend.java:464)
at org.janusgraph.diskstorage.Backend.<init>(Backend.java:149)
at org.janusgraph.graphdb.configuration.GraphDatabaseConfiguration.getBackend(GraphDatabaseConfiguration.java:1850)
at org.janusgraph.graphdb.database.StandardJanusGraph.<init>(StandardJanusGraph.java:134)
at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:107)
at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:87)
... 13 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.janusgraph.util.system.ConfigurationUtil.instantiate(ConfigurationUtil.java:58)
... 20 more
Caused by: org.elasticsearch.client.transport.NoNodeAvailableException: None of the configured nodes are available: []
at org.elasticsearch.client.transport.TransportClientNodesService.ensureNodesAreAvailable(TransportClientNodesService.java:279)
at org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:198)
at org.elasticsearch.client.transport.support.InternalTransportClusterAdminClient.execute(InternalTransportClusterAdminClient.java:86)
at org.elasticsearch.client.support.AbstractClusterAdminClient.health(AbstractClusterAdminClient.java:127)
at org.elasticsearch.action.admin.cluster.health.ClusterHealthRequestBuilder.doExecute(ClusterHealthRequestBuilder.java:92)
at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:91)
at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:65)
at org.janusgraph.diskstorage.es.ElasticSearchIndex.<init>(ElasticSearchIndex.java:215)
... 25 more
1745 [main] INFO org.apache.tinkerpop.gremlin.server.util.ServerGremlinExecutor - Initialized Gremlin thread pool. Threads in pool named with pattern gremlin-*
2190 [main] INFO org.apache.tinkerpop.gremlin.groovy.engine.ScriptEngines - Loaded gremlin-groovy ScriptEngine
2836 [main] WARN org.apache.tinkerpop.gremlin.groovy.engine.GremlinExecutor - Could not initialize gremlin-groovy ScriptEngine with scripts/empty-sample.groovy as script could not be evaluated - javax.script.ScriptException: groovy.lang.MissingPropertyException: No such property: graph for class: Script1
None of the configured nodes are available: [] why?
What can I do to make them available?
Have you verified whether Elasticsearch and Cassandra are running on those ports on localhost? If not, I would recommend checking that you're forwarding to those ports when starting your containers.
I would also recommend checking the logs for Cassandra and Elasticsearch and seeing if there is any errors in those.
I can run SparkPi example on the master node, but when I try the same command
"spark-submit --class SparkPi --master yarn-client sparkpi.jar 10"
on the slave node, I got an error:
2015-05-19 14:05:44,881 INFO [main] spark.SecurityManager (Logging.scala:logInfo(59)) - Changing view acls to: maintainer
2015-05-19 14:05:44,886 INFO [main] spark.SecurityManager (Logging.scala:logInfo(59)) - Changing modify acls to: maintainer
2015-05-19 14:05:44,887 INFO [main] spark.SecurityManager (Logging.scala:logInfo(59)) - SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(maintainer); users with modify permissions: Set(maintainer)
2015-05-19 14:05:45,389 INFO [sparkDriver-akka.actor.default-dispatcher-4] slf4j.Slf4jLogger (Slf4jLogger.scala:applyOrElse(80)) - Slf4jLogger started
2015-05-19 14:05:45,443 INFO [sparkDriver-akka.actor.default-dispatcher-4] Remoting (Slf4jLogger.scala:apply$mcV$sp(74)) - Starting remoting
2015-05-19 14:05:45,641 INFO [sparkDriver-akka.actor.default-dispatcher-3] Remoting (Slf4jLogger.scala:apply$mcV$sp(74)) - Remoting started; listening on addresses :[akka.tcp://sparkDriver#slave2.com:33055]
2015-05-19 14:05:45,644 INFO [sparkDriver-akka.actor.default-dispatcher-3] Remoting (Slf4jLogger.scala:apply$mcV$sp(74)) - Remoting now listens on addresses: [akka.tcp://sparkDriver#slave2.com:33055]
2015-05-19 14:05:45,653 INFO [main] util.Utils (Logging.scala:logInfo(59)) - Successfully started service 'sparkDriver' on port 33055.
2015-05-19 14:05:45,674 INFO [main] spark.SparkEnv (Logging.scala:logInfo(59)) - Registering MapOutputTracker
2015-05-19 14:05:45,688 INFO [main] spark.SparkEnv (Logging.scala:logInfo(59)) - Registering BlockManagerMaster
2015-05-19 14:05:45,707 INFO [main] storage.DiskBlockManager (Logging.scala:logInfo(59)) - Created local directory at /tmp/spark-local-20150519140545-c81b
2015-05-19 14:05:45,712 INFO [main] storage.MemoryStore (Logging.scala:logInfo(59)) - MemoryStore started with capacity 265.4 MB
2015-05-19 14:05:46,205 WARN [main] util.NativeCodeLoader (NativeCodeLoader.java:<clinit>(62)) - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2015-05-19 14:05:46,408 INFO [main] spark.HttpFileServer (Logging.scala:logInfo(59)) - HTTP File server directory is /tmp/spark-e95a2b5b-efea-41eb-93b9-0a9f7d6f6701
2015-05-19 14:05:46,413 INFO [main] spark.HttpServer (Logging.scala:logInfo(59)) - Starting HTTP Server
2015-05-19 14:05:46,477 INFO [main] server.Server (Server.java:doStart(272)) - jetty-8.y.z-SNAPSHOT
2015-05-19 14:05:46,499 INFO [main] server.AbstractConnector (AbstractConnector.java:doStart(338)) - Started SocketConnector#0.0.0.0:52737
2015-05-19 14:05:46,500 INFO [main] util.Utils (Logging.scala:logInfo(59)) - Successfully started service 'HTTP file server' on port 52737.
2015-05-19 14:05:46,790 INFO [main] server.Server (Server.java:doStart(272)) - jetty-8.y.z-SNAPSHOT
2015-05-19 14:05:46,805 INFO [main] server.AbstractConnector (AbstractConnector.java:doStart(338)) - Started SelectChannelConnector#0.0.0.0:4040
2015-05-19 14:05:46,805 INFO [main] util.Utils (Logging.scala:logInfo(59)) - Successfully started service 'SparkUI' on port 4040.
2015-05-19 14:05:46,808 INFO [main] ui.SparkUI (Logging.scala:logInfo(59)) - Started SparkUI at http://slave2.com:4040
2015-05-19 14:05:47,058 INFO [main] spark.SparkContext (Logging.scala:logInfo(59)) - Added JAR file:/home/maintainer/myjars/sparkpi.jar at http://[ip]:52737/jars/sparkpi.jar with timestamp 1432033547057
2015-05-19 14:05:47,190 INFO [main] client.RMProxy (RMProxy.java:createRMProxy(98)) - Connecting to ResourceManager at /0.0.0.0:8032
2015-05-19 14:09:45,861 INFO [main] client.RMProxy (RMProxy.java:createRMProxy(98)) - Connecting to ResourceManager at /0.0.0.0:8032
**2015-05-19 14:09:47,067 INFO [main] ipc.Client (Client.java:handleConnectionFailure(842)) - Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2015-05-19 14:09:48,068 INFO [main] ipc.Client (Client.java:handleConnectionFailure(842)) - Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
...**
Aside from specifying yarn.resourcemanager.hostname property in yarn-site.xml, it's also necessary to propagate configuration files to workers.
It might be done with this line (before running spark-submit):
export SPARK_YARN_DIST_FILES=$(ls $HADOOP_CONF_DIR* | sed 's#^#file://#g' | tr '\n' ',' | sed 's/,$//')
If everything's configured correctly, you'll see RM hostname instead of 0.0.0.0 in this line:
2015-05-19 14:05:47,190 INFO [main] client.RMProxy (RMProxy.java:createRMProxy(98)) - Connecting to ResourceManager at /0.0.0.0:8032
Exporting correct values for HADOOP_CONF_DIR fixed the issue.
export HADOOP_CONF_DIR=/your-path/hadoop/conf
I am trying to storm a storm topology but the storm worker refuses to start when I try to run the java command which invokes the worker process I get the following error:
Exception: java.lang.StackOverflowError thrown from the UncaughtExceptionHandler in thread "main"
I am not able to find what problem is causing this. Has anyone faced similar issue
Edit:
when I runt the worker process with flag -V I get the following error:
588 [main] INFO org.apache.zookeeper.server.ZooKeeperServer - Server environment:java.library.path=/usr/local/lib:/opt/local/lib:/usr/lib
588 [main] INFO org.apache.zookeeper.server.ZooKeeperServer - Server environment:java.io.tmpdir=/tmp
588 [main] INFO org.apache.zookeeper.server.ZooKeeperServer - Server environment:java.compiler=<NA>
588 [main] INFO org.apache.zookeeper.server.ZooKeeperServer - Server environment:os.name=Linux
588 [main] INFO org.apache.zookeeper.server.ZooKeeperServer - Server environment:os.arch=amd64
588 [main] INFO org.apache.zookeeper.server.ZooKeeperServer - Server environment:os.version=3.5.0-23-generic
588 [main] INFO org.apache.zookeeper.server.ZooKeeperServer - Server environment:user.name=storm
588 [main] INFO org.apache.zookeeper.server.ZooKeeperServer - Server environment:user.home=/home/storm
588 [main] INFO org.apache.zookeeper.server.ZooKeeperServer - Server environment:user.dir=/home/storm/storm-0.9.0.1
797 [main] ERROR org.apache.zookeeper.server.NIOServerCnxn - Thread Thread[main,5,main] died
PS: When I run the same topology in local cluster it works fine, only when i deploy in cluster mode it doesnt start.
Just found out the issue. The jar I creted to upload in the storm cluster, was kept in the storm base directory pics. This somehow was creating conflict which was not shown in the log file and actually log file never got created.
Make sure no external jars are present in the base storm folder from where one start storm. Really tricky error no idea why this happens until you just get around it.
Hope the storm guys add this into the logs so that user facing such issue can pinpoint why exactly this is happening.
While running a topology in storm we are getting error like this,
8983 [Thread-6] INFO com.netflix.curator.framework.imps.CuratorFrameworkImpl -
Starting
9144 [main] INFO **backtype.storm.daemon.nimbus** - Shutting down master
9199 [Thread-6-EventThread] INFO backtype.storm.zookeeper - Zookeeper state upd
ate: :connected:none
9241 [main] INFO backtype.storm.daemon.nimbus - Shut down master
9273 [Thread-6] INFO com.netflix.curator.framework.imps.CuratorFrameworkImpl -
Starting
9306 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2000] WARN org.apache.zookeeper.serv
er.NIOServerCnxn - EndOfStreamException: Unable to read additional data from cli
ent sessionid 0x143af55728d0003, likely client has closed socket
9354 [main] INFO backtype.storm.daemon.supervisor - Shutting down c094c3b1-a378
-4c4f-af35-9278647c217a:4beddc09-4675-4fb9-8bdc-9cf5013ce9ca
9358 [main] INFO backtype.storm.daemon.supervisor - Shut down c094c3b1-a378-4c4
f-af35-9278647c217a:4beddc09-4675-4fb9-8bdc-9cf5013ce9ca
9361 [main] INFO **backtype.storm.daemon.superviso**r - Shutting down supervisor c0
94c3b1-a378-4c4f-af35-9278647c217a
9364 [Thread-5] INFO **backtype.storm.event** - Event manager interrupted
9369 [Thread-6] INFO backtype.storm.event - Event manager interrupted
9425 [main] INFO **backtype.storm.daemon.supervisor** - Shutting down supervisor 38
6d8d71-c9b5-4b51-bd6e-f9f605034ea0
9428 [Thread-8] INFO backtype.storm.event - Event manager interrupted
9429 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2000] WARN org.apache.zookeeper.serv
er.NIOServerCnxn - EndOfStreamException: Unable to read additional data from cli
ent sessionid 0x143af55728d0007, likely client has closed socket
9429 [Thread-9] INFO backtype.storm.event - Event manager interrupted
9473 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2000] WARN org.apache.zookeeper.serv
er.NIOServerCnxn - EndOfStreamException: Unable to read additional data from cli
ent sessionid 0x143af55728d0009, likely client has closed socket
9476 [main] INFO backtype.storm.testing - Shutting down in process zookeeper
9503 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2000] WARN org.apache.zookeeper.serv
er.NIOServerCnxn - Ignoring exception
**java.nio.channels.ClosedChannelException**: null
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.jav
a:211) ~[na:1.7.0_03]
at org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.j
ava:242) ~[zookeeper-3.3.3.jar:3.3.3-1073969]
9510 [main] INFO **backtype.storm.testing** - Done shutting down in process zookeep
er
9513 [main] INFO backtype.storm.testing - Deleting temporary path C:\Users\sowm
iya\AppData\Local\Temp\c9b1bc1a-a950-4098-af77-f81a4d2b112f
9520 [main] INFO backtype.storm.testing - Deleting temporary path C:\Users\sowm
iya\AppData\Local\Temp\7e75c468-18ea-4787-a4ac-496fb108db71
9527 [main] INFO backtype.storm.testing - Unable to delete file: C:\Users\sowmi
ya\AppData\Local\Temp\7e75c468-18ea-4787-a4ac-496fb108db71\version-2\log.1
9529 [main] INFO backtype.storm.testing - Deleting temporary path C:\Users\sowm
iya\AppData\Local\Temp\fa7b3c9b-ac93-4090-b9e2-63f10019e61f
9543 [main] INFO backtype.storm.testing - Deleting temporary path C:\Users\sowm
iya\AppData\Local\Temp\55f1fd11-508e-43bb-b340-0d9b79f3af33
9579 [Thread-6-EventThread] INFO com.netflix.curator.framework.state.Connection
StateManager - State change: SUSPENDED
9580 [ConnectionStateManager-0] WARN com.netflix.curator.framework.state.Connec
tionStateManager - There are no ConnectionStateListeners registered.
9583 [Thread-6-EventThread] WARN backtype.storm.cluster - Received event :disco
nnected::none: with disconnected Zookeeper.
11232 [Thread-6-SendThread(localhost:2000)] WARN org.apache.zookeeper.ClientCnx
n - Session 0x143af55728d000b for server null, unexpected error, closing socket
connection and attempting reconnect
**java.net.ConnectException: Connection refused: no further information**
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[na:1.7.0_0
3]
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:701
) ~[na:1.7.0_03]
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1119)
~[zookeeper-3.3.3.jar:3.3.3-1073969]
13992 [Thread-6-SendThread(localhost:2000)] WARN org.apache.zookeeper.ClientCnx
n - Session 0x143af55728d000b for server null, unexpected error, closing socket
connection and attempting reconnect
java.net.ConnectException: Connection refused: no further information
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[na:1.7.0_0
3]
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:701
) ~[na:1.7.0_03]
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1119)
Whwn we are trying to run the topology jar file all the operation like nimbus,zookeeper and supervisor process going to dead.please help us to know why this is happened.
Please help us to rectify this error and help to proceed further.
Thank you,
Sowmiya
Priya
This looks like a zookeeper issue. It looks like your processes are not being able to connect to zookeeper. Can't say more without more information.