I'm trying to setup JanusGraph for development on my local machine. My goal is to have a setup similar to the Cassandra remote server mode. As storage backend, I want to use Cassandra and as index backend I planned to use Elasticsearch.
For both, I'm using Docker containers (Cassandra, Elasticsearch).
My janusgraph-server.properties file looks like this:
gremlin.graph=org.janusgraph.core.JanusGraphFactory
storage.backend=cassandra
storage.hostname=127.0.0.1
storage.cassandra.astyanax.cluster-name=cassandra_test_cluster
index.search.backend=elasticsearch
index.search.hostname=127.0.0.1
index.search.port=9300
index.search.elasticsearch.cluster-name=elasticsearch_test_cluster
Starting the gremlin-server leads to this failures:
0 [main] INFO org.apache.tinkerpop.gremlin.server.GremlinServer -
\,,,/
(o o)
-----oOOo-(3)-oOOo-----
162 [main] INFO org.apache.tinkerpop.gremlin.server.GremlinServer - Configuring Gremlin Server from conf/gremlin-server/gremlin-server.yaml
256 [main] INFO org.apache.tinkerpop.gremlin.server.util.MetricManager - Configured Metrics ConsoleReporter configured with report interval=180000ms
263 [main] INFO org.apache.tinkerpop.gremlin.server.util.MetricManager - Configured Metrics CsvReporter configured with report interval=180000ms to fileName=/tmp/gremlin-server-metrics.csv
343 [main] INFO org.apache.tinkerpop.gremlin.server.util.MetricManager - Configured Metrics JmxReporter configured with domain= and agentId=
345 [main] INFO org.apache.tinkerpop.gremlin.server.util.MetricManager - Configured Metrics Slf4jReporter configured with interval=180000ms and loggerName=org.apache.tinkerpop.gremlin.server.Settings$Slf4jReporterMetrics
800 [main] INFO com.netflix.astyanax.connectionpool.impl.ConnectionPoolMBeanManager - Registering mbean: com.netflix.MonitoredResources:type=ASTYANAX,name=ClusterJanusGraphConnectionPool,ServiceType=connectionpool
807 [main] INFO com.netflix.astyanax.connectionpool.impl.CountingConnectionPoolMonitor - AddHost: 127.0.0.1
884 [main] INFO com.netflix.astyanax.connectionpool.impl.ConnectionPoolMBeanManager - Registering mbean: com.netflix.MonitoredResources:type=ASTYANAX,name=KeyspaceJanusGraphConnectionPool,ServiceType=connectionpool
884 [main] INFO com.netflix.astyanax.connectionpool.impl.CountingConnectionPoolMonitor - AddHost: 127.0.0.1
1070 [main] INFO org.janusgraph.graphdb.configuration.GraphDatabaseConfiguration - Generated unique-instance-id=c0a8000424833-XXX-MacBook-Pro-local1
1078 [main] INFO com.netflix.astyanax.connectionpool.impl.ConnectionPoolMBeanManager - Registering mbean: com.netflix.MonitoredResources:type=ASTYANAX,name=ClusterJanusGraphConnectionPool,ServiceType=connectionpool
1079 [main] INFO com.netflix.astyanax.connectionpool.impl.CountingConnectionPoolMonitor - AddHost: 127.0.0.1
1082 [main] INFO com.netflix.astyanax.connectionpool.impl.ConnectionPoolMBeanManager - Registering mbean: com.netflix.MonitoredResources:type=ASTYANAX,name=KeyspaceJanusGraphConnectionPool,ServiceType=connectionpool
1082 [main] INFO com.netflix.astyanax.connectionpool.impl.CountingConnectionPoolMonitor - AddHost: 127.0.0.1
1099 [main] INFO org.janusgraph.diskstorage.Backend - Configuring index [search]
1179 [main] INFO org.elasticsearch.plugins - [General Orwell Taylor] loaded [], sites []
1655 [main] INFO org.janusgraph.diskstorage.es.ElasticSearchIndex - Configured remote host: 127.0.0.1 : 9300
1738 [elasticsearch[General Orwell Taylor][generic][T#2]] INFO org.elasticsearch.client.transport - [General Orwell Taylor] failed to get local cluster state for [#transport#-1][XXX-MacBook-Pro.local][inet[/127.0.0.1:9300]], disconnecting...
org.elasticsearch.transport.NodeDisconnectedException: [][inet[/127.0.0.1:9300]][cluster:monitor/state] disconnected
1743 [main] WARN org.apache.tinkerpop.gremlin.server.GremlinServer - Graph [graph] configured at [conf/gremlin-server/janusgraph-server.properties] could not be instantiated and will not be available in Gremlin Server. GraphFactory message: GraphFactory could not instantiate this Graph implementation [class org.janusgraph.core.JanusGraphFactory]
java.lang.RuntimeException: GraphFactory could not instantiate this Graph implementation [class org.janusgraph.core.JanusGraphFactory]
at org.apache.tinkerpop.gremlin.structure.util.GraphFactory.open(GraphFactory.java:82)
at org.apache.tinkerpop.gremlin.structure.util.GraphFactory.open(GraphFactory.java:70)
at org.apache.tinkerpop.gremlin.structure.util.GraphFactory.open(GraphFactory.java:104)
at org.apache.tinkerpop.gremlin.server.GraphManager.lambda$new$0(GraphManager.java:55)
at java.util.LinkedHashMap$LinkedEntrySet.forEach(LinkedHashMap.java:671)
at org.apache.tinkerpop.gremlin.server.GraphManager.<init>(GraphManager.java:53)
at org.apache.tinkerpop.gremlin.server.util.ServerGremlinExecutor.<init>(ServerGremlinExecutor.java:83)
at org.apache.tinkerpop.gremlin.server.GremlinServer.<init>(GremlinServer.java:110)
at org.apache.tinkerpop.gremlin.server.GremlinServer.main(GremlinServer.java:344)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.tinkerpop.gremlin.structure.util.GraphFactory.open(GraphFactory.java:78)
... 8 more
Caused by: java.lang.IllegalArgumentException: Could not instantiate implementation: org.janusgraph.diskstorage.es.ElasticSearchIndex
at org.janusgraph.util.system.ConfigurationUtil.instantiate(ConfigurationUtil.java:69)
at org.janusgraph.diskstorage.Backend.getImplementationClass(Backend.java:477)
at org.janusgraph.diskstorage.Backend.getIndexes(Backend.java:464)
at org.janusgraph.diskstorage.Backend.<init>(Backend.java:149)
at org.janusgraph.graphdb.configuration.GraphDatabaseConfiguration.getBackend(GraphDatabaseConfiguration.java:1850)
at org.janusgraph.graphdb.database.StandardJanusGraph.<init>(StandardJanusGraph.java:134)
at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:107)
at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:87)
... 13 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.janusgraph.util.system.ConfigurationUtil.instantiate(ConfigurationUtil.java:58)
... 20 more
Caused by: org.elasticsearch.client.transport.NoNodeAvailableException: None of the configured nodes are available: []
at org.elasticsearch.client.transport.TransportClientNodesService.ensureNodesAreAvailable(TransportClientNodesService.java:279)
at org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:198)
at org.elasticsearch.client.transport.support.InternalTransportClusterAdminClient.execute(InternalTransportClusterAdminClient.java:86)
at org.elasticsearch.client.support.AbstractClusterAdminClient.health(AbstractClusterAdminClient.java:127)
at org.elasticsearch.action.admin.cluster.health.ClusterHealthRequestBuilder.doExecute(ClusterHealthRequestBuilder.java:92)
at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:91)
at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:65)
at org.janusgraph.diskstorage.es.ElasticSearchIndex.<init>(ElasticSearchIndex.java:215)
... 25 more
1745 [main] INFO org.apache.tinkerpop.gremlin.server.util.ServerGremlinExecutor - Initialized Gremlin thread pool. Threads in pool named with pattern gremlin-*
2190 [main] INFO org.apache.tinkerpop.gremlin.groovy.engine.ScriptEngines - Loaded gremlin-groovy ScriptEngine
2836 [main] WARN org.apache.tinkerpop.gremlin.groovy.engine.GremlinExecutor - Could not initialize gremlin-groovy ScriptEngine with scripts/empty-sample.groovy as script could not be evaluated - javax.script.ScriptException: groovy.lang.MissingPropertyException: No such property: graph for class: Script1
None of the configured nodes are available: [] why?
What can I do to make them available?
Have you verified whether Elasticsearch and Cassandra are running on those ports on localhost? If not, I would recommend checking that you're forwarding to those ports when starting your containers.
I would also recommend checking the logs for Cassandra and Elasticsearch and seeing if there is any errors in those.
Related
I am running Janusgraph(0.3.1) with Cassandra(3.11.1) and Elastic search(5.6.4) using Docker. But, now I migrated to the latest versions i.e (janusgraph: latest, Cassandra: 3, elastic search: 6.6.0) and also I am running multiple graphs. This worked fine but, I am not able to use the old data. When I try to mount old volumes of Cassandra and Elastic search, I am facing some issues with the Elastic search. But, When I am running a single graph with the latest versions and old volumes, it is working fine. When I try to use multiple graphs Janusgraph is not able to configure with Elastic search.
For doing this, instead of mounting old volumes I tried to import the old data from a graph using gyro format(which I have already exported from the old server). But there also I am facing this issue.
I have tried in two ways but in both ways, I am facing some issues. Any help will be appreciated
Here is my docker-compose.yml file
version: "3"
services:
janusgraph:
image: janusgraph/janusgraph:latest
container_name: jce-janusgraph
environment:
JANUS_PROPS_TEMPLATE: cassandra-es
janusgraph.storage.backend: cql
janusgraph.storage.hostname: jce-cassandra
janusgraph.index.search.hostname: jce-elastic
volumes:
- /mnt/janusvol/janus/scripts:/opt/janusgraph/tmp/scripts
- /mnt/janusvol/janus/conf/gremlin-server.yaml:/opt/janusgraph/conf/gremlin-server/gremlin-server.yaml
- /mnt/janusvol/janus/conf/empty-sample.groovy:/opt/janusgraph/scripts/empty-sample.groovy
- /mnt/janusvol/janus/conf/janusgraph-cql-es-server-geniegraph.properties:/opt/janusgraph/conf/gremlin-server/janusgraph-cql-es-server-geniegraph.properties
- /mnt/janusvol/janus/conf/janusgraph-cql-es-server-aagraph.properties:/opt/janusgraph/conf/gremlin-server/janusgraph-cql-es-server-aagraph.properties
ports:
- "8182:8182"
networks:
- jce-network
healthcheck:
test: ["CMD", "bin/gremlin.sh", "-e", "scripts/remote-connect.groovy"]
interval: 10s
timeout: 30s
retries: 3
entrypoint:
- bin/gremlin-server.sh
- conf/gremlin-server/gremlin-server.yaml
links:
- cassandra
- elasticsearch
cassandra:
image: cassandra:3
container_name: jce-cassandra
volumes:
- /mnt/janusvol/cassandra:/var/lib/cassandra
environment:
- "CASSANDRA_CLUSTER_NAME=GenieCass"
- "HEAP_NEWSIZE=1024M"
- "MAX_HEAP_SIZE=1024M"
ports:
- "7000:7000"
- "9042:9042"
- "9160:9160"
networks:
- jce-network
elasticsearch:
image: docker.elastic.co/elasticsearch/elasticsearch:6.6.0
container_name: jce-elastic
environment:
- "ES_JAVA_OPTS=-Xms1g -Xmx1g -XX:HeapDumpPath=/var/lib/heapdump"
- "http.host=0.0.0.0"
- "network.host=0.0.0.0"
- "transport.host=127.0.0.1"
- "cluster.name=docker-cluster"
- "xpack.security.enabled=false"
- "discovery.zen.minimum_master_nodes=1"
# - "es.enforce.bootstrap.checks=true"
volumes:
- /mnt/janusvol/elas/data:/usr/share/elasticsearch/data
- /mnt/janusvol/elas/heapdump:/var/lib/heapdump
ports:
- "9200:9200"
- "9300:9300"
# command: ['chown', '-R', '1000:1000', '/usr/share/elasticsearch/data']
networks:
- jce-network
networks:
jce-network:
Here is my janusgraph logs:
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/janusgraph/lib/slf4j-log4j12-1.7.12.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/janusgraph/lib/logback-classic-1.1.3.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
0 [main] INFO com.jcabi.manifests.Manifests - 110 attributes loaded from 282 stream(s) in 2s, 110 saved, 3788 ignored: ["Agent-Class", "Ant-Version", "Archiver-Version", "Automatic-Module-Name", "Bnd-LastModified", "Boot-Class-Path", "Branch", "Build-Date", "Build-Host", "Build-Id", "Build-Java-Version", "Build-Jdk", "Build-Job", "Build-Number", "Build-Timestamp", "Build-Version", "Built-At", "Built-By", "Built-Date", "Built-OS", "Built-On", "Built-Status", "Bundle-ActivationPolicy", "Bundle-Activator", "Bundle-BuddyPolicy", "Bundle-Category", "Bundle-ClassPath", "Bundle-ContactAddress", "Bundle-Description", "Bundle-DocURL", "Bundle-License", "Bundle-ManifestVersion", "Bundle-Name", "Bundle-NativeCode", "Bundle-RequiredExecutionEnvironment", "Bundle-SymbolicName", "Bundle-Vendor", "Bundle-Version", "Can-Redefine-Classes", "Change", "Class-Path", "Created-By", "DSTAMP", "DynamicImport-Package", "Eclipse-BuddyPolicy", "Eclipse-ExtensibleAPI", "Embed-Dependency", "Embed-Transitive", "Export-Package", "Extension-Name", "Extension-name", "Fragment-Host", "Gradle-Version", "Gremlin-Lib-Paths", "Gremlin-Plugin-Dependencies", "Gremlin-Plugin-Paths", "Ignore-Package", "Implementation-Build", "Implementation-Build-Date", "Implementation-Title", "Implementation-URL", "Implementation-Vendor", "Implementation-Vendor-Id", "Implementation-Version", "Import-Package", "Include-Resource", "JCabi-Build", "JCabi-Date", "JCabi-Version", "Java-Vendor", "Java-Version", "Main-Class", "Manifest-Version", "Maven-Version", "Module-Email", "Module-Origin", "Module-Owner", "Module-Source", "Originally-Created-By", "Os-Arch", "Os-Name", "Os-Version", "Package", "Premain-Class", "Private-Package", "Provide-Capability", "Require-Bundle", "Require-Capability", "Scm-Connection", "Scm-Revision", "Scm-Url", "Specification-Title", "Specification-Vendor", "Specification-Version", "TODAY", "TSTAMP", "Time-Zone-Database-Version", "Tool", "X-Compile-Elasticsearch-Snapshot", "X-Compile-Elasticsearch-Version", "X-Compile-Lucene-Version", "X-Compile-Source-JDK", "X-Compile-Target-JDK", "hash", "implementation-version", "mode", "package", "service", "url", "version"]
57 [main] INFO org.apache.tinkerpop.gremlin.server.GremlinServer - 3.4.4
\,,,/
(o o)
-----oOOo-(3)-oOOo-----
1334 [main] INFO org.apache.tinkerpop.gremlin.server.GremlinServer - Configuring Gremlin Server from conf/gremlin-server/gremlin-server.yaml
1804 [main] INFO org.apache.tinkerpop.gremlin.server.util.MetricManager - Configured Metrics ConsoleReporter configured with report interval=180000ms
1818 [main] INFO org.apache.tinkerpop.gremlin.server.util.MetricManager - Configured Metrics CsvReporter configured with report interval=180000ms to fileName=/tmp/gremlin-server-metrics.csv
2186 [main] INFO org.apache.tinkerpop.gremlin.server.util.MetricManager - Configured Metrics JmxReporter configured with domain= and agentId=
2237 [main] INFO org.apache.tinkerpop.gremlin.server.util.MetricManager - Configured Metrics Slf4jReporter configured with interval=180000ms and loggerName=org.apache.tinkerpop.gremlin.server.Settings$Slf4jReporterMetrics
4663 [main] INFO com.datastax.driver.core - DataStax Java driver 3.7.2 for Apache Cassandra
4758 [main] INFO com.datastax.driver.core.GuavaCompatibility - Detected Guava < 19 in the classpath, using legacy compatibility layer
5244 [main] INFO com.datastax.driver.core.Native - Could not load JNR C Library, native system calls through this library will not be available (set this logger level to DEBUG to see the full stack trace).
5248 [main] INFO com.datastax.driver.core.ClockFactory - Using java.lang.System clock to generate timestamps.
6088 [main] INFO com.datastax.driver.core.NettyUtil - Found Netty's native epoll transport in the classpath, using it
10922 [main] INFO com.datastax.driver.core.policies.DCAwareRoundRobinPolicy - Using data-center name 'datacenter1' for DCAwareRoundRobinPolicy (if this is incorrect, please provide the correct datacenter name with DCAwareRoundRobinPolicy constructor)
10949 [main] INFO com.datastax.driver.core.Cluster - New Cassandra host jce-cassandra/172.20.0.3:9042 added
14634 [main] INFO org.janusgraph.core.util.ReflectiveConfigOptionLoader - Loaded and initialized config classes: 13 OK out of 13 attempts in PT1.079S
15522 [main] INFO org.reflections.Reflections - Reflections took 346 ms to scan 2 urls, producing 0 keys and 0 values
18513 [main] INFO org.janusgraph.graphdb.idmanagement.UniqueInstanceIdRetriever - Generated unique-instance-id=ac14000431-7e0aa97456d61
18661 [main] INFO com.datastax.driver.core.ClockFactory - Using java.lang.System clock to generate timestamps.
19701 [main] INFO com.datastax.driver.core.policies.DCAwareRoundRobinPolicy - Using data-center name 'datacenter1' for DCAwareRoundRobinPolicy (if this is incorrect, please provide the correct datacenter name with DCAwareRoundRobinPolicy constructor)
19707 [main] INFO com.datastax.driver.core.Cluster - New Cassandra host jce-cassandra/172.20.0.3:9042 added
19899 [main] INFO org.janusgraph.diskstorage.Backend - Configuring index [search]
24417 [main] WARN org.janusgraph.diskstorage.es.rest.RestElasticSearchClient - Unable to determine Elasticsearch server version. Default to FIVE.
java.net.ConnectException: Connection refused
at org.elasticsearch.client.RestClient$SyncResponseListener.get(RestClient.java:952)
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:229)
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:274)
at org.janusgraph.diskstorage.es.rest.RestElasticSearchClient.getMajorVersion(RestElasticSearchClient.java:118)
at org.janusgraph.diskstorage.es.rest.RestElasticSearchClient.<init>(RestElasticSearchClient.java:101)
at org.janusgraph.diskstorage.es.rest.RestClientSetup.getElasticSearchClient(RestClientSetup.java:107)
at org.janusgraph.diskstorage.es.rest.RestClientSetup.connect(RestClientSetup.java:74)
at org.janusgraph.diskstorage.es.ElasticSearchSetup$1.connect(ElasticSearchSetup.java:50)
at org.janusgraph.diskstorage.es.ElasticSearchIndex.interfaceConfiguration(ElasticSearchIndex.java:379)
at org.janusgraph.diskstorage.es.ElasticSearchIndex.<init>(ElasticSearchIndex.java:295)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.janusgraph.util.system.ConfigurationUtil.instantiate(ConfigurationUtil.java:58)
at org.janusgraph.diskstorage.Backend.getImplementationClass(Backend.java:440)
at org.janusgraph.diskstorage.Backend.getIndexes(Backend.java:427)
at org.janusgraph.diskstorage.Backend.<init>(Backend.java:150)
at org.janusgraph.graphdb.configuration.GraphDatabaseConfiguration.getBackend(GraphDatabaseConfiguration.java:1352)
at org.janusgraph.graphdb.database.StandardJanusGraph.<init>(StandardJanusGraph.java:142)
at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:161)
at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:132)
at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:112)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.tinkerpop.gremlin.structure.util.GraphFactory.open(GraphFactory.java:78)
at org.apache.tinkerpop.gremlin.structure.util.GraphFactory.open(GraphFactory.java:70)
at org.apache.tinkerpop.gremlin.structure.util.GraphFactory.open(GraphFactory.java:104)
at org.apache.tinkerpop.gremlin.server.util.DefaultGraphManager.lambda$new$0(DefaultGraphManager.java:57)
at java.util.LinkedHashMap$LinkedEntrySet.forEach(LinkedHashMap.java:671)
at org.apache.tinkerpop.gremlin.server.util.DefaultGraphManager.<init>(DefaultGraphManager.java:55)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.tinkerpop.gremlin.server.util.ServerGremlinExecutor.<init>(ServerGremlinExecutor.java:80)
at org.apache.tinkerpop.gremlin.server.GremlinServer.<init>(GremlinServer.java:122)
at org.apache.tinkerpop.gremlin.server.GremlinServer.<init>(GremlinServer.java:86)
at org.apache.tinkerpop.gremlin.server.GremlinServer.main(GremlinServer.java:345)
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:714)
at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvent(DefaultConnectingIOReactor.java:171)
at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents(DefaultConnectingIOReactor.java:145)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:348)
at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:192)
at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64)
at java.lang.Thread.run(Thread.java:748)
24582 [main] WARN org.apache.tinkerpop.gremlin.server.GremlinServer - Graph [graph] configured at [conf/gremlin-server/janusgraph-cql-es-server-geniegraph.properties] could not be instantiated and will not be available in Gremlin Server. GraphFactory message: GraphFactory could not instantiate this Graph implementation [class org.janusgraph.core.JanusGraphFactory]
java.lang.RuntimeException: GraphFactory could not instantiate this Graph implementation [class org.janusgraph.core.JanusGraphFactory]
at org.apache.tinkerpop.gremlin.structure.util.GraphFactory.open(GraphFactory.java:82)
at org.apache.tinkerpop.gremlin.structure.util.GraphFactory.open(GraphFactory.java:70)
at org.apache.tinkerpop.gremlin.structure.util.GraphFactory.open(GraphFactory.java:104)
at org.apache.tinkerpop.gremlin.server.util.DefaultGraphManager.lambda$new$0(DefaultGraphManager.java:57)
at java.util.LinkedHashMap$LinkedEntrySet.forEach(LinkedHashMap.java:671)
at org.apache.tinkerpop.gremlin.server.util.DefaultGraphManager.<init>(DefaultGraphManager.java:55)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.tinkerpop.gremlin.server.util.ServerGremlinExecutor.<init>(ServerGremlinExecutor.java:80)
at org.apache.tinkerpop.gremlin.server.GremlinServer.<init>(GremlinServer.java:122)
at org.apache.tinkerpop.gremlin.server.GremlinServer.<init>(GremlinServer.java:86)
at org.apache.tinkerpop.gremlin.server.GremlinServer.main(GremlinServer.java:345)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.tinkerpop.gremlin.structure.util.GraphFactory.open(GraphFactory.java:78)
... 13 more
Caused by: java.lang.IllegalArgumentException: Could not instantiate implementation: org.janusgraph.diskstorage.es.ElasticSearchIndex
at org.janusgraph.util.system.ConfigurationUtil.instantiate(ConfigurationUtil.java:64)
at org.janusgraph.diskstorage.Backend.getImplementationClass(Backend.java:440)
at org.janusgraph.diskstorage.Backend.getIndexes(Backend.java:427)
at org.janusgraph.diskstorage.Backend.<init>(Backend.java:150)
at org.janusgraph.graphdb.configuration.GraphDatabaseConfiguration.getBackend(GraphDatabaseConfiguration.java:1352)
at org.janusgraph.graphdb.database.StandardJanusGraph.<init>(StandardJanusGraph.java:142)
at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:161)
at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:132)
at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:112)
... 18 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.janusgraph.util.system.ConfigurationUtil.instantiate(ConfigurationUtil.java:58)
... 26 more
Caused by: org.janusgraph.diskstorage.PermanentBackendException: Connection refused
at org.janusgraph.diskstorage.es.ElasticSearchIndex.<init>(ElasticSearchIndex.java:315)
... 31 more
Caused by: java.net.ConnectException: Connection refused
at org.elasticsearch.client.RestClient$SyncResponseListener.get(RestClient.java:952)
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:229)
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:296)
at org.janusgraph.diskstorage.es.rest.RestElasticSearchClient.clusterHealthRequest(RestElasticSearchClient.java:133)
at org.janusgraph.diskstorage.es.ElasticSearchIndex.<init>(ElasticSearchIndex.java:313)
... 31 more
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:714)
at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvent(DefaultConnectingIOReactor.java:171)
at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents(DefaultConnectingIOReactor.java:145)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:348)
at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:192)
at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64)
at java.lang.Thread.run(Thread.java:748)
We can achieve this by importing the graph data in gryo/json/graphml format. The Practical Gremlin book has lots of examples of using Gremlin's io() step for example - kelvinlawrence.net/book/… Gryo specific usage with io() step can be found here: tinkerpop.apache.org/docs/current/reference/#gryo
When I tried to connect to Nifi UI using http://localhost:8080/nifi, i am getting below error
org.apache.nifi.web.server.JettyServer Failed to start web server... shutting down.
java.net.BindException: Address already in use: bind
at sun.nio.ch.Net.bind0(Native Method)
at sun.nio.ch.Net.bind(Net.java:433)
at sun.nio.ch.Net.bind(Net.java:425)
at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
at org.eclipse.jetty.server.ServerConnector.openAcceptChannel(ServerConnector.java:331)
at org.eclipse.jetty.server.ServerConnector.open(ServerConnector.java:299)
at org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:80)
at org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:235)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.server.Server.doStart(Server.java:398)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:935)
at org.apache.nifi.NiFi.<init>(NiFi.java:158)
at org.apache.nifi.NiFi.<init>(NiFi.java:72)
at org.apache.nifi.NiFi.main(NiFi.java:297)
2020-02-27 11:51:11,834 INFO [Thread-1] org.apache.nifi.NiFi Initiating shutdown of Jetty web server...
2020-02-27 11:51:11,836 INFO [Thread-1] o.eclipse.jetty.server.AbstractConnector Stopped ServerConnector#355ee205{HTTP/1.1,[http/1.1]}{0.0.0.0:8080}
2020-02-27 11:51:11,837 INFO [Thread-1] org.eclipse.jetty.server.session node 0 Stopped scavenging
Can anyone suggest what is the cause of this issue?
Nifi version- 1.9.2,installed on windows machine
Here is the nifi status logs,
12:33:16.886 [main] DEBUG org.apache.nifi.bootstrap.NotificationServiceManager - Found 0 service elements
12:33:16.896 [main] INFO org.apache.nifi.bootstrap.NotificationServiceManager - Successfully loaded the following 0 services: []
12:33:16.897 [main] INFO org.apache.nifi.bootstrap.RunNiFi - Registered no Notification Services for Notification Type NIFI_STARTED
12:33:16.897 [main] INFO org.apache.nifi.bootstrap.RunNiFi - Registered no Notification Services for Notification Type NIFI_STOPPED
12:33:16.898 [main] INFO org.apache.nifi.bootstrap.RunNiFi - Registered no Notification Services for Notification Type NIFI_DIED
12:33:16.899 [main] DEBUG org.apache.nifi.bootstrap.Command - Status File:
12:33:16.900 [main] DEBUG org.apache.nifi.bootstrap.Command - Properties: {pid=9724}
Failed to determine if Process 9724 is running; assuming that it is not
12:33:16.902 [main] INFO org.apache.nifi.bootstrap.Command - Apache NiFi is not running
The port use by nifi is already used by another process.
you can change web server port in conf/nifi.properties
I'm trying to install Apache Oozie on my local machine (Mac OS High Sierra 10.13.3) but receive a "java.net.ConnectException Connection refused" error when checking the status of the installation.
$ oozied.sh start
Setting OOZIE_HOME: /Users/akirna/oozie
Setting OOZIE_CONFIG: /Users/akirna/oozie/conf
Sourcing: /Users/akirna/oozie/conf/oozie-env.sh
setting CATALINA_OPTS="$CATALINA_OPTS -Xmx1024m"
Setting OOZIE_CONFIG_FILE: oozie-site.xml
Setting OOZIE_DATA: /Users/akirna/oozie/data
Setting OOZIE_LOG: /Users/akirna/oozie/logs
Setting OOZIE_LOG4J_FILE: oozie-log4j.properties
Setting OOZIE_LOG4J_RELOAD: 10
Setting OOZIE_HTTP_HOSTNAME: LAMU02T76ZCGTFM.uhc.com
Setting OOZIE_HTTP_PORT: 11000
Setting OOZIE_ADMIN_PORT: 11001
Setting OOZIE_HTTPS_PORT: 11443
Setting OOZIE_BASE_URL: http://LAMU02T76ZCGTFM.uhc.com:11000/oozie
Setting CATALINA_BASE: /Users/akirna/oozie/oozie-server
Setting OOZIE_HTTPS_KEYSTORE_FILE: /Users/akirna/.keystore
Setting OOZIE_HTTPS_KEYSTORE_PASS: password
Setting OOZIE_INSTANCE_ID: LAMU02T76ZCGTFM.uhc.com
Setting CATALINA_OUT: /Users/akirna/oozie/logs/catalina.out
Setting CATALINA_PID: /Users/akirna/oozie/oozie-server/temp/oozie.pid
Using CATALINA_OPTS: -Xmx1024m -Dderby.stream.error.file=/Users/akirna/oozie/logs/derby.log
Adding to CATALINA_OPTS: -Doozie.home.dir=/Users/akirna/oozie -Doozie.config.dir=/Users/akirna/oozie/conf -Doozie.log.dir=/Users/akirna/oozie/logs -Doozie.data.dir=/Users/akirna/oozie/data -Doozie.instance.id=LAMU02T76ZCGTFM.uhc.com -Doozie.config.file=oozie-site.xml -Doozie.log4j.file=oozie-log4j.properties -Doozie.log4j.reload=10 -Doozie.http.hostname=LAMU02T76ZCGTFM.uhc.com -Doozie.admin.port=11001 -Doozie.http.port=11000 -Doozie.https.port=11443 -Doozie.base.url=http://LAMU02T76ZCGTFM.uhc.com:11000/oozie -Doozie.https.keystore.file=/Users/akirna/.keystore -Doozie.https.keystore.pass=password -Djava.library.path=
Setting up oozie DB
setting CATALINA_OPTS="$CATALINA_OPTS -Xmx1024m"
Validate DB Connection
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
DONE
DB schema exists
The SQL commands have been written to: /var/folders/bx/5shvx3ln5wx4lp7c7j_l_f65gmq63g/T/ooziedb-4834223640212041380.sql
Using CATALINA_BASE: /Users/akirna/oozie/oozie-server
Using CATALINA_HOME: /Users/akirna/oozie/oozie-server
Using CATALINA_TMPDIR: /Users/akirna/oozie/oozie-server/temp
Using JRE_HOME: /Library/Java/JavaVirtualMachines/jdk1.8.0_144.jdk/Contents/Home
Using CLASSPATH: /Users/akirna/oozie/oozie-server/bin/bootstrap.jar
Using CATALINA_PID: /Users/akirna/oozie/oozie-server/temp/oozie.pid
Existing PID file found during start.
Removing/clearing stale PID file.
$ oozie admin -oozie http://localhost:11000/oozie -status output
Connection exception has occurred [ java.net.ConnectException Connection refused (Connection refused) ]. Trying after 1 sec. Retry count = 1
Connection exception has occurred [ java.net.ConnectException Connection refused (Connection refused) ]. Trying after 2 sec. Retry count = 2
Connection exception has occurred [ java.net.ConnectException Connection refused (Connection refused) ]. Trying after 4 sec. Retry count = 3
...
oozie/logs/oozie.log output:
*******************************************************************************
STARTUP MSG: Oozie BUILD_VERSION [4.3.1] compiled by [akirna] on [2018.04.18-19:11:20GMT]
STARTUP MSG: revision [unavailable]#[unavailable]
*******************************************************************************
2018-04-18 22:56:52,077 INFO XLogService:520 - SERVER[] Log4j configuration file [oozie-log4j.properties]
2018-04-18 22:56:52,077 INFO XLogService:520 - SERVER[] Log4j configuration file loaded from [/Users/akirna/oozie/conf]
2018-04-18 22:56:52,077 INFO XLogService:520 - SERVER[] Log4j reload interval [10 sec]
2018-04-18 22:56:52,084 INFO ConfigurationService:520 - SERVER[] Oozie home dir [/Users/akirna/oozie]
2018-04-18 22:56:52,085 INFO ConfigurationService:520 - SERVER[] Oozie conf dir [/Users/akirna/oozie/conf]
2018-04-18 22:56:52,085 INFO ConfigurationService:520 - SERVER[] Oozie conf file [oozie-site.xml]
2018-04-18 22:56:52,250 WARN ConfigurationService:523 - SERVER[] Invalid configuration defined, [oozie.service.ProxyUserService.proxyuser.akirna.hosts]
2018-04-18 22:56:52,251 WARN ConfigurationService:523 - SERVER[] Invalid configuration defined, [oozie.service.ProxyUserService.proxyuser.akirna.groups]
2018-04-18 22:56:52,254 WARN Services:523 - SERVER[] System ID [oozie-akir] exceeds maximum length [10], trimming
2018-04-18 22:56:52,255 INFO Services:520 - SERVER[] Exiting null Entering NORMAL
2018-04-18 22:56:52,256 INFO Services:520 - SERVER[] Initialized runtime directory [/var/folders/bx/5shvx3ln5wx4lp7c7j_l_f65gmq63g/T/oozie-akir1419795458214876479.dir]
2018-04-18 22:56:52,256 INFO ConfigurationService:520 - SERVER[] Programmatic configuration change, property[oozie.services]=[org.apache.oozie.service.LiteWorkflowAppService, org.apache.oozie.service.HadoopAccessorService]
2018-04-18 22:56:52,258 INFO ConfigurationService:520 - SERVER[] Programmatic configuration change, property[oozie.services.ext]=[]
2018-04-18 22:56:52,274 INFO HadoopAccessorService:520 - SERVER[] JOB_TRACKER_WHITELIST :[], Total entries :0
2018-04-18 22:56:52,274 INFO HadoopAccessorService:520 - SERVER[] NAME_NODE_WHITELIST :[], Total entries :0
2018-04-18 22:56:52,274 INFO HadoopAccessorService:520 - SERVER[] Oozie Kerberos Authentication [disabled]
2018-04-18 22:56:52,458 WARN NativeCodeLoader:60 - SERVER[] Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2018-04-18 22:56:52,556 INFO Services:520 - SERVER[] Initialized
2018-04-18 22:56:52,562 INFO Services:520 - SERVER[] Running with JARs for Hadoop version [3.0.0]
2018-04-18 22:56:52,563 INFO Services:520 - SERVER[] Oozie System ID [oozie-akir] started!
oozie/logs/catalina.out output:
ERROR: Oozie could not be started
REASON: java.lang.NoClassDefFoundError: org/apache/hadoop/fs/permission/AccessControlException
Stacktrace:
-----------------------------------------------------------------
java.lang.NoClassDefFoundError: org/apache/hadoop/fs/permission/AccessControlException
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at org.apache.hadoop.conf.Configuration.getClassByNameOrNull(Configuration.java:2416)
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2381)
at org.apache.oozie.util.XConfiguration.getClassByName(XConfiguration.java:220)
at org.apache.hadoop.conf.Configuration.getClasses(Configuration.java:2454)
at org.apache.oozie.service.ConfigurationService$LogChangesConfiguration.getClasses(ConfigurationService.java:411)
at org.apache.oozie.service.ConfigurationService.getClasses(ConfigurationService.java:584)
at org.apache.oozie.service.Services.loadServices(Services.java:288)
at org.apache.oozie.service.Services.init(Services.java:213)
at org.apache.oozie.servlet.ServicesLoader.contextInitialized(ServicesLoader.java:46)
at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4276)
at org.apache.catalina.core.StandardContext.start(StandardContext.java:4779)
at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:803)
at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:780)
at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:583)
at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:676)
at org.apache.catalina.startup.HostConfig.deployDescriptors(HostConfig.java:602)
at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:503)
at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1322)
at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:325)
at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:142)
at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1069)
at org.apache.catalina.core.StandardHost.start(StandardHost.java:822)
at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1061)
at org.apache.catalina.core.StandardEngine.start(StandardEngine.java:463)
at org.apache.catalina.core.StandardService.start(StandardService.java:525)
at org.apache.catalina.core.StandardServer.start(StandardServer.java:761)
at org.apache.catalina.startup.Catalina.start(Catalina.java:595)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289)
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.fs.permission.AccessControlException
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1698)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1544)
... 35 more
-----------------------------------------------------------------
I would appreciate some help on how to stress a Cassandra instance running on EC2 from my local machine (using cassandra-stress util).
Cluster on EC2: Five nodes running DSE 4.6.
Local machine: cassandra-stress as included in Cassandra 2.1.2.
After changing the Security Group the stress util invoked from my local machine is able to connect to the given instance on EC2.
I allowed inbound TCP connections on Ports 9160 and 9042 from my local machine's IP.
sh cassandra-stress write -node 54.xxx.197.xxx
Output is:
ec0007:bin planger$ sh cassandra-stress write -node 54.xxx.197.xxx
Unable to create stress keyspace: Keyspace names must be case-insensitively unique ("Keyspace1" conflicts with "Keyspace1")
Warming up WRITE with 50000 iterations...
WARN 13:45:52 Found host with 0.0.0.0 as rpc_address, using listen_address (/172.31.33.33) to contact it instead. If this is incorrect you should avoid the use of 0.0.0.0 server side.
WARN 13:45:52 Found host with 0.0.0.0 as rpc_address, using listen_address (/172.31.33.34) to contact it instead. If this is incorrect you should avoid the use of 0.0.0.0 server side.
WARN 13:45:52 Found host with 0.0.0.0 as rpc_address, using listen_address (/172.31.33.35) to contact it instead. If this is incorrect you should avoid the use of 0.0.0.0 server side.
WARN 13:45:52 Found host with 0.0.0.0 as rpc_address, using listen_address (/172.31.33.32) to contact it instead. If this is incorrect you should avoid the use of 0.0.0.0 server side.
WARN 13:45:53 Found host with 0.0.0.0 as rpc_address, using listen_address (/172.31.33.33) to contact it instead. If this is incorrect you should avoid the use of 0.0.0.0 server side.
WARN 13:45:53 Found host with 0.0.0.0 as rpc_address, using listen_address (/172.31.33.34) to contact it instead. If this is incorrect you should avoid the use of 0.0.0.0 server side.
WARN 13:45:53 Found host with 0.0.0.0 as rpc_address, using listen_address (/172.31.33.35) to contact it instead. If this is incorrect you should avoid the use of 0.0.0.0 server side.
WARN 13:45:53 Found host with 0.0.0.0 as rpc_address, using listen_address (/172.31.33.32) to contact it instead. If this is incorrect you should avoid the use of 0.0.0.0 server side.
INFO 13:45:53 Using data-center name 'Cassandra' for DCAwareRoundRobinPolicy (if this is incorrect, please provide the correct datacenter name with DCAwareRoundRobinPolicy constructor)
INFO 13:45:53 New Cassandra host /172.31.33.33:9042 added
INFO 13:45:53 New Cassandra host /172.31.33.32:9042 added
Connected to cluster: esentriDSEcluster
Datatacenter: Cassandra; Host: /54.xxx.197.xxx; Rack: rack1
INFO 13:45:53 New Cassandra host /172.31.33.35:9042 added
Datatacenter: Analytics; Host: /172.31.33.35; Rack: rack1
Datatacenter: Solr; Host: /172.31.33.34; Rack: rack1
Datatacenter: Cassandra; Host: /172.31.33.33; Rack: rack1
Datatacenter: Cassandra; Host: /172.31.33.32; Rack: rack1
INFO 13:45:53 New Cassandra host /172.31.33.34:9042 added
INFO 13:45:53 New Cassandra host /54.171.197.133:9042 added
ERROR 13:45:58 Error creating pool to /172.31.33.33:9042
com.datastax.driver.core.TransportException: [/172.31.33.33:9042] Cannot connect
at com.datastax.driver.core.Connection.<init>(Connection.java:104) ~[cassandra-driver-core-2.0.5.jar:na]
at com.datastax.driver.core.PooledConnection.<init>(PooledConnection.java:28) ~[cassandra-driver-core-2.0.5.jar:na]
at com.datastax.driver.core.Connection$Factory.open(Connection.java:458) ~[cassandra-driver-core-2.0.5.jar:na]
at com.datastax.driver.core.HostConnectionPool.<init>(HostConnectionPool.java:85) ~[cassandra-driver-core-2.0.5.jar:na]
at com.datastax.driver.core.SessionManager.replacePool(SessionManager.java:241) ~[cassandra-driver-core-2.0.5.jar:na]
at com.datastax.driver.core.SessionManager.access$400(SessionManager.java:42) ~[cassandra-driver-core-2.0.5.jar:na]
at com.datastax.driver.core.SessionManager$3.call(SessionManager.java:273) [cassandra-driver-core-2.0.5.jar:na]
at com.datastax.driver.core.SessionManager$3.call(SessionManager.java:265) [cassandra-driver-core-2.0.5.jar:na]
at java.util.concurrent.FutureTask.run(FutureTask.java:262) [na:1.7.0_55]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_55]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_55]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_55]
Caused by: org.jboss.netty.channel.ConnectTimeoutException: connection timed out: /172.31.33.33:9042
at org.jboss.netty.channel.socket.nio.NioClientBoss.processConnectTimeout(NioClientBoss.java:137) ~[netty-3.9.0.Final.jar:na]
at org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:83) ~[netty-3.9.0.Final.jar:na]
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) ~[netty-3.9.0.Final.jar:na]
at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42) ~[netty-3.9.0.Final.jar:na]
... 3 common frames omitted
The problem seems to be that the contacted EC2 instance returns the private IPs of the other four instances. How to change that? I think it should return the public IPs.
The cassandra.yaml configuration on the EC2 nodes looks like that:
seed_provider:
- class_name: org.apache.cassandra.locator.SimpleSeedProvider
parameters:
- seeds: "172.31.33.33,172.31.33.35,172.31.33.34"
listen_address: 172.31.33.32 # (different on every node)
# broadcast_address:
start_rpc: true
rpc_address: 0.0.0.0
rpc_port: 9160
Changing seeds or specifying a broadcast_address breaks the cluster.
Thanks for help in advance.
Update 1
I changed the rpc_address on every node by setting it's value to the node's public IP address. Then restarted each node.
sudo service dse stop
sudo service dse start
But this breaks the cluster. Short time after DSE startup, a node will shut down again.
From /var/log/cassandra/system.log
INFO [main] 2015-02-06 09:54:06,128 CassandraDaemon.java (line 135) Logging initialized
INFO [main] 2015-02-06 09:54:06,156 DseDaemon.java (line 382) DSE version: 4.6.0
INFO [main] 2015-02-06 09:54:06,157 DseDaemon.java (line 383) Hadoop version: 1.0.4.13
INFO [main] 2015-02-06 09:54:06,157 DseDaemon.java (line 384) Hive version: 0.12.0.5
INFO [main] 2015-02-06 09:54:06,158 DseDaemon.java (line 385) Pig version: 0.10.1
INFO [main] 2015-02-06 09:54:06,159 DseDaemon.java (line 386) Solr version: 4.6.0.3.3
INFO [main] 2015-02-06 09:54:06,159 DseDaemon.java (line 387) Sqoop version: 1.4.4.14.2
INFO [main] 2015-02-06 09:54:06,160 DseDaemon.java (line 388) Mahout version: 0.8
INFO [main] 2015-02-06 09:54:06,160 DseDaemon.java (line 389) Appender version: 3.1.0
INFO [main] 2015-02-06 09:54:06,161 DseDaemon.java (line 390) Spark version: 1.1.0.2
INFO [main] 2015-02-06 09:54:06,161 DseDaemon.java (line 391) Shark version: 1.1.0
INFO [main] 2015-02-06 09:54:06,416 DseConfig.java (line 345) Loading settings from file:/etc/dse/dse.yaml
INFO [main] 2015-02-06 09:54:06,605 DseConfig.java (line 385) Load of settings is done.
INFO [main] 2015-02-06 09:54:06,611 DseConfig.java (line 409) CQL slow log is enabled
INFO [main] 2015-02-06 09:54:06,611 DseConfig.java (line 410) CQL system info tables are not enabled
INFO [main] 2015-02-06 09:54:06,611 DseConfig.java (line 411) Resource level latency tracking is not enabled
INFO [main] 2015-02-06 09:54:06,613 DseConfig.java (line 412) Database summary stats are not enabled
INFO [main] 2015-02-06 09:54:06,614 DseConfig.java (line 413) Cluster summary stats are not enabled
INFO [main] 2015-02-06 09:54:06,614 DseConfig.java (line 414) Histogram data tables are not enabled
INFO [main] 2015-02-06 09:54:06,614 DseConfig.java (line 415) User level latency tracking is not enabled
INFO [main] 2015-02-06 09:54:06,614 DseConfig.java (line 416) Solr latency snapshots are not enabled
INFO [main] 2015-02-06 09:54:06,615 DseConfig.java (line 417) Solr slow sub-query log is not enabled
INFO [main] 2015-02-06 09:54:06,615 DseConfig.java (line 418) Solr indexing error log is not enabled
INFO [main] 2015-02-06 09:54:06,615 DseConfig.java (line 419) Solr update handler metrics are not enabled
INFO [main] 2015-02-06 09:54:06,615 DseConfig.java (line 420) Solr request handler metrics are not enabled
INFO [main] 2015-02-06 09:54:06,615 DseConfig.java (line 421) Solr index statistics reporting is not enabled
INFO [main] 2015-02-06 09:54:06,616 DseConfig.java (line 422) Solr cache statistics reporting is not enabled
INFO [main] 2015-02-06 09:54:06,629 YamlConfigurationLoader.java (line 80) Loading settings from file:/etc/dse/cassandra/cassandra.yaml
INFO [main] 2015-02-06 09:54:06,683 DatabaseDescriptor.java (line 143) Data files directories: [/raid0/cassandra/data]
INFO [main] 2015-02-06 09:54:06,683 DatabaseDescriptor.java (line 144) Commit log directory: /raid0/cassandra/commitlog
INFO [main] 2015-02-06 09:54:06,683 DatabaseDescriptor.java (line 184) DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap
INFO [main] 2015-02-06 09:54:06,683 DatabaseDescriptor.java (line 198) disk_failure_policy is stop
INFO [main] 2015-02-06 09:54:06,684 DatabaseDescriptor.java (line 199) commit_failure_policy is stop
INFO [main] 2015-02-06 09:54:06,690 DatabaseDescriptor.java (line 269) Global memtable threshold is enabled at 930MB
INFO [main] 2015-02-06 09:54:06,692 Workload.java (line 80) Setting my workload to Cassandra
INFO [main] 2015-02-06 09:54:06,693 DseDelegateSnitch.java (line 43) Initialized DseDelegateSnitch with workload Cassandra, delegating to com.datastax.bdp.snitch.DseSimpleSnitch
INFO [main] 2015-02-06 09:54:06,830 DatabaseDescriptor.java (line 408) Not using multi-threaded compaction
INFO [main] 2015-02-06 09:54:07,075 YamlConfigurationLoader.java (line 80) Loading settings from file:/etc/dse/cassandra/cassandra.yaml
INFO [main] 2015-02-06 09:54:07,090 YamlConfigurationLoader.java (line 80) Loading settings from file:/etc/dse/cassandra/cassandra.yaml
INFO [main] 2015-02-06 09:54:07,150 PerformanceObjectsController.java (line 321) ClusterSummaryStats plugin using 4 async writers
INFO [main] 2015-02-06 09:54:07,150 PerformanceObjectsController.java (line 355) ClusterSummaryStats refresh rate set to 10000 (was 0)
INFO [main] 2015-02-06 09:54:07,151 PerformanceObjectsController.java (line 321) CqlSystemInfo plugin using 1 async writers
INFO [main] 2015-02-06 09:54:07,151 PerformanceObjectsController.java (line 355) CqlSystemInfo refresh rate set to 10000 (was 0)
INFO [main] 2015-02-06 09:54:07,152 PerformanceObjectsController.java (line 321) DbSummaryStats plugin using 4 async writers
INFO [main] 2015-02-06 09:54:07,152 PerformanceObjectsController.java (line 355) DbSummaryStats refresh rate set to 10000 (was 0)
INFO [main] 2015-02-06 09:54:07,152 PerformanceObjectsController.java (line 321) HistogramDataTables plugin using 4 async writers
INFO [main] 2015-02-06 09:54:07,153 PerformanceObjectsController.java (line 355) HistogramDataTables refresh rate set to 10000 (was 0)
INFO [main] 2015-02-06 09:54:07,153 PerformanceObjectsController.java (line 321) ResourceLatencyTracking plugin using 4 async writers
INFO [main] 2015-02-06 09:54:07,153 PerformanceObjectsController.java (line 355) ResourceLatencyTracking refresh rate set to 10000 (was 0)
INFO [main] 2015-02-06 09:54:07,154 PerformanceObjectsController.java (line 321) UserLatencyTracking plugin using 1 async writers
INFO [main] 2015-02-06 09:54:07,154 PerformanceObjectsController.java (line 355) UserLatencyTracking refresh rate set to 10000 (was 0)
(…)
INFO [main] 2015-02-06 09:54:14,717 StorageService.java (line 514) Cassandra version: 2.0.11.83
INFO [main] 2015-02-06 09:54:14,718 StorageService.java (line 515) Thrift API version: 19.39.0
INFO [main] 2015-02-06 09:54:14,722 StorageService.java (line 516) CQL supported versions: 2.0.0,3.1.7 (default: 3.1.7)
INFO [main] 2015-02-06 09:54:14,745 StorageService.java (line 539) Loading persisted ring state
INFO [main] 2015-02-06 09:54:14,758 StorageService.java (line 677) Starting up server gossip
(…)
INFO [main] 2015-02-06 09:54:14,885 MessagingService.java (line 473) Starting Messaging Service on port 7000
INFO [main] 2015-02-06 09:54:14,917 YamlConfigurationLoader.java (line 80) Loading settings from file:/etc/dse/cassandra/cassandra.yaml
(…)
INFO [main] 2015-02-06 09:54:14,977 StorageService.java (line 1521) Node /172.31.33.33 state jump to normal
INFO [main] 2015-02-06 09:54:14,987 CassandraDaemon.java (line 543) Waiting for gossip to settle before accepting client requests...
INFO [main] 2015-02-06 09:54:22,988 CassandraDaemon.java (line 575) No gossip backlog; proceeding
INFO [main] 2015-02-06 09:54:23,011 AuditLogger.java (line 32) Audit logging is disabled
INFO [main] 2015-02-06 09:54:23,031 EndpointStatePersister.java (line 56) EndpointStatePersister started
WARN [main] 2015-02-06 09:54:23,031 Workload.java (line 100) Couldn't determine workload for /172.31.33.35 from value NULL
WARN [main] 2015-02-06 09:54:23,036 Workload.java (line 100) Couldn't determine workload for /172.31.33.32 from value NULL
WARN [main] 2015-02-06 09:54:23,036 Workload.java (line 100) Couldn't determine workload for /172.31.33.36 from value NULL
WARN [main] 2015-02-06 09:54:23,037 Workload.java (line 100) Couldn't determine workload for /172.31.33.34 from value NULL
INFO [main] 2015-02-06 09:54:23,038 EndpointStateTracker.java (line 80) EndpointStateTracker started
INFO [main] 2015-02-06 09:54:23,041 DseDaemon.java (line 441) Waiting for other nodes to become alive...
WARN [main] 2015-02-06 09:54:33,827 DseDaemon.java (line 444) The following nodes seems to be down: [/172.31.33.35, /172.31.33.32, /172.31.33.36, /172.31.33.34]. Some Cassandra operations may fail with UnavailableException.
INFO [main] 2015-02-06 09:54:33,827 DseDaemon.java (line 454) Wait for nodes completed
INFO [main] 2015-02-06 09:54:33,836 PluginManager.java (line 262) Activating plugin: com.datastax.bdp.plugin.DseSystemPlugin
INFO [main] 2015-02-06 09:54:33,837 PluginManager.java (line 344) Plugin activated: com.datastax.bdp.plugin.DseSystemPlugin
INFO [main] 2015-02-06 09:54:33,837 PluginManager.java (line 262) Activating plugin: com.datastax.bdp.leases.PeriodicTaskOwnershipPlugin
INFO [main] 2015-02-06 09:54:33,842 PluginManager.java (line 344) Plugin activated: com.datastax.bdp.leases.PeriodicTaskOwnershipPlugin
INFO [main] 2015-02-06 09:54:33,919 Server.java (line 156) Starting listening for CQL clients on /54.xxx.180.xxx:9042...
ERROR [main] 2015-02-06 09:54:33,934 DseDaemon.java (line 492) Unable to start DSE server.
org.jboss.netty.channel.ChannelException: **Failed to bind to**: /**54.xxx.180.xxx:9042**
at org.jboss.netty.bootstrap.ServerBootstrap.bind(ServerBootstrap.java:272)
at org.apache.cassandra.transport.Server.run(Server.java:157)
at org.apache.cassandra.transport.Server.start(Server.java:108)
at org.apache.cassandra.service.CassandraDaemon.start(CassandraDaemon.java:443)
at com.datastax.bdp.server.DseDaemon.start(DseDaemon.java:486)
at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:509)
at com.datastax.bdp.server.DseDaemon.main(DseDaemon.java:659)
Caused by: java.net.BindException: Cannot assign requested address
at sun.nio.ch.Net.bind0(Native Method)
at sun.nio.ch.Net.bind(Net.java:444)
at sun.nio.ch.Net.bind(Net.java:436)
at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
at org.jboss.netty.channel.socket.nio.NioServerBoss$RegisterTask.run(NioServerBoss.java:193)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:366)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:290)
at org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
INFO [Thread-2] 2015-02-06 09:54:33,937 DseDaemon.java (line 505) DSE shutting down...
INFO [StorageServiceShutdownHook] 2015-02-06 09:54:33,941 Gossiper.java (line 1307) Announcing shutdown
INFO [Thread-2] 2015-02-06 09:54:33,954 PluginManager.java (line 304) Deactivating plugin: com.datastax.bdp.leases.PeriodicTaskOwnershipPlugin
INFO [Thread-2] 2015-02-06 09:54:33,954 PluginManager.java (line 304) Deactivating plugin: com.datastax.bdp.plugin.DseSystemPlugin
INFO [Thread-2] 2015-02-06 09:54:33,954 PluginManager.java (line 356) All plugins are stopped.
INFO [Thread-2] 2015-02-06 09:54:33,954 CassandraDaemon.java (line 463) Cassandra shutting down...
INFO [StorageServiceShutdownHook] 2015-02-06 09:54:35,945 MessagingService.java (line 701) Waiting for messaging service to quiesce
INFO [ACCEPT-/172.31.33.33] 2015-02-06 09:54:35,945 MessagingService.java (line 941) MessagingService has terminated the accept() thread
I believe stress acts like any of the DataStax drivers and uses the RPC address for each node to communicate. Currently you have 0.0.0.0 configured which may be why its picking your internal aws address.
Three addresses avaliable in Cassandra.yaml
Listen Address - This is the ip address other Cassandra nodes will use to talk to this node. You want this to be your internal AWS IP Address for performance.
RPC Address - This is the address your client connects to, probably the one you want to configure to match your external AWS address if your client is not sitting in AWS or in the same AWS region. Also applies for stress.
Broadcast Address - If you are using multiple data centers or AWS Regions, where not all the nodes have access to each other via internal IP. You can specify the external IP address for the nodes in different data centers can still talk to each other. In many cases you don't need this setting at all, it will default to your Listen Address.
Let me know if this helps.
While running a topology in storm we are getting error like this,
8983 [Thread-6] INFO com.netflix.curator.framework.imps.CuratorFrameworkImpl -
Starting
9144 [main] INFO **backtype.storm.daemon.nimbus** - Shutting down master
9199 [Thread-6-EventThread] INFO backtype.storm.zookeeper - Zookeeper state upd
ate: :connected:none
9241 [main] INFO backtype.storm.daemon.nimbus - Shut down master
9273 [Thread-6] INFO com.netflix.curator.framework.imps.CuratorFrameworkImpl -
Starting
9306 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2000] WARN org.apache.zookeeper.serv
er.NIOServerCnxn - EndOfStreamException: Unable to read additional data from cli
ent sessionid 0x143af55728d0003, likely client has closed socket
9354 [main] INFO backtype.storm.daemon.supervisor - Shutting down c094c3b1-a378
-4c4f-af35-9278647c217a:4beddc09-4675-4fb9-8bdc-9cf5013ce9ca
9358 [main] INFO backtype.storm.daemon.supervisor - Shut down c094c3b1-a378-4c4
f-af35-9278647c217a:4beddc09-4675-4fb9-8bdc-9cf5013ce9ca
9361 [main] INFO **backtype.storm.daemon.superviso**r - Shutting down supervisor c0
94c3b1-a378-4c4f-af35-9278647c217a
9364 [Thread-5] INFO **backtype.storm.event** - Event manager interrupted
9369 [Thread-6] INFO backtype.storm.event - Event manager interrupted
9425 [main] INFO **backtype.storm.daemon.supervisor** - Shutting down supervisor 38
6d8d71-c9b5-4b51-bd6e-f9f605034ea0
9428 [Thread-8] INFO backtype.storm.event - Event manager interrupted
9429 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2000] WARN org.apache.zookeeper.serv
er.NIOServerCnxn - EndOfStreamException: Unable to read additional data from cli
ent sessionid 0x143af55728d0007, likely client has closed socket
9429 [Thread-9] INFO backtype.storm.event - Event manager interrupted
9473 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2000] WARN org.apache.zookeeper.serv
er.NIOServerCnxn - EndOfStreamException: Unable to read additional data from cli
ent sessionid 0x143af55728d0009, likely client has closed socket
9476 [main] INFO backtype.storm.testing - Shutting down in process zookeeper
9503 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2000] WARN org.apache.zookeeper.serv
er.NIOServerCnxn - Ignoring exception
**java.nio.channels.ClosedChannelException**: null
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.jav
a:211) ~[na:1.7.0_03]
at org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.j
ava:242) ~[zookeeper-3.3.3.jar:3.3.3-1073969]
9510 [main] INFO **backtype.storm.testing** - Done shutting down in process zookeep
er
9513 [main] INFO backtype.storm.testing - Deleting temporary path C:\Users\sowm
iya\AppData\Local\Temp\c9b1bc1a-a950-4098-af77-f81a4d2b112f
9520 [main] INFO backtype.storm.testing - Deleting temporary path C:\Users\sowm
iya\AppData\Local\Temp\7e75c468-18ea-4787-a4ac-496fb108db71
9527 [main] INFO backtype.storm.testing - Unable to delete file: C:\Users\sowmi
ya\AppData\Local\Temp\7e75c468-18ea-4787-a4ac-496fb108db71\version-2\log.1
9529 [main] INFO backtype.storm.testing - Deleting temporary path C:\Users\sowm
iya\AppData\Local\Temp\fa7b3c9b-ac93-4090-b9e2-63f10019e61f
9543 [main] INFO backtype.storm.testing - Deleting temporary path C:\Users\sowm
iya\AppData\Local\Temp\55f1fd11-508e-43bb-b340-0d9b79f3af33
9579 [Thread-6-EventThread] INFO com.netflix.curator.framework.state.Connection
StateManager - State change: SUSPENDED
9580 [ConnectionStateManager-0] WARN com.netflix.curator.framework.state.Connec
tionStateManager - There are no ConnectionStateListeners registered.
9583 [Thread-6-EventThread] WARN backtype.storm.cluster - Received event :disco
nnected::none: with disconnected Zookeeper.
11232 [Thread-6-SendThread(localhost:2000)] WARN org.apache.zookeeper.ClientCnx
n - Session 0x143af55728d000b for server null, unexpected error, closing socket
connection and attempting reconnect
**java.net.ConnectException: Connection refused: no further information**
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[na:1.7.0_0
3]
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:701
) ~[na:1.7.0_03]
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1119)
~[zookeeper-3.3.3.jar:3.3.3-1073969]
13992 [Thread-6-SendThread(localhost:2000)] WARN org.apache.zookeeper.ClientCnx
n - Session 0x143af55728d000b for server null, unexpected error, closing socket
connection and attempting reconnect
java.net.ConnectException: Connection refused: no further information
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[na:1.7.0_0
3]
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:701
) ~[na:1.7.0_03]
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1119)
Whwn we are trying to run the topology jar file all the operation like nimbus,zookeeper and supervisor process going to dead.please help us to know why this is happened.
Please help us to rectify this error and help to proceed further.
Thank you,
Sowmiya
Priya
This looks like a zookeeper issue. It looks like your processes are not being able to connect to zookeeper. Can't say more without more information.