ElasticsearchException failed to bind service error - elasticsearch

I really wana start elasticsearch for connect opencart.
I have set network.host: 0.0.0.0 (also try to my local IP), http.port: 9200, and give executive authority to my new account. I use centOs6, Java 1.8, and don't use root account.
I want to use elasticsearch on my server, but I have some issue.
my error log is that
[2019-03-21T18:12:27,970][INFO ][o.e.p.PluginsService ] [gxhuK_x] loaded module [x-pack-watcher]
[2019-03-21T18:12:27,971][INFO ][o.e.p.PluginsService ] [gxhuK_x] no plugins loaded
[2019-03-21T18:12:32,061][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [gxhuK_x] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: ElasticsearchException[failed to bind service]; nested: AccessDeniedException[/etc/system-release];
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-6.6.2.jar:6.6.2]
at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-6.6.2.jar:6.6.2]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:116) ~[elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) ~[elasticsearch-6.6.2.jar:6.6.2]
Caused by: org.elasticsearch.ElasticsearchException: failed to bind service
at org.elasticsearch.node.Node.<init>(Node.java:626) ~[elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.node.Node.<init>(Node.java:265) ~[elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-6.6.2.jar:6.6.2]
... 6 more
Caused by: java.nio.file.AccessDeniedException: /etc/system-release
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84) ~[?:?]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) ~[?:?]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) ~[?:?]
at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214) ~[?:?]
at java.nio.file.Files.newByteChannel(Files.java:361) ~[?:1.8.0_131]
at java.nio.file.Files.newByteChannel(Files.java:407) ~[?:1.8.0_131]
at java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:384) ~[?:1.8.0_131]
at java.nio.file.Files.newInputStream(Files.java:152) ~[?:1.8.0_131]
at java.nio.file.Files.newBufferedReader(Files.java:2784) ~[?:1.8.0_131]
at java.nio.file.Files.readAllLines(Files.java:3202) ~[?:1.8.0_131]
at java.nio.file.Files.readAllLines(Files.java:3242) ~[?:1.8.0_131]
at org.elasticsearch.monitor.os.OsProbe.readOsRelease(OsProbe.java:591) ~[elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.monitor.os.OsProbe.getPrettyName(OsProbe.java:546) ~[elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.monitor.os.OsProbe.osInfo(OsProbe.java:533) ~[elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.monitor.os.OsService.<init>(OsService.java:48) ~[elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.monitor.MonitorService.<init>(MonitorService.java:47) ~[elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.node.Node.<init>(Node.java:397) ~[elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.node.Node.<init>(Node.java:265) ~[elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-6.6.2.jar:6.6.2]
... 6 more
I have installed by tar.gz.
and I changed something of config/elasticsearch.yml
I didn't change anything else
someone know this error? please help me.
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /var/data/elasticsearch
#
# Path to log files:
#
path.logs: /var/logs/elasticsearch
#
# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
network.host: 0.0.0.0
#
# Set a custom port for HTTP:
#
http.port: 9200
#
# For more information, consult the network module documentation.
#
bootstrap.system_call_filter: false

You need to make sure that the directory on the host machine is owned by 1000:1000. Run the following command.
sudo chown 1000:1000 <directory you wish to mount>
Source

Please check the following:
Did you create /var/data/elasticsearch and /var/logs/elasticsearch directories in your filesystem?
Did you set the directories above to be writable with your user permission?
Update:
Read the source code of elasticsearch in Github. Link Here
It seemed that there some problem when the code is trying to access /etc/system-release. You might to take a look at that file. Also It mentioned that it is fallback for older Red Hat-like OS. So you might want to use another OS if possible.

In my variant trouble was in a permissions of data directory. Data directory was not default and changed to /elk-data/
To resolve the issue I make 'chmod -R elasticsearch:wheel /elk-data'

Related

Stop inter-node communication in elasticsearch (without using firewall to block ports)

I am looking to stop the inter-node communication in elasticsearch,basically to stop listening on port 9300.
Above should be achieved without using firewall to block any ports.
I tried closing the netty channel in the server bootstrap, if i do it elasticsearch wont start.
When i run elasticsearch instance it should not listen to 9300,
But i get below errors,
[2019-03-25T10:46:16,344][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [node-0] uncaught exception in thread [main]
[elasticsearch] org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: handler not set
[elasticsearch] at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
[elasticsearch] at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
[elasticsearch] at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
[elasticsearch] at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
[elasticsearch] at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
[elasticsearch] at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
[elasticsearch] at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
[elasticsearch] Caused by: java.lang.IllegalStateException: handler not set
[elasticsearch] at io.netty.bootstrap.Bootstrap.validate(Bootstrap.java:284) ~[?:?]
[elasticsearch] at io.netty.bootstrap.Bootstrap.connect(Bootstrap.java:113) ~[?:?]
[elasticsearch] at org.elasticsearch.transport.netty4.Netty4Transport.createClientBootstrap(Netty4Transport.java:189) ~[?:?]
[elasticsearch] at org.elasticsearch.transport.netty4.Netty4Transport.doStart(Netty4Transport.java:149) ~[?:?]
[elasticsearch] at org.elasticsearch.common.component.AbstractLifecycleComponent.start(AbstractLifecycleComponent.java:61) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
[elasticsearch] at org.elasticsearch.transport.TransportService.doStart(TransportService.java:228) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
[elasticsearch] at org.elasticsearch.common.component.AbstractLifecycleComponent.start(AbstractLifecycleComponent.java:61) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
[elasticsearch] at org.elasticsearch.node.Node.start(Node.java:661) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
[elasticsearch] at org.elasticsearch.bootstrap.Bootstrap.start(Bootstrap.java:263) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
[elasticsearch] at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:334) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
[elasticsearch] at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
Why not try the Single node discover ?
single-node (configure it by setting discovery.type to single-node); in this situation, a node will elect itself master and will not join a cluster with any other node. This is more clean way of doing this
Also, it is possible to evade the bootstrap checks (either by not binding transport to an external interface, or by binding transport to an external interface and setting the discovery type to single-node)
Set node.name to something other than 0.0.0.0/0

Elasticsearch Master not discovered exception

I'm running a 5 node elasticsearch cluster (2 data nodes, 2 master nodes, 1 kibana).
I'm getting the following error when use the command
curl -X GET "192.168.107.75:9200/_cat/master?v"
{"error":{"root_cause":[{"type":"master_not_discovered_exception","reason":null}
],"type":"master_not_discovered_exception","reason":null},"status":503}
I'm using the following command to run elastic
sudo systemctl start elasticsearch.service
This is the message I see in the logs:
[2018-05-28T21:02:22,074][WARN ][o.e.d.z.ZenDiscovery ] [node-master-1] not enough master nodes discovered during pinging (found [[Candidate{node={node-master-1}{kJKYkpdbTKmdIeq-RVnCAQ}{JGbXMxOXR0SyjCu746Zlwg}{192.168.107.75}{192.168.107.75:9300}, clusterStateVersion=-1}]], but needed [2]), pinging again
[2018-05-28T21:02:25,076][WARN ][o.e.d.z.ZenDiscovery ] [node-master-1] not enough master nodes discovered during pinging (found [[Candidate{node={node-master-1}{kJKYkpdbTKmdIeq-RVnCAQ}{JGbXMxOXR0SyjCu746Zlwg}{192.168.107.75}{192.168.107.75:9300}, clusterStateVersion=-1}]], but needed [2]), pinging again
[2018-05-28T21:02:28,077][WARN ][o.e.d.z.ZenDiscovery ] [node-master-1] not enough master nodes discovered during pinging (found [[Candidate{node={node-master-1}{kJKYkpdbTKmdIeq-RVnCAQ}{JGbXMxOXR0SyjCu746Zlwg}{192.168.107.75}{192.168.107.75:9300}, clusterStateVersion=-1}]], but needed [2]), pinging again
[2018-05-28T21:02:31,079][WARN ][o.e.d.z.ZenDiscovery ] [node-master-1] not enough master nodes discovered during pinging (found [[Candidate{node={node-master-1}{kJKYkpdbTKmdIeq-RVnCAQ}{JGbXMxOXR0SyjCu746Zlwg}{192.168.107.75}{192.168.107.75:9300}, clusterStateVersion=-1}]], but needed [2]), pinging again
[2018-05-28T21:02:34,081][WARN ][o.e.d.z.ZenDiscovery ] [node-master-1] not enough master nodes discovered during pinging (found [[Candidate{node={node-master-1}{kJKYkpdbTKmdIeq-RVnCAQ}{JGbXMxOXR0SyjCu746Zlwg}{192.168.107.75}{192.168.107.75:9300}, clusterStateVersion=-1}]], but needed [2]), pinging again
[2018-05-28T21:02:37,084][WARN ][o.e.d.z.ZenDiscovery ] [node-master-1] not enough master nodes discovered during pinging (found [[Candidate{node={node-master-1}{kJKYkpdbTKmdIeq-RVnCAQ}{JGbXMxOXR0SyjCu746Zlwg}{192.168.107.75}{192.168.107.75:9300}, clusterStateVersion=-1}]], but needed [2]), pinging again
[2018-05-28T21:02:40,090][WARN ][o.e.d.z.ZenDiscovery ] [node-master-1] failed to connect to master [{node-master-2}{_M4BTrFbQguT3PbY5d2_JA}{1rzJcDPSQ5OH2OZ_CnhR-g}{192.168.107.76}{192.168.107.76:9300}], retrying...
org.elasticsearch.transport.ConnectTransportException: [node-master-2][192.168.107.76:9300] connect_exception
at org.elasticsearch.transport.TcpChannel.awaitConnected(TcpChannel.java:165) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.transport.TcpTransport.openConnection(TcpTransport.java:616) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.transport.TcpTransport.connectToNode(TcpTransport.java:513) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:331) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:318) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.discovery.zen.ZenDiscovery.joinElectedMaster(ZenDiscovery.java:515) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.discovery.zen.ZenDiscovery.innerJoinCluster(ZenDiscovery.java:483) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.discovery.zen.ZenDiscovery.access$2500(ZenDiscovery.java:90) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.discovery.zen.ZenDiscovery$JoinThreadControl$1.run(ZenDiscovery.java:1253) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:573) [elasticsearch-6.2.4.jar:6.2.4]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_172]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_172]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_172]
Caused by: io.netty.channel.AbstractChannel$AnnotatedNoRouteToHostException: No route to host: 192.168.107.76/192.168.107.76:9300
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[?:?]
at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:323) ~[?:?]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:633) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:545) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:499) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) ~[?:?]
... 1 more
Caused by: java.net.NoRouteToHostException: No route to host
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[?:?]
at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:323) ~[?:?]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:633) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:545) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:499) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) ~[?:?]
... 1 more
In the ealsticsearch.yml file apart from the config for assigning different roles to nodes I'm using the following configuration:
cluster.name: test_cluster
network.host: 192.168.107.71
discovery.zen.ping.unicast.hosts: ["192.168.107.73", "192.168.107.74", "192.168.107.75", "192.168.107.76"]
#the above two configuration IPs change as per the node
discovery.zen.minimum_master_nodes: 2
The hosts are pingable and have access to each other.
Any help would be much appreciated.
I think the problem is quite clear, [node-master-2][192.168.107.76] either is not accessible from this host, or elastic process on [node-master-2] is down.
You can check if curl -XGET "192.168.107.76:9200" from this host has a valid answer.
Also elastic documents explicitly says:
It is recommended to avoid having only two master eligible nodes,
since a quorum of two is two. Therefore, a loss of either master
eligible node will result in an inoperable cluster.
This ElasticSearch install guide provides a guidance how to to fix master_not_discovered_exception exceptions. Basically you can get this error for several reasons:
Firewall rule is blocking communication
Master / Data host names cannot be resolved (won't be you case as you are using IP addresses)
Incorrect elasticsearch.yml configuration (e.g. master node is not configured as master node, or running on different port / IP address).
First and second item can easily checked with telnet (from master telnet to data node, and the other way around).

Unknown discovery type [gce]

Trying to configure GCE Discovery.
Have this in elasticsearch.yml
cloud:
gce:
project_id: ..........
zone: us-west1-a
discovery:
type: gce
Look at the log messages:
loaded plugin [discovery-gce]
but also failes with Unknown discovery type [gce]
.
[2018-01-16T09:11:25,247][INFO ][o.e.p.PluginsService ] [elasticsearch-cluster-2-node-2] loaded plugin [discovery-gce]
[2018-01-16T09:11:27,019][ERROR][o.e.b.Bootstrap ] Exception
java.lang.IllegalArgumentException: Unknown discovery type [gce]
at org.elasticsearch.discovery.DiscoveryModule.<init>(DiscoveryModule.java:110) ~[elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.node.Node.<init>(Node.java:441) ~[elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.node.Node.<init>(Node.java:245) ~[elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:322) [elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:130) [elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:121) [elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:69) [elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:134) [elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) [elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:85) [elasticsearch-6.0.0.jar:6.0.0]
What the heck is going on here? Any recommendation?
Apparently the EL maintaining team did a BWC change
In 5.1 we introduced the discovery.zen.hosts_provider setting to better reflect it. This PR removes BWC code in those plugins (GCE included) as it is not needed anymore.
- .put("discovery.type", "gce")
+ .put("discovery.zen.hosts_provider", "gce")
so the correct elasticsearch.yml configuration should reference
discovery:
zen.hosts_provider: gce

Titan: Connect to Elasticsearch from another machine

I'm using Titan 1.0.0 with Elasticsearch. I have Titan (with DynamoDB backend) working on an EC2 machine.
My main goal is connect to that Titan instance through another EC2 machine using Java.
Unfortunately I cannot connect to this machine.
My Titan instance is configured using a properties file. Here is a snippet of the Elasticsearch configuration:
# elasticsearch config
index.search.backend=elasticsearch
index.search.directory=/path/to/elasticsearch
index.search.elasticsearch.interface=NODE
index.search.elasticsearch.ext.node.data=true
index.search.elasticsearch.ext.node.client=false
index.search.elasticsearch.ext.node.local=false
This starts a full node holding data.
Now I want to connect to this node's Elasticsearch from another machine. My configuration file for this is:
storage.backend= com.amazon.titan.diskstorage.dynamodb.DynamoDBStoreManager
storage.hostname=10.0.0.249
storage.port=8182
index.search.backend=elasticsearch
index.search.elasticsearch.interface=TRANSPORT_CLIENT
index.search.elasticsearch.ext.node.data=false
index.search.elasticsearch.ext.node.client=true
index.search.hostname=10.0.0.249:9200
storage.dynamodb.client.endpoint=https://dynamodb.us-east-1.amazonaws.com
## DynamoDB client configuration: credentials
storage.dynamodb.client.credentials.class-name=com.amazonaws.auth.DefaultAWSCredentialsProviderChain
storage.dynamodb.client.credentials.constructor-args=
When I attempt to connect using Java through this line:
graph=TitanFactory.open("conf/dynamodb_remote.properties")
I get an error saying:
java.lang.IllegalArgumentException: Could not instantiate implementation: com.thinkaurelius.titan.diskstorage.es.ElasticSearchIndex
at com.thinkaurelius.titan.util.system.ConfigurationUtil.instantiate(ConfigurationUtil.java:55)
at com.thinkaurelius.titan.diskstorage.Backend.getImplementationClass(Backend.java:473)
at com.thinkaurelius.titan.diskstorage.Backend.getIndexes(Backend.java:460)
at com.thinkaurelius.titan.diskstorage.Backend.<init>(Backend.java:147)
at com.thinkaurelius.titan.graphdb.configuration.GraphDatabaseConfiguration.getBackend(GraphDatabaseConfiguration.java:1805)
at com.thinkaurelius.titan.graphdb.database.StandardTitanGraph.<init>(StandardTitanGraph.java:123)
at com.thinkaurelius.titan.core.TitanFactory.open(TitanFactory.java:94)
at com.thinkaurelius.titan.core.TitanFactory.open(TitanFactory.java:62)
at com.thinkaurelius.titan.core.TitanFactory$open.call(Unknown Source)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:110)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:122)
at groovysh_evaluate.run(groovysh_evaluate:3)
at org.codehaus.groovy.vmplugin.v7.IndyInterface.selectMethod(IndyInterface.java:215)
at org.codehaus.groovy.tools.shell.Interpreter.evaluate(Interpreter.groovy:69)
at org.codehaus.groovy.tools.shell.Groovysh.execute(Groovysh.groovy:185)
at org.codehaus.groovy.tools.shell.Shell.leftShift(Shell.groovy:119)
at org.codehaus.groovy.tools.shell.ShellRunner.work(ShellRunner.groovy:94)
at org.codehaus.groovy.tools.shell.InteractiveShellRunner.super$2$work(InteractiveShellRunner.groovy)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:324)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1207)
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuperN(ScriptBytecodeAdapter.java:130)
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuper0(ScriptBytecodeAdapter.java:150)
at org.codehaus.groovy.tools.shell.InteractiveShellRunner.work(InteractiveShellRunner.groovy:123)
at org.codehaus.groovy.tools.shell.ShellRunner.run(ShellRunner.groovy:58)
at org.codehaus.groovy.tools.shell.InteractiveShellRunner.super$2$run(InteractiveShellRunner.groovy)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:324)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1207)
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuperN(ScriptBytecodeAdapter.java:130)
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuper0(ScriptBytecodeAdapter.java:150)
at org.codehaus.groovy.tools.shell.InteractiveShellRunner.run(InteractiveShellRunner.groovy:82)
at org.codehaus.groovy.vmplugin.v7.IndyInterface.selectMethod(IndyInterface.java:215)
at org.apache.tinkerpop.gremlin.console.Console.<init>(Console.groovy:144)
at org.codehaus.groovy.vmplugin.v7.IndyInterface.selectMethod(IndyInterface.java:215)
at org.apache.tinkerpop.gremlin.console.Console.main(Console.groovy:303)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at com.thinkaurelius.titan.util.system.ConfigurationUtil.instantiate(ConfigurationUtil.java:44)
... 44 more
Caused by: org.elasticsearch.client.transport.NoNodeAvailableException: None of the configured nodes are available: []
at org.elasticsearch.client.transport.TransportClientNodesService.ensureNodesAreAvailable(TransportClientNodesService.java:279)
at org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:198)
at org.elasticsearch.client.transport.support.InternalTransportClusterAdminClient.execute(InternalTransportClusterAdminClient.java:86)
at org.elasticsearch.client.support.AbstractClusterAdminClient.health(AbstractClusterAdminClient.java:127)
at org.elasticsearch.action.admin.cluster.health.ClusterHealthRequestBuilder.doExecute(ClusterHealthRequestBuilder.java:92)
at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:91)
at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:65)
at com.thinkaurelius.titan.diskstorage.es.ElasticSearchIndex.<init>(ElasticSearchIndex.java:201)
... 49 more
I checked using wget and seems like ports 9200 and 9201 are working but 9300 is not. And probably that's why the issue exists.
Any help?
A couple suggestions based on the Titan configuration documentation
index.search.hostname should just be the hostname or IP address. It should not contain the port.
index.search.port if you decide to specify it, you should use 9300 or your Elasticsearch's value for the transport TCP port.
index.search.elasticsearch.cluster-name should match the cluster.name in the Elasticsearch config.
Updated: This seemed to work for me. In $TITAN_HOME/conf/mytitan.properties, I configured the indexing backend like this:
storage.backend=berkeleyje
storage.directory=../db/mytitan/berkeleyje
index.search.backend=elasticsearch
index.search.index-name=mytitan
index.search.elasticsearch.interface=NODE
index.search.conf-file=mytitan-elasticsearch.yml
And then $TITAN_HOME/conf/mytitan-elasticsearch.yml looks exactly like a regular ES configuration:
cluster.name: TitanElasticsearch
network.name: u1401
network.host: 192.168.14.101
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: ["192.168.14.101"]
discovery.zen.minimum_master_nodes: 1
node.name: u1401
node.master: true
node.data: true
http.port: 9200
transport.tcp.port: 9300
path.data: ./db/mytitan/elasticsearch
When I attempted to specify these properties with the prefix index.search.elasticsearch.ext..., the Transport TCP port didn't start as you noted earlier.

ElasticSearch failed to load plugins (License Module)

I have to load ElasticSearch plugins with the offline instructions but it looks like I am getting a permissions error even when I open up permissions to the files completely and running as root! I've removed and reinstalled as well but this error is consistent.
[root#ip-10-94-146-130 bin]# ./plugin install file:///home/jmblnvr/shield/license-2.1.0.zip
-> Installing from file:/home/jmblnvr/shield/license-2.1.0.zip...
Trying file:/home/jmblnvr/shield/license-2.1.0.zip ...
Downloading .DONE
Verifying file:/home/jmblnvr/shield/license-2.1.0.zip checksums if available ...
NOTE: Unable to verify checksum for downloaded plugin (unable to find .sha1 or .md5 file to verify)
Installed license into /usr/share/elasticsearch/plugins/license
[root#ip-10-94-146-130 bin]# service elasticsearch restart
Stopping elasticsearch: [FAILED]
Starting elasticsearch: Exception in thread "main" java.lang.IllegalStateException: Unable to initialize plugins
Likely root cause: java.nio.file.AccessDeniedException: /usr/share/elasticsearch/plugins/license/plugin-descriptor.properties
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214)
at java.nio.file.Files.newByteChannel(Files.java:361)
at java.nio.file.Files.newByteChannel(Files.java:407)
at java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:384)
at java.nio.file.Files.newInputStream(Files.java:152)
at org.elasticsearch.plugins.PluginInfo.readFromProperties(PluginInfo.java:86)
at org.elasticsearch.plugins.PluginsService.getPluginBundles(PluginsService.java:302)
at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:108)
at org.elasticsearch.node.Node.<init>(Node.java:148)
at org.elasticsearch.node.Node.<init>(Node.java:129)
at org.elasticsearch.node.NodeBuilder.build(NodeBuilder.java:145)
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:178)
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:285)
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:35)
Refer to the log for complete error details.
[FAILED]
From /var/log:
[root#ip-10-94-146-130 elasticsearch]# cat elasticsearch.log
[2015-12-15 11:44:24,525][INFO ][node ] [Valtorr] stopping ...
[2015-12-15 11:44:24,639][INFO ][node ] [Valtorr] stopped
[2015-12-15 11:44:24,641][INFO ][node ] [Valtorr] closing ...
[2015-12-15 11:44:24,647][INFO ][node ] [Valtorr] closed
[2015-12-15 11:44:26,097][WARN ][bootstrap ] unable to install syscall filter: seccomp unavailable: CONFIG_SECCOMP not compiled into kernel, CONFIG_SECCOMP and CONFIG_SECCOMP_FILTER are needed
[2015-12-15 11:44:26,314][INFO ][node ] [Agon] version[2.1.0], pid[21703], build[72cd1f1/2015-11-18T22:40:03Z]
[2015-12-15 11:44:26,314][INFO ][node ] [Agon] initializing ...
[2015-12-15 11:44:26,318][ERROR][bootstrap ] Exception
java.lang.IllegalStateException: Unable to initialize plugins
at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:111)
at org.elasticsearch.node.Node.<init>(Node.java:148)
at org.elasticsearch.node.Node.<init>(Node.java:129)
at org.elasticsearch.node.NodeBuilder.build(NodeBuilder.java:145)
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:178)
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:285)
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:35)
Caused by: java.nio.file.AccessDeniedException: /usr/share/elasticsearch/plugins/license/plugin-descriptor.properties
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214)
at java.nio.file.Files.newByteChannel(Files.java:361)
at java.nio.file.Files.newByteChannel(Files.java:407)
at java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:384)
at java.nio.file.Files.newInputStream(Files.java:152)
at org.elasticsearch.plugins.PluginInfo.readFromProperties(PluginInfo.java:86)
at org.elasticsearch.plugins.PluginsService.getPluginBundles(PluginsService.java:302)
at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:108)
... 6 more
I was having the same sort of exception but after installing and while loading x-pack plugin on Elasticsearch 5.2.1:
Caused by: java.nio.file.AccessDeniedException: /usr/share/elasticsearch/plugins/x-pack/plugin-descriptor.properties
In my case, I started by adding elasticsearch user to the root group:
usermod -aG root elasticsearch
Afterwards, it was only necessary to modify the ownership of the plugins folder:
chown -R elasticsearch:elasticsearch /usr/share/elasticsearch/plugins/
Thanks to Val, simple solution is after installing plugins you need to quickly change ownership/groups if you are using the RPM install of Elastic:
chown -R root:elasticsearch /usr/share/elasticsearch

Resources