I have a strange problem when trying to setup a local running elasticsaerch cluster. I have tried a lot of settings with the bind.host and the auto discovery but did not succeed. The strange thing is if I have my wireless connection connected to a network the two nodes do not see each other. If I switch it of I do not have a problem. Following is the configuration of one node that works without a wireless adapter connected.
Does anyone have a clue what to try to make my cluster work?
cluster.name: localcluster
node.name: two
index.number_of_shards: 1
index.number_of_replicas: 0
network.host: _lo0:ipv4_
zen.ping.multicast.enabled: false
zen.ping.unicast.hosts: ["127.0.0.1"]
You are missing discovery. in the zen discovery settings. Try these settings:
cluster.name: "localcluster"
network.host: _lo0:ipv4_
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: ["127.0.0.1:9300","127.0.0.1:9301","127.0.0.1:9302"]
Related
First of all, I want to be clear that I looked at several guides and suggested similar questions before opening this post, but none of them worked for our case.
Here is our situation:
Last week, our Elasticsearch stopped adding new records as it reached the state where the disk was almost full. We changed the config to get some time and now is working as expected. However, we want to add a new server with elasticsearch and form a cluster as we don't want to resize the disk because we don't want to lose anything.
Here is the configuration for the main server /etc/elasticsearch/elasticsearch.yml
cluster.name: my-cluster
node.name: master-node
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 0.0.0.0
http.max_content_length: 100mb
discover.seed_hosts: ["ip_address_server_1", "ip_address_server_2"]
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.http.ssl.enabled: true
As I said, we created another server with the same elasticsearch version (7.6.2)
cluster.name: my-cluster
node.name: another-node
node.data: true
node.master: false
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: ip_of_server
discover.seed_hosts: ["ip_address_server_1", "127.0.0.1"]
We tried restarting nodes at the same time, curls and everything but they can't see each other. At this point, I'm starting to think that the problem is the SSL configuration on the master, but I am not an elasticsearch expert, so I don't know what is exactly happening.
EDIT:
I took a look at the logs and here is what I found:
Node 2
[ico-elastic-node-2] master not discovered yet: have discovered [{node-2}{G5SfrEv0RxaxmYf8urIFtQ}{XpfNNouCQx2HmYfw2AvoQw}{ip_server_2}{ip_server_29300}{dil}{ml.machine_memory=12558602240, xpack.installed=true, ml.max_open_jobs=20}]; discovery will continue using [ip_server_1, 127.0.0.1:9300] from hosts providers and [] from last-known cluster state; node term 0, last-accepted version 0 in term 0
And the master:
[master-node] exception caught on transport layer [Netty4TcpChannel{localAddress=/ip:9300, remoteAddress=/ip:36660}], closing connection
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: No available authentication scheme
Currently master is yellow. What can we try next?
I have an ES cluster (v 5.6.12) up and running in dev mode, config below:
node1.com
cluster.name: elastic-test
node.name: "node-1"
path.data: /path/to/data
path.logs: /path/to/logs
network.host: 127.0.0.1
http.host: 0.0.0.0
discovery.zen.ping.unicast.hosts: ["node1.com", "node2.com"]
node.master: true
I am trying to connect node 2 to the same cluster:
node2.com
cluster.name: elastic-test
node.name: "node-2"
path.data: /path/to/data
path.logs: /path/to/logs
network.host: 127.0.0.1
http.host: 0.0.0.0
discovery.zen.ping.unicast.hosts: ["node1.com", "node2.com"]
node.master: true
I tried to change the network.host to their respective addresses, but this takes them out of dev mode. I also tried setting the bind and publish hosts to make the node discover-able to other nodes:
network.bind_host: 127.0.0.1
network.publish_host: node1.com
But again, this takes the nodes into production.
Is it actually possible to have multiple nodes on different servers communicate within development mode?
Short answer NO. For most use cases running a single node cluster for DEV suffices but there could be scenarios where multi node clusters are required in DEV environment, however it is not possible to currently form a multi node cluster without binding to a non local IP address.
That being said, difference between development mode and production mode with respect to Elasticsearch is just preventing ES cluster from starting if some settings are not configured appropriately. So, as long as you are able to configure the settings described in the below link then you can form a cluster and name it as DEV so users don't misidentify it as a production cluster
https://www.elastic.co/guide/en/elasticsearch/reference/5.6/system-config.html#dev-vs-prod
I have one code base to connect elastic search (localhost:9200) for the full-text-search feature. We deployed this code on two different machines (m1 & m2) under load balancing server. In this case, how to configure ES in 2 different machines to connect ES and index should reflect both sides.
I am using Elasticsearch v 5.1.2
Machine 1
cluster.name: production
node.name: database
Machine 2
cluster.name: production
node.name: app
Above setting worked on ES v 1.7.1
**Question?
What configuration should I do to make it work on ES v5.1.2?
Please help me to solve this issue.
Thanks in advance
I'm assuming these nodes aren't a part of same cluster.
Try http://MACHINE_1_IP:9200/_cat/nodes?v and check if all nodes are listed as part of cluster.
If they are not - just a quick guess, have you looked at network.host setting ? It binds to local loop by default ( That maybe something introduced in 2 + )
This can be solved by using network-module setting (ref).
Update the elasticsearch.yml on both app server by keeping same cluster name and different node name
EX :
Server_1
update the elasticseach.yml
cluster.name: Production
node.name: APP
network.host: [server_1_IP, _local_]
discovery.zen.ping.unicast.hosts: [server_1_IP, server_2_IP]
On Server_2
update the elasticseach.yml
cluster.name: Production
node.name: DB
network.host: [server_2_IP, _local_]
discovery.zen.ping.unicast.hosts: [server_1_IP, server_2_IP]
I have 2 separate machines. Port 9200 is already taken by a separate elasticsearch running, so I specify 9201 as the http.port in the yml file. i set cluster.name: MyCluster.
When I start ./elasticsearch on machine 1 and machine 2, they are not connected, but each are single node master's.
What do I need to do so that they can connect to each other and be part of the same cluster?
I also set network.host: 0.0.0.0 so I know they can see each other. I am using 2.4.0 of Elastcisearch.
In machine 1:
cluster.name: hello_world
network.host: "hostname_or_ip_1"
network.port: 9201
discovery.zen.ping.unicast.hosts: ["hostname_or_ip_2:9201"]
In machine 2:
cluster.name: hello_world
network.host: "hostname_or_ip_2"
network.port: 9201
discovery.zen.ping.unicast.hosts: ["hostname_or_ip_1:9201"]
Both cluster name should be same
discovery.zen.ping.unicast.hosts should point to correct machine
address with port
Make sure to restart elasticsearch node after editing config file
Look at unicast discovery with host:port. https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-discovery-zen.html
You might also need to be explicit about the transport.tcp.port in your elasticsearch.yml:
transport.tcp.port: 9301
Assuming I have 5 machines I want to run an elasticsearch cluster on, and they are all connected to a shared drive. I put a single copy of elasticsearch onto that shared drive so all three can see it. Do I just start the elasticsearch on that shared drive on eall of my machines and the clustering would automatically work its magic? Or would I have to configure specific settings to get the elasticsearch to realize that its running on 5 machines? If so, what are the relevant settings? Should I worry about configuring for replicas or is it handled automatically?
its super easy.
You'll need each machine to have it's own copy of ElasticSearch (simply copy the one you have now) -- the reason is that each machine / node whatever is going to keep it's own files that are sharded accross the cluster.
The only thing you really need to do is edit the config file to include the name of the cluster.
If all machines have the same cluster name elasticsearch will do the rest automatically (as long as the machines are all on the same network)
Read here to get you started:
https://www.elastic.co/guide/en/elasticsearch/guide/current/deploy.html
When you create indexes (where the data goes) you define at that time how many replicas you want (they'll be distributed around the cluster)
It is usually handled automatically.
If autodiscovery doesn't work. Edit the elastic search config file, by enabling unicast discovery
Node 1:
cluster.name: mycluster
node.name: "node1"
node.master: true
node.data: true
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: ["node1.example.com"]
Node 2:
cluster.name: mycluster
node.name: "node2"
node.master: false
node.data: true
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: ["node1.example.com"]
and so on for node 3,4,5. Make node 1 master, and the rest only as data nodes.
Edit: Please note that by ES rule, if you have N nodes, then by convention, N/2+1 nodes should be masters for fail-over mechanisms They may or may not be data nodes, though.
Also, in case auto-discovery doesn't work, most probable reason is because the network doesn't allow it (and therefore disabled). If too many auto-discovery pings take place across multiple servers, the resources to manage those pings will prevent other services from running correctly.
For ex, think of a 10,000 node cluster and all 10,000 nodes doing the auto-pings.
Elastic Search 7 changed the configurations for cluster initialisation.
What is important to note is the ES instances communicate internally using the Transport layer(TCP) and not the HTTP protocol which is normally used to perform ops on the indices. Below is sample config for 2 machines cluster.
cluster.name: cluster-new
node.name: node-1
node.master: true
node.data: true
bootstrap.memory_lock: true
network.host: 0.0.0.0
http.port: 9200
transport.host: 102.123.322.211
transport.tcp.port: 9300
discovery.seed_hosts: [“102.123.322.211:9300”,"102.123.322.212:9300”]
cluster.initial_master_nodes:
- "node-1"
- "node-2”
Machine 2 config:-
cluster.name: cluster-new
node.name: node-2
node.master: true
node.data: true
bootstrap.memory_lock: true
network.host: 0.0.0.0
http.port: 9200
transport.host: 102.123.322.212
transport.tcp.port: 9300
discovery.seed_hosts: [“102.123.322.211:9300”,"102.123.322.212:9300”]
cluster.initial_master_nodes:
- "node-1"
- "node-2”
cluster.name: This has be same across all the machines that are going to be part of a cluster.
node.name : Identifier for the ES instance. Defaults to machine name if not given.
node.master: specifies whether this ES instance is going to be master or not
node.data: specifies whether this ES instance is going to be data node or not(hold data)
bootsrap.memory_lock: disable swapping.You can start the cluster without setting this flag. But its recommended to set the lock.More info: https://www.elastic.co/guide/en/elasticsearch/reference/master/setup-configuration-memory.html
network.host: 0.0.0.0 if you want to expose the ES instance over network. 0.0.0.0 is different from 127.0.0.1( aka localhost or loopback address).
It means all IPv4 addresses on the machine. If machine has multiple ip addresses with a server listening on 0.0.0.0, the client can reach the machine from any of the IPv4 addresses.
http.port: port on which this ES instance will listen to for HTTP requests
transport.host: The IPv4 address of the host(this will be used to communicate with other ES instances running on different machines). More info: https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-transport.html
transport.tcp.port: 9300 (the port where the machine will accept the tcp connections)
discovery.seed_hosts: This was changed in recent versions. Initialise all the IPv4 addresses with TCP port(important) of ES instances that are going to be part of this cluster. This is going to be same across all ES instances that are part of this cluster.
cluster.initial_master_nodes: node names(node.name) of the ES machines that are going to participate in master election.(Quorum based decision making :- https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-discovery-quorums.html#modules-discovery-quorums)
I tried the steps that #KannarKK suggested on ES 2.0.2, however, I could not bring the cluster up and running. Evidently, I figured out something, as I had set tcp port number on Master, on the Slave configuration discovery.zen.ping.unicast.hosts needs Master's port number along with IP address ( tcp port number ) for discovery. So when I try following configuration it works for me.
Node 1
cluster.name: mycluster
node.name: "node1"
node.master: true
node.data: true
http.port : 9200
tcp.port : 9300
discovery.zen.ping.multicast.enabled: false
# I think unicast.host on master is redundant.
discovery.zen.ping.unicast.hosts: ["node1.example.com"]
Node 2
cluster.name: mycluster
node.name: "node2"
node.master: false
node.data: true
http.port : 9201
tcp.port : 9301
discovery.zen.ping.multicast.enabled: false
# The port number of Node 1
discovery.zen.ping.unicast.hosts: ["node1.example.com:9300"]