Assume there are 3 running nodes launched with multicast=false and unicast=true, but no unicast nodes are given when starting. After they all get up, they are not aware of each other.
Is there a way to tell each one IP address of the other two so they can do discovery at runtime and join to same cluster?
Yes, add the ip addresses of all the other nodes in the cluster to the
discovery.zen.ping.unicast.hosts property,
in the elasticsearch.yml file in the config folder.
Say you have three nodes, in each node, add the address of the other two nodes as below:
discovery.zen.ping.unicast.hosts: ["xx.xx.xxx.xx","yy.yy.yy.yy"]
Related
I'm trying to automate the process of horizontal scale up and scale down of elasticsearch nodes in kubernetes cluster.
Initially, I deployed an elasticsearch cluster (3 master, 3 data & 3 ingest nodes) on a Kubernetes cluster. Where, cluster.initial_master_nodes was:
cluster.initial_master_nodes:
- master-a
- master-b
- master-c
Then, I performed scale down operation, reduced the number of master node 3 to 1 (unexpected, but for testing purpose). While doing this, I deleted master-c, master-b nodes and restarted master-a node with the following setting:
cluster.initial_master_nodes:
- master-a
Since the elasticsearch nodes (i.e. pods) use persistant volume, after restarting the node, the master-a slowing the following logs:
"message": "master not discovered or elected yet, an election requires at least 2 nodes with ids from [TxdOAdryQ8GAeirXQHQL-g, VmtilfRIT6KDVv1R6MHGlw, KAJclUD2SM6rt9PxCGACSA], have discovered [] which is not a quorum; discovery will continue using [] from hosts providers and [{master-a}{VmtilfRIT6KDVv1R6MHGlw}{g29haPBLRha89dZJmclkrg}{10.244.0.95}{10.244.0.95:9300}{ml.machine_memory=12447109120, xpack.installed=true, ml.max_open_jobs=20}] from last-known cluster state; node term 5, last-accepted version 40 in term 5" }
Seems like it's trying to find master-b and master-c.
Questions:
How to overwrite cluster settings so that master-a won't search for these deleted nodes?
The cluster.initial_master_nodes setting only has an effect the first time the cluster starts up, but to avoid some very rare corner cases you should never change its value once you've set it and generally you should remove it from the config file as soon as possible. From the reference manual regarding cluster.initial_master_nodes:
You should not use this setting when restarting a cluster or adding a new node to an existing cluster.
Aside from that, Elasticsearch uses a quorum-based election protocol and says the following:
To be sure that the cluster remains available you must not stop half or more of the nodes in the voting configuration at the same time.
You have stopped two of your three master-eligible nodes at the same time, which is more than half of them, so it's expected that the cluster no longer works.
The reference manual also contains instructions for removing master-eligible nodes which you have not followed:
As long as there are at least three master-eligible nodes in the cluster, as a general rule it is best to remove nodes one-at-a-time, allowing enough time for the cluster to automatically adjust the voting configuration and adapt the fault tolerance level to the new set of nodes.
If there are only two master-eligible nodes remaining then neither node can be safely removed since both are required to reliably make progress. To remove one of these nodes you must first inform Elasticsearch that it should not be part of the voting configuration, and that the voting power should instead be given to the other node.
It goes on to describe how to safely remove the unwanted nodes from the voting configuration using POST /_cluster/voting_config_exclusions/node_name when scaling down to a single node.
Cluster state which also stores the master configuration stores on the data folder of Elasticsearch node, In your case, it seems it is reading the old-cluster state(which is 3 master nodes, with their ids).
Could you delete the data folder of your master-a, so that it can start from a clean cluster state and it should resolve your issue.
Also make sure, other data and ingest node have master.node:false setting as by default it's true.
I have 3 elastic node , How can I cluster there three nodes with always same master node , I didn't find any good docs about new elastic 7 way of specify discovery and master node:
discovery.seed_hosts: [ ]
cluster.initial_master_nodes: []
for example I have node a, b, c and I want node a to be master what what should be discovery.seed_hosts and cluster.initial_master_nodes for master node and child nodes
UPDATE
with using Daniel answer , and checking ports are open and node have same cluster name , other nodes didn't join cluster, is there any additional config needed?
UPDATE 2
looks like nodes found each other but for some reason can't choose master node with election:
master not discovered or elected yet, an election requires 2 nodes
with ids [wOZEfOs9TvqGWIHHcKXtkQ, Cs0xaF-BSBGMGB8a-swznA]
Solution
Deleting folder data of all nodes start a node and then add other nodes with first node (as master) as seed host.
Elasticsearch allows you to specify the role of a node. A node (an instance of Elasticsearch) can serve as a coordinating node, master node, voting_only node, data node, ingest node or machine learning node.
With respect to master nodes you can only configure which nodes potentially can become the (active) master, but you cannot specify which one of the so-called master-eligible nodes will be the active master node.
The only exception to this is when you only configure one master-eligible node, then obviously only this one can become the active master. But be aware that in order to get true high availability you need to have at least 3 master-eligible nodes (this ensures that your cluster will still be 100% operational even when losing one of the master-eligible nodes).
Therefore Elastic always recommends to configure 3 or 5 nodes in your cluster as master-eligible nodes. You can configure that role via the node.master property in the Elasticsearch.yml-file. Setting it to true (default) allows that node to become master, while false will ensure that this node never ever will become master and also will not participate in the master election.
Over the life-time of your cluster (master-eligible) nodes might get added and removed. Elasticsearch automatically manages your cluster and the master node election process with the ultimate goal to prevent a split brain scenario from happening, meaning you eventually end up having 2 clusters which go by the same name but with independent master nodes. To prevent that from happening when starting up your cluster for the very first time (bootstrapping your cluster) Elastic requires you to configure the cluster.initial_master_nodes property with the names of the nodes that initially will serve as master-eligible nodes. This property only needs to be configured on nodes that are master-eligible and the setting will only be considered for the very first startup of your cluster. As values you put in the names as configured with the node.name property of your master-eligible nodes.
The discovery.seed_hosts property supports the discovery process which is all about enabling a new node to establish communication with an already existing cluster and eventually joining it when the cluster.name matches. You are supposed to configure it with an array of host names (not node names!) on which you expect other instances of Elasticsearch belonging to the same cluster to be running. You don't need to add all 100 host names of the 100 nodes you may have in your cluster. It's sufficient to list host names of the most stable node names there. As master (eligible) nodes are supposed to be very stable nodes, Elastic recommends to put the host of all master-eligible nodes (typically 3) in there. Whenever you start/restart a node, it goes through this discovery process.
Conclusion
With a cluster made up of 3 nodes you would configure all of them as master-eligible nodes and list the 3 node names in the cluster.initial_master_nodes setting. And you would put all the 3 host names also in the discovery.seed_hosts setting to support the discovery process.
Useful information from the Elasticsearch reference:
Important discovery and cluster formation settings
Discovery and cluster formation settings
Bootstrapping a cluster
I've just created three dedicated master nodes and one data node for my cluster.
Now, I need to configure an initial list of nodes that will be contacted to discover and form a cluster.
I included IP addresses of the three dedicated master nodes as the values of discovery.zen.ping.unicast.hosts, but should I also include the IP address of the data node?
Adding IP address of 3 master nodes is sufficient. Data node's IP Address is not necessary.
I have ELK stack with Elasticsearch, Logstash and kibana installed on 3 different instances.
Now I want to make 3 node cluster of Elasticsearch.
I will make one node as master and 2 data nodes.
I want to know in logstash config
elasticsearch {
hosts => "http://es01:9200"
Which address I need to enter there master node or data node. and also if I have 3 master nodes then which address I need to write there.
similarly in kibana , I use
elasticsearch.url: es01:9200
In cluster env which url I need to use?
In general, the answer depends on your cluster data size and load.
Nevertheless, I'll try to answer your questions assuming the master node is not a data eligible node as well. This means it only takes care for cluster-wide actions such as creating or deleting an index, tracking which nodes are part of the cluster, and deciding which shards to allocate to which nodes. For this purposes, it is very recommended to have your master node as stable and less loaded as possible.
So, in your logstash config I would put the addresses of your two data nodes as follows:
elasticsearch{
hosts => ["http://es01:9200", "http://es02:9200"]
}
This confirmation maximize performance and fault tolerance as your master do not contain data and if one node failes it will continue to work with the other.
Please note that it is very recommended to have at least 3 master eligible nodes configured in Elasticsearch clusters since if you are loosing the (only) master node you loose data. 3 is to avoid split brain
Regarding kibana, since all nodes in the cluster "knows" each other. You basically can put any address in the cluster. But, for the same reasons as above it is recommended to fill one of your data nodes addresses.
For further reading, please refer to this documentation.
Hope I have managed to help!
I can't seem to find this in the docs.
When I specify discovery.zen.ping.unicast.hosts do I need to specify every node in the cluster? I'm hoping that I can specify a subset of the nodes, and the node will discover the cluster through whatever protocol they're using (gossip?). Will it use the discovery.zen.ping.unicast.hosts for master election, or will it use whatever state it's aware of?
No, it's enough to specify the master-eligible nodes. Since every node in the cluster should know the entire cluster setup (through the cluster state), a new node joining the cluster will "ask" the nodes from the discovery.zen.ping.unicast.hosts list to share the cluster state. From this it will learn of all other nodes' whereabouts.