I have a 3 node cluster with minimum_master_nodes set to 2. If I shut down all nodes except the master, leaving one node online, the cluster is no longer operational.
Is this by design? It seems like the node that was the master shouldd remain operational, instead I get errors like this:
{"error":"MasterNotDiscoveredException[waited for [30s]]","status":503}
All the other settings are stock and I am using the aws cloud plugin.
Yes, this is intentional.
Split brain
Imagine a situation where the other 2 nodes were still running but couldn't communicate to the the third node - you'd end up with two clusters otherwise known as a "split brain".
As the two clusters could be updating and deleting data independently of each other then recovery would be very difficult - you wouldn't have a single source of truth for the data.
By setting minimum_master_nodes to (n/2)+1 (were n is the number of nodes) you can prevent a split brain.
Single Node
If you know that the first two nodes have definitely died and not coming back - you can set the minimum_master_nodesto 1 on the remaining node (and also set to one on the other nodes before you restart them).
There is also an option no master block that lets you control what happens when you don't have a valid cluster - e.g. you could make the remaining node read-only until the cluster is re-established.
Related
I'm trying to automate the process of horizontal scale up and scale down of elasticsearch nodes in kubernetes cluster.
Initially, I deployed an elasticsearch cluster (3 master, 3 data & 3 ingest nodes) on a Kubernetes cluster. Where, cluster.initial_master_nodes was:
cluster.initial_master_nodes:
- master-a
- master-b
- master-c
Then, I performed scale down operation, reduced the number of master node 3 to 1 (unexpected, but for testing purpose). While doing this, I deleted master-c, master-b nodes and restarted master-a node with the following setting:
cluster.initial_master_nodes:
- master-a
Since the elasticsearch nodes (i.e. pods) use persistant volume, after restarting the node, the master-a slowing the following logs:
"message": "master not discovered or elected yet, an election requires at least 2 nodes with ids from [TxdOAdryQ8GAeirXQHQL-g, VmtilfRIT6KDVv1R6MHGlw, KAJclUD2SM6rt9PxCGACSA], have discovered [] which is not a quorum; discovery will continue using [] from hosts providers and [{master-a}{VmtilfRIT6KDVv1R6MHGlw}{g29haPBLRha89dZJmclkrg}{10.244.0.95}{10.244.0.95:9300}{ml.machine_memory=12447109120, xpack.installed=true, ml.max_open_jobs=20}] from last-known cluster state; node term 5, last-accepted version 40 in term 5" }
Seems like it's trying to find master-b and master-c.
Questions:
How to overwrite cluster settings so that master-a won't search for these deleted nodes?
The cluster.initial_master_nodes setting only has an effect the first time the cluster starts up, but to avoid some very rare corner cases you should never change its value once you've set it and generally you should remove it from the config file as soon as possible. From the reference manual regarding cluster.initial_master_nodes:
You should not use this setting when restarting a cluster or adding a new node to an existing cluster.
Aside from that, Elasticsearch uses a quorum-based election protocol and says the following:
To be sure that the cluster remains available you must not stop half or more of the nodes in the voting configuration at the same time.
You have stopped two of your three master-eligible nodes at the same time, which is more than half of them, so it's expected that the cluster no longer works.
The reference manual also contains instructions for removing master-eligible nodes which you have not followed:
As long as there are at least three master-eligible nodes in the cluster, as a general rule it is best to remove nodes one-at-a-time, allowing enough time for the cluster to automatically adjust the voting configuration and adapt the fault tolerance level to the new set of nodes.
If there are only two master-eligible nodes remaining then neither node can be safely removed since both are required to reliably make progress. To remove one of these nodes you must first inform Elasticsearch that it should not be part of the voting configuration, and that the voting power should instead be given to the other node.
It goes on to describe how to safely remove the unwanted nodes from the voting configuration using POST /_cluster/voting_config_exclusions/node_name when scaling down to a single node.
Cluster state which also stores the master configuration stores on the data folder of Elasticsearch node, In your case, it seems it is reading the old-cluster state(which is 3 master nodes, with their ids).
Could you delete the data folder of your master-a, so that it can start from a clean cluster state and it should resolve your issue.
Also make sure, other data and ingest node have master.node:false setting as by default it's true.
I know that it is possible to define more than one master for the ElasticSearch cluster, where only one acts as master and the others can step in if necessary. See also https://stackoverflow.com/a/15022820/2648551 .
What I don't understand is how I can determine which master is active and which could step in if necessary.
The following setting I currently have:
node-01: master (x) data(-)
node-02: master (-) data(x)
node-03: master (-) data(x)
node-04: master (-) data(x)
node-05: master (-) data(x)
node-06: master (-) data(x)
Now I want to determine that e.g. node-02 becomes additionally a master eligible. Can I rely on ES being so smart that it always takes the non-data node (node-01) as the active master, or could it be that node-02 ever acts as the active master if all nodes are present and there are no problems? Or is that something I just don't have to worry about?
I am currently using ElasticSearch 1.7 [sic!], but I am also interested in answers based on the latest versions.
A few laters and just for the context we "can" now decide which node becomes master, although not straight forward its possible.
Elasticsearch now has an method called voting_config_exclusions which can be used to move away from current master node e.g.
lets say you have 3 master-eligible nodes in your cluster
$ GET _cat/nodes?v
ip node.role master name
192.168.0.10 cdfhilmrstw - node-10
192.168.0.20 cdfhilmrstw * node-20
192.168.0.30 cdfhilmrstw - node-30
192.168.0.99 il - node-99
and Elasticsearch has selected node-20 as active master, you can run following call to remove the active node from voting.
POST /_cluster/voting_config_exclusions?node_names=node_name
This will randomly select another master-eligible node as master (if you have more than one left) Keep doing this for the active nodes until you get the right one activated as master.
Note: this doesn't remove the node, only makes it in-active master / non-voting node and allows another node to become active as master.
Once done, make sure to run below command to remove exclusions and allow all eligible nodes to become master if and when the selected node goes down.
DELETE /_cluster/voting_config_exclusions
Thank You
In short, no, you can't decide which of the master eligible nodes will become a master, because master node is elected (it was in ES 1.7, it still is in ES 6.2).
No, you can't rely on Elasticsearch being so smart to always take the non-data node as the active master. In fact, as of now (6.2) they advice to have dedicated master nodes (i.e. those that do not perform any data operations):
To ensure that your master node is stable and not under pressure, it
is a good idea in a bigger cluster to split the roles between
dedicated master-eligible nodes and dedicated data nodes.
... It is important
for the stability of the cluster that master-eligible nodes do as
little work as possible.
(Note that they are talking about a "bigger cluster".)
I can only assume that this also holds for the earlier versions and the documentation just got reacher.
There is a problem with the configuration that you have posted. Although you have many nodes, loss of one (the master node, node-01) will make your cluster non-functional. To avoid this situation you may choose one of these options:
use default strategy and make all nodes data nodes and master nodes;
make a set of dedicated master-only nodes (at least 3 of them).
It would be nice to know the reason why the ES defaults are not good enough for you, because usually they are good enough.
However, if this is the case when you need a dedicated master node, make sure you have at least 3 and that discovery.zen.minimum_master_nodes is enough to avoid the "split brain" situation:
discovery.zen.minimum_master_nodes = (master_eligible_nodes / 2) + 1
Hope that helps!
I have an elastic search cluster with 5 master nodes. I want to remove 3 master nodes. How can I do that.
I would like to make changes to the config file as well.
Please let me know the step by step procedure.
Any leads would be highly appreciated.
You should never remove 3 master nodes if you have 5 (because then you'll have an even number, 2, and you can encounter the split brain problem).
I think the safest option is to first reduce to 3, and then reduce to 1 if you really need to. To reduce to 3:
Pick one of the nodes you want to keep; update its elasticsearch.yml file to set minimum_master_nodesat 2
Restart that node
Wait for it to rejoin the cluster
Repeat steps 1-3 for the other two nodes you are keeping
Stop the nodes you're removing
If you need to then reduce to 1 node, update the elasticsearch.yml on the node you want to keep with minimum_master_nodes of 1, restart and wait for it to join the cluster, and then simply remove the other two.
You could also consider changing another node in your cluster (a client, data, or ingest node) to also serve as a master node -- that is, keep 2 master-only nodes, and update one of the other nodes to also serve as a master node.
In my production environment, I have a two-node cluster (ES 2.2.0) and each node sits on a different physical box. Inside elasticsearch.yml, I have the following:
discovery.zen.minimum_master_nodes: 2
My question is: if one box is down, can the other node continues to function normally to provide uninterrupted search services (index and search, write and read)?
If you have two nodes and each is master-eligible and you have discovery.zen.minimum_master_nodes: 2, if the network goes down and the two nodes don't see each other for a while, you'll get into a split brain situation because each node will elect itself as a master.
However, with a setting of 2, you have two possible situations:
if the non-master goes down, the other node will continue to function properly (since it is already master)
if the master goes down, the other won't be able to elect itself as the master (since it will wait for a second master-eligible node to be visible).
For this reason, with only two nodes, you need to choose between the possibility of a split brain (with minimum_master_nodes: 1) or a potentially RED cluster (with minimum_master_nodes: 2). The best way to overcome this is to include a third master-only node and then minimum_master_nodes: 2 would make sense.
Just try it out:
Start your cluster, bring down the master node, what happens?
Start your cluster, bring down the non-master node, what happens?
The purpose of minimum master nodes is to maintain the stability of the cluster.
If you have only 2 nodes in cluster and with 2 minimum master nodes settings.
If you are setting minimum master as 2, the cluster will expect 2 nodes to be UP to serve the various search services.
If one node goes down in your 2 node cluster (which had 2 minimum master node settings), theoretically cluster will goes down.
First, this setting helps prevent split brains, the existence of two masters in a single cluster.
If you have two nodes, A setting of 1 will allow your cluster to function, but doesn’t protect against split brain. It is best to have a minimum of three nodes in situations.
I have two dedicated Windows Servers (Windows Server 2012R2, 128GB memory on each server) for ES (2.2.0). If I have one node on each server and the two nodes form a cluster. What is the proper value for
discovery.zen.minimum_master_nodes
I read this general rule in elasticsearch.yml:
Prevent the "split brain" by configuring the majority of nodes (total
number of nodes / 2 + 1):
I saw this SO thread:
Proper value of ES_HEAP_SIZE for a dedicated machine with two nodes in a cluster
There is an answer saying:
As described in Elasticsearch Pre-Flight Checklist, you can set
discovery.zen.minimum_master_nodes to at least (N/2)+1 on clusters
with N > 2 nodes.
Please note "N > 2". What is the proper value in my case?
N is the number of ES nodes (not physical machines but ES processes) that can be part of the cluster.
In your case, with one node on two machines, N = 2 (note that it was 4 here), so the formula N/2 + 1 yields 2, which means that both of your nodes MUST be eligible as master nodes if you want to prevent split brain situations.
If you set that value to 1 (which is the default value!) and you experience networking issues and both of your nodes can't see each other for a brief moment, each node will think it is alone in the cluster and both will elect themselves as master. You end up in a situation where you have two masters and that's not a good thing. Whereas if you set that value to 2 and you experience networking issues, the current master node will stay elected and the second node will never decide to elect itself as new master. Whenever network is back up, the second node will rejoin the cluster and continue serving requests.
The ideal topology is to have 3 dedicated master nodes (i.e. with master: true and data:false) and have discovery.zen.minimum_master_nodes set to 2. That way you'll never have to change the setting whatever the number of data nodes are part of your cluster.
So the N > 2 constraint should indeed be N >= 2, but I guess it was somehow implied, because otherwise you're creating a fertile ground for split brain situations.
Interestingly, in ES 7 discovery.zen.minimum_master_nodes is no longer need to be defined.
discovery.zen.minimum_master_nodes value for a cluster of two nodes
https://www.elastic.co/blog/a-new-era-for-cluster-coordination-in-elasticsearch