master_not_discovered_exception ElasticSearch single node - elasticsearch

I am trying to setup elasticsearch on a single host. Here is how my configuration looks like:
elasticsearch.yml
node.name: ${HOSTNAME}
network.host: _site_, _local_
http.port: 9200
transport.tcp.port: 9300
cluster.name: "test_cluster"
node.local: true
kibana.yml
server.host: 0.0.0.0
elasticsearch.url: http://localhost:9200
On following command:
curl -XGET 'localhost:9200/_cluster/health?pretty'
I get following message:
{
"error" : {
"root_cause" : [
{
"type" : "master_not_discovered_exception",
"reason" : null
}
],
"type" : "master_not_discovered_exception",
"reason" : null
},
"status" : 503
}
In log file I see following message:
not enough master nodes discovered during pinging (found [[]], but needed [-1]), pinging again
Could someone please point me right direction here?

I spent days (sigh) on basically this. I was trying to upgrade my single node cluster from 6.x es to 7.x, and I kept dying on the hill of "master_not_discovered_exception".
What finally solved it for me was examining a completely fresh install of 7.x.
For my single-node cluster, my /etc/elasticsearch/elasticsearch.yml needed the line:
discovery.type: single-node
I hope that saves someone else from spending days like I did. In my defence, I'm very new to es.

Following changes in eleasticsearch.yml worked for me
Node
node.name: node-1
Paths
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
Network
network.host: 0.0.0.0
http.port: 9200
Discovery
discovery.seed_hosts: "127.0.0.1:9300"
cluster.initial_master_nodes: ["node-1"]
Steps:
sudo systemctl stop elasticsearch.service
sudo nano /etc/elasticsearch/elasticsearch.yml
Apply above changes, save and close the editor.
sudo /bin/systemctl daemon-reload
sudo /bin/systemctl enable elasticsearch.service
sudo systemctl start elasticsearch.service
NOTE: elasticsearch7 is used with ubuntu 20.04, it should work with other versions as well

According to this link, for the very first time you should set initial_master_nodes config like this in /etc/elasticsearch/elasticsearch.yml:
node.name: "salehnode"
cluster.initial_master_nodes:
- salehnode

On my side, after installing ES 7+, I had to set node-1 as a master-eligible node in the new cluster.
sudo nano /etc/elasticsearch/elasticsearch.yml
cluster.initial_master_nodes: ["node-1"]

First I dont think you need network.host setting for this.
In your log it is trying to get a master, but result is 0
Can you try setting properties like:
node.master: true
node.data:true
Also can you please put more logs here, if it doesnt work

You can try this in your elasticsearch.yml:
discovery.zen.minimum_master_nodes: 1
I think ES will always find discovery.zen.ping.unicast.hosts until it
finds nodes of count equal to minimum_master_nodes.

Related

Elasticsearch node can't connect to cluster

First of all, I want to be clear that I looked at several guides and suggested similar questions before opening this post, but none of them worked for our case.
Here is our situation:
Last week, our Elasticsearch stopped adding new records as it reached the state where the disk was almost full. We changed the config to get some time and now is working as expected. However, we want to add a new server with elasticsearch and form a cluster as we don't want to resize the disk because we don't want to lose anything.
Here is the configuration for the main server /etc/elasticsearch/elasticsearch.yml
cluster.name: my-cluster
node.name: master-node
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 0.0.0.0
http.max_content_length: 100mb
discover.seed_hosts: ["ip_address_server_1", "ip_address_server_2"]
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.http.ssl.enabled: true
As I said, we created another server with the same elasticsearch version (7.6.2)
cluster.name: my-cluster
node.name: another-node
node.data: true
node.master: false
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: ip_of_server
discover.seed_hosts: ["ip_address_server_1", "127.0.0.1"]
We tried restarting nodes at the same time, curls and everything but they can't see each other. At this point, I'm starting to think that the problem is the SSL configuration on the master, but I am not an elasticsearch expert, so I don't know what is exactly happening.
EDIT:
I took a look at the logs and here is what I found:
Node 2
[ico-elastic-node-2] master not discovered yet: have discovered [{node-2}{G5SfrEv0RxaxmYf8urIFtQ}{XpfNNouCQx2HmYfw2AvoQw}{ip_server_2}{ip_server_29300}{dil}{ml.machine_memory=12558602240, xpack.installed=true, ml.max_open_jobs=20}]; discovery will continue using [ip_server_1, 127.0.0.1:9300] from hosts providers and [] from last-known cluster state; node term 0, last-accepted version 0 in term 0
And the master:
[master-node] exception caught on transport layer [Netty4TcpChannel{localAddress=/ip:9300, remoteAddress=/ip:36660}], closing connection
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: No available authentication scheme
Currently master is yellow. What can we try next?

Failed to send join request to master in Elasticsearch, Unknown NamedWriteable [org.elasticsearch.cluster.metadata.MetaData$Custom][licenses]]

We have a long running single node ELK cluster running (master/data). I have decided to add additional data node. However Im getting the below error on the data node
30.X.XXX}{172.30.X.XXX:9300}{ml.enabled=true}], reason [RemoteTransportException[[master][172.30.X.XXX:9300][internal:discovery/zen/join]];
nested: IllegalStateException[failure when sending a validation request to node];
nested: RemoteTransportException[[data1][172.30.X.XXX:9300][internal:discovery/zen/join/validate]];
nested: IllegalArgumentException[Unknown NamedWriteable [org.elasticsearch.cluster.metadata.MetaData$Custom][licenses]]; ]
Below are the config files on master and new data node
Master Node:
cluster.name: my-application
node.name: master
node.master: true
node.data: true
path.data: /opt/elasticsearch
network.host: ["172.30.X.XX1","localhost"]
http.port: 9200
transport.tcp.port: 9300
discovery.zen.ping.unicast.hosts: ["172.30.X.XX1"]
discovery.zen.minimum_master_nodes: 1
Data1 Node:
cluster.name: my-application
node.name: data1
node.master: false
node.data: true
path.data: /opt/elasticsearch
network.host: ["172.30.X.XX2","localhost"]
http.port: 9200
transport.tcp.port: 9300
discovery.zen.ping.unicast.hosts: ["172.30.X.XX1"]
discovery.zen.minimum_master_nodes: 1
Tried pinging and checked telnet on 9200 and 9300 from master to data node and vice versa and it is working fine
I have tried deleting the data from /var/lib/elasticsearch/nodes/0 and restarted the data1, it didnt work
This happens if you try with a mix of xpack/commercial/non-open-source binaries of Elasticsearch and some nodes with the open-source binaries.
Unfortunately Elasticsearch tries to "trick" you into using their non-open-source version nowadays and this causes many unintended non-open-source installations.
A simple solution is to install the non-oss version everywhere, however you may not want to run the commercial version as you then need to adhere to the commercial license!
In order to convert to the open-source license on all nodes you can do the following:
You can set the following in /etc/elasticsearch/elasticsearch.yml and restart all nodes to disable some commercial features:
xpack.security.enabled: false
xpack.ml.enabled: false
Then you can change all nodes to the open-source binaries one by one in rolling fashion.
See also the following similar discussions:
https://discuss.elastic.co/t/elasticsearch-cluster-cant-join-new-node/126964
https://discuss.elastic.co/t/adding-a-new-node-on-a-different-subnet/125377/2- https://github.com/codelibs/elasticsearch-module/issues/3
https://discuss.elastic.co/t/transport-client-error-after-installing-x-pack-on-es-5-5-1/97021/4
https://discuss.elastic.co/t/bulk-indexing-with-x-pack-exception/92086/5

Elasticsearch configuration and settings for nodes

I have just installed ElasticSearch 5.0.0. When I run localhost:9200/_cat/nodes?v it gives me ip as 10.57.203.16. But I want localhost as a server. Where should I configure it ?
What if you add the below line to your elasticsearch.yml ,restart the ES and then try querying for the available nodes:
network.host: 127.0.0.1
EDIT:
Try adding a node name in your elasticsearch.yml :
node.name: elastic_test1

Elasticsearch Cluster can't join new node

I use es-2.2 version, build cluster with 3 node in different server. now some server get more memory to used, so plan to start other node in existing server.
server1:
10.1.192.31 port use default 9200 and 9300
server2:
10.1.192.32 port use default 9200 and 9300
server3:
10.216.90.225 port default 9200 and 9300
now i wanna add two new node in 31 and 32 server
newnode1: get new configure like below:
cluster.name: EScluster
node.name: ESnode-1-1
network.host: 10.1.192.32
node.master: false
node.data: true
http.port: 9202
transport.tcp.port: 9302
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: ["10.216.90.225", "10.1.192.31:9300", "10.1.192.31:9302", "10.1.192.32:9300"]
newnod2: config like below
cluster.name: EScluster
node.name: ESnode-2-1
network.host: 10.1.192.31
node.master: false
node.data: true
http.port: 9202
transport.tcp.port: 9302
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: ["10.216.90.225", "10.1.192.31:9300", "10.1.192.32:9300", "10.1.192.32:9302"]
after start get error like below:
[INFO ][discovery.zen ] [ESnode-1-1] failed to send join request to master [{ESnode-2}{C4Z7lLTASmiZYtswsljZYA}{10.1.192.31}{10.1.192.31:9300}{max_local_storage_nodes=1, master=true}], reason [RemoteTransportException[[ESnode-2][10.1.192.31:9300][internal:discovery/zen/join]]; nested: IllegalStateException[failure when sending a validation request to node]; nested: RemoteTransportException[[ESnode-1-1][10.1.192.32:9302][internal:discovery/zen/join/validate]]; nested: IllegalArgumentException[No custom metadata prototype registered for type [licenses], node like missing plugins]; ]
[2016-05-26 10:35:26,408][WARN ][transport.netty ] [ESnode-1-1] exception caught on transport layer [[id: 0x770dcb9e, /10.1.192.31:37584 => /10.1.192.32:9302]], closing connection
I met this error just now.
There are three nodes(one master node with two data node, all in docker) running on my vps.
I install marvel in master node with command:
bin/plugin install license
bin/plugin install marvel-agent
But I got No Marvel Data Found on kibana page.
Soon, I found other data node died.
The error was
IllegalArgumentException[No custom metadata prototype registered for type [licenses], node like missing plugins
and
So just install license in other two data nodes:
bin/plugin install license
restart them would work.
guys
The problem been solved, main reason like below:
[internal:discovery/zen/join/validate]]; nested: IllegalArgumentException[No custom metadata prototype registered for type [licenses], node like missing plugins]
I install some plugin like marvel but not working then disabled, but it's not install that plugin for new node。 This is big mistake, so don't try any unused plugin in your enviroment, should be carefully on it.
So, my configuration is correctly hope this can help other people, thanks.
My logs were filled with annoying license errors even though I had uninstalled marvel and was not running anything that should require a license. Nodes would not join the cluster.
I ran:
bin/plugin remove license
and restarted the nodes. Things came back online fine and the logspam stopped.

Elasticsearch Cluster set up problems

I am building a cluster with elasticsearch. I download the elasticsearch file as a zip file and unzip it in the /opt file. And these are the two IPs I am using for trial, 172.16.30.51 and 172.16.30.52.
I have come across with some problems. I have tried to amend the host files and add server IP.
sudo vi /etc/hosts
172.16.30.51 elasticnode01
172.16.30.52 elasticnode02
Also, in Server elasticnode01 :
cd /opt/elasticsearch
vi config/elasticsearch.yml
I amend the following code.
cluster.name: mycluster
node.name: "elasticnode01"
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: ["elasticnode02"]
In Server elasticnode02 :
cd /opt/elasticsearch
vi config/elasticsearch.yml
I amend the following code.
cluster.name: mycluster
node.name: "elasticnode02"
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: ["elasticnode01"]
Then finally I run the command
bin/elasticsearch &
It seems fine but as soon as I run
curl 'localhost:9200/_cat/nodes?v'
It returns
host ip heap.percent ram.percent load node.role master name
127.0.0.1 127.0.0.1 4 39 0.20 d * elasticnode01
Would anyone mind telling me what is the problem? Thanks.
Since ES 2.0, the ES server binds to localhost by default, so they won't be able to discover each other.
You need to configure network.host on both servers, like this:
On elasticnode01:
network.host: elasticnode01
On elasticnode02:
network.host: elasticnode02

Resources