Elasticsearch Cluster can't join new node - elasticsearch

Question

I use es-2.2 version, build cluster with 3 node in different server. now some server get more memory to used, so plan to start other node in existing server.
server1:
10.1.192.31 port use default 9200 and 9300
server2:
10.1.192.32 port use default 9200 and 9300
server3:
10.216.90.225 port default 9200 and 9300
now i wanna add two new node in 31 and 32 server
newnode1: get new configure like below:
cluster.name: EScluster
node.name: ESnode-1-1
network.host: 10.1.192.32
node.master: false
node.data: true
http.port: 9202
transport.tcp.port: 9302
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: ["10.216.90.225", "10.1.192.31:9300", "10.1.192.31:9302", "10.1.192.32:9300"]
newnod2: config like below
cluster.name: EScluster
node.name: ESnode-2-1
network.host: 10.1.192.31
node.master: false
node.data: true
http.port: 9202
transport.tcp.port: 9302
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: ["10.216.90.225", "10.1.192.31:9300", "10.1.192.32:9300", "10.1.192.32:9302"]
after start get error like below:
[INFO ][discovery.zen ] [ESnode-1-1] failed to send join request to master [{ESnode-2}{C4Z7lLTASmiZYtswsljZYA}{10.1.192.31}{10.1.192.31:9300}{max_local_storage_nodes=1, master=true}], reason [RemoteTransportException[[ESnode-2][10.1.192.31:9300][internal:discovery/zen/join]]; nested: IllegalStateException[failure when sending a validation request to node]; nested: RemoteTransportException[[ESnode-1-1][10.1.192.32:9302][internal:discovery/zen/join/validate]]; nested: IllegalArgumentException[No custom metadata prototype registered for type [licenses], node like missing plugins]; ]
[2016-05-26 10:35:26,408][WARN ][transport.netty ] [ESnode-1-1] exception caught on transport layer [[id: 0x770dcb9e, /10.1.192.31:37584 => /10.1.192.32:9302]], closing connection

Answer 1

I met this error just now.
There are three nodes(one master node with two data node, all in docker) running on my vps.
I install marvel in master node with command:
bin/plugin install license
bin/plugin install marvel-agent
But I got No Marvel Data Found on kibana page.
Soon, I found other data node died.
The error was
IllegalArgumentException[No custom metadata prototype registered for type [licenses], node like missing plugins
and
So just install license in other two data nodes:
bin/plugin install license
restart them would work.

Answer 2

guys
The problem been solved, main reason like below:
[internal:discovery/zen/join/validate]]; nested: IllegalArgumentException[No custom metadata prototype registered for type [licenses], node like missing plugins]
I install some plugin like marvel but not working then disabled, but it's not install that plugin for new node。 This is big mistake, so don't try any unused plugin in your enviroment, should be carefully on it.
So, my configuration is correctly hope this can help other people, thanks.

Answer 3

My logs were filled with annoying license errors even though I had uninstalled marvel and was not running anything that should require a license. Nodes would not join the cluster.
I ran:
bin/plugin remove license
and restarted the nodes. Things came back online fine and the logspam stopped.