Elasticsearch MasterNotDiscoveredException between client and master - elasticsearch

Question

In a nutshell, I have a standalone ES master instance and a client node that is created within my Java application. The client node discovers the standalone ES instance correctly if the standalone ES instance is started before the client node, which is expected.
The problem I'm facing is this - if for some reason, the client node starts before the standalone ES instance, I see a "MasterNotDiscoveredException", which is again expected. However, I continue seeing the same exception even after I start the standalone ES instance. Is there some configuration I should be changing to fix this?
I'm using ES 1.7.1 with Unicast discovery.
EDIT
Cluster information: The standalone ES instance and the client node together make up a cluster.
Client node stack trace:
11:29:35,634 INFO http [496648366, id=7BCBFQLCTWOO2, ide=tcp://172.17.78.80:61616] [Squidboy] bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address {inet[/172.17.78.80:9200]}
11:29:35,635 INFO node [496648366, id=7BCBFQLCTWOO2, ide=tcp://172.17.78.80:61616] [Squidboy] started
11:30:10,279 ERROR ApplicationLifeCycle [299961584] System startup not complete after 120 seconds ...
11:30:14,706 WARN ElasticSearchStatus [278792216] An Exception occurred during cluster health status update - java.util.concurrent.ExecutionException: org.elasticsearch.discovery.MasterNotDiscoveredException: waited for [30s]
at org.elasticsearch.common.util.concurrent.BaseFuture$Sync.getValue(BaseFuture.java:292)
at org.elasticsearch.common.util.concurrent.BaseFuture$Sync.get(BaseFuture.java:279)
at org.elasticsearch.common.util.concurrent.BaseFuture.get(BaseFuture.java:117)
at com.harry.elastic.node.ElasticSearchStatus.updateClusterHealth(ElasticSearchStatus.java:90)
at com.harry.elastic.node.ElasticSearchStatus.access$000(ElasticSearchStatus.java:37)
at com.harry.elastic.node.ElasticSearchStatus$1.run(ElasticSearchStatus.java:62)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.discovery.MasterNotDiscoveredException: waited for [30s]
at org.elasticsearch.action.support.master.TransportMasterNodeOperationAction$4.onTimeout(TransportMasterNodeOperationAction.java:164)
at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:231)
at org.elasticsearch.cluster.service.InternalClusterService$NotifyTimeout.run(InternalClusterService.java:560)
... 3 more
Client creation code:
private Node createEmbeddedClientNode() {
ImmutableSettings.Builder settingsBuilder = ImmutableSettings.settingsBuilder()
.put("discovery.zen.ping.multicast.enabled", false)
.put("discovery.zen.ping.unicast.hosts", "localhost[9300-9400]");
return nodeBuilder().settings(settingsBuilder).clusterName("harryService")
.client(true).data(false).node();
}
Master discovery configuration
"discovery": {
"zen": {
"ping": {
"multicast": {
"enabled": false
}
}
}

Answer 1

By default your client node will retry pinging your master node each 30s for 3 times and then give up. So if you started your master node after that time has passed, your client node will not discover it.
Try increasing the retries and/or the timeout, that should help.
.put("discovery.zen.fd.ping_timeout", "1m")
.put("discovery.zen.fd.ping_retries", 5)
With those settings, your client node will keep trying during 5 minutes instead of only 1.5 minutes. However, your master node should really be up already when you start your application.
Another settings that might help is the following, as it is true by default and your master will ignore client pings during master election, but since there's a single master node it might not make any difference, still worth a try:
.put("discovery.zen.master_election.filter_client", false)

Answer 2

I solved the problem by adding unicast configuration in the master node explicitly.
"discovery": {
"zen": {
"ping": {
"multicast": {
"enabled": false
},
"unicast": {
"hosts": "localhost[9300-9400]"
}
}
}
}