Cannot retrieve cluster state due to : None of the configured nodes are available - elasticsearch

Question

I'm trying to implement search-guard-5-5.6.3- in ES 5.6.3 and i have some trouble
while executing
./sgadmin.sh -ts truststore.jks -tspass 90f3cbdb3eabe04f815b -ks CN=sgadmin-keystore.jks -kspass a65d2a4fa62d7ed7a4d5 -cn cluster -h host -p 9200 -nhnv -cd ../sgconfig/
I get
Cannot retrieve cluster state due to: None of the configured nodes are
available: [{#transport#-1}{A1ZqEo4RSsqP3ZRSTXTUOg}{host}{host:9200}]. This is not an error, will keep on trying ...
Root cause: NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{A1ZqEo4RSsqP3ZRSTXTUOg}{host}{host:9200}]] (org.elasticsearch.client.transport.NoNodeAvailableException/org.elasticsearch.c lient.transport.NoNodeAvailableException)
* Try running sgadmin.sh with -icl (but no -cl) and -nhnv (If thats works you need to check your clustername as well as hostnames in your SSL certificates)
* Make also sure that your keystore or cert is a client certificate (not a node certificate) and configured properly in elasticsearch.yml
* If this is not working, try running sgadmin.sh with --diagnose and see diagnose trace log file)
* Add --accept-red-cluster to allow sgadmin to operate on a red cluster.
My cluster are correctly started, in ES log it says:
[2017-11-08T15:54:55,354][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] sslTransport protocols [TLSv1.2, TLSv1.1]
[2017-11-08T15:54:55,354][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] sslHTTP protocols [TLSv1.2, TLSv1.1]
[2017-11-08T15:54:55,356][INFO ][o.e.p.PluginsService ] [node_1] loaded module [aggs-matrix-stats]
[2017-11-08T15:54:55,357][INFO ][o.e.p.PluginsService ] [node_1] loaded module [ingest-common]
[2017-11-08T15:54:55,357][INFO ][o.e.p.PluginsService ] [node_1] loaded module [lang-expression]
[2017-11-08T15:54:55,357][INFO ][o.e.p.PluginsService ] [node_1] loaded module [lang-groovy]
[2017-11-08T15:54:55,357][INFO ][o.e.p.PluginsService ] [node_1] loaded module [lang-mustache]
[2017-11-08T15:54:55,357][INFO ][o.e.p.PluginsService ] [node_1] loaded module [lang-painless]
[2017-11-08T15:54:55,357][INFO ][o.e.p.PluginsService ] [node_1] loaded module [parent-join]
[2017-11-08T15:54:55,357][INFO ][o.e.p.PluginsService ] [node_1] loaded module [percolator]
[2017-11-08T15:54:55,357][INFO ][o.e.p.PluginsService ] [node_1] loaded module [reindex]
[2017-11-08T15:54:55,357][INFO ][o.e.p.PluginsService ] [node_1] loaded module [transport-netty3]
[2017-11-08T15:54:55,357][INFO ][o.e.p.PluginsService ] [node_1] loaded module [transport-netty4]
[2017-11-08T15:54:55,363][INFO ][o.e.p.PluginsService ] [node_1] loaded plugin [search-guard-5]
[2017-11-08T15:54:59,119][DEBUG][o.e.a.ActionModule ] Using REST wrapper from plugin com.floragunn.searchguard.SearchGuardPlugin
[2017-11-08T15:54:59,193][INFO ][c.f.s.SearchGuardPlugin ] FLS/DLS valve not bound (noop) due to java.lang.ClassNotFoundException: com.floragunn.searchguard.configuration.DlsFlsValveImpl
[2017-11-08T15:54:59,194][INFO ][c.f.s.SearchGuardPlugin ] Auditlog not available due to java.lang.ClassNotFoundException: com.floragunn.searchguard.auditlog.impl.AuditLogImpl
[2017-11-08T15:54:59,196][INFO ][c.f.s.SearchGuardPlugin ] Privileges interceptor not bound (noop) due to java.lang.ClassNotFoundException: com.floragunn.searchguard.configuration.PrivilegesInterceptorImpl
[2017-11-08T15:54:59,660][INFO ][o.e.d.DiscoveryModule ] [node_1] using discovery type [zen]
[2017-11-08T15:55:00,694][INFO ][o.e.n.Node ] [node_1] initialized
[2017-11-08T15:55:00,695][INFO ][o.e.n.Node ] [node_1] starting ...
[2017-11-08T15:55:01,017][INFO ][o.e.t.TransportService ] [node_1] publish_address {host:9300}, bound_addresses {host:9300}
[2017-11-08T15:55:01,038][INFO ][o.e.b.BootstrapChecks ] [node_1] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
[2017-11-08T15:55:01,052][INFO ][c.f.s.c.IndexBaseConfigurationRepository] Check if searchguard index exists ...
[2017-11-08T15:55:01,058][DEBUG][o.e.a.a.i.e.i.TransportIndicesExistsAction] [node_1] no known master node, scheduling a retry
[2017-11-08T15:55:04,143][INFO ][o.e.c.s.ClusterService ] [node_1] new_master {node_1}{aN2lbPkJSHWWFTllDhVeNQ}{NYFK1tN7SjC_41uRabKqRw}{mongodb-rec3.ib.fr.cly}{host:9300}, reason: zen-disco-elected-as-master ([0] nodes joined)
[2017-11-08T15:55:04,250][INFO ][c.f.s.h.SearchGuardHttpServerTransport] [node_1] publish_address {host:9200}, bound_addresses {host:9200}
[2017-11-08T15:55:04,251][INFO ][o.e.n.Node ] [node_1] started
[2017-11-08T15:55:04,542][INFO ][o.e.g.GatewayService ] [node_1] recovered [3] indices into cluster_state
[2017-11-08T15:55:05,353][INFO ][o.e.c.r.a.AllocationService] [node_1] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[fs][4]] ...]).
[2017-11-08T15:55:05,465][INFO ][c.f.s.c.IndexBaseConfigurationRepository] Node 'node_1' initialized
But wihle trying to send request http://host:9200 i am getting the following error
[2017-11-08T16:09:10,954][WARN ][c.f.s.h.SearchGuardHttpServerTransport] [node_1] Someone (/host:46422) speaks http plaintext instead of ssl, will close the channel

Answer 1

There are tow different issues here.
First, you try to connect to the HTTP port with sgadmin, but sgadmin uses the transport port. So, instead of:
-p 9200
You need to use the transport port:
-p 9300
You can also omit this setting, since 9300 is the default.
Then, you try to connect to Elasticsearch with http: http://host:9200
But most likely you have HTTPS configured in elasticsearch.yml, that's why the HTTP connection fails, and that's what the error message says:
Someone (/host:46422) speaks http plaintext instead of ssl, will close the channel
So either connect with HTTPS instead of HTTP, or disable HTTPs in elasticsearch.yml (not recommended since insecure):
searchguard.ssl.http.enabled: false
You can also find a troubleshooting article in the docs: http://docs.search-guard.com/latest/troubleshooting-sgadmin