Istio + Elasticsearch - elasticsearch

Question

I'm playing with Istio and Elasticsearch, I have virtual box machines on my laptop
I Installed Elastic search from this link Kubernetes Elasticsearch Cluster
I have a master es + data es, if I installed them without Istio, they run normally.
If I inject them with Istio the data nodes can not communicate with master (it does not find it).
root#node1:/home/arkan# k get all
NAME READY STATUS RESTARTS AGE
pod/es-data-6fdbcf956f-fdnc7 1/2 Running 1 1m
pod/es-master-6b6d5fd59b-86qpb 2/2 Running 0 1m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/elasticsearch NodePort 10.108.28.225 <none> 9200:32721/TCP 1m
service/elasticsearch-discovery ClusterIP None <none> 9300/TCP 1m
service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 22h
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
deployment.apps/es-data 1 1 1 0 1m
deployment.apps/es-master 1 1 1 1 1m
NAME DESIRED CURRENT READY AGE
replicaset.apps/es-data-6fdbcf956f 1 1 0 1m
replicaset.apps/es-master-6b6d5fd59b 1 1 1 1m
root#node1:/home/arkan# k describe pod/es-data-6fdbcf956f-fdnc7
Name: es-data-6fdbcf956f-fdnc7
Namespace: default
Node: node2/192.168.0.214
Start Time: Wed, 18 Jul 2018 21:42:50 +0300
Labels: component=elasticsearch
pod-template-hash=2986795129
role=data
Annotations: sidecar.istio.io/status={"version":"55c9e544b52e1d4e45d18a58d0b34ba4b72531e45fb6d1572c77191422556ffc","initContainers":["istio-init"],"containers":["istio-proxy"],"volumes":["istio-envoy","istio-certs...
Status: Running
IP: 172.16.104.22
Controlled By: ReplicaSet/es-data-6fdbcf956f
Init Containers:
init-sysctl:
Container ID: docker://c510035d1e823d134ad287116ef43332255758cce60cc1216ed20282b0b55e76
Image: busybox:1.27.2
Image ID: docker-pullable://busybox#sha256:bbc3a03235220b170ba48a157dd097dd1379299370e1ed99ce976df0355d24f0
Port: <none>
Host Port: <none>
Command:
sysctl
-w
vm.max_map_count=262144
State: Terminated
Reason: Completed
Exit Code: 0
Started: Wed, 18 Jul 2018 21:42:52 +0300
Finished: Wed, 18 Jul 2018 21:42:52 +0300
Ready: True
Restart Count: 0
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-mkdlq (ro)
istio-init:
Container ID: docker://42a8f5da07834533dcd4c26155fb344fa41edecb744e6a4c14c54c40610a450b
Image: docker.io/istio/proxy_init:0.8.0
Image ID: docker-pullable://istio/proxy_init#sha256:b0b288ee8270e054442abdd413da9395e2af39fed1792b85ec157700ef2c192f
Port: <none>
Host Port: <none>
Args:
-p
15001
-u
1337
-m
REDIRECT
-i
*
-x
-b
9200, 9300,
-d
State: Terminated
Reason: Completed
Exit Code: 0
Started: Wed, 18 Jul 2018 21:42:53 +0300
Finished: Wed, 18 Jul 2018 21:42:53 +0300
Ready: True
Restart Count: 0
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-mkdlq (ro)
Containers:
es-data:
Container ID: docker://505269e9be09e83d672b91a582afd7569b3afd794bbfab764d50d75e7a3f7309
Image: quay.io/pires/docker-elasticsearch-kubernetes:6.3.0
Image ID: docker-pullable://quay.io/pires/docker-elasticsearch-kubernetes#sha256:dcd3e9db3d2c6b9a448d135aebcacac30a4cca655d42efaa115aa57405cd22f3
Ports: 9200/TCP, 9300/TCP
Host Ports: 0/TCP, 0/TCP
State: Running
Started: Wed, 18 Jul 2018 21:46:08 +0300
Last State: Terminated
Reason: Error
Exit Code: 143
Started: Wed, 18 Jul 2018 21:45:18 +0300
Finished: Wed, 18 Jul 2018 21:46:07 +0300
Ready: False
Restart Count: 4
Limits:
cpu: 1
Requests:
cpu: 250m
Liveness: tcp-socket :transport delay=20s timeout=1s period=10s #success=1 #failure=3
Readiness: http-get http://:http/_cluster/health delay=20s timeout=5s period=10s #success=1 #failure=3
Environment:
NAMESPACE: default (v1:metadata.namespace)
NODE_NAME: es-data-6fdbcf956f-fdnc7 (v1:metadata.name)
CLUSTER_NAME: myesdb
NODE_MASTER: false
NODE_INGEST: false
HTTP_ENABLE: true
ES_JAVA_OPTS: -Xms256m -Xmx256m
PROCESSORS: 1 (limits.cpu)
Mounts:
/data from storage (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-mkdlq (ro)
istio-proxy:
Container ID: docker://1bf8e167ece0ac6282c336a6630c292013e36721ba027e6e8b5bb71a4bf65a25
Image: docker.io/istio/proxyv2:0.8.0
Image ID: docker-pullable://istio/proxyv2#sha256:1930f0603321b1917b2249c576ecb4141aaceeaae5fcc0760b6a88dc88daea3e
Port: <none>
Host Port: <none>
Args:
proxy
sidecar
--configPath
/etc/istio/proxy
--binaryPath
/usr/local/bin/envoy
--serviceCluster
istio-proxy
--drainDuration
45s
--parentShutdownDuration
1m0s
--discoveryAddress
istio-pilot.istio-system:15007
--discoveryRefreshDelay
10s
--zipkinAddress
zipkin.istio-system:9411
--connectTimeout
10s
--statsdUdpAddress
istio-statsd-prom-bridge.istio-system:9125
--proxyAdminPort
15000
--controlPlaneAuthPolicy
NONE
State: Running
Started: Wed, 18 Jul 2018 21:46:37 +0300
Last State: Terminated
Reason: Error
Exit Code: 255
Started: Wed, 18 Jul 2018 21:42:55 +0300
Finished: Wed, 18 Jul 2018 21:46:36 +0300
Ready: True
Restart Count: 1
Requests:
cpu: 100m
memory: 128Mi
Environment:
POD_NAME: es-data-6fdbcf956f-fdnc7 (v1:metadata.name)
POD_NAMESPACE: default (v1:metadata.namespace)
INSTANCE_IP: (v1:status.podIP)
ISTIO_META_POD_NAME: es-data-6fdbcf956f-fdnc7 (v1:metadata.name)
ISTIO_META_INTERCEPTION_MODE: REDIRECT
Mounts:
/etc/certs/ from istio-certs (ro)
/etc/istio/proxy from istio-envoy (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-mkdlq (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
storage:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
istio-envoy:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium: Memory
istio-certs:
Type: Secret (a volume populated by a Secret)
SecretName: istio.default
Optional: true
default-token-mkdlq:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-mkdlq
Optional: false
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 5m default-scheduler Successfully assigned default/es-data-6fdbcf956f-fdnc7 to node2
Normal Pulled 5m kubelet, node2 Container image "busybox:1.27.2" already present on machine
Normal Created 5m kubelet, node2 Created container
Normal Started 5m kubelet, node2 Started container
Normal Started 5m kubelet, node2 Started container
Normal Pulled 5m kubelet, node2 Container image "docker.io/istio/proxy_init:0.8.0" already present on machine
Normal Created 5m kubelet, node2 Created container
Normal Pulled 5m kubelet, node2 Container image "docker.io/istio/proxyv2:0.8.0" already present on machine
Normal Created 5m kubelet, node2 Created container
Normal Started 5m kubelet, node2 Started container
Warning Unhealthy 5m kubelet, node2 Liveness probe failed: dial tcp 172.16.104.22:9300: connect: invalid argument
Warning Unhealthy 5m kubelet, node2 Readiness probe failed: Get http://172.16.104.22:9200/_cluster/health: dial tcp 172.16.104.22:9200: connect: invalid argument
Normal Pulled 5m (x2 over 5m) kubelet, node2 Container image "quay.io/pires/docker-elasticsearch-kubernetes:6.3.0" already present on machine
Normal Created 5m (x2 over 5m) kubelet, node2 Created container
Normal Killing 5m kubelet, node2 Killing container with id docker://es-data:Container failed liveness probe.. Container will be killed and recreated.
Normal Started 5m (x2 over 5m) kubelet, node2 Started container
Warning Unhealthy 4m (x3 over 5m) kubelet, node2 Readiness probe failed: Get http://172.16.104.22:9200/_cluster/health: dial tcp 172.16.104.22:9200: connect: connection refused
Warning Unhealthy 4m (x4 over 5m) kubelet, node2 Liveness probe failed: dial tcp 172.16.104.22:9300: connect: connection refused
Warning Unhealthy 51s (x3 over 1m) kubelet, node2 (combined from similar events): Readiness probe failed: Get http://172.16.104.22:9200/_cluster/health: EOF
root#node1:/home/arkan# k logs pod/es-data-6fdbcf956f-fdnc7 -c es-data
[2018-07-18T18:46:13,037][INFO ][o.e.n.Node ] [es-data-6fdbcf956f-fdnc7] initializing ...
[2018-07-18T18:46:13,267][INFO ][o.e.e.NodeEnvironment ] [es-data-6fdbcf956f-fdnc7] using [1] data paths, mounts [[/data (/dev/mapper/node1--vg-root)]], net usable_space [13.8gb], net total_space [27.9gb], types [ext4]
[2018-07-18T18:46:13,270][INFO ][o.e.e.NodeEnvironment ] [es-data-6fdbcf956f-fdnc7] heap size [247.5mb], compressed ordinary object pointers [true]
[2018-07-18T18:46:13,272][INFO ][o.e.n.Node ] [es-data-6fdbcf956f-fdnc7] node name [es-data-6fdbcf956f-fdnc7], node ID [ymKMhUIxRq-hbrmrqzayCQ]
[2018-07-18T18:46:13,272][INFO ][o.e.n.Node ] [es-data-6fdbcf956f-fdnc7] version[6.3.0], pid[1], build[default/tar/424e937/2018-06-11T23:38:03.357887Z], OS[Linux/4.4.0-128-generic/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_151/25.151-b12]
[2018-07-18T18:46:13,273][INFO ][o.e.n.Node ] [es-data-6fdbcf956f-fdnc7] JVM arguments [-XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+DisableExplicitGC, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Xms256m, -Xmx256m, -Des.path.home=/elasticsearch, -Des.path.conf=/elasticsearch/config, -Des.distribution.flavor=default, -Des.distribution.type=tar]
[2018-07-18T18:46:15,832][WARN ][o.e.d.c.s.Settings ] [http.enabled] setting was deprecated in Elasticsearch and will be removed in a future release! See the breaking changes documentation for the next major version.
[2018-07-18T18:46:18,327][INFO ][o.e.p.PluginsService ] [es-data-6fdbcf956f-fdnc7] loaded module [aggs-matrix-stats]
[2018-07-18T18:46:18,331][INFO ][o.e.p.PluginsService ] [es-data-6fdbcf956f-fdnc7] loaded module [analysis-common]
[2018-07-18T18:46:18,331][INFO ][o.e.p.PluginsService ] [es-data-6fdbcf956f-fdnc7] loaded module [ingest-common]
[2018-07-18T18:46:18,332][INFO ][o.e.p.PluginsService ] [es-data-6fdbcf956f-fdnc7] loaded module [lang-expression]
[2018-07-18T18:46:18,332][INFO ][o.e.p.PluginsService ] [es-data-6fdbcf956f-fdnc7] loaded module [lang-mustache]
[2018-07-18T18:46:18,337][INFO ][o.e.p.PluginsService ] [es-data-6fdbcf956f-fdnc7] no plugins loaded
[2018-07-18T18:46:26,419][INFO ][o.e.x.s.a.s.FileRolesStore] [es-data-6fdbcf956f-fdnc7] parsed [0] roles from file [/elasticsearch/config/roles.yml]
[2018-07-18T18:46:28,422][INFO ][o.e.d.DiscoveryModule ] [es-data-6fdbcf956f-fdnc7] using discovery type [zen]
[2018-07-18T18:46:30,218][INFO ][o.e.n.Node ] [es-data-6fdbcf956f-fdnc7] initialized
[2018-07-18T18:46:30,218][INFO ][o.e.n.Node ] [es-data-6fdbcf956f-fdnc7] starting ...
[2018-07-18T18:46:30,630][INFO ][o.e.t.TransportService ] [es-data-6fdbcf956f-fdnc7] publish_address {172.16.104.22:9300}, bound_addresses {172.16.104.22:9300}
[2018-07-18T18:46:30,701][INFO ][o.e.b.BootstrapChecks ] [es-data-6fdbcf956f-fdnc7] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2018-07-18T18:46:33,802][WARN ][o.e.d.z.ZenDiscovery ] [es-data-6fdbcf956f-fdnc7] not enough master nodes discovered during pinging (found [[]], but needed [1]), pinging again
[2018-07-18T18:46:36,803][WARN ][o.e.d.z.ZenDiscovery ] [es-data-6fdbcf956f-fdnc7] not enough master nodes discovered during pinging (found [[]], but needed [1]), pinging again
[2018-07-18T18:46:39,805][WARN ][o.e.d.z.ZenDiscovery ] [es-data-6fdbcf956f-fdnc7] not enough master nodes discovered during pinging (found [[]], but needed [1]), pinging again
[2018-07-18T18:46:43,830][WARN ][o.e.x.s.t.n.SecurityNetty4ServerTransport] [es-data-6fdbcf956f-fdnc7] send message failed [channel: NettyTcpChannel{localAddress=0.0.0.0/0.0.0.0:59678, remoteAddress=null}]
java.nio.channels.ClosedChannelException: null
at io.netty.channel.AbstractChannel$AbstractUnsafe.write(...)(Unknown Source) ~[?:?]
[2018-07-18T18:46:44,818][WARN ][o.e.x.s.t.n.SecurityNetty4ServerTransport] [es-data-6fdbcf956f-fdnc7] send message failed [channel: NettyTcpChannel{localAddress=0.0.0.0/0.0.0.0:59684, remoteAddress=elasticsearch-discovery/172.16.166.167:9300}]
java.nio.channels.ClosedChannelException: null
at io.netty.channel.AbstractChannel$AbstractUnsafe.write(...)(Unknown Source) ~[?:?]
[2018-07-18T18:46:45,810][WARN ][o.e.d.z.ZenDiscovery ] [es-data-6fdbcf956f-fdnc7] not enough master nodes discovered during pinging (found [[]], but needed [1]), pinging again
[2018-07-18T18:46:45,902][WARN ][o.e.x.s.t.n.SecurityNetty4ServerTransport] [es-data-6fdbcf956f-fdnc7] send message failed [channel: NettyTcpChannel{localAddress=0.0.0.0/0.0.0.0:59690, remoteAddress=elasticsearch-discovery/172.16.166.167:9300}]
java.nio.channels.ClosedChannelException: null
at io.netty.channel.AbstractChannel$AbstractUnsafe.write(...)(Unknown Source) ~[?:?]
[2018-07-18T18:46:48,812][WARN ][o.e.d.z.ZenDiscovery ] [es-data-6fdbcf956f-fdnc7] not enough master nodes discovered during pinging (found [[]], but needed [1]), pinging again
[2018-07-18T18:46:48,900][WARN ][o.e.x.s.t.n.SecurityNetty4ServerTransport] [es-data-6fdbcf956f-fdnc7] send message failed [channel: NettyTcpChannel{localAddress=0.0.0.0/0.0.0.0:59714, remoteAddress=elasticsearch-discovery/172.16.166.167:9300}]
java.nio.channels.ClosedChannelException: null
at io.netty.channel.AbstractChannel$AbstractUnsafe.write(...)(Unknown Source) ~[?:?]
[2018-07-18T18:46:49,899][WARN ][o.e.x.s.t.n.SecurityNetty4ServerTransport] [es-data-6fdbcf956f-fdnc7] send message failed [channel: NettyTcpChannel{localAddress=0.0.0.0/0.0.0.0:59720, remoteAddress=null}]
java.nio.channels.ClosedChannelException: null
at io.netty.channel.AbstractChannel$AbstractUnsafe.write(...)(Unknown Source) ~[?:?]
[2018-07-18T18:46:51,815][WARN ][o.e.d.z.ZenDiscovery ] [es-data-6fdbcf956f-fdnc7] not enough master nodes discovered during pinging (found [[]], but needed [1]), pinging again
[2018-07-18T18:46:51,901][WARN ][o.e.x.s.t.n.SecurityNetty4ServerTransport] [es-data-6fdbcf956f-fdnc7] send message failed [channel: NettyTcpChannel{localAddress=0.0.0.0/0.0.0.0:59730, remoteAddress=null}]
java.nio.channels.ClosedChannelException: null
at io.netty.channel.AbstractChannel$AbstractUnsafe.write(...)(Unknown Source) ~[?:?]
[2018-07-18T18:46:52,900][WARN ][o.e.x.s.t.n.SecurityNetty4ServerTransport] [es-data-6fdbcf956f-fdnc7] send message failed [channel: NettyTcpChannel{localAddress=0.0.0.0/0.0.0.0:59734, remoteAddress=null}]
[es-data-6fdbcf956f-fdnc7] not enough master nodes discovered during pinging (found [[]], but needed [1]), pinging again
[2018-07-18T18:46:59,897][WARN ][o.e.x.s.t.n.SecurityNetty4ServerTransport] [es-data-6fdbcf956f-fdnc7] send message failed [channel: NettyTcpChannel{localAddress=0.0.0.0/0.0.0.0:59782, remoteAddress=elasticsearch-discovery/172.16.166.167:9300}]
java.nio.channels.ClosedChannelException: null
at io.netty.channel.AbstractChannel$AbstractUnsafe.write(...)(Unknown Source) ~[?:?]
[2018-07-18T18:47:00,781][WARN ][o.e.n.Node ] [es-data-6fdbcf956f-fdnc7] timed out while waiting for initial discovery state - timeout: 30s
[2018-07-18T18:47:00,801][INFO ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [es-data-6fdbcf956f-fdnc7] publish_address {172.16.104.22:9200}, bound_addresses {172.16.104.22:9200}
[2018-07-18T18:47:00,803][INFO ][o.e.n.Node ] [es-data-6fdbcf956f-fdnc7] started
[2018-07-18T18:47:00,821][WARN ][o.e.d.z.ZenDiscovery ] [es-data-6fdbcf956f-fdnc7] not enough master nodes discovered during pinging (found [[]], but needed [1]), pinging again
[2018-07-18T18:47:02,896][WARN ][o.e.x.s.t.n.SecurityNetty4ServerTransport] [es-data-6fdbcf956f-fdnc7] send message failed [channel: NettyTcpChannel{localAddress=0.0.0.0/0.0.0.0:59796, remoteAddress=elasticsearch-discovery/172.16.166.167:9300}]
java.nio.channels.ClosedChannelException: null
at io.netty.channel.AbstractChannel$AbstractUnsafe.write(...)(Unknown Source) ~[?:?]
[2018-07-18T18:47:03,822][WARN ][o.e.d.z.ZenDiscovery ] [es-data-6fdbcf956f-fdnc7] not enough master nodes discovered during pinging (found [[]], but needed [1]), pinging again
[2018-07-18T18:47:06,823][WARN ][o.e.d.z.ZenDiscovery ] [es-data-6fdbcf956f-fdnc7] not enough master nodes discovered during pinging (found [[]], but needed [1]), pinging again
[2018-07-18T18:47:15,897][WARN ][o.e.x.s.t.n.SecurityNetty4ServerTransport] [es-data-6fdbcf956f-fdnc7] send message failed [channel: NettyTcpChannel{localAddress=0.0.0.0/0.0.0.0:59880, remoteAddress=elasticsearch-discovery/172.16.166.167:9300}]
java.nio.channels.ClosedChannelException: null
at io.netty.channel.AbstractChannel$AbstractUnsafe.write(...)(Unknown Source) ~[?:?]
[2018-07-18T18:47:18,832][WARN ][o.e.d.z.ZenDiscovery ] [es-data-6fdbcf956f-fdnc7] not enough master nodes discovered during pinging (found [[]], but needed [1]), pinging again
[2018-07-18T18:47:21,835][WARN ][o.e.d.z.ZenDiscovery ] [es-data-6fdbcf956f-fdnc7] not enough master nodes discovered during pinging (found [[]], but needed [1]), pinging again
[2018-07-18T18:47:24,837][WARN ][o.e.d.z.ZenDiscovery ] [es-data-6fdbcf956f-fdnc7] not enough master nodes discovered during pinging (found [[]], but needed [1]), pinging again
[2018-07-18T18:47:24,841][WARN ][o.e.x.s.t.n.SecurityNetty4ServerTransport] [es-data-6fdbcf956f-fdnc7] send message failed [channel: NettyTcpChannel{localAddress=0.0.0.0/0.0.0.0:59936, remoteAddress=elasticsearch-discovery/172.16.166.167:9300}]
java.nio.channels.ClosedChannelException: null
at io.netty.channel.AbstractChannel$AbstractUnsafe.write(...)(Unknown Source) ~[?:?]
[2018-07-18T18:47:27,838][WARN ][o.e.d.z.ZenDiscovery ] [es-data-6fdbcf956f-fdnc7] not enough master nodes discovered during pinging (found [[]], but needed [1]), pinging again
[2018-07-18T18:47:30,840][WARN ][o.e.d.z.ZenDiscovery ] [es-data-6fdbcf956f-fdnc7] not enough master nodes discovered during pinging (found [[]], but needed [1]), pinging again
[2018-07-18T18:47:30,897][WARN ][o.e.x.s.t.n.SecurityNetty4ServerTransport] [es-data-6fdbcf956f-fdnc7] send message failed [channel: NettyTcpChannel{localAddress=0.0.0.0/0.0.0.0:59976, remoteAddress=elasticsearch-discovery/172.16.166.167:9300}]
java.nio.channels.ClosedChannelException: null
at io.netty.channel.AbstractChannel$AbstractUnsafe.write(...)(Unknown Source) ~[?:?]
[2018-07-18T18:47:33,844][WARN ][o.e.d.z.ZenDiscovery ] [es-data-6fdbcf956f-fdnc7] not enough master nodes discovered during pinging (found [[]], but needed [1]), pinging again
[2018-07-18T18:47:33,898][WARN ][o.e.x.s.t.n.SecurityNetty4ServerTransport] [es-data-6fdbcf956f-fdnc7] send message failed [channel: NettyTcpChannel{localAddress=0.0.0.0/0.0.0.0:59992, remoteAddress=elasticsearch-discovery/172.16.166.167:9300}]
java.nio.channels.ClosedChannelException: null
at io.netty.channel.AbstractChannel$AbstractUnsafe.write(...)(Unknown Source) ~[?:?]
[2018-07-18T18:47:36,847][WARN ][o.e.d.z.ZenDiscovery ] [es-data-6fdbcf956f-fdnc7] not enough master nodes discovered during pinging (found [[]], but needed [1]), pinging again
[2018-07-18T18:47:51,854][WARN ][o.e.d.z.ZenDiscovery ] [es-data-6fdbcf956f-fdnc7] not enough master nodes discovered during pinging (found [[]], but needed [1]), pinging again
[2018-07-18T18:47:51,897][WARN ][o.e.x.s.t.n.SecurityNetty4ServerTransport] [es-data-6fdbcf956f-fdnc7] send message failed [channel: NettyTcpChannel{localAddress=0.0.0.0/0.0.0.0:60110, remoteAddress=elasticsearch-discovery/172.16.166.167:9300}]
java.nio.channels.ClosedChannelException: null
at io.netty.channel.AbstractChannel$AbstractUnsafe.write(...)(Unknown Source) ~[?:?]
[2018-07-18T18:47:54,857][WARN ][o.e.d.z.ZenDiscovery ] [es-data-6fdbcf956f-fdnc7] not enough master nodes discovered during pinging (found [[]], but needed [1]), pinging again
[2018-07-18T18:47:57,858][WARN ][o.e.d.z.ZenDiscovery ] [es-data-6fdbcf956f-fdnc7] not enough master nodes discovered during pinging (found [[]], but needed [1]), pinging again
[2018-07-18T18:48:06,897][WARN ][o.e.x.s.t.n.SecurityNetty4ServerTransport] [es-data-6fdbcf956f-fdnc7] send message failed [channel: NettyTcpChannel{localAddress=0.0.0.0/0.0.0.0:60202, remoteAddress=elasticsearch-discovery/172.16.166.167:9300}]
java.nio.channels.ClosedChannelException: null
at io.netty.channel.AbstractChannel$AbstractUnsafe.write(...)(Unknown Source) ~[?:?]
[2018-07-18T18:48:09,868][WARN ][o.e.d.z.ZenDiscovery ] [es-data-6fdbcf956f-fdnc7] not enough master nodes discovered during pinging (found [[]], but needed [1]), pinging again
[2018-07-18T18:48:39,886][WARN ][o.e.d.z.ZenDiscovery ] [es-data-6fdbcf956f-fdnc7] not enough master nodes discovered during pinging (found [[]], but needed [1]), pinging again
[2018-07-18T18:48:39,897][WARN ][o.e.x.s.t.n.SecurityNetty4ServerTransport] [es-data-6fdbcf956f-fdnc7] send message failed [channel: NettyTcpChannel{localAddress=0.0.0.0/0.0.0.0:60418, remoteAddress=elasticsearch-discovery/172.16.166.167:9300}]
java.nio.channels.ClosedChannelException: null
at io.netty.channel.AbstractChannel$AbstractUnsafe.write(...)(Unknown Source) ~[?:?]
[2018-07-18T18:49:15,909][WARN ][o.e.d.z.ZenDiscovery ] [es-data-6fdbcf956f-fdnc7] not enough master nodes discovered during pinging (found [[]], but needed [1]), pinging again
[2018-07-18T18:49:15,997][WARN ][o.e.x.s.t.n.SecurityNetty4ServerTransport] [es-data-6fdbcf956f-fdnc7] send message failed [channel: NettyTcpChannel{localAddress=0.0.0.0/0.0.0.0:60640, remoteAddress=elasticsearch-discovery/172.16.166.167:9300}]
java.nio.channels.ClosedChannelException: null
at io.netty.channel.AbstractChannel$AbstractUnsafe.write(...)(Unknown Source) ~[?:?]
[2018-07-18T18:49:18,910][WARN ][o.e.d.z.ZenDiscovery ] [es-data-6fdbcf956f-fdnc7] not enough master nodes discovered during pinging (found [[]], but needed [1]), pinging again
[2018-07-18T18:49:21,912][WARN ][o.e.d.z.ZenDiscovery ] [es-data-6fdbcf956f-fdnc7] not enough master nodes discovered during pinging (found [[]], but needed [1]), pinging again
[2018-07-18T18:49:24,913][WARN ][o.e.d.z.ZenDiscovery ] [es-data-6fdbcf956f-fdnc7] not enough master nodes discovered during pinging (found [[]], but needed [1]), pinging again
[2018-07-18T18:49:27,915][WARN ][o.e.d.z.ZenDiscovery ] [es-data-6fdbcf956f-fdnc7] not enough master nodes discovered during pinging (found [[]], but needed [1]), pinging again

Answer 1

ElasticSearch expects to be able to talk to PodIPs directly.
Consider disabling the sidecar for ElasticSearch.
https://istio.io/docs/setup/kubernetes/additional-setup/sidecar-injection/#policy documents how to use the sidecar.istio.io/inject: "false" annotation.

Answer 2

You can add namespace (-n logging) to your deploy script. For example
kubectl create -f es-discovery-svc.yaml -n logging
...
Istio sidecar autoinject only work on namespace:default by default.