Getting error while installing elasticsearch Statefulset - elasticsearch

Question

I am trying to install an elastic statefulset in my GKE cluster but it's throwing an error and am unable to identify the error here this is the log that I had got inside the pod. Can someone help me? I have given the error logs as well as the elasticsearch_statefulset.yml file.
{"type": "server", "timestamp": "2021-12-16T09:30:49,473Z", "level": "WARN", "component": "o.e.d.SeedHostsResolver", "cluster.name": "k8s-logs", "node.name": "es-cluster-0", "message": "failed to resolve host [es-cluster-2.elasticsearch]",
"stacktrace": ["java.net.UnknownHostException: es-cluster-2.elasticsearch",
"at java.net.InetAddress$CachedAddresses.get(InetAddress.java:800) ~[?:?]",
"at java.net.InetAddress.getAllByName0(InetAddress.java:1495) ~[?:?]",
"at java.net.InetAddress.getAllByName(InetAddress.java:1354) ~[?:?]",
"at java.net.InetAddress.getAllByName(InetAddress.java:1288) ~[?:?]",
"at org.elasticsearch.transport.TcpTransport.parse(TcpTransport.java:548) ~[elasticsearch-7.9.1.jar:7.9.1]",
"at org.elasticsearch.transport.TcpTransport.addressesFromString(TcpTransport.java:490) ~[elasticsearch-7.9.1.jar:7.9.1]",
"at org.elasticsearch.transport.TransportService.addressesFromString(TransportService.java:855) ~[elasticsearch-7.9.1.jar:7.9.1]",
"at org.elasticsearch.discovery.SeedHostsResolver.lambda$resolveHostsLists$0(SeedHostsResolver.java:144) ~[elasticsearch-7.9.1.jar:7.9.1]",
"at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]",
"at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:651) ~[elasticsearch-7.9.1.jar:7.9.1]",
"at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]",
"at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]",
"at java.lang.Thread.run(Thread.java:832) [?:?]"] }
This is the yml I used to configure the stateful set :
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: es-cluster
namespace: kube-logging
spec:
serviceName: elasticsearch
replicas: 3
selector:
matchLabels:
app: elasticsearch
template:
metadata:
labels:
app: elasticsearch
spec:
containers:
- name: elasticsearch
image: docker.elastic.co/elasticsearch/elasticsearch:7.9.1
resources: {}
ports:
- containerPort: 9200
name: rest
protocol: TCP
- containerPort: 9300
name: inter-node
protocol: TCP
volumeMounts:
- name: data
mountPath: /usr/share/elasticsearch/data
env:
- name: cluster.name
value: k8s-logs
- name: node.name
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: discovery.seed_hosts
value: "es-cluster-0.elasticsearch,es-cluster-1.elasticsearch,es-cluster-2.elasticsearch"
- name: cluster.initial_master_nodes
value: "es-cluster-0,es-cluster-1,es-cluster-2"
- name: ES_JAVA_OPTS
value: "-Xms512m -Xmx512m"
initContainers:
- name: fix-permissions
image: busybox
command: ["sh", "-c", "chown -R 1000:1000 /usr/share/elasticsearch/data"]
securityContext:
privileged: true
volumeMounts:
- name: data
mountPath: /usr/share/elasticsearch/data
- name: increase-vm-max-map
image: busybox
command: ["sysctl", "-w", "vm.max_map_count=262144"]
securityContext:
privileged: true
- name: increase-fd-ulimit
image: busybox
command: ["sh", "-c", "ulimit -n 65536"]
securityContext:
privileged: true
volumeClaimTemplates:
- metadata:
name: data
labels:
app: elasticsearch
spec:
accessModes: [ "ReadWriteOnce" ]
storageClassName: standard
resources:
requests:
storage: 10Gi
Service file which I used :
kind: Service
apiVersion: v1
metadata:
name: elasticsearch
namespace: kube-logging
labels:
app: elasticsearch
spec:
selector:
app: elasticsearch
type: LoadBalancer
ports:
- port: 9200
name: rest
- port: 9300
name: inter-node
Output for kubectl get statefulset -n kube-logging es-cluster -oyaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
creationTimestamp: "2021-12-20T06:31:55Z"
generation: 1
name: es-cluster
namespace: kube-logging
resourceVersion: "43285"
uid: a1730c94-1aa5-461c-ba73-9af617ea4c42
spec:
podManagementPolicy: OrderedReady
replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
app: elasticsearch
serviceName: elasticsearch-headless
template:
metadata:
creationTimestamp: null
labels:
app: elasticsearch
spec:
containers:
- env:
- name: cluster.name
value: k8s-logs
- name: node.name
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
- name: discovery.seed_hosts
value: es-cluster-0.elasticsearch
- name: cluster.initial_master_nodes
value: es-cluster-0
- name: ES_JAVA_OPTS
value: -Xms512m -Xmx512m
image: docker.elastic.co/elasticsearch/elasticsearch:7.9.1
imagePullPolicy: IfNotPresent
name: elasticsearch
ports:
- containerPort: 9200
name: rest
protocol: TCP
- containerPort: 9300
name: inter-node
protocol: TCP
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /usr/share/elasticsearch/data
name: data
dnsPolicy: ClusterFirst
initContainers:
- command:
- sh
- -c
- chown -R 1000:1000 /usr/share/elasticsearch/data
image: busybox
imagePullPolicy: Always
name: fix-permissions
resources: {}
securityContext:
privileged: true
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /usr/share/elasticsearch/data
name: data
- command:
- sysctl
- -w
- vm.max_map_count=262144
image: busybox
imagePullPolicy: Always
name: increase-vm-max-map
resources: {}
securityContext:
privileged: true
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
- command:
- sh
- -c
- ulimit -n 65536
image: busybox
imagePullPolicy: Always
name: increase-fd-ulimit
resources: {}
securityContext:
privileged: true
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
terminationGracePeriodSeconds: 30
updateStrategy:
rollingUpdate:
partition: 0
type: RollingUpdate
volumeClaimTemplates:
- apiVersion: v1
kind: PersistentVolumeClaim
metadata:
creationTimestamp: null
labels:
app: elasticsearch
name: data
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
storageClassName: gold
volumeMode: Filesystem
status:
phase: Pending
status:
collisionCount: 0
currentReplicas: 1
currentRevision: es-cluster-6f5bd744c7
observedGeneration: 1
readyReplicas: 1
replicas: 1
updateRevision: es-cluster-6f5bd744c7
updatedReplicas: 1
Output for kubectl get svc -n kube-logging elasticsearch-headless -oyaml :
apiVersion: v1
kind: Service
metadata:
annotations:
cloud.google.com/neg: '{"ingress":true}'
creationTimestamp: "2021-12-20T06:31:40Z"
finalizers:
- service.kubernetes.io/load-balancer-cleanup
labels:
app: elasticsearch
name: elasticsearch-headless
namespace: kube-logging
resourceVersion: "43384"
uid: 80d10b9b-1f22-48ce-ba39-7d5801dc91ee
spec:
clusterIP: 10.8.4.128
clusterIPs:
- 10.8.4.128
externalTrafficPolicy: Cluster
ipFamilies:
- IPv4
ipFamilyPolicy: SingleStack
ports:
- name: rest
nodePort: 30625
port: 9200
protocol: TCP
targetPort: 9200
- name: inter-node
nodePort: 30056
port: 9300
protocol: TCP
targetPort: 9300
selector:
app: elasticsearch
sessionAffinity: None
type: LoadBalancer
status:
loadBalancer:
ingress:
- ip: 35.222.11.96

Answer 1

Additionally to the service you created to expose elastic search outside the cluster, you also need an headless service so that each node / Pod of the elastic cluster can communicate with each other.
I would do the following:
First, inside the spec of the StatefulSet, change spec.serviceName to another value, such as for example elasticsearch-headless
Second. create the new service with the following:
apiVersion: v1
kind: Service
metadata:
# must be the same as the StatefulSet spec.serviceName
name: elasticsearch-headless
namespace: kube-logging
labels:
app: elasticsearch
spec:
type: ClusterIP
# headless service, can be used by elastic Pods to contact each other
clusterIP: None
ports:
- name: rest
port: 9200
protocol: TCP
targetPort: 9200
- name: inter-node
port: 9300
protocol: TCP
targetPort: 9300
selector:
app: elasticsearch
Some docs on Headless Services
Also, you may be interested into checking HELM Charts and ECK because there is several stuff ready to be used in order to deploy production-ready elastic search clusters.