Elastic Search Kubernetes - Disable memory swapping - elasticsearch

I am using Elastic Search(v7.6.1) on a Kubernetes(v1.19) cluster.
The docs suggests to disable swapping:
https://www.elastic.co/guide/en/elasticsearch/reference/current/setup-configuration-memory.html
My yaml:
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
name: elastic-cluster-1
spec:
version: 7.6.1
image: docker.elastic.co/elasticsearch/elasticsearch:7.6.1
nodeSets:
- name: default
count: 3
config:
node.master: true
node.data: true
node.ingest: true
podTemplate:
metadata:
labels:
# additional labels for pods
type: elastic-master-node
spec:
nodeSelector:
node-pool: <NODE_POOL>
initContainers:
# Increase linux map count to allow elastic to store large memory maps
- name: sysctl
securityContext:
privileged: true
command: ['sh', '-c', 'sysctl -w vm.max_map_count=262144']
containers:
- name: elasticsearch
# specify resource limits and requests
resources:
limits:
memory: 11.2Gi
requests:
cpu: 3200m
env:
- name: ES_JAVA_OPTS
value: "-Xms6g -Xmx6g"
# Request persistent data storage for pods
volumeClaimTemplates:
- metadata:
name: elasticsearch-data
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 50Gi
storageClassName: ssd
- name: data
count: 2
config:
node.master: false
node.data: true
node.ingest: true
podTemplate:
metadata:
labels:
# additional labels for pods
type: elastic-data-node
spec:
nodeSelector:
node-pool: <NODE_POOL>
initContainers:
# Increase linux map count to allow elastic to store large memory maps
- name: sysctl
securityContext:
privileged: true
command: ['sh', '-c', 'sysctl -w vm.max_map_count=262144']
containers:
- name: elasticsearch
# specify resource limits and requests
resources:
limits:
memory: 11.2Gi
requests:
cpu: 3200m
env:
- name: ES_JAVA_OPTS
value: "-Xms6g -Xmx6g"
# Request persistent data storage for pods
volumeClaimTemplates:
- metadata:
name: elasticsearch-data
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 50Gi
storageClassName: ssd
# Google cloud storage credentials
secureSettings:
- secretName: "gcs-credentials"
http:
service:
spec:
# expose this cluster Service with a LoadBalancer
type: LoadBalancer
tls:
certificate:
secretName: elasticsearch-certificate
It's not clear to me how to change this yaml in order to disable swapping correctly. Changing each manually is not an option because in every restart the configuration will be lost.
How can I do this?

First of all k8s cluster by default will have swap disabled, this is actually a mandatory requirement. For most cases; especially cloud managed cluster which follows the requirement, you do not need to worry about swapping issue. Even for 1.22, enabling swap is only an alpha feature.
If for whatever reason you need to deal with this, you can consider setting bootstrap.memory_lock to true.
...
containers:
- name: elasticsearch
env:
- name: bootstrap.memory_lock
value: "true"
...

Up until recently, Kubernetes had no control over swapping.
As of 1.22, there's a new alpha feature to do this. The CRI spec does allow for swap allocations. I didn't find anything new in that regard, in the Pod specification: as far as I understand, currently, you could either allow your containers to use as much swap as they can (UnlimitedSwap), or limit swap+memory usage to whatever memory limit you set on your container (LimitedSwap).
Since you're running 1.19, this shouldn't concern you right now. A good practice while deploying your cluster would have been to make sure there is no swap at all on your nodes, or set swapiness to 0 or 1. Checking Kubespray playbooks, we can see they would still unconditionally disable swap.
You can connect your nodes (ssh), make sure there's no swap -- or disable it otherwise. There's nothing you can do in that ElasticSearch object directly.

Related

How to resize an ECK cluster

I have an elasticsearch cluster that has the storage field set to 10Gi, I want to resize this cluster (for testing purposes to 15Gi). However, after changing the storage value from 10Gi to 15Gi I can see that the cluster still did not resize and the generated PVC is still set to 10Gi.
From what I can tell the aws-ebs storage https://kubernetes.io/docs/concepts/storage/storage-classes/ allows for volume expansion when the field allowVolumeExpansion is true. But even when I have this, the volume is never expanded when I change that storage value
---
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: elasticsearch-storage
namespace: test
provisioner: kubernetes.io/aws-ebs
parameters:
type: gp2
reclaimPolicy: Delete
allowVolumeExpansion: true
---
apiVersion: elasticsearch.k8s.elastic.co/v1beta1
kind: Elasticsearch
metadata:
name: elasticsearch
namespace: test
spec:
version: 7.4.2
spec:
http:
tls:
certificate:
secretName: es-cert
nodeSets:
- name: default
count: 3
volumeClaimTemplates:
- metadata:
name: elasticsearch-data
annotations:
volume.beta.kubernetes.io/storage-class: elasticsearch-storage
spec:
accessModes:
- ReadWriteOnce
storageClassName: elasticsearch-storage
resources:
requests:
storage: 15Gi
config:
node.master: true
node.data: true
node.ingest: true
node.store.allow_mmap: false
xpack.security.authc.realms:
native:
native1:
order: 1
---
Technically it should work but your Kubernetes cluster might not be able to connect to the AWS API to expand the volume. Did you check the actual EBS volume on the EC2 console or AWS CLI? You can debug this issue by looking at the kube-controller-manager and cloud-controller manager logs.
My guess is that there is some type of permission issue that from your K8s cluster that cannot talk to your AWS/EC2 API.
If you are running EKS, make sure that the IAM cluster role that you are using has permissions for EC2/EBS. You can check the control plane logs (kube-controller-manager, kube-apiserver, cloud-controller-manager, etc) on CloudWatch.
EDIT:
The Elasticsearch operator uses StatefulSets and as of this date Volume expansion is not supported on StatefulSets.

Data is lost after changes are applied to the image

I am trying to add Elasticsearch to the EKS cluster. But whenever I apply the changes my data is lost. It looks like volume I have attached has been changed and reclaimed.
metadata:
name: elasticsearch-uat
labels:
component: elasticsearch-uat
spec:
replicas: 1
serviceName: elasticsearch-uat
template:
metadata:
...
spec:
initContainers:
- name: init-sysctl
...
containers:
- name: es
securityContext:
capabilities:
add:
- IPC_LOCK
image: 559076975273.dkr.ecr.us-west-2.amazonaws.com/elasticsearch-s3:v2
env:
...
ports:
- containerPort: 9200
name: http
protocol: TCP
- containerPort: 9300
name: transport
protocol: TCP
volumeMounts:
- mountPath: /data
name: es-storage-uat
updateStrategy:
type: RollingUpdate
volumeClaimTemplates:
- metadata:
namespace: k8
name: es-storage-uat
spec:
storageClassName: gp2
accessModes: [ ReadWriteOnce ]
resources:
requests:
storage: 2Gi
This is a statefulset
Please help me to understand this concept. I do not want my data to loose in any condition.
Thanks in advance.
From pv output it is clear that PersistentVolume reclaim policy is set to Delete. Which means that if the pvc is deleted the PersistentVolume gets auto deleted. You loose the data if the pvc is deleted.
In your case, it is appropriate to use the “Retain” policy. With the “Retain” policy, if a user deletes a PersistentVolumeClaim, the corresponding PersistentVolume is not be deleted.
Further to the above, if you are using dynamic storage then set reclaimPolicy field of the storage class to appropriate value. If no reclaimPolicy is specified when a StorageClass object is created, it will default to Delete.

Kubernetes persistent volume claim overriding existing directory's owner and permissions

In Kubernetes, I am having a directory permission problem. I am testing with a pod to create a bare-bones elasticsearch instance, built off of an ElasticSearch provided docker image.
If I use a basic .yaml file to define the container, everything starts up. The problem happens when I attempt to replace a directory created from the docker image with a directory created from mounting of the persistent volume.
The original directory was
drwxrwxr-x 1 elasticsearch root 4096 Aug 30 19:25 data
and if I mount the persistent volume, it changes the owner and permissions to
drwxr-xr-x 2 root root 4096 Aug 30 19:53 data
Now with the elasticsearch process running a the elasticsearch user, this directory can longer be accessed.
I have set the pod's security context's fsGroup to 1000, to match the group of the elasticsearch group. I have set the container's security context's runAsUser to 0. I have set various other combinations of users and group, but to no avail.
Here is my pod, persistent volume claim, and persistent volume definitions.
Any suggestions are welcome.
apiVersion: v1
kind: Pod
metadata:
name: elasticfirst
labels:
app: elasticsearch
spec:
securityContext:
fsGroup: 1000
containers:
- name: es01
image: docker.elastic.co/elasticsearch/elasticsearch:7.3.1
securityContext:
runAsUser: 0
resources:
limits:
memory: 2Gi
cpu: 200m
requests:
memory: 1Gi
cpu: 100m
env:
- name: node.name
value: es01
- name: discovery.seed_hosts
value: es01
- name: cluster.initial_master_nodes
value: es01
- name: cluster.name
value: elasticsearch-cluster
- name: bootstrap.memory_lock
value: "true"
- name: ES_JAVA_OPTS
value: "-Xms1g -Xmx2g"
ports:
- containerPort: 9200
volumeMounts:
- mountPath: "/usr/share/elasticsearch/data"
name: elastic-storage2
nodeSelector:
type: compute
volumes:
- name: elastic-storage2
persistentVolumeClaim:
claimName: elastic-storage2-pvc
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: elastic-storage2-pvc
spec:
storageClassName: local-storage
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 512Mi
apiVersion: v1
kind: PersistentVolume
metadata:
name: elastic-storage2-pv
spec:
storageClassName: local-storage
capacity:
storage: 512Mi
accessModes:
- ReadWriteOnce
hostPath:
path: /var/tmp/pv
Your question is a tiny bit confusing about what is happening versus what you want to be happening, but in general that problem is a common one; that's why many setups use an initContainer: to change the ownership of freshly provisioned PersistentVolumes (as in this example)
In such a setup, the initContainer: would run as root, but would also presumably be a very thin container whose job is only to chown and then exit, leaving your application container -- elasticsearch in your example -- free to run as an unprivileged user
spec:
initContainers:
- name: chown
image: busybox
command:
- chown
- -R
- "1000:1000"
- /the/data
volumeMounts:
- name: es-data
mountPoint: /the/data
containers:
- name: es
# etc etc

How to disable swapping in Elasticsearch on Kubernetes?

As per the official es docs, disabling swapping is one of the best performance boosts available to Elasticsearch.
However, it's proving to be difficult to configure. I've spent a number of hours researching and attempting different methods to disable swapping using the official ES docker image on Kubernetes.
When setting bootstrap.memory_lock: true as an env variable, the image fails to boot up with the error: Unable to lock JVM Memory: error=12, reason=Cannot allocate memory. This can result in part of the JVM being swapped out. Increase RLIMIT_MEMLOCK, soft limit: 65536, hard limit: 65536. As the docs point out, this is kind of expected. I've even mounted a custom /etc/security/limits.conf with the settings, but that's failed.
What is the recommended way to disable swapping when using the official es image on k8s?
And, here are the relevant sections of my yaml
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
name: elastic-data
spec:
serviceName: elastic-data
replicas: 1
template:
spec:
securityContext:
runAsUser: 0
fsGroup: 0
containers:
- name: elastic-data
image: docker.elastic.co/elasticsearch/elasticsearch-oss:6.4.0
env:
- name: ES_JAVA_OPTS
value: "-Xms2g -Xmx2g"
- name: cluster.name
value: "elastic-devs"
- name: node.name
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: discovery.zen.ping.unicast.hosts
value: "elastic-master.default.svc.cluster.local"
- name: node.master
value: "false"
- name: node.ingest
value: "false"
- name: node.data
value: "true"
- name: network.host
value: "0.0.0.0"
- name: path.data
value: /usr/share/elasticsearch/data
- name: indices.memory.index_buffer_size
value: "512MB"
- name: bootstrap.memory_lock
value: "true"
resources:
requests:
memory: "3Gi"
limits:
memory: "3Gi"
ports:
- containerPort: 9300
name: transport
- containerPort: 9200
name: http
volumeMounts:
- name: data-volume
mountPath: /usr/share/elasticsearch/data
- name: swappiness-config
mountPath: /etc/security/limits.conf
subPath: limits.conf
volumes:
- name: data-volume
persistentVolumeClaim:
claimName: pvc-es
- name: swappiness-config
configMap:
name: swappiness-config
items:
- key: limits.conf
path: limits.conf
limits.conf
elasticsearch soft memlock unlimited
elasticsearch hard memlock unlimited
elasticsearch hard nofile 65536
elasticsearch soft nofile 65536
I think, the ulimits in my yaml weren't being recognized, so I followed this post and created an image with a custom entrypoint that set the settings.
Which image-type are you using?
If its Container-Optimized OS (cos) try to switch to Ubuntu based images

elasticsearch on kubernetes - discovery of nodes

We are attempting to run Elasticsearch on top of a kubernetes / flannel / coreos cluster.
As flannel does not support multicast, we cannot use Zen multicast discovery to allow the nodes to find each other, form a cluster and communicate.
Short of hard-coding the IP addresses of all the kubernetes nodes into the ES-config-file, is there another method we can utilise to assist in discovery? Possibly using etcd2 or some other kubernetes-compatible discovery service?
Version 6.2.0 is supporting kubernetes auto discovery
update your elasticsearch.yml as following
discovery.zen.ping.unicast.hosts: "kubernetes service name"
There is a discovery plugin that uses the kubernetes API for cluster discovery:
https://github.com/fabric8io/elasticsearch-cloud-kubernetes
Install the plugin:
/usr/share/elasticsearch/bin/plugin -i io.fabric8/elasticsearch-cloud-kubernetes/1.3.0 --verbose
Create a Kubernetes service for discovery:
apiVersion: v1
kind: Service
metadata:
name: elasticsearch-cluster
spec:
ports:
- port: 9300
selector:
app: elasticsearch
And an elasticsearch.yml:
cloud.k8s.servicedns: elasticsearch-cluster
discovery.type: io.fabric8.elasticsearch.discovery.k8s.K8sDiscoveryModule
Place the containers into a Kubernetes Service. The Kubernetes API makes an 'endpoints' API available that lists the IP addresses of all of the members of a service. This endpoint set will dynamically shrink and grow as you scale the number of pods.
You can access endpoints with:
kubectl get endpoints <service-name>
or directly via the Kubernetes API, see:
https://github.com/kubernetes/kubernetes/blob/master/examples/cassandra/java/src/io/k8s/cassandra/KubernetesSeedProvider.java#L106
for an example of how this was done for Cassandra.
It worked for me only in this configuration.
Important! flannel must be enabled with vxlan.
cluster.yaml
network:
plugin: flannel
options:
flannel_backend_type: vxlan
elasticsearch.yaml
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
name: elastic-cluster
spec:
version: 7.0.1
nodeSets:
- name: node
count: 3
config:
node.master: true
node.data: true
node.ingest: true
xpack.ml.enabled: true
node.store.allow_mmap: true
indices.query.bool.max_clause_count: 100000
# Fixed flannel kubernetes network plugin
discovery.seed_hosts:
{{ range $i, $e := until (3 | int) }}
- elastic-cluster-es-node-{{ $i }}
{{ end }}
podTemplate:
spec:
containers:
- name: elasticsearch
env:
- name: ES_JAVA_OPTS
value: "-Xms4g -Xmx4g"
- name: READINESS_PROBE_TIMEOUT
value: "60"
resources:
requests:
memory: 5Gi
# cpu: 1
limits:
memory: 6Gi
volumeClaimTemplates:
- metadata:
name: elasticsearch-data
spec:
storageClassName: local-elasticsearch-storage
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 5G

Resources