I have a web App, that I am trying to deploy with Kubernetes. It's working correctly, but when I try to add resource limits, the ElasticSearch will not deploy.
elasticsearch-deployment.yaml:
apiVersion: v1
kind: Service
metadata:
name: elasticsearch-service
spec:
type: NodePort
selector:
app: elasticsearch
ports:
- port: 9200
targetPort: 9200
name: serving
- port: 9300
targetPort: 9300
name: node-to-node
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: elasticsearch-deployment
labels:
app: elasticsearch
spec:
replicas: 1
selector:
matchLabels:
app: elasticsearch
template:
metadata:
labels:
app: elasticsearch
spec:
containers:
- name: elasticsearch
image: elasticsearch:7.9.0
ports:
- containerPort: 9200
- containerPort: 9300
env:
- name: discovery.type
value: single-node
# resources:
# limits:
# memory: 8Gi
# cpu: "4"
# requests:
# memory: 4Gi
# cpu: "2"
If I uncomment the resources section of the file, the pod is stuck in pending:
> kubectl get pods
NAME READY STATUS RESTARTS AGE
backend-deployment-bd4f98697-rxsz8 1/1 Running 1 (6m9s ago) 6m40s
elasticsearch-deployment-644475545b-t75pp 0/1 Pending 0 6m40s
frontend-deployment-8bc989f89-4g6v7 1/1 Running 0 6m40s
mysql-0 1/1 Running 0 6m40s
If I check the events:
> kubectl get events
...
Warning FailedScheduling pod/elasticsearch-deployment-54d9cdd879-k69js 0/1 nodes are available: 1 Insufficient cpu.
Warning FailedScheduling pod/elasticsearch-deployment-54d9cdd879-rjj24 0/1 nodes are available: 1 Insufficient cpu.
...
The events says that the pod has Insufficient cpu, but I tried to change the resource limits to :
resources:
limits:
memory: 8Gi
cpu: "18"
requests:
memory: 4Gi
cpu: "18"
Still doesn't works, the only way for it to works is to remove the resource limit, but why?
It is because of the request, not the limit. It means your node doesn't have enough memory to schedule a pod that requests 2 CPUs. You need to set the value to a lower one (e.g. 500m).
You can check your server's allocatable CPUs. The sum of all Pod's CPU requests should be lower than this.
# kubectl describe nodes
...
Allocatable:
cpu: 28
...
In Addition to Daigo
Requests and limits are on a per-container basis. Each container in the Pod gets its own individual limit and request. When adding the limits and requests for each container together, you will get an aggregate value for the Pod.
Combining the requests and limits value of each container will represent the Pod requests which is 500m cpu and 128Mi memory and Pod limits is 1 cpu and 256Mi memory.
Requests are what the container is guaranteed to get. If a container requests a resource, k8s will only schedule it on a node that can give it that resource.
Limits, on the other hand, make sure a container won't go above a certain value or limit.
Without requests and limits defined, the scheduler might place the Pod on a node that has less than 1 GiB memory available.
Related
I have a container written in go. It deploys and runs on my DockerDesktop & on my Kubernetes cluster in DockerDesktop.
I have pushed the same container to Artefact Repository and it fails to deploy.
So I deployed it to CloudRun, and it works! Very confused.
My GKE cluster is autopilot so I assume the are no resource issues.
I expected to get a running container however i got
Cannot schedule pods: Insufficient cpu.
PodUnschedulable
Reason
Cannot schedule pods: Insufficient cpu.
Learn more
Source
gmail-sender-7944d6d4d4-tsdt9
gmail-sender-7944d6d4d4-pc9xp
gmail-sender-7944d6d4d4-kdlds
PodUnschedulable Cannot schedule pods: Insufficient memory.
My deployment file is as follows
apiVersion: apps/v1
kind: Deployment
metadata:
name: gmail-sender
labels:
app: gmail-sender
spec:
replicas: 1
selector:
matchLabels:
app: gmail-sender
template:
metadata:
labels:
app: gmail-sender
spec:
containers:
- name: gmail-sender
image: europe-west2-docker.pkg.dev/ea-website-359514/gmail-sender/gmail-sender:latest
imagePullPolicy: IfNotPresent
ports:
- containerPort: 8099
---
apiVersion: v1
kind: Service
metadata:
name: gmail-sender-cluster-ip
labels:
app: gmail-sender
spec:
ports:
- port: 8099
protocol: TCP
Looking at the error it is clear that node doesnt have sufficient memory and cpu to schedule/run the workload. Check the node configuration and ensure that resources are available on the node to host the workload
Cannot schedule pods: Insufficient cpu.
PodUnschedulable Reason Cannot schedule pods: Insufficient cpu.
Learn more Source gmail-sender-7944d6d4d4-tsdt9
gmail-sender-7944d6d4d4-pc9xp gmail-sender-7944d6d4d4-kdlds PodUnschedulable
Cannot schedule pods: Insufficient memory.
I have configured a spring-boot pod and configured the liveness and readiness probes.
When I start the pod, the describe command is showing the below output.
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 92s default-scheduler Successfully assigned pradeep-ns/order-microservice-rs-8tqrv to pool-h4jq5h014-ukl3l
Normal Pulled 43s (x2 over 91s) kubelet Container image "classpathio/order-microservice:latest" already present on machine
Normal Created 43s (x2 over 91s) kubelet Created container order-microservice
Normal Started 43s (x2 over 91s) kubelet Started container order-microservice
Warning Unhealthy 12s (x6 over 72s) kubelet Liveness probe failed: Get "http://10.244.0.206:8222/actuator/health/liveness": dial tcp 10.244.0.206:8222: connect: connection refused
Normal Killing 12s (x2 over 52s) kubelet Container order-microservice failed liveness probe, will be restarted
Warning Unhealthy 2s (x8 over 72s) kubelet Readiness probe failed: Get "http://10.244.0.206:8222/actuator/health/readiness": dial tcp 10.244.0.206:8222: connect: connection refused
The pod definition is like below
apiVersion: apps/v1
kind: ReplicaSet
metadata:
name: order-microservice-rs
labels:
app: order-microservice
spec:
replicas: 1
selector:
matchLabels:
app: order-microservice
template:
metadata:
name: order-microservice
labels:
app: order-microservice
spec:
containers:
- name: order-microservice
image: classpathio/order-microservice:latest
imagePullPolicy: IfNotPresent
env:
- name: SPRING_PROFILES_ACTIVE
value: dev
- name: SPRING_DATASOURCE_USERNAME
valueFrom:
secretKeyRef:
key: username
name: db-credentials
- name: SPRING_DATASOURCE_PASSWORD
valueFrom:
secretKeyRef:
key: password
name: db-credentials
volumeMounts:
- name: app-config
mountPath: /app/config
- name: app-logs
mountPath: /var/log
livenessProbe:
httpGet:
port: 8222
path: /actuator/health/liveness
initialDelaySeconds: 10
periodSeconds: 10
readinessProbe:
httpGet:
port: 8222
path: /actuator/health/readiness
initialDelaySeconds: 10
periodSeconds: 10
resources:
requests:
memory: "550Mi"
cpu: "500m"
limits:
memory: "550Mi"
cpu: "750m"
volumes:
- name: app-config
configMap:
name: order-microservice-config
- name: app-logs
emptyDir: {}
restartPolicy: Always
If I disable the liveness and readiness probe in the replica-set manifest and I exec into the pod, I am getting a valid response when invoking http://localhost:8222/actuator/health/liveness and http://localhost:8222/actuator/health/readiness endpoint.
Why is my pod restarting and failing when invoking the readiness and liveness endpoint with Kubernetes. Where am I going wrong?
Update
If I remove the resource section, the pods are running but when added the resource parameters, the probes are failing.
When you limit the container / spring application to 0.5 cores (500 millicores) the startup probably takes longer than the given liveness probe thresholds.
You can either increase them, or use a startupProbe with more relaxed settings (f.e. failureThreshold 10). You can reduce the period for the liveness probe in that case and get faster feedback after a successful container start was detected.
Your pod config only give 0.5 Core of CPU, and your check time was too short. The spring boot start may take a long time more than 10 seconds according your server CPU performance. This is my config of spring boot pod may give you a point.
"livenessProbe": {
"httpGet": {
"path": "/actuator/liveness",
"port": 11032,
"scheme": "HTTP"
},
"initialDelaySeconds": 90,
"timeoutSeconds": 30,
"periodSeconds": 30,
"successThreshold": 1,
"failureThreshold": 3
},
"readinessProbe": {
"httpGet": {
"path": "/actuator/health",
"port": 11032,
"scheme": "HTTP"
},
"initialDelaySeconds": 60,
"timeoutSeconds": 30,
"periodSeconds": 30,
"successThreshold": 1,
"failureThreshold": 3
},
and I did not limit the CPU and memory resource, if you limit the CPU, it will take more time. Hop this could help you.
When you are trying the request against your localhost, and it works, it is not a guarantee that it is going to work on other network interfaces. Kubelet is a node agent, so the request is going to your eth0, or equivalent, not your localhost.
You can check it by making the request from another pod to your pod's IP address, or the service backing it up.
Probably you are making your application to serve on localhost, while you have to make it serve on 0.0.0.0, or eth0.
I am using Elastic Search(v7.6.1) on a Kubernetes(v1.19) cluster.
The docs suggests to disable swapping:
https://www.elastic.co/guide/en/elasticsearch/reference/current/setup-configuration-memory.html
My yaml:
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
name: elastic-cluster-1
spec:
version: 7.6.1
image: docker.elastic.co/elasticsearch/elasticsearch:7.6.1
nodeSets:
- name: default
count: 3
config:
node.master: true
node.data: true
node.ingest: true
podTemplate:
metadata:
labels:
# additional labels for pods
type: elastic-master-node
spec:
nodeSelector:
node-pool: <NODE_POOL>
initContainers:
# Increase linux map count to allow elastic to store large memory maps
- name: sysctl
securityContext:
privileged: true
command: ['sh', '-c', 'sysctl -w vm.max_map_count=262144']
containers:
- name: elasticsearch
# specify resource limits and requests
resources:
limits:
memory: 11.2Gi
requests:
cpu: 3200m
env:
- name: ES_JAVA_OPTS
value: "-Xms6g -Xmx6g"
# Request persistent data storage for pods
volumeClaimTemplates:
- metadata:
name: elasticsearch-data
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 50Gi
storageClassName: ssd
- name: data
count: 2
config:
node.master: false
node.data: true
node.ingest: true
podTemplate:
metadata:
labels:
# additional labels for pods
type: elastic-data-node
spec:
nodeSelector:
node-pool: <NODE_POOL>
initContainers:
# Increase linux map count to allow elastic to store large memory maps
- name: sysctl
securityContext:
privileged: true
command: ['sh', '-c', 'sysctl -w vm.max_map_count=262144']
containers:
- name: elasticsearch
# specify resource limits and requests
resources:
limits:
memory: 11.2Gi
requests:
cpu: 3200m
env:
- name: ES_JAVA_OPTS
value: "-Xms6g -Xmx6g"
# Request persistent data storage for pods
volumeClaimTemplates:
- metadata:
name: elasticsearch-data
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 50Gi
storageClassName: ssd
# Google cloud storage credentials
secureSettings:
- secretName: "gcs-credentials"
http:
service:
spec:
# expose this cluster Service with a LoadBalancer
type: LoadBalancer
tls:
certificate:
secretName: elasticsearch-certificate
It's not clear to me how to change this yaml in order to disable swapping correctly. Changing each manually is not an option because in every restart the configuration will be lost.
How can I do this?
First of all k8s cluster by default will have swap disabled, this is actually a mandatory requirement. For most cases; especially cloud managed cluster which follows the requirement, you do not need to worry about swapping issue. Even for 1.22, enabling swap is only an alpha feature.
If for whatever reason you need to deal with this, you can consider setting bootstrap.memory_lock to true.
...
containers:
- name: elasticsearch
env:
- name: bootstrap.memory_lock
value: "true"
...
Up until recently, Kubernetes had no control over swapping.
As of 1.22, there's a new alpha feature to do this. The CRI spec does allow for swap allocations. I didn't find anything new in that regard, in the Pod specification: as far as I understand, currently, you could either allow your containers to use as much swap as they can (UnlimitedSwap), or limit swap+memory usage to whatever memory limit you set on your container (LimitedSwap).
Since you're running 1.19, this shouldn't concern you right now. A good practice while deploying your cluster would have been to make sure there is no swap at all on your nodes, or set swapiness to 0 or 1. Checking Kubespray playbooks, we can see they would still unconditionally disable swap.
You can connect your nodes (ssh), make sure there's no swap -- or disable it otherwise. There's nothing you can do in that ElasticSearch object directly.
I'm playing with the Elasticsearch operator Kubernetes and created two stateful sets (see https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-orchestration.html):
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
name: quickstart
spec:
version: 7.12.1
nodeSets:
- name: master-nodes
count: 3
config:
node.roles: ["master"]
volumeClaimTemplates:
- metadata:
name: elasticsearch-data
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
storageClassName: standard
- name: data-nodes
count: 3
config:
node.roles: ["data"]
volumeClaimTemplates:
- metadata:
name: elasticsearch-data
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
storageClassName: standard
The problem is that I cannot delete the stateful sets. After deletion, they're recreated automatically:
my-PC:~$ kubectl get sts
NAME READY AGE
quickstart-es-data-nodes 0/0 14m
quickstart-es-master-nodes 0/0 18m
my-PC:~$ kubectl delete sts quickstart-es-data-nodes --force --grace-period=0
warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely.
statefulset.apps "quickstart-es-data-nodes" force deleted
my-PC:~$ kubectl get sts
NAME READY AGE
quickstart-es-data-nodes 0/3 3s
quickstart-es-master-nodes 0/0 18m
Before deletion I already scaled down the statefulset to 0 to ensure that all pods are terminated. But after deletion, the stateful is recreated (see quickstart-es-data-nodes).
So, anyone having any idea how I can delete the stateful sets without being recreated?
it's due to the operator you are using for the Elasticsearch. Operator manage the statefulset and will update if you delete it.
Behind the scenes, ECK translates each NodeSet specified in the
Elasticsearch resource into a StatefulSet in Kubernetes.
if you read the documentation: https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-orchestration.html#k8s-statefulsets
https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#on-delete
You have to delete the custom object. The operator owns those StatefulSets and will continually update them to match its expected content.
I finally got the answer... I need to run the following command for deletion:
kubectl delete elasticsearch quickstart
This finally removed the quickstart examples.
I'm trying to deploy elk stack in kubernetes cluster with helm, using this chart. When I launch
helm install elk-stack stable/elastic-stack
I receive the following message:
NAME: elk-stack
LAST DEPLOYED: Mon Aug 24 07:30:31 2020
NAMESPACE: default
STATUS: deployed
REVISION: 1
NOTES:
The elasticsearch cluster and associated extras have been installed.
Kibana can be accessed:
* Within your cluster, at the following DNS name at port 9200:
elk-stack-elastic-stack.default.svc.cluster.local
* From outside the cluster, run these commands in the same shell:
export POD_NAME=$(kubectl get pods --namespace default -l "app=elastic-stack,release=elk-stack" -o jsonpath="{.items[0].metadata.name}")
echo "Visit http://127.0.0.1:5601 to use Kibana"
kubectl port-forward --namespace default $POD_NAME 5601:5601
But when I run
kubectl get pods
the result is:
NAME READY STATUS RESTARTS AGE
elk-stack-elasticsearch-client-7fcfc7b858-5f7fw 0/1 Running 0 12m
elk-stack-elasticsearch-client-7fcfc7b858-zdkwd 0/1 Running 1 12m
elk-stack-elasticsearch-data-0 0/1 Pending 0 12m
elk-stack-elasticsearch-master-0 0/1 Pending 0 12m
elk-stack-kibana-cb7d9ccbf-msw95 1/1 Running 0 12m
elk-stack-logstash-0 0/1 Pending 0 12m
Using kubectl describe pods command, I see that for elasticsearch pods the problem is:
Warning FailedScheduling 6m29s default-scheduler running "VolumeBinding" filter plugin for pod "elk-stack-elasticsearch-data-0": pod has unbound immediate PersistentVolumeClaims
and for logstash pods:
Warning FailedScheduling 7m53s default-scheduler running "VolumeBinding" filter plugin for pod "elk-stack-logstash-0": pod has unbound immediate PersistentVolumeClaims
Output of kubectl get pv,pvc,sc -A:
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
persistentvolume/elasticsearch-data 10Gi RWO Retain Bound default/elasticsearch-data manual 16d
NAMESPACE NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
default persistentvolumeclaim/claim1 Pending slow 64m
default persistentvolumeclaim/data-elk-stack-elasticsearch-data-0 Pending 120m
default persistentvolumeclaim/data-elk-stack-elasticsearch-master-0 Pending 120m
default persistentvolumeclaim/data-elk-stack-logstash-0 Pending 120m
default persistentvolumeclaim/elasticsearch-data Bound elasticsearch-data 10Gi RWO manual 16d
default persistentvolumeclaim/elasticsearch-data-elasticsearch-data-0 Pending 17d
default persistentvolumeclaim/elasticsearch-data-elasticsearch-data-1 Pending 17d
default persistentvolumeclaim/elasticsearch-data-quickstart-es-default-0 Pending 16d
default persistentvolumeclaim/elasticsearch-master-elasticsearch-master-0 Pending 17d
default persistentvolumeclaim/elasticsearch-master-elasticsearch-master-1 Pending 17d
default persistentvolumeclaim/elasticsearch-master-elasticsearch-master-2 Pending 16d
NAMESPACE NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
storageclass.storage.k8s.io/slow (default) kubernetes.io/gce-pd Delete Immediate false 66m
Storage class slow and Persistent volume claim claim1 are my experiments. I create they using kubectl create and a yaml file, the others is automatically created by helm (I think).
Output of kubectl get pvc data-elk-stack-elasticsearch-master-0 -o yaml:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
creationTimestamp: "2020-08-24T07:30:38Z"
finalizers:
- kubernetes.io/pvc-protection
labels:
app: elasticsearch
release: elk-stack
managedFields:
- apiVersion: v1
fieldsType: FieldsV1
fieldsV1:
f:metadata:
f:labels:
.: {}
f:app: {}
f:release: {}
f:spec:
f:accessModes: {}
f:resources:
f:requests:
.: {}
f:storage: {}
f:volumeMode: {}
f:status:
f:phase: {}
manager: kube-controller-manager
operation: Update
time: "2020-08-24T07:30:38Z"
name: data-elk-stack-elasticsearch-master-0
namespace: default
resourceVersion: "201123"
selfLink: /api/v1/namespaces/default/persistentvolumeclaims/data-elk-stack-elasticsearch-master-0
uid: de58f769-f9a7-41ad-a449-ef16d4b72bc6
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 4Gi
volumeMode: Filesystem
status:
phase: Pending
Can somebody please help me to fix this problem? Thanks in advance.
The reason why pod is pending is below PVCs are pending because corresponding PVs are not created.
data-elk-stack-elasticsearch-master-0
data-elk-stack-logstash-0
data-elk-stack-elasticsearch-data-0
Since you have mentioned this is for local development you can use hostPath volume for the PV. So create PV for each of the pending PVCs using the sample PV below. So you will create 3 PVs in total.
apiVersion: v1
kind: PersistentVolume
metadata:
name: elk-master
labels:
type: local
spec:
capacity:
storage: 4Gi
accessModes:
- ReadWriteOnce
hostPath:
path: "/mnt/data"
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: elk-logstash
labels:
type: local
spec:
capacity:
storage: 2Gi
accessModes:
- ReadWriteOnce
hostPath:
path: "/mnt/data"
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: elk-data
labels:
type: local
spec:
capacity:
storage: 30Gi
accessModes:
- ReadWriteOnce
hostPath:
path: "/mnt/data"