k8s spring boot pod failing readiness and liveness probe - spring-boot

Question

I have configured a spring-boot pod and configured the liveness and readiness probes.
When I start the pod, the describe command is showing the below output.
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 92s default-scheduler Successfully assigned pradeep-ns/order-microservice-rs-8tqrv to pool-h4jq5h014-ukl3l
Normal Pulled 43s (x2 over 91s) kubelet Container image "classpathio/order-microservice:latest" already present on machine
Normal Created 43s (x2 over 91s) kubelet Created container order-microservice
Normal Started 43s (x2 over 91s) kubelet Started container order-microservice
Warning Unhealthy 12s (x6 over 72s) kubelet Liveness probe failed: Get "http://10.244.0.206:8222/actuator/health/liveness": dial tcp 10.244.0.206:8222: connect: connection refused
Normal Killing 12s (x2 over 52s) kubelet Container order-microservice failed liveness probe, will be restarted
Warning Unhealthy 2s (x8 over 72s) kubelet Readiness probe failed: Get "http://10.244.0.206:8222/actuator/health/readiness": dial tcp 10.244.0.206:8222: connect: connection refused
The pod definition is like below
apiVersion: apps/v1
kind: ReplicaSet
metadata:
name: order-microservice-rs
labels:
app: order-microservice
spec:
replicas: 1
selector:
matchLabels:
app: order-microservice
template:
metadata:
name: order-microservice
labels:
app: order-microservice
spec:
containers:
- name: order-microservice
image: classpathio/order-microservice:latest
imagePullPolicy: IfNotPresent
env:
- name: SPRING_PROFILES_ACTIVE
value: dev
- name: SPRING_DATASOURCE_USERNAME
valueFrom:
secretKeyRef:
key: username
name: db-credentials
- name: SPRING_DATASOURCE_PASSWORD
valueFrom:
secretKeyRef:
key: password
name: db-credentials
volumeMounts:
- name: app-config
mountPath: /app/config
- name: app-logs
mountPath: /var/log
livenessProbe:
httpGet:
port: 8222
path: /actuator/health/liveness
initialDelaySeconds: 10
periodSeconds: 10
readinessProbe:
httpGet:
port: 8222
path: /actuator/health/readiness
initialDelaySeconds: 10
periodSeconds: 10
resources:
requests:
memory: "550Mi"
cpu: "500m"
limits:
memory: "550Mi"
cpu: "750m"
volumes:
- name: app-config
configMap:
name: order-microservice-config
- name: app-logs
emptyDir: {}
restartPolicy: Always
If I disable the liveness and readiness probe in the replica-set manifest and I exec into the pod, I am getting a valid response when invoking http://localhost:8222/actuator/health/liveness and http://localhost:8222/actuator/health/readiness endpoint.
Why is my pod restarting and failing when invoking the readiness and liveness endpoint with Kubernetes. Where am I going wrong?
Update
If I remove the resource section, the pods are running but when added the resource parameters, the probes are failing.

Answer 1

When you limit the container / spring application to 0.5 cores (500 millicores) the startup probably takes longer than the given liveness probe thresholds.
You can either increase them, or use a startupProbe with more relaxed settings (f.e. failureThreshold 10). You can reduce the period for the liveness probe in that case and get faster feedback after a successful container start was detected.

Answer 2

Your pod config only give 0.5 Core of CPU, and your check time was too short. The spring boot start may take a long time more than 10 seconds according your server CPU performance. This is my config of spring boot pod may give you a point.
"livenessProbe": {
"httpGet": {
"path": "/actuator/liveness",
"port": 11032,
"scheme": "HTTP"
},
"initialDelaySeconds": 90,
"timeoutSeconds": 30,
"periodSeconds": 30,
"successThreshold": 1,
"failureThreshold": 3
},
"readinessProbe": {
"httpGet": {
"path": "/actuator/health",
"port": 11032,
"scheme": "HTTP"
},
"initialDelaySeconds": 60,
"timeoutSeconds": 30,
"periodSeconds": 30,
"successThreshold": 1,
"failureThreshold": 3
},
and I did not limit the CPU and memory resource, if you limit the CPU, it will take more time. Hop this could help you.

Answer 3

When you are trying the request against your localhost, and it works, it is not a guarantee that it is going to work on other network interfaces. Kubelet is a node agent, so the request is going to your eth0, or equivalent, not your localhost.
You can check it by making the request from another pod to your pod's IP address, or the service backing it up.
Probably you are making your application to serve on localhost, while you have to make it serve on 0.0.0.0, or eth0.