OpenFaaS : Receiving Timeout errors during health check of function Pod - health-check

Question

Issue:
We have added health check configuration to our function. However pod becomes Unhealthy due to timeout error in liveness and readiness checks and consequently getting restarted.
However if I hit same health check url using CURL or browser it returns correct response.
Health check configuration reference.
We are using Kubernetes HPAv2 for auto-scaling (Reference).
test-function.yml
test-function:
lang: quarkus-java-with-fonts
handler: ./test-function
image: repo.azurecr.io/test-function:0.1
labels:
agentpool: openfaas
com.openfaas.scale.min: "2"
com.openfaas.scale.max: "10"
com.openfaas.scale.factor: 0
annotations:
com.openfaas.health.http.path: "/health"
com.openfaas.health.http.initialDelay: "30s"
environment:
secret_name: environment-variables
secrets:
- environment-variables
constraints:
- agentpool=openfaas
limits:
cpu: 1500m
memory: 1Gi
requests:
cpu: 500m
memory: 500Mi
Error Trace :
Liveness probe failed: Get "http://XX.XXX.XX.XX:8080/health": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
Readiness probe failed: Get "http://XX.XXX.XX.XX:8080/health": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
Any idea what could be wrong.

Answer 1

These errors:
Liveness probe failed: Get "http://XX.XXX.XX.XX:8080/health": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
Readiness probe failed: Get "http://XX.XXX.XX.XX:8080/health": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
mean that the HTTP request has failed. For the readiness and liveness probe to work properly, this type of request must be successful.
To find out where the problem is, you need to get the pod IP address. Run:
kubectl get pods -o wide
You should see an output similar to this:
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
<my-pod-name> 1/1 Running 0 25d 10.92.3.4 <my-node-name> <none> 1/1
Take your IP and run:
kubectl exec -t <another_pod> -- curl -I <pod's cluster IP>
If you get a 200 response code, it means the endpoint is properly created and configured. Any other answer suggests there is a problem with your image.
See also:
this similar question (with solution) on github
very similar question on Stack Overflow
guide how to set up liveness and readiness probes
this nice article