RMQConsumer in my code is a spring bean which is initialised at the time of Spring container initialisation.
It starts listening to RMQ messages on the queue as soon that Spring bean is ready irrespective of the whole Spring container is ready or not.
I want to start the consumption of messages only when the kubernates pod status is 1/1. As during rolling deployment, old pods are not terminated unless new pods are 1/1 status, which causes almost double the amount of pod count and spikes the consumption rate which is not desired in my case of some reasons.
What i tried
I removed the channel.basicConsume method of RMq api which was executed at Bean postConstruct. And added the code in a Rest controller method under endpoint ‘’’/postStart’’’ which i configured in kubernates deployment file under lifecycle hook of postStart, which doesnt seems to work. The hook is called before spring application even prints its first log(verified with time stamps).
Can you check if readinessProbe will help in this use case.
Please refer below article which can help.
Promoting Liveness and Readiness as core Spring Boot concepts
https://spring.io/blog/2020/03/25/liveness-and-readiness-probes-with-spring-boot
Related
I have a spring boot microservice running on K8s. I have configured liveness and readiness probes in my yaml file.
For my specific case I want to save a record in database with timestamps whenever the probes get executed in Kubernetes. How can I do this in Spring. Is there a listener kind of thing for this that listens for probe calls and take action based on that?
I have a spring batch application which is to be deployed in kubernetes. it doesn't include spring-boot-starter-web since it's just running cron jobs. is there any way to expose spring-actuator health end points without adding the starter-web dependency.
Batch applications are ephemeral by nature. It does not make sense to expose an actuator endpoint for the duration of the job (would the job still be running when you query the endopoint?). This has been discussed here: https://github.com/spring-projects/spring-boot/issues/21024.
Liveliness Probe and Readiness Probe for spring batch
Same here. A readiness probe is typically used to see if a service is ready to accept requests. I'm not sure if it makes sense to have a readiness probe for batch jobs. A liveliness probe however could make sense (to see if a job is "live", in which case one needs to define what "live" means), but I've never seen an implementation for such a probe in practice as there are other means to report if a job is running or not (such as live metrics with micrometer for instance).
I have an application running Kafka consumers and want to monitor the processing time of each message consumed from the topic. The application is a Spring boot application and exposes Kafka consumer metrics to Spring Actuator Prometheus endpoint using micrometre registry.
Can I use kafka_consumer_commit_latency_avg_seconds or kafka_consumer_commit_latency_max_seconds to monitor or alert?
Those metrics have nothing to do with record processing time. spring-kafka provides metrics for that; see here.
Monitoring Listener Performance
Starting with version 2.3, the listener container will automatically create and update Micrometer Timer s for the listener, if Micrometer is detected on the class path, and a single MeterRegistry is present in the application context. The timers can be disabled by setting the ContainerProperty micrometerEnabled to false.
Two timers are maintained - one for successful calls to the listener and one for >failures.
According to the documentation:
An application is considered ready as soon as application and
command-line runners have been called, see Spring Boot application
lifecycle and related Application Events.
So it doesn't seem to be considering external systems suchs a database. Is this correct?
How could we make the ReadinessStateHealthIndicator evaluate the state of such systems so the pod is taken away from the k8s service load balancer when they are failing of are not available?
We have a situation where we have an abundance of Spring boot applications running in containers (on OpenShift) that access centralized infrastructure (external to the pod) such as databases, queues, etc.
If a piece of central infrastructure is down, the health check returns "unhealthy" (rightfully so). The problem is that the liveliness check sees this, and restarts the pod (the readiness check then sees it's down too, so won't start the app). This is fine when only a few are available, but if many (potentially hundreds) of applications are using this, it forces restarts on all of them (crash loop).
I understand that central infrastructure being down is a bad thing. It "should" never happen. But... if it does (Murphy's law), it throws containers into a frenzy. Just seems like we're either doing something wrong, or we should reconfigure something.
A couple questions:
If you are forced to use centralized infrastructure from a Spring boot app running in a container on OpenShift/Kubernetes, should all actuator checks still be enabled for backend? (bouncing the container really won't fix the backend being down anyway)
Should the /actuator/health endpoint be set for both the liveliness probe and the readiness probe?
What common settings do folk use for the readiness/liveliness probe in a spring boot app? (timeouts/interval/etc).
Using actuator checks for liveness/readiness is the de-facto way to check for healthy app in Spring Boot Pod. Your application, once up, should ideally not go down or become unhealthy if a central piece, such as DB or Queueing service goes down , ideally you should add some sort of a resiliency which will either connect to alternate DR site or wait for certain time period for central service to come back up and app to reconnect. This is more of a technical failure on backend side causing a functional failure of your Application after it was started up cleanly.
Yes , both liveness and readiness is required as they both serve different purposes. Read this
In one of my previous projects, the settings used for readiness was around 30 seconds and liveness at around 90, but to be honest this is completely dependent on your application , if your app takes 1 minute to start , that is what your readiness time should be configured at , and your liveness should factor in the same along with any time required for making failover switch of your backend services.
Hope this helps.