I have a spring-boot application (with default Tomcat server) producing metrics for Prometheus using micrometer. The application is configured with graceful shutdown.
When shutting down I want to make sure Prometheus scrapes one last time to get the final metrics but since the server stops accepting new requests Prometheus scrape request won't be handled.
I've looked at Prometheus PushGateway which allows me to explicitly push the metrics to Prometheus although I wanted to avoid it and was looking for some solution from Spring end. Is there anyway I can avoid blocking Prometheus scrape requests upon graceful shutdown? Probably overriding some shutdown event handler...
Related
I have an application running Kafka consumers and want to monitor the processing time of each message consumed from the topic. The application is a Spring boot application and exposes Kafka consumer metrics to Spring Actuator Prometheus endpoint using micrometre registry.
Can I use kafka_consumer_commit_latency_avg_seconds or kafka_consumer_commit_latency_max_seconds to monitor or alert?
Those metrics have nothing to do with record processing time. spring-kafka provides metrics for that; see here.
Monitoring Listener Performance
Starting with version 2.3, the listener container will automatically create and update Micrometer Timer s for the listener, if Micrometer is detected on the class path, and a single MeterRegistry is present in the application context. The timers can be disabled by setting the ContainerProperty micrometerEnabled to false.
Two timers are maintained - one for successful calls to the listener and one for >failures.
I have configured micrometer and prometheus for my spring boot application, and I can see following metrics(genrated using Timer) at endpoint /actuator/prometheus:
# HELP timer_test_seconds Time spent serving test requests
# TYPE timer_test_seconds summary
timer_test_seconds_count{class="com.test.MyController",exception="none",method="testTimer",} 2.0
timer_test_seconds_sum{class="com.test.MyController",exception="none",method="testTimer",} 8.6461705
# HELP timer_test_seconds_max Time spent serving test requests
# TYPE timer_test_seconds_max gauge
timer_test_seconds_max{class="com.test.MyController",exception="none",method="testTimer",} 4.5578234
But when I run one of this query in Grafana(configured against prometheus instance), I don't see any result.
Is any configuration needed for this?
These are the metrics exposed by your service. Grafana does not interact with your service directly.
Is Prometheus up and scraping this target? You can check if Prometheus has any data at localhost:port/graph.
Is Grafana pointing to your Prometheus endpoint correctly?
Most likely, Prometheus is not scraping your target correctly. Keep in mind that Grafana is unaware of your service at all so you should first check if there is any data in Prometheus.
Currently, we have four JMS listener containers that are started during the application start. They all connect through Apache ZooKeeper and are manually started. This becomes problematic when a connection to ZooKeeper cannot be established. The (Wicket) application cannot start, even though it is not necessary for the JMS listeners be active to use the application. They simply need to listen to messages in the background, save them and a cron job will process them in batches.
Goals:
Allow the application to start and not be prevented by the message containers not being able to connect.
After the application starts, start the message listeners.
If the connection to one or any of the message listeners goes down, it should attempt to automatically reconnect.
On application shutdown (such as the Tomcat being shutdown), the application should stop the message listeners and the cron job that processes the saved messages.
Make all of this testable (as in, be able to write integration tests for this setup).
Current Setup:
Spring Boot 1.5.6
Apache ZooKeeper 3.4.6
Apache ActiveMQ 5.7
Wicket 7.7.0
Work done so far:
Define a class that implements ApplicationListener<ApplicationReadyEvent>.
Setting the autoStart property of the DefaultMessageListenerContainer to false and start each container in the onApplicationEvent in a separate thread.
Questions:
Is it necessary to start each message container in its own thread? This seems to be overkill, but the way the "start" process works is that the DefaultMessageListenerContainer is built for that listener and then it is started. There is a UI component that a user can use to start/stop the message listeners if need be, and if these are started sequentially in one thread, then the latter three message containers could be null if the first one has yet to connect on startup.
How do I accomplish goals 4 and 5?
Of course, any commments on whether I am on the right track would be helpful.
If you do not start them in a custom thread then the whole application cannot be fully started. It is not just Wicket, but the Servlet container won't change the application state from STARTING to STARTED due to the blocking request to ZooKeeper.
Another option is to use a non-blocking request to ZooKeeper but this is done by the JMS client (ActiveMQ), so you need to check whether this is supported in their docs (both ActiveMQ and ZooKeeper). I haven't used those in several years, so I cannot help you more.
Is there any recommended way to gracefully shutdown a Spring:boot 2 app in Kubernetes.
Catch a termination signal SIGTERM
Tell Tomcat to stop taking new requests. (or Jetty, Undertow or Netty/WebFlux depending on the embedded web server used). Or tell SCS to stop sending/listening for messages on Kafka.
Tell Actuator health endpoint to go SERVICE_UNAVAILABLE (503)
And then after a X seconds shutdown the application or (SIGKILL)
I'm trying to do a graceful shutdown Rest apps and SCS (kafka consumer&producer) apps
If you are on the latest version of spring-boot i.e. 2.3.5.RELEASE, then all you need to do it add the below properties to the application.properties file and you are done.
server.shutdown=graceful
spring.lifecycle.timeout-per-shutdown-phase=30s
In Kubernetes world, you can use the preStop hook. But use this when you actually want a hold before SIGTERM is initiated.
This hook is called immediately before a container is terminated. No parameters are passed to the handler. This event handler is blocking, and must complete before the call to delete the container is sent to the Docker daemon. The SIGTERM notification sent by Docker is also still sent. A more complete description of termination behavior can be found in Termination of Pods.
I'd like to get the current number of active client request threads in a spring boot app using embedded tomcat so that I can expose it over actuator's metrics endpoint. I'm not looking for active sessions, but active request processing threads. Preferably, I'd like to get this data per connector as well.
Does anyone have any ideas on a good way to get at this information in spring boot?
I don't know if this is what you are looking for, but you can get serveral values like that via JMX. You can start you current Spring Boot app and open Java Mission Control ([JDK directory]/bin). Open MBean browser and have a look at Tomcat->Thread Pool->[ConnectorName]:
You can get those values programmatically, too.