i am using Spring Boot 2.7 with server.shutdown=graceful. I have an issue with open SSE connections, as those do not terminate during the graceful shutdown.
I get the following:
2023-01-03 17:50:20.079 INFO 39014 --- [tomcat-shutdown] o.s.b.w.e.tomcat.GracefulShutdown : Graceful shutdown aborted with one or more requests still active
I would like to terminate all the SSE connections during the graceful shutdown, so that the server can stop quicker, and not wait for the 30s timeout for connections that won't ever finish.
I tried using #PreDestroy on my bean, but it is invoked after the graceful shutdown, and thus doesn't work.
Is there another Spring hook i could use to close the open SSE connections during the graceful shutdown?
I managed to solve this by implementing SmartLifecycle on my controller. The stop method will be called during the shutdown phase, and cleanup can be done there.
Related
I wrote a simple Spring app using H2 and embedded Redis server, and I have #Configuration for the embedded Redis server, so that it'll shutdown itself (#PreDestroy) before shutdown. It works fine on POSIX. I've tested it on macOS, arch, ubuntu, and it worked fine on all of them with any IDE or editor (i.e. when I clicked the stop button on IntelliJ or VS Code, both the spring app and the redis server gracefully shutdown after printing something like this).
[InstanceCleaner] --snipped-- : Stopping redis server...
[InstanceCleaner] --snipped-- : Redis exited
[ionShutdownHook] --snipped-- : Closing JPA EntityManagerFactory --snipped--
[ionShutdownHook] --snipped-- : HHH000477: Starting delayed evict --snipped--
[ionShutdownHook] --snipped-- : HikariPool-1 - Shutdown initiated...
[ionShutdownHook] --snipped-- : HikariPool-1 - Shutdown completed.
However, for some reason, when I tried to stop the app on Windows by doing the exact same thing (i.e. clicking the stop button or disconnect button or stop button on Spring extension), the spring app doesn't shutdown gracefully.
It just stops without doing anything, and it leaves the embedded redis server running. This already-running redis server prevents me from running the app again. To start the app again without any problem, I either need to terminate the redis process manually, or use other ways to shutdown the app gracefully (I found two methods, one of them is sending ctrl + c to the terminal which runs the spring app, and the other one is making a controller which triggers the shutdown and map them on /shutdown and use the url everytime I want to shutdown the app).
It's not exactly a huge hassle, but I'd like to know why I got this behavior on Windows but not on POSIX (and a fix if there is one).
Currently, we have four JMS listener containers that are started during the application start. They all connect through Apache ZooKeeper and are manually started. This becomes problematic when a connection to ZooKeeper cannot be established. The (Wicket) application cannot start, even though it is not necessary for the JMS listeners be active to use the application. They simply need to listen to messages in the background, save them and a cron job will process them in batches.
Goals:
Allow the application to start and not be prevented by the message containers not being able to connect.
After the application starts, start the message listeners.
If the connection to one or any of the message listeners goes down, it should attempt to automatically reconnect.
On application shutdown (such as the Tomcat being shutdown), the application should stop the message listeners and the cron job that processes the saved messages.
Make all of this testable (as in, be able to write integration tests for this setup).
Current Setup:
Spring Boot 1.5.6
Apache ZooKeeper 3.4.6
Apache ActiveMQ 5.7
Wicket 7.7.0
Work done so far:
Define a class that implements ApplicationListener<ApplicationReadyEvent>.
Setting the autoStart property of the DefaultMessageListenerContainer to false and start each container in the onApplicationEvent in a separate thread.
Questions:
Is it necessary to start each message container in its own thread? This seems to be overkill, but the way the "start" process works is that the DefaultMessageListenerContainer is built for that listener and then it is started. There is a UI component that a user can use to start/stop the message listeners if need be, and if these are started sequentially in one thread, then the latter three message containers could be null if the first one has yet to connect on startup.
How do I accomplish goals 4 and 5?
Of course, any commments on whether I am on the right track would be helpful.
If you do not start them in a custom thread then the whole application cannot be fully started. It is not just Wicket, but the Servlet container won't change the application state from STARTING to STARTED due to the blocking request to ZooKeeper.
Another option is to use a non-blocking request to ZooKeeper but this is done by the JMS client (ActiveMQ), so you need to check whether this is supported in their docs (both ActiveMQ and ZooKeeper). I haven't used those in several years, so I cannot help you more.
Is there a way in spring boot to control the graceful shutdown of the app.
I know that you can have #PreDestroy methods in beans but how can you control the ordering in which those #PreDestroy methods are called.
You can have multiple beans depending on each other will the shutdown of the context look for this dependency already and call the #PreDestroy methods in the right order or not?
For example what I would like to accomplish is:
1.) stop listening for new requests on rest endpoints
2.) prevent rabbit message listeners to accept new messages
3.) wait for all processing that has started before the shutdown but is not finished yet.
Spring-boot-2-3-0 has added support for graceful shutdown.
you can enable graceful shutdown by setting up server.shutdown=graceful property
To configure the timeout period you can use
spring.lifecycle.timeout-per-shutdown-phase=20s
spring boot documentation
If you can not upgrade to spring boot 2.3 then you can check below project
https://github.com/gesellix/graceful-shutdown-spring-boot
Is there any recommended way to gracefully shutdown a Spring:boot 2 app in Kubernetes.
Catch a termination signal SIGTERM
Tell Tomcat to stop taking new requests. (or Jetty, Undertow or Netty/WebFlux depending on the embedded web server used). Or tell SCS to stop sending/listening for messages on Kafka.
Tell Actuator health endpoint to go SERVICE_UNAVAILABLE (503)
And then after a X seconds shutdown the application or (SIGKILL)
I'm trying to do a graceful shutdown Rest apps and SCS (kafka consumer&producer) apps
If you are on the latest version of spring-boot i.e. 2.3.5.RELEASE, then all you need to do it add the below properties to the application.properties file and you are done.
server.shutdown=graceful
spring.lifecycle.timeout-per-shutdown-phase=30s
In Kubernetes world, you can use the preStop hook. But use this when you actually want a hold before SIGTERM is initiated.
This hook is called immediately before a container is terminated. No parameters are passed to the handler. This event handler is blocking, and must complete before the call to delete the container is sent to the Docker daemon. The SIGTERM notification sent by Docker is also still sent. A more complete description of termination behavior can be found in Termination of Pods.
i saw some code use ShutdownHook like this
Runtime.getRuntime().addShutdownHook(new Thread(){
ConfigurableApplicationContext.stop();
//close spring context;
threadpool.shutdownnow();
//close theadpool
});
is there anything useful to do like this?
i thought
when jvm exit ,maybe thread will be shutdown immediately
and spring context will close tooï¼›
what shall we do next when we need to call System.exit() ?
It really depends on your application and the lifecycle of your objects and those threads you appear to have outside of your context. If you are running the spring container inside a standalone java process, then trapping the shutdown hook like this is one way to do that. Another way is to have it listen on a tcp port and send a command to begin the shutdown process. If you are running in a web container like tomcat, then you should follow the standards on normal webapp shutdown, which Spring supports with Context Listeners.
I would also consider redesigning your app so that the threads are all managed with a bean that lives inside your spring container. For instance using a bean that is configured with directives (attributes) for start/stop methods and then that bean would use an Executor for thread pooling. This way, your shutdown is ONLY shutting down the Spring container, and Spring provides very good support for orderly shutdown of beans. One of those beans is your object holding the threads within the Executor. Its a much cleaner way than trying to integrate Spring beans with external threads.
Hope this helps.