I wrote a Spring Boot webservice that uses an embedded tomcat as container.
In case the system reboots I want to backup some information to a mysql database.
In my webservice I use #Scheduled() and #PreDestroy to run the backup.
This goes well when I stop the server with ^C.
But when I kill the process with an sysV skript (/etc/init.d) and the kill command - even though the daemon has a dependency on mysql, the mysql server is shut down before the backup is finished (resulting in SQL Exceptions in my log).
The reason for this is of course, that kill only sends a signal to stop the process.
How can I (from my sysv skript) synchroneously stop the running spring boot tomcat server?
If you include spring-boot-starter-actuator then that provides a REST endpoint for management. One of the endpoints provided is /shutdown. By hitting that endpoint, you will get a controlled shutdown of all resources, which ensures that #PreDestroy will be called. As this could be dangerous to have enabled by default, to use it you will need to add the following to your application.properties file:
endpoints.shutdown.enabled=true
Of course, once you have exposed that endpoint you need to ensure that there's a teeny bit of security applied to prevent just anybody shutting down your server.
On a related note, you may find my answer to Spring Boot application as a Service useful, where I provided the code for a full init.d script which makes use of this.
As an alternative to the "/shutdown" endpoint the Actuator also has an ApplicationPidListener (not enabled by default) that you can use to create a pidfile (which is commonly used in "init.d" style scripts to kill a process when you want to stop it). The JVM should respond to a kill (sigint) and Spring will shutdown gracefully.
Related
Currently, we have four JMS listener containers that are started during the application start. They all connect through Apache ZooKeeper and are manually started. This becomes problematic when a connection to ZooKeeper cannot be established. The (Wicket) application cannot start, even though it is not necessary for the JMS listeners be active to use the application. They simply need to listen to messages in the background, save them and a cron job will process them in batches.
Goals:
Allow the application to start and not be prevented by the message containers not being able to connect.
After the application starts, start the message listeners.
If the connection to one or any of the message listeners goes down, it should attempt to automatically reconnect.
On application shutdown (such as the Tomcat being shutdown), the application should stop the message listeners and the cron job that processes the saved messages.
Make all of this testable (as in, be able to write integration tests for this setup).
Current Setup:
Spring Boot 1.5.6
Apache ZooKeeper 3.4.6
Apache ActiveMQ 5.7
Wicket 7.7.0
Work done so far:
Define a class that implements ApplicationListener<ApplicationReadyEvent>.
Setting the autoStart property of the DefaultMessageListenerContainer to false and start each container in the onApplicationEvent in a separate thread.
Questions:
Is it necessary to start each message container in its own thread? This seems to be overkill, but the way the "start" process works is that the DefaultMessageListenerContainer is built for that listener and then it is started. There is a UI component that a user can use to start/stop the message listeners if need be, and if these are started sequentially in one thread, then the latter three message containers could be null if the first one has yet to connect on startup.
How do I accomplish goals 4 and 5?
Of course, any commments on whether I am on the right track would be helpful.
If you do not start them in a custom thread then the whole application cannot be fully started. It is not just Wicket, but the Servlet container won't change the application state from STARTING to STARTED due to the blocking request to ZooKeeper.
Another option is to use a non-blocking request to ZooKeeper but this is done by the JMS client (ActiveMQ), so you need to check whether this is supported in their docs (both ActiveMQ and ZooKeeper). I haven't used those in several years, so I cannot help you more.
I have two docker instances that I launch with docker-compose.
One holds a Cassandra instance
One holds a Spring Boot application that tries to connect to that application.
However, the Spring Boot application will always fail, because it's trying to connect to a Cassandra instance that is not ready yet to take connections.
I have tried:
Using restart:always in Docker-compose
This still doesn't always work, because the Cassandra might be up 'enough' to no longer crash the Spring Boot application, but not up 'enough' to have successfully created the Table/Column family. On top of that, this is a very hacky solution.
Using healthcheck
It seems like healthcheck in compose doesn't have restart capabilities
Using a bash script as entrypoint
In the hope that I could use netstat,ping,... whatever to determine that readiness state of Cassandra
Right now the only thing that really works is using that same bash script and sleep the process for x seconds, then start the jar. This is even more hacky...
Does anyone have an idea on how to solve this?
Thanks!
Does the spring boot service defined in the docker-compose.yml depends_on the cassandara service? If yes then the service is started only if the cassandra service is ready.
https://docs.docker.com/compose/compose-file/#depends_on
Take a look at this github repository, to find a healthcheck for the cassandra service.
https://github.com/docker-library/healthcheck
CONCLUSION
After some discussion we found out that docker-compose seems not to provide a functionality for waiting until services are up and healthy, such as Kubernetes and Openshift provide (See comments below). They recommend to use wrapper script (docker-entrypoint.sh) which waits for the depending service to come up, which make binaries necessary, the actual service shouldn't use such as the cassandra client binary. Additionally the service depending on cassandra could never get up if cassandra doesn't, which shouldn't happen.
A main thing with microservices is that they have to be resilient for failures and are not supposed to die or not to come up if a depending service is currently not available or unexpectedly disappears. Therefore the microservice should be implemented in a way so that it retries to get connection after startup or an unexpected disappearance. Unexpected is a word actually wrongly used in this context, because you should always expect such issues in a distributed environment, and even with docker-compose you will face issues like that as discussed in this topic.
The following link points to a tutorial which helped to integrate cassandra properly into a spring boot application. It provides a way to implement the retrieval of a cassandra connection with a retry behavior, therefore the service is resilient to a non existing cassandra database and will not fail to start anymore. Hope this helps others as well.
https://dzone.com/articles/containerising-a-spring-data-cassandra-application
I have Spring boot mongo db app, when i start running this application if mongo host is alive, am able to see the my app is up and running successfully,
if my mongo host is down when i start my application my app failed to start.
is there any way even thought if mongo host is down my application should be up and running.
could someone please help me on this?
am using spring boot mongo properties in my application
spring.data.mongodb.repositories.enabled=true
spring.data.mongodb.port=27017
spring.data.mongodb.database=db
spring.data.mongodb.uri=mongodb://mongo-node-1.ballu/db
have same problem with spring boot kafka also.
Sorry for the previous comment. It was for excluding auto config beans, anyway
Is there any way even though if mongo host is down my application
should be up and running.
Yes,
spring.datasource.continue-on-error=true #Whether to stop if an error occurs while initializing the database.
as per spring doc
By default, Spring Boot enables the fail-fast feature of the Spring
JDBC initializer. This means that, if the scripts cause exceptions,
the application fails to start. You can tune that behavior by setting
spring.datasource.continue-on-error.
and as of spring kafka try this(i'm not sure if this meets your requirement)
spring.kafka.admin.fail-fast=true # Whether to fail fast if the broker is not available on startup.
Is there any recommended way to gracefully shutdown a Spring:boot 2 app in Kubernetes.
Catch a termination signal SIGTERM
Tell Tomcat to stop taking new requests. (or Jetty, Undertow or Netty/WebFlux depending on the embedded web server used). Or tell SCS to stop sending/listening for messages on Kafka.
Tell Actuator health endpoint to go SERVICE_UNAVAILABLE (503)
And then after a X seconds shutdown the application or (SIGKILL)
I'm trying to do a graceful shutdown Rest apps and SCS (kafka consumer&producer) apps
If you are on the latest version of spring-boot i.e. 2.3.5.RELEASE, then all you need to do it add the below properties to the application.properties file and you are done.
server.shutdown=graceful
spring.lifecycle.timeout-per-shutdown-phase=30s
In Kubernetes world, you can use the preStop hook. But use this when you actually want a hold before SIGTERM is initiated.
This hook is called immediately before a container is terminated. No parameters are passed to the handler. This event handler is blocking, and must complete before the call to delete the container is sent to the Docker daemon. The SIGTERM notification sent by Docker is also still sent. A more complete description of termination behavior can be found in Termination of Pods.
I need to gracefully start and shutdown the spring boot application using a script . Is there any other way to do it except using an actuator module in the application. Currently i have to stop the spring boot process manually in the task manager.
Ultimately the spring boot application spins off a java process which needs to be killed. At present you are killing it manually.
You have a few options:
You can use SpringApplication.exit(ApplicationContext, ExitCodeGenerator...) method.
If your application is not a long running application then do you have some exit point where your application should stop. At that point you can System.exit(0)
You can use external manager tools, for example on unix you can use supervisor, here is a blog you can read about this.
SpringApplication.run gives you ApplicationContext which you can close.