since its 2.3 release, SpringBoot has the feature called "graceful shutdown". When a certain signal is received by the JVM, SpringBoot executes some code (because it registers a callback within the JVM) so that the application is shut down gracefully.
In the course of the shutdown process, the application context is closed, traffic is not accepted anymore, beans get destroyed etc. This is described in this section of the SpringBoot docs and elsewhere.
My question is: is it possible to get notified about the fact that SpringBoot has initiated a shutdown? By "get notified" I mean (ideally) this: I want to have a bean which would be part of the application context. This bean should get a notification (one of its methods should be called or via events) when the shutdown is initiated but before any actions are taken in that direction. I.e. #PreDestroy and other lifecycle callbacks are too late IMO. My desired notification should come earlier (when the traffic is still accepted).
I could of course register my own callback within the JVM but it wouldn't look good and also I could not be sure whether my callback is executed before or after the SpringBoot's one.
Thank you for any hints.
Related
I'm currently trying to refactor the processing of JMS messages to work in a distributed/cloud environment. To allow a better retry and error handling the messages are first stored to the database with a JPA entity and then read by spring integration jpa inbound adapter. This works fine as long as just a single instance of my service is running. However when multiple instances are running, the instances try to process the same message even after introducing a processing state on the persisted messages.
I have already tried to save the JMS messages in a JDBC message store, however then I would have to define a group identifier according to which an instance could select a message which is not really possible since the number of instances is dynamic and I can not assign a group id for each instance. Another possibility could be some kind of distributed lock with a LockRegistry but I couldn't make that work.
Do you have any hint/advice how I could implement the following requirements the best with spring integration:
JMS message should be persisted
Any instance can pick up the message and process it
If the processing fails there will be a retry for x times (could also be retried by another instance)
If an instance crashes or gets killed during the processing the message must not be lost
Is there maybe some spring-cloud component which could be helpful?
I'm happy about every hint in which direction I should go.
I have recently had to support a colleague in verifying why some system tests are not passing in wildfly, system tests that pass consistently on weblogic and glass fish.
After analysing the log, it became clear the reason is related to a JMS message sent by a backed thread getting committed to a queue too soon, when the expectation was the message would be committed when the entry point Container Managed Transaction of an MDB commits. So the message is going out before the MDB that sends it is done running.
In weblogic, to achieve the expected behaviour, you need to make sure that when you take the connection factory given by the container , which is XA configured, you set the connection.createseesion with
transacted = true and
acknowledgement = session transacted.
In a process similar to the one depicted in this URL
http://www.mastertheboss.com/jboss-server/jboss-jms/sending-jms-messages-over-xa-with-wildfly-jboss-as
Except in the snippet above auto acknowledge is set and the first parameter is set to false.
In wildly when our weblogic and glass fish configuration is used, nothing is committed and the system behaves as if the JMS message sent were to be rolled back.
If configuration as in the example above were to be used, instead what would happen is that the JMS message is immediately and the consumer MDB immediately launches being trigerred before the producer transaction actually ends, causing the system test to fail.
According to the official JMS configuration, by using a connection-pooled factory with the transaction=XA attribute, the container should immediately bind the commit of the transaction to the lifecycle of the parent transaction.
See official documentation bellow in particular in respect to the Java:/JmsXa connection factory.
https://docs.jboss.org/author/display/WFLY10/Messaging+configuration
My colleague was initially using a non pooled connection factory, but the injection info reference has since then been fixed. I have tried all possible combinations of parameters in the shed message, but my outcome is sitll:
Either sent too soon or never sent.
To conclude all the other resources are XA. Namely the oracle db is using the XA driver.
Can anyone confirm if in wildly the send JMS message only when parent transaction commits is working and if so how the session is being configured?
I will check if perhaps my colleague has not made a mistake in terms of the configuration of the connection factory used by the Men's themselves to consume messages out of the queue.but if that one is also XA... Then it is a big problem.
So the issues is fixed.
The commit of the JMS message to the queue at the end of the transaction works perfectly.
The issue was two fold:
(a) First spot of code I was looking at address the issue was not correct. Someone had decided to write his own send telegram to queue API elsewhere, and was not using the central API for sneding telegrams, so any modification I to the injection connection factory was actually not taking effect. The stale connection factories were still being used.
(b) Once the correct API was spotted it was easy to make the mechanism work by using the widlfy XA pooled connection factory mentioned in the post above.
The one thing that was tweaked was the connection.CreationSession api.
The API in JEE 7 has been enlarged and it is now better documented than in jEE 6.
To send a JMS message in a container as part of an XA transaction one should do:
connection.createSession() without any parameters.
This can easily be seen in the connection javadoc:
https://docs.oracle.com/javaee/7/api/javax/jms/Connection.html
QUOTE 1:
This method has been superseded by the method createSession(int
sessionMode) which specifies the same information using a single
argument, and by the method createSession() which is for use in a Java
EE JTA transaction. Applications should consider using those methods
instead of this one.
QUOTE 2:
In a Java EE web or EJB container, when there is an active JTA
transaction in progress:
Both arguments transacted and acknowledgeMode are ignored. The session will participate in the JTA transaction and will be committed
or rolled back when that transaction is committed or rolled back, not
by calling the session's commit or rollback methods. Since both
arguments are ignored, developers are recommended to use
createSession(), which has no arguments, instead of this method.
Which means, the code snippet in:
http://www.mastertheboss.com/jboss-server/jboss-jms/sending-jms-messages-over-xa-with-wildfly-jboss-as
Is not appropriate. What one should be doing is creating the session without any parameter and leting the container handle the rest.
Which it does just fine.
I have a Spring Boot application that is using Spring Integration. The application pulls messages from a RabbitMQ queue, transforms the data from that message, aggregates 50 transformed messages, put those messages in array and sends them to a RESTful endpoint as JSON. I am seeing memory slowly creeps up until the application crashes.
I ran a profiler on our application and there are instances of VariableLinkedBlockingQueue building up over time. The application seems to clean them up after the application starts, but after some time, the application will just build up these instances. I forced a full garbage collection through the profiler on my application and it cleaned up some instances, but they continue to build up. These instances only go up while messages are being sent to the queue. The prefetch is set to 50.
Why am I seeing these instances build up and how do I fix this?
com.rabbitmq.client.impl.WorkPool utilizes that VariableLinkedBlockingQueue and its logic is based on the register/unregister Channel as a client.
If you close Channels properly, they are unregistered from that pool and therefore their VariableLinkedBlockingQueue is garbage collected.
Not having your application to see and play is very difficult to determine where is the leak.
Spring Integration AMQP support exists already for a while and if there was such a problem do not close channel, we'd know that already.
Right now it looks like you use ConnectionFactory somehow out of the box and don't close channels/connections after usage.
I'm using Curator service discovery with Spring MVC rest controllers where each controller registers itself with ZooKeeper in #PostConstruct and de-registers itself in #PreDestroy.
The problem I'm having is that by the time the #PreDestroy method is called, the controller is already no longer servicing requests. I need to de-register the controller before the controller stops servicing requests to avoid throwing exceptions for the small number of requests that occur between the controller stopping and de-registration.
I've tried the spring ApplicationListener interface, SmartLifecycle, and ServletContextListener and in all shutdown/close related hooks, the controller has already stopped servicing requests.
I need a shutdown hook where I can deregister before the controller stops servicing requests, and I'm not sure one is available.
I don't believe such hook exists in Spring.
However, you could have a special controller, mapped to a specific reserved url, i.e. ending in /activity/suspend-traffic. You might want to have some sort of basic security for that, maybe an application specific token, i.e. /activity/suspend-traffic/{token}.
When this special controller receives the correct token, it deregisters all application controllers from ZK.
This way you wouldn't loose any request. Then, when the application is no longer receiving traffic, you could safely shutdown the server. After sending /activity/suspend-traffic/{token}, you could wait a fixed amount of time before shuting down, so that the server finishes processing the requests it has received after the suspend command.
Or you might implement a more sophisticated mechanism, i.e. via a filter or mvc interceptor that counts how many requests are "inside" the application. You could have another mapping in that special controller, i.e. /activity/request-count that returns the actual number of requests being processed. When this request returns 0, it would be safe to shutdown the server.
i saw some code use ShutdownHook like this
Runtime.getRuntime().addShutdownHook(new Thread(){
ConfigurableApplicationContext.stop();
//close spring context;
threadpool.shutdownnow();
//close theadpool
});
is there anything useful to do like this?
i thought
when jvm exit ,maybe thread will be shutdown immediately
and spring context will close tooï¼›
what shall we do next when we need to call System.exit() ?
It really depends on your application and the lifecycle of your objects and those threads you appear to have outside of your context. If you are running the spring container inside a standalone java process, then trapping the shutdown hook like this is one way to do that. Another way is to have it listen on a tcp port and send a command to begin the shutdown process. If you are running in a web container like tomcat, then you should follow the standards on normal webapp shutdown, which Spring supports with Context Listeners.
I would also consider redesigning your app so that the threads are all managed with a bean that lives inside your spring container. For instance using a bean that is configured with directives (attributes) for start/stop methods and then that bean would use an Executor for thread pooling. This way, your shutdown is ONLY shutting down the Spring container, and Spring provides very good support for orderly shutdown of beans. One of those beans is your object holding the threads within the Executor. Its a much cleaner way than trying to integrate Spring beans with external threads.
Hope this helps.