Stop properly SpringBoot jmsApplication - spring

I want to stop properly a jms application,I have the following code, where from the listener i am calling shutdown method:
public void initiateAppShutDown(int returnCode){
SpringApplication.exit(context, ()-> returnCode);
}
public void shudDownApplication(){
LOGGER.debug("Arret de l'application injecteur");
initiateAppShutDown(0);
}
i am performing the shutdown from method onMessageReceived() of the listener , but i have the following warnings , that seems that messages are rejected from the listener:
Failed to shut down 1 bean with phase value 2147483647 within timeout of 30000: [org.springframework.jms.config.internalJmsListenerEndpointRegistry]
And the following :
DefaultMessageListenerContainer: Rejecting received message because of the listener container having been stopped in the meantime:
UPDATE
here is my counter declaration
// we increment the counter. incrementer nombre de messages traités
final AtomicInteger counter = new AtomicInteger(0);
incrementCounter() Method increments counter
public void incrementCounter() {
counter.getAndIncrement();
}
public int get() {
return counter.get();
}
here is the method when the shutdown is called :
private void checkNumberMessagesProcessed() {
if(this.get() == Integer.parseInt(nbrMaxMessages)){
LOGGER.debug("fin de traitement reprise backout avec nombre de messages traites: " + this.get());
//close listeners
//shutdown();
//close application
shudDownApplication();
}
}
I still have messages treated even though i reached nbrMaxMessages=1000 parametrised,and the application restarts automatically :here is the log:
2020-09-18 11:59:34,702 DEBUG --- [Backout-1-2] g.c.s.b.c.GdrBackoutListener : end process backout with number of treated messages : 1000
2020-09-18 11:59:34,704 DEBUG --- [Backout-1-9] g.c.s.b.c.GdrBackoutListener : inject message 49585321304740080b5f1487913fe5270000000000000000 dans INTERNAL_QUEUE with number of messages to trate to 1000
2020-09-18 11:59:34,705 DEBUG --- [Backout-1-5] g.c.s.b.c.GdrBackoutListener : inject message 495853213045f0080b5f08647b4ab6760000000000000000 dans INTERNAL_QUEUE with number of messages to trate to 1000
2020-09-18 11:59:34,705 DEBUG --- [Backout-1-3] g.c.s.b.c.GdrBackoutListener : inject message 49585321304740080b5f116d7b4de1960000000000000000 dans INTERNAL_QUEUE with number of messages to trate to 1000
2020-09-18 11:59:34,705 DEBUG --- [Backout-1-4] g.c.s.b.c.GdrBackoutListener : inject message 49585321304750080b5f136b3193b7690000000000000000 dans INTERNAL_QUEUE with number of messages to trate to 1000
2020-09-18 11:59:34,708 DEBUG --- [pool-7-thread-1] g.c.s.b.c.GdrBackoutListener : Shutdown application injecteur backout
2020-09-18 11:59:34,709 DEBUG --- [Backout-1-4] g.c.s.b.c.GdrBackoutListener : inject message 495853213045f0080b5f14f4120ec5b60000000000000000 dans INTERNAL_QUEUE with number of messages to trate to 1000
2020-09-18 11:59:34,709 DEBUG --- [Backout-1-3] g.c.s.b.c.GdrBackoutListener : inject message 495853213045f0080b5f14003c676edb0000000000000000 dans INTERNAL_QUEUE with number of messages to trate to 1000
2020-09-18 11:59:34,711 DEBUG --- [Backout-1-5] g.c.s.b.c.GdrBackoutListener : inject message 49585321304750080b5f1394ecc8bc480000000000000000 dans INTERNAL_QUEUE with number of messages to trate to 1000
2020-09-18 11:59:34,711 DEBUG --- [Backout-1-2] g.c.s.b.c.GdrBackoutListener : inject message 495853213045f0080b5f15042af6ac7b0000000000000000 dans INTERNAL_QUEUE with number of messages to trate to 1000
2020-09-18 11:59:34,712 DEBUG --- [Backout-1-8] g.c.s.b.c.GdrBackoutListener : inject message 495853213045b0080b5f150c84093cf60000000000000000 dans INTERNAL_QUEUE with number of messages to trate to 1000
2020-09-18 11:59:34,712 DEBUG --- [ackout-1-10] g.c.s.b.c.GdrBackoutListener : inject message 495853213045b0080b5f0887a7225de50000000000000000 dans INTERNAL_QUEUE with number of messages to trate to 1000
2020-09-18 11:59:34,714 DEBUG --- [Backout-1-3] g.c.s.b.c.GdrBackoutListener : inject message 49585321304730080b5f1649ac4348a70000000000000000 dans INTERNAL_QUEUE with number of messages to trate to 1000
2020-09-18 11:59:34,714 DEBUG --- [Backout-1-4] g.c.s.b.c.GdrBackoutListener : inject message 49585321304730080b5f14fffe80e4a10000000000000000 dans INTERNAL_QUEUE with number of messages to trate to 1000
2020-09-18 11:59:34,716 DEBUG --- [Backout-1-5] g.c.s.b.c.GdrBackoutListener : inject message 49585321304740080b5f146dee9feec60000000000000000 dans INTERNAL_QUEUE with number of messages to trate to 1000
2020-09-18 11:59:34,716 DEBUG --- [Backout-1-2] g.c.s.b.c.GdrBackoutListener : inject message 495853213045b0080b5f158b80eac21f0000000000000000 dans INTERNAL_QUEUE with number of messages to trate to 1000
2020-09-18 11:59:34,718 INFO --- [pool-7-thread-1] s.c.a.AnnotationConfigApplicationContext : Closing org.springframework.context.annotation.AnnotationConfigApplicationContext#53dbe163: startup date [Fri Sep 18 11:59:32 CEST 2020]; parent: org.springframework.context.annotation.AnnotationConfigApplicationContext#7a5ceedd
2020-09-18 11:59:34,719 DEBUG --- [Backout-1-3] g.c.s.b.c.GdrBackoutListener : inject message 495853213045b0080b5f08b16224e18f0000000000000000 dans INTERNAL_QUEUE with number of messages to trate to 1000
2020-09-18 11:59:34,720 DEBUG --- [Backout-1-4] g.c.s.b.c.GdrBackoutListener : inject message 495853213047e0080b5f16b62be157e60000000000000000 dans INTERNAL_QUEUE with number of messages to trate to 1000
2020-09-18 11:59:34,721 DEBUG --- [Backout-1-5] g.c.s.b.c.GdrBackoutListener : inject message 49585321304730080b5f15dd2ce0d2df0000000000000000 dans INTERNAL_QUEUE with number of messages to trate to 1000
2020-09-18 11:59:34,721 DEBUG --- [Backout-1-2] g.c.s.b.c.GdrBackoutListener : inject message 495853213047e0080b5f084d3a3b00260000000000000000 dans INTERNAL_QUEUE with number of messages to trate to 1000
2020-09-18 11:59:34,723 INFO --- [pool-7-thread-1] o.s.c.s.DefaultLifecycleProcessor : Stopping beans in phase 2147483647

You're triggering a shutdown call in the same thread and it's like dead lock, your onMessage method is waiting for shutdown while JMS container is waiting for running task to be terminated and it gets eventually timedout.
To solve this issue, you can use a async executor that would trigger the shutdown, something like this.
#Component
public class MessageListener {
ScheduledExecutorService executorService = Executors.newSingleThreadScheduledExecutor();
#JmsListener(...)
public void onShutdownMessage(Message message){
// increase delay further if observer any issue, it will trigger shutdown call in 1 seconds
executorService.schedule(() -> {shudDownApplication();}, 1, TimeUnit.SECONDS);
}
}

finally i solved the problem.
i had a cyclic dependency on my listener.
i tried to tell the listener to stop itself after waiting for all threads to finish their work.
that was not correct as an approach .so i created an error handler ,then i throw errors from the JmsListener . and the shutdown is managed throw a shutDownManager class.
in the shutDownManager i created a new thread where to stop the springApplication.
this approach works fine for a thread concurrency context. here is the code :
#Component
public class AppContextManager implements ApplicationContextAware {
private static ApplicationContext _appCtx;
#Override
public void setApplicationContext(ApplicationContext ctx){
_appCtx = ctx;
}
public static ApplicationContext getAppContext(){
return _appCtx;
}
public static void exit(Integer exitCode) {
System.exit(SpringApplication.exit(_appCtx,() -> exitCode));
}
}
and in error handler ,when u want to shutdown your application you catch exception thrown fro Jmslistener ,you call this method
private void shutDown(){
//JmsListenerEndpointRegistry bean = context.getBean(JmsListenerEndpointRegistry.class);
ExecutorService executorService = Executors.newSingleThreadExecutor();
executorService.execute(() -> {
//bean.stop();
appContext.exit(0);
});
}
this allows you to shutDown silently your Listeners and the custom tread waits all the threads to finish execution

Related

How to implement Retry logic in RabbitMQ?

In my project I'm setting SimpleRetryPolicy to add custom exception and RetryOperationsInterceptor which is consuming this policy.
#Bean
public SimpleRetryPolicy rejectionRetryPolicy() {
Map<Class<? extends Throwable>, Boolean> exceptionsMap = new HashMap<Class<? extends Throwable>, Boolean>();
exceptionsMap.put(DoNotRetryException.class, false);//not retriable
exceptionsMap.put(RetryException.class, true); //retriable
return new SimpleRetryPolicy(3, exceptionsMap, true);
}
#Bean
RetryOperationsInterceptor interceptor() {
return RetryInterceptorBuilder.stateless()
.retryPolicy(rejectionRetryPolicy())
.backOffOptions(2000L, 2, 3000L)
.recoverer(
new RepublishMessageRecoverer(rabbitTemplate(), "dlExchange", "dlRoutingKey"))
.build();
}
But with these configurations retry is not working for both RetryException and DoNotRetryException, where I want RetryException to be retried finite number of time and DoNotRetryException to send to DLQ
Please help with the issue, I'm attaching repo link if in case of need.
https://github.com/aviralnimbekar/RabbitMQ/tree/main/src
Your GlobalErrorHandler does its logic before the retry happens and you override there an exception with the AmqpRejectAndDontRequeueException. And looks like you do there a publish to DLX. Consider to move your GlobalErrorHandler logic to a more general ErrorHandler for the factory.setErrorHandler(); instead.
See more info in docs: https://docs.spring.io/spring-amqp/reference/html/#exception-handling
UPDATE
After removing errorHandler = "globalErrorHandler" from your #RabbitListener, I got this in logs:
2022-08-03 16:02:08.093 INFO 16896 --- [nio-8080-exec-4] c.t.r.producer.RabbitMQProducer : Message sent -> retry
2022-08-03 16:02:08.095 INFO 16896 --- [ntContainer#0-1] c.t.r.consumer.RabbitMQConsumer : Retrying message...
2022-08-03 16:02:10.096 INFO 16896 --- [ntContainer#0-1] c.t.r.consumer.RabbitMQConsumer : Retrying message...
2022-08-03 16:02:13.099 INFO 16896 --- [ntContainer#0-1] c.t.r.consumer.RabbitMQConsumer : Retrying message...
2022-08-03 16:02:13.100 WARN 16896 --- [ntContainer#0-1] o.s.a.r.retry.RepublishMessageRecoverer : Republishing failed message to exchange 'dlExchange' with routing key dlRoutingKey
2022-08-03 16:02:17.736 INFO 16896 --- [nio-8080-exec-5] c.t.r.producer.RabbitMQProducer : Message sent -> 1231231
2022-08-03 16:02:17.738 INFO 16896 --- [ntContainer#0-1] c.t.r.consumer.RabbitMQConsumer : sending into dlq...
2022-08-03 16:02:17.739 WARN 16896 --- [ntContainer#0-1] o.s.a.r.retry.RepublishMessageRecoverer : Republishing failed message to exchange 'dlExchange' with routing key dlRoutingKey
Which definitely reflects your original requirements.

Spring boot pub/sub subcriber gets removed immediately after deployment

I have a spring boot app deployed to google cloud (via automatic github build).
Unfortunately my subscriber to pub/sub is being removed immediately after deployment :( Does anyone know reason for that?
Removing {service-activator:investobotAPIs.messageReceiver.serviceActivator} as a subscriber to the 'inputMessageChannel' channel
I've created a topic in google pub/sub and a subscription:
I've added these methods:
#RestController
#EnableAsync
public class InvestobotAPIs {
...
// [START pubsub_spring_inbound_channel_adapter]
// Create a message channel for messages arriving from the subscription `sub-one`.
#Bean
public MessageChannel inputMessageChannel() {
return new PublishSubscribeChannel();
}
// Create an inbound channel adapter to listen to the subscription `sub-one` and send
// messages to the input message channel.
#Bean
public PubSubInboundChannelAdapter inboundChannelAdapter(
#Qualifier("inputMessageChannel") MessageChannel messageChannel,
PubSubTemplate pubSubTemplate) {
PubSubInboundChannelAdapter adapter =
new PubSubInboundChannelAdapter(pubSubTemplate, "fetch-gpw-sub");
adapter.setOutputChannel(messageChannel);
adapter.setAckMode(AckMode.MANUAL);
adapter.setPayloadType(String.class);
return adapter;
}
// Define what happens to the messages arriving in the message channel.
#ServiceActivator(inputChannel = "inputMessageChannel")
public void messageReceiver(
String payload,
#Header(GcpPubSubHeaders.ORIGINAL_MESSAGE) BasicAcknowledgeablePubsubMessage message) {
logger.info("Message arrived via an inbound channel adapter from fetch-gpw! Payload: " + payload);
message.ack();
}
// [END pubsub_spring_inbound_channel_adapter]
and this is my build.graddle:
dependencies {
implementation 'org.springframework.boot:spring-boot-starter-web:2.5.5'
// get url and download files
implementation 'commons-io:commons-io:2.6'
implementation 'org.asynchttpclient:async-http-client:2.12.3'
// parse xls files
implementation 'org.apache.poi:poi:5.0.0'
implementation 'org.apache.poi:poi-ooxml:5.0.0'
// firebase messaging and db
implementation 'com.google.firebase:firebase-admin:8.1.0'
// subscribe pub/sub topic
implementation 'com.google.cloud:spring-cloud-gcp-starter-pubsub:2.0.4'
implementation 'com.google.cloud:spring-cloud-gcp-pubsub-stream-binder:2.0.4'
implementation 'com.google.cloud:spring-cloud-gcp-dependencies:2.0.4'
}
Unfortunately when it is deployed the logs show that just after deployment is gets removed:
2021-10-20 11:51:16.190 CEST2021-10-20 09:51:16.190 INFO 1 --- [ main] o.s.i.channel.PublishSubscribeChannel : Channel 'application.inputMessageChannel' has 1 subscriber(s).
Default
2021-10-20 11:51:16.190 CEST2021-10-20 09:51:16.190 INFO 1 --- [ main] o.s.i.endpoint.EventDrivenConsumer : started bean 'investobotAPIs.messageReceiver.serviceActivator'
Default
2021-10-20 11:51:16.214 CEST2021-10-20 09:51:16.214 INFO 1 --- [ main] .g.c.s.p.i.i.PubSubInboundChannelAdapter : started bean 'inboundChannelAdapter'; defined in: 'class path resource [com/miloszdobrowolski/investobotbackend/InvestobotAPIs.class]'; from source: 'com.miloszdobrowolski.investobotbackend.InvestobotAPIs.inboundChannelAdapter(org.springframework.messaging.MessageChannel,com.google.cloud.spring.pubsub.core.PubSubTemplate)'
Default
2021-10-20 11:51:16.295 CEST2021-10-20 09:51:16.294 INFO 1 --- [ main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat started on port(s): 8080 (http) with context path ''
Default
2021-10-20 11:51:19.076 CEST2021-10-20 09:51:19.075 INFO 1 --- [ main] c.m.i.InvestobotBackendApplication : Started InvestobotBackendApplication in 14.475 seconds (JVM running for 19.248)
Info
2021-10-20 11:51:22.386 CESTCloud Runinvestobot-backend-autobuild {#type: type.googleapis.com/google.cloud.audit.AuditLog, resourceName: namespaces/our-shield-329019/services/investobot-backend-autobuild, response: {…}, serviceName: run.googleapis.com, status: {…}}
Debug
2021-10-20 11:51:54.075 CESTContainer Sandbox: Unsupported syscall setsockopt(0x16,0x29,0x31,0x3e96fd1f3a9c,0x4,0x0). It is very likely that you can safely ignore this message and that this is not the cause of any error you might be troubleshooting. Please, refer to https://gvisor.dev/c/linux/amd64/setsockopt for more information.
Debug
2021-10-20 11:51:54.075 CESTContainer Sandbox: Unsupported syscall setsockopt(0x16,0x29,0x12,0x3e96fd1f3a9c,0x4,0x0). It is very likely that you can safely ignore this message and that this is not the cause of any error you might be troubleshooting. Please, refer to https://gvisor.dev/c/linux/amd64/setsockopt for more information.
Default
2021-10-20 11:52:59.196 CEST2021-10-20 09:52:59.195 WARN 1 --- [ault-executor-0] i.g.n.s.i.n.u.internal.MacAddressUtil : Failed to find a usable hardware address from the network interfaces; using random bytes: d1:79:21:a2:71:29:47:49
Default
2021-10-20 11:53:01.010 CEST2021-10-20 09:53:01.010 INFO 1 --- [ionShutdownHook] .g.c.s.p.i.i.PubSubInboundChannelAdapter : stopped bean 'inboundChannelAdapter'; defined in: 'class path resource [com/miloszdobrowolski/investobotbackend/InvestobotAPIs.class]'; from source: 'com.miloszdobrowolski.investobotbackend.InvestobotAPIs.inboundChannelAdapter(org.springframework.messaging.MessageChannel,com.google.cloud.spring.pubsub.core.PubSubTemplate)'
Default
2021-10-20 11:53:01.010 CEST2021-10-20 09:53:01.010 INFO 1 --- [ionShutdownHook] o.s.i.endpoint.EventDrivenConsumer : Removing {logging-channel-adapter:_org.springframework.integration.errorLogger} as a subscriber to the 'errorChannel' channel
Default
2021-10-20 11:53:01.010 CEST2021-10-20 09:53:01.010 INFO 1 --- [ionShutdownHook] o.s.i.channel.PublishSubscribeChannel : Channel 'application.errorChannel' has 0 subscriber(s).
Default
2021-10-20 11:53:01.011 CEST2021-10-20 09:53:01.011 INFO 1 --- [ionShutdownHook] o.s.i.endpoint.EventDrivenConsumer : stopped bean '_org.springframework.integration.errorLogger'
Default
2021-10-20 11:53:01.011 CEST2021-10-20 09:53:01.011 INFO 1 --- [ionShutdownHook] o.s.i.endpoint.EventDrivenConsumer : Removing {service-activator:investobotAPIs.messageReceiver.serviceActivator} as a subscriber to the 'inputMessageChannel' channel
Default
2021-10-20 11:53:01.011 CEST2021-10-20 09:53:01.011 INFO 1 --- [ionShutdownHook] o.s.i.channel.PublishSubscribeChannel : Channel 'application.inputMessageChannel' has 0 subscriber(s).
Default
2021-10-20 11:53:01.011 CEST2021-10-20 09:53:01.011 INFO 1 --- [ionShutdownHook] o.s.i.endpoint.EventDrivenConsumer : stopped bean 'investobotAPIs.messageReceiver.serviceActivator'
I've tried to change dependencies in gradle from com.google.cloud to org.springframework.cloud (not sure how they differ)
dependencies {
implementation 'org.springframework.boot:spring-boot-starter-web:2.5.5'
// get url and download files
implementation 'commons-io:commons-io:2.6'
implementation 'org.asynchttpclient:async-http-client:2.12.3'
// parse xls files
implementation 'org.apache.poi:poi:5.0.0'
implementation 'org.apache.poi:poi-ooxml:5.0.0'
// firebase messaging and db
implementation 'com.google.firebase:firebase-admin:8.1.0'
// subscribe pub/sub topic
// implementation 'com.google.cloud:spring-cloud-gcp-starter-pubsub:2.0.4'
// implementation 'com.google.cloud:spring-cloud-gcp-pubsub-stream-binder:2.0.4'
// implementation 'com.google.cloud:spring-cloud-gcp-dependencies:2.0.4'
implementation 'org.springframework.cloud:spring-cloud-gcp-starter-bus-pubsub:1.2.8.RELEASE'
implementation 'org.springframework.cloud:spring-cloud-gcp-pubsub:1.2.8.RELEASE'
implementation 'org.springframework.cloud:spring-cloud-gcp-autoconfigure:1.2.8.RELEASE'
implementation 'org.springframework.integration:spring-integration-core:5.5.4'
}
But it only causes a different error during deployment:
ERROR: (gcloud.run.services.update) Cloud Run error: Container failed to start. Failed to start and then listen on the port defined by the PORT environment variable. Logs for this revision might contain more information.

KafkaProducer InterruptedException during gracefull shutdown on spring boot application

For a project we are sending some events to kafka. We use spring-kafka 2.6.2.
Due to usage of spring-vault we have to restart/kill the application before the end of credentials lease (application is automatically restarted by kubernetes).
Our problem is that when using applicationContext.close() to proceed with our gracefull shutdown, KafkaProducer gets an InterruptedException Interrupted while joining ioThread inside it's close() method.
It means that in our case some pending events are not sent to kafka before shutdown as it's forced to close due to an error during destroy.
Here under a stacktrace
2020-12-18 13:57:29.007 INFO [titan-producer,222efdd2a07966ce,222efdd2a07966ce,true] 1 --- [ scheduling-1] o.s.b.w.e.tomcat.GracefulShutdown : Commencing graceful shutdown. Waiting for active requests to complete
2020-12-18 13:57:29.009 INFO [titan-producer,222efdd2a07966ce,222efdd2a07966ce,true] 1 --- [ scheduling-1] o.apache.catalina.core.StandardService : Stopping service [Tomcat]
2020-12-18 13:57:29.013 INFO [titan-producer,222efdd2a07966ce,222efdd2a07966ce,true] 1 --- [ scheduling-1] o.a.c.c.C.[Tomcat].[localhost].[/] : Destroying Spring FrameworkServlet 'dispatcherServlet'
2020-12-18 13:57:29.014 INFO [titan-producer,,,] 1 --- [tomcat-shutdown] o.s.b.w.e.tomcat.GracefulShutdown : Graceful shutdown complete
2020-12-18 13:57:29.020 WARN [titan-producer,222efdd2a07966ce,222efdd2a07966ce,true] 1 --- [ scheduling-1] o.a.c.loader.WebappClassLoaderBase : The web application [ROOT] appears to have started a thread named [kafka-producer-network-thread | titan-producer-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.base#11.0.9.1/sun.nio.ch.EPoll.wait(Native Method)
java.base#11.0.9.1/sun.nio.ch.EPollSelectorImpl.doSelect(Unknown Source)
java.base#11.0.9.1/sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)
java.base#11.0.9.1/sun.nio.ch.SelectorImpl.select(Unknown Source)
org.apache.kafka.common.network.Selector.select(Selector.java:873)
org.apache.kafka.common.network.Selector.poll(Selector.java:469)
org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:544)
org.apache.kafka.clients.producer.internals.Sender.runOnce(Sender.java:325)
org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:240)
java.base#11.0.9.1/java.lang.Thread.run(Unknown Source)
2020-12-18 13:57:29.021 WARN [titan-producer,222efdd2a07966ce,222efdd2a07966ce,true] 1 --- [ scheduling-1] o.a.c.loader.WebappClassLoaderBase : The web application [ROOT] appears to have started a thread named [micrometer-kafka-metrics] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.base#11.0.9.1/jdk.internal.misc.Unsafe.park(Native Method)
java.base#11.0.9.1/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
java.base#11.0.9.1/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)
java.base#11.0.9.1/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
java.base#11.0.9.1/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
java.base#11.0.9.1/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
java.base#11.0.9.1/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
java.base#11.0.9.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
java.base#11.0.9.1/java.lang.Thread.run(Unknown Source)
2020-12-18 13:57:29.046 INFO [titan-producer,222efdd2a07966ce,222efdd2a07966ce,true] 1 --- [ scheduling-1] o.s.s.concurrent.ThreadPoolTaskExecutor : Shutting down ExecutorService 'applicationTaskExecutor'
2020-12-18 13:57:29.048 INFO [titan-producer,222efdd2a07966ce,222efdd2a07966ce,true] 1 --- [ scheduling-1] o.s.s.c.ThreadPoolTaskScheduler : Shutting down ExecutorService 'taskScheduler'
2020-12-18 13:57:29.051 INFO [titan-producer,222efdd2a07966ce,222efdd2a07966ce,true] 1 --- [ scheduling-1] o.a.k.clients.producer.KafkaProducer : [Producer clientId=titan-producer-1] Closing the Kafka producer with timeoutMillis = 30000 ms.
2020-12-18 13:57:29.055 ERROR [titan-producer,222efdd2a07966ce,222efdd2a07966ce,true] 1 --- [ scheduling-1] o.a.k.clients.producer.KafkaProducer : [Producer clientId=titan-producer-1] Interrupted while joining ioThreadjava.lang.InterruptedException: null
at java.base/java.lang.Object.wait(Native Method)
at java.base/java.lang.Thread.join(Unknown Source)
at org.apache.kafka.clients.producer.KafkaProducer.close(KafkaProducer.java:1205)
at org.apache.kafka.clients.producer.KafkaProducer.close(KafkaProducer.java:1182)
at org.springframework.kafka.core.DefaultKafkaProducerFactory$CloseSafeProducer.closeDelegate(DefaultKafkaProducerFactory.java:901)
at org.springframework.kafka.core.DefaultKafkaProducerFactory.destroy(DefaultKafkaProducerFactory.java:428)
at org.springframework.beans.factory.support.DisposableBeanAdapter.destroy(DisposableBeanAdapter.java:258)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroyBean(DefaultSingletonBeanRegistry.java:587)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingleton(DefaultSingletonBeanRegistry.java:559)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.destroySingleton(DefaultListableBeanFactory.java:1092)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingletons(DefaultSingletonBeanRegistry.java:520)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.destroySingletons(DefaultListableBeanFactory.java:1085)
at org.springframework.context.support.AbstractApplicationContext.destroyBeans(AbstractApplicationContext.java:1061)
at org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:1030)
at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.doClose(ServletWebServerApplicationContext.java:170)
at org.springframework.context.support.AbstractApplicationContext.close(AbstractApplicationContext.java:979)
at org.springframework.cloud.sleuth.instrument.async.TraceRunnable.run(TraceRunnable.java:68)
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)2020-12-18 13:57:29.055 INFO [titan-producer,222efdd2a07966ce,222efdd2a07966ce,true] 1 --- [ scheduling-1] o.a.k.clients.producer.KafkaProducer : [Producer clientId=titan-producer-1] Proceeding to force close the producer since pending requests could not be completed within timeout 30000 ms.
2020-12-18 13:57:29.056 WARN [titan-producer,222efdd2a07966ce,222efdd2a07966ce,true] 1 --- [ scheduling-1] o.s.b.f.support.DisposableBeanAdapter : Invocation of destroy method failed on bean with name 'kafkaProducerFactory': org.apache.kafka.common.errors.InterruptException: java.lang.InterruptedException
2020-12-18 13:57:29.064 INFO [titan-producer,222efdd2a07966ce,222efdd2a07966ce,true] 1 --- [ scheduling-1] o.s.s.concurrent.ThreadPoolTaskExecutor : Shutting down ExecutorService
2020-12-18 13:57:29.065 INFO [titan-producer,222efdd2a07966ce,222efdd2a07966ce,true] 1 --- [ scheduling-1] c.l.t.p.zookeeper.ZookeeperManagerImpl : Closing zookeeperConnection
2020-12-18 13:57:29.197 INFO [titan-producer,222efdd2a07966ce,222efdd2a07966ce,true] 1 --- [ scheduling-1] org.apache.zookeeper.ZooKeeper : Session: 0x30022348ba6000b closed
2020-12-18 13:57:29.197 INFO [titan-producer,,,] 1 --- [d-1-EventThread] org.apache.zookeeper.ClientCnxn : EventThread shut down for session: 0x30022348ba6000b
2020-12-18 13:57:29.206 INFO [titan-producer,222efdd2a07966ce,222efdd2a07966ce,true] 1 --- [ scheduling-1] com.zaxxer.hikari.HikariDataSource : loadtest_fallback_titan_pendingEvents - Shutdown initiated...
2020-12-18 13:57:29.221 INFO [titan-producer,222efdd2a07966ce,222efdd2a07966ce,true] 1 --- [ scheduling-1] com.zaxxer.hikari.HikariDataSource : loadtest_fallback_titan_pendingEvents - Shutdown completed.
Here is my configuration class
#Flogger
#EnableKafka
#Configuration
#RequiredArgsConstructor
#ConditionalOnProperty(
name = "titan.producer.kafka.enabled",
havingValue = "true",
matchIfMissing = true)
public class KafkaConfiguration {
#Bean
DefaultKafkaProducerFactoryCustomizer kafkaProducerFactoryCustomizer(ObjectMapper mapper) {
return producerFactory -> producerFactory.setValueSerializer(new JsonSerializer<>(mapper));
}
#Bean
public NewTopic createTopic(TitanProperties titanProperties, KafkaProperties kafkaProperties) {
TitanProperties.Kafka kafka = titanProperties.getKafka();
String defaultTopic = kafkaProperties.getTemplate().getDefaultTopic();
int numPartitions = kafka.getNumPartitions();
short replicationFactor = kafka.getReplicationFactor();
log.atInfo()
.log("Creating Kafka Topic %s with %s partitions and %s replicationFactor", defaultTopic, numPartitions, replicationFactor);
return TopicBuilder.name(defaultTopic)
.partitions(numPartitions)
.replicas(replicationFactor)
.config(MESSAGE_TIMESTAMP_TYPE_CONFIG, LOG_APPEND_TIME.name)
.build();
}
}
and my application.yaml
spring:
application:
name: titan-producer
kafka:
client-id: ${spring.application.name}
producer:
key-serializer: org.apache.kafka.common.serialization.UUIDSerializer
value-serializer: org.springframework.kafka.support.serializer.JsonSerializer
properties:
max.block.ms: 2000
request.timeout.ms: 2000
delivery.timeout.ms: 2000 #must be greater or equal to request.timeout.ms + linger.ms
template:
default-topic: titan-dev
Our vault configuration which executes the applicationContext.close() using a scheduledTask. We do it kind randomly as we have multiple replicas of the app running in parallel and avoid all the replicas to be killed at the same time.
#Flogger
#Configuration
#ConditionalOnBean(SecretLeaseContainer.class)
#ConditionalOnProperty(
name = "titan.producer.scheduling.enabled",
havingValue = "true",
matchIfMissing = true)
public class VaultConfiguration {
#Bean
public Lifecycle scheduledAppRestart(Clock clock, TitanProperties properties, TaskScheduler scheduler, ConfigurableApplicationContext applicationContext) {
Instant now = clock.instant();
Duration maxTTL = properties.getVaultConfig().getCredsMaxLease();
Instant start = now.plusSeconds(maxTTL.dividedBy(2).toSeconds());
Instant end = now.plusSeconds(maxTTL.minus(properties.getVaultConfig().getCredsMaxLeaseExpirationThreshold()).toSeconds());
Instant randomInstant = randBetween(start, end);
return new ScheduledLifecycle(scheduler, applicationContext::close, "application restart before lease expiration", randomInstant);
}
private Instant randBetween(Instant startInclusive, Instant endExclusive) {
long startSeconds = startInclusive.getEpochSecond();
long endSeconds = endExclusive.getEpochSecond();
long random = RandomUtils.nextLong(startSeconds, endSeconds);
return Instant.ofEpochSecond(random);
}
}
The ScheduledLifecycle class we use to run the scheduledtasks
import lombok.extern.flogger.Flogger;
import org.springframework.context.SmartLifecycle;
import org.springframework.scheduling.TaskScheduler;
import java.time.Duration;
import java.time.Instant;
import java.util.concurrent.ScheduledFuture;
#Flogger
public class ScheduledLifecycle implements SmartLifecycle {
private ScheduledFuture<?> future = null;
private Duration delay = null;
private final TaskScheduler scheduler;
private final Runnable command;
private final String commandDesc;
private final Instant startTime;
public ScheduledLifecycle(TaskScheduler scheduler, Runnable command, String commandDesc, Instant startTime) {
this.scheduler = scheduler;
this.command = command;
this.commandDesc = commandDesc;
this.startTime = startTime;
}
public ScheduledLifecycle(TaskScheduler scheduler, Runnable command, String commandDesc, Instant startTime, Duration delay) {
this(scheduler, command, commandDesc, startTime);
this.delay = delay;
}
#Override
public void start() {
if (delay != null) {
log.atInfo().log("Scheduling %s: starting at %s, running every %s", commandDesc, startTime, delay);
future = scheduler.scheduleWithFixedDelay(command, startTime, delay);
} else {
log.atInfo().log("Scheduling %s: execution at %s", commandDesc, startTime);
future = scheduler.schedule(command, startTime);
}
}
#Override
public void stop() {
if (future != null) {
log.atInfo().log("Stop %s", commandDesc);
future.cancel(true);
}
}
#Override
public boolean isRunning() {
boolean running = future != null && (!future.isDone() && !future.isCancelled());
log.atFine().log("is %s running? %s", running);
return running;
}
}
Is there a bug with spring-kafka? Any idea?
Thanks
future.cancel(true);
This is interrupting the producer thread and is likely the root cause of the problem.
You should use future.cancel(false); to allow the task to terminate in an orderly fashion, without interruption.
/**
* Attempts to cancel execution of this task. This attempt will
* fail if the task has already completed, has already been cancelled,
* or could not be cancelled for some other reason. If successful,
* and this task has not started when {#code cancel} is called,
* this task should never run. If the task has already started,
* then the {#code mayInterruptIfRunning} parameter determines
* whether the thread executing this task should be interrupted in
* an attempt to stop the task.
*
* <p>After this method returns, subsequent calls to {#link #isDone} will
* always return {#code true}. Subsequent calls to {#link #isCancelled}
* will always return {#code true} if this method returned {#code true}.
*
* #param mayInterruptIfRunning {#code true} if the thread executing this
* task should be interrupted; otherwise, in-progress tasks are allowed
* to complete
* #return {#code false} if the task could not be cancelled,
* typically because it has already completed normally;
* {#code true} otherwise
*/
boolean cancel(boolean mayInterruptIfRunning);
EDIT
In addition, the ThreadPoolTaskScheduler.waitForTasksToCompleteOnShutdown is false by default.
/**
* Set whether to wait for scheduled tasks to complete on shutdown,
* not interrupting running tasks and executing all tasks in the queue.
* <p>Default is "false", shutting down immediately through interrupting
* ongoing tasks and clearing the queue. Switch this flag to "true" if you
* prefer fully completed tasks at the expense of a longer shutdown phase.
* <p>Note that Spring's container shutdown continues while ongoing tasks
* are being completed. If you want this executor to block and wait for the
* termination of tasks before the rest of the container continues to shut
* down - e.g. in order to keep up other resources that your tasks may need -,
* set the {#link #setAwaitTerminationSeconds "awaitTerminationSeconds"}
* property instead of or in addition to this property.
* #see java.util.concurrent.ExecutorService#shutdown()
* #see java.util.concurrent.ExecutorService#shutdownNow()
*/
public void setWaitForTasksToCompleteOnShutdown(boolean waitForJobsToCompleteOnShutdown) {
this.waitForTasksToCompleteOnShutdown = waitForJobsToCompleteOnShutdown;
}
You might also have to set awaitTerminationSeconds.

Is it possible to enforce message order on ActiveMQ topics using Spring Boot and JmsTemplate?

In playing around with Spring Boot, ActiveMQ, and JmsTemplate, I noticed that it appears that message order is not always preserved. In reading on ActiveMQ, "Message Groups" are offered as a potential solution to preserving message order when sending to a topic. Is there a way to do this with JmsTemplate?
Add Note: I'm starting to think that JmsTemplate is nice for "getting launched", but has too many issues.
Sample code and console output posted below...
#RestController
public class EmptyControllerSB {
#Autowired
MsgSender msgSender;
#RequestMapping(method = RequestMethod.GET, value = { "/v1/msgqueue" })
public String getAccount() {
msgSender.sendJmsMessageA();
msgSender.sendJmsMessageB();
return "Do nothing...successfully!";
}
}
#Component
public class MsgSender {
#Autowired
JmsTemplate jmsTemplate;
void sendJmsMessageA() {
jmsTemplate.convertAndSend(new ActiveMQTopic("VirtualTopic.TEST-TOPIC"), "message A");
}
void sendJmsMessageB() {
jmsTemplate.convertAndSend(new ActiveMQTopic("VirtualTopic.TEST-TOPIC"), "message B");
}
}
#Component
public class MsgReceiver {
private final String consumerOne = "Consumer.myConsumer1.VirtualTopic.TEST-TOPIC";
private final String consumerTwo = "Consumer.myConsumer2.VirtualTopic.TEST-TOPIC";
#JmsListener(destination = consumerOne )
public void receiveMessage1(String strMessage) {
System.out.println("Received on #1a -> " + strMessage);
}
#JmsListener(destination = consumerOne )
public void receiveMessage2(String strMessage) {
System.out.println("Received on #1b -> " + strMessage);
}
#JmsListener(destination = consumerTwo )
public void receiveMessage3(String strMessage) {
System.out.println("Received on #2 -> " + strMessage);
}
}
Here's the console output (note the order of output in first sequence)...
\Intel\Intel(R) Management Engine Components\DAL;C:\WINDOWS\System32\OpenSSH\;C:\Program Files (x86)\NVIDIA Corporation\PhysX\Common;C:\Program Files (x86)\gnupg\bin;C:\Users\LesR\AppData\Local\Microsoft\WindowsApps;c:\Gradle\gradle-5.0\bin;;C:\Program Files\JetBrains\IntelliJ IDEA 2018.3\bin;;.]
2019-04-03 09:23:08.408 INFO 13936 --- [ main] o.a.c.c.C.[Tomcat].[localhost].[/] : Initializing Spring embedded WebApplicationContext
2019-04-03 09:23:08.408 INFO 13936 --- [ main] o.s.web.context.ContextLoader : Root WebApplicationContext: initialization completed in 672 ms
2019-04-03 09:23:08.705 INFO 13936 --- [ main] o.s.s.concurrent.ThreadPoolTaskExecutor : Initializing ExecutorService 'applicationTaskExecutor'
2019-04-03 09:23:08.845 INFO 13936 --- [ main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat started on port(s): 8080 (http) with context path ''
2019-04-03 09:23:08.877 INFO 13936 --- [ main] mil.navy.msgqueue.MsgqueueApplication : Started MsgqueueApplication in 1.391 seconds (JVM running for 1.857)
2019-04-03 09:23:14.949 INFO 13936 --- [nio-8080-exec-1] o.a.c.c.C.[Tomcat].[localhost].[/] : Initializing Spring DispatcherServlet 'dispatcherServlet'
2019-04-03 09:23:14.949 INFO 13936 --- [nio-8080-exec-1] o.s.web.servlet.DispatcherServlet : Initializing Servlet 'dispatcherServlet'
2019-04-03 09:23:14.952 INFO 13936 --- [nio-8080-exec-1] o.s.web.servlet.DispatcherServlet : Completed initialization in 3 ms
Received on #2 -> message A
Received on #1a -> message B
Received on #1b -> message A
Received on #2 -> message B
<HIT DO-NOTHING ENDPOINT AGAIN>
Received on #1b -> message A
Received on #2 -> message A
Received on #1a -> message B
Received on #2 -> message B
BLUF - Add "?consumer.exclusive=true" to the declaration of the destination for the JmsListener annotation.
It seems that the solution is not that complex, especially if one abandons ActiveMQ's "message groups" in favor or "exclusive consumers". The drawback to the "message groups" is that the sender has to have prior knowledge of the potential partitioning of message consumers. If the producer has this knowledge, then "message groups" are a nice solution, as the solution is somewhat independent of the consumer.
But, a similar solution can be implemented from the consumer side, by having the consumer declare "exclusive consumer" on the queue. While I did not see anything in the JmsTemplate implementation that directly supports this, it seems that Spring's JmsTemplate implementation passes the queue name to ActiveMQ, and then ActiveMQ "does the right thing" and enforces the exclusive consumer behavior.
So...
Change the following...
private final String consumerOne = "Consumer.myConsumer1.VirtualTopic.TEST-TOPIC";
to...
private final String consumerOne = "Consumer.myConsumer1.VirtualTopic.TEST-TOPIC";?consumer.exclusive=true
Once I did this, only one of the two declared receive methods were invoked, and message order was maintained in all my test runs.

Not able to shutdown the jms listener which posts message to kafka spring boot application with Runtime.exit, context.close, System.exit()

I am developing a spring boot application which will listen to ibm mq with
#JmsListener(id="abc", destination="${queueName}", containerFactory="defaultJmsListenerContainerFactory")
I have a JmsListenerEndpointRegistry which starts the listenerContainer.
On message will try to push the same message with some business logic to kafka. The poster code is
kafkaTemplate.send(kafkaProp.getTopic(), uniqueId, message)
Now in case a kafka producer fails, I want my boot application to get terminated. So I have added a custom
setErrorHandler.
So I have tried
`System.exit(1)`, `configurableApplicationContextObject.close()`, `Runtime.getRuntime.exit(1)`.
But none of them work. Below is the log that gets generated after
System.exit(0) or above others.
2018-05-24 12:12:47.981 INFO 18904 --- [ Thread-4] s.c.a.AnnotationConfigApplicationContext : Closing org.springframework.context.annotation.AnnotationConfigApplicationContext#1d08376: startup date [Thu May 24 12:10:35 IST 2018]; root of context hierarchy
2018-05-24 12:12:48.027 INFO 18904 --- [ Thread-4] o.s.c.support.DefaultLifecycleProcessor : Stopping beans in phase 2147483647
2018-05-24 12:12:48.028 INFO 18904 --- [ Thread-4] o.s.c.support.DefaultLifecycleProcessor : Stopping beans in phase 0
2018-05-24 12:12:48.028 INFO 18904 --- [ Thread-4] o.s.j.e.a.AnnotationMBeanExporter : Unregistering JMX-exposed beans on shutdown
2018-05-24 12:12:48.028 INFO 18904 --- [ Thread-4] o.a.k.clients.producer.KafkaProducer : Closing the Kafka producer with timeoutMillis = 9223372036854775807 ms.
2018-05-24 12:12:48.044 INFO 18904 --- [ Thread-4] o.a.k.clients.producer.KafkaProducer : Closing the Kafka producer with timeoutMillis = 30000 ms.
But the application is still running and below are the running threads
Daemon Thread [Tomcat JDBC Pool Cleaner[14341596:1527144039908]] (Running)
Thread [DefaultMessageListenerContainer-1] (Running)
Thread [DestroyJavaVM] (Running)
Daemon Thread [JMSCCThreadPoolMaster] (Running)
Daemon Thread [RcvThread: com.ibm.mq.jmqi.remote.impl.RemoteTCPConnection#12474910[qmid=*******,fap=**,channel=****,ccsid=***,sharecnv=***,hbint=*****,peer=*******,localport=****,ssl=****]] (Running)
Thread [Thread-4] (Running)
The help is much appreciated. Thanks in advance. I simply want the application should exit.
Below is the thread dump before I call System.exit(1)
"DefaultMessageListenerContainer-1"
java.lang.Thread.State: RUNNABLE
at sun.management.ThreadImpl.getThreadInfo1(Native Method)
at sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:174)
at com.QueueErrorHandler.handleError(QueueErrorHandler.java:42)
at org.springframework.jms.listener.AbstractMessageListenerContainer.invokeErrorHandler(AbstractMessageListenerContainer.java:931)
at org.springframework.jms.listener.AbstractMessageListenerContainer.handleListenerException(AbstractMessageListenerContainer.java:902)
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:326)
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:235)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1166)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1158)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:1055)
at java.lang.Thread.run(Thread.java:745)
You should take a thread dump to see what Thread [DefaultMessageListenerContainer-1] (Running) is doing.
Now in case a kafka producer fails
What kind of failure? If the broker is down, the thread will block in the producer library for up to 60 seconds by default.
You can reduce that time by setting the max.block.ms producer property.
Couple of solutions which worked for me to solve above.
Solutions 1.
Get all threads in error handler and interrupt them all and then exist the system.
ThreadMXBean threadMXBean = ManagementFactory.getThreadMXBean();
ThreadInfo[] threadInfos = threadMXBean.getThreadInfo(threadMXBean.getAllThreadIds(), 100);
for (ThreadInfo threadInfo : threadInfos) {
Thread.currentThread().interrupt();
}
System.exit(1);
Solution 2. Define a application context manager. Like
public class AppContextManager implements ApplicationContextAware {
private static ApplicationContext _appCtx;
#Override
public void setApplicationContext(ApplicationContext ctx){
_appCtx = ctx;
}
public static ApplicationContext getAppContext(){
return _appCtx;
}
public static void exit(Integer exitCode) {
System.exit(SpringApplication.exit(_appCtx,() -> exitCode));
}
}
Then use same manager to exit in error handler
Executors.newSingleThreadExecutor().execute(new Runnable() {
public void run() {
jmsListenerEndpointRegistry.stop();
AppContextManager.exit(-1);
}
});

Resources