I have subscriber and publisher subscriber collect the messages and once it reaches specified number of message collected, it pass the control to publisher
I need to publish the message to different queue, Upon successful publish of each message I need to manually ack the subscribed queue to remove the message.
I have used return call back and confirm call back on rabbit template but callback functions are not called on successful publish of message.
I guess issue is both publisher and consumer use the same connection thats why callback is not called But I am not sure .
#SpringBootApplication
#ComponentScan(basePackages={"com.comp.dftp.scrubber.configuration","com.comp.dftp.scrubber.container",
"com.comp.dftp.scrubber.subscriber","com.comp.dftp.scrubber.publisher"})
#EnableAutoConfiguration(exclude=RabbitAutoConfiguration.class)
#EnableRabbit
public class DftpEppScrubberApplication {
private static final Logger LOGGER = LoggerFactory.getLogger(DftpEppScrubberApplication.class);
RabbitListenerEndpointRegistrar registrar;
AppConfig appConfig;
public static void main(String[] args) {
SpringApplication app = new SpringApplication(DftpEppScrubberApplication.class);
ConfigurableApplicationContext appContext = app.run(args);
}
#PreDestroy
public void graceFullExit() {
LOGGER.info("----------- Stopping Scrubber Container--------------");
if(registrar!= null && registrar.getEndpointRegistry() != null ) {
registrar.getEndpointRegistry().stop();
}
LOGGER.info("Container Stopped Sucessfully!");
/*
* If Application is not ready to exit safely we will weight for 30 sec
* and re-check again, This will be continued until the flag ReadyToGraccefullyExit
* is set true be process which has not completed Yet, Once publisher completed the process
* and reader ack back to publisher.
*/
while(!appConfig.isReadyToGraccefullyExit()) {
try{
Thread.sleep(30 * 1000);
appConfig.setReadyToGraccefullyExit(true); /* this is just for testing*/
} catch (InterruptedException e) {
LOGGER.error("Some error in gracefully exiting the application!", e);
}
}
LOGGER.info("###STOP FROM THE LIFECYCLE###");
}
}
```java
#Component
#Configuration
public class EPPQ2ListenerConfigurer implements RabbitListenerConfigurer{
public EPPQ2ListenerConfigurer(ConfigurableApplicationContext ctx) {
// TODO Auto-generated constructor stub
}
#Override
public void configureRabbitListeners(RabbitListenerEndpointRegistrar registrar) {
registrar.setMessageHandlerMethodFactory(messageHandlerMethodFactory());
}
#Bean
MessageHandlerMethodFactory messageHandlerMethodFactory() {
DefaultMessageHandlerMethodFactory messageHandlerMethodFactory = new DefaultMessageHandlerMethodFactory();
messageHandlerMethodFactory.setMessageConverter(consumerJackson2MessageConverter());
return messageHandlerMethodFactory;
}
#Bean
public MappingJackson2MessageConverter consumerJackson2MessageConverter() {
return new MappingJackson2MessageConverter();
}
}
```java
#Configuration
public class ListenerContainerFactory {
static final Logger logger = LoggerFactory.getLogger(ListenerContainerFactory.class);
#Autowired
RabbitMqConfig rabbitMqConfig;
#Autowired
EPPQ2Subscriber receiver;
public ListenerContainerFactory(ConfigurableApplicationContext ctx) {
printContainerStartMsg();
}
private void printContainerStartMsg() {
logger.info("----------- Scrubber Container Starts --------------");
}
#Bean
public CachingConnectionFactory subscriberConnectionFactory() {
CachingConnectionFactory subsCachingConnectionFactory = new CachingConnectionFactory(rabbitMqConfig.getSubscriberHosts(),
rabbitMqConfig.getSubscriberPort());
subsCachingConnectionFactory.setUsername(rabbitMqConfig.getSubscriberUsername());
subsCachingConnectionFactory.setPassword(rabbitMqConfig.getSubscriberPassword());
subsCachingConnectionFactory.setVirtualHost("hydra.services");
subsCachingConnectionFactory.setConnectionNameStrategy(f -> "subscriberConnection");
return subsCachingConnectionFactory;
}
#Bean
public SimpleRabbitListenerContainerFactory queueListenerContainer(
#Qualifier("subscriberConnectionFactory") CachingConnectionFactory subscriberConnectionFactory,
MessageListenerAdapter listenerAdapter) {
connectionFactory.setAddresses(rabbitMqConfig.getSubscriberHosts());
connectionFactory.setVirtualHost("hydra.services");
connectionFactory.setPort(rabbitMqConfig.getSubscriberPort());
connectionFactory.setUsername(rabbitMqConfig.getSubscriberUsername());
connectionFactory.setPassword(rabbitMqConfig.getSubscriberPassword());
SimpleRabbitListenerContainerFactory factory = new SimpleRabbitListenerContainerFactory();
factory.setAcknowledgeMode(AcknowledgeMode.MANUAL);
factory.setConnectionFactory(subscriberConnectionFactory);
factory.setErrorHandler(errorHandler());
return factory;
}
#Bean
MessageListenerAdapter listenerAdapter(EPPQ2Subscriber receiver) {
return new MessageListenerAdapter(receiver, "receiveMessage");
}
#Bean
public ErrorHandler errorHandler() {
return new ConditionalRejectingErrorHandler(fatalExceptionStrategy());
}
#Bean
public ScrubberFatalExceptionStrategy fatalExceptionStrategy() {
return new ScrubberFatalExceptionStrategy();
}
}
```java
#Component
public class EPPQ2Subscriber {
private static final Logger LOGGER = LoggerFactory.getLogger(EPPQ2Subscriber.class);
//#RabbitListener(queues = "#{queue.getName()}") #TODO I wann to use this in later point in time.. !
#Autowired
RabbitMqConfig rabbitMqConfig;
#Autowired
AppConfig appConfig;
#Autowired
EPPQ2PublisherImpl eppQ2Publisher;
List<Message> messageList = new ArrayList<Message>();
List<Long> deliveryTagList = new ArrayList<Long>();
/**
* Method is listener's receive message method , invoked when there is message ready to read
* #param message - Domain object of message encapsulated
* #param channel - rabitmq client channel
* #param messageId - #TODO Delete it later.
* #param messageProperties - amqp message properties contains message properties such as delivery tag etc..
*/
#RabbitListener(id="messageListener",queues = "#{rabbitMqConfig.getSubscriberQueueName()}",containerFactory="queueListenerContainer")
public void receiveMessage(Message message, Channel channel, #Header("id") String messageId,
MessageProperties messageProperties) {
LOGGER.info("Result:" + message.getClass() + ":" + message.toString());
if(messageList.size() < appConfig.getSubscriberChunkSize() ) {
messageList.add(message);
LOGGER.info("For Test Size:"+messageList.size()+ "chunk size : "+appConfig.getSubscriberChunkSize());
deliveryTagList.add(messageProperties.getDeliveryTag());
} else {
// call the service here to decrypt, read pan, call danger to scrub, encrypt pan and re-pack them in message again.
//after this branch messageList should have scrubbed and encrypted message objects ready to publish.
// Here is call for publish and ack messages..
eppQ2Publisher.sendMessages(messageList, channel, deliveryTagList);
}
}
}
````java
#Configuration
public class TopicConfiguration {
private static final Logger LOGGER = LoggerFactory.getLogger(TopicConfiguration.class);
#Autowired
RabbitMqConfig rabbitMqConfig;
#Autowired
EPPQ2Publisher eppQ2Publisher;
/**
* Caching connection factory
* #return CachingConnectionFactory
*/
#Bean
public CachingConnectionFactory cachingConnectionFactory() {
CachingConnectionFactory cachingConnectionFactory = new CachingConnectionFactory(rabbitMqConfig.getPublisherHosts(),
rabbitMqConfig.getPublisherPort());
cachingConnectionFactory.setUsername(rabbitMqConfig.getPublisherUsername());
cachingConnectionFactory.setPassword(rabbitMqConfig.getPublisherPassword());
cachingConnectionFactory.setVirtualHost("hydra.services");
cachingConnectionFactory.createConnection();
cachingConnectionFactory.setConnectionNameStrategy(f -> "publisherConnection");
return cachingConnectionFactory;
}
/**
* Bean RabbitTemplate
* #return RabbitTemplate
*/
#Bean
public RabbitTemplate template(
#Qualifier("cachingConnectionFactory") CachingConnectionFactory cachingConnectionFactory) {
final RabbitTemplate rabbitTemplate = new RabbitTemplate(cachingConnectionFactory);
rabbitTemplate.setMessageConverter(producerJackson2MessageConverter());
RetryTemplate retryTemplate = new RetryTemplate();
ExponentialBackOffPolicy backOffPolicy = new ExponentialBackOffPolicy();
backOffPolicy.setInitialInterval(500);
backOffPolicy.setMultiplier(10.0);
backOffPolicy.setMaxInterval(10000);
retryTemplate.setBackOffPolicy(backOffPolicy);
rabbitTemplate.setRetryTemplate(retryTemplate);
rabbitTemplate.setExchange(rabbitMqConfig.getPublisherTopic());
rabbitTemplate.setUsePublisherConnection(true);
rabbitTemplate.setMandatory(true);
rabbitTemplate.setConfirmCallback((correlation, ack, reason) -> {
if (correlation != null) {
LOGGER.info("Received " + (ack ? " ack " : " nack ") + "for correlation: " + correlation);
if (ack) {
// this is confirmation received..
// here is code to ack Q1. correlation.getId() and ack it !!
eppQ2Publisher.ackMessage(new Long(correlation.getId().toString()));
} else {
// no confirmation received and no need to do any thing for
// retry..
}
}
});
rabbitTemplate.setReturnCallback((message, replyCode, replyText, exchange, routingKey) -> {
LOGGER.error("Returned: " + message + "\nreplyCode: " + replyCode + "\nreplyText: " + replyText
+ "\nexchange/rk: " + exchange + "/" + routingKey);
});
return rabbitTemplate;
}
/**
* Bean Jackson2JsonMessageConverter
* #return Jackson2JsonMessageConverter
*/
#Bean
public Jackson2JsonMessageConverter producerJackson2MessageConverter() {
return new Jackson2JsonMessageConverter();
}
}
Here is some logs
----------
33morg.springframework.amqp.rabbit.listener.BlockingQueueConsumer[0;39m: Received message: (Body:'[B#1c63ef5(byte[2003])' MessageProperties [headers={}, contentLength=0, receivedDeliveryMode=NON_PERSISTENT, redelivered=true, receivedExchange=, receivedRoutingKey=hydra.Syphon.q1, deliveryTag=11, consumerTag=amq.ctag-VLCSea_a-FmED6I54TyM4w, consumerQueue=hydra.Syphon.q1])
SimpleAsyncTaskExecutor-1 org.springframework.amqp.rabbit.listener.adapter.MessagingMessageListenerAdapter[0;39m: Processing [GenericMessage [payload=byte[2003], headers={amqp_receivedDeliveryMode=NON_PERSISTENT, amqp_receivedRoutingKey=hydra.Syphon.q1, amqp_deliveryTag=11, amqp_consumerQueue=hydra.Syphon.q1, amqp_redelivered=true, id=2a24ec32-2576-a208-4cbf-f36e6170ac83, amqp_consumerTag=amq.ctag-VLCSea_a-FmED6I54TyM4w, timestamp=1554262675707}]]
SimpleAsyncTaskExecutor-1 com.discover.dftp.scrubber.subscriber.EPPQ2Subscriber[0;39m: Result:class com.discover.dftp.scrubber.domain.Message:Header [header={RETRY_COUNT=0, PUBLISH_EVENT_TYPE=AUTH}, payLoad={MTI=400, MTI_REQUEST=400, PAN=6011000000000000, PROCCODE=00, PROCCODE_REQUEST=00, FROM_ACCOUNT=00, TO_ACCOUNT=00, TRANSACTION_AMOUNT=000000000100, TRANSMISSION_MMDDHHMMSS=0518202930, STAN=000001, LOCALTIME_HHMMSS=010054, LOCALDATE_YYMMDD=180522, MERCHANT_TYPE=5311, ACQUIRING_COUNTRY_CODE=840, POS_ENTRY_MODE=02, POS_PIN_ENTRY_CAPABILITIES=0, FUNCTION_CODE=400, MESSAGE_REASON_CODE=06, ACQUIRING_ID_CODE=000000, FORWARDING_ID_CODE=000000, RETRIEVAL_REFERENCE_NUMBER=1646N472D597, APPROVAL_CODE=12345R, RESPONSE_CODE=00, MERCHANT_NUMBER=601100000000596, CARD_ACCEPTOR_NAME=Discover Acq Simulator, CARD_ACCEPTOR_CITY=Riverwoods, CARD_ACCEPTOR_STATE=IL, CARD_ACCEPTOR_COUNTRY=840, CARD_ACCEPTOR_COUNTRY_3NUMERIC=840, NRID=123456789012345, TRANSACTION_CURRENCY_CODE=840, POS_TERMINAL_ATTENDANCE_INDICATOR=0, POS_PARTIAL_APPROVAL_INDICATOR=0, POS_TERMINAL_LOCATION_INDICATOR=0, POS_TRANSACTION_STATUS_INDICATOR=0, POS_ECOMMERCE_TRAN_INDICATOR=0, POS_TYPE_OF_TERMINAL_DEVICE=0, POS_CARD_PRESENCE_INDICATOR=0, POS_CARD_CAPTURE_CAPABILITIES_INDICATOR=1, POS_TRANSACTION_SECURITY_INDICATOR=0, POS_CARD_DATA_TERMINAL_INPUT_CAPABILITY_INDICATOR=2, POS_CARDHOLDER_PRESENCE_INDICATOR=0, DFS_POS_DATA=0000010000200, GEODATA_STREET_ADDRESS=2500 LAKE COOK ROAD , GEODATA_POSTAL_CODE=600150000, GEODATA_COUNTY_CODE=840, GEODATA_STORE_NUMBER=10001, GEODATA_MALL_NAME=DISCOVER FINANCIAL SR, VERSION_INDICATOR=03141, REVERSAL_FLAG=Y, ISS_REFERENCE_ID=72967956, ISS_PROCESSOR_REFERENCE_ID=123459875}]
SimpleAsyncTaskExecutor-1 org.springframework.retry.support.RetryTemplate[0;39m: Retry: count=0
SimpleAsyncTaskExecutor-1 org.springframework.amqp.rabbit.core.RabbitTemplate[0;39m: Executing callback RabbitTemplate$$Lambda$268/19900766 on RabbitMQ Channel: Cached Rabbit Channel: AMQChannel(amqp://dftp_publisher#10.15.190.18:5672/hydra.services,1), conn: Proxy#10fa93b Shared Rabbit Connection: SimpleConnection#6329d2 [delegate=amqp://dftp_publisher#10.15.190.18:5672/hydra.services, localPort= 62375]
SimpleAsyncTaskExecutor-1 org.springframework.amqp.rabbit.core.RabbitTemplate[0;39m: Publishing message (Body:'{"HEADER":{"RETRY_COUNT":0,"PUBLISH_EVENT_TYPE":"AUTH"},"PAYLOAD":{"MTI":"400","MTI_REQUEST":"400","PAN":"6011000000000000","PROCCODE":"00","PROCCODE_REQUEST":"00","FROM_ACCOUNT":"00","TO_ACCOUNT":"00","TRANSACTION_AMOUNT":"000000000100","TRANSMISSION_MMDDHHMMSS":"0518202930","STAN":"000001","LOCALTIME_HHMMSS":"010054","LOCALDATE_YYMMDD":"180522","MERCHANT_TYPE":"5311","ACQUIRING_COUNTRY_CODE":"840","POS_ENTRY_MODE":"02","POS_PIN_ENTRY_CAPABILITIES":"0","FUNCTION_CODE":"400","MESSAGE_REASON_CODE":"06","ACQUIRING_ID_CODE":"000000","FORWARDING_ID_CODE":"000000","RETRIEVAL_REFERENCE_NUMBER":"1646N472D597","APPROVAL_CODE":"12345R","RESPONSE_CODE":"00","MERCHANT_NUMBER":"601100000000596","CARD_ACCEPTOR_NAME":"Discover Acq Simulator","CARD_ACCEPTOR_CITY":"Riverwoods","CARD_ACCEPTOR_STATE":"IL","CARD_ACCEPTOR_COUNTRY":"840","CARD_ACCEPTOR_COUNTRY_3NUMERIC":"840","NRID":"123456789012345","TRANSACTION_CURRENCY_CODE":"840","POS_TERMINAL_ATTENDANCE_INDICATOR":"0","POS_PARTIAL_APPROVAL_INDICATOR":"0","POS_TERMINAL_LOCATION_INDICATOR":"0","POS_TRANSACTION_STATUS_INDICATOR":"0","POS_ECOMMERCE_TRAN_INDICATOR":"0","POS_TYPE_OF_TERMINAL_DEVICE":"0","POS_CARD_PRESENCE_INDICATOR":"0","POS_CARD_CAPTURE_CAPABILITIES_INDICATOR":"1","POS_TRANSACTION_SECURITY_INDICATOR":"0","POS_CARD_DATA_TERMINAL_INPUT_CAPABILITY_INDICATOR":"2","POS_CARDHOLDER_PRESENCE_INDICATOR":"0","DFS_POS_DATA":"0000010000200","GEODATA_STREET_ADDRESS":"2500 LAKE COOK ROAD ","GEODATA_POSTAL_CODE":"600150000","GEODATA_COUNTY_CODE":"840","GEODATA_STORE_NUMBER":"10001","GEODATA_MALL_NAME":"DISCOVER FINANCIAL SR","VERSION_INDICATOR":"03141","REVERSAL_FLAG":"Y","ISS_REFERENCE_ID":"72967956","ISS_PROCESSOR_REFERENCE_ID":"123459875"}}' MessageProperties [headers={__TypeId__=com.discover.dftp.scrubber.domain.Message}, contentType=application/json, contentEncoding=UTF-8, contentLength=1705, deliveryMode=PERSISTENT, priority=0, deliveryTag=0])on exchange [hydra.test.exc], routingKey = [800]
SimpleAsyncTaskExecutor-1 org.springframework.retry.support.RetryTemplate[0;39m: Retry: count=0
SimpleAsyncTaskExecutor-1 org.springframework.amqp.rabbit.core.RabbitTemplate[0;39m: Executing callback RabbitTemplate$$Lambda$268/19900766 on RabbitMQ Channel: Cached Rabbit Channel: AMQChannel(amqp://dftp_publisher#10.15.190.18:5672/hydra.services,1), conn: Proxy#10fa93b Shared Rabbit Connection: SimpleConnection#6329d2 [delegate=amqp://dftp_publisher#10.15.190.18:5672/hydra.services, localPort= 62375]
SimpleAsyncTaskExecutor-1 org.springframework.amqp.rabbit.core.RabbitTemplate[0;39m: Publishing message (Body:'{"HEADER":{"RETRY_COUNT":0,"PUBLISH_EVENT_TYPE":"AUTH"},"PAYLOAD":{"MTI":"400","MTI_REQUEST":"400","PAN":"6011000000000000","PROCCODE":"00","PROCCODE_REQUEST":"00","FROM_ACCOUNT":"00","TO_ACCOUNT":"00","TRANSACTION_AMOUNT":"000000000100","TRANSMISSION_MMDDHHMMSS":"0518202930","STAN":"000001","LOCALTIME_HHMMSS":"010054","LOCALDATE_YYMMDD":"180522","MERCHANT_TYPE":"5311","ACQUIRING_COUNTRY_CODE":"840","POS_ENTRY_MODE":"02","POS_PIN_ENTRY_CAPABILITIES":"0","FUNCTION_CODE":"400","MESSAGE_REASON_CODE":"06","ACQUIRING_ID_CODE":"000000","FORWARDING_ID_CODE":"000000","RETRIEVAL_REFERENCE_NUMBER":"1646N472D597","APPROVAL_CODE":"12345R","RESPONSE_CODE":"00","MERCHANT_NUMBER":"601100000000596","CARD_ACCEPTOR_NAME":"Discover Acq Simulator","CARD_ACCEPTOR_CITY":"Riverwoods","CARD_ACCEPTOR_STATE":"IL","CARD_ACCEPTOR_COUNTRY":"840","CARD_ACCEPTOR_COUNTRY_3NUMERIC":"840","NRID":"123456789012345","TRANSACTION_CURRENCY_CODE":"840","POS_TERMINAL_ATTENDANCE_INDICATOR":"0","POS_PARTIAL_APPROVAL_INDICATOR":"0","POS_TERMINAL_LOCATION_INDICATOR":"0","POS_TRANSACTION_STATUS_INDICATOR":"0","POS_ECOMMERCE_TRAN_INDICATOR":"0","POS_TYPE_OF_TERMINAL_DEVICE":"0","POS_CARD_PRESENCE_INDICATOR":"0","POS_CARD_CAPTURE_CAPABILITIES_INDICATOR":"1","POS_TRANSACTION_SECURITY_INDICATOR":"0","POS_CARD_DATA_TERMINAL_INPUT_CAPABILITY_INDICATOR":"2","POS_CARDHOLDER_PRESENCE_INDICATOR":"0","DFS_POS_DATA":"0000010000200","GEODATA_STREET_ADDRESS":"2500 LAKE COOK ROAD ","GEODATA_POSTAL_CODE":"600150000","GEODATA_COUNTY_CODE":"840","GEODATA_STORE_NUMBER":"10001","GEODATA_MALL_NAME":"DISCOVER FINANCIAL SR","VERSION_INDICATOR":"03141","REVERSAL_FLAG":"Y","ISS_REFERENCE_ID":"72967956","ISS_PROCESSOR_REFERENCE_ID":"123459875"}}' MessageProperties [headers={__TypeId__=com.discover.dftp.scrubber.domain.Message}, contentType=application/json, contentEncoding=UTF-8, contentLength=1705, deliveryMode=PERSISTENT, priority=0, deliveryTag=0])on exchange [hydra.test.exc], routingKey = [800]
You need to enable returns and confirms in the CachingConnectionFactory configuration. See the documentation.
This feature requires a CachingConnectionFactory that has its publisherReturns property set to true (see Publisher Confirms and Returns).
...
For publisher confirms (also known as publisher acknowledgements), the template requires a CachingConnectionFactory that has its publisherConfirms property set to true.
Follow #Gary Russell's answer, I added configurations below in my codes, then ConfirmCallback starts to work. I use SpringBoot 2.5.2 and RabbitMQ 3.8.18
cachingConnectionFactory.setPublisherReturns(true);
cachingConnectionFactory.setPublisherConfirmType(CachingConnectionFactory.ConfirmType.CORRELATED);
Rabbitmq-Batch-Rabbitmq-Publish-Subscribe
We use spring boot (1.5) and integration to connect to Rabbit MQ as a service on PCF
Properties set
- concurrency = 15
- maxConcurrency = 25
- default prefetch and txSize.
How the message are processed.
The queue received 26 messages within less than 1 sec.
15 consumers started and completed with the next minute.
Average time taken by each consumer is 30 secs approx.
Few consumers took less than 15 secs.
As each consumer acknowledged, new messages(16th, 17th message) from ready becomes unacknowledged on rabbit MQ.
So far as expected, since i assume new messages are being proccessed as old messages are acknowledged.
But as new messages become unack'ed on rabbitMQ, they are not processed by any consumers. Consumers are stuck and idle.
This continues and all ready state messages become unacknowledged and stay there without any activity.
They all resume after approx 15mins.
I see this behavior always.
Any guidance ?
Here is the code that does wiring of rabbit MQ
The MessageHandler in between has code that connects to big data and does some querying.
#Configuration
#EnableRabbit
public class RabbitMQ {
protected final Logger logger = LoggerFactory.getLogger(this.getClass());
#Value("${spring.rabbitmq.listener.concurrency:3}")
private int concurrentConsumers;
#Value("${spring.rabbitmq.listener.maxConcurrency:5}")
private int maxConcurrentConsumers;
#Value("${spring.rabbitmq.template.retry.max-attempts:3}")
private int maxAttempts;
#Value("${spring.rabbitmq.template.retry.initial-interval:2000}")
private int initialInterval;
#Value("${spring.rabbitmq.template.retry.multiplier:3}")
private int multiplier;
#Value("${spring.rabbitmq.template.retry.max-interval:10000}")
private int maxInterval;
#Autowired
private BotQueues botQueues;
#Autowired
private RetryFailLogger retryRecoverer;
#Bean
public StatefulRetryOperationsInterceptor statefulRetryOperationsInterceptor() {
return RetryInterceptorBuilder.stateful()
.backOffOptions(initialInterval, multiplier, maxInterval) // initialInterval, multiplier, maxInterval
.maxAttempts(maxAttempts)
.messageKeyGenerator(message -> message.getMessageProperties().getMessageId())
.recoverer(retryRecoverer)
.build();
}
#Bean
public Jackson2JsonMessageConverter defaultJsonMessageConverter() {
Jackson2JsonMessageConverter jsonConverter = new Jackson2JsonMessageConverter();
DefaultClassMapper classMapper = new DefaultClassMapper();
classMapper.setDefaultType(JobDetailInfo.class);
jsonConverter.setClassMapper(classMapper);
return jsonConverter;
}
#Bean
public SimpleMessageListenerContainer container(ConnectionFactory connectionFactory)
throws BeansException, ClassNotFoundException {
SimpleMessageListenerContainer container = new SimpleMessageListenerContainer(connectionFactory);
container.setQueues(botQueues.inputQueues());
container.setConcurrentConsumers(concurrentConsumers);
container.setMaxConcurrentConsumers(maxConcurrentConsumers);
container.setChannelTransacted(true);
container.setAdviceChain(new Advice[] {statefulRetryOperationsInterceptor()});
return container;
}
#Bean
public AmqpInboundChannelAdapter inbound(SimpleMessageListenerContainer container,
#Qualifier("commonInputChannel") MessageChannel amqpInputChannel) {
AmqpInboundChannelAdapter adapter = new AmqpInboundChannelAdapter(container);
adapter.setMessageConverter(defaultJsonMessageConverter());
adapter.setOutputChannel(amqpInputChannel);
return adapter;
}
#Bean
#ServiceActivator(inputChannel = "commonInputChannel", outputChannel = "commonOutputChannel")
public MessageHandler messageHandler() {
return new MessageHandler();
}
#Router(inputChannel = "commonOutputChannel")
public String resolveJobChannel(JobDetailInfo jobDetailInfo) {
String returnChannel = "";
if (jobDetailInfo != null) {
switch (jobDetailInfo.getJobStatus()) {
case Scheduled:
returnChannel = "collectorChannel";
break;
default:
break;
}
}
return returnChannel;
}
}
Thread Dump:
{"threadName":"container-14","threadId":40,"blockedTime":-1,"blockedCount":45,"waitedTime":-1,"waitedCount":66654,"lockName":"java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject#1b4302a9","lockOwnerId":-1,"lockOwnerName":null,"inNative":false,"suspended":false,"threadState":"TIMED_WAITING","stackTrace":[{"methodName":"park","fileName":"Unsafe.java","lineNumber":-2,"className":"sun.misc.Unsafe","nativeMethod":true},{"methodName":"parkNanos","fileName":"LockSupport.java","lineNumber":215,"className":"java.util.concurrent.locks.LockSupport","nativeMethod":false},{"methodName":"awaitNanos","fileName":"AbstractQueuedSynchronizer.java","lineNumber":2078,"className":"java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject","nativeMethod":false},{"methodName":"poll","fileName":"LinkedBlockingQueue.java","lineNumber":467,"className":"java.util.concurrent.LinkedBlockingQueue","nativeMethod":false},{"methodName":"nextMessage","fileName":"BlockingQueueConsumer.java","lineNumber":461,"className":"org.springframework.amqp.rabbit.listener.BlockingQueueConsumer","nativeMethod":false},{"methodName":"doReceiveAndExecute","fileName":"SimpleMessageListenerContainer.java","lineNumber":1214,"className":"org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer","nativeMethod":false},{"methodName":"receiveAndExecute","fileName":"SimpleMessageListenerContainer.java","lineNumber":1189,"className":"org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer","nativeMethod":false},{"methodName":"access$1500","fileName":"SimpleMessageListenerContainer.java","lineNumber":97,"className":"org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer","nativeMethod":false},{"methodName":"run","fileName":"SimpleMessageListenerContainer.java","lineNumber":1421,"className":"org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$AsyncMessageProcessingConsumer","nativeMethod":false},{"methodName":"run","fileName":"Thread.java","lineNumber":748,"className":"java.lang.Thread","nativeMethod":false}],"lockedMonitors":[],"lockedSynchronizers":[],"lockInfo":{"className":"java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject","identityHashCode":457376425}},
{"threadName":"container-13","threadId":39,"blockedTime":-1,"blockedCount":34,"waitedTime":-1,"waitedCount":66160,"lockName":null,"lockOwnerId":-1,"lockOwnerName":null,"inNative":true,"suspended":false,"threadState":"RUNNABLE","stackTrace":[{"methodName":"socketRead0","fileName":"SocketInputStream.java","lineNumber":-2,"className":"java.net.SocketInputStream","nativeMethod":true},{"methodName":"socketRead","fileName":"SocketInputStream.java","lineNumber":116,"className":"java.net.SocketInputStream","nativeMethod":false},{"methodName":"read","fileName":"SocketInputStream.java","lineNumber":171,"className":"java.net.SocketInputStream","nativeMethod":false},{"methodName":"read","fileName":"SocketInputStream.java","lineNumber":141,"className":"java.net.SocketInputStream","nativeMethod":false},{"methodName":"readMore","fileName":"VisibleBufferedInputStream.java","lineNumber":140,"className":"org.postgresql.core.VisibleBufferedInputStream","nativeMethod":false},{"methodName":"ensureBytes","fileName":"VisibleBufferedInputStream.java","lineNumber":109,"className":"org.postgresql.core.VisibleBufferedInputStream","nativeMethod":false},{"methodName":"read","fileName":"VisibleBufferedInputStream.java","lineNumber":67,"className":"org.postgresql.core.VisibleBufferedInputStream","nativeMethod":false},{"methodName":"receiveChar","fileName":"PGStream.java","lineNumber":280,"className":"org.postgresql.core.PGStream","nativeMethod":false},{"methodName":"processResults","fileName":"QueryExecutorImpl.java","lineNumber":1916,"className":"org.postgresql.core.v3.QueryExecutorImpl","nativeMethod":false},{"methodName":"execute","fileName":"QueryExecutorImpl.java","lineNumber":288,"className":"org.postgresql.core.v3.QueryExecutorImpl","nativeMethod":false},{"methodName":"executeInternal","fileName":"PgStatement.java","lineNumber":430,"className":"org.postgresql.jdbc.PgStatement","nativeMethod":false},{"methodName":"execute","fileName":"PgStatement.java","lineNumber":356,"className":"org.postgresql.jdbc.PgStatement","nativeMethod":false},{"methodName":"executeWithFlags","fileName":"PgStatement.java","lineNumber":303,"className":"org.postgresql.jdbc.PgStatement","nativeMethod":false},{"methodName":"executeCachedSql","fileName":"PgStatement.java","lineNumber":289,"className":"org.postgresql.jdbc.PgStatement","nativeMethod":false},{"methodName":"executeWithFlags","fileName":"PgStatement.java","lineNumber":266,"className":"org.postgresql.jdbc.PgStatement","nativeMethod":false},{"methodName":"execute","fileName":"PgStatement.java","lineNumber":262,"className":"org.postgresql.jdbc.PgStatement","nativeMethod":false},{"methodName":"validate","fileName":"PooledConnection.java","lineNumber":532,"className":"org.apache.tomcat.jdbc.pool.PooledConnection","nativeMethod":false},{"methodName":"validate","fileName":"PooledConnection.java","lineNumber":443,"className":"org.apache.tomcat.jdbc.pool.PooledConnection","nativeMethod":false},{"methodName":"borrowConnection","fileName":"ConnectionPool.java","lineNumber":802,"className":"org.apache.tomcat.jdbc.pool.ConnectionPool","nativeMethod":false},{"methodName":"borrowConnection","fileName":"ConnectionPool.java","lineNumber":651,"className":"org.apache.tomcat.jdbc.pool.ConnectionPool","nativeMethod":false},{"methodName":"getConnection","fileName":"ConnectionPool.java","lineNumber":198,"className":"org.apache.tomcat.jdbc.pool.ConnectionPool","nativeMethod":false},{"methodName":"getConnection","fileName":"DataSourceProxy.java","lineNumber":132,"className":"org.apache.tomcat.jdbc.pool.DataSourceProxy","nativeMethod":false},{"methodName":"getConnection","fileName":"DatasourceConnectionProviderImpl.java","lineNumber":122,"className":"org.hibernate.engine.jdbc.connections.internal.DatasourceConnectionProviderImpl","nativeMethod":false},{"methodName":"obtainConnection","fileName":"NonContextualJdbcConnectionAccess.java","lineNumber":35,"className":"org.hibernate.internal.NonContextualJdbcConnectionAccess","nativeMethod":false},{"methodName":"acquireConnectionIfNeeded","fileName":"LogicalConnectionManagedImpl.java","lineNumber":99,"className":"org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl","nativeMethod":false},{"methodName":"getPhysicalConnection","fileName":"LogicalConnectionManagedImpl.java","lineNumber":129,"className":"org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl","nativeMethod":false},{"methodName":"getConnectionForTransactionManagement","fileName":"LogicalConnectionManagedImpl.java","lineNumber":247,"className":"org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl","nativeMethod":false},{"methodName":"begin","fileName":"LogicalConnectionManagedImpl.java","lineNumber":254,"className":"org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl","nativeMethod":false},{"methodName":"begin","fileName":"JdbcResourceLocalTransactionCoordinatorImpl.java","lineNumber":203,"className":"org.hibernate.resource.transaction.backend.jdbc.internal.JdbcResourceLocalTransactionCoordinatorImpl$TransactionDriverControlImpl","nativeMethod":false},{"methodName":"begin","fileName":"TransactionImpl.java","lineNumber":56,"className":"org.hibernate.engine.transaction.internal.TransactionImpl","nativeMethod":false},{"methodName":"beginTransaction","fileName":"HibernateJpaDialect.java","lineNumber":189,"className":"org.springframework.orm.jpa.vendor.HibernateJpaDialect","nativeMethod":false},{"methodName":"doBegin","fileName":"JpaTransactionManager.java","lineNumber":380,"className":"org.springframework.orm.jpa.JpaTransactionManager","nativeMethod":false},{"methodName":"getTransaction","fileName":"AbstractPlatformTransactionManager.java","lineNumber":373,"className":"org.springframework.transaction.support.AbstractPlatformTransactionManager","nativeMethod":false},{"methodName":"createTransactionIfNecessary","fileName":"TransactionAspectSupport.java","lineNumber":447,"className":"org.springframework.transaction.interceptor.TransactionAspectSupport","nativeMethod":false},{"methodName":"invokeWithinTransaction","fileName":"TransactionAspectSupport.java","lineNumber":277,"className":"org.springframework.transaction.interceptor.TransactionAspectSupport","nativeMethod":false},{"methodName":"invoke","fileName":"TransactionInterceptor.java","lineNumber":96,"className":"org.springframework.transaction.interceptor.TransactionInterceptor","nativeMethod":false},{"methodName":"proceed","fileName":"ReflectiveMethodInvocation.java","lineNumber":179,"className":"org.springframework.aop.framework.ReflectiveMethodInvocation","nativeMethod":false},{"methodName":"intercept","fileName":"CglibAopProxy.java","lineNumber":673,"className":"org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor","nativeMethod":false},{"methodName":"handleMessage","fileName":"","lineNumber":-1,"className":"message.handlers.MessageHandler$$EnhancerBySpringCGLIB$$cf53d23e","nativeMethod":false},{"methodName":"invoke","fileName":null,"lineNumber":-1,"className":"sun.reflect.GeneratedMethodAccessor145","nativeMethod":false},{"methodName":"invoke","fileName":"DelegatingMethodAccessorImpl.java","lineNumber":43,"className":"sun.reflect.DelegatingMethodAccessorImpl","nativeMethod":false},{"methodName":"invoke","fileName":"Method.java","lineNumber":498,"className":"java.lang.reflect.Method","nativeMethod":false},{"methodName":"execute","fileName":"ReflectiveMethodExecutor.java","lineNumber":113,"className":"org.springframework.expression.spel.support.ReflectiveMethodExecutor","nativeMethod":false},{"methodName":"getValueInternal","fileName":"MethodReference.java","lineNumber":102,"className":"org.springframework.expression.spel.ast.MethodReference","nativeMethod":false},{"methodName":"access$000","fileName":"MethodReference.java","lineNumber":49,"className":"org.springframework.expression.spel.ast.MethodReference","nativeMethod":false},{"methodName":"getValue","fileName":"MethodReference.java","lineNumber":347,"className":"org.springframework.expression.spel.ast.MethodReference$MethodValueRef","nativeMethod":false},{"methodName":"getValueInternal","fileName":"CompoundExpression.java","lineNumber":88,"className":"org.springframework.expression.spel.ast.CompoundExpression","nativeMethod":false},{"methodName":"getTypedValue","fileName":"SpelNodeImpl.java","lineNumber":131,"className":"org.springframework.expression.spel.ast.SpelNodeImpl","nativeMethod":false},{"methodName":"getValue","fileName":"SpelExpression.java","lineNumber":330,"className":"org.springframework.expression.spel.standard.SpelExpression","nativeMethod":false},{"methodName":"evaluateExpression","fileName":"AbstractExpressionEvaluator.java","lineNumber":169,"className":"org.springframework.integration.util.AbstractExpressionEvaluator","nativeMethod":false},{"methodName":"processInternal","fileName":"MessagingMethodInvokerHelper.java","lineNumber":319,"className":"org.springframework.integration.util.MessagingMethodInvokerHelper","nativeMethod":false},{"methodName":"process","fileName":"MessagingMethodInvokerHelper.java","lineNumber":155,"className":"org.springframework.integration.util.MessagingMethodInvokerHelper","nativeMethod":false},{"methodName":"processMessage","fileName":"MethodInvokingMessageProcessor.java","lineNumber":93,"className":"org.springframework.integration.handler.MethodInvokingMessageProcessor","nativeMethod":false},{"methodName":"handleRequestMessage","fileName":"ServiceActivatingHandler.java","lineNumber":89,"className":"org.springframework.integration.handler.ServiceActivatingHandler","nativeMethod":false},{"methodName":"handleMessageInternal","fileName":"AbstractReplyProducingMessageHandler.java","lineNumber":109,"className":"org.springframework.integration.handler.AbstractReplyProducingMessageHandler","nativeMethod":false},{"methodName":"handleMessage","fileName":"AbstractMessageHandler.java","lineNumber":127,"className":"org.springframework.integration.handler.AbstractMessageHandler","nativeMethod":false},{"methodName":"doDispatch","fileName":"UnicastingDispatcher.java","lineNumber":160,"className":"org.springframework.integration.dispatcher.UnicastingDispatcher","nativeMethod":false},{"methodName":"dispatch","fileName":"UnicastingDispatcher.java","lineNumber":121,"className":"org.springframework.integration.dispatcher.UnicastingDispatcher","nativeMethod":false},{"methodName":"doSend","fileName":"AbstractSubscribableChannel.java","lineNumber":89,"className":"org.springframework.integration.channel.AbstractSubscribableChannel","nativeMethod":false},{"methodName":"send","fileName":"AbstractMessageChannel.java","lineNumber":423,"className":"org.springframework.integration.channel.AbstractMessageChannel","nativeMethod":false},{"methodName":"send","fileName":"AbstractMessageChannel.java","lineNumber":373,"className":"org.springframework.integration.channel.AbstractMessageChannel","nativeMethod":false},{"methodName":"doSend","fileName":"GenericMessagingTemplate.java","lineNumber":115,"className":"org.springframework.messaging.core.GenericMessagingTemplate","nativeMethod":false},{"methodName":"doSend","fileName":"GenericMessagingTemplate.java","lineNumber":45,"className":"org.springframework.messaging.core.GenericMessagingTemplate","nativeMethod":false},{"methodName":"send","fileName":"AbstractMessageSendingTemplate.java","lineNumber":105,"className":"org.springframework.messaging.core.AbstractMessageSendingTemplate","nativeMethod":false},{"methodName":"sendMessage","fileName":"MessageProducerSupport.java","lineNumber":188,"className":"org.springframework.integration.endpoint.MessageProducerSupport","nativeMethod":false},{"methodName":"access$1100","fileName":"AmqpInboundChannelAdapter.java","lineNumber":56,"className":"org.springframework.integration.amqp.inbound.AmqpInboundChannelAdapter","nativeMethod":false},{"methodName":"processMessage","fileName":"AmqpInboundChannelAdapter.java","lineNumber":246,"className":"org.springframework.integration.amqp.inbound.AmqpInboundChannelAdapter$Listener","nativeMethod":false},{"methodName":"onMessage","fileName":"AmqpInboundChannelAdapter.java","lineNumber":203,"className":"org.springframework.integration.amqp.inbound.AmqpInboundChannelAdapter$Listener","nativeMethod":false},{"methodName":"doInvokeListener","fileName":"AbstractMessageListenerContainer.java","lineNumber":822,"className":"org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer","nativeMethod":false},{"methodName":"invokeListener","fileName":"AbstractMessageListenerContainer.java","lineNumber":745,"className":"org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer","nativeMethod":false},{"methodName":"access$001","fileName":"SimpleMessageListenerContainer.java","lineNumber":97,"className":"org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer","nativeMethod":false},{"methodName":"invokeListener","fileName":"SimpleMessageListenerContainer.java","lineNumber":189,"className":"org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$1","nativeMethod":false},{"methodName":"invoke","fileName":null,"lineNumber":-1,"className":"sun.reflect.GeneratedMethodAccessor112","nativeMethod":false},{"methodName":"invoke","fileName":"DelegatingMethodAccessorImpl.java","lineNumber":43,"className":"sun.reflect.DelegatingMethodAccessorImpl","nativeMethod":false},{"methodName":"invoke","fileName":"Method.java","lineNumber":498,"className":"java.lang.reflect.Method","nativeMethod":false},{"methodName":"invokeJoinpointUsingReflection","fileName":"AopUtils.java","lineNumber":333,"className":"org.springframework.aop.support.AopUtils","nativeMethod":false},{"methodName":"invokeJoinpoint","fileName":"ReflectiveMethodInvocation.java","lineNumber":190,"className":"org.springframework.aop.framework.ReflectiveMethodInvocation","nativeMethod":false},{"methodName":"proceed","fileName":"ReflectiveMethodInvocation.java","lineNumber":157,"className":"org.springframework.aop.framework.ReflectiveMethodInvocation","nativeMethod":false},{"methodName":"doWithRetry","fileName":"StatefulRetryOperationsInterceptor.java","lineNumber":229,"className":"org.springframework.retry.interceptor.StatefulRetryOperationsInterceptor$MethodInvocationRetryCallback","nativeMethod":false},{"methodName":"doExecute","fileName":"RetryTemplate.java","lineNumber":286,"className":"org.springframework.retry.support.RetryTemplate","nativeMethod":false},{"methodName":"execute","fileName":"RetryTemplate.java","lineNumber":210,"className":"org.springframework.retry.support.RetryTemplate","nativeMethod":false},{"methodName":"invoke","fileName":"StatefulRetryOperationsInterceptor.java","lineNumber":173,"className":"org.springframework.retry.interceptor.StatefulRetryOperationsInterceptor","nativeMethod":false},{"methodName":"proceed","fileName":"ReflectiveMethodInvocation.java","lineNumber":179,"className":"org.springframework.aop.framework.ReflectiveMethodInvocation","nativeMethod":false},{"methodName":"invoke","fileName":"JdkDynamicAopProxy.java","lineNumber":213,"className":"org.springframework.aop.framework.JdkDynamicAopProxy","nativeMethod":false},{"methodName":"invokeListener","fileName":null,"lineNumber":-1,"className":"com.sun.proxy.$Proxy137","nativeMethod":false},{"methodName":"invokeListener","fileName":"SimpleMessageListenerContainer.java","lineNumber":1276,"className":"org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer","nativeMethod":false},{"methodName":"executeListener","fileName":"AbstractMessageListenerContainer.java","lineNumber":726,"className":"org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer","nativeMethod":false},{"methodName":"doReceiveAndExecute","fileName":"SimpleMessageListenerContainer.java","lineNumber":1219,"className":"org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer","nativeMethod":false},{"methodName":"receiveAndExecute","fileName":"SimpleMessageListenerContainer.java","lineNumber":1189,"className":"org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer","nativeMethod":false},{"methodName":"access$1500","fileName":"SimpleMessageListenerContainer.java","lineNumber":97,"className":"org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer","nativeMethod":false},{"methodName":"run","fileName":"SimpleMessageListenerContainer.java","lineNumber":1421,"className":"org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$AsyncMessageProcessingConsumer","nativeMethod":false},{"methodName":"run","fileName":"Thread.java","lineNumber":748,"className":"java.lang.Thread","nativeMethod":false}],"lockedMonitors":[{"className":"org.postgresql.core.v3.QueryExecutorImpl","identityHashCode":1428002949,"lockedStackDepth":9,"lockedStackFrame":{"methodName":"execute","fileName":"QueryExecutorImpl.java","lineNumber":288,"className":"org.postgresql.core.v3.QueryExecutorImpl","nativeMethod":false}}],"lockedSynchronizers":[{"className":"java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync","identityHashCode":1110565932}],"lockInfo":null}
It's not clear why you would see that behavior; in most cases (in my experience) such issues are caused by the listener thread "stuck" in user code. It's not possible for a consumer thread to be "idle" in that way; and in any case, it wouldn't restart 15 minutes later.
Next step (for me) would be to take a thread dump to see what the threads are doing. Boot provides the /dump endpoint to do that if you can't use jstack or VisualVM.