RabbitMQ Spring Boot AMQP - consume with concurrent threads - spring-boot

I want my app to handle multiple messages received from RabbitMQ concurrently.
I have tried probably all the google-page-1 solutions but it won't work.
Here is my setup:
POM.xml
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>1.5.10.RELEASE</version>
</parent>
.
.
.
<dependency>
<groupId>org.springframework.amqp</groupId>
<artifactId>spring-rabbit</artifactId>
</dependency>
application.properties:
#############################
# RabbitMQ #
#############################
#AMQP RabbitMQ configuration
spring.rabbitmq.host=zzzzzzzz
spring.rabbitmq.port=5672
spring.rabbitmq.username=zzzzzzz
spring.rabbitmq.password=zzzzzzz
#Rabbit component names
com.cp.neworder.queue.name = new-order-queue-stg
com.cp.neworder.queue.exchange = new-order-exchange-stg
com.cp.completedorder.queue.name = completed-order-queue
com.cp.completedorder.queue.exchange = completed-order-exchange
#Rabbit MQ concurrect consumers config
spring.rabbitmq.listener.simple.concurrency=3
spring.rabbitmq.listener.simple.retry.initial-interval=3000
Configuration file:
#Configuration
public class RabbitMQConfig {
#Value("${com.cp.neworder.queue.name}")
private String newOrderQueueName;
#Value("${com.cp.neworder.queue.exchange}")
private String newOrderExchangeName;
#Bean
Queue queue() {
return new Queue(newOrderQueueName, true);
}
#Bean
TopicExchange exchange() {
return new TopicExchange(newOrderExchangeName);
}
#Bean
Binding binding(Queue queue, TopicExchange exchange) {
return BindingBuilder.bind(queue).to(exchange).with(newOrderQueueName);
}
#Bean
SimpleMessageListenerContainer container(ConnectionFactory connectionFactory, MessageListenerAdapter listenerAdapter) {
SimpleMessageListenerContainer container = new SimpleMessageListenerContainer();
container.setConnectionFactory(connectionFactory);
container.setQueueNames(newOrderQueueName);
container.setMessageListener(listenerAdapter);
return container;
}
#Bean
MessageListenerAdapter listenerAdapter(OrderMessageListener receiver) {
return new MessageListenerAdapter(receiver, "receiveOrder");
}
}
My consumer class works as intended, it just processes one request at a time. How do I know?
I save the process of my async requests in DB, so I can query how many are processing at the moment, and it's always just 1.
I can look at the RabbitMQ Management platform, and I see that it's being dequeued one by one.
What are the mistakes in my setup? How do I get it to work?
Thanks.

SimpleMessageListenerContainer has a way to set the concurrent consumers. It has setConcurrentConsumers method where you can set the number of consumers.
#Bean
SimpleMessageListenerContainer container(ConnectionFactory connectionFactory, MessageListenerAdapter listenerAdapter) {
SimpleMessageListenerContainer container = new SimpleMessageListenerContainer();
container.setConnectionFactory(connectionFactory);
container.setQueueNames(newOrderQueueName);
container.setMessageListener(listenerAdapter);
container. setConcurrentConsumers(10);
return container;
}
With this configuration, when you start the application, you will be able to see multiple consumers in the RabbitMQ admin

You are not using Boot to create the container so the boot properties aren't applied.
Try
#Bean
SimpleMessageListenerContainer container(ConnectionFactory connectionFactory, MessageListenerAdapter listenerAdapter,
RabbitProperties properties) {
SimpleMessageListenerContainer container = new SimpleMessageListenerContainer();
container.setConnectionFactory(connectionFactory);
container.setQueueNames(newOrderQueueName);
container.setMessageListener(listenerAdapter);
container.setConcurrentConsumers(properties.getListener().getSimple().getConcurrency();
return container;
}

Related

How to consume only one RabbitMQ on an application that has 3 queues?

I have a Spring Boot application that uses RabbitMQ and has 3 queues (queue1, queue2 and queue3).
In this application i have one listener, that should only listen for messages on the queue named queue1 and ignore the other 2 queues, but it is getting messages from all queues.
This is my RabbitMQ config:
#Bean
public ConnectionFactory connectionFactory() {
CachingConnectionFactory connectionFactory = new CachingConnectionFactory(this.host);
connectionFactory.setPort(this.port);
connectionFactory.setUsername(this.user);
connectionFactory.setPassword(this.password);
return connectionFactory;
}
#Bean
SimpleMessageListenerContainer container(MessageListenerAdapter listenerAdapter) {
SimpleMessageListenerContainer container = new SimpleMessageListenerContainer();
container.setConnectionFactory(this.connectionFactory());
container.setQueueNames(this.startQueueQueueName, this.printQueueQueueName, this.errorQueueQueueName);
container.setMessageListener(listenerAdapter);
return container;
}
#Bean
MessageListenerAdapter listenerAdapter(RabbitDocumentToPrintListener receiver) {
return new MessageListenerAdapter(receiver, "receiveMessage");
}
and this is my listener
public void receiveMessage(String message) throws Exception {
this.logger.debug("Received message from Rabbit");
}
I've tried adding #RabbitListener(queues = "queue1", exclusive = true) to the listener, but it didn't work.
If someone could help me making this app to consume only queue1, I'd appreciate. Thanks!

Spring boot + Rabbit MQ Process messages paralally

I have observed that I have multiple messages in queue but my worker is picking messages one by one, it's not processing messages parallally. What I am doing wrong here ? How can i process multiple messages parallally to utilizing my worker processing at most. Any best practice ?
I am not sure what concurrency value 8 doing here.
application.yml
spring:
rabbitmq:
host:
port:
username:
virtual-host:
password:
listener:
simple:
concurrency: 8
prefetch: 8
Bean config:
#Bean
Queue queue() {
return new Queue("testQ", true);
}
#Bean
TopicExchange exchange() {
return new TopicExchange("testE");
}
#Bean
Binding binding(Queue queue, TopicExchange exchange) {
return BindingBuilder.bind("testQ").to("testE").with("a.b.c");
}
#Bean
SimpleMessageListenerContainer container(ConnectionFactory connectionFactory,
MessageListenerAdapter listenerAdapter) {
SimpleMessageListenerContainer container = new SimpleMessageListenerContainer();
container.setConnectionFactory(connectionFactory);
container.setQueueNames("testQ");
container.setMessageListener(listenerAdapter);
return container;
}
#Bean
MessageListenerAdapter listenerAdapter(Receiver receiver) {
return new MessageListenerAdapter(receiver, "handleMessage");
}
I think if you create SimpleMessageListenerContainer in your code yourself, then you could specify the parameter "concurrency" right in your code. For example:
#Bean
SimpleMessageListenerContainer container(ConnectionFactory connectionFactory,
MessageListenerAdapter listenerAdapter) {
SimpleMessageListenerContainer container = new SimpleMessageListenerContainer();
container.setConnectionFactory(connectionFactory);
container.setQueueNames("testQ");
container.setConcurrency("8"); //set 'Concurrency' property for your container
container.setMessageListener(listenerAdapter);
return container;
}

Create Shared-Durable subscriber with Spring-boot

I'd like to create a shared-durable subscriber with spring-boot and IBM MQ. I have done to make a durable subscriber, but fail for the shared subscription.
When I debug the program, I found null pointer exception issue
java.lang.NoSuchMethodError: javax.jms.Session.createSharedDurableConsumer(Ljavax/jms/Topic;Ljava/lang/String;Ljava/lang/String;)Ljavax/jms/MessageConsumer;
inside class AbstractMessageListenerContainer, method createConsumer. It is because it try to invoke session.createSharedDurableConsumerMethod where the session object is of javax.jms.Session and IntelliJ point it to library geronimo-jms_1.1_spec-1.1.1.jar
My pom.xml is using JMS 2.0:
<dependency>
<groupId>javax.jms</groupId>
<artifactId>javax.jms-api</artifactId>
<version>2.0.1</version>
</dependency>
Here is the java code:
import com.ibm.mq.jms.*;
...
public class JmsConfig {
...
#Bean
public MQTopicConnectionFactory mqTopicConnectionFactory(){
MQTopicConnectionFactory factory = new MQTopicConnectionFactory();
try{
factory.setHostName(mqHostname);
factory.setChannel(mqChannel);
factory.setPort(Integer.parseInt(mqPort));
factory.setQueueManager(mqQManager);
factory.setTransportType(WMQConstants.WMQ_CM_CLIENT);
factory.setClientReconnectTimeout(0);
} catch (Exception e) {
e.printStackTrace();
}
return factory;
}
#Bean
#Primary
UserCredentialsConnectionFactoryAdapter userCredentialsConnectionFactoryAdapter(MQTopicConnectionFactory mqTopicConnectionFactory) {
UserCredentialsConnectionFactoryAdapter userCredentialsConnectionFactoryAdapter = new UserCredentialsConnectionFactoryAdapter();
userCredentialsConnectionFactoryAdapter.setTargetConnectionFactory(mqTopicConnectionFactory);
userCredentialsConnectionFactoryAdapter.setUsername(mqUsername);
if(!mqPassword.equals("")) {
userCredentialsConnectionFactoryAdapter.setPassword(mqPassword);
}
return userCredentialsConnectionFactoryAdapter;
}
#Bean
public CachingConnectionFactory cachingConnectionFactory(UserCredentialsConnectionFactoryAdapter userCredentialsConnectionFactoryAdapter) {
CachingConnectionFactory cachingConnectionFactory = new CachingConnectionFactory();
cachingConnectionFactory.setTargetConnectionFactory(userCredentialsConnectionFactoryAdapter);
cachingConnectionFactory.setSessionCacheSize(1000);
cachingConnectionFactory.setReconnectOnException(true);
return cachingConnectionFactory;
}
#Bean
public JmsOperations jmsOperations(CachingConnectionFactory cachingConnectionFactory) {
JmsTemplate jmsTemplate = new JmsTemplate(cachingConnectionFactory);
jmsTemplate.setPubSubDomain(true);
jmsTemplate.setReceiveTimeout(2000);
return jmsTemplate;
}
#Bean
public JmsListenerContainerFactory<?> topicListenerFactory(ConnectionFactory connectionFactory, DefaultJmsListenerContainerFactoryConfigurer configurer) {
DefaultJmsListenerContainerFactory factory = new DefaultJmsListenerContainerFactory();
configurer.configure(factory, connectionFactory);
factory.setPubSubDomain(true);//must follow the configurer.configure(), because it will set the factory as default as connectionFactory
factory.setConcurrency("1");
factory.setSubscriptionShared(true);
factory.setSubscriptionDurable(true);
factory.setSessionAcknowledgeMode(Session.CLIENT_ACKNOWLEDGE);
factory.setClientId(mqClientID);
return factory;
}
}
Is any missing config, or need to override the method to manage the Session object so that it can create shared-durable connection?
Finally I figured out why the program uses JMS1.1. It is because another part of the program load the AWS SQS library from pom, and this library is using JMS1.1(geronimo-jms_1.1_spec-1.1.1.jar) for ActiveMQ.
Therefore, I add exclusion in pom for the AWS SQS library to prevent it override the JMS2.0 library.

Spring configuration for Radis, how the beans get wired?

In the following example from Getting started guide of Spring how the container bean gets connectionFactory? Does Spring Boot supplies a connectionFactory on its own?
Getting Started Messaging with Spring Redis
There are 5 beans :
latch
receiver
listenerAdapter
template
container
latch gets created first. Then receiver because receiver constructor needs latch.Then listenerAdapter because it needs receiver.Both template and container need connectionFactory.
In the code I do not find any method with name connectionFactory and annotated with #Bean.
#SpringBootApplication
public class Application {
#Bean
RedisMessageListenerContainer container(RedisConnectionFactory connectionFactory,
MessageListenerAdapter listenerAdapter) {
RedisMessageListenerContainer container = new RedisMessageListenerContainer();
container.setConnectionFactory(connectionFactory);
container.addMessageListener(listenerAdapter, new PatternTopic("chat"));
return container;
}
#Bean
MessageListenerAdapter listenerAdapter(Receiver receiver) {
return new MessageListenerAdapter(receiver, "receiveMessage");
}
#Bean
Receiver receiver(CountDownLatch latch) {
return new Receiver(latch);
}
#Bean
CountDownLatch latch() {
return new CountDownLatch(1);
}
#Bean
StringRedisTemplate template(RedisConnectionFactory connectionFactory) {
return new StringRedisTemplate(connectionFactory);
}
}
It's in the classpath of your project, this is what spring boot does

Spring AMQP Transactional Processing and Retry

I am trying the Srpring AMQP features regarding transactional message processing.
I have the following setup - I have a message consumer that is annotated as #Transactional
#Transactional
public void handleMessage(EventPayload event) {
Shop shop = new Shop();
shop.setName(event.getName());
Shop savedShop = shopService.create(shop);
log.info("Created shop {} from event {}", shop, event);
}
In shopService.create I save the shop and send another message about the creation:
#Transactional(propagation = REQUIRED)
#Component
public class ShopService {
...
public Shop create(Shop shop) {
eventPublisher.publish(new EventPayload(shop.getName()));
return shopRepository.save(shop);
}
}
I want to achieve the following - the message sent in the create method should just go to the broker if the database action succeeded. If it fails the message is not sent and the received message is rolled back.
I also have a Retry configured - so I expect each message to be retried 3 times before it is rejected:
#Bean
public RetryOperationsInterceptor retryOperationsInterceptor() {
return RetryInterceptorBuilder.stateless()
.maxAttempts(3)
.backOffOptions(1000, 2.0, 10000)
.build();
}
I am observing the following behaviour:
When I configure the container as follows the message is retried 3 times but every time the message in shopService.create is sent to the broker:
#Bean
SimpleMessageListenerContainer messageListenerContainer(ConnectionFactory connectionFactory,
MessageListenerAdapter listenerAdapter) {
SimpleMessageListenerContainer container = new SimpleMessageListenerContainer();
container.setConnectionFactory(connectionFactory);
container.setQueueNames(testEventSubscriberQueue().getName());
container.setMessageListener(listenerAdapter);
container.setChannelTransacted(true);
container.setAdviceChain(new Advice[]{retryOperationsInterceptor()});
return container;
}
So I tried passing the PlatformTransactionManager to the container -
#Bean
SimpleMessageListenerContainer messageListenerContainer(ConnectionFactory connectionFactory,
MessageListenerAdapter listenerAdapter,
PlatformTransactionManager transactionManager) {
SimpleMessageListenerContainer container = new SimpleMessageListenerContainer();
container.setConnectionFactory(connectionFactory);
container.setQueueNames(testEventSubscriberQueue().getName());
container.setMessageListener(listenerAdapter);
container.setChannelTransacted(true);
container.setTransactionManager(transactionManager);
container.setAdviceChain(new Advice[]{retryOperationsInterceptor()});
return container;
}
Now the message sent in shopService.create is only send to the broker if the database transaction succeeded - which is what I want - but the message is retried indefinitely now - and not discarded after 3 retires as configured. But it seems that the backOff settings are applied - so there is some time between the retries.
The setup described does not really make sense from a business point of view - I am trying to understand and evaluate the transaction capabilities.
I am use spring-amqp 1.5.1.RELEASE
Thanks for any hints.
I had the same requirements, an #RabbitListener annotated with #Transactional, I wanted retry with backoff. It works even stateless with the following config:
#Bean
public RetryOperationsInterceptor retryOperationsInterceptor( ) {
return RetryInterceptorBuilder.stateless()
.maxAttempts( 3 )
.recoverer( new RejectAndDontRequeueRecoverer() )
.backOffOptions(1000, 2, 10000)
.build();
}
#Bean
public Jackson2JsonMessageConverter producerJackson2MessageConverter( ObjectMapper objectMapper ) {
Jackson2JsonMessageConverter jackson2JsonMessageConverter = new Jackson2JsonMessageConverter( objectMapper );
jackson2JsonMessageConverter.setCreateMessageIds( true );
return jackson2JsonMessageConverter;
}
#Bean
SimpleRabbitListenerContainerFactory rabbitListenerContainerFactory( ConnectionFactory connectionFactory,
PlatformTransactionManager transactionManager,
Jackson2JsonMessageConverter converter) {
SimpleRabbitListenerContainerFactory container = new SimpleRabbitListenerContainerFactory ();
container.setConnectionFactory(connectionFactory);
container.setChannelTransacted(true);
container.setTransactionManager(transactionManager);
container.setAdviceChain( retryOperationsInterceptor() );
container.setMessageConverter( converter );
return container;
}
To use stateless(), using RejectAndDontRequeueRecoverer was important because otherwise the retry will work but the consumer will then by default put the message back on the queue. Then the consumer will retrieve it again, applying the retry policy and then putting it back on the queue infinitely.

Resources