I am playing with the SimpleMessageListenerContainer that is subscribed to 3 different queues. The SimpleMessageListenerContainer has been configured with a RetryOperationsInterceptor that has a exponential back off policy.
My SimpleMessageListenerContainer has been configured with :
container.addQueueNames("news.politics","news.science","news.tech");
container.setMaxConcurrentConsumers(10);
container.setAdviceChain(new Advice[]{retryInterceptor});
If a message from one of the 3 queue thats been consumed enters an erroneous state resulting in an exception, the consumer triggers the exponential back of policy as predicted. However, I am noticing that the consumer STOPS doing round robin, to process messages on the other 2 queues. I was thinking since the consumer is set with "MaxConcurrentConsumer" of 10, the consumer will start to spawn consumer threads and round robin the rest of the queues.
Is this the normal behavior of SimpleMessageListenerContainer?
Can this behavior be adjusted?
For my use-case, would it be recommended to have one SimpleMessageListenerContainer per queue to keep them segregated? Perhaps come up with a decorator CompositeSimpleMessageListener container, that has internally a map of SimpleMessageListenerContainers per queue?
The algorithm to increase consumers is limited; if the retry interceptor suspends the consumer thread, it will prevent new consumers from being started.
Either increase the concurrentConsumers (to at least 3) or switch to the DirectMessageListenerContainer.
See choosing a container.
Related
When using DirectMessageListenerContainer with consumersPerQueue property of 25, I noticed 25 rabbit channels get created per listener container's subscribed queue. The rabbit channel count quickly grows out of hand in our setup, as more queues are added to the listener container dynamically. We had to increase broker channel limit to accommodate the channel growth.
What is the relationship between channels and consumers in the DirectMessageListenerContainer. From my observations it appears to be 1 channel per consumer.
Does DirectMessageListenerContainer offer any channel pooling/recycling/rebalancing to keep channel growth under control. Specifically for queues that are mostly idle.
Does the simple SimpleMessageListenerContainer handle channels pooling differently, since it can dynamically resize the consumer count.
The DMLC uses a separate channel for each consumer.
No.
The SMLC uses one channel per concurrentConsumers; since 2.0, each channel is used for multiple consumers (when there is more than one queue listened to).
However dynamically adding or removing queues is much less efficient with the SMLC because the consumer(s) are canceled and re-created when changes are made.
I can't find this information anywhere. I have two queues, #JmsListener(destination = "p1"), #JmsListener(destination = "p2"). How can I make sure I only process 1 message at a time, even though I am listening to 2 queues, and also how do I configure the polling of what queue I get messages from first, that is after processing a message I want to poll p1 first. Or do weighted polling: p1:90%, p2:10%. Etc.
Basically I am asking how to implement priority processing of messages for Spring. I'm using SQS which doesn't support priorities.
Use one of the JmsTemplate receive() or receiveAndConvert() methods instead of the message-driven model.
Use transactions if you want to ensure no message loss.
I have built an application which consists of one publisher, several queues and several consumers for each queue. Consumers on a queue (including the queue) share the channel. Other queues use a different channel. I am observing that for different queues, tasks are being worked on parallel but for a specific queue this is not happening. If I publish several messages at once to a specific queue, only one consumer works while the other ones wait until the work is ended. What should I do in order for consumers to work on parallel?
workers.each do |worker|
worker.on_delivery() do |delivery_info, metadata, payload|
perform_work(delivery_info, metadata, payload)
end
queue.subscribe_with(worker)
end
This is how I register all the consumers for a specific queue. The operation perform_work(_,_,_) is rather expensive and takes several seconds to complete.
RabbitMQ works off the back of the concept of channels, and channels are generally intended to not be shared between threads. Moreover, channels by default have a work thread pool size of one. A channel is an analog to a session.
In your case, you have multiple consumers sharing a queue and channel, and performing a long-duration job within the event handler for the channel.
There are two ways to work around this:
Allocate a channel per consumer, or
Set the work pool size of the channel on creation See this documentation
I would advocate 1 channel per consumer since it has a lower chance of causing unintended side-effects.
I have a scenario where i am creating the predefined consumers for queues in DirectMessageListenerContainer bean as follows:
#Bean
public DirectMessageListenerContainer directMessageListenerContainer(ConnectionFactory conn, DynamicTenantListenerCreator listener){
DirectMessageListenerContainer factory = new DirectMessageListenerContainer();
factory.setAcknowledgeMode(AcknowledgeMode.MANUAL);
factory.setConnectionFactory(conn);
factory.setConsumersPerQueue(5);
factory.setMessageListener(listener);
return factory;
}
As per my understanding, this will assign 5 consumers to each queue working on separate channel. for eg, if i add 5 queues to this bean so the total number of consumes will be 25 working on 5 different channels. My questions are:
What will be the impact of idle consumers on CPU in Cloud Foundry. In case some queues are having less messages, will the performance be impacted for the application as there will be some idle consumers?
Is there any way to define specific number of consumers of each queue separately?
Thanks in advance
Idle consumers will use minimal resources (just a small amount of memory, no cpu). No, you can't specify consumer counts for each queue, unless you have a separate container for each.
We are using ActiveMq in our application. We need to implement priority consumer mechanism on the queue. We have multiple consumers and we want to treat them as master/slave. So, the master consumer would have higher priority and will consume all messages and once master gets down then the consumer (with higher priority) will consume all requests from the queue.
To Implement this scenario, we came across the mechanism of priority queues. So, we initialized the queues by this way:
javax.jms.Queue queue = queueSession.createQueue("myQueue" + "?consumer.priority=" + 127);
and remaining slaves nodes were assigned lesser priorities.
But this mechanism is not working expectedly, sometimes it works fine that we get all requests on the consumer with the highest priority but sometimes consumer with lesser priority also start consuming messages while the consumer with the highest priority is working fine.
We have tried it with consumer.exclusive=true option as well, with no luck.
Note: We are running all components (ActiveMQ, producer and consumer application) locally on the same machine for now, so no network delays involved. And we are running consumers with default prefetch policy.
Is there any other approach to implement this scenario using activeMq or are we missing any configuration.