When using DirectMessageListenerContainer with consumersPerQueue property of 25, I noticed 25 rabbit channels get created per listener container's subscribed queue. The rabbit channel count quickly grows out of hand in our setup, as more queues are added to the listener container dynamically. We had to increase broker channel limit to accommodate the channel growth.
What is the relationship between channels and consumers in the DirectMessageListenerContainer. From my observations it appears to be 1 channel per consumer.
Does DirectMessageListenerContainer offer any channel pooling/recycling/rebalancing to keep channel growth under control. Specifically for queues that are mostly idle.
Does the simple SimpleMessageListenerContainer handle channels pooling differently, since it can dynamically resize the consumer count.
The DMLC uses a separate channel for each consumer.
No.
The SMLC uses one channel per concurrentConsumers; since 2.0, each channel is used for multiple consumers (when there is more than one queue listened to).
However dynamically adding or removing queues is much less efficient with the SMLC because the consumer(s) are canceled and re-created when changes are made.
Related
I have ActiveMQ Artemis. Producer generates 1000 messages and consumer one by one processing their. Now I want to process this queue with help of two consumers. I start new consumer and new messages are distributed between two runned consumers. My question: is it posible redistribute old messages between all started consumers?
Once messages are dispatched by the broker to a consumer then the broker can't simply recall them as the consumer may be processing them. It's up to the consumer to cancel the messages back to the queue (e.g. by closing its connection/session).
My recommendation would be to tune your consumerWindowSize (set on the client's URL) so that a suitable number of messages are dispatched to your consumers. The default consumerWindowSize is 1M (1024 * 1024 bytes). A smaller consumerWindowSize would mean that more clients would be able to receive messages concurrently, but it would also mean that clients would need to conduct more network round-trips to tell the broker to dispatch more messages when they run low. You'll need to run benchmarks to find the right consumerWindowSize value for your use-case and performance needs.
I can't find this information anywhere. I have two queues, #JmsListener(destination = "p1"), #JmsListener(destination = "p2"). How can I make sure I only process 1 message at a time, even though I am listening to 2 queues, and also how do I configure the polling of what queue I get messages from first, that is after processing a message I want to poll p1 first. Or do weighted polling: p1:90%, p2:10%. Etc.
Basically I am asking how to implement priority processing of messages for Spring. I'm using SQS which doesn't support priorities.
Use one of the JmsTemplate receive() or receiveAndConvert() methods instead of the message-driven model.
Use transactions if you want to ensure no message loss.
I have built an application which consists of one publisher, several queues and several consumers for each queue. Consumers on a queue (including the queue) share the channel. Other queues use a different channel. I am observing that for different queues, tasks are being worked on parallel but for a specific queue this is not happening. If I publish several messages at once to a specific queue, only one consumer works while the other ones wait until the work is ended. What should I do in order for consumers to work on parallel?
workers.each do |worker|
worker.on_delivery() do |delivery_info, metadata, payload|
perform_work(delivery_info, metadata, payload)
end
queue.subscribe_with(worker)
end
This is how I register all the consumers for a specific queue. The operation perform_work(_,_,_) is rather expensive and takes several seconds to complete.
RabbitMQ works off the back of the concept of channels, and channels are generally intended to not be shared between threads. Moreover, channels by default have a work thread pool size of one. A channel is an analog to a session.
In your case, you have multiple consumers sharing a queue and channel, and performing a long-duration job within the event handler for the channel.
There are two ways to work around this:
Allocate a channel per consumer, or
Set the work pool size of the channel on creation See this documentation
I would advocate 1 channel per consumer since it has a lower chance of causing unintended side-effects.
I am playing with the SimpleMessageListenerContainer that is subscribed to 3 different queues. The SimpleMessageListenerContainer has been configured with a RetryOperationsInterceptor that has a exponential back off policy.
My SimpleMessageListenerContainer has been configured with :
container.addQueueNames("news.politics","news.science","news.tech");
container.setMaxConcurrentConsumers(10);
container.setAdviceChain(new Advice[]{retryInterceptor});
If a message from one of the 3 queue thats been consumed enters an erroneous state resulting in an exception, the consumer triggers the exponential back of policy as predicted. However, I am noticing that the consumer STOPS doing round robin, to process messages on the other 2 queues. I was thinking since the consumer is set with "MaxConcurrentConsumer" of 10, the consumer will start to spawn consumer threads and round robin the rest of the queues.
Is this the normal behavior of SimpleMessageListenerContainer?
Can this behavior be adjusted?
For my use-case, would it be recommended to have one SimpleMessageListenerContainer per queue to keep them segregated? Perhaps come up with a decorator CompositeSimpleMessageListener container, that has internally a map of SimpleMessageListenerContainers per queue?
The algorithm to increase consumers is limited; if the retry interceptor suspends the consumer thread, it will prevent new consumers from being started.
Either increase the concurrentConsumers (to at least 3) or switch to the DirectMessageListenerContainer.
See choosing a container.
I was wondering what is the difference between a JMS Queue and JMS Topic.
ActiveMQ page says
Topics
In JMS a Topic implements publish and subscribe semantics. When you publish a message it goes to all the subscribers who are
interested - so zero to many subscribers will receive a copy of the
message. Only subscribers who had an active subscription at the time
the broker receives the message will get a copy of the message.
Queues
A JMS Queue implements load balancer semantics. A single message will be received by exactly one consumer. If there are no
consumers available at the time the message is sent it will be kept
until a consumer is available that can process the message. If a
consumer receives a message and does not acknowledge it before closing
then the message will be redelivered to another consumer. A queue can
have many consumers with messages load balanced across the available
consumers.
I want to have 'something' what will send a copy of the message to each subscriber in the same sequence as that in which the message was received by the ActiveMQ broker.
Any thoughts?
That means a topic is appropriate. A queue means a message goes to one and only one possible subscriber. A topic goes to each and every subscriber.
It is simple as that:
Queues = Insert > Withdraw (send to single subscriber) 1:1
Topics = Insert > Broadcast (send to all subscribers) 1:n
Topics are for the publisher-subscriber model, while queues are for point-to-point.
A JMS topic is the type of destination in a 1-to-many model of distribution.
The same published message is received by all consuming subscribers. You can also call this the 'broadcast' model. You can think of a topic as the equivalent of a Subject in an Observer design pattern for distributed computing. Some JMS providers efficiently choose to implement this as UDP instead of TCP. For topic's the message delivery is 'fire-and-forget' - if no one listens, the message just disappears. If that's not what you want, you can use 'durable subscriptions'.
A JMS queue is a 1-to-1 destination of messages. The message is received by only one of the consuming receivers (please note: consistently using subscribers for 'topic client's and receivers for queue client's avoids confusion). Messages sent to a queue are stored on disk or memory until someone picks it up or it expires. So queues (and durable subscriptions) need some active storage management, you need to think about slow consumers.
In most environments, I would argue, topics are the better choice because you can always add additional components without having to change the architecture. Added components could be monitoring, logging, analytics, etc.
You never know at the beginning of the project what the requirements will be like in 1 year, 5 years, 10 years. Change is inevitable, embrace it :-)
Queues
Pros
Simple messaging pattern with a transparent communication flow
Messages can be recovered by putting them back on the queue
Cons
Only one consumer can get the message
Implies a coupling between producer and consumer as it’s an one-to-one relation
Topics
Pros
Multiple consumers can get a message
Decoupling between producer and consumers (publish-and-subscribe pattern)
Cons
More complicated communication flow
A message cannot be recovered for a single listener
As for the order preservation, see this ActiveMQ page. In short: order is preserved for single consumers, but with multiple consumers order of delivery is not guaranteed.
If you have N consumers then:
JMS Topics deliver messages to N of N
JMS Queues deliver messages to 1 of N
You said you are "looking to have a 'thing' that will send a copy of the message to each subscriber in the same sequence as that in which the message was received by the ActiveMQ broker."
So you want to use a Topic in order that all N subscribers get a copy of the message.
TOPIC:: topic is one to many communication... (multipoint or publish/subscribe)
EX:-imagine a publisher publishes the movie in the youtub then all its subscribers will gets notification....
QUEVE::queve is one-to-one communication ...
Ex:-When publish a request for recharge it will go to only one qreciever ...
always remember if request goto all qreceivers then multiple recharge happened so while developing analyze which is fit for a application
Queue is JMS managed object used for holding messages waiting for subscribers to consume. When all subscribers consumed the message , message will be removed from queue.
Topic is that all subscribers to a topic receive the same message when the message is published.