I have a Rabbit Topic with multiple (say 2)subscribers which is running in a load balanced application server cluster ( say 3 ) .
So will the message will get delivered to all (2 X 3 ) subscribers of all listeners in a clustered environment or only 2 listeners ?
There's no such thing as a "topic" in rabbitmq (amqp).
The closest thing to a JMS topic for your scenario is a fanout exchange with 2 queues bound to it. Each queue gets a reference to a message sent to the exchange so both consumers (one per queue) gets a copy of the message.
If you have multiple consumers (e.g. 3) on each queue, the messages in that queue are distributed round-robin fashion to those consumers. Only one consumer per queue gets a message.
Related
I have ActiveMQ Artemis. Producer generates 1000 messages and consumer one by one processing their. Now I want to process this queue with help of two consumers. I start new consumer and new messages are distributed between two runned consumers. My question: is it posible redistribute old messages between all started consumers?
Once messages are dispatched by the broker to a consumer then the broker can't simply recall them as the consumer may be processing them. It's up to the consumer to cancel the messages back to the queue (e.g. by closing its connection/session).
My recommendation would be to tune your consumerWindowSize (set on the client's URL) so that a suitable number of messages are dispatched to your consumers. The default consumerWindowSize is 1M (1024 * 1024 bytes). A smaller consumerWindowSize would mean that more clients would be able to receive messages concurrently, but it would also mean that clients would need to conduct more network round-trips to tell the broker to dispatch more messages when they run low. You'll need to run benchmarks to find the right consumerWindowSize value for your use-case and performance needs.
We have an IBM MQ JMS queue and want to distribute the data into multiple consumers for load balancing. So if we write two JMS Clients to consume from same JMS queue what will happen? Will Messages be equally distributed across both consumers since one consumer will delete the data after it is read? Is there a possibility for data duplication, like if the same message is read by both consumers in a race condition?
My comments below are based on destructive get and not a browse get.
So if we write two JMS Clients to consume from same JMS queue what
will happen?
They will both consume messages.
Will Messages be equally distributed across both consumers since one
consumer will delete the data after it is read?
No. The "hot" consumer will be feed the next available message, assuming it is "getting" a message again before the next message arrives.
Is there a possibility for data duplication, like if the same message
is read by both consumers in a race condition?
Not if you are performing a destructive get (the default).
I have a Tibco EMS topic subscriber which I need to load balance among different instances. Each published message to the topic needs to be received by one (and only one) instance of each subscriber load balance group.
Just using global topics and balanced EMS connections (tcp://localhost:7222|tcp://localhost:7224) results in the same message received by all instances of each subscriber load balance group, producing duplicates.
Do you know any alternative for load balancing topic subscribers?
You can:
A) Bridge the topic to a queue and reconfigure your subscribers to read from the queue. Queues behave differently to topics in that a message is only obtained by one subscriber rather than all.
B) Create a number of durable subscribers on the topic with selectors that divide messages between the durables. E.g. If a message has a property 'id' that is sequentially increasing:
create durable topic DURABLENAME1 selector="(id - 2 * (id / 2)) = 0"
create durable topic DURABLENAME2 selector="(id - 2 * (id / 2)) = 1"
The selector is just a modulo so half the messages will go on one durable, and half on the other.
With EMS 8.0 new concept shared subscription is added with these only one subscription receives messages with the same subscription name go through the EMS user guide docs it may help you.
While both previous answers are valid, however the most natural approach would be to not use topics at all.
Using queues instead pf topics does the whole job (loadbalancing in round robin fashion).
I have configured Spring DefaultMessageListenerContainer as ActiveMQ consumer consuming messages from a queue. Let's call it "Test.Queue"
I have this code deployed in 4 different machines and all the machines are configured to the same ActiveMQ instance to process the messages from the same "Test.Queue" queue.
I set the max consumer size to 20 as soon as all the 4 machines are up and running, I see the number of consumers count against the queue as 80 (4 * max consumer size = 80)
Everything is fine when the messages produced and sent to the queue grows high.
When there are 1000's of messages and among the 80 consumers, let's say one of them is stuck it puts a freeze on Active MQ to stop sending messages to other consumers.
All messages are stuck in ActiveMQ forever.
As I have 4 machines with up to 80 consumers , I have no clue as to see which consumer failed to acknowledge.
I go stop and restart all the 4 machines and when I stop the machine that has the bad consumer which got stuck, then messages starts flowing again.
I don't know how to configure DefaultMessageListenerContainer to abandon the bad consumer and signal ActiveMQ immediately to start sending messages.
I was able to create the scenario even without Spring as follows:
I produced up to 5000 messages and sent them to the "Test.Queue" queue
I created 2 consumers (Consumer A, B) and in one consumer B's
onMessage() method, I put the thread to sleep for a long time (
Thread.sleep(Long.MAX_VALUE)) having the condition like when current time % 13 is 0 then put the thread to sleep.
Ran these 2 consumers.
Went to Active MQ and found that the queue has 2 consumers.
Both A and B are processing messages
At some point of time consumer B's onMessage() gets called and it puts the Thread to sleep when the condition of current time % 13 is 0 is satisified.
The consumer B is stuck and it can't acknowledge to the broker
I went back to Active MQ web console, still see the consumers as 2, but no messages are dequeued.
Now I created another consumer C and ran it to consume.
Only the consumer count in ActiveMQ went up to 3 from 2.
But Consumer C is not consuming anything as the broker failed sending any messages holding them all as it is still waiting for consumer B to acknowledge it.
Also I noticed Consumer A is not consuming anything
I go and kill consumer B , now all messages are drained.
Let's say A, B, C are managed by Spring's DefaultMessageListenerContainer, how do I tweak Spring DefaultMessageListenerContainer to take that bad consumer off the pool (in my case consumer B) after it failed to acknowledge for X number of seconds, acknowledge the broker immediately so that the broker is not holding onto messages forever.
Thanks for your time.
Appreciate if I get a solution to this problem.
here are a few options to try...
set the queue prefetch to 0 to promote better distribution across consumers and reduce 'stuck' messages on specific consumers. see http://activemq.apache.org/what-is-the-prefetch-limit-for.html
set "?useKeepAlive=false&wireFormat.maxInactivityDuration=20000" on the connection to timeout the slow consumer after a specified inactive time
set the queue policy "slowConsumerStrategy->abortSlowConsumer"...again to timeout a slow consumer
<policyEntry ...
...
<slowConsumerStrategy>
<abortSlowConsumerStrategy />
</slowConsumerStrategy>
...
</policyEntry>
I was wondering what is the difference between a JMS Queue and JMS Topic.
ActiveMQ page says
Topics
In JMS a Topic implements publish and subscribe semantics. When you publish a message it goes to all the subscribers who are
interested - so zero to many subscribers will receive a copy of the
message. Only subscribers who had an active subscription at the time
the broker receives the message will get a copy of the message.
Queues
A JMS Queue implements load balancer semantics. A single message will be received by exactly one consumer. If there are no
consumers available at the time the message is sent it will be kept
until a consumer is available that can process the message. If a
consumer receives a message and does not acknowledge it before closing
then the message will be redelivered to another consumer. A queue can
have many consumers with messages load balanced across the available
consumers.
I want to have 'something' what will send a copy of the message to each subscriber in the same sequence as that in which the message was received by the ActiveMQ broker.
Any thoughts?
That means a topic is appropriate. A queue means a message goes to one and only one possible subscriber. A topic goes to each and every subscriber.
It is simple as that:
Queues = Insert > Withdraw (send to single subscriber) 1:1
Topics = Insert > Broadcast (send to all subscribers) 1:n
Topics are for the publisher-subscriber model, while queues are for point-to-point.
A JMS topic is the type of destination in a 1-to-many model of distribution.
The same published message is received by all consuming subscribers. You can also call this the 'broadcast' model. You can think of a topic as the equivalent of a Subject in an Observer design pattern for distributed computing. Some JMS providers efficiently choose to implement this as UDP instead of TCP. For topic's the message delivery is 'fire-and-forget' - if no one listens, the message just disappears. If that's not what you want, you can use 'durable subscriptions'.
A JMS queue is a 1-to-1 destination of messages. The message is received by only one of the consuming receivers (please note: consistently using subscribers for 'topic client's and receivers for queue client's avoids confusion). Messages sent to a queue are stored on disk or memory until someone picks it up or it expires. So queues (and durable subscriptions) need some active storage management, you need to think about slow consumers.
In most environments, I would argue, topics are the better choice because you can always add additional components without having to change the architecture. Added components could be monitoring, logging, analytics, etc.
You never know at the beginning of the project what the requirements will be like in 1 year, 5 years, 10 years. Change is inevitable, embrace it :-)
Queues
Pros
Simple messaging pattern with a transparent communication flow
Messages can be recovered by putting them back on the queue
Cons
Only one consumer can get the message
Implies a coupling between producer and consumer as it’s an one-to-one relation
Topics
Pros
Multiple consumers can get a message
Decoupling between producer and consumers (publish-and-subscribe pattern)
Cons
More complicated communication flow
A message cannot be recovered for a single listener
As for the order preservation, see this ActiveMQ page. In short: order is preserved for single consumers, but with multiple consumers order of delivery is not guaranteed.
If you have N consumers then:
JMS Topics deliver messages to N of N
JMS Queues deliver messages to 1 of N
You said you are "looking to have a 'thing' that will send a copy of the message to each subscriber in the same sequence as that in which the message was received by the ActiveMQ broker."
So you want to use a Topic in order that all N subscribers get a copy of the message.
TOPIC:: topic is one to many communication... (multipoint or publish/subscribe)
EX:-imagine a publisher publishes the movie in the youtub then all its subscribers will gets notification....
QUEVE::queve is one-to-one communication ...
Ex:-When publish a request for recharge it will go to only one qreciever ...
always remember if request goto all qreceivers then multiple recharge happened so while developing analyze which is fit for a application
Queue is JMS managed object used for holding messages waiting for subscribers to consume. When all subscribers consumed the message , message will be removed from queue.
Topic is that all subscribers to a topic receive the same message when the message is published.