Requirement:
1)I need to fetch data stored in Mongo DB through Java Application and using topic exchange & binding keys, created 3 queues on RabbitMQ. I have implemented everything up to this point.
The problem starts from the 2nd point onwards.
2) When the messages should be available to multiple consumers from all the 3 queues. But when first consumer consumes the messages from 3 queues it will not be available for the rest of the consumers. How to make messages highly available to multiple consumers.
Is there any ways to achieve this or is this requirement has any alternate solutions to it.
All your consumers must provide their own unique queue and bind it to the same exchange.
There is no such a Topic abstraction in AMQP, like it is with JMS.
Even if we can publish message through the topic or fanout exchange, the message will be placed to the queue as single entry, so only one consumer will be able to pick it up from the the.
The config for my proposition may look like:
<queue id="commandQueue" name="#{node.id}.command"
auto-delete="true"/>
<fanout-exchange name="commandsExchange">
<bindings>
<binding queue="commandQueue"/>
</bindings>
</fanout-exchange>
<amqp:inbound-channel-adapter id="commandConsumer"
queue-names="#{commandQueue.name}"
channel="commandChannel"/>
With that all my application instances bind their unique queue (based on the node.id abstraction) to the same commandsExchange. And the published message to the commandsExchange will be delivered to all my nodes.
auto-delete="true" helps me to avoid extra messages for my queue, if node is dead.
HTH
Related
We have one topic with one partition due to ordering of message requirements. We have two consumers running on different servers with same set of configurations i.e. groupId, consumerId, consumerGroup. i.e.
1 Topic -> 1 Partition -> 2 Consumers
When we deploy consumers same code is deployed on both the servers. Noticed when a message comes we see both the consumers are consuming message rather than only one processing. Reason having consumers running on two separate servers is if one server crashes at least other can continue processing messages. But looks like if both up both consuming messages. Reading Kafka docs it says if we have more consumers than partitions then some stay idle don't see that happening. Anything we are missing on configuration side apart from consumerId & groupId. Thanks
As #Gary Russel said, as long as the two consumer instances have their own consumer group, they will consume every event that is written to the topic. Just put them into the same consumer-group. You can provide a consumer-group-id in the consumer.properties.
I have implemented the example from the RabbitMQ website:
RabbitMQ Example
I have expanded it to have an application with a button to send a message.
Now I started two consumer on two different computers.
When I send the message the first message is sent to computer1, then the second message is sent to computer2, the thrid to computer1 and so on.
Why is this, and how can I change the behavior to send each message to each consumer?
Why is this
As noted by Yazan, messages are consumed from a single queue in a round-robin manner. The behavior your are seeing is by design, making it easy to scale up the number of consumers for a given queue.
how can I change the behavior to send each message to each consumer?
To have each consumer receive the same message, you need to create a queue for each consumer and deliver the same message to each queue.
The easiest way to do this is to use a fanout exchange. This will send every message to every queue that is bound to the exchange, completely ignoring the routing key.
If you need more control over the routing, you can use a topic or direct exchange and manage the routing keys.
Whatever type of exchange you choose, though, you will need to have a queue per consumer and have each message routed to each queue.
you can't it's controlled by the server check Round-robin dispatching section
It decides which consumer turn is. i'm not sure if there is a set of algorithms you can pick from, but at the end server will control this (i think round robin algorithm is default)
unless you want to use routing keys and exchanges
I would see this more as a design question. Ideally, producers should create the exchanges and the consumers create the queues and each consumer can create its own queue and hook it up to an exchange. This makes sure every consumer gets its message with its private queue.
What youre doing is essentially 'worker queues' model which is used to distribute tasks among worker nodes. Since each task needs to be performed only once, the message is sent to only one node. If you want to send a message to all the nodes, you need a different model called 'pub-sub' where each message is broadcasted to all the subscribers. The following link shows a simple pub-sub tutorial
https://www.rabbitmq.com/tutorials/tutorial-three-python.html
If I have the 3 queues of priority 1,2 & 3 respectively. I want my consumer to consume first from queue withe priority 1, then 2 & so on. If in case queue with higher priority is empty, the consumer can consume from the queue with lower priority.
Is it possible to achieve from JMS or ActiveMQ or any other way?How?
You'd have to control that logic yourself using this method. To ActiveMQ, or any other JMS provider, you are just using another queue.
However, you can use a single queue for message priority. There are a couple different ways on how to do this as described in the documentation.
If you want your consumer to be as simple as possible then have the broker figure out the priority. Otherwise you'll need to mess with multiple consumers or inefficient single consumer logic with selectors to consume.
In both cases, your producer will just need to be smart enough to set the JMSPriority header to whatever priority the logic says it should be.
The only downside really is the fact that you have a broker side config to set up for that queue specifically rather than everything being automatic.
What can be the best way to aggregate messages from many different sources (actually queues/topics) into a single queue/topic and then consume it. I am trying to design an application to receive messages from different topics in JMS using weblogic.
You could write your own "aggregator" as a stand-alone Java application:
For each queue/topic have a reader in its own thread.
Each reader sends its received message again on a "aggregate queue".
Have another thread to listen on the "aggregate queue".
As a variation, you could use a JVM Queue (like java.util.concurrent.ArrayBlockingQueue) as the "aggregate queue". This is faster, does not require another MQ queue, does not need network bandwidth, but it's not persistent.
Another idea is to use a "Message driven bean (MDB)" for each incoming queue/topic:
Again, each of these MDBs just reads the message and resends it to the "aggregate queue".
Have another MDB listening on the "aggregate queue".
A few suggestions on quality requirements. I belive you have to consider them.
They will be highly relate with your technical solution.
is that message loss acceptable?
client ack could be considered.
e.g. A memory queue sit in middle, e.g. incoming queue1...n -> ArrayBlockingQueue in memory -> outgoing queue. The data in the ArrayBlockingQueue , will lost when app crash.
is that message duplicate acceptable for the single outgoing queue?
I would suggest yes.
Set applicable level PossibleDuplicateFlag to make the client aware of that.
how fast the incoming messages per second on the diff incoming queue?
one queue session has only a uniqe thread. Performance has to be considered in advance.
I was wondering what is the difference between a JMS Queue and JMS Topic.
ActiveMQ page says
Topics
In JMS a Topic implements publish and subscribe semantics. When you publish a message it goes to all the subscribers who are
interested - so zero to many subscribers will receive a copy of the
message. Only subscribers who had an active subscription at the time
the broker receives the message will get a copy of the message.
Queues
A JMS Queue implements load balancer semantics. A single message will be received by exactly one consumer. If there are no
consumers available at the time the message is sent it will be kept
until a consumer is available that can process the message. If a
consumer receives a message and does not acknowledge it before closing
then the message will be redelivered to another consumer. A queue can
have many consumers with messages load balanced across the available
consumers.
I want to have 'something' what will send a copy of the message to each subscriber in the same sequence as that in which the message was received by the ActiveMQ broker.
Any thoughts?
That means a topic is appropriate. A queue means a message goes to one and only one possible subscriber. A topic goes to each and every subscriber.
It is simple as that:
Queues = Insert > Withdraw (send to single subscriber) 1:1
Topics = Insert > Broadcast (send to all subscribers) 1:n
Topics are for the publisher-subscriber model, while queues are for point-to-point.
A JMS topic is the type of destination in a 1-to-many model of distribution.
The same published message is received by all consuming subscribers. You can also call this the 'broadcast' model. You can think of a topic as the equivalent of a Subject in an Observer design pattern for distributed computing. Some JMS providers efficiently choose to implement this as UDP instead of TCP. For topic's the message delivery is 'fire-and-forget' - if no one listens, the message just disappears. If that's not what you want, you can use 'durable subscriptions'.
A JMS queue is a 1-to-1 destination of messages. The message is received by only one of the consuming receivers (please note: consistently using subscribers for 'topic client's and receivers for queue client's avoids confusion). Messages sent to a queue are stored on disk or memory until someone picks it up or it expires. So queues (and durable subscriptions) need some active storage management, you need to think about slow consumers.
In most environments, I would argue, topics are the better choice because you can always add additional components without having to change the architecture. Added components could be monitoring, logging, analytics, etc.
You never know at the beginning of the project what the requirements will be like in 1 year, 5 years, 10 years. Change is inevitable, embrace it :-)
Queues
Pros
Simple messaging pattern with a transparent communication flow
Messages can be recovered by putting them back on the queue
Cons
Only one consumer can get the message
Implies a coupling between producer and consumer as it’s an one-to-one relation
Topics
Pros
Multiple consumers can get a message
Decoupling between producer and consumers (publish-and-subscribe pattern)
Cons
More complicated communication flow
A message cannot be recovered for a single listener
As for the order preservation, see this ActiveMQ page. In short: order is preserved for single consumers, but with multiple consumers order of delivery is not guaranteed.
If you have N consumers then:
JMS Topics deliver messages to N of N
JMS Queues deliver messages to 1 of N
You said you are "looking to have a 'thing' that will send a copy of the message to each subscriber in the same sequence as that in which the message was received by the ActiveMQ broker."
So you want to use a Topic in order that all N subscribers get a copy of the message.
TOPIC:: topic is one to many communication... (multipoint or publish/subscribe)
EX:-imagine a publisher publishes the movie in the youtub then all its subscribers will gets notification....
QUEVE::queve is one-to-one communication ...
Ex:-When publish a request for recharge it will go to only one qreciever ...
always remember if request goto all qreceivers then multiple recharge happened so while developing analyze which is fit for a application
Queue is JMS managed object used for holding messages waiting for subscribers to consume. When all subscribers consumed the message , message will be removed from queue.
Topic is that all subscribers to a topic receive the same message when the message is published.