I have a single AMQ queue that receives simple messages with string body. Consider I'm sending CLSIDs as message bodies. CLSIDs could be not unique, but I'd like to reject all messages with not unique bodies and keep only single instance of such messages in the queue. Is there any simple way to do it?
Currently I'm using a workaround. Messages from the queue are consumed by some processor that tries to insert bodies into a simple DB table with UNIQUE constraint applied to message_body field. If processor inserts the messages succesfuly - it's assigned to exchange.out.body and sent to other queue. If ConstraintViolationException is thrown - nothing is resent to other queue.
I would like to know does AMQ support something similar out of the box?
I believe you can write an interceptor for activemq where you can perform certain actions on messages. Check out: http://activemq.apache.org/interceptors.html
That being said, in my personal opinion this is bad practice. ActiveMQ is a messaging system which should only be responssible for transport of the message. All logic can beter be performed using your application ( either make sure the sender cannot send the same message more then once OR , create an intermediate consumer which indeed matches the received body with a database that contains already seen message bodies BEFORE, routing the message to the actual receiver queue)
Related
I am putting a message containing string data to rabbitmq queue.
Message publishing is called as a part of a service and the service can be called with same data (data goes to the queue) multiple times, thus chances for having duplicated data in the queue is very likely.
We have issues with this as the consumer code is inserting this data to table where this data is primary key. Consumer will be called from 4 different nodes simultaneously thus chances for having consumers consuming same data (from different messages) can happen.
I want to know if rabbitMQ publishing has any way to avoid message duplication.
Read "define a property "x-unique-message-code" to compare them is an easy and simple way" , but don't know how to do it.
I am using spring-amqp
Any help is highly appreciated.
Thank you
There is a good article from RabbitMQ about reliability: https://www.rabbitmq.com/reliability.html
There is a note like:
In the event of network failure (or a node crashing), messages can be duplicated, and consumers must be prepared to handle them. If possible, the simplest way to handle this is to ensure that your consumers handle messages in an idempotent way rather than explicitly deal with deduplication.
For this purpose the message to produce can be supplied with a messageId property.
I have implemented the example from the RabbitMQ website:
RabbitMQ Example
I have expanded it to have an application with a button to send a message.
Now I started two consumer on two different computers.
When I send the message the first message is sent to computer1, then the second message is sent to computer2, the thrid to computer1 and so on.
Why is this, and how can I change the behavior to send each message to each consumer?
Why is this
As noted by Yazan, messages are consumed from a single queue in a round-robin manner. The behavior your are seeing is by design, making it easy to scale up the number of consumers for a given queue.
how can I change the behavior to send each message to each consumer?
To have each consumer receive the same message, you need to create a queue for each consumer and deliver the same message to each queue.
The easiest way to do this is to use a fanout exchange. This will send every message to every queue that is bound to the exchange, completely ignoring the routing key.
If you need more control over the routing, you can use a topic or direct exchange and manage the routing keys.
Whatever type of exchange you choose, though, you will need to have a queue per consumer and have each message routed to each queue.
you can't it's controlled by the server check Round-robin dispatching section
It decides which consumer turn is. i'm not sure if there is a set of algorithms you can pick from, but at the end server will control this (i think round robin algorithm is default)
unless you want to use routing keys and exchanges
I would see this more as a design question. Ideally, producers should create the exchanges and the consumers create the queues and each consumer can create its own queue and hook it up to an exchange. This makes sure every consumer gets its message with its private queue.
What youre doing is essentially 'worker queues' model which is used to distribute tasks among worker nodes. Since each task needs to be performed only once, the message is sent to only one node. If you want to send a message to all the nodes, you need a different model called 'pub-sub' where each message is broadcasted to all the subscribers. The following link shows a simple pub-sub tutorial
https://www.rabbitmq.com/tutorials/tutorial-three-python.html
I need to modify some of the messages that are already present in the queues.
Can it be possible to modify the message and send new modified message on the same queue for processing.
If yes, what would be the best approach for this.
Thanks.
The short answer is no. In JMS messages are immutable once they have been sent. If you find you need to modify messages its recommended that you create a consumer with some selector which matches the messages you wish to update, consume them and send new modified messages, either to another queue or if you are careful, back to the original queue.
ref link Can I modify messages on a queue?
my opinion is if you are consume the message using selector like correlationId,then consume the particular message from queue and again post the new message with same correlationId of previous message.but this will depends up on your requirements.
I have many instances of my client application. These clients send requests to a server application via messaging and receive a reply. Normally the reply would be sent using a temporary queue.
Unfortunately I have to use the Stomp protocol which has no concept of temporary queues or topics. (Although the message broker has)
What's the best way to ensure only the original requestor receives the reply? Are there any best-practices for this unfortunate situation?
The customary solution when several requestors listen for replies on the same queue is to use correlation IDs to select messages. On the client side it looks like this:
Place a message on the request queue and commit.
Retrieve the JMSMessageID from the outbound message (the value is determined by the broker and updates the message object as a result of the send).
Receive a message from the reply queue specifying the JMSMessageID from the outbound message as the correlation ID in the selector.
Process and commit.
On the server side it looks like this:
Receive a message under syncpoint.
Process the request and prepare the response.
Set the JMSCorrelationID on the response to the value of JMSMessageID from the request.
Send the message.
Commit.
The consumer would set the selector something like this: activemq.selector:JMSCorrelationID=.
Since the broker creates a message ID that is supposed to be globally unique, the pattern of using it as the correlation ID prevents collisions that are possible when each requestor is allowed to specify it's own value.
The best way to implement this pattern with JMS (that I've found, anyway) is to create a pre-configured topic for the response messages, and use correlation selectors on the response message so that the client can get the correct one.
In more detail, this means setting a random ID on the request message (using setJMSCorrelationID()), and putting that message on the request Queue. The consumer of that request message processes it, creates the response message, sets the same correlation ID on the response message, and puts it on the response Topic. The client, meanwhile, is listening on the response topic with a selector expression which specifies the correlation ID that it's expecting.
The danger is that the response message is sent before the client can get around to listening for it, although that's probably unlikely. You can try using a pre-configured Queue for the responses rather than a topic, but I've found that topics tend to work more reliably (my JMS provider of choice is HornetQ - your mileage may vary).
All this is tell me that JMS is a very poor fit for the request/response model. The API just doesn't support it properly. This is hardly surprising, since that was never a use-case for JMS.
Something like a compute grid (Terracotta, Gigaspaces, Infinispan, etc) would likely yield better results, but that's not really an option for you.
I have a design query regarding queues. My scenario is as follows:
I have to use a messaging system, with single producer and multiple consumers (asynchronous). The producer pushes different types of messages into the messaging system. Depending upon the message type, that particular consumer has to consume that message. (Each consumer is running on a different server). If one consumer is down and a message comes for that consumer, it will be in the messaging system only. If I use a message queue, the message in the queue will block the next messages that can be consumed by the other consumers. Are queues suitable for handling this kind of situation? Or do we need to go for a topic?
Whether you use a queue or a topic should depend on whether there an instance where multiple consumers must process the same message. If that is the case then a topic is required do generate that one-to-many pattern.
On the other hand, if any one message will only ever be consumed by one consumer, then you can use a queue or topic and the consumers specify the message type as a JMS selector. In this way, all consumers can listen on the same queue and each selects a different subset of messages. In the event one application is not there, it's messages do not "block the next messages that can be consumed by the other consumers" but rather they just stack up in the queue and other consumers still receive their messages based on selection criteria.
Please also realize that queues are lightweight constructions and you can easily have one queue per consumer. Typically, things providing a service listen on a well-known queue and each queue represents a different function of the service or a different service. Thus there may be many service input queues. Similarly, reply messages are generally uniquely addressed to the application instance that made the request and go to a unique, often dynamic, reply-to queue. Both of these implementations I have described lead to a separation of traffic across queues rather than pooling different message types into the same queue. Since JMS selectors always impart an additional processing cost, using more queues is generally more performant than selecting many types of message from the same queue.
I am responding to your question about selectors in the comment section here since I have more space and can put links in...
Section 3.8.1 of the JMS 1.1 spec states:
A JMS message selector allows a client to specify, by message header, the
messages it’s interested in. Only messages whose headers and properties
match the selector are delivered. The semantics of not delivered differ a bit
depending on the MessageConsumer being used. See Section 5.8,
“QueueReceiver,” and Section 6.11, “TopicSubscriber,” for more details.
Message selectors cannot reference message body values.
A message selector matches a message if the selector evaluates to true when
the message’s header field and property values are substituted for their
corresponding identifiers in the selector.
As noted above, selectors can be on fields that are implicit in the message such as MsgID or CorrelationID or thsey can be on fields specifically set by the message producer such as a message property. Either way, the client must specify the value of any selectors used by the message consumer.