I'm looking into a message queue solution where some messages need to be delivered without delay, and other messages need to be delivered at a specified time. The delay is anywhere from hours to a week or two. I have access to a JMS message Queue, but I'm questioning whether it is a good idea to put messages on the queue with long delays.
Is delaying messages a common practice?
Is using the QueueBrowser to peek at the messages and cherry picking the messages at the right time a viable solution (assuming the message as the delivery date in it)?
Is there another solution (other than putting the messages in the database with a time stamp) and periodically querying the database?
JMS 2.0 supports message delaying; see the spec, section 7.9: You can call setDeliveryDelay on the JMSProducer with the number of milliseconds you want messages to be delayed. (Note that, confusing as it is, you can not use the setJMSDeliveryTime method on the Message object.) In JMS 1.1, some JMS implementations support proprietary headers for the same effect.
It's a quite common practice, but it has a major drawback in practical use, when the delay is longer: There's no (standardized) way to access the delayed messages: The QueueBrowser doesn't return them until their time has come. If you need more control, you're better off with polling a database.
Related
Due to some reasons, we're not allowed to use more than one queue for our LOB.
So, we have set a very large prefetch number, and the consumer of that single queue will be in charge of dispatching to other in memory queues according to some message properties. Other background tasks will fetch the message from the in memory queues and handle it.
To avoid loss of messages, is it possible to manual ack the message until the background task finishes handling the message?
MassTransit handles message acknowledgement, there is no way to work around it. Once the message consumer completes, it will be acknowledged. Messages remain in the queue until they are acknowledged. There is no way messages could be lost using this approach.
Also, your approach of using a single transport queue and then having a bunch of in-memory queues is an incredibly poor (terrible, horrible, worst possible thing ever perhaps) design choice due to a really bad rule about queue usage. MassTransit can actually do all of that dispatch for you with multiple consumers on the receive endpoint for the various message types. But a single queue is still a really, really bad idea.
I am using camel to integrate with ActiveMQ JMS. I am receiving prices for products on this queue. I am using JMSXGroupID on productId to ensure ordering across a productId. Now if I fail to process this message I move it to a DeadLetterQueue. This could be because of a connection error on a dependent service or because of error with the message itself.
In case of the former I would have to manually remove it from the DLQ and put it back into the JMS queue.
Now the problem is that I dont know if any other message on that groupId has been received and processed or not. And hence unsidelining from DLQ will disrupt the order. On the other hand if I dont unsideline it and no other message has been received the product Id will not get the correct price.
1 solution that I have in mind is to use a fast key-value store(Redis) to store the last messageId or JMSTimestamp against a productId(message group). This is updated everytime I dequeue a message. Any other solution for this?
Relying on message order in JMS is a risky business - at best.
The best thing to do is to make the receiver handle messages out of sequence as a special case (but may take advantage message order during normal operation).
You may also want to distinguish between two errors: posion messages and temporary connection problems, maybe even use two different error queues for them. In the case of a posion message (invalid payload etc.) then there is nothing you can really do about it except starting a bug investigation. In such cases, you can probably send along "something else", such as dummy message to not interfere with order.
For the issues with connection problems, you can have another strategy - ActiveMQ Redelivery Policies. If there is network trouble, it's usually no use in trying to process the second message until the first has been handled. A Redelivery Policy ensures that (given you have a single consumer, that is). There is another question at SO where the poster actually has a solution to your problem and wants to avoid it. Read it. :)
I plan to have persistent message Queues based on some implementation of AMQP and JMS API. I would like to know whether is ok (from architectural point of view) to have messages staying in the queues for hours. A day is max.
I plan to use the message broker as another persistence layer basically. Is this viable?
The technologies that I am evaluating are ActiveMQ, RabbitMQ or qupid.
I plan to use the message broker as another persistence layer
basically. Is this viable?
The broker's persistence mechanism for message retention is usually file-based, or JDBC; either one will work. It is viable? Sure, its a feature of the broker, nothing wrong with using it for the intended purpose, assuming temporary message retention is your goal; 1 day is not a big deal.
But if you're planning to retain messages for 1 day, or more, I recommend doing some calculations based on average message size and total messages per day that may end up sitting in a queue. Queue depth, by default, is usually a low number, like 10Mb, and if exceeded, the broker will probably drop subsequent messages; you want to prevent this from happening. Vendors handle this differently, so check with RabbitMq and ActiveMQ for specifics and what configuration parameters are used to control depth. I know SonicMq has what's known as the "DeadMessage" queue, a destination for expired or undeliverable messages; other products might have something similar.
It's OK to have persistent queues, and it's OK if messages are hanging around in the queues: Clients might be disconnected because of updates, network problems etc. That's one benefit of queues to decouple sender from receiver, and the queue is the buffer. However these use cases are not the normal mode of operation, it's rather an exceptional situation.
Using a messaging broker as "another persistence layer" is technically speaking possible, but in this case a database is probably more suitable, because quick message delivery/messaging and long term storage/database are different tools/scenarios. So ask yourself the question: Is it still messaging or is it already a database?
If in your use case the normal message delay (= period between sending and reception) is always beyond an hour, a database might be better, because JMS selectors are normally slower and less comfortable than database queries using where clauses.
There is another aspect: Consider the need for an online backup of your messages in a JMS provider, especially in a HA cluster mode. It might be easier to do this using a database.
I am looking looking for a message queue with these requirements. Couldn't find it; maybe the closest was the rabbitmq-lvc plugin (but I need the first value in the line to stick and stay in front).
Would anyone know a technology to support these?
message queue is FIFO
if a duplicate message is being enqueued, the message queue itself either rejects or drops it.
For example, producers put these three messages (each with a discriminator value) into the queue in this sequence: M1(discriminator=7654), M2(discriminator=2435), M3(discriminator=7654).
Now I want the message queue to see that M3 has the same discriminator value as M1 and thus drop/reject M3. Consumers receive only: M1, M2.
Thanks
Tom
I don't know the other transports but I know that WebSphere MQ doesn't do this and I believe that the explanation why would apply broadly across the category. I'd be very surprised to find that any messaging transport actually provides this. Here are a few reasons why:
Async messages are supposed to be atomic. Different vendors make their own accommodations for message affinity (a relationship between two or more messages) but as a rule, message affinity is to be avoided. Your use case not only requires the transport to deal with message affinity, but to do so over an indeterminate interval between related messages.
Message payload is a blob. For performance reasons, WMQ doesn't touch message payloads except for things like compression or code page conversion. Anything that requires parsing the message payload is a job for WebSphere Message Broker, DataPower or WebSphere ESB. I would expect any messaging transport which claims to be performant would face similar issues because parsing payloads results in longer code paths and non-linear performance degradation. The exception is message properties but WMQ uses these for selection only and I expect that is generally the case.
Stateless operation. As a transport, the state of the application may be stored in a persistent message but the state of the transport layer should not depend on the state of the application across different units of work. Again, an ESB type of product is best suited when you want to delegate management of some of the application state to the messaging layer and especially when such management spans many units of work.
Assured delivery. WMQ was designed to never lose your persistent message. If the app explicitly sets expiry the message might go away because the sender said it was OK to do so. If the message is non-persistent it might go away, but only in an exceptional condition and, again, because the sender said it was OK to do so. The use case you describe might result in a message going away not because the sender said it was OK, or even because the recipient said it was OK but because of an interaction with some unrelated 3rd party who happened to beat you to the queue with a duplicate value. What if that first message has an invalid header or code page problem and gets rolled back? What if I as an attacker spew out garbage messages with all possible 4-digit values for discriminator?
As I said, I don't know the other messaging products so there may be something out there which meets your requirement and if so I'll be interested to read about it. However in the event hat nobody replies, this post may shed some light on the reasons why.
We have quartz process that polls a ActiveMQ JMS queue.
We know that we could get several messages a minute would like to only respond to the most current message at a configured polling rate of a minute or more.
We don't need to process any of the previous messages.
Is there a way to configure the queue to get this behavior?
Its seems like a topic has the ability to do this via the subscription recovery policy using a count of 1. We would like to do this using a queue to guarantee (more or less) a single delivery of the message.
Or is there a conceptual flaw in our assumptions...
Thanks
In my opinion there is no standard operation for this, so you will have to write some code....
One possible solution would be to use a QueueBrowser together with a QueueReceiver:
Through the QueueReceiver you would get an Enumeration of the messages in the queue. For each message you can now perform a receive with a MessageSelector on the JMSMessageID as long as hasMoreElements() returns true. The last message will be the one you want to have....
When using activemq, you can use "image caching" on topics. One of the settings there is to always keep the last mesage sent..
Take a look at the Subscription recovery Policy settings:
http://activemq.apache.org/subscription-recovery-policy.html