I have 2 ActiveMQ JMS consumer queues consumer1 and consumer2.
Dead letter queues are enabled for both queues mainly for avoiding poison messages continuous retry.
Issue is now consumer2 messages should be processed in order. Where as messages going to DLQ could be processed out of order later.
Any suggestions how to handle this scenario?
Your question is not clear; the only way to guarantee order is to process them in order (concurrency=1) and don't use any prefetch.
If you prefetch messages, rejected messages might go behind the prefetch.
Turning off prefetch will severely impact performance.
Related
I am reading both concepts. Mainly Kafka. And comparing with JMS to understand better.
Kafka guarantees ordered delivery and multiple subscriber. How does kafka achieve it?
Kafka has multiple partitions. If one consumer per partition, then we can guarantee ordering. We can achieve load balancing with multiple partitions. So Both at the same time is possible.
In case of JMS, if we have multiple queues, isn't same as Kafka?
Q1: Which is better in this scenario?
Q2: Am I looking narrowly? Does kafka do more than this?
Please advise me.
Even If I am wrong about JMS, please let me know.
I was asking myself the same question before :)
As you wrote, Kafka guarantees ordered delivery only within a single partition. Period. If you are using multiple partitions (which is a must to have the parallelism), then it is possible that a consumer who listens on several partitions gets a message A from partition 1 before a message B from partition 2, even though message B arrived first.
Now, about the differences between Kafka and JMS. In JMS, you have a queue and you have a topic. With queues, when first consumer consumes a message, others cannot take it anymore. With topics, multiple consumers receive each message but it is much harder to scale. Consumer group from Kafka is a generalization of these two concepts - it allows scaling between members of the same consumer group, but it also allows broadcasting the same message between many different consumer groups.
Even more important difference is the following. Imagine that you have Kafka topic with 500 partitions and on the other hand, 500 JMS message queues. Let's also imagine that you have certain number of producers and consumers. In case of JMS, you need to configure each of them so they know which queues belong to them. What if e.g. some consumer crashes or you detect that you need to increase number of consumers? You have to reconfigure manually the whole system. This comes for free with Kafka, i.e. Kafka provides automatic rebalancing which is an extremely useful feature.
Finally, Kafka is tremendously faster, mostly because of some clever disk/memory transfer techniques and because consumers take care about the messages they consumed, not the broker like in JMS. Because of this, consumer is also able to "rewind", i.e. reread the messages from e.g. 2 days ago.
See also:
Apache Kafka order of messages with multiple partitions
Benchmarking Apache Kafka
Here's a fairly good article on the differences:
http://blog.hampisoftware.com/index.php/2016/01/20/apache-kafka-differences-from-jms/
Kafka does not guarantee message ordering across multiple partitions of a topic. Order is maintained only within a partition. In order to achieve strict ordering, you need to use one partition per topic.
We are looking for a new messaging platform, and have narrowed our choices down to RabbitMQ or Kafka.
Right now, I am leaning toward Kafka, but I have some doubts that it is a good choice given one of our requirements.
We need to have a queue that is consumed by an unknown number of consumers. That is, we need to dynamically add and remove consumers as "workers" come online to do the processing. Also, workers may drop off at any time.
So for example, we may start a queue that has no consumers at all, and then the number of consumers may grow to 30. Later it may grow to 5000 or more, and then drop back off to 3.
We do not care about message ordering for this particular use case. Is Kafka a good fit for this?
Also, we were planning on maintaining a pool of consumer threads so that the workers could grab a single message and process it. So there may be 100 consumers in the pool and only 20 workers. Is it possible that we end up with messages in the other 80 consumers which are not utilized in the workers due to message send buffering? In other words, does Kafka pre-deliver messages to consumers before they are requested like some messaging systems do?
Yes, kafka can definitely match your requirements. You can have many-to-many producers/consumers. If all your consumers are within the same consumer group all messages will be distributed evenly between all consumers. It is not a problem also if you shut down / add new consumers, kafka will manage all automatically for you.
To your last question - kafka consumers are pull-based, so it is consumer responsibility to check if there are some messages to process.
I want to understand how ApacheMQ's prefetch limit works. Are all the messages sent in one burst? What if there are concurrent consumers, what happens then?
What is the difference between prefetch limit of 0 and 1?
Read the link recommended by #Tim Bish -- the quotes I offer are from that page.
So ActiveMQ uses a prefetch limit on how many messages can be streamed
to a consumer at any point in time. Once the prefetch limit is
reached, no more messages are dispatched to the consumer until the
consumer starts sending back acknowledgements of messages (to indicate
that the message has been processed). The actual prefetch limit value
can be specified on a per consumer basis.
Specifically on the 0 versus 1 prefetch limit difference:
If you have very few messages and each message takes a very long time
to process you might want to set the prefetch value to 1 so that a
consumer is given one message at a time. Specifying a prefetch limit
of zero means the consumer will poll for more messages, one at a time,
instead of the message being pushed to the consumer.
I have a queue that is loaded with high priority JMS messages throughout the day, I want to get them out the door quickly. The queue is also being loaded periodically with lower priority messages in large batches. The problem that I see on busy days, is that there are always enough high priority messages at the front of the queue that none of the lower priority messages get selected until that volume drops off. Often they will sit on the queue until they middle of the night. The app is distributed over a number of servers, but the CPUs are not even breathing hard, the JMS seems to be the choak point.
My hunch is to implement some sort of aging algorithm that increases priority for messages that have been on the queue for a very long time, but of course, that is what middleware is supposed to do for me. I can't imagine that the JMS provider (IBM WebsphereMQ) or the application server (TIBCO BusinessWorks) doesn't have some sort of facility to cope with this. So before I go write some code, I thought I would ask, is there any way to get either of these technologies to help me out with this problem?
The BusinessWorks activity that is reading the queue is a JMS SOAP Event Source, but I could turn it into a JMS Queue Receiver activity or whatever.
All thoughts on how to solve this are welcome :-) TIA
That's like tying 1 hand behind your back and then complaining that you cannot swim properly. D'oh! First off, who's bright idea was it to mix messages. Just because you can do something does not mean you should.
The app is distributed over a number of servers, but the CPUs are not
even breathing hard, the JMS seems to be the choak point.
Well then, the solution is easy. Put high priority messages into queue "A" (the existing queue) and low priority messages into a new queue "B". Next, startup another instance of your JMS application to read the messages off queue "B".
Also, JMS is probably not the choke-point. It is what the application is doing with the message data after the JMS layer picks up the message that is taking a long time (i.e. backend work).
Finally, how many instances of your JMS application is running against the existing queue? If you are only running 1 instance, why? If you have lots of CPU capacity then why don't you run 10 instances of your JMS application. Do some true parallel processing of messages.
If you really want to keep you messages mixed on the same queue and have the high priority messages processed first, and yet your volume of messages is such that you cannot work through all the volume sometimes until the middle of the night, then you quite simply do not have enough processing applications. MQ is a parallel processing system, it is designed to allow many applications to put or get from a queue at once. Make use of this by running more of your getting applications at the same time. They will work through your high priority messages quicker and then get back to processing the lower priority ones.
From your description it's clear that you want the high priority messages to processed first. In such a case lower priority messages will have to wait.
MQ will not increase the priority of messages if they are sitting in queue for long time. How will it know that it has to change property of a message :)?. You will need to develop an application to do that.
I would think segregating messages based on priority, for example, high priority messages are put to one queue and lower priority messages to another queue could be one option you could look at.
Second option would be to look at the changing the delivery sequence (MSGDLVSQ) to FIFO. This makes to messages to be delivered to consumers in the order they arrived into queue. But note this will ignore the message priority, meaning if there is a lower priority message followed by a higher priority message, then higher priority message will wait till the lower priority message is delivered.
I am trying to implement the following,
Messages arrive at the Message broker with message priorities
They find their ways into various queues based on their message priority
So Q1 has messages with priority 1
Q2 has messages with priority 2 and so on ..
Is there a way to make the Message Broker process Q1 faster than the others.
Would it be possible to have a priority between queues ?
Q1 has higher priority to be processed than Q2 or better still processing of Q1 blocks other queues from being processed ?
Can an exchange itself be a priority queue that in turn feeds the other Queues ?
I saw that it is possible to extend the default exchanges via plugins, is there anything out there that already implements this above requirement that I have ?
Is this something feasible ? Or is this against the basic philosophy of a message broker ?
Is there any link to best practices while using prioritized messages ?
I did post this message on the Qpid nabble forum on August 28 - but 'This post has NOT been accepted by the mailing list yet'.
Thank you for your time.
In qpid you can define a queue as a "priority queue".
session.createQueue(queueName;{create:always, node:{type:queue,
x-declare:{arguments:{'x-qpid-priorities':3}}}})
In a priority queue, a message with higher priority will leap frog over messages with lower priority and will be picked up earlier. You need not define separate queues for each priority level.
The x-qpid-priorities parameter specifies how many distinct priorities are supported by the
queue.
Note though, priority based leapfrogging only works for consuming messages in a queue. Browsing doesn't respect priorities and you will see messages in the enqueue order.
Implementing separate queues for each priority isn't very useful, but if you insist on doing that, you will have to manage priority based consumption on your own. You can implement a consumer to check for messages in high priority queue, and then only check lower priority queue only if the first queue is empty.