Is it possible to create a priority queue using the Amazon simple queuing service?
Initially I couldn't find anything on the subject and this is why I create two queues. A general queue and a priority queue. I am enqueuing messages to this queue according the rule I've defined but confusion arises while dequeuing messages.
How do I do a long poll on the queues such that my combination of queues behave like a single priority queue?
I think you're on the right track by creating two queues - A normal queue and a priority queue.
You don't necessarily need a long poll in this case. Since messages in the priority queue take precedence over messages in the normal queue, you could follow an approach like this:
Poll priority queue till there are no more messages.
Poll normal queue and repeat step 1 after every message in the normal queue.
You could use Shoryuken and set a higher weight to configure the priority:
queues:
- [high_priority_queue, 5]
- [low_priority_queue, 1]
Related
So I have a Go app that listens on two different queues. Both of the messages are related, and optimally, the message from queue A arrives before the message from queue B.
The thing is, sometimes, in some edge cases, the order of arrival is inversed.
Is there some pattern that can help me to listen on both queues, but making sure queue B "holds" if a message from queue A has not arrived yet, when a specific field in both messages are equal?
I have 2 ActiveMQ JMS consumer queues consumer1 and consumer2.
Dead letter queues are enabled for both queues mainly for avoiding poison messages continuous retry.
Issue is now consumer2 messages should be processed in order. Where as messages going to DLQ could be processed out of order later.
Any suggestions how to handle this scenario?
Your question is not clear; the only way to guarantee order is to process them in order (concurrency=1) and don't use any prefetch.
If you prefetch messages, rejected messages might go behind the prefetch.
Turning off prefetch will severely impact performance.
We are looking for a new messaging platform, and have narrowed our choices down to RabbitMQ or Kafka.
Right now, I am leaning toward Kafka, but I have some doubts that it is a good choice given one of our requirements.
We need to have a queue that is consumed by an unknown number of consumers. That is, we need to dynamically add and remove consumers as "workers" come online to do the processing. Also, workers may drop off at any time.
So for example, we may start a queue that has no consumers at all, and then the number of consumers may grow to 30. Later it may grow to 5000 or more, and then drop back off to 3.
We do not care about message ordering for this particular use case. Is Kafka a good fit for this?
Also, we were planning on maintaining a pool of consumer threads so that the workers could grab a single message and process it. So there may be 100 consumers in the pool and only 20 workers. Is it possible that we end up with messages in the other 80 consumers which are not utilized in the workers due to message send buffering? In other words, does Kafka pre-deliver messages to consumers before they are requested like some messaging systems do?
Yes, kafka can definitely match your requirements. You can have many-to-many producers/consumers. If all your consumers are within the same consumer group all messages will be distributed evenly between all consumers. It is not a problem also if you shut down / add new consumers, kafka will manage all automatically for you.
To your last question - kafka consumers are pull-based, so it is consumer responsibility to check if there are some messages to process.
I have a queue that is loaded with high priority JMS messages throughout the day, I want to get them out the door quickly. The queue is also being loaded periodically with lower priority messages in large batches. The problem that I see on busy days, is that there are always enough high priority messages at the front of the queue that none of the lower priority messages get selected until that volume drops off. Often they will sit on the queue until they middle of the night. The app is distributed over a number of servers, but the CPUs are not even breathing hard, the JMS seems to be the choak point.
My hunch is to implement some sort of aging algorithm that increases priority for messages that have been on the queue for a very long time, but of course, that is what middleware is supposed to do for me. I can't imagine that the JMS provider (IBM WebsphereMQ) or the application server (TIBCO BusinessWorks) doesn't have some sort of facility to cope with this. So before I go write some code, I thought I would ask, is there any way to get either of these technologies to help me out with this problem?
The BusinessWorks activity that is reading the queue is a JMS SOAP Event Source, but I could turn it into a JMS Queue Receiver activity or whatever.
All thoughts on how to solve this are welcome :-) TIA
That's like tying 1 hand behind your back and then complaining that you cannot swim properly. D'oh! First off, who's bright idea was it to mix messages. Just because you can do something does not mean you should.
The app is distributed over a number of servers, but the CPUs are not
even breathing hard, the JMS seems to be the choak point.
Well then, the solution is easy. Put high priority messages into queue "A" (the existing queue) and low priority messages into a new queue "B". Next, startup another instance of your JMS application to read the messages off queue "B".
Also, JMS is probably not the choke-point. It is what the application is doing with the message data after the JMS layer picks up the message that is taking a long time (i.e. backend work).
Finally, how many instances of your JMS application is running against the existing queue? If you are only running 1 instance, why? If you have lots of CPU capacity then why don't you run 10 instances of your JMS application. Do some true parallel processing of messages.
If you really want to keep you messages mixed on the same queue and have the high priority messages processed first, and yet your volume of messages is such that you cannot work through all the volume sometimes until the middle of the night, then you quite simply do not have enough processing applications. MQ is a parallel processing system, it is designed to allow many applications to put or get from a queue at once. Make use of this by running more of your getting applications at the same time. They will work through your high priority messages quicker and then get back to processing the lower priority ones.
From your description it's clear that you want the high priority messages to processed first. In such a case lower priority messages will have to wait.
MQ will not increase the priority of messages if they are sitting in queue for long time. How will it know that it has to change property of a message :)?. You will need to develop an application to do that.
I would think segregating messages based on priority, for example, high priority messages are put to one queue and lower priority messages to another queue could be one option you could look at.
Second option would be to look at the changing the delivery sequence (MSGDLVSQ) to FIFO. This makes to messages to be delivered to consumers in the order they arrived into queue. But note this will ignore the message priority, meaning if there is a lower priority message followed by a higher priority message, then higher priority message will wait till the lower priority message is delivered.
I am trying to implement the following,
Messages arrive at the Message broker with message priorities
They find their ways into various queues based on their message priority
So Q1 has messages with priority 1
Q2 has messages with priority 2 and so on ..
Is there a way to make the Message Broker process Q1 faster than the others.
Would it be possible to have a priority between queues ?
Q1 has higher priority to be processed than Q2 or better still processing of Q1 blocks other queues from being processed ?
Can an exchange itself be a priority queue that in turn feeds the other Queues ?
I saw that it is possible to extend the default exchanges via plugins, is there anything out there that already implements this above requirement that I have ?
Is this something feasible ? Or is this against the basic philosophy of a message broker ?
Is there any link to best practices while using prioritized messages ?
I did post this message on the Qpid nabble forum on August 28 - but 'This post has NOT been accepted by the mailing list yet'.
Thank you for your time.
In qpid you can define a queue as a "priority queue".
session.createQueue(queueName;{create:always, node:{type:queue,
x-declare:{arguments:{'x-qpid-priorities':3}}}})
In a priority queue, a message with higher priority will leap frog over messages with lower priority and will be picked up earlier. You need not define separate queues for each priority level.
The x-qpid-priorities parameter specifies how many distinct priorities are supported by the
queue.
Note though, priority based leapfrogging only works for consuming messages in a queue. Browsing doesn't respect priorities and you will see messages in the enqueue order.
Implementing separate queues for each priority isn't very useful, but if you insist on doing that, you will have to manage priority based consumption on your own. You can implement a consumer to check for messages in high priority queue, and then only check lower priority queue only if the first queue is empty.