We have an application where we have 150 queues and now we want to introduce priority messages for all these queues.
All these queues are durable and persistent in nature.
Is it possible to convert these queues into priority queueues ? Do we have any documentation(s) available for this ? Maybe I missed it in the docs.
We have a plan that we will introduce 2 levels in priority either 0 or 1(default). https://www.rabbitmq.com/priority.html I found that
with more level of priority we need more CPU resources.
I didn't find what will be the scale or how much CPU resources will increase on which factor like no. of messages ? Do we have any stats or study available ?
Related
My topology has a bottleneck or two. The capacity metric in the Storm UI is useful for identifying these, but I'd be much more interested in the size of Bolt's queues.
My understanding is that each bolt has two queues, one for tuples pending to be executed, and another for tuple pending to be emitted. Is it possible to monitor the size of these queues?
I found some stuff online about adding an ITaskHook implementation to Bolts, but it's not remotely clear how I can use this to monitor queue size. Can the methods in ITaskHook be used for to monitor this?
You should be able to see the length of the queues for the components of your topology using the metrics mechanism. An easy way of doing is to add conf.registerMetricsConsumer(LoggingMetricsConsumer.class) to the config of your topology.
Here is an example of what I get for one of my components
4:fetch __sendqueue {write_pos=12122, read_pos=12122, capacity=1024, population=0}
4:fetch __receive {write_pos=8588, read_pos=8587, capacity=1024, population=1}
We are looking for a new messaging platform, and have narrowed our choices down to RabbitMQ or Kafka.
Right now, I am leaning toward Kafka, but I have some doubts that it is a good choice given one of our requirements.
We need to have a queue that is consumed by an unknown number of consumers. That is, we need to dynamically add and remove consumers as "workers" come online to do the processing. Also, workers may drop off at any time.
So for example, we may start a queue that has no consumers at all, and then the number of consumers may grow to 30. Later it may grow to 5000 or more, and then drop back off to 3.
We do not care about message ordering for this particular use case. Is Kafka a good fit for this?
Also, we were planning on maintaining a pool of consumer threads so that the workers could grab a single message and process it. So there may be 100 consumers in the pool and only 20 workers. Is it possible that we end up with messages in the other 80 consumers which are not utilized in the workers due to message send buffering? In other words, does Kafka pre-deliver messages to consumers before they are requested like some messaging systems do?
Yes, kafka can definitely match your requirements. You can have many-to-many producers/consumers. If all your consumers are within the same consumer group all messages will be distributed evenly between all consumers. It is not a problem also if you shut down / add new consumers, kafka will manage all automatically for you.
To your last question - kafka consumers are pull-based, so it is consumer responsibility to check if there are some messages to process.
I have a queue that is loaded with high priority JMS messages throughout the day, I want to get them out the door quickly. The queue is also being loaded periodically with lower priority messages in large batches. The problem that I see on busy days, is that there are always enough high priority messages at the front of the queue that none of the lower priority messages get selected until that volume drops off. Often they will sit on the queue until they middle of the night. The app is distributed over a number of servers, but the CPUs are not even breathing hard, the JMS seems to be the choak point.
My hunch is to implement some sort of aging algorithm that increases priority for messages that have been on the queue for a very long time, but of course, that is what middleware is supposed to do for me. I can't imagine that the JMS provider (IBM WebsphereMQ) or the application server (TIBCO BusinessWorks) doesn't have some sort of facility to cope with this. So before I go write some code, I thought I would ask, is there any way to get either of these technologies to help me out with this problem?
The BusinessWorks activity that is reading the queue is a JMS SOAP Event Source, but I could turn it into a JMS Queue Receiver activity or whatever.
All thoughts on how to solve this are welcome :-) TIA
That's like tying 1 hand behind your back and then complaining that you cannot swim properly. D'oh! First off, who's bright idea was it to mix messages. Just because you can do something does not mean you should.
The app is distributed over a number of servers, but the CPUs are not
even breathing hard, the JMS seems to be the choak point.
Well then, the solution is easy. Put high priority messages into queue "A" (the existing queue) and low priority messages into a new queue "B". Next, startup another instance of your JMS application to read the messages off queue "B".
Also, JMS is probably not the choke-point. It is what the application is doing with the message data after the JMS layer picks up the message that is taking a long time (i.e. backend work).
Finally, how many instances of your JMS application is running against the existing queue? If you are only running 1 instance, why? If you have lots of CPU capacity then why don't you run 10 instances of your JMS application. Do some true parallel processing of messages.
If you really want to keep you messages mixed on the same queue and have the high priority messages processed first, and yet your volume of messages is such that you cannot work through all the volume sometimes until the middle of the night, then you quite simply do not have enough processing applications. MQ is a parallel processing system, it is designed to allow many applications to put or get from a queue at once. Make use of this by running more of your getting applications at the same time. They will work through your high priority messages quicker and then get back to processing the lower priority ones.
From your description it's clear that you want the high priority messages to processed first. In such a case lower priority messages will have to wait.
MQ will not increase the priority of messages if they are sitting in queue for long time. How will it know that it has to change property of a message :)?. You will need to develop an application to do that.
I would think segregating messages based on priority, for example, high priority messages are put to one queue and lower priority messages to another queue could be one option you could look at.
Second option would be to look at the changing the delivery sequence (MSGDLVSQ) to FIFO. This makes to messages to be delivered to consumers in the order they arrived into queue. But note this will ignore the message priority, meaning if there is a lower priority message followed by a higher priority message, then higher priority message will wait till the lower priority message is delivered.
I am trying to implement the following,
Messages arrive at the Message broker with message priorities
They find their ways into various queues based on their message priority
So Q1 has messages with priority 1
Q2 has messages with priority 2 and so on ..
Is there a way to make the Message Broker process Q1 faster than the others.
Would it be possible to have a priority between queues ?
Q1 has higher priority to be processed than Q2 or better still processing of Q1 blocks other queues from being processed ?
Can an exchange itself be a priority queue that in turn feeds the other Queues ?
I saw that it is possible to extend the default exchanges via plugins, is there anything out there that already implements this above requirement that I have ?
Is this something feasible ? Or is this against the basic philosophy of a message broker ?
Is there any link to best practices while using prioritized messages ?
I did post this message on the Qpid nabble forum on August 28 - but 'This post has NOT been accepted by the mailing list yet'.
Thank you for your time.
In qpid you can define a queue as a "priority queue".
session.createQueue(queueName;{create:always, node:{type:queue,
x-declare:{arguments:{'x-qpid-priorities':3}}}})
In a priority queue, a message with higher priority will leap frog over messages with lower priority and will be picked up earlier. You need not define separate queues for each priority level.
The x-qpid-priorities parameter specifies how many distinct priorities are supported by the
queue.
Note though, priority based leapfrogging only works for consuming messages in a queue. Browsing doesn't respect priorities and you will see messages in the enqueue order.
Implementing separate queues for each priority isn't very useful, but if you insist on doing that, you will have to manage priority based consumption on your own. You can implement a consumer to check for messages in high priority queue, and then only check lower priority queue only if the first queue is empty.
I am going to use ActiveMQ for the first time in one of my projects (topics for durable messages). I have read that durable messages enforce a limit to the scale of number of messages per second. What are the other factors that I should be aware of (e.g. slow consumers) that puts a limit to the scale and performance characteristics of activemq and what metrics should be closely monitored and what are the values at which all hell breaks lose.
I don't expect to be pushing more than a thousand events per second in ActiveMQ for now.
here are a few tips...
increase your systemUsage limits from the defaults
increase your JVM heap size from the defaults
if using KahaDB, consider setting enableJournalDiskSyncs to false (helps throughput dramatically) or preferably use the new LevelDB
learn about producer flow control and consider disabling (frequently done)
consider using virtual topics (instead of durable topic consumers)
learn about prefetch-limit and tweak as needed
Two specific issues I ran into with activeMQ:
1) There are memory limits enforced per queue that need to be tuned. ActiveMQ won't fill up your heap unless you change the config. So you need to set -Xmx and change the config to use more memory.
2) Related to #1, by default the sender (client) blocks when limits are reached. In newer versions, there is a setting to avoid this and have an exception thrown instead. See http://activemq.apache.org/producer-flow-control.html.