We are using ActiveMq in our application. We need to implement priority consumer mechanism on the queue. We have multiple consumers and we want to treat them as master/slave. So, the master consumer would have higher priority and will consume all messages and once master gets down then the consumer (with higher priority) will consume all requests from the queue.
To Implement this scenario, we came across the mechanism of priority queues. So, we initialized the queues by this way:
javax.jms.Queue queue = queueSession.createQueue("myQueue" + "?consumer.priority=" + 127);
and remaining slaves nodes were assigned lesser priorities.
But this mechanism is not working expectedly, sometimes it works fine that we get all requests on the consumer with the highest priority but sometimes consumer with lesser priority also start consuming messages while the consumer with the highest priority is working fine.
We have tried it with consumer.exclusive=true option as well, with no luck.
Note: We are running all components (ActiveMQ, producer and consumer application) locally on the same machine for now, so no network delays involved. And we are running consumers with default prefetch policy.
Is there any other approach to implement this scenario using activeMq or are we missing any configuration.
Related
I have ActiveMQ Artemis. Producer generates 1000 messages and consumer one by one processing their. Now I want to process this queue with help of two consumers. I start new consumer and new messages are distributed between two runned consumers. My question: is it posible redistribute old messages between all started consumers?
Once messages are dispatched by the broker to a consumer then the broker can't simply recall them as the consumer may be processing them. It's up to the consumer to cancel the messages back to the queue (e.g. by closing its connection/session).
My recommendation would be to tune your consumerWindowSize (set on the client's URL) so that a suitable number of messages are dispatched to your consumers. The default consumerWindowSize is 1M (1024 * 1024 bytes). A smaller consumerWindowSize would mean that more clients would be able to receive messages concurrently, but it would also mean that clients would need to conduct more network round-trips to tell the broker to dispatch more messages when they run low. You'll need to run benchmarks to find the right consumerWindowSize value for your use-case and performance needs.
I am trying to implement a kind of Priority Channel with spring integration but I am blocked and didn't find a solution on the web.
I would like to read multiples channel (6) alternatively with a service activator. Each channel is for a priority level (CRITICAL, HIGHEST, HIGH, NORMAL, LOW, LOWEST). Message come from RabbitMQ and are distributed on correct channel with a Router.
The problem is that I would like to create a Service Activator who read alternatively in the channels using a round robin based on time.
For example, CRITICAL should have a 5 secondes running time, and then the service switch to HIGHEST for 3 seconds, and then to HIGH for 1 second, ...
Is it possible to do it properly with spring integration ?
Maybe I don't use the correct component to do it ?
Regards
The Priority Channel pattern works a bit different way.
It is a queue with sort support. When a new message arrives to the queue it is sorted to the proper place according to its priority. That absolutely does matter how your consumer of this channel works. The priority happens only in the channel. The consumer just polls messages from that queue like they are ordered for it: the CRITICAL, than HIGHEST, if CRITICAL aren't present and so on.
On the other hand, if you distribute messages by priority do different channels, why just don't have separate Service Activators for each of those channels? And each priority will be read by its own process.
There is no such a solution based on the "time to run". It just doesn't seem with a good fit for messaging architecture. Although you might can implement via scheduled task cancel() or Quartz to "perform task until...".
UPDATE
Regarding time control, I think you can come up with the solution which in the infinite loop really start()s different service activators and stop()s them after an appropriate scheduled time. All those service activators should listen to different queue channels.
I've come across a curious detail in the legacy integration solution based on WebSphere MQ 7.0.1.3 and WebSphere Message Broker 7.0.0.7. There are 2 message flows:
The 1st flow is a case of MQ Request-Reply pattern. After MQPut it has a MQGet node that gets the message by correlation ID from queue "MQ_BIS_IN".
The 2nd flow is a kind of a one-way router that starts with a MQInput node (without any filters) that listens on the queue "MQ_GW_IN".
Interestingly, "MQ_BIS_IN" is an alias for "MQ_GW_IN" queue. My first thought was that the 2 flows would interfere in a bad way, basically the "omnivorous" MQInput would ruin the Request-Reply thing. But they seem to somehow get along.
I am going to reproduce this configuration in a test environment to determine if their behaviour is stable under load. Nevertheless, does anybody know if there are some rules of precedence between concurrent read operation from the same queue? Does it matter that there's an alias to the queue?
Both the MQInput and the MQGet node can be configured to look for particular msgId's or correlation Id's only, or to pick up the items on the queue in a determined order, or only pick up complete groups of messages - so there doesn't need to be a conflict here.
I have implemented a priority queue in ActiveMQ. If the queue is being continuously flooded with the high priority messages, the low priority messages will never get processed. How does ActiveMQ handle such situations or how can this situation be avoided or handled?
ActiveMQ doesn't attempt to do anything to prevent this as it's up to you to solve it based on the needs of your application. If you have such a situation you might want to consider instead using a Queue per priority to allow for load balancing across the Queues.
Extending to Tim Bish's answer, there are some features in ActiveMQ you can use to handle this situation.
You can setup a virtual destination to filter out high and low prio messages, like this (inside a virtualDestinationInterceptor tag).
<virtualDestinations>
<compositeQueue name="ALL">
<forwardTo>
<filteredDestination selector="JMSPriority < 5" queue="LOW.PRIO"/>
<filteredDestination selector="JMSPriority > 4" queue="HIGH.PRIO"/>
</forwardTo>
</compositeQueue>
</virtualDestinations>
Then you can follow the alternative strategy presented here.
You put one consumer on the LOW.PRIO queue and multiple consumers on the HIGH.PRIO queue. Then the LOW.PRIO messages will be handled, but with less threads than high prio messages.
You can also read messages directly from the "ALL" queues with said selectors in your consumer application.
If I have the 3 queues of priority 1,2 & 3 respectively. I want my consumer to consume first from queue withe priority 1, then 2 & so on. If in case queue with higher priority is empty, the consumer can consume from the queue with lower priority.
Is it possible to achieve from JMS or ActiveMQ or any other way?How?
You'd have to control that logic yourself using this method. To ActiveMQ, or any other JMS provider, you are just using another queue.
However, you can use a single queue for message priority. There are a couple different ways on how to do this as described in the documentation.
If you want your consumer to be as simple as possible then have the broker figure out the priority. Otherwise you'll need to mess with multiple consumers or inefficient single consumer logic with selectors to consume.
In both cases, your producer will just need to be smart enough to set the JMSPriority header to whatever priority the logic says it should be.
The only downside really is the fact that you have a broker side config to set up for that queue specifically rather than everything being automatic.