I have implemented a priority queue in ActiveMQ. If the queue is being continuously flooded with the high priority messages, the low priority messages will never get processed. How does ActiveMQ handle such situations or how can this situation be avoided or handled?
ActiveMQ doesn't attempt to do anything to prevent this as it's up to you to solve it based on the needs of your application. If you have such a situation you might want to consider instead using a Queue per priority to allow for load balancing across the Queues.
Extending to Tim Bish's answer, there are some features in ActiveMQ you can use to handle this situation.
You can setup a virtual destination to filter out high and low prio messages, like this (inside a virtualDestinationInterceptor tag).
<virtualDestinations>
<compositeQueue name="ALL">
<forwardTo>
<filteredDestination selector="JMSPriority < 5" queue="LOW.PRIO"/>
<filteredDestination selector="JMSPriority > 4" queue="HIGH.PRIO"/>
</forwardTo>
</compositeQueue>
</virtualDestinations>
Then you can follow the alternative strategy presented here.
You put one consumer on the LOW.PRIO queue and multiple consumers on the HIGH.PRIO queue. Then the LOW.PRIO messages will be handled, but with less threads than high prio messages.
You can also read messages directly from the "ALL" queues with said selectors in your consumer application.
Related
Due to some reasons, we're not allowed to use more than one queue for our LOB.
So, we have set a very large prefetch number, and the consumer of that single queue will be in charge of dispatching to other in memory queues according to some message properties. Other background tasks will fetch the message from the in memory queues and handle it.
To avoid loss of messages, is it possible to manual ack the message until the background task finishes handling the message?
MassTransit handles message acknowledgement, there is no way to work around it. Once the message consumer completes, it will be acknowledged. Messages remain in the queue until they are acknowledged. There is no way messages could be lost using this approach.
Also, your approach of using a single transport queue and then having a bunch of in-memory queues is an incredibly poor (terrible, horrible, worst possible thing ever perhaps) design choice due to a really bad rule about queue usage. MassTransit can actually do all of that dispatch for you with multiple consumers on the receive endpoint for the various message types. But a single queue is still a really, really bad idea.
I am trying to implement a kind of Priority Channel with spring integration but I am blocked and didn't find a solution on the web.
I would like to read multiples channel (6) alternatively with a service activator. Each channel is for a priority level (CRITICAL, HIGHEST, HIGH, NORMAL, LOW, LOWEST). Message come from RabbitMQ and are distributed on correct channel with a Router.
The problem is that I would like to create a Service Activator who read alternatively in the channels using a round robin based on time.
For example, CRITICAL should have a 5 secondes running time, and then the service switch to HIGHEST for 3 seconds, and then to HIGH for 1 second, ...
Is it possible to do it properly with spring integration ?
Maybe I don't use the correct component to do it ?
Regards
The Priority Channel pattern works a bit different way.
It is a queue with sort support. When a new message arrives to the queue it is sorted to the proper place according to its priority. That absolutely does matter how your consumer of this channel works. The priority happens only in the channel. The consumer just polls messages from that queue like they are ordered for it: the CRITICAL, than HIGHEST, if CRITICAL aren't present and so on.
On the other hand, if you distribute messages by priority do different channels, why just don't have separate Service Activators for each of those channels? And each priority will be read by its own process.
There is no such a solution based on the "time to run". It just doesn't seem with a good fit for messaging architecture. Although you might can implement via scheduled task cancel() or Quartz to "perform task until...".
UPDATE
Regarding time control, I think you can come up with the solution which in the infinite loop really start()s different service activators and stop()s them after an appropriate scheduled time. All those service activators should listen to different queue channels.
We are using ActiveMq in our application. We need to implement priority consumer mechanism on the queue. We have multiple consumers and we want to treat them as master/slave. So, the master consumer would have higher priority and will consume all messages and once master gets down then the consumer (with higher priority) will consume all requests from the queue.
To Implement this scenario, we came across the mechanism of priority queues. So, we initialized the queues by this way:
javax.jms.Queue queue = queueSession.createQueue("myQueue" + "?consumer.priority=" + 127);
and remaining slaves nodes were assigned lesser priorities.
But this mechanism is not working expectedly, sometimes it works fine that we get all requests on the consumer with the highest priority but sometimes consumer with lesser priority also start consuming messages while the consumer with the highest priority is working fine.
We have tried it with consumer.exclusive=true option as well, with no luck.
Note: We are running all components (ActiveMQ, producer and consumer application) locally on the same machine for now, so no network delays involved. And we are running consumers with default prefetch policy.
Is there any other approach to implement this scenario using activeMq or are we missing any configuration.
If I have the 3 queues of priority 1,2 & 3 respectively. I want my consumer to consume first from queue withe priority 1, then 2 & so on. If in case queue with higher priority is empty, the consumer can consume from the queue with lower priority.
Is it possible to achieve from JMS or ActiveMQ or any other way?How?
You'd have to control that logic yourself using this method. To ActiveMQ, or any other JMS provider, you are just using another queue.
However, you can use a single queue for message priority. There are a couple different ways on how to do this as described in the documentation.
If you want your consumer to be as simple as possible then have the broker figure out the priority. Otherwise you'll need to mess with multiple consumers or inefficient single consumer logic with selectors to consume.
In both cases, your producer will just need to be smart enough to set the JMSPriority header to whatever priority the logic says it should be.
The only downside really is the fact that you have a broker side config to set up for that queue specifically rather than everything being automatic.
What can be the best way to aggregate messages from many different sources (actually queues/topics) into a single queue/topic and then consume it. I am trying to design an application to receive messages from different topics in JMS using weblogic.
You could write your own "aggregator" as a stand-alone Java application:
For each queue/topic have a reader in its own thread.
Each reader sends its received message again on a "aggregate queue".
Have another thread to listen on the "aggregate queue".
As a variation, you could use a JVM Queue (like java.util.concurrent.ArrayBlockingQueue) as the "aggregate queue". This is faster, does not require another MQ queue, does not need network bandwidth, but it's not persistent.
Another idea is to use a "Message driven bean (MDB)" for each incoming queue/topic:
Again, each of these MDBs just reads the message and resends it to the "aggregate queue".
Have another MDB listening on the "aggregate queue".
A few suggestions on quality requirements. I belive you have to consider them.
They will be highly relate with your technical solution.
is that message loss acceptable?
client ack could be considered.
e.g. A memory queue sit in middle, e.g. incoming queue1...n -> ArrayBlockingQueue in memory -> outgoing queue. The data in the ArrayBlockingQueue , will lost when app crash.
is that message duplicate acceptable for the single outgoing queue?
I would suggest yes.
Set applicable level PossibleDuplicateFlag to make the client aware of that.
how fast the incoming messages per second on the diff incoming queue?
one queue session has only a uniqe thread. Performance has to be considered in advance.