I am using last version of ActiveMQ Artemis.
My use case is pretty simple. The broker sends a lot of messages to consumer - a single queue. I have 2 consumers per queue. Consumers are set to use default prefetch size of messages (1000).
Is there a way to monitor how many messages are pre-fetch and processed from each consumer?
ActiveMQ Artemis exposes the DeliveringCount attribute on QueueControl. See the management doc for further details.
There is currently no way to track messages which are "in delivery" on a per-consumer basis.
Actually the only way to see how match messages are delivered to specific consumer (prefetched by consumer) that I found was to create and check a heap dump.
Related
I am using Nifi to get data from IBM MQ. It is working fine. My question is once the message is read from an MQ queue, does it get deleted from the queue? How to just read messages from the queue without deleting them from the queue?
My question is once the message is read from an MQ queue, does it get
deleted from the queue?
Yes, that is the default behavior.
How to just read messages from the queue without deleting them from
the queue?
You use the option: MQGMO_BROWSE_FIRST followed by MQGMO_BROWSE_NEXT on the MQGET API calls.
You can also open the queue for browse only. i.e. MQOO_BROWSE option for MQOPEN API call.
It sounds as if you would like to use a "publish/subscribe" model rather than a "point-to-point" model.
From ActiveMQ:
Topics In JMS a Topic implements publish and subscribe semantics. When
you publish a message it goes to all the subscribers who are
interested - so zero to many subscribers will receive a copy of the
message. Only subscribers who had an active subscription at the time
the broker receives the message will get a copy of the message.
Queues A JMS Queue implements load balancer semantics. A single
message will be received by exactly one consumer. If there are no
consumers available at the time the message is sent it will be kept
until a consumer is available that can process the message. If a
consumer receives a message and does not acknowledge it before closing
then the message will be redelivered to another consumer. A queue can
have many consumers with messages load balanced across the available
consumers.
If you have a queue, when a consumer consumes that message, it is removed from the queue so that the next consumer consumes the next message. With a topic, multiple consumers can be subscribed to that topic and retrieve the same message without being exclusive.
If neither of these work for you, I'm not sure what semantics you're looking for -- a "queue" which doesn't delete the message when it is consumed will never let a consumer access any but the first message.
Is there any way you can control the order delivery of messages in a topic in EMS or Tibco (using a JMS Topic Subscriber)? Something like the message selector, but instead of filtering to do ordering.
I would like to use a header like JMSXDeliveryCount, so that new messages will get a higher priority. I know that there's RedeliveryDelay, but that works only for queues, not topics.
Even JMSPriority could be an option if I can set it after getting the message with a topic subscriber. Can I do that? Maybe with a Java Code activity?
The broker will deliver messages in order as it receives them, but re-ordering messages is a distributed computing / EIP problem (see sequencer). The issue is the broker does't know how many messages are coming or how long to keep the window open for re-ordering messages. This activity is best done outside the broker in your application where that information is understood.
EIP frameworks and ESB's have done a nice job of standardizing a handler for this type of workflow.
ref: EIP Sequencer http://www.enterpriseintegrationpatterns.com/patterns/messaging/MessageSequence.html
JMSPriority is indeed an option, but it cannot be set on an already received message. The solution is to confirm the message and republished it with a different priority.
Let's suppose I have several subscribers consuming from a topic. After a message has been delivered to all the subscribers I'd like to trigger a job that would use this message in input.
So the easy way to do that would be to move messages that have been succesfully delivered to all the sucscribers to a queue from which my job would consume messages.
Is it part of JMS?
Is there any message broker able to do that directly?
If not is there a simple solution to solve this problem?
You should be able to do this using activemq's advisories.
See here for more about advisory messages: http://activemq.apache.org/advisory-message.html
So what you want to do, for the topic in question, is track:
the number of consumers
when a message is dispatched to them
when the message has been ack'd by each of the consumers
to get the number of consumers, listen to the "ActiveMQ.Advisory.Consumer.Topic." advisory topic
to get when a message is dispatched, listen to the "ActiveMQ.Advisory.MessageDelivered.Topic."
to get when a message has been ack'd, listen to "ActiveMQ.Advisory.MessageConsumed.Topic."
you could easily use Apache Camel to help out with this (listening to the topics) and aggregating whether or not all consumers have processed (ack'd) the message.. then that could kick off your further processing..
You could just create another durable subscription to route the message from the topic to queue directly. From that queue your job can consume messages. This is much easier than creating a trigger to route the messages to a queue.
So the easy way to do that would be to move messages that have been
succesfully delivered to all the sucscribers to a queue from which my
job would consume messages. Is it part of JMS?
No, this is not part of JMS specification.
I'm using ActiveMQ along with Mule (a kind of ESB based on Spring).
We got a fast producer and a slow consumer.
It's synchronous configuration with only one consumer.
Here the configuration of the consumer in spring style: http://pastebin.com/vweVd1pi
The biggest requirement is to keep the order of the messages.
However, after hours of running this code, suddenly, ActiveMQ skips 200 messages, and send the next ones.The 200 messages are still there in the activeMQ, they are not lost.
But our client (Mule), does have some custom code to check the order of the messages, using an unique identifier.
I had this issue already a few month ago. We change the consumer by using the parameter "jms.prefetchPolicy.queuePrefetch=1". It seemed to have worked well and to be the fix we needed unti now when the issue reappeared on another consumer.
Is it a bug, or a configuration issue ?
I can't talk about the requirement from a Mule perspective, but there are a couple of broker features that you should take a look at. There are two ways to guarantee message ordering in ActiveMQ:
Message groups are a way of ensuring that a set of related messages will be consumed by the same consumer in the order that they are placed on a queue. To use it you need to specify a JMSXGroupID header on related messages, and assign them an incrementing JMSXGroupSeq number. If a consumer dies, remaining messages from that group will be sent to another single consumer, while still preserving order.
Total message ordering applies to all messages on a topic. It is configured on the broker on a per-destination basis and requires no particular changes to client code. It comes with a synchronisation overhead.
Both features allow you to scale out to more than one consumer.
I need some help with topics and selectors.
I have a scenario with a topic having multiple durable subscribers (each with a selector)
not all messages going into the topic aren't read by the consumers - because of unmatching selectors.
This is correct behavior.
However the problem occurs when the unmatched messages reach a certain quantity threshold, because at that point no other messages are being delivered to the consumers
activemq tries to dispatch those old unmatchable messages, but since there is no consumer for them everything is stuck
can anybody help with this?
My setup is ActiveMq 5.5
is there some configuration option, or is it just a flawed design?
I'd say this is a flawed design given there are better alternatives and perhaps a bug in ActiveMQ.
First question: is your producer publishing to this topic setting the JMSExpiration header on those messages?
If yes, the first thing I'd do is create a Jira issue detailing the scenario you described above, because it does seem incorrect that ActiveMQ will continue hold on to and continue send messages for which no selectors apply.
As for flawed design, the minute you hear yourself saying "I need durable subscribers" and you are using ActiveMQ, you should immediately turn to using Virtual Destinations instead. Virtual Destinations have the advantages of topics in that a producer can send a message to a destination and have that message propagated to N number of other destinations for consumption, but doesn't have the disadvantages that come with having durable subscribers on a topic. Read more about Virtual Destinations here.
This is related to the way that ActiveMQ handles sparse selectors. The current implementation doesn't page into the store to look for message matching sparse selectors so you need to make some configuration changes to try and work around this. You can set the maxBrowsePageSize in the configured destination policy, the default is 400. See this page.
Virtual destinations in ActiveMQ is probably the better choice, it almost always is when thinking about using durable subscribers. You could however add some message expiration to you messages and configure the policy to expire messages on inactive durable subscribers if you use a SNAPSHOT version of ActiveMQ 5.6.
It does seem like a bug, (or dashedly inconvenient at the least) but there is a work around using Virtual Destinations.
Quote:
Note that making a topic virtual does add a small CPU overhead when
sending messages to the topic but it is fairly small. From version
5.4, dispatch from virtual topics to subscription queues can be
selectorAware such that only messages that match one of the existing
subscribers are actually dispatched. Using this option prevents the
build up of unmatched messages when selectors are used by exclusive
consumers.