I have a persistent queue, non-transacted, client-acknowledge, the consumers read with jms.prefetchPolicy.queuePrefetch=1&wireFormat.maxInactivityDuration=50000
and once a consumer processes a message, it ack's the message.
If the consumer reads the message, and before it can send an ack, the process terminates abruptly, what happens in ActiveMq? (What ActiveMq parameters come into play here?)
How is that different than if the the consumer will take 10 minutes to process the message (so the consumer task is alive and working), how does ActiveMq know the message is still being worked on? (Does it monitor the TCP/IP connection, if the connection dies, it assumes the message will not be Ack'ed?)
How do I determine if a message is a "poison pill", i.e. it makes the consumers crash? (the redelivery count seems to be valid if the consumer task does not die; is there an internal counter in the message that says "it was been read n times without being successfully ack'ed?")
As an experiment, I sent 6 messages, one of them being a "poison pill" (kills the consumer before the consumer can send the ack), with 2 simultaneous consumers running (and automatically restarting consumers to bring the count to 2 whenever a consumer dies). Looking at the queue (using jconsole, I enable jmx using broker.setUseJmx(true)), 4 messages were delivered, 2 are in-flight. Why would there be 2 in-flight instead of just one?
I've been reading the ActiveMq and JMS specs for a while without clear/conclusive answers, so any insights on what parameters come into play, and if there are any known bugs, will be greatly useful.
This is purely based on my understanding of JMS - may not be completely correct:
If the consumer reads the message, and before it can send an ack, the process terminates abruptly, what happens in ActiveMq
My understanding is that since this happens in the context of a session with the JMS provider, JMS provider knows if the session is no longer active or has failed and any message not acknowledged as part of the session will be redelivered when the session is re-established.
How do I determine if a message is a "poison pill", i.e. it makes the consumers crash?
Like you have mentioned, the JMS provider keeps track of the # of times the message was redelivered possibly in the header of the message
4 messages were delivered, 2 are in flight
Not sure about this point
Related
I have a HornetQ based JMS provider and the consumer is attached to the provider in the CLIENT_ACKNOWLEDGE_MODE . The message.acknowledge () snippet though is under an if else . Thus the consumer would sometimes would not send an acknowledgement to the server in case there was an application layer failure in processing the message .So there are two questions here -
1)Will more messages that are queued in the server will keep on
flowing to the consumer even though the consumer did not acknowledge
on of the messages as stated earlier
2)Will the unacknowledged message flow down again on restarting the
consumer .
These are some of my observations on the questions I have asked
1)The messages keep on flowing down to the consumer as per the
consumer logs even though it did not acknowledge one of the messages
to the server due to an application layer failure (Note , there was
no uncaught exception as such , just that the consumer did not
acknowledge) .
2)Secondly , on restarting the consumer as well the message did not
flow down again from the server which is surprising .
Can someone please clarify this behaviour?
It has hard to fully determine what your application is doing in terms of message acknowledgement but my guess is that you are continuing to acknowledge messages after the failed attempt to acknowledge the message in question. In that case because you are using the Client Acknowledge mode the next Acknowledge will also apply to the previous message as that is how client mode works.
Session.CLIENT_ACKNOWLEDGE: A client acknowledges a message by calling the message’s acknowledge method. In this mode, acknowledgment takes place on the session level: Acknowledging a consumed message automatically acknowledges the receipt of all messages that have been consumed by its session. For example, if a message consumer consumes ten messages and then acknowledges the fifth message delivered, all ten messages are acknowledged.
So if you read that carefully you will see how consumers in a Session acknowledging messages while in client Acknowledge mode could even affect one another.
For a better answer you'd need to break down the chain of event further so that in became more clear what is going on.
I have a set up of an ActiveMQ broker and a single consumer. Consumer gets a message that he is not able to process because a service that it depends has a bug (once fixed it will be fine). So the message keeps being redelivered (consumer redelivery) - we use JMS sessions. With our current configuration it will keep redelivering it every 10 minutes for 1 day. That obviously causes a problem because other messages are not being consumed.
In order to solve this problem I have accessed the queue through JMX and tried to delete that message but it is not there. I guess it is cached on the consumer and not visible at the broker.
Is there any way to delete this message other than restarting the application?
Is it possible to configure the redelivery mechanism so that such message (that causes a live lock eventually) is put at the end of the queue so that other messages can be processed?
The 10 minutes for 1 day redelivery policy should stay as is.
I think you're right that the messages are stuck in the consumer's prefetch buffer, and I don't know of a way to delete them from there.
I'd change your redelivery policy to send to the DLQ after the second failure, with a much shorter interval between them, like 30 seconds, and I'd configure the DLQ strategy as an individualDeadLetterStrategy so you get a separate DLQ containing only messages from this particular queue. Then set up a consumer on this DLQ to move the messages to (the end of) the main queue whenever your reprocessing condition is met (whether that's after a certain delay, or based on reading some flag value from a database, or whatever). This consumer is where you'd implement "every 10 minutes for 1 day" logic, instead of in the redelivery policy where you currently have it.
That will keep the garbage ones out of the main queue so they don't delay other messages from being consumed, but still ensure that they will be reprocessed later. And it will put them on the broker instead of in the consumer's prefetch buffer, where you can view and delete them.
The only way to get it to the back of the queue is to reproduce it to the queue. Redelivery polices can only be configured down to the destination on the connection factory.
Given that you already have a connection, it shouldn't be to hard to create a producer that can either move the given message to a DLQ or produce it back to the queue when you run into that particular bug.
Setting jms.nonBlockingRedelivery=true on the connection factory resolved the problem. Now even if there is a message redelivered it does not block processing of other Messages.
As I understand (http://activemq.apache.org/message-redelivery-and-dlq-handling.html) a redelivery can be done by either a consumer or a broker.
I have some questions though:
How does the redelivery by a consumer work underneath? Does the consumer cache the message from a broker and redeliver it locally? What does happen if a consumer terminates inbetween? Such message will be lost? I think that as long as the consumer does not acknowledge the message it shouldn't be. But in such case, the message will be still available on the broker?
Are there any guidelines when to use broker vs consumer redelivery? Any recommendations?
The consumer does cache and redeliver the message to the client locally, until the redelivery count is met, then automatically acks the message as bad (posin ack). A consumer can control if it gets marked as redelivered depending on the acknowledgment mode. If for whatever reason a consumer can't or does not want to process a message, it can also kick it back and it will be available for consumption again if it closes the session.
The broker will hold onto the message until it gets an ack from the consumer. If your consumer is set to AUTO_ACKNOWLEDGE then it is possible you could lose the message if an unhandled exception occurs or the consumer ends unexpectedly for example.
Otherwise, if your consumer is using transactions or CLIENT_ACKNOWLEDGE it will give you the control on when that occurs.
With transactions, if the consumer drops prior to a commit it will be available for the next consumer or whenever that consumer reconnects.
I've always used transaction over CLIENT_ACKNOWLEDGE so I don't want to say for sure that the message will be lost in if the Session.recover() is not called before the consumer goes down or not.
From a consumer stand point, this is also known as retry logic.
Regarding broker vs consumer redelivery: By default, the broker just keeps giving the consumer the same message until the redelivery count is met. If you tell the broker not to redeliver it after a given amount of time, then your consumer can work on consuming other messages that may be able to be processed.
When to do this is really up to what is going on in your application. Maybe a specific message needs to be put to a database and that database is not available at the moment and you want to move on to messages that go elsewhere/have another purpose?
I have a multiple consumer listening to a queue and when a message arrives its onMessage() function in invoked. Since there are multiple consumers, each consumer has its own session. The requirement is that the message should be acknowledged only if there is no issues and no exception is thrown...)
AUTO_ACK mode: My understanding is: the message is acknowledged after onMessage completes successfully. If there is an exception in onMessage(), the message will be redelivered.
Client ack: At the end of onMessage(), I explicitly call acknowledge(). In case of error the acknowledge() method will not be called so message will be redelivered.
Transacted Session: I call session.commit() at the end of the onMessage() function, in case of some exception I catch it and call session.rollback and hence the message will be redelivered.
The consumer is ABLE to detect duplicate messages and handle it appropriately. My question is that all the 3 modes are doing the same thing and solving my purpose, so which method is better than the other and why? In short why should I use transacted sessions over client ack or auto mode.
I do not want to use JTA/XA as it is not supported by all jms providers example activeMQ and it will make my application slow.
If my consumer was NOT able to handle duplicate messages, then I understand that the only option I have is to use JTA/XA because all other options can send me the message again which will result in duplicate processing. In JTA/XA also I can get the message again but it would not be considered as duplicate processing as the previous transactions would have been rollbacked.
Be careful here about the difference between ack modes as they can be more subtle than you might think.
When using auto acknowledge and throwing an exception in onMessage your message might get redelivered to another client, the same client, or go into a DLQ depending on your broker and how it's configured.
The thing that many people overlook with client acknowledge is that the client ack is actually acking all previous messages delivered for the parent Session, so if you had more than one consumer in the same session it's message could also be acked by your acknowledge call. Since in your case you have a consumer per session then you sort of fall back to the auto ack case where messages will get redelivered or might go to a DLQ.
In the local transaction case the messages are often times redelivered some number of times to the same client based on either client or Broker configuration before either being re-dispatched to some other consumer or are routed to a DLQ.
To some degree the mode you choose depends on bit on your expectations of the outcome of multiple failures and how you can configure your chosen Broker implementation to handle that.
I created a AQ in oracle and wrote 2 JMS consumers in Java to listen to the queue. I have observed sometimes that if I produce some message in to queue; the count of dequeued messages from queue is greater than what enqueued. It means that some messages are consumed twice.
I have created queue with property:- multiple_consumers => FALSE
And JMS consumers are working in CLIENT_ACKNOWLEDGE mode
Please help me learn the possible reasons for such behavior and it's solution. So, that I can replicate the problem and solve above issue and ensure that the number of message enqueued is equal to number of message dequeued in case of multiple JMS consumers listening to same AQ .
Without having seen your code, CLIENT_ACKNOWLEDGE typically says you are sending acknowledgements manually. If you do not send an ack, the message won't get deleted and the broker will try to redeliver it at a later stage (like when you restart the connection or similar). This might be the cause of your concern.