Server: Wildfly 10.0
JDK: JAVA 8
In my application, one of a feature is hosted with JMS.
The application may receive 1500+ messages (not more than 5000) every day. MDB (with maxSession=15) listens to queue and process the messages. The average processing time of a message # MDB is < 1 sec.
Issue:
Often we are facing a delay in message appearing at MDB to processes it. Sometimes message pushed to the queue are arrived at MDB after 2 hours of time delay, there is no exception or delay in pushing a message to the queue. So we are trying to understand what would be the reason behind this behavior.
Related
Little bit of backgroud: I need to improve the performance of one of our batch framework. There, batch inputs are sent to a JMS queue. Further, at the queue endpoint, we have a MDB, which is consuming the messages. Now, what i suspect here that if there are large number of messages, there is no MDB instance available to consume the messages as all of them are held up in processing the previous messages. To improve this, i am thinking of implementing a threadpool in the MDB business logic so that once the MDB has received the message and deliver it to the thread, it gets free for consuming another message.
Now before implementing this, i want to monitor my JMS queues to check if the messages are really waiting in the queues or not. So i need to know if this monitoring can be done via some WAS admin console or some JMX application. My main purpose is to check the waiting time of each jms message in the queue.
First, you can set the number of processes (MDB instances) that will consumes the Q in parallel. The default is 10 (Per member of the cluster..).
With the console: Resources -> JMS -> activation specifications, Set "Maximum concurrent MDB invocations per endpoint" which is defined as `"The maximum number of endpoints to which messages are delivered concurrently."``
As for monitoring the Q and generating some load, you can have a look at JMSToolBox on sourceforge
In the "Destination information" dialog in JMSToolBox, you will also be able to see the number of concurrent consumers on the Q
Also if you want to measure the time spend by a message in the Q, just compute the difference between the current time and the JMSTimestamp JMS standard property from the message it is process by the MDB in the onMessage() method
I have a set up of an ActiveMQ broker and a single consumer. Consumer gets a message that he is not able to process because a service that it depends has a bug (once fixed it will be fine). So the message keeps being redelivered (consumer redelivery) - we use JMS sessions. With our current configuration it will keep redelivering it every 10 minutes for 1 day. That obviously causes a problem because other messages are not being consumed.
In order to solve this problem I have accessed the queue through JMX and tried to delete that message but it is not there. I guess it is cached on the consumer and not visible at the broker.
Is there any way to delete this message other than restarting the application?
Is it possible to configure the redelivery mechanism so that such message (that causes a live lock eventually) is put at the end of the queue so that other messages can be processed?
The 10 minutes for 1 day redelivery policy should stay as is.
I think you're right that the messages are stuck in the consumer's prefetch buffer, and I don't know of a way to delete them from there.
I'd change your redelivery policy to send to the DLQ after the second failure, with a much shorter interval between them, like 30 seconds, and I'd configure the DLQ strategy as an individualDeadLetterStrategy so you get a separate DLQ containing only messages from this particular queue. Then set up a consumer on this DLQ to move the messages to (the end of) the main queue whenever your reprocessing condition is met (whether that's after a certain delay, or based on reading some flag value from a database, or whatever). This consumer is where you'd implement "every 10 minutes for 1 day" logic, instead of in the redelivery policy where you currently have it.
That will keep the garbage ones out of the main queue so they don't delay other messages from being consumed, but still ensure that they will be reprocessed later. And it will put them on the broker instead of in the consumer's prefetch buffer, where you can view and delete them.
The only way to get it to the back of the queue is to reproduce it to the queue. Redelivery polices can only be configured down to the destination on the connection factory.
Given that you already have a connection, it shouldn't be to hard to create a producer that can either move the given message to a DLQ or produce it back to the queue when you run into that particular bug.
Setting jms.nonBlockingRedelivery=true on the connection factory resolved the problem. Now even if there is a message redelivered it does not block processing of other Messages.
I created a AQ in oracle and wrote 2 JMS consumers in Java to listen to the queue. I have observed sometimes that if I produce some message in to queue; the count of dequeued messages from queue is greater than what enqueued. It means that some messages are consumed twice.
I have created queue with property:- multiple_consumers => FALSE
And JMS consumers are working in CLIENT_ACKNOWLEDGE mode
Please help me learn the possible reasons for such behavior and it's solution. So, that I can replicate the problem and solve above issue and ensure that the number of message enqueued is equal to number of message dequeued in case of multiple JMS consumers listening to same AQ .
Without having seen your code, CLIENT_ACKNOWLEDGE typically says you are sending acknowledgements manually. If you do not send an ack, the message won't get deleted and the broker will try to redeliver it at a later stage (like when you restart the connection or similar). This might be the cause of your concern.
I'm looking for a way to schedule a MDB. My requirement is that the MDB is set to feed a system from the company. This system goes out for maintenance every night, but the other systems don't know about it and may keep trying to feed it. A persistent queue is great in the way that my messages could be pilled until system goes back online.
How could I manage that? I've run into that already: schedule a message driven bean to access a queue during certain times? but it uses java 7, and worst, message is lost if the server restarts (messages is taken out of the JMS Queue and kept in memory until timer process it).
Another use of this would be to implement a "retry" queue. In case of error I want to retry processing my message, but not immediately, after a certain amount time only.
Any ideas to keep my MDB offline for a certain amount of time?
Most versions of JBoss publish a management MBean that allows you to stop delivery on a MDB.
If you're using EJB3, however, they auto-start, so you will need to register a startup class to stop starting MDBs at boot time if boots occur in your MDB's blackout period. Once past that snafu, you can schedule a simple quartz job to start and stop the MDBs according to your delivery windows.
Well, it looks like there is no way to pause a MDB in a generic way. The best solution is, like most people will answer, to use the DLQ (or DMQ).
Now, if I want to introduce a timer on a message, I set the time to live of the producer to the amount of time I want the message to wait. Then I send it to a normal queue, lets say waitingQueue which has no consumer. After expiration, the message is sent to default destination (mq.sys.dmq for Glassfish MQ, make sure to create a jms resource with mq.sys.dmq as imqDestinationName). I have a MDB listening to the error queue and responsible of sending the message again. Now, if I want to "close" a queue for some time, when a message arrives in the queue, I check if current time is allowed or not. Just set the time to live to the amount of time before next opening hours and send it to waitingQueue.
The reason I didn't use it since the beginning is that I fell into a few pitfalls. Here are a few useful properties to set when using DMQ with Glassfish 3.1.1 and its embedded MQ.
imq.message.expiration.interval=1 that's for the poll interval on each queue before sending timed out messages to the DMQ. Default is 60 seconds. If like me you want to test your application with little latency, this is what you need.
I have a persistent queue, non-transacted, client-acknowledge, the consumers read with jms.prefetchPolicy.queuePrefetch=1&wireFormat.maxInactivityDuration=50000
and once a consumer processes a message, it ack's the message.
If the consumer reads the message, and before it can send an ack, the process terminates abruptly, what happens in ActiveMq? (What ActiveMq parameters come into play here?)
How is that different than if the the consumer will take 10 minutes to process the message (so the consumer task is alive and working), how does ActiveMq know the message is still being worked on? (Does it monitor the TCP/IP connection, if the connection dies, it assumes the message will not be Ack'ed?)
How do I determine if a message is a "poison pill", i.e. it makes the consumers crash? (the redelivery count seems to be valid if the consumer task does not die; is there an internal counter in the message that says "it was been read n times without being successfully ack'ed?")
As an experiment, I sent 6 messages, one of them being a "poison pill" (kills the consumer before the consumer can send the ack), with 2 simultaneous consumers running (and automatically restarting consumers to bring the count to 2 whenever a consumer dies). Looking at the queue (using jconsole, I enable jmx using broker.setUseJmx(true)), 4 messages were delivered, 2 are in-flight. Why would there be 2 in-flight instead of just one?
I've been reading the ActiveMq and JMS specs for a while without clear/conclusive answers, so any insights on what parameters come into play, and if there are any known bugs, will be greatly useful.
This is purely based on my understanding of JMS - may not be completely correct:
If the consumer reads the message, and before it can send an ack, the process terminates abruptly, what happens in ActiveMq
My understanding is that since this happens in the context of a session with the JMS provider, JMS provider knows if the session is no longer active or has failed and any message not acknowledged as part of the session will be redelivered when the session is re-established.
How do I determine if a message is a "poison pill", i.e. it makes the consumers crash?
Like you have mentioned, the JMS provider keeps track of the # of times the message was redelivered possibly in the header of the message
4 messages were delivered, 2 are in flight
Not sure about this point