I am using ActiveMQ v5.10.0 and having an issue almost every weekend where my ActiveMQ instance stops delivering persistent messages sent on queues to the consumers. I have not been able to figure out what could be causing this.
While the issue was happening I tried following things:
I added a new consumer on the affected queue but it didn't receive
any messages.
I restarted the original consumer but it didn't receive any messages after the restart.
I purged the messages that were held on the queue but then messages started accumulating again and broker didn't deliver any of the new messages. When I purged the expiry count didn't increase neither the dequeue and dispatch counters.
I sent 100 non-persistent messages on the affected queue, surprisingly it received those messages.
I tried sending 100 persistent messages on that queue, it didn't deliver anyone of them, all the messages were held by broker.
I created a fresh new queue and sent 100 persistent messages and none of them was delivered to the consumer whereas all the non-persistent messages were delivered.
The same things happen if I send persistent or non-persistent messages from STOMP producers. Surprisingly all this happened only for queues, topic consumers were able to receive persistent as well as non-persistent messages.
I have already posted this on ActiveMQ user forum: http://activemq.2283324.n4.nabble.com/Broker-not-delivering-persistent-messages-to-consumer-on-queue-td4691245.html but no one from ActiveMQ has suggested anything.
The jstack output also isn't very helping.
More details:
1. I am not using any selectors, message groups feature
2. I have disabled producer flow control in my setup
I want some suggestions as to what configuration values might cause this issue- memory limits, message TTL etc.
Related
How can you see data accumulating in the ActiveMQ queues on the console? I have about 20-30 messages in ActiveMQ waiting for the Telegraph Listener to be activated.
However, when I look in the queues I don't see any numbers indicating there are messages in the queue. It's all zeros or ones. Despite that fact the actual functionality of durable message delivery is working.
Below is an image of what I see in the UI:
How do I view message counts/messages?
According to the screenshot all the InfluxData queues have consumers on them already which means the messages are almost certainly being consumed as they arrive. By default the console only refreshes every 5 seconds which means you may not be seeing the messages before they get consumed.
You should look at the view for the specific queue you're interested in and it will tell you how many messages have been added & consumed as well as how many are in delivery.
I have a question about some strange behaviour of consumer.
Recently we had strange situation on production environment. Two consumers on two different microservices were stuck at some messages. The first one was holding 20 messages from rabbitMQ queue and the second one 2 messages and they weren't processing them. These messages were visible as Unacked in RabbitMQ for two days. They went back to Ready state just when that two microservices were restarted. At that time when consumers took this messages the whole program was processing thousands messages per hour, so basically our Saga and all consumers were working. When these messages went back to Ready state they were processed in one second after that so I don't think that it's problem with them.
The messages are published by Saga to Exchange and besides these two stucked consumers we have also EventLogger consumer subscribed to all messages and this EventLogger processed this 22 messages normally without any problems (from his own queue). Also we have connected Application Insights to consumers and there is no information about receiving these 22 messages by these two consumers (there are information about receiving it by EventLogger).
The other day we had the same issue with one message on test environment.
Recently we updated version of MassTransit in our project from version 6.2.0 to 7.1.6 and before that we didn't notice any similar issues with consumers but maybe it's just coincidence. We also have retry, redelivery, circuit breaker and in memory outbox mechanisms but I don't think that's problem with them because the consumer didn't even start to process these 22 messages.
Do you have any suggestions what could happened to this consumers?
Usually when a consumer doesn't even start to consume the message once it has been delivered to MassTransit by RabbitMQ, it could be an issue resolving the consumer from the container, such as a dependency to another backing service (database, log server, file, network connection, device, etc.).
The message remains unacknowledged on the broker because the transport/delivery mechanism to the consumer is waiting for a resource to become available. If there isn't anything in the logs for that time period indicating an issue with a resource, it's hard to know what could have blocked those messages from being consumed. The fact that they were ultimately consumed once the services were restarted seems to indicate the message content itself was fine.
Monitoring the lack of message consumption (and likely an associated queue depth increase) would give an indication that the situation has occurred. If it happens again, I'd increase the logging detail levels to see if the issue occurs again and can then be identified.
I am using Nifi to get data from IBM MQ. It is working fine. My question is once the message is read from an MQ queue, does it get deleted from the queue? How to just read messages from the queue without deleting them from the queue?
My question is once the message is read from an MQ queue, does it get
deleted from the queue?
Yes, that is the default behavior.
How to just read messages from the queue without deleting them from
the queue?
You use the option: MQGMO_BROWSE_FIRST followed by MQGMO_BROWSE_NEXT on the MQGET API calls.
You can also open the queue for browse only. i.e. MQOO_BROWSE option for MQOPEN API call.
It sounds as if you would like to use a "publish/subscribe" model rather than a "point-to-point" model.
From ActiveMQ:
Topics In JMS a Topic implements publish and subscribe semantics. When
you publish a message it goes to all the subscribers who are
interested - so zero to many subscribers will receive a copy of the
message. Only subscribers who had an active subscription at the time
the broker receives the message will get a copy of the message.
Queues A JMS Queue implements load balancer semantics. A single
message will be received by exactly one consumer. If there are no
consumers available at the time the message is sent it will be kept
until a consumer is available that can process the message. If a
consumer receives a message and does not acknowledge it before closing
then the message will be redelivered to another consumer. A queue can
have many consumers with messages load balanced across the available
consumers.
If you have a queue, when a consumer consumes that message, it is removed from the queue so that the next consumer consumes the next message. With a topic, multiple consumers can be subscribed to that topic and retrieve the same message without being exclusive.
If neither of these work for you, I'm not sure what semantics you're looking for -- a "queue" which doesn't delete the message when it is consumed will never let a consumer access any but the first message.
I am using weblogic 10.3 .
I am trying to configure a durable subscription with persistent messaged backed by a jdbc store (in Oracle DB). I have a topic to which an MDB is listening as a durable subscriber.
Under scenario-1 : If I send the message, it hits the MDB.
Under scenario-2 : I suspend the MDB hoping that the messages send to the topic will stay around as long as they don’t get consumed by the MDB (which is the only registered durable subscriber). But when I do send the message to the topic, it briefly shows up there and then it goes away (i see it using HermesJMS).
I was under the impression that the messages since they are not being consumed by the MDB will get logged into the JDBC store in this case but the the WLSTORE table in oracle db does not get any messages persisted in it either.
I later found out that the messages do show up in the topic > durable subscribers > Show messages in the admin console.
So apparantly what’s happening is topic doesn’t keep the message, but the durable subscription that’s registered under it, keeps it until the message does get consumed.
Question -1) But the fact that messages are not going to the Oracle based jdbc store, still doesn’t make sense ? What am I doing wrong here ?
Question -2) Even when I do resume the MDB to start listening for the messages, the topic keeps showing all the message still intact under the durable subscribers (in admin console) - I was hoping they will get removed from there as they got processed by the only registered durable subscriber.
Messages do not show up in the topic, since topics and queues are different communication models. Assume you have a durable topic with 2 (durable) subscribers: A and B. You want both of them to get the message. To ensure that, A and B both have to confirm they received the message.
This is also the reason why you get all message redelivered, after reconnecting, your MDB has to call commit() on the message, to tell the server it is done processing.
This also explains why the topic itself does not store messages, they are stored per durable consumer. Because A might commit the message, but B will not (might be "offline"). So you need a copy for each consumer.
I found a couple of interesting things in further testing -
for Question 1) _ Even if I don't configure the jdbc store for the JMS server in weblogic 10, it has its own default file store which is always working without any configuration. This file store is used for storing the persistent messages for durable subscriptions and that store will keep the messages around even across server restarts.
For more reading - http://docs.oracle.com/cd/E17904_01/web.1111/e13701/store.htm#i1130575
for Question 2)_ My MDB was expecting XML messages in certain format, during my testing to keep things simple, i started sending small text messages and forgot that they will not get processed successfully in the MDB. so the MDB was failing on all those messages, and rolling back the transaction which is why the messages were never getting removed when I resumed the MDB.
This pretty much answers both my questions.
I have a persistent queue, non-transacted, client-acknowledge, the consumers read with jms.prefetchPolicy.queuePrefetch=1&wireFormat.maxInactivityDuration=50000
and once a consumer processes a message, it ack's the message.
If the consumer reads the message, and before it can send an ack, the process terminates abruptly, what happens in ActiveMq? (What ActiveMq parameters come into play here?)
How is that different than if the the consumer will take 10 minutes to process the message (so the consumer task is alive and working), how does ActiveMq know the message is still being worked on? (Does it monitor the TCP/IP connection, if the connection dies, it assumes the message will not be Ack'ed?)
How do I determine if a message is a "poison pill", i.e. it makes the consumers crash? (the redelivery count seems to be valid if the consumer task does not die; is there an internal counter in the message that says "it was been read n times without being successfully ack'ed?")
As an experiment, I sent 6 messages, one of them being a "poison pill" (kills the consumer before the consumer can send the ack), with 2 simultaneous consumers running (and automatically restarting consumers to bring the count to 2 whenever a consumer dies). Looking at the queue (using jconsole, I enable jmx using broker.setUseJmx(true)), 4 messages were delivered, 2 are in-flight. Why would there be 2 in-flight instead of just one?
I've been reading the ActiveMq and JMS specs for a while without clear/conclusive answers, so any insights on what parameters come into play, and if there are any known bugs, will be greatly useful.
This is purely based on my understanding of JMS - may not be completely correct:
If the consumer reads the message, and before it can send an ack, the process terminates abruptly, what happens in ActiveMq
My understanding is that since this happens in the context of a session with the JMS provider, JMS provider knows if the session is no longer active or has failed and any message not acknowledged as part of the session will be redelivered when the session is re-established.
How do I determine if a message is a "poison pill", i.e. it makes the consumers crash?
Like you have mentioned, the JMS provider keeps track of the # of times the message was redelivered possibly in the header of the message
4 messages were delivered, 2 are in flight
Not sure about this point