How can you see data accumulating in the ActiveMQ queues on the console? I have about 20-30 messages in ActiveMQ waiting for the Telegraph Listener to be activated.
However, when I look in the queues I don't see any numbers indicating there are messages in the queue. It's all zeros or ones. Despite that fact the actual functionality of durable message delivery is working.
Below is an image of what I see in the UI:
How do I view message counts/messages?
According to the screenshot all the InfluxData queues have consumers on them already which means the messages are almost certainly being consumed as they arrive. By default the console only refreshes every 5 seconds which means you may not be seeing the messages before they get consumed.
You should look at the view for the specific queue you're interested in and it will tell you how many messages have been added & consumed as well as how many are in delivery.
Related
I have a question about some strange behaviour of consumer.
Recently we had strange situation on production environment. Two consumers on two different microservices were stuck at some messages. The first one was holding 20 messages from rabbitMQ queue and the second one 2 messages and they weren't processing them. These messages were visible as Unacked in RabbitMQ for two days. They went back to Ready state just when that two microservices were restarted. At that time when consumers took this messages the whole program was processing thousands messages per hour, so basically our Saga and all consumers were working. When these messages went back to Ready state they were processed in one second after that so I don't think that it's problem with them.
The messages are published by Saga to Exchange and besides these two stucked consumers we have also EventLogger consumer subscribed to all messages and this EventLogger processed this 22 messages normally without any problems (from his own queue). Also we have connected Application Insights to consumers and there is no information about receiving these 22 messages by these two consumers (there are information about receiving it by EventLogger).
The other day we had the same issue with one message on test environment.
Recently we updated version of MassTransit in our project from version 6.2.0 to 7.1.6 and before that we didn't notice any similar issues with consumers but maybe it's just coincidence. We also have retry, redelivery, circuit breaker and in memory outbox mechanisms but I don't think that's problem with them because the consumer didn't even start to process these 22 messages.
Do you have any suggestions what could happened to this consumers?
Usually when a consumer doesn't even start to consume the message once it has been delivered to MassTransit by RabbitMQ, it could be an issue resolving the consumer from the container, such as a dependency to another backing service (database, log server, file, network connection, device, etc.).
The message remains unacknowledged on the broker because the transport/delivery mechanism to the consumer is waiting for a resource to become available. If there isn't anything in the logs for that time period indicating an issue with a resource, it's hard to know what could have blocked those messages from being consumed. The fact that they were ultimately consumed once the services were restarted seems to indicate the message content itself was fine.
Monitoring the lack of message consumption (and likely an associated queue depth increase) would give an indication that the situation has occurred. If it happens again, I'd increase the logging detail levels to see if the issue occurs again and can then be identified.
I have got started with ActiveMQ and able to move forward, but I am not able to figure out how I can view all messages that are sent to a particular queue.
I can see the message in the queue as long as it is not consumed by a consumer but as soon as it is consumed by a consumer, I can no longer see the message in the queue.
In my project I used to use "MQJ Explorer" and connect with a queue manager and there I could view all the messages that were sent to a particular queue, so I am expecting similar thing with ActiveMQ.
I know while sending I can have the message marked as "persistent" but still it will be there in the queue only till it is not consumed, once it is consumed I cannot see it any more.
Please let me know if things work differently with ActiveMQ.
The Queue holds messages until they are consumed at which point they are discarded. So you cannot browse all messages that were sent to the queue and then consumed. The only solution to this would be to use a Camel route or similar to mirror the Queue to some other Audit type Queue but you'd need to use a lot of care as this could easily lead to filling the message store as the mirrored messages will continue to grow and take space unless they are purged periodically.
I am using ActiveMQ v5.10.0 and having an issue almost every weekend where my ActiveMQ instance stops delivering persistent messages sent on queues to the consumers. I have not been able to figure out what could be causing this.
While the issue was happening I tried following things:
I added a new consumer on the affected queue but it didn't receive
any messages.
I restarted the original consumer but it didn't receive any messages after the restart.
I purged the messages that were held on the queue but then messages started accumulating again and broker didn't deliver any of the new messages. When I purged the expiry count didn't increase neither the dequeue and dispatch counters.
I sent 100 non-persistent messages on the affected queue, surprisingly it received those messages.
I tried sending 100 persistent messages on that queue, it didn't deliver anyone of them, all the messages were held by broker.
I created a fresh new queue and sent 100 persistent messages and none of them was delivered to the consumer whereas all the non-persistent messages were delivered.
The same things happen if I send persistent or non-persistent messages from STOMP producers. Surprisingly all this happened only for queues, topic consumers were able to receive persistent as well as non-persistent messages.
I have already posted this on ActiveMQ user forum: http://activemq.2283324.n4.nabble.com/Broker-not-delivering-persistent-messages-to-consumer-on-queue-td4691245.html but no one from ActiveMQ has suggested anything.
The jstack output also isn't very helping.
More details:
1. I am not using any selectors, message groups feature
2. I have disabled producer flow control in my setup
I want some suggestions as to what configuration values might cause this issue- memory limits, message TTL etc.
I have a JMS non-durable topic with multiple subscribers (using ActiveMQ).
Each subscriber is a web application deployed on a separate Tomcat instance.
When a message arrives, I want it delivered to all subscribers (somewhat of a distributed cache implementation).
I've noticed that not always the message gets delivered to all subscribers. When looking at the ActiveMQ admin UI I see the list of topic subscribers (as well as the list of connections) constantly changing. E.g. I see 17 subscribers, I refresh the ActiveMQ admin UI, then I see 19 subscribers, I refresh again - I see 18 etc.
What's wrong? I know that all the tomcats are up all the time. My understanding of JMS topics is that if the application is up, the message should always get delivered. Have I misunderstood something? I see no logical reason for this subscriber 'flashing'.
P.S. I'd rather not try durable topics - I don't need the message delivered if the application is not up.
I'm using ActiveMQ along with Mule (a kind of ESB based on Spring).
We got a fast producer and a slow consumer.
It's synchronous configuration with only one consumer.
Here the configuration of the consumer in spring style: http://pastebin.com/vweVd1pi
The biggest requirement is to keep the order of the messages.
However, after hours of running this code, suddenly, ActiveMQ skips 200 messages, and send the next ones.The 200 messages are still there in the activeMQ, they are not lost.
But our client (Mule), does have some custom code to check the order of the messages, using an unique identifier.
I had this issue already a few month ago. We change the consumer by using the parameter "jms.prefetchPolicy.queuePrefetch=1". It seemed to have worked well and to be the fix we needed unti now when the issue reappeared on another consumer.
Is it a bug, or a configuration issue ?
I can't talk about the requirement from a Mule perspective, but there are a couple of broker features that you should take a look at. There are two ways to guarantee message ordering in ActiveMQ:
Message groups are a way of ensuring that a set of related messages will be consumed by the same consumer in the order that they are placed on a queue. To use it you need to specify a JMSXGroupID header on related messages, and assign them an incrementing JMSXGroupSeq number. If a consumer dies, remaining messages from that group will be sent to another single consumer, while still preserving order.
Total message ordering applies to all messages on a topic. It is configured on the broker on a per-destination basis and requires no particular changes to client code. It comes with a synchronisation overhead.
Both features allow you to scale out to more than one consumer.