We're currently having problems with our ActiveMQ 5.16.1 which suddenly starts piling up messages without any apparent reason. The following image shows the ActiveMQ QueueSize:
The ActiveMQ is used as JMS message broker without any other components for e.g. high availability or load balancing. Several producers (in total and worst case around 20) produce small/simple JSON messages which are send to the broker and consumed by a JAVA-based microservice. The microservice processes the message and saves the data to an Oracle database. Average processing time for one request is about 30ms. From those 20 producers only some are active at the same time which might vary between 2 and 10 producers. Each producer sends a message every 3 secondes resulting in 20 messages/min per producer. E.g.: having 10 producers the broker will get 200 messages/min or 30 messages/sec. Preserving the order is crucital thus I'm working with JMSXGroupIds which works good so far. Messages are send via MQTT and routed (via Camel) to an JMS queue:
<route id="handleData">
<from uri="activemq://topic:some.topic.here?clientId=uniqueClientId" />
<setHeader headerName="tName">
<constant>ABC123</constant>
</setHeader>
<setHeader headerName="JMSXGroupId">
<jsonpath>$.producerId</jsonpath>
</setHeader>
<to uri="activemq://queue:myQueue" />
</route>
But for any reason the messages get stuck after some time and I can't find any significant hint why that happens. There is nothing in the log files nor the OS event log. I have to restart the ActiveMQ service in order to "reanimate" it. Afterwards all stuck messages will be processed and everything is working fine until the next "accident". This time it took about 10 days before the messages got stuck.
I already checked whether there might be a network or database-related issue. Even moved the ActiveMQ to a freshly new server in order to asure that nothing else is influencing the ActiveMQ processes. But I couldn't find any hints either. I watched the JVM, heap space growth, memory usage, etc. - everything unremarkable.
Does anybody has an idea what I could check additionally to find out what the problem is?
Add the advisoryForSlowConsumer destinationPolicy setting for those topics and watch the topic://ActiveMQ.Advisory.. for any enqueue counts that would indicate slow consumer occurred.
Your Camel route is almost identical to what a VirtualTopic does. You'll see better consistency with server-side routing in these types of scenarios, since there is no remote process (ie. Camel route) to manage connections, sessions, etc.
Bonus: MQTT transport supports using Virtual Topics to back-end the subscriptions so you any MQTT topic consumers would automatically pull from the queue.
ref: https://activemq.apache.org/virtual-destinations
Related
I have a question about some strange behaviour of consumer.
Recently we had strange situation on production environment. Two consumers on two different microservices were stuck at some messages. The first one was holding 20 messages from rabbitMQ queue and the second one 2 messages and they weren't processing them. These messages were visible as Unacked in RabbitMQ for two days. They went back to Ready state just when that two microservices were restarted. At that time when consumers took this messages the whole program was processing thousands messages per hour, so basically our Saga and all consumers were working. When these messages went back to Ready state they were processed in one second after that so I don't think that it's problem with them.
The messages are published by Saga to Exchange and besides these two stucked consumers we have also EventLogger consumer subscribed to all messages and this EventLogger processed this 22 messages normally without any problems (from his own queue). Also we have connected Application Insights to consumers and there is no information about receiving these 22 messages by these two consumers (there are information about receiving it by EventLogger).
The other day we had the same issue with one message on test environment.
Recently we updated version of MassTransit in our project from version 6.2.0 to 7.1.6 and before that we didn't notice any similar issues with consumers but maybe it's just coincidence. We also have retry, redelivery, circuit breaker and in memory outbox mechanisms but I don't think that's problem with them because the consumer didn't even start to process these 22 messages.
Do you have any suggestions what could happened to this consumers?
Usually when a consumer doesn't even start to consume the message once it has been delivered to MassTransit by RabbitMQ, it could be an issue resolving the consumer from the container, such as a dependency to another backing service (database, log server, file, network connection, device, etc.).
The message remains unacknowledged on the broker because the transport/delivery mechanism to the consumer is waiting for a resource to become available. If there isn't anything in the logs for that time period indicating an issue with a resource, it's hard to know what could have blocked those messages from being consumed. The fact that they were ultimately consumed once the services were restarted seems to indicate the message content itself was fine.
Monitoring the lack of message consumption (and likely an associated queue depth increase) would give an indication that the situation has occurred. If it happens again, I'd increase the logging detail levels to see if the issue occurs again and can then be identified.
I am using ActiveMQ v5.10.0 and having an issue almost every weekend where my ActiveMQ instance stops delivering persistent messages sent on queues to the consumers. I have not been able to figure out what could be causing this.
While the issue was happening I tried following things:
I added a new consumer on the affected queue but it didn't receive
any messages.
I restarted the original consumer but it didn't receive any messages after the restart.
I purged the messages that were held on the queue but then messages started accumulating again and broker didn't deliver any of the new messages. When I purged the expiry count didn't increase neither the dequeue and dispatch counters.
I sent 100 non-persistent messages on the affected queue, surprisingly it received those messages.
I tried sending 100 persistent messages on that queue, it didn't deliver anyone of them, all the messages were held by broker.
I created a fresh new queue and sent 100 persistent messages and none of them was delivered to the consumer whereas all the non-persistent messages were delivered.
The same things happen if I send persistent or non-persistent messages from STOMP producers. Surprisingly all this happened only for queues, topic consumers were able to receive persistent as well as non-persistent messages.
I have already posted this on ActiveMQ user forum: http://activemq.2283324.n4.nabble.com/Broker-not-delivering-persistent-messages-to-consumer-on-queue-td4691245.html but no one from ActiveMQ has suggested anything.
The jstack output also isn't very helping.
More details:
1. I am not using any selectors, message groups feature
2. I have disabled producer flow control in my setup
I want some suggestions as to what configuration values might cause this issue- memory limits, message TTL etc.
I'm working on this problem a lot of time now and can't seem to find any final solution to it.
I have a message producer that should work as a broadcaster, posting messages on two different topics. The publisher's posting process follows the flow :
Creating connection to the factory and starting it.
Creating session
Creating Message Producer by using the session and a given topic Name
Sending n* of messages
Waiting n seconds
Closing producer,session,connection
Then i have 3 consumers subscribed to those topics by using the following configuration (each consumer has it's own clientId and durableSubscriptionName):
<route id="consumerOneRoute">
<from uri="activemq:topic:topicName?clientId=consumerOneId&durableSubscriptionName=ConsumerOneName" />
<bean ref="consumerBean" method="processMessage" />
</route>
The fact is that my consumers don't always receive the messages, at least not all of them. Sometimes two of the consumers get all the messages and the 3rd one don't get any, sometimes random consumer receive random number of messages and so on...
One more fact that i noticed is that if i stop the broker and start it again, the consumers will receive the missing messages, and i really can't understand why won't that happen during the 1st lifetime of the broker.
Would anyone be so kind and try to aid me?
Thanks,
George.
P.S : I was thinking about using virtual topics though since my main purpose is to have a broadcasting producer that will allow other consumers to attach in the future i don't want to have the need of modifyin' the producer everytime by adding another virtual branch to the main topic.
I had similar issue, 1 producer sends messages via topic to many consumers, not all of them receives messages. Problem was in consumer's timeout, I manually created timeout and it was shorter then ActiveMQ could deliver last messages. Extending timeout - helped.
Take a look at Prefetch Policy. If you set it to 1 then it may fix this one for you.
...&consumer.prefetchSize=1
The only solution that worked was switching from Topics with durable consumers to Virtual Topics.
your consumers must be connected to broker when producer is sending message.
I have a single 'point to point' IBM MQ queue receiving messages from multiple producers. My application consumes the messages from the queue. I am using spring 'jmstemplate" and "DefaultMessageListenerContainer" to consume the messages asynchronously.
My application is running on 2 jvms, meaning there are 2 listeners active on each jvm listening to the same queue.
Coming to my questions, If a message arrives...
1) How does the listeners know that the message arrived in the queue?
2) Out of the two listeners, which one will receive the message ? What is the approach followed to distribute the messages to the listeners?
3) Can i scale to 'N' count of listeners for a singe queue? If i grow to 10 listeners, how does the scaling work? how are the messages distributed to listeners?
4) How does the MQ server make sure that the same message is not sent to multiple listeners?
May be these are simple questions, but not able to drill down on how the above scenarios works. Please share your thoughts...
That's a function of the IBM client library; the listener container simply polls the JMS API waiting for a message; by default, it uses a 1 second receive timeout; with TRACE level logging, you will see log messages showing this activity. The timeout can be modified by setting receiveTimeout on the container.
That is indeterminate from the client's perspective; the IBM broker knows how many consumers are registered and picks one. Some brokers allow configuration of a pre-fetch; this can help performance under high volume but can hurt performance under low volume.
Yes; the Spring Listener Container can dynamically scale the listeners based on load; you can configure min/max consumers and Spring will adjust within those bounds as necessary. Each listener is a separate consumer, as far as the broker is concerned so the work is distributed according to the broker's algorithms.
That's a function of the IBM broker (and part of the JMS contract).
If using transactions and a message is rolled back onto the queue; there is no guarantee that the same listener will get the re-delivered message.
I'm using ActiveMQ along with Mule (a kind of ESB based on Spring).
We got a fast producer and a slow consumer.
It's synchronous configuration with only one consumer.
Here the configuration of the consumer in spring style: http://pastebin.com/vweVd1pi
The biggest requirement is to keep the order of the messages.
However, after hours of running this code, suddenly, ActiveMQ skips 200 messages, and send the next ones.The 200 messages are still there in the activeMQ, they are not lost.
But our client (Mule), does have some custom code to check the order of the messages, using an unique identifier.
I had this issue already a few month ago. We change the consumer by using the parameter "jms.prefetchPolicy.queuePrefetch=1". It seemed to have worked well and to be the fix we needed unti now when the issue reappeared on another consumer.
Is it a bug, or a configuration issue ?
I can't talk about the requirement from a Mule perspective, but there are a couple of broker features that you should take a look at. There are two ways to guarantee message ordering in ActiveMQ:
Message groups are a way of ensuring that a set of related messages will be consumed by the same consumer in the order that they are placed on a queue. To use it you need to specify a JMSXGroupID header on related messages, and assign them an incrementing JMSXGroupSeq number. If a consumer dies, remaining messages from that group will be sent to another single consumer, while still preserving order.
Total message ordering applies to all messages on a topic. It is configured on the broker on a per-destination basis and requires no particular changes to client code. It comes with a synchronisation overhead.
Both features allow you to scale out to more than one consumer.