a cluster queue part of a single cluster is deleted without taking the queue out of the cluster in IBM MQ v9.1
. What issue will it cause?
Some messages might already be en-route to this queue, sitting on transmission queues, being moved by channels, and when the messages arrive at the target queue manager, the cluster channel will try to put them to the queue only to discover it has been deleted. The messages will hopefully then end up on a DLQ, but if you don't have one, and the messages are persistent, the channel will stop so that the messages stay safe, and that will mean other messages can't get through.
Altering the queue to take it out of the cluster, means that any en-route messages can still be put to the queue when they arrive, but no new messages will be sent that way as the queue is no longer in the cluster.
Then you can wait a little while - make sure all channels are running so all en-route messages can get through - and once no more messages are arriving for that queue, you can delete it later without the above problems happening.
Related
I have a question about some strange behaviour of consumer.
Recently we had strange situation on production environment. Two consumers on two different microservices were stuck at some messages. The first one was holding 20 messages from rabbitMQ queue and the second one 2 messages and they weren't processing them. These messages were visible as Unacked in RabbitMQ for two days. They went back to Ready state just when that two microservices were restarted. At that time when consumers took this messages the whole program was processing thousands messages per hour, so basically our Saga and all consumers were working. When these messages went back to Ready state they were processed in one second after that so I don't think that it's problem with them.
The messages are published by Saga to Exchange and besides these two stucked consumers we have also EventLogger consumer subscribed to all messages and this EventLogger processed this 22 messages normally without any problems (from his own queue). Also we have connected Application Insights to consumers and there is no information about receiving these 22 messages by these two consumers (there are information about receiving it by EventLogger).
The other day we had the same issue with one message on test environment.
Recently we updated version of MassTransit in our project from version 6.2.0 to 7.1.6 and before that we didn't notice any similar issues with consumers but maybe it's just coincidence. We also have retry, redelivery, circuit breaker and in memory outbox mechanisms but I don't think that's problem with them because the consumer didn't even start to process these 22 messages.
Do you have any suggestions what could happened to this consumers?
Usually when a consumer doesn't even start to consume the message once it has been delivered to MassTransit by RabbitMQ, it could be an issue resolving the consumer from the container, such as a dependency to another backing service (database, log server, file, network connection, device, etc.).
The message remains unacknowledged on the broker because the transport/delivery mechanism to the consumer is waiting for a resource to become available. If there isn't anything in the logs for that time period indicating an issue with a resource, it's hard to know what could have blocked those messages from being consumed. The fact that they were ultimately consumed once the services were restarted seems to indicate the message content itself was fine.
Monitoring the lack of message consumption (and likely an associated queue depth increase) would give an indication that the situation has occurred. If it happens again, I'd increase the logging detail levels to see if the issue occurs again and can then be identified.
I have got started with ActiveMQ and able to move forward, but I am not able to figure out how I can view all messages that are sent to a particular queue.
I can see the message in the queue as long as it is not consumed by a consumer but as soon as it is consumed by a consumer, I can no longer see the message in the queue.
In my project I used to use "MQJ Explorer" and connect with a queue manager and there I could view all the messages that were sent to a particular queue, so I am expecting similar thing with ActiveMQ.
I know while sending I can have the message marked as "persistent" but still it will be there in the queue only till it is not consumed, once it is consumed I cannot see it any more.
Please let me know if things work differently with ActiveMQ.
The Queue holds messages until they are consumed at which point they are discarded. So you cannot browse all messages that were sent to the queue and then consumed. The only solution to this would be to use a Camel route or similar to mirror the Queue to some other Audit type Queue but you'd need to use a lot of care as this could easily lead to filling the message store as the mirrored messages will continue to grow and take space unless they are purged periodically.
I have a Tibco EMS Message Queue on a production system that routes messages from a single producer to a single consumer application.
We are scaling the application vertically, but due to financial constraints, we cannot scale the two applications in pairs.
A single producer will route messages to the message broker and the message broker will need to bridge the message to 1 of 3 consumers based on a message selector.
I have set up a queue bridge and selector to route messages on the producer queue to consumer queues. This is a 1 to many queue bridge.
I noticed that the bridged consumer queues have consumers attached to them and they are receiving the messages correctly based on the selector, however the producer queue retains copies of the messages which must be manually purged at the end of day.
What is the best way to handle this scenario using bridges and selectors so that the message is retained on the broker until it is consumed (durable) but once a message is consumed by a consumer queue, the message is removed by the broker.
The easiest way to dispose those messages of the original queue is by introducing MaxMsgs and MaxBytes on the queue.
As for you requirement that the message on the original queue can only be disposes if it was consumed on one of the bridged queues, this is not possible.
That said, it also it not needed, since the bridged queue keep its own message copy, regardless of what happens to the message in the source queue. So expiring messages in the original queue has no effect on the already bridged messages
I am using ActiveMQ v5.10.0 and having an issue almost every weekend where my ActiveMQ instance stops delivering persistent messages sent on queues to the consumers. I have not been able to figure out what could be causing this.
While the issue was happening I tried following things:
I added a new consumer on the affected queue but it didn't receive
any messages.
I restarted the original consumer but it didn't receive any messages after the restart.
I purged the messages that were held on the queue but then messages started accumulating again and broker didn't deliver any of the new messages. When I purged the expiry count didn't increase neither the dequeue and dispatch counters.
I sent 100 non-persistent messages on the affected queue, surprisingly it received those messages.
I tried sending 100 persistent messages on that queue, it didn't deliver anyone of them, all the messages were held by broker.
I created a fresh new queue and sent 100 persistent messages and none of them was delivered to the consumer whereas all the non-persistent messages were delivered.
The same things happen if I send persistent or non-persistent messages from STOMP producers. Surprisingly all this happened only for queues, topic consumers were able to receive persistent as well as non-persistent messages.
I have already posted this on ActiveMQ user forum: http://activemq.2283324.n4.nabble.com/Broker-not-delivering-persistent-messages-to-consumer-on-queue-td4691245.html but no one from ActiveMQ has suggested anything.
The jstack output also isn't very helping.
More details:
1. I am not using any selectors, message groups feature
2. I have disabled producer flow control in my setup
I want some suggestions as to what configuration values might cause this issue- memory limits, message TTL etc.
I have a set up of an ActiveMQ broker and a single consumer. Consumer gets a message that he is not able to process because a service that it depends has a bug (once fixed it will be fine). So the message keeps being redelivered (consumer redelivery) - we use JMS sessions. With our current configuration it will keep redelivering it every 10 minutes for 1 day. That obviously causes a problem because other messages are not being consumed.
In order to solve this problem I have accessed the queue through JMX and tried to delete that message but it is not there. I guess it is cached on the consumer and not visible at the broker.
Is there any way to delete this message other than restarting the application?
Is it possible to configure the redelivery mechanism so that such message (that causes a live lock eventually) is put at the end of the queue so that other messages can be processed?
The 10 minutes for 1 day redelivery policy should stay as is.
I think you're right that the messages are stuck in the consumer's prefetch buffer, and I don't know of a way to delete them from there.
I'd change your redelivery policy to send to the DLQ after the second failure, with a much shorter interval between them, like 30 seconds, and I'd configure the DLQ strategy as an individualDeadLetterStrategy so you get a separate DLQ containing only messages from this particular queue. Then set up a consumer on this DLQ to move the messages to (the end of) the main queue whenever your reprocessing condition is met (whether that's after a certain delay, or based on reading some flag value from a database, or whatever). This consumer is where you'd implement "every 10 minutes for 1 day" logic, instead of in the redelivery policy where you currently have it.
That will keep the garbage ones out of the main queue so they don't delay other messages from being consumed, but still ensure that they will be reprocessed later. And it will put them on the broker instead of in the consumer's prefetch buffer, where you can view and delete them.
The only way to get it to the back of the queue is to reproduce it to the queue. Redelivery polices can only be configured down to the destination on the connection factory.
Given that you already have a connection, it shouldn't be to hard to create a producer that can either move the given message to a DLQ or produce it back to the queue when you run into that particular bug.
Setting jms.nonBlockingRedelivery=true on the connection factory resolved the problem. Now even if there is a message redelivered it does not block processing of other Messages.