We're having issues with one of the consumers, and need to debug the code. SimpleRabbitListenerContainerFactory allows to set a ConsumerTagStrategy, which should add tags during logging.
#Bean
public SimpleRabbitListenerContainerFactory analyzeTransactionListenerContainerFactory(ConnectionFactory connectionFactory, AsyncTaskExecutor asyncTaskExecutor) {
connectionFactory.getVirtualHost());
SimpleRabbitListenerContainerFactory factory = new SimpleRabbitListenerContainerFactory();
factory.setConnectionFactory(connectionFactory);
factory.setConcurrentConsumers(2);
factory.setMaxConcurrentConsumers(4);
factory.setTaskExecutor(asyncTaskExecutor);
ConsumerTagStrategy consumerTagStrategy = new ConsumerTagStrategy() {
#Override
public String createConsumerTag(String queue) {
return queue;
}
};
factory.setConsumerTagStrategy(consumerTagStrategy);
return factory;
}
However, the logs still do not have the tag. So there is no way to find what queue/consumer this message is for.
LogLevel=DEBUG; category=org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer; msg=Cancelling Consumer: tags=[{}], channel=Cached Rabbit Channel: AMQChannel(amqp://guest#10.17.1.13:5672/,47), acknowledgeMode=AUTO local queue size=0;
LogLevel=DEBUG; category=org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer; msg=Idle consumer terminating: Consumer: tags=[{}], channel=Cached Rabbit Channel: AMQChannel(amqp://guest#10.17.1.13:5672/,47), acknowledgeMode=AUTO local queue size=0;
How does one add some tags to SimpleMessageListenerContainer logging?
Well, looks like you misunderstood the Consumer Key a bit. From the RabbitMQ docs:
consumer-tag consumer-tag
Specifies the identifier for the consumer. The consumer tag is local to a channel, so two clients can use the same consumer tags. If this field is empty the server will generate a unique tag.
The client MUST NOT specify a tag that refers to an existing consumer. Error code: not-allowed
The consumer tag is valid only within the channel from which the consumer was created. I.e. a client MUST NOT create a consumer in one channel and then use it in another. Error code: not-allowed
The SimpleMessageListenerContainer populates its internal map for the consumerTags when it starts to listen to provided queues. If we are listening (consuming) there we can see them from the:
public String toString() {
return "Consumer: tags=[" + (this.consumerTags.toString()) + "], channel=" + channel
+ ", acknowledgeMode=" + acknowledgeMode + " local queue size=" + queue.size();
}
as you expecting, of course.
But if we don't consumer any more, in case of Channel#basicCancel, for example.
Or... looking to your second log message during idling. When you have an extra Consumer, but there is no message for it.
The code on the matter looks like:
boolean receivedOk = receiveAndExecute(this.consumer); // At least one message received
if (SimpleMessageListenerContainer.this.maxConcurrentConsumers != null) {
if (receivedOk) {
.....
}
else {
consecutiveMessages = 0;
if (consecutiveIdles++ > SimpleMessageListenerContainer.this.consecutiveIdleTrigger) {
considerStoppingAConsumer(this.consumer);
consecutiveIdles = 0;
}
}
}
So, maybe you worry for nothing. And you see the logs only for those extra consumers. For this case you have maxConcurrentConsumers > concurrentConsumers.
Related
I have been reading the documentation for virtual destinations here: http://activemq.apache.org/virtual-destinations.html
But I hit a bit of a snag, when I send to a topic it does not seem to follow the client id name as described on the document
My setup on the active mq is:
<destinationInterceptors>
<virtualDestinationInterceptor>
<virtualDestinations>
<virtualTopic name="Destination.>" prefix="Target.*." selectorAware="false" />
</virtualDestinations>
</virtualDestinationInterceptor>
</destinationInterceptors>
The code above describes that when I send to a Destination.Status topic with a ClientId of CustomerA.
It should send only to Target.CustomerA.Destination.Status if understand correctly, but what's happening is it's sending to Target.CustomerA.Destination.Status and Target.CustomerB.Destination.Status so basically fanning out messages to queues and ignoring the client id.
I did not see any further documentation about how to configure it, i was wondering if anyone else encountered this ?
Am I missing something here ?
Below is my producer if it's helpful.
public static class HelloWorldProducer implements Runnable {
public void run() {
try {
// Create a ConnectionFactory
ActiveMQConnectionFactory connectionFactory = new ActiveMQConnectionFactory("tcp://localhost:61617");
// Create a Connection
Connection connection = connectionFactory.createConnection();
connection.setClientID("CustomerA");
connection.start();
// Create a Session
Session session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
// Create the destination (Topic or Queue)
Destination destination = session.createTopic("Destination.Status");
// Create a MessageProducer from the Session to the Topic or Queue
MessageProducer producer = session.createProducer(destination);
producer.setDeliveryMode(DeliveryMode.NON_PERSISTENT);
// Create a messages
String text = "Hello world! From: " + Thread.currentThread().getName() + " : " + this.hashCode();
TextMessage message = session.createTextMessage(text);
// Tell the producer to send the message
System.out.println("Sent message: "+ message.hashCode() + " : " + Thread.currentThread().getName());
producer.send(message);
// Clean up
session.close();
connection.close();
}
catch (Exception e) {
System.out.println("Caught: " + e);
e.printStackTrace();
}
}
}
Any inputs will be beneficial.
The sender in this scenario has no real effect on the routing at the broker whether or not you've set a ClientID as it is just sending to a named Topic, in this case "Destination.Status". The configuration on the broker controls the routing and in your case you've configured "Destination.>" so any Queue consumer that comes along and subscribes to a Queue that matches the configuration you've set. So in your case I'd guess you have one consumer subscribing to Queue (Target.CustomerA.Destination.Status) and one to Queue (Target.CustomerB.Destination.Status) which then causes any message sent to the Topic to be fanned out to both.
If you want competing consumers then you'd need to subscribe both to Target.CustomerA.Destination.Status and then the broker would round-robin dispatch the sent message to either of the active subscribers.
The requirement is like to process the messages from dead letter queue by exposed a REST service API(Spring Boot).
So that once REST service is called, one message will be consumed from the DL queue and will publish in the main queue again for processing.
#RabbitListener(queues = "QUEUE_NAME") consumes the message immediately which is not required as per the scenario. The message only has to be consumed by the REST service API.
Any suggestion or solution?
I do not think RabbitListener will help here.
However you could implement this behaviour manually.
Spring Boot automatically creates RabbitMq connection factory so you could use it. When http call is made just read single message from the queue manually, you could use basic.get to synchronously get just one message:
#Autowire
private ConnectionFactory factory
void readSingleMessage() {
Connection connection = null;
Channel channel = null;
try {
connection = factory.newConnection();
channel = connection.createChannel();
channel.queueDeclare(QUEUE_NAME, true, false, false, null);
GetResponse response = channel.basicGet(QUEUE_NAME, true);
if (response != null) {
//Do something with the message
}
} finally {
//Check if not null
channel.close();
connection.close();
}
}
If you are using Spring; you can avoid all the boilerplate in the other answer using RabbitTemplate.receive(...).
EDIT
To manually ack/reject the message, use the execute method instead.
template.execute(channel -> {
GetResponse got = channel.basicGet("foo", false);
// ...
channel.basicAck(got.getEnvelope().getDeliveryTag(), false);
return null;
});
It's a bit lower level, but again, most of the boilerplate is taken care of for you.
I would like to create a simple code snippet that fetches all messages from the DLQ and re-sends them to the original destination (AKA resend/retry)
It can be done easily by the ActiveMQ UI (but for a single message at a time).
There is no direct JMS API for re-sending a message from a DLQ to its original queue. In fact, the JMS API doesn't even discuss dead-letter queues. It's merely a convention used by most brokers to deal with messages that can't be consumed.
You'd need to create an actual JMS consumer to receive the message from the DLQ and then create a JMS producer to send the message back to its original queue.
It's important that you use Session.TRANSACTED mode to avoid potential message loss or duplication.
If you use Session.AUTO_ACKNOWLEDGE and there is a problem between the time the message is consumed and sent (e.g the application crashes, hardware failure, etc.) then the message could be lost due to the fact that it was already acknowledged before it was sent successfully.
If you use Session.CLIENT_ACKNOWLEDGE and there is a problem between the time the message is sent and acknowledged then the message could ultimately be duplicated due to the fact that it was already sent before it was acknowledged successfully.
Both operations should be part of the JMS transaction so that the work is atomic.
Lastly, I recommend you either invoke commit() on the transacted session for each message sent or after a small batch of messages (e.g. 10). Given that you have no idea how many messages are in the DLQ it would be unwise to process every message in a single transaction. Generally you want the transaction to be as small as possible in order to minimize the window during which an error might occur and the transaction's work will need to be performed again. Also, the larger the transaction is the more heap memory will be required on the broker to keep track of the work in the transaction. Keep in mind that you can invoke commit() on the same session as many times as you want. You don't need to create a new session for each transaction.
Retrying all messages on the DLQ is already implemented in activemq as an mbean.
You can trigger the retry method with jmxterm/jolokia
e.g
Replaying all messages on queue ActiveMQ.DLQ with jolokia
curl -XGET --user admin:admin --header "Origin: http://localhost" http://localhost:8161/api/jolokia/exec/org.apache.activemq:brokerName=localhost,destinationName=ActiveMQ.DLQ,destinationType=Queue,type=Broker/retryMessages
NOTE: You can only use this method on a queue that is marked as a DLQ. It will not work for regular queues.
Also the DLQ queue can have its 'DLQ' flag set to false if the server is restarted. It is automatically set to true when a new message is sent to the DLQ
After Justin's reply I've manually implemented the retry mechanism like so:
public void retryAllDlqMessages() throws JMSException {
logger.warn("retryAllDlqMessages starting");
logger.warn("Creating a connection to {}", activemqUrl);
ActiveMQConnectionFactory connectionFactory = new ActiveMQConnectionFactory("test", "test", activemqUrl);
HashMap<String, MessageProducer> messageProducersMap = new HashMap<>();
MessageConsumer consumer = null;
try (ActiveMQConnection connection = (ActiveMQConnection) connectionFactory.createConnection();
ActiveMQSession session = (ActiveMQSession) connection.createSession(true, Session.SESSION_TRANSACTED)) {
String dlqName = getDlqName();
logger.warn("Creating a session to {}", dlqName);
ActiveMQQueue queue = (ActiveMQQueue) session.createQueue(dlqName);
logger.warn("Starting JMS Connection");
connection.start();
logger.warn("Creating a DLQ consumer");
consumer = session.createConsumer(queue);
logger.warn("Consumer start receiving");
Message message = consumer.receive(CONSUMER_RECEIVE_TIME_IN_MS);
int retriedMessages = 0;
while (message != null) {
try {
retryMessage(messageProducersMap, session, message);
retriedMessages++;
} catch (Exception e) {
logger.error("Error calling retryMessage for message = {}", message);
logger.error("Rolling back the JMS transaction...");
session.rollback();
return;
}
message = consumer.receive(CONSUMER_RECEIVE_TIME_IN_MS);
}
logger.warn("Consumer finished retrying {} messages", retriedMessages);
logger.warn("Commiting JMS Transactions of retry");
session.commit();
} finally {
if (!messageProducersMap.isEmpty()) {
logger.warn("Closing {} messageProducers in messageProducersMap", messageProducersMap.size());
for (MessageProducer producer : messageProducersMap.values()) {
producer.close();
}
}
if (consumer != null) {
logger.warn("Closing DLQ Consumer");
consumer.close();
}
}
}
private void retryMessage(HashMap<String, MessageProducer> messageProducersMap, ActiveMQSession session, Message message) {
ActiveMQObjectMessage qm = (ActiveMQObjectMessage) message;
String originalDestinationName = qm.getOriginalDestination().getQualifiedName();
logger.warn("Retry message with JmsID={} to original destination {}", qm.getJMSMessageID(), originalDestinationName);
try {
if (!messageProducersMap.containsKey(originalDestinationName)) {
logger.warn("Creating a new producer for original destination: {}", originalDestinationName);
messageProducersMap.put(originalDestinationName, session.createProducer(qm.getOriginalDestination()));
}
logger.info("Producing message to original destination");
messageProducersMap.get(originalDestinationName).send(qm);
logger.info("Message sent");
} catch (Exception e) {
logger.error("Message retry failed with exception", e);
}
}
i define a queue channel
#Bean("mail-action-laundry-list-channel")
public MessageChannel mailRecipientActionMessageChannel() {
return new QueueChannel(20);
}
the flow below, i will aggrate messages from the queue channel, i tried this:
#Bean
public IntegrationFlow mailRecipientActionLaundryListMessageFlow(#Qualifier("laundryListMessageHandler") MessageHandler laundryListMessageHandler) {
return IntegrationFlows.from("mail-action-laundry-list-channel")
.log("--> laundry list messages:：")
.aggregate(aggregatorSpec -> aggregatorSpec
.correlationExpression("#this.payload.email")
.releaseExpression("#this.size() == 5")
.messageStore(new SimpleMessageStore(100))
.groupTimeout(2000))
.transform(laundryListMessageToItemProcessDtoTransformer())
.handle(laundryListMessageHandler)
.get();
}
but why it aggrate first 5 messages from the channel always, and aggrate other message no longer
You need to configure expireGroupsUponCompletion(true) on the aggregator:
When set to true (default false), completed groups are removed from the message store, allowing subsequent messages with the same correlation to form a new group. The default behavior is to send messages with the same correlation as a completed group to the discard-channel.
Looks like your subsequent messages from the queue has the same email property. Therefore an aggregator can't form a new group for the same correlation key.
https://docs.spring.io/spring-integration/docs/5.0.3.RELEASE/reference/html/messaging-routing-chapter.html#aggregator-config
Whenever I start the following code:
ConnectionFactory factory = new ConnectionFactory();
factory.setHost("localhost");
Connection connection = factory.newConnection();
Channel channel = connection.createChannel();
String exchangeName = "direct_logs";
channel.exchangeDeclare(exchangeName, "direct");
String queueName = channel.queueDeclare().getQueue();
channel.queueBind(queueName, exchangeName, "red");
channel.basicQos(1);
final Consumer consumer = new DefaultConsumer(channel){
#Override
public void handleDelivery(String consumerTag,
Envelope envelope,
AMQP.BasicProperties properties,
byte[] body) throws IOException{
String message = new String(body, "UTF-8");
System.out.println(message);
System.out.println("message received");
}
};
channel.basicConsume(queueName, true, consumer);
It does not start an endless loop, as is implied in the documentation. Instead, it stops right away.
The only way I can have it consume for some time is to replace channel.basicConsume with a loop, as follows:
DateTime startedAt = new DateTime();
DateTime stopAt = startedAt.plusSeconds(60);
long i=0;
try {
while (stopAt.compareTo(new DateTime()) > 0) {
channel.basicConsume(queueName, true, consumer);
i++;
}
}finally {
System.out.println(new DateTime());
System.out.println(startedAt);
System.out.println(stopAt);
System.out.println(i);
}
There must be a better way to listen to messages for a while, correct? What am I missing?
It stops listening right away.
Are you sure it's stopping? What basicConsume does is register a consumer to listen to a specific queue so there is no need to execute it in a loop. You only execute it once, and the handleDelivery method of the instance of Consumer you pass will be called whenever a message arrives.
The Threads that the rabbitmq library creates should keep the JVM from exiting. In order to exit the program you should actually call connection.close()
Here is a complete receiver example from rabbitmq: https://github.com/rabbitmq/rabbitmq-tutorials/blob/master/java/Recv.java
It's actually pretty much the same as yours.
i had the same issue. the reason was that i was calling connection.close at the end. however, the basicConsume() method does not block on the current thread, rather on other threads, so the code after it, i.e. the connection.close() is called immediately.