Consumer queues are allocated in client side, broker knows nothing about this.
So how can we monitor which queue is allocated to which consumer client?
Though there is no exiting command, for each message queue per consumer group, You can find out the client using provided admin infrastructure. Here is the snippet achieving this:
private Map<MessageQueue, String> getClientConnection(DefaultMQAdminExt defaultMQAdminExt, String groupName){
Map<MessageQueue, String> results = new HashMap<MessageQueue, String>();
try{
ConsumerConnection consumerConnection = defaultMQAdminExt.examineConsumerConnectionInfo(groupName);
for (Connection connection : consumerConnection.getConnectionSet()){
String clinetId = connection.getClientId();
ConsumerRunningInfo consumerRunningInfo = defaultMQAdminExt.getConsumerRunningInfo(groupName, clinetId, false);
for(MessageQueue messageQueue : consumerRunningInfo.getMqTable().keySet()){
results.put(messageQueue, clinetId + " " + connection.getClientAddr());
}
}
}catch (Exception e){
}
return results;
}
In case you have not used the RocketMQ-Console project, please try and run it: https://github.com/rocketmq/rocketmq-console-ng
In the Consumer tab, Click "consumer detail" button, you will see message queue allocation result visually as below:
Message queues allocation result
Related
Greetings of the day.
Please help on the below requirement:
Requirement:
We want to delete message from MQ only after it is processed successfully.
Use event based message detection technique and avoid loop
So, to achieve above:
I have created message listener and consumer class below:
{
sessionIn = connectionIn.CreateSession(false, AcknowledgeMode.ClientAcknowledge);
// Create message listener and assign it to consumer
messageListener = new MessageListener(OnMessageCallback);
consumerAsync.MessageListener = messageListener;
Console.WriteLine("Message Listener set. Starting the connection now.");
// Start the connection to receive messages.
connectionWMQ.Start();
}
Reading the message from the call back event and push the message into other system:
OnMessageCallback(Message) {
if (xmsMessage is IBytesMessage)
{
IBytesMessage bytesMessage = (IBytesMessage)xmsMessage;
byte[] arrayMessage = new byte[bytesMessage.BodyLength];
bytesMessage.ReadBytes(arrayMessage);
string message = System.Text.Encoding.Default.GetString(arrayMessage);
}
}
Once the message processed, external system will fire the below over ride method:
Response method override:
protected override Task OnResponse(ReponseMessage message)
{
//Read the message and get the message id and correlation id.
//Delete the message from the queue.
//I am trying to do like this, but Its not working:
messageConsumerDelete = sessionDelete.CreateConsumer(destinationDelete, query);
if (messageConsumerDelete != null)
{
IMessage m = messageConsumerDelete.Receive(1000);
LogWrite("Receive Message=" + m);
m.Acknowledge();
}
}
Please suggest a best solution for this requirement.
I am trying to find a solution for this since weeks, but no breakthrough.
Thanks,
Balaji
I have been reading the documentation for virtual destinations here: http://activemq.apache.org/virtual-destinations.html
But I hit a bit of a snag, when I send to a topic it does not seem to follow the client id name as described on the document
My setup on the active mq is:
<destinationInterceptors>
<virtualDestinationInterceptor>
<virtualDestinations>
<virtualTopic name="Destination.>" prefix="Target.*." selectorAware="false" />
</virtualDestinations>
</virtualDestinationInterceptor>
</destinationInterceptors>
The code above describes that when I send to a Destination.Status topic with a ClientId of CustomerA.
It should send only to Target.CustomerA.Destination.Status if understand correctly, but what's happening is it's sending to Target.CustomerA.Destination.Status and Target.CustomerB.Destination.Status so basically fanning out messages to queues and ignoring the client id.
I did not see any further documentation about how to configure it, i was wondering if anyone else encountered this ?
Am I missing something here ?
Below is my producer if it's helpful.
public static class HelloWorldProducer implements Runnable {
public void run() {
try {
// Create a ConnectionFactory
ActiveMQConnectionFactory connectionFactory = new ActiveMQConnectionFactory("tcp://localhost:61617");
// Create a Connection
Connection connection = connectionFactory.createConnection();
connection.setClientID("CustomerA");
connection.start();
// Create a Session
Session session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
// Create the destination (Topic or Queue)
Destination destination = session.createTopic("Destination.Status");
// Create a MessageProducer from the Session to the Topic or Queue
MessageProducer producer = session.createProducer(destination);
producer.setDeliveryMode(DeliveryMode.NON_PERSISTENT);
// Create a messages
String text = "Hello world! From: " + Thread.currentThread().getName() + " : " + this.hashCode();
TextMessage message = session.createTextMessage(text);
// Tell the producer to send the message
System.out.println("Sent message: "+ message.hashCode() + " : " + Thread.currentThread().getName());
producer.send(message);
// Clean up
session.close();
connection.close();
}
catch (Exception e) {
System.out.println("Caught: " + e);
e.printStackTrace();
}
}
}
Any inputs will be beneficial.
The sender in this scenario has no real effect on the routing at the broker whether or not you've set a ClientID as it is just sending to a named Topic, in this case "Destination.Status". The configuration on the broker controls the routing and in your case you've configured "Destination.>" so any Queue consumer that comes along and subscribes to a Queue that matches the configuration you've set. So in your case I'd guess you have one consumer subscribing to Queue (Target.CustomerA.Destination.Status) and one to Queue (Target.CustomerB.Destination.Status) which then causes any message sent to the Topic to be fanned out to both.
If you want competing consumers then you'd need to subscribe both to Target.CustomerA.Destination.Status and then the broker would round-robin dispatch the sent message to either of the active subscribers.
I have used .net C# (IBM MQ version 9.1.5) to pull messages from the queue. So I have no issues connecting to the queue and getting messages.
I have read that there is the concept of transactions Distributed Transactions.
I tried the following:
var getMessageOptions = new MQGetMessageOptions();
getMessageOptions = new MQGetMessageOptions();
getMessageOptions.Options += MQC.MQGMO_WAIT + MQC.MQGMO_SYNCPOINT;
getMessageOptions.WaitInterval = 20000; // 20 seconds wait
Transaction oldAmbient = Transaction.Current;
using (var tx = new CommittableTransaction())
{
try
{
int i = queue.CurrentDepth;
Log.Information($"Current queue depth is {i} message(s)");
var message = new MQMessage();
queue.Get(message, getMessageOptions);
string messageStr = message.ReadString(message.DataLength);
Log.Information(messageStr);
tx.Commit();
}
catch (MQException e) when (e.Reason == 2033)
{
// Report exceptions other than "no messages in the queue"
Log.Information("No messages in the queue");
tx.Rollback();
}
catch (Exception ex)
{
Log.Error($"Exception when trying to capture a message from the queue: {ex.Message}");
tx.Rollback();
}
I am getting an error code of 2035.
Looking at the documents on Recovering Transactions, where does the "SYSTEM.DOTNET.XARECOVERY.QUEUE" live, is it on the queuemanger?
Do I need to get permissions enabled on this?
Also I see that Microsoft Distributed Transaction Manager is mentioned, is this something that we need to have running on the local host in order for distributed transactions to work?
If MQ Distributed transactions feature is being used then the user running the application should have the authority to "SYSTEM.DOTNET.XARECOVERY.QUEUE".If a transaction is incomplete "SYSTEM.DOTNET.XARECOVERY.QUEUE" queue holds the information of incomplete transaction as message in that queue,which later can be used to resolve the transaction.
Based on your scenario which you had put in comments i.e "we want to just save the message to a file. My thinking is if there is a problem with that, I could roll back the transaction." .If MQ is the only resource manager then you don't have to use Distributed transactions. Getting a message under syncpoint can also be used instead of Distributed Transactions. Distributed Transactions will be useful if more than one resource manager is being used.
To get a message under syncpoint following sample code can be used by updating hostname,channel,port,queue and queue manager name:
var getMessageOptions = new MQGetMessageOptions();
getMessageOptions = new MQGetMessageOptions();
getMessageOptions.Options += MQC.MQGMO_WAIT + MQC.MQGMO_SYNCPOINT;
getMessageOptions.WaitInterval = 20000; // 20 seconds wait
Hashtable props = new Hashtable();
props.Add(MQC.HOST_NAME_PROPERTY, "localhost");
props.Add(MQC.CHANNEL_PROPERTY, "DOTNET.SVRCONN");
props.Add(MQC.PORT_PROPERTY, 3636);
MQQueueManager qm = new MQQueueManager("QM", props);
MQQueue queue = qm.AccessQueue("Q1", MQC.MQOO_INPUT_AS_Q_DEF);
try
{
var message = new MQMessage();
queue.Get(message, getMessageOptions);
//to commit the message
qm.Commit();
string messageStr = message.ReadString(message.DataLength);
}
catch (MQException e) when (e.Reason == 2033)
{
// Report exceptions other than "no messages in the queue"
Log.Information("No messages in the queue");
}
catch (Exception ex)
{
Log.Error($"Exception when trying to capture a message from the queue:
}
I would like to create a simple code snippet that fetches all messages from the DLQ and re-sends them to the original destination (AKA resend/retry)
It can be done easily by the ActiveMQ UI (but for a single message at a time).
There is no direct JMS API for re-sending a message from a DLQ to its original queue. In fact, the JMS API doesn't even discuss dead-letter queues. It's merely a convention used by most brokers to deal with messages that can't be consumed.
You'd need to create an actual JMS consumer to receive the message from the DLQ and then create a JMS producer to send the message back to its original queue.
It's important that you use Session.TRANSACTED mode to avoid potential message loss or duplication.
If you use Session.AUTO_ACKNOWLEDGE and there is a problem between the time the message is consumed and sent (e.g the application crashes, hardware failure, etc.) then the message could be lost due to the fact that it was already acknowledged before it was sent successfully.
If you use Session.CLIENT_ACKNOWLEDGE and there is a problem between the time the message is sent and acknowledged then the message could ultimately be duplicated due to the fact that it was already sent before it was acknowledged successfully.
Both operations should be part of the JMS transaction so that the work is atomic.
Lastly, I recommend you either invoke commit() on the transacted session for each message sent or after a small batch of messages (e.g. 10). Given that you have no idea how many messages are in the DLQ it would be unwise to process every message in a single transaction. Generally you want the transaction to be as small as possible in order to minimize the window during which an error might occur and the transaction's work will need to be performed again. Also, the larger the transaction is the more heap memory will be required on the broker to keep track of the work in the transaction. Keep in mind that you can invoke commit() on the same session as many times as you want. You don't need to create a new session for each transaction.
Retrying all messages on the DLQ is already implemented in activemq as an mbean.
You can trigger the retry method with jmxterm/jolokia
e.g
Replaying all messages on queue ActiveMQ.DLQ with jolokia
curl -XGET --user admin:admin --header "Origin: http://localhost" http://localhost:8161/api/jolokia/exec/org.apache.activemq:brokerName=localhost,destinationName=ActiveMQ.DLQ,destinationType=Queue,type=Broker/retryMessages
NOTE: You can only use this method on a queue that is marked as a DLQ. It will not work for regular queues.
Also the DLQ queue can have its 'DLQ' flag set to false if the server is restarted. It is automatically set to true when a new message is sent to the DLQ
After Justin's reply I've manually implemented the retry mechanism like so:
public void retryAllDlqMessages() throws JMSException {
logger.warn("retryAllDlqMessages starting");
logger.warn("Creating a connection to {}", activemqUrl);
ActiveMQConnectionFactory connectionFactory = new ActiveMQConnectionFactory("test", "test", activemqUrl);
HashMap<String, MessageProducer> messageProducersMap = new HashMap<>();
MessageConsumer consumer = null;
try (ActiveMQConnection connection = (ActiveMQConnection) connectionFactory.createConnection();
ActiveMQSession session = (ActiveMQSession) connection.createSession(true, Session.SESSION_TRANSACTED)) {
String dlqName = getDlqName();
logger.warn("Creating a session to {}", dlqName);
ActiveMQQueue queue = (ActiveMQQueue) session.createQueue(dlqName);
logger.warn("Starting JMS Connection");
connection.start();
logger.warn("Creating a DLQ consumer");
consumer = session.createConsumer(queue);
logger.warn("Consumer start receiving");
Message message = consumer.receive(CONSUMER_RECEIVE_TIME_IN_MS);
int retriedMessages = 0;
while (message != null) {
try {
retryMessage(messageProducersMap, session, message);
retriedMessages++;
} catch (Exception e) {
logger.error("Error calling retryMessage for message = {}", message);
logger.error("Rolling back the JMS transaction...");
session.rollback();
return;
}
message = consumer.receive(CONSUMER_RECEIVE_TIME_IN_MS);
}
logger.warn("Consumer finished retrying {} messages", retriedMessages);
logger.warn("Commiting JMS Transactions of retry");
session.commit();
} finally {
if (!messageProducersMap.isEmpty()) {
logger.warn("Closing {} messageProducers in messageProducersMap", messageProducersMap.size());
for (MessageProducer producer : messageProducersMap.values()) {
producer.close();
}
}
if (consumer != null) {
logger.warn("Closing DLQ Consumer");
consumer.close();
}
}
}
private void retryMessage(HashMap<String, MessageProducer> messageProducersMap, ActiveMQSession session, Message message) {
ActiveMQObjectMessage qm = (ActiveMQObjectMessage) message;
String originalDestinationName = qm.getOriginalDestination().getQualifiedName();
logger.warn("Retry message with JmsID={} to original destination {}", qm.getJMSMessageID(), originalDestinationName);
try {
if (!messageProducersMap.containsKey(originalDestinationName)) {
logger.warn("Creating a new producer for original destination: {}", originalDestinationName);
messageProducersMap.put(originalDestinationName, session.createProducer(qm.getOriginalDestination()));
}
logger.info("Producing message to original destination");
messageProducersMap.get(originalDestinationName).send(qm);
logger.info("Message sent");
} catch (Exception e) {
logger.error("Message retry failed with exception", e);
}
}
Whenever I start the following code:
ConnectionFactory factory = new ConnectionFactory();
factory.setHost("localhost");
Connection connection = factory.newConnection();
Channel channel = connection.createChannel();
String exchangeName = "direct_logs";
channel.exchangeDeclare(exchangeName, "direct");
String queueName = channel.queueDeclare().getQueue();
channel.queueBind(queueName, exchangeName, "red");
channel.basicQos(1);
final Consumer consumer = new DefaultConsumer(channel){
#Override
public void handleDelivery(String consumerTag,
Envelope envelope,
AMQP.BasicProperties properties,
byte[] body) throws IOException{
String message = new String(body, "UTF-8");
System.out.println(message);
System.out.println("message received");
}
};
channel.basicConsume(queueName, true, consumer);
It does not start an endless loop, as is implied in the documentation. Instead, it stops right away.
The only way I can have it consume for some time is to replace channel.basicConsume with a loop, as follows:
DateTime startedAt = new DateTime();
DateTime stopAt = startedAt.plusSeconds(60);
long i=0;
try {
while (stopAt.compareTo(new DateTime()) > 0) {
channel.basicConsume(queueName, true, consumer);
i++;
}
}finally {
System.out.println(new DateTime());
System.out.println(startedAt);
System.out.println(stopAt);
System.out.println(i);
}
There must be a better way to listen to messages for a while, correct? What am I missing?
It stops listening right away.
Are you sure it's stopping? What basicConsume does is register a consumer to listen to a specific queue so there is no need to execute it in a loop. You only execute it once, and the handleDelivery method of the instance of Consumer you pass will be called whenever a message arrives.
The Threads that the rabbitmq library creates should keep the JVM from exiting. In order to exit the program you should actually call connection.close()
Here is a complete receiver example from rabbitmq: https://github.com/rabbitmq/rabbitmq-tutorials/blob/master/java/Recv.java
It's actually pretty much the same as yours.
i had the same issue. the reason was that i was calling connection.close at the end. however, the basicConsume() method does not block on the current thread, rather on other threads, so the code after it, i.e. the connection.close() is called immediately.