We are using spring integration and daily in our logs we can see below stacktrace. Other JMS adapters are working fine, we think only below one is missing something:
Spring integration configuration:
<jms:message-driven-channel-adapter concurrent-consumers="1" id="jmsInLOAN" destination="queueLOAN" channel="LOANCommonDataChannel" acknowledge="transacted" />
Please find below MQ statistics of Put and Msgs read count, there should be exact count of Message read by adapter. I am worried about spring integration's message-driven-channel-adapter of reading extra messages from queue. Any help would be appreciated.
WARN 07/Jan/2016 09:04:15,438 [org.springframework.jms.listener.DefaultMessageListenerContainer#23-1] springframework.jms.listener.DefaultMessageListenerContainer - [SYSTEM_ID=HBUSLOANIQ] [MESSAGE_ID=null] Execution of JMS message listener failed, and no ErrorHandler has been set.
org.springframework.integration.MessagingException: unsupported payload type [com.ibm.jms.JMSMessage]
at org.springframework.integration.xml.DefaultXmlPayloadConverter.convertToDocument(DefaultXmlPayloadConverter.java:76)
at org.springframework.integration.xml.DefaultXmlPayloadConverter.convertToNode(DefaultXmlPayloadConverter.java:88)
at org.springframework.integration.xml.router.XPathRouter.getChannelIdentifiers(XPathRouter.java:119)
at org.springframework.integration.router.AbstractMessageRouter.determineTargetChannels(AbstractMessageRouter.java:247)
at org.springframework.integration.router.AbstractMessageRouter.handleMessageInternal(AbstractMessageRouter.java:211)
It looks like you are passing the unconverted JMS message (com.ibm.jms.JMSMessage) to the XML Payload converter...
org.springframework.integration.MessagingException: unsupported payload type [com.ibm.jms.JMSMessage]
at org.springframework.integration.xml.DefaultXmlPayloadConverter.convertToDocument(DefaultXmlPayloadConverter.java:76)
Perhaps you have set extract-payload to false ?
Although it's not in the configuration you show.
Turning on DEBUG logging will show the payload type of messages passing through the system.
The issue was because of valid and poisonous(which has a payload type [com.ibm.jms.JMSMessage]) messages we were getting onto queue. Valid messages processed well, but poisounous messages not able to digest by application and send to the BackoutQueue.
In our case, BOQ threshold is 3 that means 3 times my application will try to consume a perticular message and if the message backout 3 times then it will be moved to BOQ queue and (msgs read - msgs put)/3 on LOAIQ == the msgs put on to BOQ queue at that sampling interval. From msgs put on BOQ queue, we can see how many messages are backout from LOAIQ queue. That is why the message read count is more than that of msg received.
Related
We have a PHP app that forwards messages from RabbitMQ to connected devices down a WebSocket connection (PHP AMQP pecl extension v1.7.1 & RabbitMQ 3.6.6).
Messages are consumed from an array of queues (1 per websocket connection), and are acknowledged by the consumer when we receive confirmation over the websocket that the message has been received (so we can requeue messages that are not delivered in an acceptable timeframe). This is done in a non-blocking fashion.
99% of the time, this works perfectly, but very occasionally we receive an error "RabbitMQ PRECONDITION_FAILED - unknown delivery tag ". This closes the channel. In my understanding, this exception is a result of one of the following conditions:
The message has already been acked or rejected.
An ack is attempted over a channel the message was not delivered on.
An ack is attempted after the message timeout (ttl) has expired.
We have implemented protections for each of the above cases but yet the problem continues.
I realise there are number of implementation details that could impact this, but at a conceptual level, are there any other failure cases that we have not considered and should be handling? or is there a better way of achieving the functionality described above?
"PRECONDITION_FAILED - unknown delivery tag" usually happens because of double ack-ing, ack-ing on wrong channels or ack-ing messages that should not be ack-ed.
So in same case you are tying to execute basic.ack two times or basic.ack using another channel
(Solution below)
Quoting Jan Grzegorowski from his blog:
If you are struggling with the 406 error message which is included in
title of this post you may be interested in reading the whole story.
Problem
I was using amqplib for conneting NodeJS based messages processor with
RabbitMQ broker. Everything seems to be working fine, but from time to
time 406 (PRECONDINTION-FAILED) message shows up in the log:
"Error: Channel closed by server: 406 (PRECONDITION-FAILED) with message "PRECONDITION_FAILED - unknown delivery tag 1"
Solution <--
Keeping things simple:
You have to ACK messages in same order as they arrive to your system
You can't ACK messages on a different channel than that they arrive on If you break any of these rules you will face 406
(PRECONDITION-FAILED) error message.
Original answer
It can happen if you set no-ack option of a Consumer to true that means you souldn't call ack function manually:
https://www.rabbitmq.com/amqp-0-9-1-reference.html#basic.consume.no-ack
The solution: set no-ack flag to false.
If you aknowledge twice the same message you can have this error.
A variation of what they said above about acking it twice:
there is an "obscure" situation where you are acking a message more than once, which is when you ack a message with multiple parameter set to true, which means all previous messages to the one you are trying to ack, will be acked too.
And so if you try to ack one of the messages that were "auto acked" by setting multiple to true then you would be trying to "ack" it multiple times and so the error, confusing but hope you understand it after a few reads.
Make sure you have the correct application.properties:
If you use the RabbitTemplate without any channel configuration, use "simple":
spring.rabbitmq.listener.simple.acknowledge-mode=manual
In this case, if you use "direct" instead of "simple", you will get the same error message. Another one looks like this:
spring.rabbitmq.listener.direct.acknowledge-mode=manual
I am using ActiveMQ for messaging and there is one requirement that if message is duplicate then it should handled by AMQ automatically.
For that I generate unique message key and set to messageproccessor.
following is code :
jmsTemplate.convertAndSend(dataQueue, event, messagePostProccessor -> {
LocalDateTime dt = LocalDateTime.now();
long ms = dt.get(ChronoField.MILLI_OF_DAY) / 1000;
String messageUniqueId = event.getResource() + event.getEntityId() + ms;
System.out.println("messageUniqueId : " + messageUniqueId);
messagePostProccessor.setJMSMessageID(messageUniqueId);
messagePostProccessor.setJMSCorrelationID(messageUniqueId);
return messagePostProccessor;
});
As it can be seen code generates unique id and then set it to messagepostproccessor.
Can somehelp me on this, is there any other configuration that I need do.
A consumer can receive duplicate messages mainly for two reasons: a producer sent the same message more times or a consumer receive the same message more times.
Apache ActiveMQ Artemis includes powerful automatic duplicate message detection, filtering out messages sent by a producer more times.
To prevent a consumer from receiving the same message more times, an idempotent consumer must be implemented, ie Apache Camel provides an Idempotent consumer component that would work with any JMS provider, see: http://camel.apache.org/idempotent-consumer.html
E 02/07/19-07:51:49.358 [-5] JMSListener exception : java.lang.ClassCastException: com.ibm.jms.JMSMessage cannot be cast to javax.jms.TextMessage
This is a ticky one. We put an xml message onto a queue and our code consumed it no problem. The next day, we put another xml message onto the same queue, and the same code threw this error.
I'm looking for suggestions on how to get more info out of what's actually happening, or if anyone else has had intermittent class cast exceptions like this with JMS.
Please note that not all 'XML' messages are sent as JMS TextMessage. I've come across the same thing. Some applications will decide to send you the XML content as JMS BytesMessage (and put the text inside the byte array body). So make sure your sending is really sending you a TextMessage.
One way to see this is to stop your consumer and look at the queued messages with MQExplorer: the Named Property mcd.Msd should show jms_text if it was sent as proper MQ JMS TextMessage. If the data is received from a non-JMS application, the MQ format matters. If the format is MQFMT_STRING, the message is received as a JMS TextMessage. Otherwise, it is received as a JMS BytesMessage !
So either change your publisher to set the value of format to be MQFMT_STRING or change your consumer application to remove the cast to TextMessage and instead also accept BytesMessages and construct the text from the body's byte array.
I am working on real-time event-based application using Spring WebSockets, Messaging and RabbitMQ. In this application, messages need to be delivered to clients in the exact order they were inserted into RabbitMQ.
"EDITED"
Our goal is to recieve a message from a browser, process it on the server in order (against a unique object determined by route paramteter), enrich the message and broadcast it to all subscribing browsers via external STOMP MQ (RabbitMQ).
Our MessageMapping method is as follows:
#MessageMapping(/commands.{route}.{data})
public CommandMessage receiveCommand(CommandMessage message, Principal principal) {
try {
// Get object to synch on using route
Object o = ...
syncrhonized(o) {
// Perform command on object
// Set message server sequence
message.setServerSequence(o.getAutoIncrementSequence());
// Log server sequence
log.debug("Message server sequence:" + message.getServerSequence());
// Send to external MQ for broadcasting to all subscribers
return message;
}
} catch (Exception e) {
...
}
return null;
}
If we configure the ClientInboundChannel and ClientOutboundChannels with a corePoolSize and maxPoolSize of 1 each, all messages are in order.
If we increase the corePoolSize and maxPoolSize on the ClientInboundChannel, messages get to MQ in the incorrect order; if we do the same increase for ClientOutboundChannel, messages get to browser in incorrect order.
]
Tests were done using a single browser client.
We turned on tracing for StompBrokerRelayMessageHandler and received logs entries like:
2014-05-06 14:26:39 TRACE [clientInboundChannel-6] o.s.m.s.s.StompBrokerRelayMessageHandler [StompBrokerRelayMessageHandler.java:412] Processing message=[Payload byte[303]][Headers={stompCommand=SEND, nativeHeaders={content-type=[application/json;charset=UTF-8], destination=[/topic/commands.BKN01.20140318]}, simpMessageType=MESSAGE, simpDestination=/topic/commands.BKN01.20140318, contentType=application/json;charset=UTF-8, simpSessionId=ehjcoxb3, id=a31d0e3d-12cc-f562-1ec2-e2d7ba0899eb, timestamp=1399400799940}]
2014-05-06 14:26:39 DEBUG [clientInboundChannel-6] o.s.m.s.s.StompBrokerRelayMessageHandler [StompBrokerRelayMessageHandler.java:658] Forwarding message to broker
2014-05-06 14:26:39 TRACE [clientInboundChannel-3] o.s.m.s.s.StompBrokerRelayMessageHandler [StompBrokerRelayMessageHandler.java:406] Ignoring message to destination=/app/commands.BKN01.20140318
2014-05-06 14:26:39 TRACE [clientInboundChannel-7] o.s.m.s.s.StompBrokerRelayMessageHandler [StompBrokerRelayMessageHandler.java:406] Ignoring message to destination=/app/commands.BKN01.20140318
2014-05-06 14:26:39 TRACE [clientInboundChannel-1] o.s.m.s.s.StompBrokerRelayMessageHandler [StompBrokerRelayMessageHandler.java:412] Processing message=[Payload byte[314]][Headers={stompCommand=SEND, nativeHeaders={content-type=[application/json;charset=UTF-8], destination=[/topic/commands.BKN01.20140318]}, simpMessageType=MESSAGE, simpDestination=/topic/commands.BKN01.20140318, contentType=application/json;charset=UTF-8, simpSessionId=ehjcoxb3, id=3cc7b4ae-8ea4-ef8a-6c4d-c3bc1ed23bcd, timestamp=1399400799947}]
2014-05-06 14:26:39 DEBUG [clientInboundChannel-1] o.s.m.s.s.StompBrokerRelayMessageHandler [StompBrokerRelayMessageHandler.java:658] Forwarding message to broker
We also turned tracing on for StompSubProtocolHandler in the org.springframework.web.socket.messaging package and received messages like:
2014-05-07 10:58:58 TRACE [http-nio-8080-exec-5] o.s.w.s.m.StompSubProtocolHandler [StompSubProtocolHandler.java:180] Received message from client session=u8wrnsr6
None of the information provides an easy way to map our message.serverSequence property (which is set prior to sending to MQ) with the various ids the log details.
Is there any way to increase inbound/outbound channel threads so ordering is intact? For example, could the channels be tied to a "route" or could a thread be pegged to a "route"?
Please help.
Thank you,
Dan
[EDITED]
Thanks, I see now. Indeed at present there is no way to ensure messages from the external broker are delivered to a client in the exact same order and that messages from a client will be sent to the external broker in the exact same order.
We could assign an index to every Message within a session and then check and buffer if necessary to enforce the order but that would be a new feature. Feel free to create a request in JIRA.
As for now you could look into creating a ChannelInterceptor that checks every message to see what session it belongs to and then set some incremental index header on it before having it sent (on the clientInboundChannel or clientOutboundChannel). Then extend StompSubProtocolHandler and StompBrokerRelayMessageHandler to check the index header and try to enforce the order of messages.
In my application, I have a queue (HornetQ) set up on JBoss 7 AS.
I have used Spring batch to do some work once the messages is received (save values in database etc.) and then the consumer commits the JMS session.
Sometimes when there is an exception while processing the message, the excecution of consumer is aborted abruptly.
And the message remains in "in delivery" state. There are about 30 messages in this state on my production queue.
I have tried restarting the consumer but the state of these messages is not changed. The only way to remove these
messages from the queue is to restart the queue. But before doing that I want a way to read these messages so
that they can be corrected and sent to the queue again to be processed.
I have tried using QueueBrowser to read them but it does not work. I have searched a lot on Google but could not
find any way to read these messages.
I am using a Transacted session, where once the message is processed, I am calling:
session.commit();
This sends the acknowledgement.
I am implementing spring's
org.springframework.jms.listener.SessionAwareMessageListener
to recieve messages and then to process them.
While processing the messages, I am using spring batch to insert some data in database.
For a perticular case, it tries to insert data too big to be inserted in a column.
It throws an exception and transaction is aborted.
Now, I have fixed my producer and consumer not to have such data, so that this case should not happen again.
But my question is what about the 30 "in delivery" state messages that are in my production queue? I want to read them so that they can be corrected and sent to the queue again to be processed. Is there any way to read these messages? Once I know their content, I can restart the queue and submit them again (after correcting them).
Thanking you in anticipation,
Suvarna
It all depends on the Transaction mode you are using.
for instance if you use transactions:
// session here is a TX Session
MessageConsumer cons = session.createConsumer(someQueue);
session.start();
Message msg = consumer.receive...
session.rollback(); // this will make the messages to be redelivered
if you are using non TX:
// session here is auto-ack
MessageConsumer cons = session.createConsumer(someQueue);
session.start();
// this means the message is ACKed as we receive, doing autoACK
Message msg = consumer.receive...
//however the consumer here could have a buffer from the server...
// if you are not using the consumer any longer.. close it
consumer.close(); // this will release messages on the client buffer
Alternatively you could also set consumerWindowSize=0 on the connectionFactory.
This is on 2.2.5 but it never changed on following releases:
http://docs.jboss.org/hornetq/2.2.5.Final/user-manual/en/html/flow-control.html
I"m covering all the possibilities I could think of since you're not being specific on how you are consuming. If you provide me more detail then I will be able to tell you more:
You can indeed read your messages in the queue using jmx (with for example jconsole)
In Jboss As7 you can do it the following way :
MBeans>jboss.as>messaging>default>myJmsQueue>Operations
listMessagesAsJson
[edit]
Since 2.3.0 You have a dedicated method for this specific case :
listDeliveringMessages
See https://issues.jboss.org/browse/HORNETQ-763