I do have the following (multi-threaded) process in place:
Browse MQ queue (with lock) and get the next available message
Do something with it which might or might not fail
a. If successful, remove message from queue and start over or b. if not successful, leave message on queue
My problem arises from the fact that my application could die unexpectedly between step 2 and 3 and the application would then produce a duplicated message upon restart.
Is there a way to mark a message as 'dirty' or 'processing' on the queue (while or after reading it) with the mark persisting even if the application restarts?
I have tried to use the marks provided by MQ, but they do not survive a restart. Another possibility would be to move the message to a 'processing' queue, remove it on success or move it back to the source queue on failure, but this requires a second queue and is not trivial code anymore.
Rough code example:
MQGetMessageOptions gmo = new MQGetMessageOptions();
gmo.options = MQConstants.MQGMO_BROWSE_FIRST | MQConstants.MQGMO_LOCK;
MQMessage message = new MQMessage();
message.correlationId = MQC.MQCI_NONE;
message.messageId = MQC.MQMI_NONE;
queue.get(message, gmo);
boolean success = processMessage(message);
// Application gets killed here after successful message processing.
// Produces duplicate after restart.
if (success) {
MQGetMessageOptions gmo2 = new MQGetMessageOptions();
gmo2.options = MQConstants.MQGMO_MSG_UNDER_CURSOR;
queue.get(new MQMessage(), gmo2);
}
Basically, I'd like to achieve this:
get message non-destructively from queue (only if not marked as "processing")
mark message as "processing" on queue
process message (including sending to some destination)
if successful delete from queue, or remove "processing" state on queue otherwise
If the application dies right after a successful third step 'process message', the message would be marked as "processing" and would not be processed again (as it might have been already).
Note: I do not want this process to have any knowledge about the message processing (other than success).
Have you tried SYNCPOINT?Commit or Backout kind of operation might help in this scenario.
Your solution is a horrible design. If you are updating a database then why are you not using 2 phase commits (i.e. XA transactions)?
Just have your MQAdmin setup up the queue manager to use the resource manager of the particular database you are using then it is as simple as:
Start transaction (2 phase commit)
Get message (destructive get NOT browse) from the queue
Update database
Commit transaction
Hence, everything in the transaction, MQGET and database update, will either be committed together or backed out together.
If your application were to crash, then the resource manager will automatically back out everything in the transaction.
Lets say you don't want to use 2 phase commit or you are not updating a database (updating a file) then you can use single phase UOW (Unit of Work).
Use MQGMO option of MQGMO_SYNCPOINT
Get message (destructive get NOT browse) from the queue
Update whatever you are updating
Issue MQCMIT
Things to know about MQ:
If an application issues an MQDISC or ends normally, with current uncommitted operations, an implied MQCMIT is executed by IBM MQ, i.e. all operations done under SYNCPOINT are committed.
If an application ends abnormally, with current uncommitted operations, an implied MQBACK is executed by IBM MQ, i.e. all operations done under SYNCPOINT are rolled back.
Related
I am facing a scenario where the reply queue I connect to, runs out of handles. I have traced it to the fact that my JMS Producers are being cached but not my JMS consumers. I am able to send and receive messages just fine so there is no problem with connecting-sending-receiving to/from the queues. I am using the CachedConnectionFactory (SessionCacheSize = 10)with the target factory as com.ibm.mq.jms.MQQueueConnectionFactory while instantiating the jmsTemplate. Code snippet is as follows
:
:
String replyQueue = "MyQueue";// replyQueue which runs out of handles
messageCreator.setReplyToQueue(new MQQueue(replyQueue));
jmsTemplate.setReceiveTimeout(receiveTimeout);
jmsTemplate.send(destination, messageCreator);// Send to destination queue
Message message = jmsTemplate.receiveSelected(replyQueue,
String.format("JMSCorrelationID = '%s'", messageCreator.getMessageId()));
:
:
From the logs (jms TRACE is enabled) Producer is cached, so the destination queue "handle count" does not increase.
// The first time around (for Producer)
Registering cached JMS MessageProducer for destination[queue:///<destination>:com.ibm.mq.jms.MQQueueSender#c9x758b
// Second time around, the cached producer is reused
Found cached JMS MessageProducer for destination [queue:///<destination>]: com.ibm.mq.jms.MQQueueSender#c9x758b
However, the handles for the replyQueue keep increasing because for every call to that queue, I see a new JMS Consumer being registered. Ultimately the calls to open the replyQueue fail because of MQRC_HANDLE_NOT_AVAILABLE
// First time around
Registering cached JMS MessageConsumer for destination [queue:///<replyQueue>]:com.ibm.mq.jms.MQQueueReceiver#b3ytd25b
// Second time around, another MessageConsumer is registered !
Registering cached JMS MessageConsumer for destination [queue:///<replyQueue>]:com.ibm.mq.jms.MQQueueReceiver#re25b
My memory is a bit dim on this, but here is what is happening. You are receiving messages based on a message selector. This selector is always changing, however. As a test, either remove the selector or make it a constant and see what happens. So when you try to cache/pool based on connection/session/consumer, the consumer is always changing. This requires a new cache entry.
After you go through your 10 sessions, a new connection will be created, but the existing one is not closed. Increase your session count to 100, for example, and your connection count on the MQ broker should climb 10 time slower.
You need to create a new consumer for every message receive as your correlation ID is always changing. So just cache connection/session. No matter what you do, you will always have to round trip to the broker to ask for the new correlation ID.
I'm pretty new to Kafka. I'm using spring cloud stream Kafka to produce and consume
#StreamListener(Sink.INPUT)
public void process(Order order) {
try {
// have my message processing
}
catch( exception e ) {
//retry here that record..
}
}
}
Just want to know how can I implement a retry ? Any help on this is highly appreciated
Hy
There are multiple ways to handle "retries" and it depends on the kind of events you encounter.
For basic issues kafka framework will retry for you to recover from an error condition, for example in case of a short network downtime the consumer and producer api implement auto retry.
In particular kafka support "built-in producer/consumer retries" to correctly handle a large variety of errors without loss of messages, but as a developer, you must still be able to handle other types of errors with the try-catch block you mention.
Error in kafka can be divided in the following categories:
(producer & consumer side) Nonretriable broker errors such as errors regarding message size, authorization errors, etc -> you must handle them in "design phase" of your app.
(producer side) Errors that occur before the message was sent to the broker—for example, serialization errors --> you must handle them in the runtime app execution
(producer & consumer sideErrors that occur when the producer exhausted all retry attempts or when the
available memory used by the producer is filled to the limit due to using all of it to store messages while retrying -> you should handle these errors.
Another point of attention regarding "how to retry" is how to handle correctly the order of commits in case of auto-commit option is set to false.
A common and simple pattern to get commit order right is to use a monotonically increasing sequence number. Increase the sequence number every time you commit and add the sequence number at the time of the commit to the commit function.
When you’re getting ready to send a retry, check if the
commit sequence number the callback got is equal to the instance
variable; if it is, there was no newer commit and it is safe to retry. If
the instance sequence number is higher, don’t retry because a
newer commit was already sent.
I am trying to understand the behavior when a message expires while it is being processed. I have the following flow setup in my test program.
queue.start -> sleepProcessor -> queue.end
The sleepProcessor takes the message from queue.start and sleeps for 5 seconds. I send a message to queue.start with a JMSExpiration of 1 second from System.currentTimeMillis(). I have setup dead letter queues for each queue named DLQ.queue.start and DLQ.queue.end.
The behavior I see is that 1 message ends up in DLQ.queue.start and another message ends up in DLQ.queue.end.
How does 1 message become 2?
The test program can be found here with source
http://s000.tinyupload.com/?file_id=04225732819763428273
I have included a maven pom.xml and the test program can be run with the following command
mvn camel:run
OS: Linux 3.5.0 (Mint 14)
JVM: 1.6
ActiveMQ: 5.7.0
Camel: 2.8.2
Any insight would be greatly appreciated.
Thank you
For the discussion, I think it is useful to show all the important parts of your code.
Your route looks as follows:
from("jms:queue.start")
.transacted()
.process(sleepProcessor)
.to("jms:queue.end");
The headers is set as follows:
final Map<String, Object> headers = new HashMap<String, Object>();
final long expiration = System.currentTimeMillis() + 1000; // <-- start time plus 1 second!
headers.put("JMSExpiration", expiration);
Finally, the processing is started as follows:
final ProducerTemplate template = camelContext.createProducerTemplate();
template.sendBodyAndHeaders("jms:queue.start", "Test message", headers);
What happend during the processing:
The route is started and the message is sent to jms:queue.start with expiration time set to 1 second
The jms:queue.start route receives the messages and the sleepProcessor waits 5 seconds. Note, that no acknowledgment is sent to the JMS broker yet (a message is only acknowledged at the end of the Camel route). During this time, the message is expired and we see the first log produced by the DLQ processor.
Afterwards, the message is put to jms:queue.end. The message header JMSExpriation indicates, that the message is expired (> start time plus 5 seconds), and the message is expired right away, and we see the second log produced by the DLQ processor looking as the message is duplicated.
Conclusion:
The route jms:queue.start is not cancelled even if the JMS message is expired during the processing. Of course, the reason for this is that it is just not possible/allowed for the JMS broker to cancel an ongoing message consumpation. We have to live with that and react accordingly in the DLQ processors.
In my application, I have a queue (HornetQ) set up on JBoss 7 AS.
I have used Spring batch to do some work once the messages is received (save values in database etc.) and then the consumer commits the JMS session.
Sometimes when there is an exception while processing the message, the excecution of consumer is aborted abruptly.
And the message remains in "in delivery" state. There are about 30 messages in this state on my production queue.
I have tried restarting the consumer but the state of these messages is not changed. The only way to remove these
messages from the queue is to restart the queue. But before doing that I want a way to read these messages so
that they can be corrected and sent to the queue again to be processed.
I have tried using QueueBrowser to read them but it does not work. I have searched a lot on Google but could not
find any way to read these messages.
I am using a Transacted session, where once the message is processed, I am calling:
session.commit();
This sends the acknowledgement.
I am implementing spring's
org.springframework.jms.listener.SessionAwareMessageListener
to recieve messages and then to process them.
While processing the messages, I am using spring batch to insert some data in database.
For a perticular case, it tries to insert data too big to be inserted in a column.
It throws an exception and transaction is aborted.
Now, I have fixed my producer and consumer not to have such data, so that this case should not happen again.
But my question is what about the 30 "in delivery" state messages that are in my production queue? I want to read them so that they can be corrected and sent to the queue again to be processed. Is there any way to read these messages? Once I know their content, I can restart the queue and submit them again (after correcting them).
Thanking you in anticipation,
Suvarna
It all depends on the Transaction mode you are using.
for instance if you use transactions:
// session here is a TX Session
MessageConsumer cons = session.createConsumer(someQueue);
session.start();
Message msg = consumer.receive...
session.rollback(); // this will make the messages to be redelivered
if you are using non TX:
// session here is auto-ack
MessageConsumer cons = session.createConsumer(someQueue);
session.start();
// this means the message is ACKed as we receive, doing autoACK
Message msg = consumer.receive...
//however the consumer here could have a buffer from the server...
// if you are not using the consumer any longer.. close it
consumer.close(); // this will release messages on the client buffer
Alternatively you could also set consumerWindowSize=0 on the connectionFactory.
This is on 2.2.5 but it never changed on following releases:
http://docs.jboss.org/hornetq/2.2.5.Final/user-manual/en/html/flow-control.html
I"m covering all the possibilities I could think of since you're not being specific on how you are consuming. If you provide me more detail then I will be able to tell you more:
You can indeed read your messages in the queue using jmx (with for example jconsole)
In Jboss As7 you can do it the following way :
MBeans>jboss.as>messaging>default>myJmsQueue>Operations
listMessagesAsJson
[edit]
Since 2.3.0 You have a dedicated method for this specific case :
listDeliveringMessages
See https://issues.jboss.org/browse/HORNETQ-763
This is Oracle 11.2.0.3.
We've got a problem where we use Oracle's JMS over OracleAQ. This works fine except we started noticing the queue getting filled with 1000s, then millions of messages over time. Some of these are in the PROCESSED state, but most are READY. We traced down this behavior to "zombie" or dead subscribers to the topic. When a Java process is terminated and doesn't get the chance to unregister itself, it leaves the subscriber record in the queue and ORacle doesn't seem to detect that it is dead. So much so that MONTHS later, a new message sent into our multi-subscriber queue will then get multiplied by the # of subscribers, which it thinks is much higher than it actually is. (We first noticed this when we reached the maximum subscriber limit.)
We've got the qmon processes running - I even tried increasing the minimum # of processes to no effect. The queue clean-up happens really nicely as long as there are no dead subscribers in the queue.
Anyone see this before, and hopefully found a solution?
Ok, So I could not have a better solution than this:
1) Create your subscriber with a name and keep track of the subscriber's name.
2) Make sure that you have a shutdown hook to application to execute below procedure, which will unsubscribe and de-register the subscriber.
3) In case of unexpected shutdown/crash, when un-subscription could not be done, there must be a cleanup task to execute below code:
DECLARE
aqAgent SYS.AQ$_AGENT;
BEGIN
for idx in (select consumer_name from
DBA_QUEUE_SUBSCRIBERS a where a.queue_name = '<Your Oracle AQ Name>') loop
aqAgent := SYS.AQ$_AGENT(idx.consumer_name, NULL, NULL);
DBMS_AQADM.REMOVE_SUBSCRIBER('<Your Oracle AQ Name>', aqAgent);
end loop;
END;
This will make sure that your system remains full-proof.