I have a list of stores with both Incoming and Outgoing Failed Message Queues. It uses Message Driven Beans (to read and write) into those Queues (Java Message Service(JMS) in WildFly 8.0)
I want to know
how to pinpoint (from the application) how and where these messages are written into the Queues
how to reduce the number of failed messages to a minimum
Messages fail because they timeout, increase the timetolive config value to avoid the messages ending up in the error queue.
To answer your first question, I dont have a specific answer, you may have to use logging statements just before you send the message to the queue along with the time. If your message has time also, later in the future you can cross verify the logs to that of the error messages.
Related
I have a spring boot app with single kafka consumer to get messages from some topic.
But sometime errors are occurred while message handling.
I want to continue to receive the following messages as usual and at the same time be able not to lose that message and receive it, for example, the next time the service is restarted with the consumer after fixing it.
Is it possible to do this?
I understand that I need to disable auto-commit and commit successful messages manually, but, in this case, if I don't throw any exception for this exception case and commit each next successful message manually, then I will lose the previous unsuccessful one, right?
If I understand your question correctly, your assumption is that the exception occurs due to a problem in your code and not while reading the message from the topic. In that case no retry or other measures will solve your problem.
What we usually do is to catch the exception and send it to another Kafka topic. Ideally, you will also add some details on why or in which code part the exception occurred. After you have fixed the bug in your application you can consume the messages from that other topic.
I understand that I need to disable auto-commit and commit successful messages manually, but, in this case, if I don't throw any exception for this exception case and commit each next successful message manually, then I will lose the previous unsuccessful one, right?
Yes, your understanding is correct. To be more precise, you will not "loose" the message but as soon as your ConsumerGroup commits a higher offset it will never try to read the lower offset again without any manual modification.
Alternative
If you only expect very rare cases where an exception could be thrown, but you just ignore it, you can always use the consumer.seek() method in pure Kafka
public void seek(TopicPartition partition, long offset)
to start reading from a particular offset out of a topic partition.
Yes you have to manually commit them. You retry a particular message 2-3 times. If it fails after retries then you can move those messages to another topic and consume those messages when you fix whatever is causing it to fail. This will not block your queue and you won't lose and messages too.
I want to continue to receive the following messages as usual and at
the same time be able not to lose that message and receive it, for
example, the next time the service is restarted with the consumer
after fixing it.
Is it possible to do this?
You don't need to do a manual commit, instead, you can choose to implement a mechanism to do a retrial, by publishing the event in another queue and delayed consuming the event. =====> Amazon SQS has delay Queue but unfortunately there is no such thing in kafka and you have to write the implementation by yourself.
Reference articles:
Article 1
Article 2
If you are retrying the message processing, then the order of the messages can change based on your implementation. Please do keep it in mind.
Do remember that kafka does consider a consumer dead in case the message processing time exceeds max.poll.interval. Read this
Is it possible to keep a history of messages (with message content would be perfect) that have already been retrieved and are no longer on a queue?
In the application I can see when the sender attempts to put the message in the queue and when the receiver attempts to pick the messages up, but I'd like to see when the message really arrived into the queue and when the messages were really received.
Does MQ Explorer have this function? How would I use it?
What you are looking for is a message tracking/auditing software for IBM MQ. You can find a list of what is available here.
It is possible to use an API exit to make copies of messages in a queue or to audit both PUT and GET operations.
It is also possible to put messages to a topic, then create as many administrative subscriptions to destination queues as required. Something can then GET and log messages from one of those destination queues. The problem with this is that MQ changes the message ID between publication and consumption whereas in a queue it remains static.
There is no native MQ function to capture messages. It's possible to use linear logs and later scrape the logs but these do not necessarily capture all messages due to optimization. (A message PUT to a waiting getter outside of syncpoint for example.) However there is at least one commercial product to scrape linear transaction logs to audit message activity.
The philosophy of MQ in general is that it is the delivery mechanism and deals with envelope data to route and deliver but does not deal with payload data. WAS, IIB and other broker/transformation engines are where IBM has put all of the functions that deal with message payloads.
I have an application listening to messages on an IBM Websphere MQ queue.
Once a message is consumed, the application performs some processing logic.
If the processing completed OK, I would like the application to acknowledge the message and have it removed from the queue.
If an error occurred while processing, I would like the message to remain in the queue.
How is this implemented? (I'm using the .NET API)
Thanks.
MQ supports a single-phase commit protocol. You specify syncpoint when you get the message, then issue COMMIT or ROLLBACK as required. The default action if the connection is lost is ROLLBACK and if the program deliberately ends without resolving the transaction a COMMIT is assumed. (This is platform dependent so the customary advice is to explicitly call COMMIT and not rely on the class destructors to do it for you.)
This works whether the message is persistent or not. However if the message has an expiry specified and expires after being rolled back there's a chance it won't be seen again.
Of course, if the program issues a ROLLBACK the message will normally be seen again since it goes back to the same spot int he queue and for a FIFO queue that's the top. If the problem with the message is not transient then this causes a poison message loop of read/rollback/repeat. To avoid that the app can check the backout count and if it exceeds some threshold requeue the message to an exception queue.
When using JMS or XMS this is done for you by the class libraries. If the input queue's BOQNAME and BOQTHRESH attributes are set the requeue is to the queue names in BOQNAME. Otherwise a requeue to the Dead Queue is attempted. IF that fails (as it should if the system is properly secured) the listener will stop receiving messages.
The usual advice is to always specify a backout queue and either let the classes use it or code the app to use it.
Please see Usage Notes for MQGET in the MQAPI Reference and the MQGetMessageOptions.NET page in the .Net class reference.
You may want to look at the MQ Reporting Options.
Expiry, Confirmation of Arrival and Confirmation of Delivery can be requested and sent via a response queue back to the sending application by the receiving Queue Manager.
Positive and Negative Acknowledgements can also be generated by the receiving application provided they use the related reporting attributes found in the Message Descriptor.
Exception can be requested and sent via a response queue back to the sending application by any Queue Manager in the transmission chain or generated by the receiving application.
1 Read the message using MQC.MQGMO_SYNCPOINT,
2 process it
3 call MQQueueManager.Commit()
If Commit() is not called explicitly, or implicitly (eg exception is thrown), all messages that have been de-queued will be re-enqueued.
We have a JMS queue and multiple competing clients are reading from this queue.
Once the message is taken and successfully processed, we want to send the acknowledge to delete ( i.e. CLIENT ACKNOWLEDGE )
However, we want to make sure that if one client has picked the message another client should not take it from the queue.
Does activeMQ provide this feature out of the box using some configuration ?
Moreover:
If the message processing failed after picking the message, so it could not be acknowledged back, in this scenario we should like other client thread to pickup the message. Is it possible out of the box with configuration , may be specifying timeout values ?
Regards,
JE
You need to take some time to understand the difference between a Topic and a Qeueue in order to understand why the first question is not an issue.
For the second question it depends a bit on the ACK mode you are using and how you are processing messages sync or async. Normally for processing where you want to control redeliveries you would do the work inside of a transaction and if the processing fails the message would be redelivered when the TX is rolled back. ActiveMQ supports redelivery policies both client side and broker side that control how many time a message will be redelivered before sent to a DLQ.
We have producer which is producing message at a rate faster than cosumer can consume. We are using Spring JMS integration as the consumer side technology stack. Currently we are using AUTO_ACKNOWLEDGE mode.
In the onMessage() method of the listener, upon the receipt we are to planning submit the client side job to a job queue and return from the onMessage() method. This means if a) processing fails or b) our server goes down while processing there is no way for us recover.
We looked at the option of using CLIENT_ACKNOWLEDGE, but this means acknowledging a message with higher timestamp automatically acknowledges all the messages with a less timestamp. This is clearly not desirable for us because a successful processing a message with newer timestamp no way means that all the messages with older timestamp are processed completely. In effect we are looking on per message acknowledgement. However, I read somewhere that this means there is some design flaw.
The other option is to use a SessionAwareMessageListener interface provided by Spring. The contract of using this interface says that if a JMSException is thrown from the onMessage the message will be redelivered. However, I was not completely sure how to use this for our purpose.
While I dig more myself into this, any help from you guys will be greatly appreciated.
Session aware message has following onMessage prototype:
onMessage(Message message, Session session)
Invoke session.recover() for the message redelivery. Upon session.recover() will send all the unacknowledged messages back to the jms destination.