I have a spring boot app with single kafka consumer to get messages from some topic.
But sometime errors are occurred while message handling.
I want to continue to receive the following messages as usual and at the same time be able not to lose that message and receive it, for example, the next time the service is restarted with the consumer after fixing it.
Is it possible to do this?
I understand that I need to disable auto-commit and commit successful messages manually, but, in this case, if I don't throw any exception for this exception case and commit each next successful message manually, then I will lose the previous unsuccessful one, right?
If I understand your question correctly, your assumption is that the exception occurs due to a problem in your code and not while reading the message from the topic. In that case no retry or other measures will solve your problem.
What we usually do is to catch the exception and send it to another Kafka topic. Ideally, you will also add some details on why or in which code part the exception occurred. After you have fixed the bug in your application you can consume the messages from that other topic.
I understand that I need to disable auto-commit and commit successful messages manually, but, in this case, if I don't throw any exception for this exception case and commit each next successful message manually, then I will lose the previous unsuccessful one, right?
Yes, your understanding is correct. To be more precise, you will not "loose" the message but as soon as your ConsumerGroup commits a higher offset it will never try to read the lower offset again without any manual modification.
Alternative
If you only expect very rare cases where an exception could be thrown, but you just ignore it, you can always use the consumer.seek() method in pure Kafka
public void seek(TopicPartition partition, long offset)
to start reading from a particular offset out of a topic partition.
Yes you have to manually commit them. You retry a particular message 2-3 times. If it fails after retries then you can move those messages to another topic and consume those messages when you fix whatever is causing it to fail. This will not block your queue and you won't lose and messages too.
I want to continue to receive the following messages as usual and at
the same time be able not to lose that message and receive it, for
example, the next time the service is restarted with the consumer
after fixing it.
Is it possible to do this?
You don't need to do a manual commit, instead, you can choose to implement a mechanism to do a retrial, by publishing the event in another queue and delayed consuming the event. =====> Amazon SQS has delay Queue but unfortunately there is no such thing in kafka and you have to write the implementation by yourself.
Reference articles:
Article 1
Article 2
If you are retrying the message processing, then the order of the messages can change based on your implementation. Please do keep it in mind.
Do remember that kafka does consider a consumer dead in case the message processing time exceeds max.poll.interval. Read this
Related
In the event where consumer is executing and the lock on the message is lost in the meantime due to duration expiring or intermittent connection loss - how can this scenario be potentially handled from within the consumer because currently if consumer finishes, the MessageLockLostException is thrown and another instance of consumer would have been started already.
To solve that - we wrote idempotent consumers, however, this is still an issue now where the first instance (the one that was executing prior to losing the lock) throws an exception because that causes the message to go the {queueName}_error by default and the MassTransit pipeline seems to ignore the fact that the message lock has been lost at that point and a new instance of consumer for the same message has been started.
So the summary of this predicament is that we can effectively get a message successfully acknowledged as well as it being in the error queue. I am looking if anyone has maybe successfully dealt with similar scenario or if there are some hooks in MassTransit where a consumer can work out whether it still has an active lock on the message it is processing.
I created a SpringBoot/Spring AMQP project where I configured a listener on a RabbitMQ queue. Question: Is there any way to leave the message in the queue? Let me explain: I consume the message and do some things (eg save on db), if something goes wrong I would like to be able to reconsume the message.
Thanks in advance
You need to think about configuring your listener container with transactions, so when DB call fails, the transaction is going to be rolled back and an AMQP message will not be acked on RabbitMQ.
See docs for more info: https://docs.spring.io/spring-amqp/docs/current/reference/html/#transactions
I don't know about the "Spring" way of accomplishing this, but what you describe is the normal behavior for AMQP consumers that do not automatically acknowledge.
From the documentation:
In automatic acknowledgement mode, a message is considered to be successfully delivered immediately after it is sent.
When you turn off automatic acknowledgment, your consumer must explicitly acknowledge the message, otherwise it will not be dequeued (or as you put it, it will be left "in the queue"). You will then need to simply ACK the message at the very end of your operation, when you are certain that your operation succeeded (and perhaps coordinated with your database transaction).
There is always the question of what to do first; acknowledge first or commit your database transaction first? Without adding complexity, you must choose what's best depending on what failure mode is less problematic for you, i.e. Would you rather tolerate a duplicated message or a missing message?
Currently I'm using bluelock's camel-spring-amqp component for my application.
What I want to achieve is:
Pull a message from RabbitMQ server.
Persist it to a database on successful processing / Send it to another "Error" queue on Exception
Tell the original queue that it is now safe to remove the message from the queue.
As of this writing, I'm able to pull from rabbit and persist to database using camel routes. What I don't really know how to do is acknowledge that my processing is done to the original queue. Here is my current route:
from("spring-amqp:EXCHANGE:queuename?autodelete=false&durable=true&type=direct&routingKey=")
.bean(Transform.class, "transform(byte[])")
.to("jpa:com.my.company.models.MyModel?entityType=java.util.ArrayList")
I realize I can set the acknowledgmentMode to NONE. But I don't know how to "manually" acknowledge once I have persisted my message.
Thanks for any help!
I'm new in Camel but I know a thing or two about RabbitMQ.
With RabbitMQConsumer the message is acknowledged if the processor doesn't throw any exception (line 133 at RabbitMQConsumer source).
So I suppose if you let your processor propagate the exception, the message won't be acknowledged. I haven't used spring-amqp but I guess it should have a similar behaviour.
We have producer which is producing message at a rate faster than cosumer can consume. We are using Spring JMS integration as the consumer side technology stack. Currently we are using AUTO_ACKNOWLEDGE mode.
In the onMessage() method of the listener, upon the receipt we are to planning submit the client side job to a job queue and return from the onMessage() method. This means if a) processing fails or b) our server goes down while processing there is no way for us recover.
We looked at the option of using CLIENT_ACKNOWLEDGE, but this means acknowledging a message with higher timestamp automatically acknowledges all the messages with a less timestamp. This is clearly not desirable for us because a successful processing a message with newer timestamp no way means that all the messages with older timestamp are processed completely. In effect we are looking on per message acknowledgement. However, I read somewhere that this means there is some design flaw.
The other option is to use a SessionAwareMessageListener interface provided by Spring. The contract of using this interface says that if a JMSException is thrown from the onMessage the message will be redelivered. However, I was not completely sure how to use this for our purpose.
While I dig more myself into this, any help from you guys will be greatly appreciated.
Session aware message has following onMessage prototype:
onMessage(Message message, Session session)
Invoke session.recover() for the message redelivery. Upon session.recover() will send all the unacknowledged messages back to the jms destination.
The logic flow is like this
A message is sent to an input queue
A ProcessorMDB's onMessage() is invoked. Within this method several operations/validations are done
In case of a poison message(msg that application code cannot handle) a RuntimeException is thrown.
This should rollback the transaction. We are seeing evidence in the log file.
There is a backout threshold defined with a backout queue name
once threshold is reached, the message is sent to backout queue
But immediately it starts going back and forth between the input queue and backout queue.
We are using MQMON tool to observe this weird behavior. It continues for ever almost even after the app server(where MDB is running) is shutdown.
We are using Weblogic 10.3.1 and WebSphere MQ 6.02
Any help will be much appreciated, looks like we are running out of ideas.
This sounds like a syncpoint issue. If the QMgr were to issue a COMMIT when a message is requeued inside of a unit of work it would affect all messages under syncpoint inside of that thread. This would cause serious problems if an application had performed several PUT or GET calls prior to hitting the poison message. Rather than issue a COMMIT outside of the program's control, the QMgr just leaves the message on the backout queue inside the unit of work and waits for the program to issue the COMMIT. This can lead to some unexpected behavior such as what you are seeing where a message lands back on the input queue.
If another message is in the queue behind the "bad" one and it is processed successfully by the same thread, everything works out perfectly. The app issues a COMMIT on the new message and this also affects the poison message on the Backout Queue. However if the thread were to exit uncleanly (without an explicit disconnect or COMMIT) then the transaction is rolled back and the poison message is returned to the input queue.
The usual way of dealing with this is that the next good message (or batch of messages if transactions are batched) in the input queue will force the COMMIT. However in some cases where the owning thread gets no new work (perhaps it was performing a GET by Correlation ID) there is nothing to push the bad message through. In these cases, it is important to make sure that the application issues a COMMIT before ending. One way to do this is to write the code to perform the GET by CORRELID with a wait interval. If the wait interval expires, the application would get a return code of 2033 and then issue a COMMIT before closing the thread. If the reply message is legitimately late for whatever reason, the COMMIT will have no effect. But if the message arrived and had been backed out and requeued, the COMMIT will cause it to stay in the Backout Queue.
One way to see exactly what is going on is to run a trace against the queue in question. You can use the built-in trace function - strmqtrc - which has a few more options in V7 than does the V6 version. However if you want very fine grained control you can use the trace exit in SupportPac MA0W. With MA0W you can see exactly what API calls are made by the program and those made on its behalf.
[EDIT] Updating the response with some info from the PMR:
The following is from the WMQ V7 Infocenter:
MessageConsumers are single threaded below the Session level, and
any requeuing of poison messages
takes place within the current unit of
work. This does not affect the
operation of the application, however
when poison messages are requeued
under a transacted or
Client_acknowledge Session, the
requeue action itself will not be
committed until the current unit of
work is committed by the application
code or, if appropriate, the
application container code."
Hence, if it is important for the customer to have poison messages
committed immediately after they are
backed out, it is recommended they
either make use of the Application
Server Facilities
(ConnectionConsumer) which can commit
the message immediately, or
another mechanism to move poison
messages from the queue.
Here is the link to this information in the V6 and V7 Information Centers. Since you are using the V6 client so you would want to refer to the V6 Infocenter. Note that with the V6 client, there is no mention in the Infocenter of ASF being able to commit the poison message immediately, even when using a ConnectionConsumer. The way I read it, this means you probably will need to upgrade to the V7 client to get the behavior you are looking for. Will be interested to see if the PMR results in a similar recommendation.