Currently I'm using bluelock's camel-spring-amqp component for my application.
What I want to achieve is:
Pull a message from RabbitMQ server.
Persist it to a database on successful processing / Send it to another "Error" queue on Exception
Tell the original queue that it is now safe to remove the message from the queue.
As of this writing, I'm able to pull from rabbit and persist to database using camel routes. What I don't really know how to do is acknowledge that my processing is done to the original queue. Here is my current route:
from("spring-amqp:EXCHANGE:queuename?autodelete=false&durable=true&type=direct&routingKey=")
.bean(Transform.class, "transform(byte[])")
.to("jpa:com.my.company.models.MyModel?entityType=java.util.ArrayList")
I realize I can set the acknowledgmentMode to NONE. But I don't know how to "manually" acknowledge once I have persisted my message.
Thanks for any help!
I'm new in Camel but I know a thing or two about RabbitMQ.
With RabbitMQConsumer the message is acknowledged if the processor doesn't throw any exception (line 133 at RabbitMQConsumer source).
So I suppose if you let your processor propagate the exception, the message won't be acknowledged. I haven't used spring-amqp but I guess it should have a similar behaviour.
Related
I created a SpringBoot/Spring AMQP project where I configured a listener on a RabbitMQ queue. Question: Is there any way to leave the message in the queue? Let me explain: I consume the message and do some things (eg save on db), if something goes wrong I would like to be able to reconsume the message.
Thanks in advance
You need to think about configuring your listener container with transactions, so when DB call fails, the transaction is going to be rolled back and an AMQP message will not be acked on RabbitMQ.
See docs for more info: https://docs.spring.io/spring-amqp/docs/current/reference/html/#transactions
I don't know about the "Spring" way of accomplishing this, but what you describe is the normal behavior for AMQP consumers that do not automatically acknowledge.
From the documentation:
In automatic acknowledgement mode, a message is considered to be successfully delivered immediately after it is sent.
When you turn off automatic acknowledgment, your consumer must explicitly acknowledge the message, otherwise it will not be dequeued (or as you put it, it will be left "in the queue"). You will then need to simply ACK the message at the very end of your operation, when you are certain that your operation succeeded (and perhaps coordinated with your database transaction).
There is always the question of what to do first; acknowledge first or commit your database transaction first? Without adding complexity, you must choose what's best depending on what failure mode is less problematic for you, i.e. Would you rather tolerate a duplicated message or a missing message?
I have a spring boot app with single kafka consumer to get messages from some topic.
But sometime errors are occurred while message handling.
I want to continue to receive the following messages as usual and at the same time be able not to lose that message and receive it, for example, the next time the service is restarted with the consumer after fixing it.
Is it possible to do this?
I understand that I need to disable auto-commit and commit successful messages manually, but, in this case, if I don't throw any exception for this exception case and commit each next successful message manually, then I will lose the previous unsuccessful one, right?
If I understand your question correctly, your assumption is that the exception occurs due to a problem in your code and not while reading the message from the topic. In that case no retry or other measures will solve your problem.
What we usually do is to catch the exception and send it to another Kafka topic. Ideally, you will also add some details on why or in which code part the exception occurred. After you have fixed the bug in your application you can consume the messages from that other topic.
I understand that I need to disable auto-commit and commit successful messages manually, but, in this case, if I don't throw any exception for this exception case and commit each next successful message manually, then I will lose the previous unsuccessful one, right?
Yes, your understanding is correct. To be more precise, you will not "loose" the message but as soon as your ConsumerGroup commits a higher offset it will never try to read the lower offset again without any manual modification.
Alternative
If you only expect very rare cases where an exception could be thrown, but you just ignore it, you can always use the consumer.seek() method in pure Kafka
public void seek(TopicPartition partition, long offset)
to start reading from a particular offset out of a topic partition.
Yes you have to manually commit them. You retry a particular message 2-3 times. If it fails after retries then you can move those messages to another topic and consume those messages when you fix whatever is causing it to fail. This will not block your queue and you won't lose and messages too.
I want to continue to receive the following messages as usual and at
the same time be able not to lose that message and receive it, for
example, the next time the service is restarted with the consumer
after fixing it.
Is it possible to do this?
You don't need to do a manual commit, instead, you can choose to implement a mechanism to do a retrial, by publishing the event in another queue and delayed consuming the event. =====> Amazon SQS has delay Queue but unfortunately there is no such thing in kafka and you have to write the implementation by yourself.
Reference articles:
Article 1
Article 2
If you are retrying the message processing, then the order of the messages can change based on your implementation. Please do keep it in mind.
Do remember that kafka does consider a consumer dead in case the message processing time exceeds max.poll.interval. Read this
I have a dead-letter queue which stores failed messages. Is there have a way to manually trigger a listener to process the messages in the dead-letter queue? I’m using Spring AMQP.
Like I said, it depends on your use case, but I can assume you are looking for something from spring amqp out-of-the-box.
just have a look at BlockingQueueConsumer, you can i.e. prefetch one message if you want, then start it manually.
Please be aware that you need choose solution according to your case, maybe you just need a flag in db to mark the job is failed, or save the msg in db.
I have a question regarding the behavior of a jdbc message store. I notice that if a failure occurs for instance if the following sequence is followed:
1)inbound-channel-adapter - Creates a message payload and sends it to a channel
2)The channel is polled for the payload by a jms:outbound-channel-adapter which adds an entry in the JMS queue. I am using ActiveMQ.
When I test for a situation like shutting down the MOM I would like to persist the message in the message-store, but this is not the default behavior. Looks like SI processes the payload in the channel and then pushes it out (deletes it from the store). I can see the message being inserted in the message-store in the database.
Am I wrong in understanding the function of a message-store? I thought that the message would persist until a successful run. Thanks for any feedback.
You need to set up durable subscriptions for that. I think most of the SI JMS components do support that.
We have producer which is producing message at a rate faster than cosumer can consume. We are using Spring JMS integration as the consumer side technology stack. Currently we are using AUTO_ACKNOWLEDGE mode.
In the onMessage() method of the listener, upon the receipt we are to planning submit the client side job to a job queue and return from the onMessage() method. This means if a) processing fails or b) our server goes down while processing there is no way for us recover.
We looked at the option of using CLIENT_ACKNOWLEDGE, but this means acknowledging a message with higher timestamp automatically acknowledges all the messages with a less timestamp. This is clearly not desirable for us because a successful processing a message with newer timestamp no way means that all the messages with older timestamp are processed completely. In effect we are looking on per message acknowledgement. However, I read somewhere that this means there is some design flaw.
The other option is to use a SessionAwareMessageListener interface provided by Spring. The contract of using this interface says that if a JMSException is thrown from the onMessage the message will be redelivered. However, I was not completely sure how to use this for our purpose.
While I dig more myself into this, any help from you guys will be greatly appreciated.
Session aware message has following onMessage prototype:
onMessage(Message message, Session session)
Invoke session.recover() for the message redelivery. Upon session.recover() will send all the unacknowledged messages back to the jms destination.