I am trying implement sample retry mechanism for RabbitMQ using Spring's StatefulRetryOperationsInterceptor.
As stated in documentation, I need to setup message key generator as the message id is absent. What I don't understand is the real usage of unique id generated per message. i.e. when I used below implementation I did not have any issue with retry:
StatefulRetryOperationsInterceptor interceptor =
RetryInterceptorBuilder.stateful()
.maxAttempts(3)
.backOffOptions(2000, 1, 2000)
.messageKeyGenerator(
new MessageKeyGenerator() {
#Override
public Object getKey(Message message) {
return 1;
}
);
container.setAdviceChain(new Advice[] {interceptor});
Stateful retry needs the originating message to be somehow unique - so the retry "state" for the message can be determined - the simplest way is to have the message publisher add a unique message id header.
However, it's possible something in your message body or some other header might be used to uniquely identify the message. Enter the MessageKeyGenerator which is used to determine the unique id.
Using a constant (1 in your case) won't work because every message has the same message key and will all be considered to be deliveries of the same message (from a retry perspective).
The framework does provide a MissingMessageIdAdvice which can provide limited support for stateful retry (if added to the advice chain before the retry advice). It adds a messageId to the incoming message.
"Limited" means full stateful retry support is not available - only one redelivery attempt is allowed.
If the redelivery fails, the message is rejected which causes it to be discarded or routed to the DLX/DLQ if so configured. In all cases the "temporary" state is removed from the cache.
Generally, if full retry support is needed and there is no messageId property and there is no way to generate a unique key with a MessageKeyGenerator, I would recommend using stateless retry.
Related
Trying to build a list of possible errors that can potentially happen during the execution of kafkaTemplate.send() method:
Errors related serialization process;
Some network issues or broker is down;
Some technical issues on broker side, for example acknowledgement not received from broker, etc.
And now I need to find a way how to handle all possible errors in the right way:
Based on the business requirements: in case of any exceptions I need to do the following things:
Retry 3 times;
If all 3 retries failed - log appropriate message.
I found that configuration property spring.kafka.producer.retries available, and I believe it exactly what I need.
But have can I configure recovery method (method that will be executed when all retries failed)?
Probably that spring.kafka.producer.retries is not what you are looking for.
This auto-configuration property is mapped directly to ConsumerConfig:
map.from(this::getRetries).to(properties.in(ProducerConfig.RETRIES_CONFIG));
and then we go and read docs for that ProducerConfig.RETRIES_CONFIG property:
private static final String RETRIES_DOC = "Setting a value greater than zero will cause the client to resend any record whose send fails with a potentially transient error."
+ " Note that this retry is no different than if the client resent the record upon receiving the error."
+ " Allowing retries without setting <code>" + MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION + "</code> to 1 will potentially change the"
+ " ordering of records because if two batches are sent to a single partition, and the first fails and is retried but the second"
+ " succeeds, then the records in the second batch may appear first. Note additionally that produce requests will be"
+ " failed before the number of retries has been exhausted if the timeout configured by"
+ " <code>" + DELIVERY_TIMEOUT_MS_CONFIG + "</code> expires first before successful acknowledgement. Users should generally"
+ " prefer to leave this config unset and instead use <code>" + DELIVERY_TIMEOUT_MS_CONFIG + "</code> to control"
+ " retry behavior.";
As you see spring-retry is fully not involved in the process and all the retries are done directly inside Kafka Client and its KafkaProducer infrastructure.
Although this is not all. Pay attention to the KafkaProducer.send() contract:
Future<RecordMetadata> send(ProducerRecord<K, V> record);
It returns a Future. And if we take a look closer to the implementation, we will see that there is a synchronous part - topic metadata request and serialization, - and enqueuing for the batch for async sending to Kafka broker. The mentioned ProducerConfig.RETRIES_CONFIG has an effect only in that Sender.completeBatch().
I believe that the Future is completed with an error when those internal retries are exhausted. So, you probably should think about using a RetryTemplate manually in the service method around KafkaTemplate to be able to control a retry (and recovery, respectively) around metadata and serialization which are really sync and blocking in the current call. The actual send you also can control in that method with retry, but if you call Future.get() to block it for a response or error from Kafka client on send.
Intro:
We're currently using the spring mail integration to receive and send emails which works without flaws if there's no exception such as a connection error to the exchange server or the database.
These mails come in as Messages and are passed to a handler method which will parse the MimeMessage to a custom mail data object. JPA saves those entities as the last step to our database.
Question/Problem:
There's a problem if the database is down or the mail can't be processed for any other reason, as the IntegrationFlow will still mark it as /SEEN once the message gets passed to the handler.
Setting this flag to false won't fix our problem, because we want Spring to set the /SEEN flag if the mail is processed and saved correctly
shouldMarkMessagesAsRead(false)
Searching for:
Would there be a possibility to set flags AFTER successfully saving the mail to the database?
We'd like to process the failed email again after the cause for the responsible error is fixed, which won't work as long Spring marks them as /SEEN no matter the result.
Reference:
The messages comes in and gets passed to the handler which will parse the mail and execute the CRUD-Repository save(mailDAO) method. The handleMimeMessage() is more or less just a mapper.
#Bean
fun imapIdleFlow(imapProperties: ImapProperties): IntegrationFlow {
imapProperties.username.let { URLEncoder.encode(it, charset) }
return IntegrationFlows
.from(
Mail.imapIdleAdapter(
ImapMailReceiver("imap://${imapProperties.username}:${imapProperties.password}#${imapProperties.host}/Inbox")
.apply {
setSimpleContent(true)
setJavaMailProperties(imapProperties.properties.toProperties())
})
.autoStartup(true)
.shouldReconnectAutomatically(true)
)
.handle(this::handleMimeMessage)
.get()
}
Is it even possible to mark the messages in the same flow afterward as you need to access the exchange a second time or would I need a second flow to get and flag the same mail?
I think it is possible with something like transaction synchronization: https://docs.spring.io/spring-integration/reference/html/mail.html#mail-tx-sync
So, you set transactional(TransactionManager transactionManager) on that Mail.imapIdleAdapter to the JpaTransactionManager to start transaction from this IMAP Idle channel adapter and propagate it to your handleMimeMessage() where you do those JPA saves.
Plus you add:
/**
* Configure a {#link TransactionSynchronizationFactory}. Usually used to synchronize
* message deletion with some external transaction manager.
* #param transactionSynchronizationFactory the transactionSynchronizationFactory.
* #return the spec.
*/
public ImapIdleChannelAdapterSpec transactionSynchronizationFactory(
TransactionSynchronizationFactory transactionSynchronizationFactory) {
To react for commit and rollback of the mentioned transaction.
The DefaultTransactionSynchronizationFactory with some TransactionSynchronizationProcessor impl can give you a desired behavior, where you take a Message and its payload from the provided IntegrationResourceHolder and perform something like message.setFlag(Flag.SEEN, true); on the MimeMessage.
You may consider to use the mentioned in docs an ExpressionEvaluatingTransactionSynchronizationProcessor.
To avoid folder reopening, you may consider to use a public ImapIdleChannelAdapterSpec autoCloseFolder(boolean autoCloseFolder) { with a false option. You need to consider to close it in that TX sync impl or some other way.
In a spring boot application, I have a class with jms listener.
public class PaymentNotification{
#JmsListener(destination="payment")
public void receive(String payload) throws Exception{
//mapstring conversion
....
paymentEvent = billingService.insert(paymentEvent); //transactional method
//call rest...
billingService.save(paymentEvent);
//send info to jms
}
}
I saw then when a error happen, data is inserted in the database, that ok, but it's like receive method is called again and again... but queue is empty when I check on the server.
If there is an error, I don't want method is called again, Is there something for that.
The JMS Message Headers might contain additional information to help with your processing. In particular JMSRedelivered could be of some value. The Oracle doc states that "If a client receives a message with the JMSRedelivered field set, it is likely, but not guaranteed, that this message was delivered earlier but that its receipt was not acknowledged at that time."
I ran the following code to explore what was available in my configuration (Spring Boot with IBM MQ).
#JmsListener(destination="DEV.QUEUE.1")
public void receive(Message message) throws Exception{
for (Enumeration<String> e = message.getPropertyNames(); e.hasMoreElements();)
System.out.println(e.nextElement());
}
From here I could find JMSXDeliveryCount is available in JMS 2.0. If that property is not available, then you may well find something similar for your own configuration.
One strategy would be to use JMSXDeliveryCount, a vendor specific property or maybe JMSRedelivered (if suitable for your needs) as a way to check before you process the message. Typically, the message would be sent to a specific blackout queue where the redelivery count exceeds a set threshold.
Depending on the messaging provider you are using it might also be possible to configure back out queue processing as properties of the queue.
I'm pretty new to Kafka. I'm using spring cloud stream Kafka to produce and consume
#StreamListener(Sink.INPUT)
public void process(Order order) {
try {
// have my message processing
}
catch( exception e ) {
//retry here that record..
}
}
}
Just want to know how can I implement a retry ? Any help on this is highly appreciated
Hy
There are multiple ways to handle "retries" and it depends on the kind of events you encounter.
For basic issues kafka framework will retry for you to recover from an error condition, for example in case of a short network downtime the consumer and producer api implement auto retry.
In particular kafka support "built-in producer/consumer retries" to correctly handle a large variety of errors without loss of messages, but as a developer, you must still be able to handle other types of errors with the try-catch block you mention.
Error in kafka can be divided in the following categories:
(producer & consumer side) Nonretriable broker errors such as errors regarding message size, authorization errors, etc -> you must handle them in "design phase" of your app.
(producer side) Errors that occur before the message was sent to the broker—for example, serialization errors --> you must handle them in the runtime app execution
(producer & consumer sideErrors that occur when the producer exhausted all retry attempts or when the
available memory used by the producer is filled to the limit due to using all of it to store messages while retrying -> you should handle these errors.
Another point of attention regarding "how to retry" is how to handle correctly the order of commits in case of auto-commit option is set to false.
A common and simple pattern to get commit order right is to use a monotonically increasing sequence number. Increase the sequence number every time you commit and add the sequence number at the time of the commit to the commit function.
When you’re getting ready to send a retry, check if the
commit sequence number the callback got is equal to the instance
variable; if it is, there was no newer commit and it is safe to retry. If
the instance sequence number is higher, don’t retry because a
newer commit was already sent.
I've been trying my hands on Spring AMQP. And I have a couple of questions:
I'd like to know what is Publisher returns and how is it different from Publisher Confirm. Of my understanding, we have a Publisher Confirm Callback that checks the status of acks. Now I looked at the documentation in Spring AMQP and Rabbit MQ. didn't really find or understand much on this.
And also why is it that if the message is tried to send to a non-existing queue, I don't get any sort of acknowledgement (ack/nack
) nor do I get any errors. Is there a way to setTimeouts for non-acknowledgements?
Short answer from the link https://www.rabbitmq.com/confirms.html :
"For unroutable messages, the broker will issue a confirm once the exchange verifies a message won't route to any queue (returns an empty list of queues). If the message is also published as mandatory, the basic.return is sent to the client before basic.ack."
In Spring AMQP if you set 'spring.rabbitmq.publisherReturns' to true this will mean messages will be 'mandatory' (unless you set mandatory to false) because of the following code:
private boolean determineMandatoryFlag() {
Boolean mandatory = this.properties.getTemplate().getMandatory();
return (mandatory != null ? mandatory : this.properties.isPublisherReturns());
}
I suggest you to read this article. There is a good description of all possible acknowledgments scenarios, including returns for the unrouted messages, like your non-existing queue.
From the Spring AMQP perspective you should bear in mind: https://docs.spring.io/spring-amqp/docs/2.0.3.RELEASE/reference/html/_reference.html#template-confirms
This feature requires a CachingConnectionFactory that has its publisherReturns property set to true.