Does RetryOperations required when I use retry-spring ?
or is it enough to have the annotation over my method?
#Retryable(value = {SQLException.class, Exception.class,RuntimeException.class, RetryException.class, ExhaustedRetryException.class}, //retry will be attempted only if the method throws an SQLException.
maxAttempts = 3, backoff = #Backoff (5000))
public boolean sendJmsMessage(final String xml) {
jmsTemplate.send.........()
// logic to send jms
}
#Recover
public void ExhaustedRetryException(ExhaustedRetryException re) {
LOG.error("Failed to deliver the message using the retry");
}
Related
Spring boot version : 2.7.6
Spring kafka version : 2.8.11
Issue:
I was trying to handle the deserialization issues in code. To handle such issues in code, I created my own class by extending
DefaultErrorHandler
and overriding the public void handleOtherException(Exception thrownException, Consumer<?, ?> consumer, MessageListenerContainer container, boolean batchListener) {}
Sample code as below
public class CustomDefaultErrorHandler extends DefaultErrorHandler {
private static Logger log = LoggerFactory.getLogger(CustomDefaultErrorHandler.class);
#Override
public void handleOtherException(Exception thrownException, Consumer<?, ?> consumer, MessageListenerContainer container, boolean batchListener) {
manageException(thrownException, consumer);
}
private void manageException(Exception ex, Consumer<?, ?> consumer) {
log.error("Error polling message: " + ex.getMessage());
if (ex instanceof RecordDeserializationException) {
RecordDeserializationException rde = (RecordDeserializationException) ex;
consumer.seek(rde.topicPartition(), rde.offset() + 1L);
consumer.commitSync();
} else {
log.error("Exception not handled");
}
}
}
If I use the #RetryableTopic along with #KafkaListener
#RetryableTopic(listenerContainerFactory = "kafkaListenerContainerFactory", backoff = #Backoff(delay = 8000, multiplier = 2.0),
dltStrategy = DltStrategy.FAIL_ON_ERROR
, traversingCauses = "true", autoCreateTopics = "true", numPartitions = "3", replicationFactor = "3",
fixedDelayTopicStrategy = FixedDelayStrategy.MULTIPLE_TOPICS, include = {RetriableException.class, RecoverableDataAccessException.class,
SQLTransientException.class, CallNotPermittedException.class}
)
#KafkaListener(topics = "${topic.name}", groupId = "order", containerFactory = "kafkaListenerContainerFactory", id = "OTR")
public void consumeOTRMessages(ConsumerRecord<String, PayloadsVO> payload, #Header(KafkaHeaders.RECEIVED_TOPIC) String topicName) throws JsonProcessingException {
logger.info("Payload :{}", payload.value());
payloadsService.savePayload(payload.value(), pegasusTopicName);
}
What I saw in debugging the code, #RetryableTopic has its own DefaultErrorHandler configurations in
ListenerContainerFactoryConfigurer
and it stops my custom handler and deserialization process wont stop on issue.
Can you please suggest any way since I wanted to use annotations for retry process in my code
I tried to configured my own implementation of
DefaultErrorHandler
by extending it and configured in
ConcurrentKafkaListenerContainerFactory
It's quite involved, but you should be able to override the RetryTopicComponentFactory bean and override listenerContainerFactoryConfigurer() to return your custom error handler.
That said, deserialization exceptions will go straight to the DLT anyway.
BTW, calling commitSync() here is worthless because there were no records returned by the poll().
I'm struggling with understanding #Retryable. What I need is to retry 3 times when I get 5xx Exception and if retry also fails then throw a custom exception in the recovery method. And if some other exception is thrown then catch it and throw a custom exception.
#Retryable(value = HttpServerErrorException.class, maxAttempts = 3, backoff = #Backoff(delay = 3000))
public String callToService(String key) {
String response;
try {
response = //assume a service call here
}catch (Exception ex) {
throw new customException("some message");
}
return response;
}
#Recover
public void retryFailed(HttpServerErrorException httpServerErrorException) {
throw new customException("some message");
}
In your case as you have added:
#Retryable(value = HttpServerErrorException.class, maxAttempts = 3, backoff = #Backoff(delay = 3000))
The #Retryable is used with:
value = HttpServerErrorException.class, so your method will be retried only if HttpServerErrorException is occure/thrown from your method code, and Note: if any other exception thrown retry will not be done, and recover method will also not be invoked, as recover method is only invoked with exception mentioned in value in #Retryable.
maxAttempts = 3, so it will retry executing your method 3 times by maximum
backoff = #Backoff(delay = 3000), so it will keep a delay of 3000ms in between retry.
And after retrying 3 times, if your method still not working, your method with #Recover will be envoked with the HttpServerErrorException
I hope it make sense and help yo understand the concept of #Retryable
Now to implement what you want you need to implement it as below:
#Retryable(value = HttpServerErrorException.class, maxAttempts = 3, backoff = #Backoff(delay = 3000))
public String callToService(String key) {
String response;
try {
response = //assume a service call here
} catch (HttpServerErrorException httpServerErrorException) {
throw httpServerErrorException;
} catch (Exception ex) {
throw new CustomException("some message");
}
return response;
}
#Recover
public void retryFailed(HttpServerErrorException httpServerErrorException) {
//do whatever you want here, when HttpServerErrorException occured more than 3 times
}
javax.JMS version 2.0.1
Provider : ibm.mq v9.0
Framework : Java Spring boot
From what I know, onMessage() is asynchronous. I am successfully retrying the message send. However, the re-sending of messages happens instantaneously after a message failure. Ideally I want the retry to happen in a sliding window style eg. First retry after 20 seconds, second retry after 40 etc.
How can I achieve this without a Thread.Sleep() which, I presume, will block the entire Java thread and is not something I want at all ?
Code is something like this
final int TIME_TO_WAIT = 20;
public void onMessage(Message , message)
{
:
:
int t = message.getIntProperty("JMSXDeliveryCount");
if(t > 1)
{
// Figure out a way to wait for (TIME_TO_WAIT * t)
}
}
catch(Exception e)
{
// Do some logging/cleanup etc.
throw new RunimeException(e);// this causes a message retry
}
I would suggest you use exponential backoff in the retry logic, but you would need to use the Delivery Delay feature.
Define a custom JmsTemplate that will use delay property from the message, you should add retry count in the message property as well so that you can delay as per your need like 20, 40, 80, 160, etc
public class DelayedJmsTemplate extends JmsTemplate {
public static String DELAY_PROPERTY_NAME = "deliveryDelay";
#Override
protected void doSend(MessageProducer producer, Message message) throws JMSException {
long delay = -1;
if (message.propertyExists(DELAY_PROPERTY_NAME)) {
delay = message.getLongProperty(DELAY_PROPERTY_NAME);
}
if (delay >= 0) {
producer.setDeliveryDelay(delay);
}
if (isExplicitQosEnabled()) {
producer.send(message, getDeliveryMode(), getPriority(), getTimeToLive());
} else {
producer.send(message);
}
}
}
Define Components, that will have the capability fo re-enqueue of the message, you can define this interface in the base message listener. The handleException method should do all the tasks of enqueue and computing delay etc. You may not always interested in enqueuing, in some cases, you would discard messages as well.
You can see a similar post-processing logic here
https://github.com/sonus21/rqueue/blob/4c9c5c88f02e5cf0ac4b16129fe5b880411d7afc/rqueue-core/src/main/java/com/github/sonus21/rqueue/listener/PostProcessingHandler.java
#Component
#Sl4j
public class MessageListener {
private final JmsTemplate jmsTemplate;
#Autowired
public MessageListener(JmsTemplate jmsTemplate) {
this.jmsTemplate = jmsTemplate;
}
#JmsListener(destination = "myDestination")
public void onMessage(Message message) throws JMSException {
try {
// do something
} catch (Exception e) {
handleException("myDestination", message, e);
}
}
// Decide whether the message should be ignored due to many retries etc
private boolean shouldBeIgnored(String destination, Message message) {
return false;
}
// add logic to compute delay
private long getDelay(String destination, Message message, int deliveryCount) {
return 100L;
}
private void handleException(String destination, Message message, Exception e) throws JMSException {
if (shouldBeIgnored(destination, message)) {
log.info("destination: {}, message: {} is ignored ", destination, message, e);
return;
}
if (message.propertyExists("JMSXDeliveryCount")) {
int t = message.getIntProperty("JMSXDeliveryCount");
long delay = getDelay(destination, message, t + 1);
message.setLongProperty(DELAY_PROPERTY_NAME, delay);
message.setIntProperty("JMSXDeliveryCount", t + 1);
jmsTemplate.send(destination, session -> message);
} else {
// no delivery count, is this the first message or should be ignored?
}
}
}
Hi I have problem with my Rabbit listener which cause infinite loop on exception (requeue message). My configuration looks:
#Bean(name = "defContainer")
public RabbitListenerContainerFactory containerFactory(ConnectionFactory connectionFactory, PlatformTransactionManager transactionManager){
SimpleRabbitListenerContainerFactory containerFactory = new SimpleRabbitListenerContainerFactory();
containerFactory.setConnectionFactory(connectionFactory);
containerFactory.setConcurrentConsumers(5);
containerFactory.setAcknowledgeMode(AcknowledgeMode.AUTO);
containerFactory.setTransactionManager(transactionManager);
containerFactory.setMessageConverter(messageConverterAmqp());
containerFactory.setDefaultRequeueRejected(false);
return new TxRabbitListenerContainerFactory(containerFactory);
}
where transactionManager is JpaTransactionManager for transaction on postgre db.
TxRabbitListenerContainerFactory is my factory which set setAlwaysRequeueWithTxManagerRollback to false:
public class TxRabbitListenerContainerFactory implements RabbitListenerContainerFactory {
private SimpleRabbitListenerContainerFactory factory;
public TxRabbitListenerContainerFactory(SimpleRabbitListenerContainerFactory factory) {
this.factory = factory;
}
#Override
public MessageListenerContainer createListenerContainer(RabbitListenerEndpoint endpoint) {
SimpleMessageListenerContainer container = factory.createListenerContainer(endpoint);
container.setAlwaysRequeueWithTxManagerRollback(false);
return container;
}
}
Now I have listner like:
#RabbitListener(bindings = #QueueBinding(
value = #Queue(value = "topic.two", durable = "true"),
exchange = #Exchange(value = "topic.def", type = "topic", durable = "true"),
key = "letter.*"
), errorHandler = "rabErrorHandler", containerFactory = "defContainer")
#Transactional
public Motorcycle topicLetters(Motorcycle motorcycle) throws Exception{
motorcycle.setId(UUID.randomUUID().toString());
Testing testing = new Testing();
testingRepository.save(testing);
throwEx();
return motorcycle;
}
where method throwEx(); throw unchecked exception.
Data from DB are properly rollbacked (not commited), but message are constantly requeued, see it in listener:
#Bean
public RabbitListenerErrorHandler rabErrorHandler(){
return new RabbitListenerErrorHandler() {
#Override
public Object handleError(Message message, org.springframework.messaging.Message<?> message1, ListenerExecutionFailedException e) throws Exception {
System.out.println("FFFFFFFFFFF");
return null;
}
};
}
How to prevvent infinite loope, and why is it happend ?
EDIT:
Logs: pasted logs
Set defaultRequeueRejected to false on the container factory.
To programmatically decide when to requeue or nor, leave that at true and throw an AmqpRejectAndDontRequeueException when you don't want it requeued.
EDIT
There's something not adding up...
protected void prepareHolderForRollback(RabbitResourceHolder resourceHolder, RuntimeException exception) {
if (resourceHolder != null) {
resourceHolder.setRequeueOnRollback(isAlwaysRequeueWithTxManagerRollback() ||
RabbitUtils.shouldRequeue(isDefaultRequeueRejected(), exception, logger));
}
}
If both booleans are false, we don't requeue.
Found issue:
Reason: caused by errorHandler handler which was specified on listener level. In some cases error handler return null - which was causing infinite loop (instead rethrowing exception and rollback transaction)
I have the following configuration
spring.rabbitmq.listener.prefetch=1
spring.rabbitmq.listener.concurrency=1
spring.rabbitmq.listener.retry.enabled=true
spring.rabbitmq.listener.retry.max-attempts=3
spring.rabbitmq.listener.retry.max-interval=1000
spring.rabbitmq.listener.default-requeue-rejected=false //I have also changed it to true but the same behavior still happens
and in my listener I throw the exception AmqpRejectAndDontRequeueException to reject the message and enforce rabbit not to try to redeliver it...But rabbit redilvers it for 3 times then finally route it to dead letter queue.
Is that the standard behavior according to my provided configuration or do I miss something?
You have to configure the retry policy to not retry for that exception.
You can't do that with properties, you have to configure the retry advice yourself.
I'll post an example later if you need help with that.
requeue-rejected is at the container level (below retry on the stack).
EDIT
#SpringBootApplication
public class So39853762Application {
public static void main(String[] args) throws Exception {
ConfigurableApplicationContext context = SpringApplication.run(So39853762Application.class, args);
Thread.sleep(60000);
context.close();
}
#RabbitListener(queues = "foo")
public void foo(String foo) {
System.out.println(foo);
if ("foo".equals(foo)) {
throw new AmqpRejectAndDontRequeueException("foo"); // won't be retried.
}
else {
throw new IllegalStateException("bar"); // will be retried
}
}
#Bean
public ListenerRetryAdviceCustomizer retryCustomizer(SimpleRabbitListenerContainerFactory containerFactory,
RabbitProperties rabbitPropeties) {
return new ListenerRetryAdviceCustomizer(containerFactory, rabbitPropeties);
}
public static class ListenerRetryAdviceCustomizer implements InitializingBean {
private final SimpleRabbitListenerContainerFactory containerFactory;
private final RabbitProperties rabbitPropeties;
public ListenerRetryAdviceCustomizer(SimpleRabbitListenerContainerFactory containerFactory,
RabbitProperties rabbitPropeties) {
this.containerFactory = containerFactory;
this.rabbitPropeties = rabbitPropeties;
}
#Override
public void afterPropertiesSet() throws Exception {
ListenerRetry retryConfig = this.rabbitPropeties.getListener().getRetry();
if (retryConfig.isEnabled()) {
RetryInterceptorBuilder<?> builder = (retryConfig.isStateless()
? RetryInterceptorBuilder.stateless()
: RetryInterceptorBuilder.stateful());
Map<Class<? extends Throwable>, Boolean> retryableExceptions = new HashMap<>();
retryableExceptions.put(AmqpRejectAndDontRequeueException.class, false);
retryableExceptions.put(IllegalStateException.class, true);
SimpleRetryPolicy policy =
new SimpleRetryPolicy(retryConfig.getMaxAttempts(), retryableExceptions, true);
ExponentialBackOffPolicy backOff = new ExponentialBackOffPolicy();
backOff.setInitialInterval(retryConfig.getInitialInterval());
backOff.setMultiplier(retryConfig.getMultiplier());
backOff.setMaxInterval(retryConfig.getMaxInterval());
builder.retryPolicy(policy)
.backOffPolicy(backOff)
.recoverer(new RejectAndDontRequeueRecoverer());
this.containerFactory.setAdviceChain(builder.build());
}
}
}
}
NOTE: You cannot currently configure the policy to retry all exceptions, "except" this one - you have to classify all exceptions you want retried (and they can't be a superclass of AmqpRejectAndDontRequeueException). I have opened an issue to support this.
The other answers posted here didn't work me when using Spring Boot 2.3.5 and Spring AMQP Starter 2.2.12, but for these versions I was able to customize the retry policy to not retry AmqpRejectAndDontRequeueException exceptions:
#Configuration
public class RabbitConfiguration {
#Bean
public RabbitRetryTemplateCustomizer customizeRetryPolicy(
#Value("${spring.rabbitmq.listener.simple.retry.max-attempts}") int maxAttempts) {
SimpleRetryPolicy policy = new SimpleRetryPolicy(maxAttempts, Map.of(AmqpRejectAndDontRequeueException.class, false), true, true);
return (target, retryTemplate) -> retryTemplate.setRetryPolicy(policy);
}
}
This lets the retry policy skip retries for AmqpRejectAndDontRequeueExceptions but retries all other exceptions as usual.
Configured this way, it traverses the causes of an exception, and skips retries if it finds an AmqpRejectAndDontRequeueException.
Traversing the causes is needed as org.springframework.amqp.rabbit.listener.adapter.MessagingMessageListenerAdapter#invokeHandler wraps all exceptions as a ListenerExecutionFailedException