Spring boot version : 2.7.6
Spring kafka version : 2.8.11
Issue:
I was trying to handle the deserialization issues in code. To handle such issues in code, I created my own class by extending
DefaultErrorHandler
and overriding the public void handleOtherException(Exception thrownException, Consumer<?, ?> consumer, MessageListenerContainer container, boolean batchListener) {}
Sample code as below
public class CustomDefaultErrorHandler extends DefaultErrorHandler {
private static Logger log = LoggerFactory.getLogger(CustomDefaultErrorHandler.class);
#Override
public void handleOtherException(Exception thrownException, Consumer<?, ?> consumer, MessageListenerContainer container, boolean batchListener) {
manageException(thrownException, consumer);
}
private void manageException(Exception ex, Consumer<?, ?> consumer) {
log.error("Error polling message: " + ex.getMessage());
if (ex instanceof RecordDeserializationException) {
RecordDeserializationException rde = (RecordDeserializationException) ex;
consumer.seek(rde.topicPartition(), rde.offset() + 1L);
consumer.commitSync();
} else {
log.error("Exception not handled");
}
}
}
If I use the #RetryableTopic along with #KafkaListener
#RetryableTopic(listenerContainerFactory = "kafkaListenerContainerFactory", backoff = #Backoff(delay = 8000, multiplier = 2.0),
dltStrategy = DltStrategy.FAIL_ON_ERROR
, traversingCauses = "true", autoCreateTopics = "true", numPartitions = "3", replicationFactor = "3",
fixedDelayTopicStrategy = FixedDelayStrategy.MULTIPLE_TOPICS, include = {RetriableException.class, RecoverableDataAccessException.class,
SQLTransientException.class, CallNotPermittedException.class}
)
#KafkaListener(topics = "${topic.name}", groupId = "order", containerFactory = "kafkaListenerContainerFactory", id = "OTR")
public void consumeOTRMessages(ConsumerRecord<String, PayloadsVO> payload, #Header(KafkaHeaders.RECEIVED_TOPIC) String topicName) throws JsonProcessingException {
logger.info("Payload :{}", payload.value());
payloadsService.savePayload(payload.value(), pegasusTopicName);
}
What I saw in debugging the code, #RetryableTopic has its own DefaultErrorHandler configurations in
ListenerContainerFactoryConfigurer
and it stops my custom handler and deserialization process wont stop on issue.
Can you please suggest any way since I wanted to use annotations for retry process in my code
I tried to configured my own implementation of
DefaultErrorHandler
by extending it and configured in
ConcurrentKafkaListenerContainerFactory
It's quite involved, but you should be able to override the RetryTopicComponentFactory bean and override listenerContainerFactoryConfigurer() to return your custom error handler.
That said, deserialization exceptions will go straight to the DLT anyway.
BTW, calling commitSync() here is worthless because there were no records returned by the poll().
Related
We are using RabbitMq with default spring boot configurations. We have a use case in which we want no parallelism for one of the listeners. That is, we want only one thread of the consumer to be running at any given point in time. We want this, because the nature of the use case is such that we want the messages to be consumed in order, thus if there are multiple threads per consumer there can be chances that the messages are processed out of order.
Since, we are using the defaults and have not explicitly tweaked the container, we are using the SimpleMessageListenerContainer. By looking at the documentation I tried fixing the number of consumers using concurrency = "1" . The annotation on the target method looks like this #RabbitListener(queues = ["queue-name"], concurrency = "1").
As per the documentation this should have ensured that there is only consumer thread.
{#link org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer
* SimpleMessageListenerContainer} if this value is a simple integer, it sets a fixed
* number of consumers in the {#code concurrentConsumers} property
2021-10-29 06:11:26.361 INFO 29752 --- [ntContainer#4-1] c.t.t.i.p.s.xxx : Created xxx
2021-10-29 06:11:26.383 INFO 29752 --- [ntContainer#0-1] c.t.t.i.p.s.xxx : Created xxx
ThreadIds to be noted here are [ntContainer#4-1] and [ntContainer#0-1].
So the question is- how can we ensure that there is only one thread per consumer at any given point in time ?
Edit: Adding the code of the consumer class for more context
#ConditionalOnProperty(value = ["rabbitmq.sharebooking.enabled"], havingValue = "true", matchIfMissing = false)
class ShareBookingConsumer #Autowired constructor(
private val shareBookingRepository: ShareBookingRepository,
private val objectMapper: ObjectMapper,
private val shareDtoToShareBookingConverter: ShareBookingDtoToShareBookingConverter
) {
private val logger = LoggerFactory.getLogger(javaClass)
init {
logger.info("start sharebooking created consumer")
}
#RabbitListener(queues = ["tax_engine.share_booking"], concurrency = "1-1", exclusive = true)
#Timed
#Transactional
fun consumeShareBookingCreatedEvent(message: Message) {
try {
consumeShareBookingCreatedEvent(message.body)
} catch (e: Exception) {
throw AmqpRejectAndDontRequeueException(e)
}
}
private fun consumeShareBookingCreatedEvent(event: ByteArray) {
toShareBookingCreationMessageEvent(event).let { creationEvent ->
RmqMetrics.measureEventMetrics(creationEvent)
val shareBooking = shareDtoToShareBookingConverter.convert(creationEvent.data)
val persisted = shareBookingRepository.save(shareBooking)
logger.info("Created shareBooking ${creationEvent.data.id}")
}
}
private fun toShareBookingCreationMessageEvent(event: ByteArray) =
objectMapper.readValue(event, shareBookingCreateEventType)
companion object {
private val shareBookingCreateEventType =
object : TypeReference<RMQMessageEnvelope<ShareBookingCreationDto>>() {}
}
}
Edit: Adding application thread analysis using visualvm
5 threads get created for 5 listeners.
[1]: https://i.stack.imgur.com/gQINE.png
Set concurrency = "1-1". Note that the concurrency of the Listener depends not only on concurrentConsumers, but also on maxConcurrentConsumers:
If you are using a custom factory:
#Bean
public SimpleRabbitListenerContainerFactory rabbitListenerContainerFactory(CachingConnectionFactory cachingConnectionFactory) {
SimpleRabbitListenerContainerFactory factory = new SimpleRabbitListenerContainerFactory();
factory.setConnectionFactory(cachingConnectionFactory);
factory.setConcurrentConsumers(1);
factory.setMaxConcurrentConsumers(1);
return factory;
}
See: https://docs.spring.io/spring-amqp/docs/current/reference/html/#simplemessagelistenercontainer for detail.
EDIT:
I did a simple test, 2 consumers&2 threads:
#RabbitListener(queues = "myQueue111", concurrency = "1-1")
public void handleMessage(Object message) throws InterruptedException {
LOGGER.info("Received message : {} in {}", message, Thread.currentThread().getName());
}
#RabbitListener(queues = "myQueue222", concurrency = "1-1")
public void handleMessag1e(Object message) throws InterruptedException {
LOGGER.info("Received message222 : {} in {}", message, Thread.currentThread().getName());
}
Try this:
#RabbitListener(queues = ["queue-name"], exclusive = true)
See https://docs.spring.io/spring-amqp/docs/current/reference/html/#exclusive-consumer
I am currently using Spring Kafka to consume messages from topic along with #Retry of Spring. So basically, I am retrying to process the consumer message in case of an error. But while doing so, I want to avoid the exception message thrown by KafkaMessageListenerContainer. Instead I want to display a custom message. I tried adding an error handler in the ConcurrentKafkaListenerContainerFactory but on doing so, my retry does not get invoked.
Can someone guide me on how to display a custom exception message along with #Retry scenario as well? Below are my code snippets:
ConcurrentKafkaListenerContainerFactory Bean Config
#Bean
ConcurrentKafkaListenerContainerFactory << ? , ? > concurrentKafkaListenerContainerFactory(ConcurrentKafkaListenerContainerFactoryConfigurer configurer, ConsumerFactory < Object, Object > kafkaConsumerFactory) {
ConcurrentKafkaListenerContainerFactory < Object, Object > kafkaListenerContainerFactory =
new ConcurrentKafkaListenerContainerFactory < > ();
configurer.configure(kafkaListenerContainerFactory, kafkaConsumerFactory);
kafkaListenerContainerFactory.setConcurrency(1);
// Set Error Handler
/*kafkaListenerContainerFactory.setErrorHandler(((thrownException, data) -> {
log.info("Retries exhausted);
}));*/
return kafkaListenerContainerFactory;
}
Kafka Consumer
#KafkaListener(
topics = "${spring.kafka.reprocess-topic}",
groupId = "${spring.kafka.consumer.group-id}",
containerFactory = "concurrentKafkaListenerContainerFactory"
)
#Retryable(
include = RestClientException.class,
maxAttemptsExpression = "${spring.kafka.consumer.max-attempts}",
backoff = #Backoff(delayExpression = "${spring.kafka.consumer.backoff-delay}")
)
public void onMessage(ConsumerRecord < String, String > consumerRecord) throws Exception {
// Consume the record
log.info("Consumed Record from topic : {} ", consumerRecord.topic());
// process the record
messageHandler.handleMessage(consumerRecord.value());
}
Below is the exception that I am getting:
You should not use #Retryable as well as the SeekToCurrentErrorHandler (which is now the default, since 2.5; so I presume you are using that version).
Instead, configure a custom SeekToCurrentErrorHandler with max attempts, back off, and retryable exceptions.
That error message is normal; it's logged by the container; it's logging level can be reduced from ERROR to INFO or DEBUG by setting the logLevel property on the SeekToCurrentErrorHandler. You can also add a custom recoverer to it, to log your custom message after the retries are exhausted.
my event retry template,
#Bean(name = "eventRetryTemplate")
public RetryTemplate eventRetryTemplate() {
RetryTemplate template = new RetryTemplate();
ExceptionClassifierRetryPolicy retryPolicy = new ExceptionClassifierRetryPolicy();
Map<Class<? extends Throwable>, RetryPolicy> policyMap = new HashMap<>();
policyMap.put(NonRecoverableException.class, new NeverRetryPolicy());
policyMap.put(RecoverableException.class, new AlwaysRetryPolicy());
retryPolicy.setPolicyMap(policyMap);
ExponentialBackOffPolicy backOffPolicy = new ExponentialBackOffPolicy();
backOffPolicy.setInitialInterval(backoffInitialInterval);
backOffPolicy.setMaxInterval(backoffMaxInterval);
backOffPolicy.setMultiplier(backoffMultiplier);
template.setRetryPolicy(retryPolicy);
template.setBackOffPolicy(backOffPolicy);
return template;
}
my kafka listener using the retry template,
#KafkaListener(
groupId = "${kafka.consumer.group.id}",
topics = "${kafka.consumer.topic}",
containerFactory = "eventContainerFactory")
public void eventListener(ConsumerRecord<String, String>
events,
Acknowledgment acknowledgment) {
eventRetryTemplate.execute(retryContext -> {
retryContext.setAttribute(EVENT, "my-event");
eventConsumer.consume(events, acknowledgment);
return null;
});
}
my kafka consumer properties,
private ConcurrentKafkaListenerContainerFactory<String, String>
getConcurrentKafkaListenerContainerFactory(
KafkaProperties kafkaProperties) {
ConcurrentKafkaListenerContainerFactory<String, String> factory =
new ConcurrentKafkaListenerContainerFactory<>();
factory.getContainerProperties().setAckMode(AckMode.MANUAL_IMMEDIATE);
factory.getContainerProperties().setAckOnError(Boolean.TRUE);
kafkaErrorEventHandler.setCommitRecovered(Boolean.TRUE);
factory.setErrorHandler(kafkaErrorEventHandler);
factory.setConcurrency(1);
factory.getContainerProperties().setAckMode(AckMode.MANUAL_IMMEDIATE);
factory.setConsumerFactory(getEventConsumerFactory(kafkaProperties));
return factory;
}
kafka error event handler is my custom error handler that extends the SeekToCurrentErrorHandler and implements the handle error method some what like this.....
#Override
public void handle(Exception thrownException, List<ConsumerRecord<?, ?>>
records,
Consumer<?, ?> consumer, MessageListenerContainer container) {
log.info("Non recoverable exception. Publishing event to Database");
super.handle(thrownException, records, consumer, container);
ConsumerRecord<String, String> consumerRecord = (ConsumerRecord<String,
String>) records.get(0);
FailedEvent event = createFailedEvent(thrownException, consumerRecord);
failedEventService.insertFailedEvent(event);
log.info("Successfully Published eventId {} to Database...",
event.getEventId());
}
here the failed event service is my custom class again which with put these failed events into a queryable relational DB (I chose this to be my DLQ).
We have a Spring Java application using RabbitMQ, and here is the scenario:
There is a consumer receiving messages from a queue and sending them to another one. We are using "SimpleRabbitListenerContainerFactory" as the container factory, but when sending the messages to the other queue inside a "parallelStream" we've got an IllegalStateException "Cannot determine target ConnectionFactory for lookup key" Exception
When we remove the "parallelStream" it works flawlessly.
public void sendMessage(final StagingMessage stagingMessage, final Long timestamp, final String country) {
final List<TransformedMessage> messages = processMessageList(stagingMessage);
messages.parallelStream().forEach(message -> {
final TransformedMessage transformedMessage = buildMessage(timestamp, ApiConstants.POST_METHOD, country);
myMessageSender.sendQueue(country, transformedMessage);
});
}
Connectio Facotory, where the lookup key is set:
#Configuration
#EnableRabbit
public class RabbitBaseConfig {
#Autowired
private QueueProperties queueProperties;
#Bean
#Primary
public ConnectionFactory connectionFactory(final ConnectionFactory connectionFactoryA, final ConnectionFactory connectionFactoryB) {
final SimpleRoutingConnectionFactory simpleRoutingConnectionFactory = new SimpleRoutingConnectionFactory();
final Map<Object, ConnectionFactory> map = new HashMap<>();
for (final String queue : queueProperties.getAQueueMap().values()) {
map.put("[" + queue + "]", connectionFactoryA);
}
for (final String queue : queueProperties.getBQueueMap().values()) {
map.put("[" + queue + "]", connectionFactoryB);
}
simpleRoutingConnectionFactory.setTargetConnectionFactories(map);
return simpleRoutingConnectionFactory;
}
#Bean
public Jackson2JsonMessageConverter jackson2JsonMessageConverter() {
return new Jackson2JsonMessageConverter();
}
}
Welcome to stack overflow!
You should always show the pertinent code and configuration beans when asking questions like this.
I assume you are using the RoutingConnectionFactory.
It uses a ThreadLocal to store the lookup key so the send has to happen on the same thread that set the key.
You generally should never go asynchronous in a listener anyway; you risk message loss. To increase concurrency, use the concurrency properties on the container.
EDIT
One technique would be to convey the lookup key in a message header:
#Bean
public RabbitTemplate template(ConnectionFactory rcf) {
RabbitTemplate rabbitTemplate = new RabbitTemplate(rcf);
Expression expression = new SpelExpressionParser().parseExpression("messageProperties.headers['cfSelector']");
rabbitTemplate.setSendConnectionFactorySelectorExpression(expression);
return rabbitTemplate;
}
#RabbitListener(queues = "foo")
public void listen1(String in) {
IntStream.range(0, 10)
.parallel()
.mapToObj(i -> in + i)
.forEach(val -> {
this.template.convertAndSend("bar", val.toUpperCase(), msg -> {
msg.getMessageProperties().setHeader("cfSelector", "[bar]");
return msg;
});
});
}
I have the following configuration
spring.rabbitmq.listener.prefetch=1
spring.rabbitmq.listener.concurrency=1
spring.rabbitmq.listener.retry.enabled=true
spring.rabbitmq.listener.retry.max-attempts=3
spring.rabbitmq.listener.retry.max-interval=1000
spring.rabbitmq.listener.default-requeue-rejected=false //I have also changed it to true but the same behavior still happens
and in my listener I throw the exception AmqpRejectAndDontRequeueException to reject the message and enforce rabbit not to try to redeliver it...But rabbit redilvers it for 3 times then finally route it to dead letter queue.
Is that the standard behavior according to my provided configuration or do I miss something?
You have to configure the retry policy to not retry for that exception.
You can't do that with properties, you have to configure the retry advice yourself.
I'll post an example later if you need help with that.
requeue-rejected is at the container level (below retry on the stack).
EDIT
#SpringBootApplication
public class So39853762Application {
public static void main(String[] args) throws Exception {
ConfigurableApplicationContext context = SpringApplication.run(So39853762Application.class, args);
Thread.sleep(60000);
context.close();
}
#RabbitListener(queues = "foo")
public void foo(String foo) {
System.out.println(foo);
if ("foo".equals(foo)) {
throw new AmqpRejectAndDontRequeueException("foo"); // won't be retried.
}
else {
throw new IllegalStateException("bar"); // will be retried
}
}
#Bean
public ListenerRetryAdviceCustomizer retryCustomizer(SimpleRabbitListenerContainerFactory containerFactory,
RabbitProperties rabbitPropeties) {
return new ListenerRetryAdviceCustomizer(containerFactory, rabbitPropeties);
}
public static class ListenerRetryAdviceCustomizer implements InitializingBean {
private final SimpleRabbitListenerContainerFactory containerFactory;
private final RabbitProperties rabbitPropeties;
public ListenerRetryAdviceCustomizer(SimpleRabbitListenerContainerFactory containerFactory,
RabbitProperties rabbitPropeties) {
this.containerFactory = containerFactory;
this.rabbitPropeties = rabbitPropeties;
}
#Override
public void afterPropertiesSet() throws Exception {
ListenerRetry retryConfig = this.rabbitPropeties.getListener().getRetry();
if (retryConfig.isEnabled()) {
RetryInterceptorBuilder<?> builder = (retryConfig.isStateless()
? RetryInterceptorBuilder.stateless()
: RetryInterceptorBuilder.stateful());
Map<Class<? extends Throwable>, Boolean> retryableExceptions = new HashMap<>();
retryableExceptions.put(AmqpRejectAndDontRequeueException.class, false);
retryableExceptions.put(IllegalStateException.class, true);
SimpleRetryPolicy policy =
new SimpleRetryPolicy(retryConfig.getMaxAttempts(), retryableExceptions, true);
ExponentialBackOffPolicy backOff = new ExponentialBackOffPolicy();
backOff.setInitialInterval(retryConfig.getInitialInterval());
backOff.setMultiplier(retryConfig.getMultiplier());
backOff.setMaxInterval(retryConfig.getMaxInterval());
builder.retryPolicy(policy)
.backOffPolicy(backOff)
.recoverer(new RejectAndDontRequeueRecoverer());
this.containerFactory.setAdviceChain(builder.build());
}
}
}
}
NOTE: You cannot currently configure the policy to retry all exceptions, "except" this one - you have to classify all exceptions you want retried (and they can't be a superclass of AmqpRejectAndDontRequeueException). I have opened an issue to support this.
The other answers posted here didn't work me when using Spring Boot 2.3.5 and Spring AMQP Starter 2.2.12, but for these versions I was able to customize the retry policy to not retry AmqpRejectAndDontRequeueException exceptions:
#Configuration
public class RabbitConfiguration {
#Bean
public RabbitRetryTemplateCustomizer customizeRetryPolicy(
#Value("${spring.rabbitmq.listener.simple.retry.max-attempts}") int maxAttempts) {
SimpleRetryPolicy policy = new SimpleRetryPolicy(maxAttempts, Map.of(AmqpRejectAndDontRequeueException.class, false), true, true);
return (target, retryTemplate) -> retryTemplate.setRetryPolicy(policy);
}
}
This lets the retry policy skip retries for AmqpRejectAndDontRequeueExceptions but retries all other exceptions as usual.
Configured this way, it traverses the causes of an exception, and skips retries if it finds an AmqpRejectAndDontRequeueException.
Traversing the causes is needed as org.springframework.amqp.rabbit.listener.adapter.MessagingMessageListenerAdapter#invokeHandler wraps all exceptions as a ListenerExecutionFailedException
I need to be able to receive notification when a ActiveMQ client consumes a MQTT message.
activemq.xml
<destinationPolicy>
<policyMap>
<policyEntries>
<policyEntry topic=">" advisoryForConsumed="true" />
</policyEntries>
</policyMap>
</destinationPolicy>
In the below code, I get MQTT messages on myTopic fine. I do not get advisory messages in processAdvisoryMessage / processAdvisoryBytesMessage.
#Component
public class MqttMessageListener {
#JmsListener(destination = "mytopic")
public void processMessage(BytesMessage message) {
}
#JmsListener(destination = "ActiveMQ.Advisory.MessageConsumed.Topic.>")
public void processAdvisoryMessage(Message message) {
System.out.println("processAdvisoryMessage Got a message");
}
#JmsListener(destination = "ActiveMQ.Advisory.MessageConsumed.Topic.>")
public void processAdvisoryBytesMessage(BytesMessage message) {
System.out.println("processAdvisoryBytesMessageGot a message");
}
}
What am I doing wrong?
I have also attempted doing this with a ActiveMQ BrokerFilter:
public class AMQMessageBrokerFilter extends GenericBrokerFilter {
#Override
public void acknowledge(ConsumerBrokerExchange consumerExchange, MessageAck ack) throws Exception {
super.acknowledge(consumerExchange, ack);
}
#Override
public void postProcessDispatch(MessageDispatch messageDispatch) {
Message message = messageDispatch.getMessage();
}
#Override
public void messageDelivered(ConnectionContext context, MessageReference messageReference) {
log.debug("messageDelivered called.");
super.messageDelivered(context, messageReference);
}
#Override
public void messageConsumed(ConnectionContext context, MessageReference messageReference) {
log.debug("messageConsumed called.");
super.messageConsumed(context, messageReference);
}
In this second scenario I was unable to both have the message and a contect with which to send the consumed notification. acknowledge/messageDelivered/messageConsumed all have a connection context but only postProcessDispatch has the message which I need part of it (payload is JSON) in order to send my outgoing message. I could be eager and use send which has both but it is safer to wait until at least it was acknowledged.
I have tried:
#Override
public void postProcessDispatch(MessageDispatch messageDispatch) {
super.postProcessDispatch(messageDispatch);
String topic = messageDispatch.getDestination().getPhysicalName();
if( topic == null || topic.equals("delivered") )
return;
try {
ActiveMQTopic responseTopic = new ActiveMQTopic("delivered");
ActiveMQTextMessage responseMsg = new ActiveMQTextMessage();
responseMsg.setPersistent(false);
responseMsg.setResponseRequired(false);
responseMsg.setProducerId(new ProducerId());
responseMsg.setText("Delivered msg: "+msg);
responseMsg.setDestination(responseTopic);
String messageKey = ":"+rand.nextLong();
MessageId msgId = new MessageId(messageKey);
responseMsg.setMessageId(msgId);
ProducerBrokerExchange producerExchange=new ProducerBrokerExchange();
ConnectionContext context = getAdminConnectionContext();
producerExchange.setConnectionContext(context);
producerExchange.setMutable(true);
producerExchange.setProducerState(new ProducerState(new ProducerInfo()));
next.send(producerExchange, responseMsg);
}
catch (Exception e) {
log.debug("Exception: "+e);
}
However the above seems to lead to unstable server. I'm thinking this is related to using the getAdminConnectionContext which seems wrong.
My factory was setting setPubSubDomain to false by default. This disables connections for advisory messages for topics. I set it to true and things started working. Note that queues will not work with this set. To get around that I created two factories and named their beans.
#Bean(name="main")
public DefaultJmsListenerContainerFactory jmsListenerContainerFactory(ConnectionFactory connectionFactory) {
DefaultJmsListenerContainerFactory factory = new DefaultJmsListenerContainerFactory();
factory.setConnectionFactory(connectionFactory);
// factory.setDestinationResolver(destinationResolver);
// factory.setPubSubDomain(true);
factory.setConcurrency("3-10");
return factory;
}