I am using application events in my service and decided to go for multicaster since I can set up the error handler and get the stacktrace in the console (normally runtime exceptions are not caught and are silently surpressed). So I defined my multicaster config as following:
#Configuration
class ApplicationEventMulticasterConfig {
companion object {
private val log = LoggerFactory.getLogger(ApplicationEventMulticasterConfig::class.java)
}
#Bean(name = ["applicationEventMulticaster"])
fun simpleApplicationEventMulticaster(multicasterExecutor: TaskExecutor): ApplicationEventMulticaster {
val eventMulticaster = SimpleApplicationEventMulticaster()
eventMulticaster.setTaskExecutor(multicasterExecutor)
eventMulticaster.setErrorHandler { throwable ->
log.error(throwable.stackTraceToString())
}
return eventMulticaster
}
#Bean(name = ["multicasterExecutor"])
fun taskExecutor(): TaskExecutor {
val executor = ThreadPoolTaskExecutor()
executor.corePoolSize = 4
executor.maxPoolSize = 40
executor.initialize()
return executor
}
}
Listener Case1:
#TransactionalEventListener
fun onEvent(event: Events.Created) {
Listener Case2:
#TransactionalEventListener(fallbackExecution=true)
fun onEvent(event: Events.Created) {
Publishing with multicaster.multicastEvent(Events.Created()). This simply does not work as expected, in case 1, the listener is not started at all (if transaction commits or rolls back) and in case 2 is listener triggered EACH time (in case of commit or failure).
If I delete the whole ApplicationEventMulticasterConfig everything is working fine but I do not have error handler set. Do you have any idea what could be wrong? It might be something with the way how I set up those beans.
Related
I am trying to implement the spring kafka consumers which needs to be paused after a certain exception while processing the event (ex: while storing the event info to DB, DB is down).
How do we handle this scenario using Resilience4j circuit breaker approach with spring boot - 2.3.8 (spring kafka)
Looking for some examples on the consumer to pause and resume also.
#Component
public class CircuitBreakerManager {
private CircuitBreaker circuitBreaker;
#Autowired
private KafkaListenerEndpointRegistry kafkaListenerEndpointRegistry;
public CircuitBreakerManager() {
CircuitBreakerConfig circuitBreakerConfig = CircuitBreakerConfig.custom()
.slidingWindowType(CircuitBreakerConfig.SlidingWindowType.COUNT_BASED)
.enableAutomaticTransitionFromOpenToHalfOpen()
.minimumNumberOfCalls(5)
.permittedNumberOfCallsInHalfOpenState(3)
.slidingWindowSize(10)
.failureRateThreshold(50)
.slowCallRateThreshold(60.0f)
.slowCallDurationThreshold(Duration.ofSeconds(3))
.build();
CircuitBreakerRegistry registry = CircuitBreakerRegistry.of(circuitBreakerConfig);
this.circuitBreaker = registry.circuitBreaker("serialization_exception");
this.circuitBreaker.getEventPublisher().onStateTransition(this::onStateChange);
}
private void onStateChange(CircuitBreakerOnStateTransitionEvent circuitBreakerEvent) {
CircuitBreaker.State toState = circuitBreakerEvent.getStateTransition()
.getToState();
System.out.println("Change in Circuit Breaker state " + toState);
switch (toState) {
case OPEN:
kafkaListenerEndpointRegistry.getListenerContainer("my_listener_id").stop();
break;
case CLOSED:
break;
case HALF_OPEN:
kafkaListenerEndpointRegistry.getListenerContainer("my_listener_id").start();
break;
}
}
}
At kafka listerner Just wanted to catch the parse error.if we get more than 5 parsing errors , the listener needs to be stopped. But i am not sure how the circuit breaker will get triggered.
#CircuitBreaker(name = RESILIENCE4J_INSTANCE_NAME)
private Event getParsedEvent(ConsumerRecord consumerRecord) {
Event event = getEvent(consumerRecord);
if (StringUtils.isEmpty(event)) {
throw new RuntimeException("Serialization Exception occurred");
}
}
return event;
}
See Pausing and Resuming Listener Containers.
Note that pause won't take effect until all the records returned from the current poll have been processed (or an exception is thrown by the listener, as long as the default error handler is in place).
We are using RabbitMq with default spring boot configurations. We have a use case in which we want no parallelism for one of the listeners. That is, we want only one thread of the consumer to be running at any given point in time. We want this, because the nature of the use case is such that we want the messages to be consumed in order, thus if there are multiple threads per consumer there can be chances that the messages are processed out of order.
Since, we are using the defaults and have not explicitly tweaked the container, we are using the SimpleMessageListenerContainer. By looking at the documentation I tried fixing the number of consumers using concurrency = "1" . The annotation on the target method looks like this #RabbitListener(queues = ["queue-name"], concurrency = "1").
As per the documentation this should have ensured that there is only consumer thread.
{#link org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer
* SimpleMessageListenerContainer} if this value is a simple integer, it sets a fixed
* number of consumers in the {#code concurrentConsumers} property
2021-10-29 06:11:26.361 INFO 29752 --- [ntContainer#4-1] c.t.t.i.p.s.xxx : Created xxx
2021-10-29 06:11:26.383 INFO 29752 --- [ntContainer#0-1] c.t.t.i.p.s.xxx : Created xxx
ThreadIds to be noted here are [ntContainer#4-1] and [ntContainer#0-1].
So the question is- how can we ensure that there is only one thread per consumer at any given point in time ?
Edit: Adding the code of the consumer class for more context
#ConditionalOnProperty(value = ["rabbitmq.sharebooking.enabled"], havingValue = "true", matchIfMissing = false)
class ShareBookingConsumer #Autowired constructor(
private val shareBookingRepository: ShareBookingRepository,
private val objectMapper: ObjectMapper,
private val shareDtoToShareBookingConverter: ShareBookingDtoToShareBookingConverter
) {
private val logger = LoggerFactory.getLogger(javaClass)
init {
logger.info("start sharebooking created consumer")
}
#RabbitListener(queues = ["tax_engine.share_booking"], concurrency = "1-1", exclusive = true)
#Timed
#Transactional
fun consumeShareBookingCreatedEvent(message: Message) {
try {
consumeShareBookingCreatedEvent(message.body)
} catch (e: Exception) {
throw AmqpRejectAndDontRequeueException(e)
}
}
private fun consumeShareBookingCreatedEvent(event: ByteArray) {
toShareBookingCreationMessageEvent(event).let { creationEvent ->
RmqMetrics.measureEventMetrics(creationEvent)
val shareBooking = shareDtoToShareBookingConverter.convert(creationEvent.data)
val persisted = shareBookingRepository.save(shareBooking)
logger.info("Created shareBooking ${creationEvent.data.id}")
}
}
private fun toShareBookingCreationMessageEvent(event: ByteArray) =
objectMapper.readValue(event, shareBookingCreateEventType)
companion object {
private val shareBookingCreateEventType =
object : TypeReference<RMQMessageEnvelope<ShareBookingCreationDto>>() {}
}
}
Edit: Adding application thread analysis using visualvm
5 threads get created for 5 listeners.
[1]: https://i.stack.imgur.com/gQINE.png
Set concurrency = "1-1". Note that the concurrency of the Listener depends not only on concurrentConsumers, but also on maxConcurrentConsumers:
If you are using a custom factory:
#Bean
public SimpleRabbitListenerContainerFactory rabbitListenerContainerFactory(CachingConnectionFactory cachingConnectionFactory) {
SimpleRabbitListenerContainerFactory factory = new SimpleRabbitListenerContainerFactory();
factory.setConnectionFactory(cachingConnectionFactory);
factory.setConcurrentConsumers(1);
factory.setMaxConcurrentConsumers(1);
return factory;
}
See: https://docs.spring.io/spring-amqp/docs/current/reference/html/#simplemessagelistenercontainer for detail.
EDIT:
I did a simple test, 2 consumers&2 threads:
#RabbitListener(queues = "myQueue111", concurrency = "1-1")
public void handleMessage(Object message) throws InterruptedException {
LOGGER.info("Received message : {} in {}", message, Thread.currentThread().getName());
}
#RabbitListener(queues = "myQueue222", concurrency = "1-1")
public void handleMessag1e(Object message) throws InterruptedException {
LOGGER.info("Received message222 : {} in {}", message, Thread.currentThread().getName());
}
Try this:
#RabbitListener(queues = ["queue-name"], exclusive = true)
See https://docs.spring.io/spring-amqp/docs/current/reference/html/#exclusive-consumer
I have MessageListener.onMessage with a thread sleep. I'm simulating actual processing time the onMessage
method will take by the above mentioned Thread sleep. However what I have noticed is that it is getting called multiple times consecutively for the remaining messages till they get processed by the onMessage method. I see this as an inefficiency.
Actual message count in to queue : 1000
Output of running number for hits
onMessage<<15656
onMessage<<15657
onMessage<<15658
onMessage<<15659
onMessage<<15660
onMessage<<15661
onMessage<<15662
onMessage<<15663
Code block
#Service
class ThreadPooledMessageListener implements MessageListener {
#Autowired
TaskExecutor threadPoolTaskExecutor;
AtomicInteger processedCount = new AtomicInteger();
#Override
public void onMessage(Message message) {
System.out.println("onMessage<<" + processedCount.incrementAndGet());
threadPoolTaskExecutor.execute(new MessageProcessor(message));
}
}
class MessageProcessor implements Runnable {
Message processingMessage;
public MessageProcessor(Message message) {
this.processingMessage = message;
}
#Override
public void run() {
System.out.println("================================"+ Thread.currentThread().getName());
System.out.println(processingMessage);
try {
Thread.sleep(1000);
} catch (InterruptedException e) {
e.printStackTrace();
}
System.out.println("================================");
}
}
What are the possible fixes for this.
As #Gary Russell has pointed out; Issue was that I have used non-spring managed container SimpleMessageListenerContainer in my code. Fixed it with spring managed bean and defined concurrency there. Works as expected.
Fixed code segment
#Bean
public SimpleMessageListenerContainer simpleMessageListenerContainer() {
SimpleMessageListenerContainer container = new SimpleMessageListenerContainer(connectionFactory);
container.setQueues(queue);
container.setMessageListener(threadPooledMessageListener);
container.setConcurrentConsumers(4);
container.start();
return container;
}
>I see this as an inefficiency.
It's not clear what you mean. Since you are handing off the processing of a message to another thread, the listener exits immediately and, of course, the next message is delivered.
This will risk message loss in the event of a failure.
If you are trying to achieve concurrency; it's better to set the container concurrentConsumers property and not do your own thread management in the listener. The container will manage the consumers for you.
I am trying to figure out the best way to handle errors that might have occurred in a service that is called after a aggregate's group timeout occurred that mimics the same flow as if the releaseExpression was met.
Here is my setup:
I have a AmqpInboundChannelAdapter that takes in messages and send them to my aggregator.
When the releaseExpression has been met and before the groupTimeout has expired, if an exception gets thrown in my ServiceActivator, the messages get sent to my dead letter queue for all the messages in that MessageGroup. (10 messages in my example below, which is only used for illustrative purposes) This is what I would expect.
If my releaseExpression hasn't been met but the groupTimeout has been met and the group times out, if an exception gets throw in my ServiceActivator, then the messages do not get sent to my dead letter queue and are acked.
After reading another blog post,
link1
it mentions that this happens because the processing happens in another thread by the MessageGroupStoreReaper and not the one that the SimpleMessageListenerContainer was on. Once processing moves away from the SimpleMessageListener's thread, the messages will be auto ack.
I added the configuration mentioned in the link above and see the error messages getting sent to my error handler. My main question, is what is considered the best way to handle this scenario to minimize message getting lost.
Here are the options I was exploring:
Use a BatchRabbitTemplate in my custom error handler to publish the failed messaged to the same dead letter queue that they would have gone to if the releaseExpression was met. (This is the approach I outlined below but I am worried about messages getting lost, if an error happens during publishing)
Investigate if there is away I could let the SimpleMessageListener know about the error that occurred and have it send the batch of messages that failed to a dead letter queue? I doubt this is possible since it seems the messages are already acked.
Don't set the SimpleMessageListenerContainer to AcknowledgeMode.AUTO and manually ack the messages when they get processed via the Service when the releaseExpression being met or the groupTimeOut happening. (This seems kinda of messy, since there can be 1..N message in the MessageGroup but wanted to see what others have done)
Ideally, I want to have a flow that will that will mimic the same flow when the releaseExpression has been met, so that the messages don't get lost.
Does anyone have recommendation on the best way to handle this scenario they have used in the past?
Thanks for any help and/or advice!
Here is my current configuration using Spring Integration DSL
#Bean
public SimpleMessageListenerContainer workListenerContainer() {
SimpleMessageListenerContainer container =
new SimpleMessageListenerContainer(rabbitConnectionFactory);
container.setQueues(worksQueue());
container.setConcurrentConsumers(4);
container.setDefaultRequeueRejected(false);
container.setTransactionManager(transactionManager);
container.setChannelTransacted(true);
container.setTxSize(10);
container.setAcknowledgeMode(AcknowledgeMode.AUTO);
return container;
}
#Bean
public AmqpInboundChannelAdapter inboundRabbitMessages() {
AmqpInboundChannelAdapter adapter = new AmqpInboundChannelAdapter(workListenerContainer());
return adapter;
}
I have defined a error channel and defined my own taskScheduler to use for the MessageStoreRepear
#Bean
public ThreadPoolTaskScheduler taskScheduler(){
ThreadPoolTaskScheduler ts = new ThreadPoolTaskScheduler();
MessagePublishingErrorHandler mpe = new MessagePublishingErrorHandler();
mpe.setDefaultErrorChannel(myErrorChannel());
ts.setErrorHandler(mpe);
return ts;
}
#Bean
public PollableChannel myErrorChannel() {
return new QueueChannel();
}
public IntegrationFlow aggregationFlow() {
return IntegrationFlows.from(inboundRabbitMessages())
.transform(Transformers.fromJson(SomeObject.class))
.aggregate(a->{
a.sendPartialResultOnExpiry(true);
a.groupTimeout(3000);
a.expireGroupsUponCompletion(true);
a.expireGroupsUponTimeout(true);
a.correlationExpression("T(Thread).currentThread().id");
a.releaseExpression("size() == 10");
a.transactional(true);
}
)
.handle("someService", "processMessages")
.get();
}
Here is my custom error flow
#Bean
public IntegrationFlow errorResponse() {
return IntegrationFlows.from("myErrorChannel")
.<MessagingException, Message<?>>transform(MessagingException::getFailedMessage,
e -> e.poller(p -> p.fixedDelay(100)))
.channel("myErrorChannelHandler")
.handle("myErrorHandler","handleFailedMessage")
.log()
.get();
}
Here is the custom error handler
#Component
public class MyErrorHandler {
#Autowired
BatchingRabbitTemplate batchingRabbitTemplate;
#ServiceActivator(inputChannel = "myErrorChannelHandler")
public void handleFailedMessage(Message<?> message) {
ArrayList<SomeObject> payload = (ArrayList<SomeObject>)message.getPayload();
payload.forEach(m->batchingRabbitTemplate.convertAndSend("some.dlq","#", m));
}
}
Here is the BatchingRabbitTemplate bean
#Bean
public BatchingRabbitTemplate batchingRabbitTemplate() {
ThreadPoolTaskScheduler scheduler = new ThreadPoolTaskScheduler();
scheduler.setPoolSize(5);
scheduler.initialize();
BatchingStrategy batchingStrategy = new SimpleBatchingStrategy(10, Integer.MAX_VALUE, 30000);
BatchingRabbitTemplate batchingRabbitTemplate = new BatchingRabbitTemplate(batchingStrategy, scheduler);
batchingRabbitTemplate.setConnectionFactory(rabbitConnectionFactory);
return batchingRabbitTemplate;
}
Update 1) to show custom MessageGroupProcessor:
public class CustomAggregtingMessageGroupProcessor extends AbstractAggregatingMessageGroupProcessor {
#Override
protected final Object aggregatePayloads(MessageGroup group, Map<String, Object> headers) {
return group;
}
}
Example Service:
#Slf4j
public class SomeService {
#ServiceActivator
public void processMessages(MessageGroup messageGroup) throws IOException {
Collection<Message<?>> messages = messageGroup.getMessages();
//Do business logic
//ack messages in the group
for (Message<?> m : messages) {
com.rabbitmq.client.Channel channel = (com.rabbitmq.client.Channel)
m.getHeaders().get("amqp_channel");
long deliveryTag = (long) m.getHeaders().get("amqp_deliveryTag");
log.debug(" deliveryTag = {}",deliveryTag);
log.debug("Channel = {}",channel);
channel.basicAck(deliveryTag, false);
}
}
}
Updated integrationFlow
public IntegrationFlow aggregationFlowWithCustomMessageProcessor() {
return IntegrationFlows.from(inboundRabbitMessages()).transform(Transformers.fromJson(SomeObject.class))
.aggregate(a -> {
a.sendPartialResultOnExpiry(true);
a.groupTimeout(3000);
a.expireGroupsUponCompletion(true);
a.expireGroupsUponTimeout(true);
a.correlationExpression("T(Thread).currentThread().id");
a.releaseExpression("size() == 10");
a.transactional(true);
a.outputProcessor(new CustomAggregtingMessageGroupProcessor());
}).handle("someService", "processMessages").get();
}
New ErrorHandler to do nack
public class MyErrorHandler {
#ServiceActivator(inputChannel = "myErrorChannelHandler")
public void handleFailedMessage(MessageGroup messageGroup) throws IOException {
if(messageGroup!=null) {
log.debug("Nack messages size = {}", messageGroup.getMessages().size());
Collection<Message<?>> messages = messageGroup.getMessages();
for (Message<?> m : messages) {
com.rabbitmq.client.Channel channel = (com.rabbitmq.client.Channel)
m.getHeaders().get("amqp_channel");
long deliveryTag = (long) m.getHeaders().get("amqp_deliveryTag");
log.debug("deliveryTag = {}",deliveryTag);
log.debug("channel = {}",channel);
channel.basicNack(deliveryTag, false, false);
}
}
}
}
Update 2 Added custom ReleaseStratgedy and change to aggegator
public class CustomMeasureGroupReleaseStratgedy implements ReleaseStrategy {
private static final int MAX_MESSAGE_COUNT = 10;
public boolean canRelease(MessageGroup messageGroup) {
return messageGroup.getMessages().size() >= MAX_MESSAGE_COUNT;
}
}
public IntegrationFlow aggregationFlowWithCustomMessageProcessorAndReleaseStratgedy() {
return IntegrationFlows.from(inboundRabbitMessages()).transform(Transformers.fromJson(SomeObject.class))
.aggregate(a -> {
a.sendPartialResultOnExpiry(true);
a.groupTimeout(3000);
a.expireGroupsUponCompletion(true);
a.expireGroupsUponTimeout(true);
a.correlationExpression("T(Thread).currentThread().id");
a.transactional(true);
a.releaseStrategy(new CustomMeasureGroupReleaseStratgedy());
a.outputProcessor(new CustomAggregtingMessageGroupProcessor());
}).handle("someService", "processMessages").get();
}
There are some flaws in your understanding.If you use AUTO, only the last message will be dead-lettered when an exception occurs. Messages successfully deposited in the group, before the release, will be ack'd immediately.
The only way to achieve what you want is to use MANUAL acks.
There is no way to "tell the listener container to send messages to the DLQ". The container never sends messages to the DLQ, it rejects a message and the broker sends it to the DLX/DLQ.
I am using Spring 4.1.x APIs, Spring Integration 4.1.x APIs and Spring Integration Java DSL 1.0.x APIs for an EIP flow where we consume messages from an Oracle database table using a JdbcPollingChannelAdpater as the entry point into the flow.
Even though we have an ErrorHandler configured on the JdbcPollingChannelAdapter's Poller, we are seeing that a session's Transaction is still rolled back and not committed when a RuntimeException is thrown and correctly handled by the ErrorHandler.
After reading through this thread: Spring Transactions - Prevent rollback after unchecked exceptions (RuntimeException), I get the feeling that it is not possible to prevent a rollback and instead force a commit. Is this correct? And, if there is a way, what is the cleanest way to force a commit instead of a rollback when an error is safely handled?
Current Configuration:
IntegrationConfig.java:
#Bean
public MessageSource<Object> jdbcMessageSource() {
JdbcPollingChannelAdapter adapter = new JdbcPollingChannelAdapter(
dataSource,
"select * from SERVICE_TABLE where rownum <= 10 for update skip locked");
adapter.setUpdateSql("delete from SERVICE_TABLE where SERVICE_MESSAGE_ID in (:id)");
adapter.setRowMapper(serviceMessageRowMapper);
adapter.setMaxRowsPerPoll(1);
adapter.setUpdatePerRow(true);
return adapter;
}
#SuppressWarnings("unchecked")
#Bean
public IntegrationFlow inFlow() {
return IntegrationFlows
.from(jdbcMessageSource(),
c -> {
c.poller(Pollers.fixedRate(100)
.maxMessagesPerPoll(10)
.transactional(transactionManager)
.errorHandler(errorHandler));
})
.channel(inProcessCh()).get();
}
ErrorHandler.java
#Component
public class ErrorHandler implements org.springframework.util.ErrorHandler {
#Autowired
private PlatformTransactionManager transactionManager;
private static final Logger logger = LogManager.getLogger();
#Override
public void handleError(Throwable t) {
logger.trace("handling error:{}", t.getMessage(), t);
// handle error code here...
// we want to force commit the transaction here?
TransactionStatus txStatus = transactionManager.getTransaction(null);
transactionManager.commit(txStatus);
}
}
--- EDITED to include ExpressionEvaluatingRequestHandlerAdvice Bean ---
#Bean
public Advice expressionEvaluatingRequestHandlerAdvice() {
ExpressionEvaluatingRequestHandlerAdvice expressionEvaluatingRequestHandlerAdvice = new ExpressionEvaluatingRequestHandlerAdvice();
expressionEvaluatingRequestHandlerAdvice.setTrapException(true);
expressionEvaluatingRequestHandlerAdvice.setOnSuccessExpression("payload");
expressionEvaluatingRequestHandlerAdvice
.setOnFailureExpression("payload");
expressionEvaluatingRequestHandlerAdvice.setFailureChannel(errorCh());
return expressionEvaluatingRequestHandlerAdvice;
}
--- EDITED to show Dummy Test Message handler ---
.handle(Message.class,
(m, h) -> {
boolean forceTestError = m.getHeaders().get("forceTestError");
if (forceTestError) {
logger.trace("simulated forced TestException");
TestException testException = new TestException(
"forced test exception");
throw testException;
}
logger.trace("simulated successful process");
return m;
}, e-> e.advice(expressionEvaluatingRequestHandlerAdvice())
--- EDITED to show ExecutorChannelInterceptor method ---
#Bean
public IntegrationFlow inFlow() {
return IntegrationFlows
.from(jdbcMessageSource(), c -> {
c.poller(Pollers.fixedRate(100).maxMessagesPerPoll(10)
.transactional(transactionManager));
})
.enrichHeaders(h -> h.header("errorChannel", errorCh(), true))
.channel(
MessageChannels.executor("testSyncTaskExecutor",
syncTaskExecutor()).interceptor(
testExecutorChannelInterceptor()))
.handle(Message.class, (m, h) -> {
boolean forceTestError = m.getHeaders().get("forceTestError");
if (forceTestError) {
logger.trace("simulated forced TestException");
TestException testException = new TestException(
"forced test exception");
throw testException;
}
logger.trace("simulated successful process");
}).channel("nullChannel").get();
}
It won't work just because your ErrorHandler works already after the finish of TX.
Here is a couple lines of source code (AbstractPollingEndpoint.Poller):
#Override
public void run() {
taskExecutor.execute(new Runnable() {
#Override
public void run() {
.............
try {
if (!pollingTask.call()) {
break;
}
count++;
}
catch (Exception e) {
....
}
}
}
});
}
Where:
The ErrorHandler is applied for the taskExecutor (SyncTaskExecutor) by default.
TransactionInterceptor being as Aspect is applied for the Proxy around that pollingTask.
Therefore TX is done around the pollingTask.call() and goes out. And only after that your ErrorHandler starts to work inside taskExecutor.execute().
To fix your issue, you need to figure out which downstream flow part isn't so critical for TX rallback and make there some try...catch or use ExpressionEvaluatingRequestHandlerAdvice to "burke" that RuntimeException.
But as you have noticed by my reasoning that must be done within TX.