Two IntegrationFlows are defined as follows:
#Bean
public IntegrationFlow myFlow() {
return IntegrationFlows.from("input.channel")
.handle("myService", "handle")
.get();
}
#Bean
public IntegrationFlow exceptionFlow() {
return IntegrationFlows.from("error.channel")
.handle(m -> System.out.println(m.getHeaders()))
.get();
}
and the handler of MyService's `handle1 method is just to print out the message and then throw an exception:
public class MyService {
public String handle(String s) {
System.out.println(s);
throw new RuntimeException("error");
}
}
In the test, a message with a defined error channel value which is exactly error.channel is put into the input.channelchannel, and it is expected to route to theerror.channel` channel.
#Test
public void myTest() {
Message<String> m = MessageBuilder.withPayload("foo").setHeader(MessageHeaders.ERROR_CHANNEL, "error.channel").build();
this.someInputChannel.send(m);
}
However, it throws the exception in the test and the message is not routed to the error channel.
That's correct behavior. The errorChannel header is consulted only when there is a thread executor. Any exception is thrown to a caller as a it is done in plain Java. In case of QueueChannel and an ExecutorChannel there is an MessagePublishingErrorHandler which wraps a task call to try..catch and sends an ErrorMessage into an errorChannel header.
In your case it is just plain Java call with that this.someInputChannel.send(m); , so you get an exception directly in this main thread.
See more info in the Reference Manual: https://docs.spring.io/spring-integration/docs/current/reference/html/#namespace-errorhandler
Related
I would like to understand how returning values work for PublishSubscribeChannel having multiple subscribers.
#Bean
public PublishSubscribeChannel channel(){
return new PublishSubscribeChannel();
}
#Bean
#ServiceActivator(inputChannel = "channel")
public MessageHandler handler1() {
//...
return handler1;
}
#Bean
#ServiceActivator(inputChannel = "channel")
public MessageHandler handler2() {
//...
return handler2;
}
#Bean
#ServiceActivator(inputChannel = "channel")
public MessageHandler handler3() {
//...
return handler3;
}
#MessagingGateway
public interface TestGateway{
#Gateway(requestChannel = "channel")
String method(String payload);
}
method expects some String as a return type. If a message is sent to all three handlers via channel, the value coming from which handler would be returned? From what I understand, messages are sent to each subscriber one by one, so would it be the value returned by the last handler?
Also, would it be possible to have handlers returning type different than the method return type, also if it wouldn't necessarily expect String?
When it comes to a scenario where any Exception occurs, I believe if setIgnoreFailures = false, the processing would stop on it and not process to the next handler. Otherwise, the last exception would be thrown.
Thanks in advance
I'm sure there is a specific business task behind your question.
But i you really are about an academic knowledge to see how Spring Integration works internally, then here is some answer for you.
Since your PublishSubscribeChannel is not configure with an Executor, then all your subscribers are called one by one, and only when the previous has done its job. And the part of that job is really a reply producing. So, if your first MessageHandler produced some reply, then exactly this one fulfills CountDownLatch in the TempraryReplyChannel for a gateway request-reply functionality.
The replies from the rest of handlers are going to be ignored and they may throw a late reply error.
Yes, you can return any type as long as it can be converted to the expected return type. See more info about ConversionService: https://docs.spring.io/spring-integration/reference/html/messaging-endpoints.html#payload-type-conversion
About ignoreFailures I'd suggest to look into a PublishSubscribeChannel source code and how it is propagated down to BroadcastingDispatcher:
private boolean invokeHandler(MessageHandler handler, Message<?> message) {
try {
handler.handleMessage(message);
return true;
}
catch (RuntimeException e) {
if (!this.ignoreFailures) {
if (e instanceof MessagingException && ((MessagingException) e).getFailedMessage() == null) { // NOSONAR
throw new MessagingException(message, "Failed to handle Message", e);
}
throw e;
}
else if (this.logger.isWarnEnabled()) {
logger.warn("Suppressing Exception since 'ignoreFailures' is set to TRUE.", e);
}
return false;
}
}
And no: otherwise none exception will be thrown. See that code again.
I've been trying to figure out how route errors to my own error handler with the following, seemingly simple configuration, but Camel is swallowing the exception without routing it to any error handler I configure. I've run out of ideas. Any help would be much appreciated.
I've got a seda route that supports multiple consumers:
#Component
public class MessageGenerator {
public static final String ERROR_GENERATOR_CHANNEL = "seda:my-error-generator?multipleConsumers=true&concurrentConsumers=3";
private final FluentProducerTemplate producerTemplate;
public MessageGenerator(FluentProducerTemplate producerTemplate) {
this.producerTemplate = producerTemplate;
}
public void generateMessage() {
producerTemplate
.to(ERROR_GENERATOR_CHANNEL)
.withBody("Hello World")
.asyncSend();
}
}
I've got two separate POJO consumers:
#Configuration
public class MessageConsumer1 {
#Consume(ERROR_GENERATOR_CHANNEL)
void receiveMessage(String message) {
System.out.println("Received message 1: " + message);
throw new NullPointerException("Error generated");
}
}
#Configuration
public class MessageConsumer2 {
#Consume(ERROR_GENERATOR_CHANNEL)
void receiveMessage(String message) {
System.out.println("Received message 2: " + message);
}
}
When I run the following example, the NullPointerException gets swallowed by the underlying Camel MulticastProcessor as we can see in the logs:
Received message 2: Hello World
Received message 1: Hello World
2022-01-15 13:40:23.711 DEBUG 32945 --- [error-generator] o.a.camel.processor.MulticastProcessor : Message exchange has failed: Multicast processing failed for number 0 for exchange: Exchange[] Exception: java.lang.NullPointerException: Error generated
2022-01-15 13:40:23.711 DEBUG 32945 --- [error-generator] o.a.camel.processor.MulticastProcessor : Message exchange has failed: Multicast processing failed for number 0 for exchange: Exchange[] Exception: java.lang.NullPointerException: Error generated
The exception only gets logged as debug and never gets propagated to any error handler I set up.
Any thoughts on how I could receive the error in my own error handler rather than Camel swallowing the exception as a debug statement?
Note1: I've attempted many variations on both default error handling and default dead letter handling to no avail. I could just be doing it wrong...
Note2: that I'm using Spring[Boot] here too, hence the #Configuration annotation.
Note1: I've attempted many variations on both default error handling and default dead letter handling to no avail. I could just be doing it wrong...
Haven't used #Consume annotations but generally if you want Camel route not to handle any errors you can use .errorHandler(noErrorHandler()). This can be used to pass the error back to parent route or all the way to the code calling ProducerTemplate.sendBody.
Example:
public class ExampleTest extends CamelTestSupport {
#Test
public void noErrorHandlerTest() {
try {
template.sendBody("direct:noErrorHandler", null);
fail();
} catch (Exception e) {
System.out.println("Caught Exception: " + e.getMessage());
}
}
#Override
protected RoutesBuilder createRouteBuilder() throws Exception {
return new RouteBuilder(){
#Override
public void configure() throws Exception {
from("direct:noErrorHandler")
.errorHandler(noErrorHandler())
.log("Throwing error")
.throwException(Exception.class, "Test Exception");
}
};
}
}
I have a quorum queue (myQueue) and it's dead letter queue (myDLQueue). We have several exceptions which we separated as Retryable or Fatal. But sometimes in below listener we make an api call that throws RateLimitException. In this case the application should increase both of retry count and retry delay.
#RabbitListener(queues = "#{myQueue.getName()}", errorHandler = "myErrorHandler")
#SendTo("#{myStatusQueue.getName()}")
public Status process(#Payload MyMessage message, #Headers MessageHeaders headers) {
int retries = headerProcessor.getRetries(headers);
if (retries > properties.getMyQueueMaxRetries()) {
throw new RetriesExceededException(retries);
}
if (retries > 0) {
logger.info("Message {} has been retried {} times. Process it again anyway", kv("task_id", message.getTaskId()), retries);
}
// here we send a request to an api. but sometimes api returns rate limit error in case we send too many requests.
// In that case makeApiCall throws RateLimitException which extends RetryableException
makeApiCall() // --> it will throw RateLimitException
if(/* a condition that needs to retry sending the message*/) {
throw new RetryableException()
}
if(/* a condition that should not retry*/){
throw new FatalException()
}
return new Status("Step 1 Success!");
}
I have also an error handler (myErrorHandler) that catches thrown exceptions from above rabbit listener and manages retry process according to the type of the exception.
public class MyErrorHandler implements RabbitListenerErrorHandler {
#Override
public Object handleError(Message amqpMessage,
org.springframework.messaging.Message<?> message,
ListenerExecutionFailedException exception) {
// Check if error is fatal or retryable
if (exception.getCause() /* ..is fatal? */) {
return new Status("FAIL!");
}
// Retryable exception, rethrow it and let message to be NACKed and retried via DLQ
throw exception;
}
}
Last part I have is a DLQHandler that listens dead letter queue messages and send them to original queue (myQueue).
#Service
public class MyDLQueueHandler {
private final MyAppProperties properties;
private final MessageHeaderProcessor headerProcessor;
private final RabbitProducerService rabbitProducerService;
public MyDLQueueHandler(MyProperties properties, MessageHeaderProcessor headerProcessor, RabbitProducerService rabbitProducerService) {
this.properties = properties;
this.headerProcessor = headerProcessor;
this.rabbitProducerService = rabbitProducerService;
}
/**
* Since message TTL is not available with quorum queues manually listen DL Queue and re-send the message with delay.
* This allows messages to be processed again.
*/
#RabbitListener(queues = {"#{myDLQueue.getName()}"}"})
public void handleError(#Payload Object message, #Headers MessageHeaders headers) {
String routingKey = headerProcessor.getRoutingKey(headers);
Map<String, Object> newHeaders = Map.of(
MessageHeaderProcessor.DELAY, properties.getRetryDelay(), // I need to send increased delay in case of RateLimitException.
MessageHeaderProcessor.RETRIES_HEADER, headerProcessor.getRetries(headers) + 1
);
rabbitProducerService.sendMessageDelayed(message, routingKey, newHeaders);
}
}
In the above handleError method inputs there is not any information related to exception instance thrown from MyErrorHandler or MyQueue listener. Currently I have to pass retry delay by reading it from app.properties. But I need to increase this delay if RateLimitException is thrown. So my question is how do I know which error is thrown from MyErrorHandler while in the MyDLQueueHandler?
When you use the normal dead letter mechanism in RabbitMQ, there is no exception information provided - the message is the original rejected message. However, Spring AMQP provides a RepublishMessageRecoverer which can be used in conjunction with a retry interceptor. In that case, exception information is published in headers.
See https://docs.spring.io/spring-amqp/docs/current/reference/html/#async-listeners
The RepublishMessageRecoverer publishes the message with additional information in message headers, such as the exception message, stack trace, original exchange, and routing key. Additional headers can be added by creating a subclass and overriding additionalHeaders().
#Bean
RetryOperationsInterceptor interceptor() {
return RetryInterceptorBuilder.stateless()
.maxAttempts(5)
.recoverer(new RepublishMessageRecoverer(amqpTemplate(), "something", "somethingelse"))
.build();
}
The interceptor is added to the container's advice chain.
https://github.com/spring-projects/spring-amqp/blob/57596c6a26be2697273cd97912049b92e81d3f1a/spring-rabbit/src/main/java/org/springframework/amqp/rabbit/retry/RepublishMessageRecoverer.java#L55-L61
public static final String X_EXCEPTION_STACKTRACE = "x-exception-stacktrace";
public static final String X_EXCEPTION_MESSAGE = "x-exception-message";
public static final String X_ORIGINAL_EXCHANGE = "x-original-exchange";
public static final String X_ORIGINAL_ROUTING_KEY = "x-original-routingKey";
The exception type can be found in the stack trace header.
I have written a spring cloud stream application where producers are publishing messages to the designated kafka topics. My query is how can I add a producer callback to receive ack/confirmation that the message has been successfully published on the topic? Like how we do in spring kafka producer.send(record, new callback { ... }) (maintaining async producer). Below is my code:
private final Sinks.Many<Message<?>> responseProcessor = Sinks.many().multicast().onBackpressureBuffer();
#Bean
public Supplier<Flux<Message<?>>> event() {
return responseProcessor::asFlux;
}
public Message<?> publishEvent(String status) {
try {
String key = ...;
response = MessageBuilder.withPayload(payload)
.setHeader(KafkaHeaders.MESSAGE_KEY, key)
.build();
responseProcessor.tryEmitNext(response);
}
How can I make sure that tryEmitNext has successfully written to the topic?
Is implementing ProducerListener a solution and possible? Couldn't find a concrete solution/documentation in Spring Cloud Stream
UPDATE
I have implemented below now, seems to work as expected
#Component
public class MyProducerListener<K, V> implements ProducerListener<K, V> {
#Override
public void onSuccess(ProducerRecord<K, V> producerRecord, RecordMetadata recordMetadata) {
// Do nothing on onSuccess
}
#Override
public void onError(ProducerRecord<K, V> producerRecord, RecordMetadata recordMetadata, Exception exception) {
log.error("Producer exception occurred while publishing message : {}, exception : {}", producerRecord, exception);
}
}
#Bean
ProducerMessageHandlerCustomizer<KafkaProducerMessageHandler<?, ?>> customizer(MyProducerListener pl) {
return (handler, destinationName) -> handler.getKafkaTemplate().setProducerListener(pl);
}
See the Kafka Producer Properties.
recordMetadataChannel
The bean name of a MessageChannel to which successful send results should be sent; the bean must exist in the application context. The message sent to the channel is the sent message (after conversion, if any) with an additional header KafkaHeaders.RECORD_METADATA. The header contains a RecordMetadata object provided by the Kafka client; it includes the partition and offset where the record was written in the topic.
ResultMetadata meta = sendResultMsg.getHeaders().get(KafkaHeaders.RECORD_METADATA, RecordMetadata.class)
Failed sends go the producer error channel (if configured); see Error Channels. Default: null
You can add a #ServiceActivator to consume from this channel asynchronously.
I am trying to figure out the best way to handle errors that might have occurred in a service that is called after a aggregate's group timeout occurred that mimics the same flow as if the releaseExpression was met.
Here is my setup:
I have a AmqpInboundChannelAdapter that takes in messages and send them to my aggregator.
When the releaseExpression has been met and before the groupTimeout has expired, if an exception gets thrown in my ServiceActivator, the messages get sent to my dead letter queue for all the messages in that MessageGroup. (10 messages in my example below, which is only used for illustrative purposes) This is what I would expect.
If my releaseExpression hasn't been met but the groupTimeout has been met and the group times out, if an exception gets throw in my ServiceActivator, then the messages do not get sent to my dead letter queue and are acked.
After reading another blog post,
link1
it mentions that this happens because the processing happens in another thread by the MessageGroupStoreReaper and not the one that the SimpleMessageListenerContainer was on. Once processing moves away from the SimpleMessageListener's thread, the messages will be auto ack.
I added the configuration mentioned in the link above and see the error messages getting sent to my error handler. My main question, is what is considered the best way to handle this scenario to minimize message getting lost.
Here are the options I was exploring:
Use a BatchRabbitTemplate in my custom error handler to publish the failed messaged to the same dead letter queue that they would have gone to if the releaseExpression was met. (This is the approach I outlined below but I am worried about messages getting lost, if an error happens during publishing)
Investigate if there is away I could let the SimpleMessageListener know about the error that occurred and have it send the batch of messages that failed to a dead letter queue? I doubt this is possible since it seems the messages are already acked.
Don't set the SimpleMessageListenerContainer to AcknowledgeMode.AUTO and manually ack the messages when they get processed via the Service when the releaseExpression being met or the groupTimeOut happening. (This seems kinda of messy, since there can be 1..N message in the MessageGroup but wanted to see what others have done)
Ideally, I want to have a flow that will that will mimic the same flow when the releaseExpression has been met, so that the messages don't get lost.
Does anyone have recommendation on the best way to handle this scenario they have used in the past?
Thanks for any help and/or advice!
Here is my current configuration using Spring Integration DSL
#Bean
public SimpleMessageListenerContainer workListenerContainer() {
SimpleMessageListenerContainer container =
new SimpleMessageListenerContainer(rabbitConnectionFactory);
container.setQueues(worksQueue());
container.setConcurrentConsumers(4);
container.setDefaultRequeueRejected(false);
container.setTransactionManager(transactionManager);
container.setChannelTransacted(true);
container.setTxSize(10);
container.setAcknowledgeMode(AcknowledgeMode.AUTO);
return container;
}
#Bean
public AmqpInboundChannelAdapter inboundRabbitMessages() {
AmqpInboundChannelAdapter adapter = new AmqpInboundChannelAdapter(workListenerContainer());
return adapter;
}
I have defined a error channel and defined my own taskScheduler to use for the MessageStoreRepear
#Bean
public ThreadPoolTaskScheduler taskScheduler(){
ThreadPoolTaskScheduler ts = new ThreadPoolTaskScheduler();
MessagePublishingErrorHandler mpe = new MessagePublishingErrorHandler();
mpe.setDefaultErrorChannel(myErrorChannel());
ts.setErrorHandler(mpe);
return ts;
}
#Bean
public PollableChannel myErrorChannel() {
return new QueueChannel();
}
public IntegrationFlow aggregationFlow() {
return IntegrationFlows.from(inboundRabbitMessages())
.transform(Transformers.fromJson(SomeObject.class))
.aggregate(a->{
a.sendPartialResultOnExpiry(true);
a.groupTimeout(3000);
a.expireGroupsUponCompletion(true);
a.expireGroupsUponTimeout(true);
a.correlationExpression("T(Thread).currentThread().id");
a.releaseExpression("size() == 10");
a.transactional(true);
}
)
.handle("someService", "processMessages")
.get();
}
Here is my custom error flow
#Bean
public IntegrationFlow errorResponse() {
return IntegrationFlows.from("myErrorChannel")
.<MessagingException, Message<?>>transform(MessagingException::getFailedMessage,
e -> e.poller(p -> p.fixedDelay(100)))
.channel("myErrorChannelHandler")
.handle("myErrorHandler","handleFailedMessage")
.log()
.get();
}
Here is the custom error handler
#Component
public class MyErrorHandler {
#Autowired
BatchingRabbitTemplate batchingRabbitTemplate;
#ServiceActivator(inputChannel = "myErrorChannelHandler")
public void handleFailedMessage(Message<?> message) {
ArrayList<SomeObject> payload = (ArrayList<SomeObject>)message.getPayload();
payload.forEach(m->batchingRabbitTemplate.convertAndSend("some.dlq","#", m));
}
}
Here is the BatchingRabbitTemplate bean
#Bean
public BatchingRabbitTemplate batchingRabbitTemplate() {
ThreadPoolTaskScheduler scheduler = new ThreadPoolTaskScheduler();
scheduler.setPoolSize(5);
scheduler.initialize();
BatchingStrategy batchingStrategy = new SimpleBatchingStrategy(10, Integer.MAX_VALUE, 30000);
BatchingRabbitTemplate batchingRabbitTemplate = new BatchingRabbitTemplate(batchingStrategy, scheduler);
batchingRabbitTemplate.setConnectionFactory(rabbitConnectionFactory);
return batchingRabbitTemplate;
}
Update 1) to show custom MessageGroupProcessor:
public class CustomAggregtingMessageGroupProcessor extends AbstractAggregatingMessageGroupProcessor {
#Override
protected final Object aggregatePayloads(MessageGroup group, Map<String, Object> headers) {
return group;
}
}
Example Service:
#Slf4j
public class SomeService {
#ServiceActivator
public void processMessages(MessageGroup messageGroup) throws IOException {
Collection<Message<?>> messages = messageGroup.getMessages();
//Do business logic
//ack messages in the group
for (Message<?> m : messages) {
com.rabbitmq.client.Channel channel = (com.rabbitmq.client.Channel)
m.getHeaders().get("amqp_channel");
long deliveryTag = (long) m.getHeaders().get("amqp_deliveryTag");
log.debug(" deliveryTag = {}",deliveryTag);
log.debug("Channel = {}",channel);
channel.basicAck(deliveryTag, false);
}
}
}
Updated integrationFlow
public IntegrationFlow aggregationFlowWithCustomMessageProcessor() {
return IntegrationFlows.from(inboundRabbitMessages()).transform(Transformers.fromJson(SomeObject.class))
.aggregate(a -> {
a.sendPartialResultOnExpiry(true);
a.groupTimeout(3000);
a.expireGroupsUponCompletion(true);
a.expireGroupsUponTimeout(true);
a.correlationExpression("T(Thread).currentThread().id");
a.releaseExpression("size() == 10");
a.transactional(true);
a.outputProcessor(new CustomAggregtingMessageGroupProcessor());
}).handle("someService", "processMessages").get();
}
New ErrorHandler to do nack
public class MyErrorHandler {
#ServiceActivator(inputChannel = "myErrorChannelHandler")
public void handleFailedMessage(MessageGroup messageGroup) throws IOException {
if(messageGroup!=null) {
log.debug("Nack messages size = {}", messageGroup.getMessages().size());
Collection<Message<?>> messages = messageGroup.getMessages();
for (Message<?> m : messages) {
com.rabbitmq.client.Channel channel = (com.rabbitmq.client.Channel)
m.getHeaders().get("amqp_channel");
long deliveryTag = (long) m.getHeaders().get("amqp_deliveryTag");
log.debug("deliveryTag = {}",deliveryTag);
log.debug("channel = {}",channel);
channel.basicNack(deliveryTag, false, false);
}
}
}
}
Update 2 Added custom ReleaseStratgedy and change to aggegator
public class CustomMeasureGroupReleaseStratgedy implements ReleaseStrategy {
private static final int MAX_MESSAGE_COUNT = 10;
public boolean canRelease(MessageGroup messageGroup) {
return messageGroup.getMessages().size() >= MAX_MESSAGE_COUNT;
}
}
public IntegrationFlow aggregationFlowWithCustomMessageProcessorAndReleaseStratgedy() {
return IntegrationFlows.from(inboundRabbitMessages()).transform(Transformers.fromJson(SomeObject.class))
.aggregate(a -> {
a.sendPartialResultOnExpiry(true);
a.groupTimeout(3000);
a.expireGroupsUponCompletion(true);
a.expireGroupsUponTimeout(true);
a.correlationExpression("T(Thread).currentThread().id");
a.transactional(true);
a.releaseStrategy(new CustomMeasureGroupReleaseStratgedy());
a.outputProcessor(new CustomAggregtingMessageGroupProcessor());
}).handle("someService", "processMessages").get();
}
There are some flaws in your understanding.If you use AUTO, only the last message will be dead-lettered when an exception occurs. Messages successfully deposited in the group, before the release, will be ack'd immediately.
The only way to achieve what you want is to use MANUAL acks.
There is no way to "tell the listener container to send messages to the DLQ". The container never sends messages to the DLQ, it rejects a message and the broker sends it to the DLX/DLQ.