I have written code that uses webclient to call another endpoint and want to add reactive error handling. However, it seems my understanding of doOnError or onErrorResume may not be correct:
webClient
.get()
.uri(someUri)
.retrieve()
.bodyToFlux(Some.class)
.onErrorResume(throwable -> {
log.error("Error occurred when calling other service: {}", throwable.getMessage());
return Flux.error(new RunTimeException("Exception type: " + throwable.getClass() + " Exception message: " + throwable.getMessage()));
});
Then intention is that this call is actually part of a larger reactive chain that calls this, and if an exception is thrown whilst running the api call (.get().retrieve()), onErrorResume should throw and pass on the exception to the higher level reactive chain caller.
I tried to unit test the validity of this by:
Mockito.when(webClient.get().uri(URI.create(uri)).retrieve()).thenThrow(new RuntimeException("Hello world exception thrown"));
But noticed that the exception just gets thrown, and the code terminates at the .retrieve step of the reactive chain, rather than proceeding to the onErrorResume step.
This is because your test throws an error when constructing the retrieve Mono, instead of returning a functional Mono that immediately emits an error when subscribed to. Thus it's not your data flow that is in error, but the pipeline handling the data flow itself.
You can solve this by returning a Mono.error:
Mockito.when(webClient.get().uri(URI.create(uri)).retrieve())
.thenReturn(Mono.error(new RuntimeException("Hello world exception thrown")));
Related
I've written a Web client to interact with some external service using spring project-reactor. The external service sometimes throttles incoming requests. How can I provide different Retry types based on the different responses/exceptions?
For Instance:
If it is a response with a 429 error code, create a Retry object with the duration provided in the response header (e.g retry-after), or else,
If it is some other exception, for example 5XX, then retry with exponential backoff:
Retry.backoff(MAX_ATTEMPTS, Duration.ofMillis(MILLIS))
The client API call code is:
.bodyValue(inputQuery)
.retrieve()
.bodyToMono(QueryResult.class)
.retryWhen(customStrategy)
.doOnError(ex -> log.debug("API invocation error: ", ex));
customStrategy can will have the logic to decide which Retry object to create. Is there anyway we can achieve this?
You can specify more than one .retryWhen in the statement and create your Retry objects with the appropriate filters.
Something like that:
.bodyToMono(String.class)
.retryWhen(
Retry
.backoff(3, Duration.ofSeconds(5))
.filter(throwable -> throwable instanceof WebClientResponseException.TooManyRequests)
)
.retryWhen(
Retry
.fixedDelay(3, Duration.ofSeconds(1))
.filter(throwable -> ((WebClientResponseException) throwable).getStatusCode().is5xxServerError())
)
.block();
Here, you can customize your Filter and Retry objects any way you want.
I am using reactor kafka to send in kafka messages and receive and process them.
While receiving the kakfa payload, I do some deserialization, and if there is an exception, I want to just log that payload ( by saving to mongo ), and then continue receiving other payloads.
For this I am using the below approach -
#EventListener(ApplicationStartedEvent.class)
public void kafkaReceiving() {
for(Flux<ReceiverRecord<String, Object>> flux: kafkaService.getFluxReceives()) {
flux.delayUntil(//some function to do something)
.doOnNext(r -> r.receiverOffset().acknowledge())
.onErrorResume(this::handleException()) // here I'll just save to mongo
.subscribe();
}
}
private Publisher<? extends ReceiverRecord<String,Object>> handleException(object ex) {
// save to mongo
return Flux.empty();
}
Here I expect that whenever I encounter an exception while receiving a payload, the onErrorResume should catch it and log to mongo and then I should be good to continue receiving more messages when I send to the kafka queue. However, I see that after the exception, even though the onErrorResume method gets invoked, but I am not able to process anymore messages sent to Kakfa topic.
Anything I might be missing here?
If you need to handle the error gracefully, you can add onErrorResume inside delayUntil:
flux
.delayUntil(r -> {
return process(r)
.onErrorReturn(e -> saveToMongo(r));
});
.doOnNext(r -> r.receiverOffset().acknowledge())
.subscribe();
Reactive operators treat error as a terminal signal, and, if your inner logic (inside delayUntil) throws an error, delayUntil will terminate the sequence, and onErrorReturn after delayUntil will not make it continue processing the events from Kafka.
As mentioned by #bsideup too, I ultimately went ahead with not throwing exception from the deserializer, since the kafka is not able to commit offset for that record, and there is no clean way of ignoring that record and going ahead with further consumption of records as we dont have the offset information of the record( since it is malformed). So even if I try to ignore the record using reactive error operators, the poll fetches the same record, and the consumer is then kind of stuck
I'm working on a project in which we are migrating from RestTemplate to WebClient.
The WebClient is implemented like this:
try {
...
return webClient
.get()
.uri(someUri)
.retrieve()
.toEntity(SomeBusinessClass.class).block()
} catch(WebClientException e) {
// do some stuff
// want to catch IOExceptions here as well
}
While refactoring the code I had to refactor the tests as well and I've come across a test in which we basically throw an ConnectException to see if our internal code catch them according to our needs. With RestTemplate's exception classes we was able to define the exception like this:
ResourceAccessException exc = new ResourceAccessException("I/O error on GET request", new ConnectException("Connection refused: connect"))
I tried to do the same with WebClient's provided exception class WebClientException but that's an abstract class and the only class inheriting from it is WebClientResponseException and that don't provide a constructor which would allow to do the same. So my only option was to do it with RuntimeException:
RuntimeException exc = new RuntimeException("I/O error on GET request", new ConnectException("Connection refused: connect"))
But since I don't want to rewrite our internal code to catch exceptions on RuntimeException level but on WebClientException level, is that not an option and I'm wondering how to do that?
I tried to find out in the Spring docs how to handle IOException's while using WebClient but couldn't find anything.
What would be the approach here?
The nicest way would almost certainly be to handle all errors in the reactive stream itself. Server response errors are usually best handled by using exchange() rather than retrieve() and then dealing with the response manually, and an underlying IOException by using the onErrorResume(), onErrorReturn() etc. reactive operators available for this purpose.
However, you mention you're migrating from blocking code, so I understand that practically that may not (yet) be on the cards. If you want to stick to catching exceptions:
But since I don't want to rewrite our internal code to catch exceptions on RuntimeException level but on WebClientException level, is that not an option and I'm wondering how to do that?
Wanting to catch all transport errors under the umbrella of WebClientException is not a sensible option. As you say, neither is just catching RuntimeException for obvious reasons.
Simplifying it, WebClientException means "I connected to the URL and sent stuff to it without an issue, but it told me to sod off" (ie. it generated an error code rather than a 200 response.)
That might be because of a 404 (resource not found), 500 (server error), 418 (you're trying to connect to a teapot, not a server), etc.
IOException on the other hand means "Couldn't even establish a connection to this URL." That could be because the connection was actively refused, the domain name couldn't be resolved, the SSL cert expired, etc.
The two are not analogous, and it would be rather odd and confusing to treat them that way.
If you want to handle them in the same block, then that's fine - naively you might just do:
catch(WebClientException|IOException e) {
// do some stuff
}
...but you can't of course, because IOException is checked. (Reactive streams in Java don't throw checked exceptions, each checked exception is mapped to a RuntimeException instead.)
However, you can map all IOException to an UncheckedIOException:
return webClient
.get()
.uri(someUri)
.retrieve()
.toEntity(SomeBusinessClass.class)
.onErrorMap(IOException.class, UncheckedIOException::new)
.block()
...and then either do catch(WebClientException|UncheckedIOException ex), or deal with them in separate catch blocks.
This certainly isn't the "nice" way to handle exceptions from a reactive mindset, but if you're aiming to migrate with the fewest possible changes, this is likely what you're after.
What is the best practice for handling exceptions in MassTransit 3+ with regard to Request/Response pattern? The docs here mention that if a ResponseAddress exists on a message, the Fault message will be sent to that address, but how does one consumer/receive the messages at that address? The ResponseAddress for Bus.Request seems to be an auto-generated MassTransit address that I don't have control over, so I don't know how to access the exception thrown in the main consumer. What am I missing? Here's my code to register the consumer and its fault consumer using Unity container:
cfg.ReceiveEndpoint(host, "request_response_queue", e =>
{
e.Consumer<IConsumer<IRequestResponse>>(container);
e.Consumer(() => container.Resolve<IMessageFaultConsumer<IRequestResponse>>() as IConsumer<Fault<IRequestResponse>>);
});
And here's my attempt at a global message fault consumer:
public interface IMessageFaultConsumer<TMessage>
{
}
public class MessageFaultConsumer<TMessage> : IConsumer<Fault<TMessage>>, IMessageFaultConsumer<TMessage>
{
public Task Consume(ConsumeContext<Fault<TMessage>> context)
{
Console.WriteLine("MessageFaultConsumer");
return Task.FromResult(0);
}
}
This approach DOES work when I use Bus.Publish as opposed to Bus.Request. I also looked into creating an IConsumeObserver and putting my global exception logging code into the ConsumeFault method, but that has the downside of being invoked every exception prior to the re-tries giving up. What is the proper way to handle exceptions for request/response?
First of all, the request/response support in MassTransit is meant to be used with the .Request() method, or the request client (MessageRequestClient or PublishRequestClient). With these methods, if the consumer of the request message throws an exception, that exception is packaged into the Fault<T>, which is sent to the ResponseAddress. Since the .Request() method, and the request client are both asynchronous, using await will throw an exception with the exception data from the fault included. That's how it is designed, await the request and it will either complete, timeout, or fault (throw an exception upon await).
If you are trying to put in some global "exception handler" code for logging purposes, you really should log those at the service boundary, and an observer is the best way to handle it. This way, you can just implement the ConsumeFault method, and log to your event sink. However, this is synchronous within the consumer pipeline, so recognize the delay that could be introduced.
The other option is to of course just consume Fault<T>, but as you mentioned, it does not get published when the request client is used with the response address in the header. In this case, perhaps your requester should publish an event indicating that operation X faulted, and you can log that -- at the business context level versus the service level.
There are many options here, it's just choosing the one that fits your use case best.
In our application we have error handling mechanism, where we throw runtime exceptions on an error. I noticed a strange behavior and I want to understand the mechanism underlying this one
1) Situation 1: Exception thrown from ServiceActivator is converted to MessageHandlingException
When an error occurs in a ServiceActivator, we throw an exception. The message we get on ErrorChannel has PayLoad as org.springframework.integration.MessageHandlingException and actual exception thrown as cause
2) Situation 2: Exception thrown from Filter is not masked with MessageHandlingException
When an error occurs in Filter, and we throw exception, then PayLoad is actual exception, and is not masked with org.springframework.integration.MessageHandlingException
I have a few questions:
Why exception throwing from ServiceActivator behaves differently than in Filter
Are there some "best practices" around error handling in Spring-integration projects, while utilizing the errorChannel and related infrastructure
Update 1:
Filter extends AbstractFileListFilter which is part of a filter chain- a custom CompositeFileFilter which implements FileListFilter
CompositeFileFilter is being used by a file:inbound-channel-adapter and which passes the output to a Channel declared below:
<int:channel
id="channelForFilesComingIn"
datatype="java.io.File"
>
<int:dispatcher task-executor="dispatchExecutor" />
</int:channel>
Update 2:
Whet we are trying to do is read files from filesystem and process them. In file reading part, using file:inbound-channel-adapter with a CompositeFilter which filters files which are not completely uploaded or don't meet naming standards.
After all filters pass, file is handed over to a ServiceActivator for processing
In any of above (Filter chain or Service) , if there is an error condition, it has to be reported to DB and by email. For achieving this we are throwing ApplicationException which are caught by errorChannel, and passed to specialized channels.
Just to make it clear, a MessageHandlingException is thrown (wraps user exception) when a Message HANDLER fails - a message handler is something that HANDLES a message.
If an exception is thrown in the MessageSource, there is no message yet so a MessageHandlingException (or any MessagingException) does not apply.
Instead, the poll fails and the exception is thrown back to the poller.
If you want to handle exceptions in a polled endpoint (MessageSource), you need to give the poller an ErrorHandlingTaskExecutor, to which you can provide an ErrorHandler and do what you want with the exception, but since there is no message yet, it is the original exception thrown by the MessageSource.
If you want to send it to the error channel, you'll need to do that in your custom ErrorHandler.