Springboot. Reactive webclient. Connection prematurely closed BEFORE response - spring

Question

I've come across with this problem
Pooled connection observed an error
reactor.netty.http.client.HttpClientOperations$PrematureCloseException:
Connection prematurely closed BEFORE response".
I'm gathering metrics from the graphite server via reactive web-client for the requested timeframes (to reduce the amount of data that transfers via http I've divided days into chunks 24/4), then combine responses into a matrix and save it to csv file -> merge to another one.
The problem appears when the number of days increases (2 or 3 works fine, but more days will be more errors with closed connections happened). Tried to use delays, it helps a bit, but to process one more day without errors.
Stack-trace:
ClosedConnectionStacktrace
Found a bit similar issue https://github.com/reactor/reactor-netty/issues/413 , but not sure.
Here's the code snippets:
discoveryMono.thenReturn(true) // discover metrics
.flux()
.flatMap(m -> Flux.fromIterable(dates) // process all days
.delayElements(Duration.ofSeconds(1L))
.flatMap(date -> Flux.range(0, 24 / intervalHours) // divide day into chunks
.delayElements(Duration.of(100L, ChronoUnit.MILLIS))
.flatMap(timeFraction -> Flux.fromIterable(sequentialTasks) // task to invoke webclient
.flatMap(task -> {
Instant from = date.plus(timeFraction * intervalHours, ChronoUnit.HOURS);
Instant until = from.plus(intervalHours, ChronoUnit.HOURS);
TaskParams taskParams = new TaskParams(itSystem, from, until, TaskParams.PollingType.FULLDAY);
log.trace("workflow | from={}, until={}", from, until);
return task.apply(taskParams)
// .doOnNext(m -> log.trace("Matrix: {}", m))
.onErrorResume(err -> {
log.error("processFullDaysInChunks | Error: {}", err);
return Mono.empty();
});
}).flatMap(params -> Flux.fromIterable(fileTasks) // tasks to check/merge files, doesn't matter
.flatMap(fileTask -> parTask.apply(params)
.onErrorResume(err -> {
log.error("processFullDaysInChunks | Error: {}", err);
return Mono.empty();
})
)
)
)
)
).subscribeOn(fullDayScheduler).subscribe();
and part of the task with webclient invokation:
private Flux<GraphiteResultDTO> getGraphiteResults(ITSystem itSystem, Instant from, Instant until) {
String fromStr = FROM_PARAMETER + Long.valueOf(from.getEpochSecond()).toString();
String untilStr = UNTIL_PARAMETER + Long.valueOf(until.getEpochSecond()).toString();
String uri = RENDER_URI + TARGET_PARAMETER + "{targetParam}" + fromStr + untilStr + FORMAT_JSON_PARAMETER;
WebClient webClient = getGraphiteWebClient(itSystem.getDataSource());
Set<String> targetParams = storage.getValueByITSystemId(itSystem.getId()).getSecond();
Flux<GraphiteResultDTO> result = Flux.fromIterable(targetParams)
.delayElements(Duration.of(10, ChronoUnit.MILLIS))
.flatMap(targetParam -> {
Map<String, String> params = Map.ofEntries(entry("targetParam", targetParam));
if (log.isTraceEnabled()) {
log.trace("getGraphiteResults | Uri={}, TargetPatam: {}", uri, targetParam);
}
return webClient.get()
.uri(uri, params)
.retrieve()
.onStatus(HttpStatus::isError, clientResponse -> {
log.trace("clientResponse | transforming body");
clientResponse.bodyToMono(String.class)
.doOnNext(errorString -> log.error("retrieve(), error={}", errorString));
// .flatMap(s -> Flux.error(clientResponse.bodyToFlux(WebClientException.class)));
return Mono.empty();
})
.bodyToFlux(GraphiteResultDTO.class)
.onErrorResume(throwable -> {
log.error("webclient | bodyToFlux error={}", throwable.getMessage());
return Flux.empty();
});
});
return result;
}

Answer 1

Resolved my problem with replacing the flatMap operator with concatMap with prefetch 1 and limiting the rate (limitRate operator). All the requests now process one by one sequentially. So there is no need to use time delays now.