I have the below list of CompletableFuture pipeline
for (Integer unit : getIds()) {
futureList.add(CompletableFuture.completedFuture(unit)
.thenApply(id -> CompletableFuture
.completedFuture(service.get1(id))
.thenCombineAsync(CompletableFuture.completedFuture(b), (a, df1) ->
CompletableFuture.completedFuture(service.validate(Optional.ofNullable(a),c,
b, dealerCode))
.thenApplyAsync(status -> CompletableFuture.completedFuture(b))
.thenCombineAsync(CompletableFuture.completedFuture(a), (b1, a1) ->
service.mapToTrans(Optional.ofNullable(a), b, c))
.handle((response, exception) -> {
if(exception == null) {
return response;
}
else {
handler.handleException(exception, results);
return null;
}
})
.thenApplyAsync(t -> service.submitRequest(Optional.ofNullable(a), c))
.thenApplyAsync((t) -> service.saveTransaction(Optional.ofNullable(t)))
.thenAccept(result1 -> results.add(result1))
)
.handle((response, exception) -> handleStage(response, exception, results))
)
.handle((response, exception) -> handleStage(response, exception, results))
);
}
The results array holds the result of all the futures. In the main method this is how I am waiting for all the futures to get completed
CompletableFuture<Void> allFutures = CompletableFuture
.allOf(futureList.toArray(new CompletableFuture[futureList.size()]));
CompletableFuture<List<Object>> allFutureExec = allFutures.thenApply(v -> {
return futureList.stream().map(pg-> pg.join()).collect(Collectors.toList());
});
allFutureExec.get();
The "results" list is supposed to contain the result of all future chain But I can see that main thread is not waiting for all the stages to get executed. The result list has the result of only few futures, though I can see from the logs that all the futures are getting executed. Please let me know if this is the correct way to wait for all the futures to get completed.
UPDATE-1
for (Integer unit : getIds()) {
futureList.add(CompletableFuture
.completedFuture(service.get1(unit))
.thenCombine(CompletableFuture.completedFuture(td), (vd, dealerInfo1) ->
CompletableFuture.completedFuture(service.validate(Optional.ofNullable(vd),sd,
td))
.thenApply(status -> CompletableFuture.completedFuture(service.mapToTrans
(Optional.ofNullable(vd), td,trade, userCredential))
.thenApplyAsync(transaction -> CompletableFuture.completedFuture(service.submit(Optional.ofNullable(vd),transaction))
.thenApply((transaction1) -> service.saveTransaction(Optional.ofNullable(transaction1)))
.thenApply(result1 -> results.add(result1)))))
);
}
Related
I want to render an object composed of two mono or flux elements (below a code snippet):
Mono<List<NodeDTO>> nodeDTOFlux = this.webClient
.get()
.uri(NODES_WITH_LIMIT + limit)
.retrieve()
.onStatus(HttpStatus::isError,
response -> response.bodyToMono(String.class).flatMap(
msg -> Mono.error(new ApiCallException(msg, response.statusCode())))
)
.bodyToFlux(new ParameterizedTypeReference<Node>() {
}).map(node -> nodeMapper.toNodeDTO(node))
.collectList();
Mono<List<EdgeDTO>> edgeDTOFlux = this.webClient
.get()
.uri(EDGES_WITH_LIMIT + limit)
.retrieve()
.onStatus(HttpStatus::isError,
response -> response.bodyToMono(String.class).flatMap(
msg -> Mono.error(new ApiCallException(msg, response.statusCode())))
)
.bodyToFlux(new ParameterizedTypeReference<Edge>() {
}).map(edge -> edgeMapper.toEdgeDTO(edge))
.collectList();
I tried with zip() method but it's not what I aim to do
I tried to return an object like this
GraphDataDTO graphDataDTO = new GraphDataDTO();
graphDataDTO.setEdgeDTOS(edgeDTOFlux);
graphDataDTO.setNodeDTOS(nodeDTOFlux);
I have a result in my console but the object returned
{
"nodeDTOS": {
"scanAvailable": true
},
"edgeDTOS": {
"scanAvailable": true
}
}
the return is done before getting all the flux.. is there any solution without blocking !
thanks in advance.
This should work:
return Mono.zip(nodeDTOFlux, edgeDTOFlux)
.map(tuple2 -> GraphDataDTO.builder().nodeDTO(tuple2.getT1()).edgeDTO(tuple2.getT2()).build())
It creates a Tuple of NodeDTO and EdgeDTO and maps it into GraphDataDTO.
Have you any ideas that can help me simplify this stream?
The main challenge here is to check in a reactive way if consumers exist on a queue and then if there are no consumers check if there is at least one message on any of the queues.
Mono<String> resumeFullSync(FullSyncContext fullSyncContext,
Function<FullSyncContext, Mono<Void>> finalizeFullSyncCallback) {
var fullSyncSpec = new FullSyncSpecification(fullSyncProperties, fullSyncContext);
return sender.declare(fullSyncSpec.getQueueSpecification())
.flatMap(declareOk -> {
if (declareOk.getConsumerCount() == 0) {
log.debug("Queue has no consumers ({})", fullSyncContext.getCategoryName());
return Flux.concat(Mono.just(declareOk),
sender.declare(fullSyncSpec.getRetry1QueueSpecification()),
sender.declare(fullSyncSpec.getRetry2QueueSpecification()),
sender.declare(fullSyncSpec.getRetry3QueueSpecification()))
.any(d -> d.getMessageCount() > 0)
.flatMap(messagesExist -> {
if (messagesExist) {
log.debug("Queues have some messages ({})", fullSyncContext.getCategoryName());
return sender.declare(fullSyncSpec.getExchangeSpecification())
.then(sender.bind(fullSyncSpec.getBindingSpecification()))
.then(sender.bind(fullSyncSpec.getRetry1BindingSpecification()))
.then(sender.bind(fullSyncSpec.getRetry2BindingSpecification()))
.then(sender.bind(fullSyncSpec.getRetry3BindingSpecification()))
.doOnNext(b -> setupFullSyncConsumer(fullSyncSpec, finalizeFullSyncCallback));
} else {
log.debug("Queues have no messages ({}). Stopping", fullSyncContext.getCategoryName());
return Mono.empty();
}
});
} else {
log.debug("Queue has consumers ({}). Stopping", fullSyncContext.getCategoryName());
return Mono.empty();
}
}
)
.thenReturn(fullSyncContext.getCategoryName());
}
I simplified it myself and now it looks better.
Mono<String> resumeFullSync(FullSyncContext fullSyncContext,
Function<FullSyncContext, Mono<Void>> finalizeFullSyncCallback) {
var fullSyncSpec = new FullSyncSpecification(fullSyncProperties, fullSyncContext);
return sender.declare(fullSyncSpec.getQueueSpecification())
.filter(declareOk -> declareOk.getConsumerCount() == 0)
.flatMapMany(declareOk -> Flux.concat(Mono.just(declareOk),
sender.declare(fullSyncSpec.getRetry1QueueSpecification()),
sender.declare(fullSyncSpec.getRetry2QueueSpecification()),
sender.declare(fullSyncSpec.getRetry3QueueSpecification())))
.any(declareOk -> declareOk.getMessageCount() > 0)
.filter(messagesExist -> messagesExist)
.flatMap(ignore -> {
log.debug("Queues have some messages ({})", fullSyncContext.getCategoryName());
return sender.declare(fullSyncSpec.getExchangeSpecification())
.then(sender.bind(fullSyncSpec.getBindingSpecification()))
.then(sender.bind(fullSyncSpec.getRetry1BindingSpecification()))
.then(sender.bind(fullSyncSpec.getRetry2BindingSpecification()))
.then(sender.bind(fullSyncSpec.getRetry3BindingSpecification()))
.doOnNext(b -> setupFullSyncConsumer(fullSyncSpec, finalizeFullSyncCallback));
})
.thenReturn(fullSyncContext.getCategoryName());
}
I was wondering if someone could eyeball the following code snippet and tell me why the SubscriberContext inside the doOnError is not trigerred
public Mono<ServerResponse> handlePlatformAuthenticationResponse(final ServerRequest serverRequest) {
Mono<MultiValueMap<String, String>> formData = serverRequest.body(BodyExtractors.toFormData());
return formData
.flatMap(this::provisionUserAndClass)
.flatMap(tuple -> Mono.subscriberContext()
.map(context -> {
// this is invoked if provisionUserAndClass completes successfully
TelemetryData telemetryData = context.get(TELEMETRY_DATA);
LTILaunchRequest<LTILaunchRequestSettings> launchRequest = tuple.getT2();
this.addLaunchDetailsToTelemetryContext(launchRequest, telemetryData);
return tuple;
}))
.doOnError(error -> Mono.subscriberContext()
.map(context -> {
// this is never invoked if provisionUserAndClass returns a Mono.error
TelemetryData telemetryData = context.get(TELEMETRY_DATA);
// log telemetryData + error message
}))
.subscriberContext(context -> context.put(TELEMETRY_DATA, new TelemetryData()));
}
private Mono<Tuple2<ClassAndUserProvisioningResponse, LTILaunchRequest<LTILaunchRequestSettings>>> provisionUserAndClass(
LTILaunchRequest<LTILaunchRequestSettings> ltiLaunchRequest) {
// returning a Mono.error just to see behavior of Mono.subscriberContext() when error occurs. Actual code will call a service method
return Mono.error(new ProvisioningException("fake"));
}
To access context in case of error you could use doOnEach operator:
.doOnEach(signal -> {
if (signal.isOnError())
{
TelemetryData telemetryData = signal.getContext().get(TELEMETRY_DATA);
Throwable error = signal.getThrowable();
// ...
}
})
Mono.subscriberContext() can only be used meaningfully in operators where you have to return a Mono, like flatMap, concatMap, etc, but not in side-effect operators where there is nothing that would subscribe to the Mono<Context>.
.doOnError(error -> Mono.subscriberContext()
.map(context -> {
// this is never invoked if provisionUserAndClass returns a Mono.error
TelemetryData telemetryData = context.get(TELEMETRY_DATA);
// log telemetryData + error message
}).subscribe())
You forgot to subscribe to the Mono.subscriberContext().
I've come across with this problem
Pooled connection observed an error
reactor.netty.http.client.HttpClientOperations$PrematureCloseException:
Connection prematurely closed BEFORE response".
I'm gathering metrics from the graphite server via reactive web-client for the requested timeframes (to reduce the amount of data that transfers via http I've divided days into chunks 24/4), then combine responses into a matrix and save it to csv file -> merge to another one.
The problem appears when the number of days increases (2 or 3 works fine, but more days will be more errors with closed connections happened). Tried to use delays, it helps a bit, but to process one more day without errors.
Stack-trace:
ClosedConnectionStacktrace
Found a bit similar issue https://github.com/reactor/reactor-netty/issues/413 , but not sure.
Here's the code snippets:
discoveryMono.thenReturn(true) // discover metrics
.flux()
.flatMap(m -> Flux.fromIterable(dates) // process all days
.delayElements(Duration.ofSeconds(1L))
.flatMap(date -> Flux.range(0, 24 / intervalHours) // divide day into chunks
.delayElements(Duration.of(100L, ChronoUnit.MILLIS))
.flatMap(timeFraction -> Flux.fromIterable(sequentialTasks) // task to invoke webclient
.flatMap(task -> {
Instant from = date.plus(timeFraction * intervalHours, ChronoUnit.HOURS);
Instant until = from.plus(intervalHours, ChronoUnit.HOURS);
TaskParams taskParams = new TaskParams(itSystem, from, until, TaskParams.PollingType.FULLDAY);
log.trace("workflow | from={}, until={}", from, until);
return task.apply(taskParams)
// .doOnNext(m -> log.trace("Matrix: {}", m))
.onErrorResume(err -> {
log.error("processFullDaysInChunks | Error: {}", err);
return Mono.empty();
});
}).flatMap(params -> Flux.fromIterable(fileTasks) // tasks to check/merge files, doesn't matter
.flatMap(fileTask -> parTask.apply(params)
.onErrorResume(err -> {
log.error("processFullDaysInChunks | Error: {}", err);
return Mono.empty();
})
)
)
)
)
).subscribeOn(fullDayScheduler).subscribe();
and part of the task with webclient invokation:
private Flux<GraphiteResultDTO> getGraphiteResults(ITSystem itSystem, Instant from, Instant until) {
String fromStr = FROM_PARAMETER + Long.valueOf(from.getEpochSecond()).toString();
String untilStr = UNTIL_PARAMETER + Long.valueOf(until.getEpochSecond()).toString();
String uri = RENDER_URI + TARGET_PARAMETER + "{targetParam}" + fromStr + untilStr + FORMAT_JSON_PARAMETER;
WebClient webClient = getGraphiteWebClient(itSystem.getDataSource());
Set<String> targetParams = storage.getValueByITSystemId(itSystem.getId()).getSecond();
Flux<GraphiteResultDTO> result = Flux.fromIterable(targetParams)
.delayElements(Duration.of(10, ChronoUnit.MILLIS))
.flatMap(targetParam -> {
Map<String, String> params = Map.ofEntries(entry("targetParam", targetParam));
if (log.isTraceEnabled()) {
log.trace("getGraphiteResults | Uri={}, TargetPatam: {}", uri, targetParam);
}
return webClient.get()
.uri(uri, params)
.retrieve()
.onStatus(HttpStatus::isError, clientResponse -> {
log.trace("clientResponse | transforming body");
clientResponse.bodyToMono(String.class)
.doOnNext(errorString -> log.error("retrieve(), error={}", errorString));
// .flatMap(s -> Flux.error(clientResponse.bodyToFlux(WebClientException.class)));
return Mono.empty();
})
.bodyToFlux(GraphiteResultDTO.class)
.onErrorResume(throwable -> {
log.error("webclient | bodyToFlux error={}", throwable.getMessage());
return Flux.empty();
});
});
return result;
}
Resolved my problem with replacing the flatMap operator with concatMap with prefetch 1 and limiting the rate (limitRate operator). All the requests now process one by one sequentially. So there is no need to use time delays now.
How can I Use Collectors to collect in a ConcurrentHashMap instread of putting manually into ConcurrentHashMap
ConcurrentHashMap<String, String> configurationMap = new ConcurrentHashMap<>();
List<Result> results = result.getResults();
results.stream().forEach(res -> {
res.getSeries().stream().forEach(series -> {
series.getValues().stream().forEach(vals ->{
configurationMap.put(vals.get(1).toString(),vals.get(2).toString());
});
});
});
//Note: vals is List<List<Object>> type
Help will be appreciated.
You can use Collectors.toConcurrentMap
results.stream()
.flatMap(res -> res.getSeries().stream())
.flatMap(series -> series.getValues().stream())
.collect(Collectors.toConcurrentMap(
vals -> vals.get(1).toString(),
vals -> vals.get(2).toString()));
We can do this as follows also:
results.stream()
.flatMap(res -> res.getSeries().stream())
.flatMap(series -> series.getValues().stream())
.collect(Collectors.toMap(
vals -> vals.get(1).toString(),
vals -> vals.get(2).toString(),
(vals1,vals2) -> vals2,
ConcurrentHashMap::new);