How to return response immediate to client in spring flux by controlling the no of thread using ExecutorService and CompletableFuture? - spring-boot

Question

I needed to call two downstream systems parallelly with non-blocking io from Spring flux-based my rest service API. But the first downstream system capacity is 10 requests at a time and the second downstream system is 100.
The first downstream system out is input to the second downstream system so I can make a more parallel request to the second system to expedite the process.
The second downstream system response is very large so unable to hold in memory to concrete all the response So immediate want to return the response to the client.
Ex workflow:
Sample Code:
#GetMapping(path = "/stream", produces = MediaType.APPLICATION_STREAM_JSON_VALUE)
public Flux<String> getstream() {
ExecutorService executor = Executors.newFixedThreadPool(10);
List<CompletableFuture> list = new ArrayList<>();
AtomicInteger ai = new AtomicInteger(1);
RestTemplate restTemplate = new RestTemplate();
for (int i = 0; i < 100; i++) {
CompletableFuture<Object> cff = CompletableFuture.supplyAsync(
() -> ai.getAndAdd(1) + " first downstream web service " +
restTemplate.getForObject("http://dummy.restapiexample.com/api/v1/employee/" + ai.get(), String.class)
).thenApplyAsync(v -> {
Random r = new Random();
Integer in = r.nextInt(1000);
return v + " second downstream web service " + in + " " + restTemplate.getForObject("http://dummy.restapiexample.com/api/v1/employee/" + ai.get() + 1, String.class) + " \n";
}, executor);
list.add(cff);
}
return Flux.fromStream(list.stream().map(m -> {
try {
return m.get().toString();
} catch (InterruptedException | ExecutionException e) {
e.printStackTrace();
}
return "";
})
);
}
This code only working for the first five threads after I am getting a response all threads completed the process. But I needed to get a response immediately to the client once I am getting the response from the second downstream system.
Note: The above code is not implemented with a second level thread pool.
Thank you in advance.

Answer 1

If you're building non-blocking system using Spring-Webflux it's better to utilise capabilities of WebClient in your example. I've created a simple test application where the below code snippet worked for me:
private final WebClient w = WebClient.create("http://localhost:8080/call"); // web client for external system
#GetMapping(path = "/stream", produces = MediaType.APPLICATION_STREAM_JSON_VALUE)
public Flux<MyClass> getstream() {
return Flux
.range(0, 100) // prepare initial 100 requests
.window(10) // combine elements in batch of 10 (probably buffer will fit better, have a look)
// .delayElements(Duration.ofSeconds(5)) for testing purpose you can use this function as well
.doOnNext(flow -> log.info("Batch of 10 is ready")) // double check tells that batch is ready
.flatMap(flow -> flow
// perform an external async call for each element in batch of 10
// they will be executed sequentially but there will not be any performance issues because
// calls are async. If you wish you can add .parallel() to the flow to make it parallel
.flatMap(element -> w.get().exchange())
.map(r -> r.bodyToMono(MyClass.class))
)
// subscribe to each response and throw received element further to the stream
.flatMap(response -> Mono.create(s -> response.subscribe(s::success)))
.window(1000) // batch of 1000 is ready
.flatMap(flow -> flow
.flatMap(element -> w.get().exchange())
.map(r -> r.bodyToMono(MyClass.class))
)
.flatMap(response -> Mono.create(s -> response.subscribe(s::success)));
}
public static class MyClass {
public Integer i;
}
UPDATE:
I've prepared a small application to reproduce your case. You can find it in my repository.