How to distribute workload to many compute and do scatter-gather scenarios with Kafka Steam? - apache-kafka-streams

Question

I am new to Kafka Stream and Alpakka Kafka.
Problem: I have been using Java Executor Service to run parallel jobs and when ALL of them are done, marking the entire process done. The issue is fault tolerance, High Availability and Not utilizing all computes to do the work. It is using just ONE HOST JVM to do work.
We have Apache Kafka as infrastructure, so I was wondering how I can use Kafka Stream to do scatter-gather or just execute child task use case implemented to distribute workload and then gather results or get an indication that all tasks are done.
Any pointer to sample work or scatter-gather or Fork join would be great with Kafka Steam or Alpakka Kafka.
Here is a Sample:
import org.springframework.http.MediaType;
import org.springframework.web.reactive.function.client.WebClient;
import java.util.LinkedList;
import java.util.List;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
public class Main {
private static final ExecutorService executorService = Executors.newFixedThreadPool(15);
public static void main(String[] args) throws Exception {
final WebClient webClient = WebClient.builder().build();
List<CompletableFuture<String>> allTasks = new LinkedList<>();
String urls[] = {"http://test1", "http://test2", "http://test3"};
// Distribute the work ( webcient can do async but I wanted to just give example).
for (final String url : urls) {
CompletableFuture<String> task = CompletableFuture.supplyAsync(() -> {
// SOME Task JUST FOR Example I have put GET call it could be any thing
String response =
webClient.get().uri(url).accept(MediaType.APPLICATION_JSON).retrieve().bodyToMono(String.class).block();
return response;
}, executorService);
allTasks.add(task);
}
// wait for all do be done (Join)
CompletableFuture.allOf(allTasks.toArray(new CompletableFuture[]{})).join();
for(CompletableFuture<String> task: allTasks){
processResponse(task.get());
}
}
public static void processResponse(String response){
System.out.println(response);
}
}