Kinesis as producer in Spring Boot Reactive Stream API - spring-boot

Question

I'm trying to build a small Spring Boot Reactive API. The API should let the users subscribe to some data, returned as SSE.
The data is located on a Kinesis Topic.
Creating the Reactive API, and the StreamListener to Kinesis is fairly easy - but can I combine these, so the Kinesis Topic are used as a producer for the event stream used by my data service.
The code looks more or less like this
//Kinesis binding, with listenerMode: rawRecords
#EnableBinding(Sink.class)
public class KinesisStreamListener {
#StreamListener(value = Sink.INPUT)
public void logger(List<Record> payload) throws Exception {
}
}
#RestController
#RequestMapping("/data")
public class DataResource {
#Autowired
DataService service;
#GetMapping(produces = {MediaType.TEXT_EVENT_STREAM_VALUE, MediaType.APPLICATION_STREAM_JSON_VALUE})
public Flux<EventObject> getData() {
return service.getData();
}
}
#Component
public class DataService {
Flux<EventObject> getData() {
Flux<Long> interval = Flux.interval(Duration.ofMillis(1000));
Flux<EventObject> dataFlux = Flux.fromStream(Stream.generate(() -> ???
));
return dataFlux.zip(interval, dataFlux).map(Tuple2::getT2);
}
}

Answer 1

Here is a sample how I would do that: https://github.com/artembilan/sandbox/tree/master/cloud-stream-kinesis-to-webflux.
Once we agree about details and some improvements it can go to the official Spring Cloud Stream Samples repository: https://github.com/spring-cloud/spring-cloud-stream-samples
The main idea is to reuse the same Flux provided by the #StreamListener via Spring Cloud Stream Reactive Support. This is is already a FluxPublish, so any new SSE connections will work as a plain Reactive subscribers.
There are a couple tricks to count with:
For the listenerMode: rawRecords, we also need to configure a contentType: application/octet-stream to avoid any conversion attempts when Binder sends a message to the Sink.INPUT channel.
Since listenerMode: rawRecords returns a List<Record> our Flux in the #StreamListener method should expect exactly this type, but not a plain Record.
Both concerns are considered as a Framework improvements.
So, let us now how it looks and works for you.