Multiple spring Kafka stream processors in one spring boot application - spring-boot

Question

I have a requirement to write 2 Spring Kafka Stream processors in one spring-boot application. Both stream processors need to consume messages from a single topic and produce the output to another single topic.
The first processor does not process all messages coming to the input topic, it processes some selected messages (group by message key) and uses windowedBy option instead.
The second processor, however, needs to process all messages which do not fall into the first processor groupBy logic.
I started with the first processor which works fine. The problem came when I try to introduce the second processor.
KafkaProcessor.java:
#Bean
public Function<KStream<String, Input>, KStream<String, Output>> processorOne() {
final AtomicReference<KeyValue<String, Output>> result = new AtomicReference<>(null);
return kStream -> kStream
.filter((key, value) -> value != null)
.filter(this::isValidKey)
.peek((key, value) -> print(value))
.groupBy((key, value) -> key)
.windowedBy(TimeWindows.ofSizeWithNoGrace(Duration.ofMinutes(5)))
.count(Materialized.as("my-state-store"))
.filter((windowedKey, count) -> count > 0)
.toStream()
.filter((messageKey, messageValue) -> {
log.info("Key: {} count: {}", messageKey.key(), messageValue);
Optional<Output> outputResult = service.process(messageKey.key());
if (outputResult.isPresent()) {
Output output = outputResult.get();
print(output);
result.set(KeyValue.pair(messageKey.key(), output));
return true;
}
return false;
})
.map((messageKey, messageValue) -> result.get());
}
Then I added, a second processor to the same class.
#Bean
public Function<KStream<String, Input>, KStream<String, Output>> processorTwo() {
final AtomicReference<KeyValue<String, Output>> result = new AtomicReference<>(null);
return kStream -> kStream
.filter((key, value) -> value != null)
.filter((key, value) -> isValidEvent(value))
.peek((key, value) -> print(value))
.filter((messageKey, messageValue) -> {
Optional<Output> outputResult = service.process(messageValue);
if (outputResult.isPresent()) {
Output output = outputResult.get();
print(output);
result.set(KeyValue.pair(String.format("%s-%s", output.getNameOne(), output.getNametwo()), output));
return true;
}
return false;
})
.map((messageKey, messageValue) -> result.get());
}
application.yaml
spring:
application:
name: my-common-processor
cloud:
stream:
function:
definition: processorOne; processorTwo
bindings:
processorOne-in-0:
destination: input-topic
group: ${spring.application.name}-processorOne
processorOne-out-0:
destination: output-topic
group: ${spring.application.name}-processorOne
processorTwo-in-0:
destination: input-topic
group: ${spring.application.name}-processorTwo
processorTwo-out-0:
destination: output-topic
group: ${spring.application.name}-processorTwo
kafka:
binder:
brokers: 127.0.0.1:9092
auto-create-topics: false
auto-add-partitions: false
streams:
binder:
configuration:
spring.json.use.type.headers: false
spring.json.trusted.packages: '*'
max.poll.records: 10
max.block.ms: 5000
default.key.serde: org.apache.kafka.common.serialization.Serdes$StringSerde
default.value.serde: org.apache.kafka.common.serialization.Serdes$StringSerde
deserialization-exception-handler: sendtodlq
auto-create-topics: false
auto-add-partitions: false
functions:
processorOne:
application-id: ${spring.application.name}-processorOne
processorTwo:
application-id: ${spring.application.name}-processorTwo
bindings:
processorOne-in-0:
consumer:
dlqName: error-topic
processorTwo-in-0:
cosumer:
dlqName: error-topic
kafka:
bootstrap-servers: 127.0.0.1:9092 // for kafka admin
Questions:
Is this the correct way to do this?
With this setup, the second processor never processes any messages even the messages which should have been processed by it.
If the setup is correct, how do I add/configure a deserialization-exception-handler for each processor?

Answer 1

This is where low level processor API implementation comes into picture. Define your project in this way.
Write a filtering processor that extends an abstract processor to filter the event from input topic(Hope your event will have a field which describes EventType1 or EventType2). Use the context forwarder inside the override process method. I would call this as EventFilteringProcessor.
if (EventType1)
context.forward(key, value, To.child(EventType1));
if (EventType2)
context.forward(key, value, To.child(EventType2));
Write two separate processor instances i.e. two separate classes that would extends AbstractProcessor. I would call these classes as EventType1Processor and EventType2Processor.
Describe your stream processor topology in the following way, in the main Spring boot application that implements the ProcessController.
Topology topology = new Topology();
topology.addSource("Source", "YourInputTopic)
.addProcessor("EventFilterClass", () -> EventFilteringProcessor.class, "Source")
.addProcessor(EventType1, () -> EventType1Processor.class, "EventFilterClass")
.addProcessor(EventType2, () -> EventType2Processor.class, "EventFilterClass");
final KafkaStreams streams = new KafkaStreams(topology, streamConfig);
streams.start();
You can have your respective business logic in the respective override process methods of EventType1Processor and EventType2Processor.
Hope this helps.