How do I access header information in kafka materialized views? - spring

Question

how do I access custom headers in a materialized view? I'm trying to build some custom dlq logic in my application and want to build a retry mechanism based on header information. The actual retry is triggered by a scheduler which should look up these header information in a materialized view.
Here are some code snippets:
Create Materialized View:
#Slf4j
#EnableBinding(DlqBinding.class)
public class DlqRetryService {
#StreamListener
public void readTable(#Input(DlqBinding.DLQ_TOPIC) KTable<String, String> table) {
}
}
public interface DlqBinding {
String DLQ_TOPIC = "dlq";
#Input(DLQ_TOPIC)
KTable<?, ?> dlqInput();
}
spring:
cloud:
stream:
kafka:
streams:
binder:
brokers: localhost:29092
configuration:
default:
key.serde: org.apache.kafka.common.serialization.Serdes$StringSerde
value.serde: org.apache.kafka.common.serialization.Serdes$StringSerde
bindings:
dlq:
consumer:
materializedAs: currentDL
keySerde: org.apache.kafka.common.serialization.Serdes$StringSerde
valueSerde: org.apache.kafka.common.serialization.Serdes$StringSerde
Scheduler:
public void processDL() {
ReadOnlyKeyValueStore<Object, Object> currentDL = interactiveQueryService.getQueryableStore("currentDL", QueryableStoreTypes.keyValueStore());
KeyValueIterator<Object, Object> all = currentDL.all();
while (all.hasNext()) {
KeyValue<Object, Object> next = all.next();
log.info("Found Entry in currentDL: {}", next);
// some retry logic would be here
}
}```

Answer 1

I don't think it is possible to access headers that way from the materialized view using interactive query.
Its not clear what you are retrying. Are you trying to re-process the records?
You can encapsulate this kind of logic by using a KStream and the transformer/processor API. Here is a blueprint of such a pattern. Keep in mind that this is a general way of accessing headers. Your application might need to adjust based on your particular use case.
#StreamListener
public void processStream(#Input(DlqBinding.DLQ_TOPIC) KStream<String, String> stream) {
stream.process(() -> new Processor() {
ProcessorContext context;
#Override
public void init(ProcessorContext context) {
this.context = context;
}
#Override
public void process(Object key, Object value) {
final Headers headers = this.context.headers();
final Iterable<Header> headerXyz = headers
.headers("HEADER_KEY");
// iterate on the header information returned and perform
// your application specific logic here.
}
#Override
public void close() {
}
});
}