spring kafka using VALUE_TYPE_METHOD - spring-boot

Question

TL/DR - can someone provide a simple example of using JsonDeserializer.VALUE_TYPE_METHOD?
I have a situation that matches the documentation exactly:
Starting with version 2.5, you can now configure the deserializer, via properties, to invoke a method to determine the target type. If present, this will override any of the other techniques discussed above. This can be useful if the data is published by an application that does not use the Spring serializer and you need to deserialize to different types depending on the data, or other headers. Set these properties to the method name - a fully qualified class name followed by the method name, separated by a period .. The method must be declared as public static, have one of three signatures (String topic, byte[] data, Headers headers), (byte[] data, Headers headers) or (byte[] data) and return a Jackson JavaType.
I setup my config like so:
spring:
kafka:
consumer:
bootstrap-servers:
- https://blah.blah.blah:443
value-deserializer: "org.springframework.kafka.support.serializer.JsonDeserializer"
properties:
"spring.json.trusted.packages": "com.my.base"
"spring.json.value.type.method": "com.my.base.Application.deserializerDelegator"
my deserializerDelegator is as such:
private static final JavaType FOO_MESSAGE_TYPE = TypeFactory.defaultInstance()
.constructType(Foo.class);
private static final JavaType BAR_MESSAGE_TYPE = TypeFactory.defaultInstance()
.constructType(Bar.class);
public static JavaType deserializerDelegator(byte[] data, Headers headers) {
Header header = headers.lastHeader("eventSubType");
return Arrays.equals(header.value(), "100".getBytes())
? FOO_MESSAGE_TYPE : BAR_MESSAGE_TYPE;
}
so now that I set all that up...how do I setup my actual consumers???? Do I create multiple consumers....one for each type? I tried this and it failed:
#Slf4j
#Component
public class Consumer {
#KafkaListener(topics = {"some-topic"}, groupId = "some-group")
public void consume(Foo message) {
// handle Foo message
}
#KafkaListener(topics = {"some-topic"}, groupId = "some-group")
public void consume(Bar message) {
// handle Bar message
}
}
which resulted in this error:
org.springframework.kafka.listener.ListenerExecutionFailedException: Listener method could not be invoked with the incoming message
Endpoint handler details:
Method [public void com.example.Consumer.consume(com.example.consumer.models.Foo)]
MessageConversionException: Cannot handle message; nested exception is org.springframework.messaging.converter.MessageConversionException: Cannot convert from [Foo] to [Bar] for GenericMessage [payload=Foo(...omitted...)]

Answer 1

If messages with different types are in the same topic, you must use a class-level #KafkaListener with #KafkaHandler methods (and an optional default).
https://docs.spring.io/spring-kafka/docs/current/reference/html/#class-level-kafkalistener
When you use #KafkaListener at the class-level, you must specify #KafkaHandler at the method level. When messages are delivered, the converted message payload type is used to determine which method to call. The following example shows how to do so:
#KafkaListener(id = "multi", topics = "myTopic")
static class MultiListenerBean {
#KafkaHandler
public void listen(String foo) {
...
}
#KafkaHandler
public void listen(Integer bar) {
...
}
#KafkaHandler(isDefault = true)
public void listenDefault(Object object) {
...
}
}
Starting with version 2.1.3, you can designate a #KafkaHandler method as the default method that is invoked if there is no match on other methods. At most, one method can be so designated. When using #KafkaHandler methods, the payload must have already been converted to the domain object (so the match can be performed).
EDIT
#SpringBootApplication
public class So73953255Application {
public static void main(String[] args) {
SpringApplication.run(So73953255Application.class, args);
}
#Bean
public NewTopic topic() {
return TopicBuilder.name("so73953255").partitions(1).replicas(1).build();
}
#Bean
ApplicationRunner runner(KafkaTemplate<String, String> template) {
return args -> {
template.send("so73953255", "{\"baz\":\"qux\"}");
template.send("so73953255", "{\"baz\":\"fiz\"}");
};
}
private static final JavaType fooType = TypeFactory.defaultInstance().constructType(Foo.class);
private static final JavaType barType = TypeFactory.defaultInstance().constructType(Bar.class);
public static JavaType typer(byte[] data, Headers headers) {
return new String(data).contains("fiz") ? barType : fooType;
}
public static class Foo {
private String baz;
public String getBaz() {
return this.baz;
}
public void setBaz(String baz) {
this.baz = baz;
}
#Override
public String toString() {
return "Foo [baz=" + this.baz + "]";
}
}
public static class Bar {
private String baz;
public String getBaz() {
return this.baz;
}
public void setBaz(String baz) {
this.baz = baz;
}
#Override
public String toString() {
return "Bar [baz=" + this.baz + "]";
}
}
}
#Component
#KafkaListener(id = "so73953255", topics = "so73953255")
class Listener {
#KafkaHandler
void fooListener(So73953255Application.Foo foo) {
System.out.println("Foo Handler: " + foo);
}
#KafkaHandler
void barListener(So73953255Application.Bar bar) {
System.out.println("Bar Handler: " + bar);
}
}
spring.kafka.consumer.value-deserializer=org.springframework.kafka.support.serializer.JsonDeserializer
spring.kafka.consumer.properties.spring.json.value.type.method=com.example.demo.So73953255Application.typer
spring.kafka.consumer.properties.spring.json.trusted.packages=*
spring.kafka.consumer.auto-offset-reset=earliest
Foo Handler: Foo [baz=qux]
Bar Handler: Bar [baz=fiz]