Spring Boot Auto Configuration Failed Loading Spring Kafka Properties - spring-boot

Spring boot failed to load properties. Here are the properties that i am using through the yaml file.
spring:
kafka:
bootstrap-servers: localhost:9092
consumer:
auto-commit-interval: 100
enable-auto-commit: true
group-id: ********************
auto-offset-reset: earliest
value-deserializer: org.springframework.kafka.support.serializer.JsonDeserializer
producer:
batch-size: 16384
buffer-memory: 33554432
retries: 0
value-serializer: org.springframework.kafka.support.serializer.JsonSerializer
listener:
poll-timeout: 20000
The exception i am getting is this
Caused by: java.lang.IllegalAccessException: Class org.apache.kafka.common.utils.Utils can not access a member of class org.springframework.kafka.support.serializer.JsonDeserializer with modifiers "protected"
I think the constructor is protected. Please provide a way to instantiate this.

That's correct. See:
protected JsonDeserializer() {
this((Class<T>) null);
}
protected JsonDeserializer(ObjectMapper objectMapper) {
this(null, objectMapper);
}
public JsonDeserializer(Class<T> targetType) {
this(targetType, new ObjectMapper());
this.objectMapper.configure(MapperFeature.DEFAULT_VIEW_INCLUSION, false);
this.objectMapper.configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false);
}
The JsonDeserializer isn't designed to be instantiated by the default constructor because it needs to know the targetType to deserialize.
You can extend this class to your particular type:
public class FooJsonDeserializer extends JsonDeserializer<Foo> { }
and use already this as class value for that value-deserializer property.
Or you can consider to customize the DefaultKafkaConsumerFactory:
#Bean
public ConsumerFactory<?, ?> kafkaConsumerFactory(KafkaProperties properties) {
Map<String, Object> consumerProperties = properties.buildConsumerProperties();
consumerProperties.put(CommonClientConfigs.METRIC_REPORTER_CLASSES_CONFIG,
MyConsumerMetricsReporter.class);
DefaultKafkaConsumerFactory<Object, Object> consumerFactory =
new DefaultKafkaConsumerFactory<>(consumerProperties);
consumerFactory.setValueDeserializer(new JsonDeserializer<>(Foo.class));
return consumerFactory;
}

Related

Producer Kafka throws deserialization exception

I have one topic and Producer/Consumer:
Dependencies (Spring Initializr)
Producer (apache kafka)
Consumer (apache kafka stream, cloud stream)
Producer:
KafkaProducerConfig
#Configuration
public class KafkaProducerConfig {
#Bean
public KafkaTemplate<String, Person> kafkaTemplate(){
return new KafkaTemplate<>(producerFactory());
}
#Bean
public ProducerFactory<String, Person> producerFactory(){
Map<String, Object> configs = new HashMap<>();
configs.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "127.0.0.1:9092");
configs.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
configs.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, JsonSerializer.class);
return new DefaultKafkaProducerFactory<>(configs);
}
}
Controller:
#RestController
public class KafkaProducerApplication {
private KafkaTemplate<String, Person> kafkaTemplate;
public KafkaProducerApplication(KafkaTemplate<String, Person> kafkaTemplate) {
this.kafkaTemplate = kafkaTemplate;
}
#GetMapping("/persons")
public Mono<List<Person>> findAll(){
var personList = Mono.just(Arrays.asList(new Person("Name1", 15),
new Person("Name2", 10)));
personList.subscribe(dataList -> kafkaTemplate.send("topic_test_spring", dataList.get(0)));
return personList;
}
}
It works correctly when accessing the endpoint and does not throw any exception in the IntelliJ console.
Consumer:
spring:
cloud:
stream:
function:
definition: personService
bindings:
personService-in-0:
destination: topic_test_spring
kafka:
bindings:
personService-in-0:
consumer:
configuration:
value:
deserializer: org.springframework.kafka.support.serializer.JsonDeserializer
binders:
brokers:
- localhost:9091
- localhost:9092
kafka:
consumer:
properties:
spring:
json:
trusted:
packages: "*"
PersonKafkaConsumer
#Configuration
public class PersonKafkaConsumer {
#Bean
public Consumer<KStream<String, Person>> personService(){
return kstream -> kstream.foreach((key, person) -> {
System.out.println(person.getName());
});
}
}
Here I get the exception when run the project.
org.apache.kafka.streams.errors.StreamsException: Deserialization exception handler is set to fail upon a deserialization error. If you would rather have the streaming pipeline continue after a deserialization error, please set the default.deserialization.exception.handler appropriately. Caused by: java.lang.IllegalArgumentException: The class 'com.example.producer.model.Person' is not in the trusted packages: [java.util, java.lang, com.nttdata.bootcamp.yanki.model, com.nttdata.bootcamp.yanki.model.]. If you believe this class is safe to deserialize, please provide its name. If the serialization is only done by a trusted source, you can also enable trust all (). org.apache.kafka.streams.errors.StreamsException: Deserialization exception handler is set to fail upon a deserialization error. If you would rather have the streaming pipeline continue after a deserialization error, please set the default.deserialization.exception.handler appropriately.
The package indicated in the exception refers to the entity's package but in the producer. The producer's properties file has no configuration.

Spring Kafka 2.6.x ErrorHandler and DeadLetterPublishingRecoverer with ConcurrentKafkaListenerContainerFactory

We are trying to use the DLT feature in Spring Kafka 2.6.x. This is the config yml:
kafka:
bootstrap-servers: localhost:9092
auto-offset-reset: earliest
consumer:
key-deserializer: org.springframework.kafka.support.serializer.ErrorHandlingDeserializer
value-deserializer: org.springframework.kafka.support.serializer.ErrorHandlingDeserializer
enable-auto-commit: false
properties:
isolation.level: read_committed
fetch.max.wait.ms: 100
spring.json.value.default.type: 'com.sample.entity.Event'
spring.json.trusted.packages: 'com.sample.entity.*'
spring.deserializer.key.delegate.class: org.apache.kafka.common.serialization.StringDeserializer
spring.deserializer.value.delegate.class: org.springframework.kafka.support.serializer.JsonDeserializer
producer:
bootstrap-servers: localhost:9092
key-serializer: org.apache.kafka.common.serialization.StringSerializer
value-serializer: org.springframework.kafka.support.serializer.JsonDeserializer
And here is the KafkaConfig class:
#EnableKafka
#Configuration
#Log4j2
public class KafkaConfig {
#Bean
public ConcurrentKafkaListenerContainerFactory<String, Event>
kafkaListenerContainerFactory(ConcurrentKafkaListenerContainerFactoryConfigurer configurer,
ConsumerFactory<Object, Object> consumerFactory) {
ConcurrentKafkaListenerContainerFactory<String, Event> factory =
new ConcurrentKafkaListenerContainerFactory<>();
factory.setConsumerFactory(consumerFactory);
return factory;
}
#Bean
public SeekToCurrentErrorHandler errorHandler(DeadLetterPublishingRecoverer deadLetterPublishingRecoverer) {
SeekToCurrentErrorHandler handler = new SeekToCurrentErrorHandler(deadLetterPublishingRecoverer);
handler.addNotRetryableExceptions(UnprocessableException.class);
return handler;
}
#Bean
public DeadLetterPublishingRecoverer publisher(KafkaOperations kafkaOperations) {
return new DeadLetterPublishingRecoverer(kafkaOperations);
}
}
It is fine without the ConcurrentKafkaListenerContainerFactory, but since we want to scale up scale down the number of instances, we want to use the ConcurrentKafkaListenerContainer.
What is the proper way to do this?
Also, I found that if it is Deserialisation Exception, the message in .DLT is not sent properly (not proper JSON), while if it is "UnprocessableException" (our custom exception that throws within listener) it is proper JSON in .DLT
Since you are wiring up your own consumer factory, you have to set the error handler on it.
but since we want to scale up scale down the number of instances, we want to use the ConcurrentKafkaListenerContainer.
Boot's auto configuration wires up a concurrent container with concurrency=1 (if there is no ....listener.concurrency property); so you can use Boot's factory.
For deserialization exceptions (all exceptions), the record.value() is whatever came in originally. If that's not what you are seeing, please provide an example of what's in the original record and the DLT.

SpringBoot Kafka: Bean method 'kafkaTemplate' in 'KafkaAutoConfiguration' is not loaded

I am using springboot and trying to write a KafkaProducer to push messages in Kafka queue.
I have created these methods in #Configuration class.
#Bean
public KafkaTemplate<String, String> kafkaTemplate(){
return new KafkaTemplate<>(producerFactory());
}
#Bean
public ProducerFactory<String, String> producerFactory() {
Map<String, Object> configProps = new HashMap<>();
configProps.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapAddress); //bootstrapAddress holds address of kafka server
configProps.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
configProps.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
return new DefaultKafkaProducerFactory<>(configProps);
}
And I have Autowired this KafkaTemplate bean in my KafkaMessageProducer class that takes care of the handling of send function of KafkaTemplate.
#Autowired
KafkaTemplate<String, String> kafkaTemplate;
But I am facing this error when I try to compile my code
Field kafkaTemplate in <pathoffile>.KafkaMessageProducer required a bean of type 'org.springframework.kafka.core.KafkaTemplate' that could not be found.
- Bean method 'kafkaTemplate' in 'KafkaAutoConfiguration' not loaded because #ConditionalOnMissingBean (types: org.springframework.kafka.core.KafkaTemplate; SearchStrategy: all) found bean 'avroKafkaTemplate'
Action:Consider revisiting the conditions above or defining a bean of type 'org.springframework.kafka.core.KafkaTemplate' in your configuration.
Also, if I try to exclude KafkaAutoConfiguration in my Spring project, I get error like "Bean cannot be loaded as KafkaAutoConfiguration is disabled'.
Any idea why I am getting this Bean error and what may be the solution?
EDIT:- I found following bean in a jar file used by my project
#Bean
#Conditional({EnableQueueCondition.class})
public KafkaTemplate<String, String> kafkaTemplate() {
KafkaTemplate<String, String> kafkaTemplate = new KafkaTemplate(this.producerFactory());
kafkaTemplate.setProducerListener(new ProducerListenerImpl());
return kafkaTemplate;
}
So, this is where the error is coming from, but I don't how to tell spring to not look to this bean, and use the bean I defined. I have tried using Primary and Qualifier annotations on the bean, it still gives the same error. Is there might be a possibility that my defined bean is not created or not found, and KafkaAutoConfiguration is then looking for the default bean that is override by avroKafkaTemplate bean? What may be the solution of this problem?
By default spring boot provide KafkaTemplate bean if you add kafka dependency in POM.
You just need to define the properties in your application.yml file
example:
server: port: 9000
spring:
kafka:
consumer:
bootstrap-servers: localhost:9092
group-id: group_id
auto-offset-reset: earliest
key-deserializer: org.apache.kafka.common.serialization.StringDeserializer
value-deserializer: org.apache.kafka.common.serialization.StringDeserializer
producer:
bootstrap-servers: localhost:9092
key-serializer: org.apache.kafka.common.serialization.StringSerializer
value-serializer: org.apache.kafka.common.serialization.StringSerializer
enable auto configuration
#Configuration
#EnableKafka
and autowire the kafkaTemplate:
#Autowired
private KafkaTemplate<String, String> kafkaTemplate;
If your case the auto config factory is looking for ProducerFactory<String, Strng>which does not match with your configuration.
#Bean
#ConditionalOnMissingBean(ProducerFactory.class)
public ProducerFactory<String, Strng> kafkaProducerFactory()
so rename your producerFactory() to kafkaProducerFactory(), it will solve your issue.
From the stacktrace, there is another KafkaTemplate bean - avroKafkaTemplate. So I guess there is another configuration, duplicating KafkaTemplate definition.

Kafka topic not created automatically on remote kafka after spring boot start(and create on local kafka server)

1) I start kafka on my machine
2) I start my spring boot server with config:
#Bean
public NewTopic MyTopic() {
return new NewTopic("my-topic", 5, (short) 1);
}
#Bean
public ProducerFactory<String, byte[]> greetingProducerFactory() {
Map<String, Object> configProps = new HashMap<>();
configProps.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
configProps.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
configProps.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, ByteArraySerializer.class);
return new DefaultKafkaProducerFactory<>(configProps);
}
#Bean
public KafkaTemplate<String, byte[]> unpMessageKafkaTemplate() {
return new KafkaTemplate<>(greetingProducerFactory());
}
result - server is start successfully and create my-topic in kafka.
But if I try do it with remote kafka on remote server - topic not create.
and in log spring write:
12:35:09.880 [ main] [INFO ] o.a.k.clients.admin.AdminClientConfig: [] AdminClientConfig values:
bootstrap.servers = [localhost:9092]
If I add this bean to config:
#Bean
public KafkaAdmin admin() {
Map<String, Object> configs = new HashMap<>();
configs.put(AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG, "remote_host:9092");
return new KafkaAdmin(configs);
}
topic create succesfully.
1) Why it happens?
2) Do I have to create KafkaAdmin ? why for the local Kafka is not required?
EDDIT
My current config:
spring:
kafka:
bootstrap-servers: remote:9092
producer:
key-serializer: org.apache.kafka.common.serialization.StringDeserializer
value-serializer: org.apache.kafka.common.serialization.ByteArraySerializer
and
#Configuration
public class KafkaTopicConfig {
#Value("${response.topics.topicName}")
private String topicName;
#Bean
public NewTopic responseTopic() {
return new NewTopic(topicName, 5, (short) 1);
}
}
After start I see:
bootstrap.servers = [remote:9092]
client.id =
connections.max.idle.ms = 300000
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
receive.buffer.bytes = 65536
...
But topic not create
KafkaAdmin is the kafka spring object that looks for NewTopic objects in your spring context and creates them. If you do not have a KafkaAdmin no creation will take place. You can explicitly create KafkaAdmin (as you show in your code snippet) or indirectly order its creation via the spring kafka configuration properties.
KafkaAdmin is a nice to have it is not related to production or consumption to/ from topics for your application code.
EDIT
You must have something wrong; I just tested it...
spring:
kafka:
bootstrap-servers: remote:9092
and
2019-03-21 09:18:18.354 INFO 58301 --- [ main] o.a.k.clients.admin.AdminClientConfig
: AdminClientConfig values:
bootstrap.servers = [remote:9092]
...
Spring Boot will automatically configure a KafkaAdmin for you, but it uses the application.yml (or application.properties). See Boot properties. Scroll down to spring.kafka.bootstrap-servers=. That's why it works with localhost (it's the default).
You also don't need a ProducerFactory or template; boot will create them for you from properties.

spring boot properties yaml

I run spring boot and kafka with auto configuration(via annotattions only) and having props defined in .yaml file, ie:
spring:
kafka:
bootstrap-servers: someserver:9999
consumer:
group-id: mygroup
....
#KafkaListener()
public void receive(ConsumerRecord<?, ?> consumerRecord) {
....
}
And it works fine, spring maps i.e. the field group-id properly.
But when i try configure kafka manually(with ConsumerFactory and ConsumerConfig) using the same yaml file then i run into problem.
In class ConsumerConfig kafka properties are named with . in name, not _ i.e.:
public static final String GROUP_ID_CONFIG = "group.id";
So I cant just load them into map and pass the map to ConsumerFactory because keys are with _ not .
I dont want to do it as ugly as shown in example provided by spring kafka team, when they map props from yaml into config class and then manually assign props in the map to be passed to factory:
#ConfigurationProperties(prefix = "kafka")
public class ConfigProperties {
private String brokerAddress;
private String topic;
private String fooTopic;
public String getBrokerAddress() {
return this.brokerAddress;
}
.....
#Bean
public ProducerFactory<String, String> producerFactory() {
Map<String, Object> props = new HashMap<>();
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, this.configProperties.getBrokerAddress());
.....
return new DefaultKafkaProducerFactory<>(props);
}
If you want to use the dotted properties in the yml file, you need to do it like boot does for arbitrary properties that are not directly supported as boot properties:
spring:
kafka:
consumer:
properties:
group.id: foo
i.e. populate a Properties property.
The reason boot provides first class support for some properties is so that IDE editors can provide content-assist, which is not possible with arbitrary properties.

Resources