Spring Kafka - Batch processing not working - spring

Question

I have Spring Kafka consumer and I want to consume 50 records each 60th seconds. I referred few documents and configured my application like this->
Consumer Configurations
#Bean
public ConsumerFactory<String, DeviceInfo> consumerFactory() {
Map<String, Object> config = new HashMap<>();
config.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, kafkaConfig.getConsumerBootstrapServers());
config.put(ConsumerConfig.GROUP_ID_CONFIG, "fixit-airwatch-etl");
config.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
config.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, JsonDeserializer.class);
config.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, AUTO_OFFSET_RESET_CONFIG);
config.put(ConsumerConfig.MAX_POLL_RECORDS_CONFIG, "50");
config.put(ConsumerConfig.MAX_POLL_INTERVAL_MS_CONFIG, "60000");
return new DefaultKafkaConsumerFactory<>(config, new StringDeserializer(),
new JsonDeserializer<>(DeviceInfo.class));
}
#Bean
public ConcurrentKafkaListenerContainerFactory<String, DeviceInfo> kafkaListenerFactory(ConsumerFactory<String, DeviceInfo> consumerFactory) {
ConcurrentKafkaListenerContainerFactory<String, DeviceInfo> factory = new ConcurrentKafkaListenerContainerFactory<>();
factory.setConsumerFactory(consumerFactory);
factory.setBatchListener(true);
factory.setBatchErrorHandler(new BatchLoggingErrorHandler());
return factory;
}
Kafka listener
#KafkaListener(topics = "${app.kafka.topic}", groupId = "etl-group", containerFactory = "kafkaListenerFactory")
public void receive(#Payload List<DeviceInfo> messages) {
log.info("Got these many records from the topic {}", messages.size());
}
application.properties
spring.kafka.listener.type=batch
Inspite of having all these configurations, looks like I'm not seeing the expected behavior. Log statements are as below.
2022-07-04 12:07:22.533 INFO 89732 --- [ntainer#0-0-C-1] c.w.g.g.f.consumer.KafkaConsumer : Got these many records from the topic 9
2022-07-04 12:07:22.533 INFO 89732 --- [ntainer#0-0-C-1] c.w.g.g.f.consumer.KafkaConsumer : Got these many records from the topic 4
2022-07-04 12:07:22.534 INFO 89732 --- [ntainer#0-0-C-1] c.w.g.g.f.consumer.KafkaConsumer : Got these many records from the topic 6
2022-07-04 12:07:22.534 INFO 89732 --- [ntainer#0-0-C-1] c.w.g.g.f.consumer.KafkaConsumer : Got these many records from the topic 8
2022-07-04 12:07:22.535 INFO 89732 --- [ntainer#0-0-C-1] c.w.g.g.f.consumer.KafkaConsumer : Got these many records from the topic 8
2022-07-04 12:07:22.535 INFO 89732 --- [ntainer#0-0-C-1] c.w.g.g.f.consumer.KafkaConsumer : Got these many records from the topic 6
2022-07-04 12:07:22.536 INFO 89732 --- [ntainer#0-0-C-1] c.w.g.g.f.consumer.KafkaConsumer : Got these many records from the topic 11
Eventhough, I mentioned the batch size as 50, it is fetching random number of records. Also, the delay between each batch processing is not what I configured. Did I miss anything in this? Please share your thoughts. TIA.

Answer 1

Your configuration looks fine, but you need to keep one thing in mind is, at every 60 secs, 50 records must be available in the topic to be consumed, for this consumer to work as expected. Or you need to adjust the value of the following properties.
fetch.min.bytes
This property allows a consumer to specify the minimum amount of data that it
wants to receive from the broker when fetching records. If a broker receives a request
for records from a consumer but the new records amount to fewer bytes than
min.fetch.bytes, the broker will wait until more messages are available before send‐
ing the records back to the consumer. This reduces the load on both the consumer
and the broker as they have to handle fewer back-and-forth messages in cases where
the topics don’t have much new activity (or for lower activity hours of the day). You
will want to set this parameter higher than the default if the consumer is using too
much CPU when there isn’t much data available, or reduce load on the brokers when
you have large number of consumers.
fetch.max.wait.ms
By setting fetch.min.bytes, you tell Kafka to wait until it has enough data to send
before responding to the consumer. fetch.max.wait.ms lets you control how long to
wait. By default, Kafka will wait up to 500 ms. This results in up to 500 ms of extra
latency in case there is not enough data flowing to the Kafka topic to satisfy the mini‐
mum amount of data to return. If you want to limit the potential latency (usually due
to SLAs controlling the maximum latency of the application), you can set
fetch.max.wait.ms to a lower value. If you set fetch.max.wait.ms to 100 ms and
fetch.min.bytes to 1 MB, Kafka will recieve a fetch request from the consumer and
will respond with data either when it has 1 MB of data to return or after 100 ms,
whichever happens first.