Spring kafka producer keep on waiting for one minute in some cases for producing data and failed with exception for some cases - spring-boot

Question

I have implemented Spring Kafka Template for producing event in my spring boot project.The code for producing an event is given below-
Producer Config:
#Beanpublic Map<String, Object> producerConfigs() throws FileNotFoundException {
Map<String, Object> props = new HashMap<>();
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG,kafkaProperties.getBootstrapServers());
props.put(CommonClientConfigs.SECURITY_PROTOCOL_CONFIG,kafkaProperties.getSecurity().getProtocol());
props.put(SslConfigs.SSL_TRUSTSTORE_LOCATION_CONFIG,ResourceUtils.getFile("classpath:client.truststoreks").getAbsolutePath());
props.put(SslConfigs.SSL_ENDPOINT_IDENTIFICATION_ALGORITHM_CONFIG,StringUtils.EMPTY);props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG,StringSerializer.class);
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG,JsonSerializer.class);
props.put(ProducerConfig.LINGER_MS_CONFIG, "100");
return props;
}
Producer Service Code:
public class KafkaProducerService<V> implements KafkaProducer<V> {
#Autowired
KafkaTemplate<String, V> kafkaTemplate;
#Autowired
KafkaTemplate<String, V> transactionLogKafkaTemplate;
public KafkaProducerService(KafkaTemplate<String, V> kafkaTemplate, KafkaTemplate<String, V> transactionLogKafkaTemplate) {
this.kafkaTemplate = kafkaTemplate;
this.transactionLogKafkaTemplate = transactionLogKafkaTemplate;
}
#Override
#Retryable({KafkaException.class, TimeoutException.class})
public void produce(String topic, String key, V value) {
log.info("Calling producer service for producing event on topic "+topic);
sendCallbackEvents(kafkaTemplate, topic, key, value);
}
private void sendCallbackEvents(KafkaTemplate<String, V> kafkaTemplate, String topic, String key, V value) {
ProducerRecord<String, V> producerRecord = new ProducerRecord(topic, key, value);
ListenableFuture<SendResult<String, V>> future = kafkaTemplate.send(producerRecord);
future.addCallback(new ListenableFutureCallback<SendResult<String, V>>() {
#Override
public void onSuccess(SendResult<String, V> result) {
log.info(String.format("Produced event to topic %s: key = %-10s value = %s", topic, key, value));
}
#Override
public void onFailure(Throwable ex) {
log.error("Producing of data on topic {} is failed", topic, ex.getCause());
}
});
}
}
P.S: We are using AWS MSK as a broker for producing an event.
But in some cases, it's taking one minute time for producing an event and getting failed with the below error in logs-
ERROR LogAccessor - Exception thrown when sending a message with key='xx' and payload='Event(key=value)' to topic topicName:
Hence it's able to produce the event due to retry logic on producer service but due to that 1-minute delay, I am facing several issues.
I tried to find out the reason for the producer service delay and failure while going through the Spring Kafka dependency classes but no luck.
I am not able to find out the exact reason why the producer is getting delayed and failing in 1st attempt for some cases. Can anyone help me in identifying the reason for that and the solution to the issue?