spring kafka: Transactions in consumer Timeout expired after 60000 milliseconds while awaiting AddOffsetsToTxn - spring-boot

Question

We have a transactional producer. And there are no issue there.
For the consumer, we see the following in the logs. Question is why is a transaction being started here while we are consuming the message (and there is this resulting exception)?
2022-12-28 18:02:05.986 DEBUG [qa] 85474 --- [tainer#0-51-C-1] o.s.k.t.KafkaTransactionManager : Creating new transaction with name [null]: PROPAGATION_REQUIRED,ISOLATION_DEFAULT
2022-12-28 18:02:10.437 DEBUG [qa] 85474 --- [tainer#0-51-C-1] o.s.k.t.KafkaTransactionManager : Created Kafka transaction on producer [CloseSafeProducer [delegate=brave.kafka.clients.TracingProducer#43e18a26]]
2022-12-28 18:02:10.438 DEBUG [qa] 85474 --- [tainer#0-51-C-1] abc.abc.kafka : NORMAL uid[n/a] cid[uPRwhWZikfcYpGfIqh7TxXFnm6VmZWkf] m[[S] Consuming message] data[record = ConsumerRecord(topic =XXXXXXX)))]
2022-12-28 18:02:10.438 DEBUG [qa] 85474 --- [tainer#0-51-C-1] abc.abc.kafka : NORMAL uid[n/a] cid[uPRwhWZikfcYpGfIqh7TxXFnm6VmZWkf] m[Processed message]
2022-12-28 18:03:10.439 INFO [qa] 85474 --- [tainer#0-51-C-1] abc.abc.common.Generic : NORMAL uid[n/a] cid[uPRwhWZikfcYpGfIqh7TxXFnm6VmZWkf] m[Throwable t] data[exception = [Ljava.lang.StackTraceElement;#6d314e1d] ex[Timeout expired after 60000 milliseconds while awaiting AddOffsetsToTxn] sts[org.apache.kafka.common.errors.TimeoutException: Timeout expired after 60000 milliseconds while awaiting AddOffsetsToTxn
]
2022-12-28 18:03:10.439 DEBUG [qa] 85474 --- [tainer#0-51-C-1] abc.abc.kafka : NORMAL uid[n/a] cid[uPRwhWZikfcYpGfIqh7TxXFnm6VmZWkf] m[[E] Consuming message]
2022-12-28 18:03:10.439 DEBUG [qa] 85474 --- [tainer#0-51-C-1] o.s.k.t.KafkaTransactionManager : Initiating transaction commit
2022-12-28 18:04:10.444 ERROR [qa] 85474 --- [tainer#0-51-C-1] o.s.k.core.DefaultKafkaProducerFactory : org.apache.kafka.common.errors.TimeoutException: Timeout expired after 60000 milliseconds while awaiting EndTxn(true)
commitTransaction failed: CloseSafeProducer [delegate=brave.kafka.clients.TracingProducer#43e18a26]
2022-12-28 18:04:10.444 DEBUG [qa] 85474 --- [tainer#0-51-C-1] o.s.k.t.KafkaTransactionManager : org.apache.kafka.common.errors.TimeoutException: Timeout expired after 60000 milliseconds while awaiting EndTxn(true)
Initiating transaction rollback after commit exception
2022-12-28 18:04:10.445 WARN [qa] 85474 --- [tainer#0-51-C-1] o.s.k.core.DefaultKafkaProducerFactory : Error during some operation; producer removed from cache: CloseSafeProducer [delegate=brave.kafka.clients.TracingProducer#43e18a26]
2022-12-28 18:04:12.436 ERROR [qa] 85474 --- [tainer#0-51-C-1] o.s.k.l.KafkaMessageListenerContainer : org.apache.kafka.common.errors.TimeoutException: Timeout expired after 60000 milliseconds while awaiting EndTxn(true)
Transaction rolled back
This is the configuration:
spring:
data:
mongodb:
uri: indb-prop
auto-index-creation: false
kafka:
producer:
key-serializer: org.apache.kafka.common.serialization.StringSerializer
value-serializer: org.springframework.kafka.support.serializer.JsonSerializer
retries: 2
client-id: ${spring.application.name}-${info.cluster}-${IP_ADDRESS}PP
transaction-id-prefix: tx-${spring.kafka.producer.client-id}-
properties:
enable.idempotence: true
spring.json.add.type.headers: false
bootstrap-servers: ${kafka_bootstrap_servers_2}
# listener:
# missing-topics-fatal: true
# type: batch
# concurrency: 15
# ack-mode: manual_immediate
# poll-timeout: 1s
consumer:
key-deserializer: org.springframework.kafka.support.serializer.ErrorHandlingDeserializer
value-deserializer: org.springframework.kafka.support.serializer.ErrorHandlingDeserializer
group-id: ${spring.application.name}-${info.cluster}-111111111112
client-id: ${spring.application.name}-${info.cluster}-${IP_ADDRESS}
# client-id: ${spring.kafka.consumer.group-id}-${IP_ADDRESS}
auto-offset-reset: earliest
enable-auto-commit: false
isolation-level: read_committed
# max-poll-records: 3
fetch-max-wait: 5s
properties:
max.poll.interval.ms: 20000000
spring.json.trusted.packages: '*'
spring.deserializer.key.delegate.class: org.apache.kafka.common.serialization.StringDeserializer
spring.deserializer.value.delegate.class: org.springframework.kafka.support.serializer.JsonDeserializer
spring.json.value.default.type: net.abc.abc.EventData
bootstrap-servers: ${kafka_bootstrap_servers}
And the consumer:
#KafkaListener(
topics = "SOME_TOPIC",
autoStartup = "true")
// #Transactional(transactionManager = "mongoTransactionManager", propagation = Propagation.REQUIRED)
#Override
public void onMessage(ConsumerRecord<String, EventData> data) {
if (data == null || data.value() == null) {
return;
}
logger.debug(m -> m.event(KAFKA)
.msg("[S] Consuming message")
.with("record", data));
try {
logger.debug(m -> m.event(KAFKA)
.msg("Processed message"));
}
finally {
logger.debug(m -> m.event(KAFKA)
.msg("[E] Consuming message"));
}
The springBoot version: '2.6.6'

Answer 1

How many broker nodes are you using? If 1, have you overridden the broker params KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR and KAFKA_TRANSACTION_STATE_LOG_MIN_ISR to 1?
Similar sounding issue here:
https://github.com/testcontainers/testcontainers-java/issues/1816
Rob.