Kafka aborting producer batches due to fatal error - spring

Question

I use Spring kafka for my application. It's working perfectly with spring boot version 2.3.2, but when I upgrade to 2.7.0, below error:
2022-05-27 15:32:33.357 "ubuntu1804" "core-service" [main,,] INFO o.a.k.c.producer.KafkaProducer.configureTransactionState:532 - [Producer clientId=producer-1] Instantiated an idempotent producer.
2022-05-27 15:32:33.392 "ubuntu1804" "core-service" [main,,] INFO o.a.k.c.s.a.AbstractLogin.login:61 - Successfully logged in.
2022-05-27 15:32:33.430 "ubuntu1804" "core-service" [main,,] INFO o.a.k.c.utils.AppInfoParser.<init>:119 - Kafka version: 3.0.1
2022-05-27 15:32:33.432 "ubuntu1804" "core-service" [main,,] INFO o.a.k.c.utils.AppInfoParser.<init>:120 - Kafka commitId: 8e30984f43e64d8b
2022-05-27 15:32:33.432 "ubuntu1804" "core-service" [main,,] INFO o.a.k.c.utils.AppInfoParser.<init>:121 - Kafka startTimeMs: 1653640353428
2022-05-27 15:32:34.350 "ubuntu1804" "core-service" [kafka-producer-network-thread | producer-1,,] INFO o.a.kafka.clients.Metadata.update:287 - [Producer clientId=producer-1] Cluster ID: XeVfqn0-QOOFk9uCnDkisw
2022-05-27 15:32:34.354 "ubuntu1804" "core-service" [kafka-producer-network-thread | producer-1,,] INFO o.a.k.c.p.i.TransactionManager.transitionToFatalError:500 - [Producer clientId=producer-1] Transiting to fatal error state due to org.apache.kafka.common.errors.ClusterAuthorizationException: Cluster authorization failed.
2022-05-27 15:32:34.386 "ubuntu1804" "core-service" [kafka-producer-network-thread | producer-1,,] ERROR o.a.k.c.p.internals.Sender.maybeAbortBatches:497 - [Producer clientId=producer-1] Aborting producer batches due to fatal error
org.apache.kafka.common.errors.ClusterAuthorizationException: Cluster authorization failed.
2022-05-27 15:32:34.387 "ubuntu1804" "core-service" [kafka-producer-network-thread | producer-1,,] ERROR o.a.k.c.p.internals.Sender.maybeAbortBatches:497 - [Producer clientId=producer-1] Aborting producer batches due to fatal error
org.apache.kafka.common.errors.ClusterAuthorizationException: Cluster authorization failed.
2022-05-27 15:32:34.388 "ubuntu1804" "core-service" [kafka-producer-network-thread | producer-1,,] ERROR o.a.k.c.p.internals.Sender.maybeAbortBatches:497 - [Producer clientId=producer-1] Aborting producer batches due to fatal error
org.apache.kafka.common.errors.ClusterAuthorizationException: Cluster authorization failed.
Here is my kafka configuration:
spring:
kafka:
producer:
bootstrap-servers: 10.98.0.1:9092,10.98.0.2:9092,10.98.0.3:9092
key-serializer: org.apache.kafka.common.serialization.StringSerializer
value-serializer: org.apache.kafka.common.serialization.StringSerializer
properties:
security:
protocol: SASL_PLAINTEXT
sasl:
mechanism: PLAIN
jaas:
config: org.apache.kafka.common.security.plain.PlainLoginModule required username=${KAFKA_USERNAME:myuser} password=${ENV_KAFKA_PASSWORD:mypassword};
Here is my logging config
logging:
level:
org:
springframework:
web: INFO
apache:
kafka: INFO
When I set logging level to OFF, there is no error anymore. But I just want to know why and how to fix it. Thank you in advance

Answer 1

It may be due to the version of the kafka client, which is not compatible with the kafka broker servion. Change the version of kafka client accordingly.
Other sources
enter link description here