KafkaTransactionManager with Spring transaction annotation - spring

Question

We are trying to implement a kafka transaction through spring boot. And noticed some interesting things.
#Configuration
public class KafkaProducerConfiguration {
#Value(value = "${spring.kafka.bootstrap-servers}")
private String bootstrapAddress;
#Bean
public ProducerFactory<Long, RatingProcessingMessage> producerFactory() {
Map<String, Object> configProps = new HashMap<>();
configProps.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapAddress);
configProps.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, LongSerializer.class);
configProps.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, JsonSerializer.class);
configProps.put(ProducerConfig.TRANSACTIONAL_ID_CONFIG, "transaction-");
return new DefaultKafkaProducerFactory<>(configProps);
}
#Bean
public KafkaTemplate<Long, Message> kafkaTemplate() {
return new KafkaTemplate<>(producerFactory());
}
}
Here it is interesting, after set ProducerConfig.TRANSACTIONAL_ID_CONFIG, the auto configuration class KafkaAutoConfiguration.class should create the bean
#Bean
#ConditionalOnProperty(
name = {"spring.kafka.producer.transaction-id-prefix"}
)
#ConditionalOnMissingBean
public KafkaTransactionManager<?, ?> kafkaTransactionManager(ProducerFactory<?, ?> producerFactory) {
return new KafkaTransactionManager(producerFactory);
}
But it doesn't happen, so spring.kafka.producer.transaction-id-prefix != ProducerConfig.TRANSACTIONAL_ID_CONFIG
Сould not find a suitable constant in ProducerConfig.
For reasons unknown to me, when calling kafkaTemplate.send() in the transaction, we see in the log JpaTransactionManager
15:56:13.939 [http-nio-8081-exec-2] INFO o.a.k.c.p.i.TransactionManager - [Producer clientId=producer-transaction-0, transactionalId=transaction-0] Invoking InitProducerId for the first time in order to acquire a producer ID
15:56:13.991 [kafka-producer-network-thread | producer-transaction-0] INFO org.apache.kafka.clients.Metadata - [Producer clientId=producer-transaction-0, transactionalId=transaction-0] Cluster ID: M-DqpGu4T_25jGui6701-w1
15:56:14.013 [kafka-producer-network-thread | producer-transaction-0] INFO o.a.k.c.p.i.TransactionManager - [Producer clientId=producer-transaction-0, transactionalId=transaction-0] Discovered transaction coordinator localhost:9094 (id: 0 rack: null)
15:56:14.185 [kafka-producer-network-thread | producer-transaction-0] INFO o.a.k.c.p.i.TransactionManager - [Producer clientId=producer-transaction-0, transactionalId=transaction-0] ProducerId set to 5 with epoch 14
15:56:14.258 [kafka-producer-network-thread | producer-transaction-0] INFO org.apache.kafka.clients.Metadata - [Producer clientId=producer-transaction-0, transactionalId=transaction-0] Resetting the last seen epoch of partition event-rating-0 to 0 since the associated topicId changed from null to dEt3oJ_SQqW0Gz1mgPc5Fg
15:56:14.259 [kafka-producer-network-thread | producer-transaction-0] INFO org.apache.kafka.clients.Metadata - [Producer clientId=producer-transaction-0, transactionalId=transaction-0] Resetting the last seen epoch of partition event-rating-5 to 0 since the associated topicId changed from null to dEt3oJ_SQqW0Gz1mgPc5Fg
15:56:14.259 [kafka-producer-network-thread | producer-transaction-0] INFO org.apache.kafka.clients.Metadata - [Producer clientId=producer-transaction-0, transactionalId=transaction-0] Resetting the last seen epoch of partition event-rating-2 to 0 since the associated topicId changed from null to dEt3oJ_SQqW0Gz1mgPc5Fg
15:56:14.259 [kafka-producer-network-thread | producer-transaction-0] INFO org.apache.kafka.clients.Metadata - [Producer clientId=producer-transaction-0, transactionalId=transaction-0] Resetting the last seen epoch of partition event-rating-8 to 0 since the associated topicId changed from null to dEt3oJ_SQqW0Gz1mgPc5Fg
15:56:14.259 [kafka-producer-network-thread | producer-transaction-0] INFO org.apache.kafka.clients.Metadata - [Producer clientId=producer-transaction-0, transactionalId=transaction-0] Resetting the last seen epoch of partition event-rating-9 to 0 since the associated topicId changed from null to dEt3oJ_SQqW0Gz1mgPc5Fg
15:56:14.259 [kafka-producer-network-thread | producer-transaction-0] INFO org.apache.kafka.clients.Metadata - [Producer clientId=producer-transaction-0, transactionalId=transaction-0] Resetting the last seen epoch of partition event-rating-4 to 0 since the associated topicId changed from null to dEt3oJ_SQqW0Gz1mgPc5Fg
15:56:14.260 [kafka-producer-network-thread | producer-transaction-0] INFO org.apache.kafka.clients.Metadata - [Producer clientId=producer-transaction-0, transactionalId=transaction-0] Resetting the last seen epoch of partition event-rating-1 to 0 since the associated topicId changed from null to dEt3oJ_SQqW0Gz1mgPc5Fg
15:56:14.260 [kafka-producer-network-thread | producer-transaction-0] INFO org.apache.kafka.clients.Metadata - [Producer clientId=producer-transaction-0, transactionalId=transaction-0] Resetting the last seen epoch of partition event-rating-6 to 0 since the associated topicId changed from null to dEt3oJ_SQqW0Gz1mgPc5Fg
15:56:14.260 [kafka-producer-network-thread | producer-transaction-0] INFO org.apache.kafka.clients.Metadata - [Producer clientId=producer-transaction-0, transactionalId=transaction-0] Resetting the last seen epoch of partition event-rating-7 to 0 since the associated topicId changed from null to dEt3oJ_SQqW0Gz1mgPc5Fg
15:56:14.406 [http-nio-8081-exec-2] DEBUG o.s.orm.jpa.JpaTransactionManager - Initiating transaction commit
15:56:14.406 [http-nio-8081-exec-2] DEBUG o.s.orm.jpa.JpaTransactionManager - Committing JPA transaction on EntityManager [SessionImpl(1789264984<open>)]
15:56:14.511 [http-nio-8081-exec-2] DEBUG o.s.orm.jpa.JpaTransactionManager - Closing JPA EntityManager [SessionImpl(1789264984<open>)] after transaction
import org.springframework.transaction.annotation.Transactional;
#AllArgsConstructor
#Service
public class TestService {
private final KafkaTemplate<Long, Message> kafkaTemplate;
#Transactional
public void test(Long id, Message message){
kafkaTemplate.send("topic", id, message);
}
}
As described above
I wonder why KafkaTransactionManager is not injected (but it is important that the transaction was successful without this bean)
For some reason JpaTransactionManager appeared in the logs

Answer 1

Boot knows nothing about
configProps.put(ProducerConfig.TRANSACTIONAL_ID_CONFIG, "transaction-");
Boot auto configuration requires you to set the property in application.yml/properties.
https://docs.spring.io/spring-boot/docs/current/reference/html/application-properties.html#application-properties.integration.spring.kafka.producer.transaction-id-prefix
Instead of defining your own producer factory, let boot auto configure it for you.
You must also have enable JPA to get its TM.