Is it possible to enable spring.cloud.stream.kafka.streams.binder.auto-create-topics in the application.yml only for topics that the application produces and not auto-create topics that the application consumes? I'd like for my producers to only be responsible for creating topics. I'm using spring-cloud-streams with the kafka-streams binder.
In a more general sense, is this the right approach to creating kafka topics?
It is a binder, not a binding, property so, no.
You can, however, use the multi-binder support and define 2 kafka binders with that property set as needed, one for producer and the other for consumer bindings.
https://docs.spring.io/spring-cloud-stream/docs/3.0.10.RELEASE/reference/html/spring-cloud-stream.html#multiple-binders
I suggest you open a new feature suggestion in a GitHub issue against the binder, allowing the binder property to be overridden at the binding level.
https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues
Related
I'm experiencing a problem, I have an application in spring boot, this application consumes from one topic and produces in another topic.
the topic that the application consumes is on-premises, the topic that the application produces is on cloud aws.
is there a way to specify a bootstrap server and schema registry for each topic?
my application.properties has the following property:
spring.kafka.bootstrap-servers=localhost:32202 spring.kafka.properties.schema.registry.url=127.0.0.1:8082
the problem here is that these properties are for both consumer and producer.
I need to specify a bootstrap server for the consumer, another for the producer.
also specify a schema-registry for the consumer, another for the producer.
I don't know if this way is the best way to deal with this problem.
spring.kafka.consumer.bootstrap-servers=consumer-localhost:32202 spring.kafka.consumer.schema.registry.url=consumer-127.0.0.1:8082 spring.kafka.producer.bootstrap-servers=producer-localhost:10010 spring.kafka.producer.schema.registry.url=producer-127.0.0.1:9090
Thanks in advance!
See the Spring Boot documentation.
The properties supported by auto configuration are shown in the “Integration Properties” section of the Appendix. Note that, for the most part, these properties (hyphenated or camelCase) map directly to the Apache Kafka dotted properties. See the Apache Kafka documentation for details.
The first few of these properties apply to all components (producers, consumers, admins, and streams) but can be specified at the component level if you wish to use different values. Apache Kafka designates properties with an importance of HIGH, MEDIUM, or LOW. Spring Boot auto-configuration supports all HIGH importance properties, some selected MEDIUM and LOW properties, and any properties that do not have a default value.
Only a subset of the properties supported by Kafka are available directly through the KafkaProperties class. If you wish to configure the producer or consumer with additional properties that are not directly supported, use the following properties:
spring.kafka.properties[prop.one]=first
spring.kafka.admin.properties[prop.two]=second
spring.kafka.consumer.properties[prop.three]=third
spring.kafka.producer.properties[prop.four]=fourth
spring.kafka.streams.properties[prop.five]=fifth
So...
spring.kafka.consumer.bootstrap-servers=consumer-localhost:32202
spring.kafka.consumer.properties.schema.registry.url=consumer-127.0.0.1:8082
spring.kafka.producer.bootstrap-servers=producer-localhost:10010
spring.kafka.producer.properties.schema.registry.url=producer-127.0.0.1:9090
This sets the common prop.one Kafka property to first (applies to producers, consumers and admins), the prop.two admin property to second, the prop.three consumer property to third, the prop.four producer property to fourth and the prop.five streams property to fifth.
For the management of the headers of the messages that are produced/consumed in the Kafka binder there is the KafkaHeaderMapper interface, whose implementation as a bean can be configured with the following property: spring.cloud.stream.kafka.binder.headerMapperBeanName.
Is there something similar for the KafkaStreams binder in Spring Cloud Stream? My intention is to be able to control how to deserialize/serialize or include/exclude message headers on stream input and output. does anyone know how to realize this?
There is not an equivalent; for streams; Spring is only involved with setting up the infrastructure/topology; it is not involved with the runtime record processing.
You can, however, use a custom serializer/deserializer and manipulate the headers there.
My understanding was that spring-kafka was created to interact with Kafka Client APIs, and later on, spring-cloud-stream project was created for "building highly scalable event-driven microservices connected with shared messaging systems", and this project includes a couple of binders, one of them is a binder that allows the interaction with Kafka Stream API:
spring-cloud-stream-binder-kafka-streams
So it was clear to me that if I want to interact with Kafka Stream API, I will use the spring-cloud-stream approach with the appropriate binder.
But, I found out that you can interact with Kafka Stream API also with the spring-kafka approach.
Need the below two dependencies. One example is here.
'org.springframework.kafka:spring-kafka'
'org.apache.kafka:kafka-streams'
So my question is - if both the approaches allow interaction with Kafka Stream API, what are the differences between the approaches?
As Gary pointed out in the comments above, spring-kafka is the lower-level library that provides the building blocks for Spring Cloud Stream Kafka Streams binder (spring-cloud-stream-binder-kafka-streams). The binder provides a programming model to write your Kafka Streams processor as a java.util.function.Function or java.util.function.Consumer.
You can have multiple such functions and each of them will build its own Kafka Streams topologies. Behind the scenes, the binder uses Spring-Kafka to build the Kafka Streams StreamsBuilder object using the StreamsBuilderFactoryBean. Binder also allows you to compose various functions. The functional model comes largely from Spring Cloud Function, but it is adapted for the Kafka Streams in the binder implementation. The short answer is that both spring-Kafka and the Spring Cloud Stream Kafka Streams binder will work, but the binder gives a programming model and extra features consistent with Spring Cloud Stream whereas spring-kafka gives various low-level building blocks.
I have a usecase where I want to get the underlying Kafka producer (KafkaTemplate) in a Spring Cloud Stream application. While navigating the code I stumbled upon KafkaProducerMessageHandler which has a getKafkaTemplate method. However, it fails to auto-wire.
Also, if I directly auto-wire KafkaTemplate, the template is initialized with default properties and it ignores the broker in the binder key of the SCSt configuration
How can I access the underlying KafkaTemplate or a producer/consumer in a Spring Cloud Stream app?
EDIT: Actually my SCSt app has multiple Kafka binders and I want to get the KafkaTemplate or Kafka producer corresponding to each binder. Is that possible somehow?
It's not entirely clear why you would need to do that, but you can capture the KafkaTemplates by adding a ProducerMessageHandlerCustomizer #Bean to the application context.
I want to set acks=all property for my producer in my spring cloud stream kafka application.
I had tried like this :
spring.cloud.stream.kafka.binder.requiredAcks=all
and
spring.cloud.stream.kafka.streams.binder.configuration=all
and
spring.cloud.stream.kafka.streams.bindings.<channel>.producer.configuration.requiredAcks=all
Unfortunately noting works for me.
Can you please help me how to set these kind of properties to application level or producer/consumer.
The configuration below is only used by the Kafka Binders (not the kafka Streams one). It is used to set the acks property of a producer instance.
spring.cloud.stream.kafka.binder.requiredAcks
To configure a Kafka Streams instance, properties must be prefixed with spring.cloud.stream.kafka.streams.binder(Spring Cloud Stream Configuration).
Into KafkaStreams, producers properties can be override by adding the prefix ".producer" (see Configuring a Streams Application; So to configure producer acks you should define the following property :
spring.cloud.stream.kafka.streams.binder.configuration.producer.acks=all
Note, that if you are building a stateful Kafka Streams application it's highly recommended to enable the exactly_once semantic.
This semantic can be configured with :
spring.cloud.stream.kafka.streams.binder.configuration.processing.guarantee=exactly-once