For the management of the headers of the messages that are produced/consumed in the Kafka binder there is the KafkaHeaderMapper interface, whose implementation as a bean can be configured with the following property: spring.cloud.stream.kafka.binder.headerMapperBeanName.
Is there something similar for the KafkaStreams binder in Spring Cloud Stream? My intention is to be able to control how to deserialize/serialize or include/exclude message headers on stream input and output. does anyone know how to realize this?
There is not an equivalent; for streams; Spring is only involved with setting up the infrastructure/topology; it is not involved with the runtime record processing.
You can, however, use a custom serializer/deserializer and manipulate the headers there.
Related
My understanding was that spring-kafka was created to interact with Kafka Client APIs, and later on, spring-cloud-stream project was created for "building highly scalable event-driven microservices connected with shared messaging systems", and this project includes a couple of binders, one of them is a binder that allows the interaction with Kafka Stream API:
spring-cloud-stream-binder-kafka-streams
So it was clear to me that if I want to interact with Kafka Stream API, I will use the spring-cloud-stream approach with the appropriate binder.
But, I found out that you can interact with Kafka Stream API also with the spring-kafka approach.
Need the below two dependencies. One example is here.
'org.springframework.kafka:spring-kafka'
'org.apache.kafka:kafka-streams'
So my question is - if both the approaches allow interaction with Kafka Stream API, what are the differences between the approaches?
As Gary pointed out in the comments above, spring-kafka is the lower-level library that provides the building blocks for Spring Cloud Stream Kafka Streams binder (spring-cloud-stream-binder-kafka-streams). The binder provides a programming model to write your Kafka Streams processor as a java.util.function.Function or java.util.function.Consumer.
You can have multiple such functions and each of them will build its own Kafka Streams topologies. Behind the scenes, the binder uses Spring-Kafka to build the Kafka Streams StreamsBuilder object using the StreamsBuilderFactoryBean. Binder also allows you to compose various functions. The functional model comes largely from Spring Cloud Function, but it is adapted for the Kafka Streams in the binder implementation. The short answer is that both spring-Kafka and the Spring Cloud Stream Kafka Streams binder will work, but the binder gives a programming model and extra features consistent with Spring Cloud Stream whereas spring-kafka gives various low-level building blocks.
I have a usecase where I want to get the underlying Kafka producer (KafkaTemplate) in a Spring Cloud Stream application. While navigating the code I stumbled upon KafkaProducerMessageHandler which has a getKafkaTemplate method. However, it fails to auto-wire.
Also, if I directly auto-wire KafkaTemplate, the template is initialized with default properties and it ignores the broker in the binder key of the SCSt configuration
How can I access the underlying KafkaTemplate or a producer/consumer in a Spring Cloud Stream app?
EDIT: Actually my SCSt app has multiple Kafka binders and I want to get the KafkaTemplate or Kafka producer corresponding to each binder. Is that possible somehow?
It's not entirely clear why you would need to do that, but you can capture the KafkaTemplates by adding a ProducerMessageHandlerCustomizer #Bean to the application context.
What benefits does spring Kafka template provide?
I have tried the existing Producer/Consumer API by Kafka. That is very simple to use, then why use Kafka template.
Kafka Template internally uses Kafka producer so you can directly use Kafka APIs. The benefit of using Kafka template is it provides different methods for sending message to Kafka topic, kind of added benefits you can see the API comparison between KafkaProducer and KafkaTemplate here:
https://kafka.apache.org/10/javadoc/org/apache/kafka/clients/producer/KafkaProducer.html
https://docs.spring.io/spring-kafka/api/org/springframework/kafka/core/KafkaTemplate.html
You can see KafkaTemplate provide many additional ways of sending data to Kafka topics because of various send methods while some calls are the same as Kafka API and are simply forwarded from KafkaTemplate to KafkaProducer.
It's up to the developer what to use. If you feel like working with KafkaTemplate is easy as you don't have to create ProducerRecord a simple send method will do all the work for you.
At a high level, the benefit is that you can externalize your properties objects more easily and you can just focus on the record processing logic
Plus Spring is integrated with lots of other components.
Note: Other options still exist like Reactor Kafka, Alpakka, Apache Camel, Smallrye reactive messaging, Vert.x... But they all wrap the same Kafka API.
So, I'd say you're (marginally) trading efficiency for convinience
Currently I'm using SCS with almost default configuration for sending and receiving message between microservices.
Somehow I've read this
https://www.confluent.io/blog/enabling-exactly-kafka-streams
and wonder that it is gonna works or not if we just put the property called "processing.guarantee" with value "exactly-once" there through properties in Spring boot application ?
In the context of your question you should look at Spring Cloud Stream as just a delegate between target system (e.g., Kafka) and your code. The binders that enable such delegation are usually implemented in such way that they propagate whatever functionality supported by the target system.
I am using Spring Cloud Stream with RabbitMQ binder. I need to call external service which does not use Spring Cloud Stream. This service is using type and correlation_id message properties.
I tried to set headers in outgoing message but, even though properties, technically are headers, they are treated in special way. So setting type header does not set a property.
I am aware about interceptors and, if using only Spring RabbitMq, it would not be a problem. But since Spring Cloud Stream represents higher level of abstraction, all binder specific settings are hidden.
Is there any possibility to set up RabbitMQ properties in outgoing stream message?
Properties are mapped from message headers keyed by AmqpHeaders constants; in this case AmqpHeaders.TYPE (amqp_type) and AmqpHeaders.CORRELATION_ID (amqp_correlationId).
All "unknown" message headers are mapped as rabbit headers.