Do anyone has a hello world example of reading message as stream using kafka stream and spring boot.
My kafka cluster is SASL_ SSL secured. So how do I connect my spring boot kafka stream application with. What to write in application.properties file.
I donot want to use spring cloud stream.
server.port=8084
topic.name=test-topic
server.servlet.context-path=/api/v1
spring.application.name=kafkatest
spring.kafka.bootstrap-servers=*************.com:9093
spring.kafka.producer.key-serializer=org.apache.kafka.common.serialization.IntegerSerializer
spring.kafka.producer.value-serializer=io.confluent.kafka.serializers.KafkaAvroSerializer
spring.kafka.jaas.enabled=true
spring.kafka.properties.security.protocol= SASL_SSL
spring.kafka.properties.security.krb5.config = file:/etc/krb5.conf
spring.kafka.properties.sasl.mechanism = GSSAPI
spring.kafka.properties.sasl.kerberos.service.name= kafka
spring.kafka.properties.sasl.jaas.config = com.sun.security.auth.module.Krb5LoginModule required useTicketCache=false serviceName="kafka" storeKey=true principal="***************" useKeyTab=true keyTab="/home/api/config/kafkaclient.keytab";
spring.kafka.ssl.trust-store-location= file:/home/api/config/truststore.p12 spring.kafka.ssl.trust-store-password=*********************
spring.kafka.ssl.trust-store-type= PKCS12
I did this way.
Add the sasl config in the properties.
> spring:
> kafka:
> client-id: ${spring.app.name}
> bootstrap-servers: <cluster_url>:9092
> properties:
> ssl.endpoint.identification.algorithm: https
> sasl.mechanism: PLAIN
> sasl.jaas.config: org.apache.kafka.common.security.plain.PlainLoginModule required
> username="xxxxx" password="xxxxxxx";
> security.protocol: SASL_SSL
And then created a bean which initializes KafkaStreamsConfiguration
#Bean
public KafkaStreamsConfiguration streamsConfig(KafkaProperties kafkaProperties) {
Map<String, Object> streamsProperties = kafkaProperties.buildStreamsProperties();
streamsProperties.put(BOOTSTRAP_SERVERS_CONFIG, server);
streamsProperties.put(APPLICATION_ID_CONFIG, applicationId);
streamsProperties.put(DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.String().getClass().getName());
streamsProperties.put(DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.String().getClass().getName());
return new KafkaStreamsConfiguration(streamsProperties);
}
Note that: I'm using KafkaProperties.buildStreamsProperties() to fetch the streams config from properties
Related
i'm working on upgrading spring boot to 2.7.8 and spring cloud to 2021.0.5.
I have Spring cloud stream kafka consumer using avro deserialization in batch-mode, and I was trying to use useNativeEncoding according to documentation.
the problem is when using an input of Message<List> the spring cloud stream code overrides (when using sleuth) the flag of native encoding to false in this class SimpleFunctionRegistry, this the message payload is empty.
without using the Message> it works fine, i.e. List.
after spending more than one day trying to debug the problem without understanding why, I took it to a side project to test it, and it stopped working after using sleuth.
The Bug
the problem is one the class SimpleFunctionRegistry on methodprivate FunctionInvocationWrapper wrapInAroundAdviceIfNecessary(FunctionInvocationWrapper function) it calls the apply and override the flag
spring cloud stream team is there any workaround? or an easy fix?
application.yaml example
spring:
cloud:
stream:
binders:
kafka-string-avro-native:
type: kafka
defaultCandidate: true
environment.spring.cloud.stream.kafka.binder.consumerProperties:
dlqProducerProperties.configuration.key.serializer: org.apache.kafka.common.serialization.StringSerializer
dlqProducerProperties.configuration.value.serializer: io.confluent.kafka.serializers.KafkaAvroSerializer
key.deserializer: org.apache.kafka.common.serialization.StringDeserializer
value.deserializer: io.confluent.kafka.serializers.KafkaAvroDeserializer
schema.registry.url: ${SCHEMA_REGISTRY_URL:http://0.0.0.0:55013}
specific.avro.reader: true
useNativeDecoding: true
bindings:
revenueEventConsumer-in-0:
binder: kafka-string-avro-native
destination: email.campaign_revenue_events
group: test-4
consumer:
concurrency: 1
batch-mode: true
use-native-decoding: true
function:
definition: revenueEventConsumer
kafka:
binder:
brokers: 0.0.0.0:55008
i found a workaround for the issue by overriding the Bean TraceFunctionAroundWrapper and overriding the setSkipInputConversion(true)
see code below
#Bean
#Primary
TraceFunctionAroundWrapper customTraceFunctionAroundWrapper(Environment environment, Tracer tracer, Propagator propagator,
Propagator.Setter<MessageHeaderAccessor> injector, Propagator.Getter<MessageHeaderAccessor> extractor,
ObjectProvider<List<FunctionMessageSpanCustomizer>> customizers) {
return new CustomTraceFunctionAroundWrapper(environment, tracer, propagator, injector, extractor,
customizers.getIfAvailable(ArrayList::new));
}
public class CustomTraceFunctionAroundWrapper extends TraceFunctionAroundWrapper {
public CustomTraceFunctionAroundWrapper(Environment environment, Tracer tracer,
Propagator propagator,
Propagator.Setter<MessageHeaderAccessor> injector,
Propagator.Getter<MessageHeaderAccessor> extractor) {
super(environment, tracer, propagator, injector, extractor);
}
public CustomTraceFunctionAroundWrapper(Environment environment, Tracer tracer, Propagator propagator, Propagator.Setter<MessageHeaderAccessor> injector,
Propagator.Getter<MessageHeaderAccessor> extractor,
List<FunctionMessageSpanCustomizer> customizers) {
super(environment, tracer, propagator, injector, extractor, customizers);
}
#Override
protected Object doApply(Object message, SimpleFunctionRegistry.FunctionInvocationWrapper targetFunction) {
targetFunction.setSkipInputConversion(true);
return super.doApply(message, targetFunction);
}
}
this is only a workaround until the bug is fixed is spring cloud stream and sleuth
I am trying to implement Kafka consumer with SSL, provide all the required configurations in the application.yml;
When I start the spring boot Kafka consumer application; Consumer is trying to connect the localhost:9092 instead of mentioned Kafka Brokers.
KafkaConfig.java
#Bean
public ConsumerFactory<String, AvroRecord> consumerFactory() throws IOException {
return new DefaultKafkaConsumerFactory<>(kafkaProps());
}
#Bean
public KafkaListenerContainerFactory<ConcurrentMessageListenerContainer<String, AvroRecord>>
kafkaListenerContainerFactory() throws IOException {
ConcurrentKafkaListenerContainerFactory<String, AvroRecord> factory =
new ConcurrentKafkaListenerContainerFactory<>();
factory.setConsumerFactory(consumerFactory());
return factory;
}
kafkaProps() is loading all the SSL and bootstrap servers related properties. Values, I can see it in the debug mode.
application.yml
kafka:
properties:
basic:
auth:
credentials:
source: USER_INFO
user: username
pass: password
enableAutoRegister: true
max_count: 100
max_delay: 5000
schema:
registry:
url: https://schema-registry:8081
ssl:
truststore:
location: <<location>>
password: pwd
keystore:
location: <<location>>
password: pwd
key:
password: pwd
ssl:
enabled: true
protocols: TLSv1.2,TLSv1.1,TLSv1
truststore:
type: JKS
location: <<location>>
password: pwd
keystore:
type: JKS
location: <<location>>
password: pwd
key:
password: pwd
security:
protocol: SSL
consumer:
bootstrap-servers: broker1:9092,broker2:9092
auto-offset-reset: earliest
key-deserializer: org.apache.kafka.common.serialization.StringDeserializer
value-deserializer: io.confluent.kafka.serializers.KafkaAvroDeserializer
max-message-size: 10241024
In the application logs, I am getting the below log
18:46:33.964 [main] INFO o.a.k.c.a.AdminClientConfig requestId=
transactionKey= | AdminClientConfig values:
bootstrap.servers = [localhost:9092]
client.dns.lookup = use_all_dns_ips
client.id =
connections.max.idle.ms = 300000
15:53:54.608 [kafka-admin-client-thread | adminclient-1] WARN o.a.k.c.NetworkClient requestId=
transactionKey= | [AdminClient clientId=adminclient-1] Connection to node -1 (localhost/127.0.0.1:9092) could not be established. Broker may not be available.
I am not able to find it, why it is connecting to localhost instead of mentioned brokers
The correct property is spring.kafka.bootstrap-servers. You appear to be missing the spring prefix completely. Also, schema.registry.url, ssl.truststore, etc are all considered singular property keys (strings) to Kafka clients, so (to my knowledge) therefore should not be "nested" in YAML objects
You only tried to set the bootstrap property on the consumer, not the AdminClient
Your client will always connect to advertised.listeners of the broker after making the initial connection to the bootstrap server string, so if that is localhost:9092, would explain the AdminClient log output
I am trying to transform with functionnal programming (and spring cloud stream) an input AVRO message from an input topic, and publish a new message on an output topic.
Here is my transform function :
#Bean
public Function<KStream<String, Data>, KStream<String, Double>> evenNumberSquareProcessor() {
return kStream -> kStream.transform(() -> new CustomProcessor(STORE_NAME), STORE_NAME);
}
The CustomProcessor is a class that implements the "Transformer" interface.
I have tried the transformation with non AVRO input and it works fine.
My difficulties is how to declare the schema registry in the application.yaml file or in the the spring application.
I have tried a lot of different configurations (it seems difficult to find the right documentation) and each time the application don't find the settings for the schema.registry.url. I have the following error :
Error creating bean with name 'kafkaStreamsFunctionProcessorInvoker':
Invocation of init method failed; nested exception is
java.lang.IllegalStateException:
org.apache.kafka.common.config.ConfigException: Missing required
configuration "schema.registry.url" which has no default value.
Here is my application.yml file :
spring:
cloud:
stream:
function:
definition: evenNumberSquareProcessor
bindings:
evenNumberSquareProcessor-in-0:
destination: input
content-type: application/*+avro
group: group-1
evenNumberSquareProcessor-out-0:
destination: output
kafka:
binder:
brokers: my-cluster-kafka-bootstrap.kafka:9092
consumer-properties:
value.deserializer: io.confluent.kafka.serializers.KafkaAvroDeserializer
schema.registry.url: http://localhost:8081
I have tried this configuration too :
spring:
cloud:
stream:
kafka:
streams:
binder:
brokers: my-cluster-kafka-bootstrap.kafka:9092
configuration:
schema.registry.url: http://localhost:8081
default.value.serde: io.confluent.kafka.streams.serdes.avro.SpecificAvroSerde
bindings:
evenNumberSquareProcessor-in-0:
consumer:
destination: input
valueSerde: io.confluent.kafka.streams.serdes.avro.SpecificAvroSerde
evenNumberSquareProcessor-out-0:
destination: output
My spring boot application is declared in this way, with the activation of the schema registry client :
#EnableSchemaRegistryClient
#SpringBootApplication
public class TransformApplication {
public static void main(String[] args) {
SpringApplication.run(TransformApplication.class, args);
}
}
Thanks for any help you could bring to me.
Regards
CG
Configure the schema registry under the configuration then it will be available to all binders. By the way. The avro serializer is under the bindings and the specific channel. If you want use the default property default.value.serde:. Your Serde might be the wrong too.
spring:
cloud:
stream:
kafka:
streams:
binder:
brokers: localhost:9092
configuration:
schema.registry.url: http://localhost:8081
default.value.serde: io.confluent.kafka.streams.serdes.avro.SpecificAvroSerde
bindings:
process-in-0:
consumer:
valueSerde: io.confluent.kafka.streams.serdes.avro.SpecificAvroSerde
Don't use the #EnableSchemaRegistryClient. Enable the schema registry on the Avro Serde. In this example, I am using the bean Data of your definition. Try to follow this example here.
#Service
public class CustomSerdes extends Serdes {
private final static Map<String, String> serdeConfig = Stream.of(
new AbstractMap.SimpleEntry<>(SCHEMA_REGISTRY_URL_CONFIG, "http://localhost:8081"))
.collect(Collectors.toMap(Map.Entry::getKey, Map.Entry::getValue));
public static Serde<Data> DataAvro() {
final Serde<Data> dataAvroSerde = new SpecificAvroSerde<>();
dataAvroSerde.configure(serdeConfig, false);
return dataAvroSerde;
}
}
I want to connect to multiple repositories i.e native (file system) and jdbc in spring cloud config. I created a spring cloud config server with below details
application.properties
server.port=8888
spring.profiles.include=native,jdbc
spring.cloud.config.server.native.search-locations=classpath:/config,classpath:/app1, classpath:/app2,classpath:/ep
encrypt.key=abcdef
spring.datasource.driver-class-name=com.mysql.jdbc.Driver
spring.datasource.url=jdbc:mysql://localhost:3306/configuration?useSSL=false
spring.cloud.config.server.jdbc.sql=SELECT properties.key, properties.value from PROPERTIES where APPLICATION=? and PROFILE=? and LABEL=?
spring.datasource.username=root
spring.datasource.password=root
spring.cloud.config.server.native.order=1
spring.cloud.config.server.jdbc.order=2
Irrespective of priority order it always fetches information from jdbc and not from native.
I tried adding the last 2 properties for order to bootstrap.properties still same behavior.
Am is missing anything ? Is my configuration correct ? Please suggest
in spring boostrap.yml loaded before application.yml so you declare server port,config search location and active profile configuration is good approach for this stack,so keep it simple boostrap.yml also spring cloud default profile is native
and in application-"profile".yml is have environment and other configuration properties
and your boostrap.yml or properites like that
server:
port: 8888
spring:
application:
name: appName
profiles:
active: native,jdbc
cloud:
config:
server:
native:
order: 1
searchLocations: classpath:/config,classpath:/app1, classpath:/app2,classpath:/ep
and create applicaiton-jdbc.properties or yml file in same layer in boostrap.yml or properties and declare jdbc properties
spring:
datasource:
driver-class-name: com.mysql.jdbc.Driver
url: 'jdbc:mysql://localhost:3306/configuration?useSSL=false'
cloud:
config:
server:
jdbc:
order: 2
sql: 'SELECT properties.key, properties.value from PROPERTIES where APPLICATION=? and PROFILE=? and LABEL=?'
username: root
password: root
and your config server configuration like this
#SpringBootApplication
#EnableConfigServer
#Import({JdbcEnvironmentRepository.class})
public class ConfigServer {
#ConfigurationProperties(prefix = "spring.datasource")
#Bean
public DataSource dataSource() {
return DataSourceBuilder.create().build();
}
#Bean
public JdbcTemplate jdbcTemplate() {
return new JdbcTemplate(dataSource());
}
public static void main(String[] arguments) {
SpringApplication.run(ConfigServer.class, arguments);
}
}
I'm trying to use Spring Cloud Stream to integrate with Kafka. The message being written is a Java POJO and while it works as expected (the message is being written to the topic and I can read off with a consumer app), there are some unknown characters being added to the start of the message which are causing trouble when trying to integrate Kafka Connect to sink the messages from the topic.
With the default setup this is the message being pushed to Kafka:
contentType "text/plain"originalContentType "application/json;charset=UTF-8"{"payload":{"username":"john"},"metadata":{"eventName":"Login","sessionId":"089acf50-00bd-47c9-8e49-dc800c1daf50","username":"john","hasSent":null,"createDate":1511186145471,"version":null}}
If I configure the Kafka producer within the Java app then the message is written to the topic without the leading characters / headers:
#Configuration
public class KafkaProducerConfig {
#Bean
public ProducerFactory<String, Object> producerFactory() {
Map<String, Object> configProps = new HashMap<>();
configProps.put(
ProducerConfig.BOOTSTRAP_SERVERS_CONFIG,
"localhost:9092");
configProps.put(
ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG,
StringSerializer.class);
configProps.put(
ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG,
JsonSerializer.class);
return new DefaultKafkaProducerFactory<String, Object>(configProps);
}
#Bean
public KafkaTemplate<String, Object> kafkaTemplate() {
return new KafkaTemplate<>(producerFactory());
}
}
Message on Kafka:
{"payload":{"username":"john"},"metadata":{"eventName":"Login","sessionId":"089acf50-00bd-47c9-8e49-dc800c1daf50","username":"john","hasSent":null,"createDate":1511186145471}
Since I'm just setting the key/value serializers I would've expected to be able to do this within the application.yml properties file, rather than doing it through the code.
However, when the yml is updated to specify the serializers it's not working as I would expect i.e. it's not generating the same message as the producer configured in Java (above):
spring:
profiles: local
cloud:
stream:
bindings:
session:
destination: session
contentType: application/json
kafka:
binder:
brokers: localhost
zkNodes: localhost
defaultZkPort: 2181
defaultBrokerPort: 9092
bindings:
session:
producer:
configuration:
value:
serializer: org.springframework.kafka.support.serializer.JsonSerializer
key:
serializer: org.apache.kafka.common.serialization.StringSerializer
Message on Kafka:
"/wILY29udGVudFR5cGUAAAAMInRleHQvcGxhaW4iE29yaWdpbmFsQ29udGVudFR5cGUAAAAgImFwcGxpY2F0aW9uL2pzb247Y2hhcnNldD1VVEYtOCJ7InBheWxvYWQiOnsidXNlcm5hbWUiOiJqb2huIn0sIm1ldGFkYXRhIjp7ImV2ZW50TmFtZSI6IkxvZ2luIiwic2Vzc2lvbklkIjoiNGI3YTBiZGEtOWQwZS00Nzg5LTg3NTQtMTQyNDUwYjczMThlIiwidXNlcm5hbWUiOiJqb2huIiwiaGFzU2VudCI6bnVsbCwiY3JlYXRlRGF0ZSI6MTUxMTE4NjI2NDk4OSwidmVyc2lvbiI6bnVsbH19"
Should it be possible to configure this solely through the application yml? Are there additional settings that are missing?
Credit to #Gary for the answer above!
For completeness, the configuration which is now working for me is below.
spring:
profiles: local
cloud:
stream:
bindings:
session:
producer:
useNativeEncoding: true
destination: session
contentType: application/json
kafka:
binder:
brokers: localhost
zkNodes: localhost
defaultZkPort: 2181
defaultBrokerPort: 9092
bindings:
session:
producer:
configuration:
value:
serializer: org.springframework.kafka.support.serializer.JsonSerializer
key:
serializer: org.apache.kafka.common.serialization.StringSerializer
See headerMode and useNativeEncoding in the producer properties (....session.producer.useNativeEncoding).
headerMode
When set to raw, disables header embedding on output. Effective only for messaging middleware that does not support message headers natively and requires header embedding. Useful when producing data for non-Spring Cloud Stream applications.
Default: embeddedHeaders.
useNativeEncoding
When set to true, the outbound message is serialized directly by client library, which must be configured correspondingly (e.g. setting an appropriate Kafka producer value serializer). When this configuration is being used, the outbound message marshalling is not based on the contentType of the binding. When native encoding is used, it is the responsibility of the consumer to use appropriate decoder (ex: Kafka consumer value de-serializer) to deserialize the inbound message. Also, when native encoding/decoding is used the headerMode property is ignored and headers will not be embedded into the message.
Default: false.
Now, spring.kafka.producer.value-serializer property can be used
yml:
spring:
kafka:
producer:
value-serializer: org.springframework.kafka.support.serializer.JsonSerializer
properties:
spring.kafka.producer.value-serializer=org.springframework.kafka.support.serializer.JsonSerializer