Spring cloud stream multi binder - Error KStreamBinderConfiguration required a single bean, but 2 were found - spring-boot

I have a use case where a single kafka stream processing MS will have a processor and a consumer which will consume the processor output message. Similar to below sample in github
https://github.com/spring-cloud/spring-cloud-stream-samples/tree/master/kafka-streams-samples/kafka-streams-message-channel
while executing the above sample I getting following error
***************************
APPLICATION FAILED TO START
***************************
Description:
Parameter 0 of method provisioningProvider in org.springframework.cloud.stream.binder.kafka.streams.KStreamBinderConfiguration required a single bean, but 2 were found:
- kafkaBinderConfigurationProperties: defined by method 'kafkaBinderConfigurationProperties' in org.springframework.cloud.stream.binder.kafka.streams.MutliBinderPropertiesConfiguration
- binderConfigurationProperties: defined by method 'binderConfigurationProperties' in class path resource [org/springframework/cloud/stream/binder/kafka/streams/KafkaStreamsBinderSupportAutoConfiguration.class]
Action:
Consider marking one of the beans as #Primary, updating the consumer to accept multiple beans, or using #Qualifier to identify the bean that should be consumed

This error occurred for me while I was using #SpringBootApplication and #EnableBinding( Processor.class ) outside of the class that was running my project, in conjunction with both of the dependencies below.
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-stream-binder-kafka</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-stream-binder-kafka-streams</artifactId>
</dependency>
Both of these dependencies provide a configuration for provisioningProvider, so to resolve the issue, we must specify which configuration to use.
In your application.yml file, set a default binder to resolve this issue.
spring:
cloud:
stream:
kafka:
binder:
brokers: localhost
defaultBrokerPort: 9092
bindings:
output:
binder: name-of-target-binder
destination: sample-topic
defaultBinder: name-of-target-binder
The documentation here, Section 7.4 Multiple Binders on the Classpath, provides further insight into this issue.

This seems to be some kind of screw-up from the side of Spring Cloud.
This cryptic thread proposed downgrading the dependencies (org.springframework.cloud:spring-cloud-stream-binder-kafka-streams and org.springframework.cloud:spring-cloud-stream for me) from 3.0.1.RELEASE to 3.0.0.RELEASE.
This was indeed the fix for my version of this problem, and I guess it is for you, too.

I had a similar issue, and I fixed it by adding in my pom:
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-actuator</artifactId>
</dependency>

Related

Spring Cloud Kafka Stream Multibinder BeanCreationException: Error creating bean with name 'kafkaStreamsFunctionProcessorInvoker'

I am trying to create a spring boot application using spring cloud kafka stream which reads input from kafka cluster 1 and send it to kafka cluster 2 using single kafkastream application.
I am getting following exception during startup.
***************************
APPLICATION FAILED TO START
***************************
Description:
A component required a bean named 'sessionBrokers-KafkaStreamsBinderConfigurationProperties' that could not be found.
Action:
Consider defining a bean named 'sessionBrokers-KafkaStreamsBinderConfigurationProperties' in your configuration.
My configuration is as below
spring.cloud.function.definition=itemConsumedStream
#Input Configuration
spring.cloud.stream.bindings.itemConsumedStream-in-0.binder=sessionBrokers
spring.cloud.stream.bindings.itemConsumedStream-in-0.destination=${session.event.data.input.topic}
spring.cloud.stream.kafka.bindings.itemConsumedStream-in-0.consumer.configuration.key.deserializer=org.apache.kafka.common.serialization.StringSerializer
spring.cloud.stream.kafka.bindings.itemConsumedStream-in-0.consumer.configuration.value.deserializer=in.custom.JsonNodeDeserializer
#Output Configuration
spring.cloud.stream.bindings.itemConsumedStream-out-0.binder=searchBrokers
spring.cloud.stream.bindings.itemConsumedStream-out-0.destination=${itemConsumed.events.output.topic}
spring.cloud.stream.kafka.bindings.itemConsumedStream-out-0.producer.configuration.key.serializer=org.apache.kafka.common.serialization.StringSerializer
spring.cloud.stream.kafka.bindings.itemConsumedStream-out-0.producer.configuration.value.serializer=in.custom.JsonSerializer
#Input Binder Configurations
spring.cloud.stream.binders.sessionBrokers.type=kafka
spring.cloud.stream.binders.sessionBrokers.environment.spring.cloud.stream.kafka.streams.binder.brokers=${session.event.data.input.kafka.brokers}
#Output Binder Configurations
spring.cloud.stream.binders.searchBrokers.type=kafka
spring.cloud.stream.binders.searchBrokers.environment.spring.cloud.stream.kafka.binder.brokers=${search.kafka.brokers}
pom dependencies are
spring boot version
2.7.1
spring cloud version
2021.0.0
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka-streams</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-stream</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-stream-binder-kafka-streams</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-stream-binder-kafka</artifactId>
</dependency>
This is working if I use single binder ie
spring.cloud.stream.kafka.streams.binder.brokers=${session.event.data.input.kafka.brokers}
instead of multiple binders

Spring boot 2.x with Camel 2.25 : Spring specific endpoints not working

I have a project with spring-boot 2.x and camel 2.25. It has different camel routes along with few REST consumer routes. Everything is good till this point.
Now I added few normal spring-boot #RestController classes with some endpoints. But these are not working (throwing 404).
When I investigated I found, that every request is coming to CamelServlet which is totally unaware of spring based normal #RestController endpoints (but knows only Camel REST consumer route endpoints). Hence throwing this error for only #RestController endpoints whereas Camel REST endpoints are still working.
Below is my configuration,
spring:
application:
name: gateway
main:
web-application-type: SERVLET
server:
servlet:
context-path: /gateway
port: 8080
camel:
springboot:
name: gateway
component:
servlet:
mapping:
enabled: true
context-path: /*
mail:
basic-property-binding: true
Below is my POM
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-validation</artifactId>
</dependency>
<dependency>
<groupId>org.apache.camel</groupId>
<artifactId>camel-spring-boot-starter</artifactId>
</dependency>
<dependency>
<groupId>org.apache.camel</groupId>
<artifactId>camel-servlet-starter</artifactId>
<scope>runtime</scope>
</dependency>
<dependency>
<groupId>org.apache.camel</groupId>
<artifactId>camel-mail-starter</artifactId>
</dependency>
Is there anything I am doing wrong? Any suggestion? Thanks in advance.
This is because of you set context-path: /* pattern means camel is going intercept(because this path is register with camel) it, before spring servlet dispatcher gonna handle it, so if you want to handle #Restcontroller then you need to define a separate context path for camel, example: context-path: camel-api/* pattern, now camel will register camel-api base route, and if the pattern is different from camel-api URL, it will handle by spring-boot
#Bean
ServletRegistrationBean servletRegistrationBean() {
ServletRegistrationBean servlet = new ServletRegistrationBean
(new CamelHttpTransportServlet(), "camel-api/*");
servlet.setName("CamelServlet");
return servlet;
}
or configure using properties.

How to replace default hikari cp to tomcat pool on spring boot 2.0

I have migrated spring boot application to 2.0 and found out some problems with hikari connection pool. When I am fetching database data this results to hikari cp timeout ie. connection is not available. I don't know why when in the previous version this worked correctly.
Therefore I tried to use tomcat pool with this config in application.yml but it did not work (in correct YAML formatting).
spring.datasource.type=org.apache.tomcat.jdbc.pool.DataSource
My pom.xml has these dependencies related to DB things:
spring-boot-jpa
spring-boot-jdbc
jdbc7
How to exclude hikari and use tomcat connection pool?
I have found out the solution.
This can be resolved in pom.xml by modifying like that:
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-jdbc</artifactId>
<exclusions>
<exclusion>
<groupId>com.zaxxer</groupId>
<artifactId>HikariCP</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.apache.tomcat</groupId>
<artifactId>tomcat-jdbc</artifactId>
</dependency>
However the hikari problem was probably with default small size of connection pool. So this problem could be resolved also with this change but not verified by myself. Just note for others. Something like that:
spring.datasource.hikari.connection-timeout=60000
spring.datasource.hikari.maximum-pool-size=5
Since Spring Boot 2.0 release, spring-boot-starter-jdbc and spring-boot-starter-data-jpa resolve HikariCP dependency by default and spring.datasource.type property has HikariDataSource as default value.So if u have both dependency in your application you should exclude it from both like below.
implementation('org.springframework.boot:spring-boot-starter-data-jpa') {
exclude group: 'com.zaxxer', module: 'HikariCP'
}
implementation('org.springframework.boot:spring-boot-starter-jdbc') {
exclude group: 'com.zaxxer', module: 'HikariCP'
}
After that you can configure other pooling technologies that u likes to use, like below
.
In your application.yml file :
spring:
datasource:
type: org.apache.tomcat.jdbc.pool.DataSource
In dependency :
implementation('org.apache.tomcat:tomcat-jdbc')
Also:
spring:
datasource:
type: org.apache.tomcat.jdbc.pool.DataSource
works with
<dependency>
<groupId>org.apache.tomcat</groupId>
<artifactId>tomcat-jdbc</artifactId>
</dependency>

Spring boot failing to start with Spring Cloud AWS Core dependency

I am writing a spring boot app that accesses stuff from an s3 bucket, but I get a NoClassDefFoundError when I use the starter spring-cloud-starter-aws dependency from the spring initializer.
Am I missing some other dependency here?
Below are my dependencies.
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-amqp</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-aws</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-cassandra</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-integration</artifactId>
</dependency>
<dependency>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
<optional>true</optional>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
</dependencies>
I also defined the dependencyManagement block for spring-cloud-dependencies and use Edgware.SR1 as my spring-cloud-version.
My app fails with the following error when starting up.
2018-01-24 12:20:25.642 INFO 1980 --- [ main] utoConfigurationReportLoggingInitializer :
Error starting ApplicationContext. To display the auto-configuration report re-run your application with 'debug' enabled.
2018-01-24 12:20:25.666 ERROR 1980 --- [ main] o.s.boot.SpringApplication : Application startup failed
org.springframework.beans.factory.BeanDefinitionStoreException: Failed to process import candidates for configuration class [com.art.productattribution.consumerintegration.ConsumerIntegrationApplication]; nested exception is java.lang.NoClassDefFoundError: com/amazonaws/AmazonClientException
at org.springframework.context.annotation.ConfigurationClassParser.processImports(ConfigurationClassParser.java:616) ~[spring-context-4.3.13.RELEASE.jar:4.3.13.RELEASE]
Not sure what am I missing here? Please let me know if you need any more details with this. The version of spring boot I am using is 1.5.9.RELEASE
The correct dependency is the spring-cloud-aws-context. Add the following in your pom file (version 1.2.2 as of Nov 22, 2017):
<!-- https://mvnrepository.com/artifact/org.springframework.cloud/spring-cloud-aws-context -->
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-aws-context</artifactId>
<version>1.2.2.RELEASE</version>
</dependency>
Below are the Spring Cloud AWS modules:
Spring Cloud AWS Core is the core module of Spring Cloud AWS providing basic services for security and configuration setup. Developers will not use this module directly but rather through other modules. The core module provides support for cloud based environment configurations providing direct access to the instance based EC2 metadata and the overall application stack specific CloudFormation metadata.
Spring Cloud AWS Context delivers access to the Simple Storage Service via the Spring resource loader abstraction. Moreover developers can send e-mails using the Simple E-Mail Service and the Spring mail abstraction. Further the developers can introduce declarative caching using the Spring caching support and the ElastiCache caching service.
Spring Cloud AWS JDBC provides automatic datasource lookup and configuration for the Relational Database Service which can be used with JDBC or any other support data access technology by Spring.
Spring Cloud AWS Messaging enables developers to receive and send messages with the Simple Queueing Service for point-to-point communication. Publish-subscribe messaging is supported with the integration of the Simple Notification Service.
Ref: http://cloud.spring.io/spring-cloud-aws/spring-cloud-aws.html#_using_amazon_web_services
It seems com.amazonaws.AmazonClientException is not found in the classpath. I think you can add the following dependency in your POM.xml file to solve this issue.
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-aws-core</artifactId>
<version>1.2.2.RELEASE</version>
</dependency>
You need to include dependency mentioned by #alltej. Also you need to add below property in application.properties file if you are running on local.
cloud.aws.stack.auto=false

Spring Cloud Contract producer cannot send message

I'm trying to incorporate Spring Cloud Contract into an existing project. I've had some success with REST but I'm struggling to set up the messaging side.
Thus far I've set up a contract on the producer, which does produce a test in target/generated-test-sources/contracts. I've also set up a base class for the test.
I can't get past this error:
2017-09-08 17:10:51.759 ERROR - --[]- [ main] o.s.c.c.v.m.stream.StreamStubMessages : Exception took place while trying to resolve the destination. Will assume the name [invites]
org.springframework.beans.factory.NoSuchBeanDefinitionException: No qualifying bean of type 'org.springframework.cloud.stream.config.ChannelBindingServiceProperties' available
at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBean(DefaultListableBeanFactory.java:353)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBean(DefaultListableBeanFactory.java:340)
at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1093)
at org.springframework.cloud.contract.verifier.messaging.stream.StreamStubMessages.resolvedDestination(StreamStubMessages.java:86)
at org.springframework.cloud.contract.verifier.messaging.stream.StreamStubMessages.receive(StreamStubMessages.java:73)
at org.springframework.cloud.contract.verifier.messaging.stream.StreamStubMessages.receive(StreamStubMessages.java:110)
at org.springframework.cloud.contract.verifier.messaging.stream.StreamStubMessages.receive(StreamStubMessages.java:36)
at org.springframework.cloud.contract.verifier.messaging.internal.ContractVerifierMessaging.receive(ContractVerifierMessaging.java:40)
at org.springframework.cloud.contract.verifier.tests.email.MessagingTest.validate_invitedContract(MessagingTest.java:27)
and later
2017-09-08 17:10:51.759 ERROR - --[]- [ main] o.s.c.c.v.m.stream.StreamStubMessages : Exception occurred while trying to read a message from a channel with name [invites]
org.springframework.beans.factory.NoSuchBeanDefinitionException: No bean named 'invites' available
at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeanDefinition(DefaultListableBeanFactory.java:687)
at org.springframework.beans.factory.support.AbstractBeanFactory.getMergedLocalBeanDefinition(AbstractBeanFactory.java:1207)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:284)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
My application.yml file under src/test/resources:
spring:
cloud:
stream:
bindings:
output:
content-type: application/json
destination: invites
I have the following dependencies:
`
<!-- Spring Cloud Contract Deps -->
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-stream</artifactId>
<version>1.2.2.RELEASE</version>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-contract-verifier</artifactId>
<version>1.1.3.RELEASE</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-stream-test-support</artifactId>
<version>1.2.2.RELEASE</version>
<scope>test</scope>
</dependency>
<!-- END SCC Deps -->`
I've combed through the docs, watched Marcin's talk and looked through the samples for Spring Cloud Contract but I'm stuck. Any help would be greatly appreciated.
First issue:
Please use the release train. In the release train, we know that there are no invalid dependencies.
Second issue:
You don't have the #EnableBinding(Source.class) annotation. That's why Stream doesn't know how to bind to the output channel.
If you go to the Spring Cloud Contract samples you'll notice this line on the main application class (https://github.com/spring-cloud-samples/spring-cloud-contract-samples/blob/master/producer/src/main/java/com/example/ProducerApplication.java#L9). After I've added this line to your code, the context started but the tests failed cause the message wasn't sent.

Resources