Spring Cloud Stream Consumer is not reading messages from KAFKA topic - spring-boot

I am trying to consume a kafka topic from spring boot application. I am using Spring cloud stream with below mentioned version
Spring-boot-starter-parent: 2.5.7
Spring cloud version: 2020.0.4
Below are the code and configuration
application.yml
spring:
zipkin:
sender:
type: kafka
kafka:
bootstrap-servers:
- localhost:19091
cloud:
stream:
bindings:
audit-in-0:
destination: com.tonitingaurav.kafka.log
group: kafka-log-group
consumer:
concurrency: 10
max-attempts: 3
default-binder: kafka
kafka:
binder:
brokers:
- localhost:19091
Message Consumer Class
#Configuration
public class LogConsumer {
#Bean
Consumer<Log> audit(){
return log -> {
System.out.println(log.getMessage());
};
}
}
Below message publisher is publishing the messages properly. Publisher is written in different micro service.
#Component
public class LogEventPublisher {
#Autowired
#Qualifier(LogProducerKafkaConfig.KAFKA_LOG_PUBLISHER)
MessageChannel messageChannel;
public void logMessage(Log log) {
Message<Log> message = MessageBuilder.withPayload(log).build();
messageChannel.send(message);
}
}
pom.xml
<dependency>
<groupId>org.springframework.kafka</groupId>
<artifactId>spring-kafka</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-stream</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-stream-binder-kafka</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-stream-binder-kafka-streams</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-stream-test-support</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.springframework.kafka</groupId>
<artifactId>spring-kafka-test</artifactId>
<scope>test</scope>
</dependency>

You already posted a very similar question here and the response with two different solutions was provided.
Also, here are the samples you can use as starting point -
https://github.com/spring-cloud/spring-cloud-stream-samples

Related

/graphql endpoint returns 404 error during integration test of spring cloud app but not by running the app with mvn spring-boot:run

spring-cloud.version:Greenwich.SR2, spring boot 2.1.7
mvn spring-boot:run makes the endpoint /graphql accessible but not mvn clean verify
I included graphql-java-servlet, javax.servlet-api, graphql-spring-boot-starter-test and graphql-spring-boot-starter, but it stills does not work
<dependency>
<groupId>io.rest-assured</groupId>
<artifactId>rest-assured</artifactId>
<scope>test</scope>
</dependency>
<!-- graphql -->
<dependency>
<groupId>com.graphql-java</groupId>
<artifactId>graphql-java</artifactId>
<version>13.0</version>
</dependency>
<dependency>
<groupId>com.graphql-java</groupId>
<artifactId>graphql-java-extended-scalars</artifactId>
<version>1.0</version>
</dependency>
<dependency>
<groupId>com.graphql-java</groupId>
<artifactId>java-dataloader</artifactId>
<version>2.2.1</version>
</dependency>
<dependency>
<groupId>com.graphql-java-kickstart</groupId>
<artifactId>graphql-spring-boot-starter</artifactId>
<version>5.10.0</version>
</dependency>
<dependency>
<groupId>com.graphql-java</groupId>
<artifactId>graphql-spring-boot-starter-test</artifactId>
<scope>test</scope>
<version>5.0.2</version>
</dependency>
<!-- Apache CXF -->
<dependency>
<groupId>org.apache.cxf</groupId>
<artifactId>cxf-rt-rs-client</artifactId>
<version>${cxf.version}</version>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.jaxrs</groupId>
<artifactId>jackson-jaxrs-json-provider</artifactId>
<version>2.10.0</version>
</dependency>
<dependency>
<groupId>io.rest-assured</groupId>
<artifactId>spring-mock-mvc</artifactId>
<version>3.3.0</version>
<scope>test</scope>
</dependency>
Here is my simple test which returns 404:
#Test
public void persons() throws Exception {
Map<String, Object> variables = new HashMap<String, Object>();
GraphQLRequest request = new GraphQLRequest(
"query persons{persons"
+ "{"
+ " id"
+ "}}",
variables,
null);
// SUT
given()
.contentType(ContentType.JSON)
.body(request)
.get(GRAPHQL_PATH)
.then()
.log().body()
.statusCode(200);
}
The test class is annotated with :
#RunWith(SpringRunner.class)
#SpringBootTest(webEnvironment = WebEnvironment.RANDOM_PORT)
#TestPropertySource(locations = "classpath:bootstrap-test.yml")
Here is my bootstrap.yml:
spring:
application:
name: crm-service
cloud:
config:
uri: http://localhost:8081
fail-fast: false
password: configPassword
username: user
main:
allow-bean-definition-overriding: true #i dont remember why but i think there is a bug with spring cloud and OAuth2ClientContext
acls-management:
permissions-config-path: permissions-config.json
acl-rules-path: acl-rules.json
eureka:
client:
register-with-eureka: false
fetch-registry: false
graphql:
servlet:
mapping: /graphql
enabled: true

A default binder has been requested, but there are no binders available for 'org.springframework.cloud.stream.messaging.DirectWithAttributesChannel'

I am trying to create the simplest as possible hello world with Spring Cloud + Kafka Streams + Spring Boot 2.
I realize I miss basic concepts. Basically, I understand that:
1 - I need to define an outbound stream to write messages to a Kafka topic, and an inbound stream to read messages from a Kafka topic
public interface LoansStreams {
String INPUT = "loans-in";
String OUTPUT = "loans-out";
#Input(INPUT)
SubscribableChannel inboundLoans();
#Output(OUTPUT)
MessageChannel outboundLoans();
}
2 - configure Spring Cloud Stream to bind to my streams
#EnableBinding(LoansStreams.class)
public class StreamsConfig {
}
3 - configure Kafka properties
spring:
cloud:
stream:
kafka:
binder:
brokers: localhost:9092
bindings:
loans-in:
destination: loans
contentType: application/json
loans-out:
destination: loans
contentType: application/json
4 - create model for exchange messages
#Getter #Setter #ToString #Builder
public class Loans {
private long timestamp;
private String result;
}
5 - write to Kafka
#Service
#Slf4j
public class LoansService {
private final LoansStreams loansStreams;
public LoansService(LoansStreams loansStreams) {
this.loansStreams = loansStreams;
}
public void sendLoan(final Loans loans) {
log.info("Sending loans {}", loans);
MessageChannel messageChannel = loansStreams.outboundLoans();
messageChannel.send(MessageBuilder
.withPayload(loans)
.setHeader(MessageHeaders.CONTENT_TYPE, MimeTypeUtils.APPLICATION_JSON)
.build());
}
}
6 - listen to Kafka topic
#Component
#Slf4j
public class LoansListener {
#StreamListener(LoansStreams.INPUT)
public void handleLoans(#Payload Loans loans) {
log.info("Received results: {}", loans);
}
}
I spent a whole day reading few blogs and I assume that the above code is at least workable. I amo not sure I realy coding the best aproach as possible. By the way, I get the error mentioned in the topic:
Error starting ApplicationContext. To display the conditions report re-run your application with 'debug' enabled.
2019-04-26 18:33:05.619 ERROR 14784 --- [ restartedMain] o.s.boot.SpringApplication : Application run failed
org.springframework.context.ApplicationContextException: Failed to start bean 'outputBindingLifecycle'; nested exception is java.lang.IllegalStateException: A default binder has been requested, but there are no binders available for 'org.springframework.cloud.stream.messaging.DirectWithAttributesChannel' : , and no default binder has been set.
Googling for solution, I found someone saying to code StreamListe returning the model so I replaced it with:
#StreamListener(LoansStreams.INPUT)
#SendTo("loans-out")
public KStream<?, Loans> process(KStream<?, Loans> l) {
log.info("Received: {}", l);
return l;
}
and then I get an error even less clear at least to me (previous error clearly mentioned some binder issue):
Error starting ApplicationContext. To display the conditions report re-run your application with 'debug' enabled.
2019-04-26 19:01:06.016 ERROR 13276 --- [ restartedMain] o.s.boot.SpringApplication : Application run failed
java.lang.IllegalArgumentException: Method must be declarative
at org.springframework.util.Assert.isTrue(Assert.java:118) ~[spring-core-5.1.6.RELEASE.jar:5.1.6.RELEASE]
at org.springframework.cloud.stream.binder.kafka.streams.KafkaStreamsStreamListenerSetupMethodOrchestrator.validateStreamListenerMethod(KafkaStreamsStreamListenerSetupMethodOrchestrator.java:510) ~[spring-cloud-stream-binder-kafka-streams-2.1.2.RELEASE.jar:2.1.2.RELEASE]
at org.springframework.cloud.stream.binder.kafka.streams.KafkaStreamsStreamListenerSetupMethodOrchestrator.orchestrateStreamListenerSetupMethod(KafkaStreamsStreamListenerSetupMethodOrchestrator.java:168) ~[spring-cloud-stream-binder-kafka-streams-2.1.2.RELEASE.jar:2.1.2.RELEASE]
at org.springframework.cloud.stream.binding.StreamListenerAnnotationBeanPostProcessor.doPostProcess(StreamListenerAnnotationBeanPostProcessor.java:226) ~[spring-cloud-stream-2.1.2.RELEASE.jar:2.1.2.RELEASE]
In case it helps somehow, I want to evoluate this idea to apply SAGAS but it is not the focus of this question. Firstly, I need get the basic up and running.
*edited
pom.xml
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>2.1.4.RELEASE</version>
<relativePath /> <!-- lookup parent from repository -->
</parent>
<groupId>com.mybank</groupId>
<artifactId>kafka-cloud-stream</artifactId>
<version>0.0.1-SNAPSHOT</version>
<name>kafka-cloud-stream</name>
<description>Spring Cloud Stream With Kafka</description>
<properties>
<java.version>11</java.version>
<spring-cloud.version>Greenwich.SR1</spring-cloud.version>
</properties>
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka-streams</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-stream</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-stream-binder-kafka-streams</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-devtools</artifactId>
<scope>runtime</scope>
</dependency>
<dependency>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
<optional>true</optional>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-stream-test-support</artifactId>
<scope>test</scope>
</dependency>
<!-- https://mvnrepository.com/artifact/org.springframework/spring-web -->
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-web</artifactId>
<!-- version>5.1.5.RELEASE</version-->
</dependency>
</dependencies>
<dependencyManagement>
<dependencies>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-dependencies</artifactId>
<version>${spring-cloud.version}</version>
<type>pom</type>
<scope>import</scope>
</dependency>
</dependencies>
</dependencyManagement>
<build>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
</plugin>
</plugins>
</build>
</project>
"A default binder has been requested, but there are no binders available ...", please add dependency as below.
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-stream-binder-kafka</artifactId>
</dependency>
You can define your default binder in application.yml (or application.properties)
spring:
cloud:
stream:
bindings:
...
default-binder: kafka
For me, with different application.properties for different contexts and multiple output bindings, the only way I could fix it was defining a general default binding like:
spring:
cloud:
stream:
default-binder: eventhub
...
And the rest of bindings type individually set in each input / output as well.
In the above pom file, you need to use binder-kafka and not binder-kafka-streams
So replace
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-stream-binder-kafka-streams</artifactId>
</dependency>
With
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-stream-binder-kafka</artifactId>
</dependency>

How to get http_ metrics from Spring-Boot Api

I have list of metrics from Spring Boot Api but it doesn't contains http_* metrics
my pom.xml contains
<prometheus.version>0.1.0</prometheus.version>
....
<dependency>
<groupId>io.prometheus</groupId>
<artifactId>simpleclient_hotspot</artifactId>
<version>${prometheus.version}</version>
</dependency>
<dependency>
<groupId>io.prometheus</groupId>
<artifactId>simpleclient_servlet</artifactId>
<version>${prometheus.version}</version>
</dependency>
<dependency>
<groupId>io.prometheus</groupId>
<artifactId>simpleclient</artifactId>
<version>${prometheus.version}</version>
</dependency>
I made DefaultExports.initialize() and set bean
#Bean
SpringBootMetricsCollector springBootMetricsCollector(Collection<PublicMetrics> publicMetrics) {
SpringBootMetricsCollector springBootMetricsCollector = new SpringBootMetricsCollector(publicMetrics);
springBootMetricsCollector.register();
return springBootMetricsCollector;
}
then i get list of metrics from spring boot api but it does not contain htpp_ * metrics
like http_request_duration_microseconds
how can i get it?

consul first bootstrap with spring cloud config

Iam using spring-cloud-config for centralized configuration and consul for service discovery. Like eureka first bootstrap - does spring support consul first bootstrap i.e on booting up a client service - I should look up the config server through consul. The otherway round works perfectly fine i.e - in config client bootstrap.properties - I provide the spring.cloud.config.uri=http://localhost:8888 which located the config server and pulls config from it. And in the config repository for my client application - I provide the consul config like :
spring.cloud.consul.host=localhost ,
spring.cloud.consul.port=8500
However, when i try to use consul first bootstrap I am unable to read the properties from config server.
Client Application (for consul first bootstrap):
pom.xml
<parent>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-parent</artifactId>
<!-- <version>Brixton.BUILD-SNAPSHOT</version> -->
<version>Brixton.M5</version>
<relativePath />
</parent>
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-aop</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.retry</groupId>
<artifactId>spring-retry</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-consul-config</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-consul-all</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-zuul</artifactId>
</dependency>
</dependencies>
bootstrap.properties:
spring.application.name=demo
spring.cloud.config.failFast=true
spring.cloud.config.retry.maxAttempts=20
spring.cloud.config.retry.initialInterval=3000
spring.cloud.consul.host=localhost
spring.cloud.consul.port=8500
DemoApplication.java
#EnableDiscoveryClient
#EnableZuulProxy
#SpringBootApplication
public class DemoSleuthApplication {
public static void main(String[] args) {
SpringApplication.run(DemoSleuthApplication.class, args);
}
}
#RefreshScope
#RestController
class RestAPIController
{
#Value(value = "${server.port}")
String port;
#Value(value = "${message}")
String message;
#RequestMapping("/message")
public String welcome(){
String s = this.restTemplate.getForObject("http://localhost:"+this.port+"/message", String.class);
return this.message + s;
}
}
In the consul K/V store
folder structure config/demo
Key/Value : spring.cloud.config.uri=http://localhost:8888
Config server git repo: not adding the config server code for brevity
demo.properties
server.port=9080
message=test
Ideally when I implement the concept of consul first bootstrap - I am thinking consul should be started and the client should identify itself using the #EnableDiscoveryClient annotation and in the consul properties - find config server url , and fetch the config properties from the server configurations. But in my case, service is being discovered and registered in consul but i am not able to read properties from config server git repo.
It was done here. It is available in SNAPSHOTS and in RC2 which will come hopefully next week.
Giving my sample code here for benefit of others. I had to do a lot of tinkering with the properties file to get to this.
As answered by #spencergibb it is available in SNAPSHOT only for now.
This time i did not use any key value properties in consul.
config server code:
pom.xml:
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-config-server</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-consul-all</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-consul-discovery</artifactId>
</dependency>
application.yml
spring:
profiles:
active: native
cloud:
config:
server:
native:
search-locations: file://${HOME}/properties
consul:
port: 8500
host: localhost
enabled: true
discovery:
enabled: true
register: true
service-name: server --registers in consul as server instead of config-server
hostname: localhost
server:
port: 8888
bootstrap.yml ::
spring:
application:
name: config-server
COnfigServerApplication.java
#EnableDiscoveryClient
#EnableConfigServer
#SpringBootApplication
public class SpringConfigServerApplication {
public static void main(String[] args) {
SpringApplication.run(SpringConfigServerApplication.class, args);
}
}
Client microservice: demo
<parent>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-parent</artifactId>
<version>Brixton.BUILD-SNAPSHOT</version>
<!-- <version>Brixton.M5</version> -->
<relativePath />
</parent>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-config</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-consul-all</artifactId>
</dependency>
bootstrap.properties
spring.application.name=demo-spring-cloud-sleuth
spring.cloud.config.failFast=true
spring.cloud.config.retry.maxAttempts=20
spring.cloud.config.retry.initialInterval=3000
spring.cloud.config.enabled=true
spring.cloud.config.discovery.enabled=true
spring.cloud.config.discovery.serviceId=config-server
spring.cloud.consul.discovery.hostName=localhost
spring.cloud.consul.discovery.register=true -- unless this is there, the service fails to register in consul.
git uri property file for client:
server.port=9082
message=message local
foo1=bar

Micro-services: Zuul & consul in Spring cloud application

I'm trying to create a Spring cloud microservice application using Zuul and Consul.
I have 2 components in my project:
api-gateway microservice using Zuul
Hello world microservice (a simple hello world Rest Webservice)
Here is the code of The api-gateway:
#SpringBootApplication
#EnableZuulProxy
#EnableDiscoveryClient
public class ZuulApplication {
public static void main(String[] args) {
SpringApplication.run(ZuulApplication.class, args);
}
}
The pom.xml
<parent>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-parent</artifactId>
<version>Brixton.M3</version>
</parent>
<properties>
<java.version>1.8</java.version>
<spring.cloud.consul.version>1.0.0.M4</spring.cloud.consul.version>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
</properties>
<dependencies>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-commons</artifactId>
</dependency>
<dependency>
<!-- Setup Spring Boot -->
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter</artifactId>
</dependency>
<dependency>
<!-- Setup Spring MVC & REST, use Embedded Tomcat -->
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-config-server</artifactId>
</dependency>
<dependency>
<!-- Spring Cloud starter -->
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-security</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-zuul</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-consul-all</artifactId>
<version>${spring.cloud.consul.version}</version>
</dependency>
</dependencies>
application.yml
zuul:
routes:
hello1:
path: /hello1/**
serviceId: microservice-example
logging:
level:
org.springframework: INFO
com.netflix: DEBUG
bootstrap.yml
spring:
application:
name: edge-server
cloud:
consul:
config:
enabled: true
host: localhost
port: 8500
Here is the code of hello microservice:
#SpringBootApplication
#EnableConfigServer
#EnableDiscoveryClient
#RestController
public class Application {
#RequestMapping(value="/hello1",method = RequestMethod.GET)
public String hello() {
System.out.print("hello1");
return "Hello1";
}
public static void main(String[] args) {
new SpringApplicationBuilder(Application.class).web(true).run(args);
}
}
bootstrap.yml:
spring:
application:
name: microservice-example
profiles:
active: native
cloud:
consul:
config:
enabled: true
host: localhost
port: 8500
But, when I start the api-gateway I got the following exception:
Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.cloud.netflix.zuul.filters.RouteLocator]: Factory method 'routeLocator' threw exception; nested exception is java.lang.IllegalStateException: Unable to locate service in consul agent: edge-server
at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:189) ~[spring-beans-4.2.3.RELEASE.jar:4.2.3.RELEASE]
at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:588) ~[spring-beans-4.2.3.RELEASE.jar:4.2.3.RELEASE]
... 69 common frames omitted
Caused by: java.lang.IllegalStateException: Unable to locate service in consul agent: edge-server
at org.springframework.cloud.consul.discovery.ConsulDiscoveryClient.getLocalServiceInstance(ConsulDiscoveryClient.java:66) ~[spring-cloud-consul-discovery-1.0.0.M4.jar:1.0.0.M4]
This issue is fixed in Brixton.M3 (1.0.0.M5). As mentioned above this was an issue with spring-cloud-consul. The new version is working fine

Resources