I am unable to set group id in Spring cloud stream kafka consumer config using below :-
spring.cloud.stream.default-binder=kafka
spring.cloud.stream.kafka.binder.brokers=${kafka.bootstrap.servers}
spring.cloud.stream.bindings.INPUT.binder=kafka
spring.cloud.stream.bindings.INPUT.destination=datapipeline.ingestion.decision.topic
spring.cloud.stream.bindings.INPUT.content-type=application/json
spring.cloud.stream.bindings.INPUT.group=input-group-1
All above property are getting set except group and getting below in console log while starting service :
group.id = anonymous.cce5a71a-66fa-49c9-874b-09d5685713f7
Kindly help on this as i think because of this my consumer unable to read from where it left.
I am using Spring boot 2.1.3.RELEASE.
Few important dependency related to this, Also I am using Spring integration starter as well :-
<spring-cloud.version>Greenwich.RELEASE</spring-cloud.version>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-stream</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-stream-binder-kafka</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-stream-binder-kafka-streams</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.kafka</groupId>
<artifactId>spring-kafka</artifactId>
</dependency>
Related
I have a Spring boot application with Prometheus Pushgateway using Micrometer, mainly based on this tutorial: https://luramarchanjo.tech/2020/01/05/spring-boot-2.2-and-prometheus-pushgateway-with-micrometer.html
pom.xml has following related dependencies:
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-core</artifactId>
</dependency>
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-registry-prometheus</artifactId>
</dependency>
<dependency>
<groupId>io.prometheus</groupId>
<artifactId>simpleclient_pushgateway</artifactId>
<version>0.16.0</version>
</dependency>
And application.properties file has:
management.metrics.export.prometheus.pushgateway.enabled=true
management.metrics.export.prometheus.pushgateway.shutdown-operation=PUSH
management.metrics.export.prometheus.pushgateway.baseUrl=localhost:9091
This works fine if I leave the application running however with my particular Spring boot application, sometimes it looses the metrics sent just before the shutdown.
I can view the following logs which indicates the PrometheusPushGatewayManager is successfully calling the shutdown() method before the application shuts down which has configured with PUSH operation in the application.properties file as above:
level":"INFO","message":"Shutting down ExecutorService","file":"ExecutorConfigurationSupport.java","line_number":"208","thread_name":"Thread-1","#version":1,"logger_name":"org.springframework.boot.actuate.metrics.export.prometheus.PrometheusPushGatewayManager$PushGatewayTaskScheduler","class":"org.springframework.scheduling.concurrent.ExecutorConfigurationSupport"
I have tried to invoke the shutdown() method on PrometheusPushGatewayManager from my application code but still having the same issue where metrics are not appearing consistently in the Pushgateway/Prometheus (randomly).
Any help or hint would be greatly appreciated it!!
I am using spring boot 2.5.8.
In my application.properties:
resilience4j.retry.configs.default.registerHealthIndicator= true
I get the following error:
'io.github.resilience4j.common.retry.configuration.RetryConfigurationProperties$InstanceProperties' has no property 'register-health-indicator'
<dependency>
<groupId>io.github.resilience4j</groupId>
<artifactId>resilience4j-spring-boot2</artifactId>
</dependency>
<dependency>
<groupId>io.github.resilience4j</groupId>
<artifactId>resilience4j-circuitbreaker</artifactId>
</dependency>
<dependency>
<groupId>io.github.resilience4j</groupId>
<artifactId>resilience4j-timelimiter</artifactId>
</dependency>
If you wanted to see health, events etc. of any resilience4j patterns below config will be sufficient;
management.endpoint.health.show-details = always
It's not a huge problem but I'm curious where some extra stream consumers are coming from, and if that's a setting I can change.
I've got a very simple spring cloud stream consumer setup against a local Kafka broker. Here's the spring config
spring:
cloud:
stream:
bindings:
consumer-in-0:
destination: test-topic
group: test-group
And the consumer class itself:
#Bean
Consumer<Message<String>> consumer() {
return message -> System.out.println("Got it: " + message.getPayload());
}
When I run the app though, I can see 3 consumers created in the output. But when I check the consumer-group members in my local broker, it's always just one consumer, and it's always the second consumer created (i.e. with client id test-group-2)
Just for clarity, I'm using Spring Boot version 2.3.4.RELEASE and cloud dependencies version Hoxton.SR10.
And here's the dependencies in the pom:
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-devtools</artifactId>
<scope>runtime</scope>
<optional>true</optional>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-stream</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-stream-binder-kafka</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.kafka</groupId>
<artifactId>spring-kafka</artifactId>
</dependency>
<dependencies>
Why am I getting 3 consumers? Why is the second one the only one that actually listens on the Kafka topic?
During start up, a temporary consumer is created to get information about the partitions provisioned for the topic.
The second consumer is the real consumer.
If you have the actuator (actually Micrometer) on the classpath the KafkaBinderMetrics creates another consumer so it can calculate the lag. It does not actually consume anything.
I am using elasticsearch7.5. I have turned on the following properties
xpack.security.enabled: true
xpack.security.authc.api_key.enabled: true
I have generated API keys to connect our springboot to elasticsearch.I am using
spring-boot-starter-data-elasticsearch
All my requests are from "RestHighClient" and able to trigger the requests.Only problem is during server startup, they are some errors where it is not able to connect to Elasticsearch.
org.elasticsearch.transport.RemoteTransportException: [ADMIN-PC][127.0.0.1:9300][cluster:monitor/nodes/liveness]
Caused by: org.elasticsearch.ElasticsearchSecurityException: missing authentication credentials for action [cluster:monitor/nodes/liveness]
In pom.xml
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-elasticsearch</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-security</artifactId>
</dependency>
<dependency>
<groupId>org.elasticsearch.client</groupId>
<artifactId>transport</artifactId>
<version>6.8.3</version>
</dependency>
in application properties:
spring.elasticsearch.server=localhost:9200
spring.data.elasticsearch.cluster-name=elasticsearch
spring.data.elasticsearch.cluster-nodes=127.0.0.1:9300
spring.data.elasticsearch.repositories.enabled=true
Can someone suggest me ,How I can fix it.
Remove these lines from your configuration:
spring.data.elasticsearch.cluster-name=elasticsearch
spring.data.elasticsearch.cluster-nodes=127.0.0.1:9300
These make Spring Boot to configure the transport client.
Written simple POC to prove and test Spring Boot and log4j2 compatibility. Once successful I will move it to real application.
Please refer my POC source code:
https://github.com/Dennyss/SpringBootLog4j2POC
I know/read about Spring version and log4j2 compatibility from:
How to set up Spring Boot and log4j2 properly?
Found and tried recommendations described here:
Spring-Boot logging with log4j2?
But still not working. The problem is that both application logs and Spring logs are printing to console only.
Please refer maven dependencies below (from POC):
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
<exclusions>
<exclusion>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-logging</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-log4j2</artifactId>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-core</artifactId>
<version>2.6.2</version>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-api</artifactId>
<version>2.6.2</version>
</dependency>
</dependencies>
If I don't exclude Spring's logback and don't add boot-starter-log4j2 then application logs are printing to application file but Spring logs are not printing at all. I feel the problem somewhere with dependencies. Appreciate any help.
According to the Spring Boot Logging documentation, the location of the logging configuration file can be specified using the logging.config property. I noticed that your start.sh script is passing -Dlog4j.configurationFile. Normally, this would be sufficient for direct Log4J 2 integration, but Spring Boot uses logging.config.
After switching to this, it writes to the log files:
java -Dlogging.config=/share/LogPOC/log4j2.xml -cp poc.jar:libs/* com.poc.logger.Application