We have a microservice architecture on the project and we use Prometheus and Grafana for monitoring. The services are implemented using Spring Boot and the integration with Prometheus is through spring-boot-actuator.
There are some Kafka consumers in the project, and for each #KafkaListener spring is generating some metrics. Here is a sample of the Prometheus time series for the metric spring_kafka_listener_seconds_count
spring_kafka_listener_seconds_count{exception="ListenerExecutionFailedException", instance="192.168.100.4:8001", job="spring-actuator", name="org.springframework.kafka.KafkaListenerEndpointContainer#0-0", result="failure"} 2
spring_kafka_listener_seconds_count{exception="none", instance="192.168.100.4:8001", job="spring-actuator", name="org.springframework.kafka.KafkaListenerEndpointContainer#0-0", result="success"} 2
spring_kafka_listener_seconds_count{exception="none", instance="192.168.100.4:8001", job="spring-actuator", name="org.springframework.kafka.KafkaListenerEndpointContainer#1-0", result="success"} 4
org.springframework.kafka.KafkaListenerEndpointContainer#0-0 - doesn't give much info regarding the #KafkaListener method of interest.
Is it possible to configure more meaningful value for the name label of these metrics?
Give the listener a meaningful id.
#KafkaListener(id = "someId", ...)
Note that, by default, the id will be used as the consumer group.id, unless you also specify group, or set idIsGroup to false.
Related
I'm experiencing a problem, I have an application in spring boot, this application consumes from one topic and produces in another topic.
the topic that the application consumes is on-premises, the topic that the application produces is on cloud aws.
is there a way to specify a bootstrap server and schema registry for each topic?
my application.properties has the following property:
spring.kafka.bootstrap-servers=localhost:32202 spring.kafka.properties.schema.registry.url=127.0.0.1:8082
the problem here is that these properties are for both consumer and producer.
I need to specify a bootstrap server for the consumer, another for the producer.
also specify a schema-registry for the consumer, another for the producer.
I don't know if this way is the best way to deal with this problem.
spring.kafka.consumer.bootstrap-servers=consumer-localhost:32202 spring.kafka.consumer.schema.registry.url=consumer-127.0.0.1:8082 spring.kafka.producer.bootstrap-servers=producer-localhost:10010 spring.kafka.producer.schema.registry.url=producer-127.0.0.1:9090
Thanks in advance!
See the Spring Boot documentation.
The properties supported by auto configuration are shown in the “Integration Properties” section of the Appendix. Note that, for the most part, these properties (hyphenated or camelCase) map directly to the Apache Kafka dotted properties. See the Apache Kafka documentation for details.
The first few of these properties apply to all components (producers, consumers, admins, and streams) but can be specified at the component level if you wish to use different values. Apache Kafka designates properties with an importance of HIGH, MEDIUM, or LOW. Spring Boot auto-configuration supports all HIGH importance properties, some selected MEDIUM and LOW properties, and any properties that do not have a default value.
Only a subset of the properties supported by Kafka are available directly through the KafkaProperties class. If you wish to configure the producer or consumer with additional properties that are not directly supported, use the following properties:
spring.kafka.properties[prop.one]=first
spring.kafka.admin.properties[prop.two]=second
spring.kafka.consumer.properties[prop.three]=third
spring.kafka.producer.properties[prop.four]=fourth
spring.kafka.streams.properties[prop.five]=fifth
So...
spring.kafka.consumer.bootstrap-servers=consumer-localhost:32202
spring.kafka.consumer.properties.schema.registry.url=consumer-127.0.0.1:8082
spring.kafka.producer.bootstrap-servers=producer-localhost:10010
spring.kafka.producer.properties.schema.registry.url=producer-127.0.0.1:9090
This sets the common prop.one Kafka property to first (applies to producers, consumers and admins), the prop.two admin property to second, the prop.three consumer property to third, the prop.four producer property to fourth and the prop.five streams property to fifth.
I have a problem to add a custom metrics in Kafka Streams.
I made a Kafka Streams application with Spring Boot like this. (Kafka Streams with Spring boot. Baeldung)
and deployed several of this app on k8s.
I want to know about avg number of processd message per second of each app instance. and it exists in Kafka Streams built-in thread metrics(process-rate). (ref. Kafka Streams Metrics)
But, that metric use thread-id at tag key and so each app instance has different metric tag key.
I'd like to use that metric value as the same tag key in each app instance.
So, I came up with a solution. It's about using that built-in metric value to add a new custom metric.
But, There's no specific information about how I get built-in metric values in source code and add a custom metric..
In ref, there's a way to add a custom metrics but no specific information about how can I apply in source code.
Is there a way to solve this problem? Or is there any other way?
I have an application running Kafka consumers and want to monitor the processing time of each message consumed from the topic. The application is a Spring boot application and exposes Kafka consumer metrics to Spring Actuator Prometheus endpoint using micrometre registry.
Can I use kafka_consumer_commit_latency_avg_seconds or kafka_consumer_commit_latency_max_seconds to monitor or alert?
Those metrics have nothing to do with record processing time. spring-kafka provides metrics for that; see here.
Monitoring Listener Performance
Starting with version 2.3, the listener container will automatically create and update Micrometer Timer s for the listener, if Micrometer is detected on the class path, and a single MeterRegistry is present in the application context. The timers can be disabled by setting the ContainerProperty micrometerEnabled to false.
Two timers are maintained - one for successful calls to the listener and one for >failures.
How health indicators should be properly configured for Spring Boot service running on top of Kafka Streams with DB connection? We use Spring Cloud Streams and Kafka Streams binding, Spring-Data JPA, Kubernetes as a container hypervisor. We have let say 3 service replicas and 9 partitions for each topic. A typical service usually joins messages from two topics and persist data in a database and publish data back to another kafka topic.
After switching to Spring Boot 2.3.1 and changing K8s liveness/readiness endpoints to the new ones:
/actuator/health/liveness
/actuator/health/readiness
we discovered that by default they do not have any health indicators included.
According to documentation:
Actuator configures the "liveness" and "readiness" probes as Health
Groups; this means that all the Health Groups features are available
for them. (...) By default, Spring Boot does not add other Health
Indicators to these groups.
I believe that this is the right approach, but I have not tested that:
management.endpoint.health.group.readiness.include: readinessState,db,binders
management.endpoint.health.group.liveness.include: livenessState,ping,diskSpace
We try to cover the following use cases:
rolling update: not available consumption slot (idle instance) when new replica is added
stream has died (runtime exception has been thrown)
DB is not available during container start up / when service is running
broker is not available
I have found a similar question, however I believe the current one is specifically related to Kafka services. They are different in it's nature from REST services.
Update:
In spring boot 2.3.1 binders health indicator checks if streams are in RUNNING or REBALANCING state for Kafka 2.5 (before only RUNNING), so I guess that rolling update case with idle instance is handled by its logic.
I am running a Kafka Stream app with Springboot 2.
I would like to have my kafka stream metrics available in the prometheus format at host:8080/actuator/prometheus
I don't manage to have this. I am not sure I understand how kafka stream metrics are exported.
Can actuator get these JMX metrics ?
Is there a way to get these metrics and expose them in Prometheus format ?
PS: didn't worked with java jmx_prometheus_agent neither
Does someone has a solution or an example ?
Thank you
You could produce all available Kafka-Streams metrics (the same as from KafkaStreams.metrics()) into Prometheus using micrometer-core and spring-kafka libraries. For integrating Kafka-Streams with micrometer, you could have KafkaStreamsMicrometerListener bean:
#Bean
KafkaStreamsMicrometerListener kafkaStreamsMicrometerListener(MeterRegistry meterRegistry) {
return new KafkaStreamsMicrometerListener(meterRegistry);
}
where MeterRegistry is from micrometer-core dependency.
If you create Kafka Streams using StreamsBuilderFactoryBean from spring-kafka, then you need to add listener into it:
streamsBuilderFactoryBean.addListener(kafkaStreamsMicrometerListener);
And if you create KafkaStreams objects directly, then on each KafkaStreams object you need to invoke
kafkaStreamsMicrometerListener.streamsAdded(beanId, kafkaStreams);
where beanId is any unique identifier per KafkaStreams object.
As a result, Kafka Streams provides multiple useful Prometheus metrics, like kafka_consumer_coordinator_rebalance_latency_avg, kafka_stream_thread_task_closed_rate, etc. KafkaStreamsMicrometerListener under the hood uses KafkaStreamsMetrics.
If you need to have Grafana Prometheus graphs with these metrics, you need to add them as Gauge metric type.
I don't have a complete example, but metrics are well accessible and documented in Confluent documentation on Monitoring Kafka Streams.
Maybe dismiss actuator and use #RestController from Spring Web along with KafkaStreams#metrics() to publish exactly what you need.