I've configured Micrometer + Prometheus in a Spring Boot application deployed on Kubernetes. This application will have multiple instances that will be measuring some business processes using tags:
process1.calls.total
process1.calls.successful
process1.calls.failed
I'd like to generate a Grafana dashboard that aggregates the metrics across all the instances of the microservice:
instance 1
process1.calls.total=20
process1.calls.successful=10
process1.calls.failed=10
instance 2
process1.calls.total=5
process1.calls.successful=2
process1.calls.failed=3
The dashboard should show:
Process1 Total calls: 25
Process1 Successful calls: 12
Process1 Successful calls: 13
How can I aggregate them?
Related
We have a microservice architecture on the project and we use Prometheus and Grafana for monitoring. The services are implemented using Spring Boot and the integration with Prometheus is through spring-boot-actuator.
There are some Kafka consumers in the project, and for each #KafkaListener spring is generating some metrics. Here is a sample of the Prometheus time series for the metric spring_kafka_listener_seconds_count
spring_kafka_listener_seconds_count{exception="ListenerExecutionFailedException", instance="192.168.100.4:8001", job="spring-actuator", name="org.springframework.kafka.KafkaListenerEndpointContainer#0-0", result="failure"} 2
spring_kafka_listener_seconds_count{exception="none", instance="192.168.100.4:8001", job="spring-actuator", name="org.springframework.kafka.KafkaListenerEndpointContainer#0-0", result="success"} 2
spring_kafka_listener_seconds_count{exception="none", instance="192.168.100.4:8001", job="spring-actuator", name="org.springframework.kafka.KafkaListenerEndpointContainer#1-0", result="success"} 4
org.springframework.kafka.KafkaListenerEndpointContainer#0-0 - doesn't give much info regarding the #KafkaListener method of interest.
Is it possible to configure more meaningful value for the name label of these metrics?
Give the listener a meaningful id.
#KafkaListener(id = "someId", ...)
Note that, by default, the id will be used as the consumer group.id, unless you also specify group, or set idIsGroup to false.
How health indicators should be properly configured for Spring Boot service running on top of Kafka Streams with DB connection? We use Spring Cloud Streams and Kafka Streams binding, Spring-Data JPA, Kubernetes as a container hypervisor. We have let say 3 service replicas and 9 partitions for each topic. A typical service usually joins messages from two topics and persist data in a database and publish data back to another kafka topic.
After switching to Spring Boot 2.3.1 and changing K8s liveness/readiness endpoints to the new ones:
/actuator/health/liveness
/actuator/health/readiness
we discovered that by default they do not have any health indicators included.
According to documentation:
Actuator configures the "liveness" and "readiness" probes as Health
Groups; this means that all the Health Groups features are available
for them. (...) By default, Spring Boot does not add other Health
Indicators to these groups.
I believe that this is the right approach, but I have not tested that:
management.endpoint.health.group.readiness.include: readinessState,db,binders
management.endpoint.health.group.liveness.include: livenessState,ping,diskSpace
We try to cover the following use cases:
rolling update: not available consumption slot (idle instance) when new replica is added
stream has died (runtime exception has been thrown)
DB is not available during container start up / when service is running
broker is not available
I have found a similar question, however I believe the current one is specifically related to Kafka services. They are different in it's nature from REST services.
Update:
In spring boot 2.3.1 binders health indicator checks if streams are in RUNNING or REBALANCING state for Kafka 2.5 (before only RUNNING), so I guess that rolling update case with idle instance is handled by its logic.
I have a question for load balancing in scope of Spring and Ribbon.
I have Microservices architecture with several services. Let's say services: A, B, C and D. All of the services are deployed in the cloud.
In front of the services stays LB that forwards requests to the corresponding service.
All of the services are implemented in Spring Boot.
Docker images are created per each service.Each service is containerised. In my local setup I am able to start all of my services in my local kubernetes cluster. For example:
kubectl get deployment
will result in:
NAME READY UP-TO-DATE AVAILABLE AGE
A 1/1 1 1 9h
B 2/2 2 2 59m
C 1/1 1 1 9h
....
Running in K8S service B can access service A, C or any other service in the namespace with:
public String getResponseFromService() {
return this.restTemplate.getForObject("http://service-a:8080/deals", String.class);
}
If I have N number of instances of service A, a round-robin rule by default is activated and random server is fetched each time when node B invokes service A.
Question:
Does it mean that k8s itself acts as Load Balancer and redirects the requests that are coming from node B to service A to one of the instances?
If the above is true, why I at all need Ribbon client LB. I know that it uses discovery client in order to check with k8s which are registered services in service registry, but if I do not care about the registry do I need the ribbon at all?
I need several instances per each service and communication between services through single endpoint (as example above).
Apologies for the question but I am pretty new to Spring Cloud Kubernete. I read a lot but still can not get this part.
Thanks in advance!
This is probably a rather peculiar question. I am using Spring Boot 2.0.2 with the default Tomcat container. In order to set up a test in our QA environment that simulates many servers, I would like to set up a Spring Boot-based REST service that listens on a very large number of ports simultaneously. I'm able to do this using the technique previously described in in another SO post (Configure Spring Boot with two ports) which basically adds connectors using TomcatServletWebServerFactory.addAdditionalTomcatConnectors().
The difficulty is that a large number of threads seem to be created for each additional port activated; some empirical measurements show the total to be 17 + (15 * number of ports). This means listening on 250 ports result in 3767 threads created and 500 ports result in 7517 threads created and I would like to go somewhat beyond that number. The test program used to take the above measurements is the bare minimum to bring up a Spring service and there is no code that creates threads explicitly so insofar, as I know, all of those threads were created by Spring/Tomcat.
Is there a way to accomplish this using Spring that doesn't use so many threads per active port? Would an alternate container like Jetty be more efficient?
You can configure the embedded tomcat container properties in spring boot configuration file and set the appropriate numbers for these properties to limit the threads created by Tomcat container -
server.tomcat.max-threads=200 # Maximum amount of worker threads.
server.tomcat.min-spare-threads=10 # Minimum amount of worker threads.
Say for example, can we limit the number of requests to be processed by an instance should be 50 and the rest has to be processed by another instance.
Do we require any configuration changes related to ribbon.