Is there any way to add `application` tag to standard prometheus micrometer metrics? - spring-boot

I have few spring-boot microservices with actuator and exposed prometheus metrics. For example:
# HELP process_uptime_seconds The uptime of the Java virtual machine
# TYPE process_uptime_seconds gauge
process_uptime_seconds 3074.971
But there is no application tag, so I'm not able to bind it to a certain application within a grafana dashboard...
Also I expect to have few application instances of some microservice, so in general it would be great to add an instance tag also.
Is there any way to customize the standard metrics with these tags?

The best way to add tags is to use the Prometheus service discovery. This keeps these tags out of your application code and keeps it from being concerned about where it exists.
However sometime if you absolutely need those extra tags (due to the service having extra insight that Prometheus service discovery isn't surfacing) you can't use the Java Simple Client (the Go client does support this though)
I turns out this feature is offered via a Micrometer feature called 'Common Tags' which wraps the Prometheus Java client. You setup your client so the tags are available via a config() call.
registry.config().commonTags("stack", "prod", "region", "us-east-1");

What I am usually doing is using Maven filtering on the resource files (e.g. application.yml) which will replace Maven-known properties like project.artifactId. Springs configuration then takes care of interpolating management.metrics.tags.application.
An application.yml example:
spring:
application:
name: ${project.artifactId}
management:
metrics:
tags:
application: ${spring.application.name}

Related

Scale SpringBoot App based on Thread Pool State

We have a Spring Boot microservice which should get some data from old / legacy system. This microservice exposes external modern REST API. Sometimes we have to issue 7-10 requests to the legacy system in order to get all the data we need for single API call. Unfortunately we can't use Reactor / WebClient and have to stick with WebServiceTemplate to issue those "legacy" calls. We can't also use Reactive Spring WebClient - Making a SOAP call
What is the best way to scale such a miroservice in Kubernetes? We have very big concerns that Thread Pool used for parallel WebServiceTemplate invocation will be depleted very fast, but I'm not sure that creating and exposing custom metric based on active threads count / thread pool size is a good idea.
Any advice will be helpful.
Enable Prometheus exporter in Spring
Make sure metrics are scraped. You're going to watch for a threadpool_size metric. Refer your k8s/prometheus distro docs to get prometheus service discovery working for you.
Write a horizontal pod autoscaler (HPA) based on a Prometheus metric:
Setup Prometheus-Adapter and follow the HPA walkthrough.
Or follow this guide https://github.com/stefanprodan/k8s-prom-hpa
Depending on what k8s distro you are using, you might have different ways to get the Prometheus and prometheus discovery:
(example platform built-in) https://cloud.google.com/stackdriver/docs/solutions/gke/prometheus
(example product) https://docs.datadoghq.com/integrations/prometheus/
(example opensource) https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack
any other prometheus solution

Grafana not showing metrics shown by springboot prometheus endpoint

I have configured micrometer and prometheus for my spring boot application, and I can see following metrics(genrated using Timer) at endpoint /actuator/prometheus:
# HELP timer_test_seconds Time spent serving test requests
# TYPE timer_test_seconds summary
timer_test_seconds_count{class="com.test.MyController",exception="none",method="testTimer",} 2.0
timer_test_seconds_sum{class="com.test.MyController",exception="none",method="testTimer",} 8.6461705
# HELP timer_test_seconds_max Time spent serving test requests
# TYPE timer_test_seconds_max gauge
timer_test_seconds_max{class="com.test.MyController",exception="none",method="testTimer",} 4.5578234
But when I run one of this query in Grafana(configured against prometheus instance), I don't see any result.
Is any configuration needed for this?
These are the metrics exposed by your service. Grafana does not interact with your service directly.
Is Prometheus up and scraping this target? You can check if Prometheus has any data at localhost:port/graph.
Is Grafana pointing to your Prometheus endpoint correctly?
Most likely, Prometheus is not scraping your target correctly. Keep in mind that Grafana is unaware of your service at all so you should first check if there is any data in Prometheus.

Spring Cloud Connector Plan Information

I am using Spring Cloud Connector to bind to databases. Is there any way to get the plan of the bound service? When I extend an AbstractCloudConfig and do
cloud().getSingletonServiceInfosByType(PostgresqlServiceInfo.class)...
I will have information on the url and how to connect to the postgres. PostgresqlServiceInfo and others do not carry along the plan data. How can I extend the service info, in order to read this information form VCAP_SERVICES?
Thanks
By design, the ServiceInfo classes in Spring Cloud Connectors carry just enough information to create the connection beans necessary for an app to consume the service resources. Connectors was designed to be platform-neutral, and fields like plan, label, and tags that are available on Cloud Foundry are not captured because they might not be available on other platforms (e.g. Heroku).
To add the plan information to a ServiceInfo, you'd need to write your own ServiceInfo class that includes a field for the value, then write a CloudFoundryServiceInfoCreator to populate the value from the VCAP_SERVICES data that the framework provides as a Map. See the project documentation for more information on creating such an extension.
Another (likely easier) option is to use the newer java-cfenv project instead of Spring Cloud Connectors. java-cfenv supports Cloud Foundry only, and gives access to the full set of information in VCAP_SERVICES. See the project documentation for an example of how you can use this library.

Access specific JMX metric information on JConsole for a Spring Boot Application

I have a Spring Boot application packaged as a WAR and deployed on a Tomcat 9 server.
It's been configured to expose the following metrics through JMX:
spring.jmx.default-domain: my-app
management.endpoints.jmx.exposure.include: health,info,metrics
I can connect to Tomcat through JConsole and see the my-app MBean that offers those 3 endpoints:
Selecting Metrics -> Operations - listNames I can get the whole list of metrics exposed, invoking the listNames method:
Now I would like to see a specific metric (e.g. jvm.memory.used), going to metrics -> Operations -> metrics:
However the metric(requiredMetricName, tag) method is disabled.
How can I get the value of a specific metric from the mbean in JConsole?
The reason it's disabled is because JConsole won't allow input of parameters of complex types. See https://stackoverflow.com/a/12025340/62667
But if you use an alternative JMX interface (e.g. add Hawtio to your application) then you could use that to invoke the operations.

Micrometer With Microservice

I am newbie to micrometer. could anyone let me know how to manage microservice metrics centrally in spring boot ?
Where i can get all registered service information and matrices and stored metrics in influxdb ?
Assuming that you're asking "How to use Micrometer with Spring Boot for collecting metrics from heterogeneous services which have multiple instances on multiple hosts" as there is nothing special with the microservice architecture compared to the assumed environment, you need to add dimensions to metrics for hosts, application instances, and so on. You can achieve it with the common tags support. See the section for it in the Spring Boot reference guide.
UPDATED:
To answer the additional question on the below comment, I created a sample showing how to use common tags with environment variables. Note that it's on the branch common-tags-2.1.x-with-env, not the master.

Resources