I am using Spring Boot 2.5.4 and Actuator with Micrometer and Prometheus support.
When I open the /actuator/prometheus endpoint, I see metrics like these:
# HELP jvm_threads_live_threads The current number of live threads including both daemon and non-daemon threads
# TYPE jvm_threads_live_threads gauge
jvm_threads_live_threads 28.0
Note that there is no timestamp attached to that metric. However, according to OpenMetrics / Prometheus spec, it is possible to add a timestamp to the metrics output.
My question: How can you tell Spring Boot Actuator to generate and add a timestamp to the metrics it creates? I have not found any documentation on it. Thanks!
References
My pom.xml dependencies look like this:
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-core</artifactId>
</dependency>
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-registry-prometheus</artifactId>
</dependency>
My application.yaml like this:
# Application name. Shows up in metrics etc.
spring:
application:
name: "some_app_name"
# Expose application on port 8080
server:
port: ${SERVER_PORT:8080}
# Expose all Actuator endpoints (don't do this in production!)
management:
endpoints:
web:
exposure:
include:
- "*"
Micrometer doesn't support this at the moment. The team's recommendation is to use the Prometheus Java Client directly for metrics where the timestamp is important to you.
Related
I am using springboot gateway 2.0.4.RELEASE.
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-registry-prometheus</artifactId>
</dependency>
After adding this dependency, it is exposing all metrics to prometheus endpoint.
I want to disable some of the metrics from exposing through /actuator/prometheus endpoint.
Let's say I want to disable
gateway_requests_seconds_count
gateway_requests_seconds_sum
gateway_requests_seconds_max
Does anyone know how can I achieve this ?
Thanks
Alpesh
I'm trying to configure spring actuator metrics along with micrometer to be sent to Datadog stastd agent.
Still, I'd like to get them all sent with a tag, so that I can filter in my Datadog dashboard just my service metrics, and not considering other services metrics.
I've added:
management:
metrics:
tags:
application: my_app
to my service metrics configuration, but I can't see this tag value in Datadog dashboard. I'm not seeing anything weird in app logs nor actuator logfile neither.
I have nothing else regarding metrics in my service, as I don't want to implement custom metrics, just want to use the one provided by actuator.
This is how the whole metrics configuration looks like:
management:
metrics:
export:
statsd:
host: ${STATSD_AGENT_HOST}
port: ${STATSD_AGENT_HOST_PORT}
flavor: datadog
tags:
application: my_app
endpoints:
web:
exposure:
include: "*"
endpoint:
health:
show-details: ALWAYS
Versions:
micrometer version: 1.6.4
actuator version: 2.4.3
spring version: 2.3.8
Any clue about what I could be missing to get the tag reaching Datadog?
Thanks!
We figured this out in the comments, I'm posting an answer that summarizes it all up: it seems the root cause was using different versions of different spring-boot modules.
It is a good rule of thumb to not define the versions yourself but use BOMs and let them define the versions for you, e.g. see: spring-boot-dependencies. This way you will use the compatible (and tested) versions.
management.metrics.tags.your-tag is the way to add tags to all of your metrics. A good way to check this is looking at /actuator/metrics.
Is it possible to expose all the Spring metrics in an rest endpoint?
In our environment Micrometer cannot forward directly the metrics to our monitoring system, then my idea was to have all the metrics exposed in a rest endpoint (or on a file, json format but the endpoint is preferable) and then having a script to fetch all the metrics at once, manipulate the payload and forward the metrics to our customized monitoring api.
If micrometer is not the right choice, what is it?
EDIT
I tried to use the actuator/prometheus endpoint. I can see the endpoint but as soon as I hit the endpoint, it hang and does not return anything:
I am using spring boot 2.2.2 and I added this dependency
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-registry-prometheus</artifactId>
<version>1.3.1</version>
</dependency>
application.yml
management:
endpoints:
web:
exposure:
include: health, metrics, prometheus
You could use Spring Boot's built-in prometheus endpoint. It exposes all of the Micrometer-based metrics. It's primarily intended for periodic fetching by a Prometheus server, but you could do similar in a script of your own that transforms the data to the necessary format and forwards it to your custom monitoring API.
At the level of idea you did everything right, and Micrometer is indeed the way to go here.
At the level of pom.xml definitions, you don't need to set the version:
If the version or micrometer/micrometer-prometheus module that your spring boot comes with clashes with your definitions it might not work properly.
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-registry-prometheus</artifactId>
</dependency>
Other than that, if you work only with core-micrometer (without even prometheus integration) you have metrics endpoint that also shows all the metrics in the "common" format. Prometheus shows the metrics in a format that prometheus can scrape. For your use case it seems like both ways can work but you'll have to be familiar with the formats in either way you chose.
Another thought: you say that the application can't forward to monitoring system, is it because some security constraints or because you use some kind of custom solution or something that micrometer can't integrate with?
If its not a security thing, you can consider to write your own micrometer registry that will act as a bridge between micrometer and the monitoring system.
One more thing I would like to answer on: you say that the endpoint doesn't finish.
I can only speculate, because basically there can be many reasons (one of which is clash of versions that I've referred above) - but all are bugs, it should not work like that.
Another reason reason is that you might have defined a Gauge in micrometer that in order to be calculated performs some "expensive" calculations: goes to the database that for some reason takes ages to access, calls some expensive http endpoint, etc.
When the rest endpoint gets invoked, it takes "snapshots" (the current values) of all the metering primitives, for gauges it actually invokes your custom code that can do anything, so I would have checked that.
Yet another reason might be connectivity / security configurations (the request doesn't reach actuator at all) but given the information you've presented I can't tell more than that...
I have got custom starter application (library) which has actuator and prometheus dependency
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-registry-prometheus</artifactId>
<version>1.5.2</version>
</dependency>
in application.properties
management.endpoints.web.exposure.include=health, info, metrics, prometheus
I am using this starter in my other applications and I want to pass also this exposition of the endpoints from the starter.
Actuator and Prometheus dependecies works in applications but not show selected endpoints in starter. Ofc I can add also line with management.endpoints.web.exposure.include=health, info, metrics, prometheus to my apps but with several apps using this starter I want to pass this once for all and change endpoints only in starter if needed.
Do you have an idea how to expose those endpoints in my app which are set in starter?
Spring Boot v2.3.2 /
Maven 3.6.3
You can pass this as command line args or as ENV vars to your jar, when you start your application.
This way you can pass it to required application, as and when needed w/o updating application.properties.
It will also safeguard against exposing the actuator endpoints when not required, as actuator endpoints reveal sensitive info about the application.
eg. java -Dmanagement.endpoints.web.exposure.include=health, info, metrics, prometheus -jar myapp.jar
I need to query for collected metrics in my application and the only way I found to do it is via Prometheus rest api as described here: https://prometheus.io/docs/prometheus/latest/querying/api/ Does any Spring or java api for fetching metrics from Prometheus exist, so that I wouldn't have to implement it from scratch?
If you want to collect metrics from your spring application using Prometheus, this is what you have to do:
In your application's pom.xml add this dependency:
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-registry-prometheus</artifactId>
<version>1.7.0</version>
</dependency>
In application.properties you need to add:
management.endpoints.web.exposure.include=*
management.endpoint.health.show-details=always
And this should be all, now when you start your Spring application you can check the collected metrics at /actuator/prometheus, you'll have all the metrics grouped here.
Good luck!