I want to scrape metrics from two endpoints for the same pods - /demo/v1.0/metrics and /actuator/prometheus. This is how my annotations in values.yaml looks like right now.
metadata:
annotations:
iam.amazonaws.com/role: xxxx
prometheus.io/app: DEMO_SERVICE
prometheus.io/env: PRODUCTION
prometheus.io/path: /demo/v1.0/metrics
prometheus.io/port: "8197"
prometheus.io/product: DEMO_SERVICE
prometheus.io/scrape: "true"
prometheus.io/service: DEMO_SERVICE
prometheus.io/team: PROCESSING
How do I add the actuator endpoint to it and is it even required? I have added the Micrometer Prometheus registry dependency to pom.xml and I'm able to see the actuator metrics locally. Thank you.
Related
I have an endpoint /api/actuator/prometheus that returns registered metrics in Prometheus format.
Is it possible to format this document prettily and host it on GitLab Pages?
I want to do something similar as Open API does but for available Prometheus metrics.
I am using Spring Boot 2.5.4 and Actuator with Micrometer and Prometheus support.
When I open the /actuator/prometheus endpoint, I see metrics like these:
# HELP jvm_threads_live_threads The current number of live threads including both daemon and non-daemon threads
# TYPE jvm_threads_live_threads gauge
jvm_threads_live_threads 28.0
Note that there is no timestamp attached to that metric. However, according to OpenMetrics / Prometheus spec, it is possible to add a timestamp to the metrics output.
My question: How can you tell Spring Boot Actuator to generate and add a timestamp to the metrics it creates? I have not found any documentation on it. Thanks!
References
My pom.xml dependencies look like this:
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-core</artifactId>
</dependency>
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-registry-prometheus</artifactId>
</dependency>
My application.yaml like this:
# Application name. Shows up in metrics etc.
spring:
application:
name: "some_app_name"
# Expose application on port 8080
server:
port: ${SERVER_PORT:8080}
# Expose all Actuator endpoints (don't do this in production!)
management:
endpoints:
web:
exposure:
include:
- "*"
Micrometer doesn't support this at the moment. The team's recommendation is to use the Prometheus Java Client directly for metrics where the timestamp is important to you.
I'm trying to configure spring actuator metrics along with micrometer to be sent to Datadog stastd agent.
Still, I'd like to get them all sent with a tag, so that I can filter in my Datadog dashboard just my service metrics, and not considering other services metrics.
I've added:
management:
metrics:
tags:
application: my_app
to my service metrics configuration, but I can't see this tag value in Datadog dashboard. I'm not seeing anything weird in app logs nor actuator logfile neither.
I have nothing else regarding metrics in my service, as I don't want to implement custom metrics, just want to use the one provided by actuator.
This is how the whole metrics configuration looks like:
management:
metrics:
export:
statsd:
host: ${STATSD_AGENT_HOST}
port: ${STATSD_AGENT_HOST_PORT}
flavor: datadog
tags:
application: my_app
endpoints:
web:
exposure:
include: "*"
endpoint:
health:
show-details: ALWAYS
Versions:
micrometer version: 1.6.4
actuator version: 2.4.3
spring version: 2.3.8
Any clue about what I could be missing to get the tag reaching Datadog?
Thanks!
We figured this out in the comments, I'm posting an answer that summarizes it all up: it seems the root cause was using different versions of different spring-boot modules.
It is a good rule of thumb to not define the versions yourself but use BOMs and let them define the versions for you, e.g. see: spring-boot-dependencies. This way you will use the compatible (and tested) versions.
management.metrics.tags.your-tag is the way to add tags to all of your metrics. A good way to check this is looking at /actuator/metrics.
I am trying to implement prometheus monitoring for spring boot apps but my spring boot apps are registered with spring cloud eureka server so I don't have an option to configure static ports for the apps in prometheus config, I can only configure scraping for eureka server like below but not for the apps registered with eureka. Can anyone please suggest?
static_configs:
- targets: ["http://mycompany.eurekaserver.net:8761"]
Came here to add new answer that, Eureka SD is supported now in Prometheus from 2.21 version onwards.
Documentation available at https://prometheus.io/docs/prometheus/latest/configuration/configuration/#eureka_sd_config
Sample configuration is available at Sample Eureka SD config
Currently eureka server has no direct integration with prometheus, so we have to use SD adapter, I would recommend below, which is working as expected without much effort.
https://github.com/twinformatics/eureka-consul-adapter
How to add/integrate rest api end points in prometheus. For eg in splunk we are monitoring micro services transactions and logs.
Lets say below spring boot:-
/abc/v1/something
/abce/v1/something2
In one line :-how can we add above micro services end points to prometheus for motioning
When using Spring Boot 2, you can easily expose your application metrics for Prometheus. See here. You'd add spring-boot-starter-actuator and the Prometheus support with micrometer-registry-prometheus to your dependencies.
This will expose the metrics under /actuator/prometheus in the Prometheus exposition format to be scraped by your Prometheus instance.
Assumung you are using Spring WebMvc, then you'll get out-of-the-box metrics for your HTTP endpoints. (Jersey support is also provided with an extra module/dependency.)