Spring boot micrometer - expose metrics in an endpoint - spring

Is it possible to expose all the Spring metrics in an rest endpoint?
In our environment Micrometer cannot forward directly the metrics to our monitoring system, then my idea was to have all the metrics exposed in a rest endpoint (or on a file, json format but the endpoint is preferable) and then having a script to fetch all the metrics at once, manipulate the payload and forward the metrics to our customized monitoring api.
If micrometer is not the right choice, what is it?
EDIT
I tried to use the actuator/prometheus endpoint. I can see the endpoint but as soon as I hit the endpoint, it hang and does not return anything:
I am using spring boot 2.2.2 and I added this dependency
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-registry-prometheus</artifactId>
<version>1.3.1</version>
</dependency>
application.yml
management:
endpoints:
web:
exposure:
include: health, metrics, prometheus

You could use Spring Boot's built-in prometheus endpoint. It exposes all of the Micrometer-based metrics. It's primarily intended for periodic fetching by a Prometheus server, but you could do similar in a script of your own that transforms the data to the necessary format and forwards it to your custom monitoring API.

At the level of idea you did everything right, and Micrometer is indeed the way to go here.
At the level of pom.xml definitions, you don't need to set the version:
If the version or micrometer/micrometer-prometheus module that your spring boot comes with clashes with your definitions it might not work properly.
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-registry-prometheus</artifactId>
</dependency>
Other than that, if you work only with core-micrometer (without even prometheus integration) you have metrics endpoint that also shows all the metrics in the "common" format. Prometheus shows the metrics in a format that prometheus can scrape. For your use case it seems like both ways can work but you'll have to be familiar with the formats in either way you chose.
Another thought: you say that the application can't forward to monitoring system, is it because some security constraints or because you use some kind of custom solution or something that micrometer can't integrate with?
If its not a security thing, you can consider to write your own micrometer registry that will act as a bridge between micrometer and the monitoring system.
One more thing I would like to answer on: you say that the endpoint doesn't finish.
I can only speculate, because basically there can be many reasons (one of which is clash of versions that I've referred above) - but all are bugs, it should not work like that.
Another reason reason is that you might have defined a Gauge in micrometer that in order to be calculated performs some "expensive" calculations: goes to the database that for some reason takes ages to access, calls some expensive http endpoint, etc.
When the rest endpoint gets invoked, it takes "snapshots" (the current values) of all the metering primitives, for gauges it actually invokes your custom code that can do anything, so I would have checked that.
Yet another reason might be connectivity / security configurations (the request doesn't reach actuator at all) but given the information you've presented I can't tell more than that...

Related

How to get spring boot cloud/actuator to support prometheus Exemplars?

The Exemplars support essentially adds the trace-id to metrics that are being scraped. I found a tutorial on how would it work with GoLang[1] but cannot figure out how to do this with spring boot libraries/functionalities. I know that Prometheus Java Client supports it as described at [2] but not sure how to get it to work with Springboot.
https://vbehar.medium.com/using-prometheus-exemplars-to-jump-from-metrics-to-traces-in-grafana-249e721d4192
https://github.com/prometheus/client_java/pull/615
Update: Exemplars are supported by Micrometer and Spring Cloud Sleuth.
I'm not 100% sure I get your question right: I'm assuming you are talking about the /actuator/prometheus endpoint.
The support for metrics in Spring is provided by Micrometer that also supports Prometheus and Spring Boot "just" sets up an actuator endpoint for it.
So the real question is: does Micrometer support exemplars? Right now it does not and we haven't had anyone asking for it so far (this issue is a little bit connected). So if you want this feature, please open an issue(Enhancement request) on GitHub.
Update: I opened an issue for this: https://github.com/micrometer-metrics/micrometer/issues/2672, please feel free to +1 or chime-in.
Adding exemplars support in Micrometer is not the end of the story, we need to add support for Spring Cloud Sleuth too and solve a few other potential issues.
Until this is implemented, I guess your best bet is using the prometheus client.

Spring boot metrics for rest api with PathVariables

In my spring boot project I would like to keep count of how many times the rest api endpoint responded with status 200. The spring boot actuator metrics endpoint came close to solving this issue for me out of the box.
However, the /metrics endpoint names provided the aggregate of responses by the endpoint method rather than each of the dynamic endpoints created through #PathVariable.
For example:
while I can get http://localhost:8084/myproject/actuator/metrics/http.server.requests?tag=status:200,uri:/api/users/{id}/books
I would like to do something like
http://localhost:8084/myproject/actuator/metrics/http.server.requests?tag=status:200,uri:/api/users/1/books
and
http://localhost:8084/myproject/actuator/metrics/http.server.requests?tag=status:200,uri:/api/users/2/books
and so on.
Is there an easy way to do this?
You can roll your own WebMvcTagsProvider. That's the place where you can hook into the tag generation. Have a look at DefaultWebMvcTagsProvider to get insight on how it is done for the default behaviour.
A note: The default tagging is done on purpose the way it is to hinder metrics explosion, because every metric name + tag combination is a new metric. So be aware of that.

Does PCF Dev show Actuator endpoints and expose log level configuration for Java app?

I'm running a demo Spring Boot app with an actuator enabled in pom.xml:
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
however the endpoints the actuator is supposed to provide (Health Checks, Traces, etc.) are not showing in the PCF Dev Apps Manager:
Is that normal behavior and they would be enabled on the public cloud deployment or am I missing something in my configuration?
Similarly, on the Logs tab I don't see a log level configuration control through which setting a log level on a particular Java package is possible. Is that the regular behavior? How to configure logging and tracing?
For logging setup , you need to add configurations in applications.properties and it is not available as separate option in the logs window. For changing your logs at run time, you need to setup a config server/environment variable and use your actuator ends point to refresh it at run time. Hope this helps.

Accessing localhost:8080/actuator endpoint (with spring boot actuator)

In spring docs I have read about endpoint named "actuator" which is provided by actuator dependency, but I haven't managed to access it on my local testing app.
Question: Does someone know how to access that endpoint? of coarse if it is possible :)
Sub-question 1: If this endpoint exists, then why it is hidden?
Sub-question 2: If this endpoint doesn't exist, how can we notify spring.io to correct documentation page (open some kind of ticket) ?
Details:
I want to access exactly "actuator" endpoint, not other endpoints provided by spring boot actuator (localhost:8080/actuator)
Yes, I have tried to enable that endpoint manually in properties file (endpoints.enabled=true OR endpoints.actuator.enabled=true)
Yes, I have tried to enable/disable endpoints.sencitive property
Yes, other endpoints of actuator work just fine
No special reason why I need that, just want to try it out (just learning new stuff :) )
Please don't just answer "there is no such endpoint dude!", there should be some kind of reason why it is written in the docs
Please use spring boot version which I am using now before answering "it is working for me with these configs" (spring boot version: 1.5.4.RELEASE)
Thank you in advance :)
You must include the Spring Hateoas dependency in order for the /actuator endpoint to become available:
<dependency>
<groupId>org.springframework.hateoas</groupId>
<artifactId>spring-hateoas</artifactId>
</dependency>
Per the docs:
Provides a hypermedia-based “discovery page” for the other endpoints.
Requires Spring HATEOAS to be on the classpath.

Spring Actuator - metrics aggregation from docker containers

I have a Spring Boot REST service application. This application uses Spring Actuator to display metrics and health information. How can I aggregate this information from two or more containers running the same application?
You need to export the metrics to a central system.
Spring Boot provides a couple of implementations of a marker interface
called Exporter which can be used to copy metric readings from the
in-memory buffers to a place where they can be analyzed and displayed.
More specifically personally I like exporting metrics to statsD
To export metrics to Statsd, make sure first that you have added
com.timgroup:java-statsd-client as a dependency of your project
(Spring Boot provides a dependency management for it). Then add a
spring.metrics.export.statsd.host value to your application.properties
file. Connections will be opened to port 8125 unless a
spring.metrics.export.statsd.port override is provided. You can use
spring.metrics.export.statsd.prefix if you want a custom prefix.
The information above is all from the Spring Boot documentation on metrics: https://docs.spring.io/spring-boot/docs/current/reference/html/production-ready-metrics.html

Resources