I'm using Spring boot 2.7.0 and I'm trying to expose the http.server.request metrics so I can scrape it in Prometheus but it's not working.
My application.yml properties look like this:
management:
endpoint:
health:
probes:
enabled: true
show-details: always
endpoints:
web:
exposure:
include: '*'
health:
livenessState:
enabled: true
readinessState:
enabled: true
metrics:
tags:
application: ${spring.application.name}
distribution:
percentiles-histogram:
http:
server:
requests: true
And I have the following dependencies:
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-registry-prometheus</artifactId>
<scope>runtime</scope>
</dependency>
but in /actuator/metrics endpoint I can't see anything about http.server.request metrics.
{
"names": [
"application.ready.time",
"application.started.time",
"disk.free",
"disk.total",
"executor.active",
"executor.completed",
"executor.pool.core",
"executor.pool.max",
"executor.pool.size",
"executor.queue.remaining",
"executor.queued",
"hikaricp.connections",
"hikaricp.connections.acquire",
"hikaricp.connections.active",
"hikaricp.connections.creation",
"hikaricp.connections.idle",
"hikaricp.connections.max",
"hikaricp.connections.min",
"hikaricp.connections.pending",
"hikaricp.connections.timeout",
"hikaricp.connections.usage",
"jdbc.connections.active",
"jdbc.connections.idle",
"jdbc.connections.max",
"jdbc.connections.min",
"jvm.buffer.count",
"jvm.buffer.memory.used",
"jvm.buffer.total.capacity",
"jvm.classes.loaded",
"jvm.classes.unloaded",
"jvm.gc.live.data.size",
"jvm.gc.max.data.size",
"jvm.gc.memory.allocated",
"jvm.gc.memory.promoted",
"jvm.gc.overhead",
"jvm.gc.pause",
"jvm.memory.committed",
"jvm.memory.max",
"jvm.memory.usage.after.gc",
"jvm.memory.used",
"jvm.threads.daemon",
"jvm.threads.live",
"jvm.threads.peak",
"jvm.threads.states",
"logback.events",
"process.cpu.usage",
"process.start.time",
"process.uptime",
"resilience4j.circuitbreaker.buffered.calls",
"resilience4j.circuitbreaker.calls",
"resilience4j.circuitbreaker.failure.rate",
"resilience4j.circuitbreaker.not.permitted.calls",
"resilience4j.circuitbreaker.slow.call.rate",
"resilience4j.circuitbreaker.slow.calls",
"resilience4j.circuitbreaker.state",
"system.cpu.count",
"system.cpu.usage",
"tomcat.sessions.active.current",
"tomcat.sessions.active.max",
"tomcat.sessions.alive.max",
"tomcat.sessions.created",
"tomcat.sessions.expired",
"tomcat.sessions.rejected"
]
}
My teammates were able to fix this problem by downgrading the spring boot version to 2.3.7.release but I'm not very fond of that. Is there any other way to expose these metrics?
It's an issue with the 2.7.0 spring boot version. Upgrade to 2.7.1
Related
I am trying to setup a openapi specification and publish the API with Apache Camel and Spring. I tried using restConfiguration, adding the property in the application.yaml, and using the #OpenApiProperty on the app. Everytime the generated yaml reads:
- servers
- url: ""
platform:
# not used
urlrewrite:
enabled: false
token: ${LOCAL_TOKEN:}
apim:
token_ep: ${x/token}
client:
username: ${apim_client_username:}
password: ${apim_client_password:}
consumerKey: ${apim_client_id:}
consumerSecret: ${apim_client_secret:}
endpointsOrchestrated:
server: myServer
emr-endpoint: xxxxxxxx
triggerName: ${PLATFORM_MODULE_ID}
env: dev
domain: ${PLATFORM_MODULE_DOMAIN}
openapi:
title: My Sample api
version: 1.0.0
camel:
dataformat:
jackson:
auto-discover-object-mapper: true
springboot:
tracing: false
rest:
host: myhost.com
port: 8080
spring:
application:
name: aaa
profiles:
active: ${ENV:local}
server:
servlet:
context-path: /
port: ${TOMCAT_PORT:8080}
host: localhost
# MAX HTTP THREADS
tomcat:
threads:
max: ${MAIN_HTTP_THREADS:200}
In my application we are making two calls from my app for getting secrets from Vault, as shown below:
Login to Vault : POST call to https::/v1/auth/approle/login -- It will take role_id and secret_id as payload and response will be client_token.
Fetch secrets : GET call to https::/v1/secret/data/abc/dev/xyz.json -- It will take headers as X-Vault-Token and X-Vault-Namespace and it will give you the response as below:
{
"request_id": "......",
"lease_id": "",
"renewable": false,
"lease_duration": 0,
"data": {
"data": {
"name": "ABC"
},
"metadata": {
"created_time": "...",
"deletion_time": "",
"destroyed": false,
"version": 1
}
}
Now I want to use Spring Cloud Vault Dependency to make things work through it. Please provide me the proper illustrations to make this work?
Assuming you are running spring boot and have a working Vault server configured for your app.
Add spring cloud vault maven dependency
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-vault-config</artifactId>
</dependency>
Add vault configuration to bootstrap.yaml
spring:
application:
name: abc
cloud:
vault:
host: <vault-server-hostname>
port: <vault-server-port>
scheme: HTTPS
namespace: <name-of-vault-namespace>
authentication: APPROLE
app-role:
role-id: <your-application-role-id>
secret-id: <your-application-secret-id>
role: <your-application-role>
If you run your app with spring profiles, like dev, it will be picked up and added to the vault path.
Now you should be able to inject secrets stored on the path secret/data/abc/dev with #Value("${<name-of-property>}
I could not find any up to date and / or working documentation and / or example on how to set this up so what I did is researching and try and error. Yet I have not been able to get it running.
Im having a config-server with the following dependencies:
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-config-server</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-bus-amqp</artifactId>
<version>2.2.1.RELEASE</version>
</dependency>
The config-server has the following bootstrap.yml:
spring:
application:
name: config-server
rabbitmq:
host: ${RABBIT_MQ_HOST:localhost}
port: ${RABBIT_MQ_PORT:5672}
username: ${RABBIT_MQ_URSER_NAME:guest}
password: ${RABBIT_MQ_URSER_PASSWORD:guest}
And the following application.yml
spring:
cloud:
config:
server:
git:
uri: https://github.com/MyGit/config-repository.git
cloneOnStart: true
management:
endpoints:
web:
exposure:
include: "*"
server:
port: 8888
All my clients are having the following dependencies:
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-bus-amqp</artifactId>
<version>2.2.1.RELEASE</version>
</dependency>
Their bootstrap.yml for example looks like this:
spring:
application:
name: user-service
cloud:
config:
uri: http://${CONFIG_HOST:localhost}:${CONFIG_PORT:8888}
rabbitmq:
host: ${RABBIT_MQ_HOST:localhost}
port: ${RABBIT_MQ_PORT:5672}
username: ${RABBIT_MQ_URSER_NAME:guest}
password: ${RABBIT_MQ_URSER_PASSWORD:guest}
And their application.yml like this:
management:
endpoints:
web:
exposure:
include: "*"
server:
port: 8000
eureka:
client:
registerWithEureka: true
fetchRegistry: true
serviceUrl:
defaultZone: http://${EUREKA_HOST:localhost}:${EUREKA_PORT:8761}/eureka
Im using springs #ConfigurationProperties, for example:
#Component
#ConfigurationProperties("user")
#Getter
#Setter
public class UserConfiguration {
private String role;
}
I'm starting all my services (including rabbitmq) using docker and docker-compose:
version: "3"
services:
rabbitmq:
image: bitnami/rabbitmq:latest
container_name: rabbitmq
environment:
RABBITMQ_USERNAME: admin
RABBITMQ_PASSWORD: admin
ports:
- "4369:4369"
- "5672:5672"
- "15672:15672"
- "25672:25672"
config-server:
build:
context: config-server
dockerfile: Dockerfile
container_name: config-server
depends_on:
- rabbitmq
environment:
RABBIT_MQ_HOST: "rabbitmq"
RABBIT_MQ_PORT: "5672"
RABBIT_MQ_URSER_NAME: "admin"
RABBIT_MQ_URSER_PASSWORD: "admin"
ports:
- "8888:8888"
eureka-server:
build:
context: eureka-server
dockerfile: Dockerfile
container_name: eureka-server
depends_on:
- rabbitmq
- config-server
environment:
RABBIT_MQ_HOST: "rabbitmq"
RABBIT_MQ_PORT: "5672"
RABBIT_MQ_URSER_NAME: "admin"
RABBIT_MQ_URSER_PASSWORD: "admin"
CONFIG_HOST: "config-server"
CONFIG_PORT: "8888"
ports:
- "8761:8761"
zuul-gateway:
build:
context: zuul-gateway
dockerfile: Dockerfile
container_name: zuul-gateway
depends_on:
- rabbitmq
- config-server
- eureka-server
environment:
RABBIT_MQ_HOST: "rabbitmq"
RABBIT_MQ_PORT: "5672"
RABBIT_MQ_URSER_NAME: "admin"
RABBIT_MQ_URSER_PASSWORD: "admin"
CONFIG_HOST: "config-server"
CONFIG_PORT: "8888"
EUREKA_HOST: "eureka-server"
EUREKA_PORT: "8761"
ports:
- "8765:8765"
user-service:
build:
context: user-service
dockerfile: Dockerfile
container_name: user-service
depends_on:
- rabbitmq
- config-server
- eureka-server
environment:
SPRING_PROFILES_ACTIVE: "development"
RABBIT_MQ_HOST: "rabbitmq"
RABBIT_MQ_PORT: "5672"
RABBIT_MQ_URSER_NAME: "admin"
RABBIT_MQ_URSER_PASSWORD: "admin"
CONFIG_HOST: "config-server"
CONFIG_PORT: "8888"
EUREKA_HOST: "eureka-server"
EUREKA_PORT: "8761"
ports:
- "8000:8000"
user-service-2:
build:
context: user-service
dockerfile: Dockerfile
container_name: user-service-2
depends_on:
- rabbitmq
- config-server
- eureka-server
environment:
SPRING_PROFILES_ACTIVE: "development"
RABBIT_MQ_HOST: "rabbitmq"
RABBIT_MQ_PORT: "5672"
RABBIT_MQ_URSER_NAME: "admin"
RABBIT_MQ_URSER_PASSWORD: "admin"
CONFIG_HOST: "config-server"
CONFIG_PORT: "8888"
EUREKA_HOST: "eureka-server"
EUREKA_PORT: "8761"
ports:
- "8001:8000"
user-service-3:
build:
context: user-service
dockerfile: Dockerfile
container_name: user-service-3
depends_on:
- rabbitmq
- config-server
- eureka-server
environment:
SPRING_PROFILES_ACTIVE: "development"
RABBIT_MQ_HOST: "rabbitmq"
RABBIT_MQ_PORT: "5672"
RABBIT_MQ_URSER_NAME: "admin"
RABBIT_MQ_URSER_PASSWORD: "admin"
CONFIG_HOST: "config-server"
CONFIG_PORT: "8888"
EUREKA_HOST: "eureka-server"
EUREKA_PORT: "8761"
ports:
- "8002:8000"
When all my services are started, at first everything seems to be fine. All services are registered with eureka, all services are callable on localhost and they can talk to each other.
But the config refresh is not working.
When I change something in my config repository and commit and push it it has no effect on my services. The configuration they have stays the same. When I manually call the bus refresh on my config-server:
http://localhost:8888/bus/refresh
with POST, like it is described in nearly every guide I could find so far, I get the response:
{
"timestamp": "2020-03-30T09:33:57.818+0000",
"status": 405,
"error": "Method Not Allowed",
"message": "Request method 'POST' not supported",
"path": "/bus/refresh"
}
When I use GET instead, I get:
{
"name": "bus",
"profiles": [
"refresh"
],
"label": null,
"version": "03759e798f3516da6a18fc8b61a265d37ddeff4e",
"state": null,
"propertySources": []
}
and it also has no effect on the confiuration of my services.
And when I call the bus refresh on any of my services I get:
{
"timestamp": "2020-03-30T09:35:39.643+0000",
"status": 404,
"error": "Not Found",
"message": "No message available",
"path": "/bus/refresh"
}
What do I need to do to get the auto configuration refresh working?
Manually calling the /actuator/refresh endpoints on the services works. They are then pulling the new config. So it seems like the rabbit mq simply is not working.
I found the solution.
Acutally my setup and configuration files are totally fine, I just called the wrong endpoint. Instead of calling:
http://localhost:8888/bus/refresh
I have to call:
http://localhost:8888/actuator/bus-refresh
This works on any service I started. It will automatically send a message to rabbitmq which will then publish the refresh to all consumers. All configurations are then updating.
Though I still dont know why bus/refresh is not working. It is used in many examples / tutorials.
We are using Spring Cloud Gateway before multiple microservices with consul as service discovery. There are several microservices developed in different languages.
Please find build.gradle for the application
buildscript {
ext {
springBootVersion = '2.1.2.RELEASE'
}
repositories {
mavenCentral()
}
dependencies {
classpath("org.springframework.boot:spring-boot-gradle-plugin:${springBootVersion}")
}
}
apply plugin: 'java'
apply plugin: 'org.springframework.boot'
apply plugin: 'io.spring.dependency-management'
group = 'com.demo'
version = '0.0.1-SNAPSHOT'
sourceCompatibility = '1.8'
repositories {
mavenCentral()
maven { url 'https://repo.spring.io/milestone' }
}
ext {
set('springCloudVersion', 'Greenwich.RELEASE')
}
dependencies {
implementation 'org.springframework.boot:spring-boot-starter-actuator'
implementation 'org.springframework.cloud:spring-cloud-starter-consul-config'
implementation 'org.springframework.cloud:spring-cloud-starter-consul-discovery'
implementation 'org.springframework.cloud:spring-cloud-starter-gateway'
implementation 'org.springframework.boot:spring-boot-starter-security'
// https://mvnrepository.com/artifact/io.netty/netty-tcnative-boringssl-static
compile group: 'io.netty', name: 'netty-tcnative-boringssl-static', version: '2.0.20.Final'
runtimeOnly 'org.springframework.boot:spring-boot-devtools'
compileOnly 'org.projectlombok:lombok'
testImplementation 'org.springframework.boot:spring-boot-starter-test'
}
dependencyManagement {
imports {
mavenBom "org.springframework.cloud:spring-cloud-dependencies:${springCloudVersion}"
}
}
Below is the example of API gateway configuration
application.yaml
server:
port: 10000
http:
port: 9000
# enable HTTP2
http2:
enabled: true
# enable compression
compression:
enabled: true
mime-types: text/html,text/xml,text/plain,text/css,text/javascript,application/javascript,application/json
ssl:
enabled: true
key-store: /var/.conf/self-signed.p12
key-store-type: PKCS12
key-store-password: "something"
key-alias: athenasowl
trust-store: /var/.conf/self-signe.p12
trust-store-password: "something"
spring:
application:
name: api-gateway
cloud:
gateway:
discovery:
locator:
enabled: true
predicates:
- Path="'/api/' + serviceId + '/**'"
filters:
- RewritePath="'/api/' + serviceId + '/(?<remaining>.*)'", "serviceId + '/${remaining}'"
management:
security:
enabled: false
server:
port: 10001
ssl:
enabled: false
endpoint:
gateway:
enabled: true
endpoints:
web:
exposure:
include: "*"
health:
sensitive: false
logging:
level:
root: DEBUG
org:
springframework:
web: INFO
pattern:
console: "%-5level %d{dd-MM-yyyy HH:mm:ss,SSS} [%F:%L] VTC : %msg%n"
file: "%-5level %d{dd-MM-yyyy HH:mm:ss,SSS} [%F:%L] VTC : %msg%n"
file: /tmp/log_files/apigateway.log
security:
basic:
enabled: false
There are a few configuration issues which we are facing, they are listed below:
Rewrite URL prefixed with /api/ to respective serviceId registered on consul: We tried to configure predicate to get path prefixed with api to rewrite path and remove api, but still it's not working. So there is another service /hello-service/ registered with consul server, but we want to do API call with /api/hello-service/
Redirect unmatched request to default path: We want to redirect all unmatched request to UI.
Redirecting HTTP to HTTPS on spring cloud gateway: We want to force all request coming to spring gateway to be https
Forwarding HTTPS request to HTTP serviceId registered with consul: Services registered with consul are on HTTP except for the API gateway, we want to be able to send HTTPS request to HTTP backend i.e. terminating HTTPS at API Gateway only.
Any help in solving the above issue would be good
Edit 1:
After some help from #spencergibb, we had setup the spring cloud gateway with https. But There are some additional issues which we faced
If HTTPS is enabled on both API gateway and service both, we received below error
javax.net.ssl.SSLException: handshake timed out at
io.netty.handler.ssl.SslHandler.handshake(...)(Unknown Source)
~[netty-handler-4.1.31.Final.jar:4.1.31.
If HTTPS is enabled on only API gateway, we received below error
There was an unexpected error (type=Not Found, status=404).
org.springframework.web.server.ResponseStatusException: 404 NOT_FOUND
and received
for path https://localhost:8443/api/hello-service/hello/message
Unable to Connect
for path http://localhost:8080/hello-service/hello/message
Please find the link for the sample applications
Instructions:
navigate to consul directory and Start consul server using command ./consul agent -dev
run api-gateway spring boot gradle project
run rest-demo spring boot gradle project
Edit 2
Thank You #spencergibb, We were able to successfully apply ssl on gateway and call the registered services on HTTP. Since Spring Webflux with Netty does not support listening on two ports, we created an additional tcp server bind to http port based on this answer.
There is still some issue we are facing with RewritePath for /api/ rule
predicates:
- name: Path
args:
pattern: "'/api/'+serviceId.toLowerCase()+'/**'"
filters:
- name: RewritePath
args:
regexp: "'/api/' + serviceId.toLowerCase() + '/(?<remaining>.*)'"
replacement: "'/${remaining}'"
below is the complete trace for the request
DEBUG 13-02-2019 03:32:01 [FilteringWebHandler.java:86] VTC : Sorted
gatewayFilterFactories:
[OrderedGatewayFilter{delegate=GatewayFilterAdapter{delegate=org.springframework.cloud.gateway.filter.AdaptCachedBodyGlobalFilter#257505fd},
order=-2147482648},
OrderedGatewayFilter{delegate=GatewayFilterAdapter{delegate=org.springframework.cloud.gateway.filter.GatewayMetricsFilter#400caab4},
order=-2147473648},
OrderedGatewayFilter{delegate=GatewayFilterAdapter{delegate=org.springframework.cloud.gateway.filter.NettyWriteResponseFilter#36e2c50b},
order=-1},
OrderedGatewayFilter{delegate=GatewayFilterAdapter{delegate=org.springframework.cloud.gateway.filter.ForwardPathFilter#66f0c66d}, order=0},
OrderedGatewayFilter{delegate=org.springframework.cloud.gateway.filter.factory.RewritePathGatewayFilterFactory$$Lambda$360/1720581802#5821f2e6,
order=0},
OrderedGatewayFilter{delegate=GatewayFilterAdapter{delegate=org.springframework.cloud.gateway.filter.RouteToRequestUrlFilter#27119239},
order=10000},
OrderedGatewayFilter{delegate=GatewayFilterAdapter{delegate=org.springframework.cloud.gateway.filter.LoadBalancerClientFilter#568a9d8f},
order=10100},
OrderedGatewayFilter{delegate=GatewayFilterAdapter{delegate=org.springframework.cloud.gateway.filter.WebsocketRoutingFilter#6ba77da3},
order=2147483646},
OrderedGatewayFilter{delegate=GatewayFilterAdapter{delegate=org.springframework.cloud.gateway.filter.NettyRoutingFilter#73c24516},
order=2147483647},
OrderedGatewayFilter{delegate=GatewayFilterAdapter{delegate=org.springframework.cloud.gateway.filter.ForwardRoutingFilter#461a9938},
order=2147483647}] TRACE 13-02-2019 03:32:01
[RouteToRequestUrlFilter.java:59] VTC : RouteToRequestUrlFilter start
TRACE 13-02-2019 03:32:02 [NettyWriteResponseFilter.java:68] VTC :
NettyWriteResponseFilter start TRACE 13-02-2019 03:32:02
[GatewayMetricsFilter.java:101] VTC : Stopping timer
'gateway.requests' with tags
[tag(outcome=CLIENT_ERROR),tag(routeId=rewrite_response_upper),tag(routeUri=http://httpbin.org:80),tag(status=NOT_FOUN
A number of things were needed
disable http2
Disable ssl configuration of httpclient
Update locator predicates and filters to use verbose configuration.
Here is the resulting portions of application.yml
server:
port: 8443
http:
port: 8080
servlet:
# enable HTTP2
# http2:
# enabled: true
# enable compression
# ... removed for brevity
spring:
application:
name: api-gateway
cloud:
consul:
enabled: true
gateway:
# httpclient:
# ssl:
# handshake-timeout-millis: 10000
# close-notify-flush-timeout-millis: 3000
# close-notify-read-timeout-millis: 0
# routes:
# - id: ui_path_route
# predicates:
# - Path="'/**'"
# filters:
# - RewritePath="'/**'", "/ui"
discovery:
instanceId: ${spring.application.name}:${vcap.application.instance_id:${spring.application.instance_id:${random.value}}}
locator:
enabled: true
predicates:
- name: Path
args:
pattern: "'/api/' + serviceId + '/**'"
filters:
- name: RewritePath
args:
regexp: "'/api/' + serviceId + '/(?<remaining>.*)'"
replacement: "'/${remaining}'"
#... removed for brevity
How should I configure the Micronaut to get the /metrics in the Prometheus format ?
Used: micronaut 1.0.0.M3
Now:
micronaut:
...
metrics:
enabled: true
export:
prometheus:
enabled: true
and result: metrics name list
{"names":["jvm.memory.max","executor.pool.size"...]}
I need to get: metrics in the prometheus format(formats)
Micronaut Micrometer has an PrometheusEndpoint from version 1.1 that will
return in Prometheus format from /prometheus and
can be enabled in application.yml by:
endpoints:
prometheus:
sensitive: false
In combination with
micronaut:
metrics:
enabled: true
export:
prometheus:
enabled: true
step: PT1M
descriptions: true
(The documentation is missing the endpoint config but will be changed in the new release)
To piggyback on the other answers, here is an Micronaut endpoint that provides the Prometheus metrics in the format we needed:
package your.package.name
import io.micrometer.prometheus.PrometheusMeterRegistry
import io.micronaut.management.endpoint.annotation.Endpoint
import io.micronaut.management.endpoint.annotation.Read
#Endpoint(id = "prometheus", value = "/prometheus", defaultEnabled = true,
defaultSensitive = false)
class PrometheusController(val prometheusMeterRegistry: PrometheusMeterRegistry) {
#Read
fun scrape(): String {
return prometheusMeterRegistry.scrape()
}
}
At the moment, we solved the problem as follows.
Added a new endpoint. Or create a controller with a mapping on /metrics.
The new endpoint added a return of scrape().
Correlated endpoint with /prometheus (new endpoint can not be mapped on /metrics).
Disconnected endpoint metrics which by default.
Config:
micronaut:
...
metrics:
enabled: true
export:
prometheus:
enabled: true
...
endpoints:
...
metrics:
enabled: false
prometheus:
enabled: true
Haven't tested this out but based on the following:
https://github.com/micronaut-projects/micronaut-core/blob/master/configurations/micrometer-registry-prometheus/src/main/java/io/micronaut/configuration/metrics/micrometer/prometheus/PrometheusMeterRegistryFactory.java
Your yaml should look like
metrics:
prometheus:
enabled: true
don't believe the export comes into play.