I am currently trying to use PrAG stack to monitor spring boot based microservices. Have 2 spring boot projects 1.5.4 (pom.xml) have the following dependencies configured to get the metrics and transform the metrics to prometheus server:
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-hateoas</artifactId>
</dependency>
<dependency>
<groupId>com.moelholm</groupId>
<artifactId>prometheus-spring-boot-starter</artifactId>
<version>1.0.1</version>
</dependency>
Prometheus Scrape Configs:
scrape_configs:
- job_name: 'Test-springboot-actuator'
scrape_interval: 2s
metrics_path: '/prometheus'
static_configs:
- targets: ['localhost:8090']
- job_name: 'secondApp'
scrape_interval: 2s
basic_auth:
username: user
password: pass
metrics_path: '/prometheus'
static_configs:
- targets: ['localhost:8080']
Test project does not use spring security , but the secondAPP uses the spring security.
Second App requires the basic authentication. Prometheus pod does not start,it somehow does not read the credentials that have been set in scrape_config.
Can anyone point me to the right direction?
One thing you can do is allow the metrics endpoint to work when security is enabled. To do that you can use this setting.
endpoints.metrics.sensitive=false
Reference.
Related
I want to set up monitoring Spring Security application, which has updated bearer token.
I connect monitoring using a combination of Grafana and Prometheus, but after a token refresh time, it disconnects.
The Spring Security application implements a token refresh mechanism after authorization.
pom.xml dependensies:
<dependency>
<groupId>io.micrometer.prometheus</groupId>
<artifactId>prometheus-rsocket-spring</artifactId>
<version>1.5.0</version>
</dependency>
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-registry-prometheus</artifactId>
</dependency>
Example of the prometheus config file I use:
scrape_configs:
- job_name: 'project_manager'
scrape_interval: 5s
bearer_token: 'my.bearer.tocken.example'
metrics_path: '/api/actuator/prometheus'
static_configs:
- targets: ['host.docker.internal:8200']
I tried use this settings in application:
management.security.enabled=false
management.metrics.sensitive=false
Monitoring works successfully for an hour, after which the token is updated and the connection between Prometheus and the application is lost.
Perhaps I can set up a refresh token in the prometneus configuration?
I have problem with properly configuring spring cloud config. I have dependencies
<spring.cloud.version>2021.0.3</spring.cloud.version>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-config</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-bootstrap</artifactId>
</dependency>
Boostrap.yml
spring:
application.name: myapp
profiles:
active: dev
config:
import: optional:configserver:http://ip:8888
cloud:
config:
enabled: true
username: admin
password: secret
Now when my application starts it loads config twice. The first load is by class ConfigServicePropertySourceLocator and is without expected profile. The second is by ConfigServerConfigDataLoader and this time it is with proper profile. It seems that the configs loaded during first load takes precedence and my application does not start.
12:32:54 [ConfigServicePropertySourceLocator:241] - Fetching config from server at : http://ip:8888
12:32:54 [ConfigServicePropertySourceLocator:165] - Located environment: name=myapp, profiles=[default], label=null, version=7d2bc5d68acd8fcca65f34f2074b1860f36e19c6, state=null
12:32:54 [MyApplication:646] - The following 1 profile is active: "dev"
12:32:54 [ConfigServerConfigDataLoader:255] - Fetching config from server at : http://ip:8888
12:32:54 [ConfigServerConfigDataLoader:255] - Located environment: name=myapp, profiles=[dev], label=null, version=7d2bc5d68acd8fcca65f34f2074b1860f36e19c6, state=null
Providing profile using param -Dspring.profiles.active=dev does not help. How to configure profile that can be read by boostrap?
Add file resources -> bootstrap.properties
spring.application.name=myapp
spring.profiles.active=dev
# ip and port of the config server
spring.cloud.config.uri=http://localhost:8888
I have a Spring Boot Application and I have this dependency:
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-actuator</artifactId>
<version>2.1.7.RELEASE</version> // From Parent
</dependency>
I have also configured my application.yml file to expose metrics:
management:
endpoints:
web:
exposure:
include: info, health, metrics
I have also created a Metrics bean for Kafka:
#Bean
public KafkaConsumerMetrics kafkaConsumerMetrics() {
return new KafkaConsumerMetrics();
}
But when I hit the endpoint GET http://localhost:8080/actuator/metrics I get a 404 Page Not Found error.
The other endpoints (info/health) works: http://localhost:8080/actuator/info and http://localhost:8080/actuator/health
What am I missing to get micrometer data?
you can try enabling metrics from the configuration, though it should be enabled by default
management:
endpoint:
metrics:
enabled: true
More details on actuator properties https://docs.spring.io/spring-boot/docs/current/reference/html/appendix-application-properties.html#actuator-properties
I'm trying to do something I think is simple. I want to use Consul for configuration, and Vault for secrets.
What I'm looking for is a simple app like this that allows me to get config and services from Consul, and secrets from Vault.
In my example, Consul and Vault are connected, and I can see Vault in the Consul services and key-value store.
In Postman, I can query Vault directly and see my data at {{vault_url}}/v1/secret/interservice-bus/data-categorizer
Right now I cannot see Consul or Vault values in the test app.
My application.yml is simple
spring:
profiles:
active: dev
main:
banner-mode: "CONSOLE"
application:
name: test-consul
logging:
level:
org: INFO
The bootstrap.yml is where I'm doing most of my configuration. Some of it is repeated, because Spring complains if different parts are missing. I'm not sure why.
spring:
profiles:
active: vault, dev
cloud:
consul:
host: 127.0.0.1
port: 8500
config:
enabled: true
discovery:
prefer-ip-address: true
config:
discovery:
enabled: true
service-id: vault
server:
vault:
host: localhost
port: 8200
scheme: http
authentication: TOKEN
token: 00000000-0000-0000-0000-000000000000
vault:
host: localhost
port: 8200
scheme: http
connection-timeout: 5000
read-timeout: 15000
authentication: TOKEN
config:
order: -10
token: 00000000-0000-0000-0000-000000000000
discovery:
enabled: true
service-id: vault
The token is obscured above, but does match the root token for the vault.
Here's my POM
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>edu.dkist</groupId>
<artifactId>test-consul</artifactId>
<version>0.0.1-SNAPSHOT</version>
<packaging>jar</packaging>
<name>test-consul</name>
<description>Small test to see if consul can work with vault</description>
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>1.5.8.RELEASE</version>
<relativePath /> <!-- lookup parent from repository -->
</parent>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
<java.version>1.8</java.version>
<spring-cloud.version>Dalston.SR4</spring-cloud.version>
</properties>
<dependencies>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-consul-config</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-consul-all</artifactId>
</dependency>
</dependencies>
<dependencyManagement>
<dependencies>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-dependencies</artifactId>
<version>${spring-cloud.version}</version>
<type>pom</type>
<scope>import</scope>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-consul-dependencies</artifactId>
<version>1.2.1.RELEASE</version>
<type>pom</type>
<scope>import</scope>
</dependency>
</dependencies>
</dependencyManagement>
<build>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
</plugin>
</plugins>
</build>
</project>
And here's the simple app I'm trying to run
package com.example.demo;
import javax.annotation.PostConstruct;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.EnableAutoConfiguration;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.cloud.client.discovery.EnableDiscoveryClient;
import org.springframework.cloud.context.config.annotation.RefreshScope;
import org.springframework.context.annotation.Configuration;
#Configuration
#EnableAutoConfiguration
#RefreshScope
#SpringBootApplication
#EnableDiscoveryClient
public class DemoApplication {
#Autowired
org.springframework.cloud.client.discovery.DiscoveryClient discoveryClient;
#Value("${test}")
private String testString;
#Value("${interservice-bus.data-categorizer.username}")
private String user;
#Value("${interservice-bus.data-categorizer.password}")
private String password;
#PostConstruct
public void setRabbitMQLocation() {
if (discoveryClient == null) {
System.out.println("******************************ERROR: Discovery Client is null******************************");
return;
}
System.out.println("****************************** Discovery Client is ACTIVE ******************************");
org.springframework.cloud.client.ServiceInstance serviceInstance =
discoveryClient.getInstances("interservice-bus")
.stream()
.findFirst()
.orElseThrow(() -> new RuntimeException("ERROR: " + "interservice-bus" + " not found"));
System.out.println("****************************** ISB is " + serviceInstance.getHost() + ":" + serviceInstance.getPort() + " ******************************");
}
public static void main(String[] args) {
SpringApplication.run(DemoApplication.class, args);
}
}
It's not finding "test". I am trying to pull the username and password from Vault, and test from Consul.
I'm getting
Error starting ApplicationContext. To display the conditions report re-run your application with 'debug' enabled.
2018-06-20 16:30:33.580 ERROR 92722 --- [ main] o.s.boot.SpringApplication : Application run failed
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'scopedTarget.demoApplication': Injection of autowired dependencies failed; nested exception is java.lang.IllegalArgumentException: Could not resolve placeholder 'test' in value "${test}"
What I'm looking for is a simple app like this that allows me to get config and services from Consul, and secrets from Vault.
From searching on the net and looking at examples, it's not clear to me if I need to pull in the Vault dependencies as well or if I'm getting everything from Consul (and thus don't need the Vault dependencies).
To get configuration from Vault you need to add Spring Cloud Vault to your pom.
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-vault-config</artifactId>
</dependency>
With Spring Cloud Dalston.SR4 (which you used in your pom) you will receive Spring Cloud Vault 1.0.2. So be aware that this version has less stuff than newer one (e.g. you cannot use discovery feature). If you want to use it, consider using at least Edgware.SR4 which comes with Spring Cloud Vault 1.1.1.RELEASE. You can always check version information here: https://github.com/spring-projects/spring-cloud/wiki#release-notes. Anyway, it still will work, but you need to configure it.
Your bootstrap.yml should look like this.
spring:
application:
name: test-consul
profiles:
active: dev
vault:
fail-fast: true
host: localhost
port: 8200
scheme: http
authentication: TOKEN
config:
order: -10 # not sure that you need this, didn't investigate
token: <your_token>
Spring Cloud Vault will search configuration in Vault based on your application name and profile using Vault HTTP API. So you need to store information according to next patterns. (See documentation for Spring Cloud Vault 1.0.2)
/secret/{application}/{profile}
/secret/{application}
/secret/{default-context}/{profile}
/secret/{default-context}
Example:
vault write secret/test-consul/dev username=test_user
In this case during application start Spring Cloud Vault will get properties from secret/test-consul/dev and create Property Source from them. To get value of this test property you should use only key. Example:
#Value("${username}")
private String user; // will be resolved to "test_user"
Getting properties from Consul KV storage is similar. bootstrap.yml file is:
spring:
profiles:
active: dev
application:
name: test-consul
cloud:
consul:
host: localhost
port: 8500
config:
enabled: true
Spring Cloud Consul will use next patterns (all of them) for getting key-values from Consul using HTTP API. See documentation for Spring Cloud Consul 1.2.1 (it comes with Dalston.SR4):
config/testApp,dev/
config/testApp/
config/application,dev/
config/application/
So you need to save data accordingly. I did it from Consul GUI by creating folders in KV Storage, using console it should look similarly to this:
consul kv put config/test-consul/test testvalue
By getting data using HTTP API Spring Cloud Consul will create a property source from properties stored there and you will be able to get this using
#Value("${test}")
private String testString; // will be resolved to "testvalue"
You can combine 2 configurations and use Spring Cloud Consul and Vault simultaneously. I did't use application.yml at all for this demo.
Hope this long post will help you :)
I have a simple Maven Spring-Boot Application (Java) and am using Prometheus to collect metric information from it. I have all of the necessary Prometheus dependencies in my pom file, and I have included the #EnablePrometheusEndpoint annotation to my #SpringBootApplication class, but when I run my application and try to access the metrics on localhost:8080/prometheus (which I think is the default endpoint where Prometheus sends instrumentation metrics?), I get a 401 error. Did I instrument my application correctly such that metrics can be collected by Prometheus using the #EnablePrometheusEndpoint annotation? Where does Prometheus show my instrumented metrics (is it in localhost:8080/prometheus)? I've also tried looking for these metrics in localhost:8080/metrics, but no luck. Any help is greatly appreciated.
#SpringBootApplication
#RestController
#EnablePrometheusEndpoint
public class Example {
//Just a logger that keeps track of relevant information:
private static final Logger LOGGER = Logger.getLogger(Example.class.getName());
//counter for counting how many times an endpoint has been hit
static final Counter myCounter = Counter.build()
.name("CounterName") //note: by convention, counters should have "_total" suffix
.help("Total requests recorded by a specific endpoint")
.labelNames("status")
.register();
#RequestMapping("/hello")
String hello() {
myCounter.labels("customLabel1").inc(); //increment the number of requests by one
LOGGER.log(Level.INFO, "Number of times /hello has been hit: " + myCounter.labels("customLabel1").get());
return "Hello world! This is an example response!";
}
#RequestMapping("/homepage")
String homePage() {
myCounter.labels("customLabel2").inc(); //increment the number of requests by one
LOGGER.log(Level.INFO, "Number of times /homepage has been hit: " + myCounter.labels("customLabel2").get());
return "this is the home page!!";
}
public static void main(String[] args) throws Exception {
SpringApplication.run(Example.class, args);
}
}
Below is my pom.xml file:
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.example</groupId>
<artifactId>springboot</artifactId>
<version>0.0.1-SNAPSHOT</version>
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>1.5.8.RELEASE</version>
</parent>
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<!-- Prometheus dependencies -->
<!-- The client -->
<dependency>
<groupId>io.prometheus</groupId>
<artifactId>simpleclient</artifactId>
<version>0.1.0</version>
</dependency>
<dependency>
<groupId>io.prometheus</groupId>
<artifactId>simpleclient_spring_boot</artifactId>
<version>0.1.0</version>
</dependency>
<dependency>
<groupId>io.prometheus</groupId>
<artifactId>simpleclient_servlet</artifactId>
<version>0.1.0</version>
</dependency>
<!-- Hotspot JVM metrics -->
<dependency>
<groupId>io.prometheus</groupId>
<artifactId>simpleclient_hotspot</artifactId>
<version>0.1.0</version>
</dependency>
<!-- Exposition HTTPServer -->
<dependency>
<groupId>io.prometheus</groupId>
<artifactId>simpleclient_httpserver</artifactId>
<version>0.1.0</version>
</dependency>
<!-- Pushgateway exposition -->
<dependency>
<groupId>io.prometheus</groupId>
<artifactId>simpleclient_pushgateway</artifactId>
<version>0.1.0</version>
</dependency>
<!-- Spring Boot Actuator for exposing metrics -->
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-actuator</artifactId>
<version>1.5.8.RELEASE</version>
</dependency>
</dependencies>
</project>
And below is my prometheus.yml file, set (I think) to scrape my spring-boot app on localhost:8080 in the last few lines:
# my global config
global:
scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
# scrape_timeout is set to the global default (10s).
# Attach these labels to any time series or alerts when communicating with
# external systems (federation, remote storage, Alertmanager).
external_labels:
monitor: 'codelab-monitor'
# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
# - "first.rules"
# - "second.rules"
# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
# The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
- job_name: 'prometheus'
# metrics_path defaults to '/metrics'
# scheme defaults to 'http'.
scrape_interval: 5s
static_configs:
- targets: ['localhost:9090']
#The following lines are meant to monitor my spring boot app
- job_name: 'hello_world_spring_boot'
scrape_interval: 5s
static_configs:
- targets: ['localhost:8080']
This is what I see when I look at my prometheus dashboard, running on localhost:9090
Prometheus dashboard status 401 message
Your Prometheus is configured to look for metrics at http://localhost:8080/metrics (default) whereas #EnablePrometheusEndpoint exposes the metrics at http://localhost:8080/prometheus.
So, you should set metrics_path to prometheus in your prometheus.yml.
The 401 happens, because by default the metrics endpoint of Spring Boot Actuator is protected.
Try adding endpoints.prometheus.sensitive: false in your application.properties or manifest. This will make your prometheus endpoint unprotected and it will be available for scrape.
For publishing custom metrics into Prometheus, you need to use a Prometheus Pushgateway.
More documentation can be found on these two links
https://github.com/prometheus/pushgateway
https://prometheus.io/docs/practices/pushing/