Trying to get metrics using Prometheus server
The yaml I am using is
global:
scrape_interval: 5s
scrape_configs:
- job_name: 'student'
metrics_path: '/student/actuator/prometheus'
static_configs:
- targets: ['<HOST IP>:8080']
The command I used to run the docker file
docker run -d -p 9090:9090 -v <prometheus.yml location>:/etc/prometheus/prometheus.yml prom/prometheus --config.file=/etc/prometheus/prometheus.yml
The error I get in Prometheus dashboard
Get http://<host ip>:8080/student/actuator/prometheus: context deadline exceeded
I resolved the issue by passing --net=host as a parameter in docker
Related
I am running a Docker bundle with these images on my server
- SpringBoot app : PORT 18081
- Granafa : PORT 3001
- PostgreSQL : PORT 5432
- Prometheus : PORT 9090
and I would like to set up Prometheus to scrape from Springboot with this prometheus.yml configuration:
#My global config
global:
scrape_interval: 15s
evaluation_interval: 15s
alerting:
alertmanagers:
- static_configs:
- targets:
# - alertmanager:9093
rule_files:
# - "first_rules.yml"
# - "second_rules.yml"
scrape_configs:
- job_name: prometheus
static_configs:
- targets: ['localhost:9090']
- job_name: spring-actuator
scrape_interval: 5s
scrape_timeout: 5s
metrics_path: /actuator/prometheus
scheme: http
static_configs:
- targets: ['172.30.0.9:18081']
where 172.30.0.9 is the docker internal IP for my SpringBoot application obtained with this command:
docker inspect -f '{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}' <container-id>
I checked the Prometheus Dashboard on ip:9090 and I was able to observe that the prometheus job is succesfully scrapped, but not the endpoint from the Spring application.
However, if I perform a curl on the VM machine curl http://172.30.0.9:18081/actuator/prometheus, I succesfully returns all the prometheus metrics.
I have tried to set as target:
localhost:18081
external_ip:18081
container-name:18081
host.docker.internal:18081
but Prometheus is still not accessing the endpoint as expected.
Did I miss anything to configure?
I see some things you may remove since they are redundant, you can try and use the following for scrape_configs (prometheus self-scrape is not necessary as well as some settings - since you defined global):
scrape_configs:
- job_name: 'spring-actuator'
metrics_path: '/actuator/prometheus'
static_configs:
- targets: ['172.30.0.9:18081']
There might be some layout issue.
My actuator-prometheus metrics are reachable under: localhost:5550/linksky/actuator/prometheus
For example, I am seeing metric named "http_server_requests_seconds_count"
I have set up my prometheus with docker-compose.yml:
services:
prometheus:
image: prom/prometheus
ports:
- 9090:9090
volumes:
- ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
networks:
monitoring:
aliases:
- prometheus
networks:
monitoring:
and my prometheus.yml
scrape_configs:
- job_name: 'linksky_monitoring'
scrape_interval: 2s
metrics_path: '/actuator/prometheus'
static_configs:
- targets: ['host.docker.internal:5550']
When I am starting prometheus, I can retrieve metric named "scrape_duration_seconds" and I see
that the scrape-job is correct:
But, when I am asking for "http_server_requests_seconds_count", I get no result.
Do I expect something wrong? Why do I have only this metric in prometheus, although the "linksky_monitoring" job seems to be running?
UPDATE and SOLUTION
I need to use a tls-connection, because each request for my spring-boot app has to be with TLS.
For this issue i have extracted key and cert from my p12-Certificate and made follow config:
scrape_configs:
- job_name: 'monitoring'
scrape_interval: 2s
metrics_path: '/jReditt/actuator/prometheus'
static_configs:
- targets: ['host.docker.internal:5550']
scheme: https
tls_config:
cert_file: '/etc/prometheus/myApp.cert'
key_file: '/etc/prometheus/myApp.key'
insecure_skip_verify: true
No, it is working fine
Your metrics_path in the prometheus.yml is wrong because it's missing a part of the endpoint. It should be like below (/linksky/actuator/prometheus)
scrape_configs:
- job_name: 'linksky_monitoring'
scrape_interval: 2s
metrics_path: '/linksky/actuator/prometheus'
static_configs:
- targets: ['host.docker.internal:5550']
I have started working on the Prometheus for my microservices. I was able to achieve it initially. Now, it's time to push the actuator endpoint under the spring security. After adding the security actuator is expecting the bearer token from the Prometheus. So, how to configure the username and password in the Prometheus job so that Prometheus will get the bearer token from the login and add it as the 'Authorization' in the header for all the requests.
I'm running the Prometheus in the docker container using the commands below
1. $ docker run --name prometheus -p 9090:9090 -v prometheus.yml:/etc/prometheus/prometheus.yml -d prom/prometheus
2. $ docker run --name grafana -d -p 3000:3000 grafana/grafana
Following is the prometheus.yml file
# my global config
global:
scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
# scrape_timeout is set to the global default (10s).
# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
# - "first_rules.yml"
# - "second_rules.yml"
# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
# The job name is added as a label `job=<job_name>` to any time series scraped from this config.
- job_name: 'prometheus'
# metrics_path defaults to '/metrics'
# scheme defaults to 'http'.
static_configs:
- targets: ['127.0.0.1:9090']
- job_name: 'NL-APPLICATION'
metrics_path: '/actuator/prometheus'
scrape_interval: 5s
scheme: http
static_configs:
- targets: ['172.17.0.1:8085']
- job_name: 'NL-ADMIN-API'
metrics_path: '/actuator/prometheus'
scrape_interval: 5s
static_configs:
- targets: ['172.17.0.1:8083']
How to Instruct Prometheus to do as follow
API call to '/login' get the Bearer token using username and password
Add the Bearer token as the 'Authorization' as a header in all actuator API call
You can either specify as a file or add the token to the config
- job_name: 'test'
metrics_path: "/metrics"
scheme: "http"
bearer_token_file: /var/run/secrets/ OR bearer_token: token_here
static_configs:
- targets: ['host.com']
I want to use Prometheus with my spring boot project, I'm new in Prometheus that way i do not know why I get error describe in picture
My prometheus.yml like below
global:
scrape_interval: 10s
scrape_configs:
- job_name: 'spring_micrometer'
metrics_path: '/actuator/prometheus'
scrape_interval: 5s
static_configs:
- targets: ['192.168.43.71:8080/app']
I run prometheus by this command docker run -d -p 9090:9090 -v <path-to-prometheus.yml>:/etc/prometheus/prometheus.yml prom/prometheus
I notice my ip not show in Prometheus targets page :
Normally Endpoint IP must be like 192.168.43.71:8080/app/actuator/prometheus but I get http://localhost:9090/metrics and when I click in it, i get error describe in picture 1
What I do wrong ?!, anyone can help me to resolve this issue and thanks.
You cannot do this - targets: ['192.168.43.71:8080/app']. Try the following:
global:
scrape_interval: 10s
scrape_configs:
- job_name: 'spring_micrometer'
metrics_path: '/app/actuator/prometheus/metrics'
scrape_interval: 5s
static_configs:
- targets: ['192.168.43.71:8080']
Why does your config not work? Take a look at the config docs here: https://prometheus.io/docs/prometheus/latest/configuration/configuration/#host
targets is a collection of host and host must be a "valid string consisting of a hostname or IP followed by an optional port number".
name: Rspec
on: [push]
jobs:
build:
runs-on: [self-hosted, linux]
services:
elasticsearch:
image: docker.elastic.co/elasticsearch/elasticsearch:7.9.2
env:
discovery.type: single-node
options: >-
--health-cmd "curl http://localhost:9200/_cluster/health"
--health-interval 10s
--health-timeout 5s
--health-retries 10
redis:
image: redis
options: --entrypoint redis-server
steps:
- uses: actions/checkout#v2
- name: running tests
run: |
sleep 60
curl -X GET http://elasticsearch:9200/
I am running tests self hosted, I see on host with docker ps the containers (redis and elasticsearch) when they up to test.
I access a container of redis, install a curl and run curl -X GET http://elasticsearch:9200/ and i see a response ok before 60 sec (wait time to service up)
On step running tests I got error message "Could not resolve host: elasticsearch"
So, inside service/container redis I see a host elasticsearch but on step running tests no. What I can do?
You have to map the ports of your service containers and use localhost:host-port as address in your steps running on the GitHub Actions runner.
If you configure the job to run directly on the runner machine and your step doesn't use a container action, you must map any required Docker service container ports to the Docker host (the runner machine). You can access the service container using localhost and the mapped port.
https://docs.github.com/en/free-pro-team#latest/actions/reference/workflow-syntax-for-github-actions#jobsjob_idservices
name: Rspec
on: [push]
jobs:
build:
runs-on: [self-hosted, linux]
services:
elasticsearch:
image: docker.elastic.co/elasticsearch/elasticsearch:7.9.2
env:
discovery.type: single-node
options: >-
--health-cmd "curl http://localhost:9200/_cluster/health"
--health-interval 10s
--health-timeout 5s
--health-retries 10
ports:
# <port on host>:<port on container>
- 9200:9200
redis:
image: redis
options: --entrypoint redis-server
steps:
- uses: actions/checkout#v2
- name: running tests
run: |
sleep 60
curl -X GET http://localhost:9200/
Alternative:
Also run your job in a container. Then the job has to access the service containers by hostname.
name: Rspec
on: [push]
jobs:
build:
services:
elasticsearch:
image: docker.elastic.co/elasticsearch/elasticsearch:7.9.2
env:
discovery.type: single-node
options: >-
--health-cmd "curl http://localhost:9200/_cluster/health"
--health-interval 10s
--health-timeout 5s
--health-retries 10
redis:
image: redis
options: --entrypoint redis-server
# Containers must run in Linux based operating systems
runs-on: [self-hosted, linux]
# Docker Hub image that this job executes in, pick any image that works for you
container: node:10.18-jessie
steps:
- uses: actions/checkout#v2
- name: running tests
run: |
sleep 60
curl -X GET http://elasticsearch:9200/