My Prometheus server gets its list of targets (or "services", in Consul's lingo) from Consul. I only want to monitor a subset of these targets. This should be possible via Prometheus's regex mechanism, but the correct configuration eludes me. How is this done?
I've scoured the web and there is not a single example showing how its done, so for posterity - the following configuration will drop all consul services marked with the 'ignore-at-prometheus' tag
# ignore consul services with 'ignore_at_prometheus' tag
# https://www.robustperception.io/little-things-matter/
relabel_configs:
- source_labels: ['__meta_consul_tags']
regex: '(.*),ignore-at-prometheus,(.*)'
action: drop
I've used a very similar solution to the problem using the following config. It allows to scrape only the services with a specific tag, rather than excluding services with a given tag.
Here's the scrape_configs section of my config:
scrape_configs:
- job_name: 'consul_registered_services'
scrape_interval: 5s
metrics_path: '/prometheus'
consul_sd_configs:
- server: 'my-consul-server:8500'
token: 'xyz'
relabel_configs:
- source_labels: ['__meta_consul_tags']
regex: '^.*,metrics_method=prometheus-servlet,.*$'
action: keep
- source_labels: ['__meta_consul_node']
target_label: instance
- source_labels: ['__meta_consul_service']
target_label: service
- source_labels: ['__meta_consul_tags']
target_label: tags
I then make sure to register all relevant services with the metrics_method=prometheus-servlet tag, and the rest will be ignored.
The documentation for the relabeling configuration is available here: https://prometheus.io/docs/operating/configuration/#relabel_config.
The documentation for the Consul service discovery configuration is available here: https://prometheus.io/docs/operating/configuration/#consul_sd_config.
Related
Please note: my prometheus is running using ubuntu terminal and my springboot application is running on windows. Seems like my ubuntu is not able to connect with the localhost of windows.
I have created springboot metrics using "actuator" and my metrics are being exposed at "http/localhost:8080/actuator/prometheus".
My application.yml configuration in my springboot application looks like this:
management:
endpoints:
web:
exposure:
include: prometheus
The configuration file of prometheus i.e. prometheus.yml is as below:
scrape_configs:
# The job name is added as a label `job=<job_name>` to any timeseries scraped from
this config.
- job_name: "services"
static_configs:
- targets: ["localhost:8080"]
metrics_path: '/actuator/prometheus'
Despite this configuration, i see "target" as down in prometheus interface. It says Get "http://localhost:8080/actuator/prometheus": dial tcp 127.0.0.1:8080: connect: connection refused . Why is prometheus not able to pick the metrics at localhost?
I want to use Prometheus with my spring boot project, I'm new in Prometheus that way i do not know why I get error describe in picture
My prometheus.yml like below
global:
scrape_interval: 10s
scrape_configs:
- job_name: 'spring_micrometer'
metrics_path: '/actuator/prometheus'
scrape_interval: 5s
static_configs:
- targets: ['192.168.43.71:8080/app']
I run prometheus by this command docker run -d -p 9090:9090 -v <path-to-prometheus.yml>:/etc/prometheus/prometheus.yml prom/prometheus
I notice my ip not show in Prometheus targets page :
Normally Endpoint IP must be like 192.168.43.71:8080/app/actuator/prometheus but I get http://localhost:9090/metrics and when I click in it, i get error describe in picture 1
What I do wrong ?!, anyone can help me to resolve this issue and thanks.
You cannot do this - targets: ['192.168.43.71:8080/app']. Try the following:
global:
scrape_interval: 10s
scrape_configs:
- job_name: 'spring_micrometer'
metrics_path: '/app/actuator/prometheus/metrics'
scrape_interval: 5s
static_configs:
- targets: ['192.168.43.71:8080']
Why does your config not work? Take a look at the config docs here: https://prometheus.io/docs/prometheus/latest/configuration/configuration/#host
targets is a collection of host and host must be a "valid string consisting of a hostname or IP followed by an optional port number".
blackbox problem
I added blackbox_exporter in my docker-compose.yml:
blackbox_exporter:
container_name: blackbox_exporter
image: prom/blackbox-exporter
restart: always
ports:
- "9115:3115"
networks:
- monitor-net
labels:
org.label-schema.group: "monitoring"
I added job into prometheus.yml:
- job_name: 'blackbox'
metrics_path: /probe
params:
module: [http_2xx] # Look for a HTTP 200 response.
static_configs:
- targets: ['google.com','amazon.com'] # Target to probe with https.
relabel_configs:
- source_labels: [__address__]
target_label: __param_target
- source_labels: [__param_target]
target_label: instance
- target_label: __address__
replacement: blackbox_exporter:9115 # The blackbox exporter's real hostname:port.
I added this dashboard in grafana: https://grafana.com/dashboards/5345 because screenshot on this page was exactly what I need.
Alas, I have only summary graphics without legend, without site-specific chapters.
You can see screenshot here:
Where my actions were wrong? What can I do with it?
In the config you posted, you relabel the blackbox exporter label from __param_target to instance but the dashboard uses target for all the filters and also for the templating variable.
Either change your config to
- source_labels: [__param_target]
target_label: target
or adjust the queries and settings in the dashboard to use instance.
I've written a very basic Spring Boot 2 application that connects to Zookeeper for service discovery (by using spring-cloud-starter-zookeeper-discovery).
The application gets registered at /services/example-service with the following value:
{"name":"example-service","id":"cb14ad15-4d33-4f1c-a420-29980ddf2fa8","address":"bf3fb9191373","port":8080,"sslPort":null,"payload":{"#class":"org.springframework.cloud.zookeeper.discovery.ZookeeperInstance","id":"application-1","name":"example-service","metadata":{}},"registrationTimeUTC":1524120820273,"serviceType":"DYNAMIC","uriSpec":{"parts":[{"value":"scheme","variable":true},{"value":"://","variable":false},{"value":"address","variable":true},{"value":":","variable":false},{"value":"port","variable":true}]}}
The address is an id because I've deployed the stack with Docker.
My Prometheus configuration looks like this:
- job_name: 'example-service'
metrics_path: '/actuator/prometheus'
serverset_sd_configs:
- servers:
- zookeeper:2181
paths:
- '/services/example-service'
The service discovery page of Prometheus shows the following discovered labels:
__address__=":0" __meta_serverset_endpoint_host="" __meta_serverset_endpoint_port="0" __meta_serverset_path="/services/example-service/cb14ad15-4d33-4f1c-a420-29980ddf2fa8" __meta_serverset_shard="0" __meta_serverset_status="" __metrics_path__="/actuator/prometheus" __scheme__="http" job="example-service"
Any idea why __address__ is :0?
Serverset discovery is a particular way of using Zookeeper, which your application is not following. In this case you probably want file service discovery.
Serverset use config as below:
{"serviceEndpoint":{"host":"localhost","port":9100},"additionalEndpoints":{},"status":"ALIVE"}
I am trying to get some custom application metrics captured in golang using the prometheus client library to show up in Prometheus.
I have the following working:
I have a go application which is exposing metrics on localhost:8080/metrics as described in this article:
https://godoc.org/github.com/prometheus/client_golang/prometheus
I have a kubernates minikube running which has Prometheus, Grafana and AlertManager running using the operator from this article:
https://github.com/coreos/prometheus-operator/tree/master/contrib/kube-prometheus
I created a docker image for my go app, when I run it and go to localhost:8080/metrics I can see the prometheus metrics showing up in a browser.
I use the following pod.yaml to deploy my docker image to a pod in k8s
apiVersion: v1
kind: Pod
metadata:
name: my-app-pod
labels:
zone: prod
version: v1
annotations:
prometheus.io/scrape: 'true'
prometheus.io/port: '8080'
spec:
containers:
- name: my-container
image: name/my-app:latest
imagePullPolicy: IfNotPresent
ports:
- containerPort: 8080
If I connect to my pod using:
kubectl exec -it my-app-pod -- /bin/bash
then do wget on "localhost:8080/metrics", I can see my metrics
So far so good, here is where I am hitting a wall. I could have multiple pods running this same image. I want to expose all the images to prometheus as targets. How do I configure my pods so that they show up in prometheus so I can report on my custom metrics?
Thanks for any help offered!
The kubernetes_sd_config directive can be used to discover all pods with a given tag. Your Prometheus.yml config file should have something like so:
- job_name: 'some-app'
kubernetes_sd_configs:
- role: pod
relabel_configs:
- source_labels: [__meta_kubernetes_pod_label_app]
regex: python-app
action: keep
The source label [__meta_kubernetes_pod_label_app] is basically using the Kubernetes api to look at pods that have a label of 'app' and whose value is captured by the regex expression, given on the line below (in this case, matching 'python-app').
Once you've done this Prometheus will automatically discover the pods you want and start scraping the metrics from your app.
Hope that helps. You can follow blog post here for more detail.
Note: it is worth mentioning that at the time of writing, kubernetes_sd_config is still in beta. Thus breaking changes to configuration may occur in future releases.
You need 2 things:
a ServiceMonitor for the Prometheus Operator, which specifies which services will be scraped for metrics
a Service which matches the ServiceMonitor and points to your pods
There is an example in the docs over here: https://coreos.com/operators/prometheus/docs/latest/user-guides/running-exporters.html
Can you share the prometheus config that you are using to scrape the metrics. The config will control what all sources to scrape the metrics from. Here are a few links that you can refer to : https://groups.google.com/forum/#!searchin/prometheus-users/Application$20metrics$20monitoring$20of$20Kubernetes$20Pods%7Csort:relevance/prometheus-users/uNPl4nJX9yk/cSKEBqJlBwAJ