Please note: my prometheus is running using ubuntu terminal and my springboot application is running on windows. Seems like my ubuntu is not able to connect with the localhost of windows.
I have created springboot metrics using "actuator" and my metrics are being exposed at "http/localhost:8080/actuator/prometheus".
My application.yml configuration in my springboot application looks like this:
management:
endpoints:
web:
exposure:
include: prometheus
The configuration file of prometheus i.e. prometheus.yml is as below:
scrape_configs:
# The job name is added as a label `job=<job_name>` to any timeseries scraped from
this config.
- job_name: "services"
static_configs:
- targets: ["localhost:8080"]
metrics_path: '/actuator/prometheus'
Despite this configuration, i see "target" as down in prometheus interface. It says Get "http://localhost:8080/actuator/prometheus": dial tcp 127.0.0.1:8080: connect: connection refused . Why is prometheus not able to pick the metrics at localhost?
Related
I am running a Docker bundle with these images on my server
- SpringBoot app : PORT 18081
- Granafa : PORT 3001
- PostgreSQL : PORT 5432
- Prometheus : PORT 9090
and I would like to set up Prometheus to scrape from Springboot with this prometheus.yml configuration:
#My global config
global:
scrape_interval: 15s
evaluation_interval: 15s
alerting:
alertmanagers:
- static_configs:
- targets:
# - alertmanager:9093
rule_files:
# - "first_rules.yml"
# - "second_rules.yml"
scrape_configs:
- job_name: prometheus
static_configs:
- targets: ['localhost:9090']
- job_name: spring-actuator
scrape_interval: 5s
scrape_timeout: 5s
metrics_path: /actuator/prometheus
scheme: http
static_configs:
- targets: ['172.30.0.9:18081']
where 172.30.0.9 is the docker internal IP for my SpringBoot application obtained with this command:
docker inspect -f '{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}' <container-id>
I checked the Prometheus Dashboard on ip:9090 and I was able to observe that the prometheus job is succesfully scrapped, but not the endpoint from the Spring application.
However, if I perform a curl on the VM machine curl http://172.30.0.9:18081/actuator/prometheus, I succesfully returns all the prometheus metrics.
I have tried to set as target:
localhost:18081
external_ip:18081
container-name:18081
host.docker.internal:18081
but Prometheus is still not accessing the endpoint as expected.
Did I miss anything to configure?
I see some things you may remove since they are redundant, you can try and use the following for scrape_configs (prometheus self-scrape is not necessary as well as some settings - since you defined global):
scrape_configs:
- job_name: 'spring-actuator'
metrics_path: '/actuator/prometheus'
static_configs:
- targets: ['172.30.0.9:18081']
There might be some layout issue.
How to configure below using the multiple agent port below, with metricbeat and jolokia to monitor multiple tomcat metric on One Linux Server...Would it be fine to configure the way below tomcat metricbeat events.module, can I get accurate separate tomcat metric in ELK?
cat /etc/metricbeat/modules.d/tomcat.yml
-module: tomcat
metricsets: ['threading', 'cache', 'memory', 'requests']
period: 10s
hosts: ['localhost:8778','localhost:7777']
path: "/jolokia/?ignoreErrors=true&canonicalNaming=false"
I've written a very basic Spring Boot 2 application that connects to Zookeeper for service discovery (by using spring-cloud-starter-zookeeper-discovery).
The application gets registered at /services/example-service with the following value:
{"name":"example-service","id":"cb14ad15-4d33-4f1c-a420-29980ddf2fa8","address":"bf3fb9191373","port":8080,"sslPort":null,"payload":{"#class":"org.springframework.cloud.zookeeper.discovery.ZookeeperInstance","id":"application-1","name":"example-service","metadata":{}},"registrationTimeUTC":1524120820273,"serviceType":"DYNAMIC","uriSpec":{"parts":[{"value":"scheme","variable":true},{"value":"://","variable":false},{"value":"address","variable":true},{"value":":","variable":false},{"value":"port","variable":true}]}}
The address is an id because I've deployed the stack with Docker.
My Prometheus configuration looks like this:
- job_name: 'example-service'
metrics_path: '/actuator/prometheus'
serverset_sd_configs:
- servers:
- zookeeper:2181
paths:
- '/services/example-service'
The service discovery page of Prometheus shows the following discovered labels:
__address__=":0" __meta_serverset_endpoint_host="" __meta_serverset_endpoint_port="0" __meta_serverset_path="/services/example-service/cb14ad15-4d33-4f1c-a420-29980ddf2fa8" __meta_serverset_shard="0" __meta_serverset_status="" __metrics_path__="/actuator/prometheus" __scheme__="http" job="example-service"
Any idea why __address__ is :0?
Serverset discovery is a particular way of using Zookeeper, which your application is not following. In this case you probably want file service discovery.
Serverset use config as below:
{"serviceEndpoint":{"host":"localhost","port":9100},"additionalEndpoints":{},"status":"ALIVE"}
I want to deploy Prometheus to Cloud Foundry without using Docker container. When I try to deploy it with the standard Cloud Foundry Go Buildpack I get the following error:
can't load package: package prometheus: no buildable Go source files in /tmp/tmp.vv4iyDzMvE/.go/src/prometheus
Which somehow makes sense, because there are actually no sources in the root directory and the project is compiled with the Prometheus utility tool.
Is there any way to deploy Prometheus to Cloud Foundry, like using another Buildpack or something?
I had the same question, but (just today) came up with a slightly different solution, that seemed easier to me.
I used the prometheus-2.2.1-linux-amd64 binary build.
I modified the prometheus.yml to use the default port 8080 as a target (last line):
# my global config
global:
scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
# scrape_timeout is set to the global default (10s).
# Alertmanager configuration
alerting:
alertmanagers:
- static_configs:
- targets:
# - alertmanager:9093
# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
# - "first_rules.yml"
# - "second_rules.yml"
# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
# The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
- job_name: 'prometheus'
# metrics_path defaults to '/metrics'
# scheme defaults to 'http'.
static_configs:
- targets: ['localhost:8080'] ###### Only changed this line
Then I added a manifest.yml
---
applications:
- name: prometheus
instances: 1
buildpack: https://github.com/cloudfoundry/binary-buildpack.git
command: ./prometheus --config.file=prometheus.yml --web.listen-address=:8080 --web.enable-admin-api
memory: 1024M
random-route: true
That is using the binary-buildpack, and tells prometheus to startup the server listening on port 8080.
2 files changes and this:
cf push
Now I have prometheus running in my space on Pivotal Web Services.
Prometheus is a TSDB. And it is intended to consume gigabytes and gigabytes of data.
On a Cloud Foundry platform, you are limited by available resources.
So, why deploy Prometheus to Cloud Foundry?
Why not spin up a standalone bosh director and deploy Prometheus through the director as a Bosh deployment, and a standalone. Then inject it as a CUPS into Cloud Foundry?
I am just curious and trying to understand the use case.
Ok, after digging around a bit I got the whole thing working as follows
manifest.yml
---
applications:
- name: prometheus
instances: 1
buildpack: https://github.com/cloudfoundry/go-buildpack.git
command: prometheus
env:
GOPACKAGENAME: github.com/prometheus/prometheus
GO_INSTALL_PACKAGE_SPEC: github.com/prometheus/prometheus/cmd/prometheus
memory: 1000M
BUT in order to listen on the right port, the only solution I could find is adding the following to the cmd/prometheus/config.go file to the beginning of the init() function
port := ":9090"
if s := os.Getenv("PORT"); s != "" {
port = ":"+s
}
and then changing the following part (also in the init() function)
cfg.fs.StringVar(
&cfg.web.ListenAddress, "web.listen-address", ":9090",
"Address to listen on for the web interface, API, and telemetry.",
)
to
cfg.fs.StringVar(
&cfg.web.ListenAddress, "web.listen-address", port,
"Address to listen on for the web interface, API, and telemetry.",
)
After that you can simply deploy the application with cf push and everything should work as a charm
I'm running a Spring Boot application as a Docker container. This works fine so far, but it's giving me some head aches when trying to use Spring Cloud Consul as well. It reads the configuration from the Consul KVS just fine, but the health checks seem to be acting up.
The default health check uses the hostname of the docker container, for example http://users-microservice/health. Obviously this won't resolve when accessed from Consul.
No problem, the documentation mentions that you can use healthCheckPath in your bootstrap.yml file to configure it. This is what I have now:
spring:
application:
name: users-microservice
cloud:
consul:
host: myserver.com
port: 8500
config:
prefix: API-CONFIG
profileSeparator: '__'
discovery:
tags: users-microservice
healthCheckPath: http://myserver.com:${server.port}/status
healthCheckInterval: 30s
Unfortunately, this variable seems to be used in a very different manner from what I expected. This is what Consul is trying to reach:
Get http://users:18090http//myserver.com:18090/status: dial tcp: unknown port tcp/18090http
How can I fix this? Is there some undocumented configuration parameter that I should set?
Use spring.cloud.consul.discovery.healthCheckUrl=http://myserver.com:${server.port}/status
healthCheckPath only changes the path, not host and port.