I've trying to find a way for send alert notification on my prometheus server to line-notify.I checked alert rules configure status on prometheus is OK and alert rules can detect event normally, this my config.yml for alertmanager
global:
resolve_timeout: 5m
route:
receiver: "line-noti"
# group_by: ['test-node-linux', 'test-node-windows', 'test-container-exporter', 'test-jmx-exporter']
group_interval: 10s
repeat_interval: 1m
receivers:
- name: 'line-noti'
webhook_configs:
- url: 'https://notify-api.line.me/api/notify'
send_resolved: true
http_config:
bearer_token: [my_token]
but it doesn't send any messages to line-notify
How can I do for solved this case?
The problem in the receiver's name, you have double quotation marks ". However, the name of receiver should be either with single apostrophes ' or completely without.
Also the url can be without apostrophes.
Try this:
global:
resolve_timeout: 5m
route:
receiver: line-noti
# group_by: ['test-node-linux', 'test-node-windows', 'test-container-exporter', 'test-jmx-exporter']
group_interval: 10s
repeat_interval: 1m
receivers:
- name: line-noti
webhook_configs:
- url: https://notify-api.line.me/api/notify
send_resolved: true
http_config:
bearer_token: [my_token]
Related
I am currently working on Alert manager and trying to handle multiple Alerts from Prometheus with different Id under same receiver:
global:
resolve_timeout: 5m
route:
group_by: ['alertname']
group_wait: 10s
group_interval: 10s
receiver: 'web.hook'
routes:
- receiver: "web.hook"
continue: true
- receiver: "abc.hook"
match:
id: 1234567
severity: CRITICAL
continue: true
receivers:
- name: 'abc.hook'
webhook_configs:
- url: 'http://localhost:8080/services/alert'
- name: 'web.hook'
webhook_configs:
- url: 'http://localhost:8005/'
inhibit_rules:
- source_match:
severity: 'critical'
target_match:
severity: 'warning'
equal: ['alertname', 'dev', 'instance']
So in above configuration under **route ** section, we have routes which contains receiver with name: "abc.hook"
I have two alerts with different Id (1234567 and 8765432)
- receiver: "abc.hook"
match:
id: 1234567
severity: CRITICAL
continue: true
Is it possible to allow both alerts with different Ids(1234567 and 8765432) under receiver mentioned above.
I tried finding something like this over google but found nothing helpful.
I also tried something like this:
- receiver: "abc.hook"
match:
id: 1234567|8765432
severity: CRITICAL
continue: true
Any helpful information how to achieve above scenario will be appreciated.
I have a service named alpha (created using python-django) that runs on http://127.0.0.1:9000 and has these two endpoings
/health returns {"health": "OK"} status 200
/codes/<str:code> returns {"code": code} status 200
I also have a kong api-gateway in the db-less declarative mode that runs on localhost port 80
in kong.yaml I have two services
services:
- name: local-alpha-health
url: http://host.docker.internal:9000/health
routes:
- name: local-alpha-health
methods:
- GET
paths:
- /alpha/health
strip_path: true
- name: local-alpha-code
url: http://host.docker.internal:9000/code/ # HOW TO WRITE THIS PART???
routes:
- name: local-alpha-code
methods:
- GET
paths:
- /alpha/code/(?<appcode>\d+) # Is this right???
strip_path: true
If I send a GET request to http://127.0.0.1/alpha/health it returns {"health": "OK"} status 200 which shows kong is working.
I want to send a request such as http://127.0.0.1/alpha/code/123 and I expect to receive {"code": 123} status 200 but I don't know how to setup kong.yaml file to do this. If I send a request to http://127.0.0.1/alpha/code/123 I get 404 from (from the alpha django application) which means kong is routing the request to alpha service but if I send a request to http://127.0.0.1/alpha/code/abc I get {"message": "no Route matched with those values"} which shows the regex is working
I could do this
services:
- name: local-alpha-health
url: http://host.docker.internal:9000/
routes:
- name: local-alpha-health
methods:
- GET
paths:
- /alpha
strip_path: true
Then a request sent to http://127.0.0.1/alpha/code/123 would go to ``http://127.0.0.1:9000/code/123` but I cannot control with regex
Any idea How to route requests to a dynamic endpoint on kong api-gateway?
This content seems related but cannot figure it out how to set it up
https://docs.konghq.com/gateway-oss/2.5.x/proxy/
Note that a request like http://127.0.0.1/alpha/code/abc will indeed not match the rule you have added, because of the \d+ (which matches one or more digits). Also, http://127.0.0.1/alpha/code/123 will reach the upstream as a request to /, since you have strip_path set to true.
I have tested your example with some minor tweaks to proxy to a local httpbin service, which has a similar endpoint (/status/<code>).
Start a local httpbin service:
$ docker run --rm -d -p "8080:80" kennethreitz/httpbin
Start Kong with the following config:
_format_version: "2.1"
services:
- name: local-alpha-code
url: http://localhost:8080
routes:
- name: local-mockbin-status
methods:
- GET
paths:
- /status/(?<appcode>\d+)
strip_path: false
Note that strip_path is set to false, so the entire matching path is proxied to the upstream.
Test it out with:
$ http :8000/status/200
HTTP/1.1 200 OK
I want to drop lines in Promtail using an AND condition from two different JSON fields.
I have JSON log lines like this.
{"timestamp":"2022-03-26T15:40:41+00:00","remote_addr":"1.2.3.4","remote_user":"","request":"GET / HTTP/1.1","status": "200","body_bytes_sent":"939","request_time":"0.000","http_referrer":"http://5.6.7.8","http_user_agent":"user agent 1"}
{"timestamp":"2022-03-26T15:40:41+00:00","remote_addr":"1.2.3.4","remote_user":"","request":"GET /path HTTP/1.1","status": "200","body_bytes_sent":"939","request_time":"0.000","http_referrer":"http://5.6.7.8","http_user_agent":"user agent 1"}
{"timestamp":"2022-03-26T15:40:41+00:00","remote_addr":"1.2.3.4","remote_user":"","request":"GET / HTTP/1.1","status": "200","body_bytes_sent":"939","request_time":"0.000","http_referrer":"http://5.6.7.8","http_user_agent":"user agent 2"}
My local Promtail config looks like this.
clients:
- url: http://localhost:3100/loki/api/v1/push
scrape_configs:
- job_name: testing-my-job-drop
pipeline_stages:
- match:
selector: '{job="my-job"}'
stages:
- json:
expressions:
http_user_agent:
request:
- drop:
source: "http_user_agent"
expression: "user agent 1"
# I want this to be AND
- drop:
source: "request"
expression: "GET / HTTP/1.1"
drop_counter_reason: my_job_healthchecks
static_configs:
- labels:
job: my-job
Using a Promtail config like this drops lines using OR from my two JSON fields.
How can I adjust my config so that I only drop lines where http_user_agent = user agent 1 AND request = GET / HTTP/1.1?
If you provide multiple options they will be treated like an AND clause, where each option has to be true to drop the log.
If you wish to drop with an OR clause, then specify multiple drop stages.
https://grafana.com/docs/loki/latest/clients/promtail/stages/drop/#drop-stage
Drop logs by time OR length
Would drop all logs older than 24h OR longer than 8kb bytes
- json:
expressions:
time:
msg:
- timestamp:
source: time
format: RFC3339
- drop:
older_than: 24h
- drop:
longer_than: 8kb
Drop logs by regex AND length
Would drop all logs that contain the word debug AND are longer than 1kb bytes
- drop:
expression: ".*debug.*"
longer_than: 1kb
clients:
- url: http://localhost:3100/loki/api/v1/push
scrape_configs:
- job_name: testing-my-job-drop
pipeline_stages:
- match:
selector: '{job="my-job"}'
stages:
- json:
expressions:
http_user_agent:
request:
- labels:
http_user_agent:
request:
#### method 1
- match:
selector: '{http_user_agent="user agent 1"}'
stages:
- drop:
source: "request"
expression: "GET / HTTP/1.1"
drop_counter_reason: my_job_healthchecks
## they are both conditions match will drop
#### method 2
- match:
selector: '{http_user_agent="user agent 1",request="GET / HTTP/1.1"}'
action: drop
#### method 3, incase regex pattern.
- match:
selector: '{http_user_agent="user agent 1"} |~ "(?i).*GET / HTTP/1.1.*"'
action: drop
static_configs:
- labels:
job: my-job
match stage include match stage.
I have two problems that I can't solve because I don't know if I'm missing something or not..
Here is my promising configuration, and I would therefore like to receive alerts via sms or via pushover, but it does not work.
global:
resolve_timeout: 5m
route:
group_by: ['critical']
group_wait: 30s
group_interval: 180s
repeat_interval: 300s
receiver: myIT
receivers:
- name: 'myIT'
email_configs:
- to: me#myfirm
from: me#myfirm
smarthost: ssl0.ovh.net:587
auth_username: 'me#myfirm'
auth_identity: 'me#myfirm'
auth_password: 'ZZZZZZZZZZZZZZZZZ'
- name: Teams
webhook_configs:
- url: 'https://teams.microsoft.com/l/channel/19%3xxxxxxxxyyyyuxxxab%40thread.tacv2/Alertes?groupId=xxxxxxxxyyyyuxxx0&t enantId=3caa0abd-0122-496f-a6cf-73cb6d3aaadd'
send_resolved: true
- name: Sms
webhook_configs:
- url: 'https://www.ovh.com/cgi-bin/sms/http2sms.cgi?&account=sms-XXXXXXX-1&login=XXXXX&password=XXXXXXX&from=XXXXXX&to=0123456789&message=Alert '
send_resolved: true
- name: pushover
pushover_configs:
- user_key: xxxxxxxxyyyyuxxx
token: xxxxxxxxyyyyuxxx
For the pushover part, it works via my grafana (and still not all the time). For the http2sms, it works all the time via a browser.
But for both it doesn't work under alertmanager. AND I would like to be able to differentiate the alerts. The simple warnign in teams or by email for example, and criticize them by sms.
Did I forget to install something?
Does anyone have a configuration that could look like this need? Thank you
Well. I found.
route:
group_by: ['critical']
group_wait: 30s
group_interval: 180s
repeat_interval: 300s
receiver: myIT
receivers:
- name: 'myIT'
email_configs:
- to: me#myfirm
from: me#myfirm
smarthost: ssl0.ovh.net:587
auth_username: 'me#myfirm'
auth_identity: 'me#myfirm'
auth_password: 'ZZZZZZZZZZZZZZZZZ'
webhook_configs:
- url: 'https://teams.microsoft.com/l/channel/19%3xxxxxxxxyyyyuxxxab%40thread.tacv2/Alertes?groupId=xxxxxxxxyyyyuxxx0&t enantId=3caa0abd-0122-496f-a6cf-73cb6d3aaadd'
send_resolved: true
pushover_configs:
- user_key: xxxxxxxxyyyyuxxx
token: xxxxxxxxyyyyuxxx
It works fine like that.
I am trying to activate my spring boot application endpoints with the alerts, for the required event that is defined in the alert rules of prometheus is broken, so that I want to add my application endpoints as a receiver to receive alerts from the prometheus alertmanager. Can anyone please suggest how to configure endpoint as a receiver to this receiver label, instead of any other push notifiers?
- receiver: 'frontend-pager'
group_by: [product, environment]
matchers:
- team="frontend"
I think 'webhook receiver' can help you. More information can refer doc https://prometheus.io/docs/alerting/latest/configuration/#webhook_config
This is an example of a webhook alert created based on blackbox_exporter's metric scraping.
prometheus rule setting
You need to create rule(s) to trigger alert, defined a rule named 'http_health_alert' here as example.
groups:
- name: http
rules:
- alert: http_health_alert
expr: probe_success == 0
for: 3m
labels:
type: http_health
annotations:
description: Health check for {{$labels.instance}} is down
Alertmanager setting
'match' is set to http_health_alert, the alert will be sent to'http://example.com/alert/receiver' via HTTP/POST method (I think you will prepare in advance).
The alert will post JSON format to the configured endpoint 'http://example.com/alert/receiver'. And you can also customize different receiving methods or receiving information in the endpoint/program for different label contents.
global:
route:
group_by: [alertname, env]
group_wait: 30s
group_interval: 3m
repeat_interval: 1h
routes:
- match:
alertname: http_health_alert
group_by: [alertname, env]
group_wait: 30s
group_interval: 3m
repeat_interval: 1h
receiver: webhook_receiver
receivers:
- name: webhook_receiver
webhook_configs:
- send_resolved: true
url: http://example.com/alert/receiver
- name: other_receiver
email_configs:
- send_resolved: true
to: xx
from: xxx