I am currently working on Alert manager and trying to handle multiple Alerts from Prometheus with different Id under same receiver:
global:
resolve_timeout: 5m
route:
group_by: ['alertname']
group_wait: 10s
group_interval: 10s
receiver: 'web.hook'
routes:
- receiver: "web.hook"
continue: true
- receiver: "abc.hook"
match:
id: 1234567
severity: CRITICAL
continue: true
receivers:
- name: 'abc.hook'
webhook_configs:
- url: 'http://localhost:8080/services/alert'
- name: 'web.hook'
webhook_configs:
- url: 'http://localhost:8005/'
inhibit_rules:
- source_match:
severity: 'critical'
target_match:
severity: 'warning'
equal: ['alertname', 'dev', 'instance']
So in above configuration under **route ** section, we have routes which contains receiver with name: "abc.hook"
I have two alerts with different Id (1234567 and 8765432)
- receiver: "abc.hook"
match:
id: 1234567
severity: CRITICAL
continue: true
Is it possible to allow both alerts with different Ids(1234567 and 8765432) under receiver mentioned above.
I tried finding something like this over google but found nothing helpful.
I also tried something like this:
- receiver: "abc.hook"
match:
id: 1234567|8765432
severity: CRITICAL
continue: true
Any helpful information how to achieve above scenario will be appreciated.
Related
I'm using deck in a CI pipeline to sync configurations to Kong from a declarative yaml file, like this:
_format_version: "1.1"
_info:
defaults: {}
select_tags:
- ms-data-export
services:
- connect_timeout: 60000
enabled: true
host: <the-host-name>
name: data-export-api
path: /api/download
port: <the-port>
protocol: http
read_timeout: 60000
retries: 5
routes:
- name: data-export
https_redirect_status_code: 426
path_handling: v0
preserve_host: false
regex_priority: 0
request_buffering: true
response_buffering: true
strip_path: true
paths:
- /api/download
protocols:
- http
plugins:
- config:
bearer_only: "yes"
client_id: kong
...
...
The error I'm getting occurs while running deck sync --kong-addr <kong-gateway> -s <the-above-yaml-file>, and when there are no actual changes to sync from the file (because the particular service already exists), and it says:
creating service data-export-api
Summary:
Created: 0
Updated: 0
Deleted: 0
Error: 1 errors occurred:
while processing event: {Create} service data-export-api failed: HTTP status 409 (message: "UNIQUE violation detected on '{name=\"data-export-api\"}'")
data-export-api is the name of the service that already exists in kong and deck tries to create.
Is there a way to avoid this error?
I have two problems that I can't solve because I don't know if I'm missing something or not..
Here is my promising configuration, and I would therefore like to receive alerts via sms or via pushover, but it does not work.
global:
resolve_timeout: 5m
route:
group_by: ['critical']
group_wait: 30s
group_interval: 180s
repeat_interval: 300s
receiver: myIT
receivers:
- name: 'myIT'
email_configs:
- to: me#myfirm
from: me#myfirm
smarthost: ssl0.ovh.net:587
auth_username: 'me#myfirm'
auth_identity: 'me#myfirm'
auth_password: 'ZZZZZZZZZZZZZZZZZ'
- name: Teams
webhook_configs:
- url: 'https://teams.microsoft.com/l/channel/19%3xxxxxxxxyyyyuxxxab%40thread.tacv2/Alertes?groupId=xxxxxxxxyyyyuxxx0&t enantId=3caa0abd-0122-496f-a6cf-73cb6d3aaadd'
send_resolved: true
- name: Sms
webhook_configs:
- url: 'https://www.ovh.com/cgi-bin/sms/http2sms.cgi?&account=sms-XXXXXXX-1&login=XXXXX&password=XXXXXXX&from=XXXXXX&to=0123456789&message=Alert '
send_resolved: true
- name: pushover
pushover_configs:
- user_key: xxxxxxxxyyyyuxxx
token: xxxxxxxxyyyyuxxx
For the pushover part, it works via my grafana (and still not all the time). For the http2sms, it works all the time via a browser.
But for both it doesn't work under alertmanager. AND I would like to be able to differentiate the alerts. The simple warnign in teams or by email for example, and criticize them by sms.
Did I forget to install something?
Does anyone have a configuration that could look like this need? Thank you
Well. I found.
route:
group_by: ['critical']
group_wait: 30s
group_interval: 180s
repeat_interval: 300s
receiver: myIT
receivers:
- name: 'myIT'
email_configs:
- to: me#myfirm
from: me#myfirm
smarthost: ssl0.ovh.net:587
auth_username: 'me#myfirm'
auth_identity: 'me#myfirm'
auth_password: 'ZZZZZZZZZZZZZZZZZ'
webhook_configs:
- url: 'https://teams.microsoft.com/l/channel/19%3xxxxxxxxyyyyuxxxab%40thread.tacv2/Alertes?groupId=xxxxxxxxyyyyuxxx0&t enantId=3caa0abd-0122-496f-a6cf-73cb6d3aaadd'
send_resolved: true
pushover_configs:
- user_key: xxxxxxxxyyyyuxxx
token: xxxxxxxxyyyyuxxx
It works fine like that.
I am trying to activate my spring boot application endpoints with the alerts, for the required event that is defined in the alert rules of prometheus is broken, so that I want to add my application endpoints as a receiver to receive alerts from the prometheus alertmanager. Can anyone please suggest how to configure endpoint as a receiver to this receiver label, instead of any other push notifiers?
- receiver: 'frontend-pager'
group_by: [product, environment]
matchers:
- team="frontend"
I think 'webhook receiver' can help you. More information can refer doc https://prometheus.io/docs/alerting/latest/configuration/#webhook_config
This is an example of a webhook alert created based on blackbox_exporter's metric scraping.
prometheus rule setting
You need to create rule(s) to trigger alert, defined a rule named 'http_health_alert' here as example.
groups:
- name: http
rules:
- alert: http_health_alert
expr: probe_success == 0
for: 3m
labels:
type: http_health
annotations:
description: Health check for {{$labels.instance}} is down
Alertmanager setting
'match' is set to http_health_alert, the alert will be sent to'http://example.com/alert/receiver' via HTTP/POST method (I think you will prepare in advance).
The alert will post JSON format to the configured endpoint 'http://example.com/alert/receiver'. And you can also customize different receiving methods or receiving information in the endpoint/program for different label contents.
global:
route:
group_by: [alertname, env]
group_wait: 30s
group_interval: 3m
repeat_interval: 1h
routes:
- match:
alertname: http_health_alert
group_by: [alertname, env]
group_wait: 30s
group_interval: 3m
repeat_interval: 1h
receiver: webhook_receiver
receivers:
- name: webhook_receiver
webhook_configs:
- send_resolved: true
url: http://example.com/alert/receiver
- name: other_receiver
email_configs:
- send_resolved: true
to: xx
from: xxx
I have the following template in my sam function:
Resources:
TagChangedFunction:
Type: AWS::Serverless::Function
Properties:
CodeUri: tag_changed_function
Handler: tag_changed/app.lambda_handler
Runtime: python3.8
Policies:
- VPCAccessPolicy: {}
- Statement:
- Sid: EC2DescribeInstancesPolicy
Effect: "Allow"
Action:
- ec2:DescribeInstances
Resource: '*'
VpcConfig:
SubnetIds:
- sg-061328bxxxxx
SecurityGroupIds:
- subnet-03afd77xxxxx
Events:
TagChanged:
Type: CloudWatchEvent
Properties:
Pattern:
source:
- aws.tag
detail-type:
- Tag Change on Resource
(I masked the SubnetIds and SecurityGroupIds in the template with xxxxx).
But when I build and try to upload my code to aws, I get the following error message:
2 validation errors detected: Value
'[subnet-061328bxxxxx]' at
'vpcConfig.securityGroupIds' failed to satisfy
constraint: Member must satisfy constraint: [Member must
have length less than or equal to 1024, Member must have
length greater than or equal to 0, Member must satisfy
regular expression pattern: ^sg-[0-9a-z]*$]; Value
'[sg-03afd77xxxxx]' at 'vpcConfig.subnetIds' failed
to satisfy constraint: Member must satisfy constraint:
[Member must have length less than or equal to 1024,
Member must have length greater than or equal to 0,
Member must satisfy regular expression pattern:
^subnet-[0-9a-z]*$] (Service: AWSLambdaInternal; Status
Code: 400; Error Code: ValidationException; Request ID:
641be279-a48f-4249-b0a1-3e221f8bbdf
(again masking with xxxxxx)
As far as I can see, the regex constraints are satisfied. Do anyone see what is wrong in the template?
If I remove the VpcConfig section, it uploads fine.
You're giving sg ID in subnet section and Subnet ID in SG section. Kindly try the below
Resources:
TagChangedFunction:
Type: AWS::Serverless::Function
Properties:
CodeUri: tag_changed_function
Handler: tag_changed/app.lambda_handler
Runtime: python3.8
Policies:
- VPCAccessPolicy: {}
- Statement:
- Sid: EC2DescribeInstancesPolicy
Effect: "Allow"
Action:
- ec2:DescribeInstances
Resource: '*'
VpcConfig:
SubnetIds:
- subnet-03afd77xxxxx
SecurityGroupIds:
- sg-061328bxxxxx
Events:
TagChanged:
Type: CloudWatchEvent
Properties:
Pattern:
source:
- aws.tag
detail-type:
- Tag Change on Resource
I've trying to find a way for send alert notification on my prometheus server to line-notify.I checked alert rules configure status on prometheus is OK and alert rules can detect event normally, this my config.yml for alertmanager
global:
resolve_timeout: 5m
route:
receiver: "line-noti"
# group_by: ['test-node-linux', 'test-node-windows', 'test-container-exporter', 'test-jmx-exporter']
group_interval: 10s
repeat_interval: 1m
receivers:
- name: 'line-noti'
webhook_configs:
- url: 'https://notify-api.line.me/api/notify'
send_resolved: true
http_config:
bearer_token: [my_token]
but it doesn't send any messages to line-notify
How can I do for solved this case?
The problem in the receiver's name, you have double quotation marks ". However, the name of receiver should be either with single apostrophes ' or completely without.
Also the url can be without apostrophes.
Try this:
global:
resolve_timeout: 5m
route:
receiver: line-noti
# group_by: ['test-node-linux', 'test-node-windows', 'test-container-exporter', 'test-jmx-exporter']
group_interval: 10s
repeat_interval: 1m
receivers:
- name: line-noti
webhook_configs:
- url: https://notify-api.line.me/api/notify
send_resolved: true
http_config:
bearer_token: [my_token]