I am using elastalert to trigger email when elasticsearch url is up/down.
I want to configure in such manner that elastalert will continuously trigger mail when server goes down and when server is up only one mail will be triggered.
Please find below example_frequency rule config
es_host: localhost
es_port: 9200
name: Example frequency rule
type: frequency
index: heartbeat-*
num_events: 1
realert:
seconds: 30
timeframe:
minutes: 3
filter:
- terms:
monitor.status: ["up", "down"]
alert:
- "email"
email:
- "AD#somemail.com"
Related
I transfer logfiles with filebeat to elasticsearch.
The data are analyzed with kibana.
Now to my problem:
Kibana shows not the timestamp from the logfile.
Kibana shows the time of the transmission in #timestamp.
I want to show the timestamp from the logfile in kibana.
But the timestamp in the logfile is overwritten.
Where is my fault?
Has anyone a solution for my problem?
Here a example from my logfile and the my filebeat config.
{"#timestamp":"2022-06-23T10:40:25.852+02:00","#version":1,"message":"Could not refresh JMS Connection]","logger_name":"org.springframework.jms.listener.DefaultMessageListenerContainer","level":"ERROR","level_value":40000}
## Filebeat configuration
## https://github.com/elastic/beats/blob/master/deploy/docker/filebeat.docker.yml
#
filebeat.config:
modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: false
filebeat.autodiscover:
providers:
# The Docker autodiscover provider automatically retrieves logs from Docker
# containers as they start and stop.
- type: docker
hints.enabled: true
filebeat.inputs:
- type: filestream
id: pls-logs
paths:
- /usr/share/filebeat/logs/*.log
parsers:
- ndjson:
processors:
- add_cloud_metadata: ~
output.elasticsearch:
hosts: ['http://elasticsearch:9200']
username: elastic
password:
## HTTP endpoint for health checking
## https://www.elastic.co/guide/en/beats/filebeat/current/http-endpoint.html
#
http.enabled: true
http.host: 0.0.0.0
Thanks for any support!
Based upon the question, this could be one potential option, which would be to use filebeat processors. What you could do is write that initial #timestamp value to another field, like event.ingested, using the following script below:
#Script to move the timestamp to the event.ingested field
- script:
lang: javascript
id: init_format
source: >
function process(event) {
var fieldTest = event.Get("#timestamp");
event.Put("event.ingested", fieldTest);
}
And then the last processor you write could move that event.ingested field to #timestamp again using the following processor:
#setting the timestamp field to the Date/time when the event originated, which would be the event.created field
- timestamp:
field: event.created
layouts:
- '2006-01-02T15:04:05Z'
- '2006-01-02T15:04:05.999Z'
- '2006-01-02T15:04:05.999-07:00'
test:
- '2019-06-22T16:33:51Z'
- '2019-11-18T04:59:51.123Z'
- '2020-08-03T07:10:20.123456+02:00'
I am trying to activate my spring boot application endpoints with the alerts, for the required event that is defined in the alert rules of prometheus is broken, so that I want to add my application endpoints as a receiver to receive alerts from the prometheus alertmanager. Can anyone please suggest how to configure endpoint as a receiver to this receiver label, instead of any other push notifiers?
- receiver: 'frontend-pager'
group_by: [product, environment]
matchers:
- team="frontend"
I think 'webhook receiver' can help you. More information can refer doc https://prometheus.io/docs/alerting/latest/configuration/#webhook_config
This is an example of a webhook alert created based on blackbox_exporter's metric scraping.
prometheus rule setting
You need to create rule(s) to trigger alert, defined a rule named 'http_health_alert' here as example.
groups:
- name: http
rules:
- alert: http_health_alert
expr: probe_success == 0
for: 3m
labels:
type: http_health
annotations:
description: Health check for {{$labels.instance}} is down
Alertmanager setting
'match' is set to http_health_alert, the alert will be sent to'http://example.com/alert/receiver' via HTTP/POST method (I think you will prepare in advance).
The alert will post JSON format to the configured endpoint 'http://example.com/alert/receiver'. And you can also customize different receiving methods or receiving information in the endpoint/program for different label contents.
global:
route:
group_by: [alertname, env]
group_wait: 30s
group_interval: 3m
repeat_interval: 1h
routes:
- match:
alertname: http_health_alert
group_by: [alertname, env]
group_wait: 30s
group_interval: 3m
repeat_interval: 1h
receiver: webhook_receiver
receivers:
- name: webhook_receiver
webhook_configs:
- send_resolved: true
url: http://example.com/alert/receiver
- name: other_receiver
email_configs:
- send_resolved: true
to: xx
from: xxx
I'm working on sending Kibana email alerts using Elastalert. I did all the setup and postfix is also working fine but I'm getting no hits and alerts. The following are my config.yaml and frequency.yaml:
frequency.yaml
# Rule name, must be unique
name: Test email alerts
# (Required)
# Type of alert.
# the frequency rule type alerts when num_events events occur with timeframe time
type: frequency
# (Required)
# Index to search, wildcard supported
index: index-*
# (Required, frequency specific)
# Alert when this many documents matching the query occur within a timeframe
num_events: 1
# (Required, frequency specific)
# num_events must occur within this amount of time to trigger an alert
timeframe:
minutes: 1
# (Required)
# A list of Elasticsearch filters used for find events
# These filters are joined with AND and nested in a filtered query
# For more info: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl.html
filter:
- term:
log: "Performed Task"
# (Required)
# The alert is use when a match is found
alert:
- "email"
# (required, email specific)
# a list of email addresses to send alerts to
email:
- "abc#gmail.com"
config.yaml
# Any .yaml file will be loaded as a rule
rules_folder: example_rules
# How often ElastAlert will query Elasticsearch
# The unit can be anything from weeks to seconds
run_every:
minutes: 1
# ElastAlert will buffer results from the most recent
# period of time, in case some log sources are not in real time
buffer_time:
minutes: 15
# The Elasticsearch hostname for metadata writeback
# Note that every rule can have its own Elasticsearch host
es_host: *host*
# The Elasticsearch port
es_port: 9200
es_username: username
es_password: password
Output of elastalert-test-rule rules_folder/frequency.yaml
/usr/lib/python3/dist-packages/requests/__init__.py:80: RequestsDependencyWarning: urllib3 (1.25.4) or chardet (3.0.4) doesn't match a supported version!
RequestsDependencyWarning)
INFO:elastalert:Note: In debug mode, alerts will be logged to console but NOT actually sent.
To send them but remain verbose, use --verbose instead.
Didn't get any results.
INFO:elastalert:Note: In debug mode, alerts will be logged to console but NOT actually sent.
To send them but remain verbose, use --verbose instead.
1 rules loaded
INFO:apscheduler.scheduler:Adding job tentatively -- it will be properly scheduled when the scheduler starts
INFO:elastalert:Queried rule Test email alerts from 2021-07-17 23:21 UTC to 2021-07-17 23:22 UTC: 0 / 0 hits
Would have written the following documents to writeback index (default is elastalert_status):
elastalert_status - {'rule_name': 'Test email alerts', 'endtime': datetime.datetime(2021, 7, 17, 23, 22, 7, 154742, tzinfo=tzutc()), 'starttime': datetime.datetime(2021, 7, 17, 23, 21, 6, 554742, tzinfo=tzutc()), 'matches': 0, 'hits': 0, '#timestamp': datetime.datetime(2021, 7, 17, 23, 22, 7, 183348, tzinfo=tzutc()), 'time_taken': 0.008371829986572266}
Can anyone please help me why I'm getting no hits?
We are running an ELK stack to aggregate all our logs and we have multiple systems. Currently, we have Filebeat configured to log to specific indices based on the system (SystemA, SystemB, SystemC).
I would like to, additionally, send all logs with level ERROR to another index where I would like to collect all errors across systems, but somehow I can't figure out how to get Filebeat to send one message to multiple indices
According to the documentation, the first condition that matches will define the index to be used, which sounds to me as if it's not possible to send a message that would match multiple patterns to multiple indices?
What I want to do:
output.elasticsearch:
hosts: '${ELASTICSEARCH_HOSTS}'
username: '${ELASTICSEARCH_USERNAME}'
password: '${ELASTICSEARCH_PASSWORD}'
index: "filebeat-external-%{+yyyy.MM.dd}"
indices:
- index: "filebeat-error-logs-%{+yyyy.MM.dd}"
when:
or:
- equals:
level: "ERROR"
- equals:
level: "error"
- index: "filebeat-service-a-%{+yyyy.MM.dd}"
when:
regexp:
container.name: "^service-a-"
- index: "filebeat-service-b-%{+yyyy.MM.dd}"
when:
regexp:
container.name: "^service-b-"
The only way I currently see is to have multiple indices per system and aggregate them in Kibana:
output.elasticsearch:
hosts: '${ELASTICSEARCH_HOSTS}'
username: '${ELASTICSEARCH_USERNAME}'
password: '${ELASTICSEARCH_PASSWORD}'
index: "filebeat-external-%{+yyyy.MM.dd}"
indices:
- index: "error-log-service-a-%{+yyyy.MM.dd}"
when:
and:
- equals:
level: "ERROR"
- regexp:
container.name: "^service-a-"
- index: "service-log-service-a-%{+yyyy.MM.dd}"
when:
and:
- not:
- equals:
level: "ERROR"
- regexp:
container.name: "^service-a-"
But this would double our number of indices and is code duplication. Am I missing something here, is there an easier way to have a general error-index but still have errors go to the service-specific indices as well?
I've trying to find a way for send alert notification on my prometheus server to line-notify.I checked alert rules configure status on prometheus is OK and alert rules can detect event normally, this my config.yml for alertmanager
global:
resolve_timeout: 5m
route:
receiver: "line-noti"
# group_by: ['test-node-linux', 'test-node-windows', 'test-container-exporter', 'test-jmx-exporter']
group_interval: 10s
repeat_interval: 1m
receivers:
- name: 'line-noti'
webhook_configs:
- url: 'https://notify-api.line.me/api/notify'
send_resolved: true
http_config:
bearer_token: [my_token]
but it doesn't send any messages to line-notify
How can I do for solved this case?
The problem in the receiver's name, you have double quotation marks ". However, the name of receiver should be either with single apostrophes ' or completely without.
Also the url can be without apostrophes.
Try this:
global:
resolve_timeout: 5m
route:
receiver: line-noti
# group_by: ['test-node-linux', 'test-node-windows', 'test-container-exporter', 'test-jmx-exporter']
group_interval: 10s
repeat_interval: 1m
receivers:
- name: line-noti
webhook_configs:
- url: https://notify-api.line.me/api/notify
send_resolved: true
http_config:
bearer_token: [my_token]