I'm working on sending Kibana email alerts using Elastalert. I did all the setup and postfix is also working fine but I'm getting no hits and alerts. The following are my config.yaml and frequency.yaml:
frequency.yaml
# Rule name, must be unique
name: Test email alerts
# (Required)
# Type of alert.
# the frequency rule type alerts when num_events events occur with timeframe time
type: frequency
# (Required)
# Index to search, wildcard supported
index: index-*
# (Required, frequency specific)
# Alert when this many documents matching the query occur within a timeframe
num_events: 1
# (Required, frequency specific)
# num_events must occur within this amount of time to trigger an alert
timeframe:
minutes: 1
# (Required)
# A list of Elasticsearch filters used for find events
# These filters are joined with AND and nested in a filtered query
# For more info: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl.html
filter:
- term:
log: "Performed Task"
# (Required)
# The alert is use when a match is found
alert:
- "email"
# (required, email specific)
# a list of email addresses to send alerts to
email:
- "abc#gmail.com"
config.yaml
# Any .yaml file will be loaded as a rule
rules_folder: example_rules
# How often ElastAlert will query Elasticsearch
# The unit can be anything from weeks to seconds
run_every:
minutes: 1
# ElastAlert will buffer results from the most recent
# period of time, in case some log sources are not in real time
buffer_time:
minutes: 15
# The Elasticsearch hostname for metadata writeback
# Note that every rule can have its own Elasticsearch host
es_host: *host*
# The Elasticsearch port
es_port: 9200
es_username: username
es_password: password
Output of elastalert-test-rule rules_folder/frequency.yaml
/usr/lib/python3/dist-packages/requests/__init__.py:80: RequestsDependencyWarning: urllib3 (1.25.4) or chardet (3.0.4) doesn't match a supported version!
RequestsDependencyWarning)
INFO:elastalert:Note: In debug mode, alerts will be logged to console but NOT actually sent.
To send them but remain verbose, use --verbose instead.
Didn't get any results.
INFO:elastalert:Note: In debug mode, alerts will be logged to console but NOT actually sent.
To send them but remain verbose, use --verbose instead.
1 rules loaded
INFO:apscheduler.scheduler:Adding job tentatively -- it will be properly scheduled when the scheduler starts
INFO:elastalert:Queried rule Test email alerts from 2021-07-17 23:21 UTC to 2021-07-17 23:22 UTC: 0 / 0 hits
Would have written the following documents to writeback index (default is elastalert_status):
elastalert_status - {'rule_name': 'Test email alerts', 'endtime': datetime.datetime(2021, 7, 17, 23, 22, 7, 154742, tzinfo=tzutc()), 'starttime': datetime.datetime(2021, 7, 17, 23, 21, 6, 554742, tzinfo=tzutc()), 'matches': 0, 'hits': 0, '#timestamp': datetime.datetime(2021, 7, 17, 23, 22, 7, 183348, tzinfo=tzutc()), 'time_taken': 0.008371829986572266}
Can anyone please help me why I'm getting no hits?
Related
I am using elastalert to trigger email when elasticsearch url is up/down.
I want to configure in such manner that elastalert will continuously trigger mail when server goes down and when server is up only one mail will be triggered.
Please find below example_frequency rule config
es_host: localhost
es_port: 9200
name: Example frequency rule
type: frequency
index: heartbeat-*
num_events: 1
realert:
seconds: 30
timeframe:
minutes: 3
filter:
- terms:
monitor.status: ["up", "down"]
alert:
- "email"
email:
- "AD#somemail.com"
I have done setup on windows 10. Getting below output when executing elastalert-test-rule for my rule.
elastalert-test-rule example_rules\example_frequency.yaml --config config.yaml
Would have written the following documents to writeback index (default is elastalert_status):
elastalert_status - {'rule_name': 'Example frequency rule', 'endtime': datetime.datetime(2020, 4, 19, 18, 49, 10, 397745, tzinfo=tzutc()), 'starttime': datetime.datetime(2019, 4, 17, 3, 13, 10, 397745, tzinfo=tzutc()), 'matches': 4, 'hits': 4, '#timestamp': datetime.datetime(2020, 4, 19, 18, 55, 56, 314841, tzinfo=tzutc()), 'time_taken': 405.48910188674927}
However, no alert is triggered.
Please find below contents of config.yaml and example_frequency.yaml
config.yaml
# This is the folder that contains the rule yaml files
# Any .yaml file will be loaded as a rule
rules_folder: example_rules
# How often ElastAlert will query Elasticsearch
# The unit can be anything from weeks to seconds
run_every:
seconds: 5
# ElastAlert will buffer results from the most recent
# period of time, in case some log sources are not in real time
buffer_time:
minutes: 15
# The Elasticsearch hostname for metadata writeback
# Note that every rule can have its own Elasticsearch host
es_host: 127.0.0.1
# The Elasticsearch port
es_port: 9200
# The AWS region to use. Set this when using AWS-managed elasticsearch
#aws_region: us-east-1
# The AWS profile to use. Use this if you are using an aws-cli profile.
# See http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html
# for details
#profile: test
# Optional URL prefix for Elasticsearch
#es_url_prefix: elasticsearch
# Connect with TLS to Elasticsearch
#use_ssl: True
# Verify TLS certificates
#verify_certs: True
# GET request with body is the default option for Elasticsearch.
# If it fails for some reason, you can pass 'GET', 'POST' or 'source'.
# See http://elasticsearch-py.readthedocs.io/en/master/connection.html?highlight=send_get_body_as#transport
# for details
#es_send_get_body_as: GET
# Option basic-auth username and password for Elasticsearch
#es_username: someusername
#es_password: somepassword
# Use SSL authentication with client certificates client_cert must be
# a pem file containing both cert and key for client
#verify_certs: True
#ca_certs: /path/to/cacert.pem
#client_cert: /path/to/client_cert.pem
#client_key: /path/to/client_key.key
# The index on es_host which is used for metadata storage
# This can be a unmapped index, but it is recommended that you run
# elastalert-create-index to set a mapping
writeback_index: elastalert_status
writeback_alias: elastalert_alerts
# If an alert fails for some reason, ElastAlert will retry
# sending the alert until this time period has elapsed
alert_time_limit:
days: 2
# Custom logging configuration
# If you want to setup your own logging configuration to log into
# files as well or to Logstash and/or modify log levels, use
# the configuration below and adjust to your needs.
# Note: if you run ElastAlert with --verbose/--debug, the log level of
# the "elastalert" logger is changed to INFO, if not already INFO/DEBUG.
#logging:
# version: 1
# incremental: false
# disable_existing_loggers: false
# formatters:
# logline:
# format: '%(asctime)s %(levelname)+8s %(name)+20s %(message)s'
#
# handlers:
# console:
# class: logging.StreamHandler
# formatter: logline
# level: DEBUG
# stream: ext://sys.stderr
#
# file:
# class : logging.FileHandler
# formatter: logline
# level: DEBUG
# filename: elastalert.log
#
# loggers:
# elastalert:
# level: WARN
# handlers: []
# propagate: true
#
# elasticsearch:
# level: WARN
# handlers: []
# propagate: true
#
# elasticsearch.trace:
# level: WARN
# handlers: []
# propagate: true
#
# '': # root logger
# level: WARN
# handlers:
# - console
# - file
# propagate: false
example_frequency.yaml
# Alert when the rate of events exceeds a threshold
# (Optional)
# Elasticsearch host
# es_host: elasticsearch.example.com
# (Optional)
# Elasticsearch port
# es_port: 14900
# (OptionaL) Connect with SSL to Elasticsearch
#use_ssl: True
# (Optional) basic-auth username and password for Elasticsearch
#es_username: someusername
#es_password: somepassword
# (Required)
# Rule name, must be unique
name: Example frequency rule
# (Required)
# Type of alert.
# the frequency rule type alerts when num_events events occur with timeframe time
type: frequency
# (Required)
# Index to search, wildcard supported
index: com-*
# (Required, frequency specific)
# Alert when this many documents matching the query occur within a timeframe
num_events: 1
# (Required, frequency specific)
# num_events must occur within this amount of time to trigger an alert
timeframe:
days: 365
# (Required)
# A list of Elasticsearch filters used for find events
# These filters are joined with AND and nested in a filtered query
# For more info: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl.html
filter:
- term:
"log_json.response.statusCode": "404"
# (Required)
# The alert is use when a match is found
alert:
- "email"
# (required, email specific)
# a list of email addresses to send alerts to
email:
- "username#mydomain.com"
realert:
minutes: 0
What is it that i am missing to receive alerts? Neither do i see any error on console.
SMTP configuration in missing, so that is why no alert is being sent.
Please try to include the smtp_host,smtp_port,smtp_ssl and smtp_auth_file in your example_frequency.yaml.
Refer to the document for Email Alert
In the elasticsearch cluster we get logs indexed from our cluster. Logstach didnt set any filter so we get all the logs with INFO, DEBUG etc levels. I want to know the query to get just the logs with one specific level.
If it's in python we could do it like this.
df = pd.DataFrame({"node.id": [123, 124, 125],
"log":["INFO: run", "DEBUG: fail", "WARN: warn"]})
log node.id
0 INFO: run 123
1 DEBUG: fail 124
2 WARN: warn 125
df[df["log"].str.contains("DEBUG")]
log node.id
1 DEBUG: fail 124
I tried using "regex", "wildcard" but doesn't seem to get the right syntax.
I have configured aggregation option in config.yaml to send summary of alerts after every 1 hour. But it is throwing following error when I try to run it.
File "elastalert.py", line 863, in run_rule
self.add_aggregated_alert(match, rule)
File "elastalert.py", line 1614, in add_aggregated_alert
alert_time = ts_now() + rule['aggregation']
TypeError: unsupported operand type(s) for +: 'datetime.datetime' and 'dict'
ERROR:root:Uncaught exception running rule Test Alert : unsupported operand type(s) for +: 'datetime.datetime' and 'dict'
INFO:elastalert:Rule Test Alert disabled
The config parameters are:
rules_folder: test_rules
run_every:
minutes: 15
buffer_time:
minutes: 30
es_host: 100.38.46.3
es_port: 9200
aggregation:
hours: 1
writeback_index: elastalert_status
alert_time_limit:
days: 2
Test Alert rule configuration:
name: Test Alert
type: metric_aggregation
index: logstash-*
buffer_time:
minutes: 30
metric_agg_key: count
metric_agg_type: sum
query_key: "name.keyword"
doc_type: counter
max_threshold: 1
min_threshold: 0
filter:
- query:
query_string:
query: "name.keyword: *timedout_count"
alert:
- "email"
email:
- "admin#abc.com"
I have followed the ElastAlert docs but not able to figure out what is causing this issue.
Thanks
From the error:
TypeError: unsupported operand type(s) for +: 'datetime.datetime' and 'dict'
And from your config.yaml
metric_agg_type: sum
It's trying (and faililng) to perform a sum aggregation on datetime and dict values that do not support summation. You'll need to choose an aggregation like count or unique count, probably, and adjust the logic of your alerts accordingly.
I have installed Elastisearch 2.1.0 and kibana 4.3.0 in single machine.
Kibana.yml Configurations :
# Kibana is served by a back end server. This controls which port to use.
server.port: 5601
# The host to bind the server to.
server.host: "IP"
# A value to use as a XSRF token. This token is sent back to the server on each request
# and required if you want to execute requests from other clients (like curl).
# server.xsrf.token: ""
# If you are running kibana behind a proxy, and want to mount it at a path,
# specify that path here. The basePath can't end in a slash.
# server.basePath: ""
# The Elasticsearch instance to use for all your queries.
elasticsearch.url: "http://IP:9200/"
# preserve_elasticsearch_host true will send the hostname specified in `elasticsearch`. If you set it to false,
# then the host you use to connect to *this* Kibana instance will be sent.
elasticsearch.preserveHost: true
# Kibana uses an index in Elasticsearch to store saved searches, visualizations
# and dashboards. It will create a new index if it doesn't already exist.
kibana.index: ".kibana"
# The default application to load.
kibana.defaultAppId: "discover"
# If your Elasticsearch is protected with basic auth, these are the user credentials
# used by the Kibana server to perform maintenance on the kibana_index at startup. Your Kibana
# users will still need to authenticate with Elasticsearch (which is proxied through
# the Kibana server)
# elasticsearch.username: "user"
# elasticsearch.password: "pass"
# SSL for outgoing requests from the Kibana Server to the browser (PEM formatted)
# server.ssl.cert: /path/to/your/server.crt
# server.ssl.key: /path/to/your/server.key
# Optional setting to validate that your Elasticsearch backend uses the same key files (PEM formatted)
# elasticsearch.ssl.cert: /path/to/your/client.crt
# elasticsearch.ssl.key: /path/to/your/client.key
# If you need to provide a CA certificate for your Elasticsearch instance, put
# the path of the pem file here.
# elasticsearch.ssl.ca: /path/to/your/CA.pem
# Set to false to have a complete disregard for the validity of the SSL
# certificate.
elasticsearch.ssl.verify: true
# Time in milliseconds to wait for elasticsearch to respond to pings, defaults to
# request_timeout setting
# elasticsearch.pingTimeout: 1500
# Time in milliseconds to wait for responses from the back end or elasticsearch.
# This must be > 0
# elasticsearch.requestTimeout: 300000
# Time in milliseconds for Elasticsearch to wait for responses from shards.
# Set to 0 to disable.
# elasticsearch.shardTimeout: 0
# Time in milliseconds to wait for Elasticsearch at Kibana startup before retrying
# elasticsearch.startupTimeout: 5000
# Set the path to where you would like the process id file to be created.
pid.file: /var/run/kibana.pid
# If you would like to send the log output to a file you can set the path below.
logging.dest: /var/log/kibana/kibana.log
# Set this to true to suppress all logging output.
# logging.silent: false
# Set this to true to suppress all logging output except for error messages.
# logging.quiet: true
# Set this to true to log all events, including system usage information and all requests.
# logging.verbose: true
While I am doing curl -IP:5601 , I am getting this output:
**HTTP/1.1 200 OK
x-app-name: kibana
x-app-version: 4.3.0
cache-control: no-cache
content-type: text/html
content-length: 217
accept-ranges: bytes
Date: Wed, 20 Jan 2016 15:28:35 GMT
Connection: keep-alive
<script>var hashRoute = '/app/kibana';
var defaultRoute = '/app/kibana';
var hash = window.location.hash;
if (hash.length) {
window.location = hashRoute + hash;
} else {
window.location = defaultRoute;
</script>
Elasticsearch and kibana both are up and running still I am not able to access Kibana GUI from the browser. It is not displaying the page.
I checked the configurations of elasticsearch.yml too.The host and IP is correct there. Curl command is giving this output for elasticsearch [Command :curl http://IP:9200/]
{
"name" : "node-1",
"cluster_name" : "elasticsearch",
"version" : {
"number" : "2.1.0",
"build_hash" : "72cd1f1a3eee09505e036106146dc1949dc5dc87",
"build_timestamp" : "2015-11-18T22:40:03Z",
"build_snapshot" : false,
"lucene_version" : "5.3.1"
},
"tagline" : "You Know, for Search"
}
Could anybody tell what could be the issue.
Did you install elasticsearch and kibana on your local machine, I mean your laptop or computer that you are workng on? Or is it running on a separate server?
If you are running it on the same machine that you are accessing the browser, then you could just access it as localhost:port
As your error includes the status
Elasticsearch is still initializing the kibana index, I would recommend you to try the steps mentioned in this page:-
Elasticsearch is still initializing the kibana index