Does ICP provide alerting on logs or this is something extra? My use case is to alert on critical exception keywords in application logs. ICP's Prometheus alerts deals with metrics/numbers. I am familiar with Graylog which uses ElasticSearch and it provides alerting feature. What is the ICP recommendation for alerting on messages/events in logs?
You can do this with elastalert: https://elastalert.readthedocs.io/en/latest/
https://github.com/Yelp/elastalert
https://github.com/bitsensor/yelp-elastalert
If you are looking for ICP supported or inbuilt log alerting then there isn't any. You may have to rely on external tools build upon elastic search as David as pointed out in the answer.
Related
I need to send email automatically whenever any error comes in my Elastic search.
Is there anyway to do it.
I dont want to use Elastic Cloud for it.
I can use Watcher in Kibana, but my question is whether the "Watcher" is available in local also along with cloud?
Please help!
Watcher is available in on-premises installations if you have at least a Gold License, it is not available with the free basic license.
The same thing for the Kibana e-mail action, it needs a Gold License.
You can check what is available at the subscription page.
If you do not have a Gold License for your on-premises cluster, you will need an external tool to query elasticsearch and send e-mails, you can build one using one of the official clients libraries (python, node.js, java etc) or you can try other tools like elastalert.
According to their subscriptions overview page, Kibana Alerting is supported in the Basic Lisence.
However I'm unable to find/activate the feature on Kibana.
I've been searching the ES forums, can't really find a clear answer whether Alerting is actually available in the Basic license or not.
Btw: security settings is activated in my cluster
I have a server stats.hostname with graphite + grafana. This is receiving some stats about geolocation from several clients. I want to know if there is some plugin/extension/external tool for alert (email) when this stats overpass some threshold.
I tried with worldPing, but I think the tool is only for checking is a site is reachable or not.
Can you suggest some solution?
Thanks!
Alerting is probably one of the most requested features of Grafana. The team at Raintank are building an alerting system on top of Grafana. You can follow the progress and the discussion here - https://github.com/grafana/grafana/issues/2209
Currently though, you can use Bosun for your alerting needs. - https://bosun.org/quickstart#graphite
It does have Graphite querying capabilities, and there's a Bosun Datasource for Grafana as well.
Alerting in grafana is available since release 4.0 from dec 2016
http://grafana.org/blog/2016/12/12/grafana-4.0-stable-release/
Currently v. 4.0.2 is available http://grafana.org/download/ for donwload.
I've setup an ELK (Elasticsearch, Logstash and Kibana) stack and created some Kibana dashboard widgets. So far everything went amazing. Now I want to send daily and weekly email with the generated reports.
What is the best way to do that. Do I need to install any plugin or I can sent it right from Kibana?
You can use ElastAlert. You will be able to mail a link with the Kibana dashboard with only the data of the period you want. The period parameter in the top right corner will be set automatically in Kibana.
There are some workarounds, such as phantomjs but not straightforward to implement. For specific events and Kibana queries there are alerting mechanisms available (Watcher, Logz.io), but I'm guessing you're looking to receive the entire dashboard by email.
There are two out-of-the box options for sending email reports from Kibana dashboard:
Skedler which allows you to schedule and send automated email reports based on your Kibana dashboard or search.
If you have Elasticsearch license/subscription, then you can use the reporting plugin.
Hope it helps.
You can use Sentinl that extends Kibana for Alerting and Reporting functionality to monitor, notify and report on data series changes using standard queries, programmable validators and a variety of configurable actions - Think of it as a free an independent "Watcher" which also has scheduled "Reporting" capabilities (PNG/PDFs snapshots).
The greatest thing about Sentinl is you can easily configure alerts through it's native App interface integrated in Kibana.
AWS has a really nice log management tool. I can make my application log messages there very easy.
Amazon ElasticBeanstalk has a "event management" tool.
The questions are:
Can I log my app messages together in the ElasticBeanstalk events? Is it the syslog of the EC2 instance?
If yes, is this a good practice? Any problem on this? I was thinking about because, if there is no problem, I would not need any other third log management service.
The events shown in Elastic Beanstalk are internal to it. You are not supposed to fudge around with them (Although nobody is really preventing you from playing around with them).
Also, there's a log snapshot feature that picks up logs related to the application. These logs are mainly related to deployment and logging messages from the application itself. So, you can use this feature in case your application code is logging messages. For example, if you are running a Ruby/Rails with passenger you would get log messages under /var/app/support/logs/passenger.log. These are not syslog messages per se and the problem with this approach is that it's not straight forward the get your custom monitoring in place. For example, how do you parse your errors and send them to say PagerDuty?
Like you've probably figured out if you want to have custom monitoring (send logs to a syslog facility) you are better off using a third party tool like Splunk Storm, PaperTrail or Loggly. Of course you can setup your own syslog server(s) but that will require you to set up all the infrastructure.
Hope this helps.