I have an event that I want to take an action off of when it happens. This event is being processed by Logstash and pushed to Elastic Search.
I see in the docs for Logstash that there are Filters and Outputs. I thought the Filter might have been what I was after, but looking closer at the docs it does not seem to have the functionality to push data to an endpoint/API: Filters are intermediary processing devices in the Logstash pipeline. You can combine filters with conditionals to perform an action on an event if it meets certain criteria.
Is it possible to take an action when the event is being processed by Logstash 7.4? Or do I need to wait to take some type of action until I do some polling-based queries for the event in ElasticSearch?
Related
I've been reviewing different ways to aggregate log messages together that have a start event but no end event. Been struggling with the logstash aggregate filter plugin not sorting correctly and was looking at retrofitting an old entity-centric model for a previous version of elasticsearch Entity-Centric Indexing - Mark Harwood | Elastic Videos when I realized elasticsearch 7.13 transforms introduce the concept of 'latest' which negates my need for a bunch of external scripts (hopefully) to do this.
I am looking at the "Getting Web Session Details by using Scripted Metric Aggregation" sample painless script https://www.elastic.co/guide/en/elasticsearch/reference/current/transform-painless-examples.html#painless-web-session which produces session details, including session duration. Because the logs do not have an end-time, I need to make use of a timeout interval, something like a 30 minute window for aggregating message events based on my group by.
Is this possible to do within the transform by adjusting that script and could anyone help?
Good day,
I have a gotten into a bit of a headache when working on indexing some data in Elasticsearch and have some questions about a good approach.
As of now, an event is received on a Kafka topic with just a part of the data that should be stored in the document. The rest of the data needs to be collected after the event is received and is available from different APIs. To reduce the amount of work, it seems that Logstash could be a good approach.
Is there a way to configure Logstash to initiate data collection from different APIs and DBs when an event is received, and then assemble the document with the combined date, or am I stuck with writing time consuming custom logic for the indexing? I have searched around a bit, but couldn't find any good answer on the problem.
What you need in logstash is to lookup/enrich you message with data from external api's, right?
You could use logstash's http_filter plugin
I would like to use elasticsearch to annotate data with geographic belonging.
I'm quite new to elasticsearch so I'm a bit unsure about the correct word to search for. I might have missed an obvious answer, but I didn't find what I was looking for in the Elasticsearch documentation.
Problem description :
I have a huge set of events stored in a ElasticSearch database. I would like to query these events, and for each event, add a kind of scripted field telling is the event point is within a given polygon.
I should be able to do this with two queries :
a first one retrieving a batch of events
another one filtering the same events with a geo_shape filter
I could then check for every event in the first list if it's also in the second one.
But it would be so much easier to have one query, with the event document annotated with an is_within_polygon boolean.
And it would also allow me to check against multiple polygons in one query.
Is it possible with Elasticsearch ?
I have problem with creating metrics and later trigger alerts base on that metric. I have two datasources, both are elasticsearch. One contains documents (logs from service) saying that message was produced to kafka, second contain documents (also logs from service) saying that message was consumed. What I want to achieve is to trigger alert if ratio of produced to consumed messages drop below 1.
Unfortunately it is impossible to use prometheus, for two reasons:
1) counter resets each time service is restarted.
2) second service doesn't have (and wont't have in reasonable time) prometheus integration.
Question is how to approach metrics and alerting based on that data sources? Is it possible? Maybe there is other way to achieve my goal?
The question is somewhat generic (meaning no mapping or code, in general, is provided), so I'll provide an approach.
You can use a watcher upon an aggregation that you will create.
It's relatively straightforward to create a percentage of consume/produce, and based upon that percentage you can trigger an alert via the watcher.
Take a look at this tutorial (official elasticsearch channel) on how to do this. Moreover, check the tutorials for your specific version of elasticsearch. From 5.x to 7.x setting alerts has been significantly improved (this means that for 7.x you might be able to do this via the UI of kibana, but for 5.x you'll probably need to add the alert via indexing json in the appropriate indices .watcher)
I haven't used grafana, but I believe the same approach can be applied. You'll need an aggregation as mentioned before and then you add the alert https://grafana.com/docs/grafana/latest/alerting/rules/
When filtering events in logstash (20+ attributes) i would like to create new event which would have one parameter from original event and store it into other ElastiSearch index.
I know this is possible using clone filter plugin. But i don't want to manually remove all attributes from original events except the one I need.
Also i could just clone that event (i'm will store new event in separate elasticsearch index) but that will duplicate unneeded attributes.
Is there any filter plugin for this purpose? Or some hidden feature? Or maybe clone filter plugin handles removal of all attributes from cloned messages?
ElastAlert is a simple framework for alerting on anomalies, spikes, or other patterns of interest from data in Elasticsearch.
http://elastalert.readthedocs.io/en/latest/elastalert.html