We are using the ELK for log aggregation. Is it possible to search for events that occured during a particular time range. Lets say I want to see all exceptions that occurred between 10am and 11am in last month.
Is it possible to extract the time part from #timestamp and do a range search on that somehow (similiar to date() in SQL)?
Thanks to Magnus who pointed me to looking at scripted fields. Take a look at:
https://www.elastic.co/blog/kibana-4-beta-3-now-more-filtery
or
https://www.elastic.co/guide/en/elasticsearch/reference/1.3/search-request-script-fields.html
Unfortunately you can not use these scripted fields in queries but only in visualisations.
So I resorted to a workaround and use logstashs drop filter to remove the events I don't want to show up in Kibana in the first-place. That is not perfect for obvious reasons but it does the job.
Related
I'm playing around the package github.com/olivere/elastic; all works fine, but I've a question: is it possible to get the last N inserted documents?
The From statement has 0 as default starting point for the Search action and I didn't understand if is possible to omit it in search.
Tldr;
Although I am not aware of a feature in elasticsearch api to retrieve the latest inserted documents.
There is a way to achieve something alike if you store the ingest time of the documents.
Then you can sort on the ingest time, and retrieve the top N documents.
Is there a way to get the date and time that an elastic search document was written?
I am running es queries via spark and would prefer NOT to look through all documents that I have already processed. Instead I would like read the only documents that were ingested between the last time the program ran and now.
What is the best most efficient way to do this?
I have looked at;
updating to add a field with an array with booleans for if its been looked at by which analytic. The negative is waiting for the update to occur.
index per time frame method, which would be to break down the current indexes into smaller ones so by hour.The negative I see is the number of open file descriptors.
??
Elasticsearch version 5.6
I posted the question on the elasticsearch discussion board and it appears using the ingest pipeline is the best option.
I am running es queries via spark and would prefer NOT to look through
all documents that I have already processed. Instead I would like read
the only documents that were ingested between the last time the
program ran and now.
A workaround could be :
While inserting data using Logstash to Elasticsearch, Logstash appends a #timestamp key to the document which represents the time (in UTC) at which the document is created or we can use an ingest pipline
After that we can query based on the timestamp.
For more on this please have a look at :
Mapping changes
There is no way to ask ES to insert a timestamp at index time
Elasticsearch doesn't have such functionality.
You need manually save with each document date. In this case you will be able to search by date range.
I am looking for a way to measure the time logstash takes to output data into elastic search.
- There is this elapsed filter
https://www.elastic.co/guide/en/logstash/current/plugins-filters-elapsed.html, which I think can be used to measure the time taken to process the message through all the configured filters but not to measure the time taken to output to elastic search
- I also tried with a batch file with something like
echo starttime = %time%
cd c:\Temp\POC\Mattias\logstash-2.0.0\logstash-2.0.0\bin
logstash agent -f first-pipeline.conf
echo endtime = %time%
The problem with this approach is logstash doesn’t stop/exit after finishing a given input file.
Any help is highly appreciated!
Thanks and regards,
Priya
The elapsed{} filter is for computing the difference between two events (start/stop pairs, etc).
Logstash sets #timestamp to the current time. If you don't replace it (via the date{} filter), it will represent the time that logstash received the document.
Elasticsearch had a feature called _timestamp that would set a field by that name to the time of the elasticsearch server. For some reason, they've deprecated that feature in version 2.
As of right now, there is no supported way to get the time that elasticsearch indexed the data, so there is no supported way to determine the lag between logstash and elasticsearch and the processing time required by elasticsearch.
I was hoping that you could add a date field in your mapping and use the null_value to default the value to 'now', but that's not supported. Hopefully, they'll support that and reinstate this very useful feature.
Is it possible to give different time periods or different search terms to each Visualization in a Kibana Dashboard?
Currently - no.
This is on the list of enhancements that the 'elastic' team will implement soon, but doesn't have any due date yet.
You could follow the open issue here: https://github.com/elastic/kibana/issues/3578
I think i've understood your question.
Lets supose this is yout data whitin elasticSearch:
timestamp level message
19:05:15 error connection failed
19:06:30 debug connection succesfull
You can reflect your percentajes of each level in differente time periods (10% of debug, 20% of errors, 14% of info and so on). For instance you can design a chart for the last 1 hour and other one for the last day in the same dashboard, so you don't need to manipulate the date picker in de header.
First you have to make a query to filter your data by the timestamp
(ex. last day):
#timestamp:[now-1d TO now]
Second, you need to save this search, and name it.
Finally, design whatever visualization you need based on this
search, and the results will be bound to it.
Repeat with different time periods.
Hope this helps. Bye.
if have the problem to execute multiple Solr queries at ones with the same phrase but for different timeranges.
Example:
search for "atom" at:
2011-04-01T10:20:22.0Z TO 2011-04-01T12:20:22.0Z
2011-03-08T10:20:22.0Z TO 2011-03-08T12:20:22.0Z
2011-02-05T10:20:22.0Z TO 2011-02-05T12:20:22.0Z
So i need a few messages from each 2 hour interval.
First of all, i thought about facet search, but i don't think, thats a way, is'n it?
2nd idea was to fire one solr request for every time range. But probably there is to much (network)overhead for that, because this example is only an simplified version.
Maybe anybody has an idea, how could i handle this? What solr functionality is the best way for this?
Thank you.
Use FieldCollapsing with the group by query option.