I'm using logstash to populate ES with a number of metrics from our live services across a number of machines. Logstash creates a new index each day and i am finding that querying ES without specifying the index, is running slowly. ( i currently maintain 5 days of indicies). If i specify the specific index eg today
.es(index=logstash-2018.01.15, q= examplequery
it runs very quickly
Is there a way i can specify todays index using the date field?
eg
.es(index=logstash-'get date', q= examplequery
You can use the query for getting the indices of today's date:
.es(index='<logstash-{now/d}>')
An interesting read with all the options available in elastic search to include date math in index names:
https://www.elastic.co/guide/en/elasticsearch/reference/current/date-math-index-names.html
By looking at the syntax I guess you are using Timelion or something that uses query string. There is a good tutorial here that includes specifying index patterns:
https://www.elastic.co/blog/timelion-tutorial-from-zero-to-hero
In your case it will be
.es(index=logstash-*, q= examplequery
or
.es(index=logstash-2018.01.*, q= examplequery
if you need this year january and the index pattern is 'logstash-YYYY.MM.dd'
Related
ONE elastic index that stores visitor data for restaurants in chennai and in New york.
The query must return data for 8-9am in chennai and 8-9am in newyork. Since the data is in one index, is there a way to ignore the timeszone in the timestamp and query elastic data?
or is it possible to solve this problem without multiple query?
The approach I am taking is to convert and store local date time (without timezone info) into a long integer yymmddhhmmss and query it. This will support lte and gte queries too.
it's not possible, no, because they are two totally different timezones/timestamps
I have been studying curator past few days and I came across this filter type "age".
On official documentation it is written as name based age filter look for a timestring within the index or snapshot name, and convert that into an epoch timestamp.
Which is not quite clear to me.
If I mention
source: name
what "name" does curator refer to?
Does it refer to name of any particular index and if yes how can I mention name of that index?
It will be really helpful if anyone suggest me some more documentation on curator.
Thanks in advance ^^
Yes, source: name reads the index name and looks for a time/date value matching timestring. For example, if you had an index named indexname-2019.06.01, you might build a filter like this:
- filtertype: age
source: name
timestring: ‘%Y.%m.%d’
unit: days
unit_count: 30
direction: older
This filter (if not following other filters in a list) will look through the names of all indices in Elasticsearch for a Year.month.day pattern, convert it to an epoch time stamp, and see if that date is more than 30 days older than the epoch time stamp at the time Curator is executed. If that is true, that index name will remain in the actionable list to do whatever action the filter is associated with.
Now, this by itself can be a dangerous filter. It will match indexname-2019.06.01 or 2019.06.01-anything or even prefix-2019.06.01-suffix. Filters in Curator were made to go together in a chain. To specify which indices you want Curator to consider, it might be wise to do a pattern filter before the age filter:
- filtertype: pattern
kind: prefix
value: indexname
- filtertype: age
source: name
timestring: ‘%Y.%m.%d’
unit: days
unit_count: 30
direction: older
Now this filter list will only look for indices which begin with indexname and have a Year.month.day time string after that. Filters in Curator are always ANDed together.
The official Curator documentation is the ultimate source of truth for all things Curator. If you have further requests for explanation, I’m happy to answer them (full disclosure: I am the author and maintainer of Curator).
I've got ES documents that looks like this:
{
"auctionOn": "2018-01-01",
"inspections: [
{
"startsOn": "2018-01-02 09:00",
"endsOn": "2018-01-02 10:00"
}
]
}
I need the following answers from a search (or multiple searches)
number of documents with an auctionOn in the future (e.g > now)
number of documents with an inspection.startsOn in the future (e.g > now)
date histogram (day breakdown) of the next 7 days, with # of documents with a auctionOn on that day
date histogram (day breakdown) of the next 7 days, with # of documents with a inspection.startsOn on that day
So, i'm trying to figure out how to efficiently get these answers. I know i can/should test out all different approaches, but i'm relatively new to ES so easier said than done.
Can someone give me a advice (or ideally, a query) on how to get these 4 values?
Ideas i had:
Query for all documents with an inspection/auction in the future. Create date histogram aggregations filtered to the next 7 days for both auction and inspections. Use range aggregations to get number of docs with auction/inspection > today.
Pros: one search for all answers. Cons: lots of documents to aggregate over?
Create seperate searches (e.g msearch) for:
query all documents with an inspection in the next 7 days. aggregate by day.
query all documents with an auction in the next 7 days. aggregate by day.
query all documents with an inspection in the future. use hits to get total
query all documents with an auction in the future. use hits to get total.
Pros: queries are simpler.. more cache hits? Cons: 4 seperate searches.
Can someone please guide me down the right path, and give me hints on how to do the query/aggregations?
Thanks
Use range query on the field auctionOn setting from as current date and to date as null.
Use range query inside nested query on the field inspection.startsOn as above.
Use date histogram aggregation using interval as day
Same as 3.) but inside nested aggregation
You can adjust all these in one query.
For example I have next documents.
{sourceIP:1.1.1.1, destIP:2.2.2.2}
{sourceIP:1.1.1.1, destIP:3.3.3.3}
{sourceIP:1.1.1.1, destIP:4.4.4.4}
Is there anyway to automatically aggregate them into one document which will contain next data?
{sourceIP:1.1.1.1, destIP:{2.2.2.2,3.3.3.3,4.4.4.4}}
So it looks like group by in SQL, but generate new documents in elasticsearch instead of old one.
I dont think there is anyway to do indexing time auto-merging of documents.
However , it should be possible to acheive whatever result you are planning to query should be possible by using one of querying options offered by Elasticsearch - while indexing one document for ,
Like ..
You can index seperate documents, query by sourceIP and use aggregations to give dest_ip
Take count of documents if its just to find dest_ips for a source_ip
Also if you want to avoid duplicate source_id + dest_id combinations , you can concat and use it as _id of document
Hope this helps.
Is there any way to get all the routings I have already used to index different documents? For example I am using month to define the routing. All documents with value as July in response date would be moved to a single shard and so on. Now can I get all the months which are used as Routing value through some query?