elasticsearch curator delete "all" indices order than 7 days - elasticsearch

background:
elasticsearch version 6.2
curator version 5.4.1.
Now I can use curator to delete one index that order 7 days, but I have more than one index and I don't want to create more than one action.yml, such as :
actions:
1:
action: delete_indices
description: >-
Delete indices older than 7 days (based on index name), for student-prefixed indices. Ignore the error if the filter does not result in an actionable list of indices (ignore_empty_list) and exit cleanly.
options:
ignore_empty_list: True
disable_action: False
filters:
- filtertype: pattern
kind: prefix
value: student=
- filtertype: age
source: name
direction: older
timestring: '%Y-%m-%d'
unit: days
unit_count: 7
According to this action.yml, It deletes student=2017-XX-XX.
But I have many indices such as teacher, parent and so on.
I replace studnet= with *= but doesn't work.
So what can I do?
Thank you very much.

You try a few things. A few examples include:
You can omit the pattern filtertype, leaving only the age. This might delete other indices with %Y-%m-%d patterns, however. In that case, you might use a different pattern filter, but to exclude patterns you don't want to delete:
- filtertype: pattern
kind: prefix
value: omit_me
exclude: true
Replacing your pattern filter with this will delete all indices with %Y-%m-%d that are older than 7 days, except indices starting with omit_me.
You might set up a regex instead of a prefix. For example:
- filtertype: pattern
kind: regex
value: '^(student|parent|teacher).*$'
This will match indices starting with student, parent, or teacher.

Related

What does "source:name" in filter means?

I have been studying curator past few days and I came across this filter type "age".
On official documentation it is written as name based age filter look for a timestring within the index or snapshot name, and convert that into an epoch timestamp.
Which is not quite clear to me.
If I mention
source: name
what "name" does curator refer to?
Does it refer to name of any particular index and if yes how can I mention name of that index?
It will be really helpful if anyone suggest me some more documentation on curator.
Thanks in advance ^^
Yes, source: name reads the index name and looks for a time/date value matching timestring. For example, if you had an index named indexname-2019.06.01, you might build a filter like this:
- filtertype: age
source: name
timestring: ‘%Y.%m.%d’
unit: days
unit_count: 30
direction: older
This filter (if not following other filters in a list) will look through the names of all indices in Elasticsearch for a Year.month.day pattern, convert it to an epoch time stamp, and see if that date is more than 30 days older than the epoch time stamp at the time Curator is executed. If that is true, that index name will remain in the actionable list to do whatever action the filter is associated with.
Now, this by itself can be a dangerous filter. It will match indexname-2019.06.01 or 2019.06.01-anything or even prefix-2019.06.01-suffix. Filters in Curator were made to go together in a chain. To specify which indices you want Curator to consider, it might be wise to do a pattern filter before the age filter:
- filtertype: pattern
kind: prefix
value: indexname
- filtertype: age
source: name
timestring: ‘%Y.%m.%d’
unit: days
unit_count: 30
direction: older
Now this filter list will only look for indices which begin with indexname and have a Year.month.day time string after that. Filters in Curator are always ANDed together.
The official Curator documentation is the ultimate source of truth for all things Curator. If you have further requests for explanation, I’m happy to answer them (full disclosure: I am the author and maintainer of Curator).

Elasicsearch curator reindexing

I'm new to curator for elasticsearch and trying to make re-indexing but I need to search indexes that contains word "debug" inside index.
I was trying to do it with
filters:
- filtertype: pattern
kind: regex
value: '^*.debug.*$'
But it doesn't work properly.
From devtools in kibana if I will do get *.debug*/_search it shows how it should be. But how can I do the same with actionfile?
Your regular expression does not match the GET statement.
GET *.debug*/_search
will not match the same indices as
value: '^*.debug.*$'
but value: '^*.debug*$' might match the same query string.

ElasticSearch query specifying an indexname using todays date

I'm using logstash to populate ES with a number of metrics from our live services across a number of machines. Logstash creates a new index each day and i am finding that querying ES without specifying the index, is running slowly. ( i currently maintain 5 days of indicies). If i specify the specific index eg today
.es(index=logstash-2018.01.15, q= examplequery
it runs very quickly
Is there a way i can specify todays index using the date field?
eg
.es(index=logstash-'get date', q= examplequery
You can use the query for getting the indices of today's date:
.es(index='<logstash-{now/d}>')
An interesting read with all the options available in elastic search to include date math in index names:
https://www.elastic.co/guide/en/elasticsearch/reference/current/date-math-index-names.html
By looking at the syntax I guess you are using Timelion or something that uses query string. There is a good tutorial here that includes specifying index patterns:
https://www.elastic.co/blog/timelion-tutorial-from-zero-to-hero
In your case it will be
.es(index=logstash-*, q= examplequery
or
.es(index=logstash-2018.01.*, q= examplequery
if you need this year january and the index pattern is 'logstash-YYYY.MM.dd'

What are concrete indices

In relation to elasticsearch, what are concrete indices.
The elasticsearch docs mention them hundreds of times, but I can't find a definition anywhere.
For example:
count - allowNoIndices:
Whether to ignore if a wildcard indices expression resolves into no ~concrete indices~.
A concrete index is simply a real index that is stored in Elasticsearch and that you can list with a /_cat/indices command such as
curl 'localhost:9200/_cat/indices?v'
As you probably know, when searching you can either specify:
one concrete index: /my_index/_search
more than one concrete indices: /my_index1,my_index2/_search
one alias: /my_alias/_search
more than one aliases: /my_alias1,my_alias2/_search
an index wildcard: /my_*/_search
In cases 1 and 2, you specify concrete indices, i.e. indices that you would see listed by the /_cat/indices command above.
In cases 3 and 4, the alias(es) you specify will resolve to concrete indices, so that in the end if my_alias is an alias for my_index1 and my_index2, then 3. is equivalent to 2.
In case 5, it's just a shortcut to not have to list all concrete indices whose name starts with the prefix my_. You often use that when you have time-based indices, such as logstash-2015* for all logstash indices of the year 2015.
To sum it up, a concrete index is an index that you have created one way or another and that will show when listing all indices of present in your Elasticsearch instance.

Kibana visualizations splitting fields with dashes in them

I am building visualizations in Kibana for AWS CloudWatch metrics, and have run into a bit of an issue creating Metric Tables.. Kibana is splitting my fields that contain dashes (instance ID, region, etc..)
Rather than having an individual row with an instance ID, for example, i-7bb06dzz, it is creating 2 rows: i & 7bb06dzz. "i" displays the aggregate count of all other fields. If I add a second split with the region, this is duplicated for every set of characters in the region name separated by dashes as well. (us , east, and 1 instead of just us-east-1).
I tried to post a screenshot, but my reputation is not high enough to do so..
Here are my visualization settings:
Metrics: Metric (Count)
Aggregations:
Split Rows: Terms: InstanceID: Top 5: Order by metric:Count
Split Rows: Terms: Region: Top 5: Order by metric:Count
No Advanced Settings have been specified. I was able to get a reasonable looking list by only specifying InstanceID, and excluding the pattern "i"... however, it doesn't do me a lot of good when I can't display the region next to it. Both values are indexed as strings and were recorded in ElasticSearch with double quotes around them.
Any recommendations on how to display the fields as intended would be much appreciated.
This is because Elasticsearch "analyzes" the field for the individual tokens in it. Logstash will store fields in both the fieldname and fieldname.raw fields - the latter is unanalyzed and will behave as you expect.

Resources