Elasticsearch Curator - delete indices except newest - elasticsearch

Question

Using Elasticsearch curator, how do I delete all indices matching a pattern, except for the newest?
I tried using filtertype: age but it does not seem to do what I need.

Answer 1

Here is an example code which you can use to delete the indices which
are older than 14 days assuming your index name have the date in it. You can get more information on the below link
https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/curator.html
import os
import sys
import json, io, boto3
import time, datetime
import curator
from elasticsearch import Elasticsearch, RequestsHttpConnection
from requests_aws4auth import AWS4Auth
import boto3
esEndPoint = ES_HOST # Add the ElasticSearch host.
region = REGION # Region where the ElasticSearch is present.
service = 'es'
credentials = boto3.Session().get_credentials()
awsauth = AWS4Auth(credentials.access_key, credentials.secret_key, region, service, session_token=credentials.token)
def lambda_handler(event, context):
esClient = connectES(esEndPoint)
index_list = curator.IndexList(esClient)
index_list.filter_by_age(source='name', direction='older', timestring='%Y.%m.%d', unit='days', unit_count=14)
print(index_list.indices)
if index_list.indices:
curator.DeleteIndices(index_list).do_action() # Delete the indices
def connectES(esEndPoint):
# Function used to connect to ES
try:
es = Elasticsearch(
hosts=[{'host': esEndPoint, 'port': 443}],
http_auth=awsauth,
use_ssl=True,
verify_certs=True,
connection_class=RequestsHttpConnection
)
return es
except Exception as E:
print("Unable to connect to {0}".format(esEndPoint))
print(E)

Answer 2

You need two filters: pattern (to match the indexes you want to delete) and age (to specify the age of the indexes to delete).
For instance the Curator configuration below is configured to delete
indexes named example_dev_*
and which are older than 10 days
Configuration:
actions:
1:
action: delete_indices
description: >-
Delete indices older than 10 days (based on index name), for example_dev_
prefixed indices.
options:
ignore_empty_list: True
disable_action: True
filters:
- filtertype: pattern
kind: prefix
value: example_dev_
- filtertype: age
source: creation_date
direction: older
unit: days
unit_count: 10
- filtertype: count
count: 1
You need to adapt both filter conditions to your needs, but that would achieve what you expect.

Answer 3

I suggest using the count filter after the pattern filter. Be sure to play with exclude true/false and dry-runs until it does what you expect.