Why auto delete index policy doesn't work in elasticsearch? - elasticsearch

Question

I created a index policy in Kibana to delete index order than 7 days. Below is the configuration:
And I have indexes who are using this policy but none of them get deleted. Below is one of the index setting configuration. It has already specified the policy to use: metrics-log-retention. Is there anything I missed?
{
"aws-logs-2022-02-01" : {
"settings" : {
"index" : {
"lifecycle" : {
"name" : "metrics-log-retention"
},
"routing" : {
"allocation" : {
"include" : {
"_tier_preference" : "data_content"
}
}
},
"number_of_shards" : "1",
"provided_name" : "aws-logs-2022-02-01",
"creation_date" : "1643673636747",
"priority" : "100",
"number_of_replicas" : "1",
"uuid" : "lLmO753nRpuw6bauKIJI2Q",
"version" : {
"created" : "7150299"
}
}
}
}
}
Below is the hot phase. I have disabled all options under hot as shown in below screenshot. but it still doesn't work.
Below is the raw data for the index policy:
{
"metrics-log-retention" : {
"version" : 4,
"modified_date" : "2022-02-10T22:24:14.492Z",
"policy" : {
"phases" : {
"hot" : {
"min_age" : "0ms",
"actions" : {
"rollover" : {
"max_size" : "50gb",
"max_primary_shard_size" : "50gb",
"max_age" : "1d"
}
}
},
"delete" : {
"min_age" : "6d",
"actions" : {
"delete" : {
"delete_searchable_snapshot" : true
}
}
}
}
},
"in_use_by" : {
"indices" : [
"aws-logs-2022-02-01",
"aws-logs-2022-02-04",
"aws-logs-2022-02-05",
"aws-logs-2022-02-02",
"aws-logs-2022-02-03",
"aws-metrics-2022-02-01",
"aws-metrics-2022-02-07",
"aws-logs-2022-02-08",
"aws-metrics-2022-02-06",
"aws-logs-2022-02-09",
"aws-logs-2022-02-06",
"aws-metrics-2022-02-09",
"aws-logs-2022-02-07",
"aws-metrics-2022-02-08",
"aws-metrics-2022-02-03",
"aws-metrics-2022-02-02",
"aws-metrics-2022-02-05",
"aws-metrics-2022-02-04",
"aws-logs-2022-02-11",
"aws-logs-2022-02-12",
"aws-logs-2022-02-10",
"aws-logs-2022-02-13",
"aws-metrics-2022-02-10",
"aws-metrics-2022-02-12",
"aws-metrics-2022-02-11",
"aws-metrics-2022-02-13"
],
"data_streams" : [ ],
"composable_templates" : [ ]
}
}
}

Answer 1

As you can see on the hot phase advanced settings, the default rollover settings are 30 days or 50GB, so your indexes will stay in the hot phase for 30 days, unless they grow over 50GB before.
Once the index gets out of the hot phase it gets into the delete phase and if you hover over the (i) icon, you can see that the 7 days are calculated AFTER the roll over from the hot phase.
So if you really want your indexes to be deleted after 7 days, you need to:
configure the hot phase to be shorter (say 6 days)
configure the delete phase to kick in after 1 day from rollover
That way, the index will be created and stay six days in the hot phase and then be deleted after one day.

Answer 2

Just add it to crontab on ES host, it will delete old indices automatically
0 7 * * * curl -u LOGIN:PASSWORD -XDELETE http://localhost:9200/aws-logs-$(date --date="7 days ago" +"%Y.%m.%d")