Total size of similar indices - elasticsearch

Question

Kibana shows statistics for every index on the monitoring page 1. How can we group indices by type to get their overall size? For example I've got a lot of winlogbeat-6.2.2-YYYY.mm.dd indices and would like to know how much space do all of them consume in total.
thanks!

Answer 1

One way to achieve what you want is to use the Index stats API, and filter out the store.size_in_bytes value, like this:
winlogbeat-6.2.2*/_stats?filter_path=_all.total.store.size_in_bytes
You'll get a response like this:
{
"_all": {
"total": {
"store": {
"size_in_bytes": 922069687976
}
}
}
}
Another way to achieve what you want involves leveraging the Cat APIs, a bit of grep and a tad of awk...
The following shell command will give you the number of bytes consumed by all your winlogbeat-6.2.2 indices:
curl -s localhost:9200/_cat/indices?bytes=b | grep winlogbeat-6.2.2 | awk '{s+=$9} END {print s}'
You'll get a single number, like this: 922069687976
Let me explain:
The first command will retrieve all indices via the _cat/indices API.
curl -s localhost:9200/_cat/indices?bytes=b
The second command keeps only the indices matching winlogbeat-6.2.2
grep winlogbeat-6.2.2
The last command does the magic of summing up all numbers in the 9th column (i.e. store.size)
awk '{s+=$9} END {print s}'
Voilà...

Answer 2

If you collect monitoring stats for your cluster, when you could use Kibana for visualization:
Prerequisites: X-Pack which will be creating .monitoring-* indices.
Create a scripted field for the index pattern .monitoring-es-6-*:
Field name: normalized_index_name (will work for for SOME-INDEX-YYYY.MM.DD pattern only as it removes everything from the index name after the last dash, a scripted field with a regex can be used, but regex fields must be explicitly allowed in the ES config):
def name = doc['index_stats.index'].value;
if (name != null) {
int lastDashIndex = name.lastIndexOf('-');
if (lastDashIndex > 0) {
return name.substring(0,lastDashIndex);
}
}
return name
Create a Line visualization. Note:
time frame should be set to 7 days
start to build the visualization from X-axis, otherwise it may not split series properly (bug?..)!