ElasticSearch Can't open file to read checksums - elasticsearch

Question

ElasticSearch ran out of disk space on a couple nodes and this is the resulting error
The error we're getting is:
[WARN][indices.store ] [odr-es-md15] Can't open file to read checksums
java.io.FileNotFoundException: No such file [_2rlb_es090_0.pos]
at org.elasticsearch.index.store.DistributorDirectory.getDirectory(DistributorDirectory.java:176)
at org.elasticsearch.index.store.DistributorDirectory.getDirectory(DistributorDirectory.java:144)
at org.elasticsearch.index.store.Store$MetadataSnapshot.buildMetadata(Store.java:482)
at org.elasticsearch.index.store.Store$MetadataSnapshot.<init>(Store.java:456)
at org.elasticsearch.index.store.Store.readMetadataSnapshot(Store.java:281)
...
...
...

Answer 1

The translog may be corrupt. I followed the directions from here:
http://unpunctualprogrammer.com/2014/05/13/corrupt-elasticsearch-translogs/
To fix this you need to look in the ES Logs,
/var/log/elasticsearch/elasticsearch.log for CentOS, and find the
error lines above. On those lines you’ll see something like
[<timestamp>][WARN ][cluste.action.shard] [<wierd name>] [logstash-2014.05.13][X]
where X (shard) is some number, likely (0,1,2,3,4), and the block
before that, logstash-date for me, and you if your doing centralized
logging like we are, is the index name. You then need to go to the
index location, /var/lib/elasticsearch/elasticsearch/nodes/0/indices/
on centos. In that directory you’ll be able to find the following
structure, logstash-date/X/translog/translog-.
That’s the file you’ll need to delete, so:
sudo service stop elasticsearch
sudo rm /var/lib/elasticsearch/elasticsearch/nodes/0/indices/logstash-date/X/translog/translog-blalblabla
repeat step 2 for all indices and shards in the error log
sudo service start elasticsearch
Watch the logs and repeat that process as needed until the ES logs
stop spitting out stack traces.