shard allocation failing after power outage elasticsearch - elasticsearch

Question

[tag:failed to recover from translog]
[2016-08-25 11:23:12,410][WARN ][indices.cluster ] [Cobalt Man] [[fct_providers][4]] marking and sending shard failed due to [failed recovery]
[fct_providers][[fct_providers][4]] IndexShardRecoveryException[failed to recovery from gateway]; nested: EngineCreationFailureException[failed to recover from translog]; nested: EngineException[failed to recover from translog]; nested: ElasticsearchException[unexpected exception reading from translog snapshot of /root/elasticsearch-2.3.4/data/elasticsearch/nodes/0/indices/fct_providers/4/translog/translog-31.tlog]; nested: EOFException[read past EOF. pos [1047332] length: [4] end: [1047332]];
at org.elasticsearch.index.shard.StoreRecoveryService.recoverFromStore(StoreRecoveryService.java:250)
at org.elasticsearch.index.shard.StoreRecoveryService.access$100(StoreRecoveryService.java:56)
at org.elasticsearch.index.shard.StoreRecoveryService$1.run(StoreRecoveryService.java:129)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: [fct_providers][[fct_providers][4]] EngineCreationFailureException[failed to recover from translog]; nested: EngineException[failed to recover from translog]; nested: ElasticsearchException[unexpected exception reading from translog snapshot of /root/elasticsearch-2.3.4/data/elasticsearch/nodes/0/indices/fct_providers/4/translog/translog-31.tlog]; nested: EOFException[read past EOF. pos [1047332] length: [4] end: [1047332]];
at org.elasticsearch.index.engine.InternalEngine.(InternalEngine.java:177)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:25)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1509)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1493)
at org.elasticsearch.index.shard.IndexShard.internalPerformTranslogRecovery(IndexShard.java:966)
at org.elasticsearch.index.shard.IndexShard.performTranslogRecovery(IndexShard.java:938)
at org.elasticsearch.index.shard.StoreRecoveryService.recoverFromStore(StoreRecoveryService.java:241)
... 5 more
Caused by: [fct_providers][[fct_providers][4]] EngineException[failed to recover from translog]; nested: ElasticsearchException[unexpected exception reading from translog snapshot of /root/elasticsearch-2.3.4/data/elasticsearch/nodes/0/indices/fct_providers/4/translog/translog-31.tlog]; nested: EOFException[read past EOF. pos [1047332] length: [4] end: [1047332]];
at org.elasticsearch.index.engine.InternalEngine.recoverFromTranslog(InternalEngine.java:240)
at org.elasticsearch.index.engine.InternalEngine.(InternalEngine.java:174)
... 11 more
Caused by: ElasticsearchException[unexpected exception reading from translog snapshot of /root/elasticsearch-2.3.4/data/elasticsearch/nodes/0/indices/fct_providers/4/translog/translog-31.tlog]; nested: EOFException[read past EOF. pos [1047332] length: [4] end: [1047332]];
at org.elasticsearch.index.translog.TranslogReader.readSize(TranslogReader.java:102)
at org.elasticsearch.index.translog.TranslogReader.access$000(TranslogReader.java:46)
at org.elasticsearch.index.translog.TranslogReader$ReaderSnapshot.readOperation(TranslogReader.java:294)
at org.elasticsearch.index.translog.TranslogReader$ReaderSnapshot.next(TranslogReader.java:287)
at org.elasticsearch.index.translog.MultiSnapshot.next(MultiSnapshot.java:70)
at org.elasticsearch.index.shard.TranslogRecoveryPerformer.recoveryFromSnapshot(TranslogRecoveryPerformer.java:105)
at org.elasticsearch.index.shard.IndexShard$1.recoveryFromSnapshot(IndexShard.java:1578)
at org.elasticsearch.index.engine.InternalEngine.recoverFromTranslog(InternalEngine.java:238)
... 12 more
Caused by: java.io.EOFException: read past EOF. pos [1047332] length: [4] end: [1047332]
at org.elasticsearch.common.io.Channels.readFromFileChannelWithEofException(Channels.java:102)
at org.elasticsearch.index.translog.ImmutableTranslogReader.readBytes(ImmutableTranslogReader.java:84)
at org.elasticsearch.index.translog.TranslogReader.readSize(TranslogReader.java:91)
... 19 more