TransportError(403, u'cluster_block_exception', u'blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];') - elasticsearch

Question

When I try to store anything in elasticsearch, An error says that:
TransportError(403, u'cluster_block_exception', u'blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];')
I already inserted about 200 millions documents in my index. But I don't have an idea why this error is happening.
I've tried:
curl -u elastic:changeme -XPUT 'localhost:9200/_cluster/settings' -H 'Content-Type: application/json' -d '{"persistent":{"cluster.blocks.read_only":false}}'
As mentioned here:
ElasticSearch entered "read only" mode, node cannot be altered
And the results is:
{"acknowledged":true,"persistent":{"cluster":{"blocks":{"read_only":"false"}}},"transient":{}}
But nothing changed. what should I do?

Answer 1

Try GET yourindex/_settings, this will show yourindex settings. If read_only_allow_delete is true, then try:
PUT /<yourindex>/_settings
{
"index.blocks.read_only_allow_delete": null
}
I got my issue fixed.
plz refer to es config guide for more detail.
The curl command for this is
curl -X PUT "localhost:9200/twitter/_settings?pretty" -H 'Content-Type: application/json' -d '
{
"index.blocks.read_only_allow_delete": null
}'

Answer 2

Last month I facing the same problem, you can try this code on your Kibana Dev Tools
curl -XPUT -H "Content-Type: application/json" http://localhost:9200/_all/_settings -d '{"index.blocks.read_only_allow_delete": null}'
I hope it helps

Answer 3

I had faced the same issue when my disk space was full,
please see the steps that I did
1- Increase the disk space
2- Update the index read-only mode, see the following curl request
curl -XPUT -H "Content-Type: application/json"
http://localhost:9200/_all/_settings -d
'{"index.blocks.read_only_allow_delete": null}'

Answer 4

This happens because of the default watermark disk usage of Elastic Search. Usually it is 95% of disk size.
This happens when Elasticsearch thinks the disk is running low on space so it puts itself into read-only mode.
By default Elasticsearch's decision is based on the percentage of disk space that's free, so on big disks this can happen even if you have many gigabytes of free space.
The flood stage watermark is 95% by default, so on a 1TB drive you need at least 50GB of free space or Elasticsearch will put itself into read-only mode.
For docs about the flood stage watermark see https://www.elastic.co/guide/en/elasticsearch/reference/6.2/disk-allocator.html.
Quoted from part of this answer
One solution is to disable it enitrely (I found it useful in my local and CI setup). To do it run the 2 commands:
curl -XPUT -H "Content-Type: application/json" http://localhost:9200/_cluster/settings -d '{ "transient": { "cluster.routing.allocation.disk.threshold_enabled": false } }'
curl -XPUT -H "Content-Type: application/json" http://localhost:9200/_all/_settings -d '{"index.blocks.read_only_allow_delete": null}'

Answer 5

Tagging into this later on as I just encountered the problem myself - I accomplished the following steps.
1) Deleted older indexes to free up space immediately - this brought me to around 23% free.
2) Update the index read-only mode.
I still had the same issue. I checked the Dev Console to see what might be locked still and none were. Restarted the cluster and had the same issue.
Finally under index management I selected the indexes with ILM lifecycle issues and picked to reapply ILM step. Had to do that a couple of times to clear them all out but it did.

Answer 6

The problem may be a disk space problem, i had this problem despite i cleaned many space my disk, so, finally i delete the data folder and it worked: sudo rm -rf /usr/share/elasticsearch/data/

Answer 7

This solved the issue;
PUT _settings { "index": { "blocks": { "read_only_allow_delete": "false" }
}