Elasticsearch - How to remove stuck persistent setting after upgrade - elasticsearch

Question

I have just upgraded my cluster from 5.6 to 6.1. I did a rolling upgrade as the documentation specified. It looks like a setting that I was using isn't available anymore in 6.1. That would've been fine, but now I can't even enable my shard allocation, so now my last node won't allocate its shards. Doing something as simple as this:
curl -XPUT 'localhost:9200/_cluster/settings?pretty' -H 'Content-Type: application/json' -d'
{
"persistent" : {
"cluster.routing.allocation.enable" : "all"
}
}
results in this:
{
"error" : {
"root_cause" : [
{
"type" : "remote_transport_exception",
"reason" : "[inoreader-es4][92.247.179.253:9300][cluster:admin/settings/update]"
}
],
"type" : "illegal_argument_exception",
"reason" : "unknown setting [indices.store.throttle.max_bytes_per_sec] did you mean [indices.recovery.max_bytes_per_sec]?"
},
"status" : 400
}
No matter what setting I try to change I always get this error.
Yes, I did set indices.store.throttle.max_bytes_per_sec as persistent setting once in 5.x, and I'm OK with having to set it to a new name now, but how can I even remove it? It's not in elasticsearch.yml.

Answer 1

You'll need to unset that value. If you are still on the old version, you can use the following command (in 5.0 unsetting with null was added):
PUT _cluster/settings
{
"persistent": {
"indices.store.throttle.max_bytes_per_sec": null
}
}
This will however fail with a "persistent setting [indices.store.throttle.max_bytes_per_sec], not recognized" in your cluster if you have already upgraded.
At the moment (Elasticsearch version 6.1.1) the removed setting will be archived under archived.indices.store.throttle.max_bytes_per_sec. You can remove this and any other archived setting with:
PUT _cluster/settings
{
"persistent": {
"archived.*": null
}
}
However, there is a bug that only lets you unset archived settings before you change any other settings.
If you have already made other settings and are affected by this bug, the only solution is to downgrade to 5.6 again, unset the configuration (command at the top of this answer), and then do the upgrade again. It's probably enough to do this on one node (stop all others) as long as it's the master and all other nodes join that master and accept its corrected cluster state. Be sure to take a snapshot before in any case.
For future versions the archived.* behavior will probably change as stated in the ticket (though it's just in the planning phase right now):
[...] we should not archive unknown and broken cluster settings.
Instead, we should fail to recover the cluster state. The solution for
users in an upgrade case would be to rollback to the previous version,
address the settings that would be unknown or broken in the next major
version, and then proceed with the upgrade.
Manually editing or even deleting the cluster state on disk sounds very risky: The cluster state includes a lot of information (check for yourself with GET /_cluster/state) like templates, indices, routing table,... Even if you have the data of the data nodes, but you lost the cluster state, you wouldn't be able to access your data (the "map" how to form indices out of the shards is missing). If I remember correctly, in more recent ES versions the data nodes cache the cluster state and will try to restore from that, but that's a last resort and I wouldn't want to rely on it. Also I'm not sure if that might not also bring back your bad setting.
PS: I can highly recommend the free upgrade assistant going from 5.6 to 6.x.