ElasticSearch too_many_nested_clauses Query contains too many nested clauses; maxClauseCount is set to 1024 - elasticsearch

Question

we are trying to run a very simple Lucene (ver 9.3.0) query using ElasticSearch (ver 8.4.1) in Elastic cloud. Our index mapping has around 800 fields.
GET index-name/_search
{
"query": {
"query_string": {
"query": "Abby OR Alta"
}
}
}
However we are getting back an exception:
{
"error" : {
"root_cause" : [
{
"type" : "too_many_nested_clauses",
"reason" : "Query contains too many nested clauses; maxClauseCount is set to 1024"
}
],
"type" : "search_phase_execution_exception",
"reason" : "all shards failed",
"phase" : "query",
"grouped" : true,
},
"status" : 500
}
Now from what I've read in this article link there was a breaking change since Lucene 9 which Elastic 8.4 uses.
The behavior changed dramatically on how this max clause value is counted. From previously being
num_terms = max_num_clauses
to
num_terms = max_num_clauses * num_fields_in_index
So in our case it would be 800 * 2 = 1600 > 1024
Now what I don't understand is why such a limitation was introduced and to what value we should actually change this setting ?
A OR B query with 800 fields in the index doesn't strike me as something unusual or being problematic from performance perspective.

Answer 1

The "easy" way out is to increase the indices.query.bool.max_clause_count limit in the configuration file to a higher value. It used to be 1024 in ES 7 and now in ES 8 it has been raised to 4096. Just be aware, though, that doing so might harm the performance of your cluster and even bring nodes down depending on your data volume.
Here is some interesting background information on how that the "ideal" value is calculated based on the hardware configuration as of ES 8.
A better way forward is to "know your data" and identify the fields to be used in your query_string query and either specify those fields in the query_string.fields array or modify your index settings to specify them as default fields to be searched on when no fields are specified in your query_string query:
PUT index/_settings
{
"index.query.default_field": [
"description",
"title",
...
]
}