Frequent circuit Breaking Exceptions occurring in ES Cluster V7.3.1? - elasticsearch

Question

We are using ES v7.3.1 and from past few days the shards in our ES cluster get unassigned because of circuit breaking exception, I am not able to understand the exact reason which leads to this exception, any help would be really helpful.
This is the detailed info that I get using command GET _cluster/allocation/explain
"unassigned_info" : {
"reason" : "ALLOCATION_FAILED",
"at" : "2021-02-12T08:05:40.154Z",
"failed_allocation_attempts" : 1,
"details" : "failed shard on node [WbHklo7iSf6jGj90cP9Y-A]: failed to perform indices:data/write/bulk[s] on replica [segment_index_573179789d2572f27bc73e6b][6], node[WbHklo7iSf6jGj90cP9Y-A], [R], s[STARTED], a[id=SNoWjhhYRXClfVqa6lsDAQ], failure RemoteTransportException[[ip-1-0-104-220][1.0.104.220:9300][indices:data/write/bulk[s][r]]]; nested: CircuitBreakingException[[parent] Data too large, data for [<transport_request>] would be [14311120802/13.3gb], which is larger than the limit of [14247644364/13.2gb], real usage: [14311004440/13.3gb], new bytes reserved: [116362/113.6kb], usages [request=0/0b, fielddata=808714408/771.2mb, in_flight_requests=15836072/15.1mb, accounting=683535018/651.8mb]]; ",
"last_allocation_status" : "no_attempt"
}

Answer 1

Your shard allocation explain API is providing all the details you need to resolve the issue
indices:data/write/bulk is causing the circuit breaker.
WbHklo7iSf6jGj90cP9Y-A node on which issue is happening.
its happening on replica shard segment_index_573179789d2572f27bc73e6b
below is the actual limit and data which you are sending which is causing data circuit breaker.
Parent circuit breaker is happening on your node, as mentioned in the log line, I have highlighted the parent and actual limit is 13.2 GB but in your bulk request you are sending 13.3 GB data.
CircuitBreakingException[[parent] Data too large, data for
[<transport_request>] would be [14311120802/13.3gb], which is larger
than the limit of [14247644364/13.2gb], real usage:
[14311004440/13.3gb], new bytes reserved: [116362/113.6kb]
Solution
You can reduce the size of your bulk request to avoid this circuit breaker

Answer 2

Can you provide the result of cluster health?
curl -XGET "http://localhost:9200/_cluster/health?pretty"
Can you provide the results of JVM pressure metric?
As there is not much information, below I explain what happened to me in a similar case of circuit breaking, and what I did in my case.
I am confident that the bulk size is small, the sum of several things(search + bulk + assigment process of new shards) causes the RAM to fill up.
In the cases that I know, "Data too large", refers to the JVM heap. Perhaps, you have reached the memory limit, trigging the circuit breaker. I dare to think(with a high probability of being wrong :/ ) that your ram per node is proportional to the % of the configured circuit breaker.
For example, if you have a circuit breaker set to 80% and it shows an error at 13.3 GB, you surely have 16 GB of RAM (100%) per node.
Depending of your circuit breaker configuration, as a temp workaround you can increses the number temporarily. If your circuit breaker is already high, it is not advisable to increase it.
Maybe your bulk its creating multiples indices, multiples indices creates multiples shards. Too many shards fill the memory, because each shards create memory, filedescriptor uses memory. There have been cases where the logic / pattern of creating new indices is incorrect, and they generate many new indices (creating many shards) inadvertently instead of creating just one index.
Maybe you were already at the limits and you did not know it, until now that you created new indexes (therefore new shards), and the problem arose.
How many shards do you have? curl -XGET "http://localhost:9200/_cat/shards" | wc -l
How many shards support each of your nodes?
When you reach the limit offset with shard your can reduce the number of replica shards, as a temp workaround. In order to complete the bulk.
An option that works for me was disabling temporary shards replica for less important indices.
Check your shards: curl -XGET "http://localhost:9200/_cat/shards"
curl -X "PUT" "http://localhost:9200/mylessimportant_indices/_settings" \
-H "Content-type: application/json" \
-d $'{
"index" : {
"number_of_replicas":0
}
}' | jq