Elasticsearch cluster dropping messages - elasticsearch

Question

In my cluster trying to insert message (from filebeat) I get
(status=400): {"type":"illegal_argument_exception","reason":"Validation Failed: 1: this action would add [2] shards, but this cluster currently has [2999]/[3000] maximum normal shards open;"}, dropping event!
looking at the cluster health its looks like it has available shards
"cluster_name" : "elastic",
"status" : "yellow",
"timed_out" : false,
"number_of_nodes" : 4,
"number_of_data_nodes" : 1,
"active_primary_shards" : 1511,
"active_shards" : 1511,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 1488,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 1159,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 57267,
"active_shards_percent_as_number" : 50.3834611537179
any ideas ?

Answer 1

Tldr;
It seems like you only have one data node.
As you can see you have 1511 allocated shards, and 1488 unassigned shards
And guess what happens if we add them up ^^.
We get the 2999.
Elasticsearch does not allow to have the primary shards and its replicas on the same node.
But since you have only one data node all those unassigned shards are most likely replicas.
Solution
short term
Create your new index with 0 replica.
POST /my-index-000001
{
"settings": {
"index" : {
"number_of_replicas" : 0
}
}
}
You could also set all other index with no replica via:
PUT /*/_settings
{
"index" : {
"number_of_replicas" : 0
}
}
Long term
You either need to change all you indices to only have one replica or add another node. So that those replicas get allocated.
At the moment you are risking data loss of the node crash.
Then as per the documentation you can change the value of cluster.routing.allocation.total_shards_per_node which must be set a 3000