How to solve unassigned_shards in elasticsearch?
When i run command curl -XGET 'localhost:9200/_cluster/health?pretty'
i got following result
{
"cluster_name" : "elasticsearch",
"status" : "yellow",
"timed_out" : false,
"number_of_nodes" : 1,
"number_of_data_nodes" : 1,
"active_primary_shards" : 145,
"active_shards" : 145,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 145,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 50.0
}
As your cluster only has one node, there will always be unassigned shards - those are replica shards. They are used to restore data when a nodes crashes. They will assign to another node automaticly as soon as your cluster has two nodes. If you dont want to have them, because u are in a local development for example, you can set the replica shard size to 0 in the index mapping. You can look up how to do that here in the Update Indices Settings documentation.
Related
In my cluster trying to insert message (from filebeat) I get
(status=400): {"type":"illegal_argument_exception","reason":"Validation Failed: 1: this action would add [2] shards, but this cluster currently has [2999]/[3000] maximum normal shards open;"}, dropping event!
looking at the cluster health its looks like it has available shards
"cluster_name" : "elastic",
"status" : "yellow",
"timed_out" : false,
"number_of_nodes" : 4,
"number_of_data_nodes" : 1,
"active_primary_shards" : 1511,
"active_shards" : 1511,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 1488,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 1159,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 57267,
"active_shards_percent_as_number" : 50.3834611537179
any ideas ?
Tldr;
It seems like you only have one data node.
As you can see you have 1511 allocated shards, and 1488 unassigned shards
And guess what happens if we add them up ^^.
We get the 2999.
Elasticsearch does not allow to have the primary shards and its replicas on the same node.
But since you have only one data node all those unassigned shards are most likely replicas.
Solution
short term
Create your new index with 0 replica.
POST /my-index-000001
{
"settings": {
"index" : {
"number_of_replicas" : 0
}
}
}
You could also set all other index with no replica via:
PUT /*/_settings
{
"index" : {
"number_of_replicas" : 0
}
}
Long term
You either need to change all you indices to only have one replica or add another node. So that those replicas get allocated.
At the moment you are risking data loss of the node crash.
Then as per the documentation you can change the value of cluster.routing.allocation.total_shards_per_node which must be set a 3000
curl -XGET 'http://localhost:9200/_cluster/health?level=shards'
{
"cluster_name" : "elasticsearch",
"status" : "yellow",
"timed_out" : false,
"number_of_nodes" : 1,
"number_of_data_nodes" : 1,
"active_primary_shards" : 1486,
"active_shards" : 1486,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 1486,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 50.0
}
How to solve unassigned shards and active_shards_percent_as_number
** Am using ES 2.4 latest **
thanking you
It's simply because you have a single node in your cluster but all your indices are configured to have one replica per shard (the default setting).
If you run the following command you'll get rid of the replicas and your cluster will turn green:
curl -XPUT 'localhost:9200/_settings' -d '{
"index" : {
"number_of_replicas" : 0
}
}'
Elasticsearch 1.7.2 on CentOS, 8GB RAM, 2 node cluster.
We posted the whole log here: http://pastebin.com/zc2iG2q4
When we look at /_cluster/health , we see 2 unassigned shards:
{
"cluster_name" : "elasticsearch-prod",
"status" : "yellow", <--------------------------
"timed_out" : false,
"number_of_nodes" : 2,
"number_of_data_nodes" : 2,
"active_primary_shards" : 5,
"active_shards" : 8,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 2, <--------------------------
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0
In the log, we see:
marking and sending shard failed due to [failed to create shard]
java.lang.OutOfMemoryError: Java heap space
And other errors.
The only memory related config value we have is:
indices.fielddata.cache.size: 75%
We are looking to:
understand the log more completely
understand what action we need to take to address the situation now (recover) and prevent it in the future
Additional details:
1) ES_HEAP_SIZE is stock, no changes. (Further, looking around, it is not clear where best to change it.... /etc/init.d/elasticsearch ?)
2) Our jvm stats are below. (And please note, as a test, I modded "/etc/init.d/elasticsearch" and and added export ES_HEAP_SIZE=4g [in place of the existing "export ES_HEAP_SIZE" line] and restarted ES.... Comparing two identical nodes, one with the changed elasticsearch file, and one stock, the values below appear identical)
"jvm" : {
"timestamp" : 1448395039780,
"uptime_in_millis" : 228297,
"mem" : {
"heap_used_in_bytes" : 81418872,
"heap_used_percent" : 7,
"heap_committed_in_bytes" : 259522560,
"heap_max_in_bytes" : 1037959168,
"non_heap_used_in_bytes" : 50733680,
"non_heap_committed_in_bytes" : 51470336,
"pools" : {
"young" : {
"used_in_bytes" : 52283368,
"max_in_bytes" : 286326784,
"peak_used_in_bytes" : 71630848,
"peak_max_in_bytes" : 286326784
},
"survivor" : {
"used_in_bytes" : 2726824,
"max_in_bytes" : 35782656,
"peak_used_in_bytes" : 8912896,
"peak_max_in_bytes" : 35782656
},
"old" : {
"used_in_bytes" : 26408680,
"max_in_bytes" : 715849728,
"peak_used_in_bytes" : 26408680,
"peak_max_in_bytes" : 715849728
}
}
},
"threads" : {
"count" : 81,
"peak_count" : 81
},
"gc" : {
"collectors" : {
"young" : {
"collection_count" : 250,
"collection_time_in_millis" : 477
},
"old" : {
"collection_count" : 1,
"collection_time_in_millis" : 22
}
}
},
"buffer_pools" : {
"direct" : {
"count" : 112,
"used_in_bytes" : 20205138,
"total_capacity_in_bytes" : 20205138
},
"mapped" : {
"count" : 0,
"used_in_bytes" : 0,
"total_capacity_in_bytes" : 0
}
}
},
Solved.
The key here is the error "java.lang.OutOfMemoryError: Java heap space"
Another day, another gem from the ES docs:
https://www.elastic.co/guide/en/elasticsearch/guide/current/heap-sizing.html
says (emphasis mine):
The default installation of Elasticsearch is configured with a 1 GB heap. For just about every deployment, this number is far too small. If you are using the default heap values, your cluster is probably configured incorrectly.
Resolution:
Edit: /etc/sysconfig/elasticsearch
Set ES_HEAP_SIZE=4g // this system has 8GB RAM
Restart ES
And tada.... the unassigned shards are magically assigned, and the cluster goes green.
I have a elasticsearch cluster with 2 data nodes with one replica node (green status). If I add a new node, the status will still be green. If I shutdown elasticsearch on the newly added node, I get a yellow status. How could I make understand to revert back to 2 nodes instead of 3 and get a green status?
Before shutdown:
{
"cluster_name" : "elastic_poiuytrez",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 5,
"number_of_data_nodes" : 3,
"active_primary_shards" : 65,
"active_shards" : 130,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0
}
curl -XGET http://localhost:9200/_cluster/settings?pretty=true
{
"persistent" : { },
"transient" : { }
}
This is what I get when I shutdown one of the node:
{
"cluster_name" : "elastic_poiuytrez",
"status" : "yellow",
"timed_out" : false,
"number_of_nodes" : 4,
"number_of_data_nodes" : 2,
"active_primary_shards" : 65,
"active_shards" : 87,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 43,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0
}
Nodes settings: http://pastebin.com/wpmsRAbK (elastic-node-1 is down)
From my experience of cluster status green/yellow/red:
a) Depending on number of replicas: Some replicas have been put on the new node, and have to be recreated or moved to remaining node(s)
b) If ES ran tight on memory, and for other reasons (like removing a node) one or more replicas can go to unassigned. The cluster will be yellow until "unassigned_shards" goes down to 0
Looking at your post, notice:
"unassigned_shards" : 43,
That will cause the cluster to be in yellow status. It may take some time to get everything shuffled around on the remaining nodes.
Watching "unassigned_shards" can show you "progress toward green"
My installation is working fine, but one index is showing yellow health due to 2 'extra' unassigned shards shown in it. How do I remove these extra shards? My current settings are 0 replicas and 2 shards.
curl -XGET 'http://localhost:9200/_cluster/health/tags?level=shards&pretty=true'
{
"cluster_name" : "elasticsearch_inspire",
"status" : "yellow",
"timed_out" : false,
"number_of_nodes" : 1,
"number_of_data_nodes" : 1,
"active_primary_shards" : 2,
"active_shards" : 2,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 2,
"indices" : {
"v1_tags" : {
"status" : "yellow",
"number_of_shards" : 2,
"number_of_replicas" : 1,
"active_primary_shards" : 2,
"active_shards" : 2,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 2,
"shards" : {
"0" : {
"status" : "yellow",
"primary_active" : true,
"active_shards" : 1,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 1
},
"1" : {
"status" : "yellow",
"primary_active" : true,
"active_shards" : 1,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 1
}
}
}
}
}
I recently had this issue where ES process was restarted on all nodes and I had some unassigned shards (cluster health was red).
I did the following to assign the shards manually and that did the trick for me (no restart was needed)
curl -XPOST 'localhost:9200/_cluster/reroute' -d '{
"commands": [
{
"allocate": {
"index": "INDEXNAME",
"shard": "SHARDNUMBER",
"node": "PICKANODE",
"allow_primary": 1
}
}
]
}'
Restarting the node seemed to be the only way to solve it.
If anyone has a less-aggressive solution, please post it :)