I'm new to Elastic Search, and I'm trying to figure out what does the cluster health represent. I have looked through the references and ES definitive guide but I can't find the answer.
This is described in the cluster health api page. The status is the key piece of information to look for.
Red = One or more primary shards are missing
Yellow = All primary shards are allocated but replicas are not
Green = All primary and
replica shards have been allocated
Related
When I run curl -X GET "elastic01:9200/_cat/indices?v"
I am observing that one of my index is having value red in health
I checked my cluster health and even That is in red
What can be done to bring the elasticsearch index health status from red to green.
Good start, you already know which index health value is in RED, which means that index is missing one or more primary shard, please identify them using this great blog post of elastic and see if some of your nodes in cluster is disconnected, holding the primary shards of RED index?
If you can't get back the nodes, holding the primary shards of the index, then as mentioned in the same blog, you have to loose the data and create empty primary shards using reroute API.
In the odd event that all nodes holding copies of this particular
shard are all permanently dead, the only recourse is to use the
reroute commands to allocate an empty/stale primary shard and accept
the fact that data has been lost.
I'm trying to move all the shards (primary and copies) from one specific elasticsearch node to others.
While doing some studies, I came to know about Cluster-level shard allocation filtering where I can specify the node name which I want to ignore while allocating shards.
PUT _cluster/settings
{
"transient" : {
"cluster.routing.allocation.exclude._name" : "data-node-1"
}
}
My questions are,
If I dynamically update the setting, will the shards be moved from the nodes that I excluded to other nodes automatically?
How can I check and make sure that all shards are moved from a specific node?
Yes, your shards will be moved automatically, if it is possible to do so:
Shards are only relocated if it is possible to do so without breaking another routing constraint, such as never allocating a primary and replica shard on the same node.
More information here
You can use the shards api to see the location of all shards. Alternatively, if you have access to a kibana Dashboard, you can see the shard allocation in the monitoring tab for shards or indices at the very bottom.
I have an elastic search cluster of 3 nodes (1 master and 2 data nodes), I have enabled xpack after that I was not able to start the master node. So I ran the elasticsearch-node repurpose command. And the cluster restarted.
But now I have the shards which are unassigned.
analytics-2019-11-19 0 p UNASSIGNED
analytics-2019-11-19 0 r UNASSIGNED
and the cluster status is red. I am new to elk. Let me know how to fix this and make the cluster green?
Thanks
In order to resolve UNASSIGNED shards issue you have to follow these steps:
Let's find out which shards are unassigned, and why run:
curl -XGET localhost:9200/_cat/shards?h=index,shard,prirep,state,unassigned.reason| grep UNASSIGNED
Via Kibana
GET _cat/shards?h=index,shard,prirep,state,unassigned.reason| grep UNASSIGNED
Let's use the cluster allocation explain API to try to garner more information about shard allocation issues
curl -XGET localhost:9200/_cluster/allocation/explain?pretty
Via Kibana
GET _cluster/allocation/explain?pretty
The resulting output will provide helpful details about why certain shards in your cluster remain unassigned.
For example:
You might see this explanation: "explanation" : "the shard cannot be allocated to the same node on which a copy of the shard already exists"
Meaning there is an index that you don’t need anymore and you can delete it to restore your cluster status to green.
If it is not the issue (the example) then it could be one of the following reasons:
-Shard allocation is purposefully delayed
-Too many shards, not enough nodes
-You need to re-enable shard allocation
-Shard data no longer exists in the cluster
-Low disk watermark
-Multiple Elasticsearch versions
Follow this guide to resolve unassigned shards issue
Hope this helps
I have script for uploading the data to ElasticSearch and it works fine with ES clusters containing 3 ES instances. But running the script against a 2-instance cluster throws that cluster into yellow status. Deleting the index restores them to green.
Found this: "A yellow cluster status means that the primary shards for all indices are allocated to nodes in a cluster, but the replica shards for at least one index are not."
How could I fix that? Should I improve my script somehow with a cluster size switch?
You certainly have in your index settings that you need 2 replicas. And as you cant have a replica and a primary shard on the same node, your cluster cant allocate all your shards in a 2 node cluster.
Could you try to decrease your number of replica to 1 ?
see here for the doc:
PUT /<your_index>/_settings
{
"index" : {
"number_of_replicas" : 1
}
}
Keep us posted !
I'm running with 3 nodes cluster on AWS EC2, one of my nodes crashed and after reboot I see 2900 unassigned shards and cluster state RED.
I configured indices to have 5 shards with 1 replica - and I don't understand why after rebooting the shards are not recovered from the replicas.
I tried to manually migrate shards with elasticsearch reroute API https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-reroute.html
But got errors:
can't cancel 2, failed to find it on node {infra-elasticsearch-1}
can't move 2, failed to find it on node {infra-elasticsearch-1}
[allocate_replica] trying to allocate a replica shard
[filebeat-demo00-2018.07.21][2], while corresponding primary shard is
still
unassigned"}],"type":"illegal_argument_exception","reason":"[allocate_replica]
trying to allocate a replica shard [filebeat-demo00-2018.07.21][2],
while corresponding primary shard is still unassigned
It's look like the some primary shard was lost (don't exists on disk) and I don't know how to the state back to GREEN.
thanks
Make sure the shard allocation is enabled in the active nodes by using the below API request
PUT _cluster/settings
{
"persistent": {
"cluster.routing.allocation.enable": null
}
}
Also you can check if the replica exists for the indexes whose primary shard has been lost by looking at the Indices information of the Monitoring app on Kibana.
To check the undergoing recovery process use the below API
GET /_recovery
I don't if this can help, but I just restarted the elasticsearch and kibana services. I waited for a few minutes, the cluster health changed from red to yellow then green in a matter of minutes.
on elastic cluster nodes:
#systemctl restart elasticsearch.service
on kibana node:
#systemctl restart kibana.service