I have been asked to restore data for a 3 node ES cluster to a new read-only cluster.
The new cluster is only for showing old log data and have very few request.
I have set up one server that will be my "cluster".
When I run my restore command I get 5 shards and 5 unassigned shards and I think that this is redundant as one must be enough.
How can I restore my data so I use as little disk space as possible?
Your cluster must be yellow since there are unassigned shards. Simply run the following command to remove the unassigned replica shards and the cluster will turn green again:
PUT index-name/_settings
{
"number_of_replicas": 0
}
Just note, though, that removing the unassigned replicas will not save you any disk space since those replica shards do not take up any space because they are unassigned anayway.
Related
I am running a three node Elasticsearch (ELK) cluster. All nodes have all and the same roles, e.g. data, master, etc. The disk on node 3 where the data folder is assigned became corrupt and that data is probably unrecoverable. The other nodes are running normally and one of them assumed the master role instead.
Will the cluster work normally if I replace the disk and make the empty directory available to elastic again, or am I risking crashing the whole cluster?
EDIT: As this is not explicitly mentioned in the answer, yes, if you add your node with an empty data folder, the cluster will continue normally as if you added a new node to the cluster, but you have to deal with the missing data. In my case, I lost the data as I do not have replicas.
Let me try to explain that in simple way.
Your data got corrupt at node-3 so if you add that that node again, it will not have the older data, i.e. the shards stored in node-3 will remain unavailable for the cluster.
Did you have the replica shards configured for the indexes?
What is the current status(yellow/red) of the cluster when you have
node-3 removed?
If a primary shard isn't available then the master-node promotes one of the active replicas to become the new primary. If there are currently no active replicas then status of the cluster will remain red.
I have an elastic search cluster of 3 nodes (1 master and 2 data nodes), I have enabled xpack after that I was not able to start the master node. So I ran the elasticsearch-node repurpose command. And the cluster restarted.
But now I have the shards which are unassigned.
analytics-2019-11-19 0 p UNASSIGNED
analytics-2019-11-19 0 r UNASSIGNED
and the cluster status is red. I am new to elk. Let me know how to fix this and make the cluster green?
Thanks
In order to resolve UNASSIGNED shards issue you have to follow these steps:
Let's find out which shards are unassigned, and why run:
curl -XGET localhost:9200/_cat/shards?h=index,shard,prirep,state,unassigned.reason| grep UNASSIGNED
Via Kibana
GET _cat/shards?h=index,shard,prirep,state,unassigned.reason| grep UNASSIGNED
Let's use the cluster allocation explain API to try to garner more information about shard allocation issues
curl -XGET localhost:9200/_cluster/allocation/explain?pretty
Via Kibana
GET _cluster/allocation/explain?pretty
The resulting output will provide helpful details about why certain shards in your cluster remain unassigned.
For example:
You might see this explanation: "explanation" : "the shard cannot be allocated to the same node on which a copy of the shard already exists"
Meaning there is an index that you don’t need anymore and you can delete it to restore your cluster status to green.
If it is not the issue (the example) then it could be one of the following reasons:
-Shard allocation is purposefully delayed
-Too many shards, not enough nodes
-You need to re-enable shard allocation
-Shard data no longer exists in the cluster
-Low disk watermark
-Multiple Elasticsearch versions
Follow this guide to resolve unassigned shards issue
Hope this helps
I have script for uploading the data to ElasticSearch and it works fine with ES clusters containing 3 ES instances. But running the script against a 2-instance cluster throws that cluster into yellow status. Deleting the index restores them to green.
Found this: "A yellow cluster status means that the primary shards for all indices are allocated to nodes in a cluster, but the replica shards for at least one index are not."
How could I fix that? Should I improve my script somehow with a cluster size switch?
You certainly have in your index settings that you need 2 replicas. And as you cant have a replica and a primary shard on the same node, your cluster cant allocate all your shards in a 2 node cluster.
Could you try to decrease your number of replica to 1 ?
see here for the doc:
PUT /<your_index>/_settings
{
"index" : {
"number_of_replicas" : 1
}
}
Keep us posted !
I'm running with 3 nodes cluster on AWS EC2, one of my nodes crashed and after reboot I see 2900 unassigned shards and cluster state RED.
I configured indices to have 5 shards with 1 replica - and I don't understand why after rebooting the shards are not recovered from the replicas.
I tried to manually migrate shards with elasticsearch reroute API https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-reroute.html
But got errors:
can't cancel 2, failed to find it on node {infra-elasticsearch-1}
can't move 2, failed to find it on node {infra-elasticsearch-1}
[allocate_replica] trying to allocate a replica shard
[filebeat-demo00-2018.07.21][2], while corresponding primary shard is
still
unassigned"}],"type":"illegal_argument_exception","reason":"[allocate_replica]
trying to allocate a replica shard [filebeat-demo00-2018.07.21][2],
while corresponding primary shard is still unassigned
It's look like the some primary shard was lost (don't exists on disk) and I don't know how to the state back to GREEN.
thanks
Make sure the shard allocation is enabled in the active nodes by using the below API request
PUT _cluster/settings
{
"persistent": {
"cluster.routing.allocation.enable": null
}
}
Also you can check if the replica exists for the indexes whose primary shard has been lost by looking at the Indices information of the Monitoring app on Kibana.
To check the undergoing recovery process use the below API
GET /_recovery
I don't if this can help, but I just restarted the elasticsearch and kibana services. I waited for a few minutes, the cluster health changed from red to yellow then green in a matter of minutes.
on elastic cluster nodes:
#systemctl restart elasticsearch.service
on kibana node:
#systemctl restart kibana.service
Below I have 2 nodes in two servers. 2 indexes each. The indexes are distributed in 2 shards and 1 replica set.
"Thor" node had a downtime so I "Iron_man" took over. That's fine.
As you can see events_v1 is an index created before the downtime and venue_v1 was created after the downtime. Shouldn't "Thor" after being back alive take over one shard automatically in the same way as it handles the newly created venue index?
If yes how should I configure the settings?
You need not to configure anything for above scenario.Because Elasticsearch's default behavior is same as you requested.if you create replica for a shard in same node is not allocated to query.If you add new node means your replica shard will be allocated in newly created node.
for more information watch this video