ElasticSearch Unassigned Shard - elasticsearch

I have 2 nodes in elastic search cluster with 8 CPU and 16 GB RAM. I have set ES_HEAP_SIZE to 10 GB.
In my yml configuration file on both machines i have set
index.number_of_shards: 5
index.number_of_replicas: 1
And both machines are allowed as master/data true.Now problem is my 0th shard of node 1 is unassigned after restart.I tried
for shard in $(curl -XGET http://localhost:9201/_cat/shards | grep UNASSIGNED | awk '{print $2}'); do
echo "processing $shard"
curl -XPOST 'localhost:9201/_cluster/reroute' -d '{
"commands" : [ {
"allocate" : {
"index" : "inxn",
"shard" : '$shard',
"node" : "node1",
"allow_primary" : true
}
}
]
}'
done
it does not give any error and says acknowledged true and show status of shard to initialize but when i view shard its still uninitialized.
Am I doing anything wrong in setting? Should I make both node as master/data true and on both machines set shard:5 and replica:1
Any help or suggestion would be greatly appreciated.
Thanks

I did a trick to solve the same , I renamed 0 folder under indices on node1 and did a force full assign 0th shard on node1 and it worked for me.
curl -XPOST 'localhost:9201/_cluster/reroute' -d '{
"commands" : [ {
"allocate" : {
"index" : "inxc",
"shard" : 0,
"node" : "node1",
"allow_primary" : true
}
}
]
}'

Related

elasticsearch 7 is in YELLOW state with unassigned shard

I'm unable to get cluster of single master node in working green condition:
elasticsearch: 7.17.0
I guess that is because I have unassigned_shards > 0
config:
apiVersion: v1
data:
elasticsearch.yml: |-
discovery:
type: single-node
network:
host: 0.0.0.0
path:
data: /bitnami/elasticsearch/data
xpack:
ml:
enabled: false
kind: ConfigMap
metadata:
labels:
app.kubernetes.io/instance: elasticsearch
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: elasticsearch
argocd.argoproj.io/instance: elasticsearch
helm.sh/chart: elasticsearch-19.5.5
name: elasticsearch
namespace: elasticsearch
kubectl logs elasticsearch-master-0
[2022-12-25T07:52:40,652][INFO ][o.e.c.r.a.AllocationService] [elasticsearch-master-0] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[.ds-ilm-history-5-2022.10.28-000014][0], [.ds-ilm-history-5-2022.09.28-000012][0]]]).
[2022-12-25T07:52:40,856][INFO ][o.e.i.g.GeoIpDownloader ] [elasticsearch-master-0] updating geoip database [GeoLite2-ASN.mmdb]
✗ curl -XGET http://localhost:9200/_cluster/health\?pretty\=true
{
"cluster_name" : "elasticsearch",
"status" : "yellow",
"timed_out" : false,
"number_of_nodes" : 1,
"number_of_data_nodes" : 1,
"active_primary_shards" : 7,
"active_shards" : 7,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 1,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 87.5
}
➜ curl -XGET http://localhost:9200/_cat/shards
magento2_product_1_v583 0 p STARTED 4868 18.9mb 10.110.4.229 elasticsearch-master-0
.ds-ilm-history-5-2022.10.28-000014 0 p STARTED 10.110.4.229 elasticsearch-master-0
.ds-ilm-history-5-2022.11.27-000015 0 p STARTED 10.110.4.229 elasticsearch-master-0
.ds-ilm-history-5-2022.08.29-000010 0 p STARTED 10.110.4.229 elasticsearch-master-0
.ds-ilm-history-5-2022.09.28-000012 0 p STARTED 10.110.4.229 elasticsearch-master-0
.geoip_databases 0 p STARTED 40 38.1mb 10.110.4.229 elasticsearch-master-0
.ds-.logs-deprecation.elasticsearch-default-2022.12.21-000022 0 p STARTED 10.110.4.229 elasticsearch-master-0
.ds-.logs-deprecation.elasticsearch-default-2022.12.21-000022 0 r UNASSIGNED
I'm trying to delete it but facing an error
➜ curl -XGET http://localhost:9200/_cat/shards | grep UNASSIGNED | awk {'print $1'} | xargs -i curl -XDELETE "http://localhost:9200/{}"
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 986 100 986 0 0 5241 0 --:--:-- --:--:-- --:--:-- 5244
{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"index [.ds-.logs-deprecation.elasticsearch-default-2022.12.21-000022] is the write index for data stream [.logs-deprecation.elasticsearch-default] and cannot be deleted"}],"type":"illegal_argument_exception","reason":"index [.ds-.logs-deprecation.elasticsearch-default-2022.12.21-000022] is the write index for data stream [.logs-deprecation.elasticsearch-default] and cannot be deleted"},"status":400}
GET /_cluster/allocation/explain:
➜ curl -XGET http://localhost:9200/_cluster/allocation/explain\?pretty\=true | jq
{
"note": "No shard was specified in the explain API request, so this response explains a randomly chosen unassigned shard. There may be other unassigned shards in this cluster which cannot be assigned for different reasons. It may not be possible to assign this shard until one of the other shards is assigned correctly. To explain the allocation of other shards (whether assigned or unassigned) you must specify the target shard in the request to this API.",
"index": ".ds-.logs-deprecation.elasticsearch-default-2022.12.21-000022",
"shard": 0,
"primary": false,
"current_state": "unassigned",
"unassigned_info": {
"reason": "CLUSTER_RECOVERED",
"at": "2022-12-25T07:52:37.022Z",
"last_allocation_status": "no_attempt"
},
"can_allocate": "no",
"allocate_explanation": "cannot allocate because allocation is not permitted to any of the nodes",
"node_allocation_decisions": [
{
"node_id": "aURccTcnSuqPC3fBfmezCg",
"node_name": "elasticsearch-master-0",
"transport_address": "10.110.4.229:9300",
"node_attributes": {
"xpack.installed": "true",
"transform.node": "true"
},
"node_decision": "no",
"deciders": [
{
"decider": "same_shard",
"decision": "NO",
"explanation": "a copy of this shard is already allocated to this node [[.ds-.logs-deprecation.elasticsearch-default-2022.12.21-000022][0], node[aURccTcnSuqPC3fBfmezCg], [P], s[STARTED], a[id=tsxhnODlSn-i__-vEvJj3A]]"
}
]
}
]
So what can be done in such scenario?
curl -v -XPUT "localhost:9200/*/_settings" -H 'Content-Type: application/json' -d '
{
"index" : {
"number_of_replicas" : 0
}
}
'
{"acknowledged":true}
curl -XGET http://localhost:9200/_cat/indices
green open magento2_product_1_v583 hvYpUxJUT16-g6_YS8qkaA 1 0 4868 0 18.9mb 18.9mb
green open .geoip_databases tDXBLQRdSFeQyi6Pk5zq2Q 1 0 40 40 38.1mb 38.1mb
Yellow status indicates that one or more of the replica shards on the Elasticsearch cluster are not allocated to a node. When you have only one node, it means your number of replicas is greater than your number of nodes. Elasticsearch will never assign a replica to the same node as the primary shard. so, if you only have one node it is perfectly normal and expected for your cluster to indicate yellow. But, if you are not convinced and want your cluster to be green, set the number of replicas to each index to be 0.
PUT /my-index/_settings
{
"index" : {
"number_of_replicas" : 0
}
}

Elasticsearch add node to cluster

I have 3 node elasticsearch cluster
192.168.2.11 - node-01
192.168.2.12 - node-02
192.168.2.13 - node-03
and i deleted node-02 from cluster using this command
curl -XPUT 192.168.2.12:9200/_cluster/settings -H 'Content-Type: application/json' -d '{
"transient" :{
"cluster.routing.allocation.exclude._ip" : "192.168.2.12"
}
}'
and ok, all my indexes moved to node-01 and node-03, but how to return back this node to the cluster?
i try this command
curl -XPUT 192.168.2.12:9200/_cluster/settings -H 'Content-Type: application/json' -d '{
"transient" :{
"cluster.routing.allocation.include._ip" : "192.168.2.12"
}
}'
but this doesn't works
:"node does not cluster setting [cluster.routing.allocation.include] filters [_ip:\"192.168.2.12\"]
The node has not been deleted but you can 'undo' your command by updating the setting you changed to null
Try updating the settings on either of the running nodes (01 or 03) with
"transient" :{
"cluster.routing.allocation.exclude._ip" : null
}
and the cluster should rebalance shards across the three nodes.
Be careful using the include._ip: "192.168.2.12" as this might stop routing indices to the other two, instead include all three ip addresses if you wanted to us this, for example
"transient" :{
"cluster.routing.allocation.include._ip" :"192.168.2.11, 192.168.2.12, 192.168.2.13"
}

Why does Elasticsearch allow me to index documents in a single node cluster that fails to meet quorum requirements?

From https://www.elastic.co/guide/en/elasticsearch/guide/2.x/distrib-write.html:
Note that the number_of_replicas is the number of replicas specified in the index settings, not the number of replicas that are currently active. If you have specified that an index should have three replicas, a quorum would be as follows:
int( (primary + 3 replicas) / 2 ) + 1 = 3
But if you start only two nodes, there will be insufficient active shard copies to satisfy the quorum, and you will be unable to index or delete any documents.
I ran the following commands on a single-node cluster and I was able to index a document successfully even though the math above says I should not be able to index documents.
curl -X DELETE http://localhost:9200/a/?pretty
curl -X PUT -siH 'Content-Type: application/json' \
http://localhost:9200/a?pretty -d '{
"settings": {
"number_of_replicas": 3
}
}'
curl -sH 'Content-Type: application/json' -X PUT http://localhost:9200/a/a/1?pretty -d '{"a": "a"}'
curl -si http://localhost:9200/_cluster/health?pretty
curl -si http://localhost:9200/a/a/1?pretty
Here is the output:
$ curl -X PUT -siH 'Content-Type: application/json' \
http://localhost:9200/a?pretty -d '{
"settings": {
"number_of_replicas": 3
}
}'
HTTP/1.1 200 OK
content-type: application/json; charset=UTF-8
content-length: 77
{
"acknowledged" : true,
"shards_acknowledged" : true,
"index" : "a"
}
$ curl -sH 'Content-Type: application/json' -X PUT http://localhost:9200/a/a/1?pretty -d '{"a": "a"}'
{
"_index" : "a",
"_type" : "a",
"_id" : "1",
"_version" : 1,
"result" : "created",
"_shards" : {
"total" : 4,
"successful" : 1,
"failed" : 0
},
"_seq_no" : 0,
"_primary_term" : 1
}
$ curl -si http://localhost:9200/_cluster/health?pretty
HTTP/1.1 200 OK
content-type: application/json; charset=UTF-8
content-length: 468
{
"cluster_name" : "docker-cluster",
"status" : "yellow",
"timed_out" : false,
"number_of_nodes" : 1,
"number_of_data_nodes" : 1,
"active_primary_shards" : 5,
"active_shards" : 5,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 15,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 25.0
}
$ curl -si http://localhost:9200/a/a/1?pretty
HTTP/1.1 200 OK
content-type: application/json; charset=UTF-8
content-length: 124
{
"_index" : "a",
"_type" : "a",
"_id" : "1",
"_version" : 1,
"found" : true,
"_source" : {
"a" : "a"
}
}
How was I able to index a document with just 1 node even though I have configured the index to have 3 replica shards and the math says I must have at least 3 nodes?
That was an old quorum rule in ES 2.x. As of ES 5.x the write concistency checks have been changed a little bit and a yellow cluster, i.e. a cluster will all primary shards allocated, will pass the consistency checks for write operations and allow you to index and delete documents.
Now the way to decide whether writes can be made is by using the wait_for_active_shards parameter in the indexing operation. By default, if all primary shards are up the index operation will be allowed. You can override that settings by specifying the number of shards that you wish to be active before an indexing operation should be authorized, e.g. wait_for_active_shards=all which is equivalent to wait_for_active_shards=4 (4 = 1 primary + 3 replicas) in your case. If you want the same quorum rule as before you'd specify wait_for_active_shards=3.
More info can be found in the official documentation and here

shard unassigned for elasticsearch cluster with docker

here is my compose file,escp is my docker images files
elasticsearch_master:
#image: elasticsearch:latest
image: escp
command: "elasticsearch \
-Des.cluster.name=dcluster \
-Des.node.name=esmaster \
-Des.node.master=true \
-Des.node.data=true \
-Des.node.client=false \
-Des.discovery.zen.minimum_master_nodes=1"
volumes:
- "${PWD}/es/config:/usr/share/elasticsearch/config"
- "${PWD}/esdata/node:/usr/share/elasticsearch/data"
- "${PWD}/es/plugins:/usr/share/elasticsearch/plugins"
environment:
- ES_HEAP_SIZE=512m
ports:
- "9200:9200"
- "9300:9300"
elasticsearch1:
#image: elasticsearch:latest
image: escp
command: "elasticsearch \
-Des.cluster.name=dcluster \
-Des.node.name=esnode1 \
-Des.node.data=true \
-Des.node.client=false \
-Des.node.master=false \
-Des.discovery.zen.minimum_master_nodes=1 \
-Des.discovery.zen.ping.unicast.hosts=elasticsearch_master"
links:
- elasticsearch_master
volumes:
- "${PWD}/es/config:/usr/share/elasticsearch/config"
- "${PWD}/esdata/node1:/usr/share/elasticsearch/data"
- "${PWD}/es/plugins:/usr/share/elasticsearch/plugins"
environment:
- ES_HEAP_SIZE=512m
elasticsearch2:
#image: elasticsearch:latest
image: escp
command: "elasticsearch \
-Des.cluster.name=dcluster \
-Des.node.name=esnode2 \
-Des.node.data=true \
-Des.node.client=false \
-Des.node.master=false \
-Des.discovery.zen.minimum_master_nodes=1 \
-Des.discovery.zen.ping.unicast.hosts=elasticsearch_master"
links:
- elasticsearch_master
volumes:
- "${PWD}/es/config:/usr/share/elasticsearch/config"
- "${PWD}/esdata/node2:/usr/share/elasticsearch/data"
- "${PWD}/es/plugins:/usr/share/elasticsearch/plugins"
environment:
- ES_HEAP_SIZE=512m
this is config file
index.number_of_shards: 1
index.number_of_replicas: 0
network.host: 0.0.0.0
after running
Name Command State Ports
--------------------------------------------------------------------------------------------------------------------
est_elasticsearch1_1 /docker-entrypoint.sh elas ... Up 9200/tcp, 9300/tcp
est_elasticsearch2_1 /docker-entrypoint.sh elas ... Up 9200/tcp, 9300/tcp
est_elasticsearch_master_1 /docker-entrypoint.sh elas ... Up 0.0.0.0:9200->9200/tcp, 0.0.0.0:9300->9300/tcp
but when i create new index there will show UNASSIGNED...
curl -s '192.168.99.100:9200/_cluster/health?pretty'
{
"cluster_name" : "dcluster",
"status" : "red",
"timed_out" : false,
"number_of_nodes" : 3,
"number_of_data_nodes" : 3,
"active_primary_shards" : 0,
"active_shards" : 0,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 1,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 0.0
}
check nodes
curl -s '192.168.99.100:9200/_cat/nodes?v'
host ip heap.percent ram.percent load node.role master name
172.17.0.2 172.17.0.2 13 33 0.00 d * esmaster
172.17.0.3 172.17.0.3 16 33 0.00 d - esnode1
172.17.0.4 172.17.0.4 13 33 0.00 d - esnode2
check shards
curl -s '192.168.99.100:9200/_cat/shards'
abcq 0 p UNASSIGNED
check allocation
curl -s '192.168.99.100:9200/_cat/allocation?v'
shards disk.indices disk.used disk.avail disk.total disk.percent host ip node
0 0b 223.4gb 9.5gb 232.9gb 95 172.17.0.4 172.17.0.4 esnode2
0 0b 223.4gb 9.5gb 232.9gb 95 172.17.0.2 172.17.0.2 esmaster
0 0b 223.4gb 9.5gb 232.9gb 95 172.17.0.3 172.17.0.3 esnode1
1 UNASSIGNED
check setting
curl 'http://192.168.99.100:9200/_cluster/settings?pretty'
{
"persistent" : { },
"transient" : { }
}
enabled reroute
curl 'http://192.168.99.100:9200/_cluster/settings?pretty'
{
"persistent" : { },
"transient" : {
"cluster" : {
"routing" : {
"allocation" : {
"enable" : "true"
}
}
}
}
}
reroute index abcq
curl -XPOST http://192.168.99.100:9200/_cluster/reroute?pretty -d '{
"commands" : [
{
"allocate" : {
"index" : "abcq",
"shard" : 0,
"node" : "esnode2",
"allow_primary" : true
}
}
]
}'
get error bellow
{
"error" : {
"root_cause" : [ {
"type" : "illegal_argument_exception",
"reason" : "[allocate] allocation of [abcq][0] on node {esnode2}{Pisl95VUSPmZa3Ga_e3sDA}{172.17.0.4}{172.17.0.4:9300}{master=false} is not allowed, reason: [YES(shard is primary)][YES(no allocation awareness enabled)][NO(more than allowed [90.0%] used disk on node, free: [4.078553722498398%])][YES(allocation disabling is ignored)][YES(primary shard can be allocated anywhere)][YES(node passes include/exclude/require filters)][YES(shard is not allocated to same node or host)][YES(total shard limit disabled: [index: -1, cluster: -1] <= 0)][YES(allocation disabling is ignored)][YES(no snapshots are currently running)][YES(below primary recovery limit of [4])]"
} ],
"type" : "illegal_argument_exception",
"reason" : "[allocate] allocation of [abcq][0] on node {esnode2}{Pisl95VUSPmZa3Ga_e3sDA}{172.17.0.4}{172.17.0.4:9300}{master=false} is not allowed, reason: [YES(shard is primary)][YES(no allocation awareness enabled)][NO(more than allowed [90.0%] used disk on node, free: [4.078553722498398%])][YES(allocation disabling is ignored)][YES(primary shard can be allocated anywhere)][YES(node passes include/exclude/require filters)][YES(shard is not allocated to same node or host)][YES(total shard limit disabled: [index: -1, cluster: -1] <= 0)][YES(allocation disabling is ignored)][YES(no snapshots are currently running)][YES(below primary recovery limit of [4])]"
},
"status" : 400
}
why i create new index get unassigned, can any one help? thanks.
guys here is how to fix problem.
more than allowed [90.0%] used disk on node
it means my disk total full, not too much space for shards allocation.
shards disk.indices disk.used disk.avail disk.total disk.percent host ip node
0 0b 223.4gb 9.5gb 232.9gb 95 172.17.0.4 172.17.0.4 esnode2
disable disk check or set it lower
curl -XPUT localhost:9200/_cluster/settings -d '{
"transient" : {
"cluster.routing.allocation.disk.threshold_enabled" : false
}
}'
curl -XPUT http://192.168.99.100:9200/_cluster/settings -d '
{
"transient" : {
"cluster.routing.allocation.disk.watermark.low": "10%",
"cluster.routing.allocation.disk.watermark.high": "10gb",
"cluster.info.update.interval": "1m"
}
}'
hope this can help ohters, more detail can check here.
https://www.elastic.co/guide/en/elasticsearch/reference/current/disk-allocator.html

Import Wikipedia's indices into Elasticsearch

For my research I should import Russian Wikipedia's dump into Elasticsearch 2.2. But instead of importing dump I decided to work with indices published by Wikimedia (http://dumps.wikimedia.org/other/cirrussearch/). To work with it I found an article https://www.elastic.co/blog/loading-wikipedia and tried to use author's scripts for my problem (just replaced some export-statements). But there's a problem in the Step 2.
It is my version of script for Step 2:
export es=localhost:9200
export site=ru.wikipedia.org
export index=ruwiki
curl -XDELETE $es/$index?pretty
curl -s 'https://'$site'/w/api.php?action=cirrus-settings-dump&format=json&formatversion=2' |
jq '{ analysis: .content.page.index.analysis, number_of_shards: 1, number_of_replicas: 0 }' |
curl -XPUT $es/$index?pretty -d #-
curl -s 'https://'$site'/w/api.php?action=cirrus-mapping-dump&format=json&formatversion=2' |
jq .content |
sed 's/"index_analyzer"/"analyzer"/' |
sed 's/"position_offset_gap"/"position_increment_gap"/' |
curl -XPUT $es/$index/_mapping/page?pretty -d #-
And the result
{
"acknowledged" : true
}
{
"acknowledged" : true
}
{
"error" : {
"root_cause" : [ {
"type" : "action_request_validation_exception",
"reason" : "Validation Failed: 1: mapping source is empty;"
} ],
"type" : "action_request_validation_exception",
"reason" : "Validation Failed: 1: mapping source is empty;"
},
"status" : 400
}
I also tried to use author's script just for test. There's the same error. I don't know what to do. Please, help to fix it.
The Wikipedia dumps are currently exported from ElasticSearch 1.7.5. Most likely (I haven't tested) the current mapping is not compatible with ES 2.2. It is likely worthwhile to try using the older version of ES.
Edit: The latest dumps are now compatible with elasticsearch 2.x

Resources