shard unassigned for elasticsearch cluster with docker - elasticsearch

Question

here is my compose file,escp is my docker images files
elasticsearch_master:
#image: elasticsearch:latest
image: escp
command: "elasticsearch \
-Des.cluster.name=dcluster \
-Des.node.name=esmaster \
-Des.node.master=true \
-Des.node.data=true \
-Des.node.client=false \
-Des.discovery.zen.minimum_master_nodes=1"
volumes:
- "${PWD}/es/config:/usr/share/elasticsearch/config"
- "${PWD}/esdata/node:/usr/share/elasticsearch/data"
- "${PWD}/es/plugins:/usr/share/elasticsearch/plugins"
environment:
- ES_HEAP_SIZE=512m
ports:
- "9200:9200"
- "9300:9300"
elasticsearch1:
#image: elasticsearch:latest
image: escp
command: "elasticsearch \
-Des.cluster.name=dcluster \
-Des.node.name=esnode1 \
-Des.node.data=true \
-Des.node.client=false \
-Des.node.master=false \
-Des.discovery.zen.minimum_master_nodes=1 \
-Des.discovery.zen.ping.unicast.hosts=elasticsearch_master"
links:
- elasticsearch_master
volumes:
- "${PWD}/es/config:/usr/share/elasticsearch/config"
- "${PWD}/esdata/node1:/usr/share/elasticsearch/data"
- "${PWD}/es/plugins:/usr/share/elasticsearch/plugins"
environment:
- ES_HEAP_SIZE=512m
elasticsearch2:
#image: elasticsearch:latest
image: escp
command: "elasticsearch \
-Des.cluster.name=dcluster \
-Des.node.name=esnode2 \
-Des.node.data=true \
-Des.node.client=false \
-Des.node.master=false \
-Des.discovery.zen.minimum_master_nodes=1 \
-Des.discovery.zen.ping.unicast.hosts=elasticsearch_master"
links:
- elasticsearch_master
volumes:
- "${PWD}/es/config:/usr/share/elasticsearch/config"
- "${PWD}/esdata/node2:/usr/share/elasticsearch/data"
- "${PWD}/es/plugins:/usr/share/elasticsearch/plugins"
environment:
- ES_HEAP_SIZE=512m
this is config file
index.number_of_shards: 1
index.number_of_replicas: 0
network.host: 0.0.0.0
after running
Name Command State Ports
--------------------------------------------------------------------------------------------------------------------
est_elasticsearch1_1 /docker-entrypoint.sh elas ... Up 9200/tcp, 9300/tcp
est_elasticsearch2_1 /docker-entrypoint.sh elas ... Up 9200/tcp, 9300/tcp
est_elasticsearch_master_1 /docker-entrypoint.sh elas ... Up 0.0.0.0:9200->9200/tcp, 0.0.0.0:9300->9300/tcp
but when i create new index there will show UNASSIGNED...
curl -s '192.168.99.100:9200/_cluster/health?pretty'
{
"cluster_name" : "dcluster",
"status" : "red",
"timed_out" : false,
"number_of_nodes" : 3,
"number_of_data_nodes" : 3,
"active_primary_shards" : 0,
"active_shards" : 0,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 1,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 0.0
}
check nodes
curl -s '192.168.99.100:9200/_cat/nodes?v'
host ip heap.percent ram.percent load node.role master name
172.17.0.2 172.17.0.2 13 33 0.00 d * esmaster
172.17.0.3 172.17.0.3 16 33 0.00 d - esnode1
172.17.0.4 172.17.0.4 13 33 0.00 d - esnode2
check shards
curl -s '192.168.99.100:9200/_cat/shards'
abcq 0 p UNASSIGNED
check allocation
curl -s '192.168.99.100:9200/_cat/allocation?v'
shards disk.indices disk.used disk.avail disk.total disk.percent host ip node
0 0b 223.4gb 9.5gb 232.9gb 95 172.17.0.4 172.17.0.4 esnode2
0 0b 223.4gb 9.5gb 232.9gb 95 172.17.0.2 172.17.0.2 esmaster
0 0b 223.4gb 9.5gb 232.9gb 95 172.17.0.3 172.17.0.3 esnode1
1 UNASSIGNED
check setting
curl 'http://192.168.99.100:9200/_cluster/settings?pretty'
{
"persistent" : { },
"transient" : { }
}
enabled reroute
curl 'http://192.168.99.100:9200/_cluster/settings?pretty'
{
"persistent" : { },
"transient" : {
"cluster" : {
"routing" : {
"allocation" : {
"enable" : "true"
}
}
}
}
}
reroute index abcq
curl -XPOST http://192.168.99.100:9200/_cluster/reroute?pretty -d '{
"commands" : [
{
"allocate" : {
"index" : "abcq",
"shard" : 0,
"node" : "esnode2",
"allow_primary" : true
}
}
]
}'
get error bellow
{
"error" : {
"root_cause" : [ {
"type" : "illegal_argument_exception",
"reason" : "[allocate] allocation of [abcq][0] on node {esnode2}{Pisl95VUSPmZa3Ga_e3sDA}{172.17.0.4}{172.17.0.4:9300}{master=false} is not allowed, reason: [YES(shard is primary)][YES(no allocation awareness enabled)][NO(more than allowed [90.0%] used disk on node, free: [4.078553722498398%])][YES(allocation disabling is ignored)][YES(primary shard can be allocated anywhere)][YES(node passes include/exclude/require filters)][YES(shard is not allocated to same node or host)][YES(total shard limit disabled: [index: -1, cluster: -1] <= 0)][YES(allocation disabling is ignored)][YES(no snapshots are currently running)][YES(below primary recovery limit of [4])]"
} ],
"type" : "illegal_argument_exception",
"reason" : "[allocate] allocation of [abcq][0] on node {esnode2}{Pisl95VUSPmZa3Ga_e3sDA}{172.17.0.4}{172.17.0.4:9300}{master=false} is not allowed, reason: [YES(shard is primary)][YES(no allocation awareness enabled)][NO(more than allowed [90.0%] used disk on node, free: [4.078553722498398%])][YES(allocation disabling is ignored)][YES(primary shard can be allocated anywhere)][YES(node passes include/exclude/require filters)][YES(shard is not allocated to same node or host)][YES(total shard limit disabled: [index: -1, cluster: -1] <= 0)][YES(allocation disabling is ignored)][YES(no snapshots are currently running)][YES(below primary recovery limit of [4])]"
},
"status" : 400
}
why i create new index get unassigned, can any one help? thanks.

Answer 1

guys here is how to fix problem.
more than allowed [90.0%] used disk on node
it means my disk total full, not too much space for shards allocation.
shards disk.indices disk.used disk.avail disk.total disk.percent host ip node
0 0b 223.4gb 9.5gb 232.9gb 95 172.17.0.4 172.17.0.4 esnode2
disable disk check or set it lower
curl -XPUT localhost:9200/_cluster/settings -d '{
"transient" : {
"cluster.routing.allocation.disk.threshold_enabled" : false
}
}'
curl -XPUT http://192.168.99.100:9200/_cluster/settings -d '
{
"transient" : {
"cluster.routing.allocation.disk.watermark.low": "10%",
"cluster.routing.allocation.disk.watermark.high": "10gb",
"cluster.info.update.interval": "1m"
}
}'
hope this can help ohters, more detail can check here.
https://www.elastic.co/guide/en/elasticsearch/reference/current/disk-allocator.html