I have a cluster with one node (by local). Health cluster is yellow. Now I add more one node, but shards can not be allocated in second node. So the health of my cluster is still yellow. I can not change this state to green, not like as this guide:health cluster example.
So how to change health state to green?
My cluster:
Cluster health:
curl -XGET 'http://localhost:9200/_cluster/health?pretty=true'
{
"cluster_name" : "astrung",
"status" : "yellow",
"timed_out" : false,
"number_of_nodes" : 2,
"number_of_data_nodes" : 2,
"active_primary_shards" : 22,
"active_shards" : 22,
"relocating_shards" : 0,
"initializing_shards" : 2,
"unassigned_shards" : 20
}
Shard status:
curl -XGET 'http://localhost:9200/_cat/shards?v'
index shard prirep state docs store ip node
_river 0 p STARTED 2 8.1kb 192.168.1.3 One
_river 0 r UNASSIGNED
megacorp 4 p STARTED 1 3.4kb 192.168.1.3 One
megacorp 4 r UNASSIGNED
megacorp 0 p STARTED 2 6.1kb 192.168.1.3 One
megacorp 0 r UNASSIGNED
megacorp 3 p STARTED 1 2.2kb 192.168.1.3 One
megacorp 3 r UNASSIGNED
megacorp 1 p STARTED 0 115b 192.168.1.3 One
megacorp 1 r UNASSIGNED
megacorp 2 p STARTED 1 2.2kb 192.168.1.3 One
megacorp 2 r UNASSIGNED
mybucket 2 p STARTED 1 2.1kb 192.168.1.3 One
mybucket 2 r UNASSIGNED
mybucket 0 p STARTED 0 115b 192.168.1.3 One
mybucket 0 r UNASSIGNED
mybucket 3 p STARTED 2 5.4kb 192.168.1.3 One
mybucket 3 r UNASSIGNED
mybucket 1 p STARTED 1 2.2kb 192.168.1.3 One
mybucket 1 r UNASSIGNED
mybucket 4 p STARTED 1 2.5kb 192.168.1.3 One
mybucket 4 r UNASSIGNED
.kibana 0 r INITIALIZING 192.168.1.3 Two
.kibana 0 p STARTED 2 8.9kb 192.168.1.3 One
.marvel-kibana 2 p STARTED 0 115b 192.168.1.3 One
.marvel-kibana 2 r UNASSIGNED
.marvel-kibana 0 r INITIALIZING 192.168.1.3 Two
.marvel-kibana 0 p STARTED 1 2.9kb 192.168.1.3 One
.marvel-kibana 3 p STARTED 0 115b 192.168.1.3 One
.marvel-kibana 3 r UNASSIGNED
.marvel-kibana 1 p STARTED 0 115b 192.168.1.3 One
.marvel-kibana 1 r UNASSIGNED
.marvel-kibana 4 p STARTED 0 115b 192.168.1.3 One
.marvel-kibana 4 r UNASSIGNED
user_ids 4 p STARTED 11 5kb 192.168.1.3 One
user_ids 4 r UNASSIGNED
user_ids 0 p STARTED 7 25.1kb 192.168.1.3 One
user_ids 0 r UNASSIGNED
user_ids 3 p STARTED 11 4.9kb 192.168.1.3 One
user_ids 3 r UNASSIGNED
user_ids 1 p STARTED 8 28.7kb 192.168.1.3 One
user_ids 1 r UNASSIGNED
user_ids 2 p STARTED 11 8.5kb 192.168.1.3 One
user_ids 2 r UNASSIGNED
I suggest updating the replication factor of all the indices to 0 and then update it back to 1.
Here's a curl to get you started
curl -XPUT 'http://localhost:9200/_settings' -H 'Content-Type: application/json' -d '
{
"index" : {
"number_of_replicas" : 0
}
}'
like #mohitt said above, update number_of_replicas to zero(for local dev only,be careful to use in production)
you can run the below in Kibana DevTool Console:
PUT _settings
{
"index" : {
"number_of_replicas" : 0
}
}
thou recovery normally takes a long time, looking at the number and size of your documents, it should take a very sort time to recover.
Looks like you have issues with the nodes contacting each other, check firewall rules, ensure ports 9200 and 9300 are reachable from each.
Related
We are looking into addressing some performance issues with our ES cluster. We were looking into shard distribution on data nodes. I know that there is an advice to have shards evenly distributed between nodes - and here is my question:
For the cluster with 8 data nodes - we have some indexes that have 2 primary shards and 3 replicas (so 8 shards in total). We have also some indexes that have 1 primary shard and 3 replicas (so 4 in total).
My question is - is that setup can be consider "evenly distributed"? We were thinking that it is not and we were thinking about having indexes with 1 primary shard - 7 replicas (so every index will be hosted on 8 nodes) - but we don't know if such setup has any sense? If not - what would you recomend instead to distirbute shards more evenly?
Here is the result of shard cat query:
id1 0 p STARTED 2138 16.1mb x.x.x.x node1
id1 0 r STARTED 2138 16.1mb x.x.x.x node2
id1 0 r STARTED 2138 16.1mb x.x.x.x node3
id1 0 r STARTED 2138 16.1mb x.x.x.x node4
id2 0 r STARTED 3379 26.8mb x.x.x.x node5
id2 0 r STARTED 3379 26.8mb x.x.x.x node3
id2 0 r STARTED 3379 26.8mb x.x.x.x node4
id2 0 p STARTED 3379 26.8mb x.x.x.x node6
id3 0 r STARTED 20086 76.1mb x.x.x.x node1
id3 0 r STARTED 20086 76.1mb x.x.x.x node5
id3 0 p STARTED 20086 76.1mb x.x.x.x node6
id3 0 r STARTED 20086 76.1mb x.x.x.x node7
id4 0 r STARTED 2754 7.3mb x.x.x.x node2
id4 0 r STARTED 2754 7.3mb x.x.x.x node3
id4 0 r STARTED 2754 7.3mb x.x.x.x node8
id4 0 p STARTED 2754 7.3mb x.x.x.x node7
id5 0 r STARTED 10239 42.3mb x.x.x.x node1
id5 0 p STARTED 10239 42.3mb x.x.x.x node4
id5 0 r STARTED 10239 42.3mb x.x.x.x node6
id5 0 r STARTED 10239 42.3mb x.x.x.x node8
id6 0 r STARTED 13388 42.4mb x.x.x.x node1
id6 0 p STARTED 13388 42.4mb x.x.x.x node5
id6 0 r STARTED 13388 42.4mb x.x.x.x node3
id6 0 r STARTED 13388 42.4mb x.x.x.x node8
id7 1 r STARTED 27483 136.2mb x.x.x.x node2
id7 1 r STARTED 27483 136.2mb x.x.x.x node3
id7 1 r STARTED 27483 136.3mb x.x.x.x node8
id7 1 p STARTED 27483 136.2mb x.x.x.x node7
id7 0 r STARTED 27189 146.5mb x.x.x.x node1
id7 0 p STARTED 27189 146.6mb x.x.x.x node5
id7 0 r STARTED 27189 146.6mb x.x.x.x node4
id7 0 r STARTED 27189 146.7mb x.x.x.x node6
.kibana 0 r STARTED 13 106.8kb x.x.x.x node2
.kibana 0 p STARTED 13 106.8kb x.x.x.x node3
id8 1 r STARTED 13555 80.8mb x.x.x.x node2
id8 1 r STARTED 13555 80.8mb x.x.x.x node4
id8 1 r STARTED 13555 80.8mb x.x.x.x node8
id8 1 p STARTED 13555 80.8mb x.x.x.x node7
id8 0 r STARTED 13390 63mb x.x.x.x node1
id8 0 p STARTED 13390 62.7mb x.x.x.x node5
id8 0 r STARTED 13390 62.7mb x.x.x.x node6
id8 0 r STARTED 13390 62.8mb x.x.x.x node7
Distributing all shards on all ES data nodes for every index doesn't make sense for various reasons.
Number of primary shards should be chosen based on their size and helps you to horizontally scale the index.
Number of replica shards helps you in high availability and increasing search performance.
It's really difficult to achieve the perfect shards balance in the ES cluster(based on a number of shards, size, and traffic), Although based on your shards size which is really small(lesser than 100MB), you can go with 1 shard and 7 replicas for your all indices, having said this you need to benchmark and choose the correct number of shards and replica based on your cluster setup and use-cases.
I closed my index and reopen it and now my shards can't assigne.
curl -s -XGET localhost:9201/_cat/shards?h=index,shard,prirep,state,unassigned.reason | grep UNASSIGNED
2018.03.27-team-logs 2 r UNASSIGNED INDEX_REOPENED
2018.03.27-team-logs 5 r UNASSIGNED INDEX_REOPENED
2018.03.27-team-logs 3 r UNASSIGNED INDEX_REOPENED
2018.03.27-team-logs 4 r UNASSIGNED INDEX_REOPENED
2018.03.27-team-logs 1 r UNASSIGNED INDEX_REOPENED
2018.03.27-team-logs 0 r UNASSIGNED INDEX_REOPENED
2018.03.28-team-logs 2 r UNASSIGNED INDEX_REOPENED
2018.03.28-team-logs 5 r UNASSIGNED INDEX_REOPENED
2018.03.28-team-logs 3 r UNASSIGNED INDEX_REOPENED
2018.03.28-team-logs 4 r UNASSIGNED INDEX_REOPENED
2018.03.28-team-logs 1 r UNASSIGNED INDEX_REOPENED
2018.03.28-team-logs 0 r UNASSIGNED INDEX_REOPENED
Could anybody explain me what does error means and how to solve it? Before I closed it everything works fine. I configured 6 shards and 1 replica. Installed Elasticsearch 6.2.
EDIT:
Output of curl -XGET "localhost:9201/_cat/shards":
2018.03.29-team-logs 1 r STARTED 1739969 206.2mb 10.207.46.247 elk-es-data-hot-1.platform.osdc2.mall.local
2018.03.29-team-logs 1 p STARTED 1739969 173mb 10.206.46.246 elk-es-data-hot-2.platform.osdc1.mall.local
2018.03.29-team-logs 2 p STARTED 1739414 169.9mb 10.207.46.247 elk-es-data-hot-1.platform.osdc2.mall.local
2018.03.29-team-logs 2 r STARTED 1739414 176.3mb 10.207.46.248 elk-es-data-hot-2.platform.osdc2.mall.local
2018.03.29-team-logs 4 p STARTED 1740185 186mb 10.206.46.247 elk-es-data-hot-1.platform.osdc1.mall.local
2018.03.29-team-logs 4 r STARTED 1740185 169.4mb 10.206.46.246 elk-es-data-hot-2.platform.osdc1.mall.local
2018.03.29-team-logs 5 r STARTED 1739660 164.3mb 10.207.46.248 elk-es-data-hot-2.platform.osdc2.mall.local
2018.03.29-team-logs 5 p STARTED 1739660 180.1mb 10.206.46.246 elk-es-data-hot-2.platform.osdc1.mall.local
2018.03.29-team-logs 3 p STARTED 1740606 171.2mb 10.207.46.248 elk-es-data-hot-2.platform.osdc2.mall.local
2018.03.29-team-logs 3 r STARTED 1740606 173.4mb 10.206.46.247 elk-es-data-hot-1.platform.osdc1.mall.local
2018.03.29-team-logs 0 r STARTED 1740166 169.7mb 10.207.46.247 elk-es-data-hot-1.platform.osdc2.mall.local
2018.03.29-team-logs 0 p STARTED 1740166 187mb 10.206.46.247 elk-es-data-hot-1.platform.osdc1.mall.local
2018.03.28-team-logs 1 p STARTED 2075020 194.2mb 10.207.46.248 elk-es-data-hot-2.platform.osdc2.mall.local
2018.03.28-team-logs 1 r UNASSIGNED
2018.03.28-team-logs 2 p STARTED 2076268 194.9mb 10.206.46.247 elk-es-data-hot-1.platform.osdc1.mall.local
2018.03.28-team-logs 2 r UNASSIGNED
2018.03.28-team-logs 4 p STARTED 2073906 194.9mb 10.207.46.247 elk-es-data-hot-1.platform.osdc2.mall.local
2018.03.28-team-logs 4 r UNASSIGNED
2018.03.28-team-logs 5 p STARTED 2072921 195mb 10.207.46.248 elk-es-data-hot-2.platform.osdc2.mall.local
2018.03.28-team-logs 5 r UNASSIGNED
2018.03.28-team-logs 3 p STARTED 2074579 194.1mb 10.206.46.246 elk-es-data-hot-2.platform.osdc1.mall.local
2018.03.28-team-logs 3 r UNASSIGNED
2018.03.28-team-logs 0 p STARTED 2073349 193.9mb 10.207.46.248 elk-es-data-hot-2.platform.osdc2.mall.local
2018.03.28-team-logs 0 r UNASSIGNED
2018.03.27-team-logs 1 p STARTED 356769 33.5mb 10.207.46.246 elk-es-data-warm-1.platform.osdc2.mall.local
2018.03.27-team-logs 1 r UNASSIGNED
2018.03.27-team-logs 2 p STARTED 356798 33.6mb 10.206.46.244 elk-es-data-warm-2.platform.osdc1.mall.local
2018.03.27-team-logs 2 r UNASSIGNED
2018.03.27-team-logs 4 p STARTED 356747 33.7mb 10.207.46.246 elk-es-data-warm-1.platform.osdc2.mall.local
2018.03.27-team-logs 4 r UNASSIGNED
2018.03.27-team-logs 5 p STARTED 357399 33.8mb 10.207.46.245 elk-es-data-warm-2.platform.osdc2.mall.local
2018.03.27-team-logs 5 r UNASSIGNED
2018.03.27-team-logs 3 p STARTED 357957 33.7mb 10.206.46.245 elk-es-data-warm-1.platform.osdc1.mall.local
2018.03.27-team-logs 3 r UNASSIGNED
2018.03.27-team-logs 0 p STARTED 356357 33.4mb 10.207.46.245 elk-es-data-warm-2.platform.osdc2.mall.local
2018.03.27-team-logs 0 r UNASSIGNED
.kibana 0 p STARTED 2 12.3kb 10.207.46.247 elk-es-data-hot-1.platform.osdc2.mall.local
.kibana 0 r UNASSIGNED
Output of curl -XGET "localhost:9201/_cat/nodes":
10.207.46.248 8 82 0 0.07 0.08 0.11 d - elk-es-data-hot-2
10.206.46.245 9 64 0 0.04 0.11 0.08 d - elk-es-data-warm-1
10.207.46.249 11 90 0 0.00 0.01 0.05 m * elk-es-master-2
10.207.46.245 9 64 0 0.00 0.01 0.05 d - elk-es-data-warm-2
10.206.46.247 12 82 0 0.00 0.06 0.08 d - elk-es-data-hot-1
10.206.46.244 10 64 0 0.08 0.04 0.05 d - elk-es-data-warm-2
10.207.46.243 5 86 0 0.00 0.01 0.05 d - elk-kibana
10.206.46.248 10 92 1 0.04 0.18 0.24 m - elk-es-master-1
10.206.46.246 6 82 0 0.02 0.07 0.09 d - elk-es-data-hot-2
10.207.46.247 9 82 0 0.06 0.06 0.05 d - elk-es-data-hot-1
10.206.46.241 6 91 0 0.00 0.02 0.05 m - master-test
10.206.46.242 8 89 0 0.00 0.02 0.05 d - es-kibana
10.207.46.246 8 64 0 0.00 0.02 0.05 d - elk-es-data-warm-1
It is expected behaviour.
Elasticsearch will not put primary and replica shard on the same
node. You will need at least 2 nodes to have have 1 replica.
You can simply set replica to 0
PUT */_settings
{
"index" : {
"number_of_replicas" : 0
}
}
UPDATE:
After running following request
GET /_cluster/allocation/explain?pretty
we can see response here
https://pastebin.com/1ag1Z7jL
"explanation" : "there are too many copies of the shard allocated to
nodes with attribute [datacenter], there are [2] total configured
shard copies for this shard id and [3] total attribute values,
expected the allocated shard count per attribute [2] to be less than
or equal to the upper bound of the required number of shards per
attribute [1]"
Probbably you have zone setting used. Elasticsearch will avoid to put primary and replica shard in same zone
https://www.elastic.co/guide/en/elasticsearch/reference/current/allocation-awareness.html
With ordinary awareness, if one zone lost contact with the other zone,
Elasticsearch would assign all of the missing replica shards to a
single zone. But in this example, this sudden extra load would cause
the hardware in the remaining zone to be overloaded.
Forced awareness solves this problem by NEVER allowing copies of the
same shard to be allocated to the same zone.
For example, lets say we have an awareness attribute called zone, and
we know we are going to have two zones, zone1 and zone2. Here is how
we can force awareness on a node:
cluster.routing.allocation.awareness.force.zone.values: zone1,zone2
cluster.routing.allocation.awareness.attributes: zone
I have three nodes working together. On those nodes I have five indexes, each having 5 primary shards and each primary shard has 2 replicas. It looks like this (i cut to see only two indices out of 5 ):
![IMG]http://i59.tinypic.com/2ez1wjt.png
As you can see on the picture:
- the node 1 has primary shards 0 and 3 (and replicas 1,2 and 4)
- the node 2 has primary shards 2 (and replicas 0, 1, 3 and 4)
- the node 3 has primary shards 1 and 4 (and replicas 0 ,2 and 3)
and this is the case for each index (the 5 of them).
I understand that if I restart my nodes this "organisation" will change but still the "look" of index will be the same as index2, 3, 4, 5. For example, after restarting, I would have:
- the node 1 has primary shards 1 and 2 (and replicas 0, 3 and 4)
- the node 2 has primary shards 3 (and replicas 0, 1, 2 and 4)
- the node 3 has primary shards 0 and 4 (and replicas 1 ,2 and 3)
and this would be the case for each index (the 5 of them).
Is there a reason why I find always the same pattern for each of my index?
Thanks
I'm using Elasticsearch on my dedicated server (not amazon). Recently it's giving me error like:
UnavailableShardsException[[tribune][4] Primary shard is not active or
isn't assigned is a known node. Timeout: [1m], request: delete
{[tribune][news][90755]}]
when ever I'm making /_cat/shards?v the result is:
index shard prirep state docs store ip node
tribune 4 p UNASSIGNED
tribune 4 r UNASSIGNED
tribune 0 p STARTED 5971 34mb ***.**.***.** Benny Beckley
tribune 0 r UNASSIGNED
tribune 3 p STARTED 5875 33.9mb ***.**.***.** Benny Beckley
tribune 3 r UNASSIGNED
tribune 1 p INITIALIZING ***.**.***.** Benny Beckley
tribune 1 r UNASSIGNED
tribune 2 p STARTED 5875 33.6mb ***.**.***.** Benny Beckley
tribune 2 r UNASSIGNED
What should happen with the final exclusive scan value in a stream compaction algorithm?
This is an example to pick out all the 'A' characters.
Sequence A:
Input: A B B A A B B A
Selection: 1 0 0 1 1 0 0 1
Scan: 0 1 1 1 2 3 3 3
0 - A
1 - A
2 - A
3 - A
Sequence B (same except the last value):
Input: A B B A A B B B
Selection: 1 0 0 1 1 0 0 0
Scan: 0 1 1 1 2 3 3 3
0 - A
1 - A
2 - A
3 - B
Clearly the second example gives the wrong final result based on doing a naive loop through the scan values writing into these addresses.
What am I missing here?
Update:
As I understand the scan algorithm, I would do the equivalent of the following:
for (int i = 0; i < scan.length(); i++)
{
result[scan[i]] = input[i];
}
In parallel this would involve a scatter instruction.
After an A, you are asuming that there will be at least another A. Therefore, you asume that the sequence ends with an A. If it doesn't, you pick the wrong final letter.
You just need to count the As. Don't start with 1. Start with 0. Only increase this count when you find an A.
Or... Update:
Input: A B B A A B B A
Selection: 1 0 0 1 1 0 0 1
Scan: 0 1 1 1 2 3 3 3 4
^
0 - A |
1 - A Four elements
2 - A
3 - A
Input: A B B A A B B B
Selection: 1 0 0 1 1 0 0 0
Scan: 0 1 1 1 2 3 3 3 3
^
0 - A |
1 - A Three elements
2 - A