I have an issue with the shards in Elastic Search. Whenever I create my index it creates an unassigned node with unassigned shards as shown below.
Can anyone please help me out here? Any help is much appreciated.
The reason is simply because you have only one node and one shard replica is always created by default.
In order to prevent this, you can remove the replicas from your index by calling this:
PUT suppliercloudproductindex1/_settings
{
"index.number_of_replicas": 0
}
Alternately, you can also specify this setting when initially creating your index in order to make sure that no replica shard gets created:
PUT suppliercloudproductindex1
{
"settings": {
"index.number_of_replicas": 0
}
}
PUT */_settings { "index.number_of_replicas": 0 }
This will update all indexes to have zero replicas. This will only work if wild card operations are allowed. Otherwise the command has to be run on every index in your cluster.
Related
What do I have:
Elasticsearch (7.7.0) cluster (amazon/opendistro-for-elasticsearch:1.8.0 Docker image)
one master node
one coordinating node
two data nodes with node.attr.data=hot
one data node with node.attr.data=warm
What do I want: prevent shard allocation and relocation from hot data nodes to warm (and cold in future) data nodes.
What have I tried:
I've put "index.routing.allocation.require.data": "hot" for all index templates, so newly created indices won't be allocated to any but hot data nodes. This works fine.
Anyway, I can't restrict shards relocation from hot nodes to warm. At the moment I'm using "cluster.routing.allocation.exclude._ip" : "warm data node ip" to prevent relocation from hot data nodes to the warm data node. But will I be able to use ILM with this filter?
I've also tried to
PUT _cluster/settings
{
"transient": {
"cluster.routing.allocation.awareness.attributes": ["data"]
}
}
PUT _cluster/settings
{
"transient": {
"cluster.routing.allocation.awareness.force.data.values": ["hot"]
}
}
and then remove the "cluster.routing.allocation.exclude._ip" filter. Shards were relocating from hot data nodes to the warm data node anyway. What am I missing?
I think you miss the allocation awareness in your ILM Policy.
Your warm phase definition should have
"allocate": {
"require": {
"data": "warm"
}
}
Remove this from your definition and it should resolve your issue.
The best article to fully inderstand the ILM for me is
https://www.elastic.co/blog/implementing-hot-warm-cold-in-elasticsearch-with-index-lifecycle-management
You will find a full example in "Optimizing your ILM policy for hot-warm-cold"
I had to update settings for my old incides:
PUT my-index/_settings
{
"index.routing.allocation.require.data": "hot"
}
Indices which are under certain index template won't be updated if the index template is updated.
I have a newly setup Elasticsearch 7.5.2 cluster. When I create an index, only one shard is created default for it.
My cluster strategy is as below:
Total Nodes: 5
--------------
Node 1 & Node 2 - Master Only
Node 3 - Master & Data Node
Node 4 & Node 5 - Data Only
Could not find any cluster setting that is restricting the shards for index creation.
Is the issue with cluster strategy or am I missing any settings here?.
Please help me to find the the issue.
Earlier Elasticsearch had default number of primary shards to 5, which is changed from Elasticsearch 7.X, which you are using, hence you are seeing just 1 primary shard.
Elasticsearch link for this change and more info on this SO answer.
Apart from API which is applicable on a particular index, which #Kamal already mentioned, you can specify this setting in your elasticsearch.yml, which would be effective on every index created until you override using the API call.
Config to add in your elasticsearch.yml
index.number_of_shards: {your desired number of shards}
Note: This is for primary shards that can't be changed dynamically, so be cautious of setting this, Unlike the number of replicas which can be changed dynamically.
That is correct. Post version 7, Elasticsearch by default creates index with shard size 1 as mentioned here
You can always specify the index shard using the below settings, while creating the index.
PUT <your_index_name>
{
"settings" : {
"index" : {
"number_of_shards" : 5
}
}
}
Hope this helps!
I am new to elastic search. I have installed elastic search in my local machine. when I add any document, the result shards total is 2, when I see all indices through _cat API primary and replica are showing as 1 only. But by default shards should be 5, but its showing only 1 for me. for every index default shards are 1 only, I didn't changed any configuration.
So starting with Elasticsearch version 7.0 the default number of shards was reduced from 5 to 1.
You can see the difference by comparing the version 6.8 and version 7.0 of the documentation.
If you still want to have 5 shards created for your index, you have to create it like the following:
PUT my_index
{
"settings": {
"number_of_shards": 5,
"number_of_replicas": 1
}
}
The resulting shards after indexing a document is two because you have also one replica. Since you operate a single-node-cluster the replica shard can not be allocated onto another node and therefore only one shard succeeded.
I know that with below config we can exclude some nodes from elastic cluster, And elastic itself relocate existing indexes on those nodes.
PUT /_cluster/settings
{
"transient" : {
"cluster.routing.allocation.exclude._ip" : "192.168.2.*"
}
}
But what I really want is to exclude some indexes from particular nodes, I tried this config
PUT test/_settings
{
"index.routing.allocation.exclude._ip": "192.168.2.*"
}
This config prohibit elastic to assign new shards to this nodes, but it seems that it does not make elastic to relocate index's shards from those node. Am I right? If I'm right how can I move existing index from particular node?
I know I can reroute shards manually but there are many shards and it is almost impossible! _reindex is another option but it takes even more!
If it matters I use elastic 2.3.5
Ok, The answer is that that config will make elastic to move indexes from excluded nodes, But elastic do it when cluster is green!
i have a cluster with single node in it..
i had created an index A with default shard size(i.e. in elasticsearch.yml file the value of index.number_of_shards: 1). When i listed all my shards, i could see single shard for index A. After this i changed the value of index.number_of_shards: 4 in elasticsearch.yml and then created another index B. again when i listed all my shards in the cluster i could only see single shard created for index B instead of 4 shards.
Does Elasticsearch check for the total number of nodes present in a cluster before creating the index and assigning the shards(in my case i had not specified any no of shards while creating the index B, so i was expecting total of 4 shards to be created for my index). can you help me with this?
The index configuration on elasticsearch.yml is being deprecated in favor of passing this configuration on the index settings/mapping instead.
So what you have to do is to remove index configurations from the elasticsearch.yml file and pass them using index settings or template. From elastic docs:
curl -XPUT 'http://localhost:9200/twitter/' -d '{
"settings" : {
"index" : {
"number_of_shards" : 3,
"number_of_replicas" : 2
}
}
Using this approach you can create different configurations for each index.
Note: elasticsearch.yml is a "global" / static configuration file of elasticsearch that is read during startup and if you change it it will not affect current running instances.
Does Elasticsearch check for the total number of nodes present in a cluster before creating the index and assigning the shards
No. It will happily create many shards for the same index on the same node. Most likely you didn't restart ES after changing elasticsearch.yml. It's not read live, so any changes to it require a restart to take effect.