I know that with below config we can exclude some nodes from elastic cluster, And elastic itself relocate existing indexes on those nodes.
PUT /_cluster/settings
{
"transient" : {
"cluster.routing.allocation.exclude._ip" : "192.168.2.*"
}
}
But what I really want is to exclude some indexes from particular nodes, I tried this config
PUT test/_settings
{
"index.routing.allocation.exclude._ip": "192.168.2.*"
}
This config prohibit elastic to assign new shards to this nodes, but it seems that it does not make elastic to relocate index's shards from those node. Am I right? If I'm right how can I move existing index from particular node?
I know I can reroute shards manually but there are many shards and it is almost impossible! _reindex is another option but it takes even more!
If it matters I use elastic 2.3.5
Ok, The answer is that that config will make elastic to move indexes from excluded nodes, But elastic do it when cluster is green!
Related
I have an ElasticSearch server with thousands of indexes. My question is how can I know on which node is my index saved on
You need to use the cat shards API.
GET /_cat/shards/index-name
It will return where the shards of the index-name is located.
What do I have:
Elasticsearch (7.7.0) cluster (amazon/opendistro-for-elasticsearch:1.8.0 Docker image)
one master node
one coordinating node
two data nodes with node.attr.data=hot
one data node with node.attr.data=warm
What do I want: prevent shard allocation and relocation from hot data nodes to warm (and cold in future) data nodes.
What have I tried:
I've put "index.routing.allocation.require.data": "hot" for all index templates, so newly created indices won't be allocated to any but hot data nodes. This works fine.
Anyway, I can't restrict shards relocation from hot nodes to warm. At the moment I'm using "cluster.routing.allocation.exclude._ip" : "warm data node ip" to prevent relocation from hot data nodes to the warm data node. But will I be able to use ILM with this filter?
I've also tried to
PUT _cluster/settings
{
"transient": {
"cluster.routing.allocation.awareness.attributes": ["data"]
}
}
PUT _cluster/settings
{
"transient": {
"cluster.routing.allocation.awareness.force.data.values": ["hot"]
}
}
and then remove the "cluster.routing.allocation.exclude._ip" filter. Shards were relocating from hot data nodes to the warm data node anyway. What am I missing?
I think you miss the allocation awareness in your ILM Policy.
Your warm phase definition should have
"allocate": {
"require": {
"data": "warm"
}
}
Remove this from your definition and it should resolve your issue.
The best article to fully inderstand the ILM for me is
https://www.elastic.co/blog/implementing-hot-warm-cold-in-elasticsearch-with-index-lifecycle-management
You will find a full example in "Optimizing your ILM policy for hot-warm-cold"
I had to update settings for my old incides:
PUT my-index/_settings
{
"index.routing.allocation.require.data": "hot"
}
Indices which are under certain index template won't be updated if the index template is updated.
I'm trying to move all the shards (primary and copies) from one specific elasticsearch node to others.
While doing some studies, I came to know about Cluster-level shard allocation filtering where I can specify the node name which I want to ignore while allocating shards.
PUT _cluster/settings
{
"transient" : {
"cluster.routing.allocation.exclude._name" : "data-node-1"
}
}
My questions are,
If I dynamically update the setting, will the shards be moved from the nodes that I excluded to other nodes automatically?
How can I check and make sure that all shards are moved from a specific node?
Yes, your shards will be moved automatically, if it is possible to do so:
Shards are only relocated if it is possible to do so without breaking another routing constraint, such as never allocating a primary and replica shard on the same node.
More information here
You can use the shards api to see the location of all shards. Alternatively, if you have access to a kibana Dashboard, you can see the shard allocation in the monitoring tab for shards or indices at the very bottom.
I am very new to elastic search. I need to know what is settings in the index.is it optional? what happens if we don't include it and what happens if we don't include shards in settings.
If you're new to Elasticsearch, it's important that you understand the basic terminologies of Elastic search first.
cluster – An Elasticsearch cluster consists of one or more nodes and is identifiable by its cluster name.
node – A single Elasticsearch instance. In most environments, each node runs on a separate box or virtual machine.
index – In Elasticsearch, an index is a collection of documents like the database in mysql.
shard – Because Elasticsearch is a distributed search engine, an index is usually split into elements known as shards that are distributed across multiple nodes. Elasticsearch automatically manages the arrangement of these shards. It also rebalances the shards as necessary, so users need not worry about the details.
replica – By default, Elasticsearch creates five primary shards and one replica for each index. This means that each index will consist of five primary shards, and each shard will have one copy.
Settings are generally used to define the overall architecture of your application. It differs based on the requirement of the application.
It contains the number of shards, no of Replica sets, etc. This information is helpful to design our Elastic according to the need of the application as below:
{
"settings" : {
"index" : {
"number_of_shards" : 3,
"number_of_replicas" : 2
}
}
}
For further clarification you can visit the official documentation of Elastic community, that is very well written here.
Setting in ElasticSearch
i have a cluster with single node in it..
i had created an index A with default shard size(i.e. in elasticsearch.yml file the value of index.number_of_shards: 1). When i listed all my shards, i could see single shard for index A. After this i changed the value of index.number_of_shards: 4 in elasticsearch.yml and then created another index B. again when i listed all my shards in the cluster i could only see single shard created for index B instead of 4 shards.
Does Elasticsearch check for the total number of nodes present in a cluster before creating the index and assigning the shards(in my case i had not specified any no of shards while creating the index B, so i was expecting total of 4 shards to be created for my index). can you help me with this?
The index configuration on elasticsearch.yml is being deprecated in favor of passing this configuration on the index settings/mapping instead.
So what you have to do is to remove index configurations from the elasticsearch.yml file and pass them using index settings or template. From elastic docs:
curl -XPUT 'http://localhost:9200/twitter/' -d '{
"settings" : {
"index" : {
"number_of_shards" : 3,
"number_of_replicas" : 2
}
}
Using this approach you can create different configurations for each index.
Note: elasticsearch.yml is a "global" / static configuration file of elasticsearch that is read during startup and if you change it it will not affect current running instances.
Does Elasticsearch check for the total number of nodes present in a cluster before creating the index and assigning the shards
No. It will happily create many shards for the same index on the same node. Most likely you didn't restart ES after changing elasticsearch.yml. It's not read live, so any changes to it require a restart to take effect.