I am using a single node AWS ES cluster. Currently, its health status is showing yellow which is obvious because there is no other node to which Amazon ES can assign a replica. I want to set the replication of all my current and upcoming indices to 0. I have indices created in this pattern:
app-one-2021.02.10
app-two-2021.01.11
so on...
These indices are currently having number_of_replicas set to 1. To disable replication for all indices I am throwing a PUT request in index pattern:
PUT /app-one-*/_settings
{
"index" : {
"number_of_replicas":0
}
}
Since I am using a wildcard here so it should set number_of_replicas to 0 in all the matching indices, which it is doing successfuly.
But if any new index is created in the future let's say app-one-2021.03.10. Then the number_of_replicas is again set to 1 in this index.
Every time I have to run a PUT request to set number_of_replicas to 0 which is tedious. Why new indices are not automatically taking number_of_replicas to 0 even if I am using a wildcard (*) in my PUT request.
Is there any way to completely set replication (number_of_replicas to 0) to 0, and doesn't matter if it's a new index or an old index. How can I achieve this?
Yes, the way is to define index templates.
Before Elasticsearch v7.8, you could only use the _template API (see docs). E.g., in your case, you can create a template matching all the app-* indices:
PUT _template/app_settings
{
"index_patterns": ["app-*"],
"settings": {
"number_of_replicas": 0
}
}
Since Elasticsearch v7.8, the old API is still supported but deprecated, and you can use the _index_template API instead (see docs).
PUT _index_template/app_settings
{
"index_patterns": ["app-*"],
"template": {
"settings": {
"number_of_replicas": 0
}
}
}
Update: add code snippets for both _template and _index_template API.
Related
Let's say I have an index- A. It contains 26k documents. Now I want to change a field status with type as Keyword. As I can't change A's status field type which is already existing, I will create a new index: B with my setting my desired type.
I followed reindex API:
POST _reindex
{
"source": {
"index": "A",
"size": 10000
},
"dest": {
"index": "B",
"version_type": "external"
}
}.
But the problem is, here I can migrate only 10k docs. How to copy the rest?
How can I copy all the docs without losing any?
delete the size: 10000 and problem will be solved.
by the way the size field in Reindex API means that what batch size elasticsearch should use to fetch and reindex docs every time. by default the batch size is 100. (you thought it means how many document you want to reindex)
I am new to Elasticsearch and I have been reading documentation in order to find a way of increasing amount of shards that my index consists of. Currently my index looks like this:
country_data 0 p STARTED 227 100.7kb 192.168.0.115 $HOSTNAME
country_data 0 r STARTED 227 100.7kb 192.168.0.116 $HOSTNAME
I wanted to increase the number of shard to 5 however I was unable to find a proper way of doing it. I learnt from another Stackoverflow question that I should be able to do it like this:
POST _reindex?slices=5
{
"source": {
"index": "country_data"
},
"dest": {
"index": "country_data_new"
}
}
However when I did that I got a copy of my country_data with same amount of shards and replicas (1 and 1). I tried to learn more about it in documentation but all I found is this: https://www.elastic.co/guide/en/elasticsearch/client/curator/current/option_slices.html
I couldn't find anything in documentation about increasing number of shards in existing index or how can I move data to new index which would have more shards. I would be grateful for any insights into this problem or at least a website where could I learn how to do it.
This can be done in any of the below mentioned way.
1st Option : You can use the elastic search Split Index API.
I suggest you to please go through the documentation once before proceeding with this method.
2nd Option : Create a new index with same mappings and give the required settings for new shards. Then use the reindex API to copy data from source index to destination index
To create the new Index:
PUT /<NEW_INDEX_NAME>
{
"settings": {
"number_of_shards": <REQUIRED_NUMBER_OF_SHARDS>
},
"mappings": {<MAPPINGS_OF_SOURCE_INDEX>}
}
}
If you don't give the number of shards in the settings while creating an index, by default it creates index with one primary and one replica shard.
To Reindex from source to newly created index:
POST _reindex
{
"source": {
"index": "<SOURCE_INDEX_NAME>"
},
"dest": {
"index": "<NEW_INDEX_NAME>"
}
}
I have elasticsearch cluster for storing logs, and i have indices like this
logs-2021.01.01
logs-2021.01.02
logs.2021.01.03 ...etc
so indices creates at daily basis, and i have index template for this indices
PUT _index_template/template_1
{
"index_patterns": ["logs*"],
"template": {
"settings": {
"number_of_shards": 6,
"number_of_replicas": 1
}
but I want to make sure that indexes that are older than 1 day have 0 replicas to save disk space, and indexes that are younger than 1 day remain with 1 replica (so that in case of server loss, I have data for today)
how can i do this using elasticsearch way? i think about bash script that executes by cron , which get all of the indices which older than 1 day and make 0 replica, but i don't want to use external scripts to do that
Thank you for you help
You can use ILM (Index life cycle management) concept of the Elasticsearch.
I this, you can create policy with different state and perform some action in each state.
You can give the condition, when the index gets migrated to next state. you can give your condition base on your scenario.
PUT _ilm/policy/my_policy
{
"policy": {
"phases": {
"warm": {
"actions": {
"allocate" : {
"number_of_replicas" : 0
}
}
}
}
}
}
https://www.elastic.co/guide/en/elasticsearch/reference/current/getting-started-index-lifecycle-management.html
https://www.elastic.co/guide/en/elasticsearch/reference/current/ilm-allocate.html
This is not the full proof policy but you can use this concept for your scenario.
When ISM policies are used, the index policy settings need to be applied during index creation but those settings are lost once a new index is created from the rollover action applied by a certain stage/phase in the policy.
For instance, having indices in the form:
pattern: msp-* [* => number, in the index template]
alias: msp-*-alias [applied during the index creation]
rollover alias: msp-*-alias
policy: msp-policy-id
Having a template index pattern msp-* (where * is a number) impedes having a rollover alias msp-*-alias for each value that * can take applied automatically. How could this situation be approached?
References:
Can variables be used in elasticsearch index templates?
https://discuss.elastic.co/t/index-lifecycle-management-dynamic-rollover-alias-and-template-name/169614
https://github.com/elastic/elasticsearch/issues/20367
https://github.com/opendistro-for-elasticsearch/index-management/issues/95
https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/ism.html
In ISM policy alias does not change after rollover. For example after multiple rollover you will have msp-000001, msp-000002, msp-000003 indices are there. While all indices should point to single static alias like msp-alias. Alias does not change after rollover.
Index setting would be applicable by template while creation of the index through rollover. Below is the example of index template.
PUT _template/msp_template
{
"index_patterns": "msp-*",
"settings": {
"number_of_shards": 2,
"number_of_replicas": 1,
"index": {
"opendistro.index_state_management.rollover_alias": "msp-alias"
}
}
}
I'm using cross cluster search and searching for a document by _id that exists in both clusters.
ES returns with 2 hits (1 in local index, 1 in remote index). I just want the one in the local index. How can I remove the duplicate from the remote cluster ?
Query :
{
"query": {
"terms": {
"_id": [ "123"]
}
}
}```
You should be able achieving this by using Field Collapsingover the _id-field and define a sorting condition in which documents from your local cluster rank higher (e.g a cluster id, or a timestamp etc)
(see Elasticsearch Reference: Field Collapsing)