in my elasticsearch, I will receive daily index with format like dstack-prod_dcbs-. I want to add ILM to them, immediately after they are revived. I dont know why ILM are not added to indexs. below you can find my command.(I have already defined "dstack-prod_dcbs-policy" ILM)
*PUT _template/dstack-prod_dcbs
{
"index_patterns": ["dstack-prod_dcbs-*"],
"settings": {
"index.lifecycle.name": "dstack-prod_dcbs-policy"
}
}*
but when I run
GET dstack-prod_dcbs/_ilm/explain*
below result returns
*{
"indices" : {
"dstack-prod_dcbs-20200821" : {
"index" : "dstack-prod_dcbs-20200821",
"managed" : false
},
"dstack-prod_dcbs-2020-09-22" : {
"index" : "dstack-prod_dcbs-2020-09-22",
"managed" : false
}
}
}*
I believe ILM is an alternative to using daily indices where indices are rolled over when a condition is met in the policy (not when it becomes a new day)
For ILM you need to define a rollover alias for the template
PUT _template/dstack-prod_dcbs
{
"index_patterns": ["dstack-prod_dcbs-*"],
"settings": {
"index.lifecycle.name": "dstack-prod_dcbs-policy",
"index.lifecycle.rollover_alias": "dstack-prod_dcbs"
}
}
Then you need to create the first index manually and assign it as the write index for the alias
PUT dstack-prod_dcbs-000001
{
"aliases": {
"dstack-prod_dcbs":{
"is_write_index": true
}
}
}
After that everything will be handled automatically and a new index will be created on rollover which will be then assigned as the write index for the alias
Related
Elasticsearch: 7.15.2
We have an alias created in this way:
POST _aliases
{
"actions": [
{
"add": {
"index": "my-index-*",
"alias": "my-alias",
"is_write_index": false
}
}
]
}
And if I get the alias info I can see
GET _alias/my-alias
{
"my-index-2021.11.30-000001" : {
"aliases" : {
"my-alias" : { }
}
}
}
However there is another index which has been created automatically by the rollover policy: my-index-2021.11.30-000002 but this index is not pointed by the alias created before my-alias
If I create a new alias from scratch with the same index pattern I can see both:
POST _aliases
{
"actions": [
{
"add": {
"index": "my-index-*",
"alias": "my-alias-2",
"is_write_index": false
}
}
]
}
GET _alias/my-alias-2
{
"my-index-2021.11.30-000001" : {
"aliases" : {
"my-alias-2" : { }
}
},
"my-index-2021.11.30-000002" : {
"aliases" : {
"my-alias-2" : { }
}
}
}
Is there something that I am missing? I was expecting to see also the *-000002 index pointed by the alias my-alias without any manual operation.
The rollover policy is just creating a new index if the index size is grater then X GBs
or maybe do I have to modify the index template in order to add the "read" alias automatically? I have already the alias specified in the template but that is the write alias for the rollover policy (which I cannot use for search because our custom elasticsearch configuration)
{
"index": {
"lifecycle": {
"rollover_alias": "my-write-alias"
}
}
}
When you create an alias using POST _aliases it will just create the alias on the matching indexes that currently exist, but if a new index is created later and matches the criteria, the alias will not be added to that index.
What you need to do is to:
create an index template containing your alias definition
assign your rollover policy to your template index setting (i.e. the index.lifecycle.name and index.lifecycle.rollover_alias settings)
Basically, like this:
PUT _index_template/my_index_template
{
"index_patterns": ["my-index-*"],
"template": {
"alias": {
"my-alias": {}
},
"mappings": {
...
},
"settings": {
"index.lifecycle.name": "my_policy",
"index.lifecycle.rollover_alias": "my_write_alias"
}
}
}
After this is set up, every time the lifecycle policy creates a new index my-index-2021.MM.dd-000xyz, that new index will be pointed to by my-alias. Also worth noting that my_write_alias will always point to the latest index of the sequence, i.e. the write index.
I've installed an Elastic Search (version 7.x) cluster and created a new index. I want to limit the maximum number of documents in this index. Let's say 10000 documents top.
The naive solution is to query the number of documents before inserting a new document into it. But this method can be not accurate and also have poor performances (2 requests...).
How to do it right?
The best practice is to use Index Life Management which is in the Basic License and enabled by default in Elastic v7.3+
You can set a rollover action on the number of document (i put 5 max docs) :
PUT _ilm/policy/my_policy
{
"policy": {
"phases": {
"hot": {
"actions": {
"rollover": {
"max_docs": 5
}
}
}
}
}
}
Now i create a template with the policy my_policy :
PUT _template/my_template
{
"index_patterns": [
"my-index*"
],
"settings": {
"index.blocks.read_only" : true,
"index.lifecycle.name": "my_policy",
"index.lifecycle.rollover_alias": "my-index"
}
}
Note that i put the setting "index.blocks.read_only" : true because when the rollover will be applied it will create a new index with read_only parameter.
Now i can create my index :
PUT my-index-000001
{
"settings": {
"index.blocks.read_only": false
},
"aliases": {
"my-index": {
"is_write_index": true
}
}
}
That's it ! After 5 documents, it will create a new read only index and the alias will be on writing on this one.
You can test by index some new docs with the alias :
PUT my-index/_doc/1
{
"field" : "value"
}
Also, by default the ilm policy will be applied every 10 minutes, you can change that in order to test with :
PUT /_cluster/settings
{
"persistent": {
"indices.lifecycle.poll_interval": "5s"
}
}
In our application , we are creating the elasticsearch index daily basis and index pattern is index-. (eg. index-17-09-2019). But our application is accessing the index through an alias which is pointing the current index. Now attaching and removing of the alias with the index is done through a cron job. Is it possible to do it through through index template as we are avoiding the cron job.
We can attach alias with the index through index template but I am not sure whether we can detach the alias with the old index and add it to the new index through index template.
That can be done with built-in index lifecycle management (ILM). Your application will be sending data to index alias and ILM will take care of the rest.
Here is the description of how it can be done, but basically you need to:
1. Create ILM job
PUT /_ilm/policy/my_policy
{
"policy": {
"phases": {
"hot": {
"actions": {
"rollover": {
"max_age": "1d"
}
}
}
}
}
}
2. Create an index template with ILM policy attached
PUT _template/my_template
{
"index_patterns": ["test-*"],
"settings": {
"number_of_shards": 1,
"number_of_replicas": 1,
"index.lifecycle.name": "my_policy",
"index.lifecycle.rollover_alias": "test-alias"
}
}
3. Start the process by creating init index
PUT test-000001
{
"aliases": {
"test-alias":{
"is_write_index": true
}
}
}
That will help you with handling creation of new index every day without external CRON job. You can also extend your policy, later on to e.g. delete old indices after 7 days after rollover.
Hope that helps.
Let's say I've an elasticsearch index with around 10M documents on it. Now I need to add a new filed with a default value e.g is_hotel_type=0 for each and every ES document. Later I'll update as per my requirments.
To do that I've modified myindex with a PUT request like below-
PUT myindex
{
"mappings": {
"rp": {
"properties": {
"is_hotel_type": {
"type": "integer"
}
}
}
}
}
Then run a painless script query with POST to update all the existing documents with the value is_hotel_type=0
POST myindex/_update_by_query
{
"query": {
"match_all": {}
},
"script" : "ctx._source.is_hotel_type = 0;"
}
But this process is very time consuming for a large index with 10M documents. Usually we can set default values on SQL while creating new columns. So my question-
Is there any way in Elasticsearch so I can add a new field with a default value.I've tried below PUT request with null_value but it doesn't work for.
PUT myindex/_mapping/rp
{
"properties": {
"is_hotel_type": {
"type": "integer",
"null_value" : 0
}
}
}
I just want to know is there any other way to do that without the script query?
I have the following rolling index defined:
POST /_aliases
{
"actions": [
{
"add": {
"index": "elmah_*",
"alias": "elmah_all"
}
}
]
}
That works great today, it picked up all my existing monthly rolling indexes. The problem is that when the index rolls over to a new month, it automatically generates the new index of elmah_2016_06, but my alias doesn't pick up this new index.
Each month I need to update my alias by running this:
POST /_aliases
{
"actions": [
{
"add": {
"index": "elmah_2016-06",
"alias": "elmah_all"
}
}
]
}
Is there any way to have ES pick this up automatically?
Yep, you can use a template.
PUT /_template/my_elmah_template
{
"template" : "elmah_*",
"alias" : {
"elmah_all" : { }
}
}
The empty object is a necessary evil because JSON expects field : value.
Whenever a matching index (based on the template parameter) is created, then it will automatically apply the template to it. In this case, the only thing that the template is doing is to add an alias to it.
You can also use a template to set settings and mappings.
The answer provided by pickypg is correct, but has a minor error. Should be "aliases" as in:
PUT /_template/my_elmah_template
{
"template" : "elmah_*",
"aliases" : {
"elmah_all" : { }
}
}
Elasticsearch >= 7.8
From Elasticsearch 7.8 upwards, composable index templates have been introduced, and the previous template syntax is deprecated (at least for index templates):
#! Legacy index templates are deprecated in favor of composable templates.
#! Deprecated field [template] used, replaced by [index_patterns]
The syntax to create an index template (by the name of my-index-template) that automatically adds the alias my-alias to new indices that match the pattern my-index-* would be:
PUT _index_template/my-index-template
{
"index_patterns": ["my-index-*"],
"template": {
"aliases": {
"my-alias": { }
}
}
}
As with the previous syntax, the behavior is identical: whenever a new index matches the pattern (or patterns, since it can be multiple ones) specified in the index_patterns parameter, the template will automatically be applied to it.
Besides adding an alias, composable index templates can be used for other operations, among them defining settings or mappings, just as in the past.
Elasticsearch < 7.8
For completeness sake, as already described in existing answers, the syntax before there were composable index templates was:
PUT _template/my-index-template
{
"template" : "my-index-*",
"aliases" : {
"my-alias" : { }
}
}