ElasticSearch - alias auto update for rolling index - elasticsearch

I have the following rolling index defined:
POST /_aliases
{
"actions": [
{
"add": {
"index": "elmah_*",
"alias": "elmah_all"
}
}
]
}
That works great today, it picked up all my existing monthly rolling indexes. The problem is that when the index rolls over to a new month, it automatically generates the new index of elmah_2016_06, but my alias doesn't pick up this new index.
Each month I need to update my alias by running this:
POST /_aliases
{
"actions": [
{
"add": {
"index": "elmah_2016-06",
"alias": "elmah_all"
}
}
]
}
Is there any way to have ES pick this up automatically?

Yep, you can use a template.
PUT /_template/my_elmah_template
{
"template" : "elmah_*",
"alias" : {
"elmah_all" : { }
}
}
The empty object is a necessary evil because JSON expects field : value.
Whenever a matching index (based on the template parameter) is created, then it will automatically apply the template to it. In this case, the only thing that the template is doing is to add an alias to it.
You can also use a template to set settings and mappings.

The answer provided by pickypg is correct, but has a minor error. Should be "aliases" as in:
PUT /_template/my_elmah_template
{
"template" : "elmah_*",
"aliases" : {
"elmah_all" : { }
}
}

Elasticsearch >= 7.8
From Elasticsearch 7.8 upwards, composable index templates have been introduced, and the previous template syntax is deprecated (at least for index templates):
#! Legacy index templates are deprecated in favor of composable templates.
#! Deprecated field [template] used, replaced by [index_patterns]
The syntax to create an index template (by the name of my-index-template) that automatically adds the alias my-alias to new indices that match the pattern my-index-* would be:
PUT _index_template/my-index-template
{
"index_patterns": ["my-index-*"],
"template": {
"aliases": {
"my-alias": { }
}
}
}
As with the previous syntax, the behavior is identical: whenever a new index matches the pattern (or patterns, since it can be multiple ones) specified in the index_patterns parameter, the template will automatically be applied to it.
Besides adding an alias, composable index templates can be used for other operations, among them defining settings or mappings, just as in the past.
Elasticsearch < 7.8
For completeness sake, as already described in existing answers, the syntax before there were composable index templates was:
PUT _template/my-index-template
{
"template" : "my-index-*",
"aliases" : {
"my-alias" : { }
}
}

Related

How to update data type of a field in elasticsearch

I am publishing a data to elasticsearch using fluentd. It has a field Data.CPU which is currently set to string. Index name is health_gateway
I have made some changes in python code which is generating the data so now this field Data.CPU has now become integer. But still elasticsearch is showing it as string. How can I update it data type.
I tried running below commands in kibana dev tools:
PUT health_gateway/doc/_mapping
{
"doc" : {
"properties" : {
"Data.CPU" : {"type" : "integer"}
}
}
}
But it gave me below error:
{
"error" : {
"root_cause" : [
{
"type" : "illegal_argument_exception",
"reason" : "Types cannot be provided in put mapping requests, unless the include_type_name parameter is set to true."
}
],
"type" : "illegal_argument_exception",
"reason" : "Types cannot be provided in put mapping requests, unless the include_type_name parameter is set to true."
},
"status" : 400
}
There is also this document which says using mutate we can convert the data type but I am not able to understand it properly.
I do not want to delete the index and recreate as I have created a visualization based on this index and after deleting it will also be deleted. Can anyone please help in this.
The short answer is that you can't change the mapping of a field that already exists in a given index, as explained in the official docs.
The specific error you got is because you included /doc/ in your request path (you probably wanted /<index>/_mapping), but fixing this alone won't be sufficient.
Finally, I'm not sure you really have a dot in the field name there. Last I heard it wasn't possible to use dots in field names.
Nevertheless, there are several ways forward in your situation... here are a couple of them:
Use a scripted field
You can add a scripted field to the Kibana index-pattern. It's quick to implement, but has major performance implications. You can read more about them on the Elastic blog here (especially under the heading "Match a number and return that match").
Add a new multi-field
You could add a new multifield. The example below assumes that CPU is a nested field under Data, rather than really being called Data.CPU with a literal .:
PUT health_gateway/_mapping
{
"doc": {
"properties": {
"Data": {
"properties": {
"CPU": {
"type": "keyword",
"fields": {
"int": {
"type": "short"
}
}
}
}
}
}
}
}
Reindex your data within ES
Use the Reindex API. Be sure to set the correct mapping on the target index.
Delete and reindex everything from source
If you are able to regenerate the data from source in a timely manner, without disrupting users, you can simply delete the index and reingest all your data with an updated mapping.
You can update the mapping, by indexing the same field in multiple ways i.e by using multi fields.
Using the below mapping, Data.CPU.raw will be of integer type
{
"mappings": {
"properties": {
"Data": {
"properties": {
"CPU": {
"type": "string",
"fields": {
"raw": {
"type": "integer"
}
}
}
}
}
}
}
}
OR you can create a new index with correct index mapping, and reindex the data in it using the reindex API

Is there a way to give aliases to all the time series indices in ElasticSearch?

I saw that following is the way to add aliases for one index.
https://www.elastic.co/guide/en/elasticsearch/reference/6.8/indices-aliases.html
Time series indices are usually formed everyday as per configuration, so how to give aliases those individual indices keeping the date part as it is?
If you scroll down the page you linked to a little bit, you'll find that you can do what you want using a glob pattern (i.e. time*)
POST /_aliases
{
"actions" : [
{ "add" : { "index" : "time*", "alias" : "all_time_indices" } }
]
}
Note, however, that if a new time series index is created it won't get the alias automatically. For that you'd need to set up an index template instead:
PUT _template/my-time-series
{
"index_patterns": ["time*"],
"aliases": {
"all_time_indices": {}
},
"settings": {
...
},
"mappings": {
...
}
}

Update configuration for actively used index without data loss

Sometimes, I need to update mappings, settings, or bind default pipelines to the actively used index.
For the time being, I am using a method with data loss as follows:
update the index template with proper mapping (or binding the default pipeline by index.default_pipeline);
create a_new_index (matching the template index_patterns);
reindex the index_to_fix to a_new_index to migrate the data already indexed;
use alias to redirect the coming indexing request to a_new_index (the alias will have the same name as index_to_fix to ensure the indexing is undisturbed) and delete the index_to_fix;
But between step 3 and step 4, there is a time gap, during which the newly indexed data are lost in the original index_to_fix.
Is there a way, to update configurations for actively used index without any data loss?
Thanks for the help of #LeBigCat, after some discussions. I think this problem could be solved in three steps.
Use Alias for CRUD
First thing first, try not to use index directly, use alias if possible; since you can't use an alias with the same name as the existed indices, directly you can't replace the index even if it's broken (badly designed). The easiest way is to use a template and include the index name directly in the alias.
PUT _template/test
{
...
"aliases" : {
"{index}-alias" : {}
}
}
Redirect the Indexing
Since the index_to_fix is being actively used, after updating the template and create a new index a_new_fix, we can use alias to redirect the indexing to a_new_fix.
POST /_aliases
{
"actions" : [
{ "add": { "index": "a_new_index", "alias": "index_to_fix-alias" } },
{ "remove": { "index": "index_to_fix", "alias": "index_to_fix-alias" } }
]
}
Migrating the Data
Simply use _reindex to migrate all the data from index_to_fix to a_new_index.
POST _reindex
{
"source": {
"index": "index_to_fix"
},
"dest": {
"index": "index_to_fix-alias"
}
}

Switch time field for index pattern in kibana without loosing scripted fields or field formatting

When a time-based index is added to kibana, you have to pick the field that will act as a time field. If you want to switch from one field to another, normally I would delete the index and re-add it back. But you end up loosing scripted fields and filed formatting this way.
Is there any way to modify the existing index time field without loosing scripted fields/formatting?
It can probably be done by messing around directly with /.kibana/index-pattern/index_pattern_name but all my attempts with changing timeFieldName directly ended up dropping scripted fields.
This is what worked for me on Kibana 7.7.0 (Elastic Cloud):
find the {id} of the document whose title field corresponds to the index you want to make change
GET .kibana/_search
{
"query": {
"match": {
"index-pattern.title": "{NAME_OF_THE_INDEX}"
}
}
}
change the timefield with following code
POST .kibana/_update/{id}
{
"doc": {
"index-pattern": {
"timeFieldName" : "{NEW_TIME_FIELD_NAME}"
}
}
}
The easiest way seems to update the corresponding document:
POST /.kibana/index-pattern/YOUR_INDEX_NAME/_update
{
"doc": {
"timeFieldName": "NEW_TIME_FIELD_NAME"
}
}
It should preserve scripted fields.
This doesn't seem to work on Kibana 5.
Instead, following is the way worked on Kibana 5.
1. find the {id} of the document whose title field corresponds to the index you want to make change
GET .kibana/index-pattern/_search
{
"_source" : "_id",
"query" : {
"match" : {
"title": "{NAME_OF_THE_INDEX}"
}
}
}
2. change the timefield with following code
POST /.kibana/index-pattern/{id}/_update
{
"doc": {
"timeFieldName" : "{NEW_TIME_FIELD_NAME}"
}
}
This worked fine with me on Kibana 5.

How to create a common mapping template for indices?

For the app i created, the indices are generated once in a week. And the type and nature of the data is not varying and that implies, I need the same mapping type for these indices. Is it possible in elasticsearch to apply the same mapping to all the indices as they are created?. This could avoid me the overhead of defining mapping each time the index is created.
Definitely, you can use what is called an index template. Since your mapping type is stable, that's the perfect condition for using index templates.
It's as easy as creating an index. See below, whenever you want to index a document in an index whose name matches my_*, ES will select that template and create the index for you using the given mappings, settings and aliases:
curl -XPUT localhost:9200/_template/template_1 -d '{
"template" : "my_*",
"settings" : {
"number_of_shards" : 1
},
"aliases" : {
"my_alias" : {}
},
"mappings" : {
"my_type" : {
"properties" : {
"my_field": { "type": "string" }
}
}
}
}'
It's basically the technique used by Logstash when it needs to index new logs for each new day in a new daily index.
You can employ index template to address your problem. The official documentation can be found here.
A use case of how to apply the same with examples can be found in this blog

Resources