I have a variety of elasticsearch indices which are created daily by logstash with the format:
"logstash-%{cluster_type}-%{cluster_name}-jobaccounting-v2-%{+YYYY.MM.dd}"
I would like to create an alias in elasticsearch which drops the version number from the index name. I am planning to point my kibana instance at the aliased index rather than the versioned index so that I can change the version numbers without impacting kibana.
index: "logstash-%{cluster_type}-%{cluster_name}-jobaccounting-v2-%{+YYYY.MM.dd}"
alias: "logstash-%{cluster_type}-%{cluster_name}-jobaccounting-%{+YYYY.MM.dd}"
Elasticsearch index templates can be used to create an alias everytime a new index is created.
https://www.elastic.co/blog/aliases-ftw
Unfortuantely, I have not found any good way to use variables in the alias name. I would like to avoid having to create a template for every cluster_type, cluster_name, and date.
If I had 2 entries for each variable cluster_name and cluster_type, I would have 4 indices every day, which would require 4 aliases for each day.
If I could use a date variable, then I could just have 4 templates rather than 4 templates for each day.
Is there a way to use a date variable in the alias name? Does taking this approach make sense?
The only available placeholder is {index}, so you can't compose more complex alias name inside template.
https://www.elastic.co/blog/aliases-ftw
Related
I have created the index template in elastic search 8.5.3. Here, I need the alias creation with a dynamic name.
For example, if the index name is, es.contact100 then the alias should be as.contact100.model.
I used this one,
{ "as-{index}": {} }
But, the output is, as-es.contact100
Please help me to resolve this.
I've an alias setup for rolling indices in elastic search. Let's call the alias : "alias" for now. It points to a number of indexes and rolls over after every 100gb. Now, let's say the number of fields in previous indices associated with alias is 100 and I've added one more field while writing to latest index. so, the number of fields become 101.
I've setup an index pattern by the name of "alias" and I can see all the indices listed via that index pattern but I am unable to visualize the 101th field I just added in the recent indices. Is there a way to do it ?
Please let me know if more details are needed regarding the same.
Hope you added the new field in the write index that your alias is pointing to, an alias can have only one write index but can have many read index and if you added the new field to a read index of your alias, you will not be able to visualise it using your alias.
I have a index template configured to apply on a index pattern products_*
How each document in products will have two fields
date
country_code
using the pipeline processor I wanted to create dynamic indexes like below
products_2021_12_24_us
products_2021_12_25_us
products_2021_12_24_uk
now i want to add alias for the indexes in the following behavior
for a index products_2021_12_24_us the alias should be products_us and products_2021_12_24
so that we can search the US products under one name and also for one date products under one name.
What should i use here?
Should i create multiple templates or one template can handle it all?
I have been exploring Elastic Search lately.
I have been going through aliases. I see ES provides an API to create multiple aliases to a single index like below:
{ "actions" : [{ "add" : { "indices" : ["test1", "test2"], "alias" : "alias1" } }] }
Refer: https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-aliases.html#indices-aliases
I'm wondering what is the use case of this.
Won't the queries on aliases get split if an alias point to multiple indices?
I have tried getting the info, but failed to do so as everywhere it's being explained how to acheive this but not the use case.
Directing me to a resource where I could get more info would also help.
A possible use case is when your application has to switch from an
old index to a newindex with zero downtime.
Let's say you want to reindex an index because of some reasons and you're not using aliases with your index then you need to update your application to use the new index name.
How this is helpful?
Assume that your application is using the alias instead of an index name.
Let's create an index:
PUT /my_index
Create its alias:
PUT /my_index/_alias/my_index_alias
Now you've decided to reindex your index (maybe you want to change the existing mapping).
Once documents have been reindexed correctly, you can switch your alias to point to the new index.
Note: You need to remove the alias from the old index at the same time as we add it to the new index. You can do it using _aliases endpoint atomically.
A good read : elastic
As per your question usage of maintaining two aliases for a single index:
Create “views” on a subset of the documents in an index.
Using multiple indices having same alias:
Group multiple indices under same name, which is helpful if you want to perform a single query on multiple index at the same time.
But you can't insert/index data using this strategy.
Lets say that you have to types of events, eventA & eventB. You want to "partition" them by time, so you use alias to map multiple indices (e.g. eventA-20220920) to one alias ('eventA' in this case). And you want make one alias for all the event types, so you need to give all the eventA-* and eventB-* indices another alias 'event'.
That way when you add a third type of event (eventC) you can just add them to the 'event' alias and don't change your queries
I currently have a cron job that is looking at a field called "ex.set" and performs these tasks:
For every index, run a terms aggregation on the field "ex.set"
For every index, get every existing alias
For every unique term appearing in an index in "ex.set", if it does not have an existing alias, create a filtered alias
The job runs every ten minutes but most of the time does not find anything. Is there a way or a plugin (compatible with 2.3.x), that will automatically detect when a new unique term has been inserted into an index on a specific field in a specific index? And then if there is a unique item trigger the creation of a filtered alias on that index? Thank you in advance for any ideas or solutions.
Yes, I believe you can use Watcher plugin to do this. It has a default license valid for 30 days, after which some features are disabled and afterwards you'd need a valid license to have it fully working again.
The basic idea is that your first two steps can be put in a chain input as search inputs which will collect the data.
Then, the additional step which compares the existent aliases with the terms from that aggregation can be considered as a script condition where you do your magic of comparing the two sets. If your condition establishes that a new alias needs to be created then, in the action part of the watch you can use a webhook action to call the create alias REST command on the index.