I saw that following is the way to add aliases for one index.
https://www.elastic.co/guide/en/elasticsearch/reference/6.8/indices-aliases.html
Time series indices are usually formed everyday as per configuration, so how to give aliases those individual indices keeping the date part as it is?
If you scroll down the page you linked to a little bit, you'll find that you can do what you want using a glob pattern (i.e. time*)
POST /_aliases
{
"actions" : [
{ "add" : { "index" : "time*", "alias" : "all_time_indices" } }
]
}
Note, however, that if a new time series index is created it won't get the alias automatically. For that you'd need to set up an index template instead:
PUT _template/my-time-series
{
"index_patterns": ["time*"],
"aliases": {
"all_time_indices": {}
},
"settings": {
...
},
"mappings": {
...
}
}
Related
I know it's possible to define two indices in an alias where one index has the is_write_index set to true while the other has it set to false -
POST /_aliases
{
"actions" : [
{
"add" : {
"index" : "test_index_1",
"alias" : "my_alias",
"is_write_index": true
}
}
]
}
POST /_aliases
{
"actions" : [
{
"add" : {
"index" : "test_index_2",
"alias" : "my_alias",
"is_write_index": false
}
}
]
}
As you can see, I've defined two indices test_index_1 and test_index_2 where the first one is a write index while the second one isn't.
Now, I want to be able to query the my_alias in such a way that searches happen only on the test_index_2 which has the is_write_index set to false while I write data to test_index_1, instead of reading from both the indices, which is the default behaviour. Meaning, I don't wish the search results come from the index where is_write_index is set to true.
Is this possible? I've tried setting index.blocks.read to true on the write index, but then search queries on the alias fail with an exception. Instead, I wish reads on the alias query only from that index which has the is_write_index set to false.
How can I achieve this?
This can be achieved by using filtered aliases.
The way you do this is you apply a custom filter while adding the write index to the alias. The filter property defines the bool condition based on which data is filtered on this index and presented as a new view of the dataset in this index. All search queries on this index happen on this new view that Elastic creates. So, if you want to avoid reading from the index you're currently writing to, apply a filter that is never satisfied across any documents in your dataset or an exists filter on some dummy field.
POST /_aliases
{
"actions": [
{
"add": {
"index": "test_index_2",
"alias": "my_alias",
"is_write_index": true,
"filter": {
"bool": {
"must_not": {
"exists": {
"field": "<field_that_always_exists_in_your_documents>"
}
}
}
}
}
}
]
}
Once you're done writing the data, update the alias by removing the filter property to allow reads from both the indices.
You are using this feature in an incorrect fashion. If you use alias for search, it will always attempt to read across all underlying indices. is_write_index is provided as feature to support rollover and index patterns, where writes are happening to 1 index, but reads happen across all indices with same alias or index pattern.
If your intent is to load data into one index, but allow application to continue to read from old index, when data loading is going on, you should use 2 separate alias - one for read and one for write and device a strategy to swap alias pointing to the indices, after your data loading is completed.
I've got an Elastic index transactions-internal and would like to point all the names like transactions-([a-z]+)-internal to this index using alias, so all the requests like
GET /transactions-a-internal/_search
GET /transactions-b-internal/_search
GET /transactions-c-internal/_search
...
etc
should give the same result as
GET /transactions-internal/_search
I've tried
POST /transactions-internal/_alias/transactions-*-internal
but it returned
Invalid alias name [...] must not contain the following characters [ , \", *, \\, <, |, ,, >, /, ?]
Is there any "smart" solution for that? I would strongly prefer co configure it on Elastic side, not anywhere else.
You're almost there. It's the other way around, i.e. POST /<index>/_alias/<alias>
POST /transactions-*-internal/_alias/transactions-internal
UPDATE:
If you want the other way around, then you can use the following (note that an alias name cannot contain wildcard characters):
POST /_aliases
{
"actions" : [
{ "add" : { "index" : "transactions-internal", "alias" : "transactions-a-internal" } },
{ "add" : { "index" : "transactions-internal", "alias" : "transactions-b-internal" } },
{ "add" : { "index" : "transactions-internal", "alias" : "transactions-c-internal" } }
]
}
Not quite sure if this is applicable to your situation but if you're starting from scratch a possible solution might be to use a index template.
PUT _index_template/transactions-internal
{
"priority": 1,
"template": {
"aliases": {
"transactions-internal": {}
}
},
"index_patterns": [
"transactions-*-internal"
],
"composed_of": []
}
As I'm quite new to elastic I don't know if this template can be applied to an existing index.But this approach will work for new indizes in v 7.12.1
Is there any way to append non matched docs at the end of the search result?
I have been working on a project where we need to search docs by geolocation data but some docs don't have the geolocation data available. As a result of that these docs not returning in the search result.
Is there any way to append non matched docs at the end of the search result?
Example mapping:
PUT /my_locations
{
"mappings": {
"_doc": {
"properties": {
"address": {
"properties": {
"city": {
"type": "text"
},
"location": {
"type": "geo_point"
}
}
}
}
}
}
}
Data with geo location:
PUT /my_locations/_doc/1
{
"address" : {
"city: "XYZ",
"location" : {
"lat" : 40.12,
"lon" : -71.34
}
}
}
Data without geo location:
PUT /my_locations/_doc/2
{
"address" : {
"city: "ABC"
}
}
Is there any way to perform geo distance query which will select the docs with geolocation data plus append the non geo docs at the end of the result?
https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-geo-distance-query.html#query-dsl-geo-distance-query
You have two separate queries
Get documents within the area
Get other documents
To get both of these in one search, would mean all of the documents appear in one result, and share ranking. It would be difficult to create a relevancy model which gets first 9 documents with address, and one without.
But you can just run two queries at once, one for say, the first 9 documents with location, and one for without any.
Example:
GET my_locations/_msearch
{}
{"size":9,"query":{"geo_distance":{"distance":"200km","pin.location":{"lat":40,"lon":-70}}}}
{}
{"size":1,"query":{"bool":{"must_not":[{"exists":{"field":"pin.location"}}]}}}
Lets take the following scenario.
I have an alias A1 pointing to index I1. Now, I would like to use rollOver feature of ES and create index I2 and make alias point to I1 and I2.
Can I always keep rolling over and make my alias A1 point to last 2 indices or in general last 'n' indices ?
You can point one alias to multiple indices like this:
POST /_aliases
{
"actions" : [
{ "add" : { "indices" : ["l1", "l2"], "alias" : "A1" } }
]
}
or even point the alias to a wildcard index pattern like this:
POST /_aliases
{
"actions" : [
{ "add" : { "index" : "l*", "alias" : "A1" } }
]
}
EDIT: With rollover, you can only point the alias to one index - the latest index. If you want an alias that points to the last 2 indices, n indices, or all of the indices matching the pattern l*, you'll have to create an additional alias using the requests I showed above.
EDIT 2: If I wanted to maintain 30 days of logs in an index, this is how I would accomplish it. I stayed consistent with the naming of indices as 'l1' and alias of 'A1'. After the first 30 days, a new index will be created called l000002 (the naming convention is incrementing the number of the last index and zero padding with a length of 6) and the alias A1 will be pointing at the index l000002. I would create a second alias to refer to 'l*' like you originally desired.
PUT /l1
{ "aliases": { "A1": {} } }
POST /A1/_rollover
{ "conditions": { "max_age": "30d" } }
POST /_aliases
{
"actions" : [
{ "add" : { "index" : "l*", "alias" : "A2" } }
]
}
I have the following rolling index defined:
POST /_aliases
{
"actions": [
{
"add": {
"index": "elmah_*",
"alias": "elmah_all"
}
}
]
}
That works great today, it picked up all my existing monthly rolling indexes. The problem is that when the index rolls over to a new month, it automatically generates the new index of elmah_2016_06, but my alias doesn't pick up this new index.
Each month I need to update my alias by running this:
POST /_aliases
{
"actions": [
{
"add": {
"index": "elmah_2016-06",
"alias": "elmah_all"
}
}
]
}
Is there any way to have ES pick this up automatically?
Yep, you can use a template.
PUT /_template/my_elmah_template
{
"template" : "elmah_*",
"alias" : {
"elmah_all" : { }
}
}
The empty object is a necessary evil because JSON expects field : value.
Whenever a matching index (based on the template parameter) is created, then it will automatically apply the template to it. In this case, the only thing that the template is doing is to add an alias to it.
You can also use a template to set settings and mappings.
The answer provided by pickypg is correct, but has a minor error. Should be "aliases" as in:
PUT /_template/my_elmah_template
{
"template" : "elmah_*",
"aliases" : {
"elmah_all" : { }
}
}
Elasticsearch >= 7.8
From Elasticsearch 7.8 upwards, composable index templates have been introduced, and the previous template syntax is deprecated (at least for index templates):
#! Legacy index templates are deprecated in favor of composable templates.
#! Deprecated field [template] used, replaced by [index_patterns]
The syntax to create an index template (by the name of my-index-template) that automatically adds the alias my-alias to new indices that match the pattern my-index-* would be:
PUT _index_template/my-index-template
{
"index_patterns": ["my-index-*"],
"template": {
"aliases": {
"my-alias": { }
}
}
}
As with the previous syntax, the behavior is identical: whenever a new index matches the pattern (or patterns, since it can be multiple ones) specified in the index_patterns parameter, the template will automatically be applied to it.
Besides adding an alias, composable index templates can be used for other operations, among them defining settings or mappings, just as in the past.
Elasticsearch < 7.8
For completeness sake, as already described in existing answers, the syntax before there were composable index templates was:
PUT _template/my-index-template
{
"template" : "my-index-*",
"aliases" : {
"my-alias" : { }
}
}