How to dynamically add an index to alias when index is dynamically created every day? I'm using Logstash to send data to our ElasticSearch engine, version 6.1.1, with the following convention:
elasticsearch {
hosts => "10.01.01.01:9200"
index => "%{[#metadata][beat]}-%{[#metadata][version]}-%{+YYYY.MM.dd}"
}
This dynamically creates a new index per day. I configured the system based on install instructions for this version.
I created an alias to be able to query across all index types (Filebeat/Winlogbeat/etc).
How can I dynamically make all dynamic indexes be added to this alias to avoid having a system administrator perform a daily task to add the index, like: (using Kibana DevTools)
POST /_aliases
{
"actions": [
{ "add": { "index": "winlogbeat-6.1.1-2018.02.16", "alias": "myaliasname"}}
]
}
Related
Is there an available JSON schema for the request object required by Elasticsearch Create Index API?
I didn't find any schema in the JSON Schema Store.
I want to validate the JSON object in IntelliJ IDEA and have the assisted edit.
Alternatively, is there any IntelliJ plugin with built-in support for editing these files? I did not find the support in any of the existing Elasticsearch* plugins.
You can use the create index API to add a new index to an Elasticsearch cluster. When creating an index, you can specify the following:
Settings for the index
Mappings for fields in the index
Index aliases
The JSON object to create Elasticsearch index looks like this:
{
"settings": {
...
},
"mappings": {
...
},
"aliases": {
...
}
}
I'm fairly new to ELK (7.10), and I would like to know how to create a map chart using GeoIP mapping.
I already have logs parsed and one field is "remote_ip" which I want to view on a map chart.
I've seen lots of instructions on how to do this but most are out of date and do not apply to my version which is 7.10. I'm using filebeats/logstash/kibana/elasticsearch.
Could someone show me the high level steps required to do this? Or point me to a detailed guide appropriate to my version? I have no idea how to begin.
I'm assuming those IP addresses are public so you can geocode them. Since your logs are already indexed, you now need to geocode them. Here is how to do it.
First, you need to modify your mapping to add a geo_point field, like this:
PUT your-index/_mapping
{
"properties": {
"remote_location": {
"type": "geo_point"
}
}
}
Once you've added that new field to your mapping, you can update your index to geocode the IP addresses. For that, you first need to create an ingest pipeline with the geoip processor:
PUT _ingest/pipeline/geoip
{
"description" : "Geocode IP address",
"processors" : [
{
"geoip" : {
"field" : "remote_ip",
"target_field": "remote_location"
}
}
]
}
Once this ingest pipeline is created you can use it to update your index using the _update_by_query endpoint like this:
POST your-index/_update_by_query?pipeline=geoip
Once the update is over, you can go into Kibana, create an index pattern and then go to Analytics > Maps and create your map.
Is it possible to use (e.g. reindex) an existing index from a newer Elasticsearch version? I tried to do it via the snapshots API, but that fails with:
the snapshot was created with Elasticsearch version [7.5.0] which is higher than the version of this node [7.4.2]
The reason we need to use the newer index is that we want to experiment with a plugin that is not yet available for the new version, but the experiments must be done on data indexed by the newer version.
The snapshot API won't work since you are trying to restore the index on an instance older than the instance that created the index.
You will need to have your index data on a 7.5 instance and use the reindex API on a 7.4.2 instance to reindex from remote
It is something like this:
POST _reindex
{
"source": {
"remote": {
"host": "http://7-5-remote-host:9200"
},
"index": "source"
},
"dest": {
"index": "dest"
}
}
You can also use a logstash pipeline to read from your 7.5 instance and index on your 7.4.2 instance.
Something like this:
input {
elasticsearch {
hosts => "http://7-5-instance:9200"
index => "your-index"
}
}
output {
elasticsearch {
hosts => "http://7-4-instance:9200"
index => "your-index"
}
}
Sometimes, I need to update mappings, settings, or bind default pipelines to the actively used index.
For the time being, I am using a method with data loss as follows:
update the index template with proper mapping (or binding the default pipeline by index.default_pipeline);
create a_new_index (matching the template index_patterns);
reindex the index_to_fix to a_new_index to migrate the data already indexed;
use alias to redirect the coming indexing request to a_new_index (the alias will have the same name as index_to_fix to ensure the indexing is undisturbed) and delete the index_to_fix;
But between step 3 and step 4, there is a time gap, during which the newly indexed data are lost in the original index_to_fix.
Is there a way, to update configurations for actively used index without any data loss?
Thanks for the help of #LeBigCat, after some discussions. I think this problem could be solved in three steps.
Use Alias for CRUD
First thing first, try not to use index directly, use alias if possible; since you can't use an alias with the same name as the existed indices, directly you can't replace the index even if it's broken (badly designed). The easiest way is to use a template and include the index name directly in the alias.
PUT _template/test
{
...
"aliases" : {
"{index}-alias" : {}
}
}
Redirect the Indexing
Since the index_to_fix is being actively used, after updating the template and create a new index a_new_fix, we can use alias to redirect the indexing to a_new_fix.
POST /_aliases
{
"actions" : [
{ "add": { "index": "a_new_index", "alias": "index_to_fix-alias" } },
{ "remove": { "index": "index_to_fix", "alias": "index_to_fix-alias" } }
]
}
Migrating the Data
Simply use _reindex to migrate all the data from index_to_fix to a_new_index.
POST _reindex
{
"source": {
"index": "index_to_fix"
},
"dest": {
"index": "index_to_fix-alias"
}
}
If I change the mapping so certain properties have new/different boost values, does that work even if the documents have already been indexed? Or do the boost values get applied when the document is indexed?
You cannot change field level boost factors after indexing data. It's not even possible for new data to be indexed once the same fields have been indexed already for previous data.
The only way to change the boost factor is to reindex your data. The pattern to do this without changing the code of your application is to use aliases. An alias points to a specific index. In case you want to change the index, you create a new index, then reindex data from the old index to the new index and finally you change the alias to point to the new index. Reindexing data is either supported by the elasticsearch library or can be achieved with a scan/scroll.
First version of mapping
Index: items_v1
Alias: items -> items_v1
Change necessary, sencond version of the index with new field level boost values :
Create new index: items_v2
Reindex data: items_v1 => items_v2
Change alias: items -> items_v2
This might be useful in other situations where you want to change your mapping.
Field level boosts are, however, not recommended. The better approach is to use boosting at query time.
Alias commands are:
Adding an alias
POST /_aliases
{
"actions": [
{ "add": {
"alias": "tems",
"index": "items_v1"
}}
]
}
Removing an alias
POST /_aliases
{
"actions": [
{ "remove": {
"alias": "tems",
"index": "items_v1"
}}
]
}
They do not.
Index time boosting is generally not recommended. Instead, you should do your boosting when you search.