We have an elasticsearch cluster consisting of 6 nodes version 6 and we have an index called bishkek in the cluster, now I want to copy only the index mappings (no data) to the new index bishkek_v2
Elasticsearch doesn't have any API that copy only mappings, so you would need to first get your mapping for bishkek index and create new index based on the mapping. To get the mapping you can run this GET Request.
GET /bishkek/_mapping
After getting the mapping you create your new Index:
PUT /bishkek_v2
{
"mappings": {
[Mapping you get from your old index]
}
}
I think this will help you to clone the index
POST /my_source_index/_clone/my_target_index
Related
My elastic search index will ingest thousands of documents per second. The service which puts documents in the index doesn't creates a new index, instead it just gets current data in nodejs and indexes docs in "log-YYYY.MM.DD". So, we know index is created automatically if not present.
Now, my question is can this practice of creating index and putting docs at the same time cause performance issues or failures given that index wil be ingesting thousands of docs per second ?
If the answer to above question is yes, how can I create a rolling index whitch date as the index name? Say today is 5 May, 2021, so I want automatic creation of index for 6 May, 2021 in the format log-2021.05.06.
For your first question, may be this can help
how many indices?
For the second question I think you can use index-alias
like
PUT /_index_template/logdate_template
{
"index_patterns": [
"log*"
],
"priority": 1,
"template": {
"aliases": {
"log":{}
},
"mappings": {
//your mappings
}
}
}
}
As here index_pattern is "log*",
in your application code you can have a job, which creates index everyday by generating date in required format and calling
PUT log-YYYY.MM.DD
The advantage of index-alias is: You can access all those indices with "log" only.
I am using a single node AWS ES cluster. Currently, its health status is showing yellow which is obvious because there is no other node to which Amazon ES can assign a replica. I want to set the replication of all my current and upcoming indices to 0. I have indices created in this pattern:
app-one-2021.02.10
app-two-2021.01.11
so on...
These indices are currently having number_of_replicas set to 1. To disable replication for all indices I am throwing a PUT request in index pattern:
PUT /app-one-*/_settings
{
"index" : {
"number_of_replicas":0
}
}
Since I am using a wildcard here so it should set number_of_replicas to 0 in all the matching indices, which it is doing successfuly.
But if any new index is created in the future let's say app-one-2021.03.10. Then the number_of_replicas is again set to 1 in this index.
Every time I have to run a PUT request to set number_of_replicas to 0 which is tedious. Why new indices are not automatically taking number_of_replicas to 0 even if I am using a wildcard (*) in my PUT request.
Is there any way to completely set replication (number_of_replicas to 0) to 0, and doesn't matter if it's a new index or an old index. How can I achieve this?
Yes, the way is to define index templates.
Before Elasticsearch v7.8, you could only use the _template API (see docs). E.g., in your case, you can create a template matching all the app-* indices:
PUT _template/app_settings
{
"index_patterns": ["app-*"],
"settings": {
"number_of_replicas": 0
}
}
Since Elasticsearch v7.8, the old API is still supported but deprecated, and you can use the _index_template API instead (see docs).
PUT _index_template/app_settings
{
"index_patterns": ["app-*"],
"template": {
"settings": {
"number_of_replicas": 0
}
}
}
Update: add code snippets for both _template and _index_template API.
I would like to control how many shards a new index should have in my logstash output file. Ex:
10-output.conf:
output {
if [type] == "mytype" {
elasticsearch {
hosts => [ "1.1.1.1:9200" ]
index => "logstash-mytype-%{+YYYY.ww}"
workers => 8
flush_size => 1000
? <====== what option to control the number of index shards goes here?
}
}
From what I understand in logstash elastic options this is not possible and new index will default to 5 shards?
The Logstash-Elasticsearch mix it's designed to work differently than what your expectation is: in Elasticsearch one defines an index template in which the number or shards is a configuration setting.
And whenever Logstash creates a new index by sending documents to this new index, Elasticsearch uses that index template (by matching the new index name with the configured template) to actually create the index.
I have created an index "index-000001" with primary shards = 5 and replica = 1. And I have created two aliases
alias-read -> index-000001
alias-write -> index-000001
for indexing and searching purposes. When I do a rollover on alias-write when it reaches its maximum capacity, it creates a new "index-000002" and updates aliases as
alias-read -> index-000001 and index-000002
alias-write -> index-000002
How do I update/delete a document existing in index-000001(what if in case all I know is the document id but not in which index the document resides) ?
Thanks
Updating using an index alias is not directly possible, the best solution for this is to use a search query using the document id or a term and get the required index. Using the index you can update your document directly.
GET alias-read/{type}/{doc_id} will get the required Document if doc_id is known.
If doc_id is not known, then find it using a unique id reference
GET alias-read/_search
{
"term" : { "field" : "value" }
}
In both cases, you will get a single document as a response.
Once the document is obtained, you can use the "_index" field to get the required index.
PUT {index_name}/{type}/{id} {
"required_field" : "new_value"
}
to update the document.
i am trying to re-index an existing index to some other index. ex. index all documents from index A to index B. index B has new mapping .but when i look for number of documents in both indexes its very much different i am getting an approx difference of 19000 documents.what could be the reason for it.here in the re-indexing code:
The nuber of documents in index B happens to be 19000 less than the documents in index A.
POST /_reindex
{
"source": {
"index": "A"
},
"dest": {
"index": "B"
}
}
EDIT: i needed to remove a type from an existing index and add some new types to that index .below are the steps that i performed.
steps to remove an existing type from index and add new type to it
remove all data from a type of index A
download new data to index A
create index B and update it mapping (latest mapping for new
data)
reindex from original index A to new index B
remove original index A -
create the original index A with updated mapping
reindex from index B to index A