from documentation:
The most basic form of _reindex just copies documents from one index
to another. This will copy documents from the twitter index into the
new_twitter index
POST /_reindex
{
"source": {
"index": "twitter"
},
"dest": {
"index": "new_twitter"
}
}
but i don't want a index with a new name... how preserve the index name (without alias)?
Related
How do I save the values that I applied the filter to a new index?
The picture is extracted only the values I want through the filter function.
I'd like to save this extracted value to a new index.
Thank you very much for letting me know.
GET 0503instgram_csv/_search?_source=message&filter_path=hits.hits._source
You can use the Reindex Api, Yuo can create new index with desired mapping and settings then project your old index with ingested data into the new one just you created. The source and destination can be any pre-existing index, index alias, or new index. However, the source and destination must be different. Consider below example, Where we created new index with the name "new_index" with some basic mappings inside PUT properties Api.
PUT /new_index
{
"settings": {
"number_of_shards": 1
},
“mappings”: {
"properties": {
"name":{
"type": "text"
},
"id":{
"type": "integer"
},
"paid": {
"type": "object"
}
}
Finally your Reindex Api may look like as below.
POST _reindex
{
"source": {
"index": "old_index"
},
"dest": {
"index": "new_index"
}
}
I have an existing elastic search index with the following document structure, without a routing_key
{
"_id",
"feild1"
"field2"
}
I need to migrate the data into a new index. The structure of the index remains the same with an added routing_key. The routing key needs to be updated to "field1_field2". Is there a simple Kibana script to migrate the data to the new index?
Combination of a simple painless and the reindex API of elastic search could be used to achieve this.
POST _reindex
{
"source": {
"index": "{old_index_name}",
"size": {batch_size}
},
"dest": {
"index": "{new_index_name}"
},
"script": {
"lang": "painless",
"inline": "if (ctx._source.participants.length > 0) {ctx._routing=ctx._source.field1+'-'+ctx._source.field2}"
}
}
Trying to test out re-index API in elasticsearch and running into issues where existing data contains fields not present in the new index's strict mapping. Is there a way to tell elasticsearch to simply ignore those fields and carry on?
Edit: To clarify, by ignore I meant not to include those fields during the re-index process.
If you have access to the index settings before running reindex you can just do:
PUT test/_mapping
{
"dynamic": "false"
}
then change it back to strict once reindexing is done.
UPDATE based on your comment
POST _reindex
{
"source": {
"index": "src"
},
"dest": {
"index": "dst"
},
"script": {
"lang": "painless",
"source": """
ctx['_source'].remove('email');
ctx['_source'].remove('username');
ctx['_source'].remove('name');
// removing from nested:
for(item in ctx['_source'].Groups){
item.remove('GroupName');
item.remove('IsActive');
}
"""
}
}
While reindexing you can include or exclude source fields according to your destination index mapping.
To exclude some specific fields while reindexing:
POST _reindex
{
"source": {
"index": "source-index",
"_source": {
"excludes": ["exclude_a", "exclude_b"]
}
},
"dest": {
"index": "dest-index"
}
}
To include any specific field while reindexing:
POST _reindex
{
"source": {
"index": "source-index",
"_source": ["include_a", "include_b"]
},
"dest": {
"index": "dest-index"
}
}
I need to clone the content of a document in my elasticsearch index (in the same index) by using the kibana console. I need exactly the same fields in the _source of the document (of course, the copy will have another id). I tryed to:
GET the document
Create a new empty instance of document
Update the new document by
manually copying the properties of the result on (1):
POST /blog/post/VAv2FWoBKgnBpki61WiD/_update { "doc" : {
"content" : "..." ...
But the problem is the field contain veeeery long properties. And sometimes I got an error since the strings seem not to be scaped when I manually copy them from the Kibana interface.
I searched in the documentation but I can not find a query to duplicate a document, and it is a quite common think to do I think...
Any clue?
Make use of Reindex API. Here is what you can do.
Summary of steps:
Create a destination_index (dummy). Make sure the mapping is exact to that of source_index
Using Reindex API, reindex that particular document from source_index to desitnation_index. During this operation, update the _id (I've mentioned the script)
Reindex this document back from desitnation_index to source_index
Reindex Query
Step 1: Copy document from source_index to destination_index. (With the script)
POST _reindex
{
"source": {
"index": "source_index",
"query": {
"match": {
"_id": "1"
}
}
},
"dest": {
"index": "destination_index"
},
"script": {
"inline": "ctx._id=2",
"lang": "painless"
}
}
Note how I've added a script in the above query that would change the _id (_id is set as 2) of the document. Your destination_index would have all the fields with exact same values as that of source except for the _id field.
Step 2: Copy that document from destination_index to source_index
POST _reindex
{
"source": {
"index": "destination_index",
"query": {
"match": {
"_id": "2"
}
}
},
"dest": {
"index": "source_index"
}
}
Now search the source_index, it would have two documents with different _ids (_id=1 and _id=2) having exact same content.
Hope this helps!
I am upgrading from ElasticSearch 5.6 to 6.0 and I have standard logstash-* indexes. In those indexes I have multiple (doc) types "attachmentsDbStats" and "attachmentsFileStats" which have the same schema. The only difference is the value of _type and type. I have created a new index attachments-* where the type is "attachement" and I want to reindex documents of both types into the new index. Obviously b/c of the new single type restriction in 6.0, both need to have the same type. I have update all of the documents in my old index such that the "type" field has a value "attachment." When I run reindex I am not able to upload the documents due to the restriction on the single type. I have attempted to update the _type field in the old indexes but that is immutable. Any ideas how to reindex and convert the type during the conversion?
Something like this should get you started:
POST _reindex
{
"source": {
"index": "logstash-*"
},
"dest": {
"index": "attachments-*"
},
"script": {
"source": """
ctx._id = ctx._type + "-" + ctx._id;
ctx._source.type = ctx._type;
ctx._type = "attachement";
"""
}
}
Combining _type and _id into the new _id field, so it's definitely unique. Move the _type field to a custom type. And set the _type to "attachement" (note that the convention used by Elastic uses "doc" as the default type, but you can pick whatever you want as long as there is a single type).
Thank you for the tip. Fixing some of the spelling errors I had above this did the trick:
POST _reindex {
"source": {
"index": "logstash-*",
"query": {
"bool": {
"must": {
"term": {
"type": "attachments"
}
}
}
}
},
"dest": {
"index": "attachments.old"
},
"script": {
"source": "ctx._id = ctx._type + '-' + ctx._id; ctx._source.type = ctx._type; ctx._type = 'attachments';"
}
}