I have an index with the mappings
{
"mappings": {
"properties": {
"title": {
"type": "text"
},
"location": {
"type": "keyword"
}
}
}
}
In location field at the moment we are storing the city name.
And we need to change the mapping structure to store also the country and state, so the mapping will be
{
"mappings": {
"properties": {
"title": {
"type": "text"
},
"location": {
"properties": {
"country": {
"type": "keyword"
},
"state": {
"type": "keyword"
},
"city": {
"type": "keyword"
}
}
}
}
}
}
What is the recommended flow for such migration?
Elasticsearch does not allow changing the definition of mapping for existing fields, just the addition of new field definitions as you can check here.
So one of the possibilities is:
create a new field definition, with a different name obviously, to store the new data type.
Stop to use the location field
The another but costly possibility is:
create a new index with the right mapping
do the reindex of the data from the old index to the new index
To reindex the data from the old index with the right format to the new index you can use a painless script:
POST /_reindex
{
"source": {
"index": "old_index_name"
},
"dest": {
"index": "new_index_name"
},
"script": {
"lang": "painless",
"params" : {
"location":{
"country" : null,
"state": null,
"city": null
}
},
"source": """
params.location.city = ctx._source.location
ctx._source.location = params.location
"""
}
}
After you can update country and state fields for the old data.
If you need the same index name, use the new index you created with the correct mapping just as a backup, then you need to delete the index with the old mapping and recreate it again with the same name using the correct mapping and bring the data that are in the other reserve index.
For more about change the mapping read CHANGE ELASTIC SEARCH MAPPING.
Follow these steps:
Create a new index
Reindex the existing index to populate the new index
Aliases can help cutover from over index to another
Related
I put an object with some field and i wanna figure out how to mapping the index to handle and show the values like elasticsearch. I dunno why opensearch separate to individual fields the values. Both app has the same index mappings but the display is different for something.
I tried to map the object type set to nested but nothing changes
PUT test
{
"mappings": {
"properties": {
"szemelyek": {
"type": "nested",
"properties": {
"szam": {
"type": "integer"
},
"nev": {
"type": "text"
}
}
}
}
}
}
I want to find out with a rest call to which template an index is bound to. So basically pass the index name and get which template it belongs to.
Basically, I know I can list all the templates and see by patterns what indices will bind to the template, but we have so many templates and so many orderings on them that it's hard to tell.
You can use the _meta mapping field for this in order to attach any custom information to your indexes.
So let's say you have an index template like this
PUT _index_template/template_1
{
"index_patterns": ["index*"],
"template": {
"settings": {
"number_of_shards": 1
},
"mappings": {
"_meta": { <---- add this
"template": "template_1" <---- add this
}, <---- add this
"_source": {
"enabled": true
},
"properties": {
"host_name": {
"type": "keyword"
},
"created_at": {
"type": "date",
"format": "EEE MMM dd HH:mm:ss Z yyyy"
}
}
},
"aliases": {
}
},
"_meta": {
"description": "my custom template"
}
}
Once you create and index that matches that template's pattern, the _meta field will also make it into the new index you're creating.
PUT index1
Then if you get that new index settings, you'll see from which template it was created:
GET index1?filter_path=**._meta.template
=>
{
"index1" : {
"mappings" : {
"_meta" : {
"template" : "template_1" <---- you get this
},
How do I save the values that I applied the filter to a new index?
The picture is extracted only the values I want through the filter function.
I'd like to save this extracted value to a new index.
Thank you very much for letting me know.
GET 0503instgram_csv/_search?_source=message&filter_path=hits.hits._source
You can use the Reindex Api, Yuo can create new index with desired mapping and settings then project your old index with ingested data into the new one just you created. The source and destination can be any pre-existing index, index alias, or new index. However, the source and destination must be different. Consider below example, Where we created new index with the name "new_index" with some basic mappings inside PUT properties Api.
PUT /new_index
{
"settings": {
"number_of_shards": 1
},
“mappings”: {
"properties": {
"name":{
"type": "text"
},
"id":{
"type": "integer"
},
"paid": {
"type": "object"
}
}
Finally your Reindex Api may look like as below.
POST _reindex
{
"source": {
"index": "old_index"
},
"dest": {
"index": "new_index"
}
}
When I create an index with mapping like this one, what does it mean the _template/ word? what does the _ mean? I ask your help to understand more about creating an index, are they stored in a kind of folder, like template/packets folder?
PUT _template/packets
{
"template": "packets-*",
"mappings": {
"pcap_file": {
"dynamic": "false",
"properties": {
"timestamp": {
"type": "date"
},
"layers": {
"properties": {
"frame": {
"properties": {
"frame_frame_len": {
"type": "long"
},
"frame_frame_protocols": {
"type": "keyword"
}
}
},
"ip": {
"properties": {
"ip_ip_src": {
"type": "ip"
},
"ip_ip_dst": {
"type": "ip"
}
}
},
"udp": {
"properties": {
"udp_udp_srcport": {
"type": "integer"
},
"udp_udp_dstport": {
"type": "integer"
}
}
}
}
}
}
}
}
}
I ask this because after typing this, I recieve he following error
! Deprecation: Deprecated field [template] used, replaced by [index_patterns]
{
"acknowledged": true
}
I copied the pattern from this link:
https://www.elastic.co/blog/analyzing-network-packets-with-wireshark-elasticsearch-and-kibana
And I'm trying to do exactly what is taught in the link, and I already can capture files with tshark and parse copy them into a packets.json file, and I will use filebeat to transfer the data to Elasticsearch, I already uploaded some data to Elasticsearch, but it wasn't indexed correctly, I just saw a lot of information with a lot of data.
My aim is to inderstand exactly how to create a new index pattern, and also how to relate what I upload to that index.
Thank you very much.
Just replace word template with index_patterns:
PUT _template/packets
{
"index_patterns": ["packets-*"],
"mappings": {
...
Index templates allow you to define templates that will automatically be applied when new indices are created.
After version 5.6 the format of Elasticsearch index templates has changed; the template field, which was used to specify one or more patterns for matching index names that would use the template at create time, was deprecated and superseded by the more appropriately named field index_patterns which works exactly the same way.
To solve the issue and get rid of the deprecation warnings you will have to update all your pre-6.0 index templates, changing the template to index_patterns.
You can list all your index templates by running this command:
curl -XGET 'http://localhost:9200/_template/*?pretty'
Or replace the asterisk with the name of one specific index template.
More about ES templates is here.
The document that I want to index is as follows
{
"Under Armour": 0.16667,
"Skechers": 0.14774,
"Nike": 0.24404,
"New Balance": 0.11905,
"SONOMA Goods for Life": 0.11236
}
Fields under this node are dynamic, which means when documents are getting added various fields(brands) will come with those documents.
If I create an index without specifying a mapping, ES says "maximum number of fields (1000) have been reached". Though we can increase this value, it is not a good practice.
In order to support the above document, I created a mapping as follows and created an index.
{
"mappings": {
"my_type": {
"dynamic_templates": [
{
"template1":{
"match_mapping_type": "double",
"match": "*",
"mapping": {
"type": "float"
}
}
}
]
}
}
}
When I add above document to the created index and checked the mapping of the index again. It looks like as below.
{
"my_index": {
"mappings": {
"my_type": {
"dynamic_templates": [
{
"template1": {
"match": "*",
"match_mapping_type": "double",
"mapping": {
"type": "float"
}
}
}
],
"properties": {
"New Balance": {
"type": "float"
},
"Nike": {
"type": "float"
},
"SONOMA Goods for Life": {
"type": "float"
},
"Skechers": {
"type": "float"
},
"Under Armour": {
"type": "float"
}
}
}
}
}
}
If you clearly see the mapping that I created earlier and the mapping when I added a document to the index is different. It added fields statically added to the mapping. When I keep adding more documents, new fields will be added to the mapping (which will end up with maximum number of fields(1000) has been reached).
My question is,
The mapping that I mentioned above is correct for the above mentioned document.
If it is correct, why new fields are added to the mapping?
According to the posts that I read, increasing the number of fields in an index is not a good practice it may increase the resource usage.
In this case, when there are enormous number of brands are there and new brands to be introduced.
The proper solution for such a case is, introduce key-value pairs. (Probably I need to do a transformation during ETL)
{
"brands": [
{
"key": "Under Armour",
"value": 0.16667
},
{
"key": "Skechers",
"value": 0.14774
},
{
"key": "Nike",
"value": 0.24404
}
]
}
When the data is formatted as above, the map won't be change.
A good reading that I found was
https://www.elastic.co/blog/found-beginner-troubleshooting#keyvalue-woes
Thanks #Val for the suggestion