How to update index mapping with dynamic as strict in future? - elasticsearch

I am new to elastic search. I have an index named users, which has a lot of fields I know. But a few more fields can be added in the future.
So when defining my mapping, I want to include the fields that I know currently with dynamic "strict", but in the future, if I want to add the new field, how will update the new mapping and if I do it, will I have to reindex everything.
I found in the ES documents that mappings are applied only during index creation time. So I am a little confused here, what's the right way to approach this.

You can always update the mapping in future, even after keeping it strict using the put mapping api. You'll not require existing data to be re-indexed unless you want the newly added field have some value for the older documents which were added before updating the mapping with the new field.
Lets assume you already have an index test with one field say field1 of type keyword. Now in future you have a requirement to add new field say field2 of integer type. You can do so by the put mapping api as below,
PUT test/_mapping
{
"properties": {
"field2": {
"type": "integer"
}
}
}
After executing the above if you check the mapping using
GET test/_mapping
You can see the new field as well in the response,
{
"test" : {
"mappings" : {
"dynamic" : "strict",
"properties" : {
"field1" : {
"type" : "keyword"
},
"field2" : {
"type" : "integer"
}
}
}
}
}

Inner objects inherit the dynamic setting from their parent object or from the mapping type. In the following example, dynamic mapping is disabled at the type level, so no new top-level fields will be added dynamically.
However, the user.social_networks object enables dynamic mapping, so you can add fields to this inner object.
https://www.elastic.co/guide/en/elasticsearch/reference/current/dynamic.html
PUT my-index-000001
{
"mappings": {
"dynamic": false,
"properties": {
"user": {
"properties": {
"name": {
"type": "text"
},
"social_networks": {
"dynamic": true,
"properties": {}
}
}
}
}
}
}

Related

can ElasticSearch only add field index ,no save the orignal value just like lucene Field.Store.NO

I have a big size field in MySQL and do not want to save the original value to ElasticSearch. Is there a method just like Lucene Field.Store.NO?
Thanks.
You just need to define the "store" mapping accordingly, eg. :
PUT your-index
{
"mappings": {
"properties": {
"some_field": {
"type": "text",
"index": true,
"store": false
}
}
}
}
You may also want to disable the _source field :
#disable-source-field
The _source field contains the original JSON document body that was passed at index time [...] Though very handy to have around, the source field does incur storage overhead within the index.
For this reason, it can be disabled as follows:
PUT your-index
{
"mappings": {
"_source": {
"enabled": false
}
}
}

how to specify a field which should not indexed?

as mentioned in the title, I want to disable index a specified field in elasticsearch, for example, I have a fields named #fileds which contains three sub-fields like name、age、salary, now I do not want to index the field #fields.age in elasticsearch, how can I achieve that? I have tried to use include_in_all parameters, but it doesn't work. mapping configuration like:
"mappings": {
"fluentd": {
"properties": {
"#fields": {
"properties": {
"age": {
"type": "text",
"include_in_all": false,
"index": "no"
}
}
}
}
}
}
when I use this mapping configuration above, I can only see #fields.age in the index's mapping, #fields.name and #fields.salary should appear on the index's mapping not the #fields.age, how can this happen? any answers will be appreciated.

How to map dynamic field value in elasticsearch?

I'm mapping a couchbase gateway document and I'd like to tell elasticsearch to avoid indexing the internal attributes added by the gateway like the "_sync", this object contains another object named "channels" which has the following form:
"channels": {
"i7de5558-32ad-48ca-bf91-858c3a1e4588": 12
}
So I guess the mapping of this object would be like:
"channels": {
"type": "object",
"properties": {
"i7de5558-32ad-48ca-bf91-858c3a1e4588": {
"type": "integer",
"index": "not_analyze"
}
}
}
The problem is that the keys are always changing, so I don't know if I should use a wildcard like this "*": {"type": "integer", "index": "not_analyze"} for this property or do something else.
Any advice please?
If the fields are of integer types, you don't have to provide them explicitly in the mapping. You can create an empty mapping ,index documents with these fields. Elasticsearch will infer the type of field and update the mapping dynamically. You can also use dynamic templates for this.
{
"mappings": {
"my_type": {
"dynamic_templates": [
{
"analysed_string_template": {
"path_match": "channels.*",
"mapping": {
"type": "integer"
}
}
}
]
}
}
}
There`s a dynamic way to do that as you need, is called dynamic template
Using templates you are able to create rules like this:
PUT /my_index
{
"mappings": {
"my_type": {
"date_detection": false
}
}
}
In your case you could create a template to set all news fields inside the channel object as not_analyzed.
Hope it will help

Set every property type to not_analyzed for custom object

I have a custom object that I wish to store in ElasticSearch as its own type in an index, but I don't want any field in the object to be analyzed. How should I go about doing this?
I have been using the ElasticSearch NEST client but can also manually create the mapping if needed.
You have a few options that will all work. Personally, I would go with either of the first two. If it's a daily index, then the second one is the better option.
Define the mapping upfront and disable dynamic fields. This is by far the safest approach and it will help you to avoid mistakes, and it will prevent fields from being added afterward.
{
"mappings": {
"_default_": {
"_all": {
"enabled": false
}
},
"mytype" : {
"dynamic" : "strict",
"properties" : {
...
}
}
}
}
Create an index template that also disables dynamic fields, but allows you to continuously roll new indices with the same mapping(s).
You can create tiered index templates so that more than one applies to any given index.
{
"template": "mytimedindex-*",
"settings": {
"number_of_shards": 2
},
"mappings": {
"_default_": {
"_all": {
"enabled": false
}
},
"mytype" : {
"dynamic" : "strict",
"properties" : {
...
}
}
}
}
Create a dynamic mapping that allows new fields, but defaults all strings to not_analyzed:
"dynamic_templates" : [ {
"strings" : {
"mapping" : {
"index" : "not_analyzed",
"type" : "string"
},
"match" : "*",
"match_mapping_type" : "string"
}
} ]
This will allow you to dynamically add fields to the mapping.

Setting up a Kibana terms panel for an Elasticsearch field that is a list of strings

I have a Kibana dashboard that contains a terms panel to show the number of instances for a particular field (let's call it field1). Field1 is, effectively, a list of strings. Each string usually contains multiple words. Since it's analyzed, Elasticsearch breaks the terms up into separate columns. I need to keep the text together, so I need a not_analyzed version. Here's my attempt to do that with a template, located at ~\config\templates\doc_template.json on a Windows box, which does not seem to be working. Elasticsearch is running as a Windows service.
{
"doc_template": {
"template": "*",
"mappings": {
"Type-*": {
"properties": {
"Field1": {
"type": "multi_field",
"fields": {
"Field1": { "index": "analyzed" },
"RawField1": { "index": "not_analyzed" }
}
}
}
}
}
}
}
In the terms panel, I expect the necessary field to be either RawField1 or Field1.RawField1, but I've tried other variations including and excluding .raw, with no luck.
New indexes are created daily. Field1 exists in 4 separate types, each of which begin with "Type-". I suspect my attempt at using a wildcard there is problematic, but I'm not sure. All data is being sent to Elasticsearch via NEST in a C# .NET application. Here's the mapping for Field1 as it currently exists for one of the types:
{
"index-2014.12.08" : {
"mappings" : {
"Type-1" : {
"properties" : {
"Field1" : {
"type" : "string"
},
"Field2" : {
"type" : "string"
},
"Field3" : {
"type" : "string"
}
}
}
}
}
}
Obviously, the mapping doesn't look like how I expect. What's the best way to remedy this issue?

Resources