Disable dynamic mapping completely in Elasticsearch - elasticsearch

I have an index template, from which I am creating an index
PUT /_index_template/example_template
{
"index_patterns": [
"example*"
],
"priority": 1,
"template": {
"aliases": {
"example":{}
},
"mappings": {
"dynamic":strict,
"_source":
{"enabled": false},
"properties": {
"SomeID":
{ "type": "keyword", "index" : true,"store":true,"ignore_above":5},
"firstName":
{ "type": "text", "index" : true,"store":true},
"lastName":
{ "type": "text", "index" : false},
"PersonInfo": {
"type": "object",
"dynamic":"true",
"properties": {
"FirstName": {
"type": "keyword",
"index": true,
"store": false
}
}
}
}
},
"settings": {
"index": {
"number_of_shards": 1,
"number_of_replicas": 3
}
}
}
}
As in the template mappings you can see I am making the dynamic as Strict, so that new fields cant be added to the mappings,
while on inner object, PersonInfo, I can set dynamic as true, which takes precedence and allow to insert a new field mapping.
PUT example10022021/_doc/1
{
"SomeID":"1234",
"firstName":"Nishikant",
"PersonInfo.service_data":"random"
}
Here service_data is getting added into mappings, as dynamic is true
"PersonInfo" : {
"dynamic" : "true",
"properties" : {
"FirstName" : {
"type" : "keyword"
},
"service_data" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
}
Is there any way to disable the dynamic mapping completely? like specifying globally?
Thanks!
Steps I took After #Val answer:
PUT /_index_template/example_template
{
"index_patterns": [
"example*"
],
"priority": 1,
"template": {
"aliases": {
"order":{}
},
"mappings": {
"dynamic": "strict",
"dynamic_templates": [
{
"objects": {
"match_mapping_type": "object",
"mapping": {
"dynamic": "strict"
}
}
}
],
"_source":
{"enabled": false},
"properties": {
"BillToID":
{ "type": "keyword", "index" : true,"store":true,"ignore_above":5},
"firstName":
{ "type": "text", "index" : true,"store":true},
"lastName":
{ "type": "text", "index" : false},
"PersonInfo": {
"type": "object",
"dynamic":true,
"properties": {
"FirstName": {
"type": "keyword",
"index": true,
"store": false
}
}
}
}
},
"settings": {
"index": {
"number_of_shards": 1,
"number_of_replicas": 3
}
}
}
}
then I create an index
PUT example10022021
then inserting a document
POST example10022021/_doc/1
{
"BillToID":"1234",
"firstName":"Nishikant",
"PersonInfo.service_data":"random"
}
this will result in 200OK, now if you check the mappings again
GET example10022021
in o/p you can see the dynamic field mapping getting added(this behavior I don't want),
"PersonInfo" : {
"dynamic" : "true",
"properties" : {
"FirstName" : {
"type" : "keyword"
},
"service_data" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
}
}

What you can do is to create another index template that applies to all indexes, i.e. using the * name pattern:
PUT /_index_template/common_template
{
"index_patterns": [
"*"
],
"priority": 0,
"template": {
"mappings": {
"dynamic": "strict",
...
If you want to also restrict the creation of dynamic fields inside inner objects, you can leverage dynamic templates, like this:
PUT /_index_template/common_template
{
"index_patterns": [
"*"
],
"priority": 1000,
"template": {
"settings": {},
"mappings": {
"dynamic": "strict",
"dynamic_templates": [
{
"objects": {
"match_mapping_type": "object",
"mapping": {
"dynamic": "strict"
}
}
}
],
"properties": {
"test": {
"type": "object",
"properties": {
"inner": {
"type": "integer"
}
}
}
}
}
}
}
With the above index template, you can create a document like this one:
POST test/_doc/
{
"test": {
"inner": 1
}
}
But not like this one:
POST test/_doc/
{
"test": {
"inner": 1,
"inner2": 2 <--- this will not be allowed
}
}

Related

ElasticSearch : "copy_to" a nested fields

I try to use the ES "copy_to" attribute to replicate an object field into a nested field, but I got an error despite my multiple tries. Here is my structure :
...
"identifiedBy": {
"type": "object",
"properties": {
"type": {
"type": "keyword",
"copy_to": "nested_identifiers.type"
},
"value": {
"type": "text",
"analyzer": "identifier-analyzer",
"copy_to": "nested_identifiers.type"
},
"note": {
"type": "text"
},
"qualifier": {
"type": "keyword"
},
"source": {
"type": "keyword",
"copy_to": "nested_identifiers.type"
},
"status": {
"type": "text"
}
}
},
"nested_identifiers": {
"type": "nested",
"properties": {
"type": {
"type": "keyword",
},
"value": {
"type": "text",
"analyzer": "identifier-analyzer",
},
"source": {
"type": "keyword",
}
}
}
...
The mapping error is
java.lang.IllegalArgumentException: Illegal combination of [copy_to] and [nested]
mappings: [copy_to] may only copy data to the current nested document or any of its
parents, however one [copy_to] directive is trying to copy data from nested object [null]
to [nested_identifiers]
I also try to place the "copy_to" at the "identifiedBy" root level : doesn't work.
I also try to use the a "fields" property into "identifiedBy" and "copy_to" this subfield : doesn't work.
Is anyone knows a solution to solve my problem ?
Thanks for your help.
Tldr;
Because of how Elasticsearch index nested documents. This is not possible ... without updating the mapping.
There is indeed a work around, using include_in_root: true setting.
Else I suggest you pre process you data before indexing it, and during this pre process copy the data over to the nested field. Maybe using an ingest pipeline ?
Ingest Pipeline
PUT /72270706/
{
"mappings": {
"properties": {
"root_type":{
"type": "keyword"
},
"nested_doc":{
"type": "nested",
"properties": {
"nested_type":{
"type": "keyword"
}
}
}
}
}
}
PUT _ingest/pipeline/set_nested_type
{
"processors": [
{
"set": {
"field": "nested_doc.nested_type",
"copy_from": "root_type"
}
}
]
}
POST /72270706/_doc?pipeline=set_nested_type
{
"root_type": "a type"
}
GET /72270706/_search
Should give you
{
"took" : 392,
"timed_out" : false,
"_shards" : {
...
},
"hits" : {
...
},
"max_score" : 1.0,
"hits" : [
{
"_index" : "72270706",
"_id" : "laOB0YABOgujegeQNA8D",
"_score" : 1.0,
"_source" : {
"root_type" : "a type",
"nested_doc" : {
"nested_type" : "a type"
}
}
}
]
}
}
To work around
...
"identifiedBy": {
"type": "object",
"properties": {
"type": {
"type": "keyword",
"copy_to": "nested_identifiers.type"
},
"value": {
"type": "text",
"analyzer": "identifier-analyzer",
"copy_to": "nested_identifiers.type"
},
"note": {
"type": "text"
},
"qualifier": {
"type": "keyword"
},
"source": {
"type": "keyword",
"copy_to": "nested_identifiers.type"
},
"status": {
"type": "text"
}
}
},
"nested_identifiers": {
"type": "nested",
"include_in_root": true,
"properties": {
"type": {
"type": "keyword",
},
"value": {
"type": "text",
"analyzer": "identifier-analyzer",
},
"source": {
"type": "keyword",
}
}
}
...
You will need to re index the existing data.
But be aware the copy_to will not copy the information to the nested object. But to another field, that has the same name but is not nested.

Elasticsearch multiple nested mapping error

I create an index for Elasticsearch and update mapping for that index, while doing this I use the same mapping as there is because I have reasons for it.
However, for the following situation, an error is returning that I don't understand, am I doing something wrong?
Thanks.
CREATE INDEX (Successful)
PUT /mytestindex
{
"mappings": {
"properties": {
"variable1": {
"type": "nested",
"include_in_parent": true,
"include_in_root": true,
"properties": {
"variable2": {
"type": "double"
},
"variable3": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword"
}
}
},
"variable4": {
"type": "double"
},
"variable5": {
"type": "nested",
"include_in_parent": true,
"include_in_root": true,
"properties": {
"variable6": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword"
}
}
}
}
}
}
}
}
}
}
UPDATE MAPPING AND RESPONSE
PUT /mytestindex/_mapping
{
"properties": {
"variable1": {
"type": "nested",
"include_in_parent": true,
"include_in_root": true,
"properties": {
"variable2": {
"type": "double"
},
"variable3": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword"
}
}
},
"variable4": {
"type": "double"
},
"variable5": {
"type": "nested",
"include_in_parent": true,
"include_in_root": true,
"properties": {
"variable6": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword"
}
}
}
}
}
}
}
}
}
{
"error" : {
"root_cause" : [
{
"type" : "mapper_exception",
"reason" : "the [include_in_root] parameter can't be updated on a nested object mapping"
}
],
"type" : "mapper_exception",
"reason" : "the [include_in_root] parameter can't be updated on a nested object mapping"
},
"status" : 500
}

Sorting in Elasticsearch make consistency problem

I'm sending my data to elasticsearch with index_number of documents. Its unique identifier. When i try to sort it with this, from python client i get this consistency problem as you see in the picture.
This is my query dsl
"size": 1,
"query": {
"match_all": {}
},
"sort": [
{
"index_number.keyword": {
"order": "asc",
"missing": "_last",
"unmapped_type": "String"
}
}
]
In logstash output
output{
elasticsearch {
hosts => ["localhost:9200"]
index => "logstash_%{+yyyy-MM-dd}"
manage_template => true
template_name => "logstash_template"
template => "..../logstash_template.json"
http_compression => true
}
}
In my logstash template.json
...
{
"index_patterns": ["logstash_*"],
"template": {
"settings":{
"number_of_shards": 1,
"number_of_replicas": 0,
"index": {
"sort.field": "index_number",
"sort.order": "asc"
}
},
"mappings": {
"dynamic_templates":{
"string_fields": {
"match": "*",
"match_mapping_type": "string",
"mapping": {"type":"keyword"}
}
},
"properties": {
"index_number": {
"type": "keyword",
"fields": {
"numeric": {
"type": "double"
}
}
}
}
}
}
}
....
Mapping on elasticsearch
{
"logstash_2020-03-12" : {
"mappings" : {
"properties" : {
.....
"index_number" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"city" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"country" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
-----
}
}
}
}
How can i solve it? Thanks for answering.
You need to add template_overwrite to your Logstash output configuration otherwise the logstash_template is not overridden if it already exists:
output{
elasticsearch {
hosts => ["localhost:9200"]
index => "logstash_%{+yyyy-MM-dd}"
manage_template => true
template_override => true <-- add this
template_name => "logstash_template"
template => "..../logstash_template.json"
http_compression => true
}
}
Make sure that your logstash_template.json file has the following format:
{
"index_patterns": [
"logstash_*"
],
"settings": {
"number_of_shards": 1,
"number_of_replicas": 0,
"index": {
"sort.field": "index_number",
"sort.order": "asc"
}
},
"mappings": {
"dynamic_templates": {
"string_fields": {
"match": "*",
"match_mapping_type": "string",
"mapping": {
"type": "keyword"
}
}
},
"properties": {
"index_number": {
"type": "keyword",
"fields": {
"numeric": {
"type": "double"
}
}
}
}
}
}
You had mappings and settings enclosed within the template section, but this is only for the new index templates which the elasticsearch Logstash output doesn't support yet. You need to use the legacy index templates.

Elasticsearch mappings api not showing my lists as nested type

Elastic search is not recognizing my list of objects as a nested type.
I would like for that to happen automatically without needing to update mapping for every such field.
I need the response of _mappings api to have some sort of identifier that distinguishes properties which are of list type.
For ex:
When i index such a document on a new test index ('mapping_index')
{
"text":"value",
"list":[{"a":"b","c":"d"},{"a":"q","c":"f"}]
}
and hit mappings api
localhost:9200/mapping_index/_mapping
I get
{
"mapping_index": {
"mappings": {
"_doc": {
"properties": {
"list": {
"properties": {
"a": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"c": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
},
"text": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
}
}
}
}
I would want something like
"type" : "nested"
for the "list" key in this response so that another service which uses these fields stored in ES can be conveyed that this "list" is a multivalue key.
I've read about dynamic templates and think it might be able to help me but i'm not really sure
(https://www.elastic.co/guide/en/elasticsearch/reference/current/dynamic-templates.html).
Any help is much appreciated.
You can use dynamic_templates
match_mapping_type: "object" will take any object type change it to nested
{
"mappings": {
"dynamic_templates": [
{
"objects": {
"match": "*",
"match_mapping_type": "object",
"mapping": {
"type": "nested"
}
}
}
]
}
}
Data:
{
"list": [
{
"a": "b",
"c": "d"
},
{
"a": "q",
"c": "f"
}
]
}
Result:
"index80" : {
"mappings" : {
"dynamic_templates" : [
{
"objects" : {
"match" : "*",
"match_mapping_type" : "object",
"mapping" : {
"type" : "nested"
}
}
}
],
"properties" : {
"list" : {
"type" : "nested",
"properties" : {
"a" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"c" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
}
}
}
}
}
}

ElasticSearch overriding mapping from text to object

I am trying to override a mapping for a field.
There is a default index template (which I can't change) and I am overriding it with a custom one.
The default index has a mapping for "message" field as text, but I need to make it treated like an object and make its fields indexable/searchable.
This is the default index template, with order 10.
{
"mappings": {
"_default_": {
"dynamic_templates": [
{
"message_field": {
"mapping": {
"index": true,
"norms": false,
"type": "text"
},
"match": "message",
"match_mapping_type": "string"
}
},
...
],
"properties": {
"message": {
"doc_values": false,
"index": true,
"norms": false,
"type": "text"
},
...
}
}
},
"order": 10,
"template": "project.*"
}
And here's my override:
{
"template" : "project.*",
"order" : 100,
"dynamic_templates": [
{
"message_field": {
"mapping": {
"type": "object"
},
"match": "message"
}
}
],
"mappings": {
"message": {
"enabled": true,
"properties": {
"tag": {"type": "string", "index": "not_analyzed"},
"requestId": {"type": "integer"},
...
}
}
}
}
This works nice, but I end up defining all fields (tag, requestId, ...) in the "message" object.
Is there a way to make all the fields in the "message" object indexable/searchable?
Here's a sample document:
{
"level": "30",
...
"kubernetes": {
"container_name": "data-sync-server",
"namespace_name": "alitest03",
...
},
"message": {
"tag": "AUDIT",
"requestId": 1234,
...
},
}
...
}
Tried lots of things, but I can't make it work.
I am using ElasticSearch version 2.4.4.
You can use the path_match property in your dynamic mapping :
Something like :
{
"template": "project.*",
"order": 100,
"mappings": {
"<your document type here>": {
"dynamic_templates": [
{
"message_field": {
"mapping": {
"type": "object"
},
"match": "message"
}
},
{
"message_properties": {
"path_match": "message.*",
"mapping": {
"type": "string",
"index": "not_analyzed"
}
}
}
]
}
}
}
But you will maybe have to distinguish between string / numeric with match_mapping_type

Resources