dynamic template: store object as a plain string - elasticsearch

Assumed that the following document will be inserted into Elasticsearch.
{
"message": "hello",
"request": {
"body": {
"data": "hi",
"some_what_dynamic": {
"nested": {
"not_ending": 10000
}
}
}
}
}
Given that field request.body is dynamic object which has nested objects and/or fields inside, I want to store the whole request.body as a string and I don't want Elasticsearch to create new field for properties inside request.body object.
How can I define mapping to achieve this?
Here's current dynamic_templates definition:
{
"template": "logstash-*",
"mappings": {
"_default_": {
"dynamic_templates": [
{
"message_field": {
"path_match": "message",
"mapping": {
"norms": false,
"type": "text"
},
"match_mapping_type": "string"
}
},
{
"string_fields": {
"mapping": {
"norms": false,
"type": "text",
"fields": {
"keyword": {
"type": "keyword"
}
}
},
"match_mapping_type": "string",
"match": "*"
}
},
{
"body_fields": {
"match": "body",
"mapping": {
"norms": false,
"type": "text"
},
"match_mapping_type": "object"
}
}
],
...
}
}
}
When I try to insert document, I got MapperParsingException with message Can't get text on a START_OBJECT at....

I guess you just need to wrap everything in the body element in double quotes? Something like this might work (javascript)?
a={
"message": "hello",
"request": {
"body": {
"data": "hi",
"some_what_dynamic": {
"nested": {
"not_ending": 10000
}
}
}
}
};
a.request.body = JSON.stringify(a.request.body);

Related

Completion Suggester in Elasticsearch not working

(Using ES 6.7)
I have an index and want to support search-as-you-type feature. For that, I want to try completion suggester but I'm having trouble in reindexing to change the mappings old index.
Here's the old index mappings
{
"old-index": {
"mappings": {
"doc": {
"properties": {
"content": {
"type": "text"
},
"project": {
"type": "keyword"
},
"title": {
"type": "text"
},
"version": {
"type": "keyword"
}
}
}
}
}
}
Here's the new test index mappings
PUT test-completion
{
"mappings": {
"doc": {
"properties": {
"content": {
"type": "text",
"fields": {
"autocomplete": {
"type": "completion",
"contexts": [
{
"name": "project",
"type": "category",
"path": "project"
},
{
"name": "version",
"type": "category",
"path": "version"
}
]
}
}
},
"title": {
"type": "text"
},
"project": {
"type": "keyword"
},
"version": {
"type": "keyword"
}
}
}
}
}
Here's the reindexing query
POST _reindex
{
"source": {
"index": "old-index"
},
"dest": {
"index": "test-completion"
}
}
And here's the query which returns no results
POST test-completion/_search
{
"suggest": {
"autocompletion_suggest": {
"prefix": "part of documentation",
"completion": {
"field": "content.autocomplete",
"fuzzy": {
"fuzziness": "AUTO"
},
"contexts": {
"project": "xyz-project",
"version": "abc-version"
}
}
}
}
}
If the prefix is set to a or b, it returns results outside of context.
Where I'm doing wrong?
https://discuss.elastic.co/t/problem-with-completion-suggester/181695

ElasticSearch overriding mapping from text to object

I am trying to override a mapping for a field.
There is a default index template (which I can't change) and I am overriding it with a custom one.
The default index has a mapping for "message" field as text, but I need to make it treated like an object and make its fields indexable/searchable.
This is the default index template, with order 10.
{
"mappings": {
"_default_": {
"dynamic_templates": [
{
"message_field": {
"mapping": {
"index": true,
"norms": false,
"type": "text"
},
"match": "message",
"match_mapping_type": "string"
}
},
...
],
"properties": {
"message": {
"doc_values": false,
"index": true,
"norms": false,
"type": "text"
},
...
}
}
},
"order": 10,
"template": "project.*"
}
And here's my override:
{
"template" : "project.*",
"order" : 100,
"dynamic_templates": [
{
"message_field": {
"mapping": {
"type": "object"
},
"match": "message"
}
}
],
"mappings": {
"message": {
"enabled": true,
"properties": {
"tag": {"type": "string", "index": "not_analyzed"},
"requestId": {"type": "integer"},
...
}
}
}
}
This works nice, but I end up defining all fields (tag, requestId, ...) in the "message" object.
Is there a way to make all the fields in the "message" object indexable/searchable?
Here's a sample document:
{
"level": "30",
...
"kubernetes": {
"container_name": "data-sync-server",
"namespace_name": "alitest03",
...
},
"message": {
"tag": "AUDIT",
"requestId": 1234,
...
},
}
...
}
Tried lots of things, but I can't make it work.
I am using ElasticSearch version 2.4.4.
You can use the path_match property in your dynamic mapping :
Something like :
{
"template": "project.*",
"order": 100,
"mappings": {
"<your document type here>": {
"dynamic_templates": [
{
"message_field": {
"mapping": {
"type": "object"
},
"match": "message"
}
},
{
"message_properties": {
"path_match": "message.*",
"mapping": {
"type": "string",
"index": "not_analyzed"
}
}
}
]
}
}
}
But you will maybe have to distinguish between string / numeric with match_mapping_type

Elasticsearch, how to check if my dynamic mapping works?

I'm providing a default mapping dynamic template at index creation in elasticsearch and wanted to check if it works as expected. Got me stumped, how can I verify if it works?
(Working with ES 2.2.2)
"mappings": {
"_default_": {
"dynamic_templates": [
{
"no_date_detection": {
"match_mapping_type": "string",
"mapping": {
"type": "string",
"date_detection": false
}
}
},
{
"language_de": {
"match_mapping_type": "*",
"match": "*_de",
"mapping": {
"type": "string",
"analyzer": "german"
}
}
},
{
"language_es": {
"match": "*_es",
"match_mapping_type": "*",
"mapping": {
"type": "string",
"analyzer": "spanish"
}
}
},
{
"language_en": {
"match": "*_en",
"match_mapping_type": "*",
"mapping": {
"type": "string",
"analyzer": "english"
}
}
}
]
}
}
It's pretty straightforward, like in the examples provided in the documentation.
GETting the mapping shows that the dynamic templates are handed down to new types
"testobject": {
"dynamic_templates": [
{
"no_date_detection": {
"mapping": {
"type": "string",
"date_detection": false
},
"match_mapping_type": "string"
}
},
{
"language_de": {
...
But when I create an object with new fields like
"description_en": "some english text"
and GET the mapping it just shows
"description_en": {
"type": "string"
}
Shouldn't this have
"analyzer": "english"
in it?
What did I do wrong, and if my dynamic mapping is correct, how can I verify that it gets applied?
Thanks in advance /Carsten
As my question "how can I verify that it gets applied?" seems unclear, I try to simplify:
I create an index with default dynamic mapping.
I create a type "testobject".
"GET /myindex/testobject/_mappings" verifies that, as expected, the dynamic templates are handed down to the type.
I create a new field in an object of type testobject.
"GET /myindex/testobject/_mappings" shows the new field but without say '"date_detection": false'. It shows it just as a simple string (see above).
How can I verify if/that the dynamic template got applied to a newly created field?
Simplified example:
PUT /myindex
{
"mappings": {
"_default_": {
"dynamic_templates": [
{
"no_date_detection": {
"match_mapping_type": "string",
"mapping": {
"type": "string",
"date_detection": false
}
}
}
]
}
}
}
PUT /myindex/gardeners/1
{
"name": "gary"
}
GET /myindex/_mapping
{
"myindex": {
"mappings": {
"gardeners": {
"dynamic_templates": [
{
"no_date_detection": {
"mapping": {
"type": "string",
"date_detection": false
},
"match_mapping_type": "string"
}
}
],
"properties": {
"name": {
"type": "string"
}
}
},
"_default_": {
"dynamic_templates": [
{
"no_date_detection": {
"mapping": {
"type": "string",
"date_detection": false
},
"match_mapping_type": "string"
}
}
]
}
}
}
}
The mapping for my new field "name"
"properties": {
"name": {
"type": "string"
}
}
doen't contain
"date_detection": false
Why doesn't it get handed down?
The dynamic templates are checked in the order they are defined and the first one that matches, that's the one that gets applied.
In your case no_date_detection template matches your field description_en because it's a string.
If you want that field to be used with the english analyzer, then you need to change the order of the templates:
"mappings": {
"_default_": {
"dynamic_templates": [
{
"language_de": {
"match_mapping_type": "*",
"match": "*_de",
"mapping": {
"type": "string",
"analyzer": "german"
}
}
},
{
"language_es": {
"match": "*_es",
"match_mapping_type": "*",
"mapping": {
"type": "string",
"analyzer": "spanish"
}
}
},
{
"language_en": {
"match": "*_en",
"match_mapping_type": "*",
"mapping": {
"type": "string",
"analyzer": "english"
}
}
},
{
"no_date_detection": {
"match_mapping_type": "string",
"mapping": {
"type": "string",
"date_detection": false
}
}
}
]
}
}
I found the error in my assumption: "date_detection": false doesn't work that way with dynamic_templates.
You have to specify the date_detection directly on the mapping (not at the level of a specific type).
https://www.elastic.co/guide/en/elasticsearch/guide/current/custom-dynamic-mapping.html
If you want this to get automatically added to new indices, you can use index templates.
Thanks to Yannick for the hint (https://discuss.elastic.co/t/mappings--default--dynamic-templates-doesnt-show-up-in-resulting-mapping/59030)

ElasticSearch - how to not store fields that are not defined in the static index?

I have set a static index with user entity in ES using
{
"mappings": {
"_default_": {
"dynamic": "false"
},
"user": {
"properties": {
"id": {
"type": "string"
},
"name": {
"type": "string"
},
"age": {
"type": "integer"
}
}
}
}
}
When I post a document with more fields than in the index it saves them to the ES.
It doesn't update the mapping but it saves the new fields.
Is there a way to remove the fields that are not in the index?
I dont want to store un-indexed fields.
In your mapping you need to use _source filtering:
{
"mappings": {
"_default_": {
"dynamic": "false"
},
"user": {
"_source": {
"includes": [
"id","name","age"
]
},
"properties": {
"id": {
"type": "string"
},
"name": {
"type": "string"
},
"age": {
"type": "integer"
}
}
}
}
}

In Elasticsearch, how to have an index with automatic mapping and all fields are not analyzed?

I tried to do:
PUT /index_name/
{ "index" : {
"analysis" : {
"analyzer" :"not_analyzed"
}
}
}
but I'm not sure it is the right way...
Edit:
I applied both methods from the answers. But I have no way to test it. If in fact it do
GET /index_name/_mapping
...
"metaData_requestHeaders_accept-language": {
"type": "string"
},
"metaData_requestHeaders_akamai-origin-hop": {
"type": "string"
},
"metaData_requestHeaders_alexatoolbar-alx_ns_ph": {
"type": "string"
},
"metaData_requestHeaders_authorization": {
"type": "string"
},
"metaData_requestHeaders_c": {
"type": "string"
},
"metaData_requestHeaders_cache-control": {
"type": "string"
},
"metaData_requestHeaders_ckiooe": {
"type": "string"
},
...
as you can see the automatic mapping does not show what analyzer is used. So I have no way to test that this is actually working. Any ideas?
Use Dynamic Templates like :
PUT my_index
{
"mappings": {
"my_type": {
"dynamic_templates": [
{
"strings": {
"match_mapping_type": "string",
"mapping": {
"type": "string",
"index": "not_analyzed"
}
}
}
]
}
}
}
After applying above template you should see something like this :
GET /my_index/_mapping
{
"my_index": {
"mappings": {
"my_type": {
"dynamic_templates": [
{
"strings": {
"mapping": {
"index": "not_analyzed",
"type": "string"
},
"match_mapping_type": "string"
}
}
],
"properties": {}
}
}
}
}
Above mapping indicate that all strings will be not_analyzed by default.
I guess what you're looking for is Elasticsearch Templates, that allow you to create mappings dynamically.
You want something like this:
PUT index_name
{
"mappings": {
"type_name": {
"dynamic_templates": [
{
"strings": {
"match_mapping_type": "string",
"mapping": {
"type": "string",
"fields": {
"raw": {
"type": "string",
"index": "not_analyzed"
}
}
}
}
}
]
}
}
}

Resources