I would like to create an index in which only some fields are indexed. I created a template with the enabled property set to false. So no field is indexed by default.
https://www.elastic.co/guide/en/elasticsearch/reference/6.4/enabled.html
Then I defined the fields I want to index with dynamic templates. After I inserted a document, no field is indexed. I guess it's because enabled:false is applied on the children of the root element, and since none should be indexed, the dynamic templates are not applied.
Is there a way to set the enabled to false for all fields not covered by the dynamic templates?
DELETE so
DELETE _template/test
PUT _template/test
{
"index_patterns": [
"*so*"
],
"settings": {
"number_of_shards": 1
},
"mappings": {
"_doc": {
"dynamic": true,
"enabled": false,
"dynamic_templates": [
{
"typeOfMaterial": {
"path_match": "*.material.typeOfMaterial",
"mapping": {
"enabled": true,
"type": "nested"
}
}
},
{
"typeOfMaterialCode": {
"path_match": "*.material.typeOfMaterial.code",
"mapping": {
"enabled": true,
"type": "keyword"
}
}
}
]
}
}
}
PUT so/_doc/1
{
"count": 5,
"AAA": {
"material": {
"typeOfMaterial": [
{
"code": "MAT1"
}
]
}
}
}
According to the documentation:
Templates are processed in order — the first matching template wins.
Based on this assumption, I would try to modify the template like this:
PUT _template/test
{
"index_patterns": [
"*so*"
],
"settings": {
"number_of_shards": 1
},
"mappings": {
"_doc": {
"dynamic": true,
"dynamic_templates": [
{
"typeOfMaterial": {
"path_match": "*.material.typeOfMaterial",
"mapping": {
"enabled": true,
"type": "nested"
}
}
},
{
"typeOfMaterialCode": {
"path_match": "*.material.typeOfMaterial.code",
"mapping": {
"enabled": true,
"type": "keyword"
}
}
},
{
"allOtherFields": {
"match": "*",
"mapping": {
"enabled": false
}
}
}
]
}
}
}
Related
Dynamic templates allow you to define custom mappings that can be applied to dynamically added fields based on:
the datatype detected by Elasticsearch, with match_mapping_type.
the name of the field, with match and unmatch or match_pattern.
the full dotted path to the field, with path_match and path_unmatch.
I was trying to have a default type keyword for all fields while some special fields with specific *Suffix or prefix* could have specified types as follows, but it turned out all fields will be keyword in the end unexpectedly.
{
"order": 99,
"index_patterns": [
"xxxx_stats_*"
],
"settings": {
"index": {
"number_of_shards": "6",
"number_of_replicas": "1"
}
},
"mappings": {
"_doc": {
"dynamic": true,
"_source": {
"enabled": true
},
"dynamic_templates": [
{
"strings": {
"match_mapping_type": "*",
"unmatch": [
"*Time",
"*At",
"is*"
],
"mapping": {
"ignore_above": 256,
"null_value": "NULL",
"type": "keyword"
}
}
},
{
"timeSuffix": {
"match_mapping_type": "*",
"match": [
"*Time",
"*At"
],
"mapping": {
"type": "long"
}
}
},
{
"isPrefix": {
"match_mapping_type": "*",
"match": "is*",
"mapping": {
"type": "boolean"
}
}
}
],
"date_detection": false,
"numeric_detection": true
}
},
"aliases": {
"{index}-alias": {
}
}
}
AFAIK match and unmatch cannot be arrays, only strings or regexes. So try this:
{
"dynamic_templates":[
{
"timeSuffix":{
"match_mapping_type":"*",
"match_pattern":"regex",
"match":"^(.*Time)|(.*At)$",
"mapping":{
"type":"long"
}
}
},
{
"isPrefix":{
"match_mapping_type":"*",
"match":"is*",
"mapping":{
"type":"boolean"
}
}
},
{
"strings":{
"match_mapping_type":"*",
"mapping":{
"ignore_above":256,
"null_value":"NULL",
"type":"keyword"
}
}
}
]
}
I also find that when you move strings to the bottom, the 2 mappings above will be resolved first. Otherwise, since every segment includes match_mapping_type":"*", the first matching segment will apply. This issue may be related.
I have an index with mappings that look like this:
"mappings": {
"default": {
"_all": {
"enabled": false
},
"properties": {
"Foo": {
"properties": {
"Bar": {
"type": "keyword"
}
}
}
}
}
I am trying to change the mapping to introduce a sub-field of Bar, called Code, whilst migrating the string currently in Bar into Bar.Code. Here is the new mapping:
"mappings": {
"default": {
"_all": {
"enabled": false
},
"properties": {
"Foo": {
"properties": {
"Bar": {
"properties": {
"Code": {
"type": "keyword"
}
}
}
}
}
}
}
In order to do this, I think I need to do a _reindex and specify a pipeline. Is that correct? If so, how does my pipeline access the original data?
I have tried variations on the following code, but without success:
PUT _ingest/pipeline/transformFooBar
{
"processors": [
{
"set": {
"field": "Bar.Code",
"value": "{{_source.Bar}}"
}
}
]
}
POST _reindex
{
"source": {
"index": "foo_v1"
},
"dest": {
"index": "foo_v2",
"pipeline": "transformFooBar"
}
}
Ah, I almost had the syntax right. The _source is not required:
// Create a pipeline with a SET processor
PUT _ingest/pipeline/transformFooBar
{
"processors": [
{
"set": {
"field": "Bar.Code",
"value": "{{Bar}}"
}
}
]
}
// Reindex using the above pipeline
POST _reindex
{
"source": {
"index": "foo_v1"
},
"dest": {
"index": "foo_v2",
"pipeline": "transformFooBar"
}
}
I have many fields within my index where the field name ends in _count (e.g. page_count, order_count etc.) and I always want these to be long. I tried to create what I thought was a default mapping as follows:
{
"mappings": {
"_default_": {
"_all": {
"enabled": false,
"norms": {
"enabled": false
}
},
"properties": {
"*_count": {
"type": "long"
}
}
}
},
"settings": {
"index.query.default_field": "message",
"number_of_replicas": 2,
"number_of_shards": 3
},
"template": "core-app-*"
}
However, this doesn't seem to work as I now have string fields in my most recent index:
"page_count":{
"type":"text",
"fields":{
"keyword":{
"type":"keyword",
"ignore_above":256
}
}
}
Is this the right way to create a mapping based on a wildcard? I'm assuming not because it doesn't seem to work... :)
You can achieve it by using dynamic templates feature in elasticsearch.
PUT _template/core-app-template
{
"template": "core-app-*",
"settings": {
"index.query.default_field": "message",
"number_of_replicas": 2,
"number_of_shards": 3
},
"mappings": {
"_default_": {
"_all": {
"enabled": false,
"norms": {
"enabled": false
}
}
},
"my_type": {
"dynamic_templates": [
{
"_count_as_long": {
"match_mapping_type": "*",
"match": "*_count",
"mapping": {
"type": "long"
}
}
}
]
}
}
}
Note: watch out index_type in above example I took liberty to define it as my_type so when you are creating this index template use your actual index_type in place of my_type
I would like to combine Index Templates and Dynamic Templates so that the dynamic mapping I define is automatically added to all indices created.
Is this possible?
Regards Morten
In the index template define the mappings as dynamic template.
For example :
PUT /_template/template_1
{
"template": "yourindex*",
"mappings": {
"my_type": {
"dynamic_templates": [
{your dynamic templates ...}
]
}
}
}
You can do something like this:
PUT /_template/my_template
{
"template": "name-*",
"mappings": {
"my_type": {
"dynamic_templates": [
{
"rule1": {
"match": "field*",
"mapping": {
"type": "string",
"index": "analyzed"
}
}
},
{
"rule2": {
"match": "another*",
"mapping": {
"type": "integer"
}
}
}
],
"properties": {
"field": {
"type": "string"
}
}
}
}
}
In Elasticsearch 7.x, document types are now deprecated. For this reason, the examples from old answers will raise exception. Solution below:
PUT /_template/test_template
{
"index_patterns" : [
"test*"
],
"mappings": {
"dynamic_templates": [
{
"strings": {
"match_mapping_type": "string",
"mapping": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 1024
}
}
}
}
}
]
}
}
P.S. Also I recommend set index type "_doc" into old applications for compatibility with these templates now (fluent-bit v1.3, for example).
I need to create an Elasticsearch mapping with an Object field whose keys are not known in advance. Also, the values can be integers or strings. But I want the values to be stored as non analyzed fields if they are strings. I tried the following mapping:
PUT /my_index/_mapping/test
{
"properties": {
"alert_text": {
"type": "object",
"index": "not_analyzed"
}
}
}
Now the index is created fine. But if I insert values like this:
POST /my_index/test
{
"alert_text": {
"1": "hello moto"
}
}
The value "hello moto" is stored as an analyzed field using standard analyzer. I want it to be stored as a non analyzed field. Is it possible if I don't know in advance what all keys can be present ?
Try dynamic templates. With this feature you can configure a set of rules for the fields that are created dynamically.
In this example I've configured the rule that I think you need, i.e, all the strings fields within alert_text are not_analyzed:
PUT /my_index
{
"mappings": {
"test": {
"properties": {
"alert_text": {
"type": "object"
}
},
"dynamic_templates": [
{
"alert_text_strings": {
"path_match": "alert_text.*",
"match_mapping_type": "string",
"mapping": {
"type": "string",
"index": "not_analyzed"
}
}
}
]
}
}
}
POST /my_index/test
{
"alert_text": {
"1": "hello moto"
}
}
After executing the requests above you can execute this query to show the current mapping:
GET /my_index/_mapping
And you will obtain:
{
"my_index": {
"mappings": {
"test": {
"dynamic_templates": [
{
"alert_text_strings": {
"mapping": {
"index": "not_analyzed",
"type": "string"
},
"match_mapping_type": "string",
"path_match": "alert_text.*"
}
}
],
"properties": {
"alert_text": {
"properties": {
"1": {
"type": "string",
"index": "not_analyzed"
}
}
}
}
}
}
}
}
Where you can see that alert_text.1 is stored as not_analyzed.