Dynamic templates allow you to define custom mappings that can be applied to dynamically added fields based on:
the datatype detected by Elasticsearch, with match_mapping_type.
the name of the field, with match and unmatch or match_pattern.
the full dotted path to the field, with path_match and path_unmatch.
I was trying to have a default type keyword for all fields while some special fields with specific *Suffix or prefix* could have specified types as follows, but it turned out all fields will be keyword in the end unexpectedly.
{
"order": 99,
"index_patterns": [
"xxxx_stats_*"
],
"settings": {
"index": {
"number_of_shards": "6",
"number_of_replicas": "1"
}
},
"mappings": {
"_doc": {
"dynamic": true,
"_source": {
"enabled": true
},
"dynamic_templates": [
{
"strings": {
"match_mapping_type": "*",
"unmatch": [
"*Time",
"*At",
"is*"
],
"mapping": {
"ignore_above": 256,
"null_value": "NULL",
"type": "keyword"
}
}
},
{
"timeSuffix": {
"match_mapping_type": "*",
"match": [
"*Time",
"*At"
],
"mapping": {
"type": "long"
}
}
},
{
"isPrefix": {
"match_mapping_type": "*",
"match": "is*",
"mapping": {
"type": "boolean"
}
}
}
],
"date_detection": false,
"numeric_detection": true
}
},
"aliases": {
"{index}-alias": {
}
}
}
AFAIK match and unmatch cannot be arrays, only strings or regexes. So try this:
{
"dynamic_templates":[
{
"timeSuffix":{
"match_mapping_type":"*",
"match_pattern":"regex",
"match":"^(.*Time)|(.*At)$",
"mapping":{
"type":"long"
}
}
},
{
"isPrefix":{
"match_mapping_type":"*",
"match":"is*",
"mapping":{
"type":"boolean"
}
}
},
{
"strings":{
"match_mapping_type":"*",
"mapping":{
"ignore_above":256,
"null_value":"NULL",
"type":"keyword"
}
}
}
]
}
I also find that when you move strings to the bottom, the 2 mappings above will be resolved first. Otherwise, since every segment includes match_mapping_type":"*", the first matching segment will apply. This issue may be related.
Related
I've index with products there. I'm trying to separate hits returned from elastic by nested field value. There's my shortened index:
{
"mapping": {
"product": {
"properties": {
"id": {
"type": "integer"
},
"model_name": {
"type": "text",
},
"variants": {
"type": "nested",
"properties": {
"attributes": {
"type": "nested",
"properties": {
"id": {
"type": "integer"
},
"name": {
"type": "text"
},
"product_attribute_id": {
"type": "integer"
},
"value": {
"type": "text"
}
}
},
"id": {
"type": "integer"
},
"product_id": {
"type": "integer"
}
}
}
}
}
}
}
And product example (there's is more variants and attributes in product - I just cut them off):
{
"_index":"product_index",
"_type":"product",
"id":192,
"model_name":"Some tshirt",
"variants":[
{
"id":1271,
"product_id":192,
"attributes":[
{
"id":29,
"name":"clothesSize",
"value":"XL",
"product_attribute_id":36740
}
]
},
{
"id":1272,
"product_id":192,
"attributes":[
{
"id":29,
"name":"clothesSize",
"value":"L",
"product_attribute_id":36741
}
]
}
]
}
The field in question is attribute id. Let's say I want to separate products by size attribute - id 29. It would be perfect if the response would look like:
"hits" : [
{
"_index":"product_index",
"_type":"product",
"id":192,
"model_name":"Some tshirt",
"variants":[
{
"id":1271,
"product_id":192,
"attributes":[
{
"id":29,
"name":"clothesSize",
"value":"XL",
"product_attribute_id":36740
}
]
}
]
},
{
"_index":"product_index",
"_type":"product",
"id":192,
"model_name":"Some tshirt",
"variants":[
{
"id":1272,
"product_id":192,
"attributes":[
{
"id":29,
"name":"clothesSize",
"value":"L",
"product_attribute_id":36741
}
]
}
]
}]
I thought about separate all variants in elastic request and then group them on application side by those attribute but i think it's not most elegant and above all, efficient way.
What are the elastic keywords that I should be interested in?
Thank you in advance for your help.
I would like to create an index in which only some fields are indexed. I created a template with the enabled property set to false. So no field is indexed by default.
https://www.elastic.co/guide/en/elasticsearch/reference/6.4/enabled.html
Then I defined the fields I want to index with dynamic templates. After I inserted a document, no field is indexed. I guess it's because enabled:false is applied on the children of the root element, and since none should be indexed, the dynamic templates are not applied.
Is there a way to set the enabled to false for all fields not covered by the dynamic templates?
DELETE so
DELETE _template/test
PUT _template/test
{
"index_patterns": [
"*so*"
],
"settings": {
"number_of_shards": 1
},
"mappings": {
"_doc": {
"dynamic": true,
"enabled": false,
"dynamic_templates": [
{
"typeOfMaterial": {
"path_match": "*.material.typeOfMaterial",
"mapping": {
"enabled": true,
"type": "nested"
}
}
},
{
"typeOfMaterialCode": {
"path_match": "*.material.typeOfMaterial.code",
"mapping": {
"enabled": true,
"type": "keyword"
}
}
}
]
}
}
}
PUT so/_doc/1
{
"count": 5,
"AAA": {
"material": {
"typeOfMaterial": [
{
"code": "MAT1"
}
]
}
}
}
According to the documentation:
Templates are processed in order — the first matching template wins.
Based on this assumption, I would try to modify the template like this:
PUT _template/test
{
"index_patterns": [
"*so*"
],
"settings": {
"number_of_shards": 1
},
"mappings": {
"_doc": {
"dynamic": true,
"dynamic_templates": [
{
"typeOfMaterial": {
"path_match": "*.material.typeOfMaterial",
"mapping": {
"enabled": true,
"type": "nested"
}
}
},
{
"typeOfMaterialCode": {
"path_match": "*.material.typeOfMaterial.code",
"mapping": {
"enabled": true,
"type": "keyword"
}
}
},
{
"allOtherFields": {
"match": "*",
"mapping": {
"enabled": false
}
}
}
]
}
}
}
I have many fields within my index where the field name ends in _count (e.g. page_count, order_count etc.) and I always want these to be long. I tried to create what I thought was a default mapping as follows:
{
"mappings": {
"_default_": {
"_all": {
"enabled": false,
"norms": {
"enabled": false
}
},
"properties": {
"*_count": {
"type": "long"
}
}
}
},
"settings": {
"index.query.default_field": "message",
"number_of_replicas": 2,
"number_of_shards": 3
},
"template": "core-app-*"
}
However, this doesn't seem to work as I now have string fields in my most recent index:
"page_count":{
"type":"text",
"fields":{
"keyword":{
"type":"keyword",
"ignore_above":256
}
}
}
Is this the right way to create a mapping based on a wildcard? I'm assuming not because it doesn't seem to work... :)
You can achieve it by using dynamic templates feature in elasticsearch.
PUT _template/core-app-template
{
"template": "core-app-*",
"settings": {
"index.query.default_field": "message",
"number_of_replicas": 2,
"number_of_shards": 3
},
"mappings": {
"_default_": {
"_all": {
"enabled": false,
"norms": {
"enabled": false
}
}
},
"my_type": {
"dynamic_templates": [
{
"_count_as_long": {
"match_mapping_type": "*",
"match": "*_count",
"mapping": {
"type": "long"
}
}
}
]
}
}
}
Note: watch out index_type in above example I took liberty to define it as my_type so when you are creating this index template use your actual index_type in place of my_type
I/m indexing documents with the following format:
{
"title": "this is the title",
"brand": "brand here",
"filters": ["filter1", "filter2", "Sin filters", "Camera IP"]
"active": true
}
Then a query looks like:
'query': {
'function_score': {
'query': {
'bool': {
'filter': [
{
'term': {
'active': True
}
}
],
'must': [
{
'terms': {
'filters': ['camera ip']
}
}
]
}
}
}
}
I can't return any document with "Camera IP" filters (or any variation of this string, lowercase and so on), but Es returns the ones with filters: "Sin filters".
The index is created with the following settings. Note that "filter" fields will fall under default template and is of type keyword
"settings":{
"index":{
"analysis":{
"analyzer":{
"keylower":{
"tokenizer":"keyword",
"filter":"lowercase"
}
}
}
}
},
"mappings": {
"_default_": {
"dynamic_templates": [
{
"string_as_keywords": {
"mapping": {
"index": "not_analyzed",
"type" : "keyword",
**"analyzer": "keylower"** # I also tried with and without changing this analyzer
},
"match": "*",
"match_mapping_type": "string"
}
},
{
"integers": {
"mapping": {
"type": "integer"
},
"match": "*",
"match_mapping_type": "long"
}
},
{
"floats": {
"mapping": {
"type": "float"
},
"match": "*",
"match_mapping_type": "double"
}
}
]
}
}
What I'm missing? It's strange it returns those with "Sin filters" filter but not with "Camera IP".
Thanks.
It seems like you want the filters to be lowercase and not be tokenized. I think the problem with your query is that you set the type of the strings a "keyword" and ES will not analyze these fields, not even changing their case:
Keyword fields are only searchable by their exact value.
That is why with your setting you can still retrieve the document with a query like this: {"query": {"term": {"filters": "Camera IP"}}}'.
Since you want the analyzer to change the casing of your text before indexing you should set the type to text by changing your mapping to something like this:
{"settings":{
"index": {
"analysis":{
"analyzer":{
"test_analyzer":{
"tokenizer":"keyword",
"filter":"lowercase"
}
}
}
}
},
"mappings": {
"_default_": {
"dynamic_templates": [
{
"string_as_keywords": {
"mapping": {
"type": "text",
"index": "not_analyzed",
"analyzer": "test_analyzer"
},
"match": "*",
"match_mapping_type": "string"
}
}
]
}
}}
Your filter 'filters': ['camera ip'] looks for camera ip whereas in the mapping you have the field filters as type keyword which elasticsearch looks for an exact match. So, in order to find that field you will need to have an exact string that you index for a match. If your use case doesn't require an exact match change the type to text, for which elasticsearch analyzes before indexing. More on text datatype here and keyword datatype here
I would like to combine Index Templates and Dynamic Templates so that the dynamic mapping I define is automatically added to all indices created.
Is this possible?
Regards Morten
In the index template define the mappings as dynamic template.
For example :
PUT /_template/template_1
{
"template": "yourindex*",
"mappings": {
"my_type": {
"dynamic_templates": [
{your dynamic templates ...}
]
}
}
}
You can do something like this:
PUT /_template/my_template
{
"template": "name-*",
"mappings": {
"my_type": {
"dynamic_templates": [
{
"rule1": {
"match": "field*",
"mapping": {
"type": "string",
"index": "analyzed"
}
}
},
{
"rule2": {
"match": "another*",
"mapping": {
"type": "integer"
}
}
}
],
"properties": {
"field": {
"type": "string"
}
}
}
}
}
In Elasticsearch 7.x, document types are now deprecated. For this reason, the examples from old answers will raise exception. Solution below:
PUT /_template/test_template
{
"index_patterns" : [
"test*"
],
"mappings": {
"dynamic_templates": [
{
"strings": {
"match_mapping_type": "string",
"mapping": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 1024
}
}
}
}
}
]
}
}
P.S. Also I recommend set index type "_doc" into old applications for compatibility with these templates now (fluent-bit v1.3, for example).