I am trying to override a mapping for a field.
There is a default index template (which I can't change) and I am overriding it with a custom one.
The default index has a mapping for "message" field as text, but I need to make it treated like an object and make its fields indexable/searchable.
This is the default index template, with order 10.
{
"mappings": {
"_default_": {
"dynamic_templates": [
{
"message_field": {
"mapping": {
"index": true,
"norms": false,
"type": "text"
},
"match": "message",
"match_mapping_type": "string"
}
},
...
],
"properties": {
"message": {
"doc_values": false,
"index": true,
"norms": false,
"type": "text"
},
...
}
}
},
"order": 10,
"template": "project.*"
}
And here's my override:
{
"template" : "project.*",
"order" : 100,
"dynamic_templates": [
{
"message_field": {
"mapping": {
"type": "object"
},
"match": "message"
}
}
],
"mappings": {
"message": {
"enabled": true,
"properties": {
"tag": {"type": "string", "index": "not_analyzed"},
"requestId": {"type": "integer"},
...
}
}
}
}
This works nice, but I end up defining all fields (tag, requestId, ...) in the "message" object.
Is there a way to make all the fields in the "message" object indexable/searchable?
Here's a sample document:
{
"level": "30",
...
"kubernetes": {
"container_name": "data-sync-server",
"namespace_name": "alitest03",
...
},
"message": {
"tag": "AUDIT",
"requestId": 1234,
...
},
}
...
}
Tried lots of things, but I can't make it work.
I am using ElasticSearch version 2.4.4.
You can use the path_match property in your dynamic mapping :
Something like :
{
"template": "project.*",
"order": 100,
"mappings": {
"<your document type here>": {
"dynamic_templates": [
{
"message_field": {
"mapping": {
"type": "object"
},
"match": "message"
}
},
{
"message_properties": {
"path_match": "message.*",
"mapping": {
"type": "string",
"index": "not_analyzed"
}
}
}
]
}
}
}
But you will maybe have to distinguish between string / numeric with match_mapping_type
Related
Running Kibana version 5.5.2.
My current setup is Logstash is taking the logs from Docker containers, runs grok filters before sending the logs to elasticsearch. The specific logs that I need to show up as long, float are two times from AWS calls to ECS and EC2 and currently a grok filter pulls them out. Here is the custom filter that pulls out the ECS timings: ECS_DESCRIBE_CONTAINER_INSTANCES (AWS)(%{SPACE})(ecs)(%{SPACE})(%{POSINT})(%{SPACE})(?<ECS_DURATION>(%{NUMBER}))(s)(%{SPACE})(?<ECS_RETRIES>(%{NONNEGINT}))(%{SPACE})(retries) so I need ECS_DURATION to be a float and ECS_RETRIES to be a long. In the docker log handler I have the following
if [ECS_DURATION] {
mutate {
convert => ["ECS_DURATION", "float"]
}
}
if [ECS_RETRIES] {
mutate {
convert => ["ECS_RETRIES", "integer"]
}
}
When I look at the field in Kibana, it still shows as a text field, but when I make the following request to elasticsearch for the mappings, it shows those fields as long and float.
GET /logstash-2020.12.18/_mapping
{
"logstash-2020.12.18": {
"mappings": {
"log": {
"_all": {
"enabled": true,
"norms": false
},
"dynamic_templates": [
{
"message_field": {
"path_match": "message",
"match_mapping_type": "string",
"mapping": {
"norms": false,
"type": "text"
}
}
},
{
"string_fields": {
"match": "*",
"match_mapping_type": "string",
"mapping": {
"fields": {
"keyword": {
"ignore_above": 256,
"type": "keyword"
}
},
"norms": false,
"type": "text"
}
}
}
],
"properties": {
"#timestamp": {
"type": "date",
"include_in_all": false
},
"#version": {
"type": "keyword",
"include_in_all": false
},
"EC2_DURATION": {
"type": "float"
},
"EC2_RETRIES": {
"type": "long"
},
"ECS_DURATION": {
"type": "float"
},
"ECS_RETRIES": {
"type": "long"
},
I even created a custom mapping template in elasticsearch with the following call
PUT /_template/aws_durations?pretty
{
"template": "logstash*",
"mappings": {
"type1": {
"_source": {
"enabled": true
},
"properties": {
"ECS_DURATION": {
"type": "half_float"
},
"ECS_RETRIES": {
"type": "byte"
},
"EC2_DURATION": {
"type": "half_float"
},
"EC2_RETRIES": {
"type": "byte"
}
}
}
}
}
Have you checked that its actually going into the if [ECS_DURATION] and if [ECS_RETRIES] conditions? (I wasnt able to comment)
hi i am using elasticsearch to index some documents. but the documents will have some fileds like goal1Completion, goal2Completion....goal100Completion. so i was trying to do mapping with dynamic Templates. so i came up with following but it throws an error:
{
"mappings": {
"date": {
"properties": {
"sessions": {
"type": "long"
},
"viewId": {
"type": "string",
"index": "not_analyzed"
},
"webPropertyId": {
"type": "string",
"index": "not_analyzed"
},
"dynamic_templates": [
{
"goalCompletions": {
"match_pattern": "regex",
"match": "goal\\d+\\w+",
"mapping": {
"type": "long"
}
}
}
]
}
}
}
}
error:"reason": "Expected map for property [fields] on field [dynamic_templates] but got a class java.lang.String"
what could be thee problem here?
You need to pull dynamic_template from properties map.
{
"mappings": {
"date": {
"properties": {
"sessions": {
"type": "long"
},
"viewId": {
"type": "string",
"index": "not_analyzed"
},
"webPropertyId": {
"type": "string",
"index": "not_analyzed"
}
},
"dynamic_templates": [ <--- Pull this out of properties
{
"goalCompletions": {
"match_pattern": "regex",
"match": "goal\\d+\\w+",
"mapping": {
"type": "long"
}
}
}
]
}
}
}
Assumed that the following document will be inserted into Elasticsearch.
{
"message": "hello",
"request": {
"body": {
"data": "hi",
"some_what_dynamic": {
"nested": {
"not_ending": 10000
}
}
}
}
}
Given that field request.body is dynamic object which has nested objects and/or fields inside, I want to store the whole request.body as a string and I don't want Elasticsearch to create new field for properties inside request.body object.
How can I define mapping to achieve this?
Here's current dynamic_templates definition:
{
"template": "logstash-*",
"mappings": {
"_default_": {
"dynamic_templates": [
{
"message_field": {
"path_match": "message",
"mapping": {
"norms": false,
"type": "text"
},
"match_mapping_type": "string"
}
},
{
"string_fields": {
"mapping": {
"norms": false,
"type": "text",
"fields": {
"keyword": {
"type": "keyword"
}
}
},
"match_mapping_type": "string",
"match": "*"
}
},
{
"body_fields": {
"match": "body",
"mapping": {
"norms": false,
"type": "text"
},
"match_mapping_type": "object"
}
}
],
...
}
}
}
When I try to insert document, I got MapperParsingException with message Can't get text on a START_OBJECT at....
I guess you just need to wrap everything in the body element in double quotes? Something like this might work (javascript)?
a={
"message": "hello",
"request": {
"body": {
"data": "hi",
"some_what_dynamic": {
"nested": {
"not_ending": 10000
}
}
}
}
};
a.request.body = JSON.stringify(a.request.body);
I tried to do:
PUT /index_name/
{ "index" : {
"analysis" : {
"analyzer" :"not_analyzed"
}
}
}
but I'm not sure it is the right way...
Edit:
I applied both methods from the answers. But I have no way to test it. If in fact it do
GET /index_name/_mapping
...
"metaData_requestHeaders_accept-language": {
"type": "string"
},
"metaData_requestHeaders_akamai-origin-hop": {
"type": "string"
},
"metaData_requestHeaders_alexatoolbar-alx_ns_ph": {
"type": "string"
},
"metaData_requestHeaders_authorization": {
"type": "string"
},
"metaData_requestHeaders_c": {
"type": "string"
},
"metaData_requestHeaders_cache-control": {
"type": "string"
},
"metaData_requestHeaders_ckiooe": {
"type": "string"
},
...
as you can see the automatic mapping does not show what analyzer is used. So I have no way to test that this is actually working. Any ideas?
Use Dynamic Templates like :
PUT my_index
{
"mappings": {
"my_type": {
"dynamic_templates": [
{
"strings": {
"match_mapping_type": "string",
"mapping": {
"type": "string",
"index": "not_analyzed"
}
}
}
]
}
}
}
After applying above template you should see something like this :
GET /my_index/_mapping
{
"my_index": {
"mappings": {
"my_type": {
"dynamic_templates": [
{
"strings": {
"mapping": {
"index": "not_analyzed",
"type": "string"
},
"match_mapping_type": "string"
}
}
],
"properties": {}
}
}
}
}
Above mapping indicate that all strings will be not_analyzed by default.
I guess what you're looking for is Elasticsearch Templates, that allow you to create mappings dynamically.
You want something like this:
PUT index_name
{
"mappings": {
"type_name": {
"dynamic_templates": [
{
"strings": {
"match_mapping_type": "string",
"mapping": {
"type": "string",
"fields": {
"raw": {
"type": "string",
"index": "not_analyzed"
}
}
}
}
}
]
}
}
}
I have an ElasticSearch type for which I want the mapping to be set dynamically. There are a few select fields on that type that I want to be analyzed, but everything else should be set to "not_analyzed".
I've come up with the following snippet. This sets all string fields to not be analyzed, but doesn't cover all other data types. I tried using the "generic" field shown in the documentation but it didn't help. Can anyone tell me how I can accomplish this?
{
"TypeName": {
"dynamic_templates": [
{
"template_name": {
"match": "*",
"match_mapping_type": "string",
"mapping": {
"index": "no",
"type": "string"
}
}
}
],
"dynamic": true,
"properties": {
"url": {
"index": "analyzed",
"type": "string"
},
"resourceUrl": {
"index": "analyzed",
"type": "string"
}
}
}
}
{
"mappings": {
"TypeName": {
"dynamic_templates": [
{
"base": {
"mapping": {
"index": "not_analyzed"
},
"match": "*",
"match_mapping_type": "*"
}
}
],
"dynamic": true,
"properties": {
"url": {
"index": "analyzed",
"type": "string"
},
"resourceUrl": {
"index": "analyzed",
"type": "string"
}
}
}
}
}
Overall, index-level template:
{
"mappings": {
"_default_": {
"dynamic_templates": [
{
"base": {
"mapping": {
"index": "not_analyzed"
},
"match": "*",
"match_mapping_type": "*"
}
}
]
}
}
}