elasticsearch mapping for timeseries - elasticsearch

We are planning to use elasticsearch as timeseries database for our metric store. After going through the following blog (https://www.elastic.co/blog/elasticsearch-as-a-time-series-data-store) on elasticsearch as timeseries database, i came up with following Mapping
{
"template": "metricsets-*",
"version": 50001,
"settings": {
"index.refresh_interval": "5s"
},
"mappings": {
"_default_": {
"_all": { "enabled": false },
"_source": { "enabled": false },
"dynamic_templates": [
{ "string_fields": { "match": "*", "match_mapping_type": "string", "mapping": { "type": "keyword" } } },
{ "tags": { "match_mapping_type": "string", "path_match": "tag.*", "mapping": { "type": "keyword" } } },
{ "metrics_object": { "match_mapping_type": "object", "mapping": { "type": "object", "index": false } } },
{ "metrics_long": { "match_mapping_type": "long", "mapping": { "type": "float", "index": false } } },
{ "metrics_double": { "match_mapping_type": "double", "mapping": { "type": "float", "index": false } } },
{ "metrics_boolean": { "match_mapping_type": "boolean", "mapping": { "type": "boolean", "index": false } } }
],
"properties": {
"#timestamp": { "type": "date" },
"#version": { "type": "keyword"}
}
}
}
}
we use metricbeat for metrics collection and logstash as ingestor. We are using elasticsearch 5.x version.
metricbeat --> logstash --> elasticsearch --> grafana/kibana.
Is my template rightly optimized for metric store? Am i missing anything?

Related

Kibana does not search on nested field

working with Elasticsearch/Kibana and trying to search on field in a nested object. However it does not seem to work. Here's mapping that I use in a template:
{
"order": 0,
"template": "ss7_signaling*",
"settings": {
"index": {
"mapping.total_fields.limit": 3000,
"number_of_shards": "5",
"refresh_interval": "30s"
},
"mappings": {
"_default_": {
"dynamic_templates": [
{
"string_fields": {
"mapping": {
"fielddata": {
"format": "disabled"
},
"index": "no",
"type": "string"
},
"match_mapping_type": "string",
"match": "*"
}
}
],
"properties": {
"message": {
"index": "not_analyzed",
"type": "string"
},
"Protocol": {
"index": "not_analyzed",
"type": "string"
},
"IMSI": {
"index": "not_analyzed",
"type": "string"
},
"nested": {
"type": "nested",
"properties": {
"name": {
"type": "string",
"index": "not_analyzed"
}
}
},
"Timestamp": {
"format": "strict_date_optional_time||epoch_millis",
"type": "date"
},
"#timestamp": {
"type": "date"
},
"#version": {
"index": "not_analyzed",
"type": "string"
}
},
"_all": {
"norms": false,
"enabled": false
}
}
},
"aliases": {
"signaling": {}
}
}
When I do search kibana on single fields - everything works fine. Still though i cannot search on nested fields like 'nested.name'.
Example of my query in kibana: nested.name:hi
Thanks.
Kibana uses the query_string query underneath, and the latter does not support querying on nested fields.
It's still being worked on but in the meantime you need to proceed differently.
UPDATE:
As of ES 7.6, it is now possible to search on nested fields

elasticsearch mapping exception when using dynamic templates

hi i am using elasticsearch to index some documents. but the documents will have some fileds like goal1Completion, goal2Completion....goal100Completion. so i was trying to do mapping with dynamic Templates. so i came up with following but it throws an error:
{
"mappings": {
"date": {
"properties": {
"sessions": {
"type": "long"
},
"viewId": {
"type": "string",
"index": "not_analyzed"
},
"webPropertyId": {
"type": "string",
"index": "not_analyzed"
},
"dynamic_templates": [
{
"goalCompletions": {
"match_pattern": "regex",
"match": "goal\\d+\\w+",
"mapping": {
"type": "long"
}
}
}
]
}
}
}
}
error:"reason": "Expected map for property [fields] on field [dynamic_templates] but got a class java.lang.String"
what could be thee problem here?
You need to pull dynamic_template from properties map.
{
"mappings": {
"date": {
"properties": {
"sessions": {
"type": "long"
},
"viewId": {
"type": "string",
"index": "not_analyzed"
},
"webPropertyId": {
"type": "string",
"index": "not_analyzed"
}
},
"dynamic_templates": [ <--- Pull this out of properties
{
"goalCompletions": {
"match_pattern": "regex",
"match": "goal\\d+\\w+",
"mapping": {
"type": "long"
}
}
}
]
}
}
}

Set index=false through dynamic mapping for all float values

I wanted to setup mapping so that any unknown field of type float automatically got the property index=false.
I used the following request:
PUT /myindex/_mapping/mytype
{
"dynamic_templates": [
{ "quantity": {
"match": "*",
"match_mapping_type": "float",
"mapping": {
"index": "false"
}
}}
],
"properties": {
"ELEMENT_ID": {
"type": "long",
"index": "true"
},
"ELEMENT_TYPE": {
"type": "keyword",
"index": "true"
}
}
}
However the unknown fields remain searchable:
GET /myindex/mytype/_search
{
"query": {
"term": { "FEEDBACK_I": "0.8202897" }
}
}
Is it possible to achieve this?
Thanks!
I suggest this approach instead (ES is more likely to match your float to a double). And, also, index property has an allowed value of no in 1.x and 2.x and true/false in 5.x:
PUT /myindex/mytype/_mapping
{
"mytype": {
"dynamic_templates": [
{
"quantity": {
"match": "*",
"match_mapping_type": "double",
"mapping": {
"type": "double",
"index": "no"
}
}
},
{
"quantity_float": {
"match": "*",
"match_mapping_type": "float",
"mapping": {
"type": "float",
"index": "no"
}
}
}
]
}
}

ElasticSearch Logstash template

I would like to index the SMTP receive log of my Exchange Server with ElasticSearch. So I created a logstash config file and it works very well but all of my fields are strings instead ip for source and target server for example. So I tried to change the default mapping in the logstash template:
I run the command curl -XGET http://localhost:9200/_template/logstash?pretty > C:\temp\logstashTemplate.txt
Edit the textfile and add my 'SourceIP' field
{
"template": "logstash-*",
"settings": {
"index": {
"refresh_interval": "5s"
}
},
"mappings": {
"_default_": {
"dynamic_templates": [{
"message_field": {
"mapping": {
"fielddata": {
"format": "disabled"
},
"index": "analyzed",
"omit_norms": true,
"type": "string"
},
"match_mapping_type": "string",
"match": "message"
}
}, {
"string_fields": {
"mapping": {
"fielddata": {
"format": "disabled"
},
"index": "analyzed",
"omit_norms": true,
"type": "string",
"fields": {
"raw": {
"ignore_above": 256,
"index": "not_analyzed",
"type": "string"
}
}
},
"match_mapping_type": "string",
"match": "*"
}
}],
"_all": {
"omit_norms": true,
"enabled": true
},
"properties": {
"#timestamp": {
"type": "date"
},
"geoip": {
"dynamic": true,
"properties": {
"ip": {
"type": "ip"
},
"latitude": {
"type": "float"
},
"location": {
"type": "geo_point"
},
"longitude": {
"type": "float"
}
}
},
"#version": {
"index": "not_analyzed",
"type": "string"
},
"SourceIP": {
"type": "ip"
}
}
}
},
"aliases": {}
}
I uploaded the edited template with the command curl -XPUT http://localhost:9200/_t
emplate/logstash -d#C:\temp\logstash.template
Restart the ElasticSearch server and index deleted/re-created
The 'SourceIP' field did not changed to type ip. What do I wrong? Can you please give me a hint? Thanks!

Create ElasticSearch dynamic template to ensure all fields are set to not_analyzed

I have an ElasticSearch type for which I want the mapping to be set dynamically. There are a few select fields on that type that I want to be analyzed, but everything else should be set to "not_analyzed".
I've come up with the following snippet. This sets all string fields to not be analyzed, but doesn't cover all other data types. I tried using the "generic" field shown in the documentation but it didn't help. Can anyone tell me how I can accomplish this?
{
"TypeName": {
"dynamic_templates": [
{
"template_name": {
"match": "*",
"match_mapping_type": "string",
"mapping": {
"index": "no",
"type": "string"
}
}
}
],
"dynamic": true,
"properties": {
"url": {
"index": "analyzed",
"type": "string"
},
"resourceUrl": {
"index": "analyzed",
"type": "string"
}
}
}
}
{
"mappings": {
"TypeName": {
"dynamic_templates": [
{
"base": {
"mapping": {
"index": "not_analyzed"
},
"match": "*",
"match_mapping_type": "*"
}
}
],
"dynamic": true,
"properties": {
"url": {
"index": "analyzed",
"type": "string"
},
"resourceUrl": {
"index": "analyzed",
"type": "string"
}
}
}
}
}
Overall, index-level template:
{
"mappings": {
"_default_": {
"dynamic_templates": [
{
"base": {
"mapping": {
"index": "not_analyzed"
},
"match": "*",
"match_mapping_type": "*"
}
}
]
}
}
}

Resources