can i set logstash default elasticsearch mapping through elasticsearch-template.json - elasticsearch

Question

I use logstash + elasticsearch to collect syslog and want to set ttl for log ageing
I find a file named elasticsearch-template.json in the logstash,the path is logstash/logstash-1.4.2/lib/logstash/outputs/elasticsearch/elasticsearch-template.json
I add ttl info in the file like this:
{
"template" : "logstash-*",
"settings" : {
"index.refresh_interval" : "5s"
},
"mappings" : {
"_default_" : {
"_all" : {"enabled" : true},
"dynamic_templates" : [ {
"string_fields" : {
"match" : "*",
"match_mapping_type" : "string",
"mapping" : {
"type" : "string", "index" : "analyzed", "omit_norms" : true,
"fields" : {
"raw" : {"type": "string", "index" : "not_analyzed", "ignore_above" : 256}
}
}
}
} ],
"_ttl": {
"enabled": true,
"default": "1d"
},
"properties" : {
"#version": { "type": "string", "index": "not_analyzed" },
"geoip" : {
"type" : "object",
"dynamic": true,
"path": "full",
"properties" : {
"location" : { "type" : "geo_point" }
}
}
}
}
}
}
then restart logstash, delete all elasticsearch index.
I check the new index's mapping in the elasticsearch, but it didn't work in this way.
How can I config the index template?

Answer 1

you need to change your logstash configuration.
if you have followed the default settings, logstash has already created a template inside elasticsearch named logstash, logstash will keep on using that template stored in elasticsearch unless you tell it not to explicitly.
modify that template file you found but in addition to that, in your logstash configuration, set the following:
output {
elasticsearch {
...
template_overwrite => true
...
}
}

Answer 2

Doesn't look like that JSON file is in the correct folder. Here is the documentation on how to use the templates:
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/indices-templates.html
About the folder:
Config
Index templates can also be placed within the config location (path.conf) under the templates directory (note, make sure to place them on all master eligible nodes). For example, a file called template_1.json can be placed under config/templates and it will be added if it matches an index. Here is a sample of the mentioned file:

Answer 3

I've created new template.json file and defined path to it into elasticsearch output block of logstash.yml config file:
stdout { codec => json_lines }
elasticsearch {
"hosts" => ["ip:port"]
"index" => "name-of-index-%{+dd.MM.YYYY}"
template => "/{path-to-logstash-folder}/templates/your-template.json"
template_overwrite => true
manage_template => false
}
document_type for Elastic I defined into input block of logstash.yml config file:
input {
file {
path => "/your-path-to-directory/*.log"
type => "name-of-type"
}
}
There is my template.json file
{
"name-of-index": {
"order": 0,
"version": 50001,
"template": "name-of-index-*",
"settings": {
"index": {
"refresh_interval": "5s"
}
},
"mappings": {
"_default_": {
"dynamic_templates": [
{
"message_field": {
"path_match": "message",
"mapping": {
"norms": false,
"type": "text"
},
"match_mapping_type": "string"
}
},
{
"string_fields": {
"mapping": {
"norms": false,
"type": "text",
"fields": {
"keyword": {
"type": "keyword"
}
}
},
"match_mapping_type": "string",
"match": "*"
}
}
],
"_all": {
"norms": false,
"enabled": true
},
"properties": {
"#timestamp": {
"include_in_all": false,
"type": "date"
},
"geoip": {
"dynamic": true,
"properties": {
"ip": {
"type": "ip"
},
"latitude": {
"type": "half_float"
},
"location": {
"type": "geo_point"
},
"longitude": {
"type": "half_float"
}
}
},
"#version": {
"include_in_all": false,
"type": "keyword"
}
}
}
},
"aliases": {}
}
}