I would like to change the type of the field location to geo_point. I'm using ES with Logstash, as y'all know, indices are generated with the name logstash-yyyy-mm-dd
I first created a logstash index and named it logstash-2016-03-29, like so:
curl -XPUT 'http://localhost:9200/logstash-2016-03-29'
then, I changed the mapping for supposedly all the indices called Logstash-* using the following:
curl -XPOST "http://localhost:9200/logstash-*/_mapping/logs" -d '{
"properties" : {
"location" : { "type":"geo_point" }
}
}'
And when I ran the Logstash configuration file, all the location fields in the index logstash-2016-03-29 were indeed of type geo_point.
However, today, the auto-generated index logstash-2016-03-30 had field location of type String instead of geo_point. I thought the type should be applied on ANY index that starts with the name logstash-*. Apparently, I was wrong. How can I fix this so that any future index created by logstash that have the location field would have that field type set to geo_point instead of String?
Thanks.
You should define it using the index template
curl -XPUT localhost:9200/_template/template_2 -d '
{
"template" : "logstash-",
"mappings" : {
"logs" : {
"properties": {
"location" : { "type" : "geo_point" }
}
}
}
}
https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-templates.html
Related
I have index mapping for type 'T1' as below:
"T1" : {
"properties" : {
"prop1" : {
"type" : "text"
}
}
}
And now I want to change the type of prop1 from text to keyword. I don't want to delete index. I have also read people suggesting to create another property with new type and replace it. But then I have to update old documents which I am not interested into. I tried to use PUT api as below but I never works.
PUT /indexName/T1/_mapping -d
{
"T1" : {
"properties" : {
"prop1" : {
"type" : "keyword"
}
}
}
}
Is there any way to achieve this?
Mapping cannot be modified, hence the PUT api you have used will not work. The new index will have to be created with the updated mapping to be used and reindexing all the data to new index.
To prevent downtime you can always use alias:
https://www.elastic.co/blog/changing-mapping-with-zero-downtime
A mapping cannot be updated once it is persisted. The only option is to create a new index with the correct mappings and reindex your data using the reindex API provided by ES.
You can read about the reindex API here:
https://www.elastic.co/guide/en/elasticsearch/reference/5.5/docs-reindex.html
In the Current [5.0] elasticsearch doc it was said that
All multi indices API support the following url query string :
ignore_unavailable and allow_no_indices
I delete all exiting indice and try to create a new one with mapping
curl -XDELETE "http://elastic:elastic#127.0.0.1:9200/mail-*?pretty=true"
curl -XPUT "http://elastic:elastic#127.0.0.1:9200/mail-*?ignore_unavailable=true&pretty=true" -d ' {
"mappings": {
"ex": {
"properties": {
...
I got this error :
"request [/mail-*] contains unrecognized parameter: [ignore_unavailable]"
i need to create this mapping because index are created by logstash with a new index every day index => "mail-%{+YYYY.MM.dd}"
if i remove the wilcard in indice name it works !
why i need to do this beacause i use the geoip filter in logstash but the geoip.location is not in the type "geo_point" and kibana tile map doesnt work without this
For the app i created, the indices are generated once in a week. And the type and nature of the data is not varying and that implies, I need the same mapping type for these indices. Is it possible in elasticsearch to apply the same mapping to all the indices as they are created?. This could avoid me the overhead of defining mapping each time the index is created.
Definitely, you can use what is called an index template. Since your mapping type is stable, that's the perfect condition for using index templates.
It's as easy as creating an index. See below, whenever you want to index a document in an index whose name matches my_*, ES will select that template and create the index for you using the given mappings, settings and aliases:
curl -XPUT localhost:9200/_template/template_1 -d '{
"template" : "my_*",
"settings" : {
"number_of_shards" : 1
},
"aliases" : {
"my_alias" : {}
},
"mappings" : {
"my_type" : {
"properties" : {
"my_field": { "type": "string" }
}
}
}
}'
It's basically the technique used by Logstash when it needs to index new logs for each new day in a new daily index.
You can employ index template to address your problem. The official documentation can be found here.
A use case of how to apply the same with examples can be found in this blog
i'm trying the following:
I have a custom Logstash filter and within this filter i have latitude and longitude values. I now want to create a new field (Serv_location) that uses the lat and lon values, so i can create a world-map with these geopoints in kibana. My Problem is that when i create the new field, it interprets it as a digit/number field in logstash and not as a therefore needed geopoint field.
currently my code looks like this to add the field:
event['serv_location'] = [geo_lat.to_f, geo_lng.to_f]
what else do i need to do, to create a geopoint field?
Edit:
Here is the mapping i did:
curl -XPUT 'http://localhost:5601/logstash-2015.04.16/_mapping/location' -d '
{
"map_location" :
{
"properties" : {
"location" : {"type" : "geo_point", "store" : true }
}
}
}
Thanks.
Please write custom mapping for this field and define "type" : "geo_point". Hopefully, this will solve your problem
I'm indexing a document with a property obj_properties, which is a hash of property name -> property value. elasticsearch is inferring that some of the property values are dates, leading to the following error when it encounters a subsequent value for the same property that can't be parsed as a date.
org.elasticsearch.index.mapper.MapperParsingException: failed to parse date field <NON-DATE FIELD within obj_properties>
So, I'd like to disable date detection for obj_properties and anything nested within it. Per
http://elasticsearch-users.115913.n3.nabble.com/Date-Detection-not-always-wanted-tp1638890p1639415.html
(Note, I believe the linked post contains a typo -- the field should be date_formats rather than date_format, but I've tried both ways)
I've created the following mapping
mapping do
indexes :name
indexes :obj_properties, type: "object", date_formats: "none"
end
but I continue to receive the same exception. The properties in obj_properties are not known ahead of time, so it's not possible to create an exhaustive mapping of types. Any ideas? Is disabling date detection the correct approach?
You can turn off date detection for a particular type by specifying it in the mapping:
curl -XPUT 'http://127.0.0.1:9200/myindex/?pretty=1' -d '
{
"mappings" : {
"mytype" : {
"date_detection" : 0
}
}
}
'
or for all types in an index by specifying it in the default mapping:
curl -XPUT 'http://127.0.0.1:9200/myindex/?pretty=1' -d '
{
"mappings" : {
"_default_" : {
"date_detection" : 0
}
}
}
'
mapping(date_detection: false) do
indexes :name
indexes :obj_properties, type: "object"
end
then curl 'http://127.0.0.1:9200/myindex/_mapping?pretty=1' will include date_detection = false mentioned here
Although i believe this applies to the entire index - not a particular field