Summary: I want to attach a TTL field with the logs in logstash and send them over to the Elastic search.
I have already gone through the documentation but could not get much of it, since it is not very clear.
This is my config file in logstash.
input {
stdin {
type => "stdin-type"
}
}
output {
stdout { debug => true debug_format => "json"}
elasticsearch {}
}
Now suppose that for each log that is read, I want to attach a TTL with it for say, 5 days.
I know how to activate the TTL option in elastic search. But What changes will I have to make in the elastic search configuration files is not very clear to me.
The documentation asks to look for the mappings folder, but there is none in the elastic search download folder.
Looking for an expert help.
Have a look here if you want to put the mapping on file system. You have to go to the config folder and create here a folder called mappings, and another one with the name of the index within mappings. Since logstash creates by default an index per day, you'd better use the _default name for the folder, so that the mapping will be applied to all indexes.
The file that you create under that folder must have the name of the type you want to apply the mapping to. I don't remember exactly what type logstash uses, thus I would use the _default_ mapping definition. Just call the file _default_.json and put the following content in it:
{
"_default_" : {
"_ttl" : { "enabled" : true }
}
}
As you can see the name of the type must appear in both the filename and in its content.
Otherwise, you could avoid putting stuff on file system. You could create an index template containing your custom mapping, like the following:
{
"template" : "logstash-*",
"mappings" : {
"_default_" : {
"_ttl" : { "enabled" : true }
}
}
}
The mapping will then be applied to all the indices whose name matches the template pattern. If you use the _default_ mapping definition the mapping will be applied as default to all the types that are going to be created.
Related
I'm fairly new to ELK (7.10), and I would like to know how to create a map chart using GeoIP mapping.
I already have logs parsed and one field is "remote_ip" which I want to view on a map chart.
I've seen lots of instructions on how to do this but most are out of date and do not apply to my version which is 7.10. I'm using filebeats/logstash/kibana/elasticsearch.
Could someone show me the high level steps required to do this? Or point me to a detailed guide appropriate to my version? I have no idea how to begin.
I'm assuming those IP addresses are public so you can geocode them. Since your logs are already indexed, you now need to geocode them. Here is how to do it.
First, you need to modify your mapping to add a geo_point field, like this:
PUT your-index/_mapping
{
"properties": {
"remote_location": {
"type": "geo_point"
}
}
}
Once you've added that new field to your mapping, you can update your index to geocode the IP addresses. For that, you first need to create an ingest pipeline with the geoip processor:
PUT _ingest/pipeline/geoip
{
"description" : "Geocode IP address",
"processors" : [
{
"geoip" : {
"field" : "remote_ip",
"target_field": "remote_location"
}
}
]
}
Once this ingest pipeline is created you can use it to update your index using the _update_by_query endpoint like this:
POST your-index/_update_by_query?pipeline=geoip
Once the update is over, you can go into Kibana, create an index pattern and then go to Analytics > Maps and create your map.
I have index mapping for type 'T1' as below:
"T1" : {
"properties" : {
"prop1" : {
"type" : "text"
}
}
}
And now I want to change the type of prop1 from text to keyword. I don't want to delete index. I have also read people suggesting to create another property with new type and replace it. But then I have to update old documents which I am not interested into. I tried to use PUT api as below but I never works.
PUT /indexName/T1/_mapping -d
{
"T1" : {
"properties" : {
"prop1" : {
"type" : "keyword"
}
}
}
}
Is there any way to achieve this?
Mapping cannot be modified, hence the PUT api you have used will not work. The new index will have to be created with the updated mapping to be used and reindexing all the data to new index.
To prevent downtime you can always use alias:
https://www.elastic.co/blog/changing-mapping-with-zero-downtime
A mapping cannot be updated once it is persisted. The only option is to create a new index with the correct mappings and reindex your data using the reindex API provided by ES.
You can read about the reindex API here:
https://www.elastic.co/guide/en/elasticsearch/reference/5.5/docs-reindex.html
Elastic 1.x allows to define in mapping default path for extracting required routing field, e.g.:
{
"comment" : {
"_routing" : {
"required" : true,
"path" : "blog.post_id"
}
}
}
Is that possible to add that field on the fly, without a downtime?
So the mapping was previously defined as:
{
"comment" : {
"_routing" : {
"required" : true
}
}
}
The update will not work. Even if the command is acknowledged, the update will not be applied.
You need to reindex the documents, as well. If that path changes and the values are different this means that documents could have ended up in a different shard than in which they are now. So, assuming that the change would have been possible, you are basically changing the hash that the documents can be routed and also GETed (gotten) from shards and it will be a mess.
For the app i created, the indices are generated once in a week. And the type and nature of the data is not varying and that implies, I need the same mapping type for these indices. Is it possible in elasticsearch to apply the same mapping to all the indices as they are created?. This could avoid me the overhead of defining mapping each time the index is created.
Definitely, you can use what is called an index template. Since your mapping type is stable, that's the perfect condition for using index templates.
It's as easy as creating an index. See below, whenever you want to index a document in an index whose name matches my_*, ES will select that template and create the index for you using the given mappings, settings and aliases:
curl -XPUT localhost:9200/_template/template_1 -d '{
"template" : "my_*",
"settings" : {
"number_of_shards" : 1
},
"aliases" : {
"my_alias" : {}
},
"mappings" : {
"my_type" : {
"properties" : {
"my_field": { "type": "string" }
}
}
}
}'
It's basically the technique used by Logstash when it needs to index new logs for each new day in a new daily index.
You can employ index template to address your problem. The official documentation can be found here.
A use case of how to apply the same with examples can be found in this blog
Am using Elasticsearch 1.4. My requirement is I will have data every hour and that needs to be uploaded. So the approach that I have taken is to create an index - "demo" and upload the data. So, the first hour data gets inserted. Now, my question is how to append the subsequent hours data into this index.
PUT /demo/userdetails/1
{
"user" : "kimchy",
"message" : "trying out Elastic Search"
}
Now I am trying to add another document
{"user": "swarna","message":"hi"}
You simply need to PUT the additional documents. In your example above you did
PUT /demo/userdetails/1 { "user" : "kimchy", "message" : "trying out Elastic Search" }
Now you would do this:
PUT /demo/userdetails/2 {"user": "swarna","message":"hi"}
In you command there demo is the index, userdetails is the type, and the number is the document id. If you omit the document id ES will make one up for you.