I am using logstash to ship json data to elasticsearch. But I need to change the key name of a particular key at logstash level before shipping to ElasticSearch.
Is it possible to do so? If yes, do I need to include some plugins for logstash?
Original data:
{"keyA": "dataA", "keyB": "dataB"}
ElasticSearch data:
{"keyC": "dataA", "keyB": "dataB"}
Yes it is possible.
Use the rename configuration option on the mutate filter to rename one or more fields.
By default, it should already come included with Logstash.
Example:
filter {
mutate {
# Renames the 'keyA' field to 'KeyC'
rename => { "keyA" => "keyC" }
}
}
Related
In a PoC that's being done in our project, we are trying out Logstash instead of our own java based indexing module to push data to ElasticSearch. The incoming json data doesn't have an #timestamp field. So when using Logstash, it's adding that field in ISO format. But we already have a specific mapping for that ES index, and it requires us to push the #timestamp in epoch-millis format.
I've tried playing with ruby filters to convert the #timestamp to epoch-millis, but no luck so far. Is there any way we can ingest records to ES through Logstash, with #timestamp being in epoch-millis format?
I'm using logstash 6.5.4 and ES 6.2.2
Update: After trying out the suggestion in the answer, my conf file looks like this:
input { stdin { } }
filter {
ruby {
code => "
epoch_ts = event.timestamp.time.localtime.strftime('%s').to_i
event.set( 'epoch', epoch_ts )
"
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
index => "myindex"
script_type => "inline"
script => 'ctx._source.#timestamp = params.event.get("epoch")'
}
stdout { codec => rubydebug }
}
But still it doesn't work. The #timestampvalue doesn't change at all. Now, I also need to remove that extra field epoch.
this ruby code should work for you:
ruby {
code => "
epoch_ts = event.timestamp.time.localtime.strftime('%s').to_f
event.set( '#timestamp', epoch_ts )
"
}
After quite a while of searching around the web, I finally gave up on this approach. Instead I forced ES to return the #timestamp in epoch_millis using this docvalue_fields approach.
i am new to the ELK stack, I want to use ELK stack to push my logs to elastic so that I can use Kibana on em. Below is the format of my custom log:
Date Time INFO - searchinfo#username#searchQuery#latitude#longitude#client_ip#responseTime
The below is an example of a log that follows the format.
2017-07-04 11:16:10 INFO - searchinfo#null#gate#0.0#0.0#180.179.209.54#598
Now I am using filebeat to push my .log files to logstash and logstash would push that data into elastic.
I need help, writing up a filter for config for logstash that would simply split using the # and then put data into respective fields into elastic index.
How can I do this?
Try to use grok plugin to parse your logs into structured data:
filter {
grok {
match => { "message" => "\A%{TIMESTAMP_ISO8601:timestamp}%{SPACE}%{WORD:var0}%{SPACE}%{NOTSPACE}%{SPACE}(?<searchinfo>[^#]*)#(?<username>[^#]*)#(?<searchQuery>[^#]*)#(?<latitude>[^#]*)#(?<longitude>[^#]*)#(?<client_ip>[^#]*)#(?<responseTime>[^#]*)" }
}
}
You can debug it online:
You need to use a grok filter to parse your log.
You can try with this:
filter {
grok {
match => { "message" => "\A%{TIMESTAMP_ISO8601:timestamp}%{SPACE}%{WORD:var0}%{SPACE}%{NOTSPACE}%{SPACE}(?<var1>[^#]*)#(?<var2>[^#]*)#(?<var3>[^#]*)#(?<var4>[^#]*)#(?<var5>[^#]*)#(?<var6>[^#]*)#(?<var7>[^#]*)" }
}
}
This will parse you log and add fields named var0, var1, etc to the parsed document. You can rename this variables as you prefer.
When I use http input plugin, Logstash adds the following fields to Elasticsearch:
headers.http_accept
headers.content_type
headers.request_path
headers.http_version
headers.request_method
...
How can I remove all these fields starting with headers.?
Since these are all pathed, that means they all are hierarchical under [headers] as far as the logstash configs go. This will probably do wonders for you:
filter {
mutate {
remove_field => [ "headers" ]
}
}
Which should drop the [headers][http_accept], [headers][content_type] and so on fields.
Using Logstash, I want to index documents into Elasticsearch and specify the type, id etc of the document that needs to be indexed. How can I specify those in my config without keeping useless fields in my documents?
Example: I want to specify the id used for insertion:
input {
stdin {
codec => json {}
}
}
output {
elasticsearch { document_id => "%{[id]}" }
}
This will insert the document in Elasticsearch with the id id but the document will keep a redundant field "id" in the mapping. How can I avoid that?
I thought of adding
filter{ mutate { remove_field => "%{[id]}"} }
in the config, but the field is removed and cannot consequently be used as document_id...
Right now this isn't possible. Logstash 1.5 introduces a #metadata field whose contents aren't included in what's eventually sent to the outputs, so you'd be able to create a [#metadata][id] field and refer to that in your output,
output {
elasticsearch { document_id => "%{[#metadata][id]}" }
}
without that field polluting the message payload indexed to Elasticsearch. See the #metadata documentation.
Summary: I want to attach a TTL field with the logs in logstash and send them over to the Elastic search.
I have already gone through the documentation but could not get much of it, since it is not very clear.
This is my config file in logstash.
input {
stdin {
type => "stdin-type"
}
}
output {
stdout { debug => true debug_format => "json"}
elasticsearch {}
}
Now suppose that for each log that is read, I want to attach a TTL with it for say, 5 days.
I know how to activate the TTL option in elastic search. But What changes will I have to make in the elastic search configuration files is not very clear to me.
The documentation asks to look for the mappings folder, but there is none in the elastic search download folder.
Looking for an expert help.
Have a look here if you want to put the mapping on file system. You have to go to the config folder and create here a folder called mappings, and another one with the name of the index within mappings. Since logstash creates by default an index per day, you'd better use the _default name for the folder, so that the mapping will be applied to all indexes.
The file that you create under that folder must have the name of the type you want to apply the mapping to. I don't remember exactly what type logstash uses, thus I would use the _default_ mapping definition. Just call the file _default_.json and put the following content in it:
{
"_default_" : {
"_ttl" : { "enabled" : true }
}
}
As you can see the name of the type must appear in both the filename and in its content.
Otherwise, you could avoid putting stuff on file system. You could create an index template containing your custom mapping, like the following:
{
"template" : "logstash-*",
"mappings" : {
"_default_" : {
"_ttl" : { "enabled" : true }
}
}
}
The mapping will then be applied to all the indices whose name matches the template pattern. If you use the _default_ mapping definition the mapping will be applied as default to all the types that are going to be created.