I use logstash-logback-encoder to send java log files to logstash, and then to elasticsearch. To parse the message in java log, I use following filter to dissect message
input {
file {
path => "/Users/MacBook-201965/Work/java/logs/oauth-logstash.log"
start_position => "beginning"
codec => "json"
}
}
filter {
if "EXECUTION_TIME" in [tags] {
dissect {
mapping => {
"message" => "%{endpoint} timeMillis:[%{execution_time_millis}] data:%{additional_data}"
}
}
mutate {
convert => { "execution_time_millis" => "integer" }
}
}
}
output {
elasticsearch {
hosts => "localhost:9200"
index => "elk-%{+YYYY}"
document_type => "log"
}
stdout {
codec => json
}
}
It dissect the message so I can get value of execution_time_millis. However the data type is string. I created the index using Kibana index pattern. How can I change the data type of execution_time_millis into long?
Here is the sample json message from logback
{
"message":"/tests/{id} timeMillis:[142] data:2282||0:0:0:0:0:0:0:1",
"logger_name":"com.timpamungkas.oauth.client.controller.ElkController",
"level_value":20000,
"endpoint":"/tests/{id}",
"execution_time_millis":"142",
"#version":1,
"host":"macbook201965s-MacBook-Air.local",
"thread_name":"http-nio-8080-exec-7",
"path":"/Users/MacBook-201965/Work/java/logs/oauth-logstash.log",
"#timestamp":"2018-01-04T11:20:20.100Z",
"level":"INFO",
"tags":[
"EXECUTION_TIME"
],
"additional_data":"2282||0:0:0:0:0:0:0:1"
}{
"message":"/tests/{id} timeMillis:[110] data:2280||0:0:0:0:0:0:0:1",
"logger_name":"com.timpamungkas.oauth.client.controller.ElkController",
"level_value":20000,
"endpoint":"/tests/{id}",
"execution_time_millis":"110",
"#version":1,
"host":"macbook201965s-MacBook-Air.local",
"thread_name":"http-nio-8080-exec-5",
"path":"/Users/MacBook-201965/Work/java/logs/oauth-logstash.log",
"#timestamp":"2018-01-04T11:20:19.780Z",
"level":"INFO",
"tags":[
"EXECUTION_TIME"
],
"additional_data":"2280||0:0:0:0:0:0:0:1"
}
Thank you
If you have already indexed the documents, you'll have to reindex the data after changing the datatype of any field.
However, you can use something like this to change the type of millis from string to integer. (long is not supported in this)
https://www.elastic.co/guide/en/logstash/current/plugins-filters-mutate.html#plugins-filters-mutate-convert
Also, try defining elasticsearch template before creating index if are going to add multiple index with index names having some regex pattern.Else, you can define your index format beforehand too an then start indexing.
Related
I have a field traceinfo.duration in my webapplog. ES maps it as a string, but i want to change it's field type to integer. My logstash.conf contains following filter section:
filter {
if "webapp-log" in [tags] {
json { source => "message" }
mutate {
convert => {"[traceinfo][duration]" => "integer"}
}
mutate {
remove_field => ["[beat][hostname]","[beat][name]"]
}
}
}
I am creating a new index with this configuration to test it. But my field type in kibana is still string for traceinfo.duration field. My logstash version is 5.3.0. Please help
I copied
{"name":"myapp","hostname":"banana.local","pid":40161,"level":30,"msg":"hi","time":"2013-01-04T18:46:23.851Z","v":0}
from https://github.com/trentm/node-bunyan and save it as my logs.json. I am trying to import only two fields (name and msg) to ElasticSearch via LogStash. The problem is that I depend on a sort of filter that I am not able to accomplish. Well I have successfully imported such line as a single message but certainly it is not worth in my real case.
That said, how can I import only name and msg to ElasticSearch? I tested several alternatives using http://grokdebug.herokuapp.com/ to reach an useful filter with no success at all.
For instance, %{GREEDYDATA:message} will bring the entire line as an unique message but how to split it and ignore all other than name and msg fields?
At the end, I am planing to use here:
input {
file {
type => "my_type"
path => [ "/home/logs/logs.log" ]
codec => "json"
}
}
filter {
grok {
match => { "message" => "data=%{GREEDYDATA:request}"}
}
#### some extra lines here probably
}
output
{
elasticsearch {
codec => json
hosts => "http://127.0.0.1:9200"
index => "indextest"
}
stdout { codec => rubydebug }
}
I have just gone through the list of available Logstash filters. The prune filter should match your need.
Assume you have installed the prune filter, your config file should look like:
input {
file {
type => "my_type"
path => [ "/home/logs/logs.log" ]
codec => "json"
}
}
filter {
prune {
whitelist_names => [
"#timestamp",
"type",
"name",
"msg"
]
}
}
output {
elasticsearch {
codec => json
hosts => "http://127.0.0.1:9200"
index => "indextest"
}
stdout { codec => rubydebug }
}
Please be noted that you will want to keep type for Elasticsearch to index it into a correct type. #timestamp is required if you will view the data on Kibana.
I used the filter plugin "rename" to rename a field in my events. Logstash didn't show any error on restart. But there doesn't seem to be any changes to the field name(checked in sense plugin). I added "rename" after indexing the file. How can I rename a field after indexing? I'm using LS-2.0, ES-2.0. This is a piece of my logstash.conf file:
filter {
grok {
match => {"message" => "^(?<EventTime>[0-9 \-\:]*)\s(?<MovieName>[\w.\-\']*)\s(?<Rating>[\d.]+)\s(?<NoOfDownloads>\d+)\s(?<NoOfViews>\d+)" }
}
mutate {
convert => {"Rating" => "float"}
}
mutate {
convert => {"NoOfDownloads" => "integer"}
}
mutate {
convert => {"NoOfViews" => "integer"}
}
mutate{
rename => { "NoOfViews" => "TotalViews" }
}
> }
You need to either reindex or create elastic search alias for the already indexed data. Direct renaming of the field is not possible on indexed data.
The following is the configuration of logstash. When I input log into logstash, it works well as expected. All the field can be accepted by elasticsearch and the value and type of all fields is correct. However, when I view the log in kinana, it says that the cost field is not indexed so that it can't been visualized. While all the string fields are indexed. I want to visualize my float field. Anyone know what's the problem?
input {
syslog {
facility_labels=>["local0"]
port=>515
}
stdin {}
}
filter{
grok {
overwrite => ["host", "message"]
match => { "message" => " %{BASE10NUM:cost} %{GREEDYDATA:message}" }
}
mutate {
convert => { "cost" => "float" }
}
}
output {
stdout{
codec=>rubydebug
}
elasticsearch{ }
}
Kibana doesn't autoreload a new field from Elastic Search. You need to reload it manually.
So you go in the Settings tab, select you index and reload the fields
ELK Stack has been successfully setup.
using grokdebug.herokuapp.com
my gork patterns are also valid and getting Dumped into ElasticSearch
filter {
if [type] == "some_log" {
grok {
match => { "message" => '%{WORD:word_1} %{TIME:time_1} %{DATE:date_1} %{NUMBER:number_1}'
}
overwrite => "message"
}
}
}
This grok parsing of input is completely correct.
and output is
output {
elasticsearch {
protocol => "http"
}
}
Problem is all the dumped variables are of String Type.
How to get them logged into their respective type in ElasticSearch ( Correct Mapping Type)
time_1, date_1 and number_1 all has same type which is of type
"time_1":{
"type":"string",
"norms":{
"enabled":false
},
"fields":{
"raw":{
"type":"string",
"index":"not_analyzed",
"ignore_above":256
}
}
}
I want date_1 to be indexed as Date Type, number_1 to be indexed as Number type in Elastic search.
PS: Is it possible to do that ?? determine the Type of Elasticsearch field from Logstash.
OR - How to send those field with proper type to ElasticSearch.
Thanks
In your grok pattern, use the form %{PATTERN:field:datatype} to turn the captured fields into something other than strings. Valid data types are "int" and "float". In your case you'd e.g. use %{NUMBER:number_1:int} to turn your number_1 field into an integer.
See the grok filter documentation under Grok Basics.
Another option is to use the mutate filter to convert the type of existing fields:
mutate {
convert => ["name-of-field", "integer"]
}
Related:
Data type conversion using logstash grok
Elasticsearch converting a string to number.
You can try to convert all the fields with ruby plugin.
In this example we combine the time_1 and date_1 together and convert them to Date format.
input {
stdin{}
}
filter {
grok {
match => [ "message" , "%{WORD:word_1} %{TIME:time_1} %{DATE:date_1} %{NUMBER:number_1}"]
overwrite => "message"
}
ruby {
code => "
datetime = event['time_1'] + ' ' + event['date_1']
event['datetime'] = Time.strptime(datetime,'%H:%M:%S %d-%m-%Y')
event['number_1'] = event['number_1'].to_i
"
}
}
output {
stdout { codec => rubydebug }
}
If you have another type that need to convert, you can try to find ruby api to convert them. Hope this can help you.