Parse integer and show in timeline in Kibana - elasticsearch

Question

Here is the Kibana UI and I want to parse some Integer in the message. The number in the end of message is the process time for one method and I what to visualize the average process time by hour in Kibana. Is that possible?
I tried some conf in logstash:
filter{
json{
source => "message"
}
grok {
match => {
"message" => "^Finish validate %{NUMBER:cto_validate_time}$"
}
}
grok {
match => {
"message" => "^Finish customize %{NUMBER:cto_customize_time}$"
}
}
}
It works. But when I create the timechart I can not get the new field.

Answer 1

Since you don't care about performance issues, you may create a scripted field named process_time in your index pattern with the following painless code. What it does is simply take the last numerical value from your message field.
def m = /.*\s(\d+)$/.matcher(doc['message.keyword'].value);
if ( m.matches() ) {
return m.group(1)
} else {
return 0
}
Then you can build a chart to show the average process time by hour. Go to the Visualize tab and create a new vertical bar chart. On the Y-Axis you'll create an Average aggregation on the process_time field and on the X-Axis you'll use a Date histogram aggregation on your timestamp field. A sample is shown below:
Note: You also need to add the following line in your elasticsearch.yml file and restart ES:
script.painless.regex.enabled: true
UPDATE
If you want to do it via Logstash you can add the following grok filter
filter{
grok {
match => {
"message" => "^Finish customize in controller %{NUMBER:cto_customize_time}$"
}
}
mutate {
convert => { "cto_customize_time" => "integer" }
}
}