custom parse with logstash and elastic search - elasticsearch

Question

I am new to logstash !
I configured and everything is working fine - so far.
My log files comes as:
2014-04-27 16:24:43 DEBUG b45e66 T+561 10.31.166.155 /v1/XXX<!session> XXX requested for category_ids: only_pro: XXX_ids:14525
If i use the following conf file:
input { file { path => "/logs/*_log" }} output { elasticsearch { host => localhost } }
It will place the following in the ES:
{
_index: "logstash-2014.04.28",
_type: "logs",
_id: "WIoUbIvCQOqnz4tMZzMohg",
_score: 1,
_source: {
message: "2014-04-27 16:24:43 DEBUG b45e66 T+561 10.31.166.155 This is my log !",
#version: "1",
#timestamp: "2014-04-28T14:25:52.165Z",
host: "MYCOMPUTER",
path: "\logs\xxx_app.log"
}
}
How do i take the string in my log so the entire text wont be at _source.message ?
e.g: I wish i could parse it to something like:
{
_index: "logstash-2014.04.28",
_type: "logs",
_id: "WIoUbIvCQOqnz4tMZzMohg",
_score: 1,
_source: {
logLevel: "DEBUG",
messageId: "b45e66",
sendFrom: "10.31.166.155",
logTimestamp: "2014-04-27 16:24:43",
message: "This is my log !",
#version: "1",
#timestamp: "2014-04-28T14:25:52.165Z",
host: "MYCOMPUTER",
path: "\logs\xxx_app.log"
}
}

Answer 1

You need to parse it through a filter, e.g. the grok filter. This can be quite a bit tricky, so be patient and try, try, try. And have a look at the predefined patterns, too.
A start for your message would be
%{DATESTAMP} %{WORD:logLevel} %{WORD:messageId} %{GREEDYDATA:someString} %{IP}
The grokdebugger is an extremely helpful tool for your assistance.
When done, your config should look like
input {
stdin {}
}
filter {
grok {
match => { "message" => "%{DATESTAMP} %{WORD:logLevel} %{WORD:messageId} %{GREEDYDATA:someString} %{IP}" }
}
}
output {
elasticsearch { host => localhost }
}