Logstash Grok Parsing Issue - elasticsearch

Question

I am using Logstash to read some log file.
Here are some data sources records
<2016-07-07 00:31:01> Start
<2016-07-07 00:31:59> Warning - Export_Sysem 6 (1) => No records to be exported
<2016-07-07 00:32:22> Export2CICAP (04) => Export PO : 34 record(s)
<2016-07-07 00:32:22> Export2CICAP (04) => Export CO : 87 record(s)
<2016-07-07 00:32:22> Export2CICAP (04) => Export FC
This is my conf file
grok{
match => {"message" => [
'<%{TIMESTAMP_ISO8601:Timestamp}> (%{WORD:Level} - )%{NOTSPACE:Job_Code} => %{GREEDYDATA:message}',
'<%{TIMESTAMP_ISO8601:Timestamp}> %{WORD:Parameter} - %{GREEDYDATA:Message}',
'<%{TIMESTAMP_ISO8601:Timestamp}> %{WORD:Status}',
]}
}
This is part of my output
{
"message" => "??2016-07-07 00:31:01> Start\r?",
"#version" => "1",
"#timestamp" => "2016-07-08T03:22:01.076Z",
"path" => "C:/CIGNA/Export.log",
"host" => "SIMSPad",
"type" => "txt",
"tags" => [
[0] "_grokparsefailure"
]
}
{
"message" => "<2016-07-07 00:31:59> Warning - Export_Sysem 6 (1) => No records to be exported\r?",
"#version" => "1",
"#timestamp" => "2016-07-06T16:31:59.000Z",
"path" => "C:/CIGNA/Export.log",
"host" => "SIMSPad",
"type" => "txt",
"Timestamp" => "2016-07-07 00:31:59",
"Parameter" => "Warning",
"Message" => "Export_Sysem 6 (1) => No records to be exported\r?"
}
{
"message" => "<2016-07-07 00:32:22> Export2CICAP (04) => Export CO : 87 record(s)\r?",
"#version" => "1",
"#timestamp" => "2016-07-06T16:32:22.000Z",
"path" => "C:/CIGNA/Export.log",
"host" => "SIMSPad",
"type" => "txt",
"Timestamp" => "2016-07-07 00:32:22",
"Status" => "Export2CICAP"
}
As seen from the output, the first output message has a grok parsing error and the other 2 outcomes did not fully parse the message. How should I modify my grok statement so it can fully parse the message?

Answer 1

For the first message, the problem comes from the two ?? that do no appear in the pattern, thus creating the _grokparsefailure.
The second and third message are not fully parsed because the first two pattern do not match the messages and so the message are parsed by the last pattern.
For the second message, if you wish to parse it with the first pattern (<%{TIMESTAMP_ISO8601:Timestamp}> (%{WORD:Level} - )%{NOTSPACE:Job_Code} => %{GREEDYDATA:message}), your pattern is false:
() around %{WORD:Level} - that do not appear in the log.
There is a space missing between :Timestamp}> and %{WORD:Level}. In the log there is two and only one in the pattern. Note that you can use %{SPACE} to avoid this problem (since %{SPACE} will match any number of space)
The %{NOTSPACE:Job_Code} match a sequence of character without any space, but there is a space in Export_Sysem 6 (1), so the Job_Code will be Export_Sysem and the => in the pattern will prevent successful matching with the first pattern.
Correct pattern :
<%{TIMESTAMP_ISO8601:Timestamp}> %{WORD:Level} - %{DATA:Job_Code} => %{GREEDYDATA:message}
For the third message, I don't see which pattern should be used.
If you add more details, I'll update my answer.
For reference: grok pattern definitions