Grok doesn't match multiline log entries properly? - elasticsearch

Question

I've been going at this for weeks now and I can't seem to wrap my head around what's wrong about this.
I'm trying to get all of these lines to fit into a multiline match with grok, but it only picks up the last one, and even discards the digit at the beginning of the line.
11:31:03.936 < : 1> 5: Load times per type (ms):
12: aaaaaa.aaaaaaaaa.aaaaaaa.aaaaaaa
1: bbbb.bbbb.bbbbbbbbbbbbb.bbbbbbbbb
3: cccc.cccccccc.ccccccccccc.cccccc
64: ddd.dddddddddddd.ddddddd.ddddddd
Expected result:
message_processed = Load times per type (ms):
12: aaaaaa.aaaaaaaaa.aaaaaaa.aaaaaaa
1: bbbb.bbbb.bbbbbbbbbbbbb.bbbbbbbbb
3: cccc.cccccccc.ccccccccccc.cccccc
64: ddd.dddddddddddd.ddddddd.ddddddd
Actual result:
message_processed = ddd.dddddddddddd.ddddddd.ddddddd
I'm using the following grok pattern:
grok {
match => [ "message" , "%{TIME:time}.*%{NUMBER:loglevel}:\s%{GREEDYDATA:message_processed}" ]
}
It is being shipped to logstash with filebeat on a windows server with the following multi-line config in filebeat.yml:
multiline.pattern: ^[0-9]{2}\:[0-9]{2}\:[0-9]{2}
multiline.negate: true
multiline.match: after
I've tried using (?m) flag but to no avail, and using multi-line codec with filebeat is a no-go according to the official documentation.
What am I doing wrong?

Answer 1

Try the following:
%{TIME:time}.*%{NUMBER:loglevel}:\s(?<message_processed>(.|\‌​r|\n)*)
As pointed out here GREEDYDATA won't match newlines while (?<message>(.|\r|\n)*) will. So the above works for me.
See the results for your example with your multiline config on https://grokconstructor.appspot.com/do/match: