FluentD - Parsing Heroku’s Logplex format - heroku

Question

We’re using FluentD to send data over to our ELK stack. Heroku sends over logs in a BULK format which includes multiple log entries, separated by a new line.
I was wondering if anyone had any experience with splitting incoming http requests in FluentD by newline? I saw examples of this in past versions < 1.0. There are also two Heroku+FluentD plugins, both of which no longer seem to work and are not maintained.
Can I use a parser to split the incoming message into multiple messages and emit each to FluentD, if so, how?
If not, is there a simpler way to get these bulk messages sent from Heroku into FluentD, split by new line?
The bulk log messages Heroku posts look something like this:
83 <40>1 2012-11-30T06:45:29+00:00 host app web.3 - State changed from starting to up 119 <40>1 2012-11-30T06:45:26+00:00 host app web.3 - Starting process with command bundle exec rackup config.ru -p 24405
So in our logging solution, we’re getting multiple rows per entry. We’ve tried multi line parsing, but that doesn’t seem to do the trick.

Answer 1

You can achieve your goal by using the following gem
Install the gem and use one of the following configs
Use this one of your events are separated with a new line
<match *.*>
#type record_splitter
tag splitted.log
input_key message
split_stratgey lines
append_new_line true
remove_new_line true
</match>
or use this one to split the lines with regex
<match *.*>
#type record_splitter
tag splitted.log
input_key message
split_stratgey regex
split_regex /\d+\s<\d+>.+/
</match>
to be able to process the log lines further, you can add another output match for the tag used (example below to send logs to elk)
<filter splitted.log>
...
</filter>
<match splitted.log>
#type rewrite_tag_filter
</match>
<match **.**>
#type elasticsearch
...
</match>