Merge logs between date/time using filebeat - elasticsearch

Question

I'm trying to push logs to elasticSearch using fileBeat ( No Logstash )
I want to send following log in single message but it get broken into multiple messages, every line becomes separate message
20161014 17:49:09.169 [ERROR] [Thread-2974] some.java.class.:70 - some.java.Exception: write failed. History: [requestHost=123-some.org.com, time=Fri Oct 14 17:49:05 GMT-07:00 2016, exception=java.net.SocketTimeoutException]
[requestHost=123-some.org.com, time=Fri Oct 14 17:49:07 GMT-07:00 2016, exception=java.net.SocketTimeoutException]
[requestHost=123-some.org.com, time=Fri Oct 14 17:49:09 GMT-07:00 2016, exception=java.net.SocketTimeoutException]
Tried 3 times
at java.lang.Thread.run(Thread.java:745)
20161014 17:49:09.169 [ERROR] [Thread-3022]
I want to merge all lines between 2 dates (1st and last line)
Here is my filebeat.yml snippet
paths:
- /test.log
multiline.pattern: '^\[0-9]{8}'
multiline.negate: true
multiline.match: after
I need to know correct regex
I'm trying to solve this without using logstash

Answer 1

Using the following Filebeat configuration with the provided log sample produces two events where each message begins with the date.
I ran ./filebeat -c filebeat.yml -e -v -d "*" with the config below to test. I also tested the pattern on the Go playground.
filebeat.yml:
filebeat:
prospectors:
- paths: ["input.txt"]
multiline:
pattern: '^[0-9]{8}'
negate: true
match: after
output:
console:
pretty: false
Output:
{
"#timestamp": "2016-10-17T14:13:31.292Z",
"beat": {
"hostname": "host.example.com",
"name": "host.example.com",
},
"input_type": "log",
"message": "20161014 17:49:09.169 [ERROR] [Thread-2974] some.java.class.:70 - some.java.Exception: write failed. History: [requestHost=123-some.org.com, time=Fri Oct 14 17:49:05 GMT-07:00 2016, exception=java.net.SocketTimeoutException]\n[requestHost=123-some.org.com, time=Fri Oct 14 17:49:07 GMT-07:00 2016, exception=java.net.SocketTimeoutException]\n[requestHost=123-some.org.com, time=Fri Oct 14 17:49:09 GMT-07:00 2016, exception=java.net.SocketTimeoutException]\n Tried 3 times\n at java.lang.Thread.run(Thread.java:745)",
"offset": 519,
"source": "input.txt",
"type": "log"
}
{
"#timestamp": "2016-10-17T14:17:21.686Z",
"beat": {
"hostname": "host.example.com",
"name": "host.example.com",
},
"input_type": "log",
"message": "20161014 17:49:09.169 [ERROR] [Thread-3022]",
"offset": 563,
"source": "input.txt",
"type": "log"
}