Logstash can't add fields? - elasticsearch

Question

I have been using logstash to read some DB restore logs. Here is some lines of sample records.
07/08/2016 6:33:22.50: START restore database
SQL2540W Restore is successful, however a warning "2539" was encountered
during Database Restore while processing in No Interrupt mode.
07/08/2016 6:33:28.93: END restore database
SQL4406W The DB2 Administration Server was started successfully.
07/08/2016 6:35:35.29: END restart server
connect reset
DB20000I The SQL command completed successfully.
07/08/2016 6:35:38.48: END p:\s6\source\system\CMD\res_uw.cmd
Here is the filter part of my conf file.
if ([message] =~ /Backup successful/){
grok{
match => {"message" => ['%{GREEDYDATA:Message}'] }
}
mutate {
add_tag => "send_to_es"
add_field => {"Timestamp" => "%{GREEDYDATA:DATETIME}"}
}
}
if ([message] =~ /warning "2539"/){
grok{
match => {"message" => ['%{GREEDYDATA:Message}'] }
}
mutate {
add_tag => "send_to_es"
add_field => {"Timestamp" => "%{GREEDYDATA:DATETIME}"}
}
}
if ([message] =~ /(END p:|END P:)/){
grok{
match => {"message" => ['%{GREEDYDATA:DATETIME}:%{SPACE}END%{SPACE}%{GREEDYDATA:Mis}'] }
remove_field => "%{GREEDYDATA:Mis}"
}
mutate {
add_tag => "send_to_es"
}
}
I want to add the data "DATETIME" extracted from the last line of my record to message to other message to index at the same time. However, it could not add the field successfully. The output will become
"message": "SQL2540W Restore is successful, however a warning \"2539\" was encountered \r\r",
"#version": "1",
"#timestamp": "2016-07-12T02:28:52.337Z",
"path": "C:/CIGNA/hkiapp67_db_restore/res_uw.log",
"host": "SIMSPad",
"type": "txt",
"Message": "SQL2540W Restore is successful, however a warning \"2539\" was encountered \r\r",
"Timestamp": "%{GREEDYDATA:DATETIME}",
"tags": [
"send_to_es"
]
How could I solve this?

Answer 1

Logstash, when receiving a line, does not have knowledge of any other line. You'll have to use a multiline codec/filter to regroup all the lines you need with a line with the date. Then you use the grok filter to extract the date and add it to your document.
The configuration of the multiline codec/filter will look like this:
multiline {
pattern => "%{DATE}"
negate => "true"
what => "next"
}
With this, all the lines not beginning with the pattern DATE will be joined with the next line.