Logstash Aggregate filter, add information to next lines - events

Question

I'm writing a logstash 2.4.0 configuration to go through HTTP logs.
We'd like to have the PORT that is passed in the Header field to be included in the Line fields below.
There is no specific end-event defined. Although I have tried adding an end event as well.
The input log file I'm currently using is:
HEADER 9200
LINE 1 2016-10-05 08:39:00 Some log data
LINE 2 2016-10-05 08:40:00 Some other log data
FOOTER
HEADER 9300
LINE 4 2016-11-05 08:39:00 Some log data in another log
LINE 5 2016-11-05 08:40:00 Some other log data in another log
FOOTER
I would like to have an output like this:
The Server_port fields are currently missing from the output
{"message" => "HEADER 9200",
"#version" => "1",
"#timestamp" => "2016-11-15T11:17:18.425Z",
"path" => "test.log",
"host" => "hostname",
"type" => "event",
"env" => "test",
"port" => 9200,
"tags" => [[0] "Header"] }
{"message" => "LINE 1 2016-10-05 08:39:00 Some log data",
"#version" => "1",
"#timestamp" => "2016-11-15T11:17:20.186Z",
"path" => "test.log",
"host" => "hostname",
"type" => "event",
"env" => "test",
"logMessage" => "1 2016-10-05 08:39:00 Some log data",
"Server_port" => 9200,
"tags" => [[0] "Line"]}
{"message" => "LINE 2 2016-10-05 08:40:00 Some other log data",
"#version" => "1",<
"#timestamp" => "2016-11-15T11:17:20.192Z",
"path" => "test.log",
"host" => "hostname",
"type" => "event",
"env" => "test",
"logMessage" => "2 2016-10-05 08:40:00 Some other log data",
"Server_port" => 9200,
"tags" => [[0] "Line"]}
{"message" => "FOOTER",
"#version" => "1",
"#timestamp" => "2016-11-15T11:17:20.195Z",
"path" => "test.log",
"host" => "hostname",
"type" => "event",
"env" => "test",
"tags" => [[0] "Footer"]}
After trying out different things, the configuration I'm currently using is as follows, with a hardcoded taskid='abcd' for testing:
input{ file{ path => "test.log"
start_position => "beginning"
sincedb_path => "/dev/null"
ignore_older => 0
type => "event"
add_field => { "env" => "test"} }
}
filter{
grok {
break_on_match => false
tag_on_failure => []
match => {"message" => ["^HEADER%{SPACE}%{INT:port:int}"]}
add_tag => ["Header"]
}
grok {
break_on_match => false
tag_on_failure => []
match => {"message" => "^LINE%{SPACE}%{GREEDYDATA:logMessage}"}
add_tag => ["Line"]
}
grok {
break_on_match => false
tag_on_failure => []
match => {"message" => "^FOOTER"}
add_tag => ["Footer"]
}
if "Header" in [tags]{
aggregate{
task_id => "abcd"
code => "map['server_port'] ||= 0; map['server_port']=event['port']"
push_map_as_event_on_timeout => true
push_previous_map_as_event => true
map_action => "create"
}
}
elseif "Line" in [tags]{
aggregate{
task_id => "abcd"
code => "event.set('server_port',map['server_port'])"
map_action => "update"
}
}
else if "Footer" in [tags]{
aggregate{
task_id => "abcd"
code => "event.set('server_port',map['server_port'])"
map_action => "update"
end_of_task => true
timeout => 120
}
}
}
output {
stdout { codec => rubydebug }
}
While this config runs without errors it's not creating the server_port fields.
Where am I going wrong?

Answer 1

After fiddling around some more I have a working test case.
I've changed the configuration as follows:
grok {
break_on_match => false
tag_on_failure => []
match => {
"message" => ["^HEADER%{SPACE}%{INT:taskid:int}%{SPACE}%{INT:port:int}"]
}
add_tag => ["Header"]
}
and
if "Header" in [tags]{
aggregate{
task_id => "%{taskid}"
code => "map['port']=event.get('port')"
map_action => "create"
}
}
elseif "Line" in [tags]{
aggregate{
task_id =>"%{taskid}"
code => "event.set('port',map['port'])"
map_action => "update"
}
}
else if "Footer" in [tags]{
aggregate{
task_id => "%{taskid}"
code => "event.set('port',map['port'])"
map_action => "update"
end_of_task => true
timeout => 120
}
}
And added a task id field to the logs:
HEADER 123 9200
LINE 123 2016-10-05 08:39:00 Some log data