Match multiple patterns in Logstash? - elasticsearch

Question

I have two types of error messages in the below format:
[2017-05-25 01:00:00,647][ERROR][marvel.agent.exporter.local] local exporter [default_local] - failed to delete indices
RemoteTransportException[[data-0][10.0.0.8:9300][indices:admin/delete]]; nested: IndexNotFoundException[no such index];
[2017-05-18 00:00:06,339][DEBUG][action.admin.indices.create] [data-2] [data-may-2017,data-apr-2017,data-mar-2017] failed to create
[data-may-2017,data-apr-2017,data-mar-2017]
My logstash configuration is like this:
input {
file {
path => "D:\logstash\logstash-2.4.0\bin\logs.txt"
start_position => "beginning"
codec => multiline {
pattern => "^\[%{TIMESTAMP_ISO8601:TIMESTAMP}\]"
negate => true
what => "previous"
}
}
}
filter {
grok {
match => [ "message", "(?m)^\[%{TIMESTAMP_ISO8601:TIMESTAMP}\]\[%{LOGLEVEL:LEVEL}%{SPACE}\]\[%{DATA:ERRORTYPE}\]%{SPACE}\[%{DATA:SERVERNAME}\]%{SPACE}(?<ERRORMESSAGE>(.|\r|\n)*)", "message", "(?m)^\[%{TIMESTAMP_ISO8601:TIMESTAMP}\]\[%{LOGLEVEL:LEVEL}%{SPACE}\]\[%{DATA:ERRORTYPE}%{SPACE}\]%{SPACE}(?<ERRORMESSAGE>(.|\r|\n)*)"]
}
}
output {
stdout { codec => rubydebug }
}
For Both the logs it is taking only the first grok pattern. Why it is not taking the second one?

Answer 1

Seems my first grok pattern is matching all the logs , so thats why logstash is taking only the first pattern. So that i had used the below config with if condition which is working fine.
input {
file {
path => "D:\logstash\logstash-2.4.0\bin\logs.txt"
start_position => "beginning"
type => "log"
codec => multiline {
pattern => "^\[%{TIMESTAMP_ISO8601:TIMESTAMP}\]"
negate => true
what => "previous"
}
}
}
filter {
if [type] == "log" {
grok {
match => [ "message", "(?m)^\[%{TIMESTAMP_ISO8601:TIMESTAMP}\]\[%{LOGLEVEL:LEVEL}%{SPACE}\]\[%{DATA:ERRORTYPE}%{SPACE}\]%{SPACE}(?<ERRORMESSAGE>(.|\r|\n)*)"]
}
# DEBUG Logs
if "grokked" not in [tags] and "DEBUG" == [LEVEL] {
grok { match => [ "ERRORMESSAGE", "(?m)^\[%{DATA:SERVERNAME}\]" ]
add_tag => [ "Debug Logs", "grokked" ]
tag_on_failure => [ ]
}
}
}
}
output {
stdout { codec => rubydebug }
}

Answer 2

Your question was:
Why it is not taking the second one?
Answer is here:
filter {
grok {
match => [ "message", "(?m)^\[%{TIMESTAMP_ISO8601:TIMESTAMP}\]\[%{LOGLEVEL:LEVEL}%{SPACE}\]\[%{DATA:ERRORTYPE}\]%{SPACE}\[%{DATA:SERVERNAME}\]%{SPACE}(?<ERRORMESSAGE>(.|\r|\n)*)", "message", "(?m)^\[%{TIMESTAMP_ISO8601:TIMESTAMP}\]\[%{LOGLEVEL:LEVEL}%{SPACE}\]\[%{DATA:ERRORTYPE}%{SPACE}\]%{SPACE}(?<ERRORMESSAGE>(.|\r|\n)*)"]
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------^
}
}
You don't have to specify source multiple times, just once.
What you did now was:
["message", "pattern", "message", "pattern"]
While in reality it has to be:
["message", "pattern", "pattern", ..., "pattern"]