Logstash Grok Filter Apache Access Log - filter

Question

I have been looking around here and there, but could not find the working resolution. I try to use Grok Filter inside the Logstash config file to filter Apache-Access log file. The log message looks like this: {"message":"00.00.0.000 - - [dd/mm/YYYY:hh:mm:ii +0000] \"GET /index.html HTTP/1.1\" 200 00"}.
On this moment I could only filter the client ip by using grok { match => [ "message", "%{IP:client_ip}" ] }.
I want to filter:
- The GET method,
- requested page (index.html),
- HTTP/1.1\,
- server response 200
- the last number 00 after 200 inside the message body
Please note that none of these does not work for me :
grok { match => { "message" => "%{COMBINEDAPACHELOG}" } }
or
grok { match => [ "message", "%{COMBINEDAPACHELOG}" ] }

Answer 1

Use the Grok Debugger to get an exact match on your log format. Its the only way.
http://grokdebug.herokuapp.com/

Answer 2

grok {
match => [ "message", "%{IP:client_ip} %{USER:ident} %{USER:auth} \[%{HTTPDATE:apache_timestamp}\] \"%{WORD:method} /%{NOTSPACE:request_page} HTTP/%{NUMBER:http_version}\" %{NUMBER:server_response} " ]
}

Answer 3

Use the following:
filter {
grok {
match => { "message" => "%{COMMONAPACHELOG}" }
}
}
As you can see from your pattern COMBINEDAPACHELOG would fail because there are some missing components:
COMBINEDAPACHELOG %{COMMONAPACHELOG} %{QS:referrer} %{QS:agent}
https://github.com/elastic/logstash/blob/v1.4.2/patterns/grok-patterns

Answer 4

You can use COMBINEDAPACHELOG pattern for this,
%{IPORHOST:clientip} %{USER:ident} %{USER:auth} \[%{HTTPDATE:timestamp}\] "(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})" %{NUMBER:response} (?:%{NUMBER:bytes}|-) %{QS:referrer} %{QS:agent}
for instance, consider this sample apache log
111.222.333.123 HOME - [01/Feb/1998:01:08:46 -0800] "GET /bannerad/ad.htm HTTP/1.0" 200 28083
"http://www.referrer.com/bannerad/ba_intro.htm" "Mozilla/4.01
(Macintosh; I; PPC)"
above filter will produce,
{
"clientip": [
[
"111.222.333.123"
]
],
"HOSTNAME": [
[
"111.222.333.123"
]
],
"IP": [
[
null
]
],
"IPV6": [
[
null
]
],
"IPV4": [
[
null
]
],
"ident": [
[
"HOME"
]
],
"USERNAME": [
[
"HOME",
"-"
]
],
"auth": [
[
"-"
]
],
"timestamp": [
[
"01/Feb/1998:01:08:46 -0800"
]
],
"MONTHDAY": [
[
"01"
]
],
"MONTH": [
[
"Feb"
]
],
"YEAR": [
[
"1998"
]
],
"TIME": [
[
"01:08:46"
]
],
"HOUR": [
[
"01"
]
],
"MINUTE": [
[
"08"
]
],
"SECOND": [
[
"46"
]
],
"INT": [
[
"-0800"
]
],
"verb": [
[
"GET"
]
],
"request": [
[
"/bannerad/ad.htm"
]
],
"httpversion": [
[
"1.0"
]
],
"BASE10NUM": [
[
"1.0",
"200",
"28083"
]
],
"rawrequest": [
[
null
]
],
"response": [
[
"200"
]
],
"bytes": [
[
"28083"
]
],
"referrer": [
[
""http://www.referrer.com/bannerad/ba_intro.htm""
]
],
"QUOTEDSTRING": [
[
""http://www.referrer.com/bannerad/ba_intro.htm"",
""Mozilla/4.01 (Macintosh; I; PPC)""
]
],
"agent": [
[
""Mozilla/4.01 (Macintosh; I; PPC)""
]
]
}
can be tested here,
https://grokdebug.herokuapp.com/