logstash and x-forwarded-for on IIS - elasticsearch

Question

I just built an ELK server on Windows so I'm new to the process. I've read through the docs but am having trouble parsing out my IIS advanced logs, especially x-forwarded-for data as we're behind a load balancer..
My advanced logging is set up to output the data like this:
$date, $time, $s-ip, $cs-uri-stem, $cs-uri-query, $s-port, $cs-username, $c-ip, $X-Forwarded-For, $csUser-Agent, $cs-Referer, $sc-status, $sc-substatus, $sc-win32-status, $time-taken
I set up my logstash.conf like this:
input {
tcp {
host => "localhost"
type => "iis"
port => 5044
}
}
filter {
if [type] == "iis" {
grok {
match => {"message" => "%{TIMESTAMP_ISO8601:log_timestamp} %{IPORHOST:site} %{URIPATH:page} %{NOTSPACE:query_string} %{NUMBER:port} %{NOTSPACE:username} %{IPORHOST:client_host} %{NOTSPACE:useragent} %{NOTSPACE:referer} %{GREEDYDATA:response} %{NUMBER:httpStatusCode:int} %{NUMBER:scSubstatus:int} %{NUMBER:scwin32status:int} %{NUMBER:timeTakenMS:int}"}
}
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
index => "iis"
document_type => "main"
}
}
I don't think this is correct as I'm not getting data. I've scoured the docs but am still having issues and am not sure if there are other steps I need to take, like mapping the fields.
I'm currently using filebeat from one server to push data to my ELK server. I'm not sure if this is the best way as well (maybe nxlog?). We don't want to install logstash on the client machines.
Can someone lend me a hand? It would be GREATLY appreciated!!
Thanks,
George

Answer 1

Since you are using Filebeat then you need to use the beats input and not the tcp input. See the documentation on how to setup Logstash for Beats.
Essentially you need to replace your tcp input with:
input {
beats {
port => 5044
}
}
And inside your Filebeat configuration file, set the document_type to iis so that your filter condition will match.
filebeat:
prospectors:
- paths:
- 'C:\path\to\your\iis\logs\*.log'
document_type: iis