ELK - How to use different source in logstash - elasticsearch

Question

I have a so far running ELK installation that I want to use to analyse log files from differenct sources:
nginx-logs
auth-logs
and so on...
I am using filebeat to collect content from logfiles and sending it to logstash with this filebeat.yml:
filebeat.inputs:
- type: log
enabled: true
paths:
- /var/log/*.log
- /var/nginx/example_com/logs/
output.logstash:
hosts: ["localhost:5044"]
In logstash I alread configured a grok-section, but only for nginx-logs. This was the only working tutorial I found. So this config receives content from filebeat, filters is (that's what grok is for?) and sends it to elasticsearch.
input {
beats {
port => 5044
}
}
filter {
grok {
patterns_dir => "/etc/logstash/patterns"
match => { "message" => "%{NGINXACCESS}" }
}
}
output {
elasticsearch {
hosts => "localhost:9200"
manage_template => false
index => "%{[#metadata][beat]}-%{[#metadata][version]}-%{+YYYY.MM.dd}"
document_type => "%{[#metadata][type]}"
}
}
That's the content of the one nginx-pattern file I am referencing:
NGUSERNAME [a-zA-Z\.\#\-\+_%]+
NGUSER %{NGUSERNAME}
NGINXACCESS %{IPORHOST:clientip} (?:-|(%{WORD}.%{WORD})) %{USER:ident} \[%{HTTPDATE:timestamp}\] "(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})" %{NUMBER:response} (?:%{NUMBER:bytes}|-) %{QS:referrer} %{QS:agent} %{QS:forwarder}
But I have trouble understanding how to manage different log-data sources. Because now Kibana only displays log content from /var/log, but there is no log data from my particular nginx folder.
What is it, that I am doing wrong here?

Answer 1

Since you are running filebeat, you already have a module available, that process nginx logs filebeat nginx module
This way, you will not need logstash to process the logs, and you only have to point the output directly to elasticsearch.
But, since you are processing multiple paths with different logs, and because elastic stack don't allow to have multiple output forms (logstash + elasticserach), you can set logstash to only process logs that do not come from nginx. This way, and using the module (that comes with sample dashboards) , your logs will do:
Filebeat -> Logstash (from input plugin to output plugin - without any filtering) -> Elasticsearch
If you really want to process the logs on your own, you are in a good path to finish. But right now, all your logs are being process by the grok pattern. So maybe the problem is with your pattern, that processes logs from nginx, and not from nginx in the same way. You can filter the logs in the filter plugin, with something like this:
#if you are using the module
filter {
if [fileset][module] == "nginx" {
}
}
if not, please check different available examples at logstash docs
Another thing you can try, it's add this to you filter. This way, if the grok fails,you will see the log in kibana, but, with the "_grok_parse_error_nginx_error" failure tag.
grok {
patterns_dir => "/etc/logstash/patterns"
match => { "message" => "%{NGINXACCESS}" }
tag_on_failure => [ "_grok_parse_error_nginx_error" ]
}