How to add host name for beats input in logstash - elasticsearch

Question

Let me explain my existing structure, I have 4 servers (Web Server, API Server, Database server, SSIS Severs) and installed filebeat and winlog in all four servers and from there I am getting all logs in my logstash, but here is the thing every log I am getting in message body, and for some messages I am getting difficulty to write correct GROK pattern, is there anyway I can get the pattern from Kibana (FYI as of now I am storing all logs in elasticsearch which I can see through Kibana.)
My Logstash config is look like -
1. Api-Pipeline
input {
beats {
host => "IP Address where my filebeat (API Server) is running"
port => 5044
}
}
2. DB Pipeline
input {
beats {
host => "IP Address where my filebeat (Database Server) is running"
port => 5044
}
}
It's working when I used only port and the moment I add host it stopped working. Can anyone help me here.
Below I am trying to achieve
Here I made change, Does it work because I need to write lengthy filters and that's why I wanted to have in separate files
Filebeat.yml on API Server
-----------------------------------------------------------------------------------------
filebeat.inputs:
- type: log
source: 'ApiServerName' // MyAPIServerName(Same Server Where I have installed filebeat)
enabled: true
paths:
- C:\Windows\System32\LogFiles\SMTPSVC1\*.log
- E:\AppLogs\*.json
scan_frequency: 10s
ignore_older: 24h
filebeat.config.modules:
path: C:\Program Files\Filebeat\modules.d\iis.yml
reload.enabled: false
setup.template.settings:
index.number_of_shards: 3
setup.kibana:
host: "kibanaServerName:5601"
output.logstash:
hosts: ["logstashServerName:5044"]
Logstash Configuration
----------------------------------------------------------------
Pipeline.yml
- pipeline.id: beats-server
config.string: |
input { beats { port => 5044 } }
output {
if [source] == 'APISERVERNAME' {
pipeline { send_to => apilog }
} else if [source] == 'DBSERVERNAME' {
pipeline { send_to => dblog }
}
else{
pipeline { send_to => defaultlog }
}
}
- pipeline.id: apilog-processing
path.config: "/Logstash/config/pipelines/apilogpipeline.conf"
- pipeline.id: dblog-processing
path.config: "/Logstash/config/pipelines/dblogpipeline.conf"
- pipeline.id: defaultlog-processing
path.config: "/Logstash/config/pipelines/defaultlogpipeline.conf"
1. apilogpipeline.conf
----------------------------------------------------------
input {
pipeline {
address => apilog
}
}
output {
file {
path => ["C:/Logs/apilog_%{+yyyy_MM_dd}.log"]
}
}
2. dbilogpipeline.conf
---------------------------------------------------------
input {
pipeline {
address => dblog
}
}
output {
file {
path => ["C:/Logs/dblog_%{+yyyy_MM_dd}.log"]
}
}
3. defaultlogpipeline.conf
---------------------------------------------------------
input {
pipeline {
address => defaultlog
}
}
output {
file {
path => ["C:/Logs/defaultlog_%{+yyyy_MM_dd}.log"]
}
}

Answer 1

It works the other way around, i.e. it's not Logstash that connects to Filebeat but Filebeat that sends data to Logstash. So in your input section, the host needs to be the name of the host where Logstash is running.
beats {
host => "logstash-host"
port => 5044
}
Then in your Filebeat configuration, you need to configure the Logstash output like this:
output.logstash:
hosts: ["logstash-host:5044"]
Since you have multiple Filebeat sources and want to apply a dedicated pipeline to each, what you can do is to define one custom field or tag in each Filebeat config (e.g. source: db, source: api-server, etc) and then in Logstash you can apply a different logic based on those values.
filebeat.inputs:
- type: log
fields:
source: 'APISERVERNAME'
fields_under_root: true
In Logstash, you can either leverage conditionals or pipeline to pipeline communication in order to apply a different logic based on event data.
On the latest link, you can see an example of the distributor pattern which is pretty much what you're after.