grok script for writing to logstash and rendering in Kibana - elasticsearch

Question

I am following filebeat->logstash->elasticsearch->kibana pipeline. filebeat successfully working and fetching the logs from the target file.
Logstash receiving the logs on input plugin and bypassing the filter plugin and sending over to the output plugin.
filebeat.yml
# ============================== Filebeat inputs ===============================
filebeat.inputs:
- type: log
enabled: true
paths:
- D:\serverslogs\ch5shdmtbuil100\TeamCity.BuildServer-logs\launcher.log
fields:
type: launcherlogs
- type: filestream
# Change to true to enable this input configuration.
enabled: false
# Paths that should be crawled and fetched. Glob based paths.
paths:
- /var/log/*.log
# =================================== Kibana ===================================
setup.kibana:
host: "localhost:5601"
# ------------------------------ Logstash Output -------------------------------
output.logstash:
# The Logstash hosts
hosts: ["localhost:5044"]
logstash.conf
input{
beats{
port => "5044"
}
}
filter {
if [fields][type] == "launcherlogs"{
grok {
match => {"message" =>%{YEAR:year}-%{MONTH:month}-%{MONTHDAY:day}%{DATA:loglevel}%{SPACE}-%{SPACE}%{DATA:class}%{SPACE}-%{GREEDYDATA:message}}
}
}
}
output{
elasticsearch{
hosts => ["http://localhost:9200"]
index => "poclogsindex"
}
}
I am able to send the logs on kibana but the grok debugger scripts is not rendering desired json on kibana.
The data json rendered on Kibana is not showing all the attributes passed in the script. Please advise.

Answer 1

Your grok pattern does not match the sample you gave in comment : several parts are missing (the brackets, the HH:mm:ss,SSS part and an additionnal space). Grok debuggers are your friends ;-)
Instead of :
%{YEAR:year}-%{MONTH:month}-%{MONTHDAY:day}%{DATA:loglevel}%{SPACE}-%{SPACE}%{DATA:class}%{SPACE}-%{GREEDYDATA:message}
Your pattern should be :
\[%{TIMESTAMP_ISO8601:timestamp}\] %{DATA:loglevel}%{SPACE}-%{SPACE}%{DATA:class}%{SPACE}-%{GREEDYDATA:message}
TIMESTAMP_ISO8601 matches this date/time.
Additionnally, I always doubled-quote the pattern, so the grok part would be :
grok {
match => {"message" => "\[%{TIMESTAMP_ISO8601:timestamp}\] %{DATA:loglevel}%{SPACE}-%{SPACE}%{DATA:class}%{SPACE}-%{GREEDYDATA:message}"}
}