Logstash with helm in Kubernetes : grok filter not working - elasticsearch

Question

I installed a filebeat -> logstash -> elasticsearch -> kibana stack in Kubernetes with helm charts :
helm repo add incubator http://storage.googleapis.com/kubernetes-charts-incubator
helm install --name elastic --namespace monitoring incubator/elasticsearch --set client.replicas=1,master.replicas=2,data.replicas=1
helm install --name logstash --namespace monitoring incubator/logstash -f logstash_values.yaml
helm install --name filebeat stable/filebeat -f filebeat_values.yaml
helm install stable/kibana --name kibana --namespace monitoring
The logs are indexed in ES, but the "message" contains the whole string, not the defined fields. My grok filter doesn't seem to work in logstash conf.
The is no documentation on https://github.com/helm/charts/tree/master/incubator/logstash about how to set the patterns.
Here is what I tried :
my log's format :
10-09-2018 11:57:55.906 [Debug] [LOG] serviceName - Technical - my specific message - correlationId - userId - data - operation - error - stackTrace escaped on one line
logstash_values.yaml (from https://github.com/helm/charts/blob/master/incubator/logstash/values.yaml) :
elasticsearch:
host: elasticsearch-client.default.svc.cluster.local
port: 9200
patterns:
main: |-
(?<time>(?:2[0123]|[01]?[0-9]):(?:[0-5][0-9]):(?:(?:[0-5]?[0-9]|60)(?:[:.,][0-9]+)?)\.(?:[0-9]){3})} [(?<logLevel>.*)] [(?<code>.*)] (?<caller>.*) - (?<logMessageType>.*) - (?<message>.*) - (?<correlationId>.*) - (?<userId>.*) - (?<data>.*) - (?<operation>.*) - (?<error>.*) - (?<stackTrace>.*)
inputs:
main: |-
input {
beats {
port => 5044
}
}
filters:
outputs:
main: |-
output {
elasticsearch {
hosts => ["${ELASTICSEARCH_HOST}:${ELASTICSEARCH_PORT}"]
manage_template => false
index => "%{[#metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[#metadata][type]}"
}
}
This becomes a Kubernetes configMap "logstash-patterns" :
apiVersion: v1
kind: ConfigMap
data:
main: (?<time>(?:2[0123]|[01]?[0-9]):(?:[0-5][0-9]):(?:(?:[0-5]?[0-9]|60)(?:[:.,][0-9]+)?)\.(?:[0-9]){3}) [(?<code>.*)] [(?<logLevel>.*)] (?<service>.*) - (?<logMessageType>.*) - (?<message>.*) - (?<correlationId>.*) - (?<userId>.*) - (?<data>.*) - (?<operation>.*) - (?<error>.*) - (?<stackTrace>.*)
I don't see any error logs in logstash pod.
Do you have any idea how to configure patterns in logstash in Kubernetes ?
Thanks.

Answer 1

I was mistaking "pattern" and "filter".
In the Helm chart, "pattern" is for specifying our custom grok patterns (https://grokdebug.herokuapp.com/patterns) :
MY_CUSTOM_ALL_CHARS .*
My grok filter should be in the filter section :
patterns:
# nothing here for me
filters:
main: |-
filter {
grok {
match => { "message" => "\{%{TIMESTAMP_ISO8601:time}\} \[%{DATA:logLevel}\] \[%{DATA:code}\] %{DATA:caller} &\$ %{DATA:logMessageType} &\$ %{DATA:message} &\$ %{DATA:correlationId} &\$ %{DATA:userId} &\$ %{DATA:data} &\$ %{DATA:operation} &\$ %{DATA:error} &\$ (?<stackTrace>.*)" }
overwrite => [ "message" ]
}
date {
match => ["time", "ISO8601"]
target => "time"
}
}