fluent-bit cannot parse kubernetes logs - elasticsearch

Question

I would like to forward Kubernetes logs from fluent-bit to elasticsearch through fluentd but fluent-bit cannot parse kubernetes logs properly. In order to install Fluent-bit and Fluentd, I use Helm charts. I tried both stable/fluentbit and fluent/fluentbit and faced with same problem:
#0 dump an error event: error_class=Fluent::Plugin::ElasticsearchErrorHandler::ElasticsearchError error="400 - Rejected by Elasticsearch [error type]: mapper_parsing_exception [reason]: 'Could not dynamically add mapping for field [app.kubernetes.io/component]. Existing mapping for [kubernetes.labels.app] must be of type object but found [text].'"
I put following lines into fluent-bit values file as shown here
remapMetadataKeysFilter:
enabled: true
match: kube.*
## List of the respective patterns and replacements for metadata keys replacements
## Pattern must satisfy the Lua spec (see https://www.lua.org/pil/20.2.html)
## Replacement is a plain symbol to replace with
replaceMap:
- pattern: "[/.]"
replacement: "_"
...nothing changed, same errors are listed.
Is there a workaround to get rid of that bug?
my values.yaml is here:
# Default values for fluent-bit.
# kind -- DaemonSet or Deployment
kind: DaemonSet
# replicaCount -- Only applicable if kind=Deployment
replicaCount: 1
image:
repository: fluent/fluent-bit
pullPolicy: Always
# tag:
imagePullSecrets: []
nameOverride: ""
fullnameOverride: ""
serviceAccount:
create: true
annotations: {}
name:
rbac:
create: true
podSecurityPolicy:
create: false
podSecurityContext:
{}
# fsGroup: 2000
securityContext:
{}
# capabilities:
# drop:
# - ALL
# readOnlyRootFilesystem: true
# runAsNonRoot: true
# runAsUser: 1000
service:
type: ClusterIP
port: 2020
annotations:
prometheus.io/path: "/api/v1/metrics/prometheus"
prometheus.io/port: "2020"
prometheus.io/scrape: "true"
serviceMonitor:
enabled: true
namespace: monitoring
interval: 10s
scrapeTimeout: 10s
# selector:
# prometheus: my-prometheus
resources:
{}
# limits:
# cpu: 100m
# memory: 128Mi
# requests:
# cpu: 100m
# memory: 128Mi
nodeSelector: {}
tolerations: []
affinity: {}
podAnnotations: {}
priorityClassName: ""
env: []
envFrom: []
extraPorts: []
# - port: 5170
# containerPort: 5170
# protocol: TCP
# name: tcp
extraVolumes: []
extraVolumeMounts: []
## https://docs.fluentbit.io/manual/administration/configuring-fluent-bit
config:
## https://docs.fluentbit.io/manual/service
service: |
[SERVICE]
Flush 1
Daemon Off
Log_Level info
Parsers_File parsers.conf
Parsers_File custom_parsers.conf
HTTP_Server On
HTTP_Listen 0.0.0.0
HTTP_Port 2020
## https://docs.fluentbit.io/manual/pipeline/inputs
inputs: |
[INPUT]
Name tail
Path /var/log/containers/*.log
Parser docker
Tag kube.*
Mem_Buf_Limit 5MB
Skip_Long_Lines On
[INPUT]
Name systemd
Tag host.*
Systemd_Filter _SYSTEMD_UNIT=kubelet.service
Read_From_Tail On
## https://docs.fluentbit.io/manual/pipeline/filters
filters: |
[FILTER]
Name kubernetes
Match kube.*
Kube_URL https://kubernetes.default.svc:443
Kube_CA_File /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
Kube_Token_File /var/run/secrets/kubernetes.io/serviceaccount/token
Kube_Tag_Prefix kube.var.log.containers.
Merge_Log On
Merge_Log_Key log_processed
K8S-Logging.Parser On
K8S-Logging.Exclude Off
[FILTER]
Name lua
Match kube.*
script /fluent-bit/etc/functions.lua
call dedot
## https://docs.fluentbit.io/manual/pipeline/outputs
outputs: |
[OUTPUT]
Name forward
Match *
Host fluentd-in-forward.elastic-system.svc.cluster.local
Port 24224
tls off
tls.verify off
## https://docs.fluentbit.io/manual/pipeline/parsers
customParsers: |
[PARSER]
Name docker_no_time
Format json
Time_Keep Off
Time_Key time
Time_Format %Y-%m-%dT%H:%M:%S.%L

Answer 1

I had the same issue, which is caused by multiple labels that are converted into json.
I renamed the conflicting keys to match with the newer format of recommended labels:
<filter **>
#type rename_key
rename_rule1 ^app$ app.kubernetes.io/name
rename_rule2 ^chart$ helm.sh/chart
rename_rule3 ^version$ app.kubernetes.io/version
rename_rule4 ^component$ app.kubernetes.io/component
rename_rule5 ^istio$ istio.io/name
</filter>

Answer 2

I think that your problem isn't in kubernetes, isn't in fluentbit/fluentd chart, your problem is in elasticsearch, concretely in the mapping.
In elsticsearch version 7.x you can't have different types (string, int, etc) for the same field.
For solve thus issue, i use "ignore_malformed": true in the index template that i use for the kubernetes logs.
https://www.elastic.co/guide/en/elasticsearch/reference/current/ignore-malformed.html
The malformed field is not indexed, but other fields in the document
are processed normally.