elasticsearch error kubernetes.labels.app - elasticsearch

Question

I have a customized kubernetes, I want to analyze all the logs in it, I found the documentation
set everything up according to the documentation, my filebeat-kubernetes.yaml configuration files turned out to be
---
apiVersion: v1
kind: ConfigMap
metadata:
name: filebeat-config
namespace: kube-system
labels:
k8s-app: filebeat
data:
filebeat.yml: |-
filebeat.config:
inputs:
# Mounted `filebeat-inputs` configmap:
path: ${path.config}/inputs.d/*.yml
# Reload inputs configs as they change:
reload.enabled: false
modules:
path: ${path.config}/modules.d/*.yml
# Reload module configs as they change:
reload.enabled: false
# To enable hints based autodiscover, remove `filebeat.config.inputs` configuration and uncomment this:
#filebeat.autodiscover:
# providers:
# - type: kubernetes
# hints.enabled: true
processors:
- add_cloud_metadata:
cloud.id: ${ELASTIC_CLOUD_ID}
cloud.auth: ${ELASTIC_CLOUD_AUTH}
output.elasticsearch:
hosts:['${ELASTICSEARCH_HOST:my_ip}:${ELASTICSEARCH_PORT:9200}']
---
apiVersion: v1
kind: ConfigMap
metadata:
name: filebeat-inputs
namespace: kube-system
labels:
k8s-app: filebeat
data:
kubernetes.yml: |-
- type: docker
containers.ids:
- "*"
processors:
- add_kubernetes_metadata:
in_cluster: true
---
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
name: filebeat
namespace: kube-system
labels:
k8s-app: filebeat
spec:
template:
metadata:
labels:
k8s-app: filebeat
spec:
serviceAccountName: filebeat
terminationGracePeriodSeconds: 30
containers:
- name: filebeat
image: docker.elastic.co/beats/filebeat:6.4.2
args: [
"-c", "/etc/filebeat.yml",
"-e",
]
env:
- name: ELASTICSEARCH_HOST
value: my_ip
- name: ELASTICSEARCH_PORT
value: "9200"
- name: ELASTIC_CLOUD_ID
value:
- name: ELASTIC_CLOUD_AUTH
value:
securityContext:
runAsUser: 0
# If using Red Hat OpenShift uncomment this:
#privileged: true
resources:
limits:
memory: 200Mi
requests:
cpu: 100m
memory: 100Mi
volumeMounts:
- name: config
mountPath: /etc/filebeat.yml
readOnly: true
subPath: filebeat.yml
- name: inputs
mountPath: /usr/share/filebeat/inputs.d
readOnly: true
- name: data
mountPath: /usr/share/filebeat/data
- name: varlibdockercontainers
mountPath: /var/lib/docker/containers
readOnly: true
volumes:
- name: config
configMap:
defaultMode: 0600
name: filebeat-config
- name: varlibdockercontainers
hostPath:
path: /var/lib/docker/containers
- name: inputs
configMap:
defaultMode: 0600
name: filebeat-inputs
# data folder stores a registry of read status for all files, so we don't send everything again on a Filebeat pod restart
- name: data
hostPath:
path: /var/lib/filebeat-data
type: DirectoryOrCreate
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: filebeat
subjects:
- kind: ServiceAccount
name: filebeat
namespace: kube-system
roleRef:
kind: ClusterRole
name: filebeat
apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
name: filebeat
labels:
k8s-app: filebeat
rules:
- apiGroups: [""] # "" indicates the core API group
resources:
- namespaces
- pods
verbs:
- get
- watch
- list
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: filebeat
namespace: kube-system
labels:
k8s-app: filebeat
---
run filebeat-kubernetes.yaml
kubectl create -f filebeat-kubernetes.yaml
I get indexes in elasticsearch
yellow open filebeat-6.4.2-2018.10.09 9A42qYPRSem4Z6ZBZQ1P7A 5 1 1129 0 457.3kb 457.3kb
yellow open filebeat-6.4.2-2018.10.11 6-8oKQ_RQBCx9D71kHhSiQ 5 1 32 0 56.4kb 56.4kb
yellow open filebeat-6.4.2-2018.10.10 Wc5xG55KRMWJXqJjfhBbUA 5 1 36826 0 29.8mb 29.8mb
but I have such errors in the elasticsearch logs
[DEBUG][o.e.a.b.TransportShardBulkAction] [filebeat-6.4.2-2018.10.11]
[3] failed to execute bulk item (index) BulkShardRequest [[filebeat-
6.4.2-2018.10.11][3]] containing [8] requests
org.elasticsearch.index.mapper.MapperParsingException: failed to parse [kubernetes.labels.app]
at org.elasticsearch.index.mapper.FieldMapper.parse(FieldMapper.java:302) ~[elasticsearch-6.4.2.jar:6.4.2]
at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrField(DocumentParser.java:481) ~[elasticsearch-6.4.2.jar:6.4.2]
at org.elasticsearch.index.mapper.DocumentParser.parseObject(DocumentParser.java:501) ~[elasticsearch-6.4.2.jar:6.4.2]
at org.elasticsearch.index.mapper.DocumentParser.innerParseObject(DocumentParser.java:390) ~[elasticsearch-6.4.2.jar:6.4.2]
at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrNested(DocumentParser.java:380) ~[elasticsearch-6.4.2.jar:6.4.2]
at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrField(DocumentParser.java:478) ~[elasticsearch-6.4.2.jar:6.4.2]
at org.elasticsearch.index.mapper.DocumentParser.parseObject(DocumentParser.java:501) ~[elasticsearch-6.4.2.jar:6.4.2]
at org.elasticsearch.index.mapper.DocumentParser.innerParseObject(DocumentParser.java:390) ~[elasticsearch-6.4.2.jar:6.4.2]
at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrNested(DocumentParser.java:380) ~[elasticsearch-6.4.2.jar:6.4.2]
at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrField(DocumentParser.java:478) ~[elasticsearch-6.4.2.jar:6.4.2]
...
kubernetes version and elasticsearch version
kubectl version
Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.3", GitCommit:"a4529464e4629c21224b3d52edfe0ea91b072862", GitTreeState:"clean", BuildDate:"2018-09-09T17:53:03Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.3", GitCommit:"a4529464e4629c21224b3d52edfe0ea91b072862", GitTreeState:"clean", BuildDate:"2018-09-09T17:53:03Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}
curl -XGET localhost:9200
{
"name" : "el3",
"cluster_name" : "elasticsearch",
"cluster_uuid" : "hmmQcpMdSYCM8P3i9gOENw",
"version" : {
"number" : "6.4.2",
"build_flavor" : "default",
"build_type" : "deb",
"build_hash" : "04711c2",
"build_date" : "2018-09-26T13:34:09.098244Z",
"build_snapshot" : false,
"lucene_version" : "7.4.0",
"minimum_wire_compatibility_version" : "5.6.0",
"minimum_index_compatibility_version" : "5.0.0"
},
"tagline" : "You Know, for Search"
}
how to fix error failed to parse [kubernetes.labels.app]? or how can i remove filebeat - label from config?
Update
I added filebeat index template in elasticsearch, my file file-index-template.json
{
"mappings": {
"_default_": {
"dynamic_templates": [
{
"template1": {
"mapping": {
"doc_values": true,
"ignore_above": 1024,
"index": "false",
"type": "{dynamic_type}"
},
"match": "*"
}
}
],
"properties": {
"#timestamp": {
"type": "date"
},
"message": {
"type": "text",
"index": "true"
},
"offset": {
"type": "long",
"doc_values": "true"
},
"geoip": {
"type": "object",
"dynamic": true,
"properties": {
"location": {
"type": "geo_point"
}
}
}
}
}
},
"settings": {
"index.refresh_interval": "5s"
},
"template": "filebeat-*"
}
added template in elasticsearch
curl -H 'Content-Type: application/json' -XPUT 'http://localhost:9200/_template/filebeat?pretty' -d#filebeat.json
check template
curl localhost:9200/_template/filebeat
{"filebeat":{"order":0,"index_patterns":["filebeat-*"],"settings":{"index":{"refresh_interval":"5s"}},"mappings":{"_default_":{"dynamic_templates":[{"template1":{"mapping":{"doc_values":true,"ignore_above":1024,"index":"false","type":"{dynamic_type}"},"match":"*"}}],"properties":{"#timestamp":{"type":"date"},"message":{"type":"text","index":"true"},"offset":{"type":"long","doc_values":"true"},"geoip":{"type":"object","dynamic":true,"properties":{"location":{"type":"geo_point"}}}}}},"aliases":{}}}
check index
curl localhost:9200/_cat/indices
yellow open filebeat-6.4.2-2018.10.17 c9EmKOQ9T7W_pl9tDRDycQ 5 1 13719988 0 13.8gb 13.8gb
yellow open filebeat-6.4.2-2018.10.14 daA_KAT_TYeL5Fn3SrT2Pw 5 1 56400 0 10.5mb 10.5mb
yellow open filebeat-6.4.2-2018.10.16 70uY3kooTjWRNaFCky24jQ 5 1 277731 0 69.3mb 69.3mb
green open .kibana DgMyQx7QSK659uBo1CccJQ 1 0 3 0 34.3kb 34.3kb
yellow open filebeat-6.4.2-2018.10.13 LsC4soOYSEqY3vwv-HOcjg 5 1 135921 0 19.1mb 19.1mb
yellow open filebeat-6.4.2-2018.10.15 hKNvyDl9SFSgw3nEU3faKg 5 1 72960 0 18.7mb 18.7mb
but still I see in the elasticsearch logs
[DEBUG][o.e.a.b.TransportShardBulkAction] [filebeat-6.4.2-2018.10.17][4] failed to execute bulk item (index) BulkShardRequest [[filebeat-6.4.2-2018.10.17][4]] containing [13] requests
org.elasticsearch.index.mapper.MapperParsingException: failed to parse [kubernetes.labels.app]
at org.elasticsearch.index.mapper.FieldMapper.parse(FieldMapper.java:302) ~[elasticsearch-6.4.2.jar:6.4.2]
at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrField(DocumentParser.java:481) ~[elasticsearch-6.4.2.jar:6.4.2]
at org.elasticsearch.index.mapper.DocumentParser.parseObject(DocumentParser.java:501) ~[elasticsearch-6.4.2.jar:6.4.2]
at org.elasticsearch.index.mapper.DocumentParser.innerParseObject(DocumentParser.java:390) ~[elasticsearch-6.4.2.jar:6.4.2]
at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrNested(DocumentParser.java:380) ~[elasticsearch-6.4.2.jar:6.4.2]
at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrField(DocumentParser.java:478) ~[elasticsearch-6.4.2.jar:6.4.2]
at org.elasticsearch.index.mapper.DocumentParser.parseObject(DocumentParser.java:501) ~[elasticsearch-6.4.2.jar:6.4.2]
at org.elasticsearch.index.mapper.DocumentParser.innerParseObject(DocumentParser.java:390) ~[elasticsearch-6.4.2.jar:6.4.2]
at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrNested(DocumentParser.java:380) ~[elasticsearch-6.4.2.jar:6.4.2]
at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrField(DocumentParser.java:478) ~[elasticsearch-6.4.2.jar:6.4.2]
at org.elasticsearch.index.mapper.DocumentParser.parseObject(DocumentParser.java:501) ~[elasticsearch-6.4.2.jar:6.4.2]
at org.elasticsearch.index.mapper.DocumentParser.innerParseObject(DocumentParser.java:390) ~[elasticsearch-6.4.2.jar:6.4.2]
at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrNested(DocumentParser.java:380) ~[elasticsearch-6.4.2.jar:6.4.2]
at org.elasticsearch.index.mapper.DocumentParser.internalParseDocument(DocumentParser.java:95) ~[elasticsearch-6.4.2.jar:6.4.2]
at org.elasticsearch.index.mapper.DocumentParser.parseDocument(DocumentParser.java:69) ~[elasticsearch-6.4.2.jar:6.4.2]
at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:263) ~[elasticsearch-6.4.2.jar:6.4.2]
at org.elasticsearch.index.shard.IndexShard.prepareIndex(IndexShard.java:725) ~[elasticsearch-6.4.2.jar:6.4.2]
at org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:702) ~[elasticsearch-6.4.2.jar:6.4.2]
at org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnPrimary(IndexShard.java:682) ~[elasticsearch-6.4.2.jar:6.4.2]
at org.elasticsearch.action.bulk.TransportShardBulkAction.lambda$executeIndexRequestOnPrimary$2(TransportShardBulkAction.java:560) ~[elasticsearch-6.4.2.jar:6.4.2]
...
Update 2
curl localhost:9200/_cat/indices
yellow open filebeat-6.4.2-2018.10.25 0RCTMniqQyucD530dz_eOQ 5 1 511 0 491.1kb 491.1kb
yellow open filebeat-6.4.2-2018.10.27 64b5ThH1TauvwMIo_ueTIg 5 1 487 0 479.4kb 479.4kb
yellow open filebeat-6.4.2-2018.10.28 Lf4UzVzESIGfGvx7VsRzFQ 5 1 283 0 357.4kb 357.4kb
yellow open filebeat-6.4.2-2018.10.24 fCUmzy2UQSy9lsNOMWmkEQ 5 1 2866 0 1.8mb 1.8mb
yellow open filebeat-6.4.2-2018.10.26 t3rPwBS4TYOhJWjtFRYk6g 5 1 323 0 428.9kb 428.9kb
yellow open filebeat-6.4.2-2018.10.22 -Rq7SbeqS_yNX3I4lwsGRg 5 1 92 0 173.2kb 173.2kb
yellow open filebeat-6.4.2-2018.10.29 yAje-vFhQqmavxSO7tlDGA 5 1 4810 0 8.5mb 8.5mb
Check elasticksearch
curl -XGET 'http://localhost:9200/filebeat-*/_search?pretty'
{
"took" : 33,
"timed_out" : false,
"_shards" : {
"total" : 35,
"successful" : 35,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : 67309,
"max_score" : 1.0,
"hits" : [
{
"_index" : "filebeat-6.4.2-2018.10.22",
"_type" : "doc",
"_id" : "-m0iwGYBP2-nX77s4y_g",
"_score" : 1.0,
"_source" : {
"#timestamp" : "2018-10-22T07:32:36.393Z",
"message" : "2018-10-22 07:32:36.393 [INFO][92] int_dataplane.go 747: Finished applying updates to dataplane. msecToApply=92.064514",
"prospector" : {
"type" : "docker"
},
"input" : {
"type" : "docker"
},
"beat" : {
"name" : "filebeat-6p7rc",
"hostname" : "filebeat-6p7rc",
"version" : "6.4.2"
},
"host" : {
"name" : "filebeat-6p7rc"
},
"source" : "/var/lib/docker/containers/02ed5c70d5341a7d3f15fecbb24dd94bc43d850fc4fd6c609a771d487518d659/02ed5c70d5341a7d3f15fecbb24dd94bc43d850fc4fd6c609a771d487518d659-json.log",
"offset" : 630130,
"stream" : "stdout"
}
},
{
"_index" : "filebeat-6.4.2-2018.10.22",
"_type" : "doc",
"_id" : "_m0iwGYBP2-nX77s4y_g",
"_score" : 1.0,
"_source" : {
"#timestamp" : "2018-10-22T07:32:38.159Z",
"beat" : {
"name" : "filebeat-6p7rc",
"hostname" : "filebeat-6p7rc",
"version" : "6.4.2"
},
"offset" : 630467,
"stream" : "stdout",
"message" : "2018-10-22 07:32:38.158 [INFO][92] health.go 150: Overall health summary=&health.HealthReport{Live:true, Ready:true}",
"source" : "/var/lib/docker/containers/02ed5c70d5341a7d3f15fecbb24dd94bc43d850fc4fd6c609a771d487518d659/02ed5c70d5341a7d3f15fecbb24dd94bc43d850fc4fd6c609a771d487518d659-json.log",
"prospector" : {
"type" : "docker"
},
"input" : {
"type" : "docker"
},
"host" : {
"name" : "filebeat-6p7rc"
}
}
},
{
"_index" : "filebeat-6.4.2-2018.10.22",
"_type" : "doc",
"_id" : "n20iwGYBP2-nX77s5jGM",
"_score" : 1.0,
"_source" : {
"#timestamp" : "2018-10-22T07:32:41.172Z",
"source" : "/var/lib/docker/containers/02ed5c70d5341a7d3f15fecbb24dd94bc43d850fc4fd6c609a771d487518d659/02ed5c70d5341a7d3f15fecbb24dd94bc43d850fc4fd6c609a771d487518d659-json.log",
"offset" : 631205,
"stream" : "stdout",
"message" : "2018-10-22 07:32:41.172 [INFO][92] table.go 438: Loading current iptables state and checking it is correct. ipVersion=0x4 table=\"raw\"",
"prospector" : {
"type" : "docker"
},
"input" : {
"type" : "docker"
},
"beat" : {
"name" : "filebeat-6p7rc",
"hostname" : "filebeat-6p7rc",
"version" : "6.4.2"
},
"host" : {
"name" : "filebeat-6p7rc"
}
}
},
{
"_index" : "filebeat-6.4.2-2018.10.22",
"_type" : "doc",
"_id" : "WG0iwGYBP2-nX77s6DIH",
"_score" : 1.0,
"_source" : {
"#timestamp" : "2018-10-22T07:32:45.710Z",
"source" : "/var/lib/docker/containers/02ed5c70d5341a7d3f15fecbb24dd94bc43d850fc4fd6c609a771d487518d659/02ed5c70d5341a7d3f15fecbb24dd94bc43d850fc4fd6c609a771d487518d659-json.log",
"offset" : 632166,
"stream" : "stdout",
"message" : "2018-10-22 07:32:45.710 [INFO][92] ipsets.go 222: Asked to resync with the dataplane on next update. family=\"inet\"",
"prospector" : {
"type" : "docker"
},
"input" : {
"type" : "docker"
},
"beat" : {
"hostname" : "filebeat-6p7rc",
"version" : "6.4.2",
"name" : "filebeat-6p7rc"
},
"host" : {
"name" : "filebeat-6p7rc"
}
}
},
{
"_index" : "filebeat-6.4.2-2018.10.22",
"_type" : "doc",
"_id" : "Wm0iwGYBP2-nX77s6DIH",
"_score" : 1.0,
"_source" : {
"#timestamp" : "2018-10-22T07:32:45.710Z",
"input" : {
"type" : "docker"
},
"beat" : {
"name" : "filebeat-6p7rc",
"hostname" : "filebeat-6p7rc",
"version" : "6.4.2"
},
"host" : {
"name" : "filebeat-6p7rc"
},
"source" : "/var/lib/docker/containers/02ed5c70d5341a7d3f15fecbb24dd94bc43d850fc4fd6c609a771d487518d659/02ed5c70d5341a7d3f15fecbb24dd94bc43d850fc4fd6c609a771d487518d659-json.log",
"offset" : 632353,
"stream" : "stdout",
"message" : "2018-10-22 07:32:45.710 [INFO][92] ipsets.go 253: Resyncing ipsets with dataplane. family=\"inet\"",
"prospector" : {
"type" : "docker"
}
}
},
{
"_index" : "filebeat-6.4.2-2018.10.22",
"_type" : "doc",
"_id" : "XG0iwGYBP2-nX77s6DIH",
"_score" : 1.0,
"_source" : {
"#timestamp" : "2018-10-22T07:32:45.711Z",
"stream" : "stdout",
"prospector" : {
"type" : "docker"
},
"input" : {
"type" : "docker"
},
"beat" : {
"name" : "filebeat-6p7rc",
"hostname" : "filebeat-6p7rc",
"version" : "6.4.2"
},
"host" : {
"name" : "filebeat-6p7rc"
},
"message" : "2018-10-22 07:32:45.711 [INFO][92] ipsets.go 295: Finished resync family=\"inet\" numInconsistenciesFound=0 resyncDuration=876.908µs",
"source" : "/var/lib/docker/containers/02ed5c70d5341a7d3f15fecbb24dd94bc43d850fc4fd6c609a771d487518d659/02ed5c70d5341a7d3f15fecbb24dd94bc43d850fc4fd6c609a771d487518d659-json.log",
"offset" : 632522
}
},
{
"_index" : "filebeat-6.4.2-2018.10.22",
"_type" : "doc",
"_id" : "QG0iwGYBP2-nX77s6TNr",
"_score" : 1.0,
"_source" : {
"#timestamp" : "2018-10-22T07:32:45.711Z",
"source" : "/var/lib/docker/containers/02ed5c70d5341a7d3f15fecbb24dd94bc43d850fc4fd6c609a771d487518d659/02ed5c70d5341a7d3f15fecbb24dd94bc43d850fc4fd6c609a771d487518d659-json.log",
"offset" : 632726,
"stream" : "stdout",
"message" : "2018-10-22 07:32:45.711 [INFO][92] int_dataplane.go 747: Finished applying updates to dataplane. msecToApply=1.061403",
"prospector" : {
"type" : "docker"
},
"input" : {
"type" : "docker"
},
"beat" : {
"hostname" : "filebeat-6p7rc",
"version" : "6.4.2",
"name" : "filebeat-6p7rc"
},
"host" : {
"name" : "filebeat-6p7rc"
}
}
},
{
"_index" : "filebeat-6.4.2-2018.10.22",
"_type" : "doc",
"_id" : "1W0iwGYBP2-nX77s8zc2",
"_score" : 1.0,
"_source" : {
"#timestamp" : "2018-10-22T07:32:58.158Z",
"message" : "2018-10-22 07:32:58.158 [INFO][92] health.go 150: Overall health summary=&health.HealthReport{Live:true, Ready:true}",
"source" : "/var/lib/docker/containers/02ed5c70d5341a7d3f15fecbb24dd94bc43d850fc4fd6c609a771d487518d659/02ed5c70d5341a7d3f15fecbb24dd94bc43d850fc4fd6c609a771d487518d659-json.log",
"offset" : 634199,
"prospector" : {
"type" : "docker"
},
"input" : {
"type" : "docker"
},
"beat" : {
"hostname" : "filebeat-6p7rc",
"version" : "6.4.2",
"name" : "filebeat-6p7rc"
},
"host" : {
"name" : "filebeat-6p7rc"
},
"stream" : "stdout"
}
},
{
"_index" : "filebeat-6.4.2-2018.10.22",
"_type" : "doc",
"_id" : "-G0iwGYBP2-nX77s8zc2",
"_score" : 1.0,
"_source" : {
"#timestamp" : "2018-10-22T07:33:00.168Z",
"message" : "2018-10-22 07:33:00.167 [INFO][92] health.go 150: Overall health summary=&health.HealthReport{Live:true, Ready:true}",
"source" : "/var/lib/docker/containers/02ed5c70d5341a7d3f15fecbb24dd94bc43d850fc4fd6c609a771d487518d659/02ed5c70d5341a7d3f15fecbb24dd94bc43d850fc4fd6c609a771d487518d659-json.log",
"offset" : 634391,
"stream" : "stdout",
"prospector" : {
"type" : "docker"
},
"input" : {
"type" : "docker"
},
"beat" : {
"name" : "filebeat-6p7rc",
"hostname" : "filebeat-6p7rc",
"version" : "6.4.2"
},
"host" : {
"name" : "filebeat-6p7rc"
}
}
},
{
"_index" : "filebeat-6.4.2-2018.10.22",
"_type" : "doc",
"_id" : "yW0iwGYBP2-nX77s_j2e",
"_score" : 1.0,
"_source" : {
"#timestamp" : "2018-10-22T07:33:18.158Z",
"offset" : 636780,
"stream" : "stdout",
"message" : "2018-10-22 07:33:18.158 [INFO][92] health.go 150: Overall health summary=&health.HealthReport{Live:true, Ready:true}",
"prospector" : {
"type" : "docker"
},
"input" : {
"type" : "docker"
},
"host" : {
"name" : "filebeat-6p7rc"
},
"beat" : {
"name" : "filebeat-6p7rc",
"hostname" : "filebeat-6p7rc",
"version" : "6.4.2"
},
"source" : "/var/lib/docker/containers/02ed5c70d5341a7d3f15fecbb24dd94bc43d850fc4fd6c609a771d487518d659/02ed5c70d5341a7d3f15fecbb24dd94bc43d850fc4fd6c609a771d487518d659-json.log"
}
}
]
}
}

Answer 1

I suppose that you haven't set up an index template for Filebeat fields, which should be parsed to Elasticsearch for further processing. You can find useful information in this Article about implementing the Filebeat index template on your cluster.
In addition, there was a similar issue reported in GitHub about parsing kubernetes.labels using Logstash event collector.