How to create indices based on kubernetes metadata - elasticsearch

Question

I am using filebeat in Kubernetes to ship logs to elastic search.
I want to create indexes based on namespaces. I'm trying to create a custom index for my different apps in a Kubernetes cluster, but this not working.
I used below conf:-
output.elasticsearch:
index: "%{[kubernetes.labels.app]:filebeat}-%{[beat.version]}-%{+yyyy.MM.dd}"
Filebeat Kube-manifest link- https://github.com/anup1384/k8s-filebeat

Answer 1

Use elasticsearch output as given below in filebeat configmap
output.elasticsearch:
index: "%{[kubernetes.namespace]:filebeat}-%{[beat.version]}-%{+yyyy.MM.dd}"

Answer 2

Create a custom index using Kubernetes metadata. So here I'm creating an index based on pod name metadata.
logstash_prefix ${record['kubernetes']['pod_name']}
For more details:
https://medium.com/faun/how-to-create-custom-indices-based-on-kubernetes-metadata-using-fluentd-beed062faa5d

Answer 3

<source>
#type tail
#id in_tail_docker_logs
read_from_head true
tag yourTag.*
path /var/log/containers/**yournamespace**.log
pos_file /var/log/file.log.pos
<parse>
#type multi_format
<pattern>
format json
time_format '%Y-%m-%dT%H:%M:%S.%N%Z'
</pattern>
<pattern>
format regexp
expression /^(?<time>.+) (?<stream>stdout|stderr)( (?<logtag>.))? (?<log>.*)$/
time_format '%Y-%m-%dT%H:%M:%S.%N%:z'
</pattern>
</parse>
</source>
<match yourTag_**>
#type elasticsearch
host "#{ENV['FLUENT_ELASTICSEARCH_HOST']}"
port "#{ENV['FLUENT_ELASTICSEARCH_PORT']}"
scheme "#{ENV['FLUENT_ELASTICSEARCH_SCHEME'] || 'https'}"
user "#{ENV['FLUENT_ELASTICSEARCH_MDSA_USER']}"
password "#{ENV['FLUENT_ELASTICSEARCH_MDSA_PASSWORD']}"
ssl_verify "#{ENV['FLUENT_ELASTICSEARCH_SSL_VERIFY'] || 'false'}"
suppress_type_name true
index_name ocp_${tag[0]}_${$.kubernetes.labels.app}_%Y%m%d ### ==> **ocp_yourTag_appName_date**
<buffer tag, time, $.kubernetes.labels.app>
#type memory
timekey 10s
timekey_wait 0s
flush_mode immediate
flush_thread_count 4
</buffer>
This will work 100%, just change with your tag and with you namespace