Unable to collect all kubernetes container/pod logs via fluentd/elasticsearch - elasticsearch

Question

I'm new with fluentd/elasticsearch stack and I'm trying to deploy it on kubernetes. While I've managed to do that, I'm having a problem that not all pod/container logs are showing up on elasticsearch (I'm using Kibana for data visualisation). In other words, I'm able to see logs from "default" kubernetes pods like weave-net and elasticsearch related pod logs (es-data, es-master...etc.) but not from "custom" pods that I'm trying to deploy.
As a simple test, I've deployed redis in the same kube namespace where fluentd/elasticsearch resides and redis service/deployment looks like this:
---
apiVersion: v1
kind: Service
metadata:
name: redis-master
labels:
app: redis
role: master
tier: backend
spec:
ports:
- port: 6379
targetPort: 6379
selector:
app: redis
role: master
tier: backend
---
apiVersion: apps/v1 # for k8s versions before 1.9.0 use apps/v1beta2 and before 1.8.0 use extensions/v1beta1
kind: Deployment
metadata:
name: redis-master
spec:
selector:
matchLabels:
app: redis
role: master
tier: backend
replicas: 1
template:
metadata:
labels:
app: redis
role: master
tier: backend
spec:
containers:
- name: master
image: k8s.gcr.io/redis:e2e # or just image: redis
resources:
requests:
cpu: 100m
memory: 100Mi
ports:
- containerPort: 6379
When I check logs from fluentd daemonpods, I see following:
2018-07-03 11:17:05 +0000 [info]: following tail of /var/log/containers/redis-master-585798d8ff-b5p5g_default_master-4c934d19a8e2b2d6143b662425fd8fc238df98433d1c0c32bf328c281ef593ad.log
which, if I'm correct, should give me an info that fluentd is picking up redis container logs. However, I'm unable to see any redis related documents stored in elasticsearch.
This is how part of the configuration for fluentd looks like (kubernetes.conf):
<source>
#type tail
#id in_tail_container_logs
path /var/log/containers/*.log
pos_file /var/log/fluentd-containers.log.pos
tag kubernetes.*
read_from_head true
format json
time_format %Y-%m-%dT%H:%M:%S.%NZ
</source>
and fluent.conf:
<match **>
#type elasticsearch
#id out_es
log_level info
include_tag_key true
host "#{ENV['FLUENT_ELASTICSEARCH_HOST']}"
port "#{ENV['FLUENT_ELASTICSEARCH_PORT']}"
scheme "#{ENV['FLUENT_ELASTICSEARCH_SCHEME'] || 'http'}"
ssl_verify "#{ENV['FLUENT_ELASTICSEARCH_SSL_VERIFY'] || 'true'}"
user "#{ENV['FLUENT_ELASTICSEARCH_USER']}"
password "#{ENV['FLUENT_ELASTICSEARCH_PASSWORD']}"
reload_connections "#{ENV['FLUENT_ELASTICSEARCH_RELOAD_CONNECTIONS'] || 'true'}"
logstash_prefix "#{ENV['FLUENT_ELASTICSEARCH_LOGSTASH_PREFIX'] || 'logstash'}"
logstash_format true
buffer_chunk_limit 2M
buffer_queue_limit 32
flush_interval 5s
max_retry_wait 30
disable_retry_limit
num_threads 8
</match>
Any hint would be very helpful.
Thanks in advance.

Answer 1

I am using fluent bit for the same purpose and I met exactly the same problem quite a few days back. Fluent bit is a light weight version of fluentd, and what worked for me might work for you as well.
What was wrong with my fluent bit was the input configuration. For the tail plugins that tail into large log files, there was some issue with the log rotation. So I lowered my refresh_interval to something like 5 secs (time period over which list of watched files are updated). Then I lowered the mem_buf_limit to something like 5MB (the total size of logs fluent bit takes into memory before flushing that out to the output plugin).
By these changes I was able to get more logs which were earlier not being collected for god knows reason.
I have asked this as an issue. Will update my answer if I get to know the reason.
Hope this helps in anyway. Mainly I suggest you to tweak your input configurations and then see the changes.