Fluentd capture when a Kubernetes Pod terminates with the 'CrashLoopBackOff'? - debugging

Question

I'm running a pod that write a simple message to the 'terminationMessagePath' then the pod exit with "CrashLoopBackOff". I would like to be able to debug through Kibana instead of having to login to each Kubernetes nodes. I queried Kibana to get the container last state value "CrashLoopBackOff" from the property reason & message and could not locate an entry.
I can see the fields for the pod in Kibana but the field that I'm looking for (in bold yaml format below) is empty.
What configuration is needed in fluentd to get the log from Kubernetes pod? or configuration need to be set from Kubernetes
$ kubectl get pod_name_1 -o=yaml
terminationMessagePath: /var/log/containers/dt.log
volumeMounts:
- mountPath: /var/log/containers
name: data
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: default-token-s0w2n
readOnly: true
dnsPolicy: ClusterFirst
nodeName: dev-master-01
restartPolicy: Always
securityContext: {}
serviceAccount: default
serviceAccountName: default
terminationGracePeriodSeconds: 30
volumes:
- hostPath:
path: /var/log/containers
name: data
- name: default-token-s0w2n
secret:
defaultMode: 420
secretName: default-token-s0w2n
status:
conditions:
- lastProbeTime: null
lastTransitionTime: 2017-07-05T14:45:11Z
status: "True"
type: Initialized
- lastProbeTime: null
lastTransitionTime: 2017-07-05T17:00:22Z
message: 'containers with unready status: [dt-termination-demo]'
reason: ContainersNotReady
status: "False"
type: Ready
- lastProbeTime: null
lastTransitionTime: 2017-07-05T14:45:11Z
status: "True"
type: PodScheduled
containerStatuses:
- containerID:
docker://9649c26527cf0e1cd3bd67ba9c606c0b78e6b4f08bacf96175627ddc7d250772
image: debian
imageID: docker pullable://docker.io/debian#sha256:
7d067f77d2ae5a23fe6920f8fbc2936c4b0d417e9d01b26372561860750815f0
lastState:
terminated:
containerID: docker://
9649c26527cf0e1cd3bd67ba9c606c0b78e6b4f08bacf96175627ddc7d250772
exitCode: 0
finishedAt: 2017-07-05T17:00:22Z
**message: |
Sleep expired**
reason: Completed
startedAt: 2017-07-05T17:00:12Z
name: dt-termination-demo
ready: false
restartCount: 30
state:
waiting:
message: Back-off 5m0s restarting failed container=dt-termination-demo
pod=dt-termination-demo-2814930607-8kshj_
default(8c247b15-6190-11e7-acb7-00505691210d)
**reason: CrashLoopBackOff**
hostIP: 192.21.19.128
phase: Running
podIP: 10.0.0.8
startTime: 2017-07-05T14:45:11Z

Answer 1

When Fluentd is deployed as a DaemonSet, it aims to collect all logs from the Node and Pods. As a guide to accomplish this please check the following Yaml file and further repository associated:
https://github.com/fluent/fluentd-kubernetes-daemonset/blob/master/fluentd-daemonset-elasticsearch.yaml
https://github.com/fluent/fluentd-kubernetes-daemonset
If you need additional assistance you can also join our Slack channel:
http://slack.fluentd.org