We have an EKS cluster running and we are looking for best practices to ship application logs from pods to Elastic.
In the EKS workshop there is an option to ship the logs to cloudwatch and then to Elastic.
Wondered if there is an option to ship the logs directly to Elastic, or to understand best practices.
Additional requirement:
We need the logs to determine from which namespace the logs is coming from and to deliver a dedicated index
You can deploy EFK stack in kubernetes cluster. Follow the reference --> https://github.com/acehko/kubernetes-examples/tree/master/efk/production
Fluentd would be deployed as DaemonSet so that one replica is run on each node collecting the logs from all pods and push them to elasticsearch
Related
I am trying to add ELK to my project which is running on kubernetes. I want to pass by filebeat -> logstach then elastic search. I prepared my filebeat.yml file and in my company the filebeat is configured as an agent in the cluster which i don't realy know what it means? I want to know how to configure the filebeat in this case ? just adding the file in the project and it will be taken into considiration once the pod started or how does it work ?
You can configure the Filebeat in some ways.
1 - You can configure it using the DeamonSet, meaning each node of your Kubernetes architecture will have one POD of Filebeat. Usually, in this architecture, you'll need to use only one filebeat.yaml configuration file and set the inputs, filters, outputs (output to Logstash, Elasticsearch, etc.), etc. In this case, your filebeat will need root access inside your cluster.
2 - Using Filebeat as a Sidecar with your application k8s resource. You can configure an emptyDir in the Deployment/StatefulSet, share it with the Filebeat Sidecar, and set the Filebeat to monitor this directory.
Does metricbeat need always an agent running separately from the ELK cluster or it provides a plugin/agent/approach to run metricbeat on the cluster side?
If I understand your question, you want to know if their is a way to monitor your cluster without installing a beat.
You can enable monitoring in the stack monitoring tab of Kibana.
If you want more, beats are standalone objects pluggables with logstash or Elasticsearch.
Latest versions of Elastic Stack (formally known as ELK ) offer more centralized configurations in Kibana, and the 7.9 version introduce a unified elastic agent in Beta to gather several beats in one and manage you "fleet" on agent within Kibana.
But information used by your beats are not directly part of Elastic (CPU, RAM, Logs, etc...)
So you'll still have to install a daemon on your system.
I am trying to decide an approach for logs processing in a EKS cluster. Idea is to use EFK. We thought we can use fluentd to push the logs to elastic search. But most of the blogs uses fluentd to send the logs to cloudwatch and then a lambda to send the cloudwatch logs to elastic search. Why is this approach preferred? What could be the drawbacks of pushing logs directly to elastic?
Thanks!
I have been using EKF in EKS and sending logs directly to elasticsearch using a dynamic index key.
My elasticsearch cluster is also running inside Kubernetes and I am running fluentd as a daemon set. I haven't found any problems yet in this approach.
I'm running an elasticsearch cluster using StatefulSets on Google Container Engine (GKE) (my k8s configs are very similar to the ones here: https://github.com/pires/kubernetes-elasticsearch-cluster/tree/master/stateful)
I created the k8s cluster with --enable-stackdriver-kubernetes
Now I want to also install & use the Stackdriver elasticsearch plugin:
https://cloud.google.com/monitoring/agent/plugins/elasticsearch
Should I install the Stackdriver monitoring agent+plugin inside the ES pods? or on the nodes?
If you're using Stackdriver logging agent to generate/export logs for pods running elasticsearch on a Kubernetes cluster, you can have Stackdriver logging enabled for the cluster (this is enabled by default and can be enabled/disabled through the Console), the Stackdriver logging agent will be deployed on the cluster.
If the logging agent is running on the cluster, logs from each container will automatically be gathered, formatted and exported by the logging agent to Stackdriver Logging [1] for the deployed pods/containers, including elasticsearch.
Kubernetes does things differently to Compute Engine instances in terms of the monitoring agent. If Stackdriver Monitoring is enabled for the cluster, pods are deployed running the Kubernetes Engine version of the Stackdriver Agent and in the case of Kubernetes these come in the form of heapster pods as explained in more detail here [2].
[1] https://cloud.google.com/kubernetes-engine/docs/how-to/loggin
[2] https://cloud.google.com/monitoring/kubernetes-engine/customizing
I am trying to do the log monitoring of Kubernetes cluster using EFK. I got Kibana dashboard but it doesn't show any logs of Kubernetes cluster.
Here is the link which I followed in my task.By default my dashboard shows like
After that i changed the index-pattern in dashboard as
Then it showed as
My dought is, how Can i view the logs of each and every pod logs in kubernetes cluster?
Could anybody suggest me how to do the log monitoring of kubernetes cluster using EFK?
Note: in order for Fluentd to work, every Kubernetes node must be
labeled with beta.kubernetes.io/fluentd-ds-ready=true, as otherwise
the Fluentd DaemonSet will ignore them.
Have you made sure to address this?