Does letting the kubelet write logs that pods emit on their stdout / stderr consume more resources such as CPU and memory, than if the pods wrote their logs directly to a mounted volume? I am asking this considering that the logs are copied from the pod to the kubelet across processes. Or does the disk IO by N pods instead of one kubelet offset any advantage?
I recognize that not letting kubelet handle the logs means kubectl logs ... would not show recent logs. But otherwise, could there be performance advantage to not writing through an intermediary like kubelet?
Related
I use company originated cloud environment which doesn't allow us to use node log collector and asks us to use a sidecar container to ship logs to ELK. However our container prints all the logs to stdout/stderr by following the 12 factor app practice.
Is there any way to collect stdout/stderr logs using a sidecar container (Prefered to use Filebeat)? I was checking documentation but it's not clearly mentioned anywhere.
There's another way in which I print all the logs to the specific directory within app container and share the volume with the sidecar Filebeat container. However, with this approach, I should think about the log rotation as well, which introduces more complexity. (Probably I need to create another container to aggregate logs and rotate logs.)
I think the recommended approach could be
container logs go stdout/stderr
Filebeat is deployed as Daemon Set
Use Hint Base Autodiscover
Consider one of the below options
Deploy Fluentd daemonset to collect the logs from all pods from a node
OR
Deploy Fluent-bit daemonset to collect logs and get it forward to Fluentd pod
Configure Fluentd pod to push the logs to Elasticsearch.
I have kubernetes pods writing logs to multiple log files using a persistent volume - nfs drive. I need a way to push the logs real time from the log files to ELastic Search.
I am trying to set up a filebeat as the sidecar container but not sure how it will help
Please suggest recommended approach with examples.
I am running a Laravel application on Kubernetes and currently have a requirement to mount a storage/log folder from multiple pods where all pods will write to the same laravel.log file. To achieve this, I used AWS EFS with the EFS-Provisioner for Kubernetes to mount the storage.
While troubleshooting a logging issue from my application, I noticed that when I log an entry to the laravel.log file from Pod A/B, it shows up in both Pod A/B when I tail the log however it does not show up in Pod C. If I log from Pod C, it will only show up in Pod C. I am working inside the container using php artisan tinker and Log::error('php-fpm'); for example as well as tail -f /var/www/api/storage/logs/laravel.log. The same behaviour happens if I echo "php-fpm" >> /var/www/api/storage/logs/laravel.log.
At first, I thought that I was possibly mounting the wrong storage however if I touch test in the folder, I can see it across Pod A, B, C. I can also see the other log files that are created with identical timestamps.
Any ideas on how to fix this?
Edit: I noticed that pod A, B which are both seeing each others log entries are in the same AZ. Pod C (and another Pod D running Nginx) are running in a different AZ. I just want to outline this, but I feel like this really shouldn't matter. It should be the same storage no matter where you are connecting from.
AWS EFS is accessed using NFS protocol and according to this stack exchange answer simultaneous writes from multiple NFS clients to the same file will be corrupted.
I'm not sure there is a way of "fixing" NFS itself but you can always just log to separate files.
Kubernetes breaks (no response from kubectl) when I have too many Pods running in the cluster (1000 pods).
There are more than enough resources (CPU and memory), so it seems to me that some kind of controller is breaking and unable to handle a large number of Pods.
The workload I need to run can be massively parallel processed, hence I have a high number of Pods.
Actually, I would like to be able to run many more times 1000 Pods. Maybe even 100,000 Pods.
My Kubernetes master node is an AWS EC2 m4.xlarge instance.
My intuition tells me that it is the master node's network performance that is holding the cluster back?
Any ideas?
Details:
I am running 1000 Pods in a Deployment.
when I do kubectl get deploy
it shows:
DESIRED CURRENT UP-TO-DATE AVAILABLE
1000 1000 1000 458
and through my application-side DB, I can see that there are only 458 Pods working.
when I do kops validate cluster
I receive the warning:
VALIDATION ERRORS
KIND NAME MESSAGE
ComponentStatus controller-manager component is unhealthy
ComponentStatus scheduler component is unhealthy
Pod kube-system/kube-controller-manager-<ip>.ec2.internal
kube-system pod
"kube-controller-manager-<ip>.ec2.internal" is not healthy
Pod
kube-system/kube-scheduler-<ip>.ec2.internal
kube-system pod "kube-scheduler-<ip>.ec2.internal" is not healthy
The fact that it takes a long time to list your pods is not really about your nodes as they will able to handle pods as much depending on the resources they have such CPUs and Memory.
The issue you are seeing is more about the kubeapi-server being able query/reply a large number of pods or resources.
So the two contention points here are the kube-apiserver and etcd where the state for everything in a Kubernetes cluster is stored. So you can focus on optimizing those two components and the faster you'll get responses from say kubectl get pods (Networking is another contention point but that's if you are issuing kubectl commands from a slow broadband connection).
You can try:
Setting up an HA external etcd cluster with pretty beefy machines and fast disks.
Upgrade the machines where your kubeapi-server(s) lives.
Follow more guidelines described here.
I've done quite a bit of research and have yet to find an answer to this. Here's what I'm trying to accomplish:
I have an ELK stack container running in a pod on a k8s cluster in GCE - the cluster also contains a PersistentVolume (format: ext4) and a PersistentVolumeClaim.
In order to scale the ELK stack to multiple pods/nodes and keep persistent data in ElasticSearch, I either need to have all pods write to the same PV (using the node/index structure of the ES file system), or have some volume logic to scale up/create these PVs/PVCs.
Currently what happens is if I spin up a second pod on the replication controller, it can't mount the PV.
So I'm wondering if I'm going about this the wrong way, and what is the best way to architect this solution to allow for persistent data in ES when my cluster/nodes autoscale.
Persistent Volumes have access semantics. on GCE I'm assuming you are using a Persistent Disk, which can either be mounted as writable to a single pod or to multiple pods as read-only. If you want multi writer semantics, you need to setup Nfs or some other storage that let's you write from multiple pods.
In case you are interested in running NFS - https://github.com/kubernetes/kubernetes/blob/release-1.2/examples/nfs/README.md
FYI: We are still working on supporting auto-provisioning of PVs as you scale your deployment. As of now it is a manual process.