As logs go away as soon as pods crashes, I would like to store them directly on my local machine. I don't want to use GCE. Also I would have multiple nodes of a service, so HostPath will not be of any use.
kubectl logs <pod-name> > log.txt will just capture a snapshot. I want the complete logs to be persistent on my local machine.
Logs are already on nodes in /var/lib/docker/containers/CONTAINER_ID/CONTAINER_ID-json.log
For collecting them you can use fluent-bit. The easiest approach would be using kubernetes to manage fluent-bit - run daemon set with host path volumes.
Here is helm chart that can do it for you: https://github.com/helm/charts/tree/master/stable/fluent-bit
Related
When I work with AWS, I can create storage volumes and bind them in kubernetes.
I would like to mount the persistent volume locally, in order to inspect the volume content and manipulate files with a local script.
Is there a handy way to mount a persistent volume to the client host, with something like: kubectl niceMountCommand my-pvc /data/local/my-pvc
I already know about kubectl cp and the possibility to add a dummy pod to access data, but I would have to adapt every script that manipulates data to exec commands through kubectl exec.
as I am green to that subject, could you pls. help.
I deploy docker image to gcloud kubernetes.
What to do to make the cluster reread the docker image when a new one would appear?
My code is:
sudo docker build -t gcr.io/${PROJECT_ID}/sf:$ENV .
sudo docker push gcr.io/${PROJECT_ID}/sf:$ENV
sudo gcloud container clusters create sf:$ENV --num-nodes=3
sudo kubectl run sfmill-web$ENV --image=gcr.io/${PROJECT_ID}/sf:$ENV --port 8088
sudo kubectl expose deployment sfmill-web$ENV --type=LoadBalancer --port 8088 --target-port 8088
kubectl set image deployment/sfmill-web$ENV sf=sf:$ENV
I encourage you to explore use Kubernetes configuration files to define resources.
You can explore the YAML for your deployment with:
kubectl get deployment/sfmill-web$ENV --output=yaml > ${PWD}/sfmill-web$ENV.yaml
You could then tweak the value of the image property and then reapply this to your cluster using:
kubectl apply --filename=${PWD}/sfmill-web$ENV.yaml
The main benefit to the configuration file approach is that you're effectively creating code to manage your infrastructure and, each time you change your code, you could check it into source control thereby knowing what you did at each stage.
Using kubectl is great but it makes it more challenging to recreate the cluster from scratch.... Which kubectl command did I perform next? Yes, you could (bash) script all your kubectl commands too which would help but configuration files remain the ideal solution.
HTH
I am using Kubernetes to deploy all my microservices provided by Azure Kubernetes Services.
Whenever I release an update of my microservice which is getting frequently from last one month, it pulls the new image from the Azure Container Registry.
I was trying to figure out where do these images reside in the cluster?
Just like Docker stores, the pulled images in /var/lib/docker & since the Kubernetes uses Docker under the hood may be it stores the images somewhere too.
But if this is the case, how can I delete the old images from the cluster that are not in use anymore?
Clusters with Linux node pools created on Kubernetes v1.19 or greater default to containerd for its container runtime (Container runtime configuration).
To manually remove unused images on a node running containerd:
Identity node names:
kubectl get nodes
Start an interactive debugging container on a node (Connect with SSH to Azure Kubernetes Service):
kubectl debug node/aks-agentpool-11045208-vmss000003 -it --image=mcr.microsoft.com/aks/fundamental/base-ubuntu:v0.0.11
Setup crictl on the debugging container (check for newer releases of crictl):
The host node's filesystem is available at /host, so configure crictl to use the host node's containerd.sock.
curl -sL https://github.com/kubernetes-sigs/cri-tools/releases/download/v1.23.0/crictl-v1.23.0-linux-amd64.tar.gz | tar xzf - -C /usr/local/bin \
&& export CONTAINER_RUNTIME_ENDPOINT=unix:///host/run/containerd/containerd.sock IMAGE_SERVICE_ENDPOINT=unix:///host/run/containerd/containerd.sock
Remove unused images on the node:
crictl rmi --prune
You are correct in guessing that it's mostly up to Docker, or rather to whatever the active CRI plugin is. The Kubelet automatically cleans up old images when disk space runs low so it's rare that you need to ever touch it directly, but if you did (and are using Docker as your runtime) then it would be the same docker image commands as per normal.
I was trying to figure out where do these images reside in the
cluster?
With the test and check, the result shows each node in the AKS cluster installed the Docker server, and the images stored like Docker as you say that the image layers stored in the directory /var/lib/docker/.
how can I delete the old images from the cluster that are not in use
anymore?
You can do this through the Docker command inside the node. Follow the steps in Connect with SSH to Azure Kubernetes Service (AKS) cluster nodes to make a connection to the node, then you could delete the image through the Docker CLI docker rmi image_name:tag, but carefully with it, make sure the image is really no more useful.
I am using kubernetes helm to deploy apps to my cluster. Everything works fine from my laptop when helm uses the cluster's kube-config file to deploy to the cluster.
I want to use helm from my CI/CD server (which is separate from my cluster) to automatically deploy apps to my cluster. I have created a k8s service account for my CI/CD server to use. But how do I create a kube-config file for the service account so that helm can use it to connect to my cluster from my CI/CD server??
Or is this not the right way to use Helm from a CI/CD server?
Helm works by using the installed kubectl to talk to your cluster. That means that if you can access your cluster via kubectl, you can use helm with that cluster.
Don't forget to make sure you're using to proper context in case you have more than one cluster in you kubcfg file. You can check that by running kubectl config current-context and comparing that to the cluster details in the kubecfg.
You can find more details in Helm's docs, check the quick start guide for more information.
why not just run your CI server inside your kubernetes cluster then you don't have to manage secrets for accessing the cluster? We do that on Jenkins X and it works great - we can run kubectl or helm inside pipelines just fine.
In this case you will want to install kubectl on whichever slave or agent you have identified for use by your CI/CD server, OR install kubectl on-the-fly in your automation, AND then make sure you have OR are able to generate a kubeconfig to use.
To answer the question:
But how do I create a kube-config file for the service account ...
You can set new clusters, credentials, and contexts for use with kubectl in a default or custom kubeconfig file using kubectl config set-cluster, kubectl config set-credentials, and kubectl config set-context. If you have KUBECONFIG env variable set and pointing to a kubeconfig file, that works or when setting new entries simply pass -kubeconfig to point to a custom file.
Here's the relevant API documentation for v1.6.
We created helmsman which provides you with declarative syntax to manage helm charts in your cluster. It configures kubectl (and therefore helm) for you wherever you run it. It can also be used from a docker container.
This is an abstract question and I hope that I am able to describe this clear.
Basically; What is the workflow in distributing of source code to Kubernetes that is running in production. As you don't run Docker with -v in production, how do you update running pods.
In production:
Do you use SaltStack to update each container in each pod?
Or
Do you rebuild Docker images and restart every pod?
Locally:
With Vagrant you can share a local folder for source code. With Docker you can use -v, but if you have Kubernetes running locally how would you mirror production as close as possible?
If you use Vagrant with boot2docker, how can you combine this with Docker -v?
Short answer is that you shouldn't "distribute source code", you should rather "build and deploy". In terms of Docker and Kubernetes, you would build by means of building and uploading the container image to the registry and then perform a rolling update with Kubernetes.
It would probably help to take a look at the specific example script, but the gist is in the usage summary in current Kubernetes CLI:
kubecfg [OPTIONS] [-u <time>] [-image <image>] rollingupdate <controller>
If you intend to try things out in development, and are looking for instant code update, I'm not sure Kubernetes helps much there. It's been designed for production systems and shadow deploys are not a kind of things one does sanely.