I am currently monitoring Usage Metrics of Pods from Kubernetes (using Metricbeat) and storing it in DB for historical analysis.
If my Pod gets restarted the Pod name gets changed and will not be able to map the old Pod information with new Pod information.
Do we have any way to connect or identify the old and new information to form the final consolidated dataset?
Related
I have created a Kubernetes application (Say deployment D1, using docker image I1), that will run on client clusters.
Requirement 1 :
Now, I want to roll updates whenever I update my docker image I1, without any efforts from client side
(Somehow, client cluster should automatically pull the latest docker image)
Requirement 2:
Whenever, I update a particular configMap, the client cluster should automatically start using the new configMap
How should I achieve this ?
Using Kubernetes Cronjobs ?
Kubernetes Operators ?
Or something else ?
I heard that k8s Operator can be useful
Starting with the Requirement 2:
Whenever, I update a particular configMap, the client cluster should
automatically start using the new configMap
If configmap is mounted to the deployment it will get auto-updated however if getting injected as the Environment restart is only option unless you are using the sidecar solution or restarting the process.
For ref : Update configmap without restarting POD
How should I achieve this ?
ImagePullpolicy is not a good option i am seeing however, in that case, manual intervention is required to restart deployment and it
pulls the latest image from the client side and it won't be in a
controlled manner.
Using Kubernetes Cronjobs ?
Cronjobs you will run which side ? If client-side it's fine to do
that way also.
Else you can keep deployment with Exposed API which will run Job to
update the deployment with the latest tag when any image gets pushed
to your docker registry.
Kubernetes Operators ?
An operator is a good native K8s option you can write in Go,
Python or your preferred language with/without Operator framework or Client Libraries.
Or something else?
If you just looking for updating the deployment, Go with running the API in the deployment or Job you can schedule in a controlled manner, no issue with the operator too would be a more native and a good approach if you can create, manage & deploy one.
If in the future you have a requirement to manage all clusters (deployment, service, firewall, network) of multiple clients from a single source of truth place you can explore the Anthos.
Config management from Git repo sync with Anthos
You can build a Kubernetes operator to watch your particular configmap and trigger cluster restart. As for the rolling updates, you can configure the deployment according to your requirement. A Deployment's rollout is triggered if and only if the Deployment's Pod template (that is, .spec.template) is changed, for example, if the labels or container images of the template are updated. Add the specifications for rolling update on your Kubernetes deployment .spec section:
type: RollingUpdate
rollingUpdate:
maxSurge: 3 //the maximum number of pods to be created beyond the desired state during the upgrade
maxUnavailable: 1 //the maximum number of unavailable pods during an update
timeoutSeconds: 100 //the time (in seconds) that waits for the rolling event to timeout
intervalSeconds: 5 //the time gap in seconds after an update
updatePeriodSeconds: 5 //time to wait between individual pods migrations or updates
I have a clickhouse cluster that is running on pods in kubernetes and I have a clickhouse-explorer that should be checking one clickhouse node each. I was thinking about using daemonset for this but I don't know how to force in the config that each pod to connect to a different service. Note I don't want to use sidecontainer for the clickhouse.
Any suggestions?
Our horizontal scaling is currently suffering because of Liquibase.
We would want our deployments to always deploy one pod which runs Liquibase (-Dspring.liquibase.enabled=true), and then all subsequent pods to not run it (-Dspring.liquibase.enabled=false).
Is there anything that Kubernetes offers which could do this out of the box?
I'm unfamiliar with Liquibase and I'm unclear how non-first Pods leverage Liquibase but, you may be able to use a lock to control access. A Pod that acquires the lock sets the property to true and, if it is unable to acquire the lock, the property is false.
One challenge will be in ensuring that the lock is released if the first Pod terminates. And, to understand the consequence on the other Pods. Is an existing Pod promoted?
Even though Kubernetes leverages etcd for its own distributed locking purposes, users are encouraged to run separate etcd instances if they need locks. Since you have to choose, you may as well choose what you prefer e.g. Redis, Zookeeper.
You could use an init Container or sidecar for the locking mechanism and a shared volume to record its state.
It feels as though Liquibase should be a distinct Deployment exposed as a Service that all Pods access.
Have you contacted Liquibase to see what it recommends for Kubernetes deployments?
I just ported my application to OpenShift Online 3 (from version 2), and now I'm struggling to understand how to manage persistent, "shared" data, that is not wiped after each build.
After reading the documentation about Persistent Volume Claims, I created a new PVC inside my project, of type RWO, using the Web dashboard. At this point I tried to understand how to access this storage from inside each pod, or if I needed to do something to mount it, and I ended up doing this:
$ oc volume dc/myapp --add --type=persistentVolumeClaim --claim-name=pvcname --mount-path=/usr/share/data
After this, it looks like the new configuration was successfully registered:
$ oc volume dc --all
deploymentconfigs/myapp
pvc/pvcname (allocated 1GiB) as volume-jh1jf
mounted at /usr/share/data
I could also see the new /usr/share/data directory from inside the pods created by the new builds.
However, after making this change, all deployments started failing with this error:
Failed to attach volume "pvc-0b747c80-a687-11e7-9eb0-122631632f42" on node "ip-172-31-48-134.ec2.internal" with: Error attaching EBS volume "vol-0008c8127ff0f4617" to instance "i-00195cc4e1d31f8ce": VolumeInUse: vol-0008c8127ff0f4617 is already attached to an instance status code: 400, request id: 722f3797-f486-4739-ab4e-fe1826ae53af. The volume is currently attached to instance "i-089e2a60e525f447c"
from which it looks like my latest change had the effect of attaching the volume to a specific instance. But then how can I mount the volume to my pods so that it survives each build and deploy?
Because you are using an EBS volume type, you must set the deployment strategy on the deployment config to Recreate instead of Rolling. This is because an EBS volume can only be mounted on a single node in the cluster at a time. This means you cannot using a rolling deployment, nor scale your application above 1 replica, as both result in more than one instance and there is no guarantee they will be deployed to the same node.
I'm setting up rethinkdb cluster inside kubernetes, but it doesn't work as expected for high availability requirement. Because when a pod is down, kubernetes will creates another pod, which runs another container of the same image, old mounted data (which is already persisted on host disk) will be erased and the new pod will join the cluster as a brand new instance. I'm running k8s in CoreOS v773.1.0 stable.
Please correct me if i'm wrong, but that way it seems impossible to setup a database cluster inside k8s.
Update: As documented here http://kubernetes.io/v1.0/docs/user-guide/pod-states.html#restartpolicy, if RestartPolicy: Always it will restart the container if exits failure. It means by "restart" that it brings up the same container, or create another one? Or maybe because I stop the pod via command kubectl stop po so it doesn't restart the same container?
That's how Kubernetes works, and other solution works probably same way. When a machine is dead, the container on it will be rescheduled to run on another machine. That other machine has no state of container. Event when it is the same machine, the container on it is created as a new one instead of restarting the exited container(with data inside it).
To persistent data, you need some kind of external storage(NFS, EBS, EFS,...). In case of k8s, you may want to look into this https://github.com/kubernetes/kubernetes/blob/master/docs/design/persistent-storage.md This Github issue also has many information https://github.com/kubernetes/kubernetes/issues/6893
And in deed, that's the way to achieve HA in my opinion. Container are all stateless, they don't hold anything inside them. Any configuration needs for them should be store outside such as using thing like Consul or Etcd. By separating this like this, it's easier to restart a container
Try using PetSets http://kubernetes.io/docs/user-guide/petset/
That allows you to name your (pet) pods. If a pod is killed, then it will come back with the same name.
Summary of the petset feature is as follows.
Stable hostname
Stable domain name
Multiple pets of a similar type will be named with a "-n" (rethink-0,
rethink-1, ... rethink-n for example)
Persistent volumes
Now apps can cluster/peer together
When a pet pod dies, a new one will be started and will assume all the same "state" (including disk) of the previous one.