Multiple replicas accessing a cache in kubernetes - go

One statistic for Prometheus logging is the duration of service calls but I want to measure the time over multiple calls of the same service.
So I figured to create a map of string to time.Time
type SomeService struct {
durations map[string]time.Time
}
On first entry the current time is stored for that account id
durations[GetId()] = time.Now()
And then in the end…in another call…the total time is stored.
startTime := c.durations[id]
duration, _ := c.durationStat.GetMetricWith(prometheus.Labels{"type": duration})
duration.Set(time.Now().Sub(startTime).Seconds())
delete(c.durations, id)
This works when there is only one replica but it breaks down in a Kubernetes cluster right? The next call might come in on another endpoint? How do you cache values in microservices so that every replica can access them?

Finally found this:
https://kubernetes.io/docs/concepts/services-networking/service
You can configure the service to have the same ip address always go to
the same pod by setting service.spec.sessionAffinity to “ClientIP” the
default is “None”
apiVersion: v1
kind: Service
metadata:
name: my-service
spec:
sessionAffinity: ClientIP
selector:
app: MyApp
This way you can safely cache simple values in memory!

Related

Custom gauge for prometheus Go SDK

I have a APP that monitors some external jobs (among other things). This does this monitoring every 5 Mins
I'm trying to create a prometheus gauge to get the count of currently running jobs.
Here is how I declared my gauge
JobStats= promauto.NewGaugeVec(
prometheus.GaugeOpts{
Namespace: "myapi",
Subsystem: "app",
Name: "job_count",
Help: "Current running jobs in the system",
ConstLabels: nil,
},
[]string{"l1", "l2", "l3"},
)
in the code that actually counts the jobs I do
metrics.JobStats.WithLabelValues(l1,l2,l3).add(float64(jobs_cnt))
when I query the /metrics endpoint I get the number
The thing is, this metrics only keeps increasing. If I restart the app this get resets to zer & again keeps increasing
I'm using grafana to graph this in a dashboard.
My question is
Get the graph to show the actual number of jobs (instead of ever increasing line)?
Should this be handled in code (like setting this to zero before every collection?) or in grafana?

How to state number of cpus in autoscaler config.yaml for specific node

I am trying to find the command for the config.yml for ray autoscaler
I know there are max_workers but that considers the cluster as a whole. I want to limit the number of cpus launched on each worker node.
for example:
worker_node:
max_cpus: 3
Head_node:
max_cpus: 4
How do I do that?
The number of CPUs per worker is determined by the worker's configuration which is provider specific. This is what the node_config field is for. For example, with AWSm if you wanted to specify a 4 CPU machine you would do something like
available_node_types:
cpu_4_ondemand:
node_config:
InstanceType: m4.xlarge
min_workers: 1
max_workers: 5
notice the InstanceType field, which is specific to EC2 (it's 4 cpus because that's how many cpus are on a m4.xlarge instance).
For Kubernetes, you would place a CRD in the node_config field. For example
node_config:
apiVersion: v1
kind: Pod
metadata:
# Automatically generates a name for the pod with this prefix.
generateName: ray-worker-
# Must match the worker node service selector above if a worker node
# service is required.
labels:
component: ray-worker
spec:
resources:
requests:
cpu: 4000m
memory: 512MiB
For more information, you may be interested in taking a look at the provider specific examples in the ray repo. For example, here are the aws examples: https://github.com/ray-project/ray/tree/master/python/ray/autoscaler/aws

clear prometheus metrics from collector

I'm trying to modify prometheus mesos exporter to expose framework states:
https://github.com/mesos/mesos_exporter/pull/97/files
A bit about mesos exporter - it collects data from both mesos /metrics/snapshot endpoint, and /state endpoint.
The issue with the latter, both with the changes in my PR and with existing metrics reported on slaves, is that metrics created lasts for ever (until exporter is restarted).
So if for example a framework was completed, the metrics reported for this framework will be stale (e.g. it will still show the framework is using CPU).
So I'm trying to figure out how I can clear those stale metrics. If I could just clear the entire mesosStateCollector each time before collect is done it would be awesome.
There is a delete method for the different p8s vectors (e.g. GaugeVec), but in order to delete a metric, I need to not only the label name, but also the label value for the relevant metric.
Ok, so seems it was easier than I thought (if only I was familiar with go-lang before approaching this task).
Just need to cast the collector to GaugeVec and reset it:
prometheus.NewGaugeVec(prometheus.GaugeOpts{
Help: "Total slave CPUs (fractional)",
Namespace: "mesos",
Subsystem: "slave",
Name: "cpus",
}, labels): func(st *state, c prometheus.Collector) {
c.(*prometheus.GaugeVec).Reset() ## <-- added this for each GaugeVec
for _, s := range st.Slaves {
c.(*prometheus.GaugeVec).WithLabelValues(s.PID).Set(s.Total.CPUs)
}
},

How to use the kubernetes go-client to get the same Pod status info that kubectl gives

Using the kubernetes go-client (k8s.io/client-go/kubernetes), I know how to get pod.Status and I find the pod.Status.Phase useful (docs). For example, I can output the Pod Status Phase of all Pods using this:
...
api := clientset.CoreV1()
pods, err := api.Pods("").List(metav1.ListOptions{})
for i, pod := range pods.Items {
podstatusPhase := string(pod.Status.Phase)
podCreationTime := pod.GetCreationTimestamp()
age := time.Since(podCreationTime.Time).Round(time.Second)
podInfo := fmt.Sprintf("[%d] Pod: %s, Phase: %s , Created: %s, Age: %s", i, pod.GetName(), podstatusPhase, podCreationTime, age.String())
fmt.Println(podInfo)
}
However, the phase is a little simplistic in that it only ever shows 5 values (Pending, Running, Succeeded, Failed, Unknown). I'd rather get the same info that kubectl get pods gives in the Status column, for example:
$ kubectl get pods
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
moby-dick-cron-scheduler-1564578660-bg4sb 0/2 ContainerCreating 0 178m <none> ip-10-30-13-151.ec2.internal <none> <none>
notifications-missed-calls-1564564740-js762 0/2 Init:0/1 0 6h49m <none> ip-10-30-13-6.ec2.internal <none> <none>
antivirus-scanner-cron-1564576740-sd6hh 0/2 Completed 0 3h30m 10.30.13.169 ip-10-30-13-151.ec2.internal <none> <none>
In particular, I'm interested in Init:0/1 and PodInitializing statuses. The Pods in these statuses just show as "Pending" when using pod.Status.Phase.
Init:0/1 means the Pod has 1 Init containers and 0 have completed successfully so far. init containers run before app containers are started.
PodInitializing means the Pod has already finished executing Init Containers.
Is there a way to get a Status such as Init:0/1 using k8s.io/client-go/kubernetes? or is there no short-cut, and I'd need to re-calculate it the same way kubectl does? I guess it uses Pod Status Conditions and container statuses to build the info. If I need to re-calculate it, maybe I can use the kubectl sourcecode? Does anyone know where I can find the relevant bit? (I have very limited golang experience)
The short answer is typically you don't have to calculate the 'Status' on the client since it's calculated at the server level.
To illustrate:
The standard way that you are trying to print with kubectl get pods, in the Kubernetes code base is called Human Readable. This method uses ServerPrint, which defaults to the Kubernetes TablePrinter. The TablePrinter type is defined here.
As you can see the PrintObj function for the TablePrinter gets delegated here. It delegates to the appropriate Kubernetes resource PrintObj. Also, that delegation comes together with the configured HumanPrintFlags and saving the original printer.
Also, you see that in humanreadable_glags.go it's including k8s.io/cli-runtime/pkg/printers, and you see that it's instantiating a printers.NewTablePrinter which is defined in k8s.io/kubernetes/pkg/printers.
The actual function to print that gets called is this PrintObj and you can see that it handles 3 cases since in some cases the server returns a table and some not (looks like < 1.16 cases).
You also see that in the above case none of the code in https://github.com/kubernetes/kubernetes/tree/4477bf02f211093b32cf58f64aa42aff77daea61/pkg/printers/internalversion is used, so that calculation happens behind the kube-apiserver side.
Keep in mind that this is the Human Readable printer and there other types of printers defined here (depending on the options): https://github.com/kubernetes/kubernetes/tree/master/staging/src/k8s.io/cli-runtime/pkg/printers
I think you need to re-calculate it. See this
I was able to show the exact same information as kubectl get pods. Please see the answer here: https://stackoverflow.com/a/74722781/7129053
you should use restClient to get raw table output
Receiving resources as Tables
the calculation is more complicated than you think,like Age field,in kubernetes,the code:
func HumanDuration(d time.Duration) string {...}

How to require one pod per minion/kublet when configuring a replication controller?

I have 4 nodes (kubelets) configured with a label role=nginx
master ~ # kubectl get node
NAME LABELS STATUS
10.1.141.34 kubernetes.io/hostname=10.1.141.34,role=nginx Ready
10.1.141.40 kubernetes.io/hostname=10.1.141.40,role=nginx Ready
10.1.141.42 kubernetes.io/hostname=10.1.141.42,role=nginx Ready
10.1.141.43 kubernetes.io/hostname=10.1.141.43,role=nginx Ready
I modified the replication controller and added these lines
spec:
replicas: 4
selector:
role: nginx
But when I fire it up I get 2 pods on one host. What I want is 1 pod on each host. What am I missing?
Prior to DaemonSet being available, you can also specify that you pod uses a host port and set the number of replicas in your replication controller to something greater than your number of nodes. The host port constraint will allow only one pod per host.
I was able to achieve this by modifying the labels as follows below
master ~ # kubectl get nodes -o wide
NAME LABELS STATUS
10.1.141.34 kubernetes.io/hostname=10.1.141.34,role=nginx1 Ready
10.1.141.40 kubernetes.io/hostname=10.1.141.40,role=nginx2 Ready
10.1.141.42 kubernetes.io/hostname=10.1.141.42,role=nginx3 Ready
10.1.141.43 kubernetes.io/hostname=10.1.141.43,role=nginx4 Ready
I then created 4 nginx replication controllers each referencing the nginx{1|2|3|4} roles and labels.
Replication controller doesn't guarantee one pod per node as the scheduler will find the best fit for each pod. I think what you want is the DaemonSet controller, which is still under development. Your workaround posted above would work too.

Resources