I would like to change the value (not the label name) of label instance in the Prometheus DB using a PromQL from metric rundeck_system_stats_uptime_since.
I managed to do this before ingestion using this:
- source_labels: [__meta_kubernetes_pod_container_name,__meta_kubernetes_pod_container_port_number]
action: replace
separator: ":"
target_label: instance
So I'm covered for future metrics, but I would like to do this for existing values for instance label.
Expected result:
rundeck_system_stats_uptime_since{app="rdk-exporter", instance="rdk-exporter:9620", [...]}
Since it's a container in k8s I'm not interested in the IP of that container/host/node etc. because it's always changing, I'm only interested in the metrics.
Thank you
You can use label_replace from MetricsQL
How it is work you can check in victoriametrics play
So for example I have three metrics
process_cpu_seconds_total{cluster_num="1"cluster_retention="1m"instance="play-1m-1-vmagent.us-east1-b.c.victoriametrics-test.internal:8429"job="vmagent"}
process_cpu_seconds_total{cluster_num="1"cluster_retention="accounting"instance="play-accounting-1-vmagent.us-east1-b.c.victoriametrics-test.internal:8429"job="vmagent"}
process_cpu_seconds_total{cluster_num="1"cluster_retention="admin"instance="play-admin-1-vmagent.us-east1-b.c.victoriametrics-test.internal:8429"job="vmagent"}
if I will use this query
label_replace(process_cpu_seconds_total{instance=~".*:8429"}, "instance", "some:$2", "instance", "(.*):(.*)")
I will get next response
process_cpu_seconds_total{cluster_num="1"cluster_retention="1m"instance="some:8429"job="vmagent"}
process_cpu_seconds_total{cluster_num="1"cluster_retention="accounting"instance="some:8429"job="vmagent"}
process_cpu_seconds_total{cluster_num="1"cluster_retention="admin"instance="some:8429"job="vmagent"}
where instance will have same host.
Hope this solution will help you.
Related
I am starting now with Kubernetes and the Operator SDK and I am trying to build my first operator and I have probably a simple question.
Question
How to detect a configuration change in the custom resource yaml in the reconcile loop and take an action according to the change?
I have some config properties specified in the my CR Spec:
apiVersion: my.example.com/v1alpha1
kind: StoreApp
metadata:
name: mystoreapp
spec:
username: technicalUser
password: abcd1234
catalogs:
- name: Bikes
description: Bikes_description
- name: Cars
description: Cars_description
I want when I add new custom resource of this kind my controller to create a new pod with my app image running inside (in a webserver). When my app is up and running for the first time I want to configure it (to add the catalogs from the spec) via HTTP request from the operator.
So far it's ok but I also what to change these catalogs while my app is up and running.
For example I want to add new catalog in the spec (through kubectl patch). My operator's reconcile method will be called and how can I understand that the spec is changed? I am not sure it's a good idea to make HTTP calls to my app to get all catalogs and compare them with the catalogs from the spec. Is this the correct way to understand there is a change?
I am thinking about two other ways to find that something is updated but I am not sure if they will work properly and are they the best way to do this.
First idea is to request the instance of StoreApp with client.Get(...) but as far as I understand this will call the API server and will get the updated version of mystoreapp. I read about some local index which acts like cache for these objects and I can check is there a difference between the cached object and the object returned from the API server. But I did not find how to get the object from this local index so I was not able to compare the two objects.
To create map in which I store the hash of the hole spec object and to check every time this hash with the hash of the object got with client.Get(...). I think this will work but there should be a better way to do this.
I read some Java Operators for K8s and there were methods like onAdd, onUpdate, onDelete. I couldn't find something similar in the Operatod SDK. Is there anything like this in the Operator SDK?
Every answer will be helpful. Thank you in advance!
Best Regards,
Hristiyan
The recommended practice is to look at the spec you received, and compare it to the state of the world/cluster, so retrieving the catalogs and comparing them to the spec is indeed the proper way to do it.
The reasoning for this recommandation is that the order of the events you get from Kubernetes is not guaranteed to be consistent, and it's also not guaranteed that you'll necessarily receive every event in a reasonable amount of time, or that you'll only receive each event once, so it's best to base your decision making on what was requested as compared to what is, rather than what specific event triggered the reconciliation.
I am trying to filter EC2 instances in prometheus.yml. Suppose that the following is part of the yml file. How can I use regex in the values to return instance that starts with, lets say, prod or qa or other labels? Is this possible without configuring relabeling?
ec2_sd_configs:
- region: us-east-1
access_key: access
secret_key: mysecret
profile: profile
filters:
- name: tag:Name
values:
- prod.*
- qa.*
It seems that Prometheus does not support regex in filtering API right now but it would be a nice feature if they can add it in future releases. What can be done in this situation is that you can add a separate tag on EC2 instances so you can filter based on those tag. Filtering at early stage is extremely helpful if you have large number of instances. Otherwise, you'll get a huge list and you need to go through a drop/keep phase via relabeling which still keeps a long list in service discovery panel and make it difficult to read.
In the next step, you can use relabeling to replace the address of each discovered instance from private IP to public IP. As a final touch, you can replace the instance name with the tag name so, for example, all instances of QA are labeled as QA.
ec2_sd_configs:
- region: value
access_key: value
secret_key: value
port: value
filters:
- name: tag:Name
values:
- qa
- prod
- some other types
relabel_configs:
- source_labels: [__meta_ec2_public_ip]
replacement: '${1}:your port number'
target_label: __address__
- source_labels: [__meta_ec2_tag_Name]
target_label: instance
I don't have any experience with AWS, but I believe its API does not support regular expressions in filtering API.
In general relabelling is the preferred way to do filtering. An example of how to achieve this would be (for consul, but that does not matter much): Prometheus: how to drop a target based on Consul tags
List of ec2 meta labels available is in prometheus docs at https://prometheus.io/docs/prometheus/latest/configuration/configuration/#ec2_sd_config
I see that Prometheus docs recommend using filters over relabelling for efficiency reasons when you have potentially thousands of instances. Using separate tag (for example "Env") that has values of "qa", "prod" etc. so they can be matched exactly (without regex/wildcard) would be an elegant solution here I'd guess?
I'm using Prometheus to do some monitoring but I can't seem to find a way to delete labels I no longer want. I tried using the DELETE /api/v1/series endpoint but it doesn't remove it from the dropdown list on the main Prometheus Graph page. Is there a way to remove them from the dropdown without restarting from scratch?
Thanks
This happens to me also, try to include the metric name when querying for labels' values like this:
label_values(node_load1, instance)
ref: http://docs.grafana.org/features/datasources/prometheus/
If you delete every relevant timeseries then it should no longer be returned. If this is not the case, please file a bug.
Prometheus doesn't provide the ability to delete particular labels, because this may result to duplicate time series with identical labelsets. For example, suppose Prometheus contains the following time series:
http_requests_total{instance="host1",job="foobar"}
http_requests_total{instance="host2",job="foobar"}
If instance label is removed, then these two time series will become identical:
http_requests_total{job="foobar"}
http_requests_total{job="foobar"}
Now neither Prometheus nor user can differentiate these two time series.
Prometheus provides only the API for deleting time series matching the given series selector - see these docs for details.
According to https://www.consul.io/docs/agent/options.html#_node_meta
I can associate with a consul node any metadata key/value pair.
Can prometheus read this metadata ?
I understand that only the following meta labels are available for prometheus:
__meta_consul_address: the address of the target
__meta_consul_node: the node name defined for the target
__meta_consul_tags: the list of tags of the target joined by the tag separator
__meta_consul_service: the name of the service the target belongs to
__meta_consul_service_address: the service address of the target
__meta_consul_service_port: the service port of the target
__meta_consul_service_id: the service ID of the target
__meta_consul_dc: the datacenter name for the target
But I would like to be absolutely sure that I miss nothing or there is no a trick to do it.
Thank you
That's not supported as the feature was only released a month ago, but feel free to send a pull request.
Yes. This was introduced into Prometheus 1.8
You can now simply reference __meta_consul_metadata_$KEYNAME
The following shows a prometheus label rewrite which filters the nodes 'location' metadata to a ldn fact which we've added to Consul agents running in London.
- source_labels: [__meta_consul_metadata_location]
separator: ;
regex: ldn
replacement: $1
action: keep
In Metric agg. by default we have Sum,count,avg,min,max,unique-count etc the functions I want to add my own customized function suppose sum/Unique_count how to implement it.
I guess it will required a code changes. Now currently all Kibana metrics are located there - https://github.com/elastic/kibana/tree/master/src/ui/public/agg_types/metrics. So, you need to clone Kibana at first, than add your_own_metric.js, which will be similar to other built-in metrics. Later on, you need to add your metrics to index.js, under https://github.com/elastic/kibana/blob/master/src/ui/public/agg_types/index.js and hopefully after you will build Kibana, you could use your custom version of it.
Some additional information - https://discuss.elastic.co/t/custom-metric-aggregation-plugin/70072/8