Prometheus Api Count of Metric for one day - go

Prometheus has echo_requests_total{code="200",host="localhost:8080",method="POST",url="/"} 3
metric. It counts requests while app is running. However I want to get count of requests for one day. For example after 7 day metric can be 5000 requests but I just want to get count of requests which sent today. I tried to find prometheus querying functions but I couldn't.

Related

For what reasons Elasticseach would not record jaeger spans at the same period everyday?

Context: I'm using Elasticsearch to record jaeger spans coming from 3 services
Issue: For an unknown reason, everyday from 3pm to midnight the next day, no spans are recorded except for one of those services
Here is the count of record, on a 5 days period, for the service that have no elasticsearch recording issue:
And here, the count of record for the other 2, on the same period:
Question: Does something could periodically happen on ES cluster that could explain this ? What could I check to assert there is no issue on my ES cluster ?

Grafana with Elastic - Show requests count toguether with average response time

I'm new at Grafana and I'm trying to create a graph that shows the requests count together with the average response time for the requests, I was able to create my requests count but now I'm struggling to add the information with the requests time, there is an option to show both information inside a panel? Or do I need to create two panels, one with the request count and another with the average time?
And another question, there is an option to show the average time in milliseconds?

Subtract two metric results elasticsearch

I have two metric already made.
1st metric represents the number of transactions started by client
2nd metric represents the number of transactions received by server
I want to get the number of transactions which failed(are sent by client but not received by server) which is simple subtraction
Can I achieve this in Kibana?
There is a plugin for Kibana 5.0.0+. It is based on the core Metric-Plugin but gives you the ability to output custom aggregates on metric-results by using custom formula and/or JavaScript.
You can check more details Here .

Query for the lack of requests in specific points in time

I have an Elasticsearch/kibana stack that stores every request the application receives. It stores gereneral information about the request (RequestTimestamp, IP, Headers, HttpStatus, Route etc), and there's at least some requests per minute.
I would like to know if there's some way to query Kibana/Elastic to know the points in time that the application didn't receive any request for, let's say, 3 minutes.
I know it can be done programmatically, but it needs to be purely done with querys (so I can show it on the Dashboard).
You could do date histogram aggregation.
You could specify 3m interval and query for a specified day.
So you would get 24*60/3 = 480 values for each day.
You could plot it on the chart and see the gaps.
If you are an expert ES user you could try filtering the aggregations using bucket selector pipeline aggregation or create a moving average using moving average aggregation.

How to aggregate same events in logstash into one new event

Lets say I get 10 login failures in about 60 seconds and I don't want to see all 10 events but only one that say there was 10 failed login attempts.
How (if it is possible) can I aggregate number of same events in logstash into one new event?
Use the metrics filter.
It will allow you to flush every XX seconds (configurable) a metric event that can consists of various valuable metrics (e.g: percentiles of duration, rates, and count). You can use the filter and specify that you want an event to be flushed every 60 seconds, with the count of number of failed login attempts. The event will be flushed regardless if the count is higher or lower than 10, but you can query elastic search, whether manually or with Kibana, for only metric events where count is higher than 10.

Resources