I'm trying influxdb continuous query with time offset. However the query didn't work.
Here is my CQ:
CREATE CONTINUOUS QUERY perf_cq_1m ON temp
BEGIN
SELECT mean(rate) AS rate INTO temp.autogen.perf_cq_1m FROM
temp.autogen.perflog GROUP BY time(1m, 2h) fill(0)
END
I'd like the query to shift forward 2h to test the functionality. But the clause GROUP BY time(1m, 2h) didn't do what it is demonstrated in the influxDB documentation.
Influx log shows that the continuous query execute with UTC time boundaries, regradless of the GROUP BY time[1m, 2h] clause. Log is as follows:
2019-03-29T02:40:00.510257Z info Continuous query execution (start) {"log_id": "0ETp9740000", "service": "continuous_querier", "trace_id": "0ETpwX~W000", "op_name": "continuous_querier_execute", "op_event": "start"}
2019-03-29T02:40:00.511279Z info Executing continuous query {"log_id": "0ETp9740000", "service": "continuous_querier", "trace_id": "0ETpwX~W000", "op_name": "continuous_querier_execute", "name": "perf_cq_1m", "db_instance": "temp", "start": "2019-03-29T02:39:00.000000Z", "end": "2019-03-29T02:40:00.000000Z"}
2019-03-29T02:40:00.512286Z info Executing query {"log_id": "0ETp9740000", "service": "query", "query": "SELECT mean(rate) AS rate INTO temp.autogen.perf_cq_1m FROM temp.autogen.perflog WHERE time >= '2019-03-29T02:39:00Z' AND time < '2019-03-29T02:40:00Z' GROUP BY time(1m, 2h) fill(0)"}
The localtime is 2019-03-29 10:40:00.
As far as i am concerned, the clause GROUP BY time(<interval>,<offset>) make CQ executes at localtime with time range in UTC time in the where time > '' and time < '' clause, rather than altering both execute time and time boundaries.
So did I miss anything?
Any help is appreciated.
Related
I have a SQL Server table Energy_CC with two columns: time [int] (Epoch time) and E_CC14 [float]. Every 30 minutes, the total amount of my energy (kWh) is appended to the data table - something like this:
time
E_CC14
1670990400
5469.00223
1670992200
5469.02791
1670994000
5469.056295
1670995800
5469.082706
1670997600
5469.10558
1670999400
5469.128534
I tried this SQL statement:
SELECT
MONTH(DATEADD(SS, i.time, '1970-01-01')),
i.E_CC14 AS mwh,
iprev.E_CC14 AS prevmwh,
(i.E_CC14 - iprev.E_CC14) AS diff
FROM
Energy_CC i
LEFT OUTER JOIN
Energy_CC iprev ON MONTH(iprev.time) = MONTH(i.time) - MONTH(DATEADD(month, -1, i.time))
AND DAY(iprev.time) = 1
WHERE
DAY(i.time) = 1
GROUP BY
MONTH(i.time), i.E_CC14, iprev.E_CC14;
I expect the result for monthly like this :
time
E_CC14
DECEMBER-22
10223
Any help would be greatly appreciated.
I'm using a transaction to see how long a device is RFM mode and the duration field increases with each table row. How I think it should work is that while the field is 'yes' it would calculate the duration that all events equal 'yes', but I have a lot of superfluous data that shouldn't be there IMO.
I only want to keep the largest duration event so I want to compare the current events duration to the next events duration and if its smaller than the current event, keep the current event.
index=crowdstrike sourcetype=crowdstrike:device:json
| transaction falcon_device.hostname startswith="falcon_device.reduced_functionality_mode=yes" endswith="falcon_device.reduced_functionality_mode=no"
| table _time duration
_time
duration
2022-10-28 06:07:45
888198
2022-10-28 05:33:44
892400
2022-10-28 04:57:44
896360
2022-08-22 18:25:53
3862
2022-08-22 18:01:53
7703
2022-08-22 17:35:53
11543
In the data above the duration goes from 896360 to 3862, and can happen on any date, and the duration runs in cycles like that where it increases until it starts over. So in the comparison I would keep the event at the 10-28 inflection point and so on at all other inflection points throughout the dataset.
How would I construct that multi event comparison?
By definition, the transaction command bundles together all events with the same hostname value, starting with the first "yes" and ending with the first "no". There is no option to include events by size, but there are options that govern the maximum time span of a transaction (maxspan), how many events can be in a transaction (maxevents), and how long the time between events can be (maxpause). That the duration value you want to keep (896360) is 10 days even though the previous transaction was only 36 minutes before it makes me wonder about the logic being used in this query. Consider using some of the options available to better define a "transaction".
What problem are you trying to solve with this query? It's possible there's another solution that doesn't use transaction (which is very non-performant).
Sans sample data, something like the following will probably work:
index=crowdstrike sourcetype=crowdstrike:device:json falcon_device.hostname=* falcon_device.reduced_functionality_mode=yes
| stats max(_time) as yestime by falcon_device.hostname
| append
[| search index=crowdstrike sourcetype=crowdstrike:device:json falcon_device.hostname=* falcon_device.reduced_functionality_mode=no
| stats max(_time) as notime by falcon_device.hostname ]
| stats values(*) as * by falcon_device.hostname
| eval elapsed_seconds=yestime-notime
Thanks for your answers but it wasn't working out. I ended up talking to some professional splunkers and got the below as a solution.
index=crowdstrike sourcetype=crowdstrike:device:json
| addinfo ```adds info_max_time```
| fields + _time, falcon_device.reduced_functionality_mode falcon_device.hostname info_max_time
| rename falcon_device.reduced_functionality_mode AS mode, falcon_device.hostname AS Hostname
| sort 0 + Hostname, -_time ``` events are not always returned in descending order per hostname, which would break streamstats```
| streamstats current=f last(mode) as new_mode last(_time) as time_change by Hostname ```compute potential time of state change```
| eval new_mode=coalesce(new_mode,mode."+++"), time_change=coalesce(time_change,info_max_time) ```take care of boundaries of search```
| table _time, Hostname, mode, new_mode, time_change
| where mode!=new_mode ```keep only state change events```
| streamstats current=f last(time_change) AS change_end by Hostname ```add start time of the next state as change_end time for the current state```
| fieldformat time_change=strftime(time_change, "%Y-%m-%d %T")
| fieldformat change_end=strftime(change_end, "%Y-%m-%d %T")
``` uncomment the following to sort by duration```
```| search change_end=* AND new_mode="yes"
| eval duration = round( (change_end-time_change)/(3600),1)
| table time_change, Hostname, new_mode, duration
| sort -duration```
We have a ingestion pipeline that will create indices every 2 hours, eg: index-2022-05-10-0 at 12am UTC, index-2022-05-10-1 at 2am UTC and so on..The problem is after 7pm UTC there is no index seen in Elasticsearch. Is it due to the timezone issue? But I know Elasticsearch uses UTC and ES servers are also configured on UTC.
What might be the issue? The new index for next day is created at UTC 12am correctly. And if I see the index creation time according to IST, its 5.30am.
Since I am working from India, and its 5.30 hours ahead of UTC so when its 7pm utc, then in IST the day changes and its 12.30am, is that the time zone issue due to which further indices are not created? Could someone please help?
Below is the pipeline code
...
"script": { "lang": "painless", "source": "Date d=new Date((long)(timestampfield)*1000); DateFormat f = new SimpleDateFormat("HH"); String crh=(Integer.parseInt(f.format(d))/2).toString(); String nvFormat="yyyy-MM-dd-"+crh; DateFormat f2=new SimpleDateFormat(nvFormat); ctx['_index']="index-"+f2.format(d);"
I have a Spring Boot application and I am using Spring Boot Actuator and Micrometer in order to track metrics about my application. I am specifically concerned about the 'http.server.requests' metric and the MAX statistic:
{
"name": "http.server.requests",
"measurements": [
{
"statistic": "COUNT",
"value": 2
},
{
"statistic": "TOTAL_TIME",
"value": 0.079653001
},
{
"statistic": "MAX",
"value": 0.032696019
}
],
"availableTags": [
{
"tag": "exception",
"values": [
"None"
]
},
{
"tag": "method",
"values": [
"GET"
]
},
{
"tag": "status",
"values": [
"200",
"400"
]
}
]
}
I suppose the MAX statistic is the maximum time of execution of a request (since I have made two requests, it's the the time of the longer processing of one of them).
Whenever I filter the metric by any tag, like localhost:9090/actuator/metrics?tag=status:200
{
"name": "http.server.requests",
"measurements": [
{
"statistic": "COUNT",
"value": 1
},
{
"statistic": "TOTAL_TIME",
"value": 0.029653001
},
{
"statistic": "MAX",
"value": 0.0
}
],
"availableTags": [
{
"tag": "exception",
"values": [
"None"
]
},
{
"tag": "method",
"values": [
"GET"
]
}
]
}
I am always getting 0.0 as a max time. What is the reason of this?
What does MAX represent (MAX Discussion)
MAX represents the maximum time taken to execute endpoint.
Analysis for /user/asset/getAllAssets
COUNT TOTAL_TIME MAX
5 115 17
6 122 17 (Execution Time = 122 - 115 = 17)
7 131 17 (Execution Time = 131 - 122 = 17)
8 187 56 (Execution Time = 187 - 131 = 56)
9 204 56 From Now MAX will be 56 (Execution Time = 204 - 187 = 17)
Will MAX be 0 if we have less number of request (or 1 request) to the particular endpoint?
No number of request for particular endPoint does not affect the MAX (see an image from Spring Boot Admin)
When MAX will be 0
There is Timer which set the value 0. When the endpoint is not being called or executed for sometime Timer sets MAX to 0. Here approximate timer value is 2 to 2.30 minutes (120 to 150 seconds)
DistributionStatisticConfig has .expiry(Duration.ofMinutes(2)) which sets the some measutement to 0 if there is no request has been made for last 2 minutes (120 seconds)
Methods such as public TimeWindowMax(Clock clock,...), private void rotate() Clock interface has been written for the same. You may see the implementation here
How I have determined the timer value?
For that, I have taken 6 samples (executed the same endpoint for 6 times). For that, I have determined the time difference between the time of calling the endpoint - time for when MAX set back to zero
MAX property belongs to enum Statistic which is used by Measurement
(In Measurement we get COUNT, TOTAL_TIME, MAX)
public static final Statistic MAX
The maximum amount recorded. When this represents a time, it is
reported in the monitoring system's base unit of time.
Notes:
This is the cases from metric for a particular endpoint (here /actuator/metrics/http.server.requests?tag=uri:/user/asset/getAllAssets).
For generalize metric of actuator/metrics/http.server.requests
MAX for some endPoint will be set backed to 0 due to a timer. In my view for MAX for /http.server.requests will be same as a particular endpoint.
UPDATE
The document has been updated for the MAX.
NOTE: Max for basic DistributionSummary implementations such as
CumulativeDistributionSummary, StepDistributionSummary is a time
window max (TimeWindowMax). It means that its value is the maximum
value during a time window. If the time window ends, it'll be reset to
0 and a new time window starts again. Time window size will be the
step size of the meter registry unless expiry in
DistributionStatisticConfig is set to other value explicitly.
You can see the individual metrics by using ?tag=url:{endpoint_tag} as defined in the response of the root /actuator/metrics/http.server.requests call. The details of the measurements values are;
COUNT: Rate per second for calls.
TOTAL_TIME: The sum of the times recorded. Reported in the monitoring system's base unit of time
MAX: The maximum amount recorded. When this represents a time, it is reported in the monitoring system's base unit of time.
As given here, also here.
The discrepancies you are seeing is due to the presence of a timer. Meaning after some time currently defined MAX value for any tagged metric can be reset back to 0. Can you add some new calls to your endpoint then immediately do a call to /actuator/metrics/http.server.requests to see a non-zero MAX value for given tag?
This is due to the idea behind getting MAX metric for each smaller period. When you are seeing these metrics, you will be able to get an array of MAX values rather than a single value for a long period of time.
You can get to see this in action within Micrometer source code. There is a rotate() method focused on resetting the MAX value to create above described behaviour.
You can see this is called for every poll() call, which is triggered every some period for metric gathering.
I created this device which sends a point to my webserver. My web server stores a Point instance which has the attributes created_at to reflect the point's creation time. My device consistently sends a request to my server at a 180s interval.
Now I want to see the periods of time my device has experienced outages in the last 7 days.
As an example, let's pretend it's August 3rt (08/03). I can query my Points table for points up to the last 3 days sorted by created_at
Points = [ point(name=p1, created_at="08/01 00:00:00"),
point(name=p2, created_at="08/01 00:03:00"),
point(name=p3, created_at="08/01 00:06:00"),
point(name=p4, created_at="08/01 00:20:00"),
point(name=p5, created_at="08/03 00:01:00"),
... ]
I would like to write an algorithm that can list out the following outages:
outages = {
"08/01": [ "00:06:00-00:20:00", "00:20:00-23:59:59" ],
"08/02": [ "00:00:00-23:59:59" ],
"08/03": [ "00:00:00-00:01:00" ],
}
Is there an elegant way to do this?