I'm having some issues with incorrect time stamps in InfluxDB. I receive data from a websocket connection, create a new point for each response, and upload each point to a bucket. When I look at the points in the InfluxDB UI, I notice that the time field is incorrect. Here are 4 examples of unmarshalled responses that I received from the websocket connection, as well as the time field in its respective point:
&types.TradeResponse{ChannelID:337, TradeArray:[]types.TradeDataResponse{types.TradeDataResponse{Price:21567.4, Volume:0.00553002, Time:time.Date(2022, time.August, 26, 3, 49, 29, 0, time.UTC), Side:"b", OrderType:"m", Misc:""}}, ChannelName:"trade", Pair:"XBT/USD"}
point time: 2022-08-26 03:49:29 +0000 UTC
&types.TradeResponse{ChannelID:337, TradeArray:[]types.TradeDataResponse{types.TradeDataResponse{Price:21567.4, Volume:0.0372093, Time:time.Date(2022, time.August, 26, 3, 49, 43, 0, time.UTC), Side:"b", OrderType:"m", Misc:""}}, ChannelName:"trade", Pair:"XBT/USD"}
point time: 2022-08-26 03:49:43 +0000 UTC
&types.TradeResponse{ChannelID:337, TradeArray:[]types.TradeDataResponse{types.TradeDataResponse{Price:21567.3, Volume:0.00045028, Time:time.Date(2022, time.August, 26, 3, 49, 59, 0, time.UTC), Side:"s", OrderType:"m", Misc:""}}, ChannelName:"trade", Pair:"XBT/USD"}
point time: 2022-08-26 03:49:59 +0000 UTC
&types.TradeResponse{ChannelID:337, TradeArray:[]types.TradeDataResponse{types.TradeDataResponse{Price:21567.3, Volume:0.00010686, Time:time.Date(2022, time.August, 26, 3, 50, 7, 0, time.UTC), Side:"s", OrderType:"m", Misc:""}}, ChannelName:"trade", Pair:"XBT/USD"}
point time: 2022-08-26 03:50:07 +0000 UTC
and here is a picture of the points in the bucket:
As you can see, the _time field is always slightly off from the time in the point. I'm currently using the go client to upload data to the DB. Here is the code that I have written for that:
func OnTradeResponse(data types.TradeResponse, tradesWriter api.WriteAPI) {
for _, trade := range data.TradeArray {
point := influxdb2.NewPoint("trade", map[string]string{"object": "trade", "pair": data.Pair}, map[string]interface{}{"price": float64(trade.Price), "volume": float64(trade.Volume), "side": trade.Side, "orderType": trade.OrderType, "misc": trade.Misc}, trade.Time.Time)
tradesWriter.WritePoint(point)
log.Println("point time: ", point.Time())
}
}
If I was just using the system time when creating new points, I could understand why there might be an issue with having an incorrect time, but since I'm using the time from the trade response as the time in the point, I don't understand what is causing the issue. Would greatly appreciate any help
Related
I am trying to creat a filter that will pull every account that has been set up after 5:00 PM from the previous day. The date and time exist in the same row. I have created a filter that works for the day but the next day, it pulls for two days. For example, here is what it looks like right now:
= Table.SelectRows(#"Sorted Rows", each [Driver ID] > #datetime(2021, 12, 29, 17, 0, 0))
I have tried changing it to the following so it would dynamically change as the days pass:
= Table.SelectRows(#"Sorted Rows", each DateTime.From([Driver ID]) > Date.AddDays(DateTime.From(Driver ID), -1))
But when I do this I get the following error:
Expression.Error: We cannot convert the value #datetime(2021, 12, 30, 0, 5, 0) to type Function.
Details:
Value=12/30/2021 12:05:00 AM
Type=[Type]
I have made sure the column type is in Date/Time format but that doesn’t seem to help.
Has anybody ran into this issue and know a good solution?
try
= Table.SelectRows(#"Sorted Rows", each Date.IsInPreviousNDays([DriverID], 1) and Time.From([DriverID])> #time( 17, 0, 0))
I'm trying to convert an ISO time string like '2021-05-11T18:21:35Z' to an int (seconds from epoch), which mktime() does, except it seems strange to me that it requires the weekday and yearday as part of the argument. In general, it seems unlikely that you would know this, and in my situation I don't.
Obviously in python this is doable with things like datetime, but in uPython these don't exist and I haven't seen a non-external-library way to do this.
Much like with regular Python, the values of weekday and yearday are ignored (they get computed from the other values and are only accepted so that you can pass mktime the tuple returned by e.g. localtime).
You can run:
MicroPython v1.14 on 2021-03-07; ESP module with ESP8266
Type "help()" for more information.
>>> import time
>>> res = time.mktime((2021, 5, 11, 18, 21, 35, 0, 0))
>>> res
674072495
>>> time.localtime(res)
(2021, 5, 11, 18, 21, 35, 1, 131)
We're using pyspark to parse a dataset containing date columns converted to an UTC timestamp with a code that looks like this:
from datetime import datetime
from pyspark.sql import SparkSession
from pyspark.sql.functions import col, to_timestamp, to_utc_timestamp
session = (SparkSession
.builder
.appName('test_spark')
.master('local[*]')
.config('spark.sql.session.timeZone', 'UTC')
.getOrCreate())
date_df = session.createDataFrame([{'date': '2017-10-29'}])
result_df = (date_df
.withColumn('date', to_timestamp(col('date')))
.withColumn('date', to_utc_timestamp(col('date'), 'Europe/Madrid')))
expected_df = session.createDataFrame([{'date': datetime(2017, 10, 29, 0, 0)}])
print("Expected")
print(expected_df.collect())
print("Result")
print(result_df.collect())
In Linux, this code prints (Spark logging omitted):
Expected
[Row(date=datetime.datetime(2017, 10, 29, 0, 0))]
Result
[Row(date=datetime.datetime(2017, 10, 29, 0, 0))]
In Windows 10, however, the output is:
Expected
[Row(date=datetime.datetime(2017, 10, 29, 0, 0))]
Result
[Row(date=datetime.datetime(2017, 10, 29, 1, 0))]
As you can see, in Windows the date parsed have 1 hour more than the result in Linux.
I have tested this with both OpenJDK JRE and Oracle JRE (always version 8) but the result is the same.
What the hell is happening here? If I modify the year or the month of the dates, the "bug" no longer happens, looks like it only appears in 29 October of 2017, what makes this day so special?
For example, by executing this:
http://:8088/ws/v1/cluster/metrics
I get an output like this:
{
"clusterMetrics": {
"appsSubmitted": 502521,
"appsCompleted": 501201,
"appsPending": 0,
"appsRunning": 19,
"appsFailed": 454,
"appsKilled": 847,
"reservedMB": 140400,
"availableMB": 12615232,
"allocatedMB": 8830800,
"reservedVirtualCores": 39,
"availableVirtualCores": 6140,
"allocatedVirtualCores": 2065,
"containersAllocated": 1692,
"containersReserved": 39,
"containersPending": 3960,
"totalMB": 21446032,
"totalVirtualCores": 8205,
"totalNodes": 199,
"lostNodes": 1,
"unhealthyNodes": 1,
"decommissionedNodes": 8,
"rebootedNodes": 0,
"activeNodes": 189
}
}
For instance, allocatedMB means what ?
Is it an instantaneous value ?
Is it averaged over an interval period ? The interval is configurable ?
The allocatedMB is the memory that has been assigned to the vcores (though not necessarily used). Yes, it is an instantaneous value. There is no interval, it's a snapshot of your cluster at that instant (minus the time it takes to compute these values from the data structures in the Resource Manager and then return it via the REST API).
If you want to translate your metrics it's saying:
You currently have 19 apps running.
These 19 apps are using a total of 2065 vcores.
These 2065 vcores have reserved 8830800 MB of memory for them
How to find the dates which are there in a week or month till date.
days_for_week should return 19,20,21 (assuming current date is 21st)
days_for_month should return 1..21 (assuming current date is 21st)
For the first, you could use Time.now.wday to get the current week day, then minus that will give you the date of beginning of this week.
For the second, it's much simpler, every month begin with 1st, right?
Assuming I'm reading your question correctly...
The second is simple:
def days_for_month
1..Date.today.day
end
The first requires a little algorithm to work back to Saturday:
def days_for_week
days = []
day = Date.today
until day.saturday?
days.unshift(day.day)
day -= 1
end
days
end
Active support provides a lot of useful methods like at_beginning_of_week, at_end_of_week, at_beginning_of_month etc ..
> Date.today.at_beginning_of_week
=> Mon, 20 May 2013
For this particular case, you could do
> (Date.today.at_beginning_of_week..Date.today).map &:day
=> [20, 21]
Similarly
> (Date.today.at_beginning_of_month..Date.today).map &:day
=> [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21]
or simply
> 1..Date.today.day