I'm trying to normalize an array of elements in a time range. Say you have 20 bank transactions that occur on Jan 1st, 2022
transaction 1 - 2022/01/01
transaction 2 - 2022/01/01
...
transaction 20 - 2022/01/01
we don't have other data than the day they occurred, but we still want to assign them an hour of the day, so they end as:
transaction 1 - 2022/01/01 00:00
transaction 2 - 2022/01/01 ??:??
...
transaction 20 - 2022/01/01 23:59
In Go I have this function that try to calculate the normalization of a time of day for an index in an array of elements:
func normal(start, end time.Time, arraySize, index float64) time.Time {
delta := end.Sub(start)
minutes := delta.Minutes()
duration := minutes * ((index+1) / arraySize)
return start.Add(time.Duration(duration) * time.Minute)
}
Howeve, I get an unexpected calculation of 2022/1/1 05:59 for index 0 in an array of 4 elements in a time range of 2022/1/1 00:00 to 2022/1/1 23:59, instead I would expect to see 2022/1/1 00:00. The only that works fine these conditions is index 3.
so, what am I doing wrong with my normalization?
EDIT:
Here is the function fixed thanks to #icza
func timeIndex(min, max time.Time, entries, position float64) time.Time {
delta := max.Sub(min)
minutes := delta.Minutes()
if position < 0 {
position = 0
}
duration := (minutes * (position / (entries - 1)))
return min.Add(time.Duration(duration) * time.Minute)
}
There is an example: Let's say our start and end date is 2022/01/01 00:00 - 2022/01/01 00:03, also we have 3 entries in our array of bank transactions and that we want to get the normalized time for the transaction nº 3 (2 in the array):
result := timeIndex(time.Date(2022, time.January, 1, 0, 0, 0, 0, time.UTC), time.Date(2022, time.January, 1, 0, 3, 0, 0, time.UTC), 3, 2)
since there is only 4 minutes between the starting and ending times (from 00:00 to 00:03) and want to find the normalized time for the last entry (index 2) in the array (size 3) the result should be:
fmt.Printf("%t", result.Equal(time.Date(2022, time.January, 1, 0, 3, 0, 0, time.UTC))
// prints "true"
or the last minute in the range, which is 00:03.
Here is a reproducible example: https://go.dev/play/p/EzwkqaNV1at
Between n points there are n-1 segments. This means if you want to include start and end in the interpolation, the number of time periods (being delta) is arraySize - 1.
Also if you add 1 to the index, you can't possibly have start as the result (you'll skip the 00:00).
So the correct algorithm is this:
func normal(start, end time.Time, arraySize, index float64) time.Time {
minutes := end.Sub(start).Minutes()
duration := minutes * (index / (arraySize - 1))
return start.Add(time.Duration(duration) * time.Minute)
}
Try it on the Go Playground.
Also note that if you have many transactions (in the order of the number of minutes in a day which is around a thousand), you may easily end up having multiple transactions having the same timestamp (same hour and minute). If you want to avoid this, use a smaller precision than minute, e.g. seconds or milliseconds:
func normal(start, end time.Time, arraySize, index float64) time.Time {
sec := end.Sub(start).Seconds()
duration := sec * (index / (arraySize - 1))
return start.Add(time.Duration(duration) * time.Second)
}
Yes, this will result in timestamps where the seconds is also not necessarily zero, but will ensure different, unique timestamps for higher transaction numbers.
If you have transactions in the order of magnitude that is close to the number of seconds in a day (which is 86400), then you can complete drop this "unit" and use time.Duration itself (which is the number of nanoseconds). This will guarantee timestamp uniqueness even for the highest number of transactions:
func normal(start, end time.Time, arraySize, index float64) time.Time {
delta := float64(end.Sub(start))
duration := delta * (index / (arraySize - 1))
return start.Add(time.Duration(duration))
}
Testing this with 1 million transactions, here are the first 15 time parts (they defer only in their sub-second part):
0 - 00:00:00.00000
1 - 00:00:00.08634
2 - 00:00:00.17268
3 - 00:00:00.25902
4 - 00:00:00.34536
5 - 00:00:00.43170
6 - 00:00:00.51804
7 - 00:00:00.60438
8 - 00:00:00.69072
9 - 00:00:00.77706
10 - 00:00:00.86340
11 - 00:00:00.94974
12 - 00:00:01.03608
13 - 00:00:01.12242
14 - 00:00:01.20876
15 - 00:00:01.29510
16 - 00:00:01.38144
17 - 00:00:01.46778
18 - 00:00:01.55412
19 - 00:00:01.64046
Try this one on the Go Playground.
Related
I have a fixed data structure given to me that has the fields YearDay and TimeOfDay. YearDay is the number of days that have passed in the current year, TimeOfDay is the number of seconds that have passed in the current day (up to 86400). YearDay is an int32, while TimeOfDay is a float64.
I want to convert this to time.Now().UnixNano() form but am unsure how to convert it. The time module has a YearDay(), but no inverse function given an yearDay (int32) (and probably a year), to give me the month and day in the month.
Ideally I'd like to somehow parse
d := time.Date(time.Year(), month, day, hour, min, sec, ms, time.UTC)
where month, day, hour, min, sec, ms was somehow predetermined, or something of equivalent that I can convert easily to any form I'd like (but mainly UnixNano()).
My best imagination would be a complicated switch statement that subtracted 31, 28(29), 30, 31 ... and to see when the int is finally negative to find the month and day, but it would have to be two switch statements with a leap year check to choose which switch block to use, while doing several remainder calculations on TimeOfDay. Is there a simpler and cleaner way?
Edit: I ended up making the following function while playing around with it, but I'll definitely be using Icza's solution. Nice to know days can overflow. Thanks!
func findMonthAndDay(yearDay int32) (int32, int32) {
year := time.Now().Year()
isLeapYear := year%400 == 0 || year%4 == 0 && year%100 != 0 // Calculates if current year is leapyear
// Determines which array to send to for loop
var monthsOfYear [12]int32
if isLeapYear {
monthsOfYear = [12]int32{31, 29, 31, 30, 31, 30, 31, 31, 30, 31, 30, 31}
} else {
monthsOfYear = [12]int32{31, 28, 31, 30, 31, 30, 31, 31, 30, 31, 30, 31}
}
var currentMonth int32
var currentDayInMonth int32
// Loop through array of months
for i := range monthsOfYear {
// If yearDay - next month #OfDays positive, not correct month
if yearDay-monthsOfYear[i] > 0 {
// Subtract month #OfDays and continue
yearDay = yearDay - monthsOfYear[i]
} else {
currentMonth = int32(i + 1) // Month found (+1 due to index at 0)
currentDayInMonth = yearDay // Remainder of YearDay is day in month
break
}
}
return currentMonth, currentDayInMonth
}
You may use Time.AddDate() to add the number of days to a time.Time value. It is OK to add days greater than 31, the implementation normalizes the result.
And convert TimeOfDay to time.Duration and use Time.Add() to add it. When converting to time.Duration, we may multiply it by 1e9 to get number of nanoseconds, so fractional seconds will be retained.
Example:
t := time.Date(2020, 1, 1, 0, 0, 0, 0, time.UTC)
fmt.Println(t)
var yearDay int32 = 100
var timeOfDay float64 = 70000.5
t = t.AddDate(0, 0, int(yearDay))
t = t.Add(time.Duration(timeOfDay * 1e9))
fmt.Println(t)
fmt.Println("Unix:", t.Unix())
fmt.Println("UnixNano:", t.UnixNano())
Output (try it on the Go Playground):
2020-01-01 00:00:00 +0000 UTC
2020-04-10 19:26:40.5 +0000 UTC
Unix: 1586546800
UnixNano: 1586546800500000000
I'm trying to find the best way to fund the cumulative product of a Gorgonia Tensor. For example:
T := ts.New(ts.WithShape(3,4), ts.WithBacking(ts.Range(ts.Float32,1,13)))
for row := 1; row < 3; row++ {
for col :=0; col < 4; col++ {
val1, _ := T.At(row-1, col).(float32)
val2, _ := T.At(row, col).(float32)
newVal := val1*val2
T.SetAt(newVal, row, col)
}
}
I want the output to be:
[1 2 3 4 ]
[5 12 21 32 ]
[45 120 231 384]
This works, but for modest sized tensors (500x50), it takes 2500 microseconds seconds per iteration. I need it to be much faster. The numpy .cumprod() takes about 10-20 microseconds for a similar computation. Is there a more efficient way to code this?
I'm using tensors because I need to do matrix multiplication, and I want to use float32 for memory constraints. If necessary, I could switch it to gonum matrices, but I would rather not have to spend the extra memory if there is a way to do it with float32.
Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 3 years ago.
Improve this question
How to get seconds of day (1 - 86400) in Go?
Just like http://joda-time.sourceforge.net/apidocs/org/joda/time/base/AbstractDateTime.html#getSecondOfDay()
Update/Clarifications
Wanted equivalent of the joda/SecondOfDay in golang
Expected to start at 0 and 86400 at end of the day
Required when rewriting in golang a java opensource function which in turn is using joda/secondOfDay
Googled '24 hour to seconds' to get 86400
While asking question I could only think of now.Unix()-yesterdayMidnight.Unix() and the simple accepted answer didn't come to my mind
Obviously didn't think about Daylight Saving
Wanted to see if there is some built in function or popular/standard library
If we define "seconds of day" as the "elapsed seconds since midnight", then to get correct result even on days when daylight saving time happens we should subtract the time representing midnight from the given time. For that, we may use Time.Sub().
func daySeconds(t time.Time) int {
year, month, day := t.Date()
t2 := time.Date(year, month, day, 0, 0, 0, 0, t.Location())
return int(t.Sub(t2).Seconds())
}
Testing it:
for _, t := range []time.Time{
time.Date(2019, 1, 1, 0, 0, 30, 0, time.UTC),
time.Date(2019, 1, 1, 0, 1, 30, 0, time.UTC),
time.Date(2019, 1, 1, 0, 12, 30, 0, time.UTC),
time.Date(2019, 1, 1, 12, 12, 30, 0, time.UTC),
} {
fmt.Println(daySeconds(t))
}
Output (try it on the Go Playground):
30
90
750
43950
Let's see how this function gives correct result when daylight saving time happens. In Hungary, 25 March 2018 is a day when the clock was turned forward 1 hour at 02:00:00, from 2 am to 3 am.
loc, err := time.LoadLocation("CET")
if err != nil {
fmt.Println(err)
return
}
t := time.Date(2018, 3, 25, 0, 0, 30, 0, loc)
fmt.Println(t)
fmt.Println(daySeconds(t))
t = t.Add(2 * time.Hour)
fmt.Println(t)
fmt.Println(daySeconds(t))
This outputs (try it on the Go Playground):
2018-03-25 00:00:30 +0100 CET
30
2018-03-25 03:00:30 +0200 CEST
7230
We print the daySeconds of a time being 30 seconds after midnight, which is 30 of course. Then we add 2 hours to the time (2 hours = 2*3600 seconds = 7200), and the daySeconds of this new time will be properly 7200 + 30 = 7230, even though the time changed 3 hours.
Note:
This function returns the nominal number of seconds of day in the (0 - 86399) range. If you're looking for the "number of seconds elapsed since midnight", which may not be in (0 - 86399) range due to daylight saving time, please see #icza's answer.
Update:
Please note also that the question refers to Joda Time implementation, which, according to Joda Time getSecondOfDay on DST switch day, seems to correspond to the nominal number of seconds implementation (as in my answer below) as opposed to the "number of seconds elapsed since midnight" (as in #icza's answer).
package main
import (
"fmt"
"time"
)
func getSecondOfDay(t time.Time) int {
return 60*60*t.Hour() + 60*t.Minute() + t.Second()
}
func main() {
t := time.Now()
fmt.Println(getSecondOfDay(t))
}
I have a loop where I start by a time.Time and I what to add a minute.
for idx := range keys {
var a = idx * time.Minute
var t = tInit.Add(time.Minute * a)
fmt.Println(t, idx)
}
Here is my error
invalid operation: idx * time.Minute (mismatched types int and time.Duration)
The operands to numeric operations must have the same type. Convert the int value idx to a time.Duration: var a = time.Duration(idx) * time.Minute
As a developer in other programing languages I found this the most counterintuitive and illogical way of doing it. I worked in Scala in the last 10 years, and it could be as simple as this:
val a = idx minutes
compared that, the Go way:
var a = time.Duration(idx) * time.Minute
is more verbose, but that wouldn't be the end of the world.
The problem is that multiplying a Duration with another Duration doesn't make any sense if what you want is to obtain another Duration as a result, because from a physical point of view that would be measured in something like seconds squared.
According to the documentation time.Minute is a constant:
const (
Nanosecond Duration = 1
Microsecond = 1000 * Nanosecond
Millisecond = 1000 * Microsecond
Second = 1000 * Millisecond
Minute = 60 * Second
Hour = 60 * Minute
)
And all those are defined in terms of the Duration type which is an alias for int64:
type Duration int64
From what I see is perfectly fine to multiply an integer literal with each one of these constants, after all that's how each one is defined in relation to the others.
So, to recap why is 60 * time.Second valid syntax (and makes perfect sense), but:
var secondsInAMinute := 60
var oneMinute = secondsInAMinute * time.Second
is invalid. This doesn't make any sense.
All those constants are of type Duration. That means they are measured in units of time (multiples of one nanosecond to be precise).
So, it seems the "correct" way to do it (correct in the sense that it compiles and works) doesn't make any physical sense. Let's look at this again:
var a = time.Duration(idx) * time.Minute
So, we are multiplying time.Duration(idx) with time.Minute.
The type for time.Minute is Duration which should be measured with a time unit. In physics it the accepted unit for time is the second. It seems Go uses integer nanoseconds instead, so time.Minute represents a Duration, represented internally in nanoseconds. That's fine.
The problem is that time.Duration(idx) also "converts" the integer idx to a Duration, so in physics it would also be represented as a unit of time, like seconds. So, accordingly, time.Duration(idx), in my opinion, represents idx nanoseconds in Go.
So, basically, when we write time.Duration(idx) * time.Minute we are muliplying idx nanoseconds (idx * 0.0000000001 seconds) with one minute (60 seconds).
So, from a physical point of view time.Duration(idx) * time.Minute would represent idx * 0.000000001 seconds * 60 seconds. Or, simplified, idx * 0.00000006 seconds squared.
Now, in what world is idx * 0.00000006 seconds squared equal to idx * 1 minute?
So, now I know, in Go, if you want to apply a multiplier to a duration, you have to multiply that Duration to another Duration, and divide that in your mind with one millisecond so that all this mess can still makes any kind of physical sense.
I understand that all these unit inconsistencies are the result of the "The operands to numeric operations must have the same type." constraint. But that doesn't make it more logical or less annoying. In my opinion that restriction of the Go language should be removed.
But, for anyone that was lost in my explanations, let's see how illogical all this is with a concrete code example:
package main
import (
"fmt"
"time"
)
func main() {
var oneMinute = 1 * time.Minute
var oneNanosecond = 1 * time.Nanosecond
var oneMinuteTimesOneNanoSecond = oneMinute * oneNanosecond
fmt.Println(oneMinuteTimesOneNanoSecond)
}
The result is exactly what I expected from this nonsensical way of doing time calculations:
1m0s
I'll learn to live with this, but I will never like it.
I use the following code in my test:
package main
import "fmt"
import "math/big"
func main() {
input := "3333333333333333333.......tested with 100'000x3 , tested with 1'000'0000x3, tested with 10'000'000x3"
bi := big.NewInt(0)
if _, ok := bi.SetString(input, 10); ok {
fmt.Printf("number = %v\n", bi)
testval := new(big.Int)
testval.SetString("3", 10)
resultat, isGanzzahl := myDiv(bi, testval)
fmt.Printf("isGanzzahl = %v, resultat = %v\n", isGanzzahl, resultat)
} else {
fmt.Printf("error parsing line %#v\n", input)
}
}
func myDiv(minuend *big.Int, subtrahend *big.Int) (*big.Int, bool) {
zerotest := big.NewInt(0)
modulus := new(big.Int)
modulus = modulus.Mod(minuend, subtrahend)
if zerotest.Cmp(modulus) == 0 {
res := big.NewInt(0)
res.Quo(minuend, subtrahend)
return res, true
} else {
return big.NewInt(0), false
}
}
100'000 x 3 / 3 == not even a quater second
1'000'000 x 3 / 3 == 9.45 seconds
10'000'000 x 3 / 3 == 16.1 minute
Im looking for a way to make this happens much much faster. If i would like to do this multithreaded in go how do i do this with go-routines? Is there a faster way to do a division with larger numbers?
As this is just for testing i planned to use Numbers in the range of 100'000'000 - 1'000'000'000 digits (which would then be 1GB of ram usage). But 1 billion digits would not work because it would take years to complete.
What would then happen if it is N / M ? Where N=1billion digit, M=10million digit. Is this even possible on a powerful home computer?
How would it look / or what do i have to change to being able to distribute this work to multiple small computer (for example AWS)?
If your number is more than 100000 digits long, you need to use Fast Fourier Transform for multiplication and division: https://en.wikipedia.org/wiki/Multiplication_algorithm#Fourier_transform_methods . The basic idea is to treat numbers as polynomials with x being power of 10 (or power of 2 if you want binary system). Multiply polynomials using Fast Fourier Transform and then propagate carry to get a number from a polynomial. I.e. if we need to multiply 19 by 19 and we use x = 101, we will have (1 * x + 9) * (1 * x + 9) = x2 + 18 * x + 81. Now we propagate carry to convert polynomial back to number: x2 + 18 * x + 81 = x2 + (18 + 8) * x + 1 = x2 + 26 * x + 1 = (1 + 2) * x2 + 6 * x + 1 = 3 * x2 + 6 * x + 1 = 361. The trick is that polynomials can be multiplied efficiently (O(N*log(N) time) using Fast Fourier Transform. The coefficients of the product polynomial are larger than digits, so you need to choose x carefully in order to avoid integer overflow or precision problems.
There unlikely to be a golang library for that so you will need to write it yourself. Here are a few short FFT implementations you can use as a starting point:
http://codeforces.com/contest/632/submission/16449753 http://codeforces.com/contest/632/submission/16445979 http://codeforces.com/contest/632/submission/16449040
http://codeforces.com/contest/632/submission/16448169
If you decide to use FFT modulo prime, see this post for a good choice of the modulo: http://petr-mitrichev.blogspot.com/2015/04/this-week-in-competitive-programming.html