Golang Equivalent of Python's `pd.to_datetime()` - go

I'm new to Go, and looking to create my own algo trading strategy backtesting library, an area I'm well experienced in with Python, to help learn the language.
I have a 5 minute OHLCV SPY5min.csv dataset, the head of which looks like this:
I use this code to read in the dataset from the file, converting everything to a list of lists of values:
package main
import (
"encoding/csv"
"log"
"os"
"fmt"
)
func ReadCsvFile(filePath string) [][]string {
f, err := os.Open(filePath)
if err != nil {
log.Fatal("Unable to read input file "+filePath, err)
}
defer f.Close()
csvReader := csv.NewReader(f)
records, err := csvReader.ReadAll()
if err != nil {
log.Fatal("Unable to parse file as CSV for "+filePath, err)
}
return records
}
func main() {
records := ReadCsvFile("./SPY5min.csv")
fmt.Println(records)
}
This returns a list of lists of string values. Cool. Now what I want to do is replicate a Pandas Dataframe like object, or perhaps separate each "column" into their own separate arrays/slices if that's easier, not sure yet.
Once that's done, I need a way to convert the strings of datetimes to actual datetime objects that I can run comparisons and loc's on. Can someone point me in the right direction?
My naive approach (pseudo) would be to:
Declare 6 array variables (datetime, open, high, low, close, volume) of len(records) in size
Iterate over the records list of lists
Insert each value into the current i of their respective arrays
Once iteration is done, mass convert the values in the datetime array to values of datetime objects?
Wondering if this is really the best way of doing this, or if there's a faster way than O(n) iteration?

You asked, "Once that's done, I need a way to convert the strings of datetimes to actual datetime objects ...". I recently answered a similar question here: https://stackoverflow.com/a/74491722/5739452
Your timestamp looks like this: "2022-11-08 4:00". The time package contains parsing and other manipulation functions. The key detail is knowing the conventions for the layout parser format. Each element of a time is recognized as a specific number. The year is 2006, the month is 01 etc.
So, for your purpose something like this should work:
package main
import (
"fmt"
"time"
)
func main() {
t := "2022-11-08 4:00"
const layout = "2006-01-02 15:04"
x, err := time.Parse(layout, t)
fmt.Println(x, err)
}

Related

unserialize php in goland

I have a file with serialized array in PHP.
The content of the file locks like this
a:2:{i:250;s:7:"my_catz";s:7:"abcd.jp";a:2:{s:11:"category_id";i:250;s:13:"category_name";s:7:"my_catz";}}
The array unserialized is this
(
[250] => my_catz
[abcd.jp] => Array
(
[category_id] => 250
[category_name] => my_catz
)
)
Now, i want to get the content of the file in GO, unserialize it convert it to an array.
In GO i can get the content of the file using
dat, err := os.ReadFile("/etc/squid3/compiled-categories.db")
if err != nil {
if e.Debug {
log.Printf("error reading /etc/squid3/compiled-categories.db: ", err)
}
}
And unserialized it using github.com/techoner/gophp library
package categorization
import (
"fmt"
"os"
"github.com/techoner/gophp"
"log"
"errors"
)
type Data struct {
Website string
Debug bool
}
func (e Data) CheckPersonalCategories() (int,string) {
if e.Debug {
log.Printf("Checking Personal Categories")
}
if _, err := os.Stat("/etc/squid3/compiled-categories.db"); errors.Is(err, os.ErrNotExist) {
if e.Debug {
log.Printf("/etc/squid3/compiled-categories.db not exit: ", err)
}
return 0,""
}
dat, err := os.ReadFile("/etc/squid3/compiled-categories.db")
if err != nil {
if e.Debug {
log.Printf("error reading /etc/squid3/compiled-categories.db: ", err)
}
}
out, _ := gophp.Unserialize(dat)
fmt.Println(out["abcd.jp"])
return 0,""
}
But I can't access to the array, for example, when I try access to array key using out["abcd.jp"] i get this error message
invalid operation: out["abcd.jp"] (type interface {} does not support indexing)
The result of out is
map[250:my_catz abcd.jp:map[category_id:250 category_name:my_catz]]
Seams that is unserializing
Don't make assumptions about what is and isn't succeeding in your code. Error responses are the only reliable way to know whether a function succeeded. In this case the assumption may hold, but ignoring errors is always a mistake. Invest time in catching errors and at least panic them - don't instead waste your time ignoring errors and then trying to debug unreliable code.
invalid operation: out["abcd.jp"] (type interface {} does not support indexing)
The package you're using unfortunately doesn't provide any documentation so you have to read the source to understand that gophp.Unserialize returns (interface{}, error). This makes sense; php can serialize any value, so Unserialize must be able to return any value.
out is therefore an interface{} whose underlying value depends on the data. To turn an interface{} into a particular value requires a type assertion. In this case, we think the underlying data should be map[string]interface{}. So we need to do a type assertion:
mout, ok := out.(map[string]interface{})
Before we get to the working code, one more point I'd like you to think about. Look at the code below: I started it from your code, but the resemblance is very slight. I took out almost all the code because it was completely irrelevant to your question. I added the input data to the code to make a minimal reproduction of your code (as I asked you to do and you declined to do). This is a very good use of your time for 2 reasons: first, it makes it a lot easier to get answers (both because it shows sufficient effort on your part and because it simplifies the description of the problem), and second, because it's excellent practice for debugging. I make minimal reproductions of code flows all the time to better understand how to do things.
You'll notice you can run this code now without any additional effort. That's the right way to provide a minimal reproducible example - not with a chunk of mostly irrelevant code which still can't be executed by anybody.
The Go Plaground is a great way to demonstrate go-specific code that others can execute and investigate. You can also see the code below at https://go.dev/play/p/QfCl08Gx53e
package main
import (
"fmt"
"github.com/techoner/gophp"
)
type Data struct {
Website string
Debug bool
}
func main() {
var dat = []byte(`a:2:{i:250;s:7:"my_catz";s:7:"abcd.jp";a:2:{s:11:"category_id";i:250;s:13:"category_name";s:7:"my_catz";}}`)
out, err := gophp.Unserialize(dat)
if err != nil {
panic(err)
}
if mout, ok := out.(map[string]interface{}); ok {
fmt.Println(mout["abcd.jp"])
}
}

How to parse timestamp with underscores in Golang

I'm trying to parse access log timestamp like "2020/11/06_18:17:25_455" in Filebeat according to Golang spec.
Here is my test program to verify layout:
package main
import (
"fmt"
"log"
"time"
)
func main() {
eventDateLayout := "2006/01/02_15:04:05_000"
eventCheckDate, err := time.Parse(eventDateLayout, "2020/11/06_18:17:25_455")
if err != nil {
log.Fatal(err)
}
fmt.Println(eventCheckDate)
}
Result:
2009/11/10 23:00:00 parsing time "2020/11/06_18:17:25_455" as
"2006/01/02_15:04:05_000": cannot parse "455" as "_000"
As I understand underscore has a special meaning in Golang, but from documentation it's not clear how to escape it.
Any ideas, please?
It doesn't seem possible to use any escape characters for the time layout (e.g. "\\_" doesn't work), so one would have to do something different.
This issue describes the same problem, but it was solved in a very non-general way that doesn't seem to apply to your format.
So your best bet seems to be replacing _ with something else/stripping it from the string, then using a layout without it. To make sure that the millisecond part ist also parsed, it must be separated with a . instead of _, then it's recognized as part of the seconds (05) format.
eventDateLayout := "2006/01/02.15:04:05"
val := strings.Replace("2020/11/06_18:17:25_455", "_", ".", 2)
eventCheckDate, err := time.Parse(eventDateLayout, val)
if err != nil {
panic(err)
}
fmt.Println(eventCheckDate)
Playground link
From time.Format
A fractional second is represented by adding a period and zeros to the
end of the seconds section of layout string, as in "15:04:05.000" to
format a time stamp with millisecond precision.
You cannot specify millisecond precision with an underscore you need 05.000 instead:
// eventDateLayout := "2006/01/02_15:04:05_000" // invalid format
eventDateLayout := "2006/01/02_15:04:05.000"
eventCheckDate, err := time.Parse(eventDateLayout, "2020/11/06_18:17:25.455")
So basically use a simple translate function to convert the final _ to a . and use the above parser.
https://play.golang.org/p/POPgXC_qe81

Is json marshalling of map stable in Go? [duplicate]

This question already has answers here:
How to produce JSON with sorted keys in Go?
(2 answers)
Closed 3 years ago.
I am writing code that will check if data changed based on a comparison of json.Marshaled hashes of maps. I've created small code to produce what I am doing in abstracted way (available also in playground)
package main
import (
"crypto/sha256"
"encoding/hex"
"encoding/json"
"fmt"
)
func main() {
fmt.Println("Hello, playground")
a := make(map[string]string)
a["a"] = "a1"
a["b"] = "b2"
sa, _ := json.Marshal(a)
ha := GenerateSHA256Hash(string(sa))
b := make(map[string]string)
b["a"] = "a1"
b["b"] = "b2"
sb, _ := json.Marshal(b)
hb := GenerateSHA256Hash(string(sb))
fmt.Println(ha)
fmt.Println(hb)
fmt.Println(ha == hb)
}
func GenerateSHA256Hash(s string) string {
hasher := sha256.New()
hasher.Write([]byte(s))
return hex.EncodeToString(hasher.Sum(nil))
}
But I recall that order of maps are unordered and in Golang spec it's written that
The iteration order over maps is not specified and is not guaranteed to be the same from one iteration to the next. If a map entry that has not yet been reached is removed during iteration, the corresponding iteration value will not be produced. If a map entry is created during iteration, that entry may be produced during the iteration or may be skipped. The choice may vary for each entry created and from one iteration to the next. If the map is nil, the number of iterations is 0.
So, in the code above I am building map, in the same way, each time and not accessing it concurrently during json.Marshalling.
Question: Will the hashes, produced in such manner, be always equal? Or will this approach be stable?
Go spec in this case is irrelevant since it's a details of the Go standard library (the encoding/json module)
As of this very moment it's implemented as
// Extract and sort the keys.
keys := v.MapKeys()
sv := make([]reflectWithString, len(keys))
for i, v := range keys {
sv[i].v = v
if err := sv[i].resolve(); err != nil {
e.error(fmt.Errorf("json: encoding error for type %q: %q", v.Type().String(), err.Error()))
}
}
sort.Slice(sv, func(i, j int) bool { return sv[i].s < sv[j].s })
Additionally, given the encoding/json documentation says
The map keys are sorted and used as JSON object keys by applying the following rules, subject to the UTF-8 coercion described for string values above:
it's safe to expect the same hash until at least Go 2.

Does go provide variable sanitization?

I am a beginner in Golang.
I have a problem with variable type assigning from user input.
When the user enters data like "2012BV352" I need to be able to ignore the BV and pass 2012352 to my next function.
There has a package name gopkg.in/validator.v2 in doc
But what it returns is whether or not the variable is safe or not.
I need to cut off the unusual things.
Any idea on how to achieve this?
You could write your own sanitizing methods and if it becomes something you'll be using more often, I'd package it out and add other methods to cover more use cases.
I provide two different ways to achieve the same result. One is commented out.
I haven't run any benchmarks so i couldn't tell you for certain which is more performant, but you could write your own tests if you wanted to figure it out. It would also expose another important aspect of Go and in my opinion one of it's more powerful tools... testing.
package main
import (
"fmt"
"log"
"regexp"
"strconv"
"strings"
)
// using a regex here which simply targets all digits and ignores everything else. I make it a global var and use MustCompile because the
// regex doesn't need to be created every time.
var extractInts = regexp.MustCompile(`\d+`)
func SanitizeStringToInt(input string) (int, error) {
m := extractInts.FindAllString(input, -1)
s := strings.Join(m, "")
return strconv.Atoi(s)
}
/*
// if you didn't want to use regex you could use a for loop
func SanitizeStringToInt(input string) (int, error) {
var s string
for _, r := range input {
if !unicode.IsLetter(r) {
s += string(r)
}
}
return strconv.Atoi(s)
}
*/
func main() {
a := "2012BV352"
n, err := SanitizeStringToInt(a)
if err != nil {
log.Fatal(err)
}
fmt.Println(n)
}

Is there a built in function in go for making copies of arbitrary maps? [duplicate]

This question already has answers here:
Copying all elements of a map into another
(4 answers)
Closed 4 months ago.
Is there a built in function in go for making copies of arbitrary maps?
I would be able to write one by hand but I found out earlier I was looking a similar question when I wanted to make a deep comparison of maps and there seemed to be a function already built in for that! So similarly, maybe I was wondering if there was an built in or some library or package for making deep copies of maps in golang. I am sure I am not the first person to want to make copies of maps in go.
By copy I mean you can create two different variables that reference a different map in memory even though they are the same content wise.
For a more general answer, you can encode your map and decode it in a new variable with encoding/gob.
The advantages of this way is that it'll even work on more complex data structure, like a slice of struct containing a slice of maps.
package main
import (
"bytes"
"encoding/gob"
"fmt"
"log"
)
func main() {
ori := map[string]int{
"key": 3,
"clef": 5,
}
var mod bytes.Buffer
enc := gob.NewEncoder(&mod)
dec := gob.NewDecoder(&mod)
fmt.Println("ori:", ori) // key:3 clef:5
err := enc.Encode(ori)
if err != nil {
log.Fatal("encode error:", err)
}
var cpy map[string]int
err = dec.Decode(&cpy)
if err != nil {
log.Fatal("decode error:", err)
}
fmt.Println("cpy:", cpy) // key:3 clef:5
cpy["key"] = 2
fmt.Println("cpy:", cpy) // key:2 clef:5
fmt.Println("ori:", ori) // key:3 clef:5
}
If you want to know more about gobs, there is a go blog post about it.
No, there is no built-in one-liner for map deep copy.
However, there is an iterative solution as well as a generic package for deep copy.
Copying a map is a few, trivial lines of code. Just write the code and enjoy letting go of the idea that every simple piece of code has to be in a library!
original := map[string]int{
"Hello": 4,
"World": 123,
}
copy := map[string]int{}
for k, v := range original {
copy[k] = v
}

Resources