Go copy bytes into struct fields with reflection - go

How can I iterate over a byte slice and assign them to the fields of a struct?
type s struct {
f1 []byte
f2 []byte
f3 []byte
}
func S s {
x := s{}
x.f1 = make([]byte, 4)
x.f1 = make([]byte, 2)
x.f1 = make([]byte, 2)
return x
}
func main() {
data := []byte{83, 117, 110, 83, 0, 1, 0, 65}
Z := S()
//pesudo code from here
i:= 0
for field in Z {
field = data[i:len(field)]
i += len(field)
}
Expecting:
f1 = [83,117,110,83]
f2 = [0,1]
f3 = [0,65]
I've done this in C/C++ before but I can't figure out how to do it in Go. I need the assigning function to be generic as I'm going to have several different structs some of which may not exist in the stream.
Ideally I want to pass in the initialized struct and my code would iterate over the struct fields filling them in.

Leverage the reflection code in the binary/encoding package.
Step 1: Declare the fields as arrays instead of slices.
type S struct {
F1 [4]byte
F2 [2]byte
F3 [2]byte
}
Step 2: Decode the data to the struct using binary.Read
var s S
data := []byte{83, 117, 110, 83, 0, 1, 0, 65}
err := binary.Read(bytes.NewReader(data), binary.LittleEndian, &s)
if err != nil {
log.Fatal(err)
}
Step 3: Done!
fmt.Print(s) // prints {[83 117 110 83] [0 1] [0 65]}
https://go.dev/play/p/H-e8Lusw0RC

You can use reflect.Copy. Like the built-in copy, it copies data into the destination up to its length. Make sure the fields you need to set are exported.
func main() {
data := []byte{83, 117, 110, 83, 0, 1, 0, 65}
z := S{
F1: make([]byte, 4),
F2: make([]byte, 2),
F3: make([]byte, 2),
}
SetBytes(&z, data)
fmt.Println(z) // {[83 117 110 83] [0 1] [0 65]}
}
func SetBytes(dst any, data []byte) {
v := reflect.ValueOf(dst)
if v.Kind() != reflect.Ptr {
panic("dst must be addressable")
}
v = v.Elem()
j := 0
for i := 0; i < v.NumField(); i++ {
field := v.Field(i)
if field.Kind() != reflect.Slice {
continue
}
j += reflect.Copy(v.Field(i), reflect.ValueOf(data[j:]))
}
}
Since data is assumed to be always []byte, you can subslice it directly.
Alternatively, you can use reflect.Value#Slice:
d := reflect.ValueOf(data)
// and later
j += reflect.Copy(v.Field(i), d.Slice(j, d.Len()))
Playground: https://go.dev/play/p/o1MR1qrW5pL

Related

How to reuse slice with sync.Pool in golang?

Consider this code:
type TestStruct struct {
Name string
}
func TestSliceWithPool(t *testing.T) {
var slicePool = sync.Pool{
New: func() interface{} {
t.Log("i am created")
s := make([]interface{}, 0)
return s
},
}
s, _ := slicePool.Get().([]interface{})
t.Logf("Lenth: %d, Cap: %d, Pointer: %p", len(s), cap(s), s)
for i := 0; i < 9000; i++ {
st := &TestStruct{Name: "test"}
s = append(s, st)
}
for _, v := range s {
if value, ok := v.(TestStruct); ok {
if value.Name != "test" {
t.Error("u are changed!")
}
}
}
s = s[:0]
slicePool.Put(s)
s2, _ := slicePool.Get().([]interface{})
t.Logf("Lenth: %d, Cap: %d, Pointer: %p", len(s), cap(s), s)
for i := 0; i < 8000; i++ {
st := &TestStruct{Name: "test2"}
s2 = append(s2, st)
}
for _, v := range s2 {
if value, ok := v.(TestStruct); ok {
if value.Name != "test2" {
t.Error("u are changed!")
}
}
}
slicePool.Put(s2)
}
The result of test is:
slice_test.go:63: i am created
slice_test.go:70: Lenth: 0, Cap: 0, Pointer: 0x1019598
slice_test.go:86: Lenth: 0, Cap: 9728, Pointer: 0xc000500000
Why is it generated only once but the address is different? And Why the cap is 9728?
Is there any problem when I use the same slice like this?
Why is it generated only once but the address is different?
Because in the first for loop you append to it beyond its capacity, which in New is set to zero, and reassign to it the result of append. For details: Why does append() modify the provided slice? (See example)
Is there any problem when I use the same slice like this?
There could be. When you reslice s with s = s[:0], you are resetting the length but not the capacity. The backing array is still the same one from the previous append-and-reassign operation.
So if you append again to s2, the capacity will be enough to not cause reallocation, and you'll end up overwriting the first elements of the backing array:
A demonstrative example:
func TestSliceWithPool(t *testing.T) {
var slicePool = sync.Pool{
New: func() interface{} {
t.Log("Created")
s := make([]interface{}, 0)
return s
},
}
s, _ := slicePool.Get().([]interface{})
for i := 0; i < 10; i++ {
s = append(s, i)
}
fmt.Println(s)
// ^ output: [0 1 2 3 4 5 6 7 8 9]
s = s[:0]
slicePool.Put(s)
s2, _ := slicePool.Get().([]interface{})
fmt.Println(s)
// ^ output: []
for i := 0; i < 5; i++ {
s2 = append(s2, i*10)
}
fmt.Println(s2)
// ^ output: [0 10 20 30 40]
fmt.Println(s2[:10])
// ^ output: [0 10 20 30 40 5 6 7 8 9]
}
This might be okay, since you now have a slice with extended capacity that doesn't need reallocation on append, but it could also be a memory leak if your application keeps around other slice headers pointing to the same backing array (as in case of s and s2), thus preventing garbage collection of the buffers.

how should I change parsing method or correct random method

I would like to generate a random 17 digits number string and parse it to uint64 by golang, here is my code:
const msgMaxValue = 100000000000000000
s := fmt.Sprintf("%17v", rand.New(rand.NewSource(time.Now().UnixNano())).Int63n(msgMaxValue))
log.Info("s:", s)
msgId, err := strconv.Atoi(s)
the error is: invalid syntax, I found there is a space in random value but don't know why, how should I fix it, thank u
have fixed it by "crypto/rand", below is code:
func CreateRandomNumber(len int) (string, error) {
var numbers = []byte{1, 2, 3, 4, 5, 7, 8, 9}
var container string
length := bytes.NewReader(numbers).Len()
for i := 0; i < len; i++ {
random, err := rand.Int(rand.Reader, big.NewInt(int64(length)))
if err != nil {
log.Error("random error:", err)
return "", err
}
container += fmt.Sprintf("%d", numbers[random.Int64()])
}
return container, nil
}
then strconv.Atoi to get a int type number is okay

How to get this specific shape for my data

In Go, I'm using this function bars, err := custplotter.NewCandlesticks(data)
from here:
https://github.com/pplcc/plotext/tree/master/custplotter
It's expecting this shape for data:
[{2 16435 16458 16435 16446 1} {3 16446 16458 16435.04 16455 1} .....]
But my code below is creating my data in this shape instead:
[[2 16435 16458 16435 16446 1] [3 16446 16458 16435.04 16455 1] .....]
Which gives me this error message:
cannot use data (type [ ][ ]string) as type custplotter.TOHLCVer in argument to custplotter.NewCandlesticks:
[ ][ ]string does not implement custplotter.TOHLCVer (missing Len method)
I believe the problem is the data shape. How can i change my code to create the required data shape (with { } instead of [ ]) ?
//read excel file******************************************
xlsx, err := excelize.OpenFile("/media/Snaps/test snaps.xlsm")
if err != nil {
fmt.Println(err)
return
}
//read all rows into df
df := xlsx.GetRows("ticker_2")
//get only TOHLCV columns and 60 rows
df3 := make([][]string, 60) // create slice for 60 rows
idx := 0
for _, row := range df[1:61] { // read 60 rows
df3row := make([]string, 6) // create slice for 6 columns
copy(df3row, row[28:34]) // copy desired columns to new row slice
df3[idx] = df3row
idx++
}
All examples of slices i found in Go litterature uses only [ [ ], [ ] ]
as per https://github.com/pplcc/plotext/blob/68ab3c6e05c34baf5af21c9f5c3341f527a110ac/examples/tohlcvexampledata.go#L42
It seems that what you need is a custplotter.TOHLCVs which is just a slice of a struct of float64.
https://github.com/pplcc/plotext/blob/master/custplotter/tohlcv.go:
type TOHLCVer interface {
// Len returns the number of time, open, high, low, close, volume tuples.
Len() int
// TOHLCV returns an time, open, high, low, close, volume tuple.
TOHLCV(int) (float64, float64, float64, float64, float64, float64)
}
// TOHLCVs implements the TOHLCVer interface using a slice.
type TOHLCVs []struct{ T, O, H, L, C, V float64 }
so basically your solution could resemble this:
df3 := make(TOHLCVs, 60) // create slice for 60 rows
idx := 0
for _, row := range df[1:61] { // read 60 rows
df3[idx].T, err = strconv.ParseFloat(row[28], 64)
df3[idx].O, err = strconv.ParseFloat(row[29], 64)
df3[idx].H, err = strconv.ParseFloat(row[30], 64)
df3[idx].L, err = strconv.ParseFloat(row[31], 64)
df3[idx].C, err = strconv.ParseFloat(row[32], 64)
df3[idx].V, err = strconv.ParseFloat(row[33], 64)
idx++
}
Or you could just implement the TOHLCVer interface too :)
type SlicesOfTOHLCV [][6]float64
func (s SlicesOfTOHLCV) Len() int {
return len(s)
}
func (s SlicesOfTOHLCV) TOHLCV(i int) (float64, float64, float64, float64, float64) {
return s[i][0], s[i][1], s[i][2], s[i][3], s[i][4], s[i][5]
}
mySlice := make(SlicesOfTOHLCV, 60)
i := 0
for _, row := range df[1:61] {
mySlice[i] = [6]float64{}
for j := 0; j < 6; j ++ {
mySlice[i][j], err = strconv.ParseFloat(row[28+j], 64)
if err != nil {
panic(err)
}
}
i ++
}

How to check response returned from tabwriter.Writer in golang

I am writting something to tabwriter.Writer object,
w := tabwriter.NewWriter(os.Stdout, 5, 1, 3, ' ', 0)
fmt.Fprintf(w, "%v\t%v\t\n", somevalue1, somevalue2)
I can print the data in w in console using w.Flush()
Is there any way so get values in w as string in one place and compare it with some value?
I want to compare what I have in w with some data.
You can implement your own io.Writer:
type W []byte
func (w *W) Write(b []byte) (int, error) {
*w = append(*w, b...)
return len(b), nil
}
You can then pass an instance of *W to tabwriter.NewWriter:
sw := &W{}
w := tabwriter.NewWriter(sw, 5, 1, 3, ' ', 0)
fmt.Fprintf(w, "%v\t%v\t\n", somevalue1, somevalue2)
// get the string value from sw
str := string(*sw)
As sugested by #Tim, you should use *bytes.Buffer instead for better performance and it already implements io.Writer:
var b bytes.Buffer
w := tabwriter.NewWriter(&b, 0, 0, 1, '.', 0)
// ...
fmt.Println(b.String())

Golang: How to printf % x for bytes in a struct?

var b [88]byte
n, err := file.Read(b[:])
fmt.Printf("bytes read: %d Bytes: [% x]\n", n, b)
The above prints bytes in hexdecimal
I have a struct like this
type SomeStruct struct {
field1 []byte
field2 []byte
}
someStructInstance := SomeStruct{[249 190 180 217], [29 1 0 0]}
fmt.Println(someStructInstance)
=> {[249 190 180 217] [29 1 0 0]}
But ideally I would like it to print hexdecimal
=> {[f9 be b4 d9] [1d 01 00 00]}
How would I go about that?
I think you will just have to define your own String function on SomeStruct. Here is an example:
package main
import "fmt"
type SomeStruct struct {
field1 []byte
field2 []byte
}
func (s SomeStruct) String() string {
return fmt.Sprintf("{[% x] [% x]}", s.field1, s.field2)
}
func main() {
someStructInstance := SomeStruct{[]byte{249, 190, 180, 217}, []byte{29, 1, 0, 0}}
fmt.Println(someStructInstance)
}
See it in running on the Go Playground: http://play.golang.org/p/eYBa1n33a2
You could use reflection to inspect the struct and print any []bytes that it has.
package main
import (
"fmt"
"reflect"
)
type SomeStruct struct {
field1 []byte
field2 []byte
}
type OtherStruct struct {
intValue int
intSlice []int
byteSlice []byte
}
var typeOfBytes = reflect.TypeOf([]byte(nil))
func printSlicesHex(obj interface{}) {
value := reflect.ValueOf(obj)
typeOfObj := value.Type()
for i := 0; i < value.NumField(); i++ {
field := value.Field(i)
if field.Type() == typeOfBytes {
bytes := field.Bytes()
printBytes(typeOfObj.Field(i).Name, bytes)
}
}
}
func printBytes(name string, bytes []byte) {
fmt.Printf("%s: [% x]\n", name, bytes)
}
func main() {
someStructInstance := SomeStruct{[]byte{249, 190, 180, 217}, []byte{29, 1, 0, 0}}
fmt.Println("Printing []bytes in SomeStruct")
printSlicesHex(someStructInstance)
fmt.Println()
otherStruct := OtherStruct{0, []int{0, 1, 2}, []byte{0, 1, 2, 3}}
fmt.Println("Printing []bytes in OtherStruct")
printSlicesHex(otherStruct)
}
For each []byte, this example prints the name of the field and its data (in hex). You could improve on this by taking a custom function to do the printing, so you don't always have to print in hex.
Playground Link

Resources