This question already has answers here:
generic function to get size of any structure in Go
(2 answers)
Closed 6 months ago.
In the case of a generic function that does byte serialization for a generic types, is there a way to proceed--other than reflection--if the different supported types have different sizes? For example:
package main
import (
"fmt"
)
type KeyType interface {
uint16 | uint32 | uint64
}
type Item[KT KeyType] struct {
Key KT
Data []byte
}
// set of generic types that hold collections of Item[T]
// sets of methods that operate on those generic types
func MarshalBinary[KT KeyType](i *Item[KT]) ([]byte, error) {
// How do I compute the size of the item and marshal it?
// It's 2 bytes for uint16, 4 for uint32, 8 for uint64,
// how do I distinguish here?
}
func main() {
i := new(Item[uint32])
i.Key = 42
fmt.Println(i)
}
Is there a way to access the size of the type within the serialization function without reflection?
I know I can proceed with reflection like this:
package main
import (
"fmt"
"reflect"
"strings"
)
type KeyType interface {
uint16 | uint32 | uint64
}
type Item[KT KeyType] struct {
Key KT
Data []byte
}
// set of generic types that hold collections of Item[T]
// sets of methods that operate on those generic types
func MarshalBinary[KT KeyType](i *Item[KT]) ([]byte, error) {
t := reflect.TypeOf(i)
var size int
if strings.Contains(t.String(), `uint32`) {
size = 4
}
fmt.Println(size)
// rest of function here
return nil, nil
}
func main() {
i := new(Item[uint32])
i.Key = 42
MarshalBinary(i)
fmt.Println(i)
}
Is there a better way? My main concern with using reflection here is the potential performance cost.
First off, I think your sample code may be incorrect, because you're using reflect.TypeOf(i), but i is of type Item[KT], and since Item includes both a KT and a []byte, it will be the size of KY plus the size of a pointer (the pointer to the byte slice). So, it will be 4 + pointer size if KT is a uint32, but you're setting it to 4.
So the question is, are you trying to get the size of i (Item[KT]), or the size of an instance of KT?
I assume it's the size of KT that you're actually looking for, since you're assigning a size of 4 if it's a uint32. You can do this without reflect by casting the value you want the size of as an interface{}, then using a standard type switch, as follows:
func MarshalBinary[KT KeyType](i *Item[KT]) ([]byte, error) {
var size int
switch (interface{})(i.Key).(type) {
case uint16:
size = 2
case uint32:
size = 4
case uint64:
size = 8
default:
panic("Unexpected type")
}
...
}
This is a bit problematic if you ever want to expand the different possible types that KeyType could be, though.
I'm currently learning Go and am following a tutorial about how to use Go with Stripe. There is this example code:
package main
import (
"fmt"
"github.com/stripe/stripe-go"
"github.com/stripe-go/customer"
)
func main() {
sc := &client.API{}
sc.Init("somekey")
c, _ := sc.Customers.Get("customerid", nil)
// ...
}
What is/could be the reason that sc stores the pointer to the struct and not the struct itself?
[To supplement the comment you received]
While in this case with the small code sample it's hard to say, in most scenarios you'll see non-trivial types passed around by pointer to enable modification. As an anti-example, consider this code which uses a variable of a struct type by value:
type S struct {
ID int
}
func (s S) UpdateID(i int) {
s.ID = i
}
func main() {
s := S{}
s.UpdateID(99)
fmt.Println(s.ID)
}
What do you think this will print? It will print 0, because methods with value receivers cannot modify the underlying type.
There's much information about this in Go - read about pointers, and about how methods should be written. This is a good reference: https://golang.org/doc/faq#methods_on_values_or_pointers, and also https://golang.org/doc/effective_go#pointers_vs_values
Back to your example: typically non-trivial types such as those representing a "client" for some services will be using pointers because method calls on such types should be able to modify the types themselves.
I'm new to golang and trying to figure out the correct way of casting a block of bytes to the correct struct. All structs start with two bytes that dictate the layout of the remaining bytes. In C I would point to the beginning of the block of memory and cast it as a simple struct that only contained those two bytes (X below) but here I get an invalid type assertion. I'm probably way off base here any help you be appreciated.
package main
import (
"fmt"
)
type A struct {
tag byte
ver byte
data1 int
data2 int
data3 int
}
type B struct {
tag byte
ver byte
data1 float32
}
type X struct {
tag byte
ver byte
}
func main() {
var a A
a.tag = 1
a.ver = 1
x := a.(X)
fmt.Printf("%d,%d", x.tag, x.ver)
}
Edit
In short I just want to create a custom method on type Foo that is the only reason why I want to perform the cast. If the solutions are very complex I will just create functions instead of methods I guess. I was just curios.
playground link
Here's my solution. It involves a few different tips:
Embed the shared struct in the individual structs.
Use encoding/binary package to load bytes into structs.
Fill header struct with first two bytes, then make a decision on which subtype to make and fill.
Always use fixed length int types for this kind of thing.
Your field names must be UpperCase to be fillable from encoding/binary
This is a pretty brittle way to manage marshalling.unmarshalling of data, but I'm sure you know that.
Here's my solution:
package main
import (
"bytes"
"encoding/binary"
"fmt"
"log"
)
type A struct {
X
Data1 int32
Data2 int32
Data3 int32
}
type B struct {
X
Data1 int32
}
type X struct {
Tag byte
Ver byte
}
func main() {
var err error
data := []byte{1, 1, 0, 0, 0, 42}
hdr := X{}
err = binary.Read(bytes.NewReader(data[:2]), binary.BigEndian, &hdr)
if err != nil {
log.Fatal(err)
}
fmt.Println(hdr.Tag, hdr.Ver)
if hdr.Tag == 1 {
b := B{}
err = binary.Read(bytes.NewReader(data), binary.BigEndian, &b)
if err != nil {
log.Fatal(err)
}
fmt.Println(b.Data1)
}
}
playground link
Go generally tries to discourage C-like memory fiddling as it leads to memory leaks, incorrect behavior, and security vulnerabilities unless extraordinary caution and testing are applied. That doesn't mean it's impossible though; in fact, the aptly-named unsafe.Pointer is exposed for exactly this purpose. Use it with caution.
I am using this construct to simulate a set
type MyType uint8
map[MyType]interface{}
I then add in all my keys and map them to nil.
I've learnt that it is also possible to use
map[MyType]struct{}
Any benefits of using the empty struct versus interface{}.
Memory usage. For example, types struct{}, interface{}, and bool,
package main
import (
"fmt"
"unsafe"
)
func main() {
var s struct{}
fmt.Println(unsafe.Sizeof(s))
var i interface{}
fmt.Println(unsafe.Sizeof(i))
var b bool
fmt.Println(unsafe.Sizeof(b))
}
Output (bytes for 32-bit architecture):
0
8
1
Output (bytes for 64-bit architecture):
0
16
1
References:
Go Data Structures: Interfaces
The empty struct and empty interface, though syntactically similar, are actually opposites. An empty struct holds no data; an empty interface can hold any type of value. If I see a map[MyType]struct{}, I know immediately that no values will be stored, only keys. If I see a map[MyType]interface{}, my first impression will be that it is a heterogenous collection of values. Even if I see code storing nil in it, I won't know for sure that some other piece of code doesn't store something else in it.
In other words, using struct{} makes your code much more readable. It also saves a little memory, as described in the other answer, but that is just a fringe benefit of using the right tool for the job.
I would like to add additional detail about empty struct , as differences are already covered by andybalholm and peterSO .
Below is the example which shows usability of empty struct .
Creates an instance of rectangle struct by using a pointer address
operator is denoted by & symbol.
package main
import "fmt"
type rectangle struct {
length int
breadth int
color string
}
func main() {
var rect1 = &rectangle{10, 20, "Green"} // Can't skip any value
fmt.Println(rect1)
var rect2 = &rectangle{}
rect2.length = 10
rect2.color = "Red"
fmt.Println(rect2) // breadth skipped
var rect3 = &rectangle{}
(*rect3).breadth = 10
(*rect3).color = "Blue"
fmt.Println(rect3) // length skipped
}
Reference : https://www.golangprograms.com/go-language/struct.html
For a thorough read you can refer : https://dave.cheney.net/2014/03/25/the-empty-struct
I get a byte slice ([]byte) from a UDP socket and want to treat it as an integer slice ([]int32) without changing the underlying array, and vice versa. In C(++) I would just cast between pointer types; how would I do this in Go?
As others have said, casting the pointer is considered bad form in Go. Here are examples of the proper Go way and the equivalent of the C array casting.
WARNING: all code untested.
The Right Way
In this example, we are using the encoding/binary package to convert each set of 4 bytes into an int32. This is better because we are specifying the endianness. We are also not using the unsafe package to break the type system.
import "encoding/binary"
const SIZEOF_INT32 = 4 // bytes
data := make([]int32, len(raw)/SIZEOF_INT32)
for i := range data {
// assuming little endian
data[i] = int32(binary.LittleEndian.Uint32(raw[i*SIZEOF_INT32:(i+1)*SIZEOF_INT32]))
}
The Wrong Way (C array casting)
In this example, we are telling Go to ignore the type system. This is not a good idea because it may fail in another implementation of Go. It is assuming things not in the language specification. However, this one does not do a full copy. This code uses unsafe to access the "SliceHeader" which is common in all slices. The header contains a pointer to the data (C array), the length, and the capacity. Instead of just converting the header to the new slice type, we first need to change the length and capacity since there are less elements if we treat the bytes as a new type.
import (
"reflect"
"unsafe"
)
const SIZEOF_INT32 = 4 // bytes
// Get the slice header
header := *(*reflect.SliceHeader)(unsafe.Pointer(&raw))
// The length and capacity of the slice are different.
header.Len /= SIZEOF_INT32
header.Cap /= SIZEOF_INT32
// Convert slice header to an []int32
data := *(*[]int32)(unsafe.Pointer(&header))
You do what you do in C, with one exception - Go does not allow to convert from one pointer type to another. Well, it does, but you must use unsafe.Pointer to tell compiler that you are aware that all rules are broken and you know what you are doing. Here is an example:
package main
import (
"fmt"
"unsafe"
)
func main() {
b := []byte{1, 0, 0, 0, 2, 0, 0, 0}
// step by step
pb := &b[0] // to pointer to the first byte of b
up := unsafe.Pointer(pb) // to *special* unsafe.Pointer, it can be converted to any pointer
pi := (*[2]uint32)(up) // to pointer to the first uint32 of array of 2 uint32s
i := (*pi)[:] // creates slice to our array of 2 uint32s (optional step)
fmt.Printf("b=%v i=%v\n", b, i)
// all in one go
p := (*[2]uint32)(unsafe.Pointer(&b[0]))
fmt.Printf("b=%v p=%v\n", b, p)
}
Obviously, you should be careful about using "unsafe" package, because Go compiler is not holding your hand anymore - for example, you could write pi := (*[3]uint32)(up) here and compiler wouldn't complain, but you would be in trouble.
Also, as other people pointed already, bytes of uint32 might be layout differently on different computers, so you should not assume these are layout as you need them to be.
So safest approach would be to read your array of bytes one by one and make whatever you need out of them.
Alex
The short answer is you can't. Go wont let you cast a slice of one type to a slice of another type. You will have loop through the array and create another array of the type you want while casting each item in the array. This is generally regarded as a good thing since typesafety is an important feature of go.
Since Go 1.17, there is a simpler way to do this using the unsafe package.
import (
"unsafe"
)
const SIZEOF_INT32 = unsafe.Sizeof(int32(0)) // 4 bytes
func main() {
var bs []byte
// Do stuff with `bs`. Maybe do some checks ensuring that len(bs) % SIZEOF_INT32 == 0
data := unsafe.Slice((*int32)(unsafe.Pointer(&bs[0])), len(bs)/SIZEOF_INT32)
// A more verbose alternative requiring `import "reflect"`
// data := unsafe.Slice((*int32)(unsafe.Pointer((*reflect.SliceHeader)(unsafe.Pointer(&bs)).Data)), len(bs)/SIZEOF_INT32)
}
Go 1.17 and beyond
Go 1.17 introduced the unsafe.Slice function, which does exactly this.
Converting a []byte to a []int32:
package main
import (
"fmt"
"unsafe"
)
func main() {
theBytes := []byte{
0x33, 0x44, 0x55, 0x66,
0x11, 0x22, 0x33, 0x44,
0x77, 0x66, 0x55, 0x44,
}
numInts := uintptr(len(theBytes)) * unsafe.Sizeof(theBytes[0]) / unsafe.Sizeof(int32(0))
theInts := unsafe.Slice((*int32)(unsafe.Pointer(&theBytes[0])), numInts)
for _, n := range theInts {
fmt.Printf("%04x\n", n)
}
}
Playground.
I had the size unknown problem and tweaked the previous unsafe method with the following code.
given a byte slice b ...
int32 slice is (*(*[]int)(Pointer(&b)))[:len(b)/4]
The array to slice example may be given a fictional large constant and the slice bounds used in the same way since no array is allocated.
You can do it with the "unsafe" package
package main
import (
"fmt"
"unsafe"
)
func main() {
var b [8]byte = [8]byte{1, 2, 3, 4, 5, 6, 7, 8}
var s *[4]uint16 = (*[4]uint16)(unsafe.Pointer(&b))
var i *[2]uint32 = (*[2]uint32)(unsafe.Pointer(&b))
var l *uint64 = (*uint64)(unsafe.Pointer(&b))
fmt.Println(b)
fmt.Printf("%04x, %04x, %04x, %04x\n", s[0], s[1], s[2], s[3])
fmt.Printf("%08x, %08x\n", i[0], i[1])
fmt.Printf("%016x\n", *l)
}
/*
* example run:
* $ go run /tmp/test.go
* [1 2 3 4 5 6 7 8]
* 0201, 0403, 0605, 0807
* 04030201, 08070605
* 0807060504030201
*/
Perhaps it was not available when the earlier answers were given, but it would seem that the binary.Read method would be a better answer than "the right way" given above.
This method allows you to read binary data from a reader directly into the value or buffer of your desired type. You can do this by creating a reader over your byte array buffer. Or, if you have control of the code that is giving you the byte array, you can replace it to read directly into your buffer without the need for the interim byte array.
See https://golang.org/pkg/encoding/binary/#Read for the documentation and a nice little example.
http://play.golang.org/p/w1m5Cs-ecz
package main
import (
"fmt"
"strings"
)
func main() {
s := []interface{}{"foo", "bar", "baz"}
b := make([]string, len(s))
for i, v := range s {
b[i] = v.(string)
}
fmt.Println(strings.Join(b, ", "))
}
func crackU32s2Bytes(us []uint32) []byte {
var bs []byte
var ptrBs = (*reflect.SliceHeader)(unsafe.Pointer(&bs))
var ptrUs = (*reflect.SliceHeader)(unsafe.Pointer(&us))
ptrBs.Data = ptrUs.Data
ptrBs.Len = ptrUs.Len*4
ptrBs.Cap = ptrBs.Len
return bs
}
func crackBytes2U32s(bs []byte) []uint32 {
var us []uint32
var ptrBs = (*reflect.SliceHeader)(unsafe.Pointer(&bs))
var ptrUs = (*reflect.SliceHeader)(unsafe.Pointer(&us))
ptrUs.Data = ptrBs.Data
ptrUs.Len = ptrBs.Len/4
ptrUs.Cap = ptrUs.Len
return us
}