I'd like to use byte array as uint32 array, then get the first element of the uint32 array. But I can not get the following code to work. Could anybody let me know how to cast a byte array to a unit32 array? Thanks.
// vim: set noexpandtab tabstop=2:
package main
import (
"unsafe"
"fmt"
)
func main() {
x := []byte{'\x01', '\x02', '\x03', '\x04', '\x06', '\x07', '\x08', '\x09'}
fmt.Printf("%#v\n", x)
fmt.Printf("%#v\n", []int32(unsafe.Pointer(x))[0])
}
It would be
fmt.Printf("%#v\n", (*(*[]int32)(unsafe.Pointer(&x)))[0])
but one should remember that it still would have len = cap = 8 (*), and carefully check for boundaries themselves.
The expression clarification
unsafe.Pointer(&x) // an uintptr to an `x` slice
(*[]int32)(...) // a type conversion
// it's the same syntax when you do float64(42)
// but in this case you need extra parentheses around the
// type name to make it unambiguous for the parser
// so, it converts a unitptr to a pointer to a int32 slice
*(...) // here it's just a pointer dereference. So you obtain a slice
// from a pointer to a slice, to index it after
(*) cap is not guaranteed to be 8 in this particular case (but it does not really matter).
Related
I am attempting to generically get the size of any struct from an interface. This works fine with objects that are passed by value, but I can't figure out how to get the size of the object when it's passed the reference using an interface.
Here's an example indicating the problem I'm having:
https://play.golang.org/p/QXXZm-j7_hZ
package main
import (
"fmt"
"reflect"
"unsafe"
)
type T struct {
c [50]byte
}
func main() {
fmt.Println("Hello, playground")
var t T
s := unsafe.Sizeof(t)
fmt.Println("expected size", s)
getSize(t, &t)
}
func getSize(valueObject interface{}, referenceObject interface{}) {
vs := []uintptr{
unsafe.Sizeof(valueObject),
unsafe.Sizeof(reflect.ValueOf(valueObject)),
reflect.TypeOf(valueObject).Size(), // THIS WORKS FOR VALUE
reflect.TypeOf(reflect.ValueOf(valueObject)).Size(),
0, //reflect.TypeOf(reflect.ValueOf(valueObject).Elem()).Size(), //EXCEPTION ACCESSING ELEM WITH VALUE
0, //reflect.TypeOf(reflect.ValueOf(valueObject).Elem()).Size(), //EXCEPTION ACCESSING ELEM WITH VALUE
0, //unsafe.Sizeof(reflect.ValueOf(valueObject).Elem()), //EXCEPTION ACCESSING ELEM WITH VALUE
0, //unsafe.Sizeof(reflect.TypeOf(reflect.ValueOf(valueObject).Elem())), //EXCEPTION ACCESSING ELEM WITH VALUE
}
fmt.Println("valueObject size", vs)
rs := []uintptr{
unsafe.Sizeof(referenceObject),
unsafe.Sizeof(reflect.ValueOf(referenceObject)),
reflect.TypeOf(referenceObject).Size(),
reflect.TypeOf(reflect.ValueOf(referenceObject)).Size(),
reflect.TypeOf(reflect.ValueOf(referenceObject).Elem()).Size(),
reflect.TypeOf(reflect.ValueOf(referenceObject).Elem()).Size(),
unsafe.Sizeof(reflect.ValueOf(referenceObject).Elem()),
unsafe.Sizeof(reflect.TypeOf(reflect.ValueOf(referenceObject).Elem())),
}
fmt.Println("referenceObject size", rs)
}
This is the output:
expected size 50
valueObject size [8 12 50 12 0 0 0 0]
referenceObject size [8 12 4 12 12 12 12 8]
As you can see, I can get the size of the object when it's passed-by-value using reflect.TypeOf(valueObject).Size(), but nothing gives me the correct size when I pass-by-reference.
There is no "reference" type in go, a pointer is a value like any other, and you can of course get the size of the pointer itself. You also are confusing the reflect.Value, the interface{} and the actual value you want the size of. Contributing to this confusion is that the package unsafe is special, and doesn't take an interface value, rather it can take any value directly, which is properly handled by the compiler.
You could handle the pointer and the struct each with a separate call, or check whether you have a pointer and call Elem():
// struct
reflect.ValueOf(i).Type().Size()
// pointer
reflect.ValueOf(i).Elem().Type().Size()
However, since optionally dereferencing a pointer is quite common, you can use reflect.Indirect to handle both types at once:
reflect.Indirect(reflect.ValueOf(i)).Type().Size()
https://play.golang.org/p/og-uMDXCmEr
For reference, a description of what some the attempts are actually taking:
// size of the interface value
unsafe.Sizeof(valueObject)
// size of reflect.Value
unsafe.Sizeof(reflect.ValueOf(valueObject))
// size of the reflect.Value type
reflect.TypeOf(reflect.ValueOf(valueObject)).Size()
// size of the interface value
unsafe.Sizeof(referenceObject)
// size of the reflect.Value
unsafe.Sizeof(reflect.ValueOf(referenceObject))
// size of the pointer value
reflect.TypeOf(referenceObject).Size()
// size of the reflect.Value type
reflect.TypeOf(reflect.ValueOf(referenceObject)).Size(),
// size of the of reflect.Value type again
reflect.TypeOf(reflect.ValueOf(referenceObject).Elem()).Size()
// size of the reflect.Value
unsafe.Sizeof(reflect.ValueOf(referenceObject).Elem()),
// size of the reflect.Type interface value
unsafe.Sizeof(reflect.TypeOf(reflect.ValueOf(referenceObject).Elem())),
I expected the following code snippet either to produce a as {0x01, 0x02} (pass by value) or as {0x03, 0x02, 0x01} (pass by reference). Strangely, it produces the output as {0x03, 0x02}. Could you please help me understand why is that?
package main
import "fmt"
func test1(t []byte) {
t[0] = 0x03
t = append(t, 0x01 )
}
func main() {
a := []byte{0x01, 0x02 }
test1(a)
_ = a
fmt.Printf("%v", a)
}
In Go []byte is not an array, it's a byte slice.
What happened is that the size of a was 2, and because of that, after its first element was changed to 3, append allocated a new bigger slice - and t was set to the address of the new slice that was allocated, but that doesn't affect the a in main.
From A Tour of Go about append:
If the backing array of s is too small to fit all the given values a bigger array will be allocated. The returned slice will point to the newly allocated array.
The slice header is passed by value, but it references the same backing array. So when you change the first value it is changed in the original array as well. When you call append a new slice header is generated and written to the space occupied by the original argument value thus staying local to the method.
I've started learning Go these days and got stuck in trying to pass a struct property's value (a slice) to a function. Apparently it's being passed as a reference (or it holds a pointer to its slice) and changes made inside the function affect it.
Here is my code, in which testFunction is supposed to receive a slice, remove its first 3 elements and print the updated values, but without affecting it externally:
package main
import (
"fmt"
)
type testStruct struct {
testArray []float64
}
var test = testStruct {
testArray: []float64{10,20,30,40,50},
}
func main() {
fmt.Println(test.testArray)
testFunction(test.testArray)
fmt.Println(test.testArray)
}
func testFunction(array []float64) {
for i:=0; i<3; i++ {
array = removeFrom(array, 0)
}
fmt.Println(array)
}
func removeFrom(array []float64, index int) []float64 {
return append(array[:index], array[index+1:]...)
}
That outputs:
[10 20 30 40 50]
[40 50]
[40 50 50 50 50]
My question is: what is causing the third fmt.Println to print this strange result?
Playground: https://play.golang.org/p/G8W3H085In
p.s.: This code is only an example. It's not my goal to remove the first elements of something. I just wanna know what is causing this strange behaviour.
Usually we don't know whether a given call to append will cause a reallocation, so we can't assume that the original slice refers to the same array as the resulting slice, nor that it refers to a different one.
To use slices correctly, it's important to remember that although the elements of the underlying array are indirect, the slice's pointer, length and capacity are not.
As a result, it's usual to assign the result of a call to append to the same slice variable:
array = append(array, ...)
So to sum up, to receive the desired result always remember to assign the append function to a new or the same slice variable.
Here is the corrected and working code:
package main
import (
"fmt"
)
type testStruct struct {
testArray []float64
}
var test = testStruct {
testArray: []float64{10,20,30,40,50},
}
func main() {
fmt.Println(test.testArray)
a := testFunction(test.testArray)
fmt.Println(a)
}
func testFunction(array []float64)[]float64 {
for i:=0; i<3; i++ {
array = removeFrom(array, 0)
}
fmt.Println(array)
return array
}
func removeFrom(array []float64, index int) []float64 {
return append(array[:index], array[index+1:]...)
}
Check it the working code on Go Playground.
Another solution is to pass the array argument via pointer reference:
func testFunction(array *[]float64) {
for i:=0; i<3; i++ {
*array = removeFrom(*array, 0)
}
fmt.Println(*array)
}
Go Playground
The slice is a composite type. It has a pointer to the data, the length and the capacity. When you pass it as an argument you're passing those values, the pointer, the length and the capacity; they are copies, always.
In your case you modify the data within the slice when you call removeFrom(), which you can do because you've copied the value of a pointer to the original data into the func, but the length and capacity remain unchanged outside the scope of that function as those are not pointers.
So, when you print it again from main() you see the altered values but it still uses the original length and capacity as any changes made to those within the scope of the other funcs were actually on copies of those values.
Here is a useful blog post about slices https://blog.golang.org/slices. It states this in particular.
It's important to understand that even though a slice contains a
pointer, it is itself a value. Under the covers, it is a struct value
holding a pointer and a length. It is not a pointer to a struct.
The reason you see [40 50 50 50 50] is because you changed the values in the slice, but you did not alter the slice itself(it's cap and len)
I am using this construct to simulate a set
type MyType uint8
map[MyType]interface{}
I then add in all my keys and map them to nil.
I've learnt that it is also possible to use
map[MyType]struct{}
Any benefits of using the empty struct versus interface{}.
Memory usage. For example, types struct{}, interface{}, and bool,
package main
import (
"fmt"
"unsafe"
)
func main() {
var s struct{}
fmt.Println(unsafe.Sizeof(s))
var i interface{}
fmt.Println(unsafe.Sizeof(i))
var b bool
fmt.Println(unsafe.Sizeof(b))
}
Output (bytes for 32-bit architecture):
0
8
1
Output (bytes for 64-bit architecture):
0
16
1
References:
Go Data Structures: Interfaces
The empty struct and empty interface, though syntactically similar, are actually opposites. An empty struct holds no data; an empty interface can hold any type of value. If I see a map[MyType]struct{}, I know immediately that no values will be stored, only keys. If I see a map[MyType]interface{}, my first impression will be that it is a heterogenous collection of values. Even if I see code storing nil in it, I won't know for sure that some other piece of code doesn't store something else in it.
In other words, using struct{} makes your code much more readable. It also saves a little memory, as described in the other answer, but that is just a fringe benefit of using the right tool for the job.
I would like to add additional detail about empty struct , as differences are already covered by andybalholm and peterSO .
Below is the example which shows usability of empty struct .
Creates an instance of rectangle struct by using a pointer address
operator is denoted by & symbol.
package main
import "fmt"
type rectangle struct {
length int
breadth int
color string
}
func main() {
var rect1 = &rectangle{10, 20, "Green"} // Can't skip any value
fmt.Println(rect1)
var rect2 = &rectangle{}
rect2.length = 10
rect2.color = "Red"
fmt.Println(rect2) // breadth skipped
var rect3 = &rectangle{}
(*rect3).breadth = 10
(*rect3).color = "Blue"
fmt.Println(rect3) // length skipped
}
Reference : https://www.golangprograms.com/go-language/struct.html
For a thorough read you can refer : https://dave.cheney.net/2014/03/25/the-empty-struct
I get a byte slice ([]byte) from a UDP socket and want to treat it as an integer slice ([]int32) without changing the underlying array, and vice versa. In C(++) I would just cast between pointer types; how would I do this in Go?
As others have said, casting the pointer is considered bad form in Go. Here are examples of the proper Go way and the equivalent of the C array casting.
WARNING: all code untested.
The Right Way
In this example, we are using the encoding/binary package to convert each set of 4 bytes into an int32. This is better because we are specifying the endianness. We are also not using the unsafe package to break the type system.
import "encoding/binary"
const SIZEOF_INT32 = 4 // bytes
data := make([]int32, len(raw)/SIZEOF_INT32)
for i := range data {
// assuming little endian
data[i] = int32(binary.LittleEndian.Uint32(raw[i*SIZEOF_INT32:(i+1)*SIZEOF_INT32]))
}
The Wrong Way (C array casting)
In this example, we are telling Go to ignore the type system. This is not a good idea because it may fail in another implementation of Go. It is assuming things not in the language specification. However, this one does not do a full copy. This code uses unsafe to access the "SliceHeader" which is common in all slices. The header contains a pointer to the data (C array), the length, and the capacity. Instead of just converting the header to the new slice type, we first need to change the length and capacity since there are less elements if we treat the bytes as a new type.
import (
"reflect"
"unsafe"
)
const SIZEOF_INT32 = 4 // bytes
// Get the slice header
header := *(*reflect.SliceHeader)(unsafe.Pointer(&raw))
// The length and capacity of the slice are different.
header.Len /= SIZEOF_INT32
header.Cap /= SIZEOF_INT32
// Convert slice header to an []int32
data := *(*[]int32)(unsafe.Pointer(&header))
You do what you do in C, with one exception - Go does not allow to convert from one pointer type to another. Well, it does, but you must use unsafe.Pointer to tell compiler that you are aware that all rules are broken and you know what you are doing. Here is an example:
package main
import (
"fmt"
"unsafe"
)
func main() {
b := []byte{1, 0, 0, 0, 2, 0, 0, 0}
// step by step
pb := &b[0] // to pointer to the first byte of b
up := unsafe.Pointer(pb) // to *special* unsafe.Pointer, it can be converted to any pointer
pi := (*[2]uint32)(up) // to pointer to the first uint32 of array of 2 uint32s
i := (*pi)[:] // creates slice to our array of 2 uint32s (optional step)
fmt.Printf("b=%v i=%v\n", b, i)
// all in one go
p := (*[2]uint32)(unsafe.Pointer(&b[0]))
fmt.Printf("b=%v p=%v\n", b, p)
}
Obviously, you should be careful about using "unsafe" package, because Go compiler is not holding your hand anymore - for example, you could write pi := (*[3]uint32)(up) here and compiler wouldn't complain, but you would be in trouble.
Also, as other people pointed already, bytes of uint32 might be layout differently on different computers, so you should not assume these are layout as you need them to be.
So safest approach would be to read your array of bytes one by one and make whatever you need out of them.
Alex
The short answer is you can't. Go wont let you cast a slice of one type to a slice of another type. You will have loop through the array and create another array of the type you want while casting each item in the array. This is generally regarded as a good thing since typesafety is an important feature of go.
Since Go 1.17, there is a simpler way to do this using the unsafe package.
import (
"unsafe"
)
const SIZEOF_INT32 = unsafe.Sizeof(int32(0)) // 4 bytes
func main() {
var bs []byte
// Do stuff with `bs`. Maybe do some checks ensuring that len(bs) % SIZEOF_INT32 == 0
data := unsafe.Slice((*int32)(unsafe.Pointer(&bs[0])), len(bs)/SIZEOF_INT32)
// A more verbose alternative requiring `import "reflect"`
// data := unsafe.Slice((*int32)(unsafe.Pointer((*reflect.SliceHeader)(unsafe.Pointer(&bs)).Data)), len(bs)/SIZEOF_INT32)
}
Go 1.17 and beyond
Go 1.17 introduced the unsafe.Slice function, which does exactly this.
Converting a []byte to a []int32:
package main
import (
"fmt"
"unsafe"
)
func main() {
theBytes := []byte{
0x33, 0x44, 0x55, 0x66,
0x11, 0x22, 0x33, 0x44,
0x77, 0x66, 0x55, 0x44,
}
numInts := uintptr(len(theBytes)) * unsafe.Sizeof(theBytes[0]) / unsafe.Sizeof(int32(0))
theInts := unsafe.Slice((*int32)(unsafe.Pointer(&theBytes[0])), numInts)
for _, n := range theInts {
fmt.Printf("%04x\n", n)
}
}
Playground.
I had the size unknown problem and tweaked the previous unsafe method with the following code.
given a byte slice b ...
int32 slice is (*(*[]int)(Pointer(&b)))[:len(b)/4]
The array to slice example may be given a fictional large constant and the slice bounds used in the same way since no array is allocated.
You can do it with the "unsafe" package
package main
import (
"fmt"
"unsafe"
)
func main() {
var b [8]byte = [8]byte{1, 2, 3, 4, 5, 6, 7, 8}
var s *[4]uint16 = (*[4]uint16)(unsafe.Pointer(&b))
var i *[2]uint32 = (*[2]uint32)(unsafe.Pointer(&b))
var l *uint64 = (*uint64)(unsafe.Pointer(&b))
fmt.Println(b)
fmt.Printf("%04x, %04x, %04x, %04x\n", s[0], s[1], s[2], s[3])
fmt.Printf("%08x, %08x\n", i[0], i[1])
fmt.Printf("%016x\n", *l)
}
/*
* example run:
* $ go run /tmp/test.go
* [1 2 3 4 5 6 7 8]
* 0201, 0403, 0605, 0807
* 04030201, 08070605
* 0807060504030201
*/
Perhaps it was not available when the earlier answers were given, but it would seem that the binary.Read method would be a better answer than "the right way" given above.
This method allows you to read binary data from a reader directly into the value or buffer of your desired type. You can do this by creating a reader over your byte array buffer. Or, if you have control of the code that is giving you the byte array, you can replace it to read directly into your buffer without the need for the interim byte array.
See https://golang.org/pkg/encoding/binary/#Read for the documentation and a nice little example.
http://play.golang.org/p/w1m5Cs-ecz
package main
import (
"fmt"
"strings"
)
func main() {
s := []interface{}{"foo", "bar", "baz"}
b := make([]string, len(s))
for i, v := range s {
b[i] = v.(string)
}
fmt.Println(strings.Join(b, ", "))
}
func crackU32s2Bytes(us []uint32) []byte {
var bs []byte
var ptrBs = (*reflect.SliceHeader)(unsafe.Pointer(&bs))
var ptrUs = (*reflect.SliceHeader)(unsafe.Pointer(&us))
ptrBs.Data = ptrUs.Data
ptrBs.Len = ptrUs.Len*4
ptrBs.Cap = ptrBs.Len
return bs
}
func crackBytes2U32s(bs []byte) []uint32 {
var us []uint32
var ptrBs = (*reflect.SliceHeader)(unsafe.Pointer(&bs))
var ptrUs = (*reflect.SliceHeader)(unsafe.Pointer(&us))
ptrUs.Data = ptrBs.Data
ptrUs.Len = ptrBs.Len/4
ptrUs.Cap = ptrUs.Len
return us
}