How to get memory size of variable? - memory-management

Does anybody know how to get memory size of the variable (int, string, []struct, etc) and print it? Is it possible?
var i int = 1
//I want to get something like this:
fmt.Println("Size of i is: %?", i)
//Also, it would be nice if I could store the value into a string

You can use the unsafe.Sizeof function for this.
It returns the size in bytes, occupied by the value you pass into it.
Here's a working example:
package main
import "fmt"
import "unsafe"
func main() {
a := int(123)
b := int64(123)
c := "foo"
d := struct {
FieldA float32
FieldB string
}{0, "bar"}
fmt.Printf("a: %T, %d\n", a, unsafe.Sizeof(a))
fmt.Printf("b: %T, %d\n", b, unsafe.Sizeof(b))
fmt.Printf("c: %T, %d\n", c, unsafe.Sizeof(c))
fmt.Printf("d: %T, %d\n", d, unsafe.Sizeof(d))
}
Take note that some platforms explicitly disallow the use of unsafe, because it is.. well, unsafe. This used to include AppEngine. Not sure if that is still the case today, but I imagine so.
As #Timur Fayzrakhmanov notes, reflect.TypeOf(variable).Size() will give you the same information. For the reflect package, the same restriction goes as for the unsafe package. I.e.: some platforms may not allow its use.

You can do it with either unsafe.Sizeof(), or reflect.Type.Size()

The size of a variable can be determined by using unsafe.Sizeof(a). The result will remain the same for a given type (i.e. int, int64, string, struct etc), irrespective of the value it holds. However, for type string, you may be interested in the size of the string that the variable references, and this is determined by using len(a) function on a given string. The following snippet illustrates that size of a variable of type string is always 8 but the length of a string that a variable references can vary:
package main
import "fmt"
import "unsafe"
func main() {
s1 := "foo"
s2 := "foobar"
fmt.Printf("s1 size: %T, %d\n", s1, unsafe.Sizeof(s1))
fmt.Printf("s2 size: %T, %d\n", s2, unsafe.Sizeof(s2))
fmt.Printf("s1 len: %T, %d\n", s1, len(s1))
fmt.Printf("s2 len: %T, %d\n", s2, len(s2))
}
Output:
s1 size: string, 8
s2 size: string, 8
s1 len: string, 3
s2 len: string, 6
The last part of your question is about assigning the length (i.e. an int value) to a string. This can be done by s := strconv.Itoa(i) where i is an int variable and the string returned by the function is assigned to s.
Note: the name of the converter function is Itoa, possibly a short form for Integer to ASCII. Most Golang programmers are likely to misread the function name as Iota.

I've written a package which calculates the actual memory size consumed by variable at runtime: https://github.com/DmitriyVTitov/size
It has single function, so basic usage is:
fmt.Println(size.Of(varName))

unsafe.Sizeof() is the correct solution.
var i int
var u uint
var up uintptr
fmt.Printf("i Type:%T Size:%d\n", i, unsafe.Sizeof(i))
fmt.Printf("u Type:%T Size:%d\n", u, unsafe.Sizeof(u))
fmt.Printf("up Type:%T Size:%d\n", up, unsafe.Sizeof(up))
The int, uint, and uintptr types are usually 32 bits wide on 32-bit systems and 64 bits wide on 64-bit systems. When you need an integer value you should use int unless you have a specific reason to use a sized or unsigned integer type.

Related

How does golang implemented the convertion between []byte and string?

I am not able to get the answer by checking the generated assemblies:
{
a := []byte{'a'}
s1 := string(a)
a[0] = 'b'
fmt.Println(s1) // a
}
{
a := "a"
b := []byte(a)
b[0] = 'b'
fmt.Println(a) // a
}
Why the observed behavior is happening?
Is there a description of how go interprets these lines of code?
What does go compiler do for type conversion?
This isn't so much a compiler issue as it is a language specification issue. The compiler can and will do strange things sometimes--what matters here is that whatever machine code the compiler ends up spitting out, it follows the rules laid out in the language specification.
As mentioned in the comments, the language specification defines the conversion of byte slices to and from string types like this:
Converting a slice of bytes to a string type yields a string whose successive bytes are the elements of the slice.
Converting a value of a string type to a slice of bytes type yields a slice whose successive elements are the bytes of the string.
In order to understand the behavior of your examples, you have to also read the definition of string types, also in the specification:
Strings are immutable: once created, it is impossible to change the contents of a string.
Because []byte is mutable, behind the scenes go must make a copy of the relevant data when converting to and from a string. This can be verified by printing the addresses of the 0th element of the []byte object and the pointer to the first element of data in the string object. Here is an example (and a Go Playground version):
package main
import (
"fmt"
"reflect"
"unsafe"
)
func main() {
a := "a"
b := []byte(a)
ah := (*reflect.StringHeader)(unsafe.Pointer(&a))
fmt.Printf("a: %4s # %#x\n", a, ah.Data)
fmt.Printf("b: %v # %p\n\n", b, b)
c := []byte{'a'}
d := string(c)
dh := (*reflect.StringHeader)(unsafe.Pointer(&d))
fmt.Printf("c: %v # %p\n", c, c)
fmt.Printf("d: %4s # %#x\n", d, dh.Data)
}
The output looks like this:
a: a # 0x4c1ab2
b: [97] # 0xc00002c008
c: [97] # 0xc00002c060
d: a # 0x554e21
Notice that the pointer locations of the string and []byte are not the same and do not overlap. Therefore there is no expectation that changes to the []byte values will effect the string values in any way.
Okay, technically the result didn't have to be this way because I didn't make any changes in my example to the values of b or c. Technically the compiler could have taken a shortcut and simply called b a length=1 []byte starting at the same memory address as a. But that optimization would not be allowed if I did something like this instead:
package main
import (
"fmt"
"reflect"
"unsafe"
)
func main() {
a := "a"
b := []byte(a)
b[0] = 'b'
ah := (*reflect.StringHeader)(unsafe.Pointer(&a))
fmt.Printf("a: %4s # %#x\n", a, ah.Data)
fmt.Printf("b: %v # %p\n\n", b, b)
}
Output:
a: a # 0x4c1ab2
b: [98] # 0xc00002c008
See this in action at the Go Playground.

How to initialise a pointer member in a struct type inline?

With a type:
type A struct {
B int, C *int
}
How do I initialise a pointer member to a non-zero value inline, without creating temporaries?
a := A{
B: 42,
C: ?,
}
For the specific example you've given you are limited in what you can do without introducing additional statements.
If you want C to point at an initialised integer variable, you will need an additional statement to define that variable since you can't take the address of an integer literal (i.e. &42 would be an error).
If you just want to initialise C as a pointer to a new zero value, you are in luck though and can set it to new(int).
If you were instead dealing with a different type that had an initialiser syntax, you would also been in luck. For example, if C was a pointer to a structure, you could initialise it to &TheStruct{...}.
If none of these are appropriate, and you are really only after code clarity for initialising *int variables, a helper function might fit your requirements. For example:
func makeIntPointer(v int) *int {
return &v
}
package main
import "fmt"
type A struct {
B int
C *int
}
func newint(i int) *int{
return &i
}
func main() {
c := newint(5)
a := &A{
B: 42,
C: c,
}
fmt.Printf(" %v" , *a.C)
fmt.Printf(" %#v" , a.C)
}
http://play.golang.org/p/s0HIMHoMRo

Why the binary.Size() return (-1)?

The code snippet likes this:
package main
import (
"fmt"
"encoding/binary"
"reflect"
)
const (
commandLen = 1
bufLen int = 4
)
func main(){
fmt.Printf("%v %v\n", reflect.TypeOf(commandLen), reflect.TypeOf(bufLen))
fmt.Printf("%d %d", binary.Size(commandLen), binary.Size(bufLen))
}
And the output is:
int int
-1 -1
I think since the types of commandLen and bufLen are int, and from "Programming in golang",
the int should be int32 or int64 which depending on the implementation, so I think the binary.Size() should return a value, not (-1).
Why the binary.Size() return (-1)?
tl;dr
int is not a fixed-length type, so it won't work. Use something that has a fixed length, for example int32.
Explanation
This might look like a bug but it is actually not a bug. The documentation of Size() says:
Size returns how many bytes Write would generate to encode the value v, which must be a
fixed-size value or a slice of fixed-size values, or a pointer to such data.
A fixed-size value is a value that is not dependent on the architecture and the size is known
beforehand. This is the case for int32 or int64 but not for int as it depends on the
environment's architecture. See the documentation of int.
If you're asking yourself why Size() enforces this, consider encoding an int on your
64 bit machine and decoding the data on a remote 32 bit machine. This is only possible if you have length encoded types, which int is not. So you either have to store the size along with the type or enforce fixed-length types which the developers did.
This is reflected in the sizeof() function of encoding/binary that computes the size:
case reflect.Uint8, reflect.Uint16, reflect.Uint32, reflect.Uint64,
reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64,
reflect.Float32, reflect.Float64, reflect.Complex64, reflect.Complex128:
return int(t.Size()), nil
As you can see, there are all number types listed but reflect.Int.

Is there a difference between new() and "regular" allocation?

In Go, is there a notable difference between the following two segments of code:
v := &Vector{}
as opposed to
v := new(Vector)
No. What they return is the same,
package main
import "fmt"
import "reflect"
type Vector struct {
x int
y int
}
func main() {
v := &Vector{}
x := new(Vector)
fmt.Println(reflect.TypeOf(v))
fmt.Println(reflect.TypeOf(x))
}
Result:
*main.Vector
*main.Vector
There is some contention on the mailing list that having both is confusing:
https://groups.google.com/forum/?fromgroups=#!topic/golang-nuts/GDXFDJgKKSs
One thing to note:
new() is the only way to get a pointer to an
unnamed integer or other basic type. You can write "p := new(int)" but
you can't write "p := &int{0}". Other than that, it's a matter of
preference.
Source : https://groups.google.com/d/msg/golang-nuts/793ZF_yeqbk/-zyUAPT-e4IJ
Yes, there is a fundamental difference between the two code fragments.
v := &Vector{}
Works only for Vector being a struct type, map type, array type or a slice type
v := new(Vector)
Works for Vector of any type.
Example: http://play.golang.org/p/nAHjL1ZEuu
Here is a difference: for a Person struct, the JSON string marshalled from &[]*Person{} is [] and from new([]*Person) is null using json.Marshal.
Check out the sample here: https://play.golang.org/p/xKkFLoMXX1s
https://golang.org/doc/effective_go.html#allocation_new
https://golang.org/doc/effective_go.html#composite_literals

Convert between slices of different types

I get a byte slice ([]byte) from a UDP socket and want to treat it as an integer slice ([]int32) without changing the underlying array, and vice versa. In C(++) I would just cast between pointer types; how would I do this in Go?
As others have said, casting the pointer is considered bad form in Go. Here are examples of the proper Go way and the equivalent of the C array casting.
WARNING: all code untested.
The Right Way
In this example, we are using the encoding/binary package to convert each set of 4 bytes into an int32. This is better because we are specifying the endianness. We are also not using the unsafe package to break the type system.
import "encoding/binary"
const SIZEOF_INT32 = 4 // bytes
data := make([]int32, len(raw)/SIZEOF_INT32)
for i := range data {
// assuming little endian
data[i] = int32(binary.LittleEndian.Uint32(raw[i*SIZEOF_INT32:(i+1)*SIZEOF_INT32]))
}
The Wrong Way (C array casting)
In this example, we are telling Go to ignore the type system. This is not a good idea because it may fail in another implementation of Go. It is assuming things not in the language specification. However, this one does not do a full copy. This code uses unsafe to access the "SliceHeader" which is common in all slices. The header contains a pointer to the data (C array), the length, and the capacity. Instead of just converting the header to the new slice type, we first need to change the length and capacity since there are less elements if we treat the bytes as a new type.
import (
"reflect"
"unsafe"
)
const SIZEOF_INT32 = 4 // bytes
// Get the slice header
header := *(*reflect.SliceHeader)(unsafe.Pointer(&raw))
// The length and capacity of the slice are different.
header.Len /= SIZEOF_INT32
header.Cap /= SIZEOF_INT32
// Convert slice header to an []int32
data := *(*[]int32)(unsafe.Pointer(&header))
You do what you do in C, with one exception - Go does not allow to convert from one pointer type to another. Well, it does, but you must use unsafe.Pointer to tell compiler that you are aware that all rules are broken and you know what you are doing. Here is an example:
package main
import (
"fmt"
"unsafe"
)
func main() {
b := []byte{1, 0, 0, 0, 2, 0, 0, 0}
// step by step
pb := &b[0] // to pointer to the first byte of b
up := unsafe.Pointer(pb) // to *special* unsafe.Pointer, it can be converted to any pointer
pi := (*[2]uint32)(up) // to pointer to the first uint32 of array of 2 uint32s
i := (*pi)[:] // creates slice to our array of 2 uint32s (optional step)
fmt.Printf("b=%v i=%v\n", b, i)
// all in one go
p := (*[2]uint32)(unsafe.Pointer(&b[0]))
fmt.Printf("b=%v p=%v\n", b, p)
}
Obviously, you should be careful about using "unsafe" package, because Go compiler is not holding your hand anymore - for example, you could write pi := (*[3]uint32)(up) here and compiler wouldn't complain, but you would be in trouble.
Also, as other people pointed already, bytes of uint32 might be layout differently on different computers, so you should not assume these are layout as you need them to be.
So safest approach would be to read your array of bytes one by one and make whatever you need out of them.
Alex
The short answer is you can't. Go wont let you cast a slice of one type to a slice of another type. You will have loop through the array and create another array of the type you want while casting each item in the array. This is generally regarded as a good thing since typesafety is an important feature of go.
Since Go 1.17, there is a simpler way to do this using the unsafe package.
import (
"unsafe"
)
const SIZEOF_INT32 = unsafe.Sizeof(int32(0)) // 4 bytes
func main() {
var bs []byte
// Do stuff with `bs`. Maybe do some checks ensuring that len(bs) % SIZEOF_INT32 == 0
data := unsafe.Slice((*int32)(unsafe.Pointer(&bs[0])), len(bs)/SIZEOF_INT32)
// A more verbose alternative requiring `import "reflect"`
// data := unsafe.Slice((*int32)(unsafe.Pointer((*reflect.SliceHeader)(unsafe.Pointer(&bs)).Data)), len(bs)/SIZEOF_INT32)
}
Go 1.17 and beyond
Go 1.17 introduced the unsafe.Slice function, which does exactly this.
Converting a []byte to a []int32:
package main
import (
"fmt"
"unsafe"
)
func main() {
theBytes := []byte{
0x33, 0x44, 0x55, 0x66,
0x11, 0x22, 0x33, 0x44,
0x77, 0x66, 0x55, 0x44,
}
numInts := uintptr(len(theBytes)) * unsafe.Sizeof(theBytes[0]) / unsafe.Sizeof(int32(0))
theInts := unsafe.Slice((*int32)(unsafe.Pointer(&theBytes[0])), numInts)
for _, n := range theInts {
fmt.Printf("%04x\n", n)
}
}
Playground.
I had the size unknown problem and tweaked the previous unsafe method with the following code.
given a byte slice b ...
int32 slice is (*(*[]int)(Pointer(&b)))[:len(b)/4]
The array to slice example may be given a fictional large constant and the slice bounds used in the same way since no array is allocated.
You can do it with the "unsafe" package
package main
import (
"fmt"
"unsafe"
)
func main() {
var b [8]byte = [8]byte{1, 2, 3, 4, 5, 6, 7, 8}
var s *[4]uint16 = (*[4]uint16)(unsafe.Pointer(&b))
var i *[2]uint32 = (*[2]uint32)(unsafe.Pointer(&b))
var l *uint64 = (*uint64)(unsafe.Pointer(&b))
fmt.Println(b)
fmt.Printf("%04x, %04x, %04x, %04x\n", s[0], s[1], s[2], s[3])
fmt.Printf("%08x, %08x\n", i[0], i[1])
fmt.Printf("%016x\n", *l)
}
/*
* example run:
* $ go run /tmp/test.go
* [1 2 3 4 5 6 7 8]
* 0201, 0403, 0605, 0807
* 04030201, 08070605
* 0807060504030201
*/
Perhaps it was not available when the earlier answers were given, but it would seem that the binary.Read method would be a better answer than "the right way" given above.
This method allows you to read binary data from a reader directly into the value or buffer of your desired type. You can do this by creating a reader over your byte array buffer. Or, if you have control of the code that is giving you the byte array, you can replace it to read directly into your buffer without the need for the interim byte array.
See https://golang.org/pkg/encoding/binary/#Read for the documentation and a nice little example.
http://play.golang.org/p/w1m5Cs-ecz
package main
import (
"fmt"
"strings"
)
func main() {
s := []interface{}{"foo", "bar", "baz"}
b := make([]string, len(s))
for i, v := range s {
b[i] = v.(string)
}
fmt.Println(strings.Join(b, ", "))
}
func crackU32s2Bytes(us []uint32) []byte {
var bs []byte
var ptrBs = (*reflect.SliceHeader)(unsafe.Pointer(&bs))
var ptrUs = (*reflect.SliceHeader)(unsafe.Pointer(&us))
ptrBs.Data = ptrUs.Data
ptrBs.Len = ptrUs.Len*4
ptrBs.Cap = ptrBs.Len
return bs
}
func crackBytes2U32s(bs []byte) []uint32 {
var us []uint32
var ptrBs = (*reflect.SliceHeader)(unsafe.Pointer(&bs))
var ptrUs = (*reflect.SliceHeader)(unsafe.Pointer(&us))
ptrUs.Data = ptrBs.Data
ptrUs.Len = ptrBs.Len/4
ptrUs.Cap = ptrUs.Len
return us
}

Resources