I'm trying to learn more regarding memory usage.
Doing some tests with interface{} and struct{} slices, I noticed that a slice of struct{} doesn't allocate any memory whereas a slice of interface{} does. It doesn't make so much sense to me, I'm actually expecting the same behavior (ie. both allocate nothing). Anyway I couldn't find any explanation regarding this particular case.
Could someone explain me why this happens?
package main
import (
"runtime"
"fmt"
)
func main() {
// Below is an example of using our PrintMemUsage() function
// Print our starting memory usage (should be around 0mb)
fmt.Println("Start")
PrintMemUsage()
fmt.Println("")
structContainer := make([]struct{}, 1000000)
for i := 0; i<1000000; i++ {
structContainer[i] = struct{}{}
}
fmt.Println("With 1kk struct{}")
PrintMemUsage()
fmt.Println("")
nilContainer := make([]interface{}, 1000000)
for i := 0; i<1000000; i++ {
nilContainer[i] = nil
}
fmt.Println("With 1kk nil interface{}")
PrintMemUsage()
fmt.Println("")
}
// PrintMemUsage outputs the current, total and OS memory being used. As well as the number
// of garage collection cycles completed.
func PrintMemUsage() {
var m runtime.MemStats
runtime.ReadMemStats(&m)
// For info on each, see: https://golang.org/pkg/runtime/#MemStats
fmt.Printf("Alloc = %v KiB", bToMb(m.Alloc))
fmt.Printf("\tTotalAlloc = %v KiB", bToMb(m.TotalAlloc))
fmt.Printf("\tSys = %v KiB", bToMb(m.Sys))
fmt.Printf("\tNumGC = %v\n", m.NumGC)
}
func bToMb(b uint64) uint64 {
return b / 1024
}
Playground link.
A variable of type interface{} can hold any value. E.g. it can hold the integer 8, it can hold the string value "hi", it can hold the struct value image.Point{X: 1, Y: 2} and pretty much everything else.
If you allocate a slice having interface{} as its element type, memory have to be allocated so that you can store any values in its elements. When using make() to allocate it, all its elements will get the zero value of the element type (which is nil for the interface{}), but memory still has to be allocated else you couldn't set elements later on.
On the other hand, the empty struct struct{} has no fields, it cannot hold any values (other than struct{}). When you allocate a slice having struct{} as its element type, memory does not need to be allocated because you won't be able to store anything in it that would require memory. So it's a simple and clever optimization not to allocate memory for such a type.
This is because an empty struct contains no value.
This is not very useful for arrays or slices. But it is useful for maps. A map without value is like a set. You can insert keys and test if they are present. The absence of value save space as you discovered.
Related
A related questions is here https://stackoverflow.com/a/12965872/6421681.
In go, you can do:
func numsInFactorial(n int) (nums []int) {
// `nums := make([]int)` is not needed
for i := 1; i <= n; i++ {
nums = append(nums, i)
}
return
}
However,the following doesn't work:
func mapWithOneKeyAndValue(k int, v int) (m map[int]int) {
m[k] = v
return
}
An error is thrown:
panic: assignment to entry in nil map
Instead, you must:
func mapWithOneKeyAndValue(k int, v int) map[int]int {
m := make(map[int]int)
m[k] = v
return
}
I can't find the documentation for this behavior.
I have read through all of effective go, and there's no mention of it there either.
I know that named return values are defined (i.e. memory is allocated; close to what new does) but not initialized (so make behavior isn't replicated).
After some experimenting, I believe this behavior can be reduced into understanding the behavior of the following code:
func main() {
var s []int // len and cap are both 0
var m map[int]int
fmt.Println(s) // works... prints an empty slice
fmt.Println(m) // works... prints an empty map
s = append(s, 10) // returns a new slice, so underlying array gets allocated
fmt.Println(s) // works... prints [10]
m[10] = 10 // program crashes, with "assignment to entry in nil map"
fmt.Println(m)
}
The issue seems that append likely calls make and allocates a new slice detecting that the capacity of s is 0. However, map never gets an explicit initialization.
The reason for this SO question is two-pronged. First, I would like to document the behavior on SO. Second, why would the language allow non-initializing definitions of slice and map? With my experience with go so far, it seems to be a pragmatic language (i.e. unused variables lead to compilation failure, gofmt forces proper formatting), so it would make sense for it to prevent the code from compiling.
Try to assign in nil slice by index - you will get "panic: runtime error: index out of range" (example: https://play.golang.org/p/-XHh1jNyn5g)
The only reason why append function works with nil, is that append function can do reallocation for the given slice.
For example, if you trying to to append 6th element to slice of 5 elements with current capacity 5, it will create the new array with new capacity, copy all the info from old one, and swap the data array pointers in the given slice. In my understanding, it is just golang implementation of dynamic arrays.
So, the nil slice is just a special case of slice with not enough capacity, so it would be reallocated on any append operation.
More details on https://blog.golang.org/go-slices-usage-and-internals
From https://blog.golang.org/go-maps-in-action
A nil map behaves like an empty map when reading, but attempts to write to a nil map will cause a runtime panic; don't do that. To initialize a map, use the built in make function
It seems like a nil map is considered a valid empty map and that's the reason they don't allocate memory for it automatically.
I've started learning Go these days and got stuck in trying to pass a struct property's value (a slice) to a function. Apparently it's being passed as a reference (or it holds a pointer to its slice) and changes made inside the function affect it.
Here is my code, in which testFunction is supposed to receive a slice, remove its first 3 elements and print the updated values, but without affecting it externally:
package main
import (
"fmt"
)
type testStruct struct {
testArray []float64
}
var test = testStruct {
testArray: []float64{10,20,30,40,50},
}
func main() {
fmt.Println(test.testArray)
testFunction(test.testArray)
fmt.Println(test.testArray)
}
func testFunction(array []float64) {
for i:=0; i<3; i++ {
array = removeFrom(array, 0)
}
fmt.Println(array)
}
func removeFrom(array []float64, index int) []float64 {
return append(array[:index], array[index+1:]...)
}
That outputs:
[10 20 30 40 50]
[40 50]
[40 50 50 50 50]
My question is: what is causing the third fmt.Println to print this strange result?
Playground: https://play.golang.org/p/G8W3H085In
p.s.: This code is only an example. It's not my goal to remove the first elements of something. I just wanna know what is causing this strange behaviour.
Usually we don't know whether a given call to append will cause a reallocation, so we can't assume that the original slice refers to the same array as the resulting slice, nor that it refers to a different one.
To use slices correctly, it's important to remember that although the elements of the underlying array are indirect, the slice's pointer, length and capacity are not.
As a result, it's usual to assign the result of a call to append to the same slice variable:
array = append(array, ...)
So to sum up, to receive the desired result always remember to assign the append function to a new or the same slice variable.
Here is the corrected and working code:
package main
import (
"fmt"
)
type testStruct struct {
testArray []float64
}
var test = testStruct {
testArray: []float64{10,20,30,40,50},
}
func main() {
fmt.Println(test.testArray)
a := testFunction(test.testArray)
fmt.Println(a)
}
func testFunction(array []float64)[]float64 {
for i:=0; i<3; i++ {
array = removeFrom(array, 0)
}
fmt.Println(array)
return array
}
func removeFrom(array []float64, index int) []float64 {
return append(array[:index], array[index+1:]...)
}
Check it the working code on Go Playground.
Another solution is to pass the array argument via pointer reference:
func testFunction(array *[]float64) {
for i:=0; i<3; i++ {
*array = removeFrom(*array, 0)
}
fmt.Println(*array)
}
Go Playground
The slice is a composite type. It has a pointer to the data, the length and the capacity. When you pass it as an argument you're passing those values, the pointer, the length and the capacity; they are copies, always.
In your case you modify the data within the slice when you call removeFrom(), which you can do because you've copied the value of a pointer to the original data into the func, but the length and capacity remain unchanged outside the scope of that function as those are not pointers.
So, when you print it again from main() you see the altered values but it still uses the original length and capacity as any changes made to those within the scope of the other funcs were actually on copies of those values.
Here is a useful blog post about slices https://blog.golang.org/slices. It states this in particular.
It's important to understand that even though a slice contains a
pointer, it is itself a value. Under the covers, it is a struct value
holding a pointer and a length. It is not a pointer to a struct.
The reason you see [40 50 50 50 50] is because you changed the values in the slice, but you did not alter the slice itself(it's cap and len)
I am trying to create a generic function that can handle actions on slices in Go... for instance, append an item of any type to a slice of that same type. This is simply a generic purpose for a more complex solution, but overall the issue boils down to this example:
package main
type car struct {
make string
color string
}
type submarine struct {
name string
length int
}
func genericAppender(thingList interface{}, thing interface{}) []interface{} {
return append(thingList, thing)
}
func main() {
cars := make([]car, 0, 10)
cars[0] = car{make: "ford", color: "red"}
cars[1] = car{make: "chevy", color: "blue"}
subs := make([]submarine, 0, 10)
subs[0] = submarine{name: "sally", length: 100}
subs[1] = submarine{name: "matilda", length: 200}
newCar := car{make: "bmw", color: "white"}
genericAppender(&cars, newCar)
}
The code playground is at this location
The above errors as follows:
prog.go:14: first argument to append must be slice; have interface {}
After this change you're still getting a runtime error (index out of range) however the problem is that thingList is not of type []interface{} but rather interface{} so you can't append to it. Here's an updated version of your code on playground that does a type assertion to convert it to an []interface{} in line with the append. In reality you need to do that on a separate line and check for errors.
https://play.golang.org/p/YMed0VDZrv
So to put some code here;
func genericAppender(thingList interface{}, thing interface{}) []interface{} {
return append(thingList.([]interface{}), thing)
}
will solve the basic problem you're facing. As noted, you still get runtime errors when indexing into the slice. Also, you could change the argument to avoid this by making it;
func genericAppender(thingList []interface{}, thing interface{}) []interface{} {
return append(thingList, thing)
}
Here's a complete example of the second type; https://play.golang.org/p/dIuW_UG7XY
Note I also corrected the runtime error. When you use make with 3 args they are, in this order, type, length, capacity. This means the length of the array is 0 so when you try to assign to indexes 0 and 1 it was causing a panic for IndexOutoFRange. Instead I removed the middle argument so it's make([]interface{}, 10) meaning the length is initially set to 10 so you can assign to those indexes.
In the answer above if you do the following then it throws error. This is what the original question was about:
//genericAppender(subs, newCar). // Throws "cannot use subs (type []submarine) as type []interface {} in argument to genericAppender"
The trick is to convert your slice of specific type into a generic []interface{}.
func convertToGeneric(thingList interface{}) []interface{} {
input := reflect.ValueOf(thingList)
length := input.Len()
out := make([]interface{},length)
for i:=0 ;i < length; i++ {
out[i] = input.Index(i).Interface()
}
return out
}
This you can call the function like this:
genericAppender(convertToGeneric(subs), newCar)
You can check modified working code here: https://play.golang.org/p/0_Zmme3c8lT
With Go 1.19 (Q4 2022), no need for interface, or "convert your slice of specific type into a generic []interface{}"
CL 363434 comes with a new slices packages:
// Package slices defines various functions useful with slices of any type.
// Unless otherwise specified, these functions all apply to the elements
// of a slice at index 0 <= i < len(s).
package slices
import "constraints"
// Grow increases the slice's capacity, if necessary, to guarantee space for
// another n elements. After Grow(n), at least n elements can be appended
// to the slice without another allocation. If n is negative or too large to
// allocate the memory, Grow panics.
func Grow[S ~[]T, T any](s S, n int) S {
return append(s, make(S, n)...)[:len(s)]
}
// Equal reports whether two slices are equal: the same length and all
// elements equal. If the lengths are different, Equal returns false.
// Otherwise, the elements are compared in index order, and the
// comparison stops at the first unequal pair.
// Floating point NaNs are not considered equal.
func Equal[T comparable](s1, s2 []T) bool {
if len(s1) != len(s2) {
return false
}
for i, v1 := range s1 {
v2 := s2[i]
if v1 != v2 {
return false
}
}
return true
}
// ...
Ian Lance Taylor confirms in issue 45955:
This package is now available at golang.org/x/exp/slices.
Per this thread, it will not be put into standard library until the 1.19 release.
We may of course adjust it based on anything we learn about having it in x/exp.
I'm trying to assign a value to a field, but my program panics with runtime error: invalid memory address or nil pointer dereference.
package main
type Node struct {
Value int
}
func (n *Node) SetValue(value int) {
n.Value = value
}
func main() {
var n *Node
n.SetValue(1)
}
This is reasonable since variable is nil.
But I've fount some Go internal structs are allowed to do this, e.g. bytes.Buffer
package main
import "bytes"
import "io"
import "os"
func main() {
var b bytes.Buffer
b.Write([]byte("Hello world"))
io.Copy(os.Stdout, &b)
}
Here is the `bytes.Buffer source code
func (b *Buffer) Write(p []byte) (n int, err error) {
b.lastRead = opInvalid
m := b.grow(len(p))
return copy(b.buf[m:], p), nil
}
Is it the thing only builtin structs can do or it's possible to accomplish this in my code?
EDIT
Here is the working example. Thanks #twotwotwo for suggestion.
package main
import "fmt"
type Node struct {
Value int
}
func (n *Node) SetValue(value int) {
n.Value = value
}
func main() {
var n Node
n.SetValue(1)
fmt.Println(n.Value)
}
The crucial thing is var b bytes.Buffer doesn't get you a nil pointer, it gets you a bytes.Buffer object with all its fields initialized with their zero values (in machine terms, with zero bytes). The spec says the zero value is "false for booleans, 0 for integers, 0.0 for floats, "" for strings, and nil for pointers, functions, interfaces, slices, channels, and maps"; follow that link for more detail.
It is possible to make your own structs whose zero values work and the Go team encourages it. struct Position { x, y int } is an easy example and Effective Go gives a more realistic one. But note that that doesn't make the nil pointer work; you would still need new(Node) or var n Node to allocate the zero Node. Same for bytes.Buffer.
Another common use of zero values: wherever your users create structs of your type directly (as folks do with, say, http.Server), the zero value is the default for any fields they don't specify. It's the default in a lot of other places: what you get for a not-found map key, if you receive from a closed channel, and probably others.
Trying to create a slice in which the type is set dynamicaly based on a pointer to a specific type, so i made the following sample
func main() {
var chicken *Chicken
//create a slice of chickens
chickens:=GetaDynamiclyTypedSlice(chicken)
//this throws cannot range over chickens (type *[]interface {}) and i cant figure how to create a slice using my above chicken pointer
for _,chicken := range chickens{
fmt.Println(chicken)
}
}
type Chicken struct{
Weight float64
}
func GetaDynamiclyTypedSlice(ptrItemType interface{})*[]interface {}{
var collection []interface{}
itemtyp := reflect.TypeOf(ptrItemType).Elem()
for i:=0;i<1000;i++{
//create an item of the wanted type
item := reflect.New(itemtyp)
//set a random float to the weight value
item.Elem().FieldByName("Weight").SetFloat(rnd.ExpFloat64())
collection = append(collection,&item)
}
return &collection
}
what should i do to be able to use range on the returned slice?
how can i use the itemtyp as the type of my slice?
There are few problems with your code.
You're returning a pointer to a reflect.Value, 99% sure that's not what you're trying to achive.
You're not dereferencing the slice like Simon mentioned.
Slices are pointer types, if you're returning *[]interface{} for performance reasons, you're actually hurting not helping.
So let's rewrite the code and optimize it! (it's late night SO, time to party):
// pass the size to preallocate the slice, also return the correct slice type.
func GetaDynamiclyTypedSlice(ptrItemType interface{}, size int) (col []interface{}) {
col = make([]interface{}, size)
itemtyp := reflect.TypeOf(ptrItemType).Elem()
for i := range col { //prettier than for i := 0; etc etc
item := reflect.New(itemtyp)
item.Elem().FieldByName("Weight").SetFloat(rand.ExpFloat64())
col[i] = item.Interface() //this is the magic word, return the actual item, not reflect.Value
}
return
}
playground
You just need to dereference the pointer (so you're not iterating over a pointer - you're iterating over a slice):
for _, chicken := range *chickens {
// ...
}
Playground link: http://play.golang.org/p/NBv9sooqEV