I am working with a very large map of pointer to struct. It is growing over the lifetime of the program (I use it as a buffer) and I wrote a function that is supposed to reduce it size when it is called.
type S struct {
a uint32
b []uint32
}
s := make(map[uint32]*S)
for k, v := range s {
delete(s, k)
s[k] = &S{a: v.a}
}
I remove b from every element of the map, so I expected the size of the map in memory to shrink (b is a slice of length > 10). However the memory is not freed, why?
The size of the map value &S, a pointer, is the same irrespective of the capacity of slice b.
package main
import (
"fmt"
"unsafe"
)
type S struct {
a uint32
b []uint32
}
func main() {
size := unsafe.Sizeof(&S{})
fmt.Println(size)
size = unsafe.Sizeof(&S{b: make([]uint32, 10)})
fmt.Println(size)
s := make(map[uint32]*S)
for k, v := range s {
delete(s, k)
s[k] = &S{a: v.a}
}
}
Output:
8
8
A slice is represented internally by
type slice struct {
array unsafe.Pointer
len int
cap int
}
When you set &S{a: v.a} you set b to initial values: array to nil and len and cap to zero. The memory formerly occupied by the underlying array is returned to the garbage collector for reuse.
The map size is bounded to the maximum size it had at any point. Because you store pointers (map[uint32]*S) and not values the deleted objects will get garbage collected eventually but usually not immediately and that why you don’t see it in top/htop like monitors.
The runtime is clever enough and reserves memory for future use if the system is not under pressure or low on resources.
See https://stackoverflow.com/a/49963553/1199408 to understand more about memory.
In your example you don't need to call delete. You will achieve what you want just by clearing the slice in the struct.
type S struct {
a uint32
b []uint32
}
s := make(map[uint32]*S)
for k, v := range s {
v.b = []uint32{}
}
Related
I'm trying to implement a very simple queue in Go using slices. This is the code that I have to enqueue five values and then discard the first two values:
package main
import (
"fmt"
)
var (
localQ []int
)
func main() {
fmt.Printf("%v %v\n", localQ, len(localQ))
for i := 0; i< 5; i++ {
localQ = enqueue(localQ, i)
fmt.Printf("%v %v\n", localQ, len(localQ))
}
localQ = dequeue(localQ, 2)
fmt.Printf("%v %v\n", localQ, len(localQ))
}
func enqueue(q []int, n int) ([]int) {
q = append(q, n)
return q
}
func dequeue(q []int, s int) ([]int) {
r := q[s:]
q = nil
return r
}
Two questions regarding the dequeue func:
1- I'm trying to ensure that the popped items are discarded and garbage collected. Does this function result them to be garbage collected?
2- What are the time and space complexities of r := q[s:]? I know there is an array under each slice. Are the array values being copied? Or is it just a pointer being copied?
Does this function result them to be garbage collected?
If the application enqueues a sufficient number of elements to cause the slice backing array to be reallocated, then the previous backing array (and its elements) will be eligible for collection.
What are the time and space complexities of r := q[s:]?
This is an O(1) operation. The operation does not allocate memory on the heap.
Does anyone know why map can be used as a value receiver, but when working with slices only the pointer receiver can be used? Why the map is changing after the method call?
Example for map:
package main
import (
"fmt"
)
type hashMap map[string]int
func (s hashMap) Add(k string, v int) {
s[k] = v
}
func main() {
var s hashMap
s = make(hashMap, 0)
fmt.Println(s)
s.Add("abc", 15)
fmt.Println(s)
}
Output:
map[]
map[abc:15]
Example for slice:
package main
import (
"fmt"
)
type slice []int
func (s *slice) Add(v int) {
(*s) = append(*s, v)
}
func main() {
var s slice
s = make(slice, 0)
fmt.Println(s)
s.Add(15)
fmt.Println(s)
}
Output:
[]
[15]
A map variable, after make, is a pointer to the map header: *hmap. The map pointer is passed by value
// A header for a Go map.
type hmap struct {
// Note: the format of the Hmap is encoded in ../../cmd/internal/gc/reflect.go and
// ../reflect/type.go. Don't change this structure without also changing that code!
count int // # live cells == size of map. Must be first (used by len() builtin)
flags uint8
B uint8 // log_2 of # of buckets (can hold up to loadFactor * 2^B items)
noverflow uint16 // approximate number of overflow buckets; see incrnoverflow for details
hash0 uint32 // hash seed
buckets unsafe.Pointer // array of 2^B Buckets. may be nil if count==0.
oldbuckets unsafe.Pointer // previous bucket array of half the size, non-nil only when growing
nevacuate uintptr // progress counter for evacuation (buckets less than this have been evacuated)
extra *mapextra // optional fields
}
A slice variable is a struct: slice. The slice struct is passed by value.
type slice struct {
array unsafe.Pointer
len int
cap int
}
Map is receiver type while struct is value type,Value type. Hence when you call using map value changes after function call.
I have the following variables:
var foo int8
var bar [5]int8
I want to count number of bytes in both variables and output the sum - so right here it should be 1 + 5 = 6. Is this possible?
You can use reflect.Size, this returns a uintptr for some reason, but is actually just the exact same thing as unsafe.SizeOf without having to use unsafe.
var x [8]byte
t := reflect.TypeOf(x)
fmt.Println(t.Size())
The snags are going to be maps, slices, strings, and pointers, which you'll get the size of the header/metadata (or just the pointer size for pointers). If that's okay, great! If not, you can handle this in various ways, this is the best I have off the top of my head:
func UnderlyingSize(x interface{}) uintptr {
v := reflect.ValueOf(x)
t := v.Type()
var size uintptr;
switch t.Kind() {
// For the builtin collections, we just multiply the len by the
// element size, for maps also do the key
case reflect.Map:
l := uintptr(v.Len())
size = t.Key().Size()*l + t.Elem().Size()*l
case reflect.Slice:
t := t.Elem();
size = t.Size() * uintptr(v.Len())
case reflect.Ptr:
t := t.Elem();
size = t.Size()
// Strings are just byte arrays, so it's just the len
case reflect.String:
size = uintptr(v.Len())
// For an interface, we need to find the underlying type
case reflect.Interface:
v := v.Elem()
size = UnderlyingSize(v)
// For anything else, including arrays, Size returns the correct value
default:
size = t.Size();
}
return size
}
There is an argument to be made for using Cap rather than Len, but it's easy enough to change yourself. You can also add t.Size() to any of these values if you want the size of the header information AND the underlying size. Note a word of warning that the real map probably takes more memory than just the key+value+header size, since there's probably some extra information under the hood.
If you have a data structure that's a collection, you'll have to implement something like this yourself, but if it's a simple struct (i.e. only made of POD structs and builtin types), you can simply add up UnderlyingSize of all the members.
You can use unsafe.Sizeof (https://play.golang.org/p/FroasKud7I):
unsafe.Sizeof(foo) + unsafe.Sizeof(bar)
I wanted to implement time based slots for holding data using golang slices. I managed to come up with a go program like this and it also works. But I have few questions regarding garbage collection and the general performance of this program. Does this program guarantee garbage collection of items once slice is equated to nil? And while shuffling slices, I hope this program does not do any deep copying.
type DataSlots struct {
slotDuration int //in milliseconds
slots [][]interface{}
totalDuration int //in milliseconds
}
func New(slotDur int, totalDur int) *DataSlots {
dat := &DataSlots{slotDuration: slotDur,
totalDuration: totalDur}
n := totalDur / slotDur
dat.slots = make([][]interface{}, n)
for i := 0; i < n; i++ {
dat.slots[i] = make([]interface{}, 0)
}
go dat.manageSlots()
return dat
}
func (self *DataSlots) addData(data interface{}) {
self.slots[0] = append(self.slots[0], data)
}
// This should be a go routine
func (self *DataSlots) manageSlots() {
n := self.totalDuration / self.slotDuration
for {
time.Sleep(time.Duration(self.slotDuration) * time.Millisecond)
for i := n - 1; i > 0; i-- {
self.slots[i] = self.slots[i-1]
}
self.slots[0] = nil
}
}
I removed critical section handling in this snippet to make it concise.
Once your slice is set too nil, any values contained in the slice are available for garbage collection, provided that the underlying array isn't shared with another slice.
Since there are no slice operations in your program, you never have multiple references to the same array, nor are you leaving data in any inaccessible portions of the underlying array.
What you need to be careful of, is when you're using slice operations:
a := []int{1, 2, 3, 4}
b := a[1:3]
a = nil
// the values 1 and 4 can't be collected, because they are
// still contained in b's underlying array
c := []int{1, 2, 3, 4}
c = append(c[1:2], 5)
// c is now []int{2, 5}, but again the values 1 and 4 are
// still in the underlying array. The 4 may be overwritten
// by a later append, but the 1 is inaccessible and won't
// be collected until the underlying array is copied.
While append does copy values when the capacity of the slice in insufficient, only the values contained in the slice are copied. There is no deep copy of any of the values.
What is the max value of *big.Int and max precision of *big.Rat?
Here are the structure definitions :
// A Word represents a single digit of a multi-precision unsigned integer.
type Word uintptr
type nat []Word
type Int struct {
neg bool // sign
abs nat // absolute value of the integer
}
type Rat struct {
// To make zero values for Rat work w/o initialization,
// a zero value of b (len(b) == 0) acts like b == 1.
// a.neg determines the sign of the Rat, b.neg is ignored.
a, b Int
}
There is no explicit limit. The limit will be your memory or, theoretically, the max array size (2^31 or 2^63, depending on your platform).
If you have practical concerns, you might be interested by the tests made in http://golang.org/src/pkg/math/big/nat_test.go, for example the one where 10^100000 is benchmarked.
And you can easily run this kind of program :
package main
import (
"fmt"
"math/big"
)
func main() {
verybig := big.NewInt(1)
ten := big.NewInt(10)
for i:=0; i<100000; i++ {
verybig.Mul(verybig, ten)
}
fmt.Println(verybig)
}
(if you want it to run fast enough for Go Playground, use a smaller exponent than 100000)
The problem won't be the max size but the used memory and the time such computations take.