http://golang.org/ref/spec#Allocation
There is a way to allocate memory, but I don't see a way to deallocate memory (without the Go GC turned on).
If I wanted to write an OS using Go, I would need to either write a low level GC for Go or disable the Go GC. In the latter case, how could I free memory?
PS - this topic has been talked about extensively on the Go mailing list, but I wanted to pose this specific question to SO.
You can free arbitrary memory by making runtime·free accessible to your program
using cgo.
Build your own package called, for example, mem and create two files:
mem.go
package mem
import "unsafe"
import "reflect"
func FreePtr(p unsafe.Pointer)
func Free(v interface {}) {
FreePtr(unsafe.Pointer(reflect.ValueOf(v).Elem().Pointer()))
}
runtime.c
// +build gc
#include <runtime.h>
void ·Free(void* foo) {
runtime·free(foo);
}
Example usage (free a slice and print number of frees):
import "free/mem"
func main() {
var m1, m2 runtime.MemStats
runtime.ReadMemStats(&m1)
c := make([]int, 10000)
inspect.Free(&c)
runtime.ReadMemStats(&m2)
fmt.Printf("%d vs %d\n", m1.Frees, m2.Frees)
}
You should see that there was one more free than before.
Related
All my questions are written in code ...
I have no idea about if golang has heap memory and stack memory.
If a func has its own stack memory, does another func can read it ??!
the code:
package main
import (
"fmt"
"math/rand"
"strconv"
"time"
)
type People struct {
name string
age int
}
func receiver(ch chan *People){
for{
// 2. Why the pointer a variable in receiver stack returned by <-ch can reach
// the stack belongs to sender?
fmt.Println((<-ch).name)
}
}
func sender(ch chan *People){
for{
time.Sleep(3*time.Second)
// 1. the People instance is created by sender and
// the pointer may points to the local stack in sender's call stack.
ch <- &People{name: strconv.Itoa(rand.Int())}
}
}
func main(){
ch := make(chan *People, 10)
go receiver(ch)
go sender(ch)
// join...
time.Sleep(time.Hour)
}
Go compiler does escape analysis to check whether an object should be on stack OR heap. The language specs doesn't make a distinction though, which means the compiler can take different course in future than exists today. By course I mean optimization and decision technique than one exists today not the genesis.
Check this stack or heap.
If you build the program with escape analysis flags:
go build -gcflags="-m"
You will notice that it shows:
(<-ch).name escapes to heap
...
"&People literal escapes to heap"
So go compiler by virtue of escape analysis knows before hand what things could be on stack and what should be on heap.
=> so the things that you have doubt on are escaped to be on heap.
Please consider this sample go code below:
package main
/*
#include <stdio.h>
#include <stdlib.h>
*/
import "C"
import (
"fmt"
"runtime"
"unsafe"
)
func main() {
// Convert Go string to C string using C.CString
cString := C.CString("Wold!")
fmt.Printf("C.CString type: %T\n", cString)
//C.free(unsafe.Pointer(cString)) // <-- this works, but I don't want to free it manually..
runtime.SetFinalizer(&cString, func(t *C.Char) {
C.free(unsafe.Pointer(t))
})
}
I am experimenting with cGo, and trying to free cString. When I try to free my variable cString using runtime.SetFinalizer I encounter:
$ go build a.go
# command-line-arguments
./a.go:22:41: could not determine kind of name for C.Char
Please point me the correct direction. Thanks!
When the cgo system is turning your wrapper into something the Go compiler understands, it has to translate each of the C types to a Go type for various purposes. It turns out this doesn't work for your case (this is the error you saw).
That's actually OK, because your code would never have worked the way you wanted in the first place. A runtime finalizer runs when Go's garbage collector is ready to release a Go object that occupies Go memory, but C.Cstring returns a pointer that is not Go memory. In particular, note the following quote from the cgo documentation:
// Go string to C string
// The C string is allocated in the C heap using malloc.
// It is the caller's responsibility to arrange for it to be
// freed, such as by calling C.free (be sure to include stdlib.h
// if C.free is needed).
func C.CString(string) *C.char
Since the returned string is on the "C heap" it will never be finalized by the Go garbage collector. Had your code compiled, it would have just been a no-op.
If you have a Go object whose lifetime parallels that of the C object, you could perhaps use that. Here's a made-up (but working) example:
package main
/*
#include <stdio.h>
#include <stdlib.h>
*/
import "C"
import (
"fmt"
"runtime"
"time"
"unsafe"
)
type S struct {
Foo int
ToFree unsafe.Pointer
}
func main() {
doit()
runtime.GC()
time.Sleep(10 * time.Millisecond) // ugly hack
}
func doit() {
cString := C.CString("Wold!")
fmt.Printf("C.CString type: %T\n", cString)
x := &S{Foo: 1, ToFree: unsafe.Pointer(cString)}
runtime.SetFinalizer(x, func(t *S) {
fmt.Println("freeing C string")
C.free(t.ToFree)
})
}
When the allocated object for x goes out of scope it becomes eligible for GC. The actual GC may never happen, so I forced one with runtime.GC() in main. This triggers the finalizer:
$ ./cfree_example
C.CString type: *main._Ctype_char
freeing C string
The "ugly hack" is in there because if main returns before the finalizer call has finished writing the freeing C string messages, it gets lost. In a real program you would not need this.
Now I'm learning go memory management, when allocating a small object, the allocator will look in the corresponding mspan in mcache, and we also know goroutine has a stack memory, so does that mean the spans in mcache belongs to stack memory(not heap)? What happened in memory if we define a varible in a func like a := 1?
this might be helpful https://go101.org/article/memory-block.html.
Also you can see what exact allocation is happening in memory using
go build -gcflags=-m your_file.go
Only things which used dynamically those only assigned to the heap.
package main
import "fmt"
func f() *int {
var x int
return &x
}
func main() {
fmt.Println(f())
}
go build -gcflags=-m encyptionDecryption.go
./encyptionDecryption.go:5:6: can inline f
./encyptionDecryption.go:11:15: inlining call to f
./encyptionDecryption.go:7:9: &x escapes to heap
./encyptionDecryption.go:6:6: moved to heap: x
./encyptionDecryption.go:11:15: f() escapes to heap
./encyptionDecryption.go:11:15: &x escapes to heap
./encyptionDecryption.go:11:15: moved to heap: x
./encyptionDecryption.go:11:13: main ... argument does not escape
This is escape analysis. You can read more here
package main
import (
"fmt"
"sync/atomic"
"unsafe"
)
func main(){
old := make(map[string]string)
new := make(map[string]string)
new["hello"] = "apple"
fmt.Println("start swap")
atomic.SwapPointer((*unsafe.Pointer)(unsafe.Pointer(&old)), unsafe.Pointer(&new))
fmt.Println("end swap")
// pending here, don't stop
fmt.Println(old)
fmt.Println("end print old")
}
I want a lock-free way to update old map with new map because of the old map will be concurrent read at the most of the time.
if I use a rwlock, there will have serious performance penalty.
so I choose Golang atomic package to implement this, but the line
fmt.Println(old), the program is stuck here, could somebody give some advice.
The function atomic.SwapPointer does not do what you want / need. As the documentation says, this is roughly equivalent to:
old = *addr
*addr = new
return old
(except that the write to *addr and read-back from *addr are done atomically). What you wanted was, I think, the atomic equivalent of:
*old, *new = *new, *old
(and no useful return value). This operation simply does not exist in the sync package. If it did, you could swap the two internal map pointers, but you'd still be treading in dangerous (might-break-in-future-compilers) waters, as multiple commenters noted.
Consider using sync.Map instead. It provides a somewhat-concurrency-safe map with internal (per entry) locking that is optimized for two use cases, as described in the linked package documentation. If your use case is one of these two, it may provide what you need.
Just for illustration (don't do this! it's silly, you could just write *old, *new = *new, *old in swap)... A non-atomic swap of old and new can be achieved using atomic.SwapPointer:
package main
import (
"fmt"
"sync/atomic"
"unsafe"
)
func read(p unsafe.Pointer) unsafe.Pointer {
return *(*unsafe.Pointer)(p)
}
func swap(old *map[string]string, new *map[string]string) {
p := atomic.SwapPointer((*unsafe.Pointer)(unsafe.Pointer(old)), read(unsafe.Pointer(new)))
_ = atomic.SwapPointer((*unsafe.Pointer)(unsafe.Pointer(new)), p)
}
func main() {
old := map[string]string{"old": "old"}
new := map[string]string{"hello": "apple", "new": "new"}
fmt.Println("before: old =", old, "new =", new)
// fmt.Println("before: old:", read(unsafe.Pointer(&old)), "new:", read(unsafe.Pointer(&new)))
swap(&old, &new)
// fmt.Println("after: old:", read(unsafe.Pointer(&old)), "new:", read(unsafe.Pointer(&new)))
fmt.Println("after: old =", old, "new =", new)
}
Uncomment the commented-out lines to see more details. Of course, calling two separate atomic.SwapPointer operations is not atomic: there is a moment when both map variables view the new map, until the second swap makes the old variable view the old map. I think the unsafe.Pointer variable p preserves the old map against GC until we get it stored back into old, but I am not at all sure about this (that's one of the darker corners of Go).
Again: don't do this. If measurement suggests it helps, try sync.Map instead.
Since go 1.17, there is a atomic.Value.Swap method that I believe does what you want.
package main
import (
"fmt"
"sync/atomic"
)
func main() {
var old atomic.Value
old.Store(make(map[string]string))
new := make(map[string]string)
new["hello"] = "apple"
fmt.Println("start swap")
new = old.Swap(new).(map[string]string)
fmt.Println("end swap")
fmt.Println(old)
fmt.Println("end print old")
fmt.Println(new)
}
That being said, it's not clear from your use case that you actually need access to the old values, and if not, you can just use atomic.Value.Store to update the pointer instead. In fact, there is a very similar example in the documentation.
That being said, I agree with the sentiment that you should measure the impact of using RWLock and sync.Map before reaching for atomic.Value.
I have an upload program that I am working on and I am running into an issue. I have n go routines that handle uploading the parts to a big file. Essentially it will split the file into 100MB chunks and upload them concurrently depending on the amount of concurrent processes you specify in the config.
The issue I'm having is when I create a buffer to read the file and upload the make([]byte, 100000000) hangs... but only if it's in a go routine. (I'm using 100000000 to simplify the upload calculations)
Here is an example.
This works: https://play.golang.org/p/tkn8JVir9S
package main
import (
"fmt"
)
func main() {
buffer := make([]byte, 100000000)
fmt.Println(len(buffer))
}
This doesn't: https://play.golang.org/p/H8626OLpqQ
package
main
import (
"fmt"
)
func main() {
go createBuffer()
for {
}
}
func createBuffer() {
buffer := make([]byte, 100000000)
fmt.Println(len(buffer))
}
It just hangs... I'm not sure if there is a memory constraint for a go routine? I tried to research and see what I could find but nothing. Any thoughts would be appreciated.
EDIT: Thanks everyone for the feedback. I will say I didn't explain the real issue very well and will try to provide more of a holistic view next time. I ended up using a channel to block to keep my goroutines ready for new files to process. This is for a DR backup uploading to a 3rd party all that requires large files to be split into 100mb chunks. I guess I should have been more clear as to the nature of my program.
This program hangs because there is an infinite loop in your code. Try running the code just like this to prove it to yourself. The goroutine is not what is causing the hanging.
func main() {
for {
}
}
If you just want to see fmt.Println(..) print, then I'd recommend having a time.Sleep call or similar.
If you would like to wait for a bunch of goroutines to complete, then I'd recommend this excellent answer to that exact question.
Package runtime
import "runtime"
func Gosched
func Gosched()
Gosched yields the processor, allowing other goroutines to run. It
does not suspend the current goroutine, so execution resumes
automatically.
When you do something strange (for {} and 100MB), you get strange results. Do something reasonable. For example,
package main
import (
"fmt"
"runtime"
)
func main() {
go createBuffer()
for {
runtime.Gosched()
}
}
func createBuffer() {
buffer := make([]byte, 100000000)
fmt.Println(len(buffer))
}
Output:
100000000
^Csignal: interrupt