Is there a way to obtain the total number of finalizers registered using runtime.SetFinalizer and which have not yet run?
We are considering adding a struct with a registered finalizer to some of our products to release memory allocated using malloc, and the object could potentially have a relatively high allocation rate. It would be nice if we could monitor the number of finalizers, to make sure that they do not pile up and trigger out-of-memory errors (like they tend to with other garbage collectors).
(I'm aware that explicit deallocation would avoid this problem, but we cannot change the existing code, which does not call a Close function or something like that.)
You can keep keep a count of these objects by incrementing and decrementing a unexported package variable when a new object is created and finalized, respectively.
For example:
package main
import (
"fmt"
"runtime"
"sync/atomic"
)
var totalObjects int32
func TotalObjects() int32 {
return atomic.LoadInt32(&totalObjects)
}
type Object struct {
p uintptr // C allocated pointer
}
func NewObject() *Object {
o := &Object{
}
// TODO: perform other initializations
atomic.AddInt32(&totalObjects, 1)
runtime.SetFinalizer(o, (*Object).finalizer)
return o
}
func (o *Object) finalizer() {
atomic.AddInt32(&totalObjects, -1)
// TODO: perform finalizations
}
func main() {
fmt.Println("Total objects:", TotalObjects())
for i := 0; i < 100; i++ {
_ = NewObject()
runtime.GC()
}
fmt.Println("Total objects:", TotalObjects())
}
https://play.golang.org/p/n35QABBIcj
It's possible to make a wrapper on runtime.SetFinalizer which does the counting for you. Of course, it's a question of using it everywhere where you use SetFinalizer.
In case this is problematic, you can also modify SetFinalizer source code directly, but that requires a modified Go compiler.
Atomic integers are used as SetFinalizer may be called on different threads, and otherwise a counter may not be accurate as without those a race condition could possibly occur. Golang guarantees that finalizers are called from a single goroutine, so it's not needed for inner function.
https://play.golang.org/p/KKCH2UwTFYw
package main
import (
"fmt"
"reflect"
"runtime"
"sync/atomic"
)
var finalizersCreated int64
var finalizersRan int64
func SetFinalizer(obj interface{}, finalizer interface{}) {
finType := reflect.TypeOf(finalizer)
funcType := reflect.FuncOf([]reflect.Type{finType.In(0)}, nil, false)
f := reflect.MakeFunc(funcType, func(args []reflect.Value) []reflect.Value {
finalizersRan++
return reflect.ValueOf(finalizer).Call([]reflect.Value{args[0]})
})
runtime.SetFinalizer(obj, f.Interface())
atomic.AddInt64(&finalizersCreated, 1)
}
func main() {
v := "a"
SetFinalizer(&v, func(a *string) {
fmt.Println("Finalizer ran")
})
fmt.Println(finalizersRan, finalizersCreated)
runtime.GC()
fmt.Println(finalizersRan, finalizersCreated)
}
Related
I have a struct in Go which contains a mutex, and I want to ensure that that mutex is never nil. To that end, I have implemented a GetMutex() function, which checks if the mutex is nil, and if it is, then assigns it a value.
My question is: is the following code thread safe? If not, what would be an idiomatic way to ensure that mux is always initialized? The only thing I can think of is to have a global mutex in this package which is used within my GetMutex() function, but perhaps there is a different approach.
package main
import (
"sync"
)
type Counter struct {
mux *sync.Mutex
counter int
}
// Is this thread safe?
func (c *Counter) GetMux() *sync.Mutex {
if c.mux == nil {
c.mux = &sync.Mutex{}
}
return c.mux
}
func (c *Counter) Inc() {
c.GetMux().Lock()
c.counter++
c.GetMux().Unlock()
}
func main() {
c := &Counter{}
c.Inc()
}
No, it's not safe if Counter.GetMux() is called from multiple goroutines concurrently: GetMux() both reads and writes the Counter.mux field.
The general way is to use a "constructor" like function that takes care of the initialization, like this:
func NewCounter() *Counter {
return &Counter{
mux: &sync.Mutex{},
}
}
And of course always create counters with this NewCounter().
Another–limited–way would be to use a non-pointer mutex value:
type Counter struct {
mux sync.Mutex
counter int
}
So when you have a Counter struct value, it–by design–includes a mutex. But if you do this, then Counter should always be used as a pointer, and Counter struct values must not be copied (else the mutex field would also be copied, but as package doc of sync states: "Values containing the types defined in this package should not be copied.").
The obvious advantage of this is that the zero value of Counter is a valid and ready counter (something you should aim for with your custom types), and no constructor function is needed.
I want to write some code like this:
var myValue interface{}
func GetMyValue() interface{} {
return atomic.Load(myValue)
}
func StoreMyValue(newValue interface{}) {
atomic.Store(myValue, newValue)
}
It seems like that i can use LoadUintptr(addr *uintptr) (val uintptr) and StoreUintptr(addr *uintptr, val uintptr) in atomic package to achive this,but i do not know how to convert between uintptr,unsafe.Pointer and interface{}.
If i do it like this:
var V interface{}
func F(v interface{}) {
p := unsafe.Pointer(&V)
atomic.StorePointer(&p, unsafe.Pointer(&v))
}
func main() {
V = 1
F(2)
fmt.Println(V)
}
the V will always be 1
If I'm not mistaken you want atomic Value. You can store and fetch values atomically with it (signatures are interface{} but you should put same type into it). It does some unsafe pointer stuff under the hood like what you wanted to do.
Sample from docs:
var config Value // holds current server configuration
// Create initial config value and store into config.
config.Store(loadConfig())
go func() {
// Reload config every 10 seconds
// and update config value with the new version.
for {
time.Sleep(10 * time.Second)
config.Store(loadConfig())
}
}()
// Create worker goroutines that handle incoming requests
// using the latest config value.
for i := 0; i < 10; i++ {
go func() {
for r := range requests() {
c := config.Load()
// Handle request r using config c.
_, _ = r, c
}
}()
}
Here's a way to use atomic.StorePointer and atomic.LoadPointer (based on your example):
package main
import (
"fmt"
"sync/atomic"
"unsafe"
)
var addr unsafe.Pointer
func GetMyValue() *interface{} {
return (*interface{})(atomic.LoadPointer(&addr))
}
func StoreMyValue(newValue *interface{}) {
atomic.StorePointer(&addr, unsafe.Pointer(newValue))
}
func main() {
var i interface{}
i = 1
StoreMyValue(&i)
fmt.Println("before:", *GetMyValue())
i = 2
StoreMyValue(&i)
fmt.Println("after", *GetMyValue())
}
Playground link
Note that this will not make your object thread-safe. Only the pointer is stored/loaded atomically. Also, I would avoid using interface{} and prefer concrete types whenever possible.
As an alternative to using 'any' (interface{}), Go 1.19 (Q3 2022) comes with new types in the sync/atomic package that make it easier to use atomic values, such as atomic.Int64 and atomic.Pointer[T].
That would be easier than using atomic.StorePointer.
This comes from issue 50860 "sync/atomic: add typed atomic values".
And CL 381317
Pointer[T] also avoids conversions using unsafe.Pointer at call sites.
You cannot do this.
You will have to protect the store/load with a mutex.
The internal representation of an interface is not specified by the language and might (is) to large to be handled by package atomic.
I'm trying to get a better understanding of go. I created a little exercise for myself: pass a pointer slice to a function and modify it.
This is what I came up with:
package main
import (
"fmt"
"unsafe"
)
func main() {
var data *[]byte
fillData(data)
fmt.Println((*data)[0:5])
}
func fillData(data *[]byte) {
b := []byte("hello")
fmt.Println(b[0:5])
fmt.Println(string(b[0:5]))
data = (*[]byte)(unsafe.Pointer(&b[0]))
}
But it gives an invalid memory address or nil pointer dereference error. I know I wouldn't use something like this in real code but I was just curious how to pass a slice and modify it without returning it.
https://play.golang.org/p/_K5ltKKKNV
When you try to update data in fillData, you make two errors. First, you update the pointer rather than what it's pointed to. Second, data is a nil pointer, so writing through that pointer will cause a nil pointer error.
Here's one possible way to write the code. data starts as a zero'ed slice, and gets updated inside fillData. This will copy the slice information (len, cap, and pointer to array) from b to *data which means that data will share information with b (importantly, including sharing the underlying array).
package main
import "fmt"
func main() {
var data []byte
fillData(&data)
fmt.Println(data, data[0:5])
}
func fillData(data *[]byte) {
b := []byte("hello")
*data = b[0:1]
}
Another way would be to have data being a pointer, and updating it. Then you have to pass a double pointer into fillData. That would look like this:
package main
import "fmt"
func main() {
var data *[]byte
fillData(&data)
fmt.Println((*data)[0:5])
}
func fillData(data **[]byte) {
b := []byte("hello")
*data = &b
}
Finally, the best way to write this code isn't to use pointers at all, and just return the slice. Unlike C or C++, it's rarely needed to use "output" parameters to functions. That's because go allows multiple return values.
package main
import "fmt"
func main() {
data := getData()
fmt.Println(data, data[0:5])
}
func getData() []byte {
return []byte("hello")[:1]
}
I'm trying to assign a value to a field, but my program panics with runtime error: invalid memory address or nil pointer dereference.
package main
type Node struct {
Value int
}
func (n *Node) SetValue(value int) {
n.Value = value
}
func main() {
var n *Node
n.SetValue(1)
}
This is reasonable since variable is nil.
But I've fount some Go internal structs are allowed to do this, e.g. bytes.Buffer
package main
import "bytes"
import "io"
import "os"
func main() {
var b bytes.Buffer
b.Write([]byte("Hello world"))
io.Copy(os.Stdout, &b)
}
Here is the `bytes.Buffer source code
func (b *Buffer) Write(p []byte) (n int, err error) {
b.lastRead = opInvalid
m := b.grow(len(p))
return copy(b.buf[m:], p), nil
}
Is it the thing only builtin structs can do or it's possible to accomplish this in my code?
EDIT
Here is the working example. Thanks #twotwotwo for suggestion.
package main
import "fmt"
type Node struct {
Value int
}
func (n *Node) SetValue(value int) {
n.Value = value
}
func main() {
var n Node
n.SetValue(1)
fmt.Println(n.Value)
}
The crucial thing is var b bytes.Buffer doesn't get you a nil pointer, it gets you a bytes.Buffer object with all its fields initialized with their zero values (in machine terms, with zero bytes). The spec says the zero value is "false for booleans, 0 for integers, 0.0 for floats, "" for strings, and nil for pointers, functions, interfaces, slices, channels, and maps"; follow that link for more detail.
It is possible to make your own structs whose zero values work and the Go team encourages it. struct Position { x, y int } is an easy example and Effective Go gives a more realistic one. But note that that doesn't make the nil pointer work; you would still need new(Node) or var n Node to allocate the zero Node. Same for bytes.Buffer.
Another common use of zero values: wherever your users create structs of your type directly (as folks do with, say, http.Server), the zero value is the default for any fields they don't specify. It's the default in a lot of other places: what you get for a not-found map key, if you receive from a closed channel, and probably others.
I know go routine can have a few blocking actions, wonder if a goroutine can call a user-defined blocking function like a regular function. A user-defined blocking function has a few steps like, step1, step2.
In another word, I would like to find out whether we can have nested blocking calls in a go routine.
UPDATE:
Original intention was to find the stack size used by goroutine, especially with nested blocking calls. Sorry for the confusion. Thanks to the answer and comments, I created the following function that has 100,000 goroutines, it took 782MB of virtual memory and 416MB of Resident memory on my Ubuntu desktop. It evens out to be 78KB of memory for each go routine stack. Is this a correct statement?
package main
import (
"fmt"
"time"
)
func f(a int) {
x := f1(a);
f2(x);
}
func f1(a int) int {
r := step("1a", a);
r = step("1b", r);
return 1000 * a;
}
func f2(a int) {
r := step("2a", a);
r = step("2b", r);
}
func step(a string, b int) int{
fmt.Printf("%s %d\n", a, b);
time.Sleep(1000 * time.Second)
return 10 * b;
}
func main() {
for i := 0; i < 100000; i++ {
go f(i);
}
//go f(20);
time.Sleep(1000 * time.Second)
}
I believe you're right, though I'm unsure of the relationship between "virtual" and "resident" memory it's possible there's some overlap.
Some things to consider: you're running 100,000 it appears, not 10,000.
The stack itself might contain things like the strings used for the printfs, method parameters, etc.
As of go 1.2 the default stack size (per go routine) is 8KB which may explain some of it.
As of go 1.3 it also uses an exponentially increasing stack size, but I doubt that's the problem you're running into.
Short answer yes.
A goroutine is a "lightweight thread", that means it can do stuff independently from other code in your program. It's almost as if you started a new program, but you can communicate with your other code using the constructs golang provides (channels, locks, etc.).
P.S. Once the main function ends, all goroutines are killed (that's why you need the time.Sleep() in the example)
Here's the quick example (won't run in the golang playground because of their constraints):
package main
import (
"fmt"
"time"
)
func saySomething(a, b func()){
a()
b()
}
func foo() {
fmt.Println("foo")
}
func bar() {
fmt.Println("bar")
}
func talkForAWhile() {
for {
saySomething(foo, bar)
}
}
func main() {
go talkForAWhile()
time.Sleep(1 * time.Second)
}