I have a map and want to manually further sharding, the simplified code is
const (
dictShardNum = 16
dictShardSize = 1 << 28
)
type shard struct {
mu sync.Mutex
m map[int64]uint32
}
type dict struct {
shards []shard
}
func newDict() *dict {
shards := make([]shard, 0, dictShardNum)
for i := 0; i < dictShardNum; i++ {
shards = append(shards, shard{ m: make(map[int64]uint32) })
}
return &dict{ shards }
}
func (d *dict) insert(n int64) uint32 {
shardNum := int(n % dictShardNum)
shard := d.shards[shardNum]
shard.mu.Lock()
defer shard.mu.Unlock()
tempID, ok := shard.m[n]
if !ok {
tempID = uint32(len(shard.m) + shardNum*dictShardSize)
shard.m[n] = tempID // fatal error: concurrent map writes
}
return tempID
}
When running I got fatal error: concurrent map writes at that line, but I did lock the mutex, not sure what's wrong with my code
Package sync
import "sync"
type Mutex
A Mutex is a mutual exclusion lock. The zero value for a Mutex is an
unlocked mutex.
A Mutex must not be copied after first use.
Your code doesn't compile!
Playground: https://play.golang.org/p/6AwS0vOZfeP
25:18: undefined: n
30:24: undefined: n
33:11: undefined: n
If I change v int64 to n int64:
A Mutex must not be copied after first use.
$ go vet mutex.go
./mutex.go:26:11: assignment copies lock value to shard: command-line-arguments.shard contains sync.Mutex
$
Playground: https://play.golang.org/p/jExE-m11ny5
package main
import (
"sync"
)
const (
dictShardNum = 16
dictShardSize = 1 << 28
)
type shard struct {
mu sync.Mutex
m map[int64]uint32
}
type dict struct {
shards []shard
}
/// a newDict function
func (d *dict) insert(n int64) uint32 {
shardNum := int(n % dictShardNum)
shard := d.shards[shardNum]
shard.mu.Lock()
defer shard.mu.Unlock()
tempID, ok := shard.m[n]
if !ok {
tempID = uint32(len(shard.m) + shardNum*dictShardSize)
shard.m[n] = tempID // fatal error: concurrent map writes
}
return tempID
}
func main() {}
Command vet
Vet examines Go source code and reports suspicious constructs
Copying locks
Flag: -copylocks
Locks that are erroneously passed by value.
I think the answer is related to copy mutex values.
The dict should be
type dict struct {
shards []*shard
}
All shards are accessed via pointer then it won't have any issue.
Related
I'm creating a structure where a developer can store a reference to something and retrieve it when needed using a reference key, but not delete the reference.
How to create a unique key for a map that is generated at the point of insertion?
So far I'm using a not exported pointer to an empty *struct{} , but wondering if there there is a better way.
package main
import "fmt"
type referenceKey **struct{}
type reference[K referenceKey, R any] struct {
s map[K]R
}
func (ref *reference[K, R]) Set(reference R) *K {
if ref.s == nil {
ref.s = make(map[K]R)
}
key := new(struct{})
refKey := K(&key)
ref.s[refKey] = reference
return &refKey
}
func (ref *reference[K, R]) Get(key *K) R {
return ref.s[*key]
}
func main() {
ref := &reference[referenceKey, int]{}
key1 := ref.Set(77)
key2 := ref.Set(15345351)
fmt.Println(ref.Get(key2))
fmt.Println(ref.Get(key1))
}
https://go.dev/play/p/SF6S5BNlP7N
EDIT:
With int I have to keep a reference to it and increment it. To reduce code I tried to use the pointer to a instance of a empty struct.
Basically I need to create a unique key that can't be created outside internal package nor can Get method be called with an invalid key.
I've also fixed the code to now allow anyone to create a new invalid key.
Based on the comments I would make it with a counter and protect the store with a mutex for concurrent use:
package main
import (
"fmt"
"sync"
)
type reference[T any] struct {
mutex sync.RWMutex
store map[uint64]T
counter uint64
}
func (r *reference[T]) Set(item T) uint64 {
r.mutex.Lock()
defer r.mutex.Unlock()
if r.store == nil {
r.store = make(map[uint64]T)
}
r.counter++
r.store[r.counter] = item
return r.counter
}
func (r *reference[T]) Get(key uint64) T {
r.mutex.RLock()
defer r.mutex.RUnlock()
return r.store[key]
}
func main() {
ref := reference[int]{}
key1 := ref.Set(77)
key2 := ref.Set(15345351)
fmt.Println(ref.Get(key2))
fmt.Println(ref.Get(key1))
}
But if in the future you need to be secure, and the keys should be random, then change the map's key to string, remove the counter and use uuid.NewString() for the new key value.
This question already has an answer here:
golang how can I use struct name as map key
(1 answer)
Closed 9 months ago.
We have a following function:
func (h *Handler) Handle(message interface{}) error {
//here there is a switch for different messages
switch m := message.(type) {
}
}
This signature is given and can't be changed. There are around 20 different message types the handler processes.
Now, there are some of these messages (around 4) which need special post-processing. In a different package.
Thus, I am thinking to do this like this:
func (h *Handler) Handle(message interface{}) error {
//here there is a switch for different messages
switch m := message.(type) {
}
//only post-process if original message processing succeeds
postProcessorPkg.Process(message)
}
Now, in the Process function, I want to quickly lookup if the message type is indeed of the ones we need postprocessing for. I don't want to do a switch again here. There are many handlers, in different packages, with varying amount of message types, and it should be generic.
So I was thinking of registering the message type in the postprocessor and then just do a lookup:
func (p *Postprocessor) Register(msgtype interface{}) {
registeredTypes[msgtype] = msgtype
}
and then
func (p *Postprocessor) Process(msgtype interface{}) error {
if ok := registeredTypes[msgtype]; !ok {
return errors.New("Unsupported message type")
}
prop := GetProp(registeredTypes[msgtype])
doSmthWithProp(prop)
}
This will all not work now because I can only "register" instances of the message, not the message type itself, as far as I know. Thus the map would only match a specific instance of a message, not its type, which is what I need.
So I guess this needs redesign. I can completely ditch the registering and the map lookup, but
I can't change the Handle function to a specific type (signature will need to remain message interface{}
I would like to avoid to have to use reflect, just because I will have a hard time defending such a solution with some colleagues.
As there is no possibility to set a type as the map key, I finally decided to implement the following solution, which is based on #Chrono Kitsune 's solution:
type Postprocess interface {
NeedsPostprocess() bool
}
type MsgWithPostProcess struct {}
func (p *MsgWithPostProcess) NeedsPostprocess() bool {
return true
}
type Msg1 struct {
MsgWithPostProcess
//other stuff
}
type Msg2 struct {
MsgWithPostProcess
//other stuff
}
type Msg3 struct {
//no postprocessing needed
}
func (p *Postprocessor) Process(msgtype interface{}) error {
if _, ok := msgtype.(Postprocess); ok {
//do postprocessing
}
}
As of my simple test I did, only Msg1 and Msg2 will be postprocessed, but not Msg3, which is what I wanted.
This question was the first hit I found on Google but the title is somewhat misleading. So I'll leave this here to add some food for thought with the title of the question in mind.
First, the issue with maps is that its key must be a comparable value. This is why for example a slice cannot be used is a map key. A slice is not comparable and is therefore not allowed. You can use an array (fixed sized slice) but not a slice for the same reason.
Second, you have in the reflect.TypeOf(...).String()a way to get a canonical string representation for types. Though it is not unambiguous unless you include the package path, as you can see here.
package main
import (
"fmt"
s2 "go/scanner"
"reflect"
s1 "text/scanner"
)
type X struct{}
func main() {
fmt.Println(reflect.TypeOf(1).String())
fmt.Println(reflect.TypeOf(X{}).String())
fmt.Println(reflect.TypeOf(&X{}).String())
fmt.Println(reflect.TypeOf(s1.Scanner{}).String())
fmt.Println(reflect.TypeOf(s2.Scanner{}).String())
fmt.Println(reflect.TypeOf(s1.Scanner{}).PkgPath(), reflect.TypeOf(s1.Scanner{}).String())
fmt.Println(reflect.TypeOf(s2.Scanner{}).PkgPath(), reflect.TypeOf(s2.Scanner{}).String())
}
int
main.X
*main.X
scanner.Scanner
scanner.Scanner
text/scanner scanner.Scanner
go/scanner scanner.Scanner
https://play.golang.org/p/NLODZNdik6r
With this information, you can (if you feel so inclined) create a map which let's go from a reflect.Type to a key and back again, like this.
package main
import (
"fmt"
s2 "go/scanner"
"reflect"
s1 "text/scanner"
)
type TypeMap struct {
m []reflect.Type
}
func (m *TypeMap) Get(t reflect.Type) int {
for i, x := range m.m {
if x == t {
return i
}
}
m.m = append(m.m, t)
return len(m.m) - 1
}
func (m *TypeMap) Reverse(t int) reflect.Type {
return m.m[t]
}
type X struct{}
func main() {
var m TypeMap
fmt.Println(m.Get(reflect.TypeOf(1)))
fmt.Println(m.Reverse(0))
fmt.Println(m.Get(reflect.TypeOf(1)))
fmt.Println(m.Reverse(0))
fmt.Println(m.Get(reflect.TypeOf(1)))
fmt.Println(m.Reverse(0))
fmt.Println(m.Get(reflect.TypeOf(X{})))
fmt.Println(m.Reverse(1))
fmt.Println(m.Get(reflect.TypeOf(&X{})))
fmt.Println(m.Reverse(2))
fmt.Println(m.Get(reflect.TypeOf(s1.Scanner{})))
fmt.Println(m.Reverse(3).PkgPath(), m.Reverse(3))
fmt.Println(m.Get(reflect.TypeOf(s2.Scanner{})))
fmt.Println(m.Reverse(4).PkgPath(), m.Reverse(4))
}
0
int
0
int
0
int
1
main.X
2
*main.X
3
text/scanner scanner.Scanner
4
go/scanner scanner.Scanner
In the above case I'm assuming that N is small. Also note the use of the identity of reflect.TypeOf, it will return the same pointer for the same type on subsequent calls.
If N is not small, you may want to do something a bit more complex.
package main
import (
"fmt"
s2 "go/scanner"
"reflect"
s1 "text/scanner"
)
type PkgPathNum struct {
PkgPath string
Num int
}
type TypeMap struct {
m map[string][]PkgPathNum
r []reflect.Type
}
func (m *TypeMap) Get(t reflect.Type) int {
k := t.String()
xs := m.m[k]
pkgPath := t.PkgPath()
for _, x := range xs {
if x.PkgPath == pkgPath {
return x.Num
}
}
n := len(m.r)
m.r = append(m.r, t)
xs = append(xs, PkgPathNum{pkgPath, n})
if m.m == nil {
m.m = make(map[string][]PkgPathNum)
}
m.m[k] = xs
return n
}
func (m *TypeMap) Reverse(t int) reflect.Type {
return m.r[t]
}
type X struct{}
func main() {
var m TypeMap
fmt.Println(m.Get(reflect.TypeOf(1)))
fmt.Println(m.Reverse(0))
fmt.Println(m.Get(reflect.TypeOf(X{})))
fmt.Println(m.Reverse(1))
fmt.Println(m.Get(reflect.TypeOf(&X{})))
fmt.Println(m.Reverse(2))
fmt.Println(m.Get(reflect.TypeOf(s1.Scanner{})))
fmt.Println(m.Reverse(3).PkgPath(), m.Reverse(3))
fmt.Println(m.Get(reflect.TypeOf(s2.Scanner{})))
fmt.Println(m.Reverse(4).PkgPath(), m.Reverse(4))
}
0
int
1
main.X
2
*main.X
3
text/scanner scanner.Scanner
4
go/scanner scanner.Scanner
https://play.golang.org/p/2fiMZ8qCQtY
Note the subtitles of pointer to type, that, X and *X actually are different types.
I have written following sample program using sync.RWMutex.
package main
import (
"fmt"
"sync"
"time"
)
// SessionData : capture session id and cc-request-number
type SessionData struct {
id string
reqNo string
}
// SessionCache : cache for the SessionData
type SessionCache struct {
sess map[SessionData]bool
sync.RWMutex
}
// InitSessionCache : Init for SessionCache
func InitSessionCache() SessionCache {
return SessionCache{sess: make(map[SessionData]bool)}
}
// Read : read value from session cache
func (s *SessionCache) Read(sd SessionData) bool {
s.RLock()
defer s.RUnlock()
_, found := s.sess[sd]
return found
}
func (s *SessionCache) Write(sd SessionData) {
s.Lock()
defer s.Unlock()
fmt.Println("Entry not found for ", sd.id, sd.reqNo, "Creating the entry now")
s.sess[sd] = true
}
func (s *SessionCache) chkDuplicate(sessionID string, Reqno string) bool {
sd := SessionData{
id: sessionID,
reqNo: Reqno,
}
found := s.Read(sd)
if !found {
s.Write(sd)
return found
}
return found
}
func main() {
mySessionData := InitSessionCache()
for i := 0; i < 10; i++ {
go mySessionData.chkDuplicate("session1", "1")
go mySessionData.chkDuplicate("session1", "1")
go mySessionData.chkDuplicate("session1", "2")
go mySessionData.chkDuplicate("session1", "2")
go mySessionData.chkDuplicate("session1", "4")
go mySessionData.chkDuplicate("session1", "2")
}
time.Sleep(300)
fmt.Println(mySessionData)
}
when I run this program in playground https://play.golang.org/p/g93UtVxZ2dl
I see that it is working correctly as the write happens only 3 times for the unique values.
Entry not found for session1 1 Creating the entry now
Entry not found for session1 2 Creating the entry now
Entry not found for session1 4 Creating the entry now
{map[{session1 1}:true {session1 2}:true {session1 4}:true] {{0 0} 0 0 0 0}}
however when I run the same program from my windows 10 machine (on VS Code) I see following output.
Entry not found for session1 1 Creating the entry now
Entry not found for session1 2 Creating the entry now
Entry not found for session1 2 Creating the entry now
Entry not found for session1 2 Creating the entry now
Entry not found for session1 4 Creating the entry now
{map[{session1 1}:true {session1 2}:true {session1 4}:true] {{0 0} 0 0 0 0}}
Am I doing something wrong?
Why does this behaves differently on my Machine and Playground?
There is no syncronisation between the call to Read and Write. All your goroutines are running concurrently, imagine if they all run up to this line and then yield to another goroutine:
found := s.Read(sd)
They will all return false because none of the goroutines have moved past this point. Now they all move on to the next line and believe that found == false, so all perform the s.Write(sd).
You need to perform the Read and Write without unlocking. Maybe something like:
func (s *SessionCache) TryWrite(sd SessionData) err {
s.Lock()
defer s.Unlock()
if _, found := s.sess[sd]; found {
return fmt.Errorf("Entry already exists")
}
s.sess[sd] = true
}
Is there a way to obtain the total number of finalizers registered using runtime.SetFinalizer and which have not yet run?
We are considering adding a struct with a registered finalizer to some of our products to release memory allocated using malloc, and the object could potentially have a relatively high allocation rate. It would be nice if we could monitor the number of finalizers, to make sure that they do not pile up and trigger out-of-memory errors (like they tend to with other garbage collectors).
(I'm aware that explicit deallocation would avoid this problem, but we cannot change the existing code, which does not call a Close function or something like that.)
You can keep keep a count of these objects by incrementing and decrementing a unexported package variable when a new object is created and finalized, respectively.
For example:
package main
import (
"fmt"
"runtime"
"sync/atomic"
)
var totalObjects int32
func TotalObjects() int32 {
return atomic.LoadInt32(&totalObjects)
}
type Object struct {
p uintptr // C allocated pointer
}
func NewObject() *Object {
o := &Object{
}
// TODO: perform other initializations
atomic.AddInt32(&totalObjects, 1)
runtime.SetFinalizer(o, (*Object).finalizer)
return o
}
func (o *Object) finalizer() {
atomic.AddInt32(&totalObjects, -1)
// TODO: perform finalizations
}
func main() {
fmt.Println("Total objects:", TotalObjects())
for i := 0; i < 100; i++ {
_ = NewObject()
runtime.GC()
}
fmt.Println("Total objects:", TotalObjects())
}
https://play.golang.org/p/n35QABBIcj
It's possible to make a wrapper on runtime.SetFinalizer which does the counting for you. Of course, it's a question of using it everywhere where you use SetFinalizer.
In case this is problematic, you can also modify SetFinalizer source code directly, but that requires a modified Go compiler.
Atomic integers are used as SetFinalizer may be called on different threads, and otherwise a counter may not be accurate as without those a race condition could possibly occur. Golang guarantees that finalizers are called from a single goroutine, so it's not needed for inner function.
https://play.golang.org/p/KKCH2UwTFYw
package main
import (
"fmt"
"reflect"
"runtime"
"sync/atomic"
)
var finalizersCreated int64
var finalizersRan int64
func SetFinalizer(obj interface{}, finalizer interface{}) {
finType := reflect.TypeOf(finalizer)
funcType := reflect.FuncOf([]reflect.Type{finType.In(0)}, nil, false)
f := reflect.MakeFunc(funcType, func(args []reflect.Value) []reflect.Value {
finalizersRan++
return reflect.ValueOf(finalizer).Call([]reflect.Value{args[0]})
})
runtime.SetFinalizer(obj, f.Interface())
atomic.AddInt64(&finalizersCreated, 1)
}
func main() {
v := "a"
SetFinalizer(&v, func(a *string) {
fmt.Println("Finalizer ran")
})
fmt.Println(finalizersRan, finalizersCreated)
runtime.GC()
fmt.Println(finalizersRan, finalizersCreated)
}
I want to write some code like this:
var myValue interface{}
func GetMyValue() interface{} {
return atomic.Load(myValue)
}
func StoreMyValue(newValue interface{}) {
atomic.Store(myValue, newValue)
}
It seems like that i can use LoadUintptr(addr *uintptr) (val uintptr) and StoreUintptr(addr *uintptr, val uintptr) in atomic package to achive this,but i do not know how to convert between uintptr,unsafe.Pointer and interface{}.
If i do it like this:
var V interface{}
func F(v interface{}) {
p := unsafe.Pointer(&V)
atomic.StorePointer(&p, unsafe.Pointer(&v))
}
func main() {
V = 1
F(2)
fmt.Println(V)
}
the V will always be 1
If I'm not mistaken you want atomic Value. You can store and fetch values atomically with it (signatures are interface{} but you should put same type into it). It does some unsafe pointer stuff under the hood like what you wanted to do.
Sample from docs:
var config Value // holds current server configuration
// Create initial config value and store into config.
config.Store(loadConfig())
go func() {
// Reload config every 10 seconds
// and update config value with the new version.
for {
time.Sleep(10 * time.Second)
config.Store(loadConfig())
}
}()
// Create worker goroutines that handle incoming requests
// using the latest config value.
for i := 0; i < 10; i++ {
go func() {
for r := range requests() {
c := config.Load()
// Handle request r using config c.
_, _ = r, c
}
}()
}
Here's a way to use atomic.StorePointer and atomic.LoadPointer (based on your example):
package main
import (
"fmt"
"sync/atomic"
"unsafe"
)
var addr unsafe.Pointer
func GetMyValue() *interface{} {
return (*interface{})(atomic.LoadPointer(&addr))
}
func StoreMyValue(newValue *interface{}) {
atomic.StorePointer(&addr, unsafe.Pointer(newValue))
}
func main() {
var i interface{}
i = 1
StoreMyValue(&i)
fmt.Println("before:", *GetMyValue())
i = 2
StoreMyValue(&i)
fmt.Println("after", *GetMyValue())
}
Playground link
Note that this will not make your object thread-safe. Only the pointer is stored/loaded atomically. Also, I would avoid using interface{} and prefer concrete types whenever possible.
As an alternative to using 'any' (interface{}), Go 1.19 (Q3 2022) comes with new types in the sync/atomic package that make it easier to use atomic values, such as atomic.Int64 and atomic.Pointer[T].
That would be easier than using atomic.StorePointer.
This comes from issue 50860 "sync/atomic: add typed atomic values".
And CL 381317
Pointer[T] also avoids conversions using unsafe.Pointer at call sites.
You cannot do this.
You will have to protect the store/load with a mutex.
The internal representation of an interface is not specified by the language and might (is) to large to be handled by package atomic.