How to eliminate Data Race even with Mutex Locks around pointer variable - go

Some starter code goes here,
func (chm *ConcurrentHashMap) NFetchWorker() {
for {
key := <-NFetchWorkerPipe
chm.mu.RLock()
data := chm.data[string(key)]
chm.mu.RUnlock()
if data.IsUsingNFetch {
chm.mu.Lock()
*(chm.data[string(key)].NFetch)--
chm.mu.Unlock()
}
}
}
go NFetchWorker()
Struct ConcurrentHashMap looks like this,
type ConcurrentHashMap struct {
data map[string]DataBlock
mu sync.RWMutex
}
Struct DataBlock looks like this,
type DataBlock struct {
...
NFetch *int32
IsUsingNFetch bool
...
}
Now when I try to run tests with race flag enabled. I get,
Write at 0x00c00012e310 by goroutine 8:
(*ConcurrentHashMap).NFetchWorker()
The line numbers point to this line.
*(chm.data[string(key)].NFetch)--
However I don't face this issue when I do not use pointer in DataBlock.NFetch.
But if I do that I lose the ability to make changes to NFetch directly from map without reassigning a whole new struct object to that hash in map, which would be relatively computationally expensive. I want to change the value of NFetch without reassigning the whole struct again for one small change while being free from DATA RACE. Any solutions??
Fairly New to Golang, I could be doing something really stupid here, feel free to point it out.
UPDATE: Adding Read and Write Op function
func (chm *ConcurrentHashMap) Get(key Key) Value {
chm.mu.RLock()
fetch, ok := chm.data[string(key)]
chm.mu.RUnlock()
if ok {
if CheckValueValidity(&fetch) {
NFetchWorkerPipe <- key
return fetch.Value
} else {
chm.mu.Lock()
delete(chm.data, string(key))
chm.mu.Unlock()
}
}
return constants.NIL
}
Write Op
func (chm *ConcurrentHashMap) Insert(key Key, value Value, options Options) {
...
chm.mu.Lock()
chm.data[string(key)] = Block{
Value: value,
NFetch: nFetchOld,
Expiry: time.Now().Add(delay),
IsUsingNFetch: foundNFetch,
IsUsingExpiry: foundTTL,
mu: mutex,
}
chm.mu.Unlock()
}

The problem is in the CheckValueValidity where you are accessing NFetch without locking it. Pointers should never be accessed without locking them.
func CheckValueValidity(value *Block) bool {
if value.IsUsingExpiry && !value.Expiry.After(time.Now()) {
return false
}
if value.IsUsingNFetch && *(value.NFetch) <= 0 {
return false
}
return true
}
This code should work
func (chm *ConcurrentHashMap) Get(key Key) Value {
chm.mu.RLock()
fetch, ok := chm.data[string(key)]
isValid := CheckValueValidity(&fetch)
chm.mu.RUnlock()
if ok {
if isValid {
NFetchWorkerPipe <- key
return fetch.Value
} else {
chm.mu.Lock()
delete(chm.data, string(key))
chm.mu.Unlock()
}
}
return constants.NIL
}

Related

Separate lock for each map key

I have a map (entities) where the key is a string and the value is a struct. Building that struct is an expensive operation because it has to go to the database to load the data.
type entityStateManagers struct {
entities map[string]*entityStateManager
mainLock *sync.Mutex
locks map[string]*sync.Mutex
}
The function below returns that struct for the provided key. If the struct is not in memory it will load it from the database.
I didn't want to have only one lock for everything because that would completely block access to the map while expensiveLoad(entityId) is running.
I tried having a separate struct (locks) with one lock per map key. With that, only access to entity_123 will have to wait for expensiveLoad(entity_123), for example.
I still have to do more testing, but it seems to be working.
func (handler entityStateManagers) getStateManager(entityId string) (*entityStateManager, error) {
handler.mainLock.Lock()
if handler.locks[entityId] == nil {
handler.locks[entityId] = &sync.Mutex{}
}
handler.mainLock.Unlock()
handler.locks[entityId].Lock()
defer handler.locks[entityId].Unlock()
if handler.entities[entityId] == nil {
handler.entities[entityId] = expensiveLoad(entityId)
}
return handler.entities[entityId], nil
}
Is this a reasonable approach? Is there something I am missing? Is there a better way for doing that?
i would say like this but i can be wrong, keeping two maps with same keys is useles, also pointers to mutexes are ehh.
type Wrapper struct {
*entityStateManager
sync.Mutex
}
type entityStateManagers struct {
entities map[string]*Wrapper
mainLock sync.Mutex
}
func (handler entityStateManagers) getStateManager(entityId string) (*entityStateManager, error) {
handler.mainLock.Lock()
val, ok := handler.entities[entityId]
if !ok {
val = &Wrapper{}
handler.entities[entityId] = val
}
val.Lock()
handler.mainLock.Unlock()
val.entityStateManager = expensiveLoad(entityId)
defer val.Unlock()
return val.entityStateManager, nil
}

Is it possible to check whether a field has been set without triggering a race condition?

I have a struct that is shared across many goroutines, and I would like to reduce the number of times that one field on the struct is computed, if possible. Probably easier shown:
type Container struct {
Items []Item
mu sync.RWMutex
}
(container *Container) func loadContainer() {
if container.Items != nil { // This is detected as a race condition by go
return
}
container.mu.Lock()
container.Items = loadItems() // Does some logic that I would like to avoid repeating
container.mu.Unlock()
}
Is there a safe way to accomplish this? It's almost like I want a race condition, where I don't mind if it's written to multiple times, but the first thread to do so should prevent subsequent reads from doing so. Pretty new to go and concurrency in general.
In your approach you would have to lock first before you can check if Items has been set. E.g.:
func (container *Container) loadContainer() {
container.mu.Lock()
defer container.mu.Unlock()
if container.Items != nil {
return
}
container.Items = loadItems()
}
But sync.Once provides a better and faster alternative. Also it would be wise to hide the Items field so it cannot accidentally be referred to. An exported function should take care of initialization, and return the value, initialized once only:
type Container struct {
once sync.Once
items []Item
}
func (container *Container) GetItems() []Item {
container.once.Do(func() {
container.Items = loadItems()
})
return container.items
}

How can I check what embedded type a concrete type is composed of?

I suspect I am trying to shoehorn go into behaving in an OOP way, but I don't know the go idiom to do what I want.
I have a Message struct that I use to pass data around in a client-server application:
type Message struct {
ID string `json:"id,omitempty"`
Type string `json:"type"`
Data interface{} `json:"data"`
}
the Data here can be different things, for example a number of Commands:
type Command struct {
User *types.UserInfo `json:"user"`
}
type CommandA struct {
Command
A *AData `json:"a_data"`
}
type CommandB struct {
CommandB
B *BData `json:"b_data"`
}
What I want to do is to check that the message data type is a Command and perform actions that are common to all commands, for example authorisation, all in one place and not having to type assert what type of command, calling the appropriate handler function and then do the auth, as this would result in massive code duplication.
The code below reflects what I would like to happen.
for {
select {
case m := <-in:
// what I would like to do, obviously not working as
// m.Data is not of type Command but the actual command type
if c, ok := m.Data.(msg.Command); ok {
// do auth and other common stuff
}
switch t := m.Data.(type) {
case *msg.CommandA:
go srv.handleCommandA(m.ID, t)
case *msg.CommandB:
go srv.handleCommandB(m.ID, t)
// etc etc
default:
// do something
}
}
}
How do I solve this go idiomatically?
You can define common command stuff in interface
type Commander interface{
DoCommonStuff()
}
implement it for Command struct
func (c Command) DoCommonStuff(){
//do stuff
}
and then assert
if c, ok := m.Data.(Commander); ok {
c.DoCommonStuff()
}
your other code should work unchanged
One approach is using reflection to extract common field value from the Data. In your example, since all Command has User field, we can use it to identify whether Message.Data is a command or not. If Command is not embedded to the data, simply return nil. Example code:
func GetUserInfo(v interface{}) *types.UserInfo {
vt := reflect.ValueOf(v)
if vt.Kind() == reflect.Ptr {
vt = vt.Elem()
}
if vt.Kind() != reflect.Struct {
return nil
}
u := vt.FieldByName("User")
if !u.IsValid() {
return nil
}
user, ok := u.Interface().(*types.UserInfo)
if !ok {
return nil
}
return user
}
//Call GetUserInfo then perform common operation
user := GetUserInfo(m.Data)
if user != nil {
//Do auth and other common stuff
}

Tricky case with value-to-references in for-range

Look to the code - what do you think the output would be? It's return "Third" instead of "Second" and took me a while to understand why.
Do you know a reason?
I get concept of pass-by-value & pass-by-reference quite well but this case is a bit tricky for people coming from languages like Python. So I decided it worth to share.
package main
import "fmt"
type Record struct {
Id int
Name string
}
var records = []Record{
Record{1, "First"},
Record{2, "Second"},
Record{3, "Third"},
}
func findRecod(id int) (foundRecord *Record) {
for _, record := range records {
if record.Id == id {
foundRecord = &record
// You think we can do a break here but imagine we need to do...
}
// ...something more here
}
return foundRecord
}
func main() {
foundRecord := findRecod(2)
if foundRecord == nil {
fmt.Println("Nothing found")
} else {
fmt.Println("Found: ", foundRecord.Name)
}
}
Run it online to check: https://play.golang.org/p/Y_iAl6m7Ms
I've spent some time figuring out what's going on.
You are returning pointer to a record variable which is reused by each iteration of the loop. Last iteration sets the pointer to the third structure.
The reuse of variable has an enormous advantage. It does not allocate memory in every iteration of the loop. This saves a lot of garbage collection time.
This is well know behaviour described in FAQ.
To fix it, return a pointer to an element of the slice. It is safe as slice elements are referencable in Go.
package main
import "fmt"
type Record struct {
Id int
Name string
}
var records = []Record{
Record{1, "First"},
Record{2, "Second"},
Record{3, "Third"},
}
func findRecod(id int) (foundRecord *Record) {
for i, record := range records {
if record.Id == id {
foundRecord = &records[i]
// You think we can do a break here but imagine we need to do...
}
// ...something more here
}
return foundRecord
}
func main() {
foundRecord := findRecod(2)
if foundRecord == nil {
fmt.Println("Nothing found")
} else {
fmt.Println("Found: ", foundRecord.Name)
}
}
Playground

Generic variadic argument in Go?

I know that Go doesn't support templates or overloaded functions, but I'm wondering if there's any way to do some kind of generic programming for variadic functions anyway?
I have many functions such as these:
func (this Document) GetString(name string, defaults ...string) string {
v, ok := this.GetValueFromDb(name)
if !ok {
if len(defaults) >= 1 {
return defaults[0]
} else {
return ""
}
}
return v.asString
}
func (this Document) GetInt(name string, defaults ...int) int {
v, ok := this.GetValueFromDb(name)
if !ok {
if len(defaults) >= 1 {
return defaults[0]
} else {
return 0
}
}
return v.asInt
}
// etc. for many different types
Is there any way to do this without having so much redundant code?
The most of what you can achieve is usage of interface{} type, something like this:
func (this Document) Get(name string, defaults ...interface{}) interface{} {
v, ok := this.GetValueFromDb(name)
if !ok {
if len(defaults) >= 1 {
return defaults[0]
} else {
return 0
}
}
return v
}
GetValueFromDb function should also be tweaked to return interface{} value and not some wrapper like now.
Then in the client code you can do the following:
value := document.Get("index", 1).(int) // Panics when the value is not int
or
value, ok := document.Get("index", 1).(int) // ok is false if the value is not int
This will yield some runtime overhead though. I'd better stick with separate functions and try to restructure the code somehow.
Here's a working example of how you could change your code.
package main
import (
"fmt"
)
type Document struct{
getSucceeds bool
}
func (d *Document) GetValueFromDb(name string) (interface{}, bool) {
return 1, d.getSucceeds
}
func (this Document) Get(name string, def ...int) interface{} {
v, ok := this.GetValueFromDb(name)
if !ok {
if len(def) >= 1 {
return def[0]
} else {
return 0
}
}
return v
}
func main() {
d1 := Document{true}
d2 := Document{false}
var int1, int2 int
int1 = d1.Get("foo", 2).(int)
int2 = d2.Get("foo", 2).(int)
fmt.Println(int1, int2)
}
Since you know what type you expect for the given name, you can write your Get method in a generic way, returning interface{}, and then assert the type at the call site. See the spec about type assertions.
There are different ways to emulate some aspects of generics in Go. There were lots of discussions on the mailing list. Often, there's a way to restructure code so it's less dependent on generics.
In the client code you can do like this :
res := GetValue("name", 1, 2, 3)
// or
// res := GetValue("name", "one", "two", "three")
if value, ok := res.(int); ok {
// process int return value
} else if value, ok := res.(string); ok {
// process string return value
}
// or
// res.(type) expression only work in switch statement
// and 'res' variable's type have to be interface type
switch value := res.(type) {
case int:
// process int return value
case string:
// process string return value
}

Resources