I'm a bit confused about mutex locken/unlocking more times after another.
I'm using a RWMutex and all goroutines will have the same mutex of course.
Is this code still race-protected when using mutexes this often?
func (r *Redis) RedisDb(dbId DatabaseId) *RedisDb {
r.Mu().RLock()
size := len(r.redisDbs) // A
r.Mu().RUnlock()
if size >= int(dbId) { // B
r.Mu().RLock()
db := r.redisDbs[dbId] // C
r.Mu().RUnlock()
if db != nil { // D
return db
}
}
// E create db...
}
Example situation I would think of can happen:
gorountine1 and goroutine2 are running both this function
both are at point A so that variable size is 3
condition B is true for both goroutines
both read C at the same time
variable db is nil for both goroutines so condition C is false
now both goroutines are going to E and create the same database 2 times, thats bad
Or do I have to lock/unlock all one time in this situation?
func (r *Redis) RedisDb(dbId DatabaseId) *RedisDb {
r.Mu().Lock()
defer r.Mu().Unlock()
size := len(r.redisDbs)
if size >= int(dbId) {
db := r.redisDbs[dbId]
if db != nil {
return db
}
}
// create db...
}
Solution
func (r *Redis) RedisDb(dbId DatabaseId) *RedisDb {
getDb := func() *RedisDb { // returns nil if db not exists
if len(r.redisDbs) >= int(dbId) {
db := r.redisDbs[dbId]
if db != nil {
return db
}
}
return nil
}
r.Mu().RLock()
db := getDb()
r.Mu().RUnlock()
if db != nil {
return db
}
// create db
r.Mu().Lock()
defer r.Mu().Unlock()
// check if db does not exists again since
// multiple "mutex readers" can come to this point
db = getDb()
if db != nil {
return db
}
// now really create it
// ...
}
Welcome to the world of synchronization. Your assessment is correct, there is opportunity for concurrency issues to arise with the first implementation. With the second, those concurrency issues are removed, but it's fully locked, with no opportunities for even concurrent read access. You don't have to do it that way, you could do an initial check with a read lock, then if that check determines creation is needed, establish a write lock, then re-check, create if still necessary, then unlock. This isn't an unusual construct. It's less efficient (due to performing the check twice) so it's up to you to make a decision on the trade-off, mostly based on how expensive the check is to do twice, and how often the function would be able to operate in the read-only path.
Related
Suppose you have a basic toy system that finds and processes all files in a directory (for some definition of "processes"). A basic diagram of how it operates could look like:
If this were a real-world distributed system, the "arrows" could actually be unbounded queues, and then it just works.
In a self-contained go application, it's tempting to model the "arrows" as channels. However, due to the self-referential nature of "generating more work by needing to list subdirectories", it's easy to see that a naive implementation would deadlock. For example (untested, forgive compile errors):
func ListDirWorker(dirs, files chan string) {
for dir := range dirs {
for _, path := range ListDir(dir) {
if isDir(path) {
dirs <- path
} else {
files <- path
}
}
}
}
}
If we imagine we've configured just a single List worker, all it takes is for a directory to have two subdirectories to basically deadlock this thing.
My brain wants there to be "unbounded channels" in golang, but the creators don't want that. What's the correct idiomatic way to model this stuff? I imagine there's something simpler than implementing a thread-safe queue and using that instead of channels. :)
Had a very similar problem to solve. Needed:
finite number of recursive workers (bounded parallelism)
content.Context for early cancelations (enforce timeout limits etc.)
partial results (some goroutines hit errors while others did not)
crawl completion (worker clean-up etc.) via recursive depth tracking
Below I describe the problem and the gist of the solution I arrived at
Problem: scrape a HR LDAP directory with no pagination support. Server-side limits also precluded bulk queries greater than 100K records. Needed small queries to work around these limitations. So recursively navigated the tree from the top (CEO) - listing employees (nodes) and recursing on managers (branches).
To avoid deadlocks - a single workItem channel was used not only by workers to read (get) work, but also to write (delegate) to other idle workers. This approach allowed for fast worker saturation.
Note: not included here, but worth adding, is to use a common API rate-limiter to avoid multiple workers collectively abusing/exceeding any server-side API rate limits.
To start the crawl, create the workers and return a results channel and an error channel. Some notes:
c.in the workItem channel must be unbuffered for delegation to work (more on this later)
c.rwg tracks collective recursion depth for all worker. When it reaches zero, all recursion is done and the crawl is complete
func (c *Crawler) Crawl(ctx context.Context, root Branch, workers int) (<-chan Result, <-chan error) {
errC := make(chan error, 1)
c.rwg = sync.WaitGroup{} // recursion depth waitgroup (to determine when all workers are done)
c.rwg.Add(1) // add to waitgroups *BEFORE* starting workers
c.in = make(chan workItem) // input channel: shared by all workers (to read from and also to write to when they need to delegate)
c.out = make(chan Result) // output channel: where all workers write their results
go func() {
workerErrC := c.createWorkers(ctx, workers)
c.in <- workItem{
branch: root, // initial place to start crawl
}
for err := range workerErrC {
if err != nil {
// tally for partial results - or abort on first error (see werr)
}
}
// summarize crawl success/failure via a single write to errC
errC <- werr // nil, partial results, aborted early etc.
close(errC)
}
return c.out, errC
}
Create a finite number of individual workers. The returned error channel receives an error for each individual worker:
func (c *Crawler) createWorkers(ctx context.Context, workers int) (<-chan error) {
errC := make(chan error)
var wg sync.WaitGroup
wg.Add(workers)
for i := 0; i < workers; i++ {
i := i
go func() {
defer wg.Done()
var err error
defer func() {
errC <- err
}()
conn := Dial("somewhere:8080") // worker prep goes here (open network connect etc.)
for workItem := range c.in {
err = c.recurse(ctx, i+1, conn, workItem)
if err != nil {
return
}
}
}()
}
go func() {
c.rwg.Wait() // wait for all recursion to finish ...
close(c.in) // ... so safe to close input channel ...
wg.Wait() // ... wait for all workers to complete ...
close(errC) // .. finally signal to caller we're truly done
}()
return errC
}
recurse logic:
for any potentially blocking channel write, always check the ctx for cancelation, so we can abort early
c.in is deliberately unbuffered to ensure delegation works (see final note)
func (c *Crawler) recurse(ctx context.Context, workID int, conn *net.Conn, wi workItem) error {
defer c.rwg.Done() // decrement recursion count
select {
case <-ctx.Done():
return ctx.Err() // canceled/timeout etc.
case c.out <- Result{ /* Item: wi.. */}: // write to results channel (manager or employee)
}
items, err := getItems(conn) // WORKER CODE (e.g. get manager employees etc.)
if err != nil {
return err
}
for _, i := range items {
// leaf case
if i.IsLeaf() {
select {
case <-ctx.Done():
return ctx.Err()
case c.out <- Result{ Item: i.Leaf }:
}
continue
}
// branch case
wi := workItem{
branch: i.Branch,
}
c.rwg.Add(1) // about to recurse (or delegate-recursion)
select {
case c.in <- wi:
// delegated to another worker!
case <-ctx.Done(): // context canceled...
c.rwg.Done() // ... so undo above `c.rwg.Add(1)`
return ctx.Err()
default:
// no-one to delegated to (all busy) - so this worker will keep working
err = c.recurse(ctx, workID, conn, wi)
if err != nil {
return err
}
}
}
return nil
}
Delegation is key:
if a worker successfully writes to the worker channel, then it knows work has been delegated to another worker.
if it cannot, then the worker knows all workers are busy working (i.e. not waiting on work items) - and so it must recurse itself
So one gets both the benefits of recursion, but also leveraging a fixed-sized worker pool.
Background: currently I am investigating the mechanism of distributed lock in order to implement it in our email service to deduplicate in the time the QPS is high.
The main article I am reading is this one https://redis.io/docs/reference/patterns/distributed-locks/#retry-on-failure from redis official.
In this article it mentioned in the retry section that
When a client is unable to acquire the lock, it should try again after a random delay in order to try to desynchronize multiple clients trying to acquire the lock for the same resource at the same time (this may result in a split brain condition where nobody wins)
Also the faster a client tries to acquire the lock in the majority of Redis instances, the smaller the window for a split brain condition (and the need for a retry), so ideally the client should try to send the SET commands to the N instances at the same time using multiplexing.
The fact that when a client needs to retry a lock, it waits a time which is comparably greater than the time needed to acquire the majority of locks, in order to probabilistically make split brain conditions during resource contention unlikely.
I read a few recommended open source implementation of redlock from git, indeed there is delay between retry, copy and pasted here for your convenience.
func (m *Mutex) LockContext(ctx context.Context) error {
if ctx == nil {
ctx = context.Background()
}
value, err := m.genValueFunc()
if err != nil {
return err
}
for i := 0; i < m.tries; i++ {
if i != 0 {
select {
case <-ctx.Done():
// Exit early if the context is done.
return ErrFailed
case <-time.After(m.delayFunc(i)):
// Fall-through when the delay timer completes.
}
}
start := time.Now()
n, err := func() (int, error) {
ctx, cancel := context.WithTimeout(ctx, time.Duration(int64(float64(m.expiry)*m.timeoutFactor)))
defer cancel()
return m.actOnPoolsAsync(func(pool redis.Pool) (bool, error) {
return m.acquire(ctx, pool, value)
})
}()
if n == 0 && err != nil {
return err
}
now := time.Now()
until := now.Add(m.expiry - now.Sub(start) - time.Duration(int64(float64(m.expiry)*m.driftFactor)))
if n >= m.quorum && now.Before(until) {
m.value = value
m.until = until
return nil
}
_, err = func() (int, error) {
ctx, cancel := context.WithTimeout(ctx, time.Duration(int64(float64(m.expiry)*m.timeoutFactor)))
defer cancel()
return m.actOnPoolsAsync(func(pool redis.Pool) (bool, error) {
return m.release(ctx, pool, value)
})
}()
if i == m.tries-1 && err != nil {
return err
}
}
return ErrFailed
}
What I do not understand is that why the existence of the retry can help to avoid brain split?
Intuitively the backoff time(delay) for the retry is used to avoid overloading the redis instance.
I've been working on a sort-of pub-sub mechanism in the application we're building. The business logic basically generates a whack-ton of events, which in turn can be used to feed data to the client using API's, or persistent in storage if the application is running with that option enabled.
What we had, and observed:
Long story short, it turns out we were dropping data we really ought not to have been dropping. The "subscriber" had a channel with a large buffer, and essentially only read data from this channel, checked a few things, and appended it to a slice. The capacity of the slice is such that memory allocations were kept to a minimum. Simulating a scenario where the subscriber channel had a buffer of, say, 1000 data-sets, we noticed data could be dropping after only 10 sets being sent. The very first event was never dropped.
The code we had at this point looks something like this:
type broker struct {
ctx context.Context
subs []*sub
}
type sub struct {
ctx context.Context
mu *sync.Mutex
ch chan []interface{}
buf []interface{}
}
func (b *broker) Send(evts ...interface{}) {
rm := make([]int, 0, len(b.subs))
defer func() {
for i := len(rm) - 1; i >= 0; i-- {
// last element
if i == len(b.subs)-1 {
b.subs = b.subs[:i]
continue
}
b.subs = append(b.subs[:i], b.subs[i+1:]...)
}
}()
for i, s := range b.subs {
select {
case <-b.ctx.Done(): // is the app still running
return
case <-s.Stopped(): // is this sub still valid
rm = append(rm, i)
case s.C() <- evts: // can we write to the channel
continue
default: // app is running, sub is valid, but channel is presumably full, skip
fmt.Println("skipped some events")
}
}
}
func NewSub(ctx context.Context, buf int) *sub {
s := &sub{
ctx: ctx,
mu: &sync.Mutex{},
ch: make(chan []interface{}, buf),
buf: make([]interface{}, 0, buf),
}
go s.loop(ctx) // start routine to consume events
return s
}
func (s *sub) C() chan<- []interface{} {
return s.ch
}
func (s *sub) Stopped() <-chan struct{} {
return s.ctx.Done()
}
func (s *sub) loop(ctx context.Context) {
defer func() {
close(s.ch)
}()
for {
select {
case <-ctx.Done():
return
case data := <-s.ch:
// do some processing
s.mu.Lock()
s.buf = append(s.buf, data...)
s.mu.Unlock()
}
}
}
func (s *sub) GetProcessedData(amt int) []*wrappedT {
s.mu.Lock()
data := s.buf
if len(data) == amt {
s.buf = s.buf[:0]
} else if len(data) > amt {
data = data[:amt]
s.buf = s.buf[amt:]
} else {
s.buf = make([]interface{}, 0, cap(s.buf))
}
s.mu.Unlock()
ret := make([]*wrappedT, 0, len(data))
for _, v := range data {
// some processing
ret = append(ret, &wrappedT{v})
}
return ret
}
Now obviously, the buffers are there to ensure that events can still be consumed when we're calling things like GetProcessedData. That type of call is usually the result of an API request, or some internal flush/persist to storage mechanism. Because of the mutex lock, we might not be reading from the internal channel. Weirdly, the channel buffers never got backed up all the way through, but not all data made its way through to the subscribers. As mentioned, the first event always did, which made us even more suspicious.
What we eventually tried (to fix):
After a fair bit of debugging, hair pulling, looking at language specs, and fruitless googling I began to suspect the select statement to be the problem. Instead of sending to the channels directly, I changed it to the rather hacky:
func (b *broker) send(s *sub, evts []interface{}) {
ctx, cfunc := context.WithTimeout(b.ctx, 100 *time.Millisecond)
defer cfunc()
select {
case <-ctx:
return
case sub.C() <- evts:
return
case <-sub.Closed():
return
}
}
func (b *broker) Send(evts ...interface{}) {
for _, s := range b.subs {
go b.send(s, evts)
}
}
Instantly, all events were correctly propagated through the system. Calling Send on the broker wasn't blocking the part of the system that actually performs the heavy lifting (that was the reason for the use of channels after all), and things are performing reasonably well.
The actual question(s):
There's a couple of things still bugging me:
The way I read the specs, the default statement ought to be the last resort, solely as a way out to prevent blocking channel operations in a select statement. Elsewhere, I read that the runtime may not consider a case ready for communication if there is no routine consuming what you're about to write to the channel, irrespective of channel buffers. Is this indeed how it works?
For the time being, the context with timeout fixes the bigger issue of data not propagating properly. However, I do feel like there should be a better way.
Has anyone ever encountered something similar, and worked out exactly what's going on?
I'm happy to provide more details where needed. I've kept the code as minimal as possible, omitting a lot of complexities WRT the broker system we're using (different event types, different types of subscribers, etc...). We don't use the interface{} type anywhere in case anyone is worried about that :P
For now, though, I think this is plenty of text for a Friday.
I've been trying to wrap my head around how paging in Apache Cassandra with the driver functions in GOlang.
I have the following code for fetching rows
/// Assume all other prerequisites.
session, _ := cluster.CreateSession()
session.SetPageSize(100)
var pagestate []byte
query := session.Query(`select * from keyspace.my_table`)
query = query.PageState(pagestate)
if err := query.Exec(); != nil {
panic(err)
}
iter := query.Iter()
for {
row := map[string]interface{}{}
if !iter.MapScan(row) {
pagestate = iter.PageState()
break
}
/// Do whatever I need with row.
}
What I'm trying to achieve:
The table I'm referencing is huge, over 18k rows huge, and I want to fetch all of them for a special operation in the most efficient manner using the driver's built in paging so the query won't time out.
The problem:
I'm not sure how to get the query to resume at the previous page state. I'm not sure if this involves running the query in a loop and managing page state outside of it or not. I understand how to get and set page state, I can't figure out how to iterate the query with a new page sate each time without a proper halt condition when all the paging is done.
My best attempt:
var pagestate []byte
query := session.Query(`select * from keyspace.my_table`)
for {
query = query.PageState(pagestate)
if err := query.Exec(); != nil {
panic(err)
}
iter := query.Iter()
/// I don't know if I'm using this bool correct or not.
/// My assumption is that this would return false when a new page isn't
/// avaliable, thus meaning that all the pages have been filled and
/// the loop can exit.
if !iter.WillSwitchPage() {
break
}
for {
row := map[string]interface{}{}
if !iter.MapScan(row) {
pagestate = iter.PageState()
break
}
/// Do whatever I need with row.
}
}
Am I doing this right, or is there a better way to achieve this?
So, as it turns out, WillSwitchPage() will never return true at any point in the loop. Not sure why. I guess it's because of how I'm using MapScan() to control the inner loop.
Anyway, I figured out a solution by checking the []byte for the page state itself at the en dof the query loop. If the driver reaches the end and doesn't fill the page, page state will have 0 elements, so I'm using that as my halt condition. This may or may not be the most elegant or intended way to handle the driver's pagination, but it's functioning as wanted.
var pagestate []byte
query := session.Query(`select * from keyspace.my_table`)
for {
query = query.PageState(pagestate)
if err := query.Exec(); != nil {
panic(err)
}
iter := query.Iter()
for {
row := map[string]interface{}{}
if !iter.MapScan(row) {
pagestate = iter.PageState()
break
}
/// Do whatever I need with row.
}
if len(pagestate) == 0 {
break
}
}
I am making a cache wrapper around a database. To account for possibly slow database calls, I was thinking of a mutex per key (pseudo Go code):
mutexes = map[string]*sync.Mutex // instance variable
mutexes[key].Lock()
defer mutexes[key].Unlock()
if value, ok := cache.find(key); ok {
return value
}
value = databaseCall(key)
cache.save(key, value)
return value
However I don't want my map to grow too much. My cache is an LRU and I want to have a fixed size for some other reasons not mentioned here. I would like to do something like
delete(mutexes, key)
when all the locks on the key are over but... that doesn't look thread safe to me... How should I do it?
Note: I found this question
In Go, can we synchronize each key of a map using a lock per key? but no answer
A map of mutexes is an efficient way to accomplish this, however the map itself must also be synchronized. A reference count can be used to keep track of entries in concurrent use and remove them when no longer needed. Here is a working map of mutexes complete with a test and benchmark.
(UPDATE: This package provides similar functionality: https://pkg.go.dev/golang.org/x/sync/singleflight )
mapofmu.go
// Package mapofmu provides locking per-key.
// For example, you can acquire a lock for a specific user ID and all other requests for that user ID
// will block until that entry is unlocked (effectively your work load will be run serially per-user ID),
// and yet have work for separate user IDs happen concurrently.
package mapofmu
import (
"fmt"
"sync"
)
// M wraps a map of mutexes. Each key locks separately.
type M struct {
ml sync.Mutex // lock for entry map
ma map[interface{}]*mentry // entry map
}
type mentry struct {
m *M // point back to M, so we can synchronize removing this mentry when cnt==0
el sync.Mutex // entry-specific lock
cnt int // reference count
key interface{} // key in ma
}
// Unlocker provides an Unlock method to release the lock.
type Unlocker interface {
Unlock()
}
// New returns an initalized M.
func New() *M {
return &M{ma: make(map[interface{}]*mentry)}
}
// Lock acquires a lock corresponding to this key.
// This method will never return nil and Unlock() must be called
// to release the lock when done.
func (m *M) Lock(key interface{}) Unlocker {
// read or create entry for this key atomically
m.ml.Lock()
e, ok := m.ma[key]
if !ok {
e = &mentry{m: m, key: key}
m.ma[key] = e
}
e.cnt++ // ref count
m.ml.Unlock()
// acquire lock, will block here until e.cnt==1
e.el.Lock()
return e
}
// Unlock releases the lock for this entry.
func (me *mentry) Unlock() {
m := me.m
// decrement and if needed remove entry atomically
m.ml.Lock()
e, ok := m.ma[me.key]
if !ok { // entry must exist
m.ml.Unlock()
panic(fmt.Errorf("Unlock requested for key=%v but no entry found", me.key))
}
e.cnt-- // ref count
if e.cnt < 1 { // if it hits zero then we own it and remove from map
delete(m.ma, me.key)
}
m.ml.Unlock()
// now that map stuff is handled, we unlock and let
// anything else waiting on this key through
e.el.Unlock()
}
mapofmu_test.go:
package mapofmu
import (
"math/rand"
"strconv"
"strings"
"sync"
"testing"
"time"
)
func TestM(t *testing.T) {
r := rand.New(rand.NewSource(42))
m := New()
_ = m
keyCount := 20
iCount := 10000
out := make(chan string, iCount*2)
// run a bunch of concurrent requests for various keys,
// the idea is to have a lot of lock contention
var wg sync.WaitGroup
wg.Add(iCount)
for i := 0; i < iCount; i++ {
go func(rn int) {
defer wg.Done()
key := strconv.Itoa(rn)
// you can prove the test works by commenting the locking out and seeing it fail
l := m.Lock(key)
defer l.Unlock()
out <- key + " A"
time.Sleep(time.Microsecond) // make 'em wait a mo'
out <- key + " B"
}(r.Intn(keyCount))
}
wg.Wait()
close(out)
// verify the map is empty now
if l := len(m.ma); l != 0 {
t.Errorf("unexpected map length at test end: %v", l)
}
// confirm that the output always produced the correct sequence
outLists := make([][]string, keyCount)
for s := range out {
sParts := strings.Fields(s)
kn, err := strconv.Atoi(sParts[0])
if err != nil {
t.Fatal(err)
}
outLists[kn] = append(outLists[kn], sParts[1])
}
for kn := 0; kn < keyCount; kn++ {
l := outLists[kn] // list of output for this particular key
for i := 0; i < len(l); i += 2 {
if l[i] != "A" || l[i+1] != "B" {
t.Errorf("For key=%v and i=%v got unexpected values %v and %v", kn, i, l[i], l[i+1])
break
}
}
}
if t.Failed() {
t.Logf("Failed, outLists: %#v", outLists)
}
}
func BenchmarkM(b *testing.B) {
m := New()
b.ResetTimer()
for i := 0; i < b.N; i++ {
// run uncontended lock/unlock - should be quite fast
m.Lock(i).Unlock()
}
}
I wrote a simple similar implementation: mapmutex
But instead of a map of mutexes, in this implementation, a mutex is used to guard the map and each item in the map is used like a 'lock'. The map itself is just simple ordinary map.