Going mutex-less - go

Alright, Go "experts". How would you write this code in idiomatic Go, aka without a mutex in next?
package main
import (
"fmt"
)
func main() {
done := make(chan int)
x := 0
for i := 0; i < 10; i++ {
go func() {
y := next(&x)
fmt.Println(y)
done <- 0
}()
}
for i := 0; i < 10; i++ {
<-done
}
fmt.Println(x)
}
var mutex = make(chan int, 1)
func next(p *int) int {
mutex <- 0
// critical section BEGIN
x := *p
*p++
// critical section END
<-mutex
return x
}
Assume you can't have two goroutines in the critical section at the same time, or else bad things will happen.
My first guess is to have a separate goroutine to handle the state, but I can't figure out a way to match up inputs / outputs.

You would use an actual sync.Mutex:
var mutex sync.Mutex
func next(p *int) int {
mutex.Lock()
defer mutex.Unlock()
x := *p
*p++
return x
}
Though you would probably also group the next functionality, state and sync.Mutex into a single struct.
Though there's no reason to do so in this case, since a Mutex is better suited for mutual exclusion around a single resource, you can use goroutines and channels to achieve the same effect
http://play.golang.org/p/RR4TQXf2ct
x := 0
var wg sync.WaitGroup
send := make(chan *int)
recv := make(chan int)
go func() {
for i := range send {
x := *i
*i++
recv <- x
}
}()
for i := 0; i < 10; i++ {
wg.Add(1)
go func() {
defer wg.Done()
send <- &x
fmt.Println(<-recv)
}()
}
wg.Wait()
fmt.Println(x)

As #favoretti mentioned, sync/atomic is a way to do it.
But, you have to use int32 or int64 rather than int (since int can be different sizes on different platforms).
Here's an example on Playground
package main
import (
"fmt"
"sync/atomic"
)
func main() {
done := make(chan int)
x := int64(0)
for i := 0; i < 10; i++ {
go func() {
y := next(&x)
fmt.Println(y)
done <- 0
}()
}
for i := 0; i < 10; i++ {
<-done
}
fmt.Println(x)
}
func next(p *int64) int64 {
return atomic.AddInt64(p, 1) - 1
}

Related

Deadlocks with buffered channels in Go

I am encountering for the below code fatal error: all goroutines are asleep - deadlock!
Am I right in using a buffered channel? I would appreciate it if you can give me pointers. I am unfortunately at the end of my wits.
func main() {
valueChannel := make(chan int, 2)
defer close(valueChannel)
var wg sync.WaitGroup
for i := 0; i < 10; i++ {
wg.Add(1)
go doNothing(&wg, valueChannel)
}
for {
v, ok := <- valueChannel
if !ok {
break
}
fmt.Println(v)
}
wg.Wait()
}
func doNothing(wg *sync.WaitGroup, numChan chan int) {
defer wg.Done()
time.Sleep(time.Duration(rand.Intn(1000)) * time.Millisecond)
numChan <- 12
}
The main goroutine blocks on <- valueChannel after receiving all values. Close the channel to unblock the main goroutine.
func main() {
valueChannel := make(chan int, 2)
var wg sync.WaitGroup
for i := 0; i < 10; i++ {
wg.Add(1)
go doNothing(&wg, valueChannel)
}
// Close channel after goroutines complete.
go func() {
wg.Wait()
close(valueChannel)
}()
// Receive values until channel is closed.
// The for / range loop here does the same
// thing as the for loop in the question.
for v := range valueChannel {
fmt.Println(v)
}
}
Run the example on the playground.
The code above works independent of the number of values sent by the goroutines.
If the main() function can determine the number of values sent by the goroutines, then receive that number of values from main():
func main() {
const n = 10
valueChannel := make(chan int, 2)
for i := 0; i < n; i++ {
go doNothing(valueChannel)
}
// Each call to doNothing sends one value. Receive
// one value for each call to doNothing.
for i := 0; i < n; i++ {
fmt.Println(<-valueChannel)
}
}
func doNothing(numChan chan int) {
time.Sleep(time.Duration(rand.Intn(1000)) * time.Millisecond)
numChan <- 12
}
Run the example on the playground.
The main problem is on the for loop of channel receiving.
The comma ok idiom is slightly different on channels, ok indicates whether the received value was sent on the channel (true) or is a zero value returned because the channel is closed and empty (false).
In this case the channel is waiting a data to be sent and since it's already finished sending the value ten times : Deadlock.
So apart of the design of the code if I just need to do the less change possible here it is:
func main() {
valueChannel := make(chan int, 2)
defer close(valueChannel)
var wg sync.WaitGroup
for i := 0; i < 10; i++ {
wg.Add(1)
go doNothing(&wg, valueChannel)
}
for i := 0; i < 10; i++ {
v := <- valueChannel
fmt.Println(v)
}
wg.Wait()
}
func doNothing(wg *sync.WaitGroup, numChan chan int) {
defer wg.Done()
time.Sleep(time.Duration(rand.Intn(1000)) * time.Millisecond)
numChan <- 12
}

X number of goroutines to update the same variable

I want to make X number of goroutines to update CountValue using parallelism (numRoutines are as much as how many CPU you have).
Solution 1:
func count(numRoutines int) (countValue int) {
var mu sync.Mutex
k := func(i int) {
mu.Lock()
defer mu.Unlock()
countValue += 5
}
for i := 0; i < numRoutines; i++ {
go k(i)
}
It becomes a data race and the returned countValue = 0.
Solution 2:
func count(numRoutines int) (countValue int) {
k := func(i int, c chan int) {
c <- 5
}
c := make(chan int)
for i := 0; i < numRoutines; i++ {
go k(i, c)
}
for i := 0; i < numRoutines; i++ {
countValue += <- c
}
return
}
I did a benchmark test on it and doing a sequential addition would work faster than using goroutines. I think it's because I have two for loops here as when I put countValue += <- c inside the first for loop, the code runs faster.
Solution 3:
func count(numRoutines int) (countValue int) {
var wg sync.WaitGroup
c := make(chan int)
k := func(i int) {
defer wg.Done()
c <- 5
}
for i := 0; i < numShards; i++ {
wg.Add(1)
go k(i)
}
go func() {
for i := range c {
countValue += i
}
}()
wg.Wait()
return
}
Still a race count :/
Is there any way better to do this?
There definitely is a better way to safely increment a variable: using sync/atomic:
import "sync/atomic"
var words int64
k := func() {
_ = atomic.AddInt64(&words, 5) // increment atomically
}
Using a channel basically eliminates the need for a mutex, or takes away the the risk of concurrent access to the variable itself, and a waitgroup here is just a bit overkill
Channels:
words := 0
done := make(chan struct{}) // or use context
ch := make(chan int, numRoutines) // buffer so each routine can write
go func () {
read := 0
for i := range ch {
words += 5 // or use i or something
read++
if read == numRoutines {
break // we've received data from all routines
}
}
close(done) // indicate this routine has terminated
}()
for i := 0; i < numRoutines; i++ {
ch <- i // write whatever value needs to be used in the counting routine on the channel
}
<- done // wait for our routine that increments words to return
close(ch) // this channel is no longer needed
fmt.Printf("Counted %d\n", words)
As you can tell, the numRoutines no longer is the number of routines, but rather the number of writes on the channel. You can move that to individual routines, still:
for i := 0; i < numRoutines; i++ {
go func(ch chan<- int, i int) {
// do stuff here
ch <- 5 * i // for example
}(ch, i)
}
WaitGroup:
Instead of using a context that you can cancel, or a channel, you can use a waitgroup + atomic to get the same result. The easiest way IMO to do so is to create a type:
type counter struct {
words int64
}
func (c *counter) doStuff(wg *sync.WaitGroup, i int) {
defer wg.Done()
_ = atomic.AddInt64(&c.words, i * 5) // whatever value you need to add
}
func main () {
cnt := counter{}
wg := sync.WaitGroup{}
wg.Add(numRoutines) // create the waitgroup
for i := 0; i < numRoutines; i++ {
go cnt.doStuff(&wg, i)
}
wg.Wait() // wait for all routines to finish
fmt.Println("Counted %d\n", cnt.words)
}
Fix for your third solution
As I mentioned in the comment: your third solution is still causing a race condition because the channel c is never closed, meaning the routine:
go func () {
for i := range c {
countValue += i
}
}()
Never returns. The waitgroup also only ensures that you've sent all values on the channel, but not that the countValue has been incremented to its final value. The fix would be to either close the channel after wg.Wait() returns so the routine can return, and add a done channel that you can close when this last routine returns, and add a <-done statement before returning.
func count(numRoutines int) (countValue int) {
var wg sync.WaitGroup
c := make(chan int)
k := func(i int) {
defer wg.Done()
c <- 5
}
for i := 0; i < numShards; i++ {
wg.Add(1)
go k(i)
}
done := make(chan struct{})
go func() {
for i := range c {
countValue += i
}
close(done)
}()
wg.Wait()
close(c)
<-done
return
}
This adds some clutter, though, and IMO is a bit messy. It might just be easier to just move the wg.Wait() call to a routine:
func count(numRoutines int) (countValue int) {
var wg sync.WaitGroup
c := make(chan int)
// add wg as argument, makes it easier to move this function outside of this scope
k := func(wg *sync.WaitGroup, i int) {
defer wg.Done()
c <- 5
}
wg.Add(numShards) // increment the waitgroup once
for i := 0; i < numShards; i++ {
go k(&wg, i)
}
go func() {
wg.Wait()
close(c) // this ends the loop over the channel
}()
// just iterate over the channel until it is closed
for i := range c {
countValue += i
}
// we've added all values to countValue
return
}

what is going on under the hood so this concurrent usage of a map is racy

in below example, the race detector will trigger an error. I am fine with it, though, as it does not change keys, the map header (if i might say), i struggle figuring out what is the reason of the race. I simply dont understand what is going on under hood so that a race detection is emitted.
package main
import (
"fmt"
"sync"
)
// scores holds values incremented by multiple goroutines.
var scores = make(map[string]int)
func main() {
var wg sync.WaitGroup
wg.Add(2)
scores["A"] = 0
scores["B"] = 0
go func() {
for i := 0; i < 1000; i++ {
// if _, ok := scores["A"]; !ok {
// scores["A"] = 1
// } else {
scores["A"]++
// }
}
wg.Done()
}()
go func() {
for i := 0; i < 1000; i++ {
scores["B"]++ // Line 28
}
wg.Done()
}()
wg.Wait()
fmt.Println("Final scores:", scores)
}
Map values are not addressable, so incrementing the integer values requires writing them back to the map itself.
The line
scores["A"]++
is equivalent to
tmp := scores["A"]
scores["A"] = tmp + 1
If you use a pointer to make the integer values addressable, and assign all the keys before the goroutines are dispatched, you can see there is no longer a race on the map itself:
var scores = make(map[string]*int)
func main() {
var wg sync.WaitGroup
wg.Add(2)
scores["A"] = new(int)
scores["B"] = new(int)
go func() {
for i := 0; i < 1000; i++ {
(*scores["A"])++
}
wg.Done()
}()
go func() {
for i := 0; i < 1000; i++ {
(*scores["B"])++
}
wg.Done()
}()
wg.Wait()
fmt.Println("Final scores:", scores)
}

Why the result is not as expected with flag "-race"?

Why the result is not as expected with flag "-race" ?
It expected the same result: 1000000 - with flag "-race" and without this
https://gist.github.com/romanitalian/f403ceb6e492eaf6ba953cf67d5a22ff
package main
import (
"fmt"
"runtime"
"sync/atomic"
"time"
)
//$ go run -race main_atomic.go
//954203
//
//$ go run main_atomic.go
//1000000
type atomicCounter struct {
val int64
}
func (c *atomicCounter) Add(x int64) {
atomic.AddInt64(&c.val, x)
runtime.Gosched()
}
func (c *atomicCounter) Value() int64 {
return atomic.LoadInt64(&c.val)
}
func main() {
counter := atomicCounter{}
for i := 0; i < 100; i++ {
go func(no int) {
for i := 0; i < 10000; i++ {
counter.Add(1)
}
}(i)
}
time.Sleep(time.Second)
fmt.Println(counter.Value())
}
The reason why the result is not the same is because time.Sleep(time.Second) does not guarantee that all of your goroutines are going to be executed in the timespan of one second. Even if you execute go run main.go, it's not guaranteed that you will get the same result every time. You can test this out if you put time.Milisecond instead of time.Second, you will see much more inconsistent results.
Whatever value you put in the time.Sleep method, it does not guarantee that all of your goroutines will be executed, it just means that it's less likely that all of your goroutines won't finish in time.
For consistent results, you would want to synchronise your goroutines a bit. You can use WaitGroup or channels.
With WaitGroup:
//rest of the code above is the same
func main() {
counter := atomicCounter{}
var wg sync.WaitGroup
for i := 0; i < 100; i++ {
wg.Add(1)
go func(no int) {
for i := 0; i < 10000; i++ {
counter.Add(1)
}
wg.Done()
}(i)
}
wg.Wait()
fmt.Println(counter.Value())
}
With channels:
func main() {
valStream := make(chan int)
doneStream := make(chan int)
result := 0
for i := 0; i < 100; i++ {
go func() {
for i := 0; i < 10000; i++ {
valStream <- 1
}
doneStream <- 1
}()
}
go func() {
counter := 0
for count := range doneStream {
counter += count
if counter == 100 {
close(doneStream)
}
}
close(valStream)
}()
for val := range valStream {
result += val
}
fmt.Println(result)
}

go routine dead lock?

I am new to golang, and I am puzzled with this deadlock (run here)
package main
import (
"fmt"
"runtime"
"time"
)
func main() {
c := make(chan string)
work := make(chan int, 1)
clvl := runtime.NumCPU()
count := 0
for i := 0; i < clvl; i++ {
go func(i int) {
for jdId := range work {
time.Sleep(time.Second * 1)
c <- fmt.Sprintf("done %d", jdId)
}
}(i)
}
go func() {
for i := 0; i < 10; i++ {
work <- i
}
close(work)
}()
for resp := range c {
fmt.Println(resp, count)
count += 1
}
}
You never close c, so your for range loop waits forever. Close it like this:
var wg sync.WaitGroup
for i := 0; i < clvl; i++ {
wg.Add(1)
go func(i int) {
defer wg.Done()
for jdId := range work {
time.Sleep(time.Second * 1)
c <- fmt.Sprintf("done %d", jdId)
}
}(i)
}
go func() {
for i := 0; i < 10; i++ {
work <- i
}
close(work)
wg.Wait()
close(c)
}()
EDIT: Fixed the panic, thanks Crast

Resources