I'm learning concurrency-related issues in Golang. I wrote some code:
package main
import (
"fmt"
"time"
)
func incr(num *int) {
*num = *num + 1
}
func main() {
var a = 0
for i := 0; i < 50; i++ {
go incr(&a)
}
incr(&a)
time.Sleep(1 * time.Second)
fmt.Println(a)
}
The result of this code is: 51
In this code I've declared a variable which I'm increasing in 50 running goroutines. What I've read and unsterstood this code should fail because multiple goroutines are writing to same memory address. In this case I should add sync.Mutex lock in order to fix that.
Code is available in the playground: https://play.golang.org/p/Tba9pfpxaHY
Could You explain what is really happening in this program?
Guess what? I ran your app and I get varying outputs: sometimes 49, sometimes 48, sometimes 50 (and sometimes 51).
If you run your app with the race detector enabled (go run -race play.go), it tells you have data races:
==================
WARNING: DATA RACE
Read at 0x00c00009a010 by goroutine 7:
main.incr()
/home/icza/gows/src/play/play.go:9 +0x3a
Previous write at 0x00c00009a010 by goroutine 6:
main.incr()
/home/icza/gows/src/play/play.go:9 +0x50
Goroutine 7 (running) created at:
main.main()
/home/icza/gows/src/play/play.go:17 +0x83
Goroutine 6 (finished) created at:
main.main()
/home/icza/gows/src/play/play.go:17 +0x83
==================
When you have data races, the behavior of your app is undefined. "Seemingly working sometimes" also fits into the "undefined" behavior, but undefined also means it can do anything else too.
See related questions:
Assign a map to another map is safety in golang?
Is it safe to read a function pointer concurrently without a lock?
golang struct concurrent read and write without Lock is also running ok?
Related
The question concerns the following code:
package main
import "fmt"
func main() {
var counters = map[int]int{}
for i := 0; i < 5; i++ {
go func(counters map[int]int, th int) {
for j := 0; j < 5; j++ {
counters[th*10+j]++
}
}(counters, i)
}
fmt.Scanln()
fmt.Println("counters result", counters)
}
Here is the output I get when I run this code with go run -race race.go
$ go run -race race.go
==================
WARNING: DATA RACE
Read at 0x00c000092150 by goroutine 8:
runtime.mapaccess1_fast64()
/usr/lib/go-1.13/src/runtime/map_fast64.go:12 +0x0
main.main.func1()
/tmp/race.go:10 +0x6b
Previous write at 0x00c000092150 by goroutine 7:
runtime.mapassign_fast64()
/usr/lib/go-1.13/src/runtime/map_fast64.go:92 +0x0
main.main.func1()
/tmp/race.go:10 +0xaf
Goroutine 8 (running) created at:
main.main()
/tmp/race.go:8 +0x67
Goroutine 7 (finished) created at:
main.main()
/tmp/race.go:8 +0x67
==================
==================
WARNING: DATA RACE
Read at 0x00c0000aa188 by main goroutine:
reflect.typedmemmove()
/usr/lib/go-1.13/src/runtime/mbarrier.go:177 +0x0
reflect.copyVal()
/usr/lib/go-1.13/src/reflect/value.go:1297 +0x7b
reflect.(*MapIter).Value()
/usr/lib/go-1.13/src/reflect/value.go:1251 +0x15e
internal/fmtsort.Sort()
/usr/lib/go-1.13/src/internal/fmtsort/sort.go:61 +0x259
fmt.(*pp).printValue()
/usr/lib/go-1.13/src/fmt/print.go:773 +0x146f
fmt.(*pp).printArg()
/usr/lib/go-1.13/src/fmt/print.go:716 +0x2ee
fmt.(*pp).doPrintln()
/usr/lib/go-1.13/src/fmt/print.go:1173 +0xad
fmt.Fprintln()
/usr/lib/go-1.13/src/fmt/print.go:264 +0x65
main.main()
/usr/lib/go-1.13/src/fmt/print.go:274 +0x13c
Previous write at 0x00c0000aa188 by goroutine 10:
main.main.func1()
/tmp/race.go:10 +0xc4
Goroutine 10 (finished) created at:
main.main()
/tmp/race.go:8 +0x67
==================
counters result map[0:1 1:1 2:1 3:1 4:1 10:1 11:1 12:1 13:1 14:1 20:1 21:1 22:1 23:1 24:1 30:1 31:1 32:1 33:1 34:1 40:1 41:1 42:1 43:1 44:1]
Found 2 data race(s)
exit status 66
Here is what I can't understand. Why there a race condition at all? Aren't we reading/writing values only one go routine can access? For example routine 0 will modify values only in counter[0] through counters[4], routine 1 will modify values only in counters[10] through counters[14], routine 2 will only modify values in counters[20] through counters[24] and so on. I'm not seeing a race condition here. Feels like I'm missing something. Will someone be able to shed some light on this?
Just an FYI I'm new to go. If you could dumb down the explanation (if it is possible) I would appreciate it.
That would be true for an array (or a slice), but a map is a complicated data structure which, among others, have the following properties:
It's free to relocate the elements stored in it in memory at any time it sees fit.
A map is initially empty, and placing an element in it (what appears as assignment in your case) involves a lot of operations on the map's internals.
Additionally, in a case like yours — incrementing an integer stored in a map — is really a map lookup, increment, and a map store.
The first and the last operations involve lookup by key.
Now consider what happens if one goroutine performs lookup at the same time another goroutine modifies the map's internal state when performing map store.
You might want to read up a bit on what is an associative array, and how it's typically implemented.
Aren't we reading/writing values only one go routine can access?
You already got a great answer from #kostix on that matter: the internals of the map are modified when you add elements to it, so it's not accurate to think that routine 0 will modify values only in counter[0] through counters[4].
But that's not all.
There's yet another data race issue in your code that's a bit more subtle and might be very difficult to catch even in tests.
To explore it, let's get rid of the "map internals" issue that #kostix mentioned, by imagining that your code is almost exactly the same, but with one tiny change: instead of using a map[int]int, imagine that you're using a []int, initialized to have at least length 56. Something like this:
// THERE'S ANOTHER RACE CONDITION HERE.
// var counters = map[int]int{}
var counters = make([]int, 56)
for i := 0; i < 5; i++ {
// go func(counters map[int]int, th int) {
go func(counters []int, th int) {
for j := 0; j < 5; j++ {
counters[th*10+j]++
}
}(counters, i)
}
fmt.Scanln()
fmt.Println("counters result", counters)
This is nearly equivalent, but gets rid of the "map internals" issue. The goal is to shift the focus away from "map internals" to show you the second issue.
There's still a race condition there. By the way, it's also similar to a race condition that exists in the first attempted solution in another answer you got, that uses a sync.Mutex but in a way that is still wrong.
The problem here is that there's no happens before relationship between the operations that change the counters and the operation that reads from it.
The fmt.Scanln() doesn't help: even though it allows you to introduce an arbitrary time delay between the code right before it (i.e., when the for loop launches the goroutines) and the code right after it (i.e., the fmt.Println()) — so that you could think "Ok, I'm just gonna wait 'a reasonably long amount of time' before pressing Enter", that doesn't eliminate the race condition.
The race condition here arises from the fact that "passage of time" (i.e., you waiting to hit Enter) does not establish a happens-before relationship between the writes to counters and the reads from it.
This notion of happens-before is absolutely fundamental for avoiding data races: you can only guarantee the absence of a data race if you can guarantee the existence of a happens-before relationship between 2 operations.
Like I mentioned, "passage of time" doesn't establish a "happens before". To establish it, you could use one of many alternatives, including primitives in the sync or atomic packages, or channels, etc.
While I'd probably suggest focusing on studying channels, and then the sync package (sync.Mutex, sync.WaitGroup, etc), and maybe only after all that the atomic package, if you do want to read more about this idea of happens before from the authoritative source, here's the link: https://golang.org/ref/mem . But be warned that it's a nasty can of worms.
Hopefully these comments here help you see why it's absolutely fundamental to follow the standard patterns for concurrency in Go. Things can be way more subtle than at first sight.
And to conclude, a quote from The Go Memory Model link I shared above:
If you must read the rest of this document to understand the behavior of your program, you are being too clever.
Don't be clever.
EDIT: for completion, here's how you could solve the problem.
There are 2 parts to the solution: (1) make sure that there's no concurrent modifications to the map; (2) make sure that there's a happens-before between all the changes to the map and the read.
For (1), you can use a sync.Mutex. Lock it before writing, unlock it after the write.
For (2), you need to ensure that the main goroutine can only get to the fmt.Println() after all the modifications are done. And remember: here, after doesn't mean "at a later point in time", but it specifically means that a happens-before relationship must be established. The 2 common patterns to solve this are to use a channel or a sync.WaitGroup. The WaitGroup solution is probably easier to reason about here, so that's what I'd use.
var mu sync.Mutex // (A)
var wg sync.WaitGroup // (A)
var counters = map[int]int{}
wg.Add(5) // (B)
for i := 0; i < 5; i++ {
go func(counters map[int]int, th int) {
for j := 0; j < 5; j++ {
mu.Lock() // (C)
counters[th*10+j]++
mu.Unlock() // (C)
}
wg.Done() // (D)
}(counters, i)
}
wg.Wait() // (E)
fmt.Scanln()
fmt.Println("counters result", counters)
(A) You don't need to initialize either the Mutex nor the WaitGroup, since their zero values are ready to use. Also, you don't need to make them pointers to anything.
(B) You .Add(5) to the WaitGroup's counter, meaning that it will have to wait for 5 .Done() signals before proceeding if you .Wait() on it. The number 5 here is because you're launching 5 goroutines, and you need to establish happens-before relationships between the changes made on all of them and the main goroutine's fmt.Println().
(C) You .Lock() and .Unlock() the Mutex around modifications to the map, to ensure that they are not done concurrently.
(D) Just before each goroutine terminates, you call wg.Done(), which decrements the WaitGroup's internal counter.
(E) Finally, you wg.Wait(). This function blocks until the wg's counter reaches 0. And here's the super important piece: the WaitGroup establishes a happens-before relationship between the calls to wg.Done() and the return of the wg.Wait() call. In other words, from a memory consistency perspective, the main goroutine is guaranteed to see all the changes performed to the map by all the goroutines!
AND FINALLY you can run that code with -race and be happy!
For you to explore further: instead of map + sync.Mutex, you could replace that with just sync.Map. But the sync.WaitGroup would still be necessary. Try to write a solution using that, it might be a nice exercise.
In addition to #kostix answer. You've to know that multiple goroutines should not access (write/read) to the same ressource at a given time.
So, in your implementation you may easly be in the case that multiple goroutines are updating (reading/writing) concurrently the same ressource (which is your map) at the same time.
What should happen ? Which value should be in this given map key ? This a what called race condition
Here is some potential fixes to your code:
Using Mutex:
package main
import (
"fmt"
"sync"
)
func main() {
var counters = map[int]int{}
var mutex = &sync.Mutex{}
for i := 0; i < 3; i++ {
go func(counters map[int]int, th int) {
for j := 0; j < 3; j++ {
mutex.Lock() // Lock the access to the map
counters[th*10+j]++
mutex.Unlock() // Release the access
}
}(counters, i)
}
fmt.Scanln()
fmt.Println("counters result", counters)
}
Output:
counters result map[0:1 1:1 2:1 10:1 11:1 12:1 20:1 21:1 22:1]
Using sync.Map:
package main
import (
"fmt"
"sync"
)
func main() {
var counters sync.Map
for i := 0; i < 3; i++ {
go func(th int) {
for j := 0; j < 3; j++ {
if result, ok := counters.Load(th*10 + j); ok {
value := result.(int) + 1
counters.Store(th*10+j, value+1)
} else {
counters.Store(th*10+j, 1)
}
}
}(i)
}
fmt.Scanln()
counters.Range(func(k, v interface{}) bool {
fmt.Println("key:", k, ", value:", v)
return true
})
}
Output:
key: 21 , value: 1
key: 10 , value: 1
key: 11 , value: 1
key: 0 , value: 1
key: 1 , value: 1
key: 20 , value: 1
key: 2 , value: 1
key: 22 , value: 1
key: 12 , value: 1
I'm logging every second the length of a map; I don't care if I have the "exact" value / race conditions (off by one is ok). I'm interested to know if this could cause a panic and if I have to enclose len() with some .RLock()/Unlock() or not.
I'm asking because concurrent reads/writes in a map will cause a panic (Go detects that) but I don't know if reading the length counts as a "read". I have tried with a test program but cannot produce a crash but I'd rather have an exact answer, at least for the sake of it.
If it matters I'm interested in both len for arrays and for maps.
Thanks!
It is a race condition. The results are undefined. For example,
racer.go:
package main
func main() {
m := make(map[int]int)
l := 0
go func() {
for {
l = len(m)
}
}()
for i := 0; i < 10000; i++ {
m[i] = i
}
}
Output:
$ go run -race racer.go
==================
WARNING: DATA RACE
Read at 0x00c00008e000 by goroutine 5:
main.main.func1()
/home/peter/gopath/src/racer.go:8 +0x5f
Previous write at 0x00c00008e000 by main goroutine:
runtime.mapassign_fast64()
/home/peter/go/src/runtime/map_fast64.go:92 +0x0
main.main()
/home/peter/gopath/src/racer.go:12 +0xba
Goroutine 5 (running) created at:
main.main()
/home/peter/gopath/src/racer.go:6 +0x92
==================
Found 1 data race(s)
exit status 66
$
References:
Wikipedia: Race condition
The Go Blog: Introducing the Go Race Detector
Go: Data Race Detector
Benign Data Races: What Could Possibly Go Wrong?
If you accept dirty data of len(), the operation will not cause deadlock, and will not cause any panic.
I met this problem in this scenario: I will decide if I need to clean the expired data in a map by if length of map reach the limitation. But I need not exactly limit the size of the map, because write operation is enclosed by .Lock()/Unlock() and will also check the limitation.
Hope this scenario will help you. However if len() is not a high frequency called, I think enclose that with lock is better.
I was disturbed by a question,
should we add lock if only one thread write variable, and other thread just read variable?
so I write such code to test it
package main
import (
"fmt"
"runtime"
"sync"
"time"
)
var lock sync.RWMutex
var i = 0
func main() {
runtime.GOMAXPROCS(2)
go func() {
for {
fmt.Println("i am here", i)
time.Sleep(time.Second)
}
}()
for {
i += 1
}
}
The result is keep print i am here 0 even after second of time. I know a little about Memory barrier or cpu cache. but how could it be cache for such a long time? I think after a few time, it should read variable I already changed.
Can anyone who is master go or computer system could help answer, please?
Update: i know it is a wrong way to update variable like this, i want to know why it is undefined in cpu/memory view.
You have a data race. Therefore, the results are undefined.
package main
import (
"fmt"
"runtime"
"sync"
"time"
)
var lock sync.RWMutex
var i = 0
func main() {
runtime.GOMAXPROCS(2)
go func() {
for {
fmt.Println("i am here", i)
time.Sleep(time.Second)
}
}()
for {
i += 1
}
}
Output:
$ go run -race racer.go
==================
WARNING: DATA RACE
Read at 0x0000005e3600 by goroutine 6:
main.main.func1()
/home/peter/gopath/src/racer.go:17 +0x63
Previous write at 0x0000005e3600 by main goroutine:
main.main()
/home/peter/gopath/src/racer.go:22 +0x7b
Goroutine 6 (running) created at:
main.main()
/home/peter/gopath/src/racer.go:15 +0x4f
==================
i am here 3622
i am here 43165250
i am here 86147697
^Csignal: interrupt
$
References:
Data Race Detector
Benign Data Races: What Could Possibly Go Wrong?
should we add lock if only one thread write variable, and other thread just read variable?
Yes. Always. No arguing here.
Your test code proves and disproves nothing as its behaviour is undefined.
finally, i find this answers, i know with a data race you will get a undefined behave, but i want to know why it behave like that currently.
this snap code is because complier just remove Add function, it never add.
so we have lesson, if you write a undefined behave, you may got a moon - -
complier will treat you code as rubbish, it does not have any value.
Is it thread-safe to modify the channel that a consumer is reading from?
Consider the following code:
func main(){
channel := make(chan int, 3)
channel_ptr := &channel
go supplier (channel_ptr)
go consumer (channel_ptr)
temp = *channel_ptr
// Important bit
*channel_ptr = make(chan int, 5)
more := true
for more{
select {
case msg := <-temp:
*channel_ptr <- msg
default:
more = false
}
}
// Block main indefinitely to keep the children alive
<-make(chan bool)
}
func consumer(c *chan int){
for true{
fmt.Println(<-(*c))
}
}
func supplier(c *chan int){
for i :=0; i < 5; i ++{
(*c)<-i
}
}
If channels and make work the way that I want them to, I should get the following properties:
The program always outputs 0 1 2 3 4
The program will never panic from trying to read from a non-initialized channel (IE, the part I labelled Important bit is atomic)
From several test runs, this seems to be true, but I can't find it anywhere in the documentation and I'm worried about subtle race conditions.
Update
Yeah, what I was doing doesn't work. This thread is probably buried at this point, but does anybody know how to dynamically resize a buffered channel?
It's not thread safe.
If you run with -race flag to use race detector, you'll see the bug:
$ run -race t.go
==================
WARNING: DATA RACE
Write at 0x00c420086018 by main goroutine:
main.main()
/Users/kjk/src/go/src/github.com/kjk/go-cookbook/start-mysql-in-docker-go/t.go:14 +0x128
Previous read at 0x00c420086018 by goroutine 6:
main.supplier()
/Users/kjk/src/go/src/github.com/kjk/go-cookbook/start-mysql-in-docker-go/t.go:37 +0x51
Goroutine 6 (running) created at:
main.main()
/Users/kjk/src/go/src/github.com/kjk/go-cookbook/start-mysql-in-docker-go/t.go:9 +0xb4
0
==================
1
2
3
==================
WARNING: DATA RACE
Read at 0x00c420086018 by goroutine 6:
main.supplier()
/Users/kjk/src/go/src/github.com/kjk/go-cookbook/start-mysql-in-docker-go/t.go:37 +0x51
Previous write at 0x00c420086018 by main goroutine:
main.main()
/Users/kjk/src/go/src/github.com/kjk/go-cookbook/start-mysql-in-docker-go/t.go:14 +0x128
Goroutine 6 (running) created at:
main.main()
/Users/kjk/src/go/src/github.com/kjk/go-cookbook/start-mysql-in-docker-go/t.go:9 +0xb4
==================
4
As a rule of thumb, you should never pass channel as a pointer. Channel already is a pointer internally.
Stepping back a bit: I don't understand what you're trying to achieve.
I guess there's a reason you're trying to pass a channel as a pointer. The pattern of using channels in Go is: you create it once and you pass it around as value. You don't pass a pointer to it and you never modify it after creation.
In your example the problem is that you have a shared piece of memory (memory address pointed to by channel_ptr) and you write to that memory in one thread while some other thread reads it. That's data race.
It's not specific to a channel, you would have the same issue if it was pointer to an int and two threads were modifying the value of an int.
The codes are simple as below:
package main
import (
"fmt"
// "sync"
"time"
)
var count = uint64(0)
//var l sync.Mutex
func add() {
for {
// l.Lock()
// fmt.Println("Start ++")
count++
// l.Unlock()
}
}
func main() {
go add()
time.Sleep(1 * time.Second)
fmt.Println("Count =", count)
}
Cases：
Running the code without changing, u will get "Count = 0". Not expected??
Only uncomment line 16 "fmt.Println("Start ++")"; u will get output with lots of "Start ++" and some value with Count like "Count = 11111". Expected??
Only uncomment line 11 "var l sync.Mutex", line 15 "l.Lock()" and line 18 "l.Unlock()" and keep line 16 commented; u will get output like "Count = 111111111". Expected.
So... something wrong with my usage in shared variable...? My question:
Why case 1 had 0 with Count?
If case 1 is expected, why case 2 happened?
Env:
1. go version go1.8 linux/amd64
2. 3.10.0-123.el7.x86_64
3. CentOS Linux release 7.0.1406 (Core)
You have a data race on count. The results are undefined.
package main
import (
"fmt"
// "sync"
"time"
)
var count = uint64(0)
//var l sync.Mutex
func add() {
for {
// l.Lock()
// fmt.Println("Start ++")
count++
// l.Unlock()
}
}
func main() {
go add()
time.Sleep(1 * time.Second)
fmt.Println("Count =", count)
}
Output:
$ go run -race racer.go
==================
WARNING: DATA RACE
Read at 0x0000005995b8 by main goroutine:
runtime.convT2E64()
/home/peter/go/src/runtime/iface.go:255 +0x0
main.main()
/home/peter/gopath/src/so/racer.go:25 +0xb9
Previous write at 0x0000005995b8 by goroutine 6:
main.add()
/home/peter/gopath/src/so/racer.go:17 +0x5c
Goroutine 6 (running) created at:
main.main()
/home/peter/gopath/src/so/racer.go:23 +0x46
==================
Count = 42104672
Found 1 data race(s)
$
References:
Benign data races: what could possibly go wrong?
Without any synchronisation you have no guarantees at all.
There could be multiple reasons, why you see 'Count = 0' in your first case:
You have a multi processor or multi core system and one unit (cpu or core) is happily churning away at the for loop, while the other sleeps for one second and prints the line you are seeing afterwards. It would be completely legal for the compiler to generate machine code, which loads the value into some register and only ever increase that register in the for loop. The memory location can be updated, when the function is done with the variable. In case of an infinite for loop, that ist never. As you, the programmer told the compiler, that there is no contention about that variable, by omitting any synchronisation.
In your mutex version the synchronisation primitives tell the compiler,
that there might be some other thread taking the mutex, so it needs to write back the value from the register to the memory location before unlocking the mutex. At least one can think about it like that. What really happens that the unlock and a later lock operation introduce a happens before relation between the two go routines and this gives the guarantee, that we will see all writes to variables from before the unlock in one thread after the lock operation in the other thread, as described in go memory model locks howsoever this is implemented.
The Go runtime scheduler doesn't run the for loop at all, until the sleep in the main go routine is done. (Isn't likely, but, if I recall correctly, there is not guarantee that this isn't happening.) Sadly there is not much official documentation available about how the scheduler works in go, but it can only schedule a goroutine at certain points, it is not really preemptive. The consequences of this are severe. For example you could make your program run forever in some versions of go, by firing up as many go routines, as you had cores, doing endless for loops only incrementing a variable. There was no core left for the main go routine (which could end the program) and the scheduler can't preempt a go routine in an endless for loop doing only simple stuff, like incrementing a variable. I don't know, if that is changed now.
as others pointed out, that is a data race, google it and read up about it.
The difference between your versions there only line 16 is commented/uncommented is likely only because of run time, as printing to a terminal can be pretty slow.
For a correct program, you need to additionally lock the mutex after your sleep in you main program and before the fmt.Println and unlock it afterwards. But there can't be a deterministic expectation about the output, as the result will vary with machine/os/...