I have a hard time understanding concurrency/paralel. in my code I made a loop of 5 cycle. Inside of the loop I added the wg.Add(1), in total I have 5 Adds. Here's the code:
package main
import (
"fmt"
"sync"
)
func main() {
var list []int
wg := sync.WaitGroup{}
for i := 0; i < 5; i++ {
wg.Add(1)
go func(c *[]int, i int) {
*c = append(*c, i)
wg.Done()
}(&list, i)
}
wg.Wait()
fmt.Println(len(list))
}
The main func waits until all the goroutines finish but when I tried to print the length of slice I get random results. ex (1,3,etc) is there something that is missing for it to get the expected result ie 5 ?
is there something that is missing for it to get the expected result ie 5 ?
Yes, proper synchronization. If multiple goroutines access the same variable where at least one of them is a write, you need explicit synchronization.
Your example can be "secured" with a single mutex:
var list []int
wg := sync.WaitGroup{}
mu := &sync.Mutex{} // A mutex
for i := 0; i < 5; i++ {
wg.Add(1)
go func(c *[]int, i int) {
mu.Lock() // Must lock before accessing shared resource
*c = append(*c, i)
mu.Unlock() // Unlock when we're done with it
wg.Done()
}(&list, i)
}
wg.Wait()
fmt.Println(len(list))
This will always print 5.
Note: the same slice is read at the end to prints its length, yet we are not using the mutex there. This is because the use of waitgroup ensures that we can only get to that point after all goroutines that modify it have completed their job, so data race cannot occur there. But in general both reads and writes have to be synchronized.
See possible duplicates:
go routine not collecting all objects from channel
Server instances with multiple users
Why does this code cause data race?
How safe are Golang maps for concurrent Read/Write operations?
golang struct concurrent read and write without Lock is also running ok?
See related questions:
Can I concurrently write different slice elements
If I am using channels properly should I need to use mutexes?
Is it safe to read a function pointer concurrently without a lock?
Concurrent access to maps with 'range' in Go
Related
Like if I have a struct with an array and I want to do something like this
type Paxos struct {
peers []string
}
for _, peer := range px.peers {
\\do stuff
}
My routines/threads will never modify the peers array, just read from it. Peers is an array of server addresses, and servers may fail but that wouldn't affect the peers array (later rpc calls would just fail)
If no writes are involved, concurrent reads are always safe, regardless of the data structure. However, as soon as even a single concurrency-unsafe write to a variable is involved, you need to serialise concurrent access (both writes and reads) to the variable.
Moreover, you can safely write to elements of a slice or an array under the condition that no more than one goroutine write to a given element.
For instance, if you run the following programme with the race detector on, it's likely to report a race condition, because multiple goroutines concurrently modify variable results without precautions:
package main
import (
"fmt"
"sync"
)
func main() {
const n = 8
var results []int
var wg sync.WaitGroup
wg.Add(n)
for i := 0; i < n; i++ {
i := i
go func() {
defer wg.Done()
results = append(results, square(i))
}()
}
wg.Wait()
fmt.Println(results)
}
func square(i int) int {
return i * i
}
However, the following programme contains no such no synchronization bug, because each element of the slice is modified by a single goroutine:
package main
import (
"fmt"
"sync"
)
func main() {
const n = 8
results := make([]int, n)
var wg sync.WaitGroup
wg.Add(n)
for i := 0; i < n; i++ {
i := i
go func() {
defer wg.Done()
results[i] = square(i)
}()
}
wg.Wait()
fmt.Println(results)
}
func square(i int) int {
return i * i
}
Yes, reads are thread-safe in Go and virtually all other languages. You're just looking up an address in memory and seeing what is there. If nothing is attempting to modify that memory, then you can have as many concurrent reads as you'd like.
I am trying to iterate a loop and call go routine on an anonymous function and adding a waitgroup on each iteration. And passing a string to same anonymous function and appending the value to slice a. Since I am looping 10000 times length of the slice is expected to be 10000. But I see random numbers. Not sure what is the issue. Can anyone help me fix this problem?
Here is my code snippet
import (
"fmt"
"sync"
)
func main() {
var wg = new(sync.WaitGroup)
var a []string
for i := 0; i <= 10000; i++ {
wg.Add(1)
go func(s string) {
a = append(a, s)
wg.Done()
}("MaxPayne")
}
wg.Wait()
fmt.Println(len(a))
}
Notice how appending a slice, you actually make a new slice, and then assign it back to the slice variable. So you have un-controlled concurrent writing to the variable a. Concurrent writing to the same value is not safe in Go (and most languages). In order to make it safe, you can serialize the writes with a mutex.
Try:
var lock sync.Mutex
var a []string
and
lock.Lock()
a = append(a, s)
lock.Unlock()
For more information about how a mutex works, see the tour and the sync package.
Here is a pattern to achieve a similar result, but without needing a mutex and still being safe.
package main
import (
"fmt"
"sync"
)
func main() {
const sliceSize = 10000
var wg = new(sync.WaitGroup)
var a = make([]string, sliceSize)
for i := 0; i < sliceSize; i++ {
wg.Add(1)
go func(s string, index int) {
a[index] = s
wg.Done()
}("MaxPayne", i)
}
wg.Wait()
}
This isn't exactly the same as your other program, but here's what it does.
Create a slice that already has the desired size of 10,000 (each element is an empty string at this point)
For each number 0...9999, create a new goroutine that is given a specific index to write a specific string into
After all goroutines have exited and the waitgroup is done waiting, then we know that each index of the slice has successfully been filled.
The memory access is now safe even without a mutex, because each goroutine is only writing to it's respective index (and each goroutine gets a unique index). Therefore, none of these concurrent memory writes conflict with each other. After initially creating the slice with the desired size, the variable a itself doesn't need to be assigned to again, so the original memory race is eliminated.
From the Go docs:
A RWMutex is a reader/writer mutual exclusion lock. The lock can be
held by an arbitrary number of readers or a single writer.
This introductory sentence is never followed by a definition of what a reader and writer may and may not do. Thus, I wonder whether a certain stretch of the definition is possible.
Suppose we have a set of many goroutines; let's call it S. Each of the goroutines in S has its own resources which it reads from and writes to frequently. Suppose I have one extra goroutine R which routinely wants to scrape the state of the resources written by the goroutines in S. Let's explore the two more obvious solutions to see where I am going.
We can use a single mutex. However, the goroutines in S would needlessly compete for the lock, since each goroutine from S accesses only its own resources anyway.
Use one mutex for each goroutine in S. This eliminates needless competition and even gives R the option to decide whether it needs to have locked all mutexex at once or merely each mutex at least once at some point in time. Requires some extra work however.
So my third idea is this: Let the goroutines in S take the read lock of a sync.RWMutex when they want to read or write (i.e. a generalized inclusive lock) and let R take the write lock of that mutex when it wants to read (i.e. a generalized exclusive lock). In other words:
Is it safe to use the read lock of sync.RWMutex for writing and its write lock for reading if the writes never race with each other by design?
The answer of your question is Yes, just like Volker said.
I would give you a more idiomatic Go solution:
every resources of goroutines in S take a sync.RWMutex
the goroutines in S read after sync.RWMutex.Rlock()
the goroutines in S write after sync.RWMutex.Lock()
extra goroutine R read after sync.RWMutex.Rlock()
I think you just go into the wrong region, it should be a simple problem. :)
If I misunderstand you, tell me.
Answering myself because of all the confusion.
Yes, it is safe to use the read lock of sync.RWMutex for writing and its write lock for reading if the writes never race with each other.
Example:
package main
import (
"fmt"
"sync"
"time"
)
func main() {
m := &sync.RWMutex{}
wg := &sync.WaitGroup{}
maxWriters := 5
wg.Add(maxWriters+1)
allData := make([]uint64, maxWriters)
go showAccumulatedData(m, wg, allData)
for i := 0; i < maxWriters; i++ {
go writeData(m, wg, &allData[i])
}
wg.Wait()
fmt.Println(accumulateData(m, allData))
}
func writeData(m *sync.RWMutex, wg *sync.WaitGroup, d *uint64) {
for i := 0; i < 1000; i++ {
m.RLock()
*d++ // Write during read-lock.
m.RUnlock()
time.Sleep(time.Millisecond)
}
wg.Done()
}
func showAccumulatedData(m *sync.RWMutex, wg *sync.WaitGroup, allData []uint64) {
for i := 0; i < 15; i++ {
fmt.Println(accumulateData(m, allData))
time.Sleep(time.Millisecond*50)
}
wg.Done()
}
func accumulateData(m *sync.RWMutex, allData []uint64) uint64 {
var accumulator uint64
m.Lock()
for _, v := range allData {
accumulator += v // Read during write-lock.
}
m.Unlock()
return accumulator
}
I'm having difficulty getting go concurrency to work correctly. I'm working with data loaded from an XML Data Source. Once I load the data into memory, i loop through the XML elements and perform an operation. The code prior to the concurrency addition has been tested and functional, and I don't believe it has any influence on the concurrency addition. I have 2 failed attempts at concurrency implementations, both with different outputs. I used locking because i dont want to enter a race condition.
For this implementation, it never enters the goroutine.
var mu sync.Mutex
// length is 197K
for i:=0;i<len(listings.Listings);i++{
go func(){
mu.Lock()
// code execution (tested prior to adding concurrency and locking)
mu.Unlock()
}()
}
For this implementation using waitGroups, a runtime out of memory occurs
var mu sync.Mutex
var wg sync.WaitGroup
// length is 197K
for i:=0;i<len(listings.Listings);i++{
wg.Add(1)
go func(){
mu.Lock()
// code execution (tested prior to adding concurrency and locking and wait group)
wg.Done()
mu.Unlock()
}()
}
wg.Wait()
I'm not really sure what's going on and could use some assistance.
You don't need Mutex here if you want to make it concurrent
197K goroitines are a lot, try lower amount of goroutines. You can accomplish it by creating N goroutines, when each of them is listening to the same channel.
https://play.golang.org/p/s4e0YyHdyPq
package main
import (
"fmt"
"sync"
)
type Listing struct{}
func main() {
var (
wg sync.WaitGroup
concurrency = 100
)
c := make(chan Listing)
wg.Add(concurrency)
for i := 0; i < concurrency; i++ {
go func(ci <-chan Listing) {
for l := range ci {
// code, l is a single Listing
fmt.Printf("%v", l)
}
wg.Done()
}(c)
}
// replace with your var
listings := []Listing{Listing{}}
for _, l := range listings {
c <- l
}
close(c)
wg.Wait()
}
This might be a rookies mistake. I have a slice with a string value and a map of channels. For each string in the slice, a channel is created and a map entry is created for it, with the string as key.
I watch the channels and pass a value to one of them, which is never found.
package main
import (
"fmt"
"time"
)
type TestStruct struct {
Test string
}
var channelsMap map[string](chan *TestStruct)
func main() {
stringsSlice := []string{"value1"}
channelsMap := make(map[string](chan *TestStruct))
for _, value := range stringsSlice {
channelsMap[value] = make(chan *TestStruct, 1)
go watchChannel(value)
}
<-time.After(3 * time.Second)
testStruct := new(TestStruct)
testStruct.Test = "Hello!"
channelsMap["value1"] <- testStruct
<-time.After(3 * time.Second)
fmt.Println("Program ended")
}
func watchChannel(channelMapKey string) {
fmt.Println("Watching channel: " + channelMapKey)
for channelValue := range channelsMap[channelMapKey] {
fmt.Printf("Channel '%s' used. Passed value: '%s'\n", channelMapKey, channelValue.Test)
}
}
Playground link: https://play.golang.org/p/IbucTqMjdGO
Output:
Watching channel: value1
Program ended
How do I execute something when the message is fed into the channel?
There are many problems with your approach.
The first one is that you're redeclaring ("shadowing") the global
variable channelsMap in your main function.
(Had you completed at least some
most basic intro to Go, you should have had no such problem.)
This means that your watchChannel (actually, all the goroutines which execute that function) read the global channelsMap while your main function writes to its local channelsMap.
What happens next, is as follows:
The range statement
in the watchChannel has a simple
map lookup expression as its source—channelsMap[channelMapKey].
In Go, this form of map lookup
never fails, but if the map has no such key (or if the map is not initialized, that is, it's nil), the so-called
"zero value"
of the appropriate type is returned.
Since the global channelsMap is always empty, any call to watchChannel performs a map lookup which always returns
the zero value of type chan *TestStruct.
The zero value for any channel is nil.
The range statement executed over a nil channel
produces zero iterations.
In other words, the for loop in watchChannel always executes
zero times.
The more complex problem, still, is not shadowing of the global variable but rather the complete absense of synchronization between the goroutines. You're using "sleeping" as a sort of band-aid in an attempt to perform implicit synchronization between goroutines
but while this does appear to be okay judged by so-called
"common sense", it's not going to work in practice for two
reasons:
Sleeping is always a naïve approach to synchronization as it solely depens of the fact all the goroutines will run relatively freely and uncontended. This is far from being true in many (if not most) production settings and hence is always the reason for subtle bugs. Don't ever do that again, please.
Nothing in the Go memory model
says that waiting against wall-clock timing is considered by the runtime as establishing the order on how execution of different goroutines relate to each other.
There exist various ways to synchronize execution between goroutines. Basically they amount to sends and receives over channels and using the types provided by the sync package.
In your particular case the simplest approach is probably using the sync.WaitGroup type.
Here is what we would
have after fixing the problems explained above:
- Initialize the map variable right at the point of its
definition and not mess with it in main.
- Use sync.WaitGroup to make main properly wait for all
the goroutines it spawned to singal they're done:
package main
import (
"fmt"
"sync"
)
type TestStruct struct {
Test string
}
var channelsMap = make(map[string](chan *TestStruct))
func main() {
stringsSlice := []string{"value1"}
var wg sync.WaitGroup
wg.Add(len(stringsSlice))
for _, value := range stringsSlice {
channelsMap[value] = make(chan *TestStruct, 1)
go watchChannel(value, &wg)
}
testStruct := new(TestStruct)
testStruct.Test = "Hello!"
channelsMap["value1"] <- testStruct
wg.Wait()
fmt.Println("Program ended")
}
func watchChannel(channelMapKey string, wg *sync.WaitGroup) {
defer wg.Done()
fmt.Println("Watching channel: " + channelMapKey)
for channelValue := range channelsMap[channelMapKey] {
fmt.Printf("Channel '%s' used. Passed value: '%s'\n", channelMapKey, channelValue.Test)
}
}
The next two problems with your code become apparent once we will
have fixed the former two—after you make the "watcher" goroutines
use the same map variable as the goroutine running main, and
make the latter properly wait for the watchers:
There is a data race
over the map variable between the
code which updates the map after the for loop spawning the
watcher goroutines ended and the code which accesses this
variable in all the watcher goroutines.
There is a deadlock
between the watcher goroutines and the main goroutine which waits for them to complete.
The reason for the deadlock is that the watcher goroutines
never receive any signal they have to quit processing and
hence are stuck forever trying to read from their respective
channels.
The ways to fix these two new problems are simple but they
might actually "break" your original idea of structuring
your code.
First, I'd remove the data race by simply making the watchers
not access the map variable. As you can see, each call to
watchChannel receives a single value to use as the key to
read a value off the shared map, and hence each watcher always
reads a single value exactly once during its run time.
The code would become much clearer if we remove this extra
map access altogether and instead pass the appropriate channel
value directly to each watcher.
A nice byproduct of this is that we do not need a global
map variable anymore.
Here's what we'll get:
package main
import (
"fmt"
"sync"
)
type TestStruct struct {
Test string
}
func main() {
stringsSlice := []string{"value1"}
channelsMap := make(map[string](chan *TestStruct))
var wg sync.WaitGroup
wg.Add(len(stringsSlice))
for _, value := range stringsSlice {
channelsMap[value] = make(chan *TestStruct, 1)
go watchChannel(value, channelsMap[value], &wg)
}
testStruct := new(TestStruct)
testStruct.Test = "Hello!"
channelsMap["value1"] <- testStruct
wg.Wait()
fmt.Println("Program ended")
}
func watchChannel(channelMapKey string, ch <-chan *TestStruct, wg *sync.WaitGroup) {
defer wg.Done()
fmt.Println("Watching channel: " + channelMapKey)
for channelValue := range ch {
fmt.Printf("Channel '%s' used. Passed value: '%s'\n", channelMapKey, channelValue.Test)
}
}
Okay, we still have the deadlock.
There are multiple approaches to solving this but they depend
on the actual circumstances, and with this toy example, any
attempt to iterate over at least a subset of them would just
muddle the waters.
Instead, let's employ the simplest one for this case: closing
a channel makes any pending receive operation on it immediately
unblock and produce the zero value for the channel's type.
For a channel being iterated over using the range statement
it simply means the stamement terminates without producing any
value from the channel.
In other words, let's just close all the channels to unblock
the range statements being run by the watcher goroutines
and then wait for these goroutines to report their completion via the wait group.
To not make the answer overly long, I also added programmatic initialization of the string slice to make the example more interesting by having multiple watchers—not just a single one—actually do useful work:
package main
import (
"fmt"
"sync"
)
type TestStruct struct {
Test string
}
func main() {
var stringsSlice []string
channelsMap := make(map[string](chan *TestStruct))
for i := 1; i <= 10; i++ {
stringsSlice = append(stringsSlice, fmt.Sprintf("value%d", i))
}
var wg sync.WaitGroup
wg.Add(len(stringsSlice))
for _, value := range stringsSlice {
channelsMap[value] = make(chan *TestStruct, 1)
go watchChannel(value, channelsMap[value], &wg)
}
for _, value := range stringsSlice {
testStruct := new(TestStruct)
testStruct.Test = fmt.Sprint("Hello! ", value)
channelsMap[value] <- testStruct
}
for _, ch := range channelsMap {
close(ch)
}
wg.Wait()
fmt.Println("Program ended")
}
func watchChannel(channelMapKey string, ch <-chan *TestStruct, wg *sync.WaitGroup) {
defer wg.Done()
fmt.Println("Watching channel: " + channelMapKey)
for channelValue := range ch {
fmt.Printf("Channel '%s' used. Passed value: '%s'\n", channelMapKey, channelValue.Test)
}
}
Playground link.
As you can see, there are things you should actually learn
about in way more greater detail before embarking on working with
concurrency.
I'd recommend to proceed in the following order:
The Go tour would make you accustomed with the bare bones of concurrency.
The Go Programming Language has two chapters dedicated to providing the readers with a gentle introduction with tackling concurrency both using channels and the types from the sync package.
Concurrency In Go goes on with presenting more hard-core details of how one deals with concurrency in Go, including advanced topics approaching the real-world problems concurrent programs face in production—such as ways to rate-limit incoming requests.
The shadowing in main of channelsMap mentioned above was a critical bug, but aside from that, the program was playing "Russian roulette" with the calls to time.After so that main wouldn't finish before the watcher goroutines did. This is unstable and unreliable, so I recommend the following approach using a channel to signal when all watcher goroutines are done:
package main
import (
"fmt"
)
type TestStruct struct {
Test string
}
var channelsMap map[string](chan *TestStruct)
func main() {
stringsSlice := []string{"value1", "value2", "value3"}
structsSlice := []TestStruct{
{"Hello1"},
{"Hello2"},
{"Hello3"},
}
channelsMap = make(map[string](chan *TestStruct))
// Signal channel to wait for watcher goroutines.
done := make(chan struct{})
for _, s := range stringsSlice {
channelsMap[s] = make(chan *TestStruct)
// Give watcher goroutines the signal channel.
go watchChannel(s, done)
}
for _, ts := range structsSlice {
for _, s := range stringsSlice {
channelsMap[s] <- &ts
}
}
// Close the channels so watcher goroutines can finish.
for _, s := range stringsSlice {
close(channelsMap[s])
}
// Wait for all watcher goroutines to finish.
for range stringsSlice {
<-done
}
// Now we're really done!
fmt.Println("Program ended")
}
func watchChannel(channelMapKey string, done chan<- struct{}) {
fmt.Println("Watching channel: " + channelMapKey)
for channelValue := range channelsMap[channelMapKey] {
fmt.Printf("Channel '%s' used. Passed value: '%s'\n", channelMapKey, channelValue.Test)
}
done <- struct{}{}
}
(Go Playground link: https://play.golang.org/p/eP57Ru44-NW)
Of importance is the use of the done channel to let watcher goroutines signal that they're finished to main. Another critical part is the closing of the channels once you're done with them. If you don't close them, the range loops in the watcher goroutines will never end, waiting forever. Once you close the channel, the range loop exits and the watcher goruoutine can send on the done channel, signaling that it has finished working.
Finally, back in main, you have to receive on the done channel once for each watcher goroutine you created. Since the number of watcher goroutines is equal to the number of items in stringsSlice, you simply range over stringsSlice to receive the correct amount of times from the done channel. Once that's finished, the main function can exit with a guarantee that all watchers have finished.