I am trying to familiarize with go routines. I have written the following simple program to store the squares of numbers from 1-10 in a map.
func main() {
squares := make(map[int]int)
var wg sync.WaitGroup
for i := 1; i <= 10; i++ {
go func(n int, s map[int]int) {
s[n] = n * n
}(i, squares)
}
wg.Wait()
fmt.Println("Squares::: ", squares)
}
At the end, it prints an empty map. But in go, maps are passed by references. Why is it printing an empty map?
As pointed out in the comments, you need to synchronize access to the map and your usage of sync.WaitGroup is incorrect.
Try this instead:
func main() {
squares := make(map[int]int)
var lock sync.Mutex
var wg sync.WaitGroup
for i := 1; i <= 10; i++ {
wg.Add(1) // Increment the wait group count
go func(n int, s map[int]int) {
lock.Lock() // Lock the map
s[n] = n * n
lock.Unlock()
wg.Done() // Decrement the wait group count
}(i, squares)
}
wg.Wait()
fmt.Println("Squares::: ", squares)
}
sync.Map is what you are actually looking for, modified the code to suit your usecase here,
https://play.golang.org/p/DPLHiMsH5R8
P.S. Had to add some sleep so that the program does not finish before all the go routines are called.
Related
I want to make X number of goroutines to update CountValue using parallelism (numRoutines are as much as how many CPU you have).
Solution 1:
func count(numRoutines int) (countValue int) {
var mu sync.Mutex
k := func(i int) {
mu.Lock()
defer mu.Unlock()
countValue += 5
}
for i := 0; i < numRoutines; i++ {
go k(i)
}
It becomes a data race and the returned countValue = 0.
Solution 2:
func count(numRoutines int) (countValue int) {
k := func(i int, c chan int) {
c <- 5
}
c := make(chan int)
for i := 0; i < numRoutines; i++ {
go k(i, c)
}
for i := 0; i < numRoutines; i++ {
countValue += <- c
}
return
}
I did a benchmark test on it and doing a sequential addition would work faster than using goroutines. I think it's because I have two for loops here as when I put countValue += <- c inside the first for loop, the code runs faster.
Solution 3:
func count(numRoutines int) (countValue int) {
var wg sync.WaitGroup
c := make(chan int)
k := func(i int) {
defer wg.Done()
c <- 5
}
for i := 0; i < numShards; i++ {
wg.Add(1)
go k(i)
}
go func() {
for i := range c {
countValue += i
}
}()
wg.Wait()
return
}
Still a race count :/
Is there any way better to do this?
There definitely is a better way to safely increment a variable: using sync/atomic:
import "sync/atomic"
var words int64
k := func() {
_ = atomic.AddInt64(&words, 5) // increment atomically
}
Using a channel basically eliminates the need for a mutex, or takes away the the risk of concurrent access to the variable itself, and a waitgroup here is just a bit overkill
Channels:
words := 0
done := make(chan struct{}) // or use context
ch := make(chan int, numRoutines) // buffer so each routine can write
go func () {
read := 0
for i := range ch {
words += 5 // or use i or something
read++
if read == numRoutines {
break // we've received data from all routines
}
}
close(done) // indicate this routine has terminated
}()
for i := 0; i < numRoutines; i++ {
ch <- i // write whatever value needs to be used in the counting routine on the channel
}
<- done // wait for our routine that increments words to return
close(ch) // this channel is no longer needed
fmt.Printf("Counted %d\n", words)
As you can tell, the numRoutines no longer is the number of routines, but rather the number of writes on the channel. You can move that to individual routines, still:
for i := 0; i < numRoutines; i++ {
go func(ch chan<- int, i int) {
// do stuff here
ch <- 5 * i // for example
}(ch, i)
}
WaitGroup:
Instead of using a context that you can cancel, or a channel, you can use a waitgroup + atomic to get the same result. The easiest way IMO to do so is to create a type:
type counter struct {
words int64
}
func (c *counter) doStuff(wg *sync.WaitGroup, i int) {
defer wg.Done()
_ = atomic.AddInt64(&c.words, i * 5) // whatever value you need to add
}
func main () {
cnt := counter{}
wg := sync.WaitGroup{}
wg.Add(numRoutines) // create the waitgroup
for i := 0; i < numRoutines; i++ {
go cnt.doStuff(&wg, i)
}
wg.Wait() // wait for all routines to finish
fmt.Println("Counted %d\n", cnt.words)
}
Fix for your third solution
As I mentioned in the comment: your third solution is still causing a race condition because the channel c is never closed, meaning the routine:
go func () {
for i := range c {
countValue += i
}
}()
Never returns. The waitgroup also only ensures that you've sent all values on the channel, but not that the countValue has been incremented to its final value. The fix would be to either close the channel after wg.Wait() returns so the routine can return, and add a done channel that you can close when this last routine returns, and add a <-done statement before returning.
func count(numRoutines int) (countValue int) {
var wg sync.WaitGroup
c := make(chan int)
k := func(i int) {
defer wg.Done()
c <- 5
}
for i := 0; i < numShards; i++ {
wg.Add(1)
go k(i)
}
done := make(chan struct{})
go func() {
for i := range c {
countValue += i
}
close(done)
}()
wg.Wait()
close(c)
<-done
return
}
This adds some clutter, though, and IMO is a bit messy. It might just be easier to just move the wg.Wait() call to a routine:
func count(numRoutines int) (countValue int) {
var wg sync.WaitGroup
c := make(chan int)
// add wg as argument, makes it easier to move this function outside of this scope
k := func(wg *sync.WaitGroup, i int) {
defer wg.Done()
c <- 5
}
wg.Add(numShards) // increment the waitgroup once
for i := 0; i < numShards; i++ {
go k(&wg, i)
}
go func() {
wg.Wait()
close(c) // this ends the loop over the channel
}()
// just iterate over the channel until it is closed
for i := range c {
countValue += i
}
// we've added all values to countValue
return
}
The code below starts a few workers. Each worker receives a value via a channel which is added to a map where the key is the worker ID and value is the number received. Finally, when I add all the values received, I should get an expected result (in this case 55 because that is what you get when you add from 1..10). In most cases, I am not seeing the expected output. What am I doing wrong here? I do not want to solve it by adding a sleep. I would like to identify the issue programmatically and fix it.
type counter struct {
value int
count int
}
var data map[string]counter
var lock sync.Mutex
func adder(wid string, n int) {
defer lock.Unlock()
lock.Lock()
d := data[wid]
d.count++
d.value += n
data[wid] = d
return
}
func main() {
fmt.Println(os.Getpid())
data = make(map[string]counter)
c := make(chan int)
for w := 1; w <= 3; w++ { //starting 3 workers here
go func(wid string) {
data[wid] = counter{}
for {
v, k := <-c
if !k {
continue
}
adder(wid, v)
}
}(strconv.Itoa(w)) // worker is given an ID
}
time.Sleep(1 * time.Second) // If this is not added, only one goroutine is recorded.
for i := 1; i <= 10; i++ {
c <- i
}
close(c)
total := 0
for i, v := range data {
fmt.Println(i, v)
total += v.value
}
fmt.Println(total)
}
Your code has two significant races:
The initialization of data[wid] = counter{} is not synchronized with other goroutines that may be reading and rewriting data.
The worker goroutines do not signal when they are done modifying data, which means your main goroutine may read data before they finish writing.
You also have a strange construct:
for {
v, k := <-c
if !k {
continue
}
adder(wid, v)
}
k will only be false when the channel c is closed, after which the goroutine spins as much as it can. This would be better written as for v := range c.
To fix the reading code in the main goroutine, we'll use the more normal for ... range c idiom and add a sync.WaitGroup, and have each worker invoke Done() on the wait-group. The main goroutine will then wait for them to finish. To fix the initialization, we'll lock the map (there are other ways to do this, e.g., to set up the map before starting any of the goroutines, or to rely on the fact that empty map slots read as zero, but this one is straightforward). I also took out the extra debug. The result is this code, also available on the Go Playground.
package main
import (
"fmt"
// "os"
"strconv"
"sync"
// "time"
)
type counter struct {
value int
count int
}
var data map[string]counter
var lock sync.Mutex
var wg sync.WaitGroup
func adder(wid string, n int) {
defer lock.Unlock()
lock.Lock()
d := data[wid]
d.count++
d.value += n
data[wid] = d
}
func main() {
// fmt.Println(os.Getpid())
data = make(map[string]counter)
c := make(chan int)
for w := 1; w <= 3; w++ { //starting 3 workers here
wg.Add(1)
go func(wid string) {
lock.Lock()
data[wid] = counter{}
lock.Unlock()
for v := range c {
adder(wid, v)
}
wg.Done()
}(strconv.Itoa(w)) // worker is given an ID
}
for i := 1; i <= 10; i++ {
c <- i
}
close(c)
wg.Wait()
total := 0
for i, v := range data {
fmt.Println(i, v)
total += v.value
}
fmt.Println(total)
}
(This can be improved easily, e.g., there's no reason for wg to be global.)
Well, I like #torek's answer but I wanted to post this answer as it contains a bunch of improvements:
Reduce the usage of locks (For such simple tasks, avoid locks. If you benchmark it, you'll notice a good difference because my code uses the lock only numworkers times).
Improve the naming of variables.
Remove usage of global vars (Use of global vars should always be as minimum as possible).
The following code adds a number from minWork to maxWork using numWorker spawned goroutines.
package main
import (
"fmt"
"sync"
)
const (
bufferSize = 1 // Buffer for numChan
numworkers = 3 // Number of workers doing addition
minWork = 1 // Sum from [minWork] (inclusive)
maxWork = 10000000 // Sum upto [maxWork] (inclusive)
)
// worker stats
type worker struct {
workCount int // Number of times, worker worked
workDone int // Amount of work done; numbers added
}
// workerMap holds a map for worker(s)
type workerMap struct {
mu sync.Mutex // Guards m for safe, concurrent r/w
m map[int]worker // Map to hold worker id to worker mapping
}
func main() {
var (
totalWorkDone int // Total Work Done
wm workerMap // WorkerMap
wg sync.WaitGroup // WaitGroup
numChan = make(chan int, bufferSize) // Channel for nums
)
wm.m = make(map[int]worker, numworkers)
for wid := 0; wid < numworkers; wid++ {
wg.Add(1)
go func(id int) {
var wk worker
// Wait for numbers
for n := range numChan {
wk.workCount++
wk.workDone += n
}
// Fill worker stats
wm.mu.Lock()
wm.m[id] = wk
wm.mu.Unlock()
wg.Done()
}(wid)
}
// Send numbers for addition by multiple workers
for i := minWork; i <= maxWork; i++ {
numChan <- i
}
// Close the channel
close(numChan)
// Wait for goroutines to finish
wg.Wait()
// Print stats
for k, v := range wm.m {
fmt.Printf("WorkerID: %d; Work: %+v\n", k, v)
totalWorkDone += v.workDone
}
// Print total work done by all workers
fmt.Printf("Work Done: %d\n", totalWorkDone)
}
Really new to the world of go and trying to learn by doing. I want to see how I would use go routines to add items to a "queue" in one go routine while another go routine listens for the queue and processes things as they enter. In this case I have a func that adds a defined amount of items to a int[] while the other attempts to print them as they are added.. I assume a flag will need to be sent to signal that one go routine has stopped adding items to the queue. I apologize for the noob question but am struggling understanding the new terminology and syntax.
package main
import "fmt"
func printFromSlice(c chan []int){
i := <-c
// wait for item to be added to channel
fmt.Print(i)
}
func addToSlice (c chan []int, i int) {
for n := 0; n < i; n++{
c <- n //I know this isnt right
}
// How do we signal this is complete??
}
func main(){
c := make(chan []int)
//add things to []i int
go addToSlice(c, 25)
//do something from []int
go printFromSlice(c)
}
**UPDATE**
modified to use the following code however now it will only execute to a single print form the ~printFromSlice` function before it closes out...
package main
func printFromSlice(c chan int){
i := <-c
// wait for item to be added to channel
println("read")
println(i)
}
func addToSlice (c chan int) {
for n := 0; n < 100; n++{
println("add")
println(n)
c <- n
}
close(c)
}
func main(){
println("starting...")
c := make(chan int)
//add things to []i int
go addToSlice(c)
//do something from []int
go printFromSlice(c)
<-c
}
You need to add a sync.WaitGroup to block the main thread until the two goroutine could finish. You could refer to this Go by Example.
package main
import "sync"
func printFromSlice(c chan int, wg *sync.WaitGroup) {
defer wg.Done() // Decrement the waitgroup counter by 1 after `printFromSlice` returns
// wait for item to be added to channel
// i := <-c // this only waits / blocks 1 time
// For each message arriving at c, read and print
for i := range c { // this would read all messages until channel is closed
println("read")
println(i)
}
}
func addToSlice(c chan int, wg *sync.WaitGroup) {
defer wg.Done() // Decrement the waitgroup counter by 1 after `addToSlice` returns
for n := 0; n < 100; n++ {
println("add")
println(n)
c <- n
}
close(c)
}
func main() {
var wg sync.WaitGroup
println("starting...")
c := make(chan int)
wg.Add(2) // Adds 2 to the waitgroup counter
//add things to []i int
go addToSlice(c, &wg) // Pass the wait group as reference so they can call wg.Done()
//do something from []int
go printFromSlice(c, &wg) // Pass the wait group as reference so they can call wg.Done()
// <-c // No need for this to block the code
wg.Wait() // Waits / blocks until waitgroup counter is 0
}
I am practicing golang by doing coding problems on LeetCode. I am trying to solve a simple sudoku puzzle (it just validates the board). No rows with same digits, no columns with same digits, no 3x3 blocks with same digits. I am trying to use concurrency for learning Go Routines/Channels/Etc...
I can't get the waitgroup to finalize
import (
"sync"
"fmt"
)
func isValidSlice(slice []byte, results chan<- bool, wg *sync.WaitGroup) {
fmt.Println(slice)
seen := make(map[byte]bool)
for _,val := range(slice) {
if seen[val] {
if val != '.'{
results <- false
defer wg.Done()
return
}
} else {
seen[val] = true
}
}
results <- true
defer wg.Done()
}
func isValidSudoku(board [][]byte) bool {
// Channel to receive solution
c := make(chan bool)
// Number of routines that will run (9 for rows, 9 for cols, 9 for 3x3 blocks)
var wg sync.WaitGroup
// Check every row
for x:= 0; x < 9; x++{
wg.Add(1)
go isValidSlice(append([]byte{}, board[x]...), c, &wg)
}
for y:= 0; y < 9; y++{
wg.Add(1)
go isValidSlice(append([]byte{}, board[0:9][y]...), c, &wg)
}
// Check every 3x3 block
for x:= 0; x <= 6; x += 3{
for y := 0; y <= 6; y += 3{
block_digits := append([]byte{}, board[x][y:y+3]...)
block_digits = append(block_digits, board[x+1][y:y+3]...)
block_digits = append(block_digits, board[x+2][y:y+3]...)
wg.Add(1)
go isValidSlice(block_digits, c, &wg)
}
}
fmt.Println("got here")
wg.Wait()
fmt.Println("never got here")
for result := range c{
if !result{
return false
}
}
return true
}
I'm expecting the wg.Wait() lock to release and code to move forward. Then I'm expecting to one of the results in the channel to be false and return false if so. Otherwise, after all elements in the channel are traverse and no false was found, I would expect a True.
Your goroutines cannot call wg.Done() because they all wait to add their value in the channel. But since you only consume the values from the channel after wg.Wait(), all goroutines but one never get to call wg.Done().
You actually don't need a WaitGroup, just remove it.
Other comments:
You should move defer wg.Done() to the first line of isValidSlice. Calling defer on the last line of your function does not make much sense.
You only need a WaitGroup if you want to close the channel properly, you can do that in an additional goroutine, see below for an example on how to do it.
func isValidSudoku(board [][]byte) bool {
// ...
fmt.Println("got here")
go func(){
wg.Wait()
close(c)
}()
fmt.Println("never got here")
for result := range c{
if !result{
go func(){
for _ := range c {
}
}()
return false
}
}
return true
}
func parallelSum (c chan int){
sum := 0
for i :=1 ; i< 100;i++{
go func(i int){
sum += i
}(i)
}
time.Sleep(1*time.Second)
c <- sum
}
I'm trying to learn the parallel ability to speed up things like OpenMP. And here is an example of the intended summing up parallel loop in Go, this function runs as a goroutine.
Note that the variable sum is not a channel here, so does this mean the variable sum access inside the for loop is a blocked operation? Is it now efficient enough? Is there a better solution?
I knew the channel feature was designed for this, my obviously wrong implement below can compile, but with 100 runtime errors like following.
goroutine 4 [chan receive]:
main.parallelSumError(0xc0000180c0)
/home/tom/src/goland_newproject/main.go:58 +0xb4 //line 58 : temp := <-sum
created by main.main
/home/tom/src/goland_newproject/main.go:128 +0x2ca //line 128: go parallelSumError(pcr), the calling function
So what's the problem here? it seems summing is not a good example for paralleled for-loop, but actually I wish to know how to use channel inside paralleled for-loop.
func parallelSum (c chan int){
sum := make(chan int)
for i :=1 ; i< 100;i++{
go func(i int){
temp := <- sum //error here why?
temp += i
sum <- temp
}(i)
}
time.Sleep(1*time.Second)
temp := <-sum
c <- temp
}
both with the same main function
func main(){
pc := make(chan int)
go parallelSum(pc)
result = <- pc
fmt.Println("parallel result:", result)
}
I don't like the idea of summing numbers through channels. I'd rather use something classical like sync.Mutex or atomic.AddUint64. But, at least, I made your code working.
We aren't able to pass a value from one channel to another (I added temp variable). Also, there is sync.WaitGroup and other stuff.
But I still don't like the idea of the code.
package main
import (
"fmt"
"sync"
)
func main() {
pc := make(chan int)
go parallelSum(pc)
result := <- pc
fmt.Println("parallel result:", result)
}
func parallelSum (c chan int){
sum := make(chan int)
wg := sync.WaitGroup{}
wg.Add(100)
for i :=1 ; i <= 100;i++{
go func(i int){
temp := <- sum
temp += i
wg.Done()
sum <- temp
}(i)
}
sum <- 0
wg.Wait()
temp := <- sum
c <- temp
}
When using go routines (i.e. go foo()), it is preferable to use communication over memory-sharing. In this matter, as you mention, channels are the golang way to handle communication.
For your specific application, the paralleled sum similar to OpenMP, it would be preferable to detect the number of CPUs and generate as many routines as wished:
package main
import (
"fmt"
"runtime"
)
func main() {
numCPU := runtime.NumCPU()
sumc := make(chan int, numCPU)
valuec := make(chan int)
endc := make(chan interface{}, numCPU)
// generate go routine per cpu
for i := 0; i < numCPU; i++ {
go sumf(sumc, valuec, endc)
}
// generate values and pass it through the channels
for i := 0; i < 100; i++ {
valuec <- i
}
// tell go routines to end up when they are done
for i := 0; i < numCPU; i++ {
endc <- nil
}
// sum results
sum := 0
for i := 0; i < numCPU; i++ {
procSum := <-sumc
sum += procSum
}
fmt.Println(sum)
}
func sumf(sumc, valuec chan int, endc chan interface{}) {
sum := 0
for {
select {
case i := <-valuec:
sum += i
case <-endc:
sumc <- sum
return
}
}
}
Hopefully, this helps.