I have the following program, I am new to gorountine, what I want to test is simple, I am calling a gorountine in a loop 100 times, it there is one time failure, the entire program fails, otherwise succeeds, and fail10Percent it delays 1 second, and check a random number if it is 4, let it fail.
package main
import (
"fmt"
"math/rand"
"time"
)
func fail10Percent(ch chan int) {
time.Sleep(1 * time.Second)
e := rand.Intn(10)
fmt.Println("Calculating rand.Intn(10) ", e)
if e == 4 {
ch <- 0
return
}
ch <- 1
}
func test() {
for i := 0; i < 100; i++ {
err := make(chan int)
go fail10Percent(err)
res := <-err
fmt.Println("=== result: ", res)
if res != 1 {
fmt.Println("failed")
return
}
}
fmt.Println("succeeded")
}
func main() {
test()
}
I expect the go fail10Percent(err) will run concurrently 100 times, which will only have 1 second delay, however, when I run it, I see the following result getting printed 1 second after 1 second, why is that, and how I can adjust my program to do what I want.
Calculating rand.Intn(10) 1
=== result: 1
Calculating rand.Intn(10) 7
=== result: 1
Calculating rand.Intn(10) 7
=== result: 1
Calculating rand.Intn(10) 9
=== result: 1
Calculating rand.Intn(10) 1
=== result: 1
Calculating rand.Intn(10) 8
=== result: 1
Calculating rand.Intn(10) 5
=== result: 1
Calculating rand.Intn(10) 0
=== result: 1
Calculating rand.Intn(10) 6
=== result: 1
Calculating rand.Intn(10) 0
=== result: 1
Calculating rand.Intn(10) 4
=== result: 0
failed
I've commented out the code for you so that you can understand.
package main
import (
"fmt"
"math/rand"
"sync"
)
func fail10Percent(ch chan int, w *sync.WaitGroup) {
defer w.Done()
num := rand.Intn(10)
fmt.Println("calculating rand.Intn(10) ", num)
if num == 4 {
ch <- 0 // Fail
return
}
ch <- 1 // Pass
}
func test() {
var ch = make(chan int, 1)
// Launch the receiver goroutine to listen if goroutine succeeded or failed based on the value sent to ch
go func() {
for recv := range ch {
switch recv {
// Fail
case 0:
fmt.Println("goroutine failed")
// Pass
case 1:
fmt.Println("goroutine succeed")
}
}
}()
// wg is a WaitGroup
var wg sync.WaitGroup
for i := 0; i < 100; i++ {
wg.Add(1)
go fail10Percent(ch, &wg)
}
// wg.Wait() to wait for all goroutines to complete
wg.Wait()
// Close the channel so that the receiver can stop
close(ch)
}
func main() {
test()
}
Update:
Simple solution without using sync.WaitGroup
package main
import (
"fmt"
"math/rand"
)
// Using a send only channel
func fail10Percent(ch chan<- int) {
num := rand.Intn(10)
fmt.Println("calculating rand.Intn(10) ", num)
if num == 4 {
ch <- 0 // Fail
return
}
ch <- 1 // Pass
}
func test() {
var ch = make(chan int, 1)
for i := 0; i < 100; i++ {
go fail10Percent(ch)
}
for i := 0; i < 100; i++ {
if recv := <-ch; recv == 0 {
fmt.Println("goroutine failed")
} else if recv == 1 {
fmt.Println("goroutine succeed")
}
}
close(ch)
}
func main() {
test()
}
Related
I made some edits from the gobyexample:
import (
"fmt"
"math/rand"
"time"
)
type DemoResult struct {
Name string
Rate int
}
func random(min, max int) int {
rand.Seed(time.Now().UTC().UnixNano())
return rand.Intn(max-min) + min
}
func worker(id int, jobs <-chan int, results chan<- DemoResult) {
for j := range jobs {
fmt.Println("worker", id, "started job", j)
time.Sleep(time.Second)
fmt.Println("worker", id, "finished job", j)
myrand := random(1, 4)
if myrand == 2 {
results <- DemoResult{Name: "succ", Rate: j}
}
// else {
// results <- DemoResult{Name: "failed", Rate: 999}
// }
}
}
func main() {
const numJobs = 5
jobs := make(chan int, numJobs)
results := make(chan DemoResult)
for w := 1; w <= 3; w++ {
go worker(w, jobs, results)
}
for j := 1; j <= numJobs; j++ {
jobs <- j
}
close(jobs)
for a := 1; a <= numJobs; a++ {
out := <-results
if out.Name == "succ" {
fmt.Printf("%v\n", out)
}
}
}
I commented the following code intentional to make it stuck forever:
// else {
// results <- DemoResult{Name: "failed", Rate: 999}
// }
It seems like we should make the result's length the same as jobs'. I was wondering if we could make it have different length?
Use a wait group to detect when the workers are done. Close the results channel when the workers are done. Receive results until the channel is closed.
func worker(wg *sync.WaitGroup, id int,
jobs <-chan int,
results chan<- DemoResult) {
// Decrement wait group counter on return from
// function.
defer wg.Done()
⋮
}
func main() {
⋮
// Declare wait group and increment counter for
// each worker.
var wg sync.WaitGroup
for w := 1; w <= 3; w++ {
wg.Add(1)
go worker(&wg, w, jobs, results)
}
⋮
// Wait for workers to decrement wait group
// counter to zero and close channel.
// Execute in goroutine so we can continue on
// to receiving values from results in main.
go func() {
wg.Wait()
close(results)
}()
⋮
// Loop until results is closed.
for out := range results {
⋮
}
}
https://go.dev/play/p/FOQwybMl7tM
I was wondering if we could make it have different length?
Absolutely but you need some way of determining when you have reached the end of the results. This is the reason your example fails - currently the function assumes there will be numJobs (one result per job) results and waits for that many.
An alternative would be to use the channels closure to indicate this i.e. (playground)
package main
import (
"fmt"
"math/rand"
"sync"
"time"
)
type DemoResult struct {
Name string
Rate int
}
func random(min, max int) int {
rand.Seed(time.Now().UTC().UnixNano())
return rand.Intn(max-min) + min
}
func worker(id int, jobs <-chan int, results chan<- DemoResult) {
for j := range jobs {
fmt.Println("worker", id, "started job", j)
time.Sleep(time.Second)
fmt.Println("worker", id, "finished job", j)
myrand := random(1, 4)
if myrand == 2 {
results <- DemoResult{Name: "succ", Rate: j}
} // else {
// results <- DemoResult{Name: "failed", Rate: 999}
//}
}
}
func main() {
const numWorkers = 3
const numJobs = 5
jobs := make(chan int, numJobs)
results := make(chan DemoResult)
var wg sync.WaitGroup
wg.Add(numWorkers)
for w := 1; w <= numWorkers; w++ {
go func() {
worker(w, jobs, results)
wg.Done()
}()
}
go func() {
wg.Wait() // Wait for go routines to complete then close results channel
close(results)
}()
for j := 1; j <= numJobs; j++ {
jobs <- j
}
close(jobs)
for out := range results {
if out.Name == "succ" {
fmt.Printf("%v\n", out)
}
}
}
i am trying to print concurrently but not able to figure out why its serial, have put the code below
package main
import (
"fmt"
"sync"
)
func main() {
fmt.Println("Hello, playground")
var wg sync.WaitGroup
wg.Add(2)
go func(){
for i := 0; i < 4; i++ {
if i%2 == 0 {
fmt.Println("hi", i)
}
}
wg.Done()
}()
go func() {
for i := 0; i < 4; i++ {
if i%2 != 0 {
fmt.Println("g", i)
}
}
wg.Done()
}()
wg.Wait()
}
expectation is
hi0
g1
hi2
g3
but i get
from g 1
from g 3
hi 0
hi 2
Such a small function is less likely to demonstrate the concurrency, because the first goroutine may complete even before the second one starts, or before context switch happens. If you add a small pause to the loop, you will observe the interleaving:
package main
import (
"fmt"
"sync"
"time"
)
func main() {
var wg sync.WaitGroup
wg.Add(2)
go func() {
for i := 0; i < 4; i++ {
if i%2 == 0 {
fmt.Println("hi", i)
}
time.Sleep(10 * time.Millisecond)
}
wg.Done()
}()
go func() {
for i := 0; i < 4; i++ {
if i%2 != 0 {
fmt.Println("from g", i)
}
time.Sleep(10 * time.Millisecond)
}
wg.Done()
}()
wg.Wait()
}
I have 3 merge sort implementations:
MergeSort: simple one without concurrency;
MergeSortSmart: with concurrency limited by buffered channel size limit. If buffer is full, calls the simple implementation;
MergeSortSmartBug: same strategy as the previous one, but with a small "refactor", passing wg pointer to a function reducing code duplication.
The first two works as expected, but the third one returns an empty slice instead of the sorted input. I couldn't understand what happened and found no answers as well.
Here is the playground link for the code: https://play.golang.org/p/DU1ypbanpVi
package main
import (
"fmt"
"math/rand"
"runtime"
"sync"
)
type pass struct{}
var semaphore = make(chan pass, runtime.NumCPU())
func main() {
rand.Seed(10)
s := make([]int, 16)
for i := 0; i < 16; i++ {
s[i] = int(rand.Int31n(1000))
}
fmt.Println(s)
fmt.Println(MergeSort(s))
fmt.Println(MergeSortSmart(s))
fmt.Println(MergeSortSmartBug(s))
}
func merge(l, r []int) []int {
tmp := make([]int, 0, len(l)+len(r))
for len(l) > 0 || len(r) > 0 {
if len(l) == 0 {
return append(tmp, r...)
}
if len(r) == 0 {
return append(tmp, l...)
}
if l[0] <= r[0] {
tmp = append(tmp, l[0])
l = l[1:]
} else {
tmp = append(tmp, r[0])
r = r[1:]
}
}
return tmp
}
func MergeSort(s []int) []int {
if len(s) <= 1 {
return s
}
n := len(s) / 2
l := MergeSort(s[:n])
r := MergeSort(s[n:])
return merge(l, r)
}
func MergeSortSmart(s []int) []int {
if len(s) <= 1 {
return s
}
n := len(s) / 2
var wg sync.WaitGroup
wg.Add(2)
var l, r []int
select {
case semaphore <- pass{}:
go func() {
l = MergeSortSmart(s[:n])
<-semaphore
wg.Done()
}()
default:
l = MergeSort(s[:n])
wg.Done()
}
select {
case semaphore <- pass{}:
go func() {
r = MergeSortSmart(s[n:])
<-semaphore
wg.Done()
}()
default:
r = MergeSort(s[n:])
wg.Done()
}
wg.Wait()
return merge(l, r)
}
func MergeSortSmartBug(s []int) []int {
if len(s) <= 1 {
return s
}
n := len(s) / 2
var wg sync.WaitGroup
wg.Add(2)
l := mergeSmart(s[:n], &wg)
r := mergeSmart(s[n:], &wg)
wg.Wait()
return merge(l, r)
}
func mergeSmart(s []int, wg *sync.WaitGroup) []int {
var tmp []int
select {
case semaphore <- pass{}:
go func() {
tmp = MergeSortSmartBug(s)
<-semaphore
wg.Done()
}()
default:
tmp = MergeSort(s)
wg.Done()
}
return tmp
}
Why does the Bug version returns an empty slice? How can I refactor the Smart version without doing two selects one after the other?
Sorry for I couldn't reproduce this behavior in a smaller example.
The problem is not with the WaitGroup itself. It's with your concurrency handling. Your mergeSmart function lunches a go routine and returns the tmp variable without waiting for the go routine to finish.
You might want to try a pattern more like this:
leftchan := make(chan []int)
rightchan := make(chan []int)
go mergeSmart(s[:n], leftchan)
go mergeSmart(s[n:], rightchan)
l := <-leftchan
r := <-rightchan
Or you can use a single channel if order doesn't matter.
mergeSmart doesn't wait on the wg, so it returns a tmp that hasn't received a value yet. You could probably repair it by passing a reference to the destination slice in to the function, instead of returning a slice.
Look at the mergeSmart function. When the select enter into the first case, the goroutine is launched and imediatly returns tmp (which is an empty array). In that case there is no way to get the right value. (See advanced debugging prints here https://play.golang.org/p/IedaY3muso2)
Maybe passing arrays preallocated by reference?
I implemented both suggestions (passing slice by reference and using channels) and the (working!) result is here: https://play.golang.org/p/DcDC_-NjjAH
package main
import (
"fmt"
"math/rand"
"runtime"
"sync"
)
type pass struct{}
var semaphore = make(chan pass, runtime.NumCPU())
func main() {
rand.Seed(10)
s := make([]int, 16)
for i := 0; i < 16; i++ {
s[i] = int(rand.Int31n(1000))
}
fmt.Println(s)
fmt.Println(MergeSort(s))
fmt.Println(MergeSortSmart(s))
fmt.Println(MergeSortSmartPointer(s))
fmt.Println(MergeSortSmartChan(s))
}
func merge(l, r []int) []int {
tmp := make([]int, 0, len(l)+len(r))
for len(l) > 0 || len(r) > 0 {
if len(l) == 0 {
return append(tmp, r...)
}
if len(r) == 0 {
return append(tmp, l...)
}
if l[0] <= r[0] {
tmp = append(tmp, l[0])
l = l[1:]
} else {
tmp = append(tmp, r[0])
r = r[1:]
}
}
return tmp
}
func MergeSort(s []int) []int {
if len(s) <= 1 {
return s
}
n := len(s) / 2
l := MergeSort(s[:n])
r := MergeSort(s[n:])
return merge(l, r)
}
func MergeSortSmart(s []int) []int {
if len(s) <= 1 {
return s
}
n := len(s) / 2
var wg sync.WaitGroup
wg.Add(2)
var l, r []int
select {
case semaphore <- pass{}:
go func() {
l = MergeSortSmart(s[:n])
<-semaphore
wg.Done()
}()
default:
l = MergeSort(s[:n])
wg.Done()
}
select {
case semaphore <- pass{}:
go func() {
r = MergeSortSmart(s[n:])
<-semaphore
wg.Done()
}()
default:
r = MergeSort(s[n:])
wg.Done()
}
wg.Wait()
return merge(l, r)
}
func MergeSortSmartPointer(s []int) []int {
if len(s) <= 1 {
return s
}
n := len(s) / 2
var l, r []int
var wg sync.WaitGroup
wg.Add(2)
mergeSmartPointer(&l, s[:n], &wg)
mergeSmartPointer(&r, s[n:], &wg)
wg.Wait()
return merge(l, r)
}
func mergeSmartPointer(tmp *[]int, s []int, wg *sync.WaitGroup) {
select {
case semaphore <- pass{}:
go func() {
*tmp = MergeSortSmartPointer(s)
<-semaphore
wg.Done()
}()
default:
*tmp = MergeSort(s)
wg.Done()
}
}
func MergeSortSmartChan(s []int) []int {
if len(s) <= 1 {
return s
}
n := len(s) / 2
lchan := make(chan []int)
rchan := make(chan []int)
go mergeSmartChan(s[:n], lchan)
go mergeSmartChan(s[n:], rchan)
l := <-lchan
r := <-rchan
return merge(l, r)
}
func mergeSmartChan(s []int, c chan []int) {
select {
case semaphore <- pass{}:
go func() {
c <- MergeSortSmartChan(s)
<-semaphore
}()
default:
c <- MergeSort(s)
}
}
I understood 100% what I was doing wrong, thanks!
And for future references, here's the benchmark of sorting a slice of 100,000 elems:
$ go test -bench=.
goos: linux
goarch: amd64
cpu: Intel(R) Core(TM) i5-9300H CPU # 2.40GHz
BenchmarkMergeSort-8 97 12230309 ns/op
BenchmarkMergeSortSmart-8 181 7209844 ns/op
BenchmarkMergeSortSmartPointer-8 163 7483136 ns/op
BenchmarkMergeSortSmartChan-8 156 8149585 ns/op
I wrote a test code, but do not understand why I get this result.
My sub() should update or return counter, based on the channel value
send 1 = counter++
send 0 = return counter
I start 10 go routines con().
They should simply send many 1 to channel (this increase counter)
I wait 1 sec and send 0 to channel. What value should I get?
I think first, I get a "random" value,
but i get 100000 (ok 10x 10000 is faster than 1 sec)
Now I change
for i:=0; i < 10; i++ {
to
for i:=0; i < 10000; i++ {
and now my returned value is 1
Why!?
Now uncomment fmt.Println(counter) in main().
As you see counter works and has this "random" number
package main
import (
"fmt"
"time"
)
var ch chan int = make(chan int)
var counter int
func main() {
go sub()
for i:=0; i < 10; i++ { //change to 10000
go con()
}
time.Sleep(1000 * time.Millisecond)
ch <- 0
fmt.Println(<- ch)
//fmt.Println(counter) //uncomment this
}
func sub() {
for c := range ch {
if c == 0 { ch <- counter }
if c == 1 { counter++ }
}
}
func con() {
for i := 0; i < 10000; i++ {
ch <- 1
}
}
with 2 channels, this work:
package main
import (
"fmt"
"time"
)
var ch chan int = make(chan int)
var ch2 chan int = make(chan int)
var counter int
func main() {
go sub()
for i:=0; i < 10000; i++ { //change to 10000
go con()
}
time.Sleep(1000 * time.Millisecond)
ch2 <- 0
fmt.Println(<- ch2)
//fmt.Println(counter) //uncomment this
}
func sub() {
for ;; {
select {
case <- ch:
counter++
case <- ch2:
ch2 <- counter
}
}
}
func con() {
for i := 0; i < 10000; i++ {
ch <- 1
}
}
Consider the following go playground
package main
import "fmt"
func main() {
var chan_array [2]chan int
chan1 := make(chan int)
chan2 := make(chan int)
chan_array[0] = chan1
chan_array[1] = chan2
for i := 0; i < 2; i++ {
go func() {
select {
case x := <- chan_array[i]:
if (x == 0) {
return
}
fmt.Println(x)
}
}()
}
chan1<- 1
chan2<- 2
chan1<- 0
chan2<- 0
}
The code above is trying to create 2 running goroutines with that listens to the channel to signal print or close.
But the above code run into dead lock.
I am not exactly sure why
Can someone point out my mistake?
Thanks
There are some problems:
What is the value of i when chan_array[i-1] runs:
for i := 0; i < 2; i++ {
go func() {
select {
case x := <- chan_array[i-1]:
if x == 0 {
return
}
fmt.Println(x)
}
}()
}
try this:
for i := 0; i < 2; i++ {
go func(i int) {
select {
case x := <-chan_array[i]:
if x == 0 {
return
}
fmt.Println(x)
}
}(i)
}
Let's simplify your code (with some corrections):
package main
import "fmt"
func main() {
chan1 := make(chan int)
chan2 := make(chan int)
go routine(chan1)
go routine(chan2)
chan1 <- 1
chan2 <- 2
chan1 <- 0
chan2 <- 0
}
func routine(ch chan int) {
select {
case x := <-ch:
if x == 0 {
return
}
fmt.Println(x)
}
}
With these:
chan1 <- 1
chan2 <- 2
fatal error:
all goroutines are asleep - deadlock!
your goroutines finished and no goroutines listening to chan1 and chan1 here:
chan1 <- 0
chan2 <- 0
Your corrected working sample code is:
package main
import "fmt"
func main() {
chan1 := make(chan int)
chan2 := make(chan int)
go routine(chan1)
go routine(chan2)
chan1 <- 1
chan2 <- 2
chan1 <- 0
chan2 <- 0
}
func routine(ch chan int) {
for {
select {
case x := <-ch:
if x == 0 {
return
}
fmt.Println(x)
}
}
}
output:
1
2
By the time the goroutines are running the variable i has already incremented. Pass it as a function parameter instead.
In fact, never rely on variables from a function closure in goroutines. It's too unreliable.