goroutine value return order [duplicate] - go

This question already has answers here:
Golang channel output order
(4 answers)
Closed 4 years ago.
Why following codes always return 2,1, not 1,2.
func test(x int, c chan int) {
c <- x
}
func main() {
c := make(chan int)
go test(1, c)
go test(2, c)
x, y := <-c, <-c // receive from c
fmt.Println(x, y)
}

If you want to know what the order is, then let your program include ordering information
This example uses a function closure to generate a sequence
The channel returns a struct of two numbers, one of which is a sequence order number
The sequence incrementer should be safe across go routines as there is a mutex lock on the sequence counter
package main
import (
"fmt"
"sync"
)
type value_with_order struct {
v int
order int
}
var (
mu sync.Mutex
)
func orgami(x int, c chan value_with_order, f func() int) {
v := new(value_with_order)
v.v = x
v.order = f()
c <- *v
}
func seq() func() int {
i := 0
return func() int {
mu.Lock()
defer mu.Unlock()
i++
return i
}
}
func main() {
c := make(chan value_with_order)
sequencer := seq()
for n := 0; n < 10; n++ {
go orgami(1, c, sequencer)
go orgami(2, c, sequencer)
go orgami(3, c, sequencer)
}
received := 0
for q := range c {
fmt.Printf("%v\n", q)
received++
if received == 30 {
close(c)
}
}
}
second version where the sequence is called from the main loop to make the sequence numbers come out in the order that the function is called
package main
import (
"fmt"
"sync"
)
type value_with_order struct {
v int
order int
}
var (
mu sync.Mutex
)
func orgami(x int, c chan value_with_order, seqno int) {
v := new(value_with_order)
v.v = x
v.order = seqno
c <- *v
}
func seq() func() int {
i := 0
return func() int {
mu.Lock()
defer mu.Unlock()
i++
return i
}
}
func main() {
c := make(chan value_with_order)
sequencer := seq()
for n := 0; n < 10; n++ {
go orgami(1, c, sequencer())
go orgami(2, c, sequencer())
go orgami(3, c, sequencer())
}
received := 0
for q := range c {
fmt.Printf("%v\n", q)
received++
if received == 30 {
close(c)
}
}
}

Related

Observer pattern or a Publish/Subscribe pattern for a cellular automaton

I am trying to code an observer pattern or a publish/submit pattern for a sort of cellular automaton.
The classical observer pattern does not to the trick because if a cell A subscribes to changes in a cell B and vice-versa, the application will run out of stack owing to the recursive approach (B.update() will call A.update() and so on and the app will run out of stack).
So I thought about using a publish/subscribe pattern where respective cells pass each other messages, rather than calling each other's update() methods.
Here is a simple example with two cells A and B:
package main
import (
"fmt"
ps "publish/pubsub"
)
func main() {
fmt.Printf("Starting\n")
chEnd := make(chan int)
// initialize
a := ps.NewNode(1, 0)
b := ps.NewNode(2, 0)
// connect nodes
a.Connect(b.ChOut)
b.Connect(a.ChOut)
// Start listening
a.Listen()
b.Listen()
// Start sending data on one arbitrary node
// to start the process.
a.ChIn <- 10
<-chEnd
}
and the corresponding lib
package pubsub
import (
"fmt"
)
type Node struct {
Id int
State int
ChOut chan int
ChIn chan int
}
func NewNode(id int, state int) Node {
chout := make(chan int)
var chin chan int
return Node{id, state, chout, chin}
}
func (p *Node) Broadcast(inItem int) {
p.ChOut <- inItem + 1
//time.Sleep(100 * time.Millisecond)
}
func (p *Node) Listen() {
go func() {
for {
select {
case inItem := <-p.ChIn:
fmt.Printf("%d: %d\n", p.Id, inItem)
p.Broadcast(inItem)
}
}
}()
}
func (p *Node) Connect(ch chan int) {
p.ChIn = ch
}
Each node has a input and an output channe.
The input channel of B is the output channel of A and vice-versa.
Every update consists merely of incrementing the data passed by the other cell.
It seems to work. So far, so good.
I tried to do something similar with a set of 4 cells A, B, C, D, in order to simulate a one dimensional cellular automaton of sorts.
In this second attempt,
each cell has two input channels (let and right) to listen to its closest left- and right-hand neighbour, respectively (ChinL and ChinR).
each cell has to output channels to communicate its latest updated state to its closest neighbours (ChoutL and ChoutR).
I must have done something wrong in the implementation of that scheme with 4 cells, because it yields odd results : the values passed back and forth between the 4 cells seem to hit a threshold instead of increasing at every consecutive step: here is the code:
package main
import (
"fmt"
ps "publish/pubsub"
)
func main() {
fmt.Printf("Starting\n")
chEnd := make(chan int)
// initialize
a := ps.NewNode(1, 0)
b := ps.NewNode(2, 0)
c := ps.NewNode(3, 0)
d := ps.NewNode(4, 0)
// connect nodes
a.ChInL = d.ChOutR
a.ChInR = b.ChOutL
b.ChInL = a.ChOutR
b.ChInR = c.ChOutL
c.ChInL = b.ChOutR
c.ChInR = d.ChOutL
d.ChInL = c.ChOutR
d.ChInR = a.ChOutL
// Start listening
go a.Listen()
go b.Listen()
go c.Listen()
go d.Listen()
go a.Broadcast()
go b.Broadcast()
go c.Broadcast()
go d.Broadcast()
// Start sending data on one arbitrary node
// to start the process.
a.ChInL <- 1
// Dummy read on channel to make main() wait
<-chEnd
}
/*
A B C D
LR LR LR LR
*/
and the corresponding lib
package pubsub
import (
"fmt"
"strings"
)
type Node struct {
Id int
State int
ChOutL chan int
ChOutR chan int
ChInL chan int
ChInR chan int
ChIO chan int
}
func NewNode(id int, state int) Node {
choutL := make(chan int)
choutR := make(chan int)
var chinL chan int
var chinR chan int
chIO := make(chan int)
return Node{id, state, choutL, choutR, chinL, chinR, chIO}
}
func (p *Node) Broadcast() {
for item := range p.ChIO {
p.ChOutL <- item + 1
p.ChOutR <- item + 1
fmt.Printf("%d: %d %s\n", p.Id, item, strings.Repeat("*", item))
}
}
func (p *Node) Listen() {
for {
//time.Sleep(100 * time.Millisecond)
select {
case inItem := <-p.ChInL:
go func() {
p.ChIO <- inItem
}()
case inItem := <-p.ChInR:
go func() {
p.ChIO <- inItem
}()
}
}
}
For completeness, here is the go.mod for the modules above:
module publish
go 1.17
For every signal a node receives on p.ChInL or p.ChInR you send out 2 signals. The first to p.ChOutL and the second to p.ChOutR. Since every node does this there will be an exponential amount of signals going around.
When running it locally the threshold is between 20 and 25. So about 2^25 33554432 signals going around. To see 26, the program will need to process 67108864 signals. So it will go past this threshold, just exponentially slower.
Now for the fix. I think you should implement some sort of tick system. So instead of sending update signals for every change to the automaton you send one update every tick like 20 times per second.
Maybe better yet, instead of using routines and channels, just make an slice of nodes and loop over them (again only updating the neighbor and not propagating it in the same tick/loop). Each iteration of the loop the state of all nodes has changed. I believe this is also how other automatons like game-of-live work. Now you can run the simulation as fast as your computer can run.
Here is one possible alternative:
package pubsub
import (
"fmt"
"math/rand"
"strings"
"time"
)
type Node struct {
Id int
State int
ChOutL chan int
ChOutR chan int
ChInL chan int
ChInR chan int
ChIO chan int
}
func NewNode(id int, state int) Node {
choutL := make(chan int)
choutR := make(chan int)
var chinL chan int
var chinR chan int
chIO := make(chan int)
return Node{id, state, choutL, choutR, chinL, chinR, chIO}
}
func (p *Node) Broadcast() {
for item := range p.ChIO {
rnd := rand.Intn(2)
if rnd == 0 {
p.ChOutL <- item + 1
} else {
p.ChOutR <- item + 1
}
fmt.Printf("%d: %d %s\n", p.Id, item, strings.Repeat("*", item))
}
}
func (p *Node) Listen() {
for {
time.Sleep(100 * time.Millisecond)
select {
case inItem := <-p.ChInL:
p.ChIO <- inItem
case inItem := <-p.ChInR:
p.ChIO <- inItem
}
}
}
Here is a second alternative:
package main
import (
"fmt"
ps "pub/pubsub"
)
func main() {
fmt.Printf("Hello\n")
chEnd := make(chan int)
A := ps.NewNode(1, 0)
B := ps.NewNode(2, 0)
C := ps.NewNode(3, 0)
D := ps.NewNode(4, 0)
B.LeftNode = &A
B.RightNode = &C
C.LeftNode = &B
C.RightNode = &D
D.LeftNode = &C
D.RightNode = &A
A.LeftNode = &D
A.RightNode = &B
A.Listen()
B.Listen()
C.Listen()
D.Listen()
A.State = 1
<-chEnd
}
//----
package pubsub
import (
"fmt"
"strings"
"sync"
"time"
)
type Node struct {
Id int
State int
LeftNode *Node
RightNode *Node
LeftState int
RightState int
}
var m sync.Mutex
func NewNode(id int, state int) Node {
return Node{id, state, nil, nil, 0, 0}
}
func (n *Node) Listen() {
go func() {
for {
m.Lock()
time.Sleep(10 * time.Millisecond)
if n.LeftState != n.LeftNode.State {
n.LeftState = n.LeftNode.State
n.State = n.LeftNode.State + 1
fmt.Printf("%d: %d %s\n", n.Id, n.State, strings.Repeat("*", n.State))
}
m.Unlock()
}
}()
go func() {
for {
m.Lock()
time.Sleep(10 * time.Millisecond)
if n.RightState != n.RightNode.State {
n.RightState = n.RightNode.State
n.State = n.RightNode.State + 1
fmt.Printf("%d: %d %s\n", n.Id, n.State, strings.Repeat("*", n.State))
}
m.Unlock()
}
}()
}

Passing a WaitGroup to a function changes behavior, why?

I have 3 merge sort implementations:
MergeSort: simple one without concurrency;
MergeSortSmart: with concurrency limited by buffered channel size limit. If buffer is full, calls the simple implementation;
MergeSortSmartBug: same strategy as the previous one, but with a small "refactor", passing wg pointer to a function reducing code duplication.
The first two works as expected, but the third one returns an empty slice instead of the sorted input. I couldn't understand what happened and found no answers as well.
Here is the playground link for the code: https://play.golang.org/p/DU1ypbanpVi
package main
import (
"fmt"
"math/rand"
"runtime"
"sync"
)
type pass struct{}
var semaphore = make(chan pass, runtime.NumCPU())
func main() {
rand.Seed(10)
s := make([]int, 16)
for i := 0; i < 16; i++ {
s[i] = int(rand.Int31n(1000))
}
fmt.Println(s)
fmt.Println(MergeSort(s))
fmt.Println(MergeSortSmart(s))
fmt.Println(MergeSortSmartBug(s))
}
func merge(l, r []int) []int {
tmp := make([]int, 0, len(l)+len(r))
for len(l) > 0 || len(r) > 0 {
if len(l) == 0 {
return append(tmp, r...)
}
if len(r) == 0 {
return append(tmp, l...)
}
if l[0] <= r[0] {
tmp = append(tmp, l[0])
l = l[1:]
} else {
tmp = append(tmp, r[0])
r = r[1:]
}
}
return tmp
}
func MergeSort(s []int) []int {
if len(s) <= 1 {
return s
}
n := len(s) / 2
l := MergeSort(s[:n])
r := MergeSort(s[n:])
return merge(l, r)
}
func MergeSortSmart(s []int) []int {
if len(s) <= 1 {
return s
}
n := len(s) / 2
var wg sync.WaitGroup
wg.Add(2)
var l, r []int
select {
case semaphore <- pass{}:
go func() {
l = MergeSortSmart(s[:n])
<-semaphore
wg.Done()
}()
default:
l = MergeSort(s[:n])
wg.Done()
}
select {
case semaphore <- pass{}:
go func() {
r = MergeSortSmart(s[n:])
<-semaphore
wg.Done()
}()
default:
r = MergeSort(s[n:])
wg.Done()
}
wg.Wait()
return merge(l, r)
}
func MergeSortSmartBug(s []int) []int {
if len(s) <= 1 {
return s
}
n := len(s) / 2
var wg sync.WaitGroup
wg.Add(2)
l := mergeSmart(s[:n], &wg)
r := mergeSmart(s[n:], &wg)
wg.Wait()
return merge(l, r)
}
func mergeSmart(s []int, wg *sync.WaitGroup) []int {
var tmp []int
select {
case semaphore <- pass{}:
go func() {
tmp = MergeSortSmartBug(s)
<-semaphore
wg.Done()
}()
default:
tmp = MergeSort(s)
wg.Done()
}
return tmp
}
Why does the Bug version returns an empty slice? How can I refactor the Smart version without doing two selects one after the other?
Sorry for I couldn't reproduce this behavior in a smaller example.
The problem is not with the WaitGroup itself. It's with your concurrency handling. Your mergeSmart function lunches a go routine and returns the tmp variable without waiting for the go routine to finish.
You might want to try a pattern more like this:
leftchan := make(chan []int)
rightchan := make(chan []int)
go mergeSmart(s[:n], leftchan)
go mergeSmart(s[n:], rightchan)
l := <-leftchan
r := <-rightchan
Or you can use a single channel if order doesn't matter.
mergeSmart doesn't wait on the wg, so it returns a tmp that hasn't received a value yet. You could probably repair it by passing a reference to the destination slice in to the function, instead of returning a slice.
Look at the mergeSmart function. When the select enter into the first case, the goroutine is launched and imediatly returns tmp (which is an empty array). In that case there is no way to get the right value. (See advanced debugging prints here https://play.golang.org/p/IedaY3muso2)
Maybe passing arrays preallocated by reference?
I implemented both suggestions (passing slice by reference and using channels) and the (working!) result is here: https://play.golang.org/p/DcDC_-NjjAH
package main
import (
"fmt"
"math/rand"
"runtime"
"sync"
)
type pass struct{}
var semaphore = make(chan pass, runtime.NumCPU())
func main() {
rand.Seed(10)
s := make([]int, 16)
for i := 0; i < 16; i++ {
s[i] = int(rand.Int31n(1000))
}
fmt.Println(s)
fmt.Println(MergeSort(s))
fmt.Println(MergeSortSmart(s))
fmt.Println(MergeSortSmartPointer(s))
fmt.Println(MergeSortSmartChan(s))
}
func merge(l, r []int) []int {
tmp := make([]int, 0, len(l)+len(r))
for len(l) > 0 || len(r) > 0 {
if len(l) == 0 {
return append(tmp, r...)
}
if len(r) == 0 {
return append(tmp, l...)
}
if l[0] <= r[0] {
tmp = append(tmp, l[0])
l = l[1:]
} else {
tmp = append(tmp, r[0])
r = r[1:]
}
}
return tmp
}
func MergeSort(s []int) []int {
if len(s) <= 1 {
return s
}
n := len(s) / 2
l := MergeSort(s[:n])
r := MergeSort(s[n:])
return merge(l, r)
}
func MergeSortSmart(s []int) []int {
if len(s) <= 1 {
return s
}
n := len(s) / 2
var wg sync.WaitGroup
wg.Add(2)
var l, r []int
select {
case semaphore <- pass{}:
go func() {
l = MergeSortSmart(s[:n])
<-semaphore
wg.Done()
}()
default:
l = MergeSort(s[:n])
wg.Done()
}
select {
case semaphore <- pass{}:
go func() {
r = MergeSortSmart(s[n:])
<-semaphore
wg.Done()
}()
default:
r = MergeSort(s[n:])
wg.Done()
}
wg.Wait()
return merge(l, r)
}
func MergeSortSmartPointer(s []int) []int {
if len(s) <= 1 {
return s
}
n := len(s) / 2
var l, r []int
var wg sync.WaitGroup
wg.Add(2)
mergeSmartPointer(&l, s[:n], &wg)
mergeSmartPointer(&r, s[n:], &wg)
wg.Wait()
return merge(l, r)
}
func mergeSmartPointer(tmp *[]int, s []int, wg *sync.WaitGroup) {
select {
case semaphore <- pass{}:
go func() {
*tmp = MergeSortSmartPointer(s)
<-semaphore
wg.Done()
}()
default:
*tmp = MergeSort(s)
wg.Done()
}
}
func MergeSortSmartChan(s []int) []int {
if len(s) <= 1 {
return s
}
n := len(s) / 2
lchan := make(chan []int)
rchan := make(chan []int)
go mergeSmartChan(s[:n], lchan)
go mergeSmartChan(s[n:], rchan)
l := <-lchan
r := <-rchan
return merge(l, r)
}
func mergeSmartChan(s []int, c chan []int) {
select {
case semaphore <- pass{}:
go func() {
c <- MergeSortSmartChan(s)
<-semaphore
}()
default:
c <- MergeSort(s)
}
}
I understood 100% what I was doing wrong, thanks!
And for future references, here's the benchmark of sorting a slice of 100,000 elems:
$ go test -bench=.
goos: linux
goarch: amd64
cpu: Intel(R) Core(TM) i5-9300H CPU # 2.40GHz
BenchmarkMergeSort-8 97 12230309 ns/op
BenchmarkMergeSortSmart-8 181 7209844 ns/op
BenchmarkMergeSortSmartPointer-8 163 7483136 ns/op
BenchmarkMergeSortSmartChan-8 156 8149585 ns/op

Does closed channel send value zero?

In the below code:
package main
import (
"fmt"
"time"
)
func asChan(vs ...int) <-chan int {
c := make(chan int)
go func() {
for _, v := range vs {
c <- v
time.Sleep(2 * time.Second)
}
close(c)
}()
return c
}
func merge(a, b <-chan int) <-chan int {
c := make(chan int)
go func() {
for {
select {
case v := <-a:
c <- v
case v := <-b:
c <- v
}
}
}()
return c
}
func main() {
a := asChan(1, 3, 5, 7)
b := asChan(2, 4, 6, 8)
c := merge(a, b)
for value := range c {
fmt.Println(value)
}
}
below is the value zero for v received after closing channel a.
but my understanding is, this line of code(case v := <-a:) in merge() function should block for input, because the channel(a) is closed
Edit:
After code changes, main() does not exit:
func merge(a, b <-chan int) chan int {
c := make(chan int)
go func() {
defer close(c)
for a != nil || b != nil { // while atleast one channel is open
select {
case v, ok := <-a:
sendOnChannel(a, c, v, ok)
case v, ok := <-b:
sendOnChannel(b, c, v, ok)
}
}
}()
return c
}
func sendOnChannel(sourceChan <-chan int, targetChan chan int, v int, channelOpen bool) {
if !channelOpen {
// The channel is closed. Set channel variable
// to nil to disable this select / case.
sourceChan = nil
} else {
targetChan <- v
}
}
Output:
$ bin/cs61a
1
2
3
4
5
6
7
8
How a generates zero in this line of code(case v := <-a)?
The specification says:
A receive operation on a closed channel can always proceed immediately, yielding the element type's zero value after any previously sent values have been received.
I don't know why the language designers chose this design, but it's useful in practice.
Use the two-value receive statement to detect when the channel yields a zero value because the channel is closed.
v, ok := <-a
if !ok {
// channel closed
}
Use the following code to implement merge:
func merge(a, b <-chan int) <-chan int {
c := make(chan int)
go func() {
defer close(c)
// While we have at least one channel ...
for a != nil || b != nil {
select {
case v, ok := <-a:
if ok {
c <- v
} else {
// The channel is closed. Set channel variable
// to nil to disable this select / case.
a = nil
}
case v, ok := <-b:
if ok {
c <- v
} else {
b = nil
}
}
}
}()
return c
}
Run it on the playground.

What is the correct way to write a distinct channel in Go?

I am a beginner in go.
I am trying to figure out an easy way to implement a channel that only output distinct values.
What I want to do is this:
package example
import (
"fmt"
"testing"
)
func TestShouldReturnDistinctValues(t *testing.T) {
var c := make([]chan int)
c <- 1
c <- 1
c <- 2
c <- 2
c <- 3
for e := range c {
// only print 1, 2 and 3.
fmt.println(e)
}
}
Should I be concern about memory leak here if I were to use a map to remember previous values?
You really can't do that, you'd have to keep a track of the values somehow, a map[int]struct{} is probably the most memory efficient way.
A simple example:
func UniqueGen(min, max int) <-chan int {
m := make(map[int]struct{}, max-min)
ch := make(chan int)
go func() {
for i := 0; i < 1000; i++ {
v := min + rand.Intn(max)
if _, ok := m[v]; !ok {
ch <- v
m[v] = struct{}{}
}
}
close(ch)
}()
return ch
}
I have done similar things before, except my problem was output inputs in ascending order. You can do this by adding a middle go routine. Here is an example:
package main
func main() {
input, output := distinct()
go func() {
input <- 1
input <- 1
input <- 2
input <- 2
input <- 3
close(input)
}()
for i := range output {
println(i)
}
}
func distinct() (input chan int, output chan int) {
input = make(chan int)
output = make(chan int)
go func() {
set := make(map[int]struct{})
for i := range input {
if _, ok := set[i]; !ok {
set[i] = struct{}{}
output <- i
}
}
close(output)
}()
return
}

Avoid panic, when trying to insert a value to a closed channel

package main
import (
"fmt"
"time"
)
func fib() chan int {
c := make(chan int)
go func() {
c <- 0
c <- 1
n, m := 0, 1
for {
temp := n + m
n = m
m = temp
c <- m // This results in panic, when the channel is closed
}
}()
return c
}
func main() {
start := time.Now()
var lastFib int
c := fib()
for i := 0; i != 1000000; i++ {
lastFib = <-c
}
close(c)
fmt.Println(lastFib)
fmt.Println(time.Now().Sub(start))
}
In the most idiomatic way, how would one avoid the panic in the goroutine, when the channel is closed? Or should i avoid using close at all?
I'm not looking into alternative methods (such as closures) to achieve the same thing, just trying to get a better understanding of channels.
Close is a good way for the goroutine sending into a channel to signal the receiving side that you are done with this channel. The other way around (your problem) is IMHO undoable, at least direct. You could add an other channel done which signal end of duty to your fibonacci generating goroutine.
Here is a modified version of your example that uses channels in an allowed (though not necessarily sensible) way:
package main
import (
"fmt"
"time"
)
func fib(c chan int) {
c <- 0
c <- 1
n, m := 0, 1
for {
temp := n + m
n = m
m = temp
c <- m
if m > 100000000 {
close(c)
break
}
}
}
func main() {
start := time.Now()
lastFib, newFib := 0, 0
ok := true
c := make(chan int)
go fib(c)
for {
newFib, ok = <-c
if !ok {
fmt.Println(lastFib)
break
}
lastFib = newFib
}
fmt.Println(time.Now().Sub(start))
}

Resources