I've bumped in to a peculiar problem which I unfortunately haven't been able to reproduce in a minimal working example. I'll try to explain it and hopefully you can give me some hint at least.
I have two protocols: A & B. For each protocol there is one central party p1, and three outer parties, lets call those pn. Each party is implemented as a separate goroutine.
Protocol A is as follows:
All parties perform a calculation, separately, and send their result of type *big.Int to p1.
p1 receives all results and puts them in a slice which it sends back to each party pn.
All parties receive the slice and perform a new calculation based on it, and send their result of type *DecryptionShare to p1.
p1 receives all data and calculates a result.
All parties output a result *big.Int.
To help with this I have three channels, one that is used for sending data p1 -> pn, one for pn -> p1 and one to output final results back to main thread (e.i. all pn read from and write to the same channels). The result 1. and 3. from pn is of different types though so that channel type is interface{}.
Protocol B first initiates protocol A and then perform further calculations, which are irrelevant.
Here's to my problem:
Protocol A on it's own works without ever showing problems.
But, when I call B ~10 % of the runs, it panics in A, even though the only thing differing is B passing on the input parameters to A.
The error showing is
panic: interface conversion: interface {} is *big.Int, not *DecryptionShare
implying that p1 receives a *big.Int while it is at step 4, although it already received every parties *big.Int in step 2.
I have tried staying at step 2 a while longer using time.Sleep and select but I never get an additional *big.Int at that step, it only occasionally shows up at step 4.
If I instead of chan interface{} use two seperate channels chan *big.Int and chan *DecryptionShare protocol B terminates correctly which also implies that everything is read correctly from channels (e.i. no thread is left blocked). I was hoping to avoid this though as I already have numerous channels in play.
Does anyone have any ideas on why this panic occurs?
EDIT:
Here's a minimal working example that doesn't produce the error though. Hopefully it can gain some insights. *DecryptionShare is replaced by int.
package tpsi
import (
"math/big"
"fmt"
"crypto/rand"
"testing"
)
type DecryptionShare struct {
index int
p *big.Int
}
func TestErs(t *testing.T) {
message_channel1 := make(chan interface{})
message_channel2 := make(chan []*big.Int)
return_channel := make(chan *big.Int)
n := 4
go CentralParty(n, message_channel2, message_channel1, return_channel)
for i := 1; i < n; i += 1 {
go OtherParty(message_channel1, message_channel2, return_channel)
}
for i := 0; i < n; i += 1 {
fmt.Println(<-return_channel)
}
t.Error("for display")
}
func CentralParty(n int, sender_channel chan<- []*big.Int, reciever_channel <-chan interface{}, return_channel chan<- *big.Int) {
p, err := rand.Prime(rand.Reader, 256)
if err != nil {panic(err)}
all_p := make([]*big.Int, n)
all_p[0] = p
for i := 1; i < n; i += 1 {
all_p[i] = (<-reciever_channel).(*big.Int)
}
for i := 1; i < n; i += 1 {
sender_channel <- all_p
}
shares := make([]*DecryptionShare, 4)
for i := 1; i < n; i += 1 {
shares[i] = (<-reciever_channel).(*DecryptionShare)
}
return_channel <- shares[1].p
}
func OtherParty(sender_channel chan<- interface{}, reciever_channel <-chan []*big.Int, return_channel chan<- *big.Int) {
p, err := rand.Prime(rand.Reader, 256)
if err != nil {panic(err)}
sender_channel <- p
all_p := <-reciever_channel
var ds DecryptionShare
ds.p = p
ds.index = all_p[0].BitLen()
sender_channel <- &ds
return_channel <- p
}
From the joint pressure of several commenters I forced myself to obtain a MWE. And as suggested by #oakad, I found the bug while doing so.
The error was (unsurprisingly) from protocol B which reused the chan interface{} once again sending the first data type *big.Int, and thereby introducing a race condition.
I completely neglected to consider race conditions across protocols.
Thank you for the comments!
Related
I'm new to golang and going over the gotour. I have following code which works perfectly as it should.
package main
import (
"fmt"
)
func fibonacci(n int, c chan int) {
x, y := 0, 1
for i := 0; i < n; i++ {
c <- x
x, y = y, x+y
}
close(c)
}
func main() {
c := make(chan int, 5)
// c := make(chan int) //doesn't work, why ?
go fibonacci(cap(c), c)
for i := range c {
fmt.Println(i)
}
}
But when I use an unbuffered channel instead of a buffered one, I don't get any output, why's that so ?
When you pass cap(c) through to the fibonacci function, what value is passed through? on the buffered channel the n == 5, on the unbuffered channel n == 0
and your for loop
for i := 0; i < 0; i++ {
Actually, this is a really bad way of handling the situation. You are requiring the number of channels to be equal to the number of iterations.
Using a channel in this way I would not recommend, think of the channel as being able to operate concurrently, which is not something you would want to do in this scenario!
If you pass the number in separately to the number of routines, then the unbuffered channel will work as expected:
https://play.golang.org/p/G1b2vjTUCsV
cap(c) will be zero if channel is un-buffered . See the modified program
I'm kind of lost here, I was trying to get a goroutine to add to an array and another goroutine to read from it, which I suspect is somewhat close to what I have below but I need to play around with the wait().
However, I am getting the error prog.go:19:14: too many variables in range, line 19 is for _, v := range c { I can't find an answer for that online, what am I doing or not doing here?
package main
import (
"fmt"
//"time"
"sync"
)
func hello(wg *sync.WaitGroup, s []int, c chan int) {
for _, v := range s {
c <- v
}
fmt.Println("Finished adding to channel")
wg.Done()
}
func hello2(wg *sync.WaitGroup, c chan int) {
fmt.Println("Channel",c)
for _, v := range c {
fmt.Println("Received",v)
}
fmt.Println("Finished taking from channel")
wg.Done()
}
func main() {
s := []int{1, 2, 3, 4, 5}
var c = make(chan int, 5)
var wg sync.WaitGroup
wg.Add(1)
go hello(&wg, s, c)
wg.Wait()
wg.Add(1)
go hello2(&wg, c)
wg.Wait()
//time.Sleep(1 * time.Second)
fmt.Println("main function")
}
When you range over a channel, iterations only produce a single value, the values that were sent on the channel. There is no index or key value like in case of slices or maps.
So you must use:
for v := range c {
fmt.Println("Received", v)
}
This is detailed in Spec: For statements:
If the range expression is a channel, at most one iteration variable is permitted, otherwise there may be up to two.
And:
For channels, the iteration values produced are the successive values sent on the channel until the channel is closed. If the channel is nil, the range expression blocks forever.
And also:
Function calls on the left are evaluated once per iteration. For each iteration, iteration values are produced as follows if the respective iteration variables are present:
Range expression 1st value 2nd value
array or slice a [n]E, *[n]E, or []E index i int a[i] E
string s string type index i int see below rune
map m map[K]V key k K m[k] V
channel c chan E, <-chan E element e E
I know you were not intending/wishing to use an iterator for your channel for range loop, but if you do wish to have an iterator it is still possible via defining the iterator yourself outside of the loop and incrementing it each time a value in the channel is read.
var wg sync.WaitGroup
var iterator = 0
for v := range c1 {
wg.Add(1)
iterator += 1
go func(v2, i2 int) {
c2 <- timeConsumingWork(v2)
fmt.Println("item ", i2, " done")
wg.Done()
}(v, iterator)
}
wg.Wait()
close(c2)
}
Here you can see the go func is accepting the channel value(v2) and the iterator value(i2), while incrementing it each time a value is read from the channel.
Additionally I see you used buffered Waitgroups to avoid blocking with the sequential calling of your two methods. Ideal Go code should avoid buffered channels whenever possible and instead call the sending and receiving functions in an interlocked fashion so that they do not need be seperated.
I have a loop throwing multiple go routines, they call a function that makes a http get petition and calculate and object.
I want to recover the result of all those routines.
I tried using channels, but hey are empty, even if I force wait for all the routines to be done.
This is the code that starts the routines:
func main() {
pairs := getPairs() //Returns an array of strings
c := make(chan result)
for _, product := range pairs {
go getScore(product.Symbol, 1, c)
}
fmt.Println(len(c))
time.Sleep(5000 * time.Millisecond)
fmt.Println(len(c))
}
And at the end of getScore() I do this, c being the name of the channel in the function and res the result of the function
c <- res
The length of the channel is 0 in both prints.
What's the best way to get the result of the functions?
A channel is a synchronization prototype against a shared memory (in simple point of view). A buffered channel has a length but not a regular channel. Buffered channel is useful in little bit cases but not as a general approaches.
The simplest way to Just add a loop by range of pair or len of pairs:
// start processing
for _, product := range pairs {
go getScore(product.Symbol, 1, c)
}
// getting a result
for i:=0; i<len(pairs); i ++ {
result := <-c
// process a result value
}
Or another way is collecting result in another grouting:
// result and sync variable
var (
wait sync.WaitGroup
result int32
)
// start processing
for _, product := range pairs {
wait.Add(1)
go getScore(product.Symbol, 1, c)
go func() {
defer wait.Done()
// simple accumulate or maybe more complicated actions
atomic.AddInt32(&result, <-c)
}()
}
// wait finishing
wait.Wait()
c := make(chan result)
Creates an unbuffered channel. Therefore send statements, such as
c <- res
cannot proceed until another goroutine is attempting a receive operation.
In other words, execute the number of receive operations in your main goroutine matching the number of sends that will be attempted from other goroutines. Like this:
for _, product := range pairs {
go getScore(product.Symbol, 1, c)
}
for x := 0; x < len(pairs); x++ {
fmt.Println(<-c)
}
See the Go Tour section on channels, and the Effective Go section on channels for more information.
I'm trying to use Go's concurrency to speed up my code,
Here's what I have:
for i:=7; i>-1; i-- {
go func (ch chan int32, ch2 chan int32, i int, arx int32, ary int32, dirf []int8, dirg []int8) {
nx := arx + int32(dirf[i])
ny := ary + int32(dirg[i])
ch <- nx
ch2 <- ny
}(ch, ch2, i, arx,ary,dirf,dirg)
}
for i:=7; i>-1; i-- {
nxx := <- ch
nyx := <- ch2
ind := nyx*w+nxx
if imData[ind] == e[i]{
process[c]=nxx
process[c+1]=nyx
c+=2
matrix[ind]=1
}
}
After running this I'm not getting the matrix slice that I expected, it's full of zeros.
But if I run the code below, it gives the matrix slice just as the code without channels but it's too slow.
for i:=7; i>-1; i-- {
go func (ch chan int32, ch2 chan int32, i int, arx int32, ary int32, dirf []int8, dirg []int8) {
nx := arx + int32(dirf[i])
ny := ary + int32(dirg[i])
ch <- nx
ch2 <- ny
}(ch, ch2, i, arx,ary,dirf,dirg)
nxx := <- ch
nyx := <- ch2
ind := nyx*w+nxx
if imData[ind] == e[i]{
process[c]=nxx
process[c+1]=nyx
c+=2
matrix[ind]=1
}
}
What's wrong with the first one? Any ideas? I'm very new at Go. So, please be clear when you are suggesting something.
Edit:
I edited the code to have values in proper order,
type data struct {
i int
nx int32
ny int32
}
for i:=7; i>-1; i-- {
go func (ch chan data, i int, arx int32, ary int32, dirf []int8, dirg []int8) {
nx := arx + int32(dirf[i])
ny := ary + int32(dirg[i])
ch <- data{i,nx,ny}
}(ch, i, arx,ary,dirf,dirg)
}
for i:=7; i>-1; i-- {
d := <- ch
nxx := d.nx
nyx := d.ny
j := d.i
ind := nyx*w+nxx
if imData[ind] == e[j]{
process[c]=nxx
process[c+1]=nyx
c+=2
matrix[ind]=1
}
}
It works now but it's still too slow.
I'm trying to speed up this main code:
for i:=7; i>-1; i-- {
nx := arx + int32(dirf[i])
ny := ary + int32(dirg[i])
ind := ny*w+nx
if imData[ind] == e[i]{
process[c]=nx
process[c+1]=ny
c+=2
matrix[ind]=1
}
}
what do you suggest with that?
In the second case, you're certain that the goroutines are executed "in the right order", since you wait for goroutines to complete before you continue on to the next one.
An example would be this minimal example on the golang playground. To fix this, you probably want pass a struct of three members, your nx, ny and i values across the channel.
I'm suspecting your "if imData[ind] == e[i]" conditional is failing in the former case, but it's hard to tell without the setup code for the channels, and more details on what those various slices hold. Have you tried running it with a print statement to see what you get from the channels?
Also, note that if the channels in question are buffered, there's no guarantee that the values in ch and ch2 are going to be in the same order. This is very possibly your issue.
Goroutine 1 could put a value on ch, but Goroutine 2 could have put a value on ch2 before Goroutine 1 gets to it. If you have 7 goroutines, it's perfectly possible to see the following ordering on the channels (or any number of others):
ch: 1, 2, 3, 4, 5, 6, 7
ch2: 1, 3, 4, 5, 6, 7, 2
If they aren't buffered, this isn't possible with your code, but it's still technically unsafe (edit: actually, it still won't match up with i in the second loop). If the data is a set of ordered pairs, you should be sending each pair as a structure over a single channel.
Incidentally, you only need to pass variables to the go func() call if they are expected to change outside that call. ch, ch2, arx, ary, dirf, and dirg all appear to be effectively constant for this block of code, and thus don't need to be passed into that go func(). You only need to pass in i, because the loop changes it immediately after firing the enclosure into a goroutine.
Now, from a pure speed perspective, you're probably better off moving the first loop inside the go func() call. Instead of creating 7 goroutines while looping in the main routine, you can fire a single routine, and it will loop over the values and send them on the channels. If the channels are buffered to at least that size, this becomes a VERY fast operation. Incidentally, this also solves the issue with channel ordering (though it's still better to send ordered pairs as a structure on a single channel), since you only have a single goroutine trying to send on the channels.
I'm currently working through the Tour of Go, and I thought that goroutines have been used similarly to Python generators, particularly with Question 66. I thought 66 looked complex, so I rewrote it to this:
package main
import "fmt"
func fibonacci(c chan int) {
x, y := 1, 1
for {
c <- x
x, y = y, x + y
}
}
func main() {
c := make(chan int)
go fibonacci(c)
for i := 0; i < 10; i++ {
fmt.Println(<-c)
}
}
This seems to work. A couple of questions:
If I turn up the buffer size on the channel to say, 10, fibonacci would fill up 10 further spots, as quickly as possible, and main would eat up the spots as quickly as it could go. Is this right? This would be more performant than a buffer size of 1 at the expense of memory, correct?
As the channel doesn't get closed by the fibonacci sender, what happens memory-wise when we go out of scope here? My expectation is that once c and go fibonacci is out of scope, the channel and everything on it gets garbage-collected. My gut tells me this is probably not what happens.
Yes, increasing the buffer size might drastically increase the execution speed of your program, because it will reduce the number of context switches. Goroutines aren't garbage-collected, but channels are. In your example, the fibonacci goroutine will run forever (waiting for another goroutine to read from the channel c), and the channel c will never be destroyed, because the fib-goroutine is still using it.
Here is another, sightly different program, which does not leak memory and is imho more similar to Python's generators:
package main
import "fmt"
func fib(n int) chan int {
c := make(chan int)
go func() {
x, y := 0, 1
for i := 0; i <= n; i++ {
c <- x
x, y = y, x+y
}
close(c)
}()
return c
}
func main() {
for i := range fib(10) {
fmt.Println(i)
}
}
Alternatively, if you do not know how many Fibonacci numbers you want to generate, you have to use another quit channel so that you can send the generator goroutine a signal when it should stop. This is whats explained in golang's tutorial https://tour.golang.org/concurrency/4.
I like #tux21b's answer; having the channel created in the fib() function makes the calling code nice and clean. To elaborate a bit, you only need a separate 'quit' channel if there's no way to tell the function when to stop when you call it. If you only ever care about "numbers up to X", you can do this:
package main
import "fmt"
func fib(n int) chan int {
c := make(chan int)
go func() {
x, y := 0, 1
for x < n {
c <- x
x, y = y, x+y
}
close(c)
}()
return c
}
func main() {
// Print the Fibonacci numbers less than 500
for i := range fib(500) {
fmt.Println(i)
}
}
If you want the ability to do either, this is a little sloppy, but I personally like it better than testing the condition in the caller and then signalling a quit through a separate channel:
func fib(wanted func (int, int) bool) chan int {
c := make(chan int)
go func() {
x, y := 0, 1
for i := 0; wanted(i, x); i++{
c <- x
x, y = y, x+y
}
close(c)
}()
return c
}
func main() {
// Print the first 10 Fibonacci numbers
for n := range fib(func(i, x int) bool { return i < 10 }) {
fmt.Println(n)
}
// Print the Fibonacci numbers less than 500
for n := range fib(func(i, x int) bool { return x < 500 }) {
fmt.Println(n)
}
}
I think it just depends on the particulars of a given situation whether you:
Tell the generator when to stop when you create it by
Passing an explicit number of values to generate
Passing a goal value
Passing a function that determines whether to keep going
Give the generator a 'quit' channel, test the values yourself, and tell it to quit when appropriate.
To wrap up and actually answer your questions:
Increasing the channel size would help performance due to fewer context switches. In this trivial example, neither performance nor memory consumption are going to be an issue, but in other situations, buffering the channel is often a very good idea. The memory used by make (chan int, 100) hardly seems significant in most cases, but it could easily make a big performance difference.
You have an infinite loop in your fibonacci function, so the goroutine running it will run (block on c <- x, in this case) forever. The fact that (once c goes out of scope in the caller) you won't ever again read from the channel you share with it doesn't change that. And as #tux21b pointed out, the channel will never be garbage collected since it's still in use. This has nothing to do with closing the channel (the purpose of which is to let the receiving end of the channel know that no more values will be coming) and everything to do with not returning from your function.
You could use closures to simulate a generator. Here is the example from golang.org.
package main
import "fmt"
// fib returns a function that returns
// successive Fibonacci numbers.
func fib() func() int {
a, b := 0, 1
return func() int {
a, b = b, a+b
return a
}
}
func main() {
f := fib()
// Function calls are evaluated left-to-right.
fmt.Println(f(), f(), f(), f(), f())
}
Using channels to emulate Python generators kind of works, but they introduce concurrency where none is needed, and it adds more complication than's probably needed. Here, just keeping the state explicitly is easier to understand, shorter, and almost certainly more efficient. It makes all your questions about buffer sizes and garbage collection moot.
type fibState struct {
x, y int
}
func (f *fibState) Pop() int {
result := f.x
f.x, f.y = f.y, f.x + f.y
return result
}
func main() {
fs := &fibState{1, 1}
for i := 0; i < 10; i++ {
fmt.Println(fs.Pop())
}
}