What I am trying to do is to essentially create a buffered channel effect using goroutines.
What I have is a function called Elem that looks as follows:
func Elem(in, out chan int) {
for {
buf := <-in
out <- buf
}
}
Originally the purpose of this function was to avoid blocking of a goroutine when it attempts to send an int value to another goroutine via a channel, and that goroutine is not receiving the value. So instead Elem acts as an intermediary which collects the value from the channel and blocks on behalf of the sending goroutine (assuming that no one is trying to receive the value).
Coming back to the point of the question, is there a way to chain an arbitrary number of these Elem goroutines together. So essentially what I am looking for is a function that takes 2 channels and an integer. The 2 channels represent the input channel for the 1st Elem goroutine and the output channel for the last Elem goroutine in the pipeline.
func ChainElems(in, out chan int, numOfElemsToChain int) {
for i:=0; i<numOfElemsToChain; i++ { //this doesn't have to be a loop I just thought that this is how you would go about it
//chaining logic
}
}
What I did is something like this:
package main
func Elem(in, out chan int) {
for {
buf := <-in
out <- buf
}
}
func main() {
in := make(chan int)
out1 := make(chan int)
out2 := make(chan int)
out3 := make(chan int)
go Elem(in, out1)
go Elem(out1, out2)
go Elem(out2, out3)
in <- 3
in <- 6
in <- 7
fmt.Println(<-out3)
fmt.Println(<-out3)
fmt.Println(<-out3)
}
However this is sort of "hardcoded" (only for 3 goroutines). I want to make this process a function for any number of Elems that I want to chain. Is this possible?
I've just installed Go on Mac, and here's the code
package main
import (
"fmt"
"time"
)
func Product(ch chan<- int) {
for i := 0; i < 100; i++ {
fmt.Println("Product:", i)
ch <- i
}
}
func Consumer(ch <-chan int) {
for i := 0; i < 100; i++ {
a := <-ch
fmt.Println("Consmuer:", a)
}
}
func main() {
ch := make(chan int, 1)
go Product(ch)
go Consumer(ch)
time.Sleep(500)
}
I "go run producer_consumer.go", there's no output on screen, and then it quits.
Any problem with my program ? How to fix it ?
This is a rather verbose answer, but to put it simply:
Using time.Sleep to wait until hopefully other routines have completed their jobs is bad.
The consumer and producer shouldn't know anything about each other, apart from the type they exchange over the channel. Your code relies on both consumer and producer knowing how many ints will be passed around. Not a realistic scenario
Channels can be iterated over (think of them as a thread-safe, shared slice)
channels should be closed
At the bottom of this rather verbose answer where I attempt to explain some basic concepts and best practices (well, better practices), you'll find your code rewritten to work and display all the values without relying on time.Sleep. I've not tested that code, but should be fine
Right, there's a couple of problems here. Just as a bullet-list:
Your channel is buffered to 1, which is fine, but it's not necessary
Your channel is never closed
You're waiting 500ns, then exit regardless of the routines having completed, or even started processing for that matter.
There's no centralised control on over the routines, once you've started them, you have 0 control. If you hit ctrl+c, you might want to cancel routines when writing code that'll handle important data. Check signal handling, and context for this
Channel buffer
Seeing as you already know how many values you're going to push onto your channel, why not simply create ch := make(chan int, 100)? That way your publisher can continue to push messages onto the channel, regardless of what the consumer does.
You don't need to do this, but adding a sensible buffer to your channel, depending on what you're trying to do, is definitely worth checking out. At the moment, though, both routines are using fmt.Println & co, which is going to be a bottleneck either way. Printing to STDOUT is thread-safe, and buffered. This means that each call to fmt.Print* is going to acquire a lock, to avoid text from both routines to be combined.
Closing the channel
You could simply push all the values onto your channel, and then close it. This is, however, bad form. The rule of thumb WRT channels is that channels are created and closed in the same routine. Meaning: you're creating the channel in the main routine, that's where it should be closed.
You need a mechanism to sync up, or at least keep tabs on whether or not your routines have completed their job. That's done using the sync package, or through a second channel.
// using a done channel
func produce(ch chan<- int) <-chan struct{} {
done := make(chan struct{})
go func() {
for i := 0; i < 100; i++ {
ch <- i
}
// all values have been published
// close done channel
close(done)
}()
return done
}
func main() {
ch := make(chan int, 1)
done := produce(ch)
go consume(ch)
<-done // if producer has done its thing
close(ch) // we can close the channel
}
func consume(ch <-chan int) {
// we can now simply loop over the channel until it's closed
for i := range ch {
fmt.Printf("Consumed %d\n", i)
}
}
OK, but here you'll still need to wait for the consume routine to complete.
You may have already noticed that the done channel technically isn't closed in the same routine that creates it either. Because the routine is defined as a closure, however, this is an acceptable compromise. Now let's see how we could use a waitgroup:
import (
"fmt"
"sync"
)
func product(wg *sync.WaitGroup, ch chan<- int) {
defer wg.Done() // signal we've done our job
for i := 0; i < 100; i++ {
ch <- i
}
}
func main() {
ch := make(chan int, 1)
wg := sync.WaitGroup{}
wg.Add(1) // I'm adding a routine to the channel
go produce(&wg, ch)
wg.Wait() // will return once `produce` has finished
close(ch)
}
OK, so this looks promising, I can have the routines tell me when they've finished their tasks. But if I add both consumer and producer to the waitgroup, I can't simply iterate over the channel. The channel will only ever get closed if both routines invoke wg.Done(), but if the consumer is stuck looping over a channel that'll never get closed, then I've created a deadlock.
Solution:
A hybrid would be the easiest solution at this point: Add the consumer to a waitgroup, and use the done channel in the producer to get:
func produce(ch chan<- int) <-chan struct{} {
done := make(chan struct{})
go func() {
for i := 0; i < 100; i++ {
ch <- i
}
close(done)
}()
return done
}
func consume(wg *sync.WaitGroup, ch <-chan int) {
defer wg.Done()
for i := range ch {
fmt.Printf("Consumer: %d\n", i)
}
}
func main() {
ch := make(chan int, 1)
wg := sync.WaitGroup{}
done := produce(ch)
wg.Add(1)
go consume(&wg, ch)
<- done // produce done
close(ch)
wg.Wait()
// consumer done
fmt.Println("All done, exit")
}
I have changed slightly(expanded time.Sleep) your code. Works fine on my Linux x86_64
func Product(ch chan<- int) {
for i := 0; i < 10; i++ {
fmt.Println("Product:", i)
ch <- i
}
}
func Consumer(ch <-chan int) {
for i := 0; i < 10; i++ {
a := <-ch
fmt.Println("Consmuer:", a)
}
}
func main() {
ch := make(chan int, 1)
go Product(ch)
go Consumer(ch)
time.Sleep(10000)
}
Output
go run s1.go
Product: 0
Product: 1
Product: 2
As JimB hinted at, time.Sleep takes a time.Duration, not an integer. The godoc shows an example of how to call this correctly. In your case, you probably want:
time.Sleep(500 * time.Millisecond)
The reason that your program is exiting quickly (but not giving you an error) is due to the (somewhat surprising) way that time.Duration is implemented.
time.Duration is simply a type alias for int64. Internally, it uses the value to represent the duration in nanoseconds. When you call time.Sleep(500), the compiler will gladly interpret the numeric literal 500 as a time.Duration. Unfortunately, that means 500 ns.
time.Millisecond is a constant equal to the number of nanoseconds in a millisecond (1,000,000). The nice thing is that requiring you to do that multiplication explicitly makes it obvious to that caller what the units are on that argument. Unfortunately, time.Sleep(500) is perfectly valid go code but doesn't do what most beginners would expect.
go version go1.11.4 darwin/amd64
A new channel and goroutine were created, and the content of the old channel was transferred to the new channel through goroutine. It should not block, but after testing, it was found to be blocked.
thanks.
type waiter struct {
ch1 chan struct{}
ch2 <-chan time.Time
limit int
count int
}
func (w *waiter) recv1Block() chan struct{} {
ch := make(chan struct{})
go func() {
for m := range w.ch1 {
ch <- m
}
}()
return ch
}
func (w *waiter) runBlock(wg *sync.WaitGroup) {
defer wg.Done()
i := 0
for i < w.limit {
select {
case <-w.recv1Block(): **// why block here?**
i++
case <-w.recv2():
}
}
w.count = i
}
why recv1Block will be block.
Every time you call recv1Block(), it creates a new channel and launches a background goroutine that tries to read all of the data from it. Since you're calling it in a loop, you will have many things all trying to consume the data from the channel; since the channel never closes, all of the goroutines will run forever.
As an exercise, you might try changing your code to pass around a chan int instead of a chan struct{}, and write a series of sequential numbers, and print them out as they're ultimately received. A sequence like this is valid:
run on goroutine #1 calls recv1Block().
recv1Block() on GR#1 spawns GR#2, and returns channel#2.
run on GR#1 blocks receiving on channel#2.
recv1Block() on GR#2 reads 0 from w.c1.
recv1Block() on GR#2 writes 0 to channel#2 (where run on GR#1 is ready to read).
recv1Block() on GR#2 reads 1 from w.c1.
recv1Block() on GR#2 wants to write 0 to channel#2 but blocks.
run on GR#1 wakes up, and receives the 0.
run on GR#1 calls recv1Block().
recv1Block() on GR#1 spawns GR#3, and returns channel #3.
recv1Block() on GR#3 reads 2 from w.c1.
...
Notice that the value 1 in this sequence will never be written anywhere, and in fact there is nothing left that could read it.
The easy solution here is to not call the channel-creating function in a loop:
func (w *waiter) runBlock(wg *sync.WaitGroup) {
defer wg.Done()
ch1 := w.recv1Block()
ch2 := w.recv2()
for {
select {
case _, ok := <-ch1:
if !ok {
return
}
w.count++
case <-ch2:
}
}
It's also standard practice to close channels when you're done with them. This will terminate a for ... range ch loop, and it will appear as readable to a select statement. In your top-level generator function:
for i := 0; i < w.limit; i++ {
w.ch1 <- struct{}{}
}
close(w.ch1)
And in your "copy the channel" function:
func (w *waiter) recv1Block() chan struct{} {
ch := make(chan struct{})
go func() {
for m := range w.ch1 {
ch <- m
}
close(ch)
}()
return ch
}
This also means that you don't need to run the main loop by "dead reckoning", expecting it to produce exactly 100 items then stop; you can stop whenever its channel exits. The consumer loop I show above does this.
i'm new to Go not sure why this is deadlock ? i want to constantly be reading results from doSomething and storing it in the function read without using a for loop
func doSomething(c chan<- string){ // recursive function does something
c <- result
return dosomething(c) }
func reads(c <-chan string){
results := ""
temp := <-c
results = results + "\n" + temp
return results
}
func main(){
go reads(c)
doSomething(c)
}
Main gorouting is trying to write multiple times to a channel in doSomething function. The read function is reading the channel only once. Therefore the write operation will wait until some other party reads from the channel. This will deadlock as the main goroutine is blocked.
If the blocking operations were not in main goroutine, the program would finish as the Go program ends whenever main goroutine ends. There would be no deadlock if main function could come to an end.
You are trying to read from an empty channel because reads executed concurrently and doSomething didn't. It is possible to solve the problem in several ways. Note, it is not about correct architecture or efficient approaches. An examples below solve "deadlock" issue of the original snippet, not more.
Read and write concurrently:
package main
func doSomething(c chan<- string) { // recursive function does something
c <- "result"
doSomething(c)
}
func reads(c <-chan string) {
results := <-c
fmt.Println("Boo", results)
}
func main() {
c := make(chan string)
go reads(c)
go doSomething(c) // Write concurrentely
}
Use select to handle channels read operation:
func reads(c <-chan string) {
// Use select
select {
case res := <-c:
fmt.Println("received message", res)
default:
fmt.Println("no results received")
}
}
I'd rather prefer combination of the first and second approaches.
Read after write (it is far from correct design as hell):
func main() {
c := make(chan string)
go doSomething(c)
reads(c) // Read after write
}
I'm just playing around with Go, taking it for a test drive so to speak. I'm having a problem where a go routine that is mean to receive 3 integers only seems to receive one.
type simpleFunction func() int
func run(fChan chan simpleFunction, result chan int) {
for{
select {
case fn := <-fChan:
fmt.Printf("sending: %d down result chan\n", fn())
result <- fn()
case <-time.After(time.Second * 2):
close(fChan)
}
}
}
func recieve(result chan int){
for {
select {
case x := <-result:
fmt.Printf("recieved: %d from result chan\n", x)
case <-time.After(time.Second * 2):
close(result)
}
}
}
So, as you can see the run routine receives functions, evaluates them, and then sends the result down the result channel.
Here's my main/test:
func main() {
fns := []simpleFunction{
func() int {return 1},
func() int {return 2},
func() int {return 3},
}
fChan := make(chan simpleFunction)
result := make(chan int)
go run(fChan, result)
go recieve(result)
for _, fn := range fns {
fmt.Printf("sending a function that returns: %d down function chan\n", fn())
fChan <- fn
}
}
And here's my output:
sending a function that returns: 1 down function chan
sending: 1 down result chan
recieved: 1 from result chan
sending a function that returns: 2 down function chan
sending a function that returns: 3 down function chan
sending: 2 down result chan
sending: 3 down result chan
So, as you can see, everything seems to go swimmingly for the first function, but it's not so hot afterwards. Any tips or suggestions?
There are a couple of issues with this code:
The program terminates when main returns. It does not wait for the run and receive goroutines to complete.
There's a race on closing the channels. There's no guarantee that the sender will top sending before the timeout.
If main does not exit, then the for { select { } } loops will spin forever printing zero values. Receive on a closed channel returns the zero value.