I am trying to play around with goroutines and channel
package main
import (
"fmt"
"math/rand"
"time"
)
func boring(msg string) <-chan string {
c := make(chan string)
go func() {
for i := 0; ; i++ {
c <- fmt.Sprintf("%s %d", msg, i)
time.Sleep(time.Duration(rand.Intn(1e3)) * time.Millisecond)
}
}()
return c
}
func main() {
c := fanInNew(boring("joe"), boring("anh"))
for i := 0; i < 10; i++ {
fmt.Println(<-c)
}
fmt.Println("You both are boring, I am leaving")
}
func fanInNew(input1, input2 <-chan string) <-chan string {
c := make(chan string)
for {
select {
case s := <-input1:
c <- s
case s := <-input2:
c <- s
}
}
return c
}
If i run this program it is giving me error, all goroutines are asleep, deadlock.
But If I put select inside anonymous goroutine, it works just fine. Working example:
package main
import (
"fmt"
"math/rand"
"time"
)
func boring(msg string) <-chan string {
c := make(chan string)
go func() {
for i := 0; ; i++ {
c <- fmt.Sprintf("%s %d", msg, i)
time.Sleep(time.Duration(rand.Intn(1e3)) * time.Millisecond)
}
}()
return c
}
func main() {
c := fanInNew(boring("joe"), boring("anh"))
for i := 0; i < 10; i++ {
fmt.Println(<-c)
}
fmt.Println("You both are boring, I am leaving")
}
func fanInNew(input1, input2 <-chan string) <-chan string {
c := make(chan string)
go func() {
for {
select {
case s := <-input1:
c <- s
case s := <-input2:
c <- s
}
}
}()
return c
}
Can you please help me to understand reasoning behind it.
The for statements loops forever, and so the <-c chan is never passed along and the chans get filled, but the main thread gets stuck waiting for c := fanInNew(a, b).
fanInNew() never returns because for loops forever (and blocks on select btw):
func fanInNew(input1, input2 <-chan string) <-chan string {
c := make(chan string)
for { // Loop forever and read from inputs 1 and 2
select {
case s := <-input1:
c <- s
case s := <-input2:
c <- s
}
}
return c
}
Then in the main thread this function never returns the c chan.
func main() {
// Never gets passed the next line
c := fanInNew(boring("joe"), boring("anh"))
}
So you can put the for loops themselves in goroutines, as you did in the second example. Also typically goroutines should return, either because you pass a message in (such as by close()ing), or because they reach a return statement.
In any case, what you have in the 2nd example is great for demonstrating the use of anonymous closures. The chan passed into the goroutine can be returned elsewhere and enable safe message passing between threads:
c := make(chan string)
go func() {
for {
select {
case s := <-input1:
c <- s
case s := <-input2:
c <- s
}
}
}()
return c
There are a few ways of ending a for loop in an anonymous goroutine, including select on a second chan, a closing channel, which when close()ed you can return. Also, typically WaitGroups can achieve that.
Related
I'm trying to learn channels in Go, so this is a contrived example of having a channel where there are multiple writers, but only one reader. This is a very basic example, but I would like to take it further, by imagining its a http server, where there is a goroutine created for each new request, and each goroutine makes a write on a channel.
func Start2(){
var wg sync.WaitGroup
wg.Add(2)
c := make(chan int)
go func(){
defer wg.Done()
i := 0
for {
i+=2
c<-i
time.Sleep(time.Duration(rand.Intn(5)) * time.Second)
if i > 10 {
break
}
}
}()
go func(){
defer wg.Done()
i := 1
for {
i+=2
c<-i
time.Sleep(time.Duration(rand.Intn(5)) * time.Second)
if i > 10 {
break
}
}
}()
for a := range c {
log.Println(a)
}
wg.Wait()
}
Why does this deadlock?
How could I organise the logic of reading from the channel, so that the code doesn't deadlock when there are no writers on the channel? (For instance, in the end goal example, if there isn't currently a request being made to the http server)?
Unbuffered channel needs at least two goroutines to operate and by exiting one of then. the Go runtime is smart enough to detect the deadlock, so you have two options here:
Since you want to use the channel in the http server - there is no deadlock here:
package main
import (
"fmt"
"log"
"net/http"
"time"
)
func main() {
http.HandleFunc("/", home)
go func() {
err := http.ListenAndServe(":8080", nil)
if err != nil {
log.Fatal(err)
}
}()
count := 0
for a := range c {
count++
fmt.Println(count, a)
}
}
var c = make(chan time.Time)
func home(w http.ResponseWriter, r *http.Request) {
fmt.Fprintln(w, "Hi")
c <- time.Now()
}
Or close the channel to force the main goroutine to exit too - just for your current example code - try it:
package main
import (
"log"
"math/rand"
"sync"
"time"
)
func main() {
wg := &sync.WaitGroup{}
wg.Add(2)
c := make(chan int)
go routine(0, c, wg)
go routine(1, c, wg)
go func() {
wg.Wait()
close(c)
}()
for a := range c {
log.Println(a)
}
}
func routine(i int, c chan int, wg *sync.WaitGroup) {
defer wg.Done()
for {
i += 2
c <- i
time.Sleep(time.Duration(rand.Intn(5)) * time.Second)
if i > 10 {
break
}
}
}
You are not closing your channel, you need to close it when you are done.
You can create a goroutine for reading from your channel and close it after the wait group.
Example:
var wg sync.WaitGroup
wg.Add(2)
c := make(chan int)
go func() {
defer wg.Done()
i := 0
for {
i += 2
c <- i
time.Sleep(time.Duration(rand.Intn(5)) * time.Second)
if i > 10 {
return
}
}
}()
go func() {
defer wg.Done()
i := 1
for {
i += 2
c <- i
time.Sleep(time.Duration(rand.Intn(5)) * time.Second)
if i > 10 {
break
}
}
}()
//read channel in a goroutine
go func() {
for a := range c {
log.Println(a)
}
}()
wg.Wait()
//close channel when done
close(c)
I have been working in Golang for a long time. But still I am facing this problem though I know the solution to my problem. But never figured out why is it happening.
For example If I have a pipeline situation for inbound and outbound channels like below:
package main
import (
"fmt"
)
func main() {
for n := range sq(sq(gen(3, 4))) {
fmt.Println(n)
}
fmt.Println("Process completed")
}
func gen(nums ...int) <-chan int {
out := make(chan int)
go func() {
for _, n := range nums {
out <- n
}
close(out)
}()
return out
}
func sq(in <-chan int) <-chan int {
out := make(chan int)
go func() {
for n := range in {
out <- n * n
}
close(out)
}()
return out
}
It does not give me a deadlock situation. But if I remove the go routine inside the outbound code as below:
func sq(in <-chan int) <-chan int {
out := make(chan int)
for n := range in {
out <- n * n
}
close(out)
return out
}
I received a deadlock error. Why is it so that looping over channels using range without go routine gives a deadlock.
This situation caused of output channel of sq function is not buffered. So sq is waiting until next function will read from output, but if sq is not async, it will not happen (Playground link):
package main
import (
"fmt"
"sync"
)
var wg sync.WaitGroup
func main() {
numsCh := gen(3, 4)
sqCh := sq(numsCh) // if there is no sq in body - we are locked here until input channel will be closed
result := sq(sqCh) // but if output channel is not buffered, so `sq` is locked, until next function will read from output channel
for n := range result {
fmt.Println(n)
}
fmt.Println("Process completed")
}
func gen(nums ...int) <-chan int {
out := make(chan int)
go func() {
for _, n := range nums {
out <- n
}
close(out)
}()
return out
}
func sq(in <-chan int) <-chan int {
out := make(chan int, 100)
for n := range in {
out <- n * n
}
close(out)
return out
}
Your function creates a channel, writes to it, then returns it. The writing will block until somebody can read the corresponding value, but that's impossible because nobody outside this function has the channel yet.
func sq(in <-chan int) <-chan int {
// Nobody else has this channel yet...
out := make(chan int)
for n := range in {
// ...but this line will block until somebody reads the value...
out <- n * n
}
close(out)
// ...and nobody else can possibly read it until after this return.
return out
}
If you wrap the loop in a goroutine then both the loop and the sq function are allowed to continue; even if the loop blocks, the return out statement can still go and eventually you'll be able to connect up a reader to the channel.
(There's nothing intrinsically bad about looping over channels outside of goroutines; your main function does it harmlessly and correctly.)
The reason of the deadlock is because the main is waiting for the sq return and finish, but the sq is waiting for someone read the chan then it can continue.
I simplified your code by removing layer of sq call, and split one sentence into 2 :
func main() {
result := sq(gen(3, 4)) // <-- block here, because sq doesn't return
for n := range result {
fmt.Println(n)
}
fmt.Println("Process completed")
}
func gen(nums ...int) <-chan int {
out := make(chan int)
go func() {
for _, n := range nums {
out <- n
}
close(out)
}()
return out
}
func sq(in <-chan int) <-chan int {
out := make(chan int)
for n := range in {
out <- n * n // <-- block here, because no one is reading from the chan
}
close(out)
return out
}
In sq method, if you put code in goroutine, then the sq will returned, and main func will not block, and consume the result queue, and the goroutine will continue, then there is no block any more.
func main() {
result := sq(gen(3, 4)) // will not blcok here, because the sq just start a goroutine and return
for n := range result {
fmt.Println(n)
}
fmt.Println("Process completed")
}
func gen(nums ...int) <-chan int {
out := make(chan int)
go func() {
for _, n := range nums {
out <- n
}
close(out)
}()
return out
}
func sq(in <-chan int) <-chan int {
out := make(chan int)
go func() {
for n := range in {
out <- n * n // will not block here, because main will continue and read the out chan
}
close(out)
}()
return out
}
The code is a bit complicated,
Let's simplify
First eq below, not has deadLock
func main() {
send := make(chan int)
receive := make(chan int)
go func() {
send<-3
send<-4
close(send)
}()
go func() {
receive<- <-send
receive<- <-send
close(receive)
}()
for v := range receive{
fmt.Println(v)
}
}
Second eq below,remove "go" has deadLock
func main() {
send := make(chan int)
receive := make(chan int)
go func() {
send<-3
send<-4
close(send)
}()
receive<- <-send
receive<- <-send
close(receive)
for v := range receive{
fmt.Println(v)
}
}
Let's simplify second code again
func main() {
ch := make(chan int)
ch <- 3
ch <- 4
close(ch)
for v := range ch{
fmt.Println(v)
}
}
The reason of the deadlock is no buffer channel waiting in main goroutine.
Two Solutions
// add more cap then "channel<-" time
func main() {
ch := make(chan int,2)
ch <- 3
ch <- 4
close(ch)
for v := range ch{
fmt.Println(v)
}
}
//async "<-channel"
func main() {
ch := make(chan int)
go func() {
for v := range ch {
fmt.Println(v)
}
}()
ch <- 3
ch <- 4
close(ch)
}
My understanding is
when the main thread is blocked for waiting for the chan to be writen or read, Go will detect if any other Go routine is running. If there is no any other Go routine running, it will have "fatal error: all goroutines are asleep - deadlock!"
I tested if by using the below simple case
func main() {
c := make(chan int)
go func() {
time.Sleep(10 * time.Second)
}()
c <- 1
}
The deadlock error is reported after 10 seconds.
Consider the toy example below. The code works perfectly but when you interchange the 2 lines marked as replace, there will be a deadlock. Is there a better way to deal with such a situation when you have different number of sends and receives?
package main
import "fmt"
import "strconv"
func main() {
a := make(chan string)
b := make(chan string)
go func() {
for i := 0; i < 2; i++ {
go func(i int) {
fmt.Println(<-a)
b <- strconv.Itoa(i) + "b" // replace
a <- strconv.Itoa(i) + "a" // replace
}(i)
}
}()
a <- "0"
for i := 0; i < 2; i++ {
fmt.Println(<-b)
}
}
EDIT: using a select statement, there's a chance that a gets picked up by the select and there's still no way to prevent a deadlock because the goroutines can't execute
package main
import "fmt"
import "strconv"
func main() {
a := make(chan string)
b := make(chan string)
c := make(chan bool)
cancel := make(chan bool)
go func() {
for i := 0; i < 2; i++ {
go func(i int) {
fmt.Println(<-a)
b <- strconv.Itoa(i) + "b" // replace
a <- strconv.Itoa(i) + "a" // replace
c <- true
}(i)
}
}()
go func() {
<-c
<-c
cancel <- true
}()
a <- "0"
loop:
for {
select {
case ain := <-a:
fmt.Println("select", ain)
case bin := <-b:
fmt.Println("select", bin)
case <-cancel:
break loop
}
}
}
Use select:
package main
import "fmt"
import "strconv"
func main() {
a := make(chan string)
b := make(chan string)
go func() {
for i := 0; i < 2; i++ {
go func(i int) {
fmt.Println(<-a)
b <- strconv.Itoa(i) + "b" // replace
a <- strconv.Itoa(i) + "a" // replace
}(i)
}
}()
// regardless of which comes in first, this will handle it
select {
case ain <- a:
fmt.Println("sent a", ain)
case bin <- b:
fmt.Println("sent b", bin)
case <- cancel:
break
}
}
That example will sit and block for an item sent on either a or b channels.
Optionally I usually set a cancel token or a timeout.
Your original code deadlocks on the switch because you sent on B, where you were only listening on A. Golang requires you to be listening on the channel BEFORE you ever send to it. This is the pattern for multiple channels, not knowing which you are going to get first.
I'm working on understanding Go's channels. I think I understand a basic bidirectional chan but I'm falling short at understanding <-chan and chan<-.
I expected them to be useful for communicating one way to a thread but I'm having issues with the thread actually reading and receiving the value.
package main
import (
"fmt"
"time"
)
func Thread(c chan<- int) {
for {
num := <-c
fmt.Println("Thread : ", num)
time.Sleep(time.Second)
}
}
func main() {
c := make(chan<- int, 3)
go Thread(c)
for i := 1; i <= 10; i++ {
c <- i
}
for len(c) > 0 {
time.Sleep(100)
}
}
I've tried using <-chan instead of chan<- in the make() but the same kind of thing happens:
C:\>go run chan.go
# command-line-arguments
.\chan.go:10: invalid operation: <-c (receive from send-only type chan<- int)
If I can't read from the channel, why bother writing to it? With that thought in mind, I figure I must be doing something wrong. I had the expectation that a send only chan would mean that one thread can only send while the other thread can only receive. This does not seem to be the case.
If I remove the <- entirely, it works, but that would make it bidirectional allowing the go routine to respond (even though it never does) and I'm looking to avoid that. It seems like I can banish numbers to a chan that I'll never be able to read from, or that I could read from a chan that's impossible to write to.
What I'm hoping to do is send integers from the main thread to the go routine for it to print using a one way channel. What am I doing wrong?
This is with go 1.3.3 on Windows if it matters. Updating to 1.4 didn't help. I might want to mention this is all x64 as well.
The Go Programming Language Specification
Channel types
A channel provides a mechanism for concurrently executing functions to
communicate by sending and receiving values of a specified element
type. The value of an uninitialized channel is nil.
ChannelType = ( "chan" | "chan" "<-" | "<-" "chan" ) ElementType .
The optional <- operator specifies the channel direction, send or
receive. If no direction is given, the channel is bidirectional. A
channel may be constrained only to send or only to receive by
conversion or assignment.
You can be explicit about channel direction by conversion or assignment. For example, by conversion,
package main
import (
"fmt"
"time"
)
func Thread(r <-chan int) {
for {
num := <-r
fmt.Println("Thread : ", num)
time.Sleep(time.Second)
}
}
func main() {
c := make(chan int, 3)
s, r := (chan<- int)(c), (<-chan int)(c)
go Thread(r)
for i := 1; i <= 10; i++ {
s <- i
}
for len(c) > 0 {
time.Sleep(100)
}
}
Output:
Thread : 1
Thread : 2
Thread : 3
. . .
Or, equivalently, by assignment,
package main
import (
"fmt"
"time"
)
func Thread(r <-chan int) {
for {
num := <-r
fmt.Println("Thread : ", num)
time.Sleep(time.Second)
}
}
func main() {
c := make(chan int, 3)
var s chan<- int = c
var r <-chan int = c
go Thread(r)
for i := 1; i <= 10; i++ {
s <- i
}
for len(c) > 0 {
time.Sleep(100)
}
}
Go allows to create send- or receive-only channels like c := make(<-chan int), however, I haven't came across a use case. There is a discussion here in github.
About the error in the code;
func Thread(c chan<- int) {
for {
num := <-c
fmt.Println("Thread : ", num)
time.Sleep(time.Second)
}
}
Function parameter c can only be sent to. Attempting to receive from c will result in a compiler error but the line num := <-c tries to receive.
At the end, fixed version;
package main
import (
"fmt"
"time"
)
func Thread(c <-chan int) {
for {
num := <-c
fmt.Println("Thread : ", num)
time.Sleep(time.Second)
}
}
func main() {
c := make(chan int, 3)
go Thread(c)
for i := 1; i <= 10; i++ {
c <- i
}
for len(c) > 0 {
time.Sleep(100)
}
}
Your Thread function is passing a send-only channel, instead of a receive-only channel. But you don't need to make() it receive-only, you can pass a bidirectional channel to a goroutine as receive-only.
package main
import (
"fmt"
"time"
)
func Thread(c <-chan int) {
for {
num := <-c
fmt.Println("Thread : ", num)
time.Sleep(time.Second)
}
}
func main() {
c := make(chan int, 3)
go Thread(c)
for i := 1; i <= 10; i++ {
c <- i
}
for len(c) > 0 {
time.Sleep(100)
}
}
You can verify this by adding the line
c <- 3
in your Thread function, throwing the error prog.go:11: invalid operation: c <- 3 (send to receive-only type <-chan int)
See https://play.golang.org/p/5KNPNLqA_k
I am a beginner in go.
I am trying to figure out an easy way to implement a channel that only output distinct values.
What I want to do is this:
package example
import (
"fmt"
"testing"
)
func TestShouldReturnDistinctValues(t *testing.T) {
var c := make([]chan int)
c <- 1
c <- 1
c <- 2
c <- 2
c <- 3
for e := range c {
// only print 1, 2 and 3.
fmt.println(e)
}
}
Should I be concern about memory leak here if I were to use a map to remember previous values?
You really can't do that, you'd have to keep a track of the values somehow, a map[int]struct{} is probably the most memory efficient way.
A simple example:
func UniqueGen(min, max int) <-chan int {
m := make(map[int]struct{}, max-min)
ch := make(chan int)
go func() {
for i := 0; i < 1000; i++ {
v := min + rand.Intn(max)
if _, ok := m[v]; !ok {
ch <- v
m[v] = struct{}{}
}
}
close(ch)
}()
return ch
}
I have done similar things before, except my problem was output inputs in ascending order. You can do this by adding a middle go routine. Here is an example:
package main
func main() {
input, output := distinct()
go func() {
input <- 1
input <- 1
input <- 2
input <- 2
input <- 3
close(input)
}()
for i := range output {
println(i)
}
}
func distinct() (input chan int, output chan int) {
input = make(chan int)
output = make(chan int)
go func() {
set := make(map[int]struct{})
for i := range input {
if _, ok := set[i]; !ok {
set[i] = struct{}{}
output <- i
}
}
close(output)
}()
return
}