I have following code:
func (q *Queue) GetStreams(qi *QueueInfo) {
channel := make(chan error, len(qi.AudioChunks))
for _, audiInfo := range qi.AudioChunks {
go audiInfo.GetStream(q.APIChunkURL, q.Sender, channel)
}
i := 0
for err := range channel {
fmt.Println(i)
i++
if err != nil {
fmt.Println("Error getting audio", err)
}
}
fmt.Println("Passed")
}
func (au *AudioChunkPayload) GetStream(path string, rs *RequestSender, channel chan error) {
completeURL := fmt.Sprintf(path, au.AudioChunkID)
err := GetEncodedAudio(rs, completeURL)
if err != nil {
channel <- err
} else {
channel <- nil
}
}
that where GetStream func is downloading some data. Most of the time, lenght of qi.AudioChunks is 8, but it is not a rule. My problem is, that program succesfully prints numer 0-7 (for 8 chunks) but it never proceed to next print (fmt.Println(Passed)). What I have had read about buffered channels, I thought that it should proceed, but obviously I am wrong. How can I make my func GetStreams proceed and finish?
Range loops over a channel until it's explicitly closed.
Buffering affects sending to a channel not receiving from the channel.
I'm trying to find the official docs but from A Tour of Go:
The loop for i := range c receives values from the channel repeatedly
until it is closed.
Related
I have two sets of code - to read a file with random lines of text, and load each line to a channel. I am unable to understand why one returns an error. but the other doesn't. Case #1 returns "fatal error: all goroutines are asleep - deadlock!" but Case # works.
Case #1
func main () {
file, err := os.Open("/Users/sample/Downloads/wordlist")
if err != nil {
log.Fatal(err)
}
lines := make (chan string)
scanner := bufio.NewScanner(file)
for scanner.Scan() {
lines <- scanner.Text()
}
close(lines)
for line := range (lines) {
fmt.Println(line)
}
}
Case #2
func main () {
file, err := os.Open("/Users/sample/Downloads/wordlist")
if err != nil {
log.Fatal(err)
}
lines := make (chan string)
go func() {
scanner := bufio.NewScanner(file)
for scanner.Scan() {
lines <- scanner.Text()
}
close(lines)
}()
for line := range (lines) {
fmt.Println(line)
}
}
The sender blocks until the receiver has received the value in unbuffered channel.
In unbuffered channel writing to channel will not happen until there must be some receiver which is waiting to receive the data.
For Example:
queue := make(chan string)
queue <- "one" /* Here main function which is also a goroutine is blocked, because
there is no goroutine to receive the channel value, hence the deadlock*/
close(queue)
for elem := range queue {
fmt.Println(elem)
}
This will leads to deadlock and it is exactly what happened in your Case #1 code.
Now In case where we have other go routine
queue := make(chan string)
go func() {
queue <- "one"
close(queue)
}()
for elem := range queue {
fmt.Println(elem)
}
This will work because main go routine is waiting for the data to be consumed before writes happen to unbuffered channel.
This is exactly happen in your Case #2
So in short, writes to unbuffered channel happens only when there is some routine waiting to read from channel, else the write operation is blocked forever and leads to deadlock.
I am trying to understand non buffered channels, so I have written a small application that iterates through an array of user input, does some work, places info on a non buffered channel and then reads it. However, I'm not able to read from the channels.
This is my code
toProcess := os.Args[1:]
var wg sync.WaitGroup
results := make(chan string)
errs := make(chan error)
for _, t := range toProcess {
wg.Add(1)
go Worker(t, "text", results, errs, &wg)
}
go func() {
for err := range errs {
if err != nil {
fmt.Println(err)
}
}
}()
go func() {
for res := range results {
fmt.Println(res)
}
}()
What am I not understanding about non buffered channels? I thought I should be placing information on it, and have another go routine reading from it.
EDIT: using two goroutines solves the issues, but it still gives me the following when there are errors:
open /Users/roosingh/go/src/github.com/nonbuff/files/22.txt: no such file or directory
fatal error: all goroutines are asleep - deadlock!
goroutine 1 [semacquire]:
sync.runtime_Semacquire(0xc42001416c)
/usr/local/Cellar/go/1.10.2/libexec/src/runtime/sema.go:56 +0x39
sync.(*WaitGroup).Wait(0xc420014160)
/usr/local/Cellar/go/1.10.2/libexec/src/sync/waitgroup.go:129 +0x72
main.main()
/Users/roosingh/go/src/github.com/nonbuff/main.go:39 +0x207
goroutine 6 [chan receive]:
main.main.func1(0xc4200780c0)
/Users/roosingh/go/src/github.com/nonbuff/main.go:25 +0x41
created by main.main
/Users/roosingh/go/src/github.com/nonbuff/main.go:24 +0x1d4
goroutine 7 [chan receive]:
main.main.func2(0xc420078060)
/Users/roosingh/go/src/github.com/nonbuff/main.go:34 +0xb2
created by main.main
/Users/roosingh/go/src/github.com/nonbuff/main.go:33 +0x1f6
So it is able to print out the error message.
My worker code is as follows;
func Worker(fn string, text string, results chan string, errs chan error, wg *sync.WaitGroup) {
file, err := os.Open(fn)
if err != nil {
errs <- err
return
}
defer func() {
file.Close()
wg.Done()
}()
reader := bufio.NewReader(file)
for {
var buffer bytes.Buffer
var l []byte
var isPrefix bool
for {
l, isPrefix, err = reader.ReadLine()
buffer.Write(l)
if !isPrefix {
break
}
if err != nil {
errs <- err
return
}
}
if err == io.EOF {
return
}
line := buffer.String()
results <- fmt. Sprintf("%s, %s", line, text)
}
if err != io.EOF {
errs <- err
return
}
return
}
As for unbuffered channels, you seem to understand the concept, meaning it's used to pass messages between goroutines but cannot hold any. Therefore, a write on an unbuffered channel will block until another goroutine is reading from the channel and a read from a channel will block until another goroutine writes to this channel.
In your case, you seem to want to read from 2 channels simultaneously in the same goroutine. Because the way channels work, you cannot range on a non closed channel and further down in the same goroutine range on another channel. Unless the first channel gets closed, you won't reach the second range.
But, it doesn't mean it's impossible! This is where the select statement comes in.
The select statement allows you to selectively read from multiple channels, meaning that it will read the first one that has something available to be read.
With that in mind, you can use the for combined with the select and rewrite your routine this way:
go func() {
for {
select {
case err := <- errs: // you got an error
fmt.Println(err)
case res := <- results: // you got a result
fmt.Println(res)
}
}
}()
Also, you don't need a waitgroup here, because you know how many workers you are starting, you could just count how many errors and results you get and stop when you reach the number of workers.
Example:
go func() {
var i int
for {
select {
case err := <- errs: // you got an error
fmt.Println(err)
i++
case res := <- results: // you got a result
fmt.Println(res)
i++
}
// all our workers are done
if i == len(toProcess) {
return
}
}
}()
I'm using two concurrent goroutines to copy stdin/stdout from my terminal to a net.Conn target. For some reason, I can't manage to completely stop the two go routines without getting a panic error (for trying to close a closed connection). This is my code:
func interact(c net.Conn, sessionMap map[int]net.Conn) {
quit := make(chan bool) //the channel to quit
copy := func(r io.ReadCloser, w io.WriteCloser) {
defer func() {
r.Close()
w.Close()
close(quit) //this is how i'm trying to close it
}()
_, err := io.Copy(w, r)
if err != nil {
//
}
}
go func() {
for {
select {
case <-quit:
return
default:
copy(c, os.Stdout)
}
}
}()
go func() {
for {
select {
case <-quit:
return
default:
copy(os.Stdin, c)
}
}
}()
}
This errors as panic: close of closed channel
I want to terminate the two go routines, and then normally proceed to another function. What am I doing wrong?
You can't call close on a channel more than once, there's no reason to call copy in a for loop, since it can only operate one time, and you're copying in the wrong direction, writing to stdin and reading from stdout.
Simply asking how to quit 2 goroutines is simple, but that's not the only thing you need to do here. Since io.Copy is blocking, you don't need the extra synchronization to determine when the call is complete. This lets you simplify the code significantly, which will make it a lot easier to reason about.
func interact(c net.Conn) {
go func() {
// You want to close this outside the goroutine if you
// expect to send data back over a half-closed connection
defer c.Close()
// Optionally close stdout here if you need to signal the
// end of the stream in a pipeline.
defer os.Stdout.Close()
_, err := io.Copy(os.Stdout, c)
if err != nil {
//
}
}()
_, err := io.Copy(c, os.Stdin)
if err != nil {
//
}
}
Also note that you may not be able to break out of the io.Copy from stdin, so you can't expect the interact function to return. Manually doing the io.Copy in the function body and checking for a half-closed connection on every loop may be a good idea, then you can break out sooner and ensure that you fully close the net.Conn.
Also could be like this
func scanReader(quit chan int, r io.Reader) chan string {
line := make(chan string)
go func(quit chan int) {
defer close(line)
scan := bufio.NewScanner(r)
for scan.Scan() {
select {
case <- quit:
return
default:
s := scan.Text()
line <- s
}
}
}(quit)
return line
}
stdIn := scanReader(quit, os.Stdin)
conIn := scanReader(quit, c)
for {
select {
case <-quit:
return
case l <- stdIn:
_, e := fmt.Fprintf(c, l)
if e != nil {
quit <- 1
return
}
case l <- conIn:
fmt.Println(l)
}
}
This is in reference to following code in The Go Programming Language - Chapter 8 p.238 copied below from this link
// makeThumbnails6 makes thumbnails for each file received from the channel.
// It returns the number of bytes occupied by the files it creates.
func makeThumbnails6(filenames <-chan string) int64 {
sizes := make(chan int64)
var wg sync.WaitGroup // number of working goroutines
for f := range filenames {
wg.Add(1)
// worker
go func(f string) {
defer wg.Done()
thumb, err := thumbnail.ImageFile(f)
if err != nil {
log.Println(err)
return
}
info, _ := os.Stat(thumb) // OK to ignore error
fmt.Println(info.Size())
sizes <- info.Size()
}(f)
}
// closer
go func() {
wg.Wait()
close(sizes)
}()
var total int64
for size := range sizes {
total += size
}
return total
}
Why do we need to put the closer in a goroutine? Why can't below work?
// closer
// go func() {
fmt.Println("waiting for reset")
wg.Wait()
fmt.Println("closing sizes")
close(sizes)
// }()
If I try running above code it gives:
waiting for reset
3547
2793
fatal error: all goroutines are asleep - deadlock!
Why is there a deadlock in above? fyi, In the method that calls makeThumbnail6 I do close the filenames channel
Your channel is unbuffered (you didn't specify any buffer size when make()ing the channel). This means that a write to the channel blocks until the value written is read. And you read from the channel after your call to wg.Wait(), so nothing ever gets read and all your goroutines get stuck on the blocking write.
That said, you do not need WaitGroup here. WaitGroups are good when you don't know when your goroutine is done, but you are sending results back, so you know. Here is a sample code that does a similar thing to what you are trying to do (with fake worker payload).
package main
import (
"fmt"
"time"
)
func main() {
var procs int = 0
filenames := []string{"file1", "file2", "file3", "file4"}
mychan := make(chan string)
for _, f := range filenames {
procs += 1
// worker
go func(f string) {
fmt.Printf("Worker processing %v\n", f)
time.Sleep(time.Second)
mychan <- f
}(f)
}
for i := 0; i < procs; i++ {
select {
case msg := <-mychan:
fmt.Printf("got %v from worker channel\n", msg)
}
}
}
Test it in the playground here https://play.golang.org/p/RtMkYbAqtGO
Although it's been a while since the question was raised, I encountered the same issue. Initially my main looked like the following:
func main() {
filenames := make(chan string, len(os.Args))
for _, f := range os.Args[1:] {
filenames <- f
}
sizes := makeThumbnails6(filenames)
close(filenames)
log.Println("Total size: ", sizes)}
This version deadlocks as the call range filenames in makeThumbnails6 is synchronous, thus close(filenames) in main was never called. The channel in makeThumbnails6 is unbuffered so goroutines block when trying to send back the size.
The solution was to move close(filenames) before making the function call in main.
The code is wrong. In short, the channel sizes is unbuffered. To fix it, we need to use a buffered channel with enough capacity when creating sizes. One-liner fix is enough, as shown. Here I just made a simple assumption that 1024 is big enough.
func makeThumbnails6(filenames chan string) int64 {
sizes := make(chan int64, 1024) // CHANGE
var wg sync.WaitGroup // number of working goroutines
for f := range filenames {
wg.Add(1)
// worker
go func(f string) {
defer wg.Done()
thumb, err := thumbnail.ImageFile(f)
if err != nil {
log.Println(err)
return
}
info, _ := os.Stat(thumb) // OK to ignore error
fmt.Println(info.Size())
sizes <- info.Size()
}(f)
}
// closer
go func() {
wg.Wait()
close(sizes)
}()
var total int64
for size := range sizes {
total += size
}
return total
}
I read Twelve Go Best Practices and encounter and interesting example on page 30.
func sendMsg(msg, addr string) error {
conn, err := net.Dial("tcp", addr)
if err != nil {
return err
}
defer conn.Close()
_, err = fmt.Fprint(conn, msg)
return err
}
func broadcastMsg(msg string, addrs []string) error {
errc := make(chan error)
for _, addr := range addrs {
go func(addr string) {
errc <- sendMsg(msg, addr)
fmt.Println("done")
}(addr)
}
for _ = range addrs {
if err := <-errc; err != nil {
return err
}
}
return nil
}
func main() {
addr := []string{"localhost:8080", "http://google.com"}
err := broadcastMsg("hi", addr)
time.Sleep(time.Second)
if err != nil {
fmt.Println(err)
return
}
fmt.Println("everything went fine")
}
The programmer mentioned, that happens to the code above:
the goroutine is blocked on the chan write
the goroutine holds a reference to the chan
the chan will never be garbage collected
Why the goroutine is blocked here? The main thread is blocked, until it receive data from goroutine. After it continues the for loop. Not?
Why the errc chan will be never garbage collected? Because I do not close the channel, after goroutine is finished?
One problem I see is that inside broadcastMsg() after goroutines have started:
for _ = range addrs {
if err := <-errc; err != nil {
return err
}
}
If a non-nil error is received from errc, broadcastMsg() returns immediately with that error and does not receive futher values from the channel, which means further goroutines will never get unblocked because errc is unbuffered.
Possible Fixes
A possible fix would be to use a buffered channel, big enough to not block any of the goroutines, in this case:
errc := make(chan error, len(addrs))
Or even if a non-nil error is received from the channel, still proceed to receive as many times as many goroutines send on it:
var errRec error
for _ = range addrs {
if err := <-errc; err != nil {
if errRec == nil {
errRec = err
}
}
}
return errRec
Or as mentioned in the linked talk on slide #33: use a "quit" channel to prevent the started goroutines to remain blocked after broadcastMsg() has completed/returned.
You have a list of two addresses (localhost, google). To each of these you're sending a message (hi), using one goroutine per address. And the goroutine sends error (which may be nil) to errc channel.
If you send something to a channel, you also need something that reads the values from that channel, otherwise it will block (unless it's a buffered channel, but even buffered channels block once their buffer is full).
So your reading loop looks like this:
for _ = range addrs {
if err := <-errc; err != nil {
return err
}
}
If the first address returns an error which is not nil, the loop returns. The subsequent error values are never read from the channel thus it blocks.