I read Twelve Go Best Practices and encounter and interesting example on page 30.
func sendMsg(msg, addr string) error {
conn, err := net.Dial("tcp", addr)
if err != nil {
return err
}
defer conn.Close()
_, err = fmt.Fprint(conn, msg)
return err
}
func broadcastMsg(msg string, addrs []string) error {
errc := make(chan error)
for _, addr := range addrs {
go func(addr string) {
errc <- sendMsg(msg, addr)
fmt.Println("done")
}(addr)
}
for _ = range addrs {
if err := <-errc; err != nil {
return err
}
}
return nil
}
func main() {
addr := []string{"localhost:8080", "http://google.com"}
err := broadcastMsg("hi", addr)
time.Sleep(time.Second)
if err != nil {
fmt.Println(err)
return
}
fmt.Println("everything went fine")
}
The programmer mentioned, that happens to the code above:
the goroutine is blocked on the chan write
the goroutine holds a reference to the chan
the chan will never be garbage collected
Why the goroutine is blocked here? The main thread is blocked, until it receive data from goroutine. After it continues the for loop. Not?
Why the errc chan will be never garbage collected? Because I do not close the channel, after goroutine is finished?
One problem I see is that inside broadcastMsg() after goroutines have started:
for _ = range addrs {
if err := <-errc; err != nil {
return err
}
}
If a non-nil error is received from errc, broadcastMsg() returns immediately with that error and does not receive futher values from the channel, which means further goroutines will never get unblocked because errc is unbuffered.
Possible Fixes
A possible fix would be to use a buffered channel, big enough to not block any of the goroutines, in this case:
errc := make(chan error, len(addrs))
Or even if a non-nil error is received from the channel, still proceed to receive as many times as many goroutines send on it:
var errRec error
for _ = range addrs {
if err := <-errc; err != nil {
if errRec == nil {
errRec = err
}
}
}
return errRec
Or as mentioned in the linked talk on slide #33: use a "quit" channel to prevent the started goroutines to remain blocked after broadcastMsg() has completed/returned.
You have a list of two addresses (localhost, google). To each of these you're sending a message (hi), using one goroutine per address. And the goroutine sends error (which may be nil) to errc channel.
If you send something to a channel, you also need something that reads the values from that channel, otherwise it will block (unless it's a buffered channel, but even buffered channels block once their buffer is full).
So your reading loop looks like this:
for _ = range addrs {
if err := <-errc; err != nil {
return err
}
}
If the first address returns an error which is not nil, the loop returns. The subsequent error values are never read from the channel thus it blocks.
Related
given the following variables
wg.Add(4)
pDayResCh := make(chan map[string]map[string]int)
pDayErrCh := make(chan error)
The following code hangs
// Convert the previous, current, and next day prayers and put them in map
go func() {
fmt.Println("executing first go routine") // TODO: Remove this line after debug
data, err := dayPrayerMapConv(previousDayPrayers)
fmt.Printf("first goroutine result %v\n", data) // TODO: Remove this line after debug
if err != nil {
fmt.Printf("first goroutine err != nil %v\n", err)
pDayErrCh <- err
}
fmt.Printf("first goroutine putting data into channel")
pDayResCh <- data
fmt.Printf("first go routine finished") // TODO: Remove this line after debug
wg.Done()
}()
pDayErr := <-pDayErrCh
close(pDayErrCh)
if pDayErr != nil {
return pDayErr
}
fmt.Println("pday err finised")
p.PreviousDayPrayers = <-pDayResCh
close(pDayResCh)
This is the result of the print statement
first goroutine result map[Asr:map[Hour:3 Minute:28] Dhuhr:map[Hour:12 Minute:23] Fajr:map[Hour:5 Minute:32] Isha:map[Hour:7 Minute:5] Maghrib:map[Hour:6 Minute:13]]
first goroutine putting data into channel
So there is data in the data variable that should have been passed into pDayResCh, but it seems to get stuck there, why?
Due to the condition in my goroutine where I check for the error
if err != nil {
fmt.Printf("first goroutine err != nil %v\n", err)
pDayErrCh <- err
}
Because error is nil, that data never gets passed into the channel, thus hangs
Removing the condition and passing the err regardless solves the issue.
I have following code:
func (q *Queue) GetStreams(qi *QueueInfo) {
channel := make(chan error, len(qi.AudioChunks))
for _, audiInfo := range qi.AudioChunks {
go audiInfo.GetStream(q.APIChunkURL, q.Sender, channel)
}
i := 0
for err := range channel {
fmt.Println(i)
i++
if err != nil {
fmt.Println("Error getting audio", err)
}
}
fmt.Println("Passed")
}
func (au *AudioChunkPayload) GetStream(path string, rs *RequestSender, channel chan error) {
completeURL := fmt.Sprintf(path, au.AudioChunkID)
err := GetEncodedAudio(rs, completeURL)
if err != nil {
channel <- err
} else {
channel <- nil
}
}
that where GetStream func is downloading some data. Most of the time, lenght of qi.AudioChunks is 8, but it is not a rule. My problem is, that program succesfully prints numer 0-7 (for 8 chunks) but it never proceed to next print (fmt.Println(Passed)). What I have had read about buffered channels, I thought that it should proceed, but obviously I am wrong. How can I make my func GetStreams proceed and finish?
Range loops over a channel until it's explicitly closed.
Buffering affects sending to a channel not receiving from the channel.
I'm trying to find the official docs but from A Tour of Go:
The loop for i := range c receives values from the channel repeatedly
until it is closed.
I am trying to understand non buffered channels, so I have written a small application that iterates through an array of user input, does some work, places info on a non buffered channel and then reads it. However, I'm not able to read from the channels.
This is my code
toProcess := os.Args[1:]
var wg sync.WaitGroup
results := make(chan string)
errs := make(chan error)
for _, t := range toProcess {
wg.Add(1)
go Worker(t, "text", results, errs, &wg)
}
go func() {
for err := range errs {
if err != nil {
fmt.Println(err)
}
}
}()
go func() {
for res := range results {
fmt.Println(res)
}
}()
What am I not understanding about non buffered channels? I thought I should be placing information on it, and have another go routine reading from it.
EDIT: using two goroutines solves the issues, but it still gives me the following when there are errors:
open /Users/roosingh/go/src/github.com/nonbuff/files/22.txt: no such file or directory
fatal error: all goroutines are asleep - deadlock!
goroutine 1 [semacquire]:
sync.runtime_Semacquire(0xc42001416c)
/usr/local/Cellar/go/1.10.2/libexec/src/runtime/sema.go:56 +0x39
sync.(*WaitGroup).Wait(0xc420014160)
/usr/local/Cellar/go/1.10.2/libexec/src/sync/waitgroup.go:129 +0x72
main.main()
/Users/roosingh/go/src/github.com/nonbuff/main.go:39 +0x207
goroutine 6 [chan receive]:
main.main.func1(0xc4200780c0)
/Users/roosingh/go/src/github.com/nonbuff/main.go:25 +0x41
created by main.main
/Users/roosingh/go/src/github.com/nonbuff/main.go:24 +0x1d4
goroutine 7 [chan receive]:
main.main.func2(0xc420078060)
/Users/roosingh/go/src/github.com/nonbuff/main.go:34 +0xb2
created by main.main
/Users/roosingh/go/src/github.com/nonbuff/main.go:33 +0x1f6
So it is able to print out the error message.
My worker code is as follows;
func Worker(fn string, text string, results chan string, errs chan error, wg *sync.WaitGroup) {
file, err := os.Open(fn)
if err != nil {
errs <- err
return
}
defer func() {
file.Close()
wg.Done()
}()
reader := bufio.NewReader(file)
for {
var buffer bytes.Buffer
var l []byte
var isPrefix bool
for {
l, isPrefix, err = reader.ReadLine()
buffer.Write(l)
if !isPrefix {
break
}
if err != nil {
errs <- err
return
}
}
if err == io.EOF {
return
}
line := buffer.String()
results <- fmt. Sprintf("%s, %s", line, text)
}
if err != io.EOF {
errs <- err
return
}
return
}
As for unbuffered channels, you seem to understand the concept, meaning it's used to pass messages between goroutines but cannot hold any. Therefore, a write on an unbuffered channel will block until another goroutine is reading from the channel and a read from a channel will block until another goroutine writes to this channel.
In your case, you seem to want to read from 2 channels simultaneously in the same goroutine. Because the way channels work, you cannot range on a non closed channel and further down in the same goroutine range on another channel. Unless the first channel gets closed, you won't reach the second range.
But, it doesn't mean it's impossible! This is where the select statement comes in.
The select statement allows you to selectively read from multiple channels, meaning that it will read the first one that has something available to be read.
With that in mind, you can use the for combined with the select and rewrite your routine this way:
go func() {
for {
select {
case err := <- errs: // you got an error
fmt.Println(err)
case res := <- results: // you got a result
fmt.Println(res)
}
}
}()
Also, you don't need a waitgroup here, because you know how many workers you are starting, you could just count how many errors and results you get and stop when you reach the number of workers.
Example:
go func() {
var i int
for {
select {
case err := <- errs: // you got an error
fmt.Println(err)
i++
case res := <- results: // you got a result
fmt.Println(res)
i++
}
// all our workers are done
if i == len(toProcess) {
return
}
}
}()
I have a function that is used to forward a message between two io.ReadWriters. Once an error happens, I need to log the error and return. But I think I may have a goroutine leakage problem in my code:
func transport(rw1, rw2 io.ReadWriter) error {
errc := make(chan error, 1) // only one buffer
go func() {
_, err := io.Copy(rw1, rw2)
errc <- err
}()
go func() {
_, err := io.Copy(rw2, rw1)
errc <- err
}()
err := <-errc // only one error catched
if err != nil && err == io.EOF {
err = nil
}
return err
}
Because there is only one error can be caught in this function, will the second goroutine exit and garbaged normally? Or should I write one more err <- errc to receive another error.
The value from one goroutine is received and the other is buffered. Both goroutines can send to the channel and exit. There is no leak.
You might want to receive both values to ensure that application detects an error when the first goroutine to send is successful and the second goroutine encounters an error.
var err error
for i := 0; i < 2; i++ {
if e := <-errc; e != nil {
err = e
}
}
Because io.Copy does not return io.EOF, there's no need to check for io.EOF when collecting the errors.
The code can be simplified to use a single goroutine:
errc := make(chan error, 1)
go func() {
_, err := io.Copy(rw1, rw2)
errc <- err
}()
_, err := io.Copy(rw2, rw1)
if e := <-errc; e != nil {
err = e
}
Assuming you have a structure like this:
ch := make(chan string)
errCh := make(chan error)
go func() {
line, _, err := bufio.NewReader(r).ReadLine()
if err != nil {
errCh <- err
} else {
ch <- string(line)
}
}()
select {
case err := <-errCh:
return "", err
case line := <-ch:
return line, nil
case <-time.After(5 * time.Second):
return "", TimeoutError
}
In the case of the 5 second timeout, the goroutine hangs until ReadLine returns, which may never happen. My project is a long-running server, so I don't want a buildup of stuck goroutines.
ReadLine will not return until either the process exits or the method reads a line. There's no deadline or timeout mechanism for pipes.
The goroutine will block if the call to ReadLine returns after the timeout. This can be fixed by using buffered channels:
ch := make(chan string, 1)
errCh := make(chan error, 1)
The application should call Wait to cleanup resources associated with the command. The goroutine is a good place to call it:
go func() {
line, _, err := bufio.NewReader(r).ReadLine()
if err != nil {
errCh <- err
} else {
ch <- string(line)
}
cmd.Wait() // <-- add this line
}()
This will cause the goroutine to block, the very thing you are trying to avoid. The alternative is that the application leaks resources for each command.