I'm writing a function that needs to run and finish as fast as possible.
It needs to make 3 REST calls and should any of these return a bad result, it needs to exit.
Each of the REST calls are being made in their own go routines and return the results to the main thread via a buffered channel.
Because I'm using buffered channels I know that the sending threads will send the results of the REST request via the buffered channel and exit - no possibility of a goroutine leak.
My question is; lets say I get the response from the first REST requests and it's a bad result (by which I mean the function as a whole needs exit), is it OK for me close the other two channels and exit without reading the contents of the other 2 buffered channels?
I have a feeling this isn't recommended and if that's they case why so?
You shouldn't close a channel which is going to be written by another goroutine. The usual pattern is to have the writer to close the channel it's writing in when it's done. If you want to cancel the call of a goroutine you should use a context.Context instead. Here is a sample synchronisation code between to goroutine using a Context to cancel another one.
package main
import (
"context"
"fmt"
"time"
)
func f(ctx context.Context, ch chan<- struct{}) {
select {
case <-time.After(time.Hour):
fmt.Println("sending data on the channel")
ch <- struct{}{}
case <-ctx.Done():
fmt.Println("closing channel")
close(ch)
}
}
func main() {
ch := make(chan struct{})
ctx, cancel := context.WithCancel(context.Background())
go f(ctx, ch)
cancel()
<-ch
}
Related
This question already has answers here:
Buffered/Unbuffered channel
(3 answers)
fatal error: all goroutines are asleep - deadlock! when channel is not buffered
(1 answer)
Why is the order of channels receiving causing/resolving a deadlock in Golang?
(3 answers)
In the Go select construct, can I have send and receive to unbuffered channel in two case statements?
(2 answers)
Closed 7 days ago.
package main
import (
"fmt"
"sync"
)
func main() {
ch := make(chan struct{})
wg := &sync.WaitGroup{}
wg.Add(1)
go func() {
defer wg.Done()
ch <- struct{}{}
}()
wg.Wait()
<-ch
fmt.Println("finished")
}
result
fnished will not be printed
package main
import (
"fmt"
"sync"
)
func main() {
ch := make(chan struct{}, 1)
wg := &sync.WaitGroup{}
wg.Add(1)
go func() {
defer wg.Done()
ch <- struct{}{}
}()
wg.Wait()
<-ch
fmt.Println("finished")
}
result
finished can be printed immediately
The only difference between the two code is that the first code use no buffer channel, but the second code use channel with a buffer.
In the first code, ch <- struct{}{} can not be run. message can not be sent to the channel.
Why does that happen? Is there any official explanation or material?
In the first example wg.Done() can only be called after a value on the channel has been sent. But since the channel is unbuffered, the send can only happen if there's a ready receiver. The main goroutine would be the receiver, but it only receives after wg.Wait(). But wg.Wait() blocks until wg.Done() is called, so this will never happen, it's a deadlock.
This is in Spec: Send statements:
Communication blocks until the send can proceed. A send on an unbuffered channel can proceed if a receiver is ready.
When using non-buffered channels, sending or receiving commands from the channel are blocking.
This means with non-buffered channel, your go routine that sends the message is not done, hence it won't call wg.Done() which will keep the wg.Wait() method in blocking state.
If you run your code on the play server https://go.dev/play/p/oDpyL6-dARP, it will panic with an informative message: fatal error: all goroutines are asleep - deadlock!
When your using buffered channels, the go routine runs because the channel has a buffer, then wg.Done() is called, which release wg.Wait() afterwards.
I am going to show a simple compilable code snipped where I get weird behaviour: after I intentionally cause a panic in processData (because I pass nil-pointer) the sending to channel l.done is blocked!
package main
import (
"context"
"fmt"
"time"
)
type Loop struct {
done chan struct{}
}
type Result struct {
ID string
}
func NewLoop() *Loop {
return &Loop{
done: make(chan struct{}),
}
}
func (l *Loop) StartAsync(ctx context.Context) {
go func() {
defer func() {
l.done <- struct{}{} // BLOCKED! But I allocated it in NewLoop ctor
fmt.Sprintf("done")
}()
for {
/**/
var data *Result
l.processData(ctx, data) // passed nil
}
}()
}
func (l *Loop) processData(ctx context.Context, data *Result) {
_ = fmt.Sprintf("%s", data.ID) // intentional panic - OK
}
func main() {
l := NewLoop()
l.StartAsync(context.Background())
time.Sleep(10 * time.Second)
}
I can recover() a panic before sending to channel and I get correct error message.
What does happen with channel? It it was closed, I would get panic on sending to closed channel
It's blocking because there isn't anything receiving from the channel. Both the receive & the send operations on an initialized and unbuffered channel will block indefinitely if the opposite operation is missing. I.e. a send to a channel will block until another goroutine receives from that channel, and, likewise, a receive from a channel will block until another goroutine sends to that channel.
So basically
done := new(chan struct{})
done<-struct{}{}
will block forever unless there is another goroutine that receives from the channel, i.e. <-done. That's how channels are supposed to behave. That's how the languages specification defines their behaviour.
about the possible fixes :
given the name of your channel, you may want to run close(l.done) rather than l.done <- struct{}{}.
using a buffered channel and l.done <- struct{}{} on completion : only one call to <-l.done will be unblocked.
Suppose you have some code looking like :
l := NewLoop()
go func(){
<-l.done
closeLoggers()
}()
go func(){
<-l.done
closeDatabase()
}()
sending one item on the done channel will make that only one consumer will receive it, and in the above example only one of the two actions will be triggered when the loop completes.
using close(l.done) : once a channel is closed all calls to receive from it will proceed.
In the above example, all actions will proceed.
As a side note: there is no need for a buffer if you use a channel only for its "open/closed" state.
I read it on (https://www.geeksforgeeks.org/channel-in-golang/) that:
"In the channel, the send and receive operation block until another side is not ready by default.
It allows goroutine to synchronize with each other without explicit locks or condition variables."
To test above statement, I have written a sample program mentioned below:
Program:
package main
import (
"fmt"
"sync"
"time"
)
func myFunc(ch chan int) {
fmt.Println("Inside goroutine:: myFunc()")
fmt.Println(10 + <-ch) //<-- According to rule, control will be blocked here until 'ch' sends some data so that it will be received in our myFunc() go routine.
}
func main() {
fmt.Println("Start Main method")
// Creating a channel
ch := make(chan int)
go myFunc(ch) //<-- This go routine started in a new thread
time.Sleep(2 * time.Second) //<--- introduced a Sleep of 2 seconds to ensure that myFunc() go routine executes before main thread
ch <- 10
fmt.Println("End Main method")
}
I was expecting below output:
Start Main method
Inside goroutine:: myFunc()
20
End Main method
But, Actual output received is:
Start Main method
Inside goroutine:: myFunc()
End Main method
Why the value sent through channel is not printed?
I think, it is because main thread finished its execution first and hence, all other goroutine also terminated.
If that is the case, then, why does the rule said - It allows goroutine to synchronize with each other without explicit locks or condition variables.
Because, to get the expected output, I have to use sync.WaitGroup to tell the main thread to wait for the other goroutine to finish. Isn't it violating the above rule as I am using locks in form of waitgroup?
PS: I am learning golang. So please forgive if I get the concept totally wrong.
The main goroutine exists before the myFunc goroutine is able to print the output. Here is an implementation which ensures that myFunc goroutine finishes before the main goroutine exits.
package main
import (
"fmt"
"sync"
"time"
)
func myFunc(ch chan int, wg *sync.WaitGroup) {
defer wg.Done()
fmt.Println("Inside goroutine:: myFunc()")
fmt.Println(10 + <-ch) //<-- According to rule, control will be blocked here until 'ch' sends some data so that it will be received in our myFunc() go routine.
}
func main() {
fmt.Println("Start Main method")
// Creating a channel
ch := make(chan int)
wg := sync.WaitGroup{}
wg.Add(1)
go myFunc(ch, &wg) //<-- This go routine started in a new thread
time.Sleep(2 * time.Second) //<--- introduced a Sleep of 2 seconds to ensure that myFunc() go routine executes before main thread
ch <- 10
wg.Wait()
fmt.Println("End Main method")
}
The channels are used here for synchronization and it works as described in documentation. It does not mean that the code starting from this point in the code will be executed at the same speed. It only means that main goroutine will not continue if myFunc goroutine is not reading from channel. And myFunc will wait for main goroutine to push data to channel. After this happen both goroutines will continue it execution independently.
Try this, used your code as basis
package main
import (
"fmt"
"time"
)
func myFunc(ch chan int, done chan struct{}) {
defer close(done) // channel will be closed in the function exit phase
fmt.Println("Inside goroutine:: myFunc()")
fmt.Println(10 + <-ch) //<-- According to rule, control will be blocked here until 'ch' sends some data so that it will be received in our myFunc() go routine.
}
func main() {
fmt.Println("Start Main method")
// Creating a channel
ch := make(chan int)
done := make(chan struct{}) // signal channel
go myFunc(ch, done) //<-- This go routine started in a new thread
time.Sleep(2 * time.Second) //<--- introduced a Sleep of 2 seconds to ensure that myFunc() go routine executes before main thread
ch <- 10
<-done // waiting for function complete
fmt.Println("End Main method")
}
Or use Jaroslaw's suggestion.
Because go is so fast... https://play.golang.org/p/LNyDAA3mGYY
After you send to channel scheduler isn't fast enoght... and program exists. I have introduced an additional context switcher for scheduler to show effect.
Yes, you are right
I think, it is because main thread finished its execution first and hence, all other goroutine also terminated.
If you check the above program execution. The sleep is before main thread writes to the channel. Now even though which goroutine() will have CPU time is completely arbitary, but in the above case if the main sleeps before the explicit sleep logic. myFunc will be blocked as there is no data in ch
Here I made a slight change to the above code to make main sleep after writing data into Channel. It gives the expected output, Without using waitgroup or quit channels.
package main
import (
"fmt"
"time"
)
func myFunc(ch chan int) {
fmt.Println("Inside goroutine:: myFunc()")
fmt.Println(10 + <-ch) //<-- According to rule, control will be blocked here until 'ch' sends some data so that it will be received in our myFunc() go routine.
}
func main() {
fmt.Println("Start Main method")
// Creating a channel
ch := make(chan int)
go myFunc(ch) //<-- This go routine started in a new thread
ch <- 10
time.Sleep(2 * time.Second) //<--- introduced a Sleep of 2 seconds to ensure that myFunc() go routine executes before main thread
fmt.Println("End Main method")
}
It is currently a race condition between the myFunc being able to print and your main function exiting.
If we look at the spec for program execution at
https://golang.org/ref/spec#Program_execution
Program execution begins by initializing the main package and then invoking the function main. When that function invocation returns, the program exits. It does not wait for other (non-main) goroutines to complete.
It is still your job to make sure that all spawned goroutines will complete before your main goroutine exists.
In your case, you could use a waitgroup as you mentioned or you could use a done channel.
https://play.golang.org/p/RVr0HXuUMgn
Given your code you could also close the channel you use to send the integer over since you are passing it to the function as bidirectional but it's not strictly idiomatic.
https://play.golang.org/p/wGvexC5ZgIi
The following code logged an error:
fatal error: all goroutines are asleep - deadlock!
package main
import "fmt"
func main() {
ch := make(chan int)
ch <- 1
fmt.Println(<-ch)
}
But when I changed the code into this:
package main
import "fmt"
func assign (ch chan int) {
ch <- 1
}
func main() {
ch := make(chan int)
go assign (ch)
fmt.Println(<-ch)
}
"1" was printed out.
Then I used buffered channels:
package main
import "fmt"
func main() {
ch := make(chan int, 2)
ch <- 1
ch <- 2
fmt.Println(<-ch)
fmt.Println(<-ch)
}
"1" and "2" can also be printed out.
I'm a little confused about the situation. Thanks in advance!
Why the deadlock happened:
In the first code snippet you have only one main goroutine and it is blocked when you are trying to write into the channel here:
ch <- 1
Because nobody reads from the channel and the main goroutine is waiting for this to continue.
See Effective Go -> Channels
If the channel is unbuffered, the sender blocks until the receiver has received the value.
The sender is main function, the receiver is also main function.
How to avoid the deadlock:
In order to solve this, you have two options:
Option 1: make the ch channel buffered like this:
ch := make(chan int, 1) // buffer length is set to 1
From A Tour of Go
Sends to a buffered channel block only when the buffer is full.
So, you can write to the channel until the buffer is full. Then somebody has to start reading from the channel.
Option 2: write to the channel from a goroutine, like you did in the second code snippet:
func assign(ch chan int) {
ch <- 1
}
func main() {
ch := make(chan int)
go assign(ch) // does not block the main goroutine
fmt.Println(<-ch) // waiting to read from the channel
}
In this case main function will be executed until fmt.Println(<-ch) and continues as soon as it can read from the channel.
When you are using unbuffered channel, goroutine is blocked during write until someone does the read.
In your first snippet, there is an unbuffered channel and single goroutine (main goroutine).
So when you are trying to write:
ch <- 1
Nobody reads from the channel yet. The main goroutine is blocked and this line is never executed:
fmt.Println(<-ch)
That's why you've got the deadlock error.
In the second example, you still using unbuffered channel, which means write operation blocks the goroutine.
But by using go you are running the second goroutine.
It means even if this new goroutine will be blocked during write (in your assign function), the main goroutine will continue to work and fmt.Println(<-ch) will be executed and do the read (which in turn unblock background goroutine and assign function will finally reach the end).
To get more understanding about channels and goroutines, this snippet will give the same result (as your second snippet):
package main
import "fmt"
func print(ch chan int) {
fmt.Println(<-ch)
}
func main() {
ch := make(chan int)
go print(ch)
ch <- 1
}
When you are working with a buffered channel (third snippet), you can do N write operations without blocking goroutine (where N is the size of the buffer).
That's why in your example you did 2 writes without blocking and is able to read them later. But if your buffer is less than the count of write operations, and nobody do the read, you will fall into the same blocking issues (see the explanation of 1&2 snippets).
I'm new to Go and am stumped on what appears to be a somewhat-rare race condition with a very small block of code running on Linux with Go version 1.2.
Basically, I create a channel for an int, start a go routine to read from the channel, and then write a single int to the channel.
package main
import "fmt"
func main() {
channel := make(chan int)
go func() {
number := <- channel
fmt.Printf("GOT IT: %d\n", number)
}()
fmt.Println("[+] putting num on channel")
channel <- 42
fmt.Println("[-] putting num on channel")
}
The output about 90% of the time is as expected:
$ go run test.go
[+] putting num on channel
GOT IT: 42
[-] putting num on channel
However, about 10% of the time, the go routine simply does not read the number from the channel and prints nothing:
$ go run test.go
[+] putting num on channel
[-] putting num on channel
I'm puzzled because this code is very similar to the example at https://gobyexample.com/channels, (which I do not have this problem with) except that I'm reading from the channel in my go routine instead of writing to the channel.
Do I have a fundamental misunderstanding of how channels work or is there something else at play here?
You should wait until your goroutine executes, and then your, for example, you can do it with sync.WaitGroup:
package main
import (
"fmt"
"sync"
)
func main() {
var wg sync.WaitGroup
channel := make(chan int)
wg.Add(1)
go func() {
number := <-channel
fmt.Printf("GOT IT: %d\n", number)
wg.Done()
}()
fmt.Println("[+] putting num on channel")
channel <- 42
wg.Wait()
fmt.Println("[-] putting num on channel")
}
(goplay: http://play.golang.org/p/VycxTw_4vu)
Also you can do it with a "notification channel", that indicates that job is done:
package main
import "fmt"
func main() {
channel := make(chan int)
done := make(chan bool)
go func() {
number := <-channel
fmt.Printf("GOT IT: %d\n", number)
done <- true
}()
fmt.Println("[+] putting num on channel")
channel <- 42
<-done
fmt.Println("[-] putting num on channel")
}
(goplay: http://play.golang.org/p/fApWQgtr4D)
You seem to be expecting the receiving goroutine to run to completion before the second fmt.Println executes. This is not guaranteed to be the case. If the program terminates, goroutines are not guaranteed to reach the end of their functions.
When you see the output that doesn't display the "GOT IT" message, the channel delivered its message, but the main function completed before the goroutine did. The program terminated, and the goroutine never gets the chance to call fmt.Printf
In the example you cited, the main function ends with this:
go func() { messages <- "ping" }()
msg := <-messages
fmt.Println(msg)
Since the main function blocks until it receives a message, the goroutine always runs to completion in this example. In your code, your goroutine executes a step after it receives from the channel, and it's undefined whether the goroutine or the main function will execute the next line after the receive.
You have two goroutines, one in main() (which is implicitly a goroutine), and the anonymous one.
They communicate over a synchronous channel, so after the channel communication, it's guaranteed that they're synchronised.
At this point, the code left in the main() goroutine looks like this:
fmt.Println("[-] putting num on a channel")
and the code left in the anonymous goroutine looks like this:
fmt.Println("GOT IT: %d\n", number)
Now you're racing: the output from these Printlns may appear in either order, or even intermingled. When the Println() from main finishes, the next thing that will happen on that goroutine is that your program will be stopped. This may prevent some or all of the Println from the anonymous goroutine from appearing.