Given this playground:
package main
import "fmt"
func main() {
go oneFunc().anotherFunc()
}
func oneFunc() something {
fmt.Println("oneFunc")
return something{}
}
type something struct{}
func (s something) anotherFunc() {
fmt.Println("anotherFunc")
}
Why is the output:
oneFunc
and "anotherFunc" is never printed?
anotherFunc doesn't execute until oneFunc returns a value. Because of this, the program exits before anotherFunc is able to run. You need to wait for anotherFunc to run before main exits.
You can do this via Go channels. For Example:
http://play.golang.org/p/dWoLB9afSj
package main
import "fmt"
func main() {
ch := make(chan int)
go oneFunc(ch).anotherFunc()
fmt.Println("Waiting...")
<-ch
fmt.Println("Done.")
}
func oneFunc(ch chan int) something {
fmt.Println("oneFunc")
return something{ch}
}
type something struct{
ch chan int
}
func (s something) anotherFunc() {
fmt.Println("anotherFunc")
s.ch <- 1
}
The way that I like to about it is this, go – like defer – consumes the last invocation, or pair parentheses, on the line and will invoke that function out of order. Every invocation before that is synchronous.
Where go makes the invocation concurrent. And defer delays invocation until current function returns.
There is a really good example of this in the defer section of Effective Go
The expression behind the go keyword is evaluated and the function value of that expression is then executed concurrently.
So, in your example oneFunc() is called, hence the oneFunc output, and the method anotherFunc on the returned instance is called concurrently.
However, your program terminates before the goroutine can run which
is why you don't see anotherFunc printed.
Solution: Use sync.WaitGroup or channels for synchronization.
To actually (empirically) verify that your go call executes anotherFunc concurrently
and not oneFunc you can print the stack in each function and compare the output.
Example (on play):
var wg = sync.WaitGroup{}
func main() {
wg.Add(1)
go oneFunc().anotherFunc()
wg.Wait()
}
func oneFunc() something {
fmt.Println("oneFunc")
buf := make([]byte, 4096)
runtime.Stack(buf, false)
fmt.Println("Stack of oneFunc:", string(buf))
return something{}
}
type something struct{}
func (s something) anotherFunc() {
defer wg.Done()
buf := make([]byte, 4096)
runtime.Stack(buf, false)
fmt.Println("Stack of anotherFunc:", string(buf))
fmt.Println("anotherFunc")
}
You will see something like this:
oneFunc
Stack of oneFunc: goroutine 1 [running]:
main.oneFunc()
/tmpfs/gosandbox-342f581d_b6e8aa8b_334a0f88_c8221b7e_20882985/prog.go:20 +0x118
main.main()
/tmpfs/gosandbox-342f581d_b6e8aa8b_334a0f88_c8221b7e_20882985/prog.go:11 +0x50
Stack of anotherFunc: goroutine 2 [running]:
main.something.anotherFunc()
/tmpfs/gosandbox-342f581d_b6e8aa8b_334a0f88_c8221b7e_20882985/prog.go:32 +0xb2
created by main.main
/tmpfs/gosandbox-342f581d_b6e8aa8b_334a0f88_c8221b7e_20882985/prog.go:11 +0x69
anotherFunc
The stack trace even tells you that the two functions are running in different
goroutines, no comparison of method calls needed.
Related
As explained on gobyexample https://gobyexample.com/channels, it says
When we run the program the "ping" message is successfully passed from one goroutine to another via our channel. but the way I see, its like we have only one go routine in the code, ie, go func (text string) { messages <- "ping"}() so I don't know from which go routine the "ping" message comes from and in which go routine it is put
package main
import "fmt"
func main() {
messages := make(chan string)
go func (text string) { messages <- "ping"}()
msg := <-messages
fmt.Println(msg)
}
Go programs start with a single goroutine. Each go statement adds another. So, in your program, there are two: the one that started when main() started, and the one that the first started when it reached the line go func ...
func main() {
messages := make(chan string) // goroutine 1
go func (text string) {
messages <- "ping" // goroutine 2
}() // goroutine 1 starting goroutine 2
msg := <-messages // goroutine 1
fmt.Println(msg) // goroutine 1
}
Say I have a function
type Foo struct {}
func (a *Foo) Bar() {
// some expensive work - does some calls to redis
}
which gets executed within a goroutine at some point in my app. Lots of these may be executing at any given point. Prior to application termination, I would like to ensure all remaining goroutines have finished their work.
Can I do something like this:
type Foo struct {
wg sync.WaitGroup
}
func (a *Foo) Close() {
a.wg.Wait()
}
func (a *Foo) Bar() {
a.wg.Add(1)
defer a.wg.Done()
// some expensive work - does some calls to redis
}
Assuming here that Bar gets executed within a goroutine and many of these may be running at a given time and that Bar should not be called once Close is called and Close is called upon a sigterm or sigint.
Does this make sense?
Usually I would see the Bar function look like this:
func (a *Foo) Bar() {
a.wg.Add(1)
go func() {
defer a.wg.Done()
// some expensive work - does some calls to redis
}()
}
Yes, WaitGroup is the right answer. You can use WaitGroup.Add at anytime that the counter is greater than zero, as per doc.
Note that calls with a positive delta that occur when the counter is zero must happen before a Wait. Calls with a negative delta, or calls with a positive delta that start when the counter is greater than zero, may happen at any time. Typically this means the calls to Add should execute before the statement creating the goroutine or other event to be waited for. If a WaitGroup is reused to wait for several independent sets of events, new Add calls must happen after all previous Wait calls have returned. See the WaitGroup example.
But one trick is that, you should always keep the counter greater than zero, before Close is called. That usually means you should call wg.Add in NewFoo (or something like that) and wg.Done in Close. And to prevent multiple calls to Done ruining the wait group, you should wrap Close into sync.Once. You may also want to prevent new Bar() from being called.
WaitGroup is one way, however, the Go team introduced the errgroup for your use case exactly. The most inconvenient part of leaf bebop's answer, is the disregard for error handling. Error handling is the reason errgroup exists. And idiomatic go code should never swallow errors.
However, keeping the signatures of your Foo struct, (except a cosmetic workerNumber)—and no error handling—my proposal looks like this:
package main
import (
"fmt"
"math/rand"
"time"
"golang.org/x/sync/errgroup"
)
type Foo struct {
errg errgroup.Group
}
func NewFoo() *Foo {
foo := &Foo{
errg: errgroup.Group{},
}
return foo
}
func (a *Foo) Bar(workerNumber int) {
a.errg.Go(func() error {
select {
// simulates the long running clals
case <-time.After(time.Second * time.Duration(rand.Intn(10))):
fmt.Println(fmt.Sprintf("worker %d completed its work", workerNumber))
return nil
}
})
}
func (a *Foo) Close() {
a.errg.Wait()
}
func main() {
foo := NewFoo()
for i := 0; i < 10; i++ {
foo.Bar(i)
}
<-time.After(time.Second * 5)
fmt.Println("Waiting for workers to complete...")
foo.Close()
fmt.Println("Done.")
}
The benefit here, is that if you introduce error handling in your code (you should), you only need to slightly modify this code: In short, errg.Wait() would return the first redis error, and Close() could propagate this up through the stack (to main, in this case).
Utilizing the context.Context package as well, you would also be able to immediately cancel any running redis call, if one fails. There are examples of this in the errgroup documentation.
I think waiting indefinitely for all the go routines to finish is not the right way.
If one of the go routines get blocked or say it hangs due to some reason and never terminates successfully, what should happen kill the process or wait for go routines to finish ?
Instead you should wait with some timeout and kill the app irrespective of whether all the routines have finished or not.
Edit: Original ans
Thanks #leaf bebop for pointing it out. I misunderstood the question.
Context package can be used to signal all the go routines to handle kill signal.
appCtx, cancel := context.WithCancel(context.Background())
Here appCtx will have to be passed to all the go routines.
On exit signal call cancel().
functions running as go routines can handle how to handle cancel context.
Using context cancellation in Go
A pattern i use a lot is: https://play.golang.org/p/ibMz36TS62z
package main
import (
"fmt"
"sync"
"time"
)
type response struct {
message string
}
func task(i int, done chan response) {
time.Sleep(1 * time.Second)
done <- response{fmt.Sprintf("%d done", i)}
}
func main() {
responses := GetResponses(10)
fmt.Println("all done", len(responses))
}
func GetResponses(n int) []response {
donequeue := make(chan response)
wg := sync.WaitGroup{}
for i := 0; i < n; i++ {
wg.Add(1)
go func(value int) {
defer wg.Done()
task(value, donequeue)
}(i)
}
go func() {
wg.Wait()
close(donequeue)
}()
responses := []response{}
for result := range donequeue {
responses = append(responses, result)
}
return responses
}
this makes it easy to throttle as well: https://play.golang.org/p/a4MKwJKj634
package main
import (
"fmt"
"sync"
"time"
)
type response struct {
message string
}
func task(i int, done chan response) {
time.Sleep(1 * time.Second)
done <- response{fmt.Sprintf("%d done", i)}
}
func main() {
responses := GetResponses(10, 2)
fmt.Println("all done", len(responses))
}
func GetResponses(n, concurrent int) []response {
throttle := make(chan int, concurrent)
for i := 0; i < concurrent; i++ {
throttle <- i
}
donequeue := make(chan response)
wg := sync.WaitGroup{}
for i := 0; i < n; i++ {
wg.Add(1)
<-throttle
go func(value int) {
defer wg.Done()
throttle <- 1
task(value, donequeue)
}(i)
}
go func() {
wg.Wait()
close(donequeue)
}()
responses := []response{}
for result := range donequeue {
responses = append(responses, result)
}
return responses
}
i'm new to Go not sure why this is deadlock ? i want to constantly be reading results from doSomething and storing it in the function read without using a for loop
func doSomething(c chan<- string){ // recursive function does something
c <- result
return dosomething(c) }
func reads(c <-chan string){
results := ""
temp := <-c
results = results + "\n" + temp
return results
}
func main(){
go reads(c)
doSomething(c)
}
Main gorouting is trying to write multiple times to a channel in doSomething function. The read function is reading the channel only once. Therefore the write operation will wait until some other party reads from the channel. This will deadlock as the main goroutine is blocked.
If the blocking operations were not in main goroutine, the program would finish as the Go program ends whenever main goroutine ends. There would be no deadlock if main function could come to an end.
You are trying to read from an empty channel because reads executed concurrently and doSomething didn't. It is possible to solve the problem in several ways. Note, it is not about correct architecture or efficient approaches. An examples below solve "deadlock" issue of the original snippet, not more.
Read and write concurrently:
package main
func doSomething(c chan<- string) { // recursive function does something
c <- "result"
doSomething(c)
}
func reads(c <-chan string) {
results := <-c
fmt.Println("Boo", results)
}
func main() {
c := make(chan string)
go reads(c)
go doSomething(c) // Write concurrentely
}
Use select to handle channels read operation:
func reads(c <-chan string) {
// Use select
select {
case res := <-c:
fmt.Println("received message", res)
default:
fmt.Println("no results received")
}
}
I'd rather prefer combination of the first and second approaches.
Read after write (it is far from correct design as hell):
func main() {
c := make(chan string)
go doSomething(c)
reads(c) // Read after write
}
in the next example, I don't understand why end value not printed when received
package main
import "fmt"
func main() {
start := make(chan int)
end := make(chan int)
go func() {
fmt.Println("Start")
fmt.Println(<-start)
}()
go func() {
fmt.Println("End")
fmt.Println(<-end)
}()
start <- 1
end <- 2
}
I know sync.WaitGroup can solve this problem.
Because the program exits when it reaches the end of func main, regardless of whether any other goroutines are running. As soon as the second function receives from the end channel, main's send on that channel is unblocked and the program finishes, before the received value gets a chance to be passed to Println.
The end value is not printed because as soon as the main goroutine (the main function is actually a goroutine) is finished (in other terms get unblocked) the other non-main goroutines does not have the chance to get completed.
When the function main() returns, the program exits. Moreover goroutines are independent units of execution and when a number of them starts one after the other you cannot depend on when a goroutine will actually be started. The logic of your code must be independent of the order in which goroutines are invoked.
One way to solve your problem (and the easiest one in your case) is to put a time.Sleep at the end of your main() function.
time.Sleep(1e9)
This will guarantee that the main goroutine will not unblock and the other goroutines will have a change to get executed.
package main
import (
"fmt"
"time"
)
func main() {
start := make(chan int)
end := make(chan int)
go func() {
fmt.Println("Start")
fmt.Println(<-start)
}()
go func() {
fmt.Println("End")
fmt.Println(<-end)
}()
start <- 1
end <- 2
time.Sleep(1e9)
}
Another solution as you mentioned is to use waitgroup.
Apart from sleep where you have to specify the time, you can use waitgroup to make you program wait until the goroutine completes execution.
package main
import "fmt"
import "sync"
var wg sync.WaitGroup
func main() {
start := make(chan int)
end := make(chan int)
wg.Add(2)
go func() {
defer wg.Done()
fmt.Println("Start")
fmt.Println(<-start)
}()
go func() {
defer wg.Done()
fmt.Println("End")
fmt.Println(<-end)
}()
start <- 1
end <- 2
wg.Wait()
}
I started to learn go language days ago. When I tried to start writing some fun codes, I am stuck by a strange behavior.
package main
import "fmt"
func recv(value int) {
if value < 0 {
return
}
fmt.Println(value)
go recv(value-1)
}
func main() {
recv(10)
}
when I run the above code, only 10 is printed. When I remove the go before the call to recv, 10 to 0 are printed out. I believe I am misusing go routine here, but I can not understand why it failed start a go routine this way.
When the main function returns, Go will not wait for any still existing goroutines to finish but instead just exit.
recv will return to main after the first "iteration" and because main has nothing more to do, the program will terminate.
One solution to this problem is to have a channel that signals that all work is done, like the following:
package main
import "fmt"
func recv(value int, ch chan bool) {
if value < 0 {
ch <- true
return
}
fmt.Println(value)
go recv(value - 1, ch)
}
func main() {
ch := make(chan bool)
recv(10, ch)
<-ch
}
Here, recv will send a single boolean before returning, and main will wait for that message on the channel.
For the logic of the program, it does not matter what type or specific value you use. bool and true are just a straightforward example. If you want to be more efficient, using a chan struct{} instead of a chan bool will save you an additional byte, since empty structs do not use any memory.
A sync.Waitgroup is another solution and specifically intended for the purpose of waiting for an arbitrary amount of goroutines to run their course.
package main
import (
"fmt"
"sync"
)
func recv(value int, wg *sync.WaitGroup) {
if value < 0 {
return
}
fmt.Println(value)
wg.Add(1) // Add 1 goroutine to the waitgroup.
go func() {
recv(value-1, wg)
wg.Done() // This goroutine is finished.
}()
}
func main() {
var wg sync.WaitGroup
recv(10, &wg)
// Block until the waitgroup signals
// all goroutines to be finished.
wg.Wait()
}
I did so and also worked. How come?
package main
import "fmt"
func recv(value int) {
if value < 0 {
return
}
fmt.Println(value)
recv(value - 1)
}
func main() {
recv(10)
}