While SayHello() executes as expected, the goroutine prints nothing.
package main
import "fmt"
func SayHello() {
for i := 0; i < 10 ; i++ {
fmt.Print(i, " ")
}
}
func main() {
SayHello()
go SayHello()
}
When your main() function ends, your program ends as well. It does not wait for other goroutines to finish.
Quoting from the Go Language Specification: Program Execution:
Program execution begins by initializing the main package and then invoking the function main. When that function invocation returns, the program exits. It does not wait for other (non-main) goroutines to complete.
See this answer for more details.
You have to tell your main() function to wait for the SayHello() function started as a goroutine to complete. You can synchronize them with channels for example:
func SayHello(done chan int) {
for i := 0; i < 10; i++ {
fmt.Print(i, " ")
}
if done != nil {
done <- 0 // Signal that we're done
}
}
func main() {
SayHello(nil) // Passing nil: we don't want notification here
done := make(chan int)
go SayHello(done)
<-done // Wait until done signal arrives
}
Another alternative is to signal the completion by closing the channel:
func SayHello(done chan struct{}) {
for i := 0; i < 10; i++ {
fmt.Print(i, " ")
}
if done != nil {
close(done) // Signal that we're done
}
}
func main() {
SayHello(nil) // Passing nil: we don't want notification here
done := make(chan struct{})
go SayHello(done)
<-done // A receive from a closed channel returns the zero value immediately
}
Notes:
According to your edits/comments: if you want the 2 running SayHello() functions to print "mixed" numbers randomly: you have no guarantee to observe such behaviour. Again, see the aforementioned answer for more details. The Go Memory Model only guarantees that certain events happen before other events, you have no guarantee how 2 concurrent goroutines are executed.
You might experiment with it, but know that the result will not be deterministic. First you have to enable multiple active goroutines to be executed with:
runtime.GOMAXPROCS(2)
And second you have to first start SayHello() as a goroutine because your current code first executes SayHello() in the main goroutine and only once it finished starts the other one:
runtime.GOMAXPROCS(2)
done := make(chan struct{})
go SayHello(done) // FIRST START goroutine
SayHello(nil) // And then call SayHello() in the main goroutine
<-done // Wait for completion
Alternatively (to icza's answer) you can use WaitGroup from sync package and anonymous function to avoid altering original SayHello.
package main
import (
"fmt"
"sync"
)
func SayHello() {
for i := 0; i < 10; i++ {
fmt.Print(i, " ")
}
}
func main() {
SayHello()
var wg sync.WaitGroup
wg.Add(1)
go func() {
defer wg.Done()
SayHello()
}()
wg.Wait()
}
In order to print numbers simultaneously run each print statement in separate routine like the following
package main
import (
"fmt"
"math/rand"
"sync"
"time"
)
func main() {
var wg sync.WaitGroup
for i := 0; i < 10; i++ {
wg.Add(1)
go func(fnScopeI int) {
defer wg.Done()
// next two strings are here just to show routines work simultaneously
amt := time.Duration(rand.Intn(250))
time.Sleep(time.Millisecond * amt)
fmt.Print(fnScopeI, " ")
}(i)
}
wg.Wait()
}
A Go program exits when the main function returns.
One option is to use something like sync.WaitGroup to wait on the other goroutines that main has spawned before returning from main.
Another option is to call runtime.Goexit() in main. From the godoc:
Goexit terminates the goroutine that calls it. No other goroutine is affected. Goexit runs all deferred calls before terminating the goroutine. Because Goexit is not a panic, any recover calls in those deferred functions will return nil.
Calling Goexit from the main goroutine terminates that goroutine without func main returning. Since func main has not returned, the program continues execution of other goroutines. If all other goroutines exit, the program crashes.
This allows main goroutine to stop executing while the background routines continue to execute. For example:
package main
import (
"fmt"
"runtime"
"time"
)
func f() {
for i := 0; ; i++ {
fmt.Println(i)
time.Sleep(10 * time.Millisecond)
}
}
func main() {
go f()
runtime.Goexit()
}
This can be cleaner than blocking forever in the main function, especially for programs that are infinite. One downside is that if all of the goroutines of a process return or exit (including the main goroutine), Go will detect this as an error and panic:
fatal error: no goroutines (main called runtime.Goexit) - deadlock!
To avoid this, at least one goroutine must call os.Exit before it returns. Calling os.Exit(0) immediately terminates the program and indicates that it did so without error. For example:
package main
import (
"fmt"
"os"
"runtime"
"time"
)
func f() {
for i := 0; i < 10; i++ {
fmt.Println(i)
time.Sleep(10 * time.Millisecond)
}
os.Exit(0)
}
func main() {
go f()
runtime.Goexit()
}
Related
Say I have a function
type Foo struct {}
func (a *Foo) Bar() {
// some expensive work - does some calls to redis
}
which gets executed within a goroutine at some point in my app. Lots of these may be executing at any given point. Prior to application termination, I would like to ensure all remaining goroutines have finished their work.
Can I do something like this:
type Foo struct {
wg sync.WaitGroup
}
func (a *Foo) Close() {
a.wg.Wait()
}
func (a *Foo) Bar() {
a.wg.Add(1)
defer a.wg.Done()
// some expensive work - does some calls to redis
}
Assuming here that Bar gets executed within a goroutine and many of these may be running at a given time and that Bar should not be called once Close is called and Close is called upon a sigterm or sigint.
Does this make sense?
Usually I would see the Bar function look like this:
func (a *Foo) Bar() {
a.wg.Add(1)
go func() {
defer a.wg.Done()
// some expensive work - does some calls to redis
}()
}
Yes, WaitGroup is the right answer. You can use WaitGroup.Add at anytime that the counter is greater than zero, as per doc.
Note that calls with a positive delta that occur when the counter is zero must happen before a Wait. Calls with a negative delta, or calls with a positive delta that start when the counter is greater than zero, may happen at any time. Typically this means the calls to Add should execute before the statement creating the goroutine or other event to be waited for. If a WaitGroup is reused to wait for several independent sets of events, new Add calls must happen after all previous Wait calls have returned. See the WaitGroup example.
But one trick is that, you should always keep the counter greater than zero, before Close is called. That usually means you should call wg.Add in NewFoo (or something like that) and wg.Done in Close. And to prevent multiple calls to Done ruining the wait group, you should wrap Close into sync.Once. You may also want to prevent new Bar() from being called.
WaitGroup is one way, however, the Go team introduced the errgroup for your use case exactly. The most inconvenient part of leaf bebop's answer, is the disregard for error handling. Error handling is the reason errgroup exists. And idiomatic go code should never swallow errors.
However, keeping the signatures of your Foo struct, (except a cosmetic workerNumber)—and no error handling—my proposal looks like this:
package main
import (
"fmt"
"math/rand"
"time"
"golang.org/x/sync/errgroup"
)
type Foo struct {
errg errgroup.Group
}
func NewFoo() *Foo {
foo := &Foo{
errg: errgroup.Group{},
}
return foo
}
func (a *Foo) Bar(workerNumber int) {
a.errg.Go(func() error {
select {
// simulates the long running clals
case <-time.After(time.Second * time.Duration(rand.Intn(10))):
fmt.Println(fmt.Sprintf("worker %d completed its work", workerNumber))
return nil
}
})
}
func (a *Foo) Close() {
a.errg.Wait()
}
func main() {
foo := NewFoo()
for i := 0; i < 10; i++ {
foo.Bar(i)
}
<-time.After(time.Second * 5)
fmt.Println("Waiting for workers to complete...")
foo.Close()
fmt.Println("Done.")
}
The benefit here, is that if you introduce error handling in your code (you should), you only need to slightly modify this code: In short, errg.Wait() would return the first redis error, and Close() could propagate this up through the stack (to main, in this case).
Utilizing the context.Context package as well, you would also be able to immediately cancel any running redis call, if one fails. There are examples of this in the errgroup documentation.
I think waiting indefinitely for all the go routines to finish is not the right way.
If one of the go routines get blocked or say it hangs due to some reason and never terminates successfully, what should happen kill the process or wait for go routines to finish ?
Instead you should wait with some timeout and kill the app irrespective of whether all the routines have finished or not.
Edit: Original ans
Thanks #leaf bebop for pointing it out. I misunderstood the question.
Context package can be used to signal all the go routines to handle kill signal.
appCtx, cancel := context.WithCancel(context.Background())
Here appCtx will have to be passed to all the go routines.
On exit signal call cancel().
functions running as go routines can handle how to handle cancel context.
Using context cancellation in Go
A pattern i use a lot is: https://play.golang.org/p/ibMz36TS62z
package main
import (
"fmt"
"sync"
"time"
)
type response struct {
message string
}
func task(i int, done chan response) {
time.Sleep(1 * time.Second)
done <- response{fmt.Sprintf("%d done", i)}
}
func main() {
responses := GetResponses(10)
fmt.Println("all done", len(responses))
}
func GetResponses(n int) []response {
donequeue := make(chan response)
wg := sync.WaitGroup{}
for i := 0; i < n; i++ {
wg.Add(1)
go func(value int) {
defer wg.Done()
task(value, donequeue)
}(i)
}
go func() {
wg.Wait()
close(donequeue)
}()
responses := []response{}
for result := range donequeue {
responses = append(responses, result)
}
return responses
}
this makes it easy to throttle as well: https://play.golang.org/p/a4MKwJKj634
package main
import (
"fmt"
"sync"
"time"
)
type response struct {
message string
}
func task(i int, done chan response) {
time.Sleep(1 * time.Second)
done <- response{fmt.Sprintf("%d done", i)}
}
func main() {
responses := GetResponses(10, 2)
fmt.Println("all done", len(responses))
}
func GetResponses(n, concurrent int) []response {
throttle := make(chan int, concurrent)
for i := 0; i < concurrent; i++ {
throttle <- i
}
donequeue := make(chan response)
wg := sync.WaitGroup{}
for i := 0; i < n; i++ {
wg.Add(1)
<-throttle
go func(value int) {
defer wg.Done()
throttle <- 1
task(value, donequeue)
}(i)
}
go func() {
wg.Wait()
close(donequeue)
}()
responses := []response{}
for result := range donequeue {
responses = append(responses, result)
}
return responses
}
in the next example, I don't understand why end value not printed when received
package main
import "fmt"
func main() {
start := make(chan int)
end := make(chan int)
go func() {
fmt.Println("Start")
fmt.Println(<-start)
}()
go func() {
fmt.Println("End")
fmt.Println(<-end)
}()
start <- 1
end <- 2
}
I know sync.WaitGroup can solve this problem.
Because the program exits when it reaches the end of func main, regardless of whether any other goroutines are running. As soon as the second function receives from the end channel, main's send on that channel is unblocked and the program finishes, before the received value gets a chance to be passed to Println.
The end value is not printed because as soon as the main goroutine (the main function is actually a goroutine) is finished (in other terms get unblocked) the other non-main goroutines does not have the chance to get completed.
When the function main() returns, the program exits. Moreover goroutines are independent units of execution and when a number of them starts one after the other you cannot depend on when a goroutine will actually be started. The logic of your code must be independent of the order in which goroutines are invoked.
One way to solve your problem (and the easiest one in your case) is to put a time.Sleep at the end of your main() function.
time.Sleep(1e9)
This will guarantee that the main goroutine will not unblock and the other goroutines will have a change to get executed.
package main
import (
"fmt"
"time"
)
func main() {
start := make(chan int)
end := make(chan int)
go func() {
fmt.Println("Start")
fmt.Println(<-start)
}()
go func() {
fmt.Println("End")
fmt.Println(<-end)
}()
start <- 1
end <- 2
time.Sleep(1e9)
}
Another solution as you mentioned is to use waitgroup.
Apart from sleep where you have to specify the time, you can use waitgroup to make you program wait until the goroutine completes execution.
package main
import "fmt"
import "sync"
var wg sync.WaitGroup
func main() {
start := make(chan int)
end := make(chan int)
wg.Add(2)
go func() {
defer wg.Done()
fmt.Println("Start")
fmt.Println(<-start)
}()
go func() {
defer wg.Done()
fmt.Println("End")
fmt.Println(<-end)
}()
start <- 1
end <- 2
wg.Wait()
}
This example taken from tour.golang.org/#63
package main
import (
"fmt"
"time"
)
func say(s string) {
for i := 0; i < 5; i++ {
time.Sleep(100 * time.Millisecond)
fmt.Println(s)
}
}
func main() {
go say("world")
say("hello")
}
The output
hello
world
hello
world
hello
world
hello
world
hello
Why world is printed only 4 times instead of 5 ?
Edit: The answer can be quoted from golang specification:
Program execution begins by initializing the main package and then
invoking the function main. When the function main returns, the
program exits. It does not wait for other (non-main) goroutines to
complete.
When your main function ends your program ends, i.e. all goroutines are terminated.
Your main terminates before go say("world") is done. If you sleep some time at the end of main you should see the last world.
Here is how you solve that synchronization problem properly - with sync.WaitGroup
Playground link
package main
import (
"fmt"
"sync"
"time"
)
func say(s string, wg *sync.WaitGroup) {
defer wg.Done()
for i := 0; i < 5; i++ {
time.Sleep(100 * time.Millisecond)
fmt.Println(s)
}
}
func main() {
wg := new(sync.WaitGroup)
wg.Add(2)
go say("world", wg)
go say("hello", wg)
wg.Wait()
fmt.Println("All done")
}
Because the calling gorouting terminates before the second one you spawned does. This causes the second to shut down. To illustrate, modify your code slightly:
package main
import (
"fmt"
"time"
)
func say(s string) {
for i := 0; i < 5; i++ {
time.Sleep(100 * time.Millisecond)
fmt.Print(i)
fmt.Println(":"+s)
}
}
func main() {
go say("world")
say("hello")
}
Try putting in a "wait" or a sleep to the end of the main function.
I started to learn go language days ago. When I tried to start writing some fun codes, I am stuck by a strange behavior.
package main
import "fmt"
func recv(value int) {
if value < 0 {
return
}
fmt.Println(value)
go recv(value-1)
}
func main() {
recv(10)
}
when I run the above code, only 10 is printed. When I remove the go before the call to recv, 10 to 0 are printed out. I believe I am misusing go routine here, but I can not understand why it failed start a go routine this way.
When the main function returns, Go will not wait for any still existing goroutines to finish but instead just exit.
recv will return to main after the first "iteration" and because main has nothing more to do, the program will terminate.
One solution to this problem is to have a channel that signals that all work is done, like the following:
package main
import "fmt"
func recv(value int, ch chan bool) {
if value < 0 {
ch <- true
return
}
fmt.Println(value)
go recv(value - 1, ch)
}
func main() {
ch := make(chan bool)
recv(10, ch)
<-ch
}
Here, recv will send a single boolean before returning, and main will wait for that message on the channel.
For the logic of the program, it does not matter what type or specific value you use. bool and true are just a straightforward example. If you want to be more efficient, using a chan struct{} instead of a chan bool will save you an additional byte, since empty structs do not use any memory.
A sync.Waitgroup is another solution and specifically intended for the purpose of waiting for an arbitrary amount of goroutines to run their course.
package main
import (
"fmt"
"sync"
)
func recv(value int, wg *sync.WaitGroup) {
if value < 0 {
return
}
fmt.Println(value)
wg.Add(1) // Add 1 goroutine to the waitgroup.
go func() {
recv(value-1, wg)
wg.Done() // This goroutine is finished.
}()
}
func main() {
var wg sync.WaitGroup
recv(10, &wg)
// Block until the waitgroup signals
// all goroutines to be finished.
wg.Wait()
}
I did so and also worked. How come?
package main
import "fmt"
func recv(value int) {
if value < 0 {
return
}
fmt.Println(value)
recv(value - 1)
}
func main() {
recv(10)
}
I've a long running server written in Go. Main fires off several goroutines where the logic of the program executes. After that main does nothing useful. Once main exits, the program will quit. The method I am using right now to keep the program running is just a simple call to fmt.Scanln(). I'd like to know how others keep main from exiting. Below is a basic example. What ideas or best practices could be used here?
I considered creating a channel and delaying exit of main by receiving on said channel, but I think that could be problematic if all my goroutines become inactive at some point.
Side note: In my server (not the example), the program isn't actually running connected to a shell, so it doesn't really make sense to interact with the console anyway. For now it works, but I'm looking for the "correct" way, assuming there is one.
package main
import (
"fmt"
"time"
)
func main() {
go forever()
//Keep this goroutine from exiting
//so that the program doesn't end.
//This is the focus of my question.
fmt.Scanln()
}
func forever() {
for ; ; {
//An example goroutine that might run
//indefinitely. In actual implementation
//it might block on a chanel receive instead
//of time.Sleep for example.
fmt.Printf("%v+\n", time.Now())
time.Sleep(time.Second)
}
}
Block forever. For example,
package main
import (
"fmt"
"time"
)
func main() {
go forever()
select {} // block forever
}
func forever() {
for {
fmt.Printf("%v+\n", time.Now())
time.Sleep(time.Second)
}
}
The current design of Go's runtime assumes that the programmer is responsible for detecting when to terminate a goroutine and when to terminate the program. The programmer needs to compute the termination condition for goroutines and also for the entire program. A program can be terminated in a normal way by calling os.Exit or by returning from the main() function.
Creating a channel and delaying exit of main() by immediately receiving on said channel is a valid approach of preventing main from exiting. But it does not solve the problem of detecting when to terminate the program.
If the number of goroutines cannot be computed before the main() function enters the wait-for-all-goroutines-to-terminate loop, you need to be sending deltas so that main function can keep track of how many goroutines are in flight:
// Receives the change in the number of goroutines
var goroutineDelta = make(chan int)
func main() {
go forever()
numGoroutines := 0
for diff := range goroutineDelta {
numGoroutines += diff
if numGoroutines == 0 { os.Exit(0) }
}
}
// Conceptual code
func forever() {
for {
if needToCreateANewGoroutine {
// Make sure to do this before "go f()", not within f()
goroutineDelta <- +1
go f()
}
}
}
func f() {
// When the termination condition for this goroutine is detected, do:
goroutineDelta <- -1
}
An alternative approach is to replace the channel with sync.WaitGroup. A drawback of this approach is that wg.Add(int) needs to be called before calling wg.Wait(), so it is necessary to create at least 1 goroutine in main() while subsequent goroutines can be created in any part of the program:
var wg sync.WaitGroup
func main() {
// Create at least 1 goroutine
wg.Add(1)
go f()
go forever()
wg.Wait()
}
// Conceptual code
func forever() {
for {
if needToCreateANewGoroutine {
wg.Add(1)
go f()
}
}
}
func f() {
// When the termination condition for this goroutine is detected, do:
wg.Done()
}
Go's runtime package has a function called runtime.Goexit that will do exactly what you want.
Calling Goexit from the main goroutine terminates that goroutine
without func main returning. Since func main has not returned,
the program continues execution of other goroutines.
If all other goroutines exit, the program crashes.
Example in the playground
package main
import (
"fmt"
"runtime"
"time"
)
func main() {
go func() {
time.Sleep(time.Second)
fmt.Println("Go 1")
}()
go func() {
time.Sleep(time.Second * 2)
fmt.Println("Go 2")
}()
runtime.Goexit()
fmt.Println("Exit")
}
Nobody mentioned signal.Notify(c chan<- os.Signal, sig ...os.Signal)
Example:
package main
import (
"fmt"
"time"
"os"
"os/signal"
"syscall"
)
func main() {
go forever()
quitChannel := make(chan os.Signal, 1)
signal.Notify(quitChannel, syscall.SIGINT, syscall.SIGTERM)
<-quitChannel
//time for cleanup before exit
fmt.Println("Adios!")
}
func forever() {
for {
fmt.Printf("%v+\n", time.Now())
time.Sleep(time.Second)
}
}
Here is a simple block forever using channels
package main
import (
"fmt"
"time"
)
func main() {
done := make(chan bool)
go forever()
<-done // Block forever
}
func forever() {
for {
fmt.Printf("%v+\n", time.Now())
time.Sleep(time.Second)
}
}
You could daemonize the process using Supervisor (http://supervisord.org/). Your function forever would just be a process that it runs, and it would handle the part of your function main. You would use the supervisor control interface to start/shutdown/check on your process.