In the below code:
package main
import (
"context"
"fmt"
"time"
)
func cancellation() {
duration := 150 * time.Millisecond
ctx, cancel := context.WithTimeout(context.Background(), duration)
defer cancel()
ch := make(chan string)
go func() {
time.Sleep(time.Duration(500) * time.Millisecond)
ch <- "paper"
}()
select {
case d := <-ch:
fmt.Println("work complete", d)
case <-ctx.Done():
fmt.Println("work cancelled")
}
time.Sleep(time.Second)
fmt.Println("--------------------------------------")
}
func main() {
cancellation()
}
Because of unbuffered channel(ch := make(chan string)), go-routine leaks due to block on send(ch <- "paper"), if main goroutine is not ready to receive.
Using buffered channel ch := make(chan string, 1) does not block send(ch <- "paper")
How to detect such go-routine leaks?
There are some packages that let you do that. Two that I've used in the past:
https://github.com/fortytw2/leaktest
https://github.com/uber-go/goleak
Generally, they use functionality from the runtime package to examine the stack before and after your code runs and report suspected leaks. It's recommended to use them in tests. I found this works well in practice and used it in a couple of projects.
Related
The problem is that both the goOne and goTwo functions are sending values to the channels ch1 and ch2 respectively, but there is no corresponding receiver for these values in the main function. This means that the channels are blocked and the program is unable to proceed. As a result, the select statement in the main function is unable to read from the channels, so it always executes the default case.
package main
import (
"fmt"
"sync"
)
func main() {
var wg sync.WaitGroup
ch1 := make(chan string)
ch2 := make(chan string)
wg.Add(2)
go goOne(&wg, ch1)
go goTwo(&wg, ch2)
select {
case <-ch1:
fmt.Println(<-ch1)
close(ch1)
case <-ch2:
fmt.Println(<-ch2)
close(ch2)
default:
fmt.Println("Default Case")
}
wg.Wait()
}
func goTwo(wg *sync.WaitGroup, ch2 chan string) {
ch2 <- "Channel 2"
wg.Done()
}
func goOne(wg *sync.WaitGroup, ch1 chan string) {
ch1 <- "Channel 1"
wg.Done()
}
Output:
Default Case
fatal error: all goroutines are asleep - deadlock!
goroutine 1 \[semacquire\]:
sync.runtime_Semacquire(0xc000108270?)
/usr/local/go/src/runtime/sema.go:62 +0x25
sync.(\*WaitGroup).Wait(0x4b9778?)
/usr/local/go/src/sync/waitgroup.go:139 +0x52
main.main()
/home/nidhey/Documents/Go_Learning/goroutines/select.go:29 +0x2af
goroutine 6 \[chan send\]:
main.goOne(0x0?, 0x0?)
/home/nidhey/Documents/Go_Learning/goroutines/select.go:39 +0x28
created by main.main
/home/nidhey/Documents/Go_Learning/goroutines/select.go:14 +0xc5
goroutine 7 \[chan send\]:
main.goTwo(0x0?, 0x0?)
/home/nidhey/Documents/Go_Learning/goroutines/select.go:33 +0x28
created by main.main
/home/nidhey/Documents/Go_Learning/goroutines/select.go:15 +0x119\```
I'm looking for a different pattern such as select to handle the case when the channels are blocked.
To fix the issue, I've added a <-ch1 or <-ch2 in the main function after wg.Wait() to receive the values sent to the channels and unblock them
It's not entirely clear what you want to do. If you want to wait for both goroutines to complete their work and get the result of their work into the channel, you don't need a weight group (because it won't be reached).
You can do something like this.
package main
import (
"fmt"
"time"
)
func main() {
ch1 := make(chan string)
ch2 := make(chan string)
go goOne(ch1)
go goTwo(ch2)
for {
select {
case v := <-ch1:
fmt.Println("Done ch1:", v)
ch1 = nil
case v := <-ch2:
fmt.Println("Done ch2:", v)
ch2 = nil
case <-time.After(time.Second):
fmt.Println("I didn't get result so lets skip it!")
ch1, ch2 = nil, nil
}
if ch1 == nil && ch2 == nil {
break
}
}
}
func goTwo(ch chan string) {
ch <- "Channel 2"
}
func goOne(_ chan string) {
//ch1 <- "Channel 1"
}
UPD:
Imagine if we are having two api end points, API1 & API2 which are returning same data but are hosted on different regions. So what I want to do, I need to make API calls for both apis in two different function ie goroutines and as soon as any one api sends us response back, I want to process the data received. So for that Im check whcih api is fetching data first using select block.
package main
import (
"context"
"fmt"
"math/rand"
"time"
)
func main() {
regions := []string{
"Europe",
"America",
"Asia",
}
// Just for different results for each run
rand.Seed(time.Now().UnixNano())
rand.Shuffle(len(regions), func(i, j int) { regions[i], regions[j] = regions[j], regions[i] })
output := make(chan string)
ctx, cancel := context.WithTimeout(context.Background(), 2 * time.Second)
for i, region := range regions {
go func(ctx context.Context, region string, output chan <- string, i int) {
// Do call with context
// If context will be cancelled just ignore it here
timeout := time.Duration(i)*time.Second
fmt.Printf("Call %s (with timeout %s)\n", region, timeout)
time.Sleep(timeout) // Simulate request timeout
select {
case <-ctx.Done():
fmt.Println("Cancel by context:", region)
case output <- fmt.Sprintf("Answer from `%s`", region):
fmt.Println("Done:", region)
}
}(ctx, region, output, i)
}
select {
case v := <-output:
cancel() // Cancel all requests in progress (if possible)
// Do not close output chan to avoid panics: When the channel is no longer used, it will be garbage collected.
fmt.Println("Result:", v)
case <-ctx.Done():
fmt.Println("Timeout by context done")
}
fmt.Println("There is we already have result or timeout, but wait a little bit to check all is okay")
time.Sleep(5*time.Second)
}
Firstly you have a race condition in that your channel publishing goroutines will probably not have been started by the time you enter the select statement, and it will immediately fall through to the default.
But assuming you resolve this (e.g. with another form of semaphore) you're on to the next issue - your select statement will either get chan1 or chan2 message, then wait at the bottom of the method, but since it is no longer in the select statement, one of your messages won't have arrived and you'll be waiting forever.
You'll need either a loop (twice in this case) or a receiver for both channels running in their own goroutines to pick up both messages and complete the waitgroup.
But in any case - as others have queried - what are you trying to achieve?
You can try something like iterating over a single channel (assuming both channels return the same type of data) and then count in the main method how many tasks are completed. Then close the channel once all the tasks are done. Example:
package main
import (
"fmt"
)
func main() {
ch := make(chan string)
go goOne(ch)
go goTwo(ch)
doneCount := 0
for v := range ch {
fmt.Println(v)
doneCount++
if doneCount == 2 {
close(ch)
}
}
}
func goTwo(ch chan string) {
ch <- "Channel 2"
}
func goOne(ch chan string) {
ch <- "Channel 1"
}
The following piece of code try to send to the channel on the main goroutine and receive from another goroutine but a few times it returns as expected but a few times it exits without printing any on the console screen
package main
import "fmt"
func main() {
ch := make(chan bool)
go func() {
data := <-ch
fmt.Printf("Received: %t", data)
}()
ch <- true
}
At the same time, the following piece of code works as expected everytime, one difference is that an additional check has been added to check if the channel is closed or not which always throws the same expected output.
Does this ensure that a check on the channel is a must than optional ? or anything wrong with the code
package main
import "fmt"
func main() {
ch := make(chan bool)
go func() {
data, ok := <-ch
if !ok {
fmt.Println("Channel closed")
return
}
fmt.Printf("Received: %t", data)
}()
ch <- true
}
You should wait for goroutine to complete before main routine exit.
package main
import (
"fmt"
"sync"
)
func main() {
ch := make(chan bool)
var wg sync.WaitGroup
wg.Add(1)
go func() {
defer wg.Done()
data := <-ch
fmt.Printf("Received: %t", data)
}()
ch <- true
wg.Wait()
}
The thing is your second piece of code doesn't print Received: true every time. I tested it several times.
As #jub0bs mentioned there is no guarantee that your goroutine finishes before the main routine. You must control it yourself.
There are two similar cases I would like to compare with you - the only difference is the way of handling values generation
the first case: values generation in one of case of select
package main
import (
"fmt"
"math/rand"
"time"
)
func main() {
generateValues := func(done <-chan interface{}) <-chan int {
values := make(chan int)
go func() {
defer fmt.Println("All values generated")
defer close(values)
for {
select {
case <-done:
fmt.Println("DONE")
return
case values <- rand.Int():
fmt.Println("Generated")
}
}
}()
return values
}
done := make(chan interface{})
values := generateValues(done)
for i := 0; i < 3; i++ {
fmt.Printf("Received value: %v\n", <-values)
}
fmt.Println("Closing the channel")
close(done)
time.Sleep(5 * time.Second)
}
Go playground: https://go.dev/play/p/edlOSqdZ9ys
the second case: value generation in default case
package main
import (
"fmt"
"math/rand"
"time"
)
func main() {
generateValues := func(done <-chan interface{}) <-chan int {
values := make(chan int)
go func() {
defer fmt.Println("All values generated")
defer close(values)
for {
select {
case <-done:
fmt.Println("DONE")
return
default:
values <- rand.Int()
fmt.Println("Generated")
}
}
}()
return values
}
done := make(chan interface{})
values := generateValues(done)
for i := 0; i < 3; i++ {
fmt.Printf("Received value: %v\n", <-values)
}
fmt.Println("Closing the channel")
close(done)
time.Sleep(5 * time.Second)
}
Go playground: https://go.dev/play/p/edlOSqdZ9ys
As you can see the second case seems leading to the situation that the 'Done' is not printed and calls related to 'defer' are not invoked. I believe we have here goroutine leaks, but cannot clearly explain it. I expected the same behaviour like in the first case.
Could someone please help in understanding the difference between them?
In the second case, the generating goroutine is not likely to receive the done message. Since the default case is always enabled, after the main goroutine receives the last value, the generating goroutine makes another round, falls into the default case, and blocks waiting to write to the values channel. While it is waiting there, the main goroutine closes the done channel and terminates.
This doesn't mean there doesn't exist an execution path where the generating goroutine doesn't receive the done channel. For this to happen, immediately after sending the last value, the generating goroutine must be preempted by the main goroutine that runs until it closes the done channel. Then if the generating goroutine is scheduled, it can receive the done signal. However, this sequence of events is highly unlikely.
Question
How to make the below code print "QUIT" after 3 seconds?
Code
package main
import (
"fmt"
"time"
)
func main() {
quit := make(chan struct{})
tasks := make(chan struct{})
go func(){
time.Sleep(1 * time.Second)
tasks <- struct{}{}
}()
go func(){
time.Sleep(3 * time.Second)
quit <- struct{}{}
}()
for {
select {
case <-quit:
fmt.Println("QUIT")
return
case <-tasks:
fmt.Println("Doing")
// do some long time jobs more than 10 seconds
time.Sleep(10 * time.Second)
}
}
}
Observations
The above code prints "Doing". and sleep 10 seconds, then print "QUIT".
How to interrupt this sleep, let it receive quit channel after 3 seconds and print "QUIT"?
It seems select is blocked by case tasks, and it will not receive from the quit channel after 3 seconds.
To signal end of an asynchronous task it is a best practice to close the channel, this is rather important to prevent many missuse that leads to various deadlocks.
In your original code I would have written close(quite) rather than quit <- struct{}{}
Remember, read on a closed does not block and always return the zero value, that is the trick.
Anyways, appart from this, an elegant way to solve your problem is to use a combination of both context.Context and time.After.
time.After will help you block on a selectable task set.
context.Context is better suited to handle this kind of signals.
https://play.golang.org/p/ZVsZw3P-YHd
package main
import (
"context"
"log"
"time"
)
func main() {
log.Println("start")
defer log.Println("end")
ctx, cancel := context.WithCancel(context.Background())
tasks := make(chan struct{})
go func() {
time.Sleep(1 * time.Second) // a job
tasks <- struct{}{}
}()
go func() {
time.Sleep(3 * time.Second) // a job
cancel()
}()
for {
select {
case <-ctx.Done():
log.Println("QUIT")
return
case <-tasks:
log.Println("Doing")
// do some long time jobs more than 10 seconds
select {
case <-ctx.Done():
return
case <-time.After(time.Second * 10):
}
}
}
}
The second job is running for 10 seconds, so it will block the loop for that time, and the signal you sent into the quit channel will not be received until this job is complete.
To interrupt the time-consuming job, maybe you can split it into another goroutine. Something like this would do:
go func() {
for {
select {
case <-tasks:
fmt.Println("Doing")
// do some long time jobs more than 10 seconds
time.Sleep(10 * time.Second)
}
}
}()
<-quit
fmt.Println("QUIT")
It would be easier to use [sync.WaitGroup.
package main
import (
"fmt"
"sync"
"time"
)
func worker(msg string, duration time.Duration, doing bool, wg *sync.WaitGroup) {
defer wg.Done()
time.Sleep(duration)
fmt.Println(msg)
if doing {
time.Sleep(10 * time.Second)
}
}
func main() {
var wg sync.WaitGroup
msgs := [2]string{"QUIT", "Doing"}
durations := [2]time.Duration{3 * time.Second, 1 * time.Second}
doing := [2]bool{false, true}
for i, msg := range msgs {
wg.Add(1)
go worker(msg, durations[i], doing[i], &wg)
}
wg.Wait()
}
Is there any way to exit a Go program, but execute all the pending defer statements?
I've been clearing up temporary files by using defer, but the deferred statements aren't executed when the program is interrupted with Ctrl+C or even os.Exit.
After exiting this program with Ctrl+C, both foo.txt and bar.txt are left over:
package main
import (
"fmt"
"io/ioutil"
"os"
"os/signal"
"syscall"
)
func main() {
ioutil.WriteFile("./foo.txt", []byte("foo"), 0644)
defer os.RemoveAll("./foo.txt")
go func() {
ioutil.WriteFile("./bar.txt", []byte("bar"), 0644)
defer os.RemoveAll("./bar.txt")
for {
// various long running things
}
}()
c := make(chan os.Signal, 1)
signal.Notify(c, os.Interrupt)
signal.Notify(c, syscall.SIGTERM)
go func() {
<-c
fmt.Println("Received OS interrupt - exiting.")
os.Exit(0)
}()
for {
// various long running things
}
}
From golang reference:
A "defer" statement invokes a function whose execution is deferred to
the moment the surrounding function returns
When you call os.Exit(0) you bypass the normal return procedure and your deferred functions are not executed.
Also, even if the deferred worked inside the main goroutine, the defers in other goroutines would not work since they would die before returning.
A better code architecture would allow you to get something similar. You need to think about your long running processes as workers. Export every long running process in workers and defer any clean up right after calling the worker. Use a select in the main goroutine to capture signals and synchronise work
package main
import (
"fmt"
"io/ioutil"
"os"
"os/signal"
"syscall"
"time"
)
func check(e error) {
if e != nil {
panic(e)
}
}
func main() {
ioutil.WriteFile("./foo.txt", []byte("foo"), 0644)
defer os.RemoveAll("./foo.txt")
// Worker 1
done := make(chan bool, 1)
go func(done chan bool) {
fmt.Println("worker 1 with bar ...")
ioutil.WriteFile("./bar.txt", []byte("bar"), 0644)
// various long running things
time.Sleep(3 * time.Second)
done <- true
}(done)
defer os.RemoveAll("./bar.txt")
// End worker1
s := make(chan os.Signal, 1)
signal.Notify(s, os.Interrupt)
signal.Notify(s, syscall.SIGTERM)
// Worker 2
done2 := make(chan bool, 1)
go func(done chan bool) {
fmt.Println("worker 2 ...")
time.Sleep(6 * time.Second)
done <- true
}(done2)
// End worker 2
select {
case <-s:
fmt.Println("Quiting with signal - exit")
case <-done:
<-done2
case <-done2:
<-done
}
return
}
This select is a quick and dirty way to handle two workers, a better way would be to use sync.WaitGroup
I would recommend not relying on defer, but defining a reusable function that can be used in a defer or in the signal block. Something like this:
package main
import (
"fmt"
"io/ioutil"
"os"
"os/signal"
"syscall"
)
func main() {
ioutil.WriteFile("./foo.txt", []byte("foo"), 0644)
cleanup := func(){
os.RemoveAll("./foo.txt")
os.RemoveAll("./bar.txt")
}
defer cleanup() //for normal return
go func() {
ioutil.WriteFile("./bar.txt", []byte("bar"), 0644)
for {
// various long running things
}
}()
c := make(chan os.Signal, 1)
signal.Notify(c, os.Interrupt)
signal.Notify(c, syscall.SIGTERM)
go func() {
<-c
fmt.Println("Received OS interrupt - exiting.")
cleanup()
os.Exit(0)
}()
for {
// various long running things
}
}