There are two similar cases I would like to compare with you - the only difference is the way of handling values generation
the first case: values generation in one of case of select
package main
import (
"fmt"
"math/rand"
"time"
)
func main() {
generateValues := func(done <-chan interface{}) <-chan int {
values := make(chan int)
go func() {
defer fmt.Println("All values generated")
defer close(values)
for {
select {
case <-done:
fmt.Println("DONE")
return
case values <- rand.Int():
fmt.Println("Generated")
}
}
}()
return values
}
done := make(chan interface{})
values := generateValues(done)
for i := 0; i < 3; i++ {
fmt.Printf("Received value: %v\n", <-values)
}
fmt.Println("Closing the channel")
close(done)
time.Sleep(5 * time.Second)
}
Go playground: https://go.dev/play/p/edlOSqdZ9ys
the second case: value generation in default case
package main
import (
"fmt"
"math/rand"
"time"
)
func main() {
generateValues := func(done <-chan interface{}) <-chan int {
values := make(chan int)
go func() {
defer fmt.Println("All values generated")
defer close(values)
for {
select {
case <-done:
fmt.Println("DONE")
return
default:
values <- rand.Int()
fmt.Println("Generated")
}
}
}()
return values
}
done := make(chan interface{})
values := generateValues(done)
for i := 0; i < 3; i++ {
fmt.Printf("Received value: %v\n", <-values)
}
fmt.Println("Closing the channel")
close(done)
time.Sleep(5 * time.Second)
}
Go playground: https://go.dev/play/p/edlOSqdZ9ys
As you can see the second case seems leading to the situation that the 'Done' is not printed and calls related to 'defer' are not invoked. I believe we have here goroutine leaks, but cannot clearly explain it. I expected the same behaviour like in the first case.
Could someone please help in understanding the difference between them?
In the second case, the generating goroutine is not likely to receive the done message. Since the default case is always enabled, after the main goroutine receives the last value, the generating goroutine makes another round, falls into the default case, and blocks waiting to write to the values channel. While it is waiting there, the main goroutine closes the done channel and terminates.
This doesn't mean there doesn't exist an execution path where the generating goroutine doesn't receive the done channel. For this to happen, immediately after sending the last value, the generating goroutine must be preempted by the main goroutine that runs until it closes the done channel. Then if the generating goroutine is scheduled, it can receive the done signal. However, this sequence of events is highly unlikely.
Related
The problem is that both the goOne and goTwo functions are sending values to the channels ch1 and ch2 respectively, but there is no corresponding receiver for these values in the main function. This means that the channels are blocked and the program is unable to proceed. As a result, the select statement in the main function is unable to read from the channels, so it always executes the default case.
package main
import (
"fmt"
"sync"
)
func main() {
var wg sync.WaitGroup
ch1 := make(chan string)
ch2 := make(chan string)
wg.Add(2)
go goOne(&wg, ch1)
go goTwo(&wg, ch2)
select {
case <-ch1:
fmt.Println(<-ch1)
close(ch1)
case <-ch2:
fmt.Println(<-ch2)
close(ch2)
default:
fmt.Println("Default Case")
}
wg.Wait()
}
func goTwo(wg *sync.WaitGroup, ch2 chan string) {
ch2 <- "Channel 2"
wg.Done()
}
func goOne(wg *sync.WaitGroup, ch1 chan string) {
ch1 <- "Channel 1"
wg.Done()
}
Output:
Default Case
fatal error: all goroutines are asleep - deadlock!
goroutine 1 \[semacquire\]:
sync.runtime_Semacquire(0xc000108270?)
/usr/local/go/src/runtime/sema.go:62 +0x25
sync.(\*WaitGroup).Wait(0x4b9778?)
/usr/local/go/src/sync/waitgroup.go:139 +0x52
main.main()
/home/nidhey/Documents/Go_Learning/goroutines/select.go:29 +0x2af
goroutine 6 \[chan send\]:
main.goOne(0x0?, 0x0?)
/home/nidhey/Documents/Go_Learning/goroutines/select.go:39 +0x28
created by main.main
/home/nidhey/Documents/Go_Learning/goroutines/select.go:14 +0xc5
goroutine 7 \[chan send\]:
main.goTwo(0x0?, 0x0?)
/home/nidhey/Documents/Go_Learning/goroutines/select.go:33 +0x28
created by main.main
/home/nidhey/Documents/Go_Learning/goroutines/select.go:15 +0x119\```
I'm looking for a different pattern such as select to handle the case when the channels are blocked.
To fix the issue, I've added a <-ch1 or <-ch2 in the main function after wg.Wait() to receive the values sent to the channels and unblock them
It's not entirely clear what you want to do. If you want to wait for both goroutines to complete their work and get the result of their work into the channel, you don't need a weight group (because it won't be reached).
You can do something like this.
package main
import (
"fmt"
"time"
)
func main() {
ch1 := make(chan string)
ch2 := make(chan string)
go goOne(ch1)
go goTwo(ch2)
for {
select {
case v := <-ch1:
fmt.Println("Done ch1:", v)
ch1 = nil
case v := <-ch2:
fmt.Println("Done ch2:", v)
ch2 = nil
case <-time.After(time.Second):
fmt.Println("I didn't get result so lets skip it!")
ch1, ch2 = nil, nil
}
if ch1 == nil && ch2 == nil {
break
}
}
}
func goTwo(ch chan string) {
ch <- "Channel 2"
}
func goOne(_ chan string) {
//ch1 <- "Channel 1"
}
UPD:
Imagine if we are having two api end points, API1 & API2 which are returning same data but are hosted on different regions. So what I want to do, I need to make API calls for both apis in two different function ie goroutines and as soon as any one api sends us response back, I want to process the data received. So for that Im check whcih api is fetching data first using select block.
package main
import (
"context"
"fmt"
"math/rand"
"time"
)
func main() {
regions := []string{
"Europe",
"America",
"Asia",
}
// Just for different results for each run
rand.Seed(time.Now().UnixNano())
rand.Shuffle(len(regions), func(i, j int) { regions[i], regions[j] = regions[j], regions[i] })
output := make(chan string)
ctx, cancel := context.WithTimeout(context.Background(), 2 * time.Second)
for i, region := range regions {
go func(ctx context.Context, region string, output chan <- string, i int) {
// Do call with context
// If context will be cancelled just ignore it here
timeout := time.Duration(i)*time.Second
fmt.Printf("Call %s (with timeout %s)\n", region, timeout)
time.Sleep(timeout) // Simulate request timeout
select {
case <-ctx.Done():
fmt.Println("Cancel by context:", region)
case output <- fmt.Sprintf("Answer from `%s`", region):
fmt.Println("Done:", region)
}
}(ctx, region, output, i)
}
select {
case v := <-output:
cancel() // Cancel all requests in progress (if possible)
// Do not close output chan to avoid panics: When the channel is no longer used, it will be garbage collected.
fmt.Println("Result:", v)
case <-ctx.Done():
fmt.Println("Timeout by context done")
}
fmt.Println("There is we already have result or timeout, but wait a little bit to check all is okay")
time.Sleep(5*time.Second)
}
Firstly you have a race condition in that your channel publishing goroutines will probably not have been started by the time you enter the select statement, and it will immediately fall through to the default.
But assuming you resolve this (e.g. with another form of semaphore) you're on to the next issue - your select statement will either get chan1 or chan2 message, then wait at the bottom of the method, but since it is no longer in the select statement, one of your messages won't have arrived and you'll be waiting forever.
You'll need either a loop (twice in this case) or a receiver for both channels running in their own goroutines to pick up both messages and complete the waitgroup.
But in any case - as others have queried - what are you trying to achieve?
You can try something like iterating over a single channel (assuming both channels return the same type of data) and then count in the main method how many tasks are completed. Then close the channel once all the tasks are done. Example:
package main
import (
"fmt"
)
func main() {
ch := make(chan string)
go goOne(ch)
go goTwo(ch)
doneCount := 0
for v := range ch {
fmt.Println(v)
doneCount++
if doneCount == 2 {
close(ch)
}
}
}
func goTwo(ch chan string) {
ch <- "Channel 2"
}
func goOne(ch chan string) {
ch <- "Channel 1"
}
In this example, we have a worker. The idea here is simulate clean shutdown of all go routines based on a condition.
In this case, go routines get spun - based on workers count. Each go routine reads the channel, does some work and sends output to the outputChannel.
The main go routine reads this output and prints it. To simulate a stop condition, the doneChannel is closed. Expected outcome is that select inside each go routine will pick this up and execute return which in turn will call the defer println. The actual output is that it never gets called and main exits.
Not sure what's the reason behind this.
package main
import (
"log"
"time"
)
const jobs = 100
const workers = 1
var timeout = time.After(5 * time.Second)
func main() {
doneChannel := make(chan interface{})
outputChannel := make(chan int)
numberStream := generator()
for i := 1; i <= workers; i++ {
go worker(doneChannel, numberStream, outputChannel)
}
// listen for output
loop:
for {
select {
case i := <-outputChannel:
log.Println(i)
case <-timeout:
// before you timeout cleanup go routines
break loop
}
}
close(doneChannel)
time.Sleep(5 * time.Second)
log.Println("main exited")
}
func generator() <-chan int {
defer log.Println("generator completed !")
c := make(chan int)
go func() {
for i := 1; i <= jobs; i++ {
c <- i
}
defer close(c)
}()
return c
}
func worker(done <-chan interface{}, c <-chan int, output chan<- int) {
// this will be a go routine
// Do some work and send results to output Channel.
// Incase if the done channel is called kill the go routine.
defer log.Println("go routines exited")
for {
select {
case <-done:
log.Println("here")
return
case i := <-c:
time.Sleep(1 * time.Second) // worker delay
output <- i * 100
}
}
}
When your main loop finishes during the timeout, you continue your program and
Close done channel
Print message
Exit
There is no reason to wait for any goroutine to process the signal of this channel.
If you add a small sleep you will see some messages
In real scenarios we use a waitgroup to be sure all goroutine finish properly
The following piece of code try to send to the channel on the main goroutine and receive from another goroutine but a few times it returns as expected but a few times it exits without printing any on the console screen
package main
import "fmt"
func main() {
ch := make(chan bool)
go func() {
data := <-ch
fmt.Printf("Received: %t", data)
}()
ch <- true
}
At the same time, the following piece of code works as expected everytime, one difference is that an additional check has been added to check if the channel is closed or not which always throws the same expected output.
Does this ensure that a check on the channel is a must than optional ? or anything wrong with the code
package main
import "fmt"
func main() {
ch := make(chan bool)
go func() {
data, ok := <-ch
if !ok {
fmt.Println("Channel closed")
return
}
fmt.Printf("Received: %t", data)
}()
ch <- true
}
You should wait for goroutine to complete before main routine exit.
package main
import (
"fmt"
"sync"
)
func main() {
ch := make(chan bool)
var wg sync.WaitGroup
wg.Add(1)
go func() {
defer wg.Done()
data := <-ch
fmt.Printf("Received: %t", data)
}()
ch <- true
wg.Wait()
}
The thing is your second piece of code doesn't print Received: true every time. I tested it several times.
As #jub0bs mentioned there is no guarantee that your goroutine finishes before the main routine. You must control it yourself.
I am trying to understand context in golang. I copied an example from https://golang.org/pkg/context/#example_WithCancel and changed it a bit:
Playgroud: https://play.golang.org/p/Aczc2CqcVZR
package main
import (
"context"
"fmt"
"time"
)
func main() {
// gen generates integers in a separate goroutine and
// sends them to the returned channel.
// The callers of gen need to cancel the context once
// they are done consuming generated integers not to leak
// the internal goroutine started by gen.
gen := func(ctx context.Context) <-chan int {
dst := make(chan int)
n := 1
go func() {
for {
select {
case <-ctx.Done():
fmt.Println("DONE")
return // returning not to leak the goroutine
case dst <- n:
n++
}
}
fmt.Println("END")
}()
return dst
}
ctx, cancel := context.WithCancel(context.Background())
defer time.Sleep(1 * time.Second)
defer fmt.Println("Before cancel")
defer cancel() // cancel when we are finished consuming integers
defer fmt.Println("After cancel")
channel := gen(ctx)
for n := range channel {
fmt.Println(n)
if n == 5 {
break
}
}
fmt.Println( <-channel)
}
When commenting out the
defer time.Sleep(1 * time.Second)
the "DONE" never gets printed. Playgroud: (https://play.golang.org/p/K0OcyZaj_xK)
I would expect the go routine which was started in the anonymous function still to be active. Once cancel() is called due to being deferred, the select should no longer block as
case <-ctx.Done():
should be available. However it seems to just end, unless I wait for 1 second and give it time. This behavior seems very wrong.
This behavior seems very wrong.
It's not. That's how program execution is specified. After main and its deferred functions return, the program exits.
Program execution begins by initializing the main package and then invoking the function main. When that function invocation returns, the program exits. It does not wait for other (non-main) goroutines to complete.
https://golang.org/ref/spec#Program_execution
I'm here to find out the most idiomatic way to do the follow task.
Task:
Write data from a channel to a file.
Problem:
I have a channel ch := make(chan int, 100)
I need to read from the channel and write the values I read from the channel to a file. My question is basically how do I do so given that
If channel ch is full, write the values immediately
If channel ch is not full, write every 5s.
So essentially, data needs to be written to the file at least every 5s (assuming that data will be filled into the channel at least every 5s)
Whats the best way to use select, for and range to do my above task?
Thanks!
There is no such "event" as "buffer of channel is full", so you can't detect that [*]. This means you can't idiomatically solve your problem with language primitives using only 1 channel.
[*] Not entirely true: you could detect if the buffer of a channel is full by using select with default case when sending on the channel, but that requires logic from the senders, and repetitive attempts to send.
I would use another channel from which I would receive as values are sent on it, and "redirect", store the values in another channel which has a buffer of 100 as you mentioned. At each redirection you may check if the internal channel's buffer is full, and if so, do an immediate write. If not, continue to monitor the "incoming" channel and a timer channel with a select statement, and if the timer fires, do a "regular" write.
You may use len(chInternal) to check how many elements are in the chInternal channel, and cap(chInternal) to check its capacity. Note that this is "safe" as we are the only goroutine handling the chInternal channel. If there would be multiple goroutines, value returned by len(chInternal) could be outdated by the time we use it to something (e.g. comparing it).
In this solution chInternal (as its name says) is for internal use only. Others should only send values on ch. Note that ch may or may not be a buffered channel, solution works in both cases. However, you may improve efficiency if you also give some buffer to ch (so chances that senders get blocked will be lower).
var (
chInternal = make(chan int, 100)
ch = make(chan int) // You may (should) make this a buffered channel too
)
func main() {
delay := time.Second * 5
timer := time.NewTimer(delay)
for {
select {
case v := <-ch:
chInternal <- v
if len(chInternal) == cap(chInternal) {
doWrite() // Buffer is full, we need to write immediately
timer.Reset(delay)
}
case <-timer.C:
doWrite() // "Regular" write: 5 seconds have passed since last write
timer.Reset(delay)
}
}
}
If an immediate write happens (due to a "buffer full" situation), this solution will time the next "regular" write 5 seconds after this. If you don't want this and you want the 5-second regular writes be independent from the immediate writes, simply do not reset the timer following the immediate write.
An implementation of doWrite() may be as follows:
var f *os.File // Make sure to open file for writing
func doWrite() {
for {
select {
case v := <-chInternal:
fmt.Fprintf(f, "%d ", v) // Write v to the file
default: // Stop when no more values in chInternal
return
}
}
}
We can't use for ... range as that only returns when the channel is closed, but our chInternal channel is not closed. So we use a select with a default case so when no more values are in the buffer of chInternal, we return.
Improvements
Using a slice instead of 2nd channel
Since the chInternal channel is only used by us, and only on a single goroutine, we may also choose to use a single []int slice instead of a channel (reading/writing a slice is much faster than a channel).
Showing only the different / changed parts, it could look something like this:
var (
buf = make([]int, 0, 100)
)
func main() {
// ...
for {
select {
case v := <-ch:
buf = append(buf, v)
if len(buf) == cap(buf) {
// ...
}
}
func doWrite() {
for _, v := range buf {
fmt.Fprintf(f, "%d ", v) // Write v to the file
}
buf = buf[:0] // "Clear" the buffer
}
With multiple goroutines
If we stick to leave chInternal a channel, the doWrite() function may be called on another goroutine to not block the other one, e.g. go doWrite(). Since data to write is read from a channel (chInternal), this requires no further synchronization.
if you just use 5 seconds write, to increase the file write performance,
you may fill the channel any time you need,
then writer goroutine writes that data to the buffered file,
see this very simple and idiomatic sample without using timer
with just using for...range:
package main
import (
"bufio"
"fmt"
"os"
"sync"
)
var wg sync.WaitGroup
func WriteToFile(filename string, ch chan int) {
f, e := os.Create(filename)
if e != nil {
panic(e)
}
w := bufio.NewWriterSize(f, 4*1024*1024)
defer wg.Done()
defer f.Close()
defer w.Flush()
for v := range ch {
fmt.Fprintf(w, "%d ", v)
}
}
func main() {
ch := make(chan int, 100)
wg.Add(1)
go WriteToFile("file.txt", ch)
for i := 0; i < 500000; i++ {
ch <- i // do the job
}
close(ch) // Finish the job and close output file
wg.Wait()
}
and notice the defers order.
and in case of 5 seconds write, you may add one interval timer just to flush the buffer of this file to the disk, like this:
package main
import (
"bufio"
"fmt"
"os"
"sync"
"time"
)
var wg sync.WaitGroup
func WriteToFile(filename string, ch chan int) {
f, e := os.Create(filename)
if e != nil {
panic(e)
}
w := bufio.NewWriterSize(f, 4*1024*1024)
ticker := time.NewTicker(5 * time.Second)
quit := make(chan struct{})
go func() {
for {
select {
case <-ticker.C:
if w.Buffered() > 0 {
fmt.Println(w.Buffered())
w.Flush()
}
case <-quit:
ticker.Stop()
return
}
}
}()
defer wg.Done()
defer f.Close()
defer w.Flush()
defer close(quit)
for v := range ch {
fmt.Fprintf(w, "%d ", v)
}
}
func main() {
ch := make(chan int, 100)
wg.Add(1)
go WriteToFile("file.txt", ch)
for i := 0; i < 25; i++ {
ch <- i // do the job
time.Sleep(500 * time.Millisecond)
}
close(ch) // Finish the job and close output file
wg.Wait()
}
here I used time.NewTicker(5 * time.Second) for interval timer with quit channel, you may use time.AfterFunc() or time.Tick() or time.Sleep().
with some optimizations ( removing quit channel):
package main
import (
"bufio"
"fmt"
"os"
"sync"
"time"
)
var wg sync.WaitGroup
func WriteToFile(filename string, ch chan int) {
f, e := os.Create(filename)
if e != nil {
panic(e)
}
w := bufio.NewWriterSize(f, 4*1024*1024)
ticker := time.NewTicker(5 * time.Second)
defer wg.Done()
defer f.Close()
defer w.Flush()
for {
select {
case v, ok := <-ch:
if ok {
fmt.Fprintf(w, "%d ", v)
} else {
fmt.Println("done.")
ticker.Stop()
return
}
case <-ticker.C:
if w.Buffered() > 0 {
fmt.Println(w.Buffered())
w.Flush()
}
}
}
}
func main() {
ch := make(chan int, 100)
wg.Add(1)
go WriteToFile("file.txt", ch)
for i := 0; i < 25; i++ {
ch <- i // do the job
time.Sleep(500 * time.Millisecond)
}
close(ch) // Finish the job and close output file
wg.Wait()
}
I hope this helps.