func main() {
rand.Seed(time.Now().Unix())
ctx, cancelFunc := context.WithCancel(context.Background())
anies := make(chan any)
go doSomething(ctx, anies)
intn := rand.Intn(2)
if intn == 0 { //BRANCH1
cancelFunc()
close(anies)
}
time.Sleep(time.Second)
}
func doSomething(ctx context.Context, anies chan any) {
for {
if ctx.Err() == nil { //LINE2
anies <- 1 //LINE3
}
}
}
Can it be possible that somewhen BRANCH1 happens between LINE2 AND LINE3 and I will got panic.
Yes, a panic is possible. Here's an example timeline where a panic occurs. The lines are in increasing time order. The N: prefix represents the goroutine.
1: start goroutine 2
2: call ctx.Err(), it returns nil
1: call cancelFunc()
1: close channel anies
2: send to channel anies. panic because channel is closed.
Related
The problem is that both the goOne and goTwo functions are sending values to the channels ch1 and ch2 respectively, but there is no corresponding receiver for these values in the main function. This means that the channels are blocked and the program is unable to proceed. As a result, the select statement in the main function is unable to read from the channels, so it always executes the default case.
package main
import (
"fmt"
"sync"
)
func main() {
var wg sync.WaitGroup
ch1 := make(chan string)
ch2 := make(chan string)
wg.Add(2)
go goOne(&wg, ch1)
go goTwo(&wg, ch2)
select {
case <-ch1:
fmt.Println(<-ch1)
close(ch1)
case <-ch2:
fmt.Println(<-ch2)
close(ch2)
default:
fmt.Println("Default Case")
}
wg.Wait()
}
func goTwo(wg *sync.WaitGroup, ch2 chan string) {
ch2 <- "Channel 2"
wg.Done()
}
func goOne(wg *sync.WaitGroup, ch1 chan string) {
ch1 <- "Channel 1"
wg.Done()
}
Output:
Default Case
fatal error: all goroutines are asleep - deadlock!
goroutine 1 \[semacquire\]:
sync.runtime_Semacquire(0xc000108270?)
/usr/local/go/src/runtime/sema.go:62 +0x25
sync.(\*WaitGroup).Wait(0x4b9778?)
/usr/local/go/src/sync/waitgroup.go:139 +0x52
main.main()
/home/nidhey/Documents/Go_Learning/goroutines/select.go:29 +0x2af
goroutine 6 \[chan send\]:
main.goOne(0x0?, 0x0?)
/home/nidhey/Documents/Go_Learning/goroutines/select.go:39 +0x28
created by main.main
/home/nidhey/Documents/Go_Learning/goroutines/select.go:14 +0xc5
goroutine 7 \[chan send\]:
main.goTwo(0x0?, 0x0?)
/home/nidhey/Documents/Go_Learning/goroutines/select.go:33 +0x28
created by main.main
/home/nidhey/Documents/Go_Learning/goroutines/select.go:15 +0x119\```
I'm looking for a different pattern such as select to handle the case when the channels are blocked.
To fix the issue, I've added a <-ch1 or <-ch2 in the main function after wg.Wait() to receive the values sent to the channels and unblock them
It's not entirely clear what you want to do. If you want to wait for both goroutines to complete their work and get the result of their work into the channel, you don't need a weight group (because it won't be reached).
You can do something like this.
package main
import (
"fmt"
"time"
)
func main() {
ch1 := make(chan string)
ch2 := make(chan string)
go goOne(ch1)
go goTwo(ch2)
for {
select {
case v := <-ch1:
fmt.Println("Done ch1:", v)
ch1 = nil
case v := <-ch2:
fmt.Println("Done ch2:", v)
ch2 = nil
case <-time.After(time.Second):
fmt.Println("I didn't get result so lets skip it!")
ch1, ch2 = nil, nil
}
if ch1 == nil && ch2 == nil {
break
}
}
}
func goTwo(ch chan string) {
ch <- "Channel 2"
}
func goOne(_ chan string) {
//ch1 <- "Channel 1"
}
UPD:
Imagine if we are having two api end points, API1 & API2 which are returning same data but are hosted on different regions. So what I want to do, I need to make API calls for both apis in two different function ie goroutines and as soon as any one api sends us response back, I want to process the data received. So for that Im check whcih api is fetching data first using select block.
package main
import (
"context"
"fmt"
"math/rand"
"time"
)
func main() {
regions := []string{
"Europe",
"America",
"Asia",
}
// Just for different results for each run
rand.Seed(time.Now().UnixNano())
rand.Shuffle(len(regions), func(i, j int) { regions[i], regions[j] = regions[j], regions[i] })
output := make(chan string)
ctx, cancel := context.WithTimeout(context.Background(), 2 * time.Second)
for i, region := range regions {
go func(ctx context.Context, region string, output chan <- string, i int) {
// Do call with context
// If context will be cancelled just ignore it here
timeout := time.Duration(i)*time.Second
fmt.Printf("Call %s (with timeout %s)\n", region, timeout)
time.Sleep(timeout) // Simulate request timeout
select {
case <-ctx.Done():
fmt.Println("Cancel by context:", region)
case output <- fmt.Sprintf("Answer from `%s`", region):
fmt.Println("Done:", region)
}
}(ctx, region, output, i)
}
select {
case v := <-output:
cancel() // Cancel all requests in progress (if possible)
// Do not close output chan to avoid panics: When the channel is no longer used, it will be garbage collected.
fmt.Println("Result:", v)
case <-ctx.Done():
fmt.Println("Timeout by context done")
}
fmt.Println("There is we already have result or timeout, but wait a little bit to check all is okay")
time.Sleep(5*time.Second)
}
Firstly you have a race condition in that your channel publishing goroutines will probably not have been started by the time you enter the select statement, and it will immediately fall through to the default.
But assuming you resolve this (e.g. with another form of semaphore) you're on to the next issue - your select statement will either get chan1 or chan2 message, then wait at the bottom of the method, but since it is no longer in the select statement, one of your messages won't have arrived and you'll be waiting forever.
You'll need either a loop (twice in this case) or a receiver for both channels running in their own goroutines to pick up both messages and complete the waitgroup.
But in any case - as others have queried - what are you trying to achieve?
You can try something like iterating over a single channel (assuming both channels return the same type of data) and then count in the main method how many tasks are completed. Then close the channel once all the tasks are done. Example:
package main
import (
"fmt"
)
func main() {
ch := make(chan string)
go goOne(ch)
go goTwo(ch)
doneCount := 0
for v := range ch {
fmt.Println(v)
doneCount++
if doneCount == 2 {
close(ch)
}
}
}
func goTwo(ch chan string) {
ch <- "Channel 2"
}
func goOne(ch chan string) {
ch <- "Channel 1"
}
I wish to put a timeout on a function called foo. Consider the following
func fooWithTimeout(d time.Duration) error {
ch := make(chan error, 1)
go func() {
ch <- foo()
}()
select {
case err := <-ch:
return err
case <-time.After(d):
return errors.New("foo has timed out")
}
}
If foo has timed out, then will foo ever be able to write to channel ch or is there a risk the goroutine blocks or panics?
What happens to channel ch once fooWithTimeout has exited?
Is this code potentially problematic?
Should I add defer close(ch) within go func(){...}() just before calling foo?
Does it matter that I use a buffered (with size 1) or an unbuffered channel in this example?
After the timer tick, fooWithTimeout will return. The goroutine will continue running until foo returns.
If foo times out, it will write to channel ch because it is buffered.
The channel ch will be garbage collected eventually if foo returns.
You don't need to close the channel. Once it is out of scope, it will be garbage collected.
A large burst of calls fooWithTimeout will create large amount of resources. Each call creates two goroutines. The proper way of timing this out is to change foo to use a context.
Building on https://stackoverflow.com/a/73611534/1079543, here is foo with a context:
package main
import (
"context"
"fmt"
"log"
"time"
)
func foo(ctx context.Context) (string, error) {
ch := make(chan string, 1)
go func() {
fmt.Println("Sleeping...")
time.Sleep(time.Second * 1)
fmt.Println("Wake up...")
ch <- "foo"
}()
select {
case <-ctx.Done():
return "", fmt.Errorf("context cancelled: %w", ctx.Err())
case result := <-ch:
return result, nil
}
}
func main() {
ctx, cancel := context.WithTimeout(context.Background(), time.Second*3)
defer cancel()
res, err := foo(ctx)
if err != nil {
log.Fatalf("foo failed: %v", err)
}
log.Printf("res: %s", res)
}
I built the following go code.
The idea is to build a done channel and a generator that generates a channel of int.
Link them in a 2 stage pipe
chanNumbers := pipeb(done, pipea(done, gen(done)))
After some seconds, cancel the done channel.
I expect to see the generator and two stage of the pipe to cancel and return, but the text "PipeX is terminating now" only appears randomly and I really do not understand why.
Would someone have an idea?
package main
import (
"fmt"
"time"
)
func gen(done <-chan interface{}) <-chan int {
ret := make(chan int)
cx := 0
go func() {
for {
select {
case <-done:
fmt.Println("**Generator Terminates now")
time.Sleep(2 * time.Second)
fmt.Println("**Generator has terminated now")
close(ret)
return
case ret <- cx:
fmt.Printf("Gen : we push %d \n", cx)
cx = cx + 1
}
}
}()
fmt.Println("Generator has created and returned its channel")
return ret
}
func pipea(done <-chan interface{}, in <-chan int) <-chan int {
ret := make(chan int)
go func() {
for {
select {
case <-done:
fmt.Println("**pipeA terminates")
time.Sleep(2 * time.Second)
fmt.Println("**pipeA has terminated now")
close(ret)
return
case tmp, ok := (<-in):
if ok {
fmt.Printf("pipeA : we push %d \n", tmp)
ret <- tmp
} else {
in = nil
}
}
}
}()
return ret
}
func pipeb(done <-chan interface{}, in <-chan int) <-chan int {
ret := make(chan int)
go func() {
for {
select {
case <-done:
fmt.Println("**pipeB terminates")
time.Sleep(2 * time.Second)
fmt.Println("**pipeB has terminated now")
close(ret)
return
case tmp, ok := (<-in):
if ok {
fmt.Printf("pipeB : we push %d \n", tmp)
ret <- tmp
} else {
in = nil
}
}
}
}()
return ret
}
func main() {
done := make(chan interface{})
chanNumbers := pipeb(done, pipea(done, gen(done)))
go func() {
time.Sleep(2 * time.Second)
close(done)
}()
forloop:
for {
select {
case n := <-chanNumbers:
fmt.Printf("Received in main element : %d\n", n)
case <-done:
break forloop
}
}
//end of the main program
fmt.Println("Sleeping some seconds before termiating")
time.Sleep(8 * time.Second)
fmt.Println("exit...")
}
You have four go-routines running:
gen, your generator, writing to an unbuffered output channel until done
pipeA, reading from gen, writing to an unbuffered output channel until done
pipeB, reading from pipeA, writing to an unbuffered output channel until done
main, reading from pipeB until done
Now when you close done, it totally depends on the order in which the go-routines see that.
If main is the first to see that done is closed, it will break the for-loop and stop consuming from pipeB. But if pipeB is still trying to write to the output channel (ret <- tmp), it will block there; so it will never get to the <- done part.
There are two options to fix this:
Only listen to done in your generator, and have the other go-routines use for n := range in { }.
Put your sending logic in a select as well so your generator and pipes can detect when done is closed.
Alternatively, you might want to use buffered output channels, but even then this problem can still occur.
I am trying to understand non buffered channels, so I have written a small application that iterates through an array of user input, does some work, places info on a non buffered channel and then reads it. However, I'm not able to read from the channels.
This is my code
toProcess := os.Args[1:]
var wg sync.WaitGroup
results := make(chan string)
errs := make(chan error)
for _, t := range toProcess {
wg.Add(1)
go Worker(t, "text", results, errs, &wg)
}
go func() {
for err := range errs {
if err != nil {
fmt.Println(err)
}
}
}()
go func() {
for res := range results {
fmt.Println(res)
}
}()
What am I not understanding about non buffered channels? I thought I should be placing information on it, and have another go routine reading from it.
EDIT: using two goroutines solves the issues, but it still gives me the following when there are errors:
open /Users/roosingh/go/src/github.com/nonbuff/files/22.txt: no such file or directory
fatal error: all goroutines are asleep - deadlock!
goroutine 1 [semacquire]:
sync.runtime_Semacquire(0xc42001416c)
/usr/local/Cellar/go/1.10.2/libexec/src/runtime/sema.go:56 +0x39
sync.(*WaitGroup).Wait(0xc420014160)
/usr/local/Cellar/go/1.10.2/libexec/src/sync/waitgroup.go:129 +0x72
main.main()
/Users/roosingh/go/src/github.com/nonbuff/main.go:39 +0x207
goroutine 6 [chan receive]:
main.main.func1(0xc4200780c0)
/Users/roosingh/go/src/github.com/nonbuff/main.go:25 +0x41
created by main.main
/Users/roosingh/go/src/github.com/nonbuff/main.go:24 +0x1d4
goroutine 7 [chan receive]:
main.main.func2(0xc420078060)
/Users/roosingh/go/src/github.com/nonbuff/main.go:34 +0xb2
created by main.main
/Users/roosingh/go/src/github.com/nonbuff/main.go:33 +0x1f6
So it is able to print out the error message.
My worker code is as follows;
func Worker(fn string, text string, results chan string, errs chan error, wg *sync.WaitGroup) {
file, err := os.Open(fn)
if err != nil {
errs <- err
return
}
defer func() {
file.Close()
wg.Done()
}()
reader := bufio.NewReader(file)
for {
var buffer bytes.Buffer
var l []byte
var isPrefix bool
for {
l, isPrefix, err = reader.ReadLine()
buffer.Write(l)
if !isPrefix {
break
}
if err != nil {
errs <- err
return
}
}
if err == io.EOF {
return
}
line := buffer.String()
results <- fmt. Sprintf("%s, %s", line, text)
}
if err != io.EOF {
errs <- err
return
}
return
}
As for unbuffered channels, you seem to understand the concept, meaning it's used to pass messages between goroutines but cannot hold any. Therefore, a write on an unbuffered channel will block until another goroutine is reading from the channel and a read from a channel will block until another goroutine writes to this channel.
In your case, you seem to want to read from 2 channels simultaneously in the same goroutine. Because the way channels work, you cannot range on a non closed channel and further down in the same goroutine range on another channel. Unless the first channel gets closed, you won't reach the second range.
But, it doesn't mean it's impossible! This is where the select statement comes in.
The select statement allows you to selectively read from multiple channels, meaning that it will read the first one that has something available to be read.
With that in mind, you can use the for combined with the select and rewrite your routine this way:
go func() {
for {
select {
case err := <- errs: // you got an error
fmt.Println(err)
case res := <- results: // you got a result
fmt.Println(res)
}
}
}()
Also, you don't need a waitgroup here, because you know how many workers you are starting, you could just count how many errors and results you get and stop when you reach the number of workers.
Example:
go func() {
var i int
for {
select {
case err := <- errs: // you got an error
fmt.Println(err)
i++
case res := <- results: // you got a result
fmt.Println(res)
i++
}
// all our workers are done
if i == len(toProcess) {
return
}
}
}()
I have the following sample code. I want to maintain 4 goroutines running at all times. They have the possibility of panicking. In the case of the panic, I have a recover where I restart the goroutine.
The way I implemented works but I am not sure whether its the correct and proper way to do this. Any thoughts
package main
import (
"fmt"
"time"
)
var gVar string
var pCount int
func pinger(c chan int) {
for i := 0; ; i++ {
fmt.Println("adding ", i)
c <- i
}
}
func printer(id int, c chan int) {
defer func() {
if err := recover(); err != nil {
fmt.Println("HERE", id)
fmt.Println(err)
pCount++
if pCount == 5 {
panic("TOO MANY PANICS")
} else {
go printer(id, c)
}
}
}()
for {
msg := <-c
fmt.Println(id, "- ping", msg, gVar)
if msg%5 == 0 {
panic("PANIC")
}
time.Sleep(time.Second * 1)
}
}
func main() {
var c chan int = make(chan int, 2)
gVar = "Preflight"
pCount = 0
go pinger(c)
go printer(1, c)
go printer(2, c)
go printer(3, c)
go printer(4, c)
var input string
fmt.Scanln(&input)
}
You can extract the recover logic in a function such as:
func recoverer(maxPanics, id int, f func()) {
defer func() {
if err := recover(); err != nil {
fmt.Println("HERE", id)
fmt.Println(err)
if maxPanics == 0 {
panic("TOO MANY PANICS")
} else {
go recoverer(maxPanics-1, id, f)
}
}
}()
f()
}
And then use it like:
go recoverer(5, 1, func() { printer(1, c) })
Like Zan Lynx's answer, I'd like to share another way to do it (although it's pretty much similar to OP's way.) I used an additional buffered channel ch. When a goroutine panics, the recovery function inside the goroutine send its identity i to ch. In for loop at the bottom of main(), it detects which goroutine's in panic and whether to restart by receiving values from ch.
Run in Go Playground
package main
import (
"fmt"
"time"
)
func main() {
var pCount int
ch := make(chan int, 5)
f := func(i int) {
defer func() {
if err := recover(); err != nil {
ch <- i
}
}()
fmt.Printf("goroutine f(%v) started\n", i)
time.Sleep(1000 * time.Millisecond)
panic("goroutine in panic")
}
go f(1)
go f(2)
go f(3)
go f(4)
for {
i := <-ch
pCount++
if pCount >= 5 {
fmt.Println("Too many panics")
break
}
fmt.Printf("Detected goroutine f(%v) panic, will restart\n", i)
f(i)
}
}
Oh, I am not saying that the following is more correct than your way. It is just another way to do it.
Create another function, call it printerRecover or something like it, and do your defer / recover in there. Then in printer just loop on calling printerRecover. Add in function return values to check if you need the goroutine to exit for some reason.
The way you implemented is correct. Just for me the approach to maintain exactly 4 routines running at all times looks not much go_way, either handling routine's ID, either spawning in defer which may leads unpredictable stack due to closure. I don't think you can efficiently balance resource this way. Why don't you like to simple spawn worker when it needed
func main() {
...
go func(tasks chan int){ //multiplexer
for {
task = <-tasks //when needed
go printer(task) //just spawns handler
}
}(ch)
...
}
and let runtime do its job? This way things are done in stdlib listeners/servers and them known to be efficient enough. goroutines are very lightweight to spawn and runtime is quite smart to balance load. Sure you must to recover either way. It is my very personal opinion.