package main
import (
"fmt"
"sync"
)
func main(){
ch1 := make(chan int,100)
ct := 0
var wg sync.WaitGroup
wg.Add(1)
go func(){
//defer close(ch1)
for i:= 0; i < 10;i ++{
ch1 <- i
}
}()
go func(){
defer wg.Done()
for x := range ch1{
fmt.Println(x)
}
}()
wg.Wait()
fmt.Println("numbers:",ct)
}
Why this code will return
fatal error: all goroutines are asleep - deadlock!
I've found if I closed the channel there will be no deadlock, but I don't know why it's like that.
Do I have to close the channel after I inputted all items into the channel?
for range over a channel only terminates if / when the channel is closed. If you don't close the channel and you don't send more values on it, the for range statement will block forever, and so will the main goroutine at wg.Wait().
The "sender" party should close the channel once it sent all values, signalling the "receiver" party that no more values will come on the channel.
So yes, you should close the channel:
go func() {
defer close(ch1)
for i := 0; i < 10; i++ {
ch1 <- i
}
}()
The for loop that reads from the channel will continue reading unless you close the channel. That is the reason for the deadlock, because the goroutine that reads from the channel is the only active goroutine, and there is no other goroutines that can write to it. When you close, the for loop terminates.
range unclosed ch1 will causing block;
even close ch1, you can still receive data from ch1;
It can be said, close(ch1) will send a special message to ch1, which can be used to notify the ch1 receiver that no more data will be received. So even if there is data in the ch1, you can close() without causing the receiver to not receive the remaining data.
The channel does not need to release resources through close. As long as there is no goroutine holding the channel, the related resources will be automatically released.
uncomment defer close(ch1) will solve this problems.
here
Related
This question already has answers here:
Buffered/Unbuffered channel
(3 answers)
fatal error: all goroutines are asleep - deadlock! when channel is not buffered
(1 answer)
Why is the order of channels receiving causing/resolving a deadlock in Golang?
(3 answers)
In the Go select construct, can I have send and receive to unbuffered channel in two case statements?
(2 answers)
Closed 7 days ago.
package main
import (
"fmt"
"sync"
)
func main() {
ch := make(chan struct{})
wg := &sync.WaitGroup{}
wg.Add(1)
go func() {
defer wg.Done()
ch <- struct{}{}
}()
wg.Wait()
<-ch
fmt.Println("finished")
}
result
fnished will not be printed
package main
import (
"fmt"
"sync"
)
func main() {
ch := make(chan struct{}, 1)
wg := &sync.WaitGroup{}
wg.Add(1)
go func() {
defer wg.Done()
ch <- struct{}{}
}()
wg.Wait()
<-ch
fmt.Println("finished")
}
result
finished can be printed immediately
The only difference between the two code is that the first code use no buffer channel, but the second code use channel with a buffer.
In the first code, ch <- struct{}{} can not be run. message can not be sent to the channel.
Why does that happen? Is there any official explanation or material?
In the first example wg.Done() can only be called after a value on the channel has been sent. But since the channel is unbuffered, the send can only happen if there's a ready receiver. The main goroutine would be the receiver, but it only receives after wg.Wait(). But wg.Wait() blocks until wg.Done() is called, so this will never happen, it's a deadlock.
This is in Spec: Send statements:
Communication blocks until the send can proceed. A send on an unbuffered channel can proceed if a receiver is ready.
When using non-buffered channels, sending or receiving commands from the channel are blocking.
This means with non-buffered channel, your go routine that sends the message is not done, hence it won't call wg.Done() which will keep the wg.Wait() method in blocking state.
If you run your code on the play server https://go.dev/play/p/oDpyL6-dARP, it will panic with an informative message: fatal error: all goroutines are asleep - deadlock!
When your using buffered channels, the go routine runs because the channel has a buffer, then wg.Done() is called, which release wg.Wait() afterwards.
How do I wait for all go routines to finish and then read all the data from a channel?
Why is this example stuck waiting for the go routines to finish?
Go Playground
package main
import (
"fmt"
"sync"
"time"
)
func doStuff(i int, wg *sync.WaitGroup, messages chan<- string) {
defer wg.Done()
time.Sleep(time.Duration(i) * time.Second)
messages <- fmt.Sprintf("Doing stuff...%d", i)
}
func doMoreStuff(i int, wg *sync.WaitGroup, messages chan<- string) {
defer wg.Done()
time.Sleep(time.Duration(i) * time.Second)
messages <- fmt.Sprintf("Doing more stuff...%d", i)
}
func main() {
var wg sync.WaitGroup
var messages = make(chan string)
for start := time.Now();; {
elapsedTime := time.Since(start)
fmt.Println(elapsedTime)
if elapsedTime > time.Duration(3) * time.Second {
fmt.Println("BREAK")
break
}
for i := 0; i < 10; i++ {
wg.Add(1)
go doStuff(i, &wg, messages)
wg.Add(1)
go doMoreStuff(i, &wg, messages)
}
time.Sleep(time.Duration(1) * time.Second)
}
fmt.Println("WAITING")
wg.Wait()
fmt.Println("DONE")
for message := range messages {
fmt.Println(message)
}
}
If you want to wait for all goroutines to finish that send messages on a channel, and you want to start reading the messages after that, then you have no choice but to use a buffered channel that can "host" all the messages that can be sent on it by the goroutines.
This isn't something practical. And even if you'd go down this path, you would be able to receive and print the messages, but the for range loop doing so would never terminate because that only terminates when it receives all messages that were sent on the channel before it was closed, but you never close the channel. You may check this "half-working" solution on the Go Playground.
Instead launch a goroutine that waits for others to finish, and then close the channel:
go func() {
fmt.Println("WAITING")
wg.Wait()
close(messages)
}()
So now you may use for range to receive messages in the main goroutine:
for message := range messages {
fmt.Println(message)
}
fmt.Println("DONE")
Try this one on the Go Playground.
This solution is still not perfect: it first has to launch all goroutines, and only then it tries to receive values, and all the launched goroutines will be blocked on their send operation (until main is ready to receive those).
A better solution may be to launch a goroutine that receives the values, preferably before the goroutines that send messages on the channel (else they would be blocked on their sends anyway):
go func() {
for message := range messages {
fmt.Println(message)
}
}()
And wait for the goroutines and close the channel at the end of main():
fmt.Println("WAITING")
wg.Wait()
close(messages)
The problem with this is that when main() ends, so does your app, it does not wait for other non-main goroutines to finish. And this means it will not wait for the "consumer" goroutine to receive the messages.
To wait for that "consumer", you may use an additional sync.WaitGroup:
var wg2 sync.WaitGroup
wg2.Add(1)
go func() {
defer wg2.Done()
for message := range messages {
fmt.Println(message)
}
}()
// And the end of `main()`:
fmt.Println("WAITING")
wg.Wait()
close(messages)
fmt.Println("DONE")
wg2.Wait()
Try this one on the Go Playground.
The example is stuck because you are using an unbuffered channel. Both go-routines are blocked waiting to write to the channel since nothing is ready to read from it.
You can use a buffered channel but then you would just be using it as a data storage. Channels are more useful for communication. The question is why do you want to wait until all the writers are finished? In the general case of an unknown (unlimited) number of writers you would no know how big to make your channel.
This is homework and beginner question. I edited the question since I made a mistake that I found.
I am trying to do parallel frequency maps from a text and I get an error in the last operation (reducing maps)
The code seem to work up to this point.
If I close the channel I get error:
"panic: send on closed channel"
If I do not close the channel I get:
"fatal error: all goroutines are asleep - deadlock!"
func WordCount(text string) {
text = strings.ToLower(text)
re := regexp.MustCompile("\\w+")
sentence := re.FindAllString(text, -1)
numberOfGroups := 4
piece := len(sentence) / numberOfGroups
wordChannel := make(chan map[string]int)
wg := new(sync.WaitGroup)
wg.Add(numberOfGroups)
for i := 0; i < numberOfGroups; i ++ {
go processToCounting(sentence[i*piece:(i+1)*piece], wordChannel, wg)
}
wg.Wait()
fmt.Print(<-wordChannel)
fmt.Print("\n")
finalMap := make(map[string]int)
close(wordChannel)
for i := 0; i < numberOfGroups; i++ {
for k, v := range <- wordChannel {
finalMap[k] += v
}
}
}
func processToCounting(textSlice []string, wordChannel chan map[string]int, wg *sync.WaitGroup) {
freq := make(map[string]int)
for _, v := range textSlice {
freq[v]++
}
wg.Done()
wordChannel <- freq
}
1. First question: panic
If I close the channel I get error: "panic: send on closed channel"
Why? One of you goroutines is trying to write to the channel which you already closed in the calling (main) goroutine. In your case in WordCount function.
In the current version of your code the panic is not reproducible with my test sentence, but you can easily cause this e.g. if you would call close(wordChannel) before wg.Wait().
Lets look at the Bug in processToCounting which may cause the panic:
wg.Done() // tells to the WaitGroup that the gouroutine is Done (decrements the counter of goroutines)
wordChannel <- freq // trying to write to the channel
Here wg.Done() signals to the WaitGroup that the goroutine is Done before actual writing to the channel has happened. The calling goroutine (WordCount function) at some point thinks that all gouroutines are done (wg.Wait() line) and closes the channel. But one of your goroutine which have not finished writing, will try to write to the closed channel. Then you will get the panic.
How to fix:
Use defer in processToCounting function
defer wg.Done() // triggers wg.Done right before the function returns (but after writing to the channel in your case)
What to read:
See for beginners in A Tour of Go / Concurrency
Sending on a closed channel will cause a panic.
and documentation for: close
Sending to or closing a closed channel causes a run-time panic.
2. Second question: deadlock
If I do not close the channel I get: "fatal error: all goroutines are asleep - deadlock!"
You have a for-loop which reads from the channel. This for-loop is locked forever. Waiting for new values from the channel, but nobody will write there anymore.
See in A Tour of Go / Concurrency
The loop for i := range c receives values from the channel repeatedly until it is closed.
and see documentation for Channels
Receivers always block until there is data to receive
Q1: why close(wordChannel) triggers a panic
A:
you put wg.Done() before pushing result into the channel
the channel is a non-buffer channel, which means that it will be stuck to write until there is a read. but the read happens after you close channel. the sequence is close channel -> read channel -> write channel -> panic
Q2: why there is a deadlock.
A: fmt.Print(<-wordChannel) read a msg from the channel, so that the last loop can't read numberOfGroups message. it waits for the last message forever.
for i := 0; i < numberOfGroups; i++ {
for k, v := range <- wordChannel {
finalMap[k] += v
}
}
I have issue when using Go routine with channel. The code looks like this:
func main() {
c := make(chan int)
var wg sync.WaitGroup
wg.Add(1)
go func (c chan int, x int) {
c <- x
fmt.Println(x)
close(c)
defer wg.Done()
}(c,10)
wg.Wait()
}
When run the code I got this error:
fatal error: all goroutines are asleep - deadlock!
I cannot understand why this issue happens. Please help me to understand
You have 2 goroutines in your example: the main goroutine running the main() function, and the other you launch inside it. The main goroutine waits for the other to complete (to call wg.Done()), and the other goroutine blocks in the line where it attempts to send a value on channel c. Since nobody is receiving from that channel, and because that channel is unbuffered, this goroutine will never advance, so all your 2 goroutines will block forever.
Note that defer wg.Done() should be the first statement in the goroutine. If it's the last, defer won't make any difference.
If the channel would have a buffer of at least one, the send operation could proceed:
c := make(chan int, 1)
And output will be (try it on the Go Playground):
10
If we leave the channel unbuffered, there must be another goroutine that receives from the channel, e.g.:
wg.Add(1)
go func() {
defer wg.Done()
x := <-c
fmt.Println("Received:", x)
}()
Then output will be (try it on the Go Playground):
10
Received: 10
I'm not able to close channel when there is no knowledge about its
length
package main
import (
"fmt"
"time"
)
func gen(ch chan int) {
var i int
for {
time.Sleep(time.Millisecond * 10)
ch <- i
i++
// when no more data (e.g. from db, or event stream)
if i > 100 {
break
}
}
// hot to close it properly?
close(ch)
}
func receiver(ch chan int) {
for i := range ch {
fmt.Println("received:", i)
}
}
func main() {
ch := make(chan int)
for i := 0; i < 10; i++ {
go gen(ch)
}
receiver(ch)
}
It gives me error
panic: send on closed channel
goroutine 8 [running]:
main.gen(0xc82001a0c0)
/home/exu/src/github.com/exu/go-workshops/100-concurrency-channels/16-close-problem.go:12 +0x57
created by main.main
/home/exu/src/github.com/exu/go-workshops/100-concurrency-channels/16-close-problem.go:35 +0xbd
goroutine 1 [panicwait]:
runtime.gopark(0x0, 0x0, 0x50b8e0, 0x9, 0x10, 0x1)
/usr/lib/go/src/runtime/proc.go:185 +0x163
runtime.main()
/usr/lib/go/src/runtime/proc.go:121 +0x2f4
runtime.goexit()
/usr/lib/go/src/runtime/asm_amd64.s:1696 +0x1
goroutine 6 [sleep]:
time.Sleep(0x989680)
/usr/lib/go/src/runtime/time.go:59 +0xf9
main.gen(0xc82001a0c0)
/home/exu/src/github.com/exu/go-workshops/100-concurrency-channels/16-close-problem.go:11 +0x29
created by main.main
/home/exu/src/github.com/exu/go-workshops/100-concurrency-channels/16-close-problem.go:33 +0x79
goroutine 7 [sleep]:
time.Sleep(0x989680)
/usr/lib/go/src/runtime/time.go:59 +0xf9
main.gen(0xc82001a0c0)
/home/exu/src/github.com/exu/go-workshops/100-concurrency-channels/16-close-problem.go:11 +0x29
created by main.main
/home/exu/src/github.com/exu/go-workshops/100-concurrency-channels/16-close-problem.go:34 +0x9b
exit status 2
It's logical - first goroutine closing channel when the second one tries to send to it. What will be the best approach to close channel in this situation?
Once a channel is closed, you can't send further values on it else it panics. This is what you experience.
This is because you start multiple goroutines that use the same channel and they send values on it. And you close the channel in each of it. And since they are not synchronized, once the first goroutine reaches the point where it closes it, others may (and they will) still continue to send values on it: panic!
You can close the channel only once (attempting to close an already closed channel also panics). And you should do it when all the goroutines that send values on it are done. In order to do this, you need to detect when all the sender goroutines are done. An idiomatic way to detect this is to use sync.WaitGroup.
For each started sender goroutine we add 1 to the WaitGroup using WaitGroup.Add(). And each goroutine that is done sending the values can signal this by calling WaitGroup.Done(). Best to do this as a deferred statement, so if your goroutine would terminate abruptly (e.g. panics), WaitGroup.Done() would still be called, and would not leave other goroutines hanging (waiting for an absolution - a "missing" WaitGroup.Done() call that would never come...).
And WaitGroup.Wait() will wait until all sender goroutines are done, and only after this and only once will it close the channel. We want to detect this "global" done event and close the channel while processing the values sent on it is in progress, so we have to do this in its own goroutine.
The receiver goroutine will run until the channel is closed since we used the for ... range construct on the channel. And since it runs in the main goroutine, the program will not exit until all the values are properly received and processed from the channel. The for ... range construct loops until all the values are received that were sent before the channel was closed.
Note that the solution below works with buffered and unbuffered channel too without modification (try using a buffered channel with ch := make(chan int, 100)).
Correct solution (try it on the Go Playground):
func gen(ch chan int, wg *sync.WaitGroup) {
defer wg.Done()
var i int
for {
time.Sleep(time.Millisecond * 10)
ch <- i
i++
// when no more data (e.g. from db, or event stream)
if i > 100 {
break
}
}
}
func receiver(ch chan int) {
for i := range ch {
fmt.Println("received:", i)
}
}
func main() {
ch := make(chan int)
wg := &sync.WaitGroup{}
for i := 0; i < 10; i++ {
wg.Add(1)
go gen(ch, wg)
}
go func() {
wg.Wait()
close(ch)
}()
receiver(ch)
}
Note:
Note that it's important that receiver(ch) runs in the main goroutine, and the code what waits for the WaitGroup and closes the channel in its own (non-main) goroutine; and not the other way around. If you would switch these 2, it might cause an "early exit", that is not all values might be received and processed from the channel. The reason for this is because a Go program exits when the main goroutine finishes (spec: Program execution). It does not wait for other (non-main) goroutines to finish. So if waiting and closing the channel would be in the main goroutine, after closing the channel the program could exit at any moment, not waiting for the other goroutine that in this case would loop to receive values from the channel.
"One general principle of using Go channels is don't close a channel from the receiver side and don't close a channel if the channel has multiple concurrent senders."
Every channel will be GCed eventually once it is marked for cleanup, so it is okay to leave channel un-closed the only difference it will make is that that channel will be available for gc after a few cycles maybe if not closed explicitly.
Nevertheless it is always good if you can close the channel off. Please go through the following links for detailed explaination.
Articles this and this shows various ways to close a channel in case of 1:N, N:1 or M:N (senders:receivers)