If I have multiple goroutines reading and writing to the same channel, are there any ways to close the channel at certain point, for example after a fix amount of elements are written.
func foo(c chan int) {
for i := range c {
// how to close the channel after 100 integers are written?
c <- i + 1
c <- i + 2
}
}
func bar() {
c := make(chan int, 200)
c <- 0
go foo(c)
go foo(c)
go foo(c)
}
Give conditional case would work for it:
for i := 0; i <= cap(c); i++ {
// how to close the channel after 100 integers are written?
c <- i + 1
// since you send 2 values to channel for each i loop, you need to half of it.
if i == 49 { //if you asking why 49, you have sent one value on bar function. Also to make it perfectly 100 integer in total.
close(ch)
break // terminating the function after the closing channel, it will run in panic because the function still running and keep sending value on channel which was already closed.
}
c <- i + 2
}
Output :
0
1
2
2
3
3
...
50
50
Just give some information about channel, don't use range on channel if you havent sent anything on it...
If you put "i range channel" before you have value on channel, it will not read anything.
Back to again to your code
If you run this code :
func bar() {
c := make(chan int, 200)
c <- 0
go foo(c)
go foo(c) // run in panic
go foo(c) // run in panic
}
It will run in panic, since the channel already closed which mean your channel not accepting any values again.
Take a look at this picture, let say you make a buffered channel with 3 capacity. And you going to close after receiving 2 values to send it will run in panic if you try to send values on it again.
Related
I am new to Golang concurrency and have been working to understand this piece of code mentioned below.
I witness few things which I am unable to explain why it happens:
when using i smaller than equal to 100000 for i <= 100000 { in main function, it sometimes prints different values for nResults and countWrites (in last two statements)
fmt.Printf("number of result writes %d\n", nResults) fmt.Printf("Number of job writes %d\n", jobWrites)
when using i more than 1000000 it gives panic: send on closed channel
How can I make sure that the values send to jobs is not on closed channel and later after all values are received in results we can close the channel without deadlock?
package main
import (
"fmt"
"sync"
)
func worker(wg *sync.WaitGroup, id int, jobs <-chan int, results chan<- int, countWrites *int64) {
defer wg.Done()
for j := range jobs {
*countWrites += 1
go func(j int) {
if j%2 == 0 {
results <- j * 2
} else {
results <- j
}
}(j)
}
}
func main() {
wg := &sync.WaitGroup{}
jobs := make(chan int)
results := make(chan int)
var i int = 1
var jobWrites int64 = 0
for i <= 10000000 {
go func(j int) {
if j%2 == 0 {
i += 99
j += 99
}
jobWrites += 1
jobs <- j
}(i)
i += 1
}
var nResults int64 = 0
for w := 1; w < 1000; w++ {
wg.Add(1)
go worker(wg, w, jobs, results, &nResults)
}
close(jobs)
wg.Wait()
var sum int32 = 0
var count int64 = 0
for r := range results {
count += 1
sum += int32(r)
if count == nResults {
close(results)
}
}
fmt.Println(sum)
fmt.Printf("number of result writes %d\n", nResults)
fmt.Printf("Number of job writes %d\n", jobWrites)
}
Quite a few problems in your code.
Sending on closed channel
One general principle of using Go channels is
don't close a channel from the receiver side and don't close a channel if the channel has multiple concurrent senders
(https://go101.org/article/channel-closing.html)
The solution for you is simple: don't have multiple concurrent senders, and then you can close the channel from the sender side.
Instead of starting millions of separate goroutine for each job you add to the channel, run one goroutine that executes the whole loop to add all jobs to the channel. And close the channel after the loop. The workers will consume the channel as fast as they can.
Data races by modifying shared variables in multiple goroutines
You're modifying two shared variables without taking special steps:
nResults, which you pass to the countWrites *int64 in the worker.
i in the loop that writes to the jobs channel: you're adding 99 to it from multiple goroutines, making it unpredictable how many values you actually write to the jobs channel
To solve 1, there are many options, including using sync.Mutex. However since you're just adding to it, the easiest solution is to use atomic.AddInt64(countWrites, 1) instead of *countWrites += 1
To solve 2, don't use one goroutine per write to the channel, but one goroutine for the entire loop (see above)
I just started learning golang, while I was going through concurrency I accidentally wrote this code:
import (
"fmt"
)
func squares(c chan int) {
for i := 0; i < 4; i++ {
num := <- c
fmt.Println(num * num)
}
}
func main() {
fmt.Println("main start")
c := make(chan int)
go squares(c)
c <- 1
c <- 2
c <- 3
c <- 4
go squares(c)
c <- 5
c <- 6
c <- 7
c <- 8
fmt.Println("main stop")
}
Originally I was suppose to assign c := make(chan int, 3).
I am having trouble understanding the output of the code I've written.
main start
1
4
9
25
36
49
16
main stop
I would like to understand how the code is executed.
I was expecting error: all goroutines are asleep - deadlock!
Many thanks!
I am not sure to really understand what you wanted to achieve, especially the reason of that weird loop :
for i := 0; i < 4; i++ {
num := <- c
fmt.Println(num * num)
}
But anyway.
First have all, some explanations on how works channels and goroutines.
A Channel is a thread safe messaging pipe used to share data accross differents executions contexts. A channel is creating with make instruction, then
c := make(chan int, 3)
means create a channel of int type with a "buffer" size of 3. This element is very important to understand. Channel do follow a producer / consumer pattern with that bases rules :
For producer :
if I try to push some data in a channel with some "free space", it doesn't block and next instructions are executed
If I try to push some data in a channel without "free space", it blocks till previous has been treated
For consumer :
If I try to take element from an empty channel, it block untill some data come
If I try to take element from a non empty channel, it take the first (Channel is a FIFO)
Please note that all blocking operations may be turned non-blocking using some special pattern available through select instruction, but that's another subject :).
Goroutine are light sub-processes, routines (No thread). It is no place here to explain all in details, but what is important is 1 goroutine === 1 execution stack.
So in your case, the output is quite predictable till there is only one consumer. Once you start the second Goroutine, the consumption order is still predictable (the size of the channel is only one), but the execution order not !
One thing is noticable: you have only 7 output... That because your main goroutine ends before it happends, terminating the whole program execution. That point explain why you didn't have the all goroutines are asleep - deadlock!.
If you want to have that, you just should add <- c somewhere at the end of the main :
package main
import (
"fmt"
)
func squares(c chan int) {
for i := 0; i < 4; i++ {
num := <-c
fmt.Println(num * num)
}
}
func main() {
fmt.Println("main start")
c := make(chan int)
go squares(c)
c <- 1
c <- 2
c <- 3
c <- 4
go squares(c)
c <- 5
c <- 6
c <- 7
c <- 8
<-c
fmt.Println("main stop")
}
You will have the behavior I think you expect :
main start
1
4
9
25
36
49
64
16
fatal error: all goroutines are asleep - deadlock!
Edit : on step by step, execution :
// a goroutine is created and c is empty. because
// the code of `squares` act as a consumer, the goroutine
// "block" at instruction `num := <-c`, but not the main goroutine
go squares(c)
// 1 is pushed to the channel. Till "c" is not full the main goroutine
// doesn't block but the other goroutine may be "released"
c <- 1
// if goroutine of squares has not consume 1 yet, main goroutine block
// untill so, but is released just after
c <- 2
// it continues with same logic
c <- 3
c <- 4
// till main goroutine encountered `<- c` (instruction I added) .
// Here, it behave as a consumer of "c". At this point all
// goroutine are waiting as consuler on "c" => deadlock
I am learning basic Go via a Udemy course. In the goroutines section, there is an example of throttling which has thrown my understanding of how wait groups work.
package main
import (
"fmt"
"math/rand"
"sync"
"time"
)
func main() {
c1 := make(chan int)
c2 := make(chan int)
go populate(c1)
go fanOutIn(c1, c2)
for v := range c2 {
fmt.Println(v)
}
fmt.Println("about to exit")
}
func populate(c chan int) {
for i := 0; i < 100; i++ {
c <- i
}
close(c)
}
func fanOutIn(c1, c2 chan int) {
var wg sync.WaitGroup
const goroutines = 10
wg.Add(goroutines)
for i := 0; i < goroutines; i++ {
go func() {
for v := range c1 {
func(v2 int) {
c2 <- timeConsumingWork(v2)
}(v)
}
wg.Done()
}()
}
wg.Wait()
close(c2)
}
func timeConsumingWork(n int) int {
time.Sleep(time.Microsecond * time.Duration(rand.Intn(500)))
return n + rand.Intn(1000)
}
The part that isn't inline with my understanding is in the function fanOutIn where the we set up the WaitGroup, and Add(10).
Why am I getting 100 values printed out? Only a single value (i := 0) can be put onto c1, and the value is never explicitly removed from the channel. The code then hits wg.Done(), and the wait group queue is reduced to 9 and so on.
In my current understanding, I would expect to see 10 values, of 0 + rand.Intn(1000).
The function that is spun off reads as follows (including the go at the front and the parentheses to call it):
go func() {
for v := range c1 {
func(v2 int) {
c2 <- timeConsumingWork(v2)
}(v)
}
wg.Done()
}()
This code is a little weird and bizarre. Let's shrink it down even further, discarding the wg.Done and keeping only the for loop itself:
for v := range c1 {
func(v2 int) {
c2 <- timeConsumingWork(v2)
}(v)
}
There is an inner unnamed function that is pretty useless here; we can discard it without changing the behavior of the program, to get:
for v := range c1 {
c2 <- timeConsumingWork(v)
}
which is at last a simple loop. One key question is now this: How many iterations do you expect from this loop? Note: It is not necessarily any constant number. Perhaps a better way to phrase the question is: When does this loop end?
The for loop reads a channel. This kind of loop ends when a read from the channel indicates that there is no more data, i.e., that the channel is closed and its queue is empty. (See the Go specification section on for loops.)
So this innermost loop, for v := range c1, is not going to terminate until channel c1 is closed and there is no more data in its queue. This channel was created with:
c1 := make(chan int)
so it has no queue, so we need not even think about that: it terminates after a close(c1) closes it. You should now look for a close that closes c1.
Where is our close?
Here's the place that closes c1:
func populate(c chan int) {
for i := 0; i < 100; i++ {
c <- i
}
close(c)
}
We call this with c1 as its argument, so its final close(c) closes c1. Now you can ask: When do we reach this close call? The answer is obvious: after i >= 100 in the loop, i.e., after we have sent 100 values, zero through 99 respectively, into channel c1.
What fanOutIn does is spin off 10 goroutines. Each of the 10 goroutines runs the first anonymous function I quoted above. That anonymous function has a loop that runs an indeterminate number of times, repeating until channel c1 is closed. Each trip through the loop takes a value of the channel, so initially, if the ten goroutines all manage to start before there are any values available, all ten goroutines will be waiting for values.
When the producer function puts one value into the channel, one of the ten waiting goroutines will get it and begin using it. If that goroutine takes a long time to get back to the top of its own for loop, another goroutine will take the next-produced value. So what happens here is that up to ten produced values propagate through the channel to up to ten goroutines.1 Each of those (up to ten) goroutines spends some nontrivial amount of time using its value, then sends a final-product-value to channel c2 and goes back to the top of its own indefinite for loop.
Only when the producer has closed its channel c (which is our c1 here) will the ten goroutines see a closed-channel-empty-queue, allowing them to exit their for loops. When they do exit their for loops, each of them will call wg.Done() (once each) and terminate.
So, once the close(c1) has occurred (via the close(c) in populate), eventually all ten of these anonymous goroutines will have called wg.Done(). At that point, the wg.Wait() in fanOutIn will return. This will call close(c2) and return from fanOutIn, terminating that goroutine as well.
Meanwhile, in main, we use for v := range c2 to read from channel c2. This for loop will run when values are written into c2 by any of the ten goroutines. It will exit only when c2 itself is closed (its queue must also be empty but again c2 has a zero-length queue). So main will not proceed past the for loop until c2 is closed, which cannot happen until wg.Wait() returns, which cannot happen until ten wg.Done() calls have happened, which cannot happen until channel c1 is closed.
This means that main cannot get past its own for loop until populate has called close(c), and that happens only after generating exactly 100 values.
1As discussed in comments below, the phrase up to here can be important: we don't really know how many goroutines will really consume values. A lot depends on how much work each goroutine does, what kind of work that is, and how many CPUs your Go runtime has available.
The following code keeps printing 0.
package main
import (
"fmt"
)
func main() {
c := make(chan int)
go func() {
for i := 0; i < 10; i++ {
c <- i // INPUT
}
close(c)
}()
for {
fmt.Print(<-c) // RECEIVER; OUTPUT
}
}
From my understanding, it should print 0 to 9 and then go on an infinite loop. Why does it keep printing zero?
Your understanding is (almost) correct, except that the infinite loop in the main goroutine will not block but will receive and print relentlessly.
The goroutine you start sends the numbers 0, 1, ... 9 on the channel, then closes it. And receiving from a closed channel does not block, on the contrary, it can proceed immediately, and it yields the zero value of the element type of the channel, which is 0 for the type int. This is stated in Spec: Receive operator:
Receiving from a nil channel blocks forever. A receive operation on a closed channel can always proceed immediately, yielding the element type's zero value after any previously sent values have been received.
So you see exactly what you should. First it prints the numbers 0..9, then keeps printing 0 very fast (without any delay), so probably you don't even notice the initial 0..9 numbers.
Slightly modifying your example so that the loop in the main goroutine exits after 15 iterations will immediately show what happens:
c := make(chan int)
go func() {
for i := 0; i < 10; i++ {
c <- i // INPUT
}
close(c)
}()
for i := 0; i < 15; i++ {
fmt.Print(<-c) // RECEIVER; OUTPUT
}
Output (try it on the Go Playground):
012345678900000
If your goal is to quit once all sent numbers have been processed (printed), use the for range on the channel:
for i := range c {
fmt.Print(i) // RECEIVER; OUTPUT
}
If your goal is to block until new numbers are available once those 10 numbers are processed, then don't close the channel. That way the next receive operation will block. In this case both your original loop and the for range loop would work, but for range is better as that always exits if / when the channel is closed.
Check out this question and memorize the channel axioms: How does a non initialized channel behave?
I am stuck in a strange situation where the write operation to the channel never happens.
package main
import (
"fmt"
"time"
)
func main() {
c := make(chan int)
s := make(chan bool)
k := make(chan bool)
fmt.Println("start")
go func() {
fmt.Println("routine 1")
s <- true
}()
go func() {
fmt.Println("routine 2")
for {
select {
case <-s :
for i := 0; i < 3; i++ {
fmt.Println("before")
c <- i
fmt.Println("after")
}
case <- k :
fmt.Println("k ready")
break
}
}
}()
go func() {
fmt.Println("routine 3")
for {
select {
case x := <- c :
fmt.Println("x=", x)
k <- true
fmt.Println("k done")
}
}
}()
time.Sleep(1000 * time.Millisecond)
}
And here is the output:
start
routine 1
routine 2
before
routine 3
x= 0
after
before
I wonder why the write to the channel k blocks, but the log statement fmt.Println("k ready") is never printed.
Here is what I think :
go routine 1 writes true to channel s
go routine 2 writes 0 to
channel c and waits because buffer size is 0, it will not be able to
write '1' to it unless someone reads channel c
go routine 3 comes into the picture, reads channel c (now go routine 2
can write to c once go routine 2 resumes) prints the value of x. NOW IT SHOULD BE ABLE TO WRITE TO CHANNEL K but that is not happening
According to me it should be able to write to channel k then case 2 of goroutine should execute and print "k ready"
Can anyone explain me why write to the channel blocked?
As a fix I know I can increase the buffer size of channel c and everything will get printed but I am not interested in fixing this, instead I want to understand this scenario.
A nice blog to understand the above case.
You have a deadlock.
goroutine 1 writes to s then quits
goroutine 2 reads from s, and writes to c
goroutine 3 reads from c, and writes to k, and this blocks because nothing is reading from k, because goroutine 2 is blocked in the write to k above.
goroutine 2 writes to c again which blocks as goroutine 3 is still trying to write to k and thus is not reading from c
Contrary to what you say, you don't have a buffer size of 1. You have a buffer size of zero (i.e. an unbuffered channel), so a write will block until something reads. This is probably the source of your misunderstanding. Per the language specification:
A new, initialized channel value can be made using the built-in function make, which takes the channel type and an optional capacity as arguments:
make(chan int, 100)
The capacity, in number of elements, sets the size of the buffer in the channel. If the capacity is zero or absent, the channel is unbuffered and communication succeeds only when both a sender and receiver are ready. Otherwise, the channel is buffered and communication succeeds without blocking if the buffer is not full (sends) or not empty (receives). A nil channel is never ready for communication.