When reading from channels in Go, I observed that it does not follow perfect synchronization between the publishing function and the consuming function. Strangely enough, if it was a quirk with the CPU scheduling, I would have got different results some of the time. The consumer in main seems to consume in multiples two at a time and print them.
Consider the following example:
package main
import (
"fmt"
)
func squares(ch chan int) {
for i:=0; i<9; i++ {
val:= i*i;
fmt.Printf("created val %v \n", val);
ch <- i*i;
fmt.Printf("After posting val %v \n", val);
}
close(ch)
}
func main() {
c := make(chan int)
go squares(c)
for val := range c{
fmt.Println(val);
}
}
No matter how many times I run it on Go Playground, I see the following output.
created val 0
After posting val 0
created val 1
0
1
After posting val 1
created val 4
After posting val 4
created val 9
4
9
After posting val 9
created val 16
After posting val 16
created val 25
16
25
After posting val 25
created val 36
After posting val 36
created val 49
36
49
After posting val 49
created val 64
After posting val 64
64
Shouldn't I be expecting because go would block the squares method until main has printed it?
created val 0
0
After posting val 0
...
If not then why? If I want perfect synchronization like the above, what should be my way?
You are using an unbuffered channel, so this is what happens.
NOTE: This is not meant as a technically 100% accurate description.
main routine starts. It runs until for val := range c {. Then it is put to "sleep" as there is no value to be read from c.
goroutine for squares which was just created is being "awoken". It creates a value and can post it as the other goroutine is already "waiting" for a value on the channel. Then it creates another value and can't post it as the channel now blocks.
main routine is "awoken", reads the value, prints it, reads the next value which the other goroutine is already waiting to push. prints it and then is stuck again as there is no new value available.
goroutine for sqares is "awoken" as it was able to push its value. It prints "After posting", creates a new value, posts it as the other routine is already waiting for one, and creates another one. Then it gets stuck as the other routine is not ready to receive the next value.
back to 3)
If you want a "smoother" workflow where the routines don't synchronize on every single value that is being passed through the channel, make a buffered channel:
c := make(chan int, 3)
Related
I'm trying to understand, why my code doesn't behave as I expect it. The problem is that I would expect, that my code would behave like that:
Define channel
Run goroutine and start looping
Put value into channel, print "finished"
Starting second iteration, blocking call(there is already value in the channel), move to main goroutine
Printing 1, trying to run second iteration, blocking call for main goroutine, coming back to second goroutine
Cycle repeats
It works like that, but only with time.Sleep, but for some reason when commenting out time.Sleep it behaves totally different. What's even more interesting that sometimes for really small values of time like Nanos etc this code returns even more different results. Could someone explain me, why it works like that? My guess is that maybe Println is too slow on display, but it sounds weird to me..
Thanks
** As expected: **
finished
1
finished
2
finished
3
finished
6
finished
4
finished
8
finished all
** Not expected **
finished
1
2
finished
finished
3
finished
6
4
finished
finished
8
finished all
func main() {
var c chan int = make(chan int)
go sendingThrowingResults(c)
for val := range c {
fmt.Println(val)
}
fmt.Println("finished all")
}
func sendingThrowingResults(c chan int) {
var results []int = []int{1, 2, 3, 6, 4, 8}
for _, val := range results {
//time.Sleep(100 * time.Millisecond)
c <- val
fmt.Println("finished")
}
defer close(c)
}
A channel operation needs both sides to participate. A write only happens when a reader is ready. Once that happens, there is no guarantee on which goroutine will run first.
Thus, once the channel write happens, one of the two printlns will work, in some random order.
Doesn't go routine and the channels worked in the order they were called.
and go routine share values between the region variables?
main.go
var dataSendChannel = make(chan int)
func main() {
a(dataSendChannel)
time.Sleep(time.Second * 10)
}
func a(c chan<- int) {
for i := 0; i < 1000; i++ {
go b(dataSendChannel)
c <- i
}
}
func b(c <-chan int) {
val := <-c
fmt.Println(val)
}
output
> go run main.go
0
1
54
3
61
5
6
7
8
9
Channels are ordered. Goroutines are not. Goroutines may run, or stall, more or less at random, whenever they are logically allowed to run. They must stop and wait whenever you force them to do so, e.g., by attempting to write on a full channel, or using a mutex.Lock() call on an already-locked mutex, or any of those sorts of things.
Your dataSendChannel is unbuffered, so an attempt to write to it will pause until some goroutine is actively attempting to read from it. Function a spins off one goroutine that will attempt one read (go b(...)), then writes and therefore waits for at least one reader to be reading. Function b immediately begins reading, waiting for data. This unblocks function a, which can now write some integer value. Function a can now spin off another instance of b, which begins reading; this may happen before, during, or after the b that got a value begins calling fmt.Println. This second instance of b must now wait for someone—which in this case is always function a, running the loop—to send another value, but a does that as quickly as it can. The second instance of b can now begin calling fmt.Println, but it might, mostly-randomly, not get a chance to do that yet. The first instance of b might already be in fmt.Println, or maybe it isn't yet, and the second one might run first—or maybe both wait around for a while and a third instance of b spins up, reads some value from the channel, and so on.
There's no guarantee which instance of b actually gets into fmt.Println when, so the values you see printed will come out in some semi-random order. If you want the various b instances to sequence themselves, they will need to do that somehow.
So I'm trying to understand how parallel computing works while also learning Go. I understand the difference between concurrency and parallelism, however, what I'm a little stuck on is how Go (or the OS) determines that something should be executed in parallel...
Is there something I have to do when writing my code, or is it all handled by the schedulers?
In the example below, I have two functions that are run in separate Go routines using the go keyword. Because the default GOMAXPROCS is the number of processors available on your machine (and I'm also explicitly setting it) I would expect that these two functions run at the same time and thus the output would be a mix of number in particular order - And furthermore that each time it is run the output would be different. However, this is not the case. Instead, they are running one after the other and to make matters more confusing function two is running before function one.
Code:
func main() {
runtime.GOMAXPROCS(6)
var wg sync.WaitGroup
wg.Add(2)
fmt.Println("Starting")
go func() {
defer wg.Done()
for smallNum := 0; smallNum < 20; smallNum++ {
fmt.Printf("%v ", smallNum)
}
}()
go func() {
defer wg.Done()
for bigNum := 100; bigNum > 80; bigNum-- {
fmt.Printf("%v ", bigNum)
}
}()
fmt.Println("Waiting to finish")
wg.Wait()
fmt.Println("\nFinished, Now terminating")
}
Output:
go run main.go
Starting
Waiting to finish
100 99 98 97 96 95 94 93 92 91 90 89 88 87 86 85 84 83 82 81 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19
Finished, Now terminating
I am following this article, although just about every example I've come across does something similar.
Concurrency, Goroutines and GOMAXPROCS
Is this working the way is should and I'm not understanding something correctly, or is my code not right?
Is there something I have to do when writing my code,
No.
or is it all handled by the schedulers?
Yes.
In the example below, I have two functions that are run in separate Go routines using the go keyword. Because the default GOMAXPROCS is the number of processors available on your machine (and I'm also explicitly setting it) I would expect that these two functions run at the same time
They might or might not, you have no control here.
and thus the output would be a mix of number in particular order - And furthermore that each time it is run the output would be different. However, this is not the case. Instead, they are running one after the other and to make matters more confusing function two is running before function one.
Yes. Again you cannot force parallel computation.
Your test is flawed: You just don't do much in each goroutine. In your example goroutine 2 might be scheduled to run, starts running and completes before goroutine 1 started running. "Starting" a goroutine with go doesn't force it to start executing right away, all there is done is creating a new goroutine which can run. From all goroutines which can run some are scheduled onto your processors. All this scheduling cannot be controlled, it is fully automatic. As you seem to know this is the difference between concurrent and parallel. You have control over concurrency in Go but not (much) on what is done actually in parallel on two or more cores.
More realistic examples with actual, long-running goroutines which do actual work will show interleaved output.
It's all handled by the scheduler.
With only two loops of 20 short instructions, you will be hard pressed to see the effects of concurrency or parallelism.
Here is another toy example : https://play.golang.org/p/xPKITzKACZp
package main
import (
"fmt"
"runtime"
"sync"
"sync/atomic"
"time"
)
const (
ConstMaxProcs = 2
ConstRunners = 4
ConstLoopcount = 1_000_000
)
func runner(id int, wg *sync.WaitGroup, cptr *int64) {
var times int
for i := 0; i < ConstLoopcount; i++ {
val := atomic.AddInt64(cptr, 1)
if val > 1 {
times++
}
atomic.AddInt64(cptr, -1)
}
fmt.Printf("[runner %d] cptr was > 1 on %d occasions\n", id, times)
wg.Done()
}
func main() {
runtime.GOMAXPROCS(ConstMaxProcs)
var cptr int64
wg := &sync.WaitGroup{}
wg.Add(ConstRunners)
start := time.Now()
for id := 1; id <= ConstRunners; id++ {
go runner(id, wg, &cptr)
}
wg.Wait()
fmt.Printf("completed in %s\n", time.Now().Sub(start))
}
As with your example : you don't have control on the scheduler, this example just has more "surface" to witness some effects of concurrency.
It's hard to witness the actual difference between concurrency and parallelism from within the program, you can view your processor's activity while it runs, or check the global execution time.
The playground does not give sub-second precision on its clock, if you want to see the actual timing, copy/paste the code in a local file and tune the constants to see various effects.
Note that some other effects (probably : branch prediction on the if val > 1 {...} check and/or memory invalidation around the shared cptr variable) make the execution very volatile on my machine, so don't expect a straight "running with ConstMaxProcs = 4 is 4 times quicker than ConstMaxProcs = 1".
Following this post about go-routines leaking, https://www.ardanlabs.com/blog/2018/11/goroutine-leaks-the-forgotten-sender.html, i tried to solve my leaking code. But adding a buffer to the channel it did not do the trick.
My code
package main
import (
"fmt"
"runtime"
"time"
)
func main() {
fmt.Println(runtime.NumGoroutine())
leaking()
time.Sleep(5)
fmt.Println(runtime.NumGoroutine())
}
func leaking() {
errChang := make(chan int, 1)
go func() {
xx := return666()
errChang <- xx
}()
fmt.Println("hola")
return
fmt.Println(<-errChang)
}
func return666() int {
time.Sleep(time.Second * 1)
return 6
}
My initial code did not use a buffer, leading to the go-routine in the leaking function, .. to leak. Following the post i expected that by adding a buffer to the channel, it would have avoided leaking.
Here, in the Go Playground, is your original code with a few slight modifications:
the delays are reduced, except for time.Sleep(5) which becomes time.Sleep(time.Second);
a return is removed because it becomes unnecessary;
a fmt.Println is commented out, because with both the return and the uncommented fmt.Println, go vet complains about the unreachable fmt.Println;
the channel stored in errChang is changed to unbuffered.
When run, its output is:
1
hola
2
(with a small delay before the 2), showing that indeed, the anonymous goroutine you started in function leaking is still running.
If we uncomment the commented-out fmt.Println, the output is:
1
hola
6
1
(with the same slight delay before the final 1) because we now wait for (and then print) the value computed in return666 and sent over channel errChang.
If we keep the commented-out fmt.Println commented out and make the channel buffered, the output becomes:
1
hola
1
as the anonymous goroutine is now able to push its value (6) into the channel.
The channel itself would be garbage collected, along with the single value stored inside it, as there are no remaining references to the channel at this point. Note, however, that simply making the channel buffered is not always sufficient. If we send two values down the channel, the program returns to printing:
1
hola
2
as the anonymous goroutine succeeds in putting 6 into the channel but then blocks trying to put 42 in as well.
I just started using go and wrote my first program but the output is not as expected. I have writtern a async routine addUrl which adds url to channel 5000 times and consumeUrl removes from the channel and prints it.
The routine runs only 9 time. Why is it? Below is the code and output
package main
import "fmt"
import "time"
var urlCount = 0
func main(){
urlHolder := make(chan string,5000)
fmt.Printf("Starting program")
go addUrls(urlHolder)
time.Sleep(time.Millisecond * 100)
go consumeUrls(urlHolder)
fmt.Printf("Done")
}
func addUrls(urlHolder chan string){
var myurl string = "https://example.com/"
for i:=0; i<5000 ; i++ {
urlHolder<-myurl
fmt.Printf(" %d url added \n",i)
time.Sleep(time.Millisecond * 10)
}
}
func consumeUrls(urlHolder chan string) {
urlCount++
urlsConsumed := <- urlHolder
fmt.Printf("Pulled url %d",urlCount," ",urlsConsumed,"\n")
time.Sleep(time.Millisecond * 20)
}
The output is
Starting program
0 url added
1 url added
2 url added
3 url added
4 url added
5 url added
6 url added
7 url added
8 url added
Done
Why is it terminating at 8 when loop is 5000?
You are using time.Sleep to wait for main to finish but you really should be using WaitGroups.
That way you don't have to try to pick some arbitrary time and hope it's enough for your program to finish, or worry about setting too much time and your program sits around doing nothing.
I've added the implementation of WaitGroups to your code here:
https://play.golang.org/p/1zn2JYefaA
Also, the way your consumeUrls function is written is not looping properly and you won't get everything in your channel printed. But since that wasn't your specific question I won't address it here.
Actually The problem was like Nipun said, the main got terminated early.