Consider the following code
package main
import (
"fmt"
"runtime"
"sync"
)
func main() {
messages := make(chan bool)
var wg sync.WaitGroup
var x = 1000
wg.Add(runtime.NumCPU())
for i := 0; i < runtime.NumCPU(); i++ {
go func(x int) {
defer wg.Done()
var i = 0
for i < x {
i += 1
fmt.Println(i * i)
}
messages <- true
}(x)
}
go func() {
for i := range messages {
fmt.Println(i)
}
}()
wg.Wait()
}
And the following last couple of line output
980100
982081
984064
true
988036
990025
992016
994009
996004
998001
1000000
Since message <- true is always at the end of a for loop and
for i := range messages {
fmt.Println(i)
}
prints after the channel receive the message.
I expect true to be printed always at the end like
988036
990025
992016
994009
996004
998001
1000000
true
But I find that is only sometimes true, why is that?
What you're doing is:
Start a number of goroutines, equal to the number of CPUs on your system.
Start one additional goroutine, which reads and prints the values from the messages channel.
Wait for the goroutines from #1 to terminate
Exit
Because you're only waiting for the first batch of goroutines to terminate, there is no guarantee that all (or even any) of the messages values will be printed before the program terminates.
Related
In this example, we have a worker. The idea here is simulate clean shutdown of all go routines based on a condition.
In this case, go routines get spun - based on workers count. Each go routine reads the channel, does some work and sends output to the outputChannel.
The main go routine reads this output and prints it. To simulate a stop condition, the doneChannel is closed. Expected outcome is that select inside each go routine will pick this up and execute return which in turn will call the defer println. The actual output is that it never gets called and main exits.
Not sure what's the reason behind this.
package main
import (
"log"
"time"
)
const jobs = 100
const workers = 1
var timeout = time.After(5 * time.Second)
func main() {
doneChannel := make(chan interface{})
outputChannel := make(chan int)
numberStream := generator()
for i := 1; i <= workers; i++ {
go worker(doneChannel, numberStream, outputChannel)
}
// listen for output
loop:
for {
select {
case i := <-outputChannel:
log.Println(i)
case <-timeout:
// before you timeout cleanup go routines
break loop
}
}
close(doneChannel)
time.Sleep(5 * time.Second)
log.Println("main exited")
}
func generator() <-chan int {
defer log.Println("generator completed !")
c := make(chan int)
go func() {
for i := 1; i <= jobs; i++ {
c <- i
}
defer close(c)
}()
return c
}
func worker(done <-chan interface{}, c <-chan int, output chan<- int) {
// this will be a go routine
// Do some work and send results to output Channel.
// Incase if the done channel is called kill the go routine.
defer log.Println("go routines exited")
for {
select {
case <-done:
log.Println("here")
return
case i := <-c:
time.Sleep(1 * time.Second) // worker delay
output <- i * 100
}
}
}
When your main loop finishes during the timeout, you continue your program and
Close done channel
Print message
Exit
There is no reason to wait for any goroutine to process the signal of this channel.
If you add a small sleep you will see some messages
In real scenarios we use a waitgroup to be sure all goroutine finish properly
I am new to go and I am trying to learn some basic use of signal functions in goroutines. I have an infinite for loop in go. Through this for loop, I pass values to a goroutine through a channel. I also have a threshold value after which I will like to stop sending values indefinitely to the goroutine (i.e. close the channel). When the threshold value is reached, I will like to break the for loop. Following is what I have tried so far.
In this particular example, thresholdValue = 10 and I would like to print values from 0 , ..., 9 and then stop.
I followed this post on medium and this post on stackoverflow. I picked elements from these posts which I could use.
This is what I have done at the present. In the main function of my code, I purposefully make the for loop an infinite loop. My main intention is to learn how to have the goroutine readValues() take the threshold value and then stop transmission of values indefinitely in the channel.
package main
import (
"fmt"
)
func main() {
ch := make(chan int)
quitCh := make(chan struct{}) // signal channel
thresholdValue := 10 //I want to stop the incoming data to readValues() after this value
go readValues(ch, quitCh, thresholdValue)
for i:=0; ; i++{
ch <- i
}
}
func readValues(ch chan int, quitCh chan struct{}, thresholdValue int) {
for value := range ch {
fmt.Println(value)
if (value == thresholdValue){
close(quitCh)
}
}
}
The goroutine in my code still misses the threshold. I will appreciate any direction as to how I should proceed from here.
to show good faith, this is the program rewritten.
package main
import (
"log"
"sync"
"time"
)
func main() {
ch := make(chan int, 5) // capacity increased for demonstration
thresholdValue := 10
var wg sync.WaitGroup
wg.Add(1)
go func() {
readValues(ch)
wg.Done()
}()
for i := 0; i < thresholdValue; i++ {
ch <- i
}
close(ch)
log.Println("sending done.")
wg.Wait()
}
func readValues(ch chan int) {
for value := range ch {
<-time.After(time.Second) // for demonstratin purposes.
log.Println(value)
}
}
In this version readValues exits because the for loop did exit and that main closed ch.
In other words a stop condition take effects and triggers the exit sequence (signal end of input then wait for the processing to finish)
I use sync.WaitGroup with goroutine before, but I want to control the goroutine concurrency,
so I write my waitgroup with concurrency limit like:
package wglimit
import (
"sync"
)
// WaitGroupLimit ...
type WaitGroupLimit struct {
ch chan int
wg *sync.WaitGroup
}
// New ...
func New(size int) *WaitGroupLimit {
if size <= 0 {
size = 1
}
return &WaitGroupLimit{
ch: make(chan int, size), // buffer chan to limit concurrency
wg: &sync.WaitGroup{},
}
}
// Add ...
func (wgl *WaitGroupLimit) Add(delta int) {
for i := 0; i < delta; i++ {
wgl.ch <- 1
wgl.wg.Add(1)
}
}
// Done ...
func (wgl *WaitGroupLimit) Done() {
wgl.wg.Done()
<-wgl.ch
}
// Wait ...
func (wgl *WaitGroupLimit) Wait() {
close(wgl.ch)
wgl.wg.Wait()
}
And then I use it to control the goroutine concurrency, for example:
jobs := ["1", "2", "3", "4"] // some jobs
// wg := sync.WaitGroup{} // have no concurrency limit
wg := wglimit.New(2) // limit 2 goroutine
for _, job := range jobs {
wg.Add(1)
go func(job string) {
// job worker
defer wg.Done()
}(job)
}
wg.Wait()
And it looks like worked when running.
But Test Failed:
package wglimit
import (
"runtime"
"testing"
"time"
)
func TestGoLimit(t *testing.T) {
var limit int = 5
wglimit := New(limit)
for i := 0; i < 10000; i++ {
wglimit.Add(1)
go func() {
defer wglimit.Done()
time.Sleep(time.Millisecond)
if runtime.NumGoroutine() > limit+2 {
println(runtime.NumGoroutine()) // will print 9 , cocurrent limit fail ?
t.Errorf("FAIL")
}
}()
}
wglimit.Wait()
}
When testing, the goroutine numbers is bigger than my limit, it seems like the cocurrent limit fail.
Anything wrong with my WaitGroupLimit code and why?
Anything wrong with my WaitGroupLimit code [...]?
No.
The problem is runtime.NumGoroutine() doesn't do what you seem to think it does. It counts all goroutines, i.e. not only the ones you start but also the goroutines the runtime uses itself, e.g. for concurrent garbage collection. NumGoroutine is thus higher than your limit.
Your code is fine, your test isn't. Do not try to get clever in testing and test what you code really does: It blocks on Add until the limited resource is available. Test that and not a goroutine count which is just a (bad) proxy for the desired behaviour in your test.
I am trying to understand how channels work in golang. The code I have is very simple but the output is surprising.
As the documentation states: reading and writing from/to a channel is blocking the current goroutine, so I thought writing to a channel would block the channel until the main routine yields.
package main
func rtn(messages chan<- string) {
defer close(messages)
println("p1")
messages <- "ping1"
//for i := 0; i < 10000000; i++ { }
println("p2")
messages <- "ping2"
}
func main() {
messages := make(chan string)
go rtn(messages)
for msg := range messages {
println(msg)
}
}
I thought it would print
p1
ping1
p2
ping2
but it actually prints
p1
p2
ping1
ping2
You are using an unbuffered channels, that works as a point of synchronization between the main and second goroutines.
In this case, you only know that when the second goroutine is here messages <- "ping1" the main one is at the line for msg := range messages. Thus, there is no guarantee that main loop reaches println(msg) immediately. I.e., in the meantime the second goroutine could have moved on and reached lines println("p2") and messages <- "ping2".
As a counterexample, I am adding a channel just to enforce the complete synchronization between prints.
package main
func rtn(messages chan<- string, syncChan chan struct{}) {
defer close(messages)
println("p1")
messages <- "ping1"
//Wait for main goroutine to print its message
<-syncChan
//for i := 0; i < 10000000; i++ { }
println("p2")
messages <- "ping2"
//Wait for main goroutine to print its message
<-syncChan
}
func main() {
messages := make(chan string)
syncChan := make(chan struct{})
go rtn(messages, syncChan)
for msg := range messages {
println(msg)
//Notify the second goroutine that is free to go
syncChan <- struct{}{}
}
}
Which prints the output you expected:
p1
ping1
p2
ping2
Here is another example that produce the output you are looking for. In this case the main goroutine is forcefully blocked by the time.Sleep(). This will make the second goroutine be ready to send before the receiver is ready to receive. Therefore, the sender will actually block on the send operation.
package main
import (
"time"
)
func rtn(messages chan<- string) {
defer close(messages)
println("p1")
messages <- "ping1"
//for i := 0; i < 10000000; i++ { }
println("p2")
messages <- "ping2"
}
func main() {
messages := make(chan string)
go rtn(messages)
//Put main goroutine to sleep. This will make the
//sender goroutine ready before the receiver.
//Therefore it will have to actually block!
time.Sleep(time.Millisecond * 500)
for msg := range messages {
println(msg)
}
}
i am new in golang recently. i have a question about goroutine when use time.sleep function. here the code.
package main
import (
"fmt"
"time"
)
func go1(msg_chan chan string) {
for {
msg_chan <- "go1"
}
}
func go2(msg_chan chan string) {
for {
msg_chan <- "go2"
}
}
func count(msg_chan chan string) {
for {
msg := <-msg_chan
fmt.Println(msg)
time.Sleep(time.Second * 1)
}
}
func main() {
var c chan string
c = make(chan string)
go go1(c)
go go2(c)
go count(c)
var input string
fmt.Scanln(&input)
}
and output is
go1
go2
go1
go2
go1
go2
i think when count function is execute sleep function, go1 and go2 will execute in random sequence. so the out put maybe like
go1
go1
go2
go2
go2
go1
when i delete the sleep code in count function. the result as i supposed , it's random.
i am stucked in this issue.
thanks.
First thing to notice is that there are three go routines and all of them are independent of each other. The only thing that combines the two go routines with count routine is the channel on which both go routines are sending the values.
time.Sleep is not making the go routines synchronous. On using time.Sleep you are actually letting the count go routine to wait for that long which let other go routine to send the value on the channel which is available for the count go routine to be able to receive.
One more thing that you can do to check it is increase the number of CPU's which will give you the random result.
func GOMAXPROCS(n int) int
GOMAXPROCS sets the maximum number of CPUs that can be executing
simultaneously and returns the previous setting. If n < 1, it does not
change the current setting. The number of logical CPUs on the local
machine can be queried with NumCPU. This call will go away when the
scheduler improves.
The number of CPUs available simultaneously to executing goroutines is
controlled by the GOMAXPROCS shell environment variable, whose default
value is the number of CPU cores available. Programs with the
potential for parallel execution should therefore achieve it by
default on a multiple-CPU machine. To change the number of parallel
CPUs to use, set the environment variable or use the similarly-named
function of the runtime package to configure the run-time support to
utilize a different number of threads. Setting it to 1 eliminates the
possibility of true parallelism, forcing independent goroutines to
take turns executing.
Considering the part where output of the go routine is random, it is always random. But channels most probably work in queue which is FIFO(first in first out) as it depends on which value is available on the channel to b received. So whichever be the value available on the channel to be sent is letting the count go routine to wait and print that value.
Take for an example even if I am using time.Sleep the output is random:
package main
import (
"fmt"
"time"
)
func go1(msg_chan chan string) {
for i := 0; i < 10; i++ {
msg_chan <- fmt.Sprintf("%s%d", "go1:", i)
}
}
func go2(msg_chan chan string) {
for i := 0; i < 10; i++ {
msg_chan <- fmt.Sprintf("%s%d", "go2:", i)
}
}
func count(msg_chan chan string) {
for {
msg := <-msg_chan
fmt.Println(msg)
time.Sleep(time.Second * 1)
}
}
func main() {
var c chan string
c = make(chan string)
go go1(c)
go go2(c)
go count(c)
time.Sleep(time.Second * 20)
fmt.Println("finished")
}
This sometimes leads to race condition which is why we use synchronization either using channels or wait.groups.
package main
import (
"fmt"
"sync"
"time"
)
var wg sync.WaitGroup
func go1(msg_chan chan string) {
defer wg.Done()
for {
msg_chan <- "go1"
}
}
func go2(msg_chan chan string) {
defer wg.Done()
for {
msg_chan <- "go2"
}
}
func count(msg_chan chan string) {
defer wg.Done()
for {
msg := <-msg_chan
fmt.Println(msg)
time.Sleep(time.Second * 1)
}
}
func main() {
var c chan string
c = make(chan string)
wg.Add(1)
go go1(c)
wg.Add(1)
go go2(c)
wg.Add(1)
go count(c)
wg.Wait()
fmt.Println("finished")
}
Now coming to the part where you are using never ending for loop to send the values on a channel. So if you remove the time.Sleep your process will hang since the loop will never stop to send the values on the channel.