How can i use gomaxprocs? The code below sets gomaxprocs, but then more workers are spawned that set. I expect 2 processes but 5 are still run.
package main
import (
"fmt"
"runtime"
"sync"
"time"
)
func worker(i int, waiter chan struct{}, wg *sync.WaitGroup) {
defer func(waiter chan struct{}, wg *sync.WaitGroup) {
fmt.Printf("worker %d done\n", i)
wg.Done()
<-waiter
}(waiter, wg)
fmt.Printf("worker %d starting\n", i)
time.Sleep(time.Second)
}
func main() {
runtime.GOMAXPROCS(2)
var concurrency = 5
var items = 10
waiter := make(chan struct{}, concurrency)
var wg sync.WaitGroup
for i := 0; i < items; i++ {
wg.Add(1)
waiter <- struct{}{}
go worker(i, waiter, &wg)
}
wg.Wait()
}
Go has three concepts for what C/C++ programmers think of as a thread: G, P, M.
M = actual thread
G = Goroutines (i.e., the code in your program)
P = Processor
There is no Go API for limiting the number of Ms. There is no API for limiting the number of Gs - a new one gets created every time go func(...) is called. The GOMAXPROCS thing is there to limit Ps.
Each P is used to track the runtime state of some running Goroutine.
You should think of GOMAXPROCS as the peak number of Ms devoted to running Goroutines. (There are other Ms that don't run Goroutines, but handle garbage collection tasks and serve as template threads for creating new Ms as needed etc. Some Ms are devoted to holding runtime state while some Go code is blocked inside a system call.)
So, in terms of the code in your program, GOMAXPROCS is a constraint for how parallel its Go code execution can be. When a running Goroutine reaches a point where it becomes blocked, it is parked and its P is used to resume execution of some other Goroutine that is not blocked.
I've just installed Go on Mac, and here's the code
package main
import (
"fmt"
"time"
)
func Product(ch chan<- int) {
for i := 0; i < 100; i++ {
fmt.Println("Product:", i)
ch <- i
}
}
func Consumer(ch <-chan int) {
for i := 0; i < 100; i++ {
a := <-ch
fmt.Println("Consmuer:", a)
}
}
func main() {
ch := make(chan int, 1)
go Product(ch)
go Consumer(ch)
time.Sleep(500)
}
I "go run producer_consumer.go", there's no output on screen, and then it quits.
Any problem with my program ? How to fix it ?
This is a rather verbose answer, but to put it simply:
Using time.Sleep to wait until hopefully other routines have completed their jobs is bad.
The consumer and producer shouldn't know anything about each other, apart from the type they exchange over the channel. Your code relies on both consumer and producer knowing how many ints will be passed around. Not a realistic scenario
Channels can be iterated over (think of them as a thread-safe, shared slice)
channels should be closed
At the bottom of this rather verbose answer where I attempt to explain some basic concepts and best practices (well, better practices), you'll find your code rewritten to work and display all the values without relying on time.Sleep. I've not tested that code, but should be fine
Right, there's a couple of problems here. Just as a bullet-list:
Your channel is buffered to 1, which is fine, but it's not necessary
Your channel is never closed
You're waiting 500ns, then exit regardless of the routines having completed, or even started processing for that matter.
There's no centralised control on over the routines, once you've started them, you have 0 control. If you hit ctrl+c, you might want to cancel routines when writing code that'll handle important data. Check signal handling, and context for this
Channel buffer
Seeing as you already know how many values you're going to push onto your channel, why not simply create ch := make(chan int, 100)? That way your publisher can continue to push messages onto the channel, regardless of what the consumer does.
You don't need to do this, but adding a sensible buffer to your channel, depending on what you're trying to do, is definitely worth checking out. At the moment, though, both routines are using fmt.Println & co, which is going to be a bottleneck either way. Printing to STDOUT is thread-safe, and buffered. This means that each call to fmt.Print* is going to acquire a lock, to avoid text from both routines to be combined.
Closing the channel
You could simply push all the values onto your channel, and then close it. This is, however, bad form. The rule of thumb WRT channels is that channels are created and closed in the same routine. Meaning: you're creating the channel in the main routine, that's where it should be closed.
You need a mechanism to sync up, or at least keep tabs on whether or not your routines have completed their job. That's done using the sync package, or through a second channel.
// using a done channel
func produce(ch chan<- int) <-chan struct{} {
done := make(chan struct{})
go func() {
for i := 0; i < 100; i++ {
ch <- i
}
// all values have been published
// close done channel
close(done)
}()
return done
}
func main() {
ch := make(chan int, 1)
done := produce(ch)
go consume(ch)
<-done // if producer has done its thing
close(ch) // we can close the channel
}
func consume(ch <-chan int) {
// we can now simply loop over the channel until it's closed
for i := range ch {
fmt.Printf("Consumed %d\n", i)
}
}
OK, but here you'll still need to wait for the consume routine to complete.
You may have already noticed that the done channel technically isn't closed in the same routine that creates it either. Because the routine is defined as a closure, however, this is an acceptable compromise. Now let's see how we could use a waitgroup:
import (
"fmt"
"sync"
)
func product(wg *sync.WaitGroup, ch chan<- int) {
defer wg.Done() // signal we've done our job
for i := 0; i < 100; i++ {
ch <- i
}
}
func main() {
ch := make(chan int, 1)
wg := sync.WaitGroup{}
wg.Add(1) // I'm adding a routine to the channel
go produce(&wg, ch)
wg.Wait() // will return once `produce` has finished
close(ch)
}
OK, so this looks promising, I can have the routines tell me when they've finished their tasks. But if I add both consumer and producer to the waitgroup, I can't simply iterate over the channel. The channel will only ever get closed if both routines invoke wg.Done(), but if the consumer is stuck looping over a channel that'll never get closed, then I've created a deadlock.
Solution:
A hybrid would be the easiest solution at this point: Add the consumer to a waitgroup, and use the done channel in the producer to get:
func produce(ch chan<- int) <-chan struct{} {
done := make(chan struct{})
go func() {
for i := 0; i < 100; i++ {
ch <- i
}
close(done)
}()
return done
}
func consume(wg *sync.WaitGroup, ch <-chan int) {
defer wg.Done()
for i := range ch {
fmt.Printf("Consumer: %d\n", i)
}
}
func main() {
ch := make(chan int, 1)
wg := sync.WaitGroup{}
done := produce(ch)
wg.Add(1)
go consume(&wg, ch)
<- done // produce done
close(ch)
wg.Wait()
// consumer done
fmt.Println("All done, exit")
}
I have changed slightly(expanded time.Sleep) your code. Works fine on my Linux x86_64
func Product(ch chan<- int) {
for i := 0; i < 10; i++ {
fmt.Println("Product:", i)
ch <- i
}
}
func Consumer(ch <-chan int) {
for i := 0; i < 10; i++ {
a := <-ch
fmt.Println("Consmuer:", a)
}
}
func main() {
ch := make(chan int, 1)
go Product(ch)
go Consumer(ch)
time.Sleep(10000)
}
Output
go run s1.go
Product: 0
Product: 1
Product: 2
As JimB hinted at, time.Sleep takes a time.Duration, not an integer. The godoc shows an example of how to call this correctly. In your case, you probably want:
time.Sleep(500 * time.Millisecond)
The reason that your program is exiting quickly (but not giving you an error) is due to the (somewhat surprising) way that time.Duration is implemented.
time.Duration is simply a type alias for int64. Internally, it uses the value to represent the duration in nanoseconds. When you call time.Sleep(500), the compiler will gladly interpret the numeric literal 500 as a time.Duration. Unfortunately, that means 500 ns.
time.Millisecond is a constant equal to the number of nanoseconds in a millisecond (1,000,000). The nice thing is that requiring you to do that multiplication explicitly makes it obvious to that caller what the units are on that argument. Unfortunately, time.Sleep(500) is perfectly valid go code but doesn't do what most beginners would expect.
I have a hard time understanding concurrency/paralel. in my code I made a loop of 5 cycle. Inside of the loop I added the wg.Add(1), in total I have 5 Adds. Here's the code:
package main
import (
"fmt"
"sync"
)
func main() {
var list []int
wg := sync.WaitGroup{}
for i := 0; i < 5; i++ {
wg.Add(1)
go func(c *[]int, i int) {
*c = append(*c, i)
wg.Done()
}(&list, i)
}
wg.Wait()
fmt.Println(len(list))
}
The main func waits until all the goroutines finish but when I tried to print the length of slice I get random results. ex (1,3,etc) is there something that is missing for it to get the expected result ie 5 ?
is there something that is missing for it to get the expected result ie 5 ?
Yes, proper synchronization. If multiple goroutines access the same variable where at least one of them is a write, you need explicit synchronization.
Your example can be "secured" with a single mutex:
var list []int
wg := sync.WaitGroup{}
mu := &sync.Mutex{} // A mutex
for i := 0; i < 5; i++ {
wg.Add(1)
go func(c *[]int, i int) {
mu.Lock() // Must lock before accessing shared resource
*c = append(*c, i)
mu.Unlock() // Unlock when we're done with it
wg.Done()
}(&list, i)
}
wg.Wait()
fmt.Println(len(list))
This will always print 5.
Note: the same slice is read at the end to prints its length, yet we are not using the mutex there. This is because the use of waitgroup ensures that we can only get to that point after all goroutines that modify it have completed their job, so data race cannot occur there. But in general both reads and writes have to be synchronized.
See possible duplicates:
go routine not collecting all objects from channel
Server instances with multiple users
Why does this code cause data race?
How safe are Golang maps for concurrent Read/Write operations?
golang struct concurrent read and write without Lock is also running ok?
See related questions:
Can I concurrently write different slice elements
If I am using channels properly should I need to use mutexes?
Is it safe to read a function pointer concurrently without a lock?
Concurrent access to maps with 'range' in Go
My specific issue is that I have an unbuffered channel and am spawning multiple goroutines bounded with a semaphore to perform work:
func main() {
sem := make(chan struct{}, 10) // allow ten concurrent parsers
wg := &sync.WaitGroup{}
wg.Add(1)
DoSomething("http://example.com", sem, wg)
wg.Wait()
// all done
}
func DoSomething(u string, sem chan struct{}, wg *sync.WaitGroup) {
defer wg.Done()
sem <- struct{}{} // grab
defer func() { <-sem }() // release
var newSomethings []string
// ...
for u := range newSomethings {
wg.Add(1)
go DoSomething(u)
}
}
If there are multiple DoSomething goroutines on the stack, blocked on the sem write (or inversely on a read) When a write happens is there any ordering to which go routine gets through with the write?? I would guess it were random but I could imagine:
it is random
writes/receives happen in the order they are registered
implementation dependent
I looked at a couple of resources and was unable to find a solution:
https://github.com/golang/go/issues/247
https://golang.org/ref/spec#Receive_operator
https://golang.org/ref/spec#Channel_types
I'm wondering if this is undefined and/or implementation dependent, or if this logic is located and defined somewhere within go core?
The order that goroutines blocked on a send operation are serviced is not defined, but it's implemented as a FIFO. You can see the implementation in runtime/chan.go, which uses a linked list to track the channel's senders and receivers.
We can try to make an example showing the effective ordering like so:
func main() {
ch := make(chan int)
ready := make(chan int)
for i := 0; i < 10; i++ {
i := i
go func() {
ready <- 1
ch <- i
}()
<-ready
runtime.Gosched()
}
for i := 0; i < 10; i++ {
v := <-ch
if i != v {
panic("out of order!")
}
fmt.Println(v)
}
}
https://play.golang.org/p/u0ukR-5Ptw4
This still isn't technically correct, because there's no way to observe blocking on a send operation, so there's still a race between the ready send and the send to ch on the next line. We can try to eliminate that with the runtime.Gosched call here, or even a time.Sleep, but without explicit synchronization there's no guarantee of a "happens before" relationship.
Regardless, this queues up the goroutines and shows the expected output order, and if they weren't queued up already, it would be more likely to process the values out of order.
You can see by this example that we can't truly determine the order that the goroutines are queued up, it is almost always non-deterministic, and therefore reasoning about this isn't usually useful in practice.
How do I deal with a situation where undetected deadlock occurs when reading results of execution of uncertain number tasks from a channel in a complex program, e.g. web server?
package main
import (
"fmt"
"math/rand"
"time"
)
func main() {
rand.Seed(time.Now().UTC().UnixNano())
results := make(chan int, 100)
// we can't know how many tasks there will be
for i := 0; i < rand.Intn(1<<8)+1<<8; i++ {
go func(i int) {
time.Sleep(time.Second)
results <- i
}(i)
}
// can't close channel here
// because it is still written in
//close(results)
// something else is going on other threads (think web server)
// therefore a deadlock won't be detected
go func() {
for {
time.Sleep(time.Second)
}
}()
for j := range results {
fmt.Println(j)
// we just stuck in here
}
}
In case of simpler programs go detects a deadlock and properly fails. Most examples either fetch a known number of results, or write to the channel sequentially.
The trick is to use sync.WaitGroup and wait for the tasks to finish in a non-blocking way.
var wg sync.WaitGroup
// we can't know how many tasks there will be
for i := 0; i < rand.Intn(1<<8)+1<<8; i++ {
wg.Add(1)
go func(i int) {
time.Sleep(time.Second)
results <- i
wg.Done()
}(i)
}
// wait for all tasks to finish in other thread
go func() {
wg.Wait()
close(results)
}()
// execution continues here so you can print results
See also: Go Concurrency Patterns: Pipelines and cancellation - The Go Blog