I don't understand why the deadlock is occurring in this code. I've tried several different things to get the deadlock to stop (several different versions using WorkGroup). This is my first day in Go, and I am pretty disappointed so far with the complexity of fairly simple and straightforward operations. I feel like I'm missing something big and obvious, but all of the docs I have found on this are seemingly very different from what, to me, is a very basic mode of operation. All of the docs use primitive types for channels (int, string) not more complex types, all with very basic for loops OR on they are on the other end of the spectrum where the functions are fairly complicated orchestrations.
I guess I'm really looking for a middle of the road sample of "this is how it's usually done" with goroutines.
package main
import "fmt"
//import "sync"
import "time"
type Item struct {
name string
}
type Truck struct {
Cargo []Item
name string
}
func UnloadTrucks(c chan Truck) {
for t := range c {
fmt.Printf("%s has %d items in cargo: %s\n", t.name, len(t.Cargo), t.Cargo[0].name)
}
}
func main() {
trucks := make([]Truck, 2)
ch := make(chan Truck)
for i, _ := range trucks {
trucks[i].name = fmt.Sprintf("Truck %d", i+1)
fmt.Printf("Building %s\n", trucks[i].name)
}
for t := range trucks {
go func(tr Truck) {
itm := Item{}
itm.name = "Groceries"
fmt.Printf("Loading %s\n", tr.name)
tr.Cargo = append(tr.Cargo, itm)
ch <- tr
}(trucks[t])
}
time.Sleep(50 * time.Millisecond)
fmt.Println("Unloading Trucks")
UnloadTrucks(ch)
fmt.Println("Done")
}
You never close the "truck" channel ch, so UnloadTrucks never returns.
You can close the channel after all workers are done, by using a WaitGroup:
go func() {
wg.Wait()
close(ch)
}()
UnloadTrucks(ch)
http://play.golang.org/p/1V7UbYpsQr
Related
How can I get a goroutine's runtime ID?
I'm getting interleaved logs from an imported package - one approach would be to add a unique identifier to the logs of each goroutine.
I've found some references to runtime.GoID:
func worker() {
id := runtime.GoID()
log.Println("Goroutine ID:", id)
}
But it looks like this is now outdated/has been removed - https://pkg.go.dev/runtime?
Go deliberately chooses not to provide an ID since it would encourage worse software and hurt the overall ecosystem: https://go.dev/doc/faq#no_goroutine_id
Generally, the desire to de-anonymize goroutines is a design flaw and is strongly not recommended. There is almost always going to be a much better way to solve the issue at hand. Eg, if you need a unique identifier, it should be passed into the function or potentially via context.Context.
However, internally the runtime needs IDs for the implementation. For educational purposes you can find them with something like:
package main
import (
"bytes"
"errors"
"fmt"
"runtime"
"strconv"
)
func main() {
fmt.Println(goid())
done := make(chan struct{})
go func() {
fmt.Println(goid())
done <- struct{}{}
}()
go func() {
fmt.Println(goid())
done <- struct{}{}
}()
<-done
<-done
}
var (
goroutinePrefix = []byte("goroutine ")
errBadStack = errors.New("invalid runtime.Stack output")
)
// This is terrible, slow, and should never be used.
func goid() (int, error) {
buf := make([]byte, 32)
n := runtime.Stack(buf, false)
buf = buf[:n]
// goroutine 1 [running]: ...
buf, ok := bytes.CutPrefix(buf, goroutinePrefix)
if !ok {
return 0, errBadStack
}
i := bytes.IndexByte(buf, ' ')
if i < 0 {
return 0, errBadStack
}
return strconv.Atoi(string(buf[:i]))
}
Example output:
1 <nil>
19 <nil>
18 <nil>
They can also be found (less portably) via assembly by accessing the goid field in the g struct. This is how packages like github.com/petermattis/goid typically do it.
Are there are any concurrency issues if we access mutually exclusive fields of a struct inside different go co-routines?
I remember reading somewhere that if two parallel threads access same object they might get run on different cores of the cpu both having different cpu level caches with different copies of the object in question. (Not related to Go)
Will the below code be sufficient to achieve the functionality correctly or does additional synchronization mechanism needs to be used?
package main
import (
"fmt"
"sync"
)
type structure struct {
x string
y string
}
func main() {
val := structure{}
wg := new(sync.WaitGroup)
wg.Add(2)
go func1(&val, wg)
go func2(&val, wg)
wg.Wait()
fmt.Println(val)
}
func func1(val *structure, wg *sync.WaitGroup) {
val.x = "Test 1"
wg.Done()
}
func func2(val *structure, wg *sync.WaitGroup) {
val.y = "Test 2"
wg.Done()
}
Edit: - for people who ask why not channels unfortunately this is not the actual code I am working on. Both the func has calls to different api and get the data in a struct, those struct has a pragma.DoNotCopy in them ask protobuf auto generator why they thought it was a good idea. So those data can't be sent over the channel, or else i have to create another struct to send the data over or ask the linter to stop complaining. Or i can send a pointer to the object but feel that is also sharing the memory.
You must synchronize when at least one of accesses to shared resources is a write.
Your code is doing write access, yes, but different struct fields have different memory locations. So you are not accessing shared variables.
If you run your program with the race detector, eg. go run -race main.go it will not print a warning.
Now add fmt.Println(val.y) in func1 and run again, it will print:
WARNING: DATA RACE
Write at 0x00c0000c0010 by goroutine 8:
... rest of race warning
The preferred way in Go would be to communicate memory as opposed to share the memory.
In practice that would mean you should use the Go channels as I show in this blogpost.
https://marcofranssen.nl/concurrency-in-go
If you really want to stick with sharing memory you will have to use a Mutex.
https://tour.golang.org/concurrency/9
However that would cause context switching and Go routines synchronization that slows down your program.
Example using channels
package main
import (
"fmt"
"time"
)
type structure struct {
x string
y string
}
func main() {
val := structure{}
c := make(chan structure)
go func1(c)
go func2(c)
func(c chan structure) {
for {
select {
case v, ok := <-c:
if !ok {
return
}
if v.x != "" {
fmt.Printf("Received %v\n", v)
val.x = v.x
}
if v.y != "" {
fmt.Printf("Received %v\n", v)
val.y = v.y
}
if val.x != "" && val.y != "" {
close(c)
}
}
}
}(c)
fmt.Printf("%v\n", val)
}
func func1(c chan<- structure) {
time.Sleep(1 * time.Second)
c <- structure{x: "Test 1"}
}
func func2(c chan<- structure) {
c <- structure{y: "Test 2"}
}
I don't understand why the deadlock is occurring in this code. I've tried several different things to get the deadlock to stop (several different versions using WorkGroup). This is my first day in Go, and I am pretty disappointed so far with the complexity of fairly simple and straightforward operations. I feel like I'm missing something big and obvious, but all of the docs I have found on this are seemingly very different from what, to me, is a very basic mode of operation. All of the docs use primitive types for channels (int, string) not more complex types, all with very basic for loops OR on they are on the other end of the spectrum where the functions are fairly complicated orchestrations.
I guess I'm really looking for a middle of the road sample of "this is how it's usually done" with goroutines.
package main
import "fmt"
//import "sync"
import "time"
type Item struct {
name string
}
type Truck struct {
Cargo []Item
name string
}
func UnloadTrucks(c chan Truck) {
for t := range c {
fmt.Printf("%s has %d items in cargo: %s\n", t.name, len(t.Cargo), t.Cargo[0].name)
}
}
func main() {
trucks := make([]Truck, 2)
ch := make(chan Truck)
for i, _ := range trucks {
trucks[i].name = fmt.Sprintf("Truck %d", i+1)
fmt.Printf("Building %s\n", trucks[i].name)
}
for t := range trucks {
go func(tr Truck) {
itm := Item{}
itm.name = "Groceries"
fmt.Printf("Loading %s\n", tr.name)
tr.Cargo = append(tr.Cargo, itm)
ch <- tr
}(trucks[t])
}
time.Sleep(50 * time.Millisecond)
fmt.Println("Unloading Trucks")
UnloadTrucks(ch)
fmt.Println("Done")
}
You never close the "truck" channel ch, so UnloadTrucks never returns.
You can close the channel after all workers are done, by using a WaitGroup:
go func() {
wg.Wait()
close(ch)
}()
UnloadTrucks(ch)
http://play.golang.org/p/1V7UbYpsQr
I see lots of tutorials and examples on how to make Go wait for x number of goroutines to finish, but what I'm trying to do is have ensure there are always x number running, so a new goroutine is launched as soon as one ends.
Specifically I have a few hundred thousand 'things to do' which is processing some stuff that is coming out of MySQL. So it works like this:
db, err := sql.Open("mysql", connection_string)
checkErr(err)
defer db.Close()
rows,err := db.Query(`SELECT id FROM table`)
checkErr(err)
defer rows.Close()
var id uint
for rows.Next() {
err := rows.Scan(&id)
checkErr(err)
go processTheThing(id)
}
checkErr(err)
rows.Close()
Currently that will launch several hundred thousand threads of processTheThing(). What I need is that a maximum of x number (we'll call it 20) goroutines are launched. So it starts by launching 20 for the first 20 rows, and from then on it will launch a new goroutine for the next id the moment that one of the current goroutines has finished. So at any point in time there are always 20 running.
I'm sure this is quite simple/standard, but I can't seem to find a good explanation on any of the tutorials or examples or how this is done.
You may find Go Concurrency Patterns article interesting, especially Bounded parallelism section, it explains the exact pattern you need.
You can use channel of empty structs as a limiting guard to control number of concurrent worker goroutines:
package main
import "fmt"
func main() {
maxGoroutines := 10
guard := make(chan struct{}, maxGoroutines)
for i := 0; i < 30; i++ {
guard <- struct{}{} // would block if guard channel is already filled
go func(n int) {
worker(n)
<-guard
}(i)
}
}
func worker(i int) { fmt.Println("doing work on", i) }
Here I think something simple like this will work :
package main
import "fmt"
const MAX = 20
func main() {
sem := make(chan int, MAX)
for {
sem <- 1 // will block if there is MAX ints in sem
go func() {
fmt.Println("hello again, world")
<-sem // removes an int from sem, allowing another to proceed
}()
}
}
Thanks to everyone for helping me out with this. However, I don't feel that anyone really provided something that both worked and was simple/understandable, although you did all help me understand the technique.
What I have done in the end is I think much more understandable and practical as an answer to my specific question, so I will post it here in case anyone else has the same question.
Somehow this ended up looking a lot like what OneOfOne posted, which is great because now I understand that. But OneOfOne's code I found very difficult to understand at first because of the passing functions to functions made it quite confusing to understand what bit was for what. I think this way makes a lot more sense:
package main
import (
"fmt"
"sync"
)
const xthreads = 5 // Total number of threads to use, excluding the main() thread
func doSomething(a int) {
fmt.Println("My job is",a)
return
}
func main() {
var ch = make(chan int, 50) // This number 50 can be anything as long as it's larger than xthreads
var wg sync.WaitGroup
// This starts xthreads number of goroutines that wait for something to do
wg.Add(xthreads)
for i:=0; i<xthreads; i++ {
go func() {
for {
a, ok := <-ch
if !ok { // if there is nothing to do and the channel has been closed then end the goroutine
wg.Done()
return
}
doSomething(a) // do the thing
}
}()
}
// Now the jobs can be added to the channel, which is used as a queue
for i:=0; i<50; i++ {
ch <- i // add i to the queue
}
close(ch) // This tells the goroutines there's nothing else to do
wg.Wait() // Wait for the threads to finish
}
Create channel for passing data to goroutines.
Start 20 goroutines that processes the data from channel in a loop.
Send the data to the channel instead of starting a new goroutine.
Grzegorz Żur's answer is the most efficient way to do it, but for a newcomer it could be hard to implement without reading code, so here's a very simple implementation:
type idProcessor func(id uint)
func SpawnStuff(limit uint, proc idProcessor) chan<- uint {
ch := make(chan uint)
for i := uint(0); i < limit; i++ {
go func() {
for {
id, ok := <-ch
if !ok {
return
}
proc(id)
}
}()
}
return ch
}
func main() {
runtime.GOMAXPROCS(4)
var wg sync.WaitGroup //this is just for the demo, otherwise main will return
fn := func(id uint) {
fmt.Println(id)
wg.Done()
}
wg.Add(1000)
ch := SpawnStuff(10, fn)
for i := uint(0); i < 1000; i++ {
ch <- i
}
close(ch) //should do this to make all the goroutines exit gracefully
wg.Wait()
}
playground
This is a simple producer-consumer problem, which in Go can be easily solved using channels to buffer the paquets.
To put it simple: create a channel that accept your IDs. Run a number of routines which will read from the channel in a loop then process the ID. Then run your loop that will feed IDs to the channel.
Example:
func producer() {
var buffer = make(chan uint)
for i := 0; i < 20; i++ {
go consumer(buffer)
}
for _, id := range IDs {
buffer <- id
}
}
func consumer(buffer chan uint) {
for {
id := <- buffer
// Do your things here
}
}
Things to know:
Unbuffered channels are blocking: if the item wrote into the channel isn't accepted, the routine feeding the item will block until it is
My example lack a closing mechanism: you must find a way to make the producer to wait for all consumers to end their loop before returning. The simplest way to do this is with another channel. I let you think about it.
I've wrote a simple package to handle concurrency for Golang. This package will help you limit the number of goroutines that are allowed to run concurrently:
https://github.com/zenthangplus/goccm
Example:
package main
import (
"fmt"
"goccm"
"time"
)
func main() {
// Limit 3 goroutines to run concurrently.
c := goccm.New(3)
for i := 1; i <= 10; i++ {
// This function have to call before any goroutine
c.Wait()
go func(i int) {
fmt.Printf("Job %d is running\n", i)
time.Sleep(2 * time.Second)
// This function have to when a goroutine has finished
// Or you can use `defer c.Done()` at the top of goroutine.
c.Done()
}(i)
}
// This function have to call to ensure all goroutines have finished
// after close the main program.
c.WaitAllDone()
}
Also can take a look here: https://github.com/LiangfengChen/goutil/blob/main/concurrent.go
The example can refer the test case.
func TestParallelCall(t *testing.T) {
format := "test:%d"
data := make(map[int]bool)
mutex := sync.Mutex{}
val, err := ParallelCall(1000, 10, func(pos int) (interface{}, error) {
mutex.Lock()
defer mutex.Unlock()
data[pos] = true
return pos, errors.New(fmt.Sprintf(format, pos))
})
for i := 0; i < 1000; i++ {
if _, ok := data[i]; !ok {
t.Errorf("TestParallelCall pos not found: %d", i)
}
if val[i] != i {
t.Errorf("TestParallelCall return value is not right (%d,%v)", i, val[i])
}
if err[i].Error() != fmt.Sprintf(format, i) {
t.Errorf("TestParallelCall error msg is not correct (%d,%v)", i, err[i])
}
}
}
I was trying to implement an example Go code for using returned channels from go routines without any "reading block" in the main function. Here, a fanIn function accepts channels from two other routines and return which got as input.
Here, the expected output is Random Outputs from two inner routines. But the actual output is always one "ann" followed by a "john", not at all random in any case.
Why am I not getting random output?
Go Playground: http://play.golang.org/p/46CiihtPwD
Actual output:
you say: ann,0
you say: john,0
you say: ann,1
you say: john,1
......
Code:
package main
import (
"fmt"
"time"
)
func main() {
final := fanIn(boring("ann"), boring("john"))
for i := 0; i < 100; i++ {
fmt.Println("you say:", <-final)
}
time.Sleep(4 * time.Second)
}
func boring(msg string) chan string {
c1 := make(chan string)
go func() {
for i := 0; ; i++ {
c1 <- fmt.Sprintf("%s,%d", msg, i)
time.Sleep(time.Second)
}
}()
return c1
}
func fanIn(input1, input2 <-chan string) chan string {
c := make(chan string)
go func() {
for {
c <- <-input1
}
}()
go func() {
for {
c <- <-input2
}
}()
return c
}
No particular reason, that's just how Go happens to schedule the relevant goroutines (basically, you're getting "lucky" that there's a pattern). You can't rely on it. If you really want an actual reliably random result, you'll have to manually mix in randomness somehow.
There's also the Multiplex function from https://github.com/eapache/channels/ (doc: https://godoc.org/github.com/eapache/channels#Multiplex) which does effectively the same thing as your fanIn function. I don't think it would behave any differently in terms of randomness though.