How can I get a goroutine's runtime ID?
I'm getting interleaved logs from an imported package - one approach would be to add a unique identifier to the logs of each goroutine.
I've found some references to runtime.GoID:
func worker() {
id := runtime.GoID()
log.Println("Goroutine ID:", id)
}
But it looks like this is now outdated/has been removed - https://pkg.go.dev/runtime?
Go deliberately chooses not to provide an ID since it would encourage worse software and hurt the overall ecosystem: https://go.dev/doc/faq#no_goroutine_id
Generally, the desire to de-anonymize goroutines is a design flaw and is strongly not recommended. There is almost always going to be a much better way to solve the issue at hand. Eg, if you need a unique identifier, it should be passed into the function or potentially via context.Context.
However, internally the runtime needs IDs for the implementation. For educational purposes you can find them with something like:
package main
import (
"bytes"
"errors"
"fmt"
"runtime"
"strconv"
)
func main() {
fmt.Println(goid())
done := make(chan struct{})
go func() {
fmt.Println(goid())
done <- struct{}{}
}()
go func() {
fmt.Println(goid())
done <- struct{}{}
}()
<-done
<-done
}
var (
goroutinePrefix = []byte("goroutine ")
errBadStack = errors.New("invalid runtime.Stack output")
)
// This is terrible, slow, and should never be used.
func goid() (int, error) {
buf := make([]byte, 32)
n := runtime.Stack(buf, false)
buf = buf[:n]
// goroutine 1 [running]: ...
buf, ok := bytes.CutPrefix(buf, goroutinePrefix)
if !ok {
return 0, errBadStack
}
i := bytes.IndexByte(buf, ' ')
if i < 0 {
return 0, errBadStack
}
return strconv.Atoi(string(buf[:i]))
}
Example output:
1 <nil>
19 <nil>
18 <nil>
They can also be found (less portably) via assembly by accessing the goid field in the g struct. This is how packages like github.com/petermattis/goid typically do it.
Related
Are there are any concurrency issues if we access mutually exclusive fields of a struct inside different go co-routines?
I remember reading somewhere that if two parallel threads access same object they might get run on different cores of the cpu both having different cpu level caches with different copies of the object in question. (Not related to Go)
Will the below code be sufficient to achieve the functionality correctly or does additional synchronization mechanism needs to be used?
package main
import (
"fmt"
"sync"
)
type structure struct {
x string
y string
}
func main() {
val := structure{}
wg := new(sync.WaitGroup)
wg.Add(2)
go func1(&val, wg)
go func2(&val, wg)
wg.Wait()
fmt.Println(val)
}
func func1(val *structure, wg *sync.WaitGroup) {
val.x = "Test 1"
wg.Done()
}
func func2(val *structure, wg *sync.WaitGroup) {
val.y = "Test 2"
wg.Done()
}
Edit: - for people who ask why not channels unfortunately this is not the actual code I am working on. Both the func has calls to different api and get the data in a struct, those struct has a pragma.DoNotCopy in them ask protobuf auto generator why they thought it was a good idea. So those data can't be sent over the channel, or else i have to create another struct to send the data over or ask the linter to stop complaining. Or i can send a pointer to the object but feel that is also sharing the memory.
You must synchronize when at least one of accesses to shared resources is a write.
Your code is doing write access, yes, but different struct fields have different memory locations. So you are not accessing shared variables.
If you run your program with the race detector, eg. go run -race main.go it will not print a warning.
Now add fmt.Println(val.y) in func1 and run again, it will print:
WARNING: DATA RACE
Write at 0x00c0000c0010 by goroutine 8:
... rest of race warning
The preferred way in Go would be to communicate memory as opposed to share the memory.
In practice that would mean you should use the Go channels as I show in this blogpost.
https://marcofranssen.nl/concurrency-in-go
If you really want to stick with sharing memory you will have to use a Mutex.
https://tour.golang.org/concurrency/9
However that would cause context switching and Go routines synchronization that slows down your program.
Example using channels
package main
import (
"fmt"
"time"
)
type structure struct {
x string
y string
}
func main() {
val := structure{}
c := make(chan structure)
go func1(c)
go func2(c)
func(c chan structure) {
for {
select {
case v, ok := <-c:
if !ok {
return
}
if v.x != "" {
fmt.Printf("Received %v\n", v)
val.x = v.x
}
if v.y != "" {
fmt.Printf("Received %v\n", v)
val.y = v.y
}
if val.x != "" && val.y != "" {
close(c)
}
}
}
}(c)
fmt.Printf("%v\n", val)
}
func func1(c chan<- structure) {
time.Sleep(1 * time.Second)
c <- structure{x: "Test 1"}
}
func func2(c chan<- structure) {
c <- structure{y: "Test 2"}
}
I don't understand why the deadlock is occurring in this code. I've tried several different things to get the deadlock to stop (several different versions using WorkGroup). This is my first day in Go, and I am pretty disappointed so far with the complexity of fairly simple and straightforward operations. I feel like I'm missing something big and obvious, but all of the docs I have found on this are seemingly very different from what, to me, is a very basic mode of operation. All of the docs use primitive types for channels (int, string) not more complex types, all with very basic for loops OR on they are on the other end of the spectrum where the functions are fairly complicated orchestrations.
I guess I'm really looking for a middle of the road sample of "this is how it's usually done" with goroutines.
package main
import "fmt"
//import "sync"
import "time"
type Item struct {
name string
}
type Truck struct {
Cargo []Item
name string
}
func UnloadTrucks(c chan Truck) {
for t := range c {
fmt.Printf("%s has %d items in cargo: %s\n", t.name, len(t.Cargo), t.Cargo[0].name)
}
}
func main() {
trucks := make([]Truck, 2)
ch := make(chan Truck)
for i, _ := range trucks {
trucks[i].name = fmt.Sprintf("Truck %d", i+1)
fmt.Printf("Building %s\n", trucks[i].name)
}
for t := range trucks {
go func(tr Truck) {
itm := Item{}
itm.name = "Groceries"
fmt.Printf("Loading %s\n", tr.name)
tr.Cargo = append(tr.Cargo, itm)
ch <- tr
}(trucks[t])
}
time.Sleep(50 * time.Millisecond)
fmt.Println("Unloading Trucks")
UnloadTrucks(ch)
fmt.Println("Done")
}
You never close the "truck" channel ch, so UnloadTrucks never returns.
You can close the channel after all workers are done, by using a WaitGroup:
go func() {
wg.Wait()
close(ch)
}()
UnloadTrucks(ch)
http://play.golang.org/p/1V7UbYpsQr
I have the following sample code. I want to maintain 4 goroutines running at all times. They have the possibility of panicking. In the case of the panic, I have a recover where I restart the goroutine.
The way I implemented works but I am not sure whether its the correct and proper way to do this. Any thoughts
package main
import (
"fmt"
"time"
)
var gVar string
var pCount int
func pinger(c chan int) {
for i := 0; ; i++ {
fmt.Println("adding ", i)
c <- i
}
}
func printer(id int, c chan int) {
defer func() {
if err := recover(); err != nil {
fmt.Println("HERE", id)
fmt.Println(err)
pCount++
if pCount == 5 {
panic("TOO MANY PANICS")
} else {
go printer(id, c)
}
}
}()
for {
msg := <-c
fmt.Println(id, "- ping", msg, gVar)
if msg%5 == 0 {
panic("PANIC")
}
time.Sleep(time.Second * 1)
}
}
func main() {
var c chan int = make(chan int, 2)
gVar = "Preflight"
pCount = 0
go pinger(c)
go printer(1, c)
go printer(2, c)
go printer(3, c)
go printer(4, c)
var input string
fmt.Scanln(&input)
}
You can extract the recover logic in a function such as:
func recoverer(maxPanics, id int, f func()) {
defer func() {
if err := recover(); err != nil {
fmt.Println("HERE", id)
fmt.Println(err)
if maxPanics == 0 {
panic("TOO MANY PANICS")
} else {
go recoverer(maxPanics-1, id, f)
}
}
}()
f()
}
And then use it like:
go recoverer(5, 1, func() { printer(1, c) })
Like Zan Lynx's answer, I'd like to share another way to do it (although it's pretty much similar to OP's way.) I used an additional buffered channel ch. When a goroutine panics, the recovery function inside the goroutine send its identity i to ch. In for loop at the bottom of main(), it detects which goroutine's in panic and whether to restart by receiving values from ch.
Run in Go Playground
package main
import (
"fmt"
"time"
)
func main() {
var pCount int
ch := make(chan int, 5)
f := func(i int) {
defer func() {
if err := recover(); err != nil {
ch <- i
}
}()
fmt.Printf("goroutine f(%v) started\n", i)
time.Sleep(1000 * time.Millisecond)
panic("goroutine in panic")
}
go f(1)
go f(2)
go f(3)
go f(4)
for {
i := <-ch
pCount++
if pCount >= 5 {
fmt.Println("Too many panics")
break
}
fmt.Printf("Detected goroutine f(%v) panic, will restart\n", i)
f(i)
}
}
Oh, I am not saying that the following is more correct than your way. It is just another way to do it.
Create another function, call it printerRecover or something like it, and do your defer / recover in there. Then in printer just loop on calling printerRecover. Add in function return values to check if you need the goroutine to exit for some reason.
The way you implemented is correct. Just for me the approach to maintain exactly 4 routines running at all times looks not much go_way, either handling routine's ID, either spawning in defer which may leads unpredictable stack due to closure. I don't think you can efficiently balance resource this way. Why don't you like to simple spawn worker when it needed
func main() {
...
go func(tasks chan int){ //multiplexer
for {
task = <-tasks //when needed
go printer(task) //just spawns handler
}
}(ch)
...
}
and let runtime do its job? This way things are done in stdlib listeners/servers and them known to be efficient enough. goroutines are very lightweight to spawn and runtime is quite smart to balance load. Sure you must to recover either way. It is my very personal opinion.
I have a simple concurrency use case in go, and I cannot figure out an elegant solution to my problem.
I want to write a method fetchAll that queries an unspecified number of resources from remote servers in parallel. If any of the fetches fails, I want to return that first error immediately.
My initial implementation leaks goroutines:
package main
import (
"fmt"
"math/rand"
"sync"
"time"
)
func fetchAll() error {
wg := sync.WaitGroup{}
errs := make(chan error)
leaks := make(map[int]struct{})
defer fmt.Println("these goroutines leaked:", leaks)
// run all the http requests in parallel
for i := 0; i < 4; i++ {
leaks[i] = struct{}{}
wg.Add(1)
go func(i int) {
defer wg.Done()
defer delete(leaks, i)
// pretend this does an http request and returns an error
time.Sleep(time.Duration(rand.Intn(100)) * time.Millisecond)
errs <- fmt.Errorf("goroutine %d's error returned", i)
}(i)
}
// wait until all the fetches are done and close the error
// channel so the loop below terminates
go func() {
wg.Wait()
close(errs)
}()
// return the first error
for err := range errs {
if err != nil {
return err
}
}
return nil
}
func main() {
fmt.Println(fetchAll())
}
Playground: https://play.golang.org/p/Be93J514R5
I know from reading https://blog.golang.org/pipelines that I can create a signal channel to cleanup the other threads. Alternatively, I could probably use context to accomplish it. But it seems like such a simple use case should have a simpler solution that I'm missing.
Using Error Group makes this even simpler. This automatically waits for all the supplied Go Routines to complete successfully, or cancels all those remaining in the case of any one routine returning an error (in which case that error is the one bubble back up to the caller).
package main
import (
"context"
"fmt"
"math/rand"
"time"
"golang.org/x/sync/errgroup"
)
func fetchAll(ctx context.Context) error {
errs, ctx := errgroup.WithContext(ctx)
// run all the http requests in parallel
for i := 0; i < 4; i++ {
errs.Go(func() error {
// pretend this does an http request and returns an error
time.Sleep(time.Duration(rand.Intn(100)) * time.Millisecond)
return fmt.Errorf("error in go routine, bailing")
})
}
// Wait for completion and return the first error (if any)
return errs.Wait()
}
func main() {
fmt.Println(fetchAll(context.Background()))
}
All but one of your goroutines are leaked, because they're still waiting to send to the errs channel - you never finish the for-range that empties it. You're also leaking the goroutine who's job is to close the errs channel, because the waitgroup is never finished.
(Also, as Andy pointed out, deleting from map is not thread-safe, so that'd need protection from a mutex.)
However, I don't think maps, mutexes, waitgroups, contexts etc. are even necessary here. I'd rewrite the whole thing to just use basic channel operations, something like the following:
package main
import (
"fmt"
"math/rand"
"time"
)
func fetchAll() error {
var N = 4
quit := make(chan bool)
errc := make(chan error)
done := make(chan error)
for i := 0; i < N; i++ {
go func(i int) {
// dummy fetch
time.Sleep(time.Duration(rand.Intn(100)) * time.Millisecond)
err := error(nil)
if rand.Intn(2) == 0 {
err = fmt.Errorf("goroutine %d's error returned", i)
}
ch := done // we'll send to done if nil error and to errc otherwise
if err != nil {
ch = errc
}
select {
case ch <- err:
return
case <-quit:
return
}
}(i)
}
count := 0
for {
select {
case err := <-errc:
close(quit)
return err
case <-done:
count++
if count == N {
return nil // got all N signals, so there was no error
}
}
}
}
func main() {
rand.Seed(time.Now().UnixNano())
fmt.Println(fetchAll())
}
Playground link: https://play.golang.org/p/mxGhSYYkOb
EDIT: There indeed was a silly mistake, thanks for pointing it out. I fixed the code above (I think...). I also added some randomness for added Realism™.
Also, I'd like to stress that there really are multiple ways to approach this problem, and my solution is but one way. Ultimately it comes down to personal taste, but in general, you want to strive towards "idiomatic" code - and towards a style that feels natural and easy to understand for you.
Here's a more complete example using errgroup suggested by joth. It shows processing successful data, and will exit on the first error.
https://play.golang.org/p/rU1v-Mp2ijo
package main
import (
"context"
"fmt"
"golang.org/x/sync/errgroup"
"math/rand"
"time"
)
func fetchAll() error {
g, ctx := errgroup.WithContext(context.Background())
results := make(chan int)
for i := 0; i < 4; i++ {
current := i
g.Go(func() error {
// Simulate delay with random errors.
time.Sleep(time.Duration(rand.Intn(100)) * time.Millisecond)
if rand.Intn(2) == 0 {
return fmt.Errorf("goroutine %d's error returned", current)
}
// Pass processed data to channel, or receive a context completion.
select {
case results <- current:
return nil
// Close out if another error occurs.
case <-ctx.Done():
return ctx.Err()
}
})
}
// Elegant way to close out the channel when the first error occurs or
// when processing is successful.
go func() {
g.Wait()
close(results)
}()
for result := range results {
fmt.Println("processed", result)
}
// Wait for all fetches to complete.
return g.Wait()
}
func main() {
fmt.Println(fetchAll())
}
As long as each goroutine completes, you won't leak anything. You should create the error channel as buffered with the buffer size equal to the number of goroutines so that the send operations on the channel won't block. Each goroutine should always send something on the channel when it finishes, whether it succeeds or fails. The loop at the bottom can then just iterate for the number of goroutines and return if it gets a non-nil error. You don't need the WaitGroup or the other goroutine that closes the channel.
I think the reason it appears that goroutines are leaking is that you return when you get the first error, so some of them are still running.
By the way, maps are not goroutine safe. If you share a map among goroutines and some of them are making changes to the map, you need to protect it with a mutex.
This answer includes the ability to get the responses back into doneData -
package main
import (
"fmt"
"math/rand"
"os"
"strconv"
)
var doneData []string // responses
func fetchAll(n int, doneCh chan bool, errCh chan error) {
partialDoneCh := make(chan string)
for i := 0; i < n; i++ {
go func(i int) {
if r := rand.Intn(100); r != 0 && r%10 == 0 {
// simulate an error
errCh <- fmt.Errorf("e33or for reqno=" + strconv.Itoa(r))
} else {
partialDoneCh <- strconv.Itoa(i)
}
}(i)
}
// mutation of doneData
for d := range partialDoneCh {
doneData = append(doneData, d)
if len(doneData) == n {
close(partialDoneCh)
doneCh <- true
}
}
}
func main() {
// rand.Seed(1)
var n int
var e error
if len(os.Args) > 1 {
if n, e = strconv.Atoi(os.Args[1]); e != nil {
panic(e)
}
} else {
n = 5
}
doneCh := make(chan bool)
errCh := make(chan error)
go fetchAll(n, doneCh, errCh)
fmt.Println("main: end")
select {
case <-doneCh:
fmt.Println("success:", doneData)
case e := <-errCh:
fmt.Println("failure:", e, doneData)
}
}
Execute using go run filename.go 50 where N=50 i.e amount of parallelism
I don't understand why the deadlock is occurring in this code. I've tried several different things to get the deadlock to stop (several different versions using WorkGroup). This is my first day in Go, and I am pretty disappointed so far with the complexity of fairly simple and straightforward operations. I feel like I'm missing something big and obvious, but all of the docs I have found on this are seemingly very different from what, to me, is a very basic mode of operation. All of the docs use primitive types for channels (int, string) not more complex types, all with very basic for loops OR on they are on the other end of the spectrum where the functions are fairly complicated orchestrations.
I guess I'm really looking for a middle of the road sample of "this is how it's usually done" with goroutines.
package main
import "fmt"
//import "sync"
import "time"
type Item struct {
name string
}
type Truck struct {
Cargo []Item
name string
}
func UnloadTrucks(c chan Truck) {
for t := range c {
fmt.Printf("%s has %d items in cargo: %s\n", t.name, len(t.Cargo), t.Cargo[0].name)
}
}
func main() {
trucks := make([]Truck, 2)
ch := make(chan Truck)
for i, _ := range trucks {
trucks[i].name = fmt.Sprintf("Truck %d", i+1)
fmt.Printf("Building %s\n", trucks[i].name)
}
for t := range trucks {
go func(tr Truck) {
itm := Item{}
itm.name = "Groceries"
fmt.Printf("Loading %s\n", tr.name)
tr.Cargo = append(tr.Cargo, itm)
ch <- tr
}(trucks[t])
}
time.Sleep(50 * time.Millisecond)
fmt.Println("Unloading Trucks")
UnloadTrucks(ch)
fmt.Println("Done")
}
You never close the "truck" channel ch, so UnloadTrucks never returns.
You can close the channel after all workers are done, by using a WaitGroup:
go func() {
wg.Wait()
close(ch)
}()
UnloadTrucks(ch)
http://play.golang.org/p/1V7UbYpsQr