I am trying to loop a slice of function and then invoke every function in it. However I am getting strange results. Here is my code:
package main
import (
"fmt"
"sync"
)
func A() {
fmt.Println("A")
}
func B() {
fmt.Println("B")
}
func C() {
fmt.Println("C")
}
func main() {
type fs func()
var wg sync.WaitGroup
f := []fs{A, B, C}
for a, _ := range f {
wg.Add(1)
go func() {
defer wg.Done()
f[a]()
}()
}
wg.Wait()
}
I was thinking that it will invoke function A,B and then C but my output gets only Cs.
C
C
C
Please suggest whats wrong and the logic behind it. Also how can I get desired behavior.
Go Playground
Classic go gotcha :)
Official Go FAQ
for a, _ := range f {
wg.Add(1)
a:=a // this will make it work
go func() {
defer wg.Done()
f[a]()
}()
}
Your func() {}() is a closure that closes over a. And a is a shared across all the go func go routines because for loop reuses the same var (meaning same address in memory, hence same value), so naturally they all see last value of a.
Solution is either re-declare a:=a before closure (like above). This will create new var (new address in memory) which is then new for each invocation of go func.
Or pass it in as parameter to the go function, in which case you pass a copy of value of a like so:
go func(i int) {
defer wg.Done()
f[i]()
}(a)
You don't even need to have go routines this https://play.golang.org/p/nkP9YfeOWF for example demonstrates the same gotcha. The key here is 'closure'.
The problem seems to be that you are not passing the desired value to the goroutine and the variable value is being taken from the outer scope. That being said, the range iteration finishes even before the first goroutine is executed and that is why you are always getting index a == 2, which is function C.
You can test this if you simply put time.Sleep(100) inside your range, just to allow the goroutine to catch up with the main thread before continuing to the next iteration --> GO playground
for a, _ := range f {
wg.Add(1)
go func() {
defer wg.Done()
f[a]()
}()
time.Sleep(100)
}
Output
A
B
C
Although what you want to do is just simply pass a parameter to the goroutine which will make a copy for the function.
func main() {
type fs func()
var wg sync.WaitGroup
f := []fs{A, B, C}
for _, v := range f {
wg.Add(1)
go func(f fs) {
defer wg.Done()
f()
}(v)
}
wg.Wait()
}
GO Playground
Related
I would like to implement a parallelization function, which in the absence of generics, looks like so:
func Parallelize(s []interface{}, c chan interface{}, f func(interface{}, chan interface{})) {
var wg sync.WaitGroup
defer wg.Done()
for _, si := range s {
wg.Add(1)
go f(si, c)
}
wg.Wait()
close(c)
}
I'd like to enable passing objects of any type, but to ensure that the first argument is a slice of objects, second is a channel and third is a function that accepts an object and channel.
Apparently, the go compiler doesn't like the arguments. It is not allowing me to call this function like so:
a := make([]*A)
c := make(chan *A)
f := func(_a *A, _c chan A) {
...
}
Parallelize(a, c, f)
What is the right way of going about this?
There are several ways you can do this, though the best in my opinion is not to do it. This is one of those common patterns that is best left explicit because it is much easier to read and maintain. However, if you insist:
One way to do it is realizing that you don't really need to pass the slice elements:
func Parallelize(n int, c chan interface{}, f func(int, chan interface{})) {
var wg sync.WaitGroup
defer wg.Done()
for i:=0;i<n;i++ {
wg.Add(1)
go f(i, c)
}
wg.Wait()
close(c)
}
And call it using:
Parallelize(len(slice), ch, func(i int,ch chan interface{}) {
// use slice[i]
})
You also don't need to pass the channel:
func Parallelize(n int, f func(int)) {
var wg sync.WaitGroup
defer wg.Done()
for i:=0;i<n;i++ {
wg.Add(1)
go f(i)
}
wg.Wait()
}
And call it:
Parallelize(len(slice), func(i int) {
// use slice[i] and chan ch
})`
close(ch)
Another way of doing this is using reflection. It'll be uglier, and you'll have to deal with runtime errors instead of compile time errors.
I have a function that recursively spawns goroutines to walk a DOM tree, putting the nodes they find into a channel shared between all of them.
import (
"golang.org/x/net/html"
"sync"
)
func walk(doc *html.Node, ch chan *html.Node) {
var wg sync.WaitGroup
defer close(ch)
var f func(*html.Node)
f = func(n *html.Node) {
defer wg.Done()
ch <- n
for c := n.FirstChild; c != nil; c = c.NextSibling {
wg.Add(1)
go f(c)
}
}
wg.Add(1)
go f(doc)
wg.Wait()
}
Which I'd call like
// get the webpage using http
// parse the html into doc
ch := make(chan *html.Node)
go walk(doc, ch)
for c := range ch {
if someCondition(c) {
// do something with c
// quit all goroutines spawned by walk
}
}
I am wondering how I could quit all of these goroutines--i.e. close ch--once I have found a node of a certain type or some other condition has been fulfilled. I have tried using a quit channel that'd be polled before spawning the new goroutines and close ch if a value was received but that lead to race conditions where some goroutines tried sending on the channel that had just been closed by another one. I was pondering using a mutex but it seems inelegant and against the spirit of go to protect a channel with a mutex. Is there an idiomatic way to do this using channels? If not, is there any way at all? Any input appreciated!
The context package provides similar functionality. Using context.Context with a few Go-esque patterns, you can achieve what you need.
To start you can check this article to get a better feel of cancellation with context: https://www.sohamkamani.com/blog/golang/2018-06-17-golang-using-context-cancellation/
Also make sure to check the official GoDoc: https://golang.org/pkg/context/
So to achieve this functionality your function should look more like:
func walk(ctx context.Context, doc *html.Node, ch chan *html.Node) {
var wg sync.WaitGroup
defer close(ch)
var f func(*html.Node)
f = func(n *html.Node) {
defer wg.Done()
ch <- n
for c := n.FirstChild; c != nil; c = c.NextSibling {
select {
case <-ctx.Done():
return // quit the function as it is cancelled
default:
wg.Add(1)
go f(c)
}
}
}
select {
case <-ctx.Done():
return // perhaps it was cancelled so quickly
default:
wg.Add(1)
go f(doc)
wg.Wait()
}
}
And when calling the function, you will have something like:
// ...
ctx, cancelFunc := context.WithCancel(context.Background())
walk(ctx, doc, ch)
for value := range ch {
// ...
if someCondition {
cancelFunc()
// the for loop will automatically exit as the channel is being closed for the inside
}
}
I'm using goroutines in my project and I want to to assign the values to the struct fields but I don't know that how I will assign the values get by using mongodb quires to the struct fields I'm showing my struct and the query too.
type AppLoadNew struct{
StripeTestKey string `json:"stripe_test_key" bson:"stripe_test_key,omitempty"`
Locations []Locations `json:"location" bson:"location,omitempty"`
}
type Locations struct{
Id int `json:"_id" bson:"_id"`
Location string `json:"location" bson:"location"`
}
func GoRoutine(){
values := AppLoadNew{}
go func() {
data, err := GetStripeTestKey(bson.M{"is_default": true})
if err == nil {
values.StripeTestKey := data.TestStripePublishKey
}
}()
go func() {
location, err := GetFormLocation(bson.M{"is_default": true})
if err == nil {
values.Locations := location
}
}()
fmt.Println(values) // Here it will nothing
// empty
}
Can you please help me that I will assign all the values to the AppLoadNew struct.
In Go no value is safe for concurrent read and write (from multiple goroutines). You must synchronize access.
Reading and writing variables from multiple goroutines can be protected using sync.Mutex or sync.RWMutex, but in your case there is something else involved: you should wait for the 2 launched goroutines to complete. For that, the go-to solution is sync.WaitGroup.
And since the 2 goroutines write 2 different fields of a struct (which act as 2 distinct variables), they don't have to be synchronized to each other (see more on this here: Can I concurrently write different slice elements). Which means using a sync.WaitGroup is sufficient.
This is how you can make it safe and correct:
func GoRoutine() {
values := AppLoadNew{}
wg := &sync.WaitGroup{}
wg.Add(1)
go func() {
defer wg.Done()
data, err := GetStripeTestKey(bson.M{"is_default": true})
if err == nil {
values.StripeTestKey = data.StripeTestKey
}
}()
wg.Add(1)
go func() {
defer wg.Done()
location, err := GetFormLocation(bson.M{"is_default": true})
if err == nil {
values.Locations = location
}
}()
wg.Wait()
fmt.Println(values)
}
See a (slightly modified) working example on the Go Playground.
See related / similar questions:
Reading values from a different thread
golang struct concurrent read and write without Lock is also running ok?
How to make a variable thread-safe
You can use sync package with WaitGroup, here is an example:
package main
import (
"fmt"
"sync"
"time"
)
type Foo struct {
One string
Two string
}
func main() {
f := Foo{}
var wg sync.WaitGroup
wg.Add(1)
go func() {
defer wg.Done()
// Perform long calculations
<-time.After(time.Second * 1)
f.One = "foo"
}()
wg.Add(1)
go func() {
defer wg.Done()
// Perform long calculations
<-time.After(time.Second * 2)
f.Two = "bar"
}()
fmt.Printf("Before %+v\n", f)
wg.Wait()
fmt.Printf("After %+v\n", f)
}
The output:
Before {One: Two:}
After {One:foo Two:bar}
I have an issue. Here is example: https://play.golang.org/p/QSWY2INQuSE
func Avg(c chan string, wg *sync.WaitGroup) {
defer wg.Done()
c <- "test"
}
func main() {
var wg sync.WaitGroup
c := make(chan string)
timer1 := time.NewTicker(5 * time.Second)
for {
select {
case <-timer1.C:
wg.Add(1)
go Avg(c, &wg)
wg.Wait()
}
}
fmt.Println(<-c)
}
Why data does not reach fmt.Println(<-c)
Thank you!
Because you have an endless for, so the last fmt.Println() statement is never reached.
You have to break out of the loop if you want the last fmt.Println() statement to ever execute, for example:
loop:
for {
select {
case <-timer1.C:
wg.Add(1)
go Avg(c, &wg)
wg.Wait()
break loop
}
}
fmt.Println(<-c)
Note that you have to use a label, else the break would only break out of the select statement (and not from the for loop).
Also note that this alone won't work, as the channel is unbuffered, and thus Avg() will be blocked forever, trying to send a value on c while noone is ever trying to receive from it.
This simple example can be made working if you create the channel to be buffered:
c := make(chan string, 1) // Buffer for 1 value
Now it works and prints (try it on the Go Playground):
test
I have some issues with the following code:
package main
import (
"fmt"
"sync"
)
// This program should go to 11, but sometimes it only prints 1 to 10.
func main() {
ch := make(chan int)
var wg sync.WaitGroup
wg.Add(2)
go Print(ch, wg) //
go func(){
for i := 1; i <= 11; i++ {
ch <- i
}
close(ch)
defer wg.Done()
}()
wg.Wait() //deadlock here
}
// Print prints all numbers sent on the channel.
// The function returns when the channel is closed.
func Print(ch <-chan int, wg sync.WaitGroup) {
for n := range ch { // reads from channel until it's closed
fmt.Println(n)
}
defer wg.Done()
}
I get a deadlock at the specified place. I have tried setting wg.Add(1) instead of 2 and it solves my problem. My belief is that I'm not successfully sending the channel as an argument to the Printer function. Is there a way to do that? Otherwise, a solution to my problem is replacing the go Print(ch, wg)line with:
go func() {
Print(ch)
defer wg.Done()
}
and changing the Printer function to:
func Print(ch <-chan int) {
for n := range ch { // reads from channel until it's closed
fmt.Println(n)
}
}
What is the best solution?
Well, first your actual error is that you're giving the Print method a copy of the sync.WaitGroup, so it doesn't call the Done() method on the one you're Wait()ing on.
Try this instead:
package main
import (
"fmt"
"sync"
)
func main() {
ch := make(chan int)
var wg sync.WaitGroup
wg.Add(2)
go Print(ch, &wg)
go func() {
for i := 1; i <= 11; i++ {
ch <- i
}
close(ch)
defer wg.Done()
}()
wg.Wait() //deadlock here
}
func Print(ch <-chan int, wg *sync.WaitGroup) {
for n := range ch { // reads from channel until it's closed
fmt.Println(n)
}
defer wg.Done()
}
Now, changing your Print method to remove the WaitGroup of it is a generally good idea: the method doesn't need to know something is waiting for it to finish its job.
I agree with #Elwinar's solution, that the main problem in your code caused by passing a copy of your Waitgroup to the Print function.
This means the wg.Done() is operated on a copy of wg you defined in the main. Therefore, wg in the main could not get decreased, and thus a deadlock happens when you wg.Wait() in main.
Since you are also asking about the best practice, I could give you some suggestions of my own:
Don't remove defer wg.Done() in Print. Since your goroutine in main is a sender, and print is a receiver, removing wg.Done() in receiver routine will cause an unfinished receiver. This is because only your sender is synced with your main, so after your sender is done, your main is done, but it's possible that the receiver is still working. My point is: don't leave some dangling goroutines around after your main routine is finished. Close them or wait for them.
Remember to do panic recovery everywhere, especially anonymous goroutine. I have seen a lot of golang programmers forgetting to put panic recovery in goroutines, even if they remember to put recover in normal functions. It's critical when you want your code to behave correctly or at least gracefully when something unexpected happened.
Use defer before every critical calls, like sync related calls, at the beginning since you don't know where the code could break. Let's say you removed defer before wg.Done(), and a panic occurrs in your anonymous goroutine in your example. If you don't have panic recover, it will panic. But what happens if you have a panic recover? Everything's fine now? No. You will get deadlock at wg.Wait() since your wg.Done() gets skipped because of panic! However, by using defer, this wg.Done() will be executed at the end, even if panic happened. Also, defer before close is important too, since its result also affects the communication.
So here is the code modified according to the points I mentioned above:
package main
import (
"fmt"
"sync"
)
func main() {
ch := make(chan int)
var wg sync.WaitGroup
wg.Add(2)
go Print(ch, &wg)
go func() {
defer func() {
if r := recover(); r != nil {
println("panic:" + r.(string))
}
}()
defer func() {
wg.Done()
}()
for i := 1; i <= 11; i++ {
ch <- i
if i == 7 {
panic("ahaha")
}
}
println("sender done")
close(ch)
}()
wg.Wait()
}
func Print(ch <-chan int, wg *sync.WaitGroup) {
defer func() {
if r := recover(); r != nil {
println("panic:" + r.(string))
}
}()
defer wg.Done()
for n := range ch {
fmt.Println(n)
}
println("print done")
}
Hope it helps :)