Say I have the following code (playground):
package main
import (
"fmt"
"sync"
)
func createStr() *string {
tmp := "foo"
return &tmp
}
func main() {
var (
s *string
wg sync.WaitGroup
)
go func() {
wg.Add(1)
defer wg.Done()
s = createStr()
}()
wg.Wait()
fmt.Printf("s after: %v", s)
}
I would have expected s to not equal nil.
However, if I add a small wait, I get s != nil (playground):
package main
import (
"fmt"
"sync"
"time"
)
func createStr() *string {
tmp := "foo"
return &tmp
}
func main() {
var (
s *string
wg sync.WaitGroup
)
go func() {
wg.Add(1)
defer wg.Done()
s = createStr()
}()
wg.Wait()
time.Sleep(time.Second)
fmt.Printf("s after: %v", s)
}
This just caused a bug in a program I wrote. What is happening with sync.WaitGroup that's not causing my program to wait for s to be assigned a string pointer in my go func?
Your placement of Add method for sync.WaitGroup is wrong. Do not use Add inside a goroutine (anonymous goroutine here) but use it in the goroutine (main goroutine here) that's going to wait for it.
A probable situation that was happening in your code was that wg.Wait() didn't wait as the goroutine's wg.Add(1) wasn't called yet and hence s == nil. The following code fixes the issue:
Go Playground
package main
import (
"fmt"
"sync"
)
func createStr() *string {
tmp := "foo"
return &tmp
}
func main() {
var (
s *string
wg sync.WaitGroup
)
// Use wg.Add() here
wg.Add(1)
go func() {
defer wg.Done()
s = createStr()
}()
wg.Wait()
fmt.Printf("s after: %v", *s)
}
Related
I'm trying to figure out why I have a dead lock with waitgroup.Wait()
package main
import (
"fmt"
"sync"
)
var wg sync.WaitGroup
func foo(c chan int, i int) {
defer wg.Done()
c <- i
}
func main() {
ch := make(chan int)
for i := 0; i < 10; i++ {
wg.Add(1)
go foo(ch, i)
}
wg.Wait()
close(ch)
for item := range ch {
fmt.Println(item)
}
}
When I run it like this, it prints fatal error: all goroutines are asleep - deadlock!
I tried to change ch to a buffered channel and that solved the problem. But I really want to know why is there a dead lock.
I've commented out the parts where your program's logic is not correct:
package main
import (
"fmt"
"sync"
)
var wg sync.WaitGroup
func foo(c chan int, i int) {
defer wg.Done()
c <- i
}
func main() {
ch := make(chan int) // unbuffered channel
for i := 0; i < 10; i++ {
wg.Add(1)
go foo(ch, i)
}
// wg.Wait is waiting for all goroutines to finish but that's
// only possible if the send to channel succeeds. In this case,
// it is not possible as your receiver "for item := range ch" is below
// this. Hence, a deadlock.
wg.Wait()
// Ideally, it should be the sender's duty to close the channel.
// And closing a channel before the receiver where the channel
// is unbuffered is not correct.
close(ch)
for item := range ch {
fmt.Println(item)
}
}
Corrected program:
package main
import (
"fmt"
"sync"
)
var wg sync.WaitGroup
func foo(c chan int, i int) {
defer wg.Done()
c <- i
}
func main() {
ch := make(chan int)
go func() {
for item := range ch {
fmt.Println(item)
}
}()
for i := 0; i < 10; i++ {
wg.Add(1)
go foo(ch, i)
}
wg.Wait()
close(ch)
}
Deadlock in channels communication between two packages - Golang.
I have two packages that are communicated by two channels. One is main and the other has a function. When I run it, I obtain a deadlock.
package main
import (
functionspackage "GoEjemplos/subFolder"
"fmt"
"sync"
)
func main() {
var wg sync.WaitGroup
ChannelSendData := make(chan functionspackage.FunctionStruct, 1)
defer close(ChannelSendData)
data := functionspackage.FunctionStruct{
FieldOne: 3.56,
FieldTwo: 23,
}
ChannelSendData <- data
wg.Add(1)
go functionspackage.FunctionExt(ChannelSendData, &wg)
recibe := <-functionspackage.ChannelOutFunct
fmt.Println("channelOut: ", recibe)
close(functionspackage.ChannelOutFunct)
wg.Wait()
}
The other package is
package functionspackage
import "sync"
type FunctionStruct struct {
FieldOne float64
FieldTwo int
}
var ChannelOutFunct chan float64
func FunctionExt(RecibeChan chan FunctionStruct, wg *sync.WaitGroup) (ChannelOutFunct chan float64) {
reciveData := <-RecibeChan
result := reciveData.FieldOne * float64(reciveData.FieldTwo)
ChannelOutFunct <- result
wg.Done()
return ChannelOutFunct
}
This is the deadlock.
PS C:\Go-Project\src\GoEjemplos> go run main.go
fatal error: all goroutines are asleep - deadlock!
goroutine 1 [chan receive (nil chan)]:
main.main()
C:/Go-Project/src/GoEjemplos/main.go:32 +0x13d
goroutine 19 [chan send (nil chan)]:
GoEjemplos/subFolder.FunctionExt(0xc0000d4000, 0xc0000a2070, 0xc0000c9f18)
C:/Go-Project/src/GoEjemplos/subFolder/functionsPackage.go:19 +0x85
created by main.main
C:/Go-Project/src/GoEjemplos/main.go:30 +0x11a
exit status 2
PS C:\Go-Project\src\GoEjemplos>
Could you explain it to me where is the problem?
Thanks!
The channel functionspackage.ChannelOutFunct is not initialized, thus it is a nil channel. Writing to a nil-channel or reading from a nil-channel will always block.
https://dave.cheney.net/2014/03/19/channel-axioms
I got it to work. This is the code
package main
import (
packagefunctions "GoEjemplos/subFolder"
"fmt"
"sync"
)
func main() {
var wg sync.WaitGroup
var ChannelSendData = make(chan packagefunctions.FunctionStruct, 0)
defer close(ChannelSendData)
var ChanReturn = make(chan float64)
defer close(ChanReturn)
data := packagefunctions.FunctionStruct{
FieldOne: 3.56,
FieldTwo: 2,
}
wg.Add(1)
go func() { ChannelSendData <- data }()
wg.Add(1)
go func() {
ChanReturn = packagefunctions.FunctionExt(ChannelSendData, &wg)
recibeChanReturn := <-ChanReturn
fmt.Println("channelOut: ", recibeChanReturn)
wg.Done()
}()
wg.Wait()
}
the other package is
package packagefunctions
import (
"fmt"
"sync"
)
type FunctionStruct struct {
FieldOne float64
FieldTwo int
}
func FunctionExt(ChanIn chan FunctionStruct, wg *sync.WaitGroup) chan float64 {
reciveData, ok := <-ChanIn
if ok == false {
fmt.Println("channel closed")
}
var result float64
result = reciveData.FieldOne * float64(reciveData.FieldTwo)
var ChannelReturn = make(chan float64, 1)
defer close(ChannelReturn)
ChannelReturn <- result
wg.Done()
return ChannelReturn
}
With the following code:
package main
import (
"github.com/davecgh/go-spew/spew"
"sync"
"time"
)
func callbackWithTimeout(cbFunc func() ([]byte, error), timeout time.Duration) {
defer wg.Done() //I don't want this function to know about sync.WaitGroup
time.Sleep(timeout)
d, e := cbFunc()
spew.Dump(d)
spew.Dump(e)
}
var wg sync.WaitGroup
func main() {
wg.Add(1)
go func() {
cbFunc := func() ([]byte, error) {
//I feel like I should be able to defer here instead
return nil, nil
}
callbackWithTimeout(cbFunc, time.Duration(4*time.Second))
}()
println("some line")
wg.Wait()
}
In function callbackWithTimeout, I don't want to use defer wg.Done() because it's not callbackWithTimeout()'s concern to wg.Done(). How do I go about implementing such a thing? i.e., remove any sync.WaitGroup in callbackWithTimeout? I have a bit of problem understanding the separation of concerns here as a callback'er function should not have to know about waitgroups but in this case it seems, I have no other choice?
I feel like it should be a caller's responsibility to wg.Done() (which in this case is the cbFunc) but lack any concise reference to documentation or ideas on how to implement it in Go because by definition, all a callback function does is call the function back. So, where I am doing it wrong?
Silly assumptions were made by yours truly during refactoring. Working code below. Many thanks.
package main
import (
"errors"
"github.com/davecgh/go-spew/spew"
"sync"
"time"
)
func callbackWithTimeout(cbFunc func() ([]byte, error), timeout time.Duration) {
time.Sleep(timeout)
d, e := cbFunc()
spew.Dump(d)
spew.Dump(e)
}
func main() {
var wg sync.WaitGroup
wg.Add(1)
go func() {
defer wg.Done()
callbackWithTimeout(func() ([]byte, error) {
b := []byte{1, 2, 3, 4}
e := errors.New("error123")
return b, e
}, time.Duration(2*time.Second))
}()
println("some line")
wg.Wait()
}
May be like this?
package main
import (
"sync"
"time"
"github.com/davecgh/go-spew/spew"
)
func callbackWithTimeout(cbFunc func() ([]byte, error), timeout time.Duration) {
time.Sleep(timeout)
d, e := cbFunc()
spew.Dump(d)
spew.Dump(e)
}
func main() {
var wg sync.WaitGroup
wg.Add(1)
go func() {
defer wg.Done() // move it here
cbFunc := func() ([]byte, error) {
//I feel like I should be able to defer here instead
return nil, nil
}
callbackWithTimeout(cbFunc, time.Duration(4*time.Second))
}()
println("some line")
wg.Wait()
}
Have loook at this contrived example:
package main
import "fmt"
func printElo() {
fmt.Printf("Elo\n")
}
func printHello() {
fmt.Printf("Hello\n")
}
func main() {
fmt.Printf("This will print.")
i := 0
for i < 10 {
go printElo()
go printHello()
i++
}
}
The output of this program would be just "This will print". Output of goroutines printElo() and printHello will not be emitted because, I guess, the main() function thread will finish before the goroutines have a chance to even start executing.
What is the idiomatic way to make similar code work in Golang and not terminate prematurely?
Simplest, cleanest and "scalable" way to do it is to use a sync.WaitGroup:
var wg = &sync.WaitGroup{}
func printElo() {
defer wg.Done()
fmt.Printf("Elo\n")
}
func printHello() {
defer wg.Done()
fmt.Printf("Hello\n")
}
func main() {
fmt.Printf("This will print.")
i := 0
for i < 10 {
wg.Add(1)
go printElo()
wg.Add(1)
go printHello()
i++
}
wg.Wait()
}
Output (try it on the Go Playground):
This will print.Hello
Elo
Hello
Elo
Hello
Elo
Hello
Elo
Hello
Elo
Hello
Elo
Hello
Elo
Hello
Elo
Hello
Elo
Hello
Elo
Simple "rules" to follow when doing it with sync.WaitGroup:
call WaitGroup.Add() in the "original" goroutine (that starts a new) before the go statement
recommended to call WaitGroup.Done() deferred, so it gets called even if the goroutine panics
if you want to pass WaitGroup to other functions (and not use a package level variable), you must pass a pointer to it, else the WaitGroup (which is a struct) would be copied, and the Done() method called on the copy wouldn't be observed on the original
As already mentioned sync.WaitGroup is a right way in production code. But when developing for test and debug purposes you can just add select{} statement at the end or the main().
func main(){
go routine()
...
select{}
}
main() then never returns and you would kill it with for example Ctrl-C. It isn't idiomatic, never used in production, but just very quick easy hack when developing.
You can use sync package and take a look at waitgroups. You can take a look at a working Goplayground I set up.
Essentially
package main
import (
"fmt"
"sync"
)
//Takes a reference to the wg and sleeps when work is done
func printElo(wg *sync.WaitGroup) {
fmt.Printf("Elo\n")
defer wg.Done()
}
//Takes a reference to the wg and sleeps when work is done
func printHello(wg *sync.WaitGroup) {
fmt.Printf("Hello\n")
defer wg.Done()
}
func main() {
//Create a new WaitGroup
var wg sync.WaitGroup
fmt.Println("This will print.")
for i := 0; i < 10; i++ {
//Add a new entry to the waitgroup
wg.Add(1)
//New Goroutine which takes a reference to the wg
go printHello(&wg)
//Add a new entry to the waitgroup
wg.Add(1)
//New Goroutine which takes a reference to the wg
go printElo(&wg)
}
//Wait until everything is done
wg.Wait()
}
If you want just to play with results you can use "hack" with waiting for input:
package main
import (
"fmt"
"bufio"
"os"
)
func printElo() {
fmt.Printf("Elo\n")
}
func printHello() {
fmt.Printf("Hello\n")
}
func main() {
fmt.Printf("This will print.")
i := 0
for i < 10 {
go printElo()
go printHello()
i++
}
reader := bufio.NewReader(os.Stdin)
reader.ReadString('\n')
}
If want to learn how to do synchronization read about sync package:
package main
import (
"fmt"
"sync"
)
var wg sync.WaitGroup
func printElo() {
fmt.Printf("Elo\n")
wg.Done()
}
func printHello() {
fmt.Printf("Hello\n")
wg.Done()
}
func main() {
fmt.Printf("This will print.")
i := 0
for i < 10 {
wg.Add(2)
go printElo()
go printHello()
i++
}
wg.Wait()
}
I spend my evening looking at how to fix this error but I haven't succeeded. When I run the program I have the following error : "all goroutines are asleep - deadlock!". I understand this is because the main program is exited before the routine has the possibility to do its tasks and I thought using sync.WaitGroup would help but not really :/
I want to set a number of routine and use channels to send urls in order to check http status code. I want to limit the number of concurrency call to a website. I've followed examples doing the same thing with string instead of struct and it worked.
Any help would be well appreciated :)
package main
import (
"fmt"
"sync"
"time"
)
const (
numPollers = 2 // number of Poller goroutines to launch
)
var urls = []string{
"http://www.google.com/",
"http://golang.org/",
"http://blog.golang.org/",
"http://golangtutorials.blogspot.fr",
"https://gobyexample.com/",
}
// Resource represents an HTTP URL to be polled by this program.
type Resource struct {
url string
}
func Poller(in <-chan *Resource, wg *sync.WaitGroup) {
//defer wg.Done()
for r := range in {
fmt.Printf("Finished: %v - %v\n", r.url, time.Now())
}
wg.Done()
}
func main() {
var wg sync.WaitGroup
pending := make(chan *Resource)
wg.Add(len(urls))
go Poller(pending, &wg)
go func() {
for _, url := range urls {
wg.Add(1)
fmt.Println("SENT > Pending url " + url)
pending <- &Resource{url: url}
}
}()
wg.Wait()
fmt.Printf("Finished all goroutines: %v\n", time.Now())
}
https://play.golang.org/p/B-HSiDo2Qg
First, you have too many calls to wg.Add(). You call that once for each goroutine you're running. See http://golang.org/pkg/sync/#WaitGroup. Second, you didn't close the channel after you were done writing to it. Here's a modified version of your code:
package main
import (
"fmt"
"sync"
"time"
)
const (
numPollers = 2 // number of Poller goroutines to launch
)
var urls = []string{
"http://www.google.com/",
"http://golang.org/",
"http://blog.golang.org/",
"http://golangtutorials.blogspot.fr",
"https://gobyexample.com/",
}
// Resource represents an HTTP URL to be polled by this program.
type Resource struct {
url string
}
func Poller(in <-chan *Resource, wg *sync.WaitGroup) {
defer wg.Done()
for r := range in {
fmt.Printf("Finished: %v - %v\n", r.url, time.Now())
}
}
func main() {
var wg sync.WaitGroup
pending := make(chan *Resource)
wg.Add(2)
go Poller(pending, &wg)
go func() {
defer close(pending)
defer wg.Done()
for _, url := range urls {
fmt.Println("SENT > Pending url " + url)
pending <- &Resource{url: url}
}
}()
wg.Wait()
fmt.Printf("Finished all goroutines: %v\n", time.Now())
}
and https://play.golang.org/p/ucUlZEZMZM
You forget to close channel, and your wait group is too long. This works for me: https://play.golang.org/p/yasIzaCbmQ