I wrote a short script to write a file concurrently.
One goroutine is supposed to write strings to a file while the others are supposed to send the messages through a channel to it.
However, for some really strange reason the file is created but no message is added to it through the channel.
package main
import (
"fmt"
"os"
"sync"
)
var wg sync.WaitGroup
var output = make(chan string)
func concurrent(n uint64) {
output <- fmt.Sprint(n)
defer wg.Done()
}
func printOutput() {
f, err := os.OpenFile("output.txt", os.O_CREATE|os.O_RDWR|os.O_APPEND, 0666);
if err != nil {
panic(err)
}
defer f.Close()
for msg := range output {
f.WriteString(msg+"\n")
}
}
func main() {
wg.Add(2)
go concurrent(1)
go concurrent(2)
wg.Wait()
close(output)
printOutput()
}
The printOutput() goroutine is executed completely, if I tried to write something after the for loop it would actually get into the file. So this leads me to think that range output might be null
You need to have something taking from the output channel as it is blocking until something removes what you put on it.
Not the only/best way to do it but: I moved printOutput() to above the other funcs and run it as a go routine and it prevents the deadlock.
package main
import (
"fmt"
"os"
"sync"
)
var wg sync.WaitGroup
var output = make(chan string)
func concurrent(n uint64) {
output <- fmt.Sprint(n)
defer wg.Done()
}
func printOutput() {
f, err := os.OpenFile("output.txt", os.O_CREATE|os.O_RDWR|os.O_APPEND, 0666)
if err != nil {
panic(err)
}
defer f.Close()
for msg := range output {
f.WriteString(msg + "\n")
}
}
func main() {
go printOutput()
wg.Add(2)
go concurrent(1)
go concurrent(2)
wg.Wait()
close(output)
}
One of the reason why you get a null output is because channels are blocking for both send/receive.
According to your flow, the code snippet below will never reach wg.Done(), as sending channel is expecting a receiving end to pull the data out. This is a typical deadlock example.
func concurrent(n uint64) {
output <- fmt.Sprint(n) // go routine is blocked until data in channel is fetched.
defer wg.Done()
}
Let's examine the main func:
func main() {
wg.Add(2)
go concurrent(1)
go concurrent(2)
wg.Wait() // the main thread will be waiting indefinitely here.
close(output)
printOutput()
}
My take on the problem:
package main
import (
"fmt"
"os"
"sync"
)
var wg sync.WaitGroup
var output = make(chan string)
var donePrinting = make(chan struct{})
func concurrent(n uint) {
defer wg.Done() // It only makes sense to defer
// wg.Done() before you do something.
// (like sending a string to the output channel)
output <- fmt.Sprint(n)
}
func printOutput() {
f, err := os.OpenFile("output.txt", os.O_CREATE|os.O_RDWR|os.O_APPEND, 0666)
if err != nil {
panic(err)
}
defer f.Close()
for msg := range output {
f.WriteString(msg + "\n")
}
donePrinting <- struct{}{}
}
func main() {
wg.Add(2)
go printOutput()
go concurrent(1)
go concurrent(2)
wg.Wait()
close(output)
<-donePrinting
}
Each concurrent function will deduct from the wait-group.
After the two concurrent goroutines finish, the wg.Wait() will unblock, and the next instruction (close(output)) will be executed. You have to wait for the two goroutines to finish before closing the channel. If, instead, you try the following:
go printOutput()
go concurrent(1)
go concurrent(2)
close(output)
wg.Wait()
you could end up with the close(output) instruction executing before any one of the concurrent goroutines concludes. If the channel closes before the concurrent goroutines run, they will crash at runtime, (while trying to write to a closed channel).
If, then, you don’t wait up for the printOutput() goroutine to finish, you could actually quit main() before printOutput() has gotten the chance to finish writing to its file.
Because I want to wait for the printOutput() goroutine to finish before I quit the program, I also created a separate channel just to signal that printOutput() is done.
The <-donePrinting instruction blocks until main receives something over the donePrinting channel.
Once main receives anything (even the empty structure that printOutput() sends), it will unblock and run to conclusion.
https://play.golang.org/p/nXJoYLI758m
Related
I found some example with "Catch values from Goroutines"-> link
There is show how to fetch value but if I want to return value from several goroutines, it wont work.So, does anybody know, how to do it?
package main
import (
"fmt"
"io/ioutil"
"log"
"net/http"
"sync"
)
// WaitGroup is used to wait for the program to finish goroutines.
var wg sync.WaitGroup
func responseSize(url string, nums chan int) {
// Schedule the call to WaitGroup's Done to tell goroutine is completed.
defer wg.Done()
response, err := http.Get(url)
if err != nil {
log.Fatal(err)
}
defer response.Body.Close()
body, err := ioutil.ReadAll(response.Body)
if err != nil {
log.Fatal(err)
}
// Send value to the unbuffered channel
nums <- len(body)
}
func main() {
nums := make(chan int) // Declare a unbuffered channel
wg.Add(1)
go responseSize("https://www.golangprograms.com", nums)
go responseSize("https://gobyexample.com/worker-pools", nums)
go responseSize("https://stackoverflow.com/questions/ask", nums)
fmt.Println(<-nums) // Read the value from unbuffered channel
wg.Wait()
close(nums) // Closes the channel
// >> loading forever
Also, this example, worker pools
Is it possible to get value from result: fmt.Println(<-results) <- will be error.
Yes, just read from the channel multiple times:
answerOne := <-nums
answerTwo := <-nums
answerThree := <-nums
Channels function like thread-safe queues, allowing you to queue up values and read them out one by one
P.S. You should either add 3 to the wait group or not have one at all. The <-nums will block until a value is available on nums so it is not necessary
The following piece of code try to send to the channel on the main goroutine and receive from another goroutine but a few times it returns as expected but a few times it exits without printing any on the console screen
package main
import "fmt"
func main() {
ch := make(chan bool)
go func() {
data := <-ch
fmt.Printf("Received: %t", data)
}()
ch <- true
}
At the same time, the following piece of code works as expected everytime, one difference is that an additional check has been added to check if the channel is closed or not which always throws the same expected output.
Does this ensure that a check on the channel is a must than optional ? or anything wrong with the code
package main
import "fmt"
func main() {
ch := make(chan bool)
go func() {
data, ok := <-ch
if !ok {
fmt.Println("Channel closed")
return
}
fmt.Printf("Received: %t", data)
}()
ch <- true
}
You should wait for goroutine to complete before main routine exit.
package main
import (
"fmt"
"sync"
)
func main() {
ch := make(chan bool)
var wg sync.WaitGroup
wg.Add(1)
go func() {
defer wg.Done()
data := <-ch
fmt.Printf("Received: %t", data)
}()
ch <- true
wg.Wait()
}
The thing is your second piece of code doesn't print Received: true every time. I tested it several times.
As #jub0bs mentioned there is no guarantee that your goroutine finishes before the main routine. You must control it yourself.
I recently looked at go and got hooked, it looks so interesting! After completing the tutorial I wanted to build something by myself: I want to list all of my songs from my music library. I think I can profit from go's concurrency here. While on routine is walking down the directory tree it pushes music files (path to those files) into a channel which are then picked up by another routine that reads the ID3 tags, so I don't have to wait until every file has been found.
This is my simple and naive approach:
package main
import (
"fmt"
"os"
"path/filepath"
"strings"
"sync"
)
const searchPath = "/Users/luma/Music/test" // 5GB of music.
func main() {
files := make(chan string)
var wg sync.WaitGroup
wg.Add(2)
go printHashes(files, &wg)
go searchFiles(searchPath, files, &wg)
wg.Wait()
}
func searchFiles(searchPath string, files chan<- string, wg *sync.WaitGroup) {
visit := func(path string, f os.FileInfo, err error) error {
if !f.IsDir() && strings.Contains(".mp4.mp3.flac", filepath.Ext(f.Name())) {
files <- path
}
return err
}
if err := filepath.Walk(searchPath, visit); err != nil {
fmt.Println(err)
}
wg.Done()
}
func printHashes(files <-chan string, wg *sync.WaitGroup) {
for range files {
fmt.Println(<-files)
}
wg.Done()
}
This program doesn't read the tags, yet. Instead it just prints the file path. This works, it lists all music files extremely fast! But I see this error after the program finishes:
fatal error: all goroutines are asleep - deadlock!
goroutine 1 [semacquire]:
sync.runtime_Semacquire(0xc42007205c)
/usr/local/Cellar/go/1.7.4_2/libexec/src/runtime/sema.go:47 +0x30
sync.(*WaitGroup).Wait(0xc420072050)
/usr/local/Cellar/go/1.7.4_2/libexec/src/sync/waitgroup.go:131 +0x97
main.main()
/Users/luma/Code/Go/src/github.com/LuMa/test/main.go:22 +0xfa
goroutine 17 [chan receive]:
main.printHashes(0xc42008e000, 0xc420072050)
/Users/luma/Code/Go/src/github.com/LuMa/test/main.go:42 +0xb4
created by main.main
/Users/luma/Code/Go/src/github.com/LuMa/test/main.go:19 +0xab
exit status 2
What is causing the deadlock?
Because you need close files channel.
In your case, you don't close it, so
for range files {
fmt.Println(<-files)
} will wait get value from files channel. so wg.Done() will never done in printHashes.
func searchFiles(searchPath string, files chan<- string, wg *sync.WaitGroup) {
visit := func(path string, f os.FileInfo, err error) error {
if !f.IsDir() && strings.Contains(".mp4.mp3.flac", filepath.Ext(f.Name())) {
files <- path
}
return err
}
if err := filepath.Walk(searchPath, visit); err != nil {
fmt.Println(err)
}
wg.Done()
close(files) // close the chanel, because you don't put thing into the channel anymore.
}
Within searchFiles, you want to close(files) when done sending. This convention is called sender-closes (receivers never close). Also, remove the call to wg.Done() as you are not done... There could still be items on the channel.
The close(files) will signal the for range files to close and exit the loop, which will call your wg.Done() to signal the main function that everything is done.
(Untested on mobile)
package main
import (
"fmt"
"os"
"path/filepath"
"strings"
"sync"
)
const searchPath = "/Users/luma/Music/test" // 5GB of music.
func main() {
files := make(chan string)
var wg sync.WaitGroup
wg.Add(1)
go printHashes(files)
go searchFiles(searchPath, files, &wg)
wg.Wait()
}
func searchFiles(searchPath string, files chan<- string) {
visit := func(path string, f os.FileInfo, err error) error {
if !f.IsDir() && strings.Contains(".mp4.mp3.flac", filepath.Ext(f.Name())) {
files <- path
}
return err
}
if err := filepath.Walk(searchPath, visit); err != nil {
fmt.Println(err)
}
close(files)
}
func printHashes(files <-chan string, wg *sync.WaitGroup) {
defer wg.Done()
for range files {
fmt.Println(<-files)
}
}
Note that while this may seem fast, using a single goroutine is fine and unblocks the main goroutine too. But, you may not gain any advantage if you try to read multiple files for id3 tags in multiple goroutines - they will all share the same file i/o lock at the syscall level. The only way that would advantageous would be if the processing of data far out weighs the file i/o locking (e.g. something big in computation, because processing is far faster than syscall locks).
PS, welcome to the Go community!
I have some issues with the following code:
package main
import (
"fmt"
"sync"
)
// This program should go to 11, but sometimes it only prints 1 to 10.
func main() {
ch := make(chan int)
var wg sync.WaitGroup
wg.Add(2)
go Print(ch, wg) //
go func(){
for i := 1; i <= 11; i++ {
ch <- i
}
close(ch)
defer wg.Done()
}()
wg.Wait() //deadlock here
}
// Print prints all numbers sent on the channel.
// The function returns when the channel is closed.
func Print(ch <-chan int, wg sync.WaitGroup) {
for n := range ch { // reads from channel until it's closed
fmt.Println(n)
}
defer wg.Done()
}
I get a deadlock at the specified place. I have tried setting wg.Add(1) instead of 2 and it solves my problem. My belief is that I'm not successfully sending the channel as an argument to the Printer function. Is there a way to do that? Otherwise, a solution to my problem is replacing the go Print(ch, wg)line with:
go func() {
Print(ch)
defer wg.Done()
}
and changing the Printer function to:
func Print(ch <-chan int) {
for n := range ch { // reads from channel until it's closed
fmt.Println(n)
}
}
What is the best solution?
Well, first your actual error is that you're giving the Print method a copy of the sync.WaitGroup, so it doesn't call the Done() method on the one you're Wait()ing on.
Try this instead:
package main
import (
"fmt"
"sync"
)
func main() {
ch := make(chan int)
var wg sync.WaitGroup
wg.Add(2)
go Print(ch, &wg)
go func() {
for i := 1; i <= 11; i++ {
ch <- i
}
close(ch)
defer wg.Done()
}()
wg.Wait() //deadlock here
}
func Print(ch <-chan int, wg *sync.WaitGroup) {
for n := range ch { // reads from channel until it's closed
fmt.Println(n)
}
defer wg.Done()
}
Now, changing your Print method to remove the WaitGroup of it is a generally good idea: the method doesn't need to know something is waiting for it to finish its job.
I agree with #Elwinar's solution, that the main problem in your code caused by passing a copy of your Waitgroup to the Print function.
This means the wg.Done() is operated on a copy of wg you defined in the main. Therefore, wg in the main could not get decreased, and thus a deadlock happens when you wg.Wait() in main.
Since you are also asking about the best practice, I could give you some suggestions of my own:
Don't remove defer wg.Done() in Print. Since your goroutine in main is a sender, and print is a receiver, removing wg.Done() in receiver routine will cause an unfinished receiver. This is because only your sender is synced with your main, so after your sender is done, your main is done, but it's possible that the receiver is still working. My point is: don't leave some dangling goroutines around after your main routine is finished. Close them or wait for them.
Remember to do panic recovery everywhere, especially anonymous goroutine. I have seen a lot of golang programmers forgetting to put panic recovery in goroutines, even if they remember to put recover in normal functions. It's critical when you want your code to behave correctly or at least gracefully when something unexpected happened.
Use defer before every critical calls, like sync related calls, at the beginning since you don't know where the code could break. Let's say you removed defer before wg.Done(), and a panic occurrs in your anonymous goroutine in your example. If you don't have panic recover, it will panic. But what happens if you have a panic recover? Everything's fine now? No. You will get deadlock at wg.Wait() since your wg.Done() gets skipped because of panic! However, by using defer, this wg.Done() will be executed at the end, even if panic happened. Also, defer before close is important too, since its result also affects the communication.
So here is the code modified according to the points I mentioned above:
package main
import (
"fmt"
"sync"
)
func main() {
ch := make(chan int)
var wg sync.WaitGroup
wg.Add(2)
go Print(ch, &wg)
go func() {
defer func() {
if r := recover(); r != nil {
println("panic:" + r.(string))
}
}()
defer func() {
wg.Done()
}()
for i := 1; i <= 11; i++ {
ch <- i
if i == 7 {
panic("ahaha")
}
}
println("sender done")
close(ch)
}()
wg.Wait()
}
func Print(ch <-chan int, wg *sync.WaitGroup) {
defer func() {
if r := recover(); r != nil {
println("panic:" + r.(string))
}
}()
defer wg.Done()
for n := range ch {
fmt.Println(n)
}
println("print done")
}
Hope it helps :)
I was simply experimenting in golang. I came across an interesting result. This is my code.
package main
import (
"fmt"
"sync"
)
func main() {
var wg sync.WaitGroup
var str1, str2 string
wg.Add(2)
go func() {
fmt.Scanf("%s", &str1)
wg.Done()
}()
go func() {
fmt.Scanf("%s", &str2)
wg.Done()
}()
wg.Wait()
fmt.Printf("%s %s\n", str1, str2)
}
I gave the following input.
beat
it
I was expecting the result to be either
it beat
or
beat it
But I got.
eat bit
Can any one please help me figure out why it is so?
fmt.Scanf isn't an atomic operation. Here's the implementation : http://golang.org/src/pkg/fmt/scan.go#L1115
There's no semaphor, nothing preventing two parallel executions. So what happens is simply that the executions are really parallel, and as there's no buffering, any byte reading is an IO operation and thus a perfect time for the go scheduler to change goroutine.
The problem is that you are sharing a single resource (the stdin byte stream) across multiple goroutines.
Each goroutine could be spawn at different non-deterministic times. i.e:
first goroutine 1 read all stdin, then start goroutine 2
first goroutine 2 read all stdin, then start goroutine 1
first goroutine 1 block on read, then start goroutine 2 read one char and then restart goroutine 1
... and so on and on ...
In most cases is enough to use only one goroutine to access a linear resource as a byte stream and attach a channel to it and then spawn multiple consumers that listen to that channel.
For example:
package main
import (
"fmt"
"io"
"sync"
)
func main() {
var wg sync.WaitGroup
words := make(chan string, 10)
wg.Add(1)
go func() {
for {
var buff string
_, err := fmt.Scanf("%s", &buff)
if err != nil {
if err != io.EOF {
fmt.Println("Error: ", err)
}
break
}
words <- buff
}
close(words)
wg.Done()
}()
// Multiple consumers
for i := 0; i < 5; i += 1 {
go func() {
for word := range words {
fmt.Printf("%s\n", word)
}
}()
}
wg.Wait()
}