GO language: fatal error: all goroutines are asleep - deadlock - go

Code below works fine with hard coded JSON data however doesn't work when I read JSON data from a file. I'm getting fatal error: all goroutines are asleep - deadlock error when using sync.WaitGroup.
WORKING EXAMPLE WITH HARD-CODED JSON DATA:
package main
import (
"bytes"
"fmt"
"os/exec"
"time"
)
func connect(host string) {
cmd := exec.Command("ssh", host, "uptime")
var out bytes.Buffer
cmd.Stdout = &out
err := cmd.Run()
if err != nil {
fmt.Println(err)
}
fmt.Printf("%s: %q\n", host, out.String())
time.Sleep(time.Second * 2)
fmt.Printf("%s: DONE\n", host)
}
func listener(c chan string) {
for {
host := <-c
go connect(host)
}
}
func main() {
hosts := [2]string{"user1#111.79.154.111", "user2#111.79.190.222"}
var c chan string = make(chan string)
go listener(c)
for i := 0; i < len(hosts); i++ {
c <- hosts[i]
}
var input string
fmt.Scanln(&input)
}
OUTPUT:
user#user-VirtualBox:~/go$ go run channel.go
user1#111.79.154.111: " 09:46:40 up 86 days, 18:16, 0 users, load average: 5"
user2#111.79.190.222: " 09:46:40 up 86 days, 17:27, 1 user, load average: 9"
user1#111.79.154.111: DONE
user2#111.79.190.222: DONE
NOT WORKING - EXAMPLE WITH READING JSON DATA FILE:
package main
import (
"bytes"
"fmt"
"os/exec"
"time"
"encoding/json"
"os"
"sync"
)
func connect(host string) {
cmd := exec.Command("ssh", host, "uptime")
var out bytes.Buffer
cmd.Stdout = &out
err := cmd.Run()
if err != nil {
fmt.Println(err)
}
fmt.Printf("%s: %q\n", host, out.String())
time.Sleep(time.Second * 2)
fmt.Printf("%s: DONE\n", host)
}
func listener(c chan string) {
for {
host := <-c
go connect(host)
}
}
type Content struct {
Username string `json:"username"`
Ip string `json:"ip"`
}
func main() {
var wg sync.WaitGroup
var source []Content
var hosts []string
data := json.NewDecoder(os.Stdin)
data.Decode(&source)
for _, value := range source {
hosts = append(hosts, value.Username + "#" + value.Ip)
}
var c chan string = make(chan string)
go listener(c)
for i := 0; i < len(hosts); i++ {
wg.Add(1)
c <- hosts[i]
defer wg.Done()
}
var input string
fmt.Scanln(&input)
wg.Wait()
}
OUTPUT
user#user-VirtualBox:~/go$ go run deploy.go < hosts.txt
user1#111.79.154.111: " 09:46:40 up 86 days, 18:16, 0 users, load average: 5"
user2#111.79.190.222: " 09:46:40 up 86 days, 17:27, 1 user, load average: 9"
user1#111.79.154.111 : DONE
user2#111.79.190.222: DONE
fatal error: all goroutines are asleep - deadlock!
goroutine 1 [semacquire]:
sync.runtime_Semacquire(0xc210000068)
/usr/lib/go/src/pkg/runtime/sema.goc:199 +0x30
sync.(*WaitGroup).Wait(0xc210047020)
/usr/lib/go/src/pkg/sync/waitgroup.go:127 +0x14b
main.main()
/home/user/go/deploy.go:64 +0x45a
goroutine 3 [chan receive]:
main.listener(0xc210038060)
/home/user/go/deploy.go:28 +0x30
created by main.main
/home/user/go/deploy.go:53 +0x30b
exit status 2
user#user-VirtualBox:~/go$
HOSTS.TXT
[
{
"username":"user1",
"ip":"111.79.154.111"
},
{
"username":"user2",
"ip":"111.79.190.222"
}
]

Go program ends when the main function ends.
From the language specification
Program execution begins by initializing the main package and then invoking the function main. When that function invocation returns, the program exits. It does not wait for other (non-main) goroutines to complete.
Therefore, you need to wait for your goroutines to finish. The common solution for this is to use sync.WaitGroup object.
The simplest possible code to synchronize goroutine:
package main
import "fmt"
import "sync"
var wg sync.WaitGroup // 1
func routine() {
defer wg.Done() // 3
fmt.Println("routine finished")
}
func main() {
wg.Add(1) // 2
go routine() // *
wg.Wait() // 4
fmt.Println("main finished")
}
And for synchronizing multiple goroutines
package main
import "fmt"
import "sync"
var wg sync.WaitGroup // 1
func routine(i int) {
defer wg.Done() // 3
fmt.Printf("routine %v finished\n", i)
}
func main() {
for i := 0; i < 10; i++ {
wg.Add(1) // 2
go routine(i) // *
}
wg.Wait() // 4
fmt.Println("main finished")
}
WaitGroup usage in order of execution.
Declaration of global variable. Making it global is the easiest way to make it visible to all functions and methods.
Increasing the counter. This must be done in main goroutine because there is no guarantee that newly started goroutine will execute before 4 due to memory model guarantees.
Decreasing the counter. This must be done at the exit of goroutine. Using deferred call, we make sure that it will be called whenever function ends no matter but no matter how it ends.
Waiting for the counter to reach 0. This must be done in main goroutine to prevent program exit.
* The actual parameters are evaluated before starting new gouroutine. Thus it is needed to evaluate them explicitly before wg.Add(1) so the possibly panicking code would not leave increased counter.
Use
param := f(x)
wg.Add(1)
go g(param)
instead of
wg.Add(1)
go g(f(x))

Thanks for the very nice and detailed explanation Grzegorz Żur.
One thing that I want to point it out that typically the func that needs to be threaded wont be in main(), so we would have something like this:
package main
import (
"bufio"
"fmt"
"io"
"io/ioutil"
"math/rand"
"os"
"reflect"
"regexp"
"strings"
"sync"
"time"
)
var wg sync.WaitGroup // VERY IMP to declare this globally, other wise one //would hit "fatal error: all goroutines are asleep - deadlock!"
func doSomething(arg1 arg1Type) {
// cured cancer
}
func main() {
r := rand.New(rand.NewSource(time.Now().UnixNano()))
randTime := r.Intn(10)
wg.Add(1)
go doSomething(randTime)
wg.Wait()
fmt.Println("Waiting for all threads to finish")
}
The thing that I want to point it out is that global declaration of wg is very crucial for all threads to finish before main()

try this code snippest
package main
import (
"bytes"
"fmt"
"os/exec"
"time"
"sync"
)
func connect(host string, wg *sync.WaitGroup) {
defer wg.Done()
cmd := exec.Command("ssh", host, "uptime")
var out bytes.Buffer
cmd.Stdout = &out
err := cmd.Run()
if err != nil {
fmt.Println(err)
}
fmt.Printf("%s: %q\n", host, out.String())
time.Sleep(time.Second * 2)
fmt.Printf("%s: DONE\n", host)
}
func listener(c chan string,wg *sync.WaitGroup) {
for {
host,ok := <-c
// check channel is closed or not
if !ok{
break
}
go connect(host)
}
}
func main() {
var wg sync.WaitGroup
hosts := [2]string{"user1#111.79.154.111", "user2#111.79.190.222"}
var c chan string = make(chan string)
go listener(c)
for i := 0; i < len(hosts); i++ {
wg.Add(1)
c <- hosts[i]
}
close(c)
var input string
fmt.Scanln(&input)
wg.Wait()
}

Related

Goroutine is blocked in execution [duplicate]

This question already has answers here:
Deadlock after attempting to print values of channel using 'range'
(3 answers)
Deadlock in the program when using for range on buffered channel
(1 answer)
Waiting for a WaitGroup and ranging over a channel results in a "fatal error: all goroutines are asleep - deadlock!"
(2 answers)
go routine for range over channels
(5 answers)
golang concurrency sync issue
(1 answer)
Closed 26 days ago.
I try to work on a piece of code with goroutine to scan open port on a subnet.
Here is the code :
package main
import (
"fmt"
"log"
"net"
"time"
)
func is_445_open(ip string, ch chan string){
connexion, err := net.DialTimeout("tcp", ip + ":445", 3*time.Second )
if err != nil {
return
}
defer connexion.Close()
ch <- ip
}
func ip_suivante(ip net.IP) {
for j := len(ip) - 1; j >= 0; j-- {
ip[j]++
if ip[j] > 0 {
break
}
}
}
func main() {
ch := make(chan string)
ipv4Addr, ipv4Net, err := net.ParseCIDR("200.31.0.0/24")
if err != nil {
log.Fatal(err)
}
for ipv4Addr := ipv4Addr.Mask(ipv4Net.Mask); ipv4Net.Contains(ipv4Addr); ip_suivante(ipv4Addr) {
//fmt.Println(ipv4Addr.String())
go is_445_open(ipv4Addr.String(), ch)
}
for v := range ch{
fmt.Println(v)
}
fmt.Println("Done")
}
I use ParseCIDR to loop over every IP of the network range then i connect to the port 445 to test if it is open.
To speed up the process, i want to use goroutine and write the ip where 445 is open to a channel.
Then loop over it to print them on the console.
When i use the loop to print every IP on my console the program never end.
It print the ip correctly but never the "done" at the end.
I have tried to close the channel before looping but when i do, no more data is printed and the program only show "done".
Do you have any tips ?
EDIT :
Finally figured out. You MUST put the code which manage the waitgroup and the channel closing into another goroutine to make it work.
Cannot undestand why.
But here is the code modified accordingly :
package main
import (
"fmt"
"log"
"net"
"time"
"sync"
)
func is_445_open(ip string, ch chan string, wg *sync.WaitGroup){
connexion, err := net.DialTimeout("tcp", ip + ":445", 3*time.Second )
if err != nil {
wg.Done()
return
}
defer connexion.Close()
ch <- ip
wg.Done()
}
func ip_suivante(ip net.IP) {
for j := len(ip) - 1; j >= 0; j-- {
ip[j]++
if ip[j] > 0 {
break
}
}
}
func main() {
wg := &sync.WaitGroup{}
ch := make(chan string)
ipv4Addr, ipv4Net, err := net.ParseCIDR("192.168.1.0/24")
if err != nil {
log.Fatal(err)
}
for ipv4Addr := ipv4Addr.Mask(ipv4Net.Mask); ipv4Net.Contains(ipv4Addr); ip_suivante(ipv4Addr) {
//fmt.Println(ipv4Addr.String())
wg.Add(1)
go is_445_open(ipv4Addr.String(), ch, wg)
}
go func(cha chan string, wg *sync.WaitGroup){
wg.Wait()
close(cha)
}(ch,wg)
for v := range ch{
fmt.Println(v)
}
fmt.Println("Done")
}

How to limit the number of goroutines in recursion?

I'm using goroutine to traverse all files in the specified directory. I want to know how to limit the number of goroutines under recursive traversal
I wrote this code, but it prompted panic chan send
`
package main
import (
"fmt"
"os"
"runtime"
"sync"
"time"
)
var (
wg sync.WaitGroup
// limit = make(chan bool, 20)
// totalchan = make(chan bool)
)
func Work(path string, size int, finished bool) {
fmt.Printf("gn1111: %d\n", runtime.NumGoroutine())
defer func() { wg.Done() }()
fl, err := os.ReadDir(path)
if err == nil {
for _, file := range fl {
if file.IsDir() {
// limit <- true
wg.Add(1)
go Work(path+file.Name()+"/", size, false)
} else {
fmt.Printf("gn2222: %d\n", runtime.NumGoroutine())
}
}
}
// if !finished {
// <-limit
// }
}
func main() {
path := "C:/Windows/"
size := 0
start := time.Now()
wg.Add(1)
go Work(path, size, true)
wg.Wait()
fmt.Printf("total= %d, cost: %v\n", 0, time.Since(start))
}
`
the errgroup has a Limiting functionality
SetLimit limits the number of active goroutines in this group to at most n. A negative value indicates no limit.
Any subsequent call to the Go method will block until it can add an active goroutine without exceeding the configured limit.
The limit must not be modified while any goroutines in the group are active.

golang: all goroutines are asleep - deadlock

I want to print 1 to 100 use two goroutine:
package main
import (
"fmt"
"sync"
)
var my_chan chan int
var wg sync.WaitGroup
func worker() {
for true {
number := <-my_chan
fmt.Println(number)
number++
if number > 100 {
wg.Done()
return
}
my_chan <- number
}
}
func main() {
wg.Add(2)
my_chan := make(chan int)
init_num := 1
go worker()
go worker()
my_chan <- init_num
wg.Wait()
}
When I run the above code, I get the following error:
fatal error: all goroutines are asleep - deadlock!
Can anyone tell me where I am doing wrong?
Replace the channel creation with this:
my_chan = make(chan int)
Otherwise you are redeclaring my_chan in main, and all goroutines try to read from a nil channel. That will block.
Then it will count to 100 and deadlock. The check for number being larger than 100 will work for one of the goroutines, while the other one will be stuck waiting to read/write.
Thanks for Burak Serdar's answer, the following code works:
package main
import (
"fmt"
"sync"
)
var my_chan chan int
var wg sync.WaitGroup
func worker() {
for true {
number := <-my_chan
if number > 100 {
wg.Done()
my_chan <- 101
return
}
fmt.Println(number)
number++
my_chan <- number
}
}
func main() {
wg.Add(2)
my_chan = make(chan int)
init_num := 1
go worker()
go worker()
my_chan <- init_num
wg.Wait()
}

Deadlock on closing chan

I would like to understand why this case deadlock and why it's not in the other case.
If I close the channel inside the goroutine, it works fine, but if I close it after the WaitGroup.Wait() it cause a deadlock.
package main
import (
"fmt"
"io/ioutil"
"os"
"sync"
)
var (
wg = sync.WaitGroup{}
links = make(chan string)
)
func rec_readdir(depth int, path string) {
files, _ := ioutil.ReadDir(path)
for _, f := range files {
if symlink, err := os.Readlink(path + "/" + f.Name()); err == nil {
links <- path + "/" + symlink
}
rec_readdir(depth+1, path+"/"+f.Name())
}
if depth == 0 {
wg.Done()
// close(links) // if close here ok
}
}
func main() {
wg.Add(1)
go rec_readdir(0, ".")
for slink := range links {
fmt.Println(slink)
}
wg.Wait()
close(links) // if close here deadlock
}
https://play.golang.org/p/Ntl_zsV5nwO
for slink := range links will continue looping until the channel is closed. So you obviously can't close after that loop. When you do, you get a deadlock, as you have observed.

Why all goroutines are asleep - deadlock. Identifying bottleneck

package main
import (
"fmt"
"runtime"
"sync"
"time"
)
func main() {
intInputChan := make(chan int, 50)
var wg sync.WaitGroup
for i := 0; i < 3; i++ {
wg.Add(1)
go worker(intInputChan, wg)
}
for i := 1; i < 51; i++ {
fmt.Printf("Inputs. %d \n", i)
intInputChan <- i
}
close(intInputChan)
wg.Wait()
fmt.Println("Existing Main App... ")
panic("---------------")
}
func worker(input chan int, wg sync.WaitGroup) {
defer func() {
fmt.Println("Executing defer..")
wg.Done()
}()
for {
select {
case intVal, ok := <-input:
time.Sleep(100 * time.Millisecond)
if !ok {
input = nil
return
}
fmt.Printf("%d %v\n", intVal, ok)
default:
runtime.Gosched()
}
}
}
error thrown is.
fatal error: all goroutines are asleep - deadlock!
goroutine 1 [semacquire]:
sync.(*WaitGroup).Wait(0xc082004600)
c:/go/src/sync/waitgroup.go:132 +0x170
main.main()
E:/Go/go_projects/go/src/Test.go:22 +0x21a
I just tried it (playground) passing a wg *sync.WaitGroup and it works.
Passing sync.WaitGroup means passing a copy of the sync.WaitGroup (passing by value): the goroutine mentions Done() to a different sync.WaitGroup.
var wg sync.WaitGroup
for i := 0; i < 3; i++ {
wg.Add(1)
go worker(intInputChan, &wg)
}
Note the &wg: you are passing by value the pointer to the original sync.WaitGroup, for the goroutine to use.
As mentioned, don't pass types from the sync package around by value, right near the top of the sync package documentation: "Values containing the types defined in this package should not be copied." That also includes the types themselves (sync.Mutex, sync.WaitGroup, etc).
However, several notes:
You can use just a single call to wg.Add if you know how many you're going to add (but as documented make sure it's done before anything can call Wait).
You don't want to call runtime.Gosched like that; it makes the workers busy loop.
You can use range to read from the channel to simplify stopping when it's closed.
For small functions you can use a closure and not bother to pass the channel or wait group at all.
That turns it into this:
package main
import (
"fmt"
"sync"
"time"
)
func main() {
const numWorkers = 3
c := make(chan int, 10)
var wg sync.WaitGroup
wg.Add(numWorkers)
for i := 0; i < numWorkers; i++ {
go func() {
defer func() {
fmt.Println("Executing defer…")
wg.Done()
}()
for v := range c {
fmt.Println("recv:", v)
time.Sleep(100 * time.Millisecond)
}
}()
}
for i := 1; i < 51; i++ {
fmt.Println("send:", i)
c <- i
}
fmt.Println("closing…")
close(c)
fmt.Println("waiting…")
wg.Wait()
fmt.Println("Exiting Main App... ")
}
playground

Resources