Below is a simple go example. I have omitted error handling etc intentionally to make the example short. I have a simple for loop calling the writeOutput function 5 times using the go keyword to make the function run concurrently.
What I expect to happen is 5 files are created in /tmp/ with the contents of test.
What happens is that no files are created.
However if I remove the go keyword the code executes as expected. Im overlooking something super obvious. My background is dynamically typed languages like PHP/Ruby so just getting to grips with go and can't understand why 5 files are created when the go keyword exists.
package main
import (
"os"
"math/rand"
"strconv"
)
func main() {
for i := 0; i < 5; i++ {
go writeOutput()
}
}
func writeOutput() {
filename := strconv.Itoa(rand.Intn(10000))
file, _ := os.Create("/tmp/" + filename)
defer file.Close()
file.WriteString("test")
}
I managed to solve this with a wait group as suggested in the comments.
package main
import (
"math/rand"
"os"
"strconv"
"sync"
)
func main() {
var wg sync.WaitGroup
for i := 0; i < 5; i++ {
wg.Add(1)
go func() {
defer wg.Done()
writeOutput()
}()
}
wg.Wait()
}
func writeOutput() {
filename := strconv.Itoa(rand.Intn(10000))
file, _ := os.Create("/tmp/" + filename)
defer file.Close()
file.WriteString("test")
}
Related
Introduction
zerolog fields
I'm using github.com/rs/zerolog in my golang project.
I know that I can add fields to the output by using something like this:
package main
import (
"os"
"github.com/rs/zerolog"
)
func main() {
logger := zerolog.New(os.Stderr).With().Timestamp().Logger()
logger.Int("myIntField", 42)
logger.Info("a regular log output") // this log entry will also contain the integer field `myIntField`
}
But what I would like to have is something evaluates at runtime of the line logger.Info("a regular log output") what the value of a field myIntField is.
The setting
I have a producer/consumer setup (for example see https://goplay.tools/snippet/hkoMAwqKcwj) with go-routines and I have two integers that are atomically counted down the number of consumer and producer go-routines still in business. Upon tear down of the consumer and producer I want to display these numbers at runtime.
Here's the code when using log instead of zerolog:
package main
import (
"fmt"
"log"
"os"
"sync"
"sync/atomic"
)
func main() {
numProducers := int32(3)
numConsumers := int32(3)
producersRunning := numProducers
consumersRunning := numConsumers
var wg sync.WaitGroup
l := log.New(os.Stderr, "", 0)
// producers
for i := int32(0); i < numProducers; i++ {
idx := i
wg.Add(1)
go (func() {
// producer tear down
defer func() {
atomic.AddInt32(&producersRunning, -1)
l.Printf("producer-%3d . producersRunning: %3d\n", idx, producersRunning)
wg.Done()
}()
// this is where the actual producer works is happening
})()
}
// consumers
for i := int32(0); i < numConsumers; i++ {
idx := i
wg.Add(1)
go (func() {
// consumer tear down
defer func() {
atomic.AddInt32(&consumersRunning, -1)
l.Printf("consumer-%3d . consumersRunning: %3d\n", idx, consumersRunning)
wg.Done()
}()
// this is where the actual consumer works is happening
})()
}
fmt.Println("waiting")
wg.Wait()
}
It outputs something like this:
waiting
producer- 1 . producersRunning: 2
producer- 0 . producersRunning: 1
consumer- 1 . consumersRunning: 2
producer- 2 . producersRunning: 0
consumer- 2 . consumersRunning: 1
consumer- 0 . consumersRunning: 0
A logger per consumer / producer
With zerolog you can create loggers an pass them to each go-rountine:
logger := zerolog.New(os.Stderr)
go myConsumer(logger.With().Str("is", "consumer").Logger())
go myProducer(logger.With().Str("is", "producer").Logger())
Then you can easily find out in the logs if a message came from a consumer or a producer just by looking at the is field in each log line.
But what if I want to always print the number of currently active consumers/producers in each log line? You might be tempted to do something like this:
go myConsumer(logger.With().Str("is", "consumer").Int("consumersRunning", consumersRunning).Logger())
go myProducer(logger.With().Str("is", "producer").Int("producersRunning", producersRunning).Logger())
But of course, this will only print the momentary value of consumersRunning and producersRunning at the time of creating the go-routine. Instead I would like the log output to reflect the values at the time of the log output.
Summary
I hope my question is clear. I'm not sure if it is against the concept of zero-ness but a function like
func (e *Event) DeferredInt(key string, i func()int) *Event
would probably work, if only it existed.
Is there another way to achieve the same effect?
Potential workaround
I mean one way could be to replace the logger variable with a function call like this:
logFunc := func() zerolog.Logger {
return logger.With().Int("runningConcumers", runningConsumers).Logger()
}
And then a log entry can be created with logFunc().Msg("hello"). This defers the evaluation of runningConsumers but also creates a logger for each log entry which feels like overkill.
By now I hope I haven't confused you.
You can add a hook. Hook is evaluated for each logging event
https://go.dev/play/p/Q7doafJGaeE
package main
import (
"os"
"github.com/rs/zerolog"
)
type IntHook struct {
Count int
}
func (h *IntHook) Run(e *zerolog.Event, l zerolog.Level, msg string) {
e.Int("count", h.Count)
h.Count++
}
func main() {
var intHook IntHook
log := zerolog.New(os.Stdout).Hook(&intHook)
log.Info().Msg("hello world")
log.Info().Msg("hello world one more time")
}
Output is
{"level":"info","count":0,"message":"hello world"}
{"level":"info","count":1,"message":"hello world one more time"}
Pointer is required to save Count between calls to Hook.Run
May be for you a HookFunc is better. It is a stateless function that is called for each event. Here is an example of a function hook that calls PRNG for each message: https://go.dev/play/p/xu6aXpUmE0v
package main
import (
"math/rand"
"os"
"github.com/rs/zerolog"
)
func RandomHook(e *zerolog.Event, l zerolog.Level, msg string) {
e.Int("random", rand.Intn(100))
}
func main() {
var randomHook zerolog.HookFunc = RandomHook
log := zerolog.New(os.Stdout).Hook(randomHook)
log.Info().Msg("hello world")
log.Info().Msg("hello world one more time")
}
Output
{"level":"info","random":81,"message":"hello world"}
{"level":"info","random":87,"message":"hello world one more time"}
You can use a zerolog Hook to achieve this. Hooks are interfaces with a Run method which is called before the event data is written to the given io.Writer (in your case os.Stderr).
Here is some example code:
type counter struct {
name string
value int32
}
func (c *counter) inc() { atomic.AddInt32(&c.value, 1) }
func (c *counter) dec() { atomic.AddInt32(&c.value, -1) }
func (c *counter) get() { atomic.LoadInt32(&c.value) }
func (c *counter) Run(e *zerolog.Event, _ zerolog.Level, _ string) {
e.Int32(c.name, c.get())
}
int main() {
numConsumers, numProducers := 3, 3
consumersRunning := &counter{
name: "consumersRunning",
value: int32(numConsumers),
}
producersRunning := &counter{
name: "producersRunning",
value: int32(numProducers),
}
logger := zerolog.New(os.Stderr)
consumerLogger := logger.With().Str("is", "consumer").Logger().Hook(consumersRunning)
producerLogger := logger.With().Str("is", "producer").Logger().Hook(producersRunning)
// your other code
}
You will use the inc and dec methods of the counters to modify the numbers of consumers/producers running.
I would like to understand why this case deadlock and why it's not in the other case.
If I close the channel inside the goroutine, it works fine, but if I close it after the WaitGroup.Wait() it cause a deadlock.
package main
import (
"fmt"
"io/ioutil"
"os"
"sync"
)
var (
wg = sync.WaitGroup{}
links = make(chan string)
)
func rec_readdir(depth int, path string) {
files, _ := ioutil.ReadDir(path)
for _, f := range files {
if symlink, err := os.Readlink(path + "/" + f.Name()); err == nil {
links <- path + "/" + symlink
}
rec_readdir(depth+1, path+"/"+f.Name())
}
if depth == 0 {
wg.Done()
// close(links) // if close here ok
}
}
func main() {
wg.Add(1)
go rec_readdir(0, ".")
for slink := range links {
fmt.Println(slink)
}
wg.Wait()
close(links) // if close here deadlock
}
https://play.golang.org/p/Ntl_zsV5nwO
for slink := range links will continue looping until the channel is closed. So you obviously can't close after that loop. When you do, you get a deadlock, as you have observed.
I'm trying to understand go channel and go routine. To do so, I'm doing online exercises. I found one here: http://whipperstacker.com/2015/10/05/3-trivial-concurrency-exercises-for-the-confused-newbie-gopher/
I resolved the 3rd one (named "Internet cafe").
But there's something I resolved by "luck", and it's bother me because I don't understand my issue and why my "hack" fixed it.
In my code below, I replace "enterChan <- next" by "go func() { enterChan <- next }()", and it solved my deadlock.
Can someone explain to me why it deadlock before, and why it works with this hack ? Is it a proper solution, or an ugly one ?
Don't hesitate to criticize my code, I'm searching to improve :)
Many thanks!
This is my code:
package main
import (
"fmt"
"math/rand"
"strconv"
"time"
)
const (
maxNumberOfUser = 8
)
func useComputer(tourist string, leaverChan chan string) {
seed := rand.NewSource(time.Now().UnixNano())
random := rand.New(seed)
fmt.Println(tourist, "is online")
d := random.Intn(120-15) + 15
time.Sleep(time.Duration(d) * time.Millisecond * 10)
fmt.Println(tourist, "is done, having spent", d, "minutes online.")
leaverChan <- tourist
}
func manageUsers(enterChan, leaverChan chan string, stopChan chan struct{}) {
nbUsed := 0
queue := make([]string, 0)
for {
select {
case tourist := <-enterChan:
if nbUsed < maxNumberOfUser {
nbUsed++
go useComputer(tourist, leaverChan)
} else {
fmt.Println(tourist, "waiting for turn.")
queue = append(queue, tourist)
}
case tourist := <-leaverChan:
nbUsed--
fmt.Println(tourist, "is leaving, number of free place is now:", maxNumberOfUser-nbUsed)
if len(queue) > 0 {
next := queue[0]
queue = queue[1:]
go func() {
enterChan <- next
}()
} else if nbUsed == 0 {
close(stopChan)
return
}
}
}
}
func main() {
enterChan := make(chan string)
leaverChan := make(chan string)
stopChan := make(chan struct{})
go manageUsers(enterChan, leaverChan, stopChan)
for i := 1; i <= 25; i++ {
enterChan <- "Tourist " + strconv.Itoa(i)
}
<-stopChan
fmt.Println("The place is empty, let's close up and go to the beach!")
}
As explained by #dev.bmax, enterChan is not a buffered channel and you are trying to send data to a channel that is not being read. For example, below code results in a deadlock error:
package main
import (
"fmt"
)
func main() {
stream := make(chan int)
<-stream
stream<-1
fmt.Println("Yayyy!! no deadlock!")
}
you can run the above code here and check that it has deadlock. But if I change it to this:
package main
import (
"fmt"
)
func main() {
stream := make(chan int)
go func(){<-stream}()
stream<-1
fmt.Println("Yayyy!! no deadlock!")
}
we have no deadlock. Because we started another GoRoutine which is reading from the channel while the main GoRoutine is pushing data to channel. Hope this helps!
Code below works fine with hard coded JSON data however doesn't work when I read JSON data from a file. I'm getting fatal error: all goroutines are asleep - deadlock error when using sync.WaitGroup.
WORKING EXAMPLE WITH HARD-CODED JSON DATA:
package main
import (
"bytes"
"fmt"
"os/exec"
"time"
)
func connect(host string) {
cmd := exec.Command("ssh", host, "uptime")
var out bytes.Buffer
cmd.Stdout = &out
err := cmd.Run()
if err != nil {
fmt.Println(err)
}
fmt.Printf("%s: %q\n", host, out.String())
time.Sleep(time.Second * 2)
fmt.Printf("%s: DONE\n", host)
}
func listener(c chan string) {
for {
host := <-c
go connect(host)
}
}
func main() {
hosts := [2]string{"user1#111.79.154.111", "user2#111.79.190.222"}
var c chan string = make(chan string)
go listener(c)
for i := 0; i < len(hosts); i++ {
c <- hosts[i]
}
var input string
fmt.Scanln(&input)
}
OUTPUT:
user#user-VirtualBox:~/go$ go run channel.go
user1#111.79.154.111: " 09:46:40 up 86 days, 18:16, 0 users, load average: 5"
user2#111.79.190.222: " 09:46:40 up 86 days, 17:27, 1 user, load average: 9"
user1#111.79.154.111: DONE
user2#111.79.190.222: DONE
NOT WORKING - EXAMPLE WITH READING JSON DATA FILE:
package main
import (
"bytes"
"fmt"
"os/exec"
"time"
"encoding/json"
"os"
"sync"
)
func connect(host string) {
cmd := exec.Command("ssh", host, "uptime")
var out bytes.Buffer
cmd.Stdout = &out
err := cmd.Run()
if err != nil {
fmt.Println(err)
}
fmt.Printf("%s: %q\n", host, out.String())
time.Sleep(time.Second * 2)
fmt.Printf("%s: DONE\n", host)
}
func listener(c chan string) {
for {
host := <-c
go connect(host)
}
}
type Content struct {
Username string `json:"username"`
Ip string `json:"ip"`
}
func main() {
var wg sync.WaitGroup
var source []Content
var hosts []string
data := json.NewDecoder(os.Stdin)
data.Decode(&source)
for _, value := range source {
hosts = append(hosts, value.Username + "#" + value.Ip)
}
var c chan string = make(chan string)
go listener(c)
for i := 0; i < len(hosts); i++ {
wg.Add(1)
c <- hosts[i]
defer wg.Done()
}
var input string
fmt.Scanln(&input)
wg.Wait()
}
OUTPUT
user#user-VirtualBox:~/go$ go run deploy.go < hosts.txt
user1#111.79.154.111: " 09:46:40 up 86 days, 18:16, 0 users, load average: 5"
user2#111.79.190.222: " 09:46:40 up 86 days, 17:27, 1 user, load average: 9"
user1#111.79.154.111 : DONE
user2#111.79.190.222: DONE
fatal error: all goroutines are asleep - deadlock!
goroutine 1 [semacquire]:
sync.runtime_Semacquire(0xc210000068)
/usr/lib/go/src/pkg/runtime/sema.goc:199 +0x30
sync.(*WaitGroup).Wait(0xc210047020)
/usr/lib/go/src/pkg/sync/waitgroup.go:127 +0x14b
main.main()
/home/user/go/deploy.go:64 +0x45a
goroutine 3 [chan receive]:
main.listener(0xc210038060)
/home/user/go/deploy.go:28 +0x30
created by main.main
/home/user/go/deploy.go:53 +0x30b
exit status 2
user#user-VirtualBox:~/go$
HOSTS.TXT
[
{
"username":"user1",
"ip":"111.79.154.111"
},
{
"username":"user2",
"ip":"111.79.190.222"
}
]
Go program ends when the main function ends.
From the language specification
Program execution begins by initializing the main package and then invoking the function main. When that function invocation returns, the program exits. It does not wait for other (non-main) goroutines to complete.
Therefore, you need to wait for your goroutines to finish. The common solution for this is to use sync.WaitGroup object.
The simplest possible code to synchronize goroutine:
package main
import "fmt"
import "sync"
var wg sync.WaitGroup // 1
func routine() {
defer wg.Done() // 3
fmt.Println("routine finished")
}
func main() {
wg.Add(1) // 2
go routine() // *
wg.Wait() // 4
fmt.Println("main finished")
}
And for synchronizing multiple goroutines
package main
import "fmt"
import "sync"
var wg sync.WaitGroup // 1
func routine(i int) {
defer wg.Done() // 3
fmt.Printf("routine %v finished\n", i)
}
func main() {
for i := 0; i < 10; i++ {
wg.Add(1) // 2
go routine(i) // *
}
wg.Wait() // 4
fmt.Println("main finished")
}
WaitGroup usage in order of execution.
Declaration of global variable. Making it global is the easiest way to make it visible to all functions and methods.
Increasing the counter. This must be done in main goroutine because there is no guarantee that newly started goroutine will execute before 4 due to memory model guarantees.
Decreasing the counter. This must be done at the exit of goroutine. Using deferred call, we make sure that it will be called whenever function ends no matter but no matter how it ends.
Waiting for the counter to reach 0. This must be done in main goroutine to prevent program exit.
* The actual parameters are evaluated before starting new gouroutine. Thus it is needed to evaluate them explicitly before wg.Add(1) so the possibly panicking code would not leave increased counter.
Use
param := f(x)
wg.Add(1)
go g(param)
instead of
wg.Add(1)
go g(f(x))
Thanks for the very nice and detailed explanation Grzegorz Żur.
One thing that I want to point it out that typically the func that needs to be threaded wont be in main(), so we would have something like this:
package main
import (
"bufio"
"fmt"
"io"
"io/ioutil"
"math/rand"
"os"
"reflect"
"regexp"
"strings"
"sync"
"time"
)
var wg sync.WaitGroup // VERY IMP to declare this globally, other wise one //would hit "fatal error: all goroutines are asleep - deadlock!"
func doSomething(arg1 arg1Type) {
// cured cancer
}
func main() {
r := rand.New(rand.NewSource(time.Now().UnixNano()))
randTime := r.Intn(10)
wg.Add(1)
go doSomething(randTime)
wg.Wait()
fmt.Println("Waiting for all threads to finish")
}
The thing that I want to point it out is that global declaration of wg is very crucial for all threads to finish before main()
try this code snippest
package main
import (
"bytes"
"fmt"
"os/exec"
"time"
"sync"
)
func connect(host string, wg *sync.WaitGroup) {
defer wg.Done()
cmd := exec.Command("ssh", host, "uptime")
var out bytes.Buffer
cmd.Stdout = &out
err := cmd.Run()
if err != nil {
fmt.Println(err)
}
fmt.Printf("%s: %q\n", host, out.String())
time.Sleep(time.Second * 2)
fmt.Printf("%s: DONE\n", host)
}
func listener(c chan string，wg *sync.WaitGroup) {
for {
host，ok := <-c
// check channel is closed or not
if !ok{
break
}
go connect(host)
}
}
func main() {
var wg sync.WaitGroup
hosts := [2]string{"user1#111.79.154.111", "user2#111.79.190.222"}
var c chan string = make(chan string)
go listener(c)
for i := 0; i < len(hosts); i++ {
wg.Add(1)
c <- hosts[i]
}
close(c)
var input string
fmt.Scanln(&input)
wg.Wait()
}
This example taken from tour.golang.org/#63
package main
import (
"fmt"
"time"
)
func say(s string) {
for i := 0; i < 5; i++ {
time.Sleep(100 * time.Millisecond)
fmt.Println(s)
}
}
func main() {
go say("world")
say("hello")
}
The output
hello
world
hello
world
hello
world
hello
world
hello
Why world is printed only 4 times instead of 5 ?
Edit: The answer can be quoted from golang specification:
Program execution begins by initializing the main package and then
invoking the function main. When the function main returns, the
program exits. It does not wait for other (non-main) goroutines to
complete.
When your main function ends your program ends, i.e. all goroutines are terminated.
Your main terminates before go say("world") is done. If you sleep some time at the end of main you should see the last world.
Here is how you solve that synchronization problem properly - with sync.WaitGroup
Playground link
package main
import (
"fmt"
"sync"
"time"
)
func say(s string, wg *sync.WaitGroup) {
defer wg.Done()
for i := 0; i < 5; i++ {
time.Sleep(100 * time.Millisecond)
fmt.Println(s)
}
}
func main() {
wg := new(sync.WaitGroup)
wg.Add(2)
go say("world", wg)
go say("hello", wg)
wg.Wait()
fmt.Println("All done")
}
Because the calling gorouting terminates before the second one you spawned does. This causes the second to shut down. To illustrate, modify your code slightly:
package main
import (
"fmt"
"time"
)
func say(s string) {
for i := 0; i < 5; i++ {
time.Sleep(100 * time.Millisecond)
fmt.Print(i)
fmt.Println(":"+s)
}
}
func main() {
go say("world")
say("hello")
}
Try putting in a "wait" or a sleep to the end of the main function.