I've created function 1 that is run concurrently using goroutines and waitgroups.
This function imports function 2 from another package, which has its own logging configuration.
If I use some techniques to stop log interleaving in function 1, like sync.Mutex or buffered log channel, this works.
But how can I stop the logs from the imported function 2 getting all mixed up?
Related
I worked on languages like PHP and Python but recently I got a Go project to maintain which utilizes several workers and goroutines.
In PHP or Python, we can know about the execution of code as its all synchronous calls. But I am finding it very difficult to understand how the code execution takes place in Go, especially in workers and goroutine. Can anyone suggest me any tools by using which I can get the execution flow(the name of the function being called)?
For example, if there is one API in which there are various goroutines and inside those goroutines, other goroutines are called, so how to know in which sequence these functions are called.
Go runtime trace tool will help you to trace , but you need to add trace option in go code start
package main
import (
"os"
"runtime/trace"
)
func main() {
trace.Start(os.Stderr) //start the trace
defer trace.Stop() // defer to the end
.... //rest of the code
}
Then use ,
go run main.go 2> trace.out
then use the trace tool
go tool trace trace.out.
This link has more info https://blog.gopheracademy.com/advent-2017/go-execution-tracer/
[I]f there is one API in which there are various goroutines and inside those goroutines, other goroutines are called, [...]
There isn‘t.
I have a program in which I have 256 goroutines generating test data and sending them to a channel.
In the consuming part of the program, I set up a select like this:
select {
case c := <-theChan:
// Do some stuff with c
default:
//
}
What surprise me is that while the 256 goroutines keep sending items to the channel and the processing of the items take time? The program runs into the default branch several times.
I wonder how does the select statement decide that theChan is empty and run into default.
It depends on the scheduler, but between the time you consume a value from the channel and the time another goroutine gets execution time allocated by the scheduler (which would add a value in the channel), the main goroutine may have enough time to run the case and go back to the select statement before a value is added to the channel, it would then run the default case.
You could reduce this by using a buffered channel.
I'm just wondering if there is potential for corruption as a result of writing the same value to a global variable at the same time. My brain is telling me there is nothing wrong with this because its just a location in memory, but I figure I should probably double check this assumption.
I have concurrent processes writing to a global map var linksToVisit map[string]bool. The map is actually tracking what links on a website need to be further crawled.
However it can be the case that concurrent processes may have the same link on their respective pages and therefore each will mark that same link as true concurrently. There's nothing wrong with NOT using locks in this case right? NOTE: I never change the value back to false so either the key exists and it's value is true or it doesn't exist.
I.e.
var linksToVisit = map[string]bool{}
...
// somewhere later a goroutine finds a link and marks it as true
// it is never marked as false anywhere
linksToVisit[someLink] = true
What happens if concurrent processes write to a global variable the
same value?
The results of a data race are undefined.
Run the Go data race detector.
References:
Wikipedia: Race condition
Benign Data Races: What Could Possibly Go Wrong?
The Go Blog: Introducing the Go Race Detector
Go: Data Race Detector
Go 1.8 Release Notes
Concurrent Map Misuse
In Go 1.6, the runtime added lightweight, best-effort detection of
concurrent misuse of maps. This release improves that detector with
support for detecting programs that concurrently write to and iterate
over a map.
As always, if one goroutine is writing to a map, no other goroutine
should be reading (which includes iterating) or writing the map
concurrently. If the runtime detects this condition, it prints a
diagnosis and crashes the program. The best way to find out more about
the problem is to run the program under the race detector, which will
more reliably identify the race and give more detail.
For example,
package main
import "time"
var linksToVisit = map[string]bool{}
func main() {
someLink := "someLink"
go func() {
for {
linksToVisit[someLink] = true
}
}()
go func() {
for {
linksToVisit[someLink] = true
}
}()
time.Sleep(100 * time.Millisecond)
}
Output:
$ go run racer.go
fatal error: concurrent map writes
$
$ go run -race racer.go
==================
WARNING: DATA RACE
Write at 0x00c000078060 by goroutine 6:
runtime.mapassign_faststr()
/home/peter/go/src/runtime/map_faststr.go:190 +0x0
main.main.func2()
/home/peter/gopath/src/racer.go:16 +0x6a
Previous write at 0x00c000078060 by goroutine 5:
runtime.mapassign_faststr()
/home/peter/go/src/runtime/map_faststr.go:190 +0x0
main.main.func1()
/home/peter/gopath/src/racer.go:11 +0x6a
Goroutine 6 (running) created at:
main.main()
/home/peter/gopath/src/racer.go:14 +0x88
Goroutine 5 (running) created at:
main.main()
/home/peter/gopath/src/racer.go:9 +0x5b
==================
fatal error: concurrent map writes
$
It is better to use locks if you are changing the same value concurrently using multiple go routines. Since mutex and locks are used whenever it comes to secure the value from accessing when another function is changing the same just like writing to database table while accessing the same table.
For your question on using maps with different keys it is not preferable in Go as:
The typical use of maps did not require safe access from multiple
goroutines, and in those cases where it did, the map was probably part
of some larger data structure or computation that was already
synchronized. Therefore requiring that all map operations grab a mutex
would slow down most programs and add safety to few.
Map access is unsafe only when updates are occurring. As long as all
goroutines are only reading—looking up elements in the map, including
iterating through it using a for range loop—and not changing the map
by assigning to elements or doing deletions, it is safe for them to
access the map concurrently without synchronization.
So In case of update of maps it is not recommended. For more information Check FAQ on why maps operations not defined atomic.
Also it is noticed that if you realy wants to go for there should be a way to synchronize them.
Maps are not safe for concurrent use: it's not defined what happens
when you read and write to them simultaneously. If you need to read
from and write to a map from concurrently executing goroutines, the
accesses must be mediated by some kind of synchronization mechanism.
One common way to protect maps is with sync.RWMutex.
Concurrent map write is not ok, so you will most likely get a fatal error. So I think a lock should be used
As of Go 1.6, simultaneous map writes will cause a panic. Use a sync.Map to synchronize access.
See the map value assign implementation:
https://github.com/golang/go/blob/fe8a0d12b14108cbe2408b417afcaab722b0727c/src/runtime/hashmap.go#L519
I wrote a log collector program in go, which runs a bunch of goroutines as follow:
routine A runs HTTP server, allow users to view log information
routine B runs UDP server, allow log messages to be sent to it from LAN
routine C runs a timer, which periodically query/download zipped log archives from an internal HTTP file server (not part of the program)
routine B & C both send processed messages to a Channel
routine D runs a for {} loop with a select statement which receives message from the Channel and flush it to disk
there are a few other go routines such as a routine to scan the log archives generated by routine D to create SQLite indices etc.
The program has a problem that after a few hours running, the log viewer http server still works well but there are NO messages coming in either from the UDP or fileserver routines. I know that there are endless log messages sending from various channels, also if I restart the program, it start to process incoming logs again.
I added -race to the compiler, and it indeed find out some problematic code, and I fixed these, but still, problem persists. What's more, although there are racy problems, the old version code running on our production server works well, regardless of the racy code.
My question is, how can I proceed to pinpoint the problem. The following is key loop in my log processing routine:
for {
select {
case msg := <-logCh:
logque.Cache(msg)
case <-time.After(time.Second):
}
if time.Since(lastFlush) >= 3 * time.Second {
logque.Flush()
lastFlush = time.Now()
}
}
I finally found the code that created the blocking. In the following code:
for {
select {
case msg := <-logCh:
logque.Cache(msg)
case <-time.After(time.Second):
}
if time.Since(lastFlush) >= 3 * time.Second {
logque.Flush()
lastFlush = time.Now()
}
}
Inside logque.Flush() there are some code that generate log messages which in turn write into the channel, eventually caused the channel's buffer being filled up. This only occurs when I turn on debug mode, production code does not do this in the Flush() method.
To answer my own question, the method I used to nail down the problem is pretty simple:
if len(logch) >= LOG_CHANNEL_CAP {
//drop the message or store it into
//secondary buffer...
return
}
logch <- msg
To settle some misunderstandings I have about goroutines, I went to the Go playground and ran this code:
package main
import (
"fmt"
)
func other(done chan bool) {
done <- true
go func() {
for {
fmt.Println("Here")
}
}()
}
func main() {
fmt.Println("Hello, playground")
done := make(chan bool)
go other(done)
<-done
fmt.Println("Finished.")
}
As I expected, Go playground came back with an error: Process took too long.
This seems to imply that the goroutine created within other runs forever.
But when I run the same code on my own machine, I get this output almost instantaneously:
Hello, playground.
Finished.
This seems to imply that the goroutine within other exits when the main goroutine finishes. Is this true? Or does the main goroutine finish, while the other goroutine continues to run in the background?
Edit: Default GOMAXPROCS has changed on the Go Playground, it now defaults to 8. In the "old" days it defaulted to 1. To get the behavior described in the question, set it to 1 explicitly with runtime.GOMAXPROCS(1).
Explanation of what you see:
On the Go Playground, GOMAXPROCS is 1 (proof).
This means one goroutine is executed at a time, and if that goroutine does not block, the scheduler is not forced to switch to other goroutines.
Your code (like every Go app) starts with a goroutine executing the main() function (the main goroutine). It starts another goroutine that executes the other() function, then it receives from the done channel - which blocks. So the scheduler must switch to the other goroutine (executing other() function).
In your other() function when you send a value on the done channel, that makes both the current (other()) and the main goroutine runnable. The scheduler chooses to continue to run other(), and since GOMAXPROCS=1, main() is not continued. Now other() launches another goroutine executing an endless loop. The scheduler chooses to execute this goroutine which takes forever to get to a blocked state, so main() is not continued.
And then the timeout of the Go Playground's sandbox comes as an absolution:
process took too long
Note that the Go Memory Model only guarantees that certain events happen before other events, you have no guarantee how 2 concurrent goroutines are executed. Which makes the output non-deterministic.
You are not to question any execution order that does not violate the Go Memory Model. If you want the execution to reach certain points in your code (to execute certain statements), you need explicit synchronization (you need to synchronize your goroutines).
Also note that the output on the Go Playground is cached, so if you run the app again, it won't be run again, but instead the cached output will be presented immediately. If you change anything in the code (e.g. insert a space or a comment) and then you run it again, it then will be compiled and run again. You will notice it by the increased response time. Using the current version (Go 1.6) you will see the same output every time though.
Running locally (on your machine):
When you run it locally, most likely GOMAXPROCS will be greater than 1 as it defaults to the number of CPU cores available (since Go 1.5). So it doesn't matter if you have a goroutine executing an endless loop, another goroutine will be executed simultaneously, which will be the main(), and when main() returns, your program terminates; it does not wait for other non-main goroutines to complete (see Spec: Program execution).
Also note that even if you set GOMAXPROCS to 1, your app will most likely exit in a "short" time as the scheduler imlementation will switch to other goroutines and not just execute the endless loop forever (however, as stated above, this is non-deterministic). And when it does, it will be the main() goroutine, and so when main() finishes and returns, your app terminates.
Playing with your app on the Go Playground:
As mentioned, by default GOMAXPROCS is 1 on the Go Playground. However it is allowed to set it to a higher value, e.g.:
runtime.GOMAXPROCS(2)
Without explicit synchronization, execution still remains non-deterministic, however you will observe a different execution order and a termination without running into a timeout:
Hello, playground
Here
Here
Here
...
<Here is printed 996 times, then:>
Finished.
Try this variant on the Go Playground.
What you will see on screen is nondeterministic. Or more precisely if by any chance the true value you pass to channel is delayed you would see some "Here".
But usually the Stdout is buffered, it means it's not printed instantaneously but the data gets accumulated and after it gets to maximum buffer size it's printed. In your case before the "here" is printed the main function is already finished thus the process finishes.
The rule of thumb is: main function must be alive otherwise all other goroutines gets killed.