I have a program where memory keep growing. I'm not sure if it's a memory leak or just a buffer that keep growing.
I successfully isolated the problem, but I still can't find the problem.
There is some strange behavoir: if I remove the compression part, the leak disappear.
So I assume it's there. BUT if I (only) remove the clause with chanTest in the switch, the leak disappear too.
Could someone confirm the problem and explain me why it has such behavoir?
I'm using go1.0.3
Thanks!
Here is the program: ( it compress some dummy data every 100ms )
package main
import (
"bytes"
"compress/zlib"
"fmt"
"time"
)
func main() {
timeOut := time.NewTicker(100 * time.Millisecond)
chanTest := make(chan int32)
for {
L: for { // timer part
select {
case resp := <- chanTest: // strange clause
fmt.Println("received stuff", resp)
case <-timeOut.C:
fmt.Println("break")
break L
}
}
timeOut = time.NewTicker(100 * time.Millisecond)
// compression part
data := []byte{1, 2, 3, 4, 5, 6, 7}
var b bytes.Buffer
w := zlib.NewWriter(&b)
w.Write(data)
w.Close()
b.Reset()
}
}
You're starting a new Ticker inside the loop without calling .Stop() on the original. Since the Ticker runs at an interval, you end up with multiple Tickers continuing to run at the same time.
Calling .Stop() to halt the previous one would technically work:
timeOut.Stop()
timeOut = time.NewTicker(100 * time.Millisecond)
...but seems to defeat the purpose of it. Instead, just don't make new Tickers in the loop, and the original will continue to run:
// timeOut = time.NewTicker(100 * time.Millisecond) // not needed
There is a fix for this issue in golang tip newer than 25-Feb-2013 (revision 1c50db40d078). If you run hg pull; hg update to update Go's source code and recompile your Go distribution then the memory consumption issue should disappear.
However, this alone does not make the program correct - it only fixes the unusually high memory consumption. A correct implementation needs to be calling timeout.Stop().
Related
In Go, a sync.Mutex or chan is used to prevent concurrent access of shared objects. However, in some cases I am just interested in the "latest" value of a variable or field of an object.
Or I like to write a value and do not care if another go-routine overwrites it later or has just overwritten it before.
Update: TLDR; Just don't do this. It is not safe. Read the answers, comments, and linked documents!
Update 2021: The Go memory model is going to be specified more thoroughly and there are three great articles by Russ Cox that will teach you more about the surprising effects of unsynchronized memory access. These articles summarize a lot of the below discussions and learnings.
Here are two variants good and bad of an example program, where both seem to produce "correct" output using the current Go runtime:
package main
import (
"flag"
"fmt"
"math/rand"
"time"
)
var bogus = flag.Bool("bogus", false, "use bogus code")
func pause() {
time.Sleep(time.Duration(rand.Uint32()%100) * time.Millisecond)
}
func bad() {
stop := time.After(100 * time.Millisecond)
var name string
// start some producers doing concurrent writes (DANGER!)
for i := 0; i < 10; i++ {
go func(i int) {
pause()
name = fmt.Sprintf("name = %d", i)
}(i)
}
// start consumer that shows the current value every 10ms
go func() {
tick := time.Tick(10 * time.Millisecond)
for {
select {
case <-stop:
return
case <-tick:
fmt.Println("read:", name)
}
}
}()
<-stop
}
func good() {
stop := time.After(100 * time.Millisecond)
names := make(chan string, 10)
// start some producers concurrently writing to a channel (GOOD!)
for i := 0; i < 10; i++ {
go func(i int) {
pause()
names <- fmt.Sprintf("name = %d", i)
}(i)
}
// start consumer that shows the current value every 10ms
go func() {
tick := time.Tick(10 * time.Millisecond)
var name string
for {
select {
case name = <-names:
case <-stop:
return
case <-tick:
fmt.Println("read:", name)
}
}
}()
<-stop
}
func main() {
flag.Parse()
if *bogus {
bad()
} else {
good()
}
}
The expected output is as follows:
...
read: name = 3
read: name = 3
read: name = 5
read: name = 4
...
Any combination of read: and read: name=[0-9] is correct output for this program. Receiving any other string as output would be an error.
When running this program with go run --race bogus.go it is safe.
However, go run --race bogus.go -bogus warns of the concurrent reads and writes.
For map types and when appending to slices I always need a mutex or a similar method of protection to avoid segfaults or unexpected behavior. However, reading and writing literals (atomic values) to variables or field values seems to be safe.
Question: Which Go data types can I safely read and safely write concurrently without a mutext and without producing segfaults and without reading garbage from memory?
Please explain why something is safe or unsafe in Go in your answer.
Update: I rewrote the example to better reflect the original code, where I had the the concurrent writes issue. The important leanings are already in the comments. I will accept an answer that summarizes these learnings with enough detail (esp. on the Go-runtime).
However, in some cases I am just interested in the latest value of a variable or field of an object.
Here is the fundamental problem: What does the word "latest" mean?
Suppoose that, mathematically speaking, we have a sequence of values Xi, with 0 <= i < N. Then obviously Xj is "later than" Xi if j > i. That's a nice simple definition of "latest" and is probably the one you want.
But when two separate CPUs within a single machine—including two goroutines in a Go program—are working at the same time, time itself loses meaning. We cannot say whether i < j, i == j, or i > j. So there is no correct definition for the word latest.
To solve this kind of problem, modern CPU hardware, and Go as a programming language, gives us certain synchronization primitives. If CPUs A and B execute memory fence instructions, or synchronization instructions, or use whatever other hardware provisions exist, the CPUs (and/or some external hardware) will insert whatever is required for the notion of "time" to regain its meaning. That is, if the CPU uses barrier instructions, we can say that a memory load or store that was executed before the barrier is a "before" and a memory load or store that is executed after the barrier is an "after".
(The actual implementation, in some modern hardware, consists of load and store buffers that can rearrange the order in which loads and stores go to memory. The barrier instruction either synchronizes the buffers, or places an actual barrier in them, so that loads and stores cannot move across the barrier. This particular concrete implementation gives an easy way to think about the problem, but isn't complete: you should think of time as simply not existing outside the hardware-provided synchronization, i.e., all loads from, and stores to, some location are happening simultaneously, rather than in some sequential order, except for these barriers.)
In any case, Go's sync package gives you a simple high level access method to these kinds of barriers. Compiled code that executes before a mutex Lock call really does complete before the lock function returns, and the code that executes after the call really does not start until after the lock function returns.
Go's channels provide the same kinds of before/after time guarantees.
Go's sync/atomic package provides much lower level guarantees. In general you should avoid this in favor of the higher level channel or sync.Mutex style guarantees. (Edit to add note: You could use sync/atomic's Pointer operations here, but not with the string type directly, as Go strings are actually implemented as a header containing two separate values: a pointer, and a length. You could solve this with another layer of indirection, by updating a pointer that points to the string object. But before you even consider doing that, you should benchmark the use of the language's preferred methods and verify that these are a problem, because code that works at the sync/atomic level is hard to write and hard to debug.)
Which Go data types can I safely read and safely write concurrently without a mutext and without producing segfaults and without reading garbage from memory?
None.
It really is that simple: You cannot, under no circumstance whatsoever, read and write concurrently to anything in Go.
(Btw: Your "correct" program is not correct, it is racy and even if you get rid of the race condition it would not deterministically produce the output.)
Why can't you use channels
package main
import (
"fmt"
"sync"
)
func main() {
var wg sync.WaitGroup // wait group to close channel
var buffer int = 1 // buffer of the channel
// channel to get the share data
cName := make(chan string, buffer)
for i := 0; i < 10; i++ {
wg.Add(1) // add to wait group
go func(i int) {
cName <- fmt.Sprintf("name = %d", i)
wg.Done() // decrease wait group.
}(i)
}
go func() {
wg.Wait() // wait of wait group to be 0
close(cName) // close the channel
}()
// process all the data
for n := range cName {
println("read:", n)
}
}
The above code returns the following output
read: name = 0
read: name = 5
read: name = 1
read: name = 2
read: name = 3
read: name = 4
read: name = 7
read: name = 6
read: name = 8
read: name = 9
https://play.golang.org/p/R4n9ssPMOeS
Article about channels
I'm trying to make a channel with never ending ticker, but it works only 2 times.
Could you help me to understand where is the problem?
Code:
package main
import (
"fmt"
"time"
)
var mark = [2]float64{8.9876, 762.098568}
func tick(out chan <- [2]float64){
c := time.NewTicker(time.Millisecond *500)
for range c.C{
out <- mark
}
}
func main() {
fmt.Println("Start")
md := make(chan [2]float64)
go tick(md)
for range <-md{
fmt.Println(<-md)
}
}
Output:
Start
[8.9876 762.098568]
[8.9876 762.098568]
Example: https://play.golang.org/p/P2FaUwbW-3
This:
for range <-md{
is not the same as:
for range md{
The latter ranges over the channel (what you want), while the former ranges over the value received from the channel when the loop starts, which happens to be a two-element array, hence the two executions. You're also ignoring the value received from the channel in the for statement, and reading from it again in the loop body, ignoring every other message on the channel (though this makes no difference in the example, since every value is identical, it would make a significant difference in practice). What you really want is:
for foo := range md{
fmt.Println(foo)
}
Here's a working version of your playground example, slightly modified to avoid "program took too long" errors because in its current form it never stops: https://play.golang.org/p/RSUJFvluU5
I'm playing around with the code presented on https://tour.golang.org/concurrency/5. My idea was that I could simplify the code by getting rid of the quit channel and still keep the correct program behavior - just for learning purposes.
Here is the code I've got (simplified it even more for better readability):
package main
import "fmt"
import "time"
func sendNumbers(c chan int) {
for i := 0; i < 2; i++ {
c <- i
}
}
func main() {
c := make(chan int)
go func() {
for i := 0; i < 2; i++ {
fmt.Println(<-c)
}
}()
time.Sleep(0 * time.Second)
sendNumbers(c)
}
In this code, the go routine I spawn should be able to receive 2 numbers before it returns. The sendNumbers() function I call next sends exactly 2 numbers to the channel c.
So, the expected output of the program is 2 lines: 0 and 1. However, what I'm getting when I run the code on the page, is just a single line containing 0.
Note the
time.Sleep(0 * time.Second)
though that I deliberately added before calling sendNumbers(c). If I change it to
time.Sleep(1 * time.Second)
The output becomes as expected:
0
1
So, I'm confused with what happens. Can someone explain what is going on? Shouldn't both sends and receives block regardless of how much time is passed before I call sendNumbers()?
In Go, the program exits as soon as the main function returns regardless of whether other goroutines are still running or not. If you want to make sure the main function does not exit prematurely, you need some synchronization mechanism. https://nathanleclaire.com/blog/2014/02/15/how-to-wait-for-all-goroutines-to-finish-executing-before-continuing/
You don’t absolutely have to use synchronization primitives, you could also do it with channels only, arguably a more idiomatic way to do it.
I have something like this mock (code below) which distributes the same keyword out to multiple goroutines, except the goroutines all take different amount of times doing things with the keyword but can operate independently of each other so they don't need any synchronization. The solution given below to distribute clearly synchronizes the goroutines.
I just want to toss this idea out there to see how other people would deal with this type of distribution, as I assume it is fairly common and someone else has thought about it before.
Here are some other solutions I have thought up and why they seem kinda meh to me:
One goroutine for each keyword
Each time a new keyword comes in spawn a goroutine to handle the distribution
Give the keyword a bitmask or something for each goroutine to update
This way once all of the workers have touched the keyword it can be deleted and we can move on
Give each worker its own stack to work off of
This seems kinda appealing, just give each worker a stack to add each keyword to, but we would eventually run into a problem of a ton of memory being taken up since it is planned to run so long
The problem with all of these is that my code is supposed to run for a long time, unwatched, and that would lead to either a huge build up of keywords or goroutines due to the lazy worker taking longer than the others. It almost seems like it'd be nice to give each worker its own Amazon SQS queue or implement something similar to that myself.
EDIT:
Store the keyword outside the program
I just thought of doing it this way instead, I could perhaps just store the keyword outside the program until they all grab it and then delete it once it has been used up. This sits ok with me actually, I don't have a problem with using up disk space
Anyway here is an example of the approach that waits for all to finish:
package main
import (
"flag"
"fmt"
"math/rand"
"os"
"os/signal"
"strconv"
"time"
)
var (
shutdown chan struct{}
count = flag.Int("count", 5, "number to run")
)
type sleepingWorker struct {
name string
sleep time.Duration
ch chan int
}
func NewQuicky(n string) sleepingWorker {
var rq sleepingWorker
rq.name = n
rq.ch = make(chan int)
rq.sleep = time.Duration(rand.Intn(5)) * time.Second
return rq
}
func (r sleepingWorker) Work() {
for {
fmt.Println(r.name, "is about to sleep, number:", <-r.ch)
time.Sleep(r.sleep)
}
}
func NewLazy() sleepingWorker {
var rq sleepingWorker
rq.name = "Lazy slow worker"
rq.ch = make(chan int)
rq.sleep = 20 * time.Second
return rq
}
func distribute(gen chan int, workers ...sleepingWorker) {
for kw := range gen {
for _, w := range workers {
fmt.Println("sending keyword to:", w.name)
select {
case <-shutdown:
return
case w.ch <- kw:
fmt.Println("keyword sent to:", w.name)
}
}
}
}
func main() {
flag.Parse()
shutdown = make(chan struct{})
go func() {
c := make(chan os.Signal, 1)
signal.Notify(c, os.Interrupt)
<-c
close(shutdown)
}()
x := make([]sleepingWorker, *count)
for i := 0; i < (*count)-1; i++ {
x[i] = NewQuicky(strconv.Itoa(i))
go x[i].Work()
}
x[(*count)-1] = NewLazy()
go x[(*count)-1].Work()
gen := make(chan int)
go distribute(gen, x...)
go func() {
i := 0
for {
i++
select {
case <-shutdown:
return
case gen <- i:
}
}
}()
<-shutdown
os.Exit(0)
}
Let's assume I understand the problem correctly:
There's not too much you can do about it I'm afraid. You have limited resources (assuming all resources are limited) so if data to your input is written faster then you process it, there will be some synchronisation needed. At the end the whole process will run as quickly as the slowest worker anyway.
If you really need data from the workers available as soon as possible, the best you can do is to add some kind of buffering. But the buffer must be limited in size (even if you run in the cloud it would be limited by your wallet) so assuming never ending torrent of input it will only postpone the choke until some time in the future where you will start seeing "synchronisation" again.
All the ideas you presented in your questions are based on buffering the data. Even if you run a routine for every keyword-worker pair, this will buffer one element in every routine and, unless you implement the limit on total number of routines, you'll run out of memory. And even if you always leave some room for the quickest worker to spawn a new routine, the input queue won't be able to deliver new items as it would be choked on the slowest worker.
Buffering would solve your problem if on average you input is slower than processing time, but you have occasional spikes. If your buffer is big enough you can than accommodate the increase of throughput and maybe your quickest worker won't notice a thing.
Solution?
As go comes with buffered channels, this is the easiest to implement (also suggested by icza in the comment). Just give each worker a buffer. If you know which worker is the slowest, you can give it a bigger buffer. In this scenario you're limited by the memory of your machine.
If you're not happy with the single-machine memory limit then yes, per one of your ideas, you can "simply" store the buffer (queue) for each worker on the hard drive. But this is also limited and just postpones the blocking scenario until later. This is essentially the same as your Amazon SQS proposal (you could keep buffer in the cloud, but you need either limit it reasonably or prepare for the bill.)
The final note, depending on the system you're building, it might be not a good idea to buffer items in such a massive scale allowing to build up the backlog for the slower workers – it's often not desirable to have a worker hours, days, weeks behind the input flow and this is what would happen with an infinite buffer. The real answer then would be: improve your slowest worker to process things faster. (And add some buffering to improve the experience.)
I need to wait until x.Addr is being updated but it seems the for loop is not run. I suspect this is due the go scheduler and I'm wondering why it works this way or if there is any way I can fix it(without channels).
package main
import "fmt"
import "time"
type T struct {
Addr *string
}
func main() {
x := &T{}
go update(x)
for x.Addr == nil {
if x.Addr != nil {
break
}
}
fmt.Println("Hello, playground")
}
func update(x *T) {
time.Sleep(2 * time.Second)
y := ""
x.Addr = &y
}
There are two (three) problems with your code.
First, you are right that there is no point in the loop at which you give control to the scheduler and such it can't execute the update goroutine. To fix this you can set GOMAXPROCS to something bigger than one and then multiple goroutines can run in parallel.
(However, as it is this won't help as you pass x by value to the update function which means that the main goroutine will never see the update on x. To fix this problem you have to pass x by pointer. Now obsolete as OP fixed the code.)
Finally, note that you have a data race on Addr as you are not using atomic loads and stores.