This question already has answers here:
Cannot free memory once occupied by bytes.Buffer
(2 answers)
Closed 4 years ago.
I've written a simple TCP server.
The problem is that when stress-testing it, it seems that the memory usage is increasing dramatically, and not decreasing when he test is done.
When the server is started, it takes ~700KB.
During and after the stress-test, the memory usage jumps to ~7MB.
Here's my code:
package main
import (
"net"
"log"
"fmt"
"bufio"
)
func main() {
ln, err := net.Listen("tcp", ":8888")
if err != nil {
log.Fatal(err)
}
defer ln.Close()
for {
conn, err := ln.Accept()
if err != nil {
fmt.Println(err)
continue
}
go handle(conn)
}
}
func handle(conn net.Conn) {
defer conn.Close()
fmt.Println("Accepted", conn.LocalAddr())
for {
buf, err := bufio.NewReader(conn).ReadString('\n')
if err != nil {
break
}
msg := string(buf[:len(buf)-2])
fmt.Println("Received", msg)
conn.Write([]byte("OK\n"))
}
}
Any help is much appreciated.
Note: I'm using tcpkali for loading it. This is the command line:
tcpkali -em "testing\r\n" -c 100 -r 1000 -T 60 127.0.0.1:8888
EDIT: Following some comments below, I ran some tests and here are the results:
Started the server and ran tcpkali.
After the first run, RSS was at 8516.
After a second run, RSS climbed to 8572.
Server is now idle. 5 minutes later, RSS climbed to 8588.
5 more minutes later, RSS climbed to 8608, and seems stable.
After 15 minutes of break, I ran tcpkali again, and RSS climbed to 8684.
A few minutes break, another tcpkali run, RSS climbs to 8696.
A few minutes break, another tcpkali run, RSS climbs to 8704.
A few minutes break, another tcpkali run, RSS climbs to 8712.
Now, I don't know what you call this, but I call this a memory leak. Something is wrong here. No memory is freed, and RSS is keep climbing every time I run a test. Obviously, this thing cannot be deployed to production as it will eventually consume all available memory.
I also tried calling os.FreeOSMemory() but nothing happens.
My system is Go 1.9.4 on macOS 10.13.1. Is this environment related or am I missing something?
LAST UPDATE:
Following #Steffen Ullrich answer and the tests that failed on my environment, I gave it a try on Ubuntu server, and the memory is freed after a few minutes of idle time.
Seems like there's an issue with macOS.
Go does not release memory it allocated from the OS immediately. The reason is probably that allocating memory is costly (needs system calls) and the chance is high that it will be needed in the near future anyway again. But, if memory gets long enough unused it will be released eventually so that the RSS of the process decreases again.
Doing your test again with slight modifications will show this (it did at least for me):
Start you program and look at the RSS.
Run tcpkali, wait for tcpkali to end and look at the RSS again. It is much higher now since lots of memory was needed for the program to do the intended task.
Don't stop the program but run tcpkali again and wait again for it to end. When looking at the RSS you should see that it did not grow (much) further. This means that the program used the already allocated memory again and did not need to allocate new memory from the system.
Now monitor the RSS and wait. After a while (about 10 minutes on my system) you should see the RSS go down again. This is because the program has determined now that the allocated but unused memory will probably not be used any longer and returned the memory back to the OS.
Note that not all memory might be given back. According to Understanding Go Lang Memory Usage it will not return (in go 1.3) the memory used for the stacks of the go routines since it is more likely that this will be needed in the future.
For testing you might also add some debug.FreeOSMemory() (from runtime/debug) at strategic places (like when you break out of the loop in the goroutine) so that the memory gets returned earlier to the OS. But given that the lazy return of memory is for performance such explicit freeing might impact the performance.
Related
I've got a small peice of code named test.go. It counts time(ns) when doing two writings that write a same byte slice to 2 files, one with the flag syscall.O_DIRECT and the other not.
The code is below:
package main;
import (
"os"
"time"
"fmt"
"strconv"
"bytes"
"syscall"
// "os/exec"
)
func main() {
num, _ := strconv.Atoi(os.Args[1]);
writeContent:= bytes.Repeat( ([]byte)("1"), num );
t1:= time.Now().UnixNano();
fd1, err := syscall.Open("abc.txt", syscall.O_WRONLY | syscall.O_DIRECT | syscall.O_TRUNC, 0);
syscall.Write(fd1, writeContent);
if err != nil {panic(err);}
t2:= time.Now().UnixNano();
fmt.Println("sysW1:", t2-t1);
t1= time.Now().UnixNano();
fd2, err := syscall.Open("abc.txt", syscall.O_WRONLY | syscall.O_TRUNC, 0);
syscall.Write(fd2, writeContent);
if err != nil {panic(err);}
t2= time.Now().UnixNano();
fmt.Println("sysW2:", t2-t1);
}
The program is runned in linux command line like this:(after being compiled with go build ./test.go)
./test 1024
I had expected writing file with syscall.O_DIRECT flag to be faster, but the result showed that writing files with syscall.O_DIRECT flag was about 30 times slower than writing without it :(
result:
sysW1: 1107377
sysW2: 37155
Why? I tought writing with syscall.O_DIRECT does less copying and would be faster, but it now turns out to be much slower. Please help me explain it :(
PX: I will not provide playground link since the result is always 0 when running the program on the playground in some reason.
O_DIRECT doesn't do what you think. While it does less memory copying (since it doesn't copy to the cache before copying to the device driver), that doesn't give you a performance boost.
The filesystem cache ensures that the system call can return early before the data is written to the device, and buffer data to send data in larger chunks.
With O_DIRECT, the system call waits until the data is completely transferred to the device.
From the man page for the open call:
O_DIRECT (since Linux 2.4.10)
Try to minimize cache effects of the I/O to and from this
file. In general this will degrade performance, but it is
useful in special situations, such as when applications do
their own caching. File I/O is done directly to/from
user-space buffers. The O_DIRECT flag on its own makes an
effort to transfer data synchronously, but does not give
the guarantees of the O_SYNC flag that data and necessary
metadata are transferred.
See also: What does O_DIRECT really mean?
You don't need to manually release the cache after using it.
The cache is considered free available memory by the Linux kernel. If a process needs memory that is occupied by the cache, the kernel will flush/release the cache at that point. The cache doesn't "use up" memory.
Consider the following code:
go func() {
for now := range time.Tick(time.Minute) {
fmt.Println(now, statusUpdate())
}
}()
I need the for loop runs forever and never need to stop it.
Will this cause memory leak or not?
I knew if I ever need to break the for loop, it will cause memory leak. But what if I don't need to break the for loop?
The doc says
While Tick is useful for clients that have no need to shut down the Ticker, be aware that without a way to shut it down the underlying Ticker cannot be recovered by the garbage collector; it "leaks".
I just want to get it right.
First, let's see a definition of what a "memory leak" is, from Wikipedia:
In computer science, a memory leak is a type of resource leak that occurs when a computer program incorrectly manages memory allocations in a way that memory which is no longer needed is not released.
It's important to note that in the doc you quoted, it does not specifically mention a "memory leak", just a "leak" (meaning "resource leak"). The resources in question are not only the memory used by the ticker, but the goroutine that runs it. Because of this, I'll interpret this definition as applying to "resource leaks" more broadly.
As mentioned by the doc you quoted, time.Tick does not make it possible to release the ticker's resources.
So by this definition, if the ticker is no longer needed at any subsequent point in the program, then a resource leak has occurred. If the ticker is always needed for the remainder of the program after created, then it's not a leak.
Further in the Wikipedia definition, however, is this note:
A memory leak may also happen when an object is stored in memory but cannot be accessed by the running code.
Again, time.Tick does not make it possible to release the ticker's resources.
So by this continued definition, you might say that the use of time.Tick is always a resource leak.
In practical terms, as long as you range over the time.Tick without breaking, you have reasonable assurance that the ticker is going to continue to be used for the remainder of the program, and there will be no "leak". If you have any doubts of if a ticker will be used forever, then use time.NewTicker and Stop() it appropriately:
go func() {
ticker := time.NewTicker(time.Minute)
defer ticker.Stop()
for now := range ticker.C {
fmt.Println(now, statusUpdate())
// some exception
if (something) {
break
}
}
}()
Think of a large project which deals with tons of concurrent requests handled by its own goroutine. It happens that there is a bug in the code and one of these requests will cause panic due to a nil reference.
In Java, C# and many other languages, this would end up in a exception which would stop the request without any harm to other healthy requests. In go, that would crash the entire program.
AFAIK, I'd have to have recover() for every single new go routine creation. Is that the only way to prevent entire program from crashing?
UPDATE: adding recover() call for every gorouting creation seems OK. What about third-party libraries? If third party creates goroutines without recover() safe net, it seems there is NOTHING to be done.
If you go the defer-recover-all-the-things, I suggest investing some time to make sure that a clear error message is collected with enough information to promptly act on it.
Writing the panic message to stderr/stdout is not great as it will be very hard to find where the problem is. In my experience the best approach is to invest a bit of time to get your Go programs to handle errors in a reasonable way. errors.Wrap from "github.com/pkg/errors" for instance allows you to wrap all errors and get a stack-trace.
Recovering panic is often a necessary evil. Like you say, it's not ideal to crash the entire program just because one requested caused a panic. In most cases recovering panics will not back-fire, but it is possible for a program to end up in a undefined not-recoverable state that only a manual restart can fix. That being said, my suggestion in this case is to make sure your Go program exposes a way to create a core dump.
Here's how to write a core dump to stderr when SIGQUIT is sent to the Go program (eg. kill pid -QUIT)
go func() {
// Based on answers to this stackoverflow question:
// https://stackoverflow.com/questions/19094099/how-to-dump-goroutine-stacktraces
sigs := make(chan os.Signal, 1)
signal.Notify(sigs, syscall.SIGQUIT)
for {
<-sigs
fmt.Fprintln(os.Stderr, "=== received SIGQUIT ===")
fmt.Fprintln(os.Stderr, "*** goroutine dump...")
var buf []byte
var bufsize int
var stacklen int
// Create a stack buffer of 1MB and grow it to at most 100MB if
// necessary
for bufsize = 1e6; bufsize < 100e6; bufsize *= 2 {
buf = make([]byte, bufsize)
stacklen = runtime.Stack(buf, true)
if stacklen < bufsize {
break
}
}
fmt.Fprintln(os.Stderr, string(buf[:stacklen]))
fmt.Fprintln(os.Stderr, "*** end of dump")
}
}()
there is no way you can handle panic without recover function, a good practice would be using a middleware like function for your safe function, checkout this snippet
https://play.golang.org/p/d_fQWzXnlAm
I am trying to build a small tool that will allow me to run a program and track memory usage through Go. I am using r.exec = exec.Command(r.Command, r.CommandArgs...) to run the command, and runtime.MemStats to track memory usage (in a separate go routine):
func monitorRuntime() {
m := &runtime.MemStats{}
f, err := os.Create(fmt.Sprintf("mmem_%s.csv", getFileTimeStamp()))
if err != nil {
panic(err)
}
f.WriteString("Time;Allocated;Total Allocated; System Memory;Num Gc;Heap Allocated;Heap System;Heap Objects;Heap Released;\n")
for {
runtime.ReadMemStats(m)
f.WriteString(fmt.Sprintf("%s;%d;%d;%d;%d;%d;%d;%d;%d;\n", getTimeStamp(), m.Alloc, m.TotalAlloc, m.Sys, m.NumGC, m.HeapAlloc, m.HeapSys, m.HeapObjects, m.HeapReleased))
time.Sleep(5 * time.Second)
}
}
When I tested my code with simple program that just sits there (for about 12 hours), I noticed that Go is constantly allocating more memory:
System Memory
Heap Allocation
I did a few more tests such as running the monitorRuntime() function without any other code, or using pprof, such as:
package main
import (
"net/http"
_ "net/http/pprof"
)
func main() {
http.ListenAndServe(":8080", nil)
}
Yet I still noticed that memory allocation keeps going up just like in the graphs.
How can I accurately track memory usage of the program I want to run through Go?
I know one way, which I used in the past, is to use /proc/$PID/statm, but that file doesn't exist in every operating system (such as MacOS or Windows)
There isn't a way in standard Go to get the memory usage of a program called from exec.Command. runtime.ReadMemStats only returns memory tracked by the go runtime (which, in this case, is only the file handling and sprintf).
Your best bet would be to execute platform specific commands to get memory usage.
On Linux (RedHat) the following will show memory usage:
ps -No pid,comm,size,vsize,args:90
I am running a program in Go which sends data continuously after reading a file /proc/stat.
Using ioutil.ReadFile("/proc/stat")
After running for about 14 hrs i got err: too many files open /proc/stat
Click here for snippet of code.
I doubt that defer f.Close is ignored by Go sometimes or it is skipping it.
The snippet of code (in case play.golang.org dies sooner than stackoverflow.com):
package main
import ("fmt";"io/ioutil")
func main() {
for {
fmt.Println("Hello, playground")
fData,err := ioutil.ReadFile("/proc/stat")
if err != nil {
fmt.Println("Err is ",err)
}
fmt.Println("FileData",string(fData))
}
}
The reason probably is that somewhere in your program:
you are forgetting to close files, or
you are leaning on the garbage collector to automatically close files on object finalization, but Go's conservative garbage collector fails to do so. In this case you should check your program's memory consumption (whether it is steadily increasing while the program is running).
In either case, try to check the contents of /proc/PID/fd to see whether the number of open files is increasing while the program is running.
If you are sure you Do the f.Close(),it still has the proble,Maybe it is because your other connection,for example the connection to MYSQL,also will be cause the problem,especially,in a loop,and you forget to close the connection.
Always do :
db.connection....
**defer db.Close()**
If it is in loop
loop
db.connection....
**defer db.Close()**
end
Do not put the db.connection before the loop