In Go, when using a make statement,
for example, allocating megabytes of memory
make([]byte, 1024*1024*d)
Is there a way to determine how much memory is free, before asking for more memory?
Thank you for all your input.
I have decided to use a Max_Memory configuration option, since the use case for this is to utilize n Megabytes of memory up to about 75% max available, on test servers, which are only running this application, as a way to trigger autoscaling in a test environment.
Yes there is. You can use the gopsutil package:
package main
import (
"fmt"
"github.com/shirou/gopsutil/mem"
)
func main() {
vm, err := mem.VirtualMemory()
if err != nil {
panic(err)
}
fmt.Printf("Total:%d, Available:%d, Used:%d", vm.Total, vm.Available, vm.Used)
}
There are also lots of ways to get this information that are OS-specific. None of them are perfect, but they'll give you something.
You could also have Max Memory as a configuration variable and use: http://godoc.org/runtime#MemStats.
Related
I've got a small peice of code named test.go. It counts time(ns) when doing two writings that write a same byte slice to 2 files, one with the flag syscall.O_DIRECT and the other not.
The code is below:
package main;
import (
"os"
"time"
"fmt"
"strconv"
"bytes"
"syscall"
// "os/exec"
)
func main() {
num, _ := strconv.Atoi(os.Args[1]);
writeContent:= bytes.Repeat( ([]byte)("1"), num );
t1:= time.Now().UnixNano();
fd1, err := syscall.Open("abc.txt", syscall.O_WRONLY | syscall.O_DIRECT | syscall.O_TRUNC, 0);
syscall.Write(fd1, writeContent);
if err != nil {panic(err);}
t2:= time.Now().UnixNano();
fmt.Println("sysW1:", t2-t1);
t1= time.Now().UnixNano();
fd2, err := syscall.Open("abc.txt", syscall.O_WRONLY | syscall.O_TRUNC, 0);
syscall.Write(fd2, writeContent);
if err != nil {panic(err);}
t2= time.Now().UnixNano();
fmt.Println("sysW2:", t2-t1);
}
The program is runned in linux command line like this:(after being compiled with go build ./test.go)
./test 1024
I had expected writing file with syscall.O_DIRECT flag to be faster, but the result showed that writing files with syscall.O_DIRECT flag was about 30 times slower than writing without it :(
result:
sysW1: 1107377
sysW2: 37155
Why? I tought writing with syscall.O_DIRECT does less copying and would be faster, but it now turns out to be much slower. Please help me explain it :(
PX: I will not provide playground link since the result is always 0 when running the program on the playground in some reason.
O_DIRECT doesn't do what you think. While it does less memory copying (since it doesn't copy to the cache before copying to the device driver), that doesn't give you a performance boost.
The filesystem cache ensures that the system call can return early before the data is written to the device, and buffer data to send data in larger chunks.
With O_DIRECT, the system call waits until the data is completely transferred to the device.
From the man page for the open call:
O_DIRECT (since Linux 2.4.10)
Try to minimize cache effects of the I/O to and from this
file. In general this will degrade performance, but it is
useful in special situations, such as when applications do
their own caching. File I/O is done directly to/from
user-space buffers. The O_DIRECT flag on its own makes an
effort to transfer data synchronously, but does not give
the guarantees of the O_SYNC flag that data and necessary
metadata are transferred.
See also: What does O_DIRECT really mean?
You don't need to manually release the cache after using it.
The cache is considered free available memory by the Linux kernel. If a process needs memory that is occupied by the cache, the kernel will flush/release the cache at that point. The cache doesn't "use up" memory.
This question already has answers here:
Cannot free memory once occupied by bytes.Buffer
(2 answers)
Closed 4 years ago.
I've written a simple TCP server.
The problem is that when stress-testing it, it seems that the memory usage is increasing dramatically, and not decreasing when he test is done.
When the server is started, it takes ~700KB.
During and after the stress-test, the memory usage jumps to ~7MB.
Here's my code:
package main
import (
"net"
"log"
"fmt"
"bufio"
)
func main() {
ln, err := net.Listen("tcp", ":8888")
if err != nil {
log.Fatal(err)
}
defer ln.Close()
for {
conn, err := ln.Accept()
if err != nil {
fmt.Println(err)
continue
}
go handle(conn)
}
}
func handle(conn net.Conn) {
defer conn.Close()
fmt.Println("Accepted", conn.LocalAddr())
for {
buf, err := bufio.NewReader(conn).ReadString('\n')
if err != nil {
break
}
msg := string(buf[:len(buf)-2])
fmt.Println("Received", msg)
conn.Write([]byte("OK\n"))
}
}
Any help is much appreciated.
Note: I'm using tcpkali for loading it. This is the command line:
tcpkali -em "testing\r\n" -c 100 -r 1000 -T 60 127.0.0.1:8888
EDIT: Following some comments below, I ran some tests and here are the results:
Started the server and ran tcpkali.
After the first run, RSS was at 8516.
After a second run, RSS climbed to 8572.
Server is now idle. 5 minutes later, RSS climbed to 8588.
5 more minutes later, RSS climbed to 8608, and seems stable.
After 15 minutes of break, I ran tcpkali again, and RSS climbed to 8684.
A few minutes break, another tcpkali run, RSS climbs to 8696.
A few minutes break, another tcpkali run, RSS climbs to 8704.
A few minutes break, another tcpkali run, RSS climbs to 8712.
Now, I don't know what you call this, but I call this a memory leak. Something is wrong here. No memory is freed, and RSS is keep climbing every time I run a test. Obviously, this thing cannot be deployed to production as it will eventually consume all available memory.
I also tried calling os.FreeOSMemory() but nothing happens.
My system is Go 1.9.4 on macOS 10.13.1. Is this environment related or am I missing something?
LAST UPDATE:
Following #Steffen Ullrich answer and the tests that failed on my environment, I gave it a try on Ubuntu server, and the memory is freed after a few minutes of idle time.
Seems like there's an issue with macOS.
Go does not release memory it allocated from the OS immediately. The reason is probably that allocating memory is costly (needs system calls) and the chance is high that it will be needed in the near future anyway again. But, if memory gets long enough unused it will be released eventually so that the RSS of the process decreases again.
Doing your test again with slight modifications will show this (it did at least for me):
Start you program and look at the RSS.
Run tcpkali, wait for tcpkali to end and look at the RSS again. It is much higher now since lots of memory was needed for the program to do the intended task.
Don't stop the program but run tcpkali again and wait again for it to end. When looking at the RSS you should see that it did not grow (much) further. This means that the program used the already allocated memory again and did not need to allocate new memory from the system.
Now monitor the RSS and wait. After a while (about 10 minutes on my system) you should see the RSS go down again. This is because the program has determined now that the allocated but unused memory will probably not be used any longer and returned the memory back to the OS.
Note that not all memory might be given back. According to Understanding Go Lang Memory Usage it will not return (in go 1.3) the memory used for the stacks of the go routines since it is more likely that this will be needed in the future.
For testing you might also add some debug.FreeOSMemory() (from runtime/debug) at strategic places (like when you break out of the loop in the goroutine) so that the memory gets returned earlier to the OS. But given that the lazy return of memory is for performance such explicit freeing might impact the performance.
On any given computer (OSX, WIn, Lin, etc) there are any number of connected network adapters... whether it's Wi-Fi, BlueTooth, Ethernet or other... And depending on the routing there may be multiple active devices.
In my NARROW use case I want to know what the current CONNECTED default adapter type (Wi-Fi etc...) although once that's known there are easily some others as well as some details.
Here is an example shell script that mostly works and converting it to Go is easy enough... it just seems to me there must be a native GO way.
You can use Interfaces() from net package
Sample:
package main
import (
"fmt"
"net"
)
func main() {
l, err := net.Interfaces()
if err != nil {
panic(err)
}
for _, f := range l {
if f.Flags&net.FlagUp > 0 {
fmt.Printf("%s is up\n", f.Name)
}
}
}
On Linux, you'd need to use the RTNETLINK interface, and there appears to be
a number of packages
which implement wrappers around this layer.
Basically, you'd issue an RTM_GETLINK request and then look at the type of each interface (ifi_type field in that manual).
The available types are here — you'd need those which are 802.2 (Ethernet) and 802.11 (Wi-Fi).
You could also try to implement a "low-tech" approach by first using
net.Interfaces() and then querying their classes by trying to read
a file named "/sys/class/net/{iface_name}/type" (as explained here). That type is what the ifi_type field described above contains.
Note that this approach has some drawbacks:
Interfaces might came and go anytime, so there's an inherent race between
obtaining their list via net.Interfaces() and querying each interface
via that /sys virtual filesystem.
That /sys filesystem might not be mounted on a particular system.
That would be quite unusual for a typical desktop or a server system
but not so much for some "weird" installation (such as embedded etc).
I am trying to build a small tool that will allow me to run a program and track memory usage through Go. I am using r.exec = exec.Command(r.Command, r.CommandArgs...) to run the command, and runtime.MemStats to track memory usage (in a separate go routine):
func monitorRuntime() {
m := &runtime.MemStats{}
f, err := os.Create(fmt.Sprintf("mmem_%s.csv", getFileTimeStamp()))
if err != nil {
panic(err)
}
f.WriteString("Time;Allocated;Total Allocated; System Memory;Num Gc;Heap Allocated;Heap System;Heap Objects;Heap Released;\n")
for {
runtime.ReadMemStats(m)
f.WriteString(fmt.Sprintf("%s;%d;%d;%d;%d;%d;%d;%d;%d;\n", getTimeStamp(), m.Alloc, m.TotalAlloc, m.Sys, m.NumGC, m.HeapAlloc, m.HeapSys, m.HeapObjects, m.HeapReleased))
time.Sleep(5 * time.Second)
}
}
When I tested my code with simple program that just sits there (for about 12 hours), I noticed that Go is constantly allocating more memory:
System Memory
Heap Allocation
I did a few more tests such as running the monitorRuntime() function without any other code, or using pprof, such as:
package main
import (
"net/http"
_ "net/http/pprof"
)
func main() {
http.ListenAndServe(":8080", nil)
}
Yet I still noticed that memory allocation keeps going up just like in the graphs.
How can I accurately track memory usage of the program I want to run through Go?
I know one way, which I used in the past, is to use /proc/$PID/statm, but that file doesn't exist in every operating system (such as MacOS or Windows)
There isn't a way in standard Go to get the memory usage of a program called from exec.Command. runtime.ReadMemStats only returns memory tracked by the go runtime (which, in this case, is only the file handling and sprintf).
Your best bet would be to execute platform specific commands to get memory usage.
On Linux (RedHat) the following will show memory usage:
ps -No pid,comm,size,vsize,args:90
I'm testing this Go code on my VirtualBoxed Ubuntu 11.4
package main
import ("fmt";"time";"big")
var c chan *big.Int
func sum( start,stop,step int64) {
bigStop := big.NewInt(stop)
bigStep := big.NewInt(step)
bigSum := big.NewInt(0)
for i := big.NewInt(start);i.Cmp(bigStop)<0 ;i.Add(i,bigStep){
bigSum.Add(bigSum,i)
}
c<-bigSum
}
func main() {
s := big.NewInt( 0 )
n := time.Nanoseconds()
step := int64(4)
c = make( chan *big.Int , int(step))
stop := int64(100000000)
for j:=int64(0);j<step;j++{
go sum(j,stop,step)
}
for j:=int64(0);j<step;j++{
s.Add(s,<-c)
}
n = time.Nanoseconds() - n
fmt.Println(s,float64(n)/1000000000.)
}
Ubuntu has access to all my 4 cores. I checked this with simultaneous run of several executables and System Monitor.
But when I'm trying to run this code, it's using only one core and is not gaining any profit of parallel processing.
What I'm doing wrong?
You probably need to review the Concurrency section of the Go FAQ, specifically these two questions, and work out which (if not both) apply to your case:
Why doesn't my multi-goroutine program
use multiple CPUs?
You must set the GOMAXPROCS shell environment
variable or use the similarly-named function
of the runtime package to allow the run-time
support to utilize more than one OS thread.
Programs that perform parallel computation
should benefit from an increase in GOMAXPROCS.
However, be aware that concurrency is not parallelism.
Why does using GOMAXPROCS > 1
sometimes make my program slower?
It depends on the nature of your
program. Programs that contain several
goroutines that spend a lot of time
communicating on channels will
experience performance degradation
when using multiple OS threads. This
is because of the significant
context-switching penalty involved in
sending data between threads.
Go's goroutine scheduler is not as
good as it needs to be. In future, it
should recognize such cases and
optimize its use of OS threads. For
now, GOMAXPROCS should be set on a
per-application basis.
For more detail on this topic see the
talk entitled Concurrency is not Parallelism.