I have been trying to dive in to Go (golang) performance analysis, based on articles like https://software.intel.com/en-us/blogs/2014/05/10/debugging-performance-issues-in-go-programs .
However, in the actual profiled programs, the generated CPU profiles have very little information. The go tool either tells that the profile is empty or it has no information about any function calls. This happens on both OS X and Linux.
I generated a minimal example of this situation - I am gathering the profile in a similar manner and facing the same issues in actual programs, too.
Here's the source code for miniprofile/main.go:
package main
import (
"fmt"
"os"
"runtime/pprof"
)
func do_something(prev string, limit int) {
if len(prev) < limit {
do_something(prev+"a", limit)
}
}
func main() {
f, err := os.Create("./prof")
if err != nil {
fmt.Println(err)
os.Exit(1)
}
pprof.StartCPUProfile(f)
defer pprof.StopCPUProfile()
do_something("", 100000)
}
I am expecting to see a CPU profile that tells that almost all the time has been spent on different recursive calls to do_something.
However, this happens (the minimal app above is called miniprofile) - not very useful:
$ go version
go version go1.6.2 darwin/amd64
$ go install .
$ miniprofile
$ go tool pprof --text prof
1.91s of 1.91s total ( 100%)
flat flat% sum% cum cum%
1.91s 100% 100% 1.91s 100%
Am I doing something in a horribly wrong way?
You're missing the binary argument to pprof:
go tool pprof --text miniprofile prof
Related
I am working on a small web server that serves files and provides access to each user's home directory.
If the source was to be in C I had the option of answering each request under different threads and to make sure each thread gets to run with the user of the caller as its users.
Is there any approach to achieve something similar to that in Go?
Ideally, the part of the code that handles the request, the goroutine or the method that gets called should be run under the user account of the caller.
I have done some research and it seems in Go we can stick a single goroutine to the current thread but I can't see how it is possible to create a new thread and then attach a goroutine to that thread.
It is not possible to run a goroutine or method as a different user because they both run within the same context as the parent process. Goroutines are equivalent to green threads and don't even necessarily spawn off proper OS thread per routine.
This answer might also depend on OS, but I don't think this will work on windows either.
if you are spawning another process via the cmd package, then this answer may be useful
Running external commands through os/exec under another user
Yes, you can do that with the use of the Linux syscall setuid (not the built in function setuid). I just found this question and thought that it has to be possible, as I use this in other programming languages too. So I got my problem solved and wanted to report back how to do this.
However, it is correct what SJP wrote about the threads and there lies exactly the answer to my problem, but it will not solve your problem, due to the threading issue - whole story in this very long issue 1435. Therein is also a suggestion in how to solve a specific subset of the setuid problem and that solved my problem.
But back to code ... you need to call LockOSThread in order to fix the current go routine to the thread you're currently executing in and in that, you can change the context with the syscall setuid.
Here is a working example for Linux:
package main
import (
"fmt"
"log"
"os"
"runtime"
"sync"
"syscall"
"time"
)
func printUID() {
fmt.Printf("Real UID: %d\n", syscall.Getuid())
fmt.Printf("Effective UID: %d\n", syscall.Geteuid())
}
func main() {
printUID()
var wg sync.WaitGroup
wg.Add(2)
go func(wg *sync.WaitGroup) {
defer wg.Done()
time.Sleep(2 * time.Second)
printUID()
}(&wg)
go func(wg *sync.WaitGroup) {
runtime.LockOSThread()
defer runtime.UnlockOSThread()
defer wg.Done()
_, _, serr := syscall.Syscall(syscall.SYS_SETUID, 1, 0, 0)
if serr != 0 {
log.Fatal(serr)
os.Exit(1)
}
printUID()
}(&wg)
wg.Wait()
printUID()
}
You will receive operation not supported if you use syscall.Setuid:
serr := syscall.Setuid(1)
instead of
_, _, serr := syscall.Syscall(syscall.SYS_SETUID, 1, 0, 0)
[This answer is similar to the one by #A.Steinel but, alas, I have insufficient reputation to actually comment on that one. Hopefully, this offers a little more of a complete worked example and, importantly, demonstrates keeping the runtime free of the confusion of threads running with different UIDs.]
First, to strictly do what you asked requires a number of hacks and isn't all that secure...
[Go likes to operate with POSIX semantics, and what you want to do is break POSIX semantics by operating with two or more UIDs at the same time in a single process. Go wants POSIX semantics because it runs goroutines on whatever thread is available, and the runtime needs them to all behave the same for this to work reliably. Since Linux's setuid() syscall doesn't honor POSIX semantics, Go opted to not implement syscall.Setuid() until very recently when it became possible to implement it with POSIX semantics in go1.16.
Note, glibc, if you call setuid(), wraps the syscall itself with a fix-up mechanism (glibc/nptl/setxid) and will change the UID values for all the threads in the program simultaneously. So, even in C, you will have to do some hacking to work around this detail.]
That being said, you can make goroutines work the way you want with the runtime.LockOSThread() call, but not confuse the Go runtime by discarding the locked threads immediately after each specialized use.
Something like this (call it uidserve.go):
// Program uidserve serves content as different uids. This is adapted
// from the https://golang.org/pkg/net/http/#ListenAndServe example.
package main
import (
"fmt"
"log"
"net/http"
"runtime"
"syscall"
)
// Simple username to uid mapping.
var prefixUIDs = map[string]uintptr{
"apple": 100,
"banana": 101,
"cherry": 102,
}
type uidRunner struct {
uid uintptr
}
func (u *uidRunner) ServeHTTP(w http.ResponseWriter, r *http.Request) {
runtime.LockOSThread()
// Note, we never runtime.UnlockOSThread().
if _, _, e := syscall.RawSyscall(syscall.SYS_SETUID, u.uid, 0, 0); e != 0 {
http.Error(w, "permission problem", http.StatusInternalServerError)
return
}
fmt.Fprintf(w, "query %q executing as UID=%d\n", r.URL.Path, syscall.Getuid())
}
func main() {
for u, uid := range prefixUIDs {
h := &uidRunner{uid: uid}
http.Handle(fmt.Sprint("/", u, "/"), h)
}
http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
fmt.Fprintf(w, "general query %q executing as UID=%d\n", r.URL.Path, syscall.Getuid())
})
log.Fatal(http.ListenAndServe(":8080", nil))
}
Build it like this:
$ go build uidserve.go
Next, to get this to work, you have to grant this program some privilege. That is do one or the other of (setcap is a tool from the libcap suite):
$ sudo /sbin/setcap cap_setuid=ep ./uidserve
or, alternatively, the more traditional way of running setuid-root:
$ sudo chown root ./uidserve
$ sudo chmod +s ./uidserve
Now, if you run ./uidserve and connect to your browser to localhost:8080 you can try fetching the following URLs:
localhost:8080/something which shows something like general query "/something" executing as UID=your UID here.
localhost:8080/apple/pie which shows something like query "/apple/pie" executing as UID=100.
etc.
Hope that helps show how to do what you asked. [Since it involves lots of hacks, however, I wouldn't recommend doing this for real though...]
I'm trying to retrieve the current uptime of my Go application.
I've seen there's a package syscall which provides a type Sysinfo_t and a method Sysinfo(*Sysinfo_t) which apparently allows you to retrieve the Uptime (since it's a field of the Sysinfo_t struct)
What I've done so far is:
sysi := &syscall.Sysinfo_t{}
if err := syscall.Sysinfo(sysi); err != nil {
return http.StatusInternalServerError, nil
}
The problem is that at compile time I get this:
/path/to/file/res_system.go:43: undefined: syscall.Sysinfo_t
/path/to/file/res_system.go:45: undefined: syscall.Sysinfo
I've searched a bit and apparently that method and type are available only on Linux and I need the application to run both on Linux and OsX (which I'm currently using).
Is there a cross-compatible way to retrieve the application uptime?
NOTE: I'd rather not use any third party libraries (unless they're absolutely necessary)
Simple way to get uptime is to store service start time:
https://play.golang.org/p/by_nkvhzqD
package main
import (
"fmt"
"time"
)
var startTime time.Time
func uptime() time.Duration {
return time.Since(startTime)
}
func init() {
startTime = time.Now()
}
func main() {
fmt.Println("started")
time.Sleep(time.Second * 1)
fmt.Printf("uptime %s\n", uptime())
time.Sleep(time.Second * 5)
fmt.Printf("uptime %s\n", uptime())
}
You should use Since function from time package.
create time value when application start:
startTime := time.Now()
then ask whenever you want:
uptime := time.Since(startTime)
Package syscall was frozen on Go 1.4.
NOTE: This package is locked down. Code outside the standard Go repository should be migrated to use the corresponding package in the golang.org/x/sys repository. That is also where updates required by new systems or versions should be applied. See https://golang.org/s/go1.4-syscall for more information.
Use Sysinfo from golang.org/x/sys it should support this in a cross-platform way, at least on Unix.
The unix package in Go Standard Library go1.19.4 on macOS 13.1 Darwin xnu can now determine process start time using unix.SysctlKinfoProc
I have an open source Go library doing this here: https://github.com/haraldrudell/parl/blob/main/mains/process-start.go
ie.
import "github.com/haraldrudell/parl/mains"
println(mains.ProcessStartTime())
unix.SysctlKinfoProc uses macOS libSystem ie. it is supported by Apple, Inc. and uses direct kernel calls and no dumbities
Code is basically:
if unixKinfoProc, err = unix.SysctlKinfoProc(kernProcPid, os.Getpid()); perrors.Is(&err, "unix.SysctlKinfoProc: %T %+[1]v", err) {
panic(err)
}
var unixTimeval unix.Timeval = unixKinfoProc.Proc.P_starttime
sec, nsec := unixTimeval.Unix()
createTime = time.Unix(sec, nsec)
Difficulties
import "syscall" has been starved on most of its functionality which has been extracted to platform specific code in import "golang.org/x/sys/unix" and import "golang.org/x/sys/windows".
macOS GOOS==Darwin sorts under unix. The code in unix and windows is platform-specific, ie. if windows is imported on unix, the result is
error while importing golang.org/x/sys/windows: build constraints exclude all Go files in …
This means the program has to have a portable layer defining a portable function name, and that function is implemented for each supported platform like _darwin.go _linux.go and _windows.go which has to be tested on the real operating system.
The alternative is to use a third-party package where portability is already implemented. What you do then is to browse to Go Package search and pick a well-written candidate.
Solution
I browsed to Go Package search for Sysinfo: https://pkg.go.dev/search?q=sysinfo
Top result is gosysinfo "github.com/elastic/go-sysinfo". This package is awkwardly written as can be seen by a hyphen in its name and a peculiar package structure. It works, and the code goes like:
import (
gosysinfo "github.com/elastic/go-sysinfo"
"github.com/elastic/go-sysinfo/types"
"github.com/haraldrudell/parl"
)
func goSysinfo() {
var process types.Process
var err error
if process, err = gosysinfo.Self(); err != nil {
panic(parl.Errorf("go-sysinfo.Self: %w", err))
}
var processInfo types.ProcessInfo
if processInfo, err = process.Info(); err != nil {
panic(parl.Errorf("go-sysinfo.Info: %w", err))
}
startTime := processInfo.StartTime
fmt.Printf("Process start time: %s\n", startTime.Format(parl.Rfc3339s))
}
→
Process start time: 2022-03-22 10:15:05-07:00
I want to profile my benchmarks generated by go test -c, but the go tool pprof needs a profile file usually generated inside the main function like this:
func main() {
flag.Parse()
if *cpuprofile != "" {
f, err := os.Create(*cpuprofile)
if err != nil {
log.Fatal(err)
}
pprof.StartCPUProfile(f)
defer pprof.StopCPUProfile()
}
How can I create a profile file within my benchmarks ?
As described in https://pkg.go.dev/cmd/go#hdr-Testing_flags you can specify the profile file using the flag -cpuprofile.
For example
go test -cpuprofile cpu.out
Use the -cpuprofile flag to go test as documented at http://golang.org/cmd/go/#hdr-Description_of_testing_flags
This post explains how to profile benchmarks with an example: Benchmark Profiling with pprof.
The following benchmark simulates some CPU work.
package main
import (
"math/rand"
"testing"
)
func BenchmarkRand(b *testing.B) {
for n := 0; n < b.N; n++ {
rand.Int63()
}
}
To generate a CPU profile for the benchmark test, run:
go test -bench=BenchmarkRand -benchmem -cpuprofile profile.out
The -memprofile and -blockprofile flags can be used to generate memory allocation and blocking call profiles.
To analyze the profile use the Go tool:
go tool pprof profile.out
(pprof) top
Showing nodes accounting for 1.16s, 100% of 1.16s total
Showing top 10 nodes out of 22
flat flat% sum% cum cum%
0.41s 35.34% 35.34% 0.41s 35.34% sync.(*Mutex).Unlock
0.37s 31.90% 67.24% 0.37s 31.90% sync.(*Mutex).Lock
0.12s 10.34% 77.59% 1.03s 88.79% math/rand.(*lockedSource).Int63
0.08s 6.90% 84.48% 0.08s 6.90% math/rand.(*rngSource).Uint64 (inline)
0.06s 5.17% 89.66% 1.11s 95.69% math/rand.Int63
0.05s 4.31% 93.97% 0.13s 11.21% math/rand.(*rngSource).Int63
0.04s 3.45% 97.41% 1.15s 99.14% benchtest.BenchmarkRand
0.02s 1.72% 99.14% 1.05s 90.52% math/rand.(*Rand).Int63
0.01s 0.86% 100% 0.01s 0.86% runtime.futex
0 0% 100% 0.01s 0.86% runtime.allocm
The bottleneck in this case is the mutex, caused by the default source in math/rand being synchronized.
Other profile presentations and output formats are also possible, e.g. tree. Type help for more options.
Note, that any initialization code before the benchmark loop will also be profiled.
http://golang.org/ref/spec#Allocation
There is a way to allocate memory, but I don't see a way to deallocate memory (without the Go GC turned on).
If I wanted to write an OS using Go, I would need to either write a low level GC for Go or disable the Go GC. In the latter case, how could I free memory?
PS - this topic has been talked about extensively on the Go mailing list, but I wanted to pose this specific question to SO.
You can free arbitrary memory by making runtime·free accessible to your program
using cgo.
Build your own package called, for example, mem and create two files:
mem.go
package mem
import "unsafe"
import "reflect"
func FreePtr(p unsafe.Pointer)
func Free(v interface {}) {
FreePtr(unsafe.Pointer(reflect.ValueOf(v).Elem().Pointer()))
}
runtime.c
// +build gc
#include <runtime.h>
void ·Free(void* foo) {
runtime·free(foo);
}
Example usage (free a slice and print number of frees):
import "free/mem"
func main() {
var m1, m2 runtime.MemStats
runtime.ReadMemStats(&m1)
c := make([]int, 10000)
inspect.Free(&c)
runtime.ReadMemStats(&m2)
fmt.Printf("%d vs %d\n", m1.Frees, m2.Frees)
}
You should see that there was one more free than before.
How can I get the list of currently running processes in Go?
The OS package provides some functions: http://golang.org/pkg/os/
but doesn't give anything to see the list of running processes.
There is no such function in the standard library and most likely never will be.
In most cases, the list of processes isn't required by programs. Go programs usually want to wait for a single or a smaller number of processes, not for all processes. PIDs of processes are usually obtained by other means than searching the list of all processes.
If you are on Linux, the list of processes can be obtained by reading contents of /proc directory. See question Linux API to list running processes?
This library:
github.com/mitchellh/go-ps
worked for me.
import (
ps "github.com/mitchellh/go-ps"
... // other imports here...
)
func whatever(){
processList, err := ps.Processes()
if err != nil {
log.Println("ps.Processes() Failed, are you using windows?")
return
}
// map ages
for x := range processList {
var process ps.Process
process = processList[x]
log.Printf("%d\t%s\n",process.Pid(),process.Executable())
// do os.* stuff on the pid
}
}
I suggest to use for this purpose the following library:
https://github.com/shirou/gopsutil/
Here is an example to get total processes and running ones:
package main
import (
"fmt"
"github.com/shirou/gopsutil/host"
"github.com/shirou/gopsutil/load"
)
func main() {
infoStat, _ := host.Info()
fmt.Printf("Total processes: %d\n", infoStat.Procs)
miscStat, _ := load.Misc()
fmt.Printf("Running processes: %d\n", miscStat.ProcsRunning)
}
The library allows to get several other data.
Take a look at the documentation for available informations provided according to the target operative system.
If you only need the process information, can just run "ps" command from your go code, then parse the text output.
A complete solution can refer to Exercise 29 in Book "Learning Go" # http://www.miek.nl/files/go/
you can use this library github.com/shirou/gopsutil
package main
import (
"fmt"
"github.com/shirou/gopsutil/v3/process"
)
func main() {
processes, _ := process.Processes()
for _, process := range processes {
name, _ := process.Name()
fmt.Println(name)
}
}
in this library,you can also get process info other
For linux
I found a fairly simple solution to get the list of running processes without using a large library:
package main
import (
"fmt"
"os"
"path/filepath"
)
func main() {
matches, err := filepath.Glob("/proc/*/exe")
for _, file := range matches {
target, _ := os.Readlink(file)
if len(target) > 0 {
fmt.Printf("%+v\n", target)
}
}
}
It will print the path for each running process. If you need just the process name, then you can get it with filepath.Base(target)
This works by de-referencing the symlink for the /proc/[procID]/exe file, which is a symlink to the executable file. This is much simpler than reading and extracting the process name from the /proc/[procID]/status file (as suggested in other solutions I found).
PS: This might not work on all distribution because it relies on the exe file in the process' folder, which might not present in all flavors of Linux.