Trying to find out time.Sleep() call in the execution path.
To simplify the problem, consider the code below.
func BenchmarkSleep(b *testing.B) {
for i:=0; i < b.N; i++ {
time.Sleep(time.Millisecond * 10)
}
}
I tried to profile (cpu, block, mutex) the code and expected to see the time.Sleep samples in the profile.
What should I do.
Have you tried:
go test -trace trace.out
followed by:
go tool trace trace.out
... to view it in a browser?
You can then generate a pprof-like profile from the trace:
go tool trace -pprof=TYPE trace.out > TYPE.pprof
Supported profile types are:
net: network blocking profile
sync: synchronization blocking profile
syscall: syscall blocking profile
sched: scheduler latency profile
And view that too:
go tool pprof TYPE.pprof
More generally...
The trace package: https://golang.org/pkg/runtime/trace/
Can be used with the trace tool: https://golang.org/cmd/trace/
To get diagnostics and visualise the profiling data: https://golang.org/doc/diagnostics.html
Related
Steps:
go test -bench . -benchmem -memprofile memprofile.out -cpuprofile cpuprofile.out
go tool pprof -web memprofile.out
All i was able to do.
the problem was pprof -web does not include to memory profile graph
such things as runtime operations (both Group and FastGroup contain only map/slice-related stuff, no func calls).
the only workaround here is to add -lines option to see on graph not only Func, but Func:N where N is the line where exactly allocs were in Func
Issue
I have a go package, with a test suite.
When I run the test suite for this package, the total runtime is ~ 7 seconds :
$ go test ./mydbpackage/ -count 1
ok mymodule/mydbpackage 7.253s
However, when I add a -cpuprofile=cpu.out option, the sampling does not cover the whole run :
$ go test ./mydbpackage/ -count 1 -cpuprofile=cpu.out
ok mymodule/mydbpackage 7.029s
$ go tool pprof -text -cum cpu.out
File: mydbpackage.test
Type: cpu
Time: Aug 6, 2020 at 9:42am (CEST)
Duration: 5.22s, Total samples = 780ms (14.95%) # <--- depending on the runs, I get 400ms to 1s
Showing nodes accounting for 780ms, 100% of 780ms total
flat flat% sum% cum cum%
0 0% 0% 440ms 56.41% testing.tRunner
10ms 1.28% 1.28% 220ms 28.21% database/sql.withLock
10ms 1.28% 2.56% 180ms 23.08% runtime.findrunnable
0 0% 2.56% 180ms 23.08% runtime.mcall
...
Looking at the collected samples :
# sample from another run :
$ go tool pprof -traces cpu.out | grep "ms " # get the first line of each sample
10ms runtime.nanotime
10ms fmt.(*readRune).ReadRune
30ms syscall.Syscall
10ms runtime.scanobject
10ms runtime.gentraceback
...
# 98 samples collected, for a total sum of 1.12s
The issue I see is : for some reason, the sampling profiler stops collecting samples, or is blocked/slowed down at some point.
Context
go version is 1.14.6, platform is linux/amd64
$ go version
go version go1.14.6 linux/amd64
This package contains code that interact with a database, and the tests are run against a live postgresql server.
One thing I tried : t.Skip() internally calls runtime.Goexit(), so I replaced calls to t.Skip and variants with a simple return ; but it didn't change the outcome.
Question
Why aren't more samples collected ? I there some known pattern that blocks/slows down the sampler, or terminates the sampler earlier than it should ?
#Volker guided me to the answer in his comments :
-cpuprofile creates a profile in which only goroutines actively using the CPU are sampled.
In my use case : my go code spends a lot of time waiting for the answers of the postgresql server.
Generating a trace using go test -trace=trace.out, and then extracting a network blocking profile using go tool trace -pprof=net trace.out > network.out yielded much more relevant information.
For reference, on top of opening the complete trace using go tool trace trace.out, here are the values you can pass to -pprof= :
from go tool trace docs :
net: network blocking profile
sync: synchronization blocking profile
syscall: syscall blocking profile
sched: scheduler latency profile
I have the following in my main block:
func main() {
defer profile.Start().Stop()
fmt.Println("running version", version, "built on", date)
fmt.Println()
cmd.Execute()
time.Sleep(2 * time.Second)
}
where cmd is a cobra subcommand. I do a go build, and then I run the binary. I can see that it generates a pprof file:
2018/09/13 18:43:26 profile: cpu profiling enabled, /tmp/profile705487093/cpu.pprof
... output deleted ...
2018/09/13 18:43:31 profile: cpu profiling disabled, /tmp/profile705487093/cpu.pprof
Then I'm trying to analyze it, using:
go tool pprof /root/code/debug/evented /tmp/profile705487093/cpu.pprof
But when pprof opens, I see this:
File: evented
Type: cpu
Time: Sep 13, 2018 at 6:43pm (UTC)
Duration: 5.49s, Total samples = 0
In case it helps, I'm running go version go1.11 linux/amd64 on a Ubuntu 16.04.5 LTS. Not sure whether it matters, but I'm trying to inspect the pprof output on a DigitalOcean droplet.
Is there something that I'm doing wrong? Thanks!
After looking a bit through the comments of the profile pkg, I managed to get some samples, by doing this:
runtime.SetCPUProfileRate(5000)
before calling the defer profile.Start().Stop() line.
I am unable to use variables inside my go routine started in a go plugin.
The following code is an example of what I am trying to do.
The code
// Can be an non-empty struct as well
var channel = make(chan string)
log.Println(channel)
go func(ch chan string) {
log.Println(ch)
}(channel)
crahes with a coredump when executed in a go plugin on darwin.
-Yes Go plugins work on darwin: http://prntscr.com/iq8czy
I recently reported this issue to golang: Issue related to go plugin crashes
This only appears to be the case on darwin (OS X).
I'm new to golang. I was debugging my go application.
While I tried to run "info goroutines", it threw out:
Undefined info command: "goroutines".
Try "help info
What did I miss in my gdb configuration?
The article "Debugging Go Code with GDB" does mention:
(gdb) info goroutines
But only in the context of loading extension scripts for a given binary.
The tool chain uses this to extend GDB with a handful of commands to inspect internals of the runtime code (such as goroutines) and to pretty print the built-in map, slice and channel types.
If you'd like to see how this works, or want to extend it, take a look at src/pkg/runtime/runtime-gdb.py in the Go source distribution.
It depends on some special magic types (hash<T,U>) and variables (runtime.m and runtime.g) that the linker (src/cmd/ld/dwarf.c) ensures are described in the DWARF code.
If you're interested in what the debugging information looks like, run 'objdump -W 6.out' and browse through the .debug_* sections.
So make sure your debug session is run with those extensions activated.
in the gdb session run
source $GOROOT/src/runtime/runtime-gdb.py
where $GOROOT is go lives (see go env | grep ROOT)
you should use https://github.com/go-delve/delve as recommended by golang docs https://golang.org/doc/gdb