I have a couple of go programs that have a unused inport of net/http/pprof in them.
import _ "net/http/pprof"
...
//http.ListenAndServe("127.0.0.1:6060", nil)
I was wondering what the overhead of this import is in term of CPU and Mem. Aka. Should I remove then in prod (yes), but what would be the impact if I forgot?
Related: what are the exact sideeffects of this import? It registers some http handlers, but does it also inject things in go's malloc functions?
The overhead of importing the net/http/pprof package is pretty limited: it just installs some handlers for the http server. See the source code at:
http://golang.org/src/pkg/net/http/pprof/pprof.go
CPU profiling is not activated at initialization time, it is only activated for a period of time (30 seconds by default) when the /debug/pprof/profile REST service is called. So compiling with this package should not impact much the performance of the application (except that extra goroutines for the http server are needed).
Now, during the execution of /debug/pprof/profile, CPU sampling is activated, so a performance overhead is expected. I guess it can be used for production code provided the access to the corresponding port is restricted to the administrators of the application.
Related
Since importing net/http/pprof package is enough to enable the golang profiler in a program, my question is, is the profiler enabled by default for all golang programs, and importing the package just exposes the metrics endpoint? If that is the case, how could I disable the profiler to eliminate performance overhead?
Is profiling enabled by default for all go programs?
No.
is the profiler enabled by default for all golang programs, and importing the package just exposes the metrics endpoint?
No.
If that is the case, how could I disable the profiler to eliminate performance overhead?
No, that's not the case.
If you look at the source code of the pprof.go file, you can see in its handlers, that the profiling, tracing etc. are running only when you do a GET on any of its endpoints.
If you specify a time, for example: http://localhost:6060/debug/pprof/trace?seconds=10 the server will take 10 seconds to respond with the trace data. So the profiling is happening only if you call on an endpoint.
You can find some examples in the first comment block of the pprof.go file.
I'm kind of new to the pprof tool, and am wondering if its ok to keep running this in production. From the articles I have seen, it seems to be ok and standard, however I'm confused as to how this does not affect performance since it does a sampling N times every second and how come this does not lead to a degradation in performance.
Jaana Dogan does say in her article "Continuous Profiling of Go programs"
Profiling in production
pprof is safe to use in production.
We target an additional 5% overhead for CPU and heap allocation profiling.
The collection is happening for 10 seconds for every minute from a single instance. If you have multiple replicas of a Kubernetes pod, we make sure we do amortized collection.
For example, if you have 10 replicas of a pod, the overhead will be 0.5%. This makes it possible for users to keep the profiling always on.
We currently support CPU, heap, mutex and thread profiles for Go programs.
Why?
Before explaining how you can use the profiler in production, it would be helpful to explain why you would ever want to profile in production. Some very common cases are:
Debug performance problems only visible in production.
Understand the CPU usage to reduce billing.
Understand where the contention cumulates and optimize.
Understand the impact of new releases, e.g. seeing the difference between canary and production.
Enrich your distributed traces by correlating them with profiling samples to understand the root cause of latency.
So if you are using pprof for the right reason, yes, you can leave it in production.
But for basic monitoring, as commented, the system is enough.
As noted in "Continuous Profiling and Go" by Vladimir Varankin
Depending on the state of the infrastructure in the company, an “unexpected” HTTP server inside the application’s process can raise questions from your systems operations department ;)
At the same time, depending on the peculiar nature of a company, the very ability to access something inside a production application, that doesn’t directly relate to application’s business logic, can raise questions from the security department ;)) I
So the overhead is not the only criteria to consider when leaving active such a feature.
I am writing a game in Go. I was told my code was leaking memory. What would be the right approach to pinpoint the cause of the leaks?
I know pprof exists but as far as I know it can be used when your app uses the http package and my app is a simple game and it does not make use of any netowrk connectivity.
You can certainly create profiles without an http server using the runtime/pprof package. There's also github.com/pkg/profile by Dave Cheney, a wrapper for runtime/pprof with a simpler API. You can use signal handlers to enable and disable profiling, for instance.
That being said, nothing stops you from starting an HTTP server, just for pprof. I find that much more convenient because then you can easily create any profile on demand.
// +build debug
package main
import (
"log"
"net/http"
_ "net/http/pprof/"
)
func init() {
log.Println("Launching pprof server")
go func() { log.Fatal(http.ListenAndServe("localhost:3030", nil)) }()
}
This uses a build tag so you can enable the server in development but exclude it from production builds. Compile with -tags debug to enable the server. See Build Constraints for details.
You can use https://golang.org/pkg/runtime/pprof/ package to write memory profiles on disk (you can do it let's say once in a minute). After that just run go tool pprof (inuse_space mode) on few of them and look if there are some parts where inuse_space is growing
Apart from the pprof discussed before, it might also be useful to check heap allocation by doing an escape analysis of your code (e.g. if file is main.go) by:
go build -gcflags=-m main.go
This immediately gives you an idea of things like if you are passing pointers around unnecessarily which can be avoided.
Here is a short article which goes through various ways you can make your program more efficient.
I want to profile a simple webserver that I wrote in Go. It accepts requests, maps the request to avro object and sends it to Kafka in a go routine. The requirement is that it answers immediately and sends an object to the Kafka later. Locally it answers in under 1 ms on average. I have been trying to profile it by starting the script with davecheney/profile package and sending test requests with jmeter. I can see in the output that the profile file is generated but it remains empty, no matter how long jemeter is sending the requests. I'm running it on Mac El Capitan. I read that there were some issues with profiling on Mac but it would be working on El Capitan. Do you have any tips?
Firstly, I'm not sure whether you're trying to do latency profiling. If so, be aware that Go's CPU profiler only reports time spent by a function executing on the CPU and doesn't include time spent sleeping, etc. If CPU Profiling really is what you're looking for, read on.
If you're running a webserver, just include the following in your imports (in the file where main() is) and rebuild:
import _ "net/http/pprof"
Then, while applying load through jmeter, run:
go tool pprof /path/to/webserver/binary http://<server>/debug/pprof/profile
The net/http/pprof package provides profiling hooks that allow you to profile your webserver on demand any time, even while it's running in production. You may want to use a different, firewalled port for it, though, if your webserver is meant to be exposed publicly.
A Twisted app I have was constantly getting killed due to memory problems. The program grew in size, consuming all of the system's memory before being shut down by the os. Restart and repeat.
This is on a virtual server, so I doubled the memory, and the issue resolved - the daemon stabilized at around 1.25GB of memory
Does anyone have advice on how I can best profile this to tell what/where all the memory is getting sucked up into ?
If info on the app helps, I'm using the twisted reactor and internet.timer.TimerService to poll a database for items to update through three 'services'. the items to process are pushed into a twisted.internet.defer.DeferredList , and their processing occurs in a deferToThread block. In the deferred process there are a handful of blocking operations ( fetching web pages, etc ) and a lot of HTML parsing ( beautiful soup and other libraries ). I've suggested the reactor.threadpool size to be 10 and each 'service' defers to thread using a SemaphoreService that has 10 tokens. I really expected this daemon to max out at around 400MB of memory, not 3x that.
This is more of a generic share of thoughts how I debug memory leak/usage problems in my twisted applications.
Twisted has a ssh server support, and is something which I add in to almost all of my projects in development.
The ssh provides a interactive python interpreter access to the method which has python garbage collector available and a number of helper functions which allow me to a) inspect count of the instances from a same class, b) start and stop inspection of changes of that count over time and c) to get all references of that class. The nice thing with the interactive interpreter is that it allows ad-hoc introspection of offending instances, their relation to other objects and the state of process they are in. This so far has always proven a valuable instrument to pinpoint exact location where I have forgot / unforseen the ref release problems in my projects.