I'd like to profile my Go HTTP server application to see if there are places where it can be optimized. I'm using the fasthttp package with the fasthttp/router package, and I'm struggling to figure out how to hook up pprof.
The basic setup looks like this, obviously very abridged:
func main() {
flag.Parse()
r := router.New()
r.GET("/", index)
r.GET("/myroute", myrouteFn)
h := r.Handler
if err := limitedListenAndServe(*addr, fasthttplogger.Tiny(h)); err != nil {
log.Fatalf("Error in ListenAndServe: %s", err)
}
}
First, I tried following a fairly straightforward guide like this one, and added this line in my main() function (per the guide) in addition to the corresponding import. That changed the above to this:
func main() {
flag.Parse()
r := router.New()
r.GET("/", index)
r.GET("/myroute", myrouteFn)
h := r.Handler
if err := limitedListenAndServe(*addr, fasthttplogger.Tiny(h)); err != nil {
log.Fatalf("Error in ListenAndServe: %s", err)
}
defer profile.Start().Stop()
}
After doing that, I ran my program, made a bunch of requests that I was interested in profiling, and then terminated the server. That created a cpu.pprof file, but the file was empty (zero bytes) when I went to run it through the go tool pprof command to generate the graph.
After a bit more sleuthing I found this Gist that I suspect would work if I were using totally vanilla fasthttp.
Trying to combine that with my application, I'm still stuck. Conceptually, I think the solution is to use the fasthttpadaptor to convert the net/http/pprof to a fasthttp route. But the Gist that I was looking at is using its own router to do the mux, I'd and rather not rewrite all the routes using a different router in my server.
How would I go about making the profiling data available here?
You can use net/http/pprof for profiling.
fasthttp provides custom implementation for the same. You can use that same as the net/http/pprof.
To use this, register the handler as:
import "github.com/valyala/fasthttp/pprofhandler"
...
r.GET("/debug/pprof/{profile:*}", pprofhandler.PprofHandler)
Then you can use this to profile.
go tool pprof http://host:port/debug/pprof/profile
Or you can also visit http://host:port/debug/pprof/ to see more types of profiles available.
Related
I want to wrap Read function net.Conn.Read(). The purpose of this to read the SSL handshake messages. https://pkg.go.dev/net#TCPConn.Read
nc, err := net.Dial("tcp", "google.com:443")
if err != nil {
fmt.Println(err)
}
tls.Client(nc, &tls.Config{})
Are there any ways to do?
Thanks in advance
Use the following code to intercept Read on a net.Conn:
type wrap {
// Conn is the wrapped net.Conn.
// Because it's an embedded field, the
// net.Conn methods are automatically promoted
// to wrap.
net.Conn
}
// Read calls through to the wrapped read and
// prints the bytes that flow through. Replace
// the print statement with whatever is appropriate
// for your application.
func (w wrap) Read(p []byte) (int, error) {
n, err := w.Conn.Read()
fmt.Printf("%x\n", p[:n]) // example
return n, err
}
Wrap like this:
tnc, err :=tls.Client(wrap{nc}, &tls.Config{})
The previous answer gets the job done indeed.
However I would recommend Liz Rice's talk: GopherCon 2018: Liz Rice - The Go Programmer's Guide to Secure Connections
Going through her code in Github, you might find a more elegant way to achieve what you want.
Start with the client code on line 26.
Am I correct to assume that with the Go language, these two formulations are always equivalent ?
func f() {
// Do stuff
}
go f()
and
func f() {
go func(){
// do stuff
}()
)
The question was basically answered in the comments, but although in the simple case both examples do the same thing, one may be preferred over the other depending on what the actual goal is.
One that the comments mention is allowing the user of your code to decide on concurrency vs you (the writer) deciding. I think this rule of thumb is generally preferred especially for people writing packages for others to use (even if perhaps the others are in your own team). I've also seen this rule of thumb espoused elsewhere on "the internet", and I think arose because in the early days of Go, people were using (and abusing) concurrency features just because they were available. For example, returning a channel from which you'd receive a value instead of just returning the value.
Another difference is that in the top example, f() may not be able to close on variables that you might want accessible when run as a goroutine--you'd have to pass everything into f() as a parameter. In the second example the anonymous function in go func() {...} could close over something in f().
One example where I prefer the second style is starting servers. For example:
func (app *Application) start() {
if app.HttpsServer != nil {
go func() {
err := app.HttpsServer.ListenAndServeTLS(
app.Config.TLSCertificateFile,
app.Config.TLSKeyFile)
if err != nil && err != http.ErrServerClosed {
// unexpected error
log.Printf(log.Critical, "error with https server: %s", err)
}
}()
}
go func() {
err := app.HttpServer.ListenAndServe()
if err != nil && err != http.ErrServerClosed {
// unexpected error
log.Printf(log.Critical, "error with http server: %s", err)
}
}()
}
Here the intention is that Application is configured and controlled in main(), the servers (one on https, one on http) are started and program flow returns to main(). In my specific case, main() waits for a signal from the OS then shuts down the servers and exits. Both goroutines close over app and have access to the data it contains. Is this "good" or "bad"...who knows, but it works well for me.
So essentially... "It depends".
I am trying to figure out how to get a simple bq load command to work with https://godoc.org/cloud.google.com/go/bigquery#Table.LoaderFrom
Running it manually it looks like this:
bq load --source_format=AVRO --ignore_unknown_values --replace=true mydataset.mytable gs://mybucket/table/*
And running it in my golang with exec.Command() successfully looks like this:
exec.Command("bq", "load", "--source_format=AVRO", "--ignore_unknown_values",
"--replace=true", "mydataset.mytable",
"gs://mybucket/table/*")
However, I cannot get this program to run without a segmentation fault when trying to get the load and job.wait to run successfully it seems to be getting a segmentation violation at the job.wait line of the program
package main
import (
"context"
"log"
"cloud.google.com/go/bigquery"
)
func main(){
ctx := context.Background()
client, err := bigquery.NewClient(ctx, "my-project-id")
if err != nil {
// TODO: Handle error.
}
gcsRef := bigquery.NewGCSReference("gs://mybucket/table/*")
gcsRef.SourceFormat = "AVRO"
gcsRef.IgnoreUnknownValues = true
// TODO: set other options on the GCSReference.
ds := client.Dataset("mydataset")
loader := ds.Table("mytable").LoaderFrom(gcsRef)
// TODO: set other options on the Loader.
job, err := loader.Run(ctx)
if err != nil {
// TODO: Handle error.
}
status, err := job.Wait(ctx) //seg faults right here
if err != nil {
// TODO: Handle error.
}
if status.Err() != nil {
// TODO: Handle error.
}
}
The panic is probably coming from a nil pointer reference to the job variable.
I would suggest including a log.Fatal(err)
In all of your err!= nil blocks.
This will help get you closer to why job is not being assigned correctly.
When you're writing one off scripts like this one in go log.Fatal is a great way to exit the program and print exactly what the issue is.
With go you're always trying to bubble errors up the stack to determine if the code should continue to execute, if things can be recovered, or if it's just a fatal thing and you should end the program.
For more info on the logging package checkout here: https://golang.org/pkg/log/
If you're just starting out learning go here are some awesome resources that can help give you ideas on how different types of programs can be designed.
https://github.com/dashpradeep99/https-github.com-miguellgt-books/tree/master/go
Best,
Christopher
I would love to be able to use Go-IPFS within my Go program, however it is totally undocumented. This is where my reasearch lead me:
package main
import (
"context"
"fmt"
"io/ioutil"
"log"
"os"
"path/filepath"
"gx/ipfs/QmSP88ryZkHSRn1fnngAaV2Vcn63WUJzAavnRM9CVdU1Ky/go-ipfs-cmdkit/files"
"github.com/ipfs/go-ipfs/core"
"github.com/ipfs/go-ipfs/core/coreunix"
)
func main() {
ctx := context.Background()
node, err := core.NewNode(ctx, &core.BuildCfg{})
if err != nil {
log.Fatalf("Failed to start IPFS node: %v", err)
}
reader, err := coreunix.Cat(ctx, node, "QmPZ9gcCEpqKTo6aq61g2nXGUhM4iCL3ewB6LDXZCtioEB")
if err != nil {
log.Fatalf("Failed to look up IPFS welcome page: %v", err)
}
blob, err := ioutil.ReadAll(reader)
if err != nil {
log.Fatalf("Failed to retrieve IPFS welcome page: %v", err)
}
fmt.Println(string(blob))
}
However I am not sure about the difference of
context.Background() Vs context.TODO() Vs context.WithCancel(context.Background()).
And most importantly, how can choose where IPFS will put the IPFS repo and make sure it also initializes it?
How can I enable and use Pubsub along with its commands subscribe and publish?
How can I add and pin a file to IPFS with the possibility to also input a stream for big files?
Is coreunix.Cat suitable to read files with a stream as well?
How can I keep the node "listening" like when you run the ipfs daemon from the CLI and have everything runnng on all the ports like webui, swarm, etc.?
How about this to add files? Does this use streams or reads the entire file into memory? How can this be improved?
func addFile(ctx *context.Context, node *core.IpfsNode, path *string) error {
file, err := os.Open(*path)
if err != nil {
return err
}
adder, err := coreunix.NewAdder(*ctx, node.Pinning, node.Blockstore, node.DAG)
if err != nil {
return err
}
filename := filepath.Base(*path)
fileReader := files.NewReaderFile(filename, filename, file, nil)
adder.AddFile(fileReader)
adder.PinRoot()
return nil
}
You may want to breakdown your question into smaller pieces, I have been playing with a source code of go-ipfs for a while and here's the general instruction I could give you:
Most of the data structures like, context, DAG, IPFSNode and so on are defined in form of go structures, and you should be able to find them in gx/.../... directory where also you should be able to see detailed information about each variable used ( simple string search through the directory should get you to your needed source file)
All the methods are defined in folder github.com/../.. directory
Clear the concept of pointers as they are using pointers most of the time to pass parameters to functions
I need to know how to detect when a file changes using Go. I know that Unix provides a function named fcntl() which notifies when a specific file is changed but I have not found this one in Go.
Please help me.
Here's a simple cross-platform version:
func watchFile(filePath string) error {
initialStat, err := os.Stat(filePath)
if err != nil {
return err
}
for {
stat, err := os.Stat(filePath)
if err != nil {
return err
}
if stat.Size() != initialStat.Size() || stat.ModTime() != initialStat.ModTime() {
break
}
time.Sleep(1 * time.Second)
}
return nil
}
And usage would be like this:
doneChan := make(chan bool)
go func(doneChan chan bool) {
defer func() {
doneChan <- true
}()
err := watchFile("/path/to/file")
if err != nil {
fmt.Println(err)
}
fmt.Println("File has been changed")
}(doneChan)
<-doneChan
Not as efficient as a proper system call but it's simple and works everywhere, and might be enough for some uses.
I would add to the peterSO's answer that if you in fact want to read the data appended to a file by some other process(es) — what tail program does in Unix, — it's probably better to just make tail itself do the hard job of monitoring the file and consume what it outputs. This can be achieved by running tail using the StdoutPipe function from the exec package.
Using tail for this kind of task is preferable in my eyes because tail has been taught to use a bunch of clever tricks including detection of file replacements (commonly occuring when one monitors a log file which is being rotated by logrotate or something similar).
There is currently an experimental package here. It should be merged into core as os/fsnotify in go1.3
Have a look at https://github.com/howeyc/fsnotify. It wraps the inotify subsystem of the Linux kernel and should work in Go1.
As of Go1 inotify has been removed from the package. Have a look at the syscall package now...
Package inotify implements a wrapper for the Linux inotify
system.