Understanding of Prometheus - go

Question

I would like to try simple example of using Prometheus.
I have downloaded server binaries
I have started simple code, but with few modifications
var addr = flag.String("listen-address", ":8080", "The address to listen on for HTTP requests.")
func main() {
flag.Parse()
http.Handle("/metrics", promhttp.Handler())
http.Handle("/test/{id}", myHandler(promhttp.Handler()))
log.Fatal(http.ListenAndServe(*addr, nil))
}
func myHandler(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
fmt.Fprintf(w, "hello, you've hit %s\n", r.URL.Path)
next.ServeHTTP(w, r)
})
}
Questions:
I assume Prometheus is monitoring tool and I would like to monitor endpoints metrics and /test/{id} separately. Did I get the idea correctly by creating several handlers and using promhttp.Handler() as middleware?
What else apart of quantity and latency of requests can be monitored in e.g. simple web app with database?

Answer 1

To follow up on #David Maze's answer, the default handler promhttp.Handler is for reporting metrics. (gathers from all registered handlers and reports them on request).
Unfortunately, it is not a generic http middleware that gives you any metrics out of the box.
I have seen many of go's web frameworks have some sort of community prometheus middleware (gin's) that give metrics out of the box (latency, response code, request counts, etc).
The go prometheus client library has examples of how to add metrics to your application.
var (
httpRequests = prometheus.NewCounter(
prometheus.CounterOpts{
Name: "http_requests_total",
Help: "Number of http requests.",
},
)
)
func init() {
// Metrics have to be registered to be exposed:
prometheus.MustRegister(httpRequests)
}
func myHandler(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
httpRequests.Inc()
fmt.Fprintf(w, "hello, you've hit %s\n", r.URL.Path)
next.ServeHTTP(w, r)
})
}
As for your second question lots :) Google advocates for monitoring 4 golden signals:
https://landing.google.com/sre/book/chapters/monitoring-distributed-systems.html#xref_monitoring_golden-signals
These are
Traffic - Throughput - Counts/Time
Latency - distribution / histogram
Errors - HTTP Response codes/explicit error counts
Saturation - resource queues ie if there is a goroutine pool how many goroutines are active at a given time
In my experie8inces it has also been helpful to have visibility of all the interactions between your appl8ication and your database (ie the 4 golden signals applied to your database):
number of calls made to db from app
latencies of calls made
results (err/success) of your calls made to determine availability (success / total)
saturation available from your db driver (https://golang.org/pkg/database/sql/#DBStats)