How to re-evaluate promhttp.Handler on database change? - go

Question

I'm perhaps abusing promhttp.Handler() to realise the use case for my microservice to tell me the:
version
if it has database connectivity
If there is a better way to monitor my microservices, do let me know!
I'm not sure how to structure the handle in such a way that when /metrics are called, the db.Ping() is re-evaluated.
https://s.natalian.org/2019-06-02/msping.mp4
package main
import (
"log"
"net/http"
"os"
_ "github.com/go-sql-driver/mysql"
"github.com/gorilla/mux"
"github.com/jmoiron/sqlx"
"github.com/prometheus/client_golang/prometheus"
"github.com/prometheus/client_golang/prometheus/promhttp"
)
const version = "0.0.1"
type App struct {
Router *mux.Router
DB *sqlx.DB
}
func main() {
a := App{}
a.Initialize()
log.Fatal(http.ListenAndServe(":"+os.Getenv("PORT"), a.Router))
}
func (a *App) Initialize() {
connectionString := "root:secret#tcp(localhost:3306)/rest_api_example?multiStatements=true&sql_mode=TRADITIONAL&timeout=5s"
var err error
a.DB, err = sqlx.Open("mysql", connectionString)
if err != nil {
log.Fatal(err)
}
microservicecheck := prometheus.NewGaugeVec(
prometheus.GaugeOpts{
Name: "mscheck",
Help: "Version with DB ping check",
},
[]string{
"commit",
},
)
if a.DB.Ping() == nil {
microservicecheck.WithLabelValues(version).Set(1)
} else {
microservicecheck.WithLabelValues(version).Set(0)
}
prometheus.MustRegister(microservicecheck)
a.Router = mux.NewRouter()
a.initializeRoutes()
}
func (a *App) initializeRoutes() {
a.Router.Handle("/metrics", promhttp.Handler()).Methods("GET")
}
https://play.golang.org/p/9DdXnz77S55

Answer 1

You could also add a middleware hook that does a preflight routine (i.e. your ping test) before calling promhttp.Handler(). However, at collection time, I think metrics should already have been tallied; and not generated at the instance of collection. So...
Try a separate go-routine that will poll the health of the DB connection at regular intervals. This avoids any messy hooks or custom collectors:
var pingPollingFreq = 5 * time.Second // this should probably match the Prometheus scrape interval
func (a *App) Initialize() {
// ...
prometheus.MustRegister(microservicecheck)
go func() {
for {
if a.DB.Ping() == nil {
microservicecheck.WithLabelValues(version).Set(1)
} else {
microservicecheck.WithLabelValues(version).Set(0)
}
time.Sleep(pingPollingFreq)
}
}()
// ...
}