I am trying to build a REST API with go-chi and Gorm.
I am not sure how I should pass the Gorm DB instance to the route handlers.
Or if I should create one instance per handler, which does not sound right to me.
Should I use middleware, dependency injection or other? What would be recommended pattern here?
package main
import (
"encoding/json"
"fmt"
"github.com/go-chi/chi/v5"
"log"
"net/http"
"os"
"time"
)
func main() {
r := chi.NewRouter()
r.Get("/", indexHandler)
port := os.Getenv("PORT")
if port == "" {
port = "8080"
log.Printf("Defaulting to port %s", port)
}
db := Connect()
migrations(db)
logStartServer(port)
log.Fatal(http.ListenAndServe(fmt.Sprintf(":%s", port), r))
}
func logStartServer(port string) {
log.Printf("Listening on port %s", port)
log.Printf("Open http://localhost:%s in the browser", port)
}
func indexHandler(w http.ResponseWriter, r *http.Request) {
//How can I access db here?
//result := db.Find(&users)
policy := InsurancePolicy{ValidFrom: time.Now()}
err := json.NewEncoder(w).Encode(policy)
if err != nil {
w.WriteHeader(http.StatusInternalServerError)
}
}
Use methods instead of functions. This allows you to pass any information needed by the handlers using the receiver of those methods:
type MyHandler struct {
DB *gorm.DB
}
func (m MyHandler) IndexHandler(w http.ResponseWriter, r *http.Request) {
// Use m.DB here
}
In main:
handler:=mypkg.MyHandler{DB:gormDB}
r.Get("/", handler.IndexHandler)
In some cases, a closure makes more sense.
func GetIndexHandler(db *gorm.DB) func(http.ResponseWriter,*http.Request) {
return func(w http.ResponseWriter,req *http.Request) {
// Implement index handler here, using db
}
}
func main() {
...
r.Get("/", GetIndexHandler(db))
Declaring the DB instance as a global variable is quite convenient if your project is small.
A number of ways for organising DB access are documented here quite well. Pick the one which fits your needs.
In the DB/query function itself. I personally make a separate package for controllers and a separate package for services. I handle all the request validation and HTTP stuff in the controller (which has my handler functions). Then, if everything checks out, I call a service package. The service package is the one that calls the DB as well as any other services or API integrations.
Yet, where ever you call the DB, usually you are calling into a db package that has a bunch of query functions with friendly names like db.GetAccountByID or something like that. Well, that db function is exactly where you pass the *sql.DB or *gorm.DB object.
An example would be...
package db
func GetAccountByID(id int, db *gorm.DB) (*model.Account, error) {
if db == nil {
db = conn // conn is the package level db connection object
}
//...
}
Generally, when the server starts, I create the DB connection (which functions as a connection pool) and so it's not really necessary to pass it into the function. So, why do it? Well, it's because of testing. You don't want your DB handler reaching out to a package level DB connection object because it becomes more difficult to do isolated testing of that function.
So, this function signature gives you that testability and the initial if condition still uses that single central DB connection object if nil is passed in for the DB value, which is always is nil unless you are testing.
This is just one approach but one I've used successfully for years now.
Related
I am playing around with Dependency Injection and HTTP servers in Golang, and are trying to wrap my head around how to make available a logger that contains information related to the current request that is being handled. What I want to be able to do is to call a logging function inside the current http.Handler and functions/methods that is being called inside from the http.Handler. Essentially leave a request ID in every log event, so I can trace exactly what happened.
My idea was to use Dependency Injection and define all the logic as methods on interfaces that would have the logger injected. I like this approach, since it makes the dependencies painstakingly clear.
I've seen other approaches, such as Zerolog's hlog package that puts the logger into the requests context.Context, and I could then modify my code to receive a context.Context as an argument, from where I could pull out the logger and call it. While this will work, I don't like it very much, in that it kinda hides away the dependency.
My issue is that I can't seem to come up with a way of using Dependency Injection in conjunction with a HTTP handling server, where the injected dependencies are concurrency safe.
As an example, I whipped up this code (ignore the fact that it doesn't use interfaces)
package main
import (
"crypto/rand"
"fmt"
mrand "math/rand"
"net/http"
"time"
)
func main() {
l := LoggingService{}
ss := SomethingService{
LoggingService: &l,
}
handler := Handler{
SomethingService: &ss,
LoggingService: &l,
}
http.Handle("/", &handler)
http.ListenAndServe("localhost:3333", nil)
}
type Handler struct {
SomethingService *SomethingService
LoggingService *LoggingService
}
func (h *Handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
reqID := generateRequestID()
h.LoggingService.AddRequestID(reqID)
h.LoggingService.Log("A request")
out := h.SomethingService.DoSomething("nothing")
w.Write([]byte(out))
}
func generateRequestID() string {
buf := make([]byte, 10)
rand.Read(buf)
return fmt.Sprintf("%x", buf)
}
type SomethingService struct {
LoggingService *LoggingService
}
func (s *SomethingService) DoSomething(input string) string {
s.LoggingService.Log(fmt.Sprintf("Let's do something with %s", input))
mrand.Seed(time.Now().UnixNano())
n := mrand.Intn(3)
s.LoggingService.Log(fmt.Sprintf("Sleeping %d seconds...", n))
time.Sleep(time.Duration(n) * time.Second)
s.LoggingService.Log("Done")
return "something"
}
type LoggingService struct {
RequestID string
}
func (l *LoggingService) AddRequestID(id string) {
l.RequestID = id
}
func (l *LoggingService) Log(msg string) {
if l.RequestID != "" {
fmt.Printf("[%s] %s\n", l.RequestID, msg)
} else {
fmt.Printf("%s\n", msg)
}
}
The way that I've understood Go's HTTP server is that the handler is invoked, for each request, in it's own goroutine. Because the LoggingService is passed as a pointer, each of these goroutines are essentially accessing the same instance of LoggingService, which can result in a race condition where one request sets l.RequestID after which another request logs its requests, but ends up with the request ID from the first request.
I am sure others have run into a need to log something in a function that is called by a HTTP request and to be able to trace that log event to that specific request? However, I haven't found any examples where people are doing this.
I was thinking of delaying the instantiation of the dependencies until inside the handler. While that works, it has the side-effect of creating many instances of the dependencies - This is fine for the LoggingService, but something like the SomethingService could be talking to a DB, where it would be a waste to create a database client for every request (since most DB implementations have concurrency safe clients).
Most Go/GORM examples I've seen show Automigrate being called immediately after opening the database connection, including GORM documentation here. For API services, this would be an expensive/wanted call with every API requests. So, I assume, for API services, Automigrate should be removed from regular flow and handled separately. Is my understanding correct?
From GORM Documentation
...
db, err := gorm.Open(sqlite.Open("test.db"), &gorm.Config{})
if err != nil {
panic("failed to connect database")
}
// Migrate the schema
db.AutoMigrate(&Product{})
...
It wouldn't happen on every API request. Not even close. It'd happen every time the application is started, so basically: connect to the DB in main, and run AutoMigrate there. Pass the connection as a dependency to your handlers/service packages/wherever you need them. The HTTP handler can just access it there.
Basically this:
package main
func main() {
db, err := gorm.Open(sqlite.Open("test.db"), &gorm.Config{})
if err != nil {
fmt.Printf("Failed to connect to DB: %v", err)
os.Exit(1)
}
// see below how this is handled
fRepo := foo.New(db) // all repos here
fRepo.Migrate() // this handles migrations
// create request handlers
fHandler := handlers.NewFoo(fRepo) // migrations have already been handled
mux := http.NewServeMux()
mux.HandleFunc("/foo/list", fHandler.List) // set up handlers
// start server etc...
}
Have the code that interacts with the DB in some package like this:
package foo
// The DB connection interface as you use it
type Connection interface {
Create()
Find()
AutoMigrate(any)
}
type Foo struct {
db Connection
}
func New(db Connection) *Foo {
return &Foo{
db: db,
}
}
func (f *Foo) Migrate() {
f.db.AutoMigrate(&Stuff{}) // all types this repo deals with go here
}
func (f *Foo) GetAll() ([]Stuff, error) {
ret := []Stuff{}
res := f.db.Find(&ret)
return ret, res.Error
}
Then have your handlers structured in a sensible way, and provide them with the repository (aka foo package stuff):
package handlers
type FooRepo interface {
GetAll() ([]Stuff, error)
}
type FooHandler struct {
repo FooRepo
}
func NewFoo(repo FooRepo) *FooHandler {
return &FooHandler{
repo: repo,
}
}
func (f *FooHandler) List(res http.ResponseWriter, req *http.Request) {
all, err := f.repo.GetAll()
if err != nil {
res.WriteHeader(http.StatusInternalServerError)
io.WriteString(w, err.Error())
return
}
// write response as needed
}
Whenever you deploy an updated version of your application, the main function will call AutoMigrate, and the application will handle requests without constantly re-connecting to the DB or attempting to handle migrations time and time again.
I don't know why you'd think that your application would have to run through the setup for each request, especially given that your main function (or some function you call from main) explicitly creates an HTTP server, and listens on a specific port for requests. The DB connection and subsequent migrations should be handled before you start listening for requests. It's not part of handling requests, ever...
I am trying to prototype this little golang app and hoping to get some advice on how to go about managing my database and redis connection objects.
I want to create a a 'service layer' that will have all the product related logic, so maybe a ProductService.
I want ProductService to have a reference to both redis and my database client.
What would this ProductService look like roughly, and if I need to create a single instance of this and use it throughout the app do I define it in a var?
func main() {
db, err := gorm.Open("postgres", "host=localhost user=blankman dbname=blank_development sslmode=disable password=")
if err != nil {
log.Fatalf("Got error when connect database, the error is '%v'", err)
}
defer db.Close()
redis := redis.NewClient(&redis.Options{
Addr: "localhost:6379",
Password: "",
DB: 0,
})
pong, err := redis.Ping().Result()
fmt.Println(pong, err)
router := mux.NewRouter()
router.HandleFunc("/products", GetProducts).Methods("GET")
log.Fatal(http.ListenAndServe(":3001", router))
}
My GetProducts handler has your regular signature:
func GetProducts(w http.ResponseWriter, r *http.Request)
How am I suppose to pass in the ProductsService into this handler? It looks like the request/response are somehow automatically passed to this handler by MUX, so unsure how it can get a reference to the ProductService?
Create the product service with the fields you need:
type ProductService struct {
db *gorm.DB
redis *redis.Client
}
Make GetProducts a method of ProductService:
func (s *ProductService) GetProducts(w http.ResponseWriter, r *http.Request) {
// s.db is the database, s.redis is the redis client
}
Initialize ProductService in main. Use method values as handler functions:
s := &ProductService{db: db, redis: redis}
router.HandleFunc("/products", s.GetProducts).Methods("GET")
An alternative to the method value is to use a closure to adapt a function to a handler function:
func (s *ProductService) Handler(fn func(*ProductService, http.ResponseWriter, *http.Request)) func(http.ResponseWriter, *http.Request) {
return func(w http.ResponseWriter, r *http.Request) {
fn(s, w, r)
}
}
Use
router.HandleFunc("/products", s.Handler(PostPoduct)).Methods("POST")
to register a function like this:
func PostProduct(s *ProductService, w http.ResponseWriter, r *http.Request) {
}
With this second approach, the ProductService layer can be kept separate from the individual handlers.
Another option is to avoid a struct for the service and build your handlers using a closure (with a function that closes over your dependencies) like this:
func MakeGetProductsHander(db *gorm.DB, client *redis.Client) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
// Use db and client in here...
}
}
Then apply the handler like this:
router.HandleFunc("/products", MakeGetProductsHander(db, client)).Methods("GET")
This injects your dependencies into your handlers in a more explicit and obvious way.
Side note: rename your Redis client variable to client or something else since you are shadowing the package name by calling it redis.
I am having an issue with creating a middleware that will be chained to other routes and requires access to the database and am not sure how to approach this problem.
I store all of my app context, including the database in a struct called AppContext. I want to create a function handler that looks something like this:
func SomeHandler(appC *AppContext, next http.Handler) http.Handler {
fn := func(w http.ResponseWriter, r *http.Request) {
// Access the database using appC.db
// Logic that requires access to the database.
next.ServeHTTP(w, r)
}
return http.HandlerFunc(fn)
}
}
In main.go, I have tried:
someHandler := middleware.SomeHandler(&appC)
However, I get the error not enough arguments in call to middleware.SomeHandler. What would be the best way to approach this problem?
The error you're getting is due to not providing the second argument, next http.Handler.
In the case of how to go about the middleware, I recommend taking a look at the implementation of the http.ServeMux https://golang.org/src/net/http/server.go?s=57308:57433#L1890 it essentially does what you're trying to do (and then some) for routing though. So using a http.Handler struct might be more easy here than using a Handler function, this way the Handler(s) you'd have as the next http.Handler argument in your function are just a field within the struct that the parent handler can call from within its ServeHTTP().
So to wrap up my point, you might want to use a struct that implements the http.Handler interface. That way it can have child handlers and the DB access. That way you don't have to keep passing this AppContext either.
I'd not do a premature setup of the context. It is specifically there to be request scoped.
I'd rather create a small and dedicated middleware for passing a database session out of the pool into the context and retrieve said session created for the request in the main handler.
func DBSession(sess *mgo.Session, next http.Handler) http.Handler {
return http.HandlerFunc(
func(w http.ResponseWriter, r *http.Request) {
// Create a new session for the scope of the request
s := sess.Copy()
// Make it close when exiting scope
defer s.Close()
// Add the new session (actually a pointer to it)
// acessible via the name "_sess"
ctx = context.WithValue(r.Context(), "_sess", s)
// execute the next handler
next(w, r.WithContext(ctx))
})
}
Now you can use this middleware in you main.go
package main
import (
"net/http"
"gopkg.in/mgo.v2"
)
func sayHello(w http.ResponseWriter, r *http.Request) http.Handler{
return http.HandlerFunc(
func (w http.ResponseWriter, r *http.Request) {
s := r.Context().Value("_sess").(*mgo.Session)
// Do something with your database here.
}
)
}
func main() {
// Ignoring err for brevity
sess, _ := mgo.Dial("localhost:27017")
// Magic happens here: the sayHello Handler is wrapped in the
// DBSession middleware, which makes the session available
// as context param _sess and closes the session after sayHello
// executed. No clutter in your controller for setting up and tearing down
// the session, no way you can forget to close it.
http.Handle("/", middleware.DBSession(sess, sayHello))
}
I'm using Gin, https://gin-gonic.github.io/gin/, to build a simple RESTful JSON API with Golang.
The routes are setup with something like this:
func testRouteHandler(c *gin.Context) {
// do smth
}
func main() {
router := gin.Default()
router.GET("/test", testRouteHandler)
router.Run(":8080")
}
My question is how can I pass down an argument to the testRouteHandler function? For example a common database connection could be something that one wants to reuse among routes.
Is the best way to have this in a global variable? Or is there some way in Go to pass along an extra variable to the testRouteHandler function? Are there optional arguments for functions in Go?
PS. I'm just getting started in learning Go, so could be something obvious that I'm missing :)
I would avoid stuffing 'application scoped' dependencies (e.g. a DB connection pool) into a request context. Your two 'easiest' options are:
Make it a global. This is OK for smaller projects, and *sql.DB is thread-safe.
Pass it explicitly in a closure so that the return type satisfies gin.HandlerFunc
e.g.
// SomeHandler returns a `func(*gin.Context)` to satisfy Gin's router methods
// db could turn into an 'Env' struct that encapsulates all of your
// app dependencies - e.g. DB, logger, env vars, etc.
func SomeHandler(db *sql.DB) gin.HandlerFunc {
fn := func(c *gin.Context) {
// Your handler code goes in here - e.g.
rows, err := db.Query(...)
c.String(200, results)
}
return gin.HandlerFunc(fn)
}
func main() {
db, err := sql.Open(...)
// handle the error
router := gin.Default()
router.GET("/test", SomeHandler(db))
router.Run(":8080")
}
Using the link i posted on comments, I have created a simple example.
package main
import (
"log"
"github.com/gin-gonic/gin"
"github.com/jinzhu/gorm"
_ "github.com/mattn/go-sqlite3"
)
// ApiMiddleware will add the db connection to the context
func ApiMiddleware(db gorm.DB) gin.HandlerFunc {
return func(c *gin.Context) {
c.Set("databaseConn", db)
c.Next()
}
}
func main() {
r := gin.New()
// In this example, I'll open the db connection here...
// In your code you would probably do it somewhere else
db, err := gorm.Open("sqlite3", "./example.db")
if err != nil {
log.Fatal(err)
}
r.Use(ApiMiddleware(db))
r.GET("/api", func(c *gin.Context) {
// Don't forget type assertion when getting the connection from context.
dbConn, ok := c.MustGet("databaseConn").(gorm.DB)
if !ok {
// handle error here...
}
// do your thing here...
})
r.Run(":8080")
}
This is just a simple POC. But i believe it's a start.
Hope it helps.
Late to the party, so far here is my proposal. Encapsulate methods into the object with private/public vars in it:
package main
import (
"log"
"github.com/gin-gonic/gin"
"github.com/jinzhu/gorm"
_ "github.com/mattn/go-sqlite3"
)
type HandlerA struct {
Db gorm.DB
}
func (this *HandlerA) Get(c *gin.Context) {
log.Info("[%#f]", this.Db)
// do your thing here...
}
func main() {
r := gin.New()
// Init, should be separate, but it's ok for this sample:
db, err := gorm.Open("sqlite3", "./example.db")
if err != nil {
log.Fatal(err)
}
Obj := new(HandlerA)
Obj.Db = db // Or init inside Object
r := gin.New()
Group := r.Group("api/v1/")
{
Group.GET("/storage", Obj.Get)
}
r.Run(":8080")
}
Handler closures are a good option, but that works best when the argument is used in that handler alone.
If you have route groups, or long handler chains, where the same argument is needed in multiple places, you should set values into the Gin context.
You can use function literals, or named functions that return gin.HandlerFunc to do that in a clean way.
Example injecting configs into a router group:
Middleware package:
func Configs(conf APIV1Config) gin.HandlerFunc {
return func(c *gin.Context) {
c.Set("configKey", conf) // key could be an unexported struct to ensure uniqueness
}
}
Router:
conf := APIV1Config{/* some api configs */}
// makes conf available to all routes in this group
g := r.Group("/api/v1", middleware.Configs(conf))
{
// ... routes that all need API V1 configs
}
This is also easily unit-testable. Assuming that you test the single handlers, you can set the necessary values into the mock context:
w := httptest.NewRecorder()
c, _ := gin.CreateTestContext(w)
c.Set("configKey", /* mock configs */)
apiV1FooHandler(c)
Now in the case of application-scoped dependencies (db connections, remote clients, ...), I agree that setting these directly into the Gin context is a poor solution.
What you should do then, is to inject providers into the Gin context, using the pattern outlined above:
Middleware package:
// provider could be an interface for easy mocking
func DBProvider(provider database.Provider) gin.HandlerFunc {
return func(c *gin.Context) {
c.Set("providerKey", provider)
}
}
Router:
dbProvider := /* init provider with db connection */
r.Use(DBProvider(dbProvider)) // global middleware
// or
g := r.Group("/users", DBProvider(dbProvider)) // users group only
Handler (you can greatly reduce the boilerplate code by putting these context getters in some helper function):
// helper function
func GetDB(c *gin.Context) *sql.DB {
provider := c.MustGet("providerKey").(database.Provider)
return provider.GetConn()
}
func createUserHandler(c *gin.Context) {
db := GetDB(c) // same in all other handlers
// ...
}
I like wildneuro's example but would do a one liner to setup the handler
package main
import (
"log"
"github.com/gin-gonic/gin"
"github.com/jinzhu/gorm"
_ "github.com/mattn/go-sqlite3"
)
type HandlerA struct {
Db gorm.DB
}
func (this *HandlerA) Get(c *gin.Context) {
log.Info("[%#f]", this.Db)
// do your thing here...
}
func main() {
r := gin.New()
// Init, should be separate, but it's ok for this sample:
db, err := gorm.Open("sqlite3", "./example.db")
if err != nil {
log.Fatal(err)
}
r := gin.New()
Group := r.Group("api/v1/")
{
Group.GET("/storage", (&HandlerA{Db: db}).Get)
}
r.Run(":8080")
}
Let me try to explain in detail so that you won't get confused.
Depending on the incoming route, you want to call a controller function. Lets say your incoming route is /books and your controller is BooksController
Your BooksController will try to fetch the books from the database and returns a response.
Now, you want this a handler within your BooksController so that you can access database.
I would do something like this. Let's assume that you are using dynamoDB and the aws sdk provides *dynamodb.DynamoDB. Depending on your db, change this variable.
Create a struct as below.
type serviceConnection struct {
db *dynamoDB.DynamoDB
// You can have all services declared here
// which you want to use it in your controller
}
In your main function, get the db connection information. Let's say you already have a function initDatabaseConnection which returns a handler to db, something like below.
db := initDatabaseConnection() -> returns *dynamodb.DynamoDB
Set db to a struct variable.
conn := new(serviceConnection)
conn.db = db
Call the gin request method with a receiver handler as below.
r := gin.Default()
r.GET("/books", conn.BooksController)
As you see, the gin handler is a controller method which has your struct instance as a receiver.
Now, create a controller method with serviceConnection struct receiver.
func (conn *serviceConnection) BooksController(c *gin.Context) {
books := getBooks(conn.db)
}
As you see here, you have access to all the serviceConnection struct variables and you can use them in your controller.
Alright, I have given you a simple example. It should work. You can extend it as per your need
func main() {
router := gin.Default()
router.GET("/test/:id/:name", testRouteHandler)
router.Run(":8080")
}
func testRouteHandler(c *gin.Context) {
id := c.Params.ByName("id")
name := c.Params.ByName("name")
}
Now you will have to call your handler as below
http://localhost:8080/test/1/myname