Best Practise opening DB - go

Note: I am not sure if this is the most accurate title for this post, if not, please advise on a better one.
Currently I am creating a server where I have a couple of handlers (using goji). After receiving a request, I want to interact with a MongoDB database I have (using mgo). My question is:
I am assuming doing this kind of stuff every time I am handling a request is expensive:
uri := os.Getenv("MONGOHQ_URL")
if uri == "" {
panic("no DB connection string provided")
}
session, err := mgo.Dial(uri)
So, would it be better for me to have a global var that I can access from inside the handlers? So I would go with something like this:
var session *mgo.Session
func main() {
session = setupDB()
defer session.Close()
goji.Get("/user", getUser)
goji.Serve()
}
func getUser(c web.C, w http.ResponseWriter, r *http.Request) {
// Use the session var here
}
My question is related to what would be the best practise here? Opening the DB every time a request comes in, or keep it open for the entire duration of the application.

What about wraping your handler in a Controller struct like this: (http://play.golang.org/p/NK6GO_lqgk)
package main
import (
"fmt"
"log"
"net/http"
"os"
"github.com/zenazn/goji"
"github.com/zenazn/goji/web"
)
type Controller struct {
session *Session
}
func NewController() (*Controller, error) {
if uri := os.Getenv("MONGOHQ_URL"); uri == "" {
return nil, fmt.Errorf("no DB connection string provided")
}
session, err := mgo.Dial(uri)
if err != nil {
return nil, err
}
return &Controller{
session: session,
}, nil
}
func (c *Controller) getUser(c web.C, w http.ResponseWriter, r *http.Request) {
// Use the session var here
}
func main() {
ctl, err := NewController()
if err != nil {
log.Fatal(err)
}
defer ctl.session.Close()
goji.Get("/user", ctl.getUser)
goji.Serve()
}
This way, you can embed your session in your handler and add any other data that you might need.

Related

How to simulate multiple different HTTP responses using Go's httptest?

I have created some Go functions that make HTTP GET calls to services that are out there on the internet and parse the results.
I am now working on writing test-cases for these functions.
In my test cases, I'm using the go package httptest to simulate calls to these external services. Below is my code. Error checking is purposefully removed for brevity. Here is the go-playground.
package main
import (
"fmt"
"io"
"context"
"net/http"
"net/http/httptest"
)
func handlerResponse() http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusOK)
w.Write([]byte(`{"A":"B"}`))
})
}
func buildMyRequest(ctx context.Context, url string) *http.Request {
request, _ := http.NewRequestWithContext(ctx, "GET", url, nil)
return request
}
func myPrint(response *http.Response) {
b := make([]byte, 60000)
for {
_, err := response.Body.Read(b)
if err == io.EOF {
break
}
}
fmt.Println(string(b))
}
func main() {
srv := httptest.NewServer(handlerResponse())
client := http.Client{}
myResponse1, _ := client.Do(buildMyRequest(context.Background(), srv.URL))
fmt.Println("myResponse1:")
myPrint(myResponse1)
myResponse2, _ := client.Do(buildMyRequest(context.Background(), srv.URL))
fmt.Println("myResponse2:")
myPrint(myResponse2)
}
This is the output it produces:
myResponse1:
{"A":"B"}
myResponse2:
{"A":"B"}
As you can see, I have created some dummy HTTP response data {"A":"B"} and when you send an HTTP request to srv.URL, it actually hits an ephemeral HTTP server which responds with the dummy data. Cool!
When you send the second HTTP request to srv.URL, it again responds with the same dummy data. But this is where my problem arises. I want the ephemeral HTTP server to return some different data the second time {"C":"D"} and third time {"E":"F"} it receives a request.
How can I change the first line of the main() function so that the server responds with my desired data on subsequent HTTP calls?
you could use a hack like follows ( playground : here)
package main
import (
"fmt"
"io"
"context"
"net/http"
"net/http/httptest"
"sync"
)
type responseWriter struct{
resp map[int]string
count int
lock *sync.Mutex
}
func NewResponseWriter()*responseWriter{
r := new(responseWriter)
r.lock = new(sync.Mutex)
r.resp = map[int]string{
0: `{"E":"F"}`,
1: `{"A":"B"}`,
2: `{"C":"D"}`,
}
r.count = 0
return r
}
func (r *responseWriter)GetResp()string{
r.lock.Lock()
defer r.lock.Unlock()
r.count ++
return r.resp[r.count%3]
}
func handlerResponse(rr *responseWriter) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusOK)
w.Write([]byte(rr.GetResp()))
})
}
func buildMyRequest(ctx context.Context, url string) *http.Request {
request, _ := http.NewRequestWithContext(ctx, "GET", url, nil)
return request
}
func myPrint(response *http.Response) {
b := make([]byte, 60000)
for {
_, err := response.Body.Read(b)
if err == io.EOF {
break
}
}
fmt.Println(string(b))
}
func main() {
rr := NewResponseWriter()
srv := httptest.NewServer(handlerResponse(rr))
client := http.Client{}
myResponse1, err := client.Do(buildMyRequest(context.Background(), srv.URL))
if err != nil{
fmt.Println(err)
return
}
defer myResponse1.Body.Close()
fmt.Println("myResponse1:")
myPrint(myResponse1)
myResponse2, err := client.Do(buildMyRequest(context.Background(), srv.URL))
if err != nil{
fmt.Println(err)
return
}
defer myResponse2.Body.Close()
fmt.Println("myResponse2:")
myPrint(myResponse2)
}

How correctly to make controllers to routes in Golang?

I am new in Golang and need some help. I am tring to create REST API web service without ORM.
Right now I am successfully connected to PostgreSQL database. In database I have table which called factors. I want to create CRUD operations. The problem is with controllers logic.
main.go:
package main
import (
"github.com/gorilla/mux"
"log"
"net/http"
"rest_api/configurations"
"rest_api/controllers"
)
func main() {
db, err := configurations.PostgreSQLDatabase()
if err != nil {
log.Fatal(err)
}
router := mux.NewRouter()
router.StrictSlash(true)
subrouter := router.PathPrefix("/api").Subrouter()
subrouter.HandleFunc("/factors", controllers.GetFactors(db)).Methods("GET")
log.Fatal(http.ListenAndServe(":8000", router))
}
models/factors.go:
package models
type Factor struct {
ID int `json:"id"`
Name string `json:"name"`
}
How correctly looks like the GetFactors controller? Can someone show me please. For instance I pass db object to GetFactors controller as in the example below. Unfortunately it seems like it's incorrect.
controllers/factors.go:
func GetFactors(db *sql.DB, w http.ResponseWriter, req *http.Request) {
// some code
}
configurations/PostgreSQL.go:
func PostgreSQLDatabase() (*sql.DB, error) {
// Load environment variables from ".env" file.
err := godotenv.Load(".env")
if err != nil {
log.Fatal(err)
}
// Initialize database-related variables.
dbUser := os.Getenv("PostgreSQL_USER")
dbPassword := os.Getenv("PostgreSQL_PASSWORD")
dbHost := os.Getenv("PostgreSQL_HOST")
dbName := os.Getenv("PostgreSQL_DB_NAME")
dbURL := fmt.Sprintf("user=%s password=%s host=%s dbname=%s sslmode=disable", dbUser, dbPassword, dbHost, dbName)
// Create PostgreSQL database connection pool.
db, err := sql.Open("postgres", dbURL)
if err != nil {
return nil, err
}
// Ping PostgreSQL database to make sure it's alive.
err = db.Ping()
if err != nil {
log.Fatal(err)
} else {
log.Println("Web service successfully connected to remote PostgreSQL database.")
}
return db, nil
}
A pattern I like to use is to define your own Router struct that has a mux.Router as a field as well as encapsulates things like your database connection, application config and etc.
I find doing it this way makes it easily update your routes when they require different resources and development proceeds.
First create a router object that takes in the database connection on creation and makes it available to all routes you wish to use.
router.go
package main
import (
"net/http"
"database/sql"
"github.com/gorilla/mux"
)
type Router struct {
router *mux.Router
db *sql.DB
}
func NewRouter(db *sql.DB) (*Router, error) {
router := mux.NewRouter()
router.StrictSlash(true)
subrouter := router.PathPrefix("/api").Subrouter()
r := &Router{
router: router,
db: db,
}
subrouter.HandleFunc("/factors", r.GetFactors).Methods(http.MethodGet)
return r, nil
}
func (r *Router) GetFactors(w http.ResponseWriter, req *http.Request) {
// Now you can access your database via `r.db`
}
// Needed so we can pass our custom router to ListenAndServe.
func (r *Router) ServeHTTP(w http.ResponseWriter, req *http.Request) {
r.router.ServeHTTP(w, req)
}
Then in main.go you can simply create your custom router, passing it your database connection. Then the custom router can be passed directly to ListenAndServe.
main.go
package main
import (
"log"
"net/http"
"rest_api/configurations"
"rest_api/controllers"
)
func main() {
db, err := configurations.PostgreSQLDatabase()
if err != nil {
log.Fatal(err)
}
router, err := NewRouter(db)
if err != nil {
log.Fatalf("error initializing router: %v", err)
}
log.Fatal(http.ListenAndServe(":8000", router))
}
Hopefully this helps.
Your func GetFactors must looks like:
func GetFactors(w http.ResponseWriter, r *http.Request) {}
and in main file you must have:
subrouter.HandleFunc("/factors", controllers.GetFactors).Methods("GET")
and with purpose to get DB connection you must have func like GetDB in your package "rest_api/configurations".
In "rest_api/configurations" you must have something like:
var db *PostgreSQLDatabase
func init() {
var err error
db, err = configurations.PostgreSQLDatabase()
if err != nil {
log.Fatal(err)
}
}
func GetDB() *PostgreSQLDatabase {
return db
}
There is no correct way, it mostly opinion-based.
The semantic of HandlerFunc function should be like func(w http.ResponseWriter, r *http.Request), in order to pass database you can use closures, here is an example.
main.go
// ... some code here
subrouter.HandleFunc("/factors", controllers.GetFactors(db)).Methods("GET")
// ... some code here
controllers/factors.go
func GetFactors(db *sql.DB) http.HandlerFunc {
return http.HandlerFunc(func(w http.ResponseWriter, req *http.Request) {
// some code
})
}
Another option:
I'm not quite sure about this, but you can adjucts it to your needs. Initialize a Controller struct and pass db to it:
main.go
// ... some code here
db, err := configurations.PostgreSQLDatabase()
if err != nil {
log.Fatal(err)
}
ctrl := controllers.Controller{DB: db}
subrouter.HandleFunc("/factors", ctrl.GetFactors).Methods("GET")
// ... some code here
Denote a method on the Controller struct.
Define a struct in the controllers
controllers/factors.go
type Controller struct {
DB *PostgreSQLDatabase
}
func (c Controller) GetFactors(w http.ResponseWriter, req *http.Request) {
// some code
// c.DB.MySqlMethod()
}

Passing Data from Datastore to http.Writer

I was recently introduced to the wonders of the language known as go. I set myself a task of writing a RESTful API using GoLang and Google's Datastore. I am able to retrieve data from Datastore and Print it to the console using fmt.Println however the issue comes into play when I try to use the data from Datastore and pass it to the http.Handler.
I was wondering if someone could inform me of where I am going wrong or even point me in the right direction.
Here is what I have done so far
package main
import (
"log"
"fmt"
"context"
"net/http"
// "encoding/json"
"cloud.google.com/go/datastore"
)
type Item struct {
Id string `datastore:"id"`
Name string `datastore:"title"`
View int `datastore:"views"`
Brand string `datastore:"brand"`
id int64 // interger from "Name/ID" fild in datastore entities list
}
func main() {
http.HandleFunc("/", ListTasks)
http.ListenAndServe(":8080", nil)
}
//func ListTasks(w http.ResponseWriter, r *http.Request) ([]*Item, error) {
func ListTasks(w http.ResponseWriter, r *http.Request) {
ctx := context.Background()
client, err := datastore.NewClient(ctx, "my-client")
if err != nil {
log.Fatalln(err)
}
var tasks []*Item
query := datastore.NewQuery("my-query")
keys, err := client.GetAll(ctx, query, &tasks)
if err != nil {
return nil, err
}
for i, key := range keys {
tasks[i].id = key.ID
}
return tasks, nil
}
I've also looked into http Wrappers, but I'm unaware if using a wrapper is 100% necessary or if I'm just adding more to my plate.
I've removed the return tasks, nil as it appeared to be unnecessary, modified the return, nil err to log.Fatalln(nil, err) and also encoded tasks as instructed by #ThunderCat and #tkausl. My issue has been resolved, thank you.
Here is my working code
package main
import (
"log"
"context"
"net/http"
"encoding/json"
"cloud.google.com/go/datastore"
)
type Item struct {
Id string `datastore:"id"`
Name string `datastore:"title"`
View int `datastore:"views"`
Brand string `datastore:"brand"`
id int64 // interger from "Name/ID" fild in datastore entities list
}
func main() {
http.HandleFunc("/", ListTasks)
http.ListenAndServe(":8080", nil)
}
func ListTasks(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/json")
ctx := context.Background()
client, err := datastore.NewClient(ctx, "my-client")
if err != nil {
log.Fatalln(err)
}
var tasks []*Item
query := datastore.NewQuery("my-query")
keys, err := client.GetAll(ctx, query, &tasks)
if err != nil {
log.Fatalln(nil, err)
}
for i, key := range keys {
tasks[i].id = key.ID
}
json.NewEncoder(w).Encode(tasks)
// return tasks, nil
}
It now returns [{"Id":"24X660","Name":"Fiesta","View":129,"Brand":"Ford"}]
Also thank you to #static_cast for correcting my formatting errors.

Web address routing works fine with http.ListenAndServe, but fails with cgi.Serve()

I'm working on a website using Go. The server constraints require that I use CGI. When I test the following code locally using http.ListenAndServe() (commented out below), the various handlers are called correctly depending on the address requested. However, if I use cgi.Serve() instead, the default router is executed for all addresses (i.e., the handler for "/" is always executed). I'd appreciate any clues as to how to fix the issue.
EDIT: Here is the simplest test case I can think of to show the problem:
//=============SIMPLIFIED CODE================//
package main
import (
"fmt"
"net/http"
"net/http/cgi"
)
func defaultHandler(w http.ResponseWriter, r *http.Request) {
fmt.Fprintf(w, "Default")
}
func otherHandler(w http.ResponseWriter, r *http.Request) {
fmt.Fprintf(w, "Other")
}
func main() {
http.HandleFunc("/other", otherHandler)
http.HandleFunc("/", defaultHandler)
/*
//Works fine
err := http.ListenAndServe(":8090", nil)
if err != nil {
panic(err)
}
*/
//Always fires defaultHandler no matter the address requested
err := cgi.Serve(nil)
if err != nil {
panic(err)
}
}
//=============CODE FROM ORIGINAL POST===================//
package main
import (
"fmt"
"net/http"
"net/http/cgi"
"net/url"
"os"
"github.com/go-cas/cas"
)
func logoutHandler(w http.ResponseWriter, r *http.Request) {
cas.RedirectToLogout(w, r)
}
func calendarHandler(w http.ResponseWriter, r *http.Request) {
if !cas.IsAuthenticated(r) {
cas.RedirectToLogin(w, r)
}
fmt.Fprintf(w, "Calendar for %s", cas.Username(r))
}
func defaultHandler(w http.ResponseWriter, r *http.Request) {
if !cas.IsAuthenticated(r) {
cas.RedirectToLogin(w, r)
}
fmt.Fprintf(w, "Hi there %s!", cas.Username(r))
}
func main() {
u, _ := url.Parse("https://www.examplecasserver.com")
client := cas.NewClient(&cas.Options{
URL: u,
})
http.Handle("/logout", client.HandleFunc(logoutHandler))
http.Handle("/calendar", client.HandleFunc(calendarHandler))
http.Handle("/", client.HandleFunc(defaultHandler))
/*
err := http.ListenAndServe(":8080", nil)
if err != nil {
log.Fatal("ListenAndServe: ", err)
}
*/
err := cgi.Serve(nil)
if err != nil {
panic(err)
}
}
The CGI program expects some variables to be set in order to build the request.
Probably there is some issue with the configuration of your web server in which the variables are either not set correctly or not named correctly.
To verify this:
1) Add this before calling cgi.Serve and you'll see how the right handler is called (otherHandler)
os.Setenv("REQUEST_METHOD", "get")
os.Setenv("SERVER_PROTOCOL", "HTTP/1.1")
os.Setenv("SCRIPT_NAME", "/other")
2) Add this at the beginning of the main to check how the variables are being set by the web server:
fmt.Println(os.Environ())
In that output, look for the CGI meta variables defined in the spec:
http://www.ietf.org/rfc/rfc3875
Look for the section "Request Meta-Variables" in that page, you are probably looking for the SCRIPT_NAME or PATH_INFO variables.
EDIT
From the variable values you pasted below, it seems the issue is that the REQUEST_URI contains an additional path component:
REQUEST_URI=/main.cgi/other
So the easiest fix would be for you to map the routes accordingly:
http.HandleFunc("/main.cgi/other", otherHandler)
http.HandleFunc("/", defaultHandler) // or maybe /main.cgi

Database connection best practice

I've an app that uses net/http. I register some handlers with http that need to fetch some stuff from a database before we can proceed to writing the response and be done with the request.
My question is in about which the best pratice is to connect to this database. I want this to work at one request per minute or 10 request per second.
I could connect to database within each handler every time a request comes in. (This would spawn a connection to mysql for each request?)
package main
import (
"database/sql"
_ "github.com/go-sql-driver/mysql"
"net/http"
"fmt"
)
func main() {
http.HandleFunc("/",func(w http.ResponseWriter, r *http.Request) {
db, err := sql.Open("mysql","dsn....")
if err != nil {
panic(err)
}
defer db.Close()
row := db.QueryRow("select...")
// scan row
fmt.Fprintf(w,"text from database")
})
http.ListenAndServe(":8080",nil)
}
I could connect to database at app start. Whenever I need to use the database I Ping it and if it's closed I reconnect to it. If it's not closed I continue and use it.
package main
import (
"database/sql"
_ "github.com/go-sql-driver/mysql"
"net/http"
"fmt"
"sync"
)
var db *sql.DB
var mutex sync.RWMutex
func GetDb() *sql.DB {
mutex.Lock()
defer mutex.Unlock()
err := db.Ping()
if err != nil {
db, err = sql.Open("mysql","dsn...")
if err != nil {
panic(err)
}
}
return db
}
func main() {
var err error
db, err = sql.Open("mysql","dsn....")
if err != nil {
panic(err)
}
http.HandleFunc("/",func(w http.ResponseWriter, r *http.Request) {
row := GetDb().QueryRow("select...")
// scan row
fmt.Fprintf(w,"text from database")
})
http.ListenAndServe(":8080",nil)
}
Which of these ways are the best or is there another way which is better. Is it a bad idea to have multiple request use the same database connection?
It's unlikly I will create an app that runs into mysql connection limit, but I don't want to ignore the fact that there's a limit.
The best way is to create the database once at app start-up, and use this handle afterwards. Additionnaly, the sql.DB type is safe for concurrent use, so you don't even need mutexes to lock their use. And to finish, depending on your driver, the database handle will automatically reconnect, so you don't need to do that yourself.
var db *sql.DB
var Database *Database
func init(){
hostName := os.Getenv("DB_HOST")
port := os.Getenv("DB_PORT")
username := os.Getenv("DB_USER")
password := os.Getenv("DB_PASS")
database := os.Getenv("DB_NAME")
var err error
db, err = sql.Open("mysql", fmt.Sprintf("%s:%s#tcp(%s:%d)/%s", username, password, hostName, port, database))
defer db.Close()
if err != nil {
panic(err)
}
err = db.Ping()
if err != nil {
panic(err)
}
Database := &Database{conn: db}
}
type Database struct {
conn *sql.DB
}
func (d *Database) GetConn() *sql.DB {
return d.conn
}
func main() {
row := Database.GetConn().QueryRow("select * from")
}
I'd recommend make the connection to your database on init().
Why? cause init() is guaranteed to run before main() and you definitely want to make sure you have your db conf set up right before the real work begins.
var db *sql.DB
func GetDb() (*sql.DB, error) {
db, err = sql.Open("mysql","dsn...")
if err != nil {
return nil, err
}
return db, nil
}
func init() {
db, err := GetDb()
if err != nil {
panic(err)
}
err = db.Ping()
if err != nil {
panic(err)
}
}
I did not test the code above but it should technically look like this.

Resources