I've an app that uses net/http. I register some handlers with http that need to fetch some stuff from a database before we can proceed to writing the response and be done with the request.
My question is in about which the best pratice is to connect to this database. I want this to work at one request per minute or 10 request per second.
I could connect to database within each handler every time a request comes in. (This would spawn a connection to mysql for each request?)
package main
import (
"database/sql"
_ "github.com/go-sql-driver/mysql"
"net/http"
"fmt"
)
func main() {
http.HandleFunc("/",func(w http.ResponseWriter, r *http.Request) {
db, err := sql.Open("mysql","dsn....")
if err != nil {
panic(err)
}
defer db.Close()
row := db.QueryRow("select...")
// scan row
fmt.Fprintf(w,"text from database")
})
http.ListenAndServe(":8080",nil)
}
I could connect to database at app start. Whenever I need to use the database I Ping it and if it's closed I reconnect to it. If it's not closed I continue and use it.
package main
import (
"database/sql"
_ "github.com/go-sql-driver/mysql"
"net/http"
"fmt"
"sync"
)
var db *sql.DB
var mutex sync.RWMutex
func GetDb() *sql.DB {
mutex.Lock()
defer mutex.Unlock()
err := db.Ping()
if err != nil {
db, err = sql.Open("mysql","dsn...")
if err != nil {
panic(err)
}
}
return db
}
func main() {
var err error
db, err = sql.Open("mysql","dsn....")
if err != nil {
panic(err)
}
http.HandleFunc("/",func(w http.ResponseWriter, r *http.Request) {
row := GetDb().QueryRow("select...")
// scan row
fmt.Fprintf(w,"text from database")
})
http.ListenAndServe(":8080",nil)
}
Which of these ways are the best or is there another way which is better. Is it a bad idea to have multiple request use the same database connection?
It's unlikly I will create an app that runs into mysql connection limit, but I don't want to ignore the fact that there's a limit.
The best way is to create the database once at app start-up, and use this handle afterwards. Additionnaly, the sql.DB type is safe for concurrent use, so you don't even need mutexes to lock their use. And to finish, depending on your driver, the database handle will automatically reconnect, so you don't need to do that yourself.
var db *sql.DB
var Database *Database
func init(){
hostName := os.Getenv("DB_HOST")
port := os.Getenv("DB_PORT")
username := os.Getenv("DB_USER")
password := os.Getenv("DB_PASS")
database := os.Getenv("DB_NAME")
var err error
db, err = sql.Open("mysql", fmt.Sprintf("%s:%s#tcp(%s:%d)/%s", username, password, hostName, port, database))
defer db.Close()
if err != nil {
panic(err)
}
err = db.Ping()
if err != nil {
panic(err)
}
Database := &Database{conn: db}
}
type Database struct {
conn *sql.DB
}
func (d *Database) GetConn() *sql.DB {
return d.conn
}
func main() {
row := Database.GetConn().QueryRow("select * from")
}
I'd recommend make the connection to your database on init().
Why? cause init() is guaranteed to run before main() and you definitely want to make sure you have your db conf set up right before the real work begins.
var db *sql.DB
func GetDb() (*sql.DB, error) {
db, err = sql.Open("mysql","dsn...")
if err != nil {
return nil, err
}
return db, nil
}
func init() {
db, err := GetDb()
if err != nil {
panic(err)
}
err = db.Ping()
if err != nil {
panic(err)
}
}
I did not test the code above but it should technically look like this.
Related
Let's assume that I want to make a client, for example, MySQL client (but my question is generic and not just for MySQL client, any client), this is a sample code
func main() {
db, err := sql.Open("mysql", "root:<yourMySQLdatabasepassword>#tcp(127.0.0.1:3306)/test")
if err != nil {
panic(err.Error())
}
defer db.Close()
}
I want to move it into a function so I don't have to initialize it all over the place and here my question begins I can do it in various ways and I want to know which one is the recommended way
Here is functional way
func client() (client *sql.DB, err error) {
client, err = sql.Open("mysql", "root:<yourMySQLdatabasepassword>#tcp(127.0.0.1:3306)/test")
if err != nil {
panic(err.Error())
}
return
}
Global variable way
var client, err = sql.Open("mysql", "root:<yourMySQLdatabasepassword>#tcp(127.0.0.1:3306)/test")
init func way
var Client *sql.DB
func init() {
Client, err := sql.Open("mysql", "root:root#tcp(localhost:3306)/otp")
if err != nil {
panic(err.Error())
}
}
and finally struct way
type MSQL struct {
// Fields...
}
func (m *MSQL) Client() *sql.DB {
client, err := sql.Open("mysql", "root:<yourMySQLdatabasepassword>#tcp(127.0.0.1:3306)/test")
if err != nil {
panic(err.Error())
}
return client
}
// NewMysql().where(...)
I'm really confused here to select which one to follow
Exposing a public variable var Client *sql.DB is error prone as I can do <package>.Client = null from anywhere.
struct way is the way to go, As you can work with multiple connection each to a specific database at any time.
You can also use New() to pass in a config object or other props to customise other db props. like Max Connection.
package database;
type Db struct{
*db sql.DB
}
func New(driver, connection string) *Db {
db, err := sql.Open(driver, connection)
if err != nil {
panic(err)
}
if err = db.Ping(); err != nil {
panic(err)
}
/* configure details
db.SetConnMaxLifetime(conLifeTime)
db.SetMaxOpenConns(maxOpenConns)
db.SetMaxIdleConns(maxIdleConns)
*/
return &Db{db}
}
func (d *Db) Close() error {
return d.db.Close();
}
Its generic, you can pass any driver & connection string for creating a connection to any database.
connection database.New("mysql", "root:<yourMySQLdatabasepassword>#tcp(127.0.0.1:3306)/test")
usage database.Db.Where(...)
close connection database.Db.Close()
I am using Mongo-driver with gin framework. I have written code to connect mongodb in DB package and if I write query inside db/connect.go, it works but when I use same dbcon in other package it doesn't.
db/connect.go:
var dbcon *mongo.Database
func ConfigDB() (*mongo.Database) {
ctx := context.Background()
client, err := mongo.Connect(
ctx,
options.Client().ApplyURI("mongodb://localhost:27017/todo"),
)
if err != nil {
log.Fatal(err)
}
dbcon = client.Database("todo")
}
if I use the code below in same db/connect.go, then it works but when I use the same code in handler/task.go, then it won't.
func CreateTask() () {
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
res, err := dbcon.Collection("ttest").InsertOne(ctx, bson.D{
{"task", "test4"},
{"createdAt", "test"},
{"modifiedAt","test3"},
})
if err != nil {
fmt.Println( err))
}
}
I have to implement a mongo-driver in my project, but due to above issue I am facing problem to implement.
You'll have to import to import the db/connect.go file into the handler/task.go. This is not working because they are in different packages.
In my opinion you could refactor your code like this
func ConfigDB() (*mongo.Database) {
ctx := context.Background()
client, err := mongo.Connect(
ctx,
options.Client().ApplyURI("mongodb://localhost:27017/todo"),
)
if err != nil {
log.Fatal(err)
}
return client.Database("todo")
}
import (
"db/connect"
)
func CreateTask() () {
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
res, err := ConfigDB().Collection("test").InsertOne(ctx, bson.D{
{"task", "test4"},
{"createdAt", "test"},
{"modifiedAt","test3"},
})
if err != nil {
fmt.Println( err))
}
}
Here I post a complete working example.
I catch the mongo session connection in a global variable. So that, anywhere in the project you can access the active connection.
package main
import (
"fmt"
"net/http"
"os"
"github.com/gin-gonic/gin"
"gopkg.in/mgo.v2"
"gopkg.in/mgo.v2/bson"
)
// SESSION ensure global mongodb connection
var SESSION *mgo.Session
func init() {
// mongodb manual connection using host ip. Replace your host IP address there
session, err := mgo.Dial("172.17.0.2")
// session, err := mgo.Dial("<HostIP>")
Must(err)
fmt.Println(err)
SESSION = session
}
func main() {
port := os.Getenv("PORT")
gin.SetMode(gin.ReleaseMode)
// gin.SetMode(gin.DebugMode)
r := gin.Default()
r.Use(mapMongo)
if port == "" {
port = "8000"
}
r.POST("/api/v1/task", CreateTask)
http.ListenAndServe(":"+port, r)
}
// close connection
func mapMongo(c *gin.Context) {
s := SESSION.Clone()
defer s.Close()
c.Set("mongo", s.DB("mongotask"))
c.Next()
}
// Must to catch the mongo panic issues
func Must(err error) {
if err != nil {
panic(err.Error())
}
}
// NewTask Struct/model
type NewTask struct {
Id bson.ObjectId `json:"_id,omitempty" bson:"_id,omitempty"`
Task string
}
// Mongo bson generate New unique Id each request
func (self *NewTask) Init() {
self.Id = bson.NewObjectId()
}
const (
// CollectionTask is the collection name
CollectionTask = "taskCollection"
)
// CreateTask to create new Task message
func CreateTask(c *gin.Context) {
var newTask NewTask
err := c.BindJSON(&newTask)
if err != nil {
c.Error(err)
return
}
mongodb := c.MustGet("mongo").(*mgo.Database)
con := mongodb.C(CollectionTask)
// fmt.Println(newTask)
con.Insert(newTask)
if err != nil {
c.Error(err)
}
}
I have two packages, main and db.
However, I get "DB declared and not used" error.
db.go
package db
import (
"database/sql"
)
var DB *sql.DB
func Connect() {
DB, err := sql.Open("mysql", "root:xxxx#/xxxx")
if err != nil {
panic(err.Error())
}
}
func Close() {
DB.Close()
}
main.go
package main
import (
"database/sql"
// "fmt"
_ "github.com/go-sql-driver/mysql"
"html/template"
"net/http"
"github.com/****/****/config"
"github.com/****/****/db"
)
var tpl *template.Template
func init() {
tpl = template.Must(template.ParseGlob("templates/*.gohtml"))
}
func main() {
Connect()
defer Close()
loadRoutes()
http.ListenAndServe(":8080", nil)
}
Golang is strict about variable declaration, it is also mentioned in the Golang FAQs:
The presence of an unused variable may indicate a bug, while unused
imports just slow down compilation, an effect that can become
substantial as a program accumulates code and programmers over time.
For these reasons, Go refuses to compile programs with unused
variables or imports, trading short-term convenience for long-term
build speed and program clarity.
It's easy to address the situation, though. Use the blank identifier to let unused things persist while you're developing.
_, err := sql.Open("mysql", "root:Berlin2018#/jplatform")
But since you want db instance by creating a connection. I suggest to use it either by returning from the function OR You can check for the connection if it is working or not by sending ping to the database server as:
var DB *sql.DB
func Connect() {
DB, err := sql.Open("mysql", "root:Berlin2018#/jplatform")
if err = DB.Ping(); err != nil {
log.Panic(err)
}
}
Or you can create a struct which you can use anywhere you want including use of method receiver for every function which require db connection for querying the database as
type Env struct {
db *sql.DB
}
func Connect() {
db, err := sql.Open("mysql", "root:Berlin2018#/jplatform")
_ = &Env{db: db}
}
func(env *Env) getDataFromDatabase(){}
You are not using your DB variable on db.go:
package db
import (
"database/sql"
)
var DB *sql.DB
func Connect() {
con, err := sql.Open("mysql", "root:Berlin2018#/jplatform")
if err != nil {
panic(err.Error())
}
DB = con
}
func Close() {
DB.Close()
}
I was performing some RPC tests, and stumbled across a problem I can't seem to solve. In my testing I create three separate RPC servers, all of which I try to close and shutdown. However upon performing my last test (TestRpcCodecServerClientComm), it seems my client connection is connecting to the first RPC server I started (I know this because I at some point attached IDs to the RPCHandlers), even though I attempted everything I could to make sure it was shutdown. Though the code is not there I have attempted to inspect every single error I could, but that did not bring about anything.
rpc.go
package rbot
import (
"io"
"net"
"net/rpc"
"net/rpc/jsonrpc"
)
func RpcCodecClientWithPort(port string) (rpc.ClientCodec, error) {
conn, err := net.Dial("tcp", "localhost:"+port)
if err != nil {
return nil, err
}
return jsonrpc.NewClientCodec(conn), nil
}
func RpcCodecServer(conn io.ReadWriteCloser) rpc.ServerCodec {
return jsonrpc.NewServerCodec(conn)
}
rpc_test.go
package rbot
import (
"errors"
"fmt"
"net"
"net/rpc"
"testing"
)
type RPCHandler struct {
RPCServer net.Listener
conn rpc.ServerCodec
done chan bool
TestPort string
stop bool
GotRPC bool
}
func (r *RPCHandler) SetupTest() {
r.stop = false
r.GotRPC = false
r.done = make(chan bool)
r.TestPort = "5556"
}
// TODO: Create separate function to handle erroring
func (r *RPCHandler) CreateRPCServer() error {
rpc.RegisterName("TestMaster", TestAPI{r})
var err error
r.RPCServer, err = net.Listen("tcp", ":"+r.TestPort)
if err != nil {
return err
}
go func() {
for {
conn, err := r.RPCServer.Accept()
if err != nil || r.stop {
r.done <- true
return
}
r.conn = RpcCodecServer(conn)
rpc.ServeCodec(r.conn)
}
}()
return nil
}
func (r *RPCHandler) CloseRPCServer() error {
r.stop = true
if r.conn != nil {
err := r.conn.Close()
if err != nil {
fmt.Println(err)
}
}
err := r.RPCServer.Close()
<-r.done
return err
}
type TestAPI struct {
t *RPCHandler
}
func (tapi TestAPI) Send(msg string, result *string) error {
if msg == "Got RPC?" {
tapi.t.GotRPC = true
return nil
}
return errors.New("Didn't receive right message")
}
// Check if we can create and close an RPC server successfully using the RPC server codec.
func TestRpcCodecServer(t *testing.T) {
r := RPCHandler{}
r.SetupTest()
err := r.CreateRPCServer()
if err != nil {
t.Fatalf("Could not create rpc server! %s:", err.Error())
}
err = r.CloseRPCServer()
if err != nil {
t.Fatalf("Could not close RPC server! %s:", err.Error())
}
}
// Check if we can create a client without erroring.
func TestRpcCodecClientWithPortt(t *testing.T) {
r := RPCHandler{}
r.SetupTest()
r.CreateRPCServer()
defer r.CloseRPCServer()
RPCClient, err := RpcCodecClientWithPort(r.TestPort)
defer RPCClient.Close()
if err != nil {
t.Fatalf("Could not create an RPC client! %s:", err.Error())
}
}
// Let's double check and make sure our server and client can speak to each other
func TestRpcCodecServerClientComm(t *testing.T) {
r := RPCHandler{}
r.SetupTest()
r.CreateRPCServer()
defer r.CloseRPCServer()
RPCCodec, _ := RpcCodecClientWithPort(r.TestPort)
RPCClient := rpc.NewClientWithCodec(RPCCodec)
defer RPCClient.Close()
var result string
err := RPCClient.Call("TestMaster.Send", "Got RPC?", &result)
if err != nil {
t.Fatalf("Error while trying to send RPC message: %s", err.Error())
}
if !r.GotRPC {
t.Fatalf("Could not send correct message over RPC")
}
}
Not sure if I'm just mishandling the connection or something of the like, any help would be much appreciated.
For the Record The RPC api does receive the correct string message
While not the source of your problems, your test configuration has a few race conditions which you should take care of before they cause problems. Always check for issues with the -race option. You should also let the OS allocate the port so you don't run into conflicts. See for example how httptest.Server works.
Your failure here is that you're not creating a new rpc.Server for each test, you're reusing the rpc.DefaultServer. The first call to CreateRPCServer registers a TestAPI under the name TestMaster. Each subsequent call uses the already registered instance.
If you create a new rpc.Server each time you setup the test and register a new TestAPI, the final test will pass.
srv := rpc.NewServer()
srv.RegisterName("TestMaster", testAPI)
...
// and then use srv to handle the new connection
srv.ServeCodec(RpcCodecServer(conn))
Bolt obtains a file lock on the data file so multiple processes cannot open the same database at the same time. Opening an already open Bolt database will cause it to hang until the other process closes it.
As this is the case,is there any connection pooling concept like various clients connecting and accessing the database at the same time.? Is this possible in boltdb?Like there are various connections reading and writing in the database at the same time.How it can be implemented?
A Bolt database is usually embedded into a larger program and is not used over the network like you would with shared databases (think SQLite vs MySQL). Using Bolt is a bit like having a persistent map[[]byte][]byte if that were possible. Depending on what you are doing, you might want to just use something like Redis.
That said, if you need to use Bolt this way, it is not very difficult to wrap with a simple server. Here is an example that writes/reads keys from a Bolt DB over HTTP. You can use Keep-Alive for connection pooling.
Code at: https://github.com/skyec/boltdb-server
package main
import (
"flag"
"fmt"
"io/ioutil"
"log"
"net/http"
"time"
"github.com/boltdb/bolt"
"github.com/gorilla/mux"
)
type server struct {
db *bolt.DB
}
func newServer(filename string) (s *server, err error) {
s = &server{}
s.db, err = bolt.Open(filename, 0600, &bolt.Options{Timeout: 1 * time.Second})
return
}
func (s *server) Put(bucket, key, contentType string, val []byte) error {
return s.db.Update(func(tx *bolt.Tx) error {
b, err := tx.CreateBucketIfNotExists([]byte(bucket))
if err != nil {
return err
}
if err = b.Put([]byte(key), val); err != nil {
return err
}
return b.Put([]byte(fmt.Sprintf("%s-ContentType", key)), []byte(contentType))
})
}
func (s *server) Get(bucket, key string) (ct string, data []byte, err error) {
s.db.View(func(tx *bolt.Tx) error {
b := tx.Bucket([]byte(bucket))
r := b.Get([]byte(key))
if r != nil {
data = make([]byte, len(r))
copy(data, r)
}
r = b.Get([]byte(fmt.Sprintf("%s-ContentType", key)))
ct = string(r)
return nil
})
return
}
func (s *server) ServeHTTP(w http.ResponseWriter, r *http.Request) {
vars := mux.Vars(r)
if vars["bucket"] == "" || vars["key"] == "" {
http.Error(w, "Missing bucket or key", http.StatusBadRequest)
return
}
switch r.Method {
case "POST", "PUT":
data, err := ioutil.ReadAll(r.Body)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
err = s.Put(vars["bucket"], vars["key"], r.Header.Get("Content-Type"), data)
w.WriteHeader(http.StatusOK)
case "GET":
ct, data, err := s.Get(vars["bucket"], vars["key"])
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
w.Header().Add("Content-Type", ct)
w.Write(data)
}
}
func main() {
var (
addr string
dbfile string
)
flag.StringVar(&addr, "l", ":9988", "Address to listen on")
flag.StringVar(&dbfile, "db", "/var/data/bolt.db", "Bolt DB file")
flag.Parse()
log.Println("Using Bolt DB file:", dbfile)
log.Println("Listening on:", addr)
server, err := newServer(dbfile)
if err != nil {
log.Fatalf("Error: %s", err)
}
router := mux.NewRouter()
router.Handle("/v1/buckets/{bucket}/keys/{key}", server)
http.Handle("/", router)
log.Fatal(http.ListenAndServe(addr, nil))
}
There is no connection pooling concept in boltdb, because there is no connection. It is not a client/server database, it is an embedded database (like sqlite or Berkeley-DB).
Boltdb is designed so that multiple goroutines of the same process can access the database at the same time (using different transactions). The model is single writer, multiple readers. Boltdb is not designed to support accesses from multiple processes.
If you need a Go program to use an embedded database supporting access from multiple processes at the same time, you may want to have a look at the wrappers over LMDB, such as:
https://github.com/szferi/gomdb
https://github.com/armon/gomdb