I am trying to group my database code into a sub-package, which would sort of encapsulate my database connection and all my prepared statements.
I can create the database connection and the statements in the package's init function, but I need to close them at some point.
In a program, where these variables are initialized in the code that uses them I would use defer db.Close(), etc, but how to go about this here?
I would prefer not to expose the package's global variables (connection and statements) so that the caller could access them to close them. Is there a way to do it in a more elegant manner?
I suspect that I might have a wrong paradigm in mind, and I am trying to create objects (this is essentially what it is here) in a language that does not have them. If so then I would appreciate any help with how to do it go-way.
package database
import (
"database/sql"
_ "github.com/lib/pq"
)
var db *sql.DB
var stmtSelectUser *sql.Stmt
func GetUser(email string) string {
var name string
stmtSelectUser.QueryRow(email).Scan(&name)
return name
}
func init() {
var e error;
db, e = sql.Open("postgres", "host=localhost dbname=pictocat sslmode=disable")
stmtSelectUser, e = db.Prepare("select * from users where email='$1'")
}
It's not really clear from your example code what your problem is, but in general:
Either the function running a statement needs to finish the statement and Close() it or you need to add a function to your package so the caller can close it; it's as simple as that.
You don't need to expose all of the database stuff in your package, just a function that in turns calls Close().
Related
My Go packages sometimes require setup - it may be a connection pool for a DB, the creation of a *Regexp struct that I use throughout the package, or any number of other things.
Given that these functions/setup routines have to be called in order for my packages to function correctly, that the setup may be computationally expensive to run, and that the setups only have to be run once, I usually put them an init() function within my package - e.g.:
//Taken from example code here: https://stackoverflow.com/a/36358573/13296826
var myDb *sql.DB
func init() {
var err error
myDb, err = sql.Open("driver-name", "database=test1")
if err != nil {
panic(err)
}
}
To set up a (hypothetical) connection pool to a DB where the params are hardcoded. Or:
var myRegexp *regexp.Regexp
func init() {
//Will panic if MustCompile() fails.
myRegexp = regexp.MustCompile("[a-zA-Z0-9]{3}")
}
To compile a regexp struct for use throughout the package.
The two problems that I can see with this approach are:
Package-level variables (even unexported ones) are seen by many as bad practice (e.g. https://stackoverflow.com/a/50844500/13296826 & https://www.reddit.com/r/golang/comments/8d8qes/when_is_it_okay_to_use_package_level_variables/) - this viewpoint is often down to the fact that a package-level variable can lead to race conditions (although both sql.DB and regexp.Regexp are safe for concurrent use).
The init() functions in both of the above code examples can result in an error or panic. Since the init() is called when the package is initialised (and not directly invoked by calling package.init()), handling an error or recovering from a panic isn't possible outside of the init() (according to this: https://stackoverflow.com/a/30659042/13296826).
So, is there a better way to handle initialising a package-level variable that accounts for any errors/panics that may occur?
So I'm just getting to grips with Golang. I'm writing an application for funsies just to understand stuff and get my head around it.
I have a whole bunch of functions that will interact with a DB where I pass in *SQL.DB for the function to use. I can test those easily enough using a mocked interface from sqlmock.
No problem there.
I'm now writing the Initialisation function for the application which will initiate the DB connection which will be attached to a struct and from there passed into utility functions.
However, I am struggling to find a way to easily test that connection without having the hassle of setting up an actual database.
So I guessing that I have probably either badly structured my app, or I've missed something, potentially pretty obvious.
So here is some example code to illustrate my predicament.
util.go
package main
import (
"log"
"database/sql"
"github.com/go-sql-driver/mysql"
)
func DoDBStuff(db *sql.DB) {
rows, err := db.Query("SELECT column1, column2 FROM example")
if err != nil {
log.Error(err)
}
// do stuff with rows
}
util_test.go
package main
import (
"testing"
"github.com/DATA-DOG/go-sqlmock"
)
func TestDoDBStuff(t *testing.T) {
db, mock, err := sqlmock.New()
if err != nil {
t.Fatalf("An error '%s' was not expected when opening a stub database connection", err)
}
defer db.Close()
rows := sqlmock.NewRows([]string{"col1", "col2"})
rows.AddRow("val1", "val2")
rows.AddRow("val3", "val4")
mock.ExpectQuery("^SELECT column1, column2 from example$").WillReturnRows(rows)
DoDBStuff(db)
if err := mock.ExpectationsWereMet(); err != nil {
t.Errorf("there were unfulfilled expectations: %s", err)
}
}
That all works fine, I can test my DB queries.
However Now I want to test Initialising the App.
package main
import (
"database/sql"
"github.com/go-sql-driver/mysql"
)
type App {
DB *sql.DB
// some other data structures
}
func (a *App) InitApp(connectionString string) {
a.DB = sql.Open("mysql", connectionString)
// other init stuff
}
But as I can't pass in the SQL I don't think it can be mocked, certainly not easily. So I'm struggling a bit on how to move forward.
I am intending for this to sit behind a rest API, so on startup, the app will need to initialize before being able to process any requests.
Ideally, I'd like to be able to test the REST interface without having to set up a database, delay testing with real data until I can feed the code into a Dev environment.
So really I want to know:
Is what I'm intending possible?
Is there a better approach?
If not what am I missing? Poor test design or poor code set up?
Edit:
Folling #peter's comment I just want to clarify.
I want to test the functionality of the InitDB() function but with the sql.Open call I would need to have a Database for it to connect to, If I don't then I the call would fail and I could not effectively test the function.
There is Dockertest which creates a Docker container running whatever you want (i.e. MySQL), and you can test against that. It's designed for being able to do proper integration testing, so it should be able to do what you want (you wont need sqlmock anymore too).
I am moving to Go from Node.js and I am concerned whether a construct that I would use in Node is safe to do in Go and whether there is a more idiomatic way to accomplish the same thing. I am using the Echo framework and want to set a route specific struct that will be available within the context object. I could generate the struct for every call within middleware, but it's expensive to do so. Instead, I set the struct once in an outer func that then returns an inner func that refers to the struct in the outer func. My hope is that I then only incur the generation cost once and then have the correct struct associated with my route for every call.
e.POST(path, POST.GenericPostHandler, func(next echo.HandlerFunc) echo.HandlerFunc {
operation := getOperationMap(path)
return func(c echo.Context) error {
c.Set("op", operation)
return next(c)
}
})
Are there any concerns with this code? Will it cause problems with GC? Is there a more efficient way to accomplish the same thing? I assume a copy of the struct is made every time the middleware is called.
This code is safe, won't cause GC problems, and is a good, idiomatic pattern that can be used in Go.
In your example, only one operation will be created, moved to the heap, and then shared by each request as they are handled by Echo.
I often use this exact pattern myself when I need to initialize an expensive struct that will be used when handling all requests.
If operationMap never changes after initialization, You can declare operationMap as a singleton instance like following:
package main
import (
"fmt"
"sync"
)
var (
operationMapInst map[string]string // I don't know the exact type of map, so you should change the type.
operationMapOnce sync.Once
)
func getOperationMap() map[string]string {
// operationMapOnce.Do() runs only once
// when the first time getOperationMap() is called.
operationMapOnce.Do(func() {
// Initialize operationMapInst.
operationMapInst = map[string]string{"/": "root", "/ver": "version"}
fmt.Println("operaionMap has initialized!")
})
return operationMapInst
}
func main() {
// The initialization logic runs only once.
// Because getOperationMap() returns map,
// syntax for the value for a path should be getOperationMap()[path],
// not getOperationMap(path).
rootOp, ok := getOperationMap()["/"]
fmt.Println(rootOp, ok)
// repetition
rootOp, ok = getOperationMap()["/"]
fmt.Println(rootOp, ok)
verOp, ok := getOperationMap()["/ver"]
fmt.Println(verOp, ok)
verOp, ok = getOperationMap()["/ver"]
fmt.Println(verOp, ok)
}
You can run this code here.
I recommend http://marcio.io/2015/07/singleton-pattern-in-go/ for understanding singleton pattern in Go.
Beego ORM is somehow incomplete for now (for example it doesn't support foreign key constraints). So I've decided to use gorm with Beego. What is proper way of doing that? I've seen the sample code from gorm:
import (
"github.com/jinzhu/gorm"
_ "github.com/jinzhu/gorm/dialects/postgres"
)
func main() {
db, err := gorm.Open("postgres", "host=myhost user=gorm dbname=gorm sslmode=disable password=mypassword")
defer db.Close()
}
But do I have to connect to database each time in every controller function? Is there a way to use something like long polling connections?
gorm uses sql.DB type embedded in gorm.DB under the hood which
DB is a database handle representing a pool of zero or more
underlying connections. It's safe for concurrent use by multiple
goroutines. The sql package creates and frees connections
automatically; it also maintains a free pool of idle connections.
So you can use obtained DB globally in your code,
if you want level of isolation in request handling use transaction
tr:=db.Begin()
So, as #Uvelichitel pointed out, your option is to define your db connection at the global level and to use it from a desired place (probably main function to open a connection and model layer to query for results).
So you could basically have a file containing your db connection logics:
// appname/conn.go
package db
import (
"github.com/jinzhu/gorm"
...
)
var (
// this one gonna contain an open connection
// make sure to call Connect() before using it
Conn *gorm.DB
)
func Connect(dbConnString string) (*gorm.DB, error) {
db, err := gorm.Open("postgres", dbConnString)
Conn = db
return db, err
}
After you call db.Connect from your main.go you are free to use an opened connection db.Conn from anywhere of your application (just make sure you're importing this package to the places of use).
import "appname/db"
func main() {
conn, _ := db.Connect("host=localhost user=postgres ...")
// db.Conn is initialized and ready for usage anywhere else
The same result could be achieved within a single main.go file, moving global variable declaration and a connection logics straight there.
sql.Open() returns a variable of type *sql.DB
I have a function that calls 10 other functions that all need to make database calls
Is it more correct/efficient to:
Send the *sql.DB pointer to every function, or
Create a new *sql.DB object in each function
Meaning
func DoLotsOfThings() {
db, _ := sql.Open()
defer db.Close()
DoTask1(db)
DoTask2(db)
}
or
func DoLotsOfThings() {
DoTask1()
DoTask2()
}
func DoTask1() {
db, _ := sql.Open()
defer db.Close()
}
func DoTask1() {
db, _ := sql.Open()
defer db.Close()
}
The reason why I'm asking is because I am currently sending the pointer to each function and my driver seems to break. I'm using http://code.google.com/p/odbc , which leads me to believe each function should have its own, and that I can rely on the driver's internals.
EDIT
RE driver breakage, it only happens under high traffic environments. And it only happens after say, ten minutes or so of time. Which leads me to believe that there is some sort of memory leak that makes using the driver stop working. However I defer db.Close() for every instance of *sql.DB, so I don't know what else I can do to resolve this issue.
andybalholm says the connection pooling is handled internally, which seems to be accurate, because it only breaks after I attempt to execute something, not when I invoke sql.Open()
If I leave my Go app running, it will not be able to execute any sort of SQL queries, but if I attempt to run other Go tests separately connecting to MSSQL and running queries, it works.
Declare a var db *sql.DB globally, and then reuse it across your code. Here is an example (simplified):
var db *sql.DB
func DoLotsOfThings() {
DoTask1(db)
DoTask2(db)
}
func main() {
db, _ = sql.Open() # or whatever you use
defer db.Close()
DoLotsOfThings()
}
Declaring *sql.DB globally also have some additional benefits such as SetMaxIdleConns (regulating connection pool size) or preparing SQL statements across your application.
You shouldn't need to open database connections all over the place. The database/sql package does connection pooling internally, opening and closing connections as needed, while providing the illusion of a single connection that can be used concurrently.
Probably you need to look elsewhere for the cause of your driver breakage. Some more details about that would make it easier for people to figure out what is going on.