sql.Open() returns a variable of type *sql.DB
I have a function that calls 10 other functions that all need to make database calls
Is it more correct/efficient to:
Send the *sql.DB pointer to every function, or
Create a new *sql.DB object in each function
Meaning
func DoLotsOfThings() {
db, _ := sql.Open()
defer db.Close()
DoTask1(db)
DoTask2(db)
}
or
func DoLotsOfThings() {
DoTask1()
DoTask2()
}
func DoTask1() {
db, _ := sql.Open()
defer db.Close()
}
func DoTask1() {
db, _ := sql.Open()
defer db.Close()
}
The reason why I'm asking is because I am currently sending the pointer to each function and my driver seems to break. I'm using http://code.google.com/p/odbc , which leads me to believe each function should have its own, and that I can rely on the driver's internals.
EDIT
RE driver breakage, it only happens under high traffic environments. And it only happens after say, ten minutes or so of time. Which leads me to believe that there is some sort of memory leak that makes using the driver stop working. However I defer db.Close() for every instance of *sql.DB, so I don't know what else I can do to resolve this issue.
andybalholm says the connection pooling is handled internally, which seems to be accurate, because it only breaks after I attempt to execute something, not when I invoke sql.Open()
If I leave my Go app running, it will not be able to execute any sort of SQL queries, but if I attempt to run other Go tests separately connecting to MSSQL and running queries, it works.
Declare a var db *sql.DB globally, and then reuse it across your code. Here is an example (simplified):
var db *sql.DB
func DoLotsOfThings() {
DoTask1(db)
DoTask2(db)
}
func main() {
db, _ = sql.Open() # or whatever you use
defer db.Close()
DoLotsOfThings()
}
Declaring *sql.DB globally also have some additional benefits such as SetMaxIdleConns (regulating connection pool size) or preparing SQL statements across your application.
You shouldn't need to open database connections all over the place. The database/sql package does connection pooling internally, opening and closing connections as needed, while providing the illusion of a single connection that can be used concurrently.
Probably you need to look elsewhere for the cause of your driver breakage. Some more details about that would make it easier for people to figure out what is going on.
Related
So I'm just getting to grips with Golang. I'm writing an application for funsies just to understand stuff and get my head around it.
I have a whole bunch of functions that will interact with a DB where I pass in *SQL.DB for the function to use. I can test those easily enough using a mocked interface from sqlmock.
No problem there.
I'm now writing the Initialisation function for the application which will initiate the DB connection which will be attached to a struct and from there passed into utility functions.
However, I am struggling to find a way to easily test that connection without having the hassle of setting up an actual database.
So I guessing that I have probably either badly structured my app, or I've missed something, potentially pretty obvious.
So here is some example code to illustrate my predicament.
util.go
package main
import (
"log"
"database/sql"
"github.com/go-sql-driver/mysql"
)
func DoDBStuff(db *sql.DB) {
rows, err := db.Query("SELECT column1, column2 FROM example")
if err != nil {
log.Error(err)
}
// do stuff with rows
}
util_test.go
package main
import (
"testing"
"github.com/DATA-DOG/go-sqlmock"
)
func TestDoDBStuff(t *testing.T) {
db, mock, err := sqlmock.New()
if err != nil {
t.Fatalf("An error '%s' was not expected when opening a stub database connection", err)
}
defer db.Close()
rows := sqlmock.NewRows([]string{"col1", "col2"})
rows.AddRow("val1", "val2")
rows.AddRow("val3", "val4")
mock.ExpectQuery("^SELECT column1, column2 from example$").WillReturnRows(rows)
DoDBStuff(db)
if err := mock.ExpectationsWereMet(); err != nil {
t.Errorf("there were unfulfilled expectations: %s", err)
}
}
That all works fine, I can test my DB queries.
However Now I want to test Initialising the App.
package main
import (
"database/sql"
"github.com/go-sql-driver/mysql"
)
type App {
DB *sql.DB
// some other data structures
}
func (a *App) InitApp(connectionString string) {
a.DB = sql.Open("mysql", connectionString)
// other init stuff
}
But as I can't pass in the SQL I don't think it can be mocked, certainly not easily. So I'm struggling a bit on how to move forward.
I am intending for this to sit behind a rest API, so on startup, the app will need to initialize before being able to process any requests.
Ideally, I'd like to be able to test the REST interface without having to set up a database, delay testing with real data until I can feed the code into a Dev environment.
So really I want to know:
Is what I'm intending possible?
Is there a better approach?
If not what am I missing? Poor test design or poor code set up?
Edit:
Folling #peter's comment I just want to clarify.
I want to test the functionality of the InitDB() function but with the sql.Open call I would need to have a Database for it to connect to, If I don't then I the call would fail and I could not effectively test the function.
There is Dockertest which creates a Docker container running whatever you want (i.e. MySQL), and you can test against that. It's designed for being able to do proper integration testing, so it should be able to do what you want (you wont need sqlmock anymore too).
Given the following function:
func CreateDB() (*xorm.Engine orm, error) {
}
I want to detect within that function whether it was called like this:
_, err := CreateDB()
...or like this:
orm, err := CreateDB()
I want to check the orm variable and close the connection if it's not required. Is this possible?
You can't do that and it is also not a good design. You can check from which function it was called using reflection and see if that function is using the orm variable or not.
https://golang.org/pkg/runtime/#Caller
Go uses connection pool and if you are worried about the open connection, you may use timeout for each connection made.
I'm using this to receive SNMP traps: https://github.com/soniah/gosnmp
Now, lets say I want to programmatically break out of the (taken from here):
err := tl.Listen("0.0.0.0:9162")
What are my best approaches to this?
I'm somewhat new to Golang and didnt find a way to break out of a goroutine that I have no way of modifying ("3rd party").
Thanks,
Short answer: You can't. There's no way to kill a goroutine (short of killing the entire program) from outside the goroutine.
Long answer: A goroutine can listen for some sort of "terminate" signal (via channels, signals, or any other mechanism). But ultimately, the goroutine must terminate from within.
Looking at the library in your example, it appears this functionality is not provided.
Standard https://golang.org/pkg/net/#Conn interface provides special methods SetDeadline (together with SetReadDeadline and SetWriteDeadline) to set a hard connection break time for staled connections. As I see in the source code:
type GoSNMP struct {
// Conn is net connection to use, typically established using GoSNMP.Connect()
Conn net.Conn
...
// Timeout is the timeout for the SNMP Query
Timeout time.Duration
...
net.Conn interface is exported - so you may try to get direct access to it to set up a deadline.
type TrapListener struct {
OnNewTrap func(s *SnmpPacket, u *net.UDPAddr)
Params *GoSNMP
...
}
In its turn TrapListener exports GoSNMP struct so you may have access to it. Try this:
tl := TrapListener{...}
tl.Params.Conn.SetDeadline(time.Now().Add(1*time.Second))
tl.Listen(...)
However this line disensures me - looks like it doesn't use stored connection and its options:
func (t *TrapListener) Listen(addr string) (err error) {
...
conn, err := net.ListenUDP("udp", udpAddr)
....
}
But you may try :)
Beego ORM is somehow incomplete for now (for example it doesn't support foreign key constraints). So I've decided to use gorm with Beego. What is proper way of doing that? I've seen the sample code from gorm:
import (
"github.com/jinzhu/gorm"
_ "github.com/jinzhu/gorm/dialects/postgres"
)
func main() {
db, err := gorm.Open("postgres", "host=myhost user=gorm dbname=gorm sslmode=disable password=mypassword")
defer db.Close()
}
But do I have to connect to database each time in every controller function? Is there a way to use something like long polling connections?
gorm uses sql.DB type embedded in gorm.DB under the hood which
DB is a database handle representing a pool of zero or more
underlying connections. It's safe for concurrent use by multiple
goroutines. The sql package creates and frees connections
automatically; it also maintains a free pool of idle connections.
So you can use obtained DB globally in your code,
if you want level of isolation in request handling use transaction
tr:=db.Begin()
So, as #Uvelichitel pointed out, your option is to define your db connection at the global level and to use it from a desired place (probably main function to open a connection and model layer to query for results).
So you could basically have a file containing your db connection logics:
// appname/conn.go
package db
import (
"github.com/jinzhu/gorm"
...
)
var (
// this one gonna contain an open connection
// make sure to call Connect() before using it
Conn *gorm.DB
)
func Connect(dbConnString string) (*gorm.DB, error) {
db, err := gorm.Open("postgres", dbConnString)
Conn = db
return db, err
}
After you call db.Connect from your main.go you are free to use an opened connection db.Conn from anywhere of your application (just make sure you're importing this package to the places of use).
import "appname/db"
func main() {
conn, _ := db.Connect("host=localhost user=postgres ...")
// db.Conn is initialized and ready for usage anywhere else
The same result could be achieved within a single main.go file, moving global variable declaration and a connection logics straight there.
I am trying to group my database code into a sub-package, which would sort of encapsulate my database connection and all my prepared statements.
I can create the database connection and the statements in the package's init function, but I need to close them at some point.
In a program, where these variables are initialized in the code that uses them I would use defer db.Close(), etc, but how to go about this here?
I would prefer not to expose the package's global variables (connection and statements) so that the caller could access them to close them. Is there a way to do it in a more elegant manner?
I suspect that I might have a wrong paradigm in mind, and I am trying to create objects (this is essentially what it is here) in a language that does not have them. If so then I would appreciate any help with how to do it go-way.
package database
import (
"database/sql"
_ "github.com/lib/pq"
)
var db *sql.DB
var stmtSelectUser *sql.Stmt
func GetUser(email string) string {
var name string
stmtSelectUser.QueryRow(email).Scan(&name)
return name
}
func init() {
var e error;
db, e = sql.Open("postgres", "host=localhost dbname=pictocat sslmode=disable")
stmtSelectUser, e = db.Prepare("select * from users where email='$1'")
}
It's not really clear from your example code what your problem is, but in general:
Either the function running a statement needs to finish the statement and Close() it or you need to add a function to your package so the caller can close it; it's as simple as that.
You don't need to expose all of the database stuff in your package, just a function that in turns calls Close().