So I'm just getting to grips with Golang. I'm writing an application for funsies just to understand stuff and get my head around it.
I have a whole bunch of functions that will interact with a DB where I pass in *SQL.DB for the function to use. I can test those easily enough using a mocked interface from sqlmock.
No problem there.
I'm now writing the Initialisation function for the application which will initiate the DB connection which will be attached to a struct and from there passed into utility functions.
However, I am struggling to find a way to easily test that connection without having the hassle of setting up an actual database.
So I guessing that I have probably either badly structured my app, or I've missed something, potentially pretty obvious.
So here is some example code to illustrate my predicament.
util.go
package main
import (
"log"
"database/sql"
"github.com/go-sql-driver/mysql"
)
func DoDBStuff(db *sql.DB) {
rows, err := db.Query("SELECT column1, column2 FROM example")
if err != nil {
log.Error(err)
}
// do stuff with rows
}
util_test.go
package main
import (
"testing"
"github.com/DATA-DOG/go-sqlmock"
)
func TestDoDBStuff(t *testing.T) {
db, mock, err := sqlmock.New()
if err != nil {
t.Fatalf("An error '%s' was not expected when opening a stub database connection", err)
}
defer db.Close()
rows := sqlmock.NewRows([]string{"col1", "col2"})
rows.AddRow("val1", "val2")
rows.AddRow("val3", "val4")
mock.ExpectQuery("^SELECT column1, column2 from example$").WillReturnRows(rows)
DoDBStuff(db)
if err := mock.ExpectationsWereMet(); err != nil {
t.Errorf("there were unfulfilled expectations: %s", err)
}
}
That all works fine, I can test my DB queries.
However Now I want to test Initialising the App.
package main
import (
"database/sql"
"github.com/go-sql-driver/mysql"
)
type App {
DB *sql.DB
// some other data structures
}
func (a *App) InitApp(connectionString string) {
a.DB = sql.Open("mysql", connectionString)
// other init stuff
}
But as I can't pass in the SQL I don't think it can be mocked, certainly not easily. So I'm struggling a bit on how to move forward.
I am intending for this to sit behind a rest API, so on startup, the app will need to initialize before being able to process any requests.
Ideally, I'd like to be able to test the REST interface without having to set up a database, delay testing with real data until I can feed the code into a Dev environment.
So really I want to know:
Is what I'm intending possible?
Is there a better approach?
If not what am I missing? Poor test design or poor code set up?
Edit:
Folling #peter's comment I just want to clarify.
I want to test the functionality of the InitDB() function but with the sql.Open call I would need to have a Database for it to connect to, If I don't then I the call would fail and I could not effectively test the function.
There is Dockertest which creates a Docker container running whatever you want (i.e. MySQL), and you can test against that. It's designed for being able to do proper integration testing, so it should be able to do what you want (you wont need sqlmock anymore too).
Related
I have a file with serialized array in PHP.
The content of the file locks like this
a:2:{i:250;s:7:"my_catz";s:7:"abcd.jp";a:2:{s:11:"category_id";i:250;s:13:"category_name";s:7:"my_catz";}}
The array unserialized is this
(
[250] => my_catz
[abcd.jp] => Array
(
[category_id] => 250
[category_name] => my_catz
)
)
Now, i want to get the content of the file in GO, unserialize it convert it to an array.
In GO i can get the content of the file using
dat, err := os.ReadFile("/etc/squid3/compiled-categories.db")
if err != nil {
if e.Debug {
log.Printf("error reading /etc/squid3/compiled-categories.db: ", err)
}
}
And unserialized it using github.com/techoner/gophp library
package categorization
import (
"fmt"
"os"
"github.com/techoner/gophp"
"log"
"errors"
)
type Data struct {
Website string
Debug bool
}
func (e Data) CheckPersonalCategories() (int,string) {
if e.Debug {
log.Printf("Checking Personal Categories")
}
if _, err := os.Stat("/etc/squid3/compiled-categories.db"); errors.Is(err, os.ErrNotExist) {
if e.Debug {
log.Printf("/etc/squid3/compiled-categories.db not exit: ", err)
}
return 0,""
}
dat, err := os.ReadFile("/etc/squid3/compiled-categories.db")
if err != nil {
if e.Debug {
log.Printf("error reading /etc/squid3/compiled-categories.db: ", err)
}
}
out, _ := gophp.Unserialize(dat)
fmt.Println(out["abcd.jp"])
return 0,""
}
But I can't access to the array, for example, when I try access to array key using out["abcd.jp"] i get this error message
invalid operation: out["abcd.jp"] (type interface {} does not support indexing)
The result of out is
map[250:my_catz abcd.jp:map[category_id:250 category_name:my_catz]]
Seams that is unserializing
Don't make assumptions about what is and isn't succeeding in your code. Error responses are the only reliable way to know whether a function succeeded. In this case the assumption may hold, but ignoring errors is always a mistake. Invest time in catching errors and at least panic them - don't instead waste your time ignoring errors and then trying to debug unreliable code.
invalid operation: out["abcd.jp"] (type interface {} does not support indexing)
The package you're using unfortunately doesn't provide any documentation so you have to read the source to understand that gophp.Unserialize returns (interface{}, error). This makes sense; php can serialize any value, so Unserialize must be able to return any value.
out is therefore an interface{} whose underlying value depends on the data. To turn an interface{} into a particular value requires a type assertion. In this case, we think the underlying data should be map[string]interface{}. So we need to do a type assertion:
mout, ok := out.(map[string]interface{})
Before we get to the working code, one more point I'd like you to think about. Look at the code below: I started it from your code, but the resemblance is very slight. I took out almost all the code because it was completely irrelevant to your question. I added the input data to the code to make a minimal reproduction of your code (as I asked you to do and you declined to do). This is a very good use of your time for 2 reasons: first, it makes it a lot easier to get answers (both because it shows sufficient effort on your part and because it simplifies the description of the problem), and second, because it's excellent practice for debugging. I make minimal reproductions of code flows all the time to better understand how to do things.
You'll notice you can run this code now without any additional effort. That's the right way to provide a minimal reproducible example - not with a chunk of mostly irrelevant code which still can't be executed by anybody.
The Go Plaground is a great way to demonstrate go-specific code that others can execute and investigate. You can also see the code below at https://go.dev/play/p/QfCl08Gx53e
package main
import (
"fmt"
"github.com/techoner/gophp"
)
type Data struct {
Website string
Debug bool
}
func main() {
var dat = []byte(`a:2:{i:250;s:7:"my_catz";s:7:"abcd.jp";a:2:{s:11:"category_id";i:250;s:13:"category_name";s:7:"my_catz";}}`)
out, err := gophp.Unserialize(dat)
if err != nil {
panic(err)
}
if mout, ok := out.(map[string]interface{}); ok {
fmt.Println(mout["abcd.jp"])
}
}
I have a tests which looks like:
package tst
import (
"testing"
"github.com/demas/cowl-go/pkg/postgres"
"log"
"os"
"fmt"
"github.com/jmoiron/sqlx"
"github.com/demas/cowl-go/pkg/quzx-crawler"
"github.com/SlyMarbo/rss"
"time"
_ "github.com/lib/pq"
)
func TestMain(m *testing.M) {
prepare()
retCode := m.Run()
os.Exit(retCode)
}
func prepare() {
connectionString := fmt.Sprintf("user=%s password=%s host=%s port=%s dbname=%s sslmode=disable",
os.Getenv("DBUSER"),
os.Getenv("DBPASS"),
os.Getenv("DBHOST"),
os.Getenv("DBPORT"),
os.Getenv("DBNAME"))
db, err := sqlx.Open("postgres", connectionString)
if err != nil {
log.Fatal(err)
}
db.Exec(`DELETE FROM Settings`)
db.Exec(`DELETE FROM HackerNews`)
// ....
}
Tests works fine if I keep in the root project folder, but if I move them to tst folder I get error message:
D:\development\gopath\src\github.com\demas\cowl-go\tst>go test -v
2017/03/31 16:30:06 sql: unknown driver "postgres" (forgotten import?)
exit status 1
FAIL github.com/demas/cowl-go/tst 0.085s
Why ?
As already mentioned by #JimB in the comments, the error means that you're trying to open a db connection, using sqlx.Open, without first importing a db driver. This can be fixed by, in your case, adding this _ "github.com/lib/pq" import spec.
If, even after adding that import, you're still seeing the same error, then that means that one of your dependencies is also trying to open a db connection without first importing the necessary driver.
Please note that while log.Fatal is a nice and clean way to stop your program it can sometimes be lacking, as you already know. You might want to consider using panic instead, its output is much more chaotic but, on the other hand, you'll get the line number and file name that caused the panic and eventually you'll learn to parse it quickly.
Beego ORM is somehow incomplete for now (for example it doesn't support foreign key constraints). So I've decided to use gorm with Beego. What is proper way of doing that? I've seen the sample code from gorm:
import (
"github.com/jinzhu/gorm"
_ "github.com/jinzhu/gorm/dialects/postgres"
)
func main() {
db, err := gorm.Open("postgres", "host=myhost user=gorm dbname=gorm sslmode=disable password=mypassword")
defer db.Close()
}
But do I have to connect to database each time in every controller function? Is there a way to use something like long polling connections?
gorm uses sql.DB type embedded in gorm.DB under the hood which
DB is a database handle representing a pool of zero or more
underlying connections. It's safe for concurrent use by multiple
goroutines. The sql package creates and frees connections
automatically; it also maintains a free pool of idle connections.
So you can use obtained DB globally in your code,
if you want level of isolation in request handling use transaction
tr:=db.Begin()
So, as #Uvelichitel pointed out, your option is to define your db connection at the global level and to use it from a desired place (probably main function to open a connection and model layer to query for results).
So you could basically have a file containing your db connection logics:
// appname/conn.go
package db
import (
"github.com/jinzhu/gorm"
...
)
var (
// this one gonna contain an open connection
// make sure to call Connect() before using it
Conn *gorm.DB
)
func Connect(dbConnString string) (*gorm.DB, error) {
db, err := gorm.Open("postgres", dbConnString)
Conn = db
return db, err
}
After you call db.Connect from your main.go you are free to use an opened connection db.Conn from anywhere of your application (just make sure you're importing this package to the places of use).
import "appname/db"
func main() {
conn, _ := db.Connect("host=localhost user=postgres ...")
// db.Conn is initialized and ready for usage anywhere else
The same result could be achieved within a single main.go file, moving global variable declaration and a connection logics straight there.
I am trying to group my database code into a sub-package, which would sort of encapsulate my database connection and all my prepared statements.
I can create the database connection and the statements in the package's init function, but I need to close them at some point.
In a program, where these variables are initialized in the code that uses them I would use defer db.Close(), etc, but how to go about this here?
I would prefer not to expose the package's global variables (connection and statements) so that the caller could access them to close them. Is there a way to do it in a more elegant manner?
I suspect that I might have a wrong paradigm in mind, and I am trying to create objects (this is essentially what it is here) in a language that does not have them. If so then I would appreciate any help with how to do it go-way.
package database
import (
"database/sql"
_ "github.com/lib/pq"
)
var db *sql.DB
var stmtSelectUser *sql.Stmt
func GetUser(email string) string {
var name string
stmtSelectUser.QueryRow(email).Scan(&name)
return name
}
func init() {
var e error;
db, e = sql.Open("postgres", "host=localhost dbname=pictocat sslmode=disable")
stmtSelectUser, e = db.Prepare("select * from users where email='$1'")
}
It's not really clear from your example code what your problem is, but in general:
Either the function running a statement needs to finish the statement and Close() it or you need to add a function to your package so the caller can close it; it's as simple as that.
You don't need to expose all of the database stuff in your package, just a function that in turns calls Close().
sql.Open() returns a variable of type *sql.DB
I have a function that calls 10 other functions that all need to make database calls
Is it more correct/efficient to:
Send the *sql.DB pointer to every function, or
Create a new *sql.DB object in each function
Meaning
func DoLotsOfThings() {
db, _ := sql.Open()
defer db.Close()
DoTask1(db)
DoTask2(db)
}
or
func DoLotsOfThings() {
DoTask1()
DoTask2()
}
func DoTask1() {
db, _ := sql.Open()
defer db.Close()
}
func DoTask1() {
db, _ := sql.Open()
defer db.Close()
}
The reason why I'm asking is because I am currently sending the pointer to each function and my driver seems to break. I'm using http://code.google.com/p/odbc , which leads me to believe each function should have its own, and that I can rely on the driver's internals.
EDIT
RE driver breakage, it only happens under high traffic environments. And it only happens after say, ten minutes or so of time. Which leads me to believe that there is some sort of memory leak that makes using the driver stop working. However I defer db.Close() for every instance of *sql.DB, so I don't know what else I can do to resolve this issue.
andybalholm says the connection pooling is handled internally, which seems to be accurate, because it only breaks after I attempt to execute something, not when I invoke sql.Open()
If I leave my Go app running, it will not be able to execute any sort of SQL queries, but if I attempt to run other Go tests separately connecting to MSSQL and running queries, it works.
Declare a var db *sql.DB globally, and then reuse it across your code. Here is an example (simplified):
var db *sql.DB
func DoLotsOfThings() {
DoTask1(db)
DoTask2(db)
}
func main() {
db, _ = sql.Open() # or whatever you use
defer db.Close()
DoLotsOfThings()
}
Declaring *sql.DB globally also have some additional benefits such as SetMaxIdleConns (regulating connection pool size) or preparing SQL statements across your application.
You shouldn't need to open database connections all over the place. The database/sql package does connection pooling internally, opening and closing connections as needed, while providing the illusion of a single connection that can be used concurrently.
Probably you need to look elsewhere for the cause of your driver breakage. Some more details about that would make it easier for people to figure out what is going on.