Beego ORM is somehow incomplete for now (for example it doesn't support foreign key constraints). So I've decided to use gorm with Beego. What is proper way of doing that? I've seen the sample code from gorm:
import (
"github.com/jinzhu/gorm"
_ "github.com/jinzhu/gorm/dialects/postgres"
)
func main() {
db, err := gorm.Open("postgres", "host=myhost user=gorm dbname=gorm sslmode=disable password=mypassword")
defer db.Close()
}
But do I have to connect to database each time in every controller function? Is there a way to use something like long polling connections?
gorm uses sql.DB type embedded in gorm.DB under the hood which
DB is a database handle representing a pool of zero or more
underlying connections. It's safe for concurrent use by multiple
goroutines. The sql package creates and frees connections
automatically; it also maintains a free pool of idle connections.
So you can use obtained DB globally in your code,
if you want level of isolation in request handling use transaction
tr:=db.Begin()
So, as #Uvelichitel pointed out, your option is to define your db connection at the global level and to use it from a desired place (probably main function to open a connection and model layer to query for results).
So you could basically have a file containing your db connection logics:
// appname/conn.go
package db
import (
"github.com/jinzhu/gorm"
...
)
var (
// this one gonna contain an open connection
// make sure to call Connect() before using it
Conn *gorm.DB
)
func Connect(dbConnString string) (*gorm.DB, error) {
db, err := gorm.Open("postgres", dbConnString)
Conn = db
return db, err
}
After you call db.Connect from your main.go you are free to use an opened connection db.Conn from anywhere of your application (just make sure you're importing this package to the places of use).
import "appname/db"
func main() {
conn, _ := db.Connect("host=localhost user=postgres ...")
// db.Conn is initialized and ready for usage anywhere else
The same result could be achieved within a single main.go file, moving global variable declaration and a connection logics straight there.
Related
Now I have upgraded my gorm package to the new version which is "gorm.io/gorm" but I am using package (github.com/qor/admin) that use the old version (github.com/jinzhu/gorm) of the package.
I need to pass gorm.DB(new version) value to a function of the package "github.com/qor/admin" that take gorm.DB(old version) as a parameter
package main
import (
adminPkg "github.com/qor/admin"
database "github.com/youssefsiam38/myfolder/db"
)
func main() {
db, err := database.Connection() // retrun db of type *gorm.io/gorm.DB
if err != nil {
panic(err)
}
admin := adminPkg.New(&adminPkg.AdminConfig{DB: db})
}
The err
vet: ./main.go:14:50: cannot use db (variable of type *gorm.DB) as *gorm.DB value in struct literal
You can't. Those two objects are not related, even though the name and implementations seem to indicate otherwise.
The github.com/qor/admin library has an issue open for that, so I'd stay tuned and/or contribute to migrate to the new version of gorm (and maybe rollback the lib upgrade if github.com/qor/admin is critical for your operations :)
It's good to note that if those libs were using interfaces, this could be fixable by third-parties. Stay in school kids, and use interfaces.
I'm using goose to manage my database migrations but I need to write SQL sentences directly in the migrations file. There is a way to generate the SQL directly from the GORM model?
Unfortunately using the gorm.Session{DryRun: true} option doesn't make the migration SQL statement/s available to the caller as it does with normal queries.
The only way I can see right now would be to capture the SQL that is run by the migration when it's being logged by reimplementing the gorm.io/gorm/logger.Interface interface. Specifically, the Trace method.
type Interface interface {
LogMode(LogLevel) Interface
Info(context.Context, string, ...interface{})
Warn(context.Context, string, ...interface{})
Error(context.Context, string, ...interface{})
Trace(ctx context.Context, begin time.Time, fc func() (string, int64), err error)
}
Inside Trace you can call that fc function argument to get the SQL and RowsAffected, which you can do whatever you want with.
For example:
import (
"time"
"context"
"gorm.io/gorm/logger"
)
type RecorderLogger struct {
logger.Interface
Statements []string
}
func (r *RecorderLogger) Trace(ctx context.Context, begin time.Time, fc func() (string, int64), err error) {
sql, _ := fc()
r.Statements = append(r.Statements, sql)
}
Now use it as:
recorder := RecorderLogger{logger.Default.LogMode(logger.Info)}
session := db.Session(&gorm.Session{
Logger: &recorder
})
session.AutoMigrate(&Model{}, ...)
// or
session.Migrator().CreateTable(&Model{}, ...) // or any method therein
// now recorder.Statements contains the statements run during migration
This is very hacky, and you may run into problems because AutoMigrate modifies the current state of the database and migrates it up to what your model requires (up to a point) and for that to work your current database must reflect the current state of your production database (or whatever database your hope to migrate). So, you could build that tool that helps you get the migration script started if you're careful, but to properly gain the advantages of a migration system like goose you'll need to get your hands dirty with the SQL :)
you can using this lib: https://github.com/sunary/sqlize
It's allowed you create sql from models, also support migration by differ between models and existing sql.
I personally would use the migration functionality that is available inside Gorm, but for your case we can do the following.
Firstly there is a feature in Gorm called Dry Run and you can use this to see the SQL statements that get executed when performing queries. Unfortunately I can't see that it is possible when using migrations. So what I suggest is to use github.com/DATA-DOG/go-sqlmock
I would usually use this for testing purposes, but you could use it temporarily to get the SQL needed for your separate migrations.
package main
import (
"database/sql"
"time"
"github.com/DATA-DOG/go-sqlmock"
"gorm.io/driver/mysql"
"gorm.io/gorm"
)
type Model struct {
ID uint64 `gorm:"primaryKey"`
Name string `gorm:"index"`
Description string
CreatedAt time.Time
LastLogin sql.NullTime
}
func main() {
sqlDB, _, err := sqlmock.New()
if err != nil {
panic(err)
}
gormDB, err := gorm.Open(mysql.New(mysql.Config{
Conn: sqlDB,
SkipInitializeWithVersion: true,
}), &gorm.Config{})
if err != nil {
panic(err)
}
defer sqlDB.Close()
gormDB.AutoMigrate(&Model{})
}
This will give you a result like this
all expectations were already fulfilled, call to ExecQuery 'CREATE TABLE `models` (`id` bigint unsigned AUTO_INCREMENT,`name` varchar(191),`description` longtext,`created_at` datetime(3) NULL,`last_login` datetime(3) NULL,PRIMARY KEY (`id`),INDEX idx_models_name (`name`))' with args [] was not expected
[0.003ms] [rows:0] CREATE TABLE `models` (`id` bigint unsigned AUTO_INCREMENT,`name` varchar(191),`description` longtext,`created_at` datetime(3) NULL,`last_login` datetime(3) NULL,PRIMARY KEY (`id`),INDEX idx_models_name (`name`))
which contains the SQL statement required. This feels incredibly hacky but will give you the result you need
So I'm just getting to grips with Golang. I'm writing an application for funsies just to understand stuff and get my head around it.
I have a whole bunch of functions that will interact with a DB where I pass in *SQL.DB for the function to use. I can test those easily enough using a mocked interface from sqlmock.
No problem there.
I'm now writing the Initialisation function for the application which will initiate the DB connection which will be attached to a struct and from there passed into utility functions.
However, I am struggling to find a way to easily test that connection without having the hassle of setting up an actual database.
So I guessing that I have probably either badly structured my app, or I've missed something, potentially pretty obvious.
So here is some example code to illustrate my predicament.
util.go
package main
import (
"log"
"database/sql"
"github.com/go-sql-driver/mysql"
)
func DoDBStuff(db *sql.DB) {
rows, err := db.Query("SELECT column1, column2 FROM example")
if err != nil {
log.Error(err)
}
// do stuff with rows
}
util_test.go
package main
import (
"testing"
"github.com/DATA-DOG/go-sqlmock"
)
func TestDoDBStuff(t *testing.T) {
db, mock, err := sqlmock.New()
if err != nil {
t.Fatalf("An error '%s' was not expected when opening a stub database connection", err)
}
defer db.Close()
rows := sqlmock.NewRows([]string{"col1", "col2"})
rows.AddRow("val1", "val2")
rows.AddRow("val3", "val4")
mock.ExpectQuery("^SELECT column1, column2 from example$").WillReturnRows(rows)
DoDBStuff(db)
if err := mock.ExpectationsWereMet(); err != nil {
t.Errorf("there were unfulfilled expectations: %s", err)
}
}
That all works fine, I can test my DB queries.
However Now I want to test Initialising the App.
package main
import (
"database/sql"
"github.com/go-sql-driver/mysql"
)
type App {
DB *sql.DB
// some other data structures
}
func (a *App) InitApp(connectionString string) {
a.DB = sql.Open("mysql", connectionString)
// other init stuff
}
But as I can't pass in the SQL I don't think it can be mocked, certainly not easily. So I'm struggling a bit on how to move forward.
I am intending for this to sit behind a rest API, so on startup, the app will need to initialize before being able to process any requests.
Ideally, I'd like to be able to test the REST interface without having to set up a database, delay testing with real data until I can feed the code into a Dev environment.
So really I want to know:
Is what I'm intending possible?
Is there a better approach?
If not what am I missing? Poor test design or poor code set up?
Edit:
Folling #peter's comment I just want to clarify.
I want to test the functionality of the InitDB() function but with the sql.Open call I would need to have a Database for it to connect to, If I don't then I the call would fail and I could not effectively test the function.
There is Dockertest which creates a Docker container running whatever you want (i.e. MySQL), and you can test against that. It's designed for being able to do proper integration testing, so it should be able to do what you want (you wont need sqlmock anymore too).
I am trying to group my database code into a sub-package, which would sort of encapsulate my database connection and all my prepared statements.
I can create the database connection and the statements in the package's init function, but I need to close them at some point.
In a program, where these variables are initialized in the code that uses them I would use defer db.Close(), etc, but how to go about this here?
I would prefer not to expose the package's global variables (connection and statements) so that the caller could access them to close them. Is there a way to do it in a more elegant manner?
I suspect that I might have a wrong paradigm in mind, and I am trying to create objects (this is essentially what it is here) in a language that does not have them. If so then I would appreciate any help with how to do it go-way.
package database
import (
"database/sql"
_ "github.com/lib/pq"
)
var db *sql.DB
var stmtSelectUser *sql.Stmt
func GetUser(email string) string {
var name string
stmtSelectUser.QueryRow(email).Scan(&name)
return name
}
func init() {
var e error;
db, e = sql.Open("postgres", "host=localhost dbname=pictocat sslmode=disable")
stmtSelectUser, e = db.Prepare("select * from users where email='$1'")
}
It's not really clear from your example code what your problem is, but in general:
Either the function running a statement needs to finish the statement and Close() it or you need to add a function to your package so the caller can close it; it's as simple as that.
You don't need to expose all of the database stuff in your package, just a function that in turns calls Close().
sql.Open() returns a variable of type *sql.DB
I have a function that calls 10 other functions that all need to make database calls
Is it more correct/efficient to:
Send the *sql.DB pointer to every function, or
Create a new *sql.DB object in each function
Meaning
func DoLotsOfThings() {
db, _ := sql.Open()
defer db.Close()
DoTask1(db)
DoTask2(db)
}
or
func DoLotsOfThings() {
DoTask1()
DoTask2()
}
func DoTask1() {
db, _ := sql.Open()
defer db.Close()
}
func DoTask1() {
db, _ := sql.Open()
defer db.Close()
}
The reason why I'm asking is because I am currently sending the pointer to each function and my driver seems to break. I'm using http://code.google.com/p/odbc , which leads me to believe each function should have its own, and that I can rely on the driver's internals.
EDIT
RE driver breakage, it only happens under high traffic environments. And it only happens after say, ten minutes or so of time. Which leads me to believe that there is some sort of memory leak that makes using the driver stop working. However I defer db.Close() for every instance of *sql.DB, so I don't know what else I can do to resolve this issue.
andybalholm says the connection pooling is handled internally, which seems to be accurate, because it only breaks after I attempt to execute something, not when I invoke sql.Open()
If I leave my Go app running, it will not be able to execute any sort of SQL queries, but if I attempt to run other Go tests separately connecting to MSSQL and running queries, it works.
Declare a var db *sql.DB globally, and then reuse it across your code. Here is an example (simplified):
var db *sql.DB
func DoLotsOfThings() {
DoTask1(db)
DoTask2(db)
}
func main() {
db, _ = sql.Open() # or whatever you use
defer db.Close()
DoLotsOfThings()
}
Declaring *sql.DB globally also have some additional benefits such as SetMaxIdleConns (regulating connection pool size) or preparing SQL statements across your application.
You shouldn't need to open database connections all over the place. The database/sql package does connection pooling internally, opening and closing connections as needed, while providing the illusion of a single connection that can be used concurrently.
Probably you need to look elsewhere for the cause of your driver breakage. Some more details about that would make it easier for people to figure out what is going on.