I have been using GORM for my application based on AWS lambda. I used gorm.Open() for every Handler function,
db, err := gorm.Open(mysql.Open(dsn), &gorm.Config{
Logger: logger.Default.LogMode(logger.Info),
})
so can someone help me confirm that does gorm.Open(...) automatically close the connection or not? Or I must use generic database interface bellow?
// Get generic database object sql.DB to use its functions
sqlDB, err := db.DB()
// Ping
sqlDB.Ping()
// Close
sqlDB.Close()
// Returns database statistics
sqlDB.Stats()
A gorm.DB object is intended to be reused, like a sql.DB handle. You rarely have to explicitly close these objects. Just create it once and reuse it.
gorm.DB contains a sql.DB which uses a connection pool to manage the connections. If it is closed, it will stop accepting new queries, wait for running queries to finish and close all connections.
Related
I am using the gosnowflake 1.40 driver. I am seeing my sessions cycle after 2 queries as seen in the image below, less than 1 second apart.
Connection setup looks something like this:
dsn, err := sf.DSN(sfConfig)
if err != nil {
panic("cannot get snowflake session: " + err.Error())
}
DBSession, err = sql.Open("snowflake", dsn)
if err != nil {
panic("cannot get snowflake session: " + err.Error())
}
return DBSession, nil
I use the following query pattern inside a function:
result = dbSession.QueryRow(command)
This session cycling pattern is not ideal, as I'd like to be able to assume a role and run multiple commands. Can someone point me to what I can do to make the Snowflake sessions persist? I don't have this problem using the WebUI.
DB maintains a pool of connections. Each connection in the pool will have a unique session ID. From the documentation:
DB is a database handle representing a pool of zero or more underlying connections. It's safe for concurrent use by multiple goroutines.
The sql package creates and frees connections automatically; it also maintains a free pool of idle connections.
You have a couple options for bypassing the default behavior of cycling through the pool of connections:
Obtain a specific Conn instance
from the connection pool using
DB.Conn(). The documentation
specifically states:
Queries run on the same Conn will be run in the same database session.
Modify the connection pool parameters using
DB.SetMaxOpenConns().
I suspect that setting this to 1 will also obtain the desired behavior.
However, this introduces scalability/concurrency concerns that are
addressed by having a connection pool in the first place.
Note, I'm not familiar with the Snowflake driver in particular. There may be other options that the driver supports.
I'm writing a TCP server which simultaneously accepts multiple connections from mobile devices and some WiFi devices (IOT). The connections needs to be maintained once established, with the 30 seconds timeout if there is no heartbeat received. So it is something like the following:
// clientsMap map[string] conn
func someFunction() {
conn, err := s.listener.Accept()
// I store the conn in clientsMap
// so I can access it, for brevity not
// shown here, then:
go serve(connn)
}
func serve(conn net.Conn) {
timeoutDuration := 30 * time.Second
conn.SetReadDeadline(time.Now().Add(timeoutDuration))
for {
msgBuffer := make([]byte, 2048)
msgBufferLen, err := conn.Read(msgBuffer)
// do something with the stuff
}
}
So there is one goroutine for each client. And each client, once connected to the server, is pending on the read. The server then processes the stuff read.
The problem is that I sometimes need to read things off one client, and then pass data to another (Between a mobile device and a WiFi device). I have stored the connections in clientsMap. So I can always access that. But since each client is handled by one goroutine, shall I be passing the data from one client to another by using a channel? But if the goroutine is blocked waiting for a pending read, how do I make it also wait for data from a channel? Or shall I just obtain the connection for the other party from the clientsMap and write to it?
The documentation for net.Conn clearly states:
Multiple goroutines may invoke methods on a Conn simultaneously.
So yes, it is okay to simply Write to the connections. You should take care to issue a single Write call per message you want to send. If you call Write more than once you risk interleaving messages from different mobile devices. This implies calling Write directly and not via some other API (in other words don't wrap the connection). For instance, the following would not be safe:
json.NewEncoder(conn).Encode(myValue) // use json.Marshal(myValue) instead
io.Copy(conn, src) // use io.ReadAll(src) instead
I've started using Google Cloud Datastore in one of the project in the company I currently work in.
https://godoc.org/cloud.google.com/go/datastore
In the provided example, they use a context and pass it to the connection instance
ctx := context.Background()
dsClient, err := datastore.NewClient(ctx, "my-project")
Through the documentation you will see that they pass a context to all the functions that makes operations on the database, I am not sure if they are passing the same pointer or create a new pointer for each operation.
The current setup that I have is a global variable for the context in a package called "store" which I keep all the structs functions that communicate with the db, and I use that global variable each time. I don't know what is the effect of that, I am not sure why the context is used, Should I get reference of context.Background() each time I make operations on the database ?
context.Background is the global context--so no need for you global variable. Most of the time you'll want to use a child of that context, with a cancel or a timeout.
ctx, cancel := context.WithCancel(context.Background)
//or
ctx, cancel := context.WithTimeout(context.Background, time.Second * 30)
Then you can use the cancel function to close down your application nicely, or cancel and retry if a request is hanging. If you never plan on canceling or timing out operations, then using context.Background is fine.
Also context.Context is an interface so it's always passed by reference, so all uses of a certain instance point to the same context.
Hypothetically speaking, is it good practice to connect to a database for each request and close in when the request has completed?
I'm using mongodb with mgo for the database.
In my project, I would like to connect to a certain database by getting the database name from the request header (of course, this is combined with an authentication mechanism, e.g. JWT in my app). The flow goes something like:
User authentication:
POST to http://api.app.com/authenticate
// which checks the user in a "global" database,
// authenticates them and returns a signed JWT token
// The token is stored in bolt.db for the authentication mechanism
Some RESTful operations
POST to http://api.app.com/v1/blog/posts
// JWT middleware for each request to /v1* is set up
// `Client-Domain` in header is set to a database's name, e.g 'app-com'
// so we open a connection to that database and close when
// request finishes
So my questions are:
Is this feasible? - I've read about connection pools and reusing them but I haven't read much about them yet
Is there a better way of achieving the desired functionality?
How do I ensure the session is only closed when the request has completed?
The reason why I need to do this is because we have multiple vendors that have the same database collections with different entries with restricted access to their own databases.
Update / Solution
I ended up using Go's built in Context by Copying a session and using it anywhere I need to do any CRUD ops
Something like:
func main() {
...
// Configure connection and set in global var
model.DBSession, err = mgo.DialWithInfo(mongoDBDialInfo)
defer model.DBSession.Close()
...
n := negroni.Classic()
n.Use(negroni.HandlerFunc(Middleware))
...
}
func Middleware(res http.ResponseWriter, req *http.Request, next http.HandlerFunc) {
...
db := NewDataStore(clientDomain)
// db.Close() is an alias for ds.session.Close(), code for this function is not included in this post
// Im still experimenting with this, I need to make sure the session is only closed after a request has completed, currently it does not always do so
defer db.Close()
ctx := req.Context()
ctx = context.WithValue(ctx, auth.DataStore, db)
req = req.WithContext(ctx)
...
}
func NewDataStore(db string) *DataStore {
store := &DataStore{
db: DBSession.Copy().DB(db),
session: DBSession.Copy(),
}
return store
}
And then use it in a HandlerFunc, example /v1/system/users:
func getUsers(res http.ResponseWriter, req *http.Request) {
db := req.Context().Value(auth.DataStore).(*model.DataStore)
users := make([]SystemUser{}, 0)
// db.C() is an alias for ds.db.C(), code for this function is not included in this post
db.C("system_users").Find(nil).All(&users)
}
40% response time decrease over the original method I experimented with.
Hypothetically speaking is not a good practice because:
The database logic is scattered among several packages.
It's difficult to test
You can't apply DI (mainly it will be hard to maintain the code)
Replying to your questions:
Yes is feasible BUT you will not use the connection pool inside them go package (take a look to the code here if you want know more about Connection Pool)
A better way is to create a global variable that contains the database connection and close when the application is going to stop (and not close the connection every request)
How do I ensure the session is only closed when the request has complete<- you should checkout the answer fro your db query and then close the connection (but I don't recommend to close the connection after a request because you'll need to open again for another request and close again etc...)
As the title says I don't know if having multiple sql.Open statements is a good or bad thing or what or if I should have a file with just an init that is something like:
var db *sql.DB
func init() {
var err error
db, err = sql.Open
}
just wondering what the best practice would be. Thanks!
You should at least check the error.
As mentioned in "Connecting to a database":
Note that Open does not directly open a database connection: this is deferred until a query is made. To verify that a connection can be made before making a query, use the Ping function:
if err := db.Ping(); err != nil {
log.Fatal(err)
}
After use, the database is closed using Close.
If possible, limit the number of opened connection to a database to a minimum.
See "Go/Golang sql.DB reuse in functions":
You shouldn't need to open database connections all over the place.
The database/sql package does connection pooling internally, opening and closing connections as needed, while providing the illusion of a single connection that can be used concurrently.
As elithrar points out in the comment, database.sql/#Open does mention:
The returned DB is safe for concurrent use by multiple goroutines and maintains its own pool of idle connections.
Thus, the Open function should be called just once.
It is rarely necessary to close a DB.
As mentioned here
Declaring *sql.DB globally also have some additional benefits such as SetMaxIdleConns (regulating connection pool size) or preparing SQL statements across your application.
You can use a function init, which will run even if you don't have a main():
var db *sql.DB
func init() {
db, err = sql.Open(DBparms....)
}
init() is always called, regardless if there's main or not, so if you import a package that has an init function, it will be executed.
You can have multiple init() functions per package, they will be executed in the order they show up in the code (after all variables are initialized of course).