I've started using Google Cloud Datastore in one of the project in the company I currently work in.
https://godoc.org/cloud.google.com/go/datastore
In the provided example, they use a context and pass it to the connection instance
ctx := context.Background()
dsClient, err := datastore.NewClient(ctx, "my-project")
Through the documentation you will see that they pass a context to all the functions that makes operations on the database, I am not sure if they are passing the same pointer or create a new pointer for each operation.
The current setup that I have is a global variable for the context in a package called "store" which I keep all the structs functions that communicate with the db, and I use that global variable each time. I don't know what is the effect of that, I am not sure why the context is used, Should I get reference of context.Background() each time I make operations on the database ?
context.Background is the global context--so no need for you global variable. Most of the time you'll want to use a child of that context, with a cancel or a timeout.
ctx, cancel := context.WithCancel(context.Background)
//or
ctx, cancel := context.WithTimeout(context.Background, time.Second * 30)
Then you can use the cancel function to close down your application nicely, or cancel and retry if a request is hanging. If you never plan on canceling or timing out operations, then using context.Background is fine.
Also context.Context is an interface so it's always passed by reference, so all uses of a certain instance point to the same context.
Related
I'm currently working on migrating our code from global sign package to go mongo-driver, not sure where should I use context.TODO() and context.Background(), it’s really confusing, I know both it returns non-nil empty, so should I use context.Background() in the main function & init() functions? And use context.TODO() in other places? Can anyone help with this?
Trying to check to see which param should I use context.TODO() or context.Background().
Check their documentation:
context.Background():
func Background() Context
Background returns a non-nil, empty Context. It is never canceled, has no values, and has no deadline. It is typically used by the main function, initialization, and tests, and as the top-level Context for incoming requests.
context.TODO():
func TODO() Context
TODO returns a non-nil, empty Context. Code should use context.TODO when it's unclear which Context to use or it is not yet available (because the surrounding function has not yet been extended to accept a Context parameter).
According to the doc, when you need a context but you don't have one (yet) and don't know what to use, use context.TODO(). This is exactly your case, using context.TODO() properly documents this. Also, static analysis tools and IDEs may give support discovering these contexts and give you warning later to address the issue.
Also note that if you do have a context when you have to use the mongo driver, then consider using that context, or deriving a new one from it.
For example, if you are writing an HTTP handler where you need to query some documents, the HTTP request http.Request already has a context which you can access using Request.Context(). This is a prime candidate to use when you call other API functions that require a context.
Why, you may ask? Because the request's context is cancelled when the HTTP client abandons / aborts the request, which means whatever you do, the client will not receive it, so for example if the HTTP handler just serves information, you may as well abort generating it, saving some resources. So if the request context is cancelled while you're executing a MongoDB query, you may as well cancel the query too, because you won't need (won't process) the result anyway. Cancelling the context passed to the query execution (Collection.Find() for example) is the way to tell the MongoDB server to cancel the query if possible (because you won't need the result).
So using the request context in this case can save you some CPU time and memory both on the HTTP server and on the MongoDB server as well.
Another example: let's say you have to produce the HTTP response within 10 seconds (may be a platform limitation), during which you have to execute a MongoDB operation, and you have to post process the results which takes 4 seconds. In this scenario if the MongoDB operation takes more than 6 seconds, you would not be able to complete the post processing to fit in the 10-second limit. So you might as well cancel the MongoDB operation if it doesn't complete in 6 seconds.
An easy way to solve this is to derive a context with 6 seconds timeout:
ctx, cancel := context.WithTimeout(r.Context(), 6 * time.Second)
defer cancel()
// ctx automatically times out after 6 seconds
curs, err := c.Find(ctx, bson.M{"some": "filter"})
In this example ctx will time out after 6 seconds, so if the c.Find() operation doesn't get finished by that, the passed ctx will signal that it can be cancelled (in which case an error will be returned). Since ctx is derived from r.Context(), if the request's context get's cancelled earlier (e.g. the HTTP client aborts), ctx will also be cancelled.
Now let's say you're not writing an HTTP handler but some other code (standalone app, background worker etc.), where you may not have a context, but you have your standards that you don't want to wait more than 30 seconds for a query. This is a prime example where you may derive a context from context.Background() using a 30-sec timeout:
ctx, cancel := context.WithTimeout(context.Background(), 30 * time.Second)
defer cancel()
// ctx automatically times out after 30 seconds
curs, err := c.Find(ctx, bson.M{"some": "filter"})
If the query finishes within 30 seconds, all good, you may use the results. If not, the context gets cancelled after 30 sec, and so c.Find() will also (likely) return an error.
context.TODO:
Code should use context.TODO when it's unclear which Context to use or it is not yet available (because the surrounding function has not yet been extended to accept a Context parameter).
context.Background:
Background returns a non-nil, empty Context. It is never canceled, has no values, and has no deadline. It is typically used by the main function, initialization, and tests, and as the top-level Context for incoming requests.
TODO is a placeholder to make it clear that the code should be updated with a more appropriate context in the future, but none is available yet. Background is used when such an indication isn't necessary and you simply need an empty root context with no values or cancellation. The values they return are identical; the only difference is semantic, in that one might scan code for TODO contexts (or use a linter to do so) as places that need to be updated/addressed.
I'm writing a custom terraform provider. I need to wrap the calls to the backed API in a "session", opening it before any calls are made and then closing it once all the terraform calls have completed.
Opening the session is straightforward, I can do that in the ConfigureContextFunc of the schema.Provider. Is there a way to set up a callback (or something) at the end of the application so I can close/"finalize" the session? I can imagine something specific to my resources, but that seems hacky. In my dream world I'd also be able to fail the apply if the close had an error.
Absent a nice finalize call is there a way to access the plan that I could use to determine that the current call is the last needed for the apply?
Update: I thought I could use a StopContext:
stopCtx, ok := schema.StopContext(ctx)
...
go func(ctx context.Context) {
// Wait for stop context cancellation
<-stopCtx.Done()
...
}
However, this is both deprecated and seems to only get called when stopping due to some outside trigger, like SIGINT, and not a regular exit (at least that's what I've been seeing).
After a fair amount of floundering I have what I believe to be a reasonable solution, and it even matches #Ben Hoyt's comment. We can defer and teardown in main.
func main() {
var prov *schema.Provider
defer func() {
xyz.PrividerTeardown(prov)
}()
plugin.Serve(&plugin.ServeOpts{
ProviderFunc: func() *schema.Provider {
prov = xyz.Provider()
return prov
},
})
}
I'll note that my comment on the question about not being around when the provider is made is incorrect. The provider is made in a main function in the provider code. In my (flimsy) defense I used the example scaffolding so that code came pre-written.
One thing that threw me was the docs for plugin.Serve
Serve serves a plugin. This function never returns and should be the final function called in the main function of the plugin.
It turns out that when the terraform action is done and the plugin is no longer needed, Serve does return and allows our defer to run. I did need to keep track of success or failure of all the calls that were made while Serve was active to know the status in my ProviderTeardown, but it is working.
I have been using GORM for my application based on AWS lambda. I used gorm.Open() for every Handler function,
db, err := gorm.Open(mysql.Open(dsn), &gorm.Config{
Logger: logger.Default.LogMode(logger.Info),
})
so can someone help me confirm that does gorm.Open(...) automatically close the connection or not? Or I must use generic database interface bellow?
// Get generic database object sql.DB to use its functions
sqlDB, err := db.DB()
// Ping
sqlDB.Ping()
// Close
sqlDB.Close()
// Returns database statistics
sqlDB.Stats()
A gorm.DB object is intended to be reused, like a sql.DB handle. You rarely have to explicitly close these objects. Just create it once and reuse it.
gorm.DB contains a sql.DB which uses a connection pool to manage the connections. If it is closed, it will stop accepting new queries, wait for running queries to finish and close all connections.
Will context.Done() unblock when a context variable goes out of scope and cancel is not explicitly called?
Let's say I have the following code:
func DoStuff() {
ctx, _ := context.WithCancel(context.Background())
go DoWork(ctx)
return
}
Will ctx.Done() unblock in DoWork after the return in DoStuff()?
I found this thread, https://groups.google.com/forum/#!topic/golang-nuts/BbvTlaQwhjw, where the person asking how to use Context.Done() claims that context.Done() will unblock when the context variable leaves scope but no one validated this, and I didn't see anything in the docs.
No, it doesn't cancel automatically when the context leaves scope. Typically one calls defer cancel() (using the callback from ctx.WithCancel()) oneself to make sure that the context is cancelled.
https://blog.golang.org/context provides a good overview of how to use contexts correctly (including the defer pattern above). Also, the source code https://golang.org/src/context/context.go is quite readable and you can see there's no magic that would provide automatic cancellation.
"Unblocking" is not the clearest terminology. Done() returns a channel (or nil) that will receive a struct{} and/or close when the context is "cancelled". What exactly that chan is, or when it is sent on, is up to the individual implementation. It may be sent/closed at some fixed time as with WithDeadline, or manually done as with WithCancel.
The key though, is that this is never "automatic" or guaranteed to happen. If you make a context with WithCancel and read from the Done() channel, that read will block indefinitely until the Cancel() method is called. If that never happens, then you have a wasted goroutine and your application's memory will increase each time you do it.
Once the context is completely out of scope (no executing goroutine is listening to it or has a reference to the parent context), it will get garbage collected and everything will go away.
EDIT: After reading the source though, it looks like WithCancel and friends spawn goroutines to propigate the cancellation. Therefore you must make sure Cancel gets called at some point to avoid goroutine leaks.
As the title says I don't know if having multiple sql.Open statements is a good or bad thing or what or if I should have a file with just an init that is something like:
var db *sql.DB
func init() {
var err error
db, err = sql.Open
}
just wondering what the best practice would be. Thanks!
You should at least check the error.
As mentioned in "Connecting to a database":
Note that Open does not directly open a database connection: this is deferred until a query is made. To verify that a connection can be made before making a query, use the Ping function:
if err := db.Ping(); err != nil {
log.Fatal(err)
}
After use, the database is closed using Close.
If possible, limit the number of opened connection to a database to a minimum.
See "Go/Golang sql.DB reuse in functions":
You shouldn't need to open database connections all over the place.
The database/sql package does connection pooling internally, opening and closing connections as needed, while providing the illusion of a single connection that can be used concurrently.
As elithrar points out in the comment, database.sql/#Open does mention:
The returned DB is safe for concurrent use by multiple goroutines and maintains its own pool of idle connections.
Thus, the Open function should be called just once.
It is rarely necessary to close a DB.
As mentioned here
Declaring *sql.DB globally also have some additional benefits such as SetMaxIdleConns (regulating connection pool size) or preparing SQL statements across your application.
You can use a function init, which will run even if you don't have a main():
var db *sql.DB
func init() {
db, err = sql.Open(DBparms....)
}
init() is always called, regardless if there's main or not, so if you import a package that has an init function, it will be executed.
You can have multiple init() functions per package, they will be executed in the order they show up in the code (after all variables are initialized of course).