I am currently using the redigo library for my project where i create a redis pool
I use defer to release the redis connection every time i get one from the pool
c := redisPool.Get()
defer c.Close()
But it will block forever in this scenario if MaxActive has been set
func function1() {
c := redisPool.Get()
defer c.Close()
function2()
...
}
func function2() {
c := redisPool.Get()
defer c.Close()
...
}
should i use only one redis connection in one goroutine?
You have a few options here.
You can Close() when you are done, returning the connection to the pool and then calling function2. Upside: works, not too complex. Downside: management of returning the connection in the case of multiple exit points from the function.
You can change function2 to take a redis.Conn argument that it uses and just pass that connection off. Upside: defer still works for function1. Downside: you need a connection to call function2 and need to do connection management from the calling site. In your example that is easy enough.
Make sure you have at least N*2 max connections, where N is the max number of concurrent goroutines that will be running. Upside: Your code stays as-is without changes. Downside: limited in the number of concurrent calls to function1 you can make.
You can use following approach to make sure application won't lock/ break.
set wait: true in the pool configuration.
// If Wait is true and the pool is at the MaxActive limit, then Get()
waits // for a connection to be returned to the pool before returning.
Confirm that the server's maxclient limit is larger than MaxActive The default maxclient is 10k.
Most applications can keep connection use low by avoiding long or blocking operations (other than calls to Redis) between the call to Get() and the call to Close().
Hope this helps.
Related
I have been using GORM for my application based on AWS lambda. I used gorm.Open() for every Handler function,
db, err := gorm.Open(mysql.Open(dsn), &gorm.Config{
Logger: logger.Default.LogMode(logger.Info),
})
so can someone help me confirm that does gorm.Open(...) automatically close the connection or not? Or I must use generic database interface bellow?
// Get generic database object sql.DB to use its functions
sqlDB, err := db.DB()
// Ping
sqlDB.Ping()
// Close
sqlDB.Close()
// Returns database statistics
sqlDB.Stats()
A gorm.DB object is intended to be reused, like a sql.DB handle. You rarely have to explicitly close these objects. Just create it once and reuse it.
gorm.DB contains a sql.DB which uses a connection pool to manage the connections. If it is closed, it will stop accepting new queries, wait for running queries to finish and close all connections.
I'm writing a TCP server which simultaneously accepts multiple connections from mobile devices and some WiFi devices (IOT). The connections needs to be maintained once established, with the 30 seconds timeout if there is no heartbeat received. So it is something like the following:
// clientsMap map[string] conn
func someFunction() {
conn, err := s.listener.Accept()
// I store the conn in clientsMap
// so I can access it, for brevity not
// shown here, then:
go serve(connn)
}
func serve(conn net.Conn) {
timeoutDuration := 30 * time.Second
conn.SetReadDeadline(time.Now().Add(timeoutDuration))
for {
msgBuffer := make([]byte, 2048)
msgBufferLen, err := conn.Read(msgBuffer)
// do something with the stuff
}
}
So there is one goroutine for each client. And each client, once connected to the server, is pending on the read. The server then processes the stuff read.
The problem is that I sometimes need to read things off one client, and then pass data to another (Between a mobile device and a WiFi device). I have stored the connections in clientsMap. So I can always access that. But since each client is handled by one goroutine, shall I be passing the data from one client to another by using a channel? But if the goroutine is blocked waiting for a pending read, how do I make it also wait for data from a channel? Or shall I just obtain the connection for the other party from the clientsMap and write to it?
The documentation for net.Conn clearly states:
Multiple goroutines may invoke methods on a Conn simultaneously.
So yes, it is okay to simply Write to the connections. You should take care to issue a single Write call per message you want to send. If you call Write more than once you risk interleaving messages from different mobile devices. This implies calling Write directly and not via some other API (in other words don't wrap the connection). For instance, the following would not be safe:
json.NewEncoder(conn).Encode(myValue) // use json.Marshal(myValue) instead
io.Copy(conn, src) // use io.ReadAll(src) instead
I have studied the Godoc of the gorilla/websocket package.
In the Godoc it is clearly stated that
Concurrency
Connections support one concurrent reader and one concurrent writer.
Applications are responsible for ensuring that no more than one goroutine calls the write methods (NextWriter, SetWriteDeadline, WriteMessage, WriteJSON, EnableWriteCompression, SetCompressionLevel) concurrently and that no more than one goroutine calls the read methods (NextReader, SetReadDeadline, ReadMessage, ReadJSON, SetPongHandler, SetPingHandler) concurrently.
The Close and WriteControl methods can be called concurrently with all other
methods.
However, in one of the example provided by the package
func (c *Conn) readPump() {
defer func() {
hub.unregister <- c
c.ws.Close()
}()
c.ws.SetReadLimit(maxMessageSize)
c.ws.SetReadDeadline(time.Now().Add(pongWait))
c.ws.SetPongHandler(func(string) error {
c.ws.SetReadDeadline(time.Now().Add(pongWait)); return nil
})
for {
_, message, err := c.ws.ReadMessage()
if err != nil {
if websocket.IsUnexpectedCloseError(err, websocket.CloseGoingAway) {
log.Printf("error: %v", err)
}
break
}
message = bytes.TrimSpace(bytes.Replace(message, newline, space, -1))
hub.broadcast <- message
}
}
Source: https://github.com/gorilla/websocket/blob/a68708917c6a4f06314ab4e52493cc61359c9d42/examples/chat/conn.go#L50
This line
c.ws.SetPongHandler(func(string) error {
c.ws.SetReadDeadline(time.Now().Add(pongWait)); return nil
})
and this line
_, message, err := c.ws.ReadMessage()
seems to be not synchronized because the first line is a callback function so it should be invoked in a Goroutine created in the package and the second line is executing in the Goroutine that invoke serveWs
More importantly, how should I ensure that no more than one goroutine calls the SetReadDeadline, ReadMessage, SetPongHandler, SetPingHandler concurrently?
I tries to use a Mutex lock and lock it whenever I call the above functions, and unlock it afterwards, but quickly I realize a problem. It is usual (also in the example) that ReadMessage is being called in a for-loop. But if the Mutext is locked before the ReadMessage, then no other Read-functions can acquire the lock and execute until next message is received
Is there any better way in handling this concurrency issue? Thanks in advance.
The best way to ensure that there are no concurrent calls to the read methods is to execute all of the read methods from a single goroutine.
All of the Gorilla websocket examples use this approach including the example pasted in the question. In the example, all calls to the read methods are from the readPump method. The readPump method is called once for a connection on a single goroutine. It follows that the connection read methods are not called concurrently.
The section of the documentation on control messages says that the application must read the connection to process control messages. Based on this and Gorilla's own examples, I think it's safe to assume that the ping, pong and close handlers will be called from the application's reading goroutine as it is in the current implementation. It would be nice if the documentation could be more explicit about this. Maybe file an issue?
As the title says I don't know if having multiple sql.Open statements is a good or bad thing or what or if I should have a file with just an init that is something like:
var db *sql.DB
func init() {
var err error
db, err = sql.Open
}
just wondering what the best practice would be. Thanks!
You should at least check the error.
As mentioned in "Connecting to a database":
Note that Open does not directly open a database connection: this is deferred until a query is made. To verify that a connection can be made before making a query, use the Ping function:
if err := db.Ping(); err != nil {
log.Fatal(err)
}
After use, the database is closed using Close.
If possible, limit the number of opened connection to a database to a minimum.
See "Go/Golang sql.DB reuse in functions":
You shouldn't need to open database connections all over the place.
The database/sql package does connection pooling internally, opening and closing connections as needed, while providing the illusion of a single connection that can be used concurrently.
As elithrar points out in the comment, database.sql/#Open does mention:
The returned DB is safe for concurrent use by multiple goroutines and maintains its own pool of idle connections.
Thus, the Open function should be called just once.
It is rarely necessary to close a DB.
As mentioned here
Declaring *sql.DB globally also have some additional benefits such as SetMaxIdleConns (regulating connection pool size) or preparing SQL statements across your application.
You can use a function init, which will run even if you don't have a main():
var db *sql.DB
func init() {
db, err = sql.Open(DBparms....)
}
init() is always called, regardless if there's main or not, so if you import a package that has an init function, it will be executed.
You can have multiple init() functions per package, they will be executed in the order they show up in the code (after all variables are initialized of course).
My case is: long running server with connection to Redis. This server wait for SIGTERM signal for terminating. What is the right way to guarantee to release connection after terminating of my application?
I know about defer - is really great, but not for persistent connection, because I do not want to open connection to Redis for each operation.
Thanks!
You would still use defer if you want to ensure some block of code executes before exit. The difference is in it's scope. The scope of your connection and defer statement should be the same. I have no idea what your app is but to provide a concrete example, you need to defer the connection close in the main of you command line app, not in the methods that read and write.
You said "because I do not want to open connection to Redis for each operation" but that only makes defer problematic if you defer the close in the scope of some method that does a single IO operation. If you instead do the defer in the scope above a single operation (where all operations occur) then it will do waht you want;
init connection
defer connectionClose
begin execution of code that does db IO
block here if above is async
program is exiting, my defer is called here
EDIT: As pointed out in the comments, the execution of deferred statements in not guaranteed. I just want to make it clear that you can defer the connection closing at the top level of application.