I am using the gosnowflake 1.40 driver. I am seeing my sessions cycle after 2 queries as seen in the image below, less than 1 second apart.
Connection setup looks something like this:
dsn, err := sf.DSN(sfConfig)
if err != nil {
panic("cannot get snowflake session: " + err.Error())
}
DBSession, err = sql.Open("snowflake", dsn)
if err != nil {
panic("cannot get snowflake session: " + err.Error())
}
return DBSession, nil
I use the following query pattern inside a function:
result = dbSession.QueryRow(command)
This session cycling pattern is not ideal, as I'd like to be able to assume a role and run multiple commands. Can someone point me to what I can do to make the Snowflake sessions persist? I don't have this problem using the WebUI.
DB maintains a pool of connections. Each connection in the pool will have a unique session ID. From the documentation:
DB is a database handle representing a pool of zero or more underlying connections. It's safe for concurrent use by multiple goroutines.
The sql package creates and frees connections automatically; it also maintains a free pool of idle connections.
You have a couple options for bypassing the default behavior of cycling through the pool of connections:
Obtain a specific Conn instance
from the connection pool using
DB.Conn(). The documentation
specifically states:
Queries run on the same Conn will be run in the same database session.
Modify the connection pool parameters using
DB.SetMaxOpenConns().
I suspect that setting this to 1 will also obtain the desired behavior.
However, this introduces scalability/concurrency concerns that are
addressed by having a connection pool in the first place.
Note, I'm not familiar with the Snowflake driver in particular. There may be other options that the driver supports.
Related
I have been using GORM for my application based on AWS lambda. I used gorm.Open() for every Handler function,
db, err := gorm.Open(mysql.Open(dsn), &gorm.Config{
Logger: logger.Default.LogMode(logger.Info),
})
so can someone help me confirm that does gorm.Open(...) automatically close the connection or not? Or I must use generic database interface bellow?
// Get generic database object sql.DB to use its functions
sqlDB, err := db.DB()
// Ping
sqlDB.Ping()
// Close
sqlDB.Close()
// Returns database statistics
sqlDB.Stats()
A gorm.DB object is intended to be reused, like a sql.DB handle. You rarely have to explicitly close these objects. Just create it once and reuse it.
gorm.DB contains a sql.DB which uses a connection pool to manage the connections. If it is closed, it will stop accepting new queries, wait for running queries to finish and close all connections.
I'm writing a TCP server which simultaneously accepts multiple connections from mobile devices and some WiFi devices (IOT). The connections needs to be maintained once established, with the 30 seconds timeout if there is no heartbeat received. So it is something like the following:
// clientsMap map[string] conn
func someFunction() {
conn, err := s.listener.Accept()
// I store the conn in clientsMap
// so I can access it, for brevity not
// shown here, then:
go serve(connn)
}
func serve(conn net.Conn) {
timeoutDuration := 30 * time.Second
conn.SetReadDeadline(time.Now().Add(timeoutDuration))
for {
msgBuffer := make([]byte, 2048)
msgBufferLen, err := conn.Read(msgBuffer)
// do something with the stuff
}
}
So there is one goroutine for each client. And each client, once connected to the server, is pending on the read. The server then processes the stuff read.
The problem is that I sometimes need to read things off one client, and then pass data to another (Between a mobile device and a WiFi device). I have stored the connections in clientsMap. So I can always access that. But since each client is handled by one goroutine, shall I be passing the data from one client to another by using a channel? But if the goroutine is blocked waiting for a pending read, how do I make it also wait for data from a channel? Or shall I just obtain the connection for the other party from the clientsMap and write to it?
The documentation for net.Conn clearly states:
Multiple goroutines may invoke methods on a Conn simultaneously.
So yes, it is okay to simply Write to the connections. You should take care to issue a single Write call per message you want to send. If you call Write more than once you risk interleaving messages from different mobile devices. This implies calling Write directly and not via some other API (in other words don't wrap the connection). For instance, the following would not be safe:
json.NewEncoder(conn).Encode(myValue) // use json.Marshal(myValue) instead
io.Copy(conn, src) // use io.ReadAll(src) instead
I have read online that the best practice when using Cassandra is to have 1 Cluster and 1 Session for the lifetime of your service.
My questions are:
In case our Cassandra server goes down, how do I make sure that my Cluster and/or Session will keep trying to reconnect until our Cassandra server gets back online?
Should only the Cluster attempt to reconnect, or only the Session, or both?
We are using Go and github.com/gocql/gocql for our service.
I have seen the following snippet in the gocql documentation, but it looks like it only has a limited number of retries:
cluster.ReconnectionPolicy = &gocql.ConstantReconnectionPolicy{MaxRetries: 10, Interval: 8 * time.Second}
I also found the below snippet online, but it doesn't look like it's designed to handle this scenario:
var cluster *gocql.ClusterConfig
var session *gocql.Session
func getCassandraSession() *gocql.Session {
if session == nil || session.Closed() {
if cluster == nil {
cluster = gocql.NewCluster("127.0.0.1:9042")
cluster.Keyspace = "demodb"
cluster.Consistency = gocql.One
cluster.ProtoVersion = 4
}
var err error
if session, err = cluster.CreateSession(); err != nil {
panic(err)
}
}
return session
}
Are any of the above methods sufficient to ensure that reconnections are attempted until our Cassandra server gets back online? And if not, what's the best practice for this scenario?
Special thanks to #Jim Wartnick for this. I just tried turning Cassandra off on my local machine and then turning it back on and gocql instantly reconnected without having to use any of the above snippets in my question.
As long as your Cluster and Session have connected to Cassandra at least once, even if Cassandra goes down, they will instantly reconnect to it as soon as Cassandra gets back online.
Many thanks to everyone who helped!
I am currently using the redigo library for my project where i create a redis pool
I use defer to release the redis connection every time i get one from the pool
c := redisPool.Get()
defer c.Close()
But it will block forever in this scenario if MaxActive has been set
func function1() {
c := redisPool.Get()
defer c.Close()
function2()
...
}
func function2() {
c := redisPool.Get()
defer c.Close()
...
}
should i use only one redis connection in one goroutine?
You have a few options here.
You can Close() when you are done, returning the connection to the pool and then calling function2. Upside: works, not too complex. Downside: management of returning the connection in the case of multiple exit points from the function.
You can change function2 to take a redis.Conn argument that it uses and just pass that connection off. Upside: defer still works for function1. Downside: you need a connection to call function2 and need to do connection management from the calling site. In your example that is easy enough.
Make sure you have at least N*2 max connections, where N is the max number of concurrent goroutines that will be running. Upside: Your code stays as-is without changes. Downside: limited in the number of concurrent calls to function1 you can make.
You can use following approach to make sure application won't lock/ break.
set wait: true in the pool configuration.
// If Wait is true and the pool is at the MaxActive limit, then Get()
waits // for a connection to be returned to the pool before returning.
Confirm that the server's maxclient limit is larger than MaxActive The default maxclient is 10k.
Most applications can keep connection use low by avoiding long or blocking operations (other than calls to Redis) between the call to Get() and the call to Close().
Hope this helps.
As the title says I don't know if having multiple sql.Open statements is a good or bad thing or what or if I should have a file with just an init that is something like:
var db *sql.DB
func init() {
var err error
db, err = sql.Open
}
just wondering what the best practice would be. Thanks!
You should at least check the error.
As mentioned in "Connecting to a database":
Note that Open does not directly open a database connection: this is deferred until a query is made. To verify that a connection can be made before making a query, use the Ping function:
if err := db.Ping(); err != nil {
log.Fatal(err)
}
After use, the database is closed using Close.
If possible, limit the number of opened connection to a database to a minimum.
See "Go/Golang sql.DB reuse in functions":
You shouldn't need to open database connections all over the place.
The database/sql package does connection pooling internally, opening and closing connections as needed, while providing the illusion of a single connection that can be used concurrently.
As elithrar points out in the comment, database.sql/#Open does mention:
The returned DB is safe for concurrent use by multiple goroutines and maintains its own pool of idle connections.
Thus, the Open function should be called just once.
It is rarely necessary to close a DB.
As mentioned here
Declaring *sql.DB globally also have some additional benefits such as SetMaxIdleConns (regulating connection pool size) or preparing SQL statements across your application.
You can use a function init, which will run even if you don't have a main():
var db *sql.DB
func init() {
db, err = sql.Open(DBparms....)
}
init() is always called, regardless if there's main or not, so if you import a package that has an init function, it will be executed.
You can have multiple init() functions per package, they will be executed in the order they show up in the code (after all variables are initialized of course).