Hey I'm getting an error message : conn busy from pgx
I don't know how to solve this. Here is my function :
func (r *proverbRepo) SelectPendingProverbs(table string) (proverbs []domain.Proverb, err error) {
query := fmt.Sprintf("SELECT id, proverb literal FROM %s", table)
rows, err := r.Db.Query(context.Background(), query)
defer rows.Close()
if err != nil {
return
}
for rows.Next() {
var prov domain.Proverb
if err = rows.Scan(&prov.ID, &prov.Literal); err != nil {
return
}
proverbs = append(proverbs, prov)
}
return
}
r.Db is pgx.Connect(context.Background(), os.Getenv("PSQL_URL"))
I'm fetching two different table in a very short interval from two separate front end requests.
The first request goes through, the other one returns the conn busy error message.
I really don't know what to look for, would somebody help me ?
pgx.Connect() returns a pgx.Conn which cannot be used concurrently. This is what the godocs of this type state:
Conn is a PostgreSQL connection handle. It is not safe for concurrent usage. Use a connection pool to manage access to multiple database connections from multiple goroutines.
So if you replace pgx.Connect() with pgxpool.Connect() from github.com/jackc/pgx/pgxpool you should be fine.
The r.Db returned by pgx.Connect(context.Background(), os.Getenv("PSQL_URL")) if of type *pgx.Conn and represents a single connection which is not concurrency safe. Usually, you would like to use a connection pool to handle the concurrency for you and allows reusing open connection.
To use a connection pool replace the import github.com/jackc/pgx/v4 with github.com/jackc/pgx/v4/pgxpool and connect with pgxpool.Connect() instead of pgx.Connect() and the api will be the same:
r.Pool := pgxpool.Connect(context.Background(), os.Getenv("PSQL_URL"))
r.Pool.Query(context.Background(), query)
...
if at any given point you need to use a single connection to access some lower-level feature you can safely acquire a concurrent safe connection as follow:
conn, err := r.Pool.Acquire(context.Background())
if err != nil {
fmt.Fprintln(os.Stderr, "Error acquiring connection:", err)
os.Exit(1)
}
defer conn.Release()
...
Related
Just getting started with Go and I'm wondering about the following situation:
I have a pretty simple codebase where I simply want to open/close a database connection and execute a simple query. I can do this as follows (just showing the important bits here):
import (
"database/sql"
_ "github.com/lib/pq"
)
func (db *Database) ExecQueryA() {
dbConn, err := sql.Open("postgres", db.psqlconn)
if err != nil {
panic(err)
}
defer dbConn.Close()
_, err = db.Exec(...
if err != nil {
panic(err)
}
}
The above idea works fine, but what if I want to write x more of these functions, I do not want to duplicate this part:
dbConn, err := sql.Open("postgres", db.psqlconn)
if err != nil {
panic(err)
}
defer dbConn.Close()
At the start of each function (i.e. I want to avoid code duplication). In python I would write a context manager for this, I.e. I would use a with .. statement which would open and close the database connection for me. When using Go, what is the best way to avoid code duplication in this use case?
As Brits points out in the comment to your question, the *sql.DB does not need to be open and closed every time you intend to use it. Instead a single shared instance of *sql.DB, Opened once at the launch of your app, is a common and recommended practice.
... the Open function should be called just once. It is rarely necessary
to close a DB.
Note that *sql.DB is not a connection, instead, it is a pool that manages multiple connections, opens as many as necessary (and possible), keeps idle ones around if necessary, frees them if unnecessary, etc. And most of all, it is safe for concurrent use.
DB is a database handle representing a pool of zero or more underlying
connections. It's safe for concurrent use by multiple goroutines.
To answer your actual question, one pattern to reduce the repetition of obtaining-and-releasing resources is to pass a function literal to a wrapper function:
func (db *Database) run(f func(c *sql.DB)) {
c, err := sql.Open("postgres", db.psqlconn)
if err != nil {
panic(err)
}
defer c.Close()
f(c)
}
func (db *Database) ExecQueryA() {
db.run(func(c *sql.DB) {
_, err := c.Exec(...
if err != nil {
panic(err)
}
})
}
Do I need a mutex in this case? I am refreshing the token with a goroutine, the token is used in another goroutine. In other words, will my token be empty at some point so that the response will be a 401?
If yes, is it part of the structure c *threatq or is it a simple variable, I mean, a "standalone" one inside my code.
// IndicatorChannelIterator returns indicators from ThreatQ into a channel.
func (c *threatq) IndicatorChannelIterator() (<-chan *models.Indicator, error) {
// Authenticate
token, err := c.authenticate(c.clientID, c.email, c.password)
if err != nil {
return nil, fmt.Errorf("Error while authenticating to TQ : %s", err)
}
// Periodically refresh the token
ticker := time.NewTicker(30 * time.Minute)
go func() {
for range ticker.C {
token, err = c.authenticate(c.clientID, c.email, c.password)
if err != nil {
logrus.Errorf("Error while authenticating to TQ : %s", err)
}
}
}()
// Prepare the query
query := &Query{}
// Get the first page
firstTQResponse, err := c.advancedSearch(query, token, 0)
if err != nil {
return nil, fmt.Errorf("Error while getting the first page from TQ : %s", err)
}
// Create the channel
indicators := make(chan *models.Indicator)
// Request the others
go func() {
req := 1
total := firstTQResponse.Total
for offset := 0; offset < total; offset += c.perPage {
// Search the indicators
tqResponse, err := c.advancedSearch(query, token, offset)
if err != nil {
logrus.Errorf("Error while getting the indicators from TQ : %s", err)
continue
}
...
The rule is simple: if a variable is accessed from multiple goroutines and at least one of them is a write, explicit synchronization is needed.
This is true in your case: one of your goroutine writes the token variable (and also the err variable!), and another reads it, so you must synchronize access.
Since token is not a field of the threatq structure, putting the mutex that protects it would not be wise. Always put the mutex close to the data it ought to protect.
Some notes: as mentioned earlier, you also write and read the local err variable from multiple goroutines. You should not do this, instead create another local variable to hold the error from other goroutines (unless you want to "translfer" the error between goroutines, but this is not the case here).
See related questions:
Immutability of string and concurrency
Should we synchronize variable assignment in goroutine?
golang struct concurrent read and write without Lock is also running ok?
Reading values from a different thread
Why does this code cause data race?
Yes, you could also try to run this test with -race flag enabled. Go's race detector will probably tell you that token is a shared variable across multiple goroutines. Thus, it must be protected with a Mutex or RWMutex.
In your case I think that RWMutex is more appropriate because there is one goroutine that changes (i.e. write) the state of token every 30 mins and another goroutine that reads its value.
If you don't protect the shared variable with a lock, the second goroutine might read an old value of token, that could be expired.
I am using this gocql package.
I am trying to get this example working.
func main() {
// connect to the cluster
cluster := gocql.NewCluster("192.168.1.1", "192.168.1.2", "192.168.1.3")
cluster.ProtoVersion = 3
cluster.Keyspace = "example"
cluster.Consistency = gocql.Quorum
session, err := cluster.CreateSession()
defer session.Close()
if err != nil {
fmt.Printf("%v\n", err)
return
}
uuid := gocql.TimeUUID()
fmt.Printf("UUID : %v", uuid)
query := session.Query(`INSERT INTO tweet (timeline, id, text) VALUES
(?, ?, ?)`, "me", uuid, "hello world")
fmt.Println("About to exec")
err = query.Exec()
// insert a tweet
if err != nil {
log.Fatal(err)
}
var id gocql.UUID
var text string
/* Search for a specific set of records whose 'timeline' column matches
* the value 'me'. The secondary index that we created earlier will be
* used for optimizing the search */
if err := session.Query(`SELECT id, text FROM tweet WHERE timeline = ?
LIMIT 1`,"me").Consistency(gocql.One).Scan(&id, &text); err != nil {
log.Fatal(err)
}
fmt.Println("Tweet:", id, text)
// list all tweets
iter := session.Query(`SELECT id, text FROM tweet WHERE timeline = ?`,
"me").Iter()
for iter.Scan(&id, &text) {
fmt.Println("Tweet:", id, text)
}
if err := iter.Close(); err != nil {
log.Fatal(err)
}
}
Using the Cassandra shell I have created the keyspace "example" and the table "tweet" and they all work fine.
However, when I run the program, it gives me this error:
2017/04/14 20:52:55 gocql: unable to dial control conn 192.168.1.3: dial tcp
192.168.1.3:9042: i/o timeout
gocql: unable to create session: control: unable to connect to initial
hosts: dial tcp 192.168.1.3:9042: i/o timeoutpanic: runtime error: invalid
memory address or nil pointer dereference
[signal 0xc0000005 code=0x0 addr=0x0 pc=0x60236d]
goroutine 1 [running]:
github.com/gocql/gocql.(*Session).Close(0x0)
/home/mda/.local/go/src/github.com/gocql/gocql/session.go:344 +0x2d
For some reason, gocql is unable to dial the localhost connection, and it times out. I am not sure how to fix this and a stackoverflow and google search hasn't helped so far.
Any ideas?
Looks like the ip addresses are not according to the cluster situation on your computer. After giving me the nodetool status I think you should update the cluster := gocql.NewCluster to have just the 127.0.0.1 address and not the addresses from example.
I think in reality you are just running single node cassandra local instance. Which is fine for development.
Basically:
cluster := gocql.NewCluster("127.0.0.1")
and just drop this line all together
cluster.ProtoVersion = 3
Hi in database/sql package I can execute a query in two ways:
First way: using Sql.Stmt
var DeletePermissionStmt *sql.Stmt
DeletePermissionStmt, err = database.Prepare(`DELETE FROM permission WHERE permission_id=$1`)
if err != nil {
log.Errorf("can't prepare delete permission statement: %s", err.Error())
}
transaction, err := database.Begin() // assume postgres database is defined previously
if err != nil {
log.WithFields(logFields).Errorf("can't start transaction: %s", err.Error())
return err
}
_, err := transaction.Stmt(DeletePermissionStmt).Exec(permission_id)
Second way: using string
var DeletePermissionStmt string
DeletePermissionStmt = `DELETE FROM permission WHERE permission_id=$1`
transaction, err := database.Begin() // assume postgres database is defined previously
if err != nil {
log.WithFields(logFields).Errorf("can't start transaction: %s", err.Error())
return err
}
_, err := transaction.Exec(DeletePermissionStmt,permission_id)
The only difference That I know that its is not possible to use sql.Stmt when you are returning something for example Insert Into FOO(f1,f2,f3) Values(v1,v2,v3) returning f_id
Is there any other differences? and when should I use each one?
Using the Stmt helps you avoid sql injection from the user.
From wikipedia:
Prepared statements are resilient against SQL injection, because
parameter values, which are transmitted later using a different
protocol, need not be correctly escaped. If the original statement
template is not derived from external input, SQL injection cannot
occur.
http.Serve either returns an error as soon as it is called or blocks if successfully executing.
How can I make it so that if it blocks it does so in its own goroutine? I currently have the following code:
func serveOrErr(l net.Listener, handler http.Handler) error {
starting := make(chan struct{})
serveErr := make(chan error)
go func() {
starting <- struct{}{}
if err := http.Serve(l, handler); err != nil {
serveErr <- err
}
}()
<-starting
select {
case err := <-serveErr:
return err
default:
return nil
}
}
This seemed like a good start and works on my test machine but I believe that there are no guarantees that serveErr <- err would be called before case err := <-serveErr therefore leading to inconsistent results due to a data race if http.Serve were to produce an error.
http.Serve either returns an error as soon as it is called or blocks if successfully executing
This assumption is not correct. And I believe it rarely occurs. http.Serve calls net.Listener.Accept in the loop – an error can occur any time (socket closed, too many open file descriptors etc.). It's http.ListenAndServe, usually being used for running http servers, which often fails early while binding listening socket (no permissions, address already in use).
In my opinion what you're trying to do is wrong, unless really your net.Listener.Accept is failing on the first call for some reason. Is it? If you want to be 100% sure your server is working, you could try to connect to it (and maybe actually transmit something), but once you successfully bound the socket I don't see it really necessary.
You could use a timeout on your select statement, e.g.
timeout := time.After(5 * time.Millisecond) // TODO: ajust the value
select {
case err := <-serveErr:
return err
case _ := <- timeout:
return nil
}
This way your select will block until serveErr has a value or the specified timeout has elapsed. Note that the execution of your function will therefore block the calling goroutine for up to the duration of the specified timeout.
Rob Pike's excellent talk on go concurrency patterns might be helpful.