pgxpool's config.AfterRelease is not firing - go

I have the following code:
func (s *SqlServerDatabase) ConnectPool() {
config, err := pgxpool.ParseConfig(s.Url)
if err != nil {
fmt.Println("config Database Fail")
fmt.Print(err)
}
config.AfterRelease = func(conn *pgx.Conn) bool {
fmt.Println("After Releasing")
return true
}
config.BeforeAquire = func(ctx context.Context, conn *pgx.Conn) bool {
fmt.Println("Before Aquiring")
return true
}
conn, err := pgxpool.ConnectConfig(context.Background(), config)
s.PoolConn = conn
}
func (s *SqlServerDatabase) PoolQuery(query string) (pgx.Rows, error) {
conn, err := s.PoolConn.Acquire(context.Background())
if err != nil {
log.Printf("Couldn't get a connection with the database. Reason %v", err)
} else {
// release the connection to the pool after using it
defer conn.Release()
results, err := conn.Query(context.Background(), query)
if err != nil {
log.Printf("Couldn't execute query. Reason %v", err)
} else {
// show the results boy, you got it.
fmt.Printf("%T\n", results)
}
return results, err
}
return nil, err
}
So :
I create the connection in ConnectPool. thats where I set up the config and connect.
Then In my query method I acquire one of the pools the BeforeAcquire method fires and prints.
But the AfterRelease method never does.
I release the connection in the deferred call so I'm not sure why its not running.

See the code at pgxpool/conn.go l.30 :
/*20*/ func (c *Conn) Release() {
...
/*30*/ if conn.IsClosed() || conn.PgConn().IsBusy() ||
conn.PgConn().TxStatus() != 'I' ||
(now.Sub(res.CreationTime()) > c.p.maxConnLifetime) {
res.Destroy()
return
}
/* 'c.afterRelease' is checked and used only after that block */
So : under some conditions, the connection is detroyed right away, and AfterRelease is indeed never executed.
note: the code link reflects the state of the master branch on 2021-05-05.

Related

Pgxpool returns "pool closed" error on Scan

I'm trying to implement pgxpool in a new go app. I keep getting a "pool closed" error after attempting a scan into a struct.
The pgx logger into gives me this after connecting. I thought the pgxpool was meant to remain open.
{"level":"info","msg":"closed connection","pid":5499,"time":"2022-02-24T16:36:33+10:30"}
Here is my router code
func router() http.Handler {
var err error
config, err := pgxpool.ParseConfig(os.Getenv("DATABASE_URL"))
if err != nil {
log.Fatalln(err)
}
log.Println(os.Getenv("DATABASE_URL"))
logrusLogger := &logrus.Logger{
Out: os.Stderr,
Formatter: new(logrus.JSONFormatter),
Hooks: make(logrus.LevelHooks),
Level: logrus.InfoLevel,
ExitFunc: os.Exit,
ReportCaller: false,
}
config.ConnConfig.Logger = NewLogger(logrusLogger)
db, err := pgxpool.ConnectConfig(context.Background(), config)
if err != nil {
log.Fatalln(err)
}
defer db.Close()
--- minio connection
rs := newAppResource(db, mc)
Then, in a helper file I setup the resource
type appResource struct {
db *pgxpool.Pool
mc *minio.Client
}
// newAppResource function to pass global var
func newAppResource(db *pgxpool.Pool, mc *minio.Client) *appResource {
return &appResource{
db: db,
mc: mc,
}
}
There "pool closed" error occurs at the end of this code
func (rs *appResource) login(w http.ResponseWriter, r *http.Request) {
var user User
var login Login
d := json.NewDecoder(r.Body)
d.DisallowUnknownFields() // catch unwanted fields
err := d.Decode(&login)
if err != nil {
http.Error(w, err.Error(), http.StatusBadRequest)
return
}
if err != nil {
fmt.Println("can't decode JSON", err)
}
if login.Email == "" {
log.Println("empty email")
return
}
log.Println(login.Email)
log.Println(login.Password)
if login.Password == "" {
log.Println("empty password")
return
}
// optional extra check
if d.More() {
http.Error(w, "extraneous data after JSON object", http.StatusBadRequest)
return
}
sqlStatement := "SELECT user_id, password FROM users WHERE active = 'true' AND email = ?"
row := rs.db.QueryRow(context.Background(), sqlStatement, login.Email)
err = row.Scan(&user.UserId, &user.Password)
if err == sql.ErrNoRows {
log.Println("user not found")
http.Error(w, err.Error(), http.StatusUnauthorized)
return
}
if err != nil {
log.Println(err)
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
It appears that you are doing something like the following:
func router() http.Handler {
db, err := pgxpool.ConnectConfig(context.Background(), config)
if err != nil {
log.Fatalln(err)
}
defer db.Close()
return appResource{db: db}
}
The issue with this is that the defer db.Close() runs when the function router() ends and this is before the returned pgxPool.Pool is actually used (the http.Handler returned will be used later when http requests are processed). Attempting to use a closed pgxPool.Pool results in the error you are seeing.
The simplest solution is to simply remove the defer db.Close() however you might also consider calling db.Close() as part of a clean shutdown process (it needs to remain open as long as you are handling requests).
You are using pgxpool which does differ from the standard library; however I believe that the advice given in the standard library docs applies here:
It is rarely necessary to close a DB.

Prepared statement not returning results from pgxpool

I have the following code:
func (s *SqlServerDatabase) ConnectPool() {
config, err := pgxpool.ParseConfig(s.Url)
if err != nil {
fmt.Println("config Database Fail")
fmt.Print(err)
}
config.AfterRelease = func(conn *pgx.Conn) bool {
fmt.Println("After Releasing")
return true
}
config.BeforeAcquire = func(ctx context.Context, conn *pgx.Conn) bool {
fmt.Println("Before Aquiring")
return true
}
conn, err := pgxpool.ConnectConfig(context.Background(), config)
s.PoolConn = conn
}
func (s *SqlServerDatabase) PoolPreparedStatementQuery(statement string) { //} (pgx.Rows, error) {
conn, err := s.PoolConn.Acquire(context.Background())
fmt.Println("End of PS")
if err != nil {
log.Printf("Couldn't get a connection with the database. Reason %v", err)
} else {
// release the connection to the pool after using it
defer conn.Release()
tx, err := conn.BeginTx(context.Background(), pgx.TxOptions{})
if err != nil {
fmt.Println(err)
}
if _, err := tx.Prepare(context.Background(), statement, ""); err != nil {
fmt.Println(err)
}
results, err := tx.Query(context.Background(), statement)
if err != nil {
log.Printf("Couldn't execute query. Reason %v", err)
} else {
// show the results boy, you got it.
fmt.Printf("%T\n", results)
fmt.Println("results:", results)
fmt.Println("Before loop")
for results.Next() {
fmt.Println("inside")
fmt.Println(results.Scan())
}
}
}
fmt.Println("End of PS")
}
Im pretty new to Go and hadnt even heard of prepared statements until yesterday so please bear with me.
I create a connection in ConnectPool. Then in PoolPreparedStatement I try to execute the prepared statement. I am calling it like:
db.PoolPreparedStatementQuery("EXECUTE test_ps")
where test ps is a prepared statement i created on my postgresql database and was able to execute with that command.
I call BeginTx cos Prepare is only on the Tx class.
Im not sure i need to do that though? The documentation says Prepare is used to create a prepared statement but i dont need to create it, just execute it. It says it doesnt matter if you call it on an existing statement though so i guess it doesnt matter. When I execute the prepared statement and try and loop through the results there are none. So im not sure what im doing wrong.

How I can close the connection by timeout if the client doesn't respond in 10 seconds?

I have code (I use https://github.com/fiorix/go-smpp):
// -----------------------------------------------
// handleConnection new clients.
// -----------------------------------------------
func (_srv *ServerSmpp) handleConnection(_cfg *ConfigSmpp, c *conn) {
defer c.Close()
if err := _srv.auth(_cfg, c); err != nil {
if err != io.EOF {
log.Printf("smpp_server: server auth failed: %s\n", err)
}
return
}
notify := make(chan error)
go func() {
for {
pb, err := c.Read()
if err != nil {
notify <- err
return
}
err = _srv.Handler(_srv.RemoteProvider, c, pb)
if err != nil {
fmt.Printf("%s\n", err)
notify <- err
return
}
}
}()
for {
select {
case err:= <-notify:
if io.EOF == err {
fmt.Printf("Smpp server (read): %s\n", err)
return
}
case <-time.After(time.Second * 10):
fmt.Printf("Client disconnected by timeout.\n")
return
}
}
}
Code for invoked handleConnection:
func (_srv *ServerSmpp) Serve(_cfg *ConfigSmpp) {
for {
client, err := _srv.NetListener.Accept()
if err != nil {
break
}
c := newConn(client)
go _srv.handleConnection(_cfg, c)
}
}
When this code work, the server disconnects all clients by timeout 10 sec, but how I can disconnect the client when it's doesn't work 10 sec?
Your client object seems to be a net.Conn,
choose a way to call client.SetReadDeadline() with the appropriate time.Time value before blocking on client.Read() :
c.client.SetDeadline( time.Now().Add(10 * time.Second )
pb, err := c.Read() { ...

Error Handling within a for loop in Go results probably in a next iteration

I struggle with a specific Go implementation for sending log files to different locations:
package main
func isDestinationSIEM(json_msg string, json_obj *jason.Object, siem_keys []string) (bool) {
if json_obj != nil {
dest, err := json_obj.GetString("destination")
if err == nil {
if strings.Contains(dest,"SIEM") {
return true
}
}
for _, key := range siem_keys {
if strings.Contains(json_msg, key) {
return true
}
}
}
return false
}
func sendToSIEM(siem_dst string, json_msg string) (error) {
// Create connection to syslog server
roots := x509.NewCertPool()
ok := roots.AppendCertsFromPEM([]byte(rootPEM))
if !ok {
fmt.Println("failed to parse root certificate")
}
config := &tls.Config{RootCAs: roots, InsecureSkipVerify: true}
conn, err := tls.Dial("tcp", siem_dst, config)
if err != nil {
fmt.Println("Error connecting SIEM")
fmt.Println(err.Error())
} else {
// Send log message
_, err = fmt.Fprintf(conn, json_msg)
if err != nil {
fmt.Println("Error sending SIEM message: ", json_msg)
fmt.Println(err.Error())
}
}
defer conn.Close()
return err
}
func main() {
// simplified code otherwise there would have been too much
// but the 'devil' is this for loop
for _, obj := range objects {
// first check
isSIEM := isDestinationSIEM(obj, siem_keys)
if isSIEM {
err := sendToSIEM(obj)
if err != nil {
// print error
}
isAUDIT:= isDestinationSIEM(obj)
if isAUDIT {
err := sendToAUDIT(obj)
if err != nil {
// print error
}
} // end of for
}
When the 'if isSIEM' returns an error, the second check 'if isAUDIT' is not conducted.
Why is this? If an error is returned, does the loop start with the next iteration?
The error looks like this:
runtime error: invalid memory address or nil pointer dereference: errorString (which lists a couple of go packages)
The error looks like this: runtime error: invalid memory address or nil pointer dereference: errorString (which lists a couple of go packages)
It means you catch the panic() and your program has been stopped that means your circle for is stopped too.
Here details how works with panic https://blog.golang.org/defer-panic-and-recover

rpc.ServerCodec Still Serving?

I was performing some RPC tests, and stumbled across a problem I can't seem to solve. In my testing I create three separate RPC servers, all of which I try to close and shutdown. However upon performing my last test (TestRpcCodecServerClientComm), it seems my client connection is connecting to the first RPC server I started (I know this because I at some point attached IDs to the RPCHandlers), even though I attempted everything I could to make sure it was shutdown. Though the code is not there I have attempted to inspect every single error I could, but that did not bring about anything.
rpc.go
package rbot
import (
"io"
"net"
"net/rpc"
"net/rpc/jsonrpc"
)
func RpcCodecClientWithPort(port string) (rpc.ClientCodec, error) {
conn, err := net.Dial("tcp", "localhost:"+port)
if err != nil {
return nil, err
}
return jsonrpc.NewClientCodec(conn), nil
}
func RpcCodecServer(conn io.ReadWriteCloser) rpc.ServerCodec {
return jsonrpc.NewServerCodec(conn)
}
rpc_test.go
package rbot
import (
"errors"
"fmt"
"net"
"net/rpc"
"testing"
)
type RPCHandler struct {
RPCServer net.Listener
conn rpc.ServerCodec
done chan bool
TestPort string
stop bool
GotRPC bool
}
func (r *RPCHandler) SetupTest() {
r.stop = false
r.GotRPC = false
r.done = make(chan bool)
r.TestPort = "5556"
}
// TODO: Create separate function to handle erroring
func (r *RPCHandler) CreateRPCServer() error {
rpc.RegisterName("TestMaster", TestAPI{r})
var err error
r.RPCServer, err = net.Listen("tcp", ":"+r.TestPort)
if err != nil {
return err
}
go func() {
for {
conn, err := r.RPCServer.Accept()
if err != nil || r.stop {
r.done <- true
return
}
r.conn = RpcCodecServer(conn)
rpc.ServeCodec(r.conn)
}
}()
return nil
}
func (r *RPCHandler) CloseRPCServer() error {
r.stop = true
if r.conn != nil {
err := r.conn.Close()
if err != nil {
fmt.Println(err)
}
}
err := r.RPCServer.Close()
<-r.done
return err
}
type TestAPI struct {
t *RPCHandler
}
func (tapi TestAPI) Send(msg string, result *string) error {
if msg == "Got RPC?" {
tapi.t.GotRPC = true
return nil
}
return errors.New("Didn't receive right message")
}
// Check if we can create and close an RPC server successfully using the RPC server codec.
func TestRpcCodecServer(t *testing.T) {
r := RPCHandler{}
r.SetupTest()
err := r.CreateRPCServer()
if err != nil {
t.Fatalf("Could not create rpc server! %s:", err.Error())
}
err = r.CloseRPCServer()
if err != nil {
t.Fatalf("Could not close RPC server! %s:", err.Error())
}
}
// Check if we can create a client without erroring.
func TestRpcCodecClientWithPortt(t *testing.T) {
r := RPCHandler{}
r.SetupTest()
r.CreateRPCServer()
defer r.CloseRPCServer()
RPCClient, err := RpcCodecClientWithPort(r.TestPort)
defer RPCClient.Close()
if err != nil {
t.Fatalf("Could not create an RPC client! %s:", err.Error())
}
}
// Let's double check and make sure our server and client can speak to each other
func TestRpcCodecServerClientComm(t *testing.T) {
r := RPCHandler{}
r.SetupTest()
r.CreateRPCServer()
defer r.CloseRPCServer()
RPCCodec, _ := RpcCodecClientWithPort(r.TestPort)
RPCClient := rpc.NewClientWithCodec(RPCCodec)
defer RPCClient.Close()
var result string
err := RPCClient.Call("TestMaster.Send", "Got RPC?", &result)
if err != nil {
t.Fatalf("Error while trying to send RPC message: %s", err.Error())
}
if !r.GotRPC {
t.Fatalf("Could not send correct message over RPC")
}
}
Not sure if I'm just mishandling the connection or something of the like, any help would be much appreciated.
For the Record The RPC api does receive the correct string message
While not the source of your problems, your test configuration has a few race conditions which you should take care of before they cause problems. Always check for issues with the -race option. You should also let the OS allocate the port so you don't run into conflicts. See for example how httptest.Server works.
Your failure here is that you're not creating a new rpc.Server for each test, you're reusing the rpc.DefaultServer. The first call to CreateRPCServer registers a TestAPI under the name TestMaster. Each subsequent call uses the already registered instance.
If you create a new rpc.Server each time you setup the test and register a new TestAPI, the final test will pass.
srv := rpc.NewServer()
srv.RegisterName("TestMaster", testAPI)
...
// and then use srv to handle the new connection
srv.ServeCodec(RpcCodecServer(conn))

Resources