Prepared statement not returning results from pgxpool - go

I have the following code:
func (s *SqlServerDatabase) ConnectPool() {
config, err := pgxpool.ParseConfig(s.Url)
if err != nil {
fmt.Println("config Database Fail")
fmt.Print(err)
}
config.AfterRelease = func(conn *pgx.Conn) bool {
fmt.Println("After Releasing")
return true
}
config.BeforeAcquire = func(ctx context.Context, conn *pgx.Conn) bool {
fmt.Println("Before Aquiring")
return true
}
conn, err := pgxpool.ConnectConfig(context.Background(), config)
s.PoolConn = conn
}
func (s *SqlServerDatabase) PoolPreparedStatementQuery(statement string) { //} (pgx.Rows, error) {
conn, err := s.PoolConn.Acquire(context.Background())
fmt.Println("End of PS")
if err != nil {
log.Printf("Couldn't get a connection with the database. Reason %v", err)
} else {
// release the connection to the pool after using it
defer conn.Release()
tx, err := conn.BeginTx(context.Background(), pgx.TxOptions{})
if err != nil {
fmt.Println(err)
}
if _, err := tx.Prepare(context.Background(), statement, ""); err != nil {
fmt.Println(err)
}
results, err := tx.Query(context.Background(), statement)
if err != nil {
log.Printf("Couldn't execute query. Reason %v", err)
} else {
// show the results boy, you got it.
fmt.Printf("%T\n", results)
fmt.Println("results:", results)
fmt.Println("Before loop")
for results.Next() {
fmt.Println("inside")
fmt.Println(results.Scan())
}
}
}
fmt.Println("End of PS")
}
Im pretty new to Go and hadnt even heard of prepared statements until yesterday so please bear with me.
I create a connection in ConnectPool. Then in PoolPreparedStatement I try to execute the prepared statement. I am calling it like:
db.PoolPreparedStatementQuery("EXECUTE test_ps")
where test ps is a prepared statement i created on my postgresql database and was able to execute with that command.
I call BeginTx cos Prepare is only on the Tx class.
Im not sure i need to do that though? The documentation says Prepare is used to create a prepared statement but i dont need to create it, just execute it. It says it doesnt matter if you call it on an existing statement though so i guess it doesnt matter. When I execute the prepared statement and try and loop through the results there are none. So im not sure what im doing wrong.

Related

rows.Next() halts after some number of rows

Im newbie in Golang, so it may be simple for professionals but I got stuck with no idea what to do next.
I'm making some migration app that extract some data from oracle DB and after some conversion insert it to Postges one-by-one.
The result of native Query in DB console returns about 400k of rows and takes about 13 sec to end.
The data from Oracle extracts with rows.Next() with some strange behavior:
First 25 rows extracted fast enough, then about few sec paused, then new 25 rows until it pauses "forever".
Here is the function:
func GetHrTicketsFromOra() (*sql.Rows, error) {
rows, err := oraDB.Query("select id,STATE_ID,REMEDY_ID,HEADER,CREATE_DATE,TEXT,SOLUTION,SOLUTION_USER_LOGIN,LAST_SOLUTION_DATE from TICKET where SOLUTION_GROUP_ID = 5549")
if err != nil {
println("Error while getting rows from Ora")
return nil, err
}
log.Println("Finished legacy tickets export")
return rows, err
}
And here I export data:
func ConvertRows(rows *sql.Rows, c chan util.ArchTicket, m chan int) error {
log.Println("Conversion start")
defer func(rows *sql.Rows) {
err := rows.Close()
if err != nil {
log.Println("ORA connection closed", err)
return
}
}(rows)
for rows.Next() {
log.Println("Reading the ticket")
ot := util.OraTicket{}
at := util.ArchTicket{}
err := rows.Scan(&ot.ID, &ot.StateId, &ot.RemedyId, &ot.Header, &ot.CreateDate, &ot.Text, &ot.Solution, &ot.SolutionUserLogin, &ot.LastSolutionDate)
if err != nil {
log.Println("Error while reading row", err)
return err
}
at = convertLegTOArch(ot)
c <- at
}
if err := rows.Err(); err != nil {
log.Println("Error while reading row", err)
return err
}
m <- 1
return nil
}
UPD. I use "github.com/sijms/go-ora/v2" driver
UPD2. Seems like the root cause of the problem is in TEXT and SOLUTION fields of the result rows. They are varchar and can be big enough. Deleting them from the direct query changes the time of execution from 13sec to 258ms. But I still have no idea what to do with that.
UPD3.
Minimal reproducible example
package main
import (
"database/sql"
_ "github.com/sijms/go-ora/v2"
"log"
)
var oraDB *sql.DB
var con = "oracle://login:password#ora_db:1521/database"
func InitOraDB(dataSourceName string) error {
var err error
oraDB, err = sql.Open("oracle", dataSourceName)
if err != nil {
return err
}
return oraDB.Ping()
}
func GetHrTicketsFromOra() {
var ot string
rows, err := oraDB.Query("select TEXT from TICKET where SOLUTION_GROUP_ID = 5549")
if err != nil {
println("Error while getting rows from Ora")
}
for rows.Next() {
log.Println("Reading the ticket")
err := rows.Scan(&ot)
if err != nil {
log.Println("Reading failed", err)
}
log.Println("Read:")
}
log.Println("Finished legacy tickets export")
}
func main() {
err := InitOraDB(con)
if err != nil {
log.Println("Error connection Ora")
}
GetHrTicketsFromOra()
}

There is already an object named 'company_names' in the database

I use Gorm for Golang like this code,
func Connect() (*gorm.DB, error) {
var (
err error
db *gorm.DB
)
dsn := "sqlserver://User:12345#127.0.0.1:1433?database=gorm"
db, err = gorm.Open(sqlserver.Open(dsn), &gorm.Config{})
if err != nil {
return nil, err
}
err = db.Debug().AutoMigrate(&models.CompanyName{}, &models.CarModel{}, &models.CreateYear{}, &models.Diversity{})
if err != nil {
return nil, err
}
return db, nil
}
and in main.go
func main() {
db, err := Connect()
if err != nil {
panic("database connection failed")
}
ech := echo.New()
}
Ok in the first time when we run code, gorm create tables in database,but in the second time I got an error thet
mssql: There is already an object named 'table_names' in the database.
Do I have to delete db.Debug().AutoMigrate)(?

Pgxpool returns "pool closed" error on Scan

I'm trying to implement pgxpool in a new go app. I keep getting a "pool closed" error after attempting a scan into a struct.
The pgx logger into gives me this after connecting. I thought the pgxpool was meant to remain open.
{"level":"info","msg":"closed connection","pid":5499,"time":"2022-02-24T16:36:33+10:30"}
Here is my router code
func router() http.Handler {
var err error
config, err := pgxpool.ParseConfig(os.Getenv("DATABASE_URL"))
if err != nil {
log.Fatalln(err)
}
log.Println(os.Getenv("DATABASE_URL"))
logrusLogger := &logrus.Logger{
Out: os.Stderr,
Formatter: new(logrus.JSONFormatter),
Hooks: make(logrus.LevelHooks),
Level: logrus.InfoLevel,
ExitFunc: os.Exit,
ReportCaller: false,
}
config.ConnConfig.Logger = NewLogger(logrusLogger)
db, err := pgxpool.ConnectConfig(context.Background(), config)
if err != nil {
log.Fatalln(err)
}
defer db.Close()
--- minio connection
rs := newAppResource(db, mc)
Then, in a helper file I setup the resource
type appResource struct {
db *pgxpool.Pool
mc *minio.Client
}
// newAppResource function to pass global var
func newAppResource(db *pgxpool.Pool, mc *minio.Client) *appResource {
return &appResource{
db: db,
mc: mc,
}
}
There "pool closed" error occurs at the end of this code
func (rs *appResource) login(w http.ResponseWriter, r *http.Request) {
var user User
var login Login
d := json.NewDecoder(r.Body)
d.DisallowUnknownFields() // catch unwanted fields
err := d.Decode(&login)
if err != nil {
http.Error(w, err.Error(), http.StatusBadRequest)
return
}
if err != nil {
fmt.Println("can't decode JSON", err)
}
if login.Email == "" {
log.Println("empty email")
return
}
log.Println(login.Email)
log.Println(login.Password)
if login.Password == "" {
log.Println("empty password")
return
}
// optional extra check
if d.More() {
http.Error(w, "extraneous data after JSON object", http.StatusBadRequest)
return
}
sqlStatement := "SELECT user_id, password FROM users WHERE active = 'true' AND email = ?"
row := rs.db.QueryRow(context.Background(), sqlStatement, login.Email)
err = row.Scan(&user.UserId, &user.Password)
if err == sql.ErrNoRows {
log.Println("user not found")
http.Error(w, err.Error(), http.StatusUnauthorized)
return
}
if err != nil {
log.Println(err)
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
It appears that you are doing something like the following:
func router() http.Handler {
db, err := pgxpool.ConnectConfig(context.Background(), config)
if err != nil {
log.Fatalln(err)
}
defer db.Close()
return appResource{db: db}
}
The issue with this is that the defer db.Close() runs when the function router() ends and this is before the returned pgxPool.Pool is actually used (the http.Handler returned will be used later when http requests are processed). Attempting to use a closed pgxPool.Pool results in the error you are seeing.
The simplest solution is to simply remove the defer db.Close() however you might also consider calling db.Close() as part of a clean shutdown process (it needs to remain open as long as you are handling requests).
You are using pgxpool which does differ from the standard library; however I believe that the advice given in the standard library docs applies here:
It is rarely necessary to close a DB.

pgxpool's config.AfterRelease is not firing

I have the following code:
func (s *SqlServerDatabase) ConnectPool() {
config, err := pgxpool.ParseConfig(s.Url)
if err != nil {
fmt.Println("config Database Fail")
fmt.Print(err)
}
config.AfterRelease = func(conn *pgx.Conn) bool {
fmt.Println("After Releasing")
return true
}
config.BeforeAquire = func(ctx context.Context, conn *pgx.Conn) bool {
fmt.Println("Before Aquiring")
return true
}
conn, err := pgxpool.ConnectConfig(context.Background(), config)
s.PoolConn = conn
}
func (s *SqlServerDatabase) PoolQuery(query string) (pgx.Rows, error) {
conn, err := s.PoolConn.Acquire(context.Background())
if err != nil {
log.Printf("Couldn't get a connection with the database. Reason %v", err)
} else {
// release the connection to the pool after using it
defer conn.Release()
results, err := conn.Query(context.Background(), query)
if err != nil {
log.Printf("Couldn't execute query. Reason %v", err)
} else {
// show the results boy, you got it.
fmt.Printf("%T\n", results)
}
return results, err
}
return nil, err
}
So :
I create the connection in ConnectPool. thats where I set up the config and connect.
Then In my query method I acquire one of the pools the BeforeAcquire method fires and prints.
But the AfterRelease method never does.
I release the connection in the deferred call so I'm not sure why its not running.
See the code at pgxpool/conn.go l.30 :
/*20*/ func (c *Conn) Release() {
...
/*30*/ if conn.IsClosed() || conn.PgConn().IsBusy() ||
conn.PgConn().TxStatus() != 'I' ||
(now.Sub(res.CreationTime()) > c.p.maxConnLifetime) {
res.Destroy()
return
}
/* 'c.afterRelease' is checked and used only after that block */
So : under some conditions, the connection is detroyed right away, and AfterRelease is indeed never executed.
note: the code link reflects the state of the master branch on 2021-05-05.

Error Handling within a for loop in Go results probably in a next iteration

I struggle with a specific Go implementation for sending log files to different locations:
package main
func isDestinationSIEM(json_msg string, json_obj *jason.Object, siem_keys []string) (bool) {
if json_obj != nil {
dest, err := json_obj.GetString("destination")
if err == nil {
if strings.Contains(dest,"SIEM") {
return true
}
}
for _, key := range siem_keys {
if strings.Contains(json_msg, key) {
return true
}
}
}
return false
}
func sendToSIEM(siem_dst string, json_msg string) (error) {
// Create connection to syslog server
roots := x509.NewCertPool()
ok := roots.AppendCertsFromPEM([]byte(rootPEM))
if !ok {
fmt.Println("failed to parse root certificate")
}
config := &tls.Config{RootCAs: roots, InsecureSkipVerify: true}
conn, err := tls.Dial("tcp", siem_dst, config)
if err != nil {
fmt.Println("Error connecting SIEM")
fmt.Println(err.Error())
} else {
// Send log message
_, err = fmt.Fprintf(conn, json_msg)
if err != nil {
fmt.Println("Error sending SIEM message: ", json_msg)
fmt.Println(err.Error())
}
}
defer conn.Close()
return err
}
func main() {
// simplified code otherwise there would have been too much
// but the 'devil' is this for loop
for _, obj := range objects {
// first check
isSIEM := isDestinationSIEM(obj, siem_keys)
if isSIEM {
err := sendToSIEM(obj)
if err != nil {
// print error
}
isAUDIT:= isDestinationSIEM(obj)
if isAUDIT {
err := sendToAUDIT(obj)
if err != nil {
// print error
}
} // end of for
}
When the 'if isSIEM' returns an error, the second check 'if isAUDIT' is not conducted.
Why is this? If an error is returned, does the loop start with the next iteration?
The error looks like this:
runtime error: invalid memory address or nil pointer dereference: errorString (which lists a couple of go packages)
The error looks like this: runtime error: invalid memory address or nil pointer dereference: errorString (which lists a couple of go packages)
It means you catch the panic() and your program has been stopped that means your circle for is stopped too.
Here details how works with panic https://blog.golang.org/defer-panic-and-recover

Resources