I am having trouble with scanning from a pgx query in Golang. The id field is always that of the last record. If I un-comment the var person Person declaration at the top of the function, every id is 3. There are 3 records with id's from 1 to 3 in my db. When I comment that declaration and declare the variable in the rows.Next() loop I get the correct id's. I can't figure out why the personId isn't being correctly overwritten
output from marshalled JSON with the var declared at the top of the function.
[{"person_id":3,"first_name":"Mark","last_name":"Brown"},{"person_id":3,"first_name":"Sam","last_name":"Smith"},{"person_id":3,"first_name":"Bob","last_name":"Jones"}]
output after declaring person every iteration of the scan loop
[{"person_id":1,"first_name":"Mark","last_name":"Brown"},{"person_id":2,"first_name":"Sam","last_name":"Smith"},{"person_id":3,"first_name":"Bob","last_name":"Jones"}]
I have this struct
// Person model
type Person struct {
PersonId *int64 `json:"person_id"`
FirstName *string `json:"first_name"`
LastName *string `json:"last_name"`
}
Here is my query function
func getPersons(rs *appResource, companyId int64) ([]Person, error) {
// var person Person
var persons []Person
queryString := `SELECT
user_id,
first_name,
last_name,
FROM users
WHERE company_id = $1`
rows, err := rs.db.Query(context.Background(), queryString, companyId)
if err != nil {
return persons, err
}
for rows.Next() {
var person Person
err = rows.Scan(
&person.PersonId,
&person.FirstName,
&person.LastName)
if err != nil {
return persons, err
}
log.Println(*person.PersonId) // 1, 2, 3 for both var patterns
persons = append(persons, person)
}
if rows.Err() != nil {
return persons, rows.Err()
}
return persons, err
}
I believe that you have discovered a bug (or, at least, unexpected behaviour) in github.com/jackc/pgx/v4. When running Scan it appears that if the pointer (so person.PersonId) is not nil then whatever it is pointing to is being reused. To prove this I replicated the issue and confirmed that you can also fix it with:
persons = append(persons, person)
person.PersonId = nil
I can duplicate the issue with this simplified code:
conn, err := pgx.Connect(context.Background(), "postgresql://user:password#127.0.0.1:5432/schema?sslmode=disable")
if err != nil {
panic(err)
}
defer conn.Close(context.Background())
queryString := `SELECT num::int FROM generate_series(1, 3) num`
var scanDst *int64
var slc []*int64
rows, err := conn.Query(context.Background(), queryString)
if err != nil {
panic(err)
}
for rows.Next() {
err = rows.Scan(&scanDst)
if err != nil {
panic(err)
}
slc = append(slc, scanDst)
// scanDst = nil
}
if rows.Err() != nil {
panic(err)
}
for _, i := range slc {
fmt.Printf("%v %d\n", i, *i)
}
The output from this is:
0xc00009f168 3
0xc00009f168 3
0xc00009f168 3
You will note that the pointer is the same in each case. I have done some further testing:
Uncommenting scanDst = nil in the above fixes the issue.
When using database/sql (with the "github.com/jackc/pgx/stdlib" driver) the code works as expected.
If PersonId is *string (and query uses num::text) it works as expected.
The issue appears to boil down to the following in convert.go:
if v := reflect.ValueOf(dst); v.Kind() == reflect.Ptr {
el := v.Elem()
switch el.Kind() {
// if dst is a pointer to pointer, strip the pointer and try again
case reflect.Ptr:
if el.IsNil() {
// allocate destination
el.Set(reflect.New(el.Type().Elem()))
}
return int64AssignTo(srcVal, srcStatus, el.Interface())
So this handles the case where the destination is a pointer (for some datatypes). The code checks if it is nil and, if so, creates a new instance of the relevant type as a destination. If it's not nil it just reuses the pointer. Note: I've not used reflect for a while so there may be issues with my interpretation.
As the behaviour differs from database/sql and is likely to cause confusion I believe it's probably a bug (I guess it could be an attempt to reduce allocations). I have had a quick look at the issues and could not find anything reported (will have a more detailed look later).
Related
I have database store function:
func (p *ProductsRep) FindAll(PageNumber int, PaginationSize int, Query string) []*postgresmodels.Product {
Also I have SQL query look like this:
SELECT * FROM table_name.
Then I want to concat conditional action like WHERE some_value=3 if some value (in this case Query) exists then I want to get SELECT * FROM table_name WHERE some_value=3.
I tried to use fmt.Sprintf to concat, or strings.Join, or bytes.Buffer.WriteString. But everytime I getting this error:
I replace real value for understanding:
pq: column "Some value" does not exist.
How can I do "adaptive" queries, which depends on inputed function values.
I believe you are trying to query rows in the database by using parameters.
You need to make sure you don't pass this data in as RAW values, due to the potential risk of SQL injection. You can make queries by using store procedures
You can use the function Query to pass in your query with your parameters. In the example case this is $1. If you wanted to you could add $2, $3... etc depending on how many parameters you wanted to query
Here is two examples
Postgres
using "github.com/jackc/pgx/v4" driver
ctx := context.Background()
type Bar struct {
ID int64
SomeValue string
}
rows, err := conn.Query(ctx, `SELECT * FROM main WHERE some_value=$1`, "foo")
if err != nil {
fmt.Println("ERRO")
panic(err) // handle error
}
defer rows.Close()
var items []Bar
for rows.Next() {
var someValue string
var id int64
if err := rows.Scan(&id, &someValue); err != nil {
log.Fatal(err) // handle error
}
item := Bar{
ID: id,
SomeValue: someValue,
}
items = append(items, item)
}
fmt.Println(items)
MySQL Driver
https://golang.org/pkg/database/sql/#DB.QueryRow
type Bar struct {
ID int64
SomeValue string
}
rows, err := conn.Query(`SELECT * FROM main WHERE some_value=$1`, "foo")
if err != nil {
fmt.Println("ERRO")
panic(err) // handle error
}
defer rows.Close()
var items []Bar
for rows.Next() {
var someValue string
var id int64
if err := rows.Scan(&id, &someValue); err != nil {
log.Fatal(err) // handle error
}
item := Bar{
ID: id,
SomeValue: someValue,
}
items = append(items, item)
}
fmt.Println(items)
Go here, using gorm to help me with database stuff.
I have the following function which is working for me:
func (d DbPersister) FetchOrderById(orderId string) (Order,error) {
order := &Order{}
if err := d.GormDB.Table("orders").
Select(`orders`.`order_id`,
`orders`.`quantity`,
`addresses`.`line_1`,
`addresses`.`state`,
`addresses`.`zip`).
Joins("join addresses on addresses.address_id = orders._address_id").
Where("orders.order_id = ?", orderId).
First(order).Error; err != nil {
return Order{}, err
}
return *order,nil
}
So, you give it an orderId, and it fetches that from the DB and maps it to an Order instance.
I now want to look up all a particular user's orders:
func (d DbPersister) FetchAllOrdersByCustomerId(userId string) ([]Order,error) {
orders := []&Order{}
if err := d.GormDB.Table("orders").
Select(`orders`.`order_id`,
`orders`.`quantity`,
`orders`.`status`,
`addresses`.`line_1`,
`addresses`.`state`,
`addresses`.`zip`).
Joins("join addresses on addresses.address_id = orders.shipping_address_id").
Joins("join users on users.user_id = orders.user_id").
Where("users.user_id = ?", userId).
First(orders).Error; err != nil {
return []Order{}, err
}
return orders,nil
}
However, I'm getting compiler errors. I don't believe First(orders) is the correct function to be calling here. Basically do a join between orders and users and give me all of a particular user's orders, which should be a list of Order instances. Can anyone spot where I'm going awry?
Firstly orders := []&Order{} should be orders := make([]Order,0)
Then use Find() instead of First() to get multiple values.
For structs, slices, maps, and other composite types, if no data is contained you can return nil
So your code should be
func (d DbPersister) FetchAllOrdersByCustomerId(userId string) ([]Order,error) {
orders := make([]Order,0)
if err := d.GormDB.Table("orders").
Select(`orders`.`order_id`,
`orders`.`quantity`,
`orders`.`status`,
`addresses`.`line_1`,
`addresses`.`state`,
`addresses`.`zip`).
Joins("join addresses on addresses.address_id = orders.shipping_address_id").
Joins("join users on users.user_id = orders.user_id").
Where("users.user_id = ?", userId).
Find(&orders).Error; err != nil {
return nil, err
}
return orders,nil
}
I still struggle understanding the benefits of prepared statement in Go / psql.
Let's assume I have a struct
type Brand struct {
Id int `json:"id,omitempty"`
Name string `json:"name,omitempty"`
Issued_at *time.Time `json:"issued_at,omitempty"`
}
And some table brands, where id is a unique field. Now I want to retrieve the element from that table using and id.
I can write the following function using QueryRow.
func GetBrand1(id int) (Brand, error) {
brand := Brand{}
if err := Db.QueryRow("SELECT name, issued_at FROM brands WHERE id = $1", id).Scan(&brand.Name, &brand.Issued_at); err != nil {
if err == sql.ErrNoRows {
return brand, nil
}
return brand, err
}
brand.Id = id
return brand, nil
}
and I can do the same (I hope it is the same) using prepared statement:
func GetBrand2(id int) (Brand, error) {
brand := Brand{}
stmt, err := Db.Prepare("SELECT name, issued_at FROM brands WHERE id = $1")
if err != nil {
return brand, err
}
defer stmt.Close()
rows, err := stmt.Query(id)
if err != nil {
return brand, err
}
defer rows.Close()
for rows.Next() {
rows.Scan(&brand.Name, &brand.Issued_at)
brand.Id = id
return brand, err
}
if err = rows.Err(); err != nil {
return brand, err
}
return brand, err
}
Now in my application I am planning to execute GetBrand* function many times (with different parameters). Will is one of this implementations is more preferable to another (in terms of sql-requests/memory/anything). Or may be they both suck and I would be better doing something else.
I have read this and a followed up link and I saw that:
db.Query() actually prepares, executes, and closes a prepared
statement. That’s three round-trips to the database. If you’re not
careful, you can triple the number of database interactions your
application makes
but I think that prepared statement in the second case will be removed at the end of the function.
In both of those examples, there's roughly the same database overhead. If you're going to use a statement a lot, prepare it once in a wider scope so it's reusable.
You would only be making one round trip to the database with that pattern.
If you're ever using databases in conjunction with user input, you should always prepare the statement beforehand.
If not, you run a risk of DB Insertion (SQL Insertion ex).
I have created an object mapping in Go that is not relational, it is very simple.
I have several structs that looks like this:
type Message struct {
Id int64
Message string
ReplyTo sql.NullInt64 `db:"reply_to"`
FromId int64 `db:"from_id"`
ToId int64 `db:"to_id"`
IsActive bool `db:"is_active"`
SentTime int64 `db:"sent_time"`
IsViewed bool `db:"is_viewed"`
Method string `db:"-"`
AppendTo int64 `db:"-"`
}
To create a new message I just run this function:
func New() *Message {
return &Message{
IsActive: true,
SentTime: time.Now().Unix(),
Method: "new",
}
}
And then I have a message_crud.go file for this struct that looks like this:
To find a message by a unique column (for example by id) I run this function:
func ByUnique(column string, value interface{}) (*Message, error) {
query := fmt.Sprintf(`
SELECT *
FROM message
WHERE %s = ?
LIMIT 1;
`, column)
message := &Message{}
err := sql.DB.QueryRowx(query, value).StructScan(message)
if err != nil {
return nil, err
}
return message, nil
}
And to save a message (insert or update in the database) I run this method:
func (this *Message) save() error {
s := ""
if this.Id == 0 {
s = "INSERT INTO message SET %s;"
} else {
s = "UPDATE message SET %s WHERE id=:id;"
}
query := fmt.Sprintf(s, sql.PlaceholderPairs(this))
nstmt, err := sql.DB.PrepareNamed(query)
if err != nil {
return err
}
res, err := nstmt.Exec(*this)
if err != nil {
return err
}
if this.Id == 0 {
lastId, err := res.LastInsertId()
if err != nil {
return err
}
this.Id = lastId
}
return nil
}
The sql.PlaceholderPairs() function looks like this:
func PlaceholderPairs(i interface{}) string {
s := ""
val := reflect.ValueOf(i).Elem()
count := val.NumField()
for i := 0; i < count; i++ {
typeField := val.Type().Field(i)
tag := typeField.Tag
fname := strings.ToLower(typeField.Name)
if fname == "id" {
continue
}
if t := tag.Get("db"); t == "-" {
continue
} else if t != "" {
s += t + "=:" + t
} else {
s += fname + "=:" + fname
}
s += ", "
}
s = s[:len(s)-2]
return s
}
But every time I create a new struct, for example a User struct I have to copy paste the "crud section" above and create a user_crud.go file and replace the words "Message" with "User", and the words "message" with "user". I repeat alot of code and it is not very dry. Is there something I could do to not repeat this code for things I would reuse? I always have a save() method, and always have a function ByUnique() where I can return a struct and search by a unique column.
In PHP this was easy because PHP is not statically typed.
Is this possible to do in Go?
Your ByUnique is almost generic already. Just pull out the piece that varies (the table and destination):
func ByUnique(table string, column string, value interface{}, dest interface{}) error {
query := fmt.Sprintf(`
SELECT *
FROM %s
WHERE %s = ?
LIMIT 1;
`, table, column)
return sql.DB.QueryRowx(query, value).StructScan(dest)
}
func ByUniqueMessage(column string, value interface{}) (*Message, error) {
message := &Message{}
if err := ByUnique("message", column, value, &message); err != nil {
return nil, err
}
return message, error
}
Your save is very similar. You just need to make a generic save function along the lines of:
func Save(table string, identifier int64, source interface{}) { ... }
Then inside of (*Message)save, you'd just call the general Save() function. Looks pretty straightforward.
Side notes: do not use this as the name of the object inside a method. See the link from #OneOfOne for more on that. And do not get obsessed about DRY. It is not a goal in itself. Go focuses on code being simple, clear, and reliable. Do not create something complicated and fragile just to avoid typing a simple line of error handling. This doesn't mean that you shouldn't extract duplicated code. It just means that in Go it is usually better to repeat simple code a little bit rather than create complicated code to avoid it.
EDIT: If you want to implement Save using an interface, that's no problem. Just create an Identifier interface.
type Ider interface {
Id() int64
SetId(newId int64)
}
func (msg *Message) Id() int64 {
return msg.Id
}
func (msg *Message) SetId(newId int64) {
msg.Id = newId
}
func Save(table string, source Ider) error {
s := ""
if source.Id() == 0 {
s = fmt.Sprintf("INSERT INTO %s SET %%s;", table)
} else {
s = fmt.Sprintf("UPDATE %s SET %%s WHERE id=:id;", table)
}
query := fmt.Sprintf(s, sql.PlaceholderPairs(source))
nstmt, err := sql.DB.PrepareNamed(query)
if err != nil {
return err
}
res, err := nstmt.Exec(source)
if err != nil {
return err
}
if source.Id() == 0 {
lastId, err := res.LastInsertId()
if err != nil {
return err
}
source.SetId(lastId)
}
return nil
}
func (msg *Message) save() error {
return Save("message", msg)
}
The one piece that might blow up with this is the call to Exec. I don't know what package you're using, and it's possible that Exec won't work correctly if you pass it an interface rather than the actual struct, but it probably will work. That said, I'd probably just pass the identifier rather than adding this overhead.
You probably want to use an ORM.
They eliminate a lot of the boilerplate code you're describing.
See this question for "What is an ORM?"
Here is a list of ORMs for go: https://github.com/avelino/awesome-go#orm
I have never used one myself, so I can't recommend any. The main reason is that an ORM takes a lot of control from the developer and introduces a non-negligible performance overhead. You need to see for yourself if they fit your use-case and/or if you are comfortable with the "magic" that's going on in those libraries.
I don't recommend doing this, i personally would prefer being explicit about scanning into structs and creating queries.
But if you really want to stick to reflection you could do:
func ByUnique(obj interface{}, column string, value interface{}) error {
// ...
return sql.DB.QueryRowx(query, value).StructScan(obj)
}
// Call with
message := &Message{}
ByUnique(message, ...)
And for your save:
type Identifiable interface {
Id() int64
}
// Implement Identifiable for message, etc.
func Save(obj Identifiable) error {
// ...
}
// Call with
Save(message)
The approach i use and would recommend to you:
type Redirect struct {
ID string
URL string
CreatedAt time.Time
}
func FindByID(db *sql.DB, id string) (*Redirect, error) {
var redirect Redirect
err := db.QueryRow(
`SELECT "id", "url", "created_at" FROM "redirect" WHERE "id" = $1`, id).
Scan(&redirect.ID, &redirect.URL, &redirect.CreatedAt)
switch {
case err == sql.ErrNoRows:
return nil, nil
case err != nil:
return nil, err
}
return &redirect, nil
}
func Save(db *sql.DB, redirect *Redirect) error {
redirect.CreatedAt = time.Now()
_, err := db.Exec(
`INSERT INTO "redirect" ("id", "url", "created_at") VALUES ($1, $2, $3)`,
redirect.ID, redirect.URL, redirect.CreatedAt)
return err
}
This has the advantage of using the type system and mapping only things it should actually map.
I have been doing some golang programming and usually has been lot of fun. Now I have this code I need to port from C# to go and it is just not working.
The idea is to fill a tree of employees from database but inner slices are not being filled up on each call.
Better to write the code here
func (db *DalBase) TitleAllChildren(tx *gorp.Transaction) (items []Title, err error) {
var dbChildren []entities.Title
_, err = tx.Select(&dbChildren, "select * from title where idparent is null order by name")
if err != nil {
return
}
items = make([]Title, 0)
for i := range dbChildren {
currItem := &dbChildren[i]
item := &Title{Id: currItem.Id, Name: currItem.Name}
err = db.TitleChildrenRecursive(tx, item)
if err != nil {
return
}
items = append(items, *item)
}
return
}
func (db *DalBase) TitleChildrenRecursive(tx *gorp.Transaction, u *Title) (err error) {
var dbChildren []entities.Title
_, err = tx.Select(&dbChildren, "select * from title where idparent = $1 order by name", u.Id)
if err != nil {
return
}
if len(dbChildren) != 0 {
u.Items = make([]Title, 0)
for i := range dbChildren {
currItem := &dbChildren[i]
item := &Title{Id: currItem.Id, Name: currItem.Name}
err = db.TitleChildrenRecursive(tx, item)
if err != nil {
return
}
u.Items = append(item.Items, *item)
}
}
return
}
I have logged the values on each recursive call and they are being filled inside the function, but when it bubbles up to the parent, slices are empty.
I wouldn't like to use pointers to slices, is it possible to implement?
Edit: here is the struct I am trying to fill
type Title struct {
Id string `json:"id"`
Name string `json:"name"`
Items []Title `json:"items"`
}
You won't need a pointer to a slice, as long you are passing around a pointer to the struct which contains your slice.
Each time you call TitleChildrenRecursive, you're replacing your slice with a new one before anything is appended:
u.Items = make([]Title, 0)
There's no need to make a new slice, since append works correctly with a nil slice.
You should also change []Title to []*Title, so that if any append operations happen to children items after they are added to the slice, it's reflected throughout the tree.