Is there any way to get the rows that have been updated using the update command in Gorm, using a single operation.
I know this is like a million years old but for the sake of completion here's the Gorm way of doing it - clauses.
result := r.Gdb.Model(&User{}).Clauses(clause.Returning{}).Where("id = ?", "000-000-000").Updates(content)
Ref: Gorm Returning Data From Modified Rows
It's not pretty, but since you are using postgres you can do:
realDB := db.DB()
rows, err := realDB.Query("UPDATE some_table SET name = 'a' WHERE name = 'b' RETUNING id, name")
//you could probably do db.Raw but I'm not sure
if err != nil {
log.Fatal(err)
}
defer rows.Close()
for rows.Next() {
var id int
var name string
err := rows.Scan(&id, &name)
if err != nil {
log.Fatal(err)
}
log.Println(id, name)
}
This is a decent solution if you know the number of rows you're updating is relatively small (<1000)
var ids []int
var results []YourModel
// Get the list of rows that will be affected
db.Where("YOUR CONDITION HERE").Table("your_table").Select("id").Find(&ids)
query := db.Where("id IN (?)", ids)
// Do the update
query.Model(&YourModel{}).Updates(YourModel{field: "value"})
// Get the updated rows
query.Find(&results)
This is safe against race conditions since it uses the list of IDs to do the update instead of repeating the WHERE clause. As you can imagine this becomes a bit less practical when you start talking about thousands of rows.
Related
Essentially, using GORMDB, my current code looks something like this:
res = []*modelExample
DB.Model(&modelExample{}).
Order("task_id ").
Find(res)
And what I do with res is that I will manually loop through and append the models with the same task_id into one list, and then append this list to be worked on. The reason why I need to do this is because there are some specific operations i need to do on specific columns that I need to extract which I can't do in GORM.
However, is there a way to do this more efficiently where I return like a list of list, which I can then for loop and do my operation on each list element?
You should be able to achieve your needs with the following code snippet:
package main
import (
"fmt"
"gorm.io/driver/postgres"
"gorm.io/gorm"
)
type modelExample struct {
TaskId int
Name string
}
func main() {
dsn := "host=localhost user=postgres password=postgres dbname=postgres port=5432 sslmode=disable"
db, err := gorm.Open(postgres.Open(dsn), &gorm.Config{})
if err != nil {
panic(err)
}
db.AutoMigrate(&modelExample{})
// here you should populate the database with some data
// querying
res := make(map[int][]modelExample, 0)
rows, err := db.Table("model_examples").Select("task_id, name").Rows()
if err != nil {
panic(err)
}
defer rows.Close()
// scanning
for rows.Next() {
var taskId int
var name string
rows.Scan(&taskId, &name)
if _, isFound := res[taskId]; !isFound {
res[taskId] = []modelExample{{taskId, name}}
continue
}
res[taskId] = append(res[taskId], modelExample{taskId, name})
}
// always good idea to check for errors when scanning
if err = rows.Err(); err != nil {
panic(err)
}
for _, v := range res {
fmt.Println(v)
}
}
After the initial setup, let's take a closer look at the querying section.
First, you're going to get all the records from the table. The records you get are stored in the rows variable.
In the for loop, you scan all of the records. Each record will be either added as a new map entry or appended to an existing one (if the taskId is already present in the map).
This is the easiest way to create different lists based on a specific column (e.g. the TaskId). Actually, from what I understood, you need to split the records rather than grouping them with an aggregation function (e.g. COUNT, SUM, and so on).
The other code I added was just put in for clarity.
Let me know if this solves your issue or if you need something else, thanks!
I'm new to Go, so sorry for the silly question in advance!
I'm using Gin framework and want to make multiple queries to the database within the same handler (database/sql + lib/pq)
userIds := []int{}
bookIds := []int{}
var id int
/* Handling first query here */
rows, err := pgClient.Query(getUserIdsQuery)
defer rows.Close()
if err != nil {
return
}
for rows.Next() {
err := rows.Scan(&id)
if err != nil {
return
}
userIds = append(userIds, id)
}
/* Handling second query here */
rows, err = pgClient.Query(getBookIdsQuery)
defer rows.Close()
if err != nil {
return
}
for rows.Next() {
err := rows.Scan(&id)
if err != nil {
return
}
bookIds = append(bookIds, id)
}
I have a couple of questions regarding this code (any improvements and best practices would be appreciated)
Does Go properly handle defer rows.Close() in such a case? I mean I have reassignment of rows variable later down the code, so will compiler track both and properly close at the end of a function?
Is it ok to reuse id shared var or should I redeclare it while iterating within rows.Next() loop?
What's the better approach of having even more queries within one handler? Should I have some kind of Writer that accepts query and slice and populate it with ids retrieved?
Thanks.
I've never worked with go-pg library, and my answer is mostly focused on the other stuff, which are generic, and are not specific to golang or go-pg.
Regardless of the fact that the rows here has the same reference while being shared between 2 queries (so one rows.Close() call would suffice, unless the library has some special implementation), defining two variables is cleaner, like userRows and bookRows.
Although I already said that I have not worked with go-pg, I believe that you wont need to iterate through rows and scan the id for all the rows manually, I believe that the lib has provided some API like this (based on the quick look on the documentations):
userIds := []int{}
err := pgClient.Query(&userIds, "select id from users where ...", args...)
Regarding your second question, it depends on what you mean by "ok". Since your doing some synchronous iteration, I don't think it would result into bugs, but when it comes to coding style, personally, I wouldn't do this.
I think that the best thing to do in your case is this:
// repo layer
func getUserIds(args whatever) ([]int, err) {...}
// these can be exposed, based on your packaging logic
func getBookIds(args whatever) ([]int, err) {...}
// service layer, or wherever you want to aggregate both queries
func getUserAndBookIds() ([]int, []int, err) {
userIds, err := getUserIds(...)
// potential error handling
bookIds, err := getBookIds(...)
// potential error handling
return userIds, bookIds, nil // you have done err handling earlier
}
I think this code is easier to read/maintain. You won't face the variable reassignment and other issues.
You can take a look at the go-pg documentations for more details on how to improve your query.
Looking to retrieve table names from my postgresql database. Now, I know in Go you can use sql and the pq driver, but I'm using GORM for doing queries in my REST API.
The table_name type in PostgreSQL is "information_schema.sql_identifier". This is what I was trying to do, but the type isn't string.
var tables []string
if err := db.Table("information_schema.tables").Select("table_name").Where("table_schema = ?", "public").Find(&tables).Error; err != nil {
panic(err)
}
TL;DR
To select a single column values into a slice using Gorm, you can use db.Pluck helper:
var tables []string
if err := db.Table("information_schema.tables").Where("table_schema = ?", "public").Pluck("table_name", &tables).Error; err != nil {
panic(err)
}
TS;WM
Considering this, the SELECT statement returns a set of rows with one or more columns. In order to map those to Go code, we need a sort of struct so that Gorm can understand which column is mapped to which field of the struct. Even when you only select 1 single column, it's just a struct with 1 single field.
type Table struct {
TableName string
// more fields if needed...
}
So your output variable should be []*Table:
var tables []*Table
if err := db.Table("information_schema.tables").Select("table_name").Where("table_schema = ?", "public").Find(&tables).Error; err != nil {
panic(err)
}
Note: it could be []Table as well if you don't want to modify the element inside the slice.
If you don't want to define the struct, you can use the the db.Pluck function which is just a helper of this sort of code:
rows, err := db.Table("information_schema.tables").Select("table_name").Where("table_schema = ?", "public").Rows()
defer rows.Close()
var tables []string
var name string
for rows.Next() {
row.Scan(&name)
tables = append(tables, name)
}
I have a chapter table have about 2000000 rows, I want to to update each row for some specific conditions:
func main(){
rows, err := db.Query("SELECT id FROM chapters where title = 'custom_type'")
if err != nil {
panic(err)
}
for rows.Next() {
var id int
_ = rows.Scan(&id)
fmt.Println(id)
go updateRowForSomeReason(id)
}
}
func updateRowForSomeReason(id int) {
row, err := db.Query(fmt.Sprintf("SELECT id FROM chapters where parent_id = %v", id))
if err != nil {
panic(err) <----- // here is the panic occurs
}
for rows.Next() {
// ignore update code for simplify
}
}
Inside updateRowForSomeReason, I execute update statement for each row.
It will works with few seconds, after that the error will print:
323005
323057
323125
323244
323282
323342
323459
323498
323556
323618
323693
325343
325424
325468
325624
325816
326001
326045
326082
326226
326297
panic: sql: database is closed
This doesn't seem to be a Go problem as such, more a question of how to optimally structure your SQL within your code. You're taking a result set from executing a query on 2,000,000 rows:
rows, err := db.Query("SELECT id FROM chapters where title = 'custom_type'")
then executing another query for each row in this result set:
row, err := db.Query(fmt.Sprintf("SELECT id FROM chapters where parent_id = %v", id))
and then executing some more code for each of those, apparently one by one:
for rows.Next() {
// ignore update code for simplify
}
This is effectively two levels of nesting of statements, which is very inefficient way to load all these results into program memory and then execute independent UPDATE statements:
SELECT
+---->SELECT
+---->UPDATE
Instead, you could be doing all the work in the database itself, which would be much more efficient. You don't show what the UPDATE statement is, but this is the key part. Let's say you want to set a publish flag. You could do something like this:
UPDATE chapters
SET publish=true
WHERE parent_id in
(SELECT id FROM chapters
WHERE title='custom_type')
RETURNING id;
By using a nested query, you can combine all of the three separate queries into one single query. The database has all the information it needs to optimise the operation and build the most efficient query plan, and you are only executing a single db.Query operation. The RETURNING clause lets you retrieve a list of the ids that ended up being updated in the operation. So the code would be as simple as:
func main(){
rows, err := db.Query("UPDATE chapters SET publish=true WHERE parent_id in" +
"(SELECT id FROM chapters WHERE title='custom_type')" +
"RETURNING id;")
if err != nil {
panic(err)
}
for rows.Next() {
var id int
_ = rows.Scan(&id)
fmt.Println(id)
}
}
I have a go program connected to a bigquery table. This is the table's schema:
name STRING NULLABLE
age INTEGER NULLABLE
amount INTEGER NULLABLE
I have succeded at queryng the data of this table and printing all rows on console with this code:
ctx := context.Background()
client, err := bigquery.NewClient(ctx, projectID)
q := client.Query("SELECT * FROM test.test_user LIMIT 1000")
it, err := q.Read(ctx)
if err != nil {
log.Fatal(err)
}
for {
var values []bigquery.Value
err := it.Next(&values)
if err == iterator.Done {
break
}
if err != nil {
// TODO: Handle error.
}
fmt.Println(values)
}
And I also have succeded to insert data on the table from a struct using this code:
type test struct {
Name string
Age int
Amount int
}
u := client.Dataset("testDS").Table("test_user").Uploader()
savers := []*bigquery.StructSaver{
{Struct: test{Name: "Jack", Age: 23, Amount:123}, InsertID: "id1"},
}
if err := u.Put(ctx, savers); err != nil {
log.Fatal(err)
}
fmt.Printf("rows inserted!!")
Now, what I am failing to do is updating rows. What I want to do is selecting all the rows and update all of them with an operation (for example: amount = amount * 2)
How can I achieve this using golang?
Updating rows is not specific to Go, or any other client library. If you want to update data in BigQuery, you need to use DML (Data Manipulation Language) via SQL. So, essentially you already have the main part working (running a query) - you just need to change this SQL to use DML.
But, a word of caution: BigQuery is a OLAP service. Don't use it for OLTP. Also, there are quotas with using DML. Make sure you familiarise yourself with them.