I'm building a query with multiple result sets, following the example given, here. Important bits copied below for convenience.
My question is, should rows.Err() be called after each for rows.Next() {...} loop, rather than once at the very end as shown in the example? Why / why not? My understanding is that rows.Err() will catch errors, "... encountered during iteration", whereas rows.NextResultSet() will catch, "... [errors] advancing to [the next result set]". Seems like I would want to catch errors during each iteration, no? Thanks!
rows, err := db.Query(q, age)
if err != nil {
log.Fatal(err)
}
defer rows.Close()
for rows.Next() {
var (
id int64
name string
)
if err := rows.Scan(&id, &name); err != nil {
log.Fatal(err)
}
fmt.Printf("id %d name is %s\n", id, name)
}
if !rows.NextResultSet() {
log.Fatal("expected more result sets", rows.Err())
}
var roleMap = map[int64]string{
1: "user",
2: "admin",
3: "gopher",
}
for rows.Next() {
var (
id int64
role int64
)
if err := rows.Scan(&id, &role); err != nil {
log.Fatal(err)
}
fmt.Printf("id %d has role %s\n", id, roleMap[role])
}
if err := rows.Err(); err != nil {
log.Fatal(err)
}
You can find from source code of Next or Err or NextResultSet function that there is lasterr member of Rows structure which can be set by several ways and Err always return it if it isn't EOF.
Those ways are:
Next gets some error, sets lasterr and always returns false (so for rows.Next() {... doesn't iterate more and it is enough to check rows.Err() only after the loop.
NextResultSet also sets lasterr whenever it gets error and also returns false.
So answer is yes, it should. Because if NextResultSet after Next returns false its error replaces error from Next.
Related
So the docs here state specifically to use iterator.Done:
Next loads the next row into dst. Its return value is iterator.Done if there are no more results. Once Next returns iterator.Done, all subsequent calls will return iterator.Done.
However if I attempt to use Done it generates a compiler error. Indeed, Done is not defined on the RowIterator docs here.
My code (almost identical to the docs):
it, err := job.Read(ctx)
if err != nil {
fmt.Println(err)
}
for {
var rec MyType
err := it.Next(&rec)
// the docs say to use Done, but it provides an error
if err == it.Done {
break
}
if err != nil {
fmt.Println(err)
}
rows = append(rows, rec)
}
When I try to build it, I get:
./test.go:94:15: it.Done undefined (type *"cloud.google.com/go/bigquery".RowIterator has no field or method Done)
What am I missing?
iterator.Done is a variable defined in the iterator package. So replace it.Done with iterator.Done. This is shown in this example:
package main
import (
"cloud.google.com/go/bigquery"
"context"
"fmt"
"google.golang.org/api/iterator"
)
func main() {
ctx := context.Background()
client, err := bigquery.NewClient(ctx, "project-id")
if err != nil {
// TODO: Handle error.
}
q := client.Query("select name, num from t1")
it, err := q.Read(ctx)
if err != nil {
// TODO: Handle error.
}
for {
var row []bigquery.Value
err := it.Next(&row)
if err == iterator.Done {
break
}
if err != nil {
// TODO: Handle error.
}
fmt.Println(row)
}
}
You have confused between the iterator returned from the Job.Read call i.e. the RowIterator and the generic Google API iterator in https://pkg.go.dev/google.golang.org/api/iterator
You should check the return value from the latter i.e. err == iterator.Done to check if the the iteration is complete. The sample codes under the documentation has useful examples explained - https://github.com/GoogleCloudPlatform/golang-samples/tree/master/bigquery
This question already has answers here:
Go Error Handling Techniques [closed]
(11 answers)
Closed 2 years ago.
I wrote a function that should do a simple thing:
look up for a specific address in a table and return the ID, if
already existing
if not, create a new record for this particular address
return the ID of this newly created record
As RDMS I use mysql here. I put everything in a transaction to avoid race conditions in my concurrent go-routines that makes calls to this function.
However, the tons of constant checks for err makes the code ugly and full test coverage hard to get.
Is there anything I can improve here in terms of better code quality?
func getAddressId(db *sql.DB, address string) (int64, error) {
tx, err := db.Begin()
if err != nil {
tx.Rollback()
return 0, err
}
stmt, err := tx.Prepare("SELECT id FROM address WHERE `address`=?")
if err != nil {
tx.Rollback()
return 0, err
}
defer stmt.Close()
var result sql.NullInt64
err = stmt.QueryRow(address).Scan(&result)
if err != nil && err != sql.ErrNoRows {
tx.Rollback()
return 0, err
}
if result.Valid {
tx.Commit()
return result.Int64, nil
}
stmt, err = tx.Prepare("INSERT INTO address (address) VALUES (?)")
if err != nil {
tx.Rollback()
return 0, err
}
var res sql.Result = nil
res, err = stmt.Exec(address)
if err != nil {
tx.Rollback()
return 0, err
}
tx.Commit()
var id int64 = 0
id, err = res.LastInsertId()
return id, err
}
First, and most importantly, there's very little wrong with the above code. There are a few pieces I'd adjust (and will below), but generally it is very clear, straightforward, and (almost) hard to get wrong. There is nothing ugly about that.
Second, see Error Handling and Go for thoughts on error handling Go, though I won't be using those techniques here because they're not necessary.
Now there is one thing that's a bit bad, which is that it's easy to forget to call tx.Rollback() or tx.Commit() in the right places. In my opinion, that's reasonable to fix (but it's really more style than substance). The below isn't tested.
// Name your return values so that we can use bare returns.
func getAddressId(db *sql.DB, address string) (id int64, err error) {
tx, err := db.Begin()
if err != nil {
return // This is a bare return. No need to write "0, err" everywhere.
}
// From this point on, if we exit with an error, then rollback, otherwise commit.
defer func() {
if err != nil {
tx.Rollback()
} else {
tx.Commit()
}
}()
stmt, err := tx.Prepare("SELECT id FROM address WHERE `address`=?")
if err != nil {
return
}
defer stmt.Close() // I'm not sure this is correct, because you reuse stmt
// This is purely style, but you can tighten up `err = ...; if err` logic like this:
var result sql.NullInt64
if err = stmt.QueryRow(address).Scan(&result); err != nil && err != sql.ErrNoRows {
return
}
if result.Valid {
id = result.Int64
return
}
if stmt, err = tx.Prepare("INSERT INTO address (address) VALUES (?)"); err != nil {
return
}
res, err := stmt.Exec(address)
if err != nil {
return
}
id = res.LastInsertId()
}
That said, I think this function is doing way too much, and if you break it up, it becomes easier to understand. For example (again, untested):
func getExistingAddressId(tx *sql.Tx, address string) (id int64, err error) {
stmt, err := tx.Prepare("SELECT id FROM address WHERE `address`=?")
if err != nil {
return
}
// I believe you need to close both statements, and splitting it up makes that clearer
defer stmt.Close()
var result sql.NullInt64
if err = stmt.QueryRow(address).Scan(&result); err != nil && err != sql.ErrNoRows {
return
}
// This is probably over-complicated. If !Valid, then .Int64 is 0.
if result.Valid {
return result.Int64, nil
}
return 0, nil
}
func insertNewAddress(tx *sql.Tx, address string) (id int64, err error) {
stmt, err := tx.Prepare("INSERT INTO address (address) VALUES (?)")
if err != nil {
return
}
defer stmt.Close()
res, err := stmt.Exec(address)
if err != nil {
return
}
return res.LastInsertId()
}
func getAddressId(db *sql.DB, address string) (id int64, err error) {
tx, err := db.Begin()
if err != nil {
return
}
defer func() {
if err != nil {
tx.Rollback()
} else {
tx.Commit()
}
}()
if id, err = getExistingAddressId(tx, address); err != nil || id != 0 {
return
}
return insertNewAddress(tx, address)
}
Using named return values like this is a matter of style, and you could certainly not do it that way and it would be just as clear. But the point (a) defer is a powerful way to avoid duplicating logic that must always run and (b) if a function becomes a mess of error handling, it probably is doing too much.
As a side note, I strongly suspect you could get rid of the Prepare calls here, would would simplify things significantly. You only use the Statements one time. If you cached that Statements and reused them, then it would make sense to Prepare them. If you do that, then the code simplifies to:
func getExistingAddressId(tx *sql.Tx, address string) (int64, error) {
var result sql.NullInt64
if err := tx.QueryRow("SELECT id FROM address WHERE `address`=?", address).
Scan(&result); err != nil && err != sql.ErrNoRows {
return 0, err
}
return result.Int64, nil
}
func insertNewAddress(tx *sql.Tx, address string) (int64, error) {
res, err := tx.Exec("INSERT INTO address (address) VALUES (?)", address)
if err != nil {
return 0, err
}
return res.LastInsertId()
}
func getAddressId(db *sql.DB, address string) (id int64, err error) {
tx, err := db.Begin()
if err != nil {
return 0, err
}
defer func() {
if err != nil {
tx.Rollback()
} else {
tx.Commit()
}
}()
if id, err = getExistingAddressId(tx, address); err != nil || id != 0 {
return
}
return insertNewAddress(tx, address)
}
Rather than trying to simplify Go syntax, this simplifies the operation, which as a side effect makes the syntax simpler.
A small subtlety that may go overlooked if you're not very familiar with named return values. In return insertNewAddress(...), the return value of the function call gets assigned to id and err before the defer runs, so the if err != nil check will correctly reflect the returned value. This can be a bit tricky, so you may prefer to write this all more explicitly, especially now that the function is so much shorter.
func getAddressId(db *sql.DB, address string) (int64, error) {
tx, err := db.Begin()
if err != nil {
return 0, err
}
var id Int64
id, err = getExistingAddressId(tx, address)
if err == nil && id == 0 {
id, err = insertNewAddress(tx, address)
}
if err != nil {
tx.Rollback()
return 0, err
}
tx.Commit()
return id, nil
}
And now the code is very straightforward, with no tricks, which IMO is Go at its best.
Given that the SQL is running perfectly in Query Editor. Still after assigning it to a struct, the data seems to have different values. Why is it like that?
var RunQuery = func(req *http.Request, query string)(*bigquery.RowIterator, error){
ctx := appengine.NewContext(req)
ctxWithDeadline, _ := context.WithTimeout(ctx, 30*time.Minute)
bqClient, bqErr := bigquery.NewClient(ctxWithDeadline, project, option.WithCredentialsFile(serviceAccount))
if bqErr != nil {
log.Errorf(ctx, "%v", bqErr)
return nil, bqErr
}
q := bqClient.Query(query)
job, err := q.Run(ctx)
if err != nil {
log.Errorf(ctx, "%v", err)
return nil, err
}
status, err := job.Wait(ctx)
if err != nil {
log.Errorf(ctx, "%v", err)
return nil, err
}
if err := status.Err(); err != nil {
log.Errorf(ctx, "%v", err)
return nil, err
}
it, err := job.Read(ctx)
if err != nil {
log.Errorf(ctx, "%v", err)
return nil, err
}
log.Infof(ctx, "Total Rows: %v", it.TotalRows)
return it, nil
}
type Customers struct {
CustomerName string `bigquery:"customer_name"`
CustomerAge int `bigquery:"customer_age"`
}
var rowsRead int
func main() {
query := `SELECT
name as customer_name,
age as customer_age
FROM customer_table
WHERE customerStatus = '0'`
customerInformation, customerInfoErr := RunQuery(req, query, false)
if customerInfoErr != nil {
log.Errorf(ctx, "Fetching customer information error :: %v", customerInfoErr)
return
}
for {
var row Customers
err := customerInformation.Next(&row)
log.Infof(ctx, "row %v", row)
if err == iterator.Done {
log.Infof(ctx, "ITERATION COMPLETE. Rows read %v", rowsRead)
break
}
rowsRead++
}
}
Let's say i have Query Results of
customer_name|customer_age
cat | 2
dog | 3
horse | 10
But after assigning it to a struct the results was
customer_name|customer_age
"" | 2
dog | ""
"" | ""
Why is it like this? i even tested it on chunk where i set the limit to 1000, still the same results. But the query results in Query Editor is what i expect
Solved it using Value Loader bigquery.Value. Instead of using expected struct in mapping the query results. used map[string]bigquery.Value. Still don't know why mapping query results with expected struct is not working perfectly. Here is my solution.
for {
row := make(map[string]bigquery.Value)
err := customerInformation.Next(&row)
log.Infof(ctx, "row %v", row)
if err == iterator.Done {
log.Infof(ctx, "ITERATION COMPLETE. Rows read %v", rowsRead)
break
}
rowsRead++
}
From the documentation:
If dst is a pointer to a struct, each column in the schema will be matched with an exported field of the struct that has the same name, ignoring the case. Unmatched schema columns and struct fields will be ignored.
cloud.google.com/go/bigquery
Here you try to resolve customer_age to a struct property named CustomerAge. If you update it to Customer_Age or customer_age it should work.
I am querying a mysql database in a GO function and want to return key value pairs in a map but can't quite figure out how to accomplish this. So far I have this function:
func GetData(callIds []string) map[string]Records {
//db insert
db, err := sql.Open("mysql", mySql)
if err != nil {
fmt.Printf(err.Error())
}
defer db.Close()
//db query
var foo string
err = db.QueryRow("select foo from bardata where callId = %v", 1).Scan(&foo)
if err != nil {
fmt.Printf(err.Error())
}
fmt.Println(foo)
return nil
I want to return a map with the key being callId and value being foo for each row returned from the query.
First, you need to build up your query. As it is, you're not even using your function input. Since we have a variable number of arguments, we need to do a little work to construct the right number of placeholders:
query := `select callid, foo from bardata where callid in (` +
strings.Repeat(`?,`, len(callIds) - 1) + `?)`
then, execute with the values passed in:
rows, err := db.Query(query, callIds...)
if err != nil {
// handle it
}
defer rows.Close()
then collect the results:
ret := map[string]string{}
for rows.Next() {
var callid, foo string
err = rows.Scan(&callid, &foo)
if err != nil {
// handle it
}
ret[callid] = foo
}
return ret
Caveats:
This will cause a placeholder mismatch error if callIds is an empty slice. If that's possible, then you need to detect it and handle it separately (maybe by returning an error or an empty map — querying the DB shouldn't be necessary).
This returns a map[string]string where the values are whatever "foo" is. In your question you have the function returning a map[string]Records but there's no information about what a Records might be or how to fetch one.
You might want to handle sql.ErrNoRows differently from other errors.
package main
import (
"fmt"
"github.com/bobby96333/goSqlHelper"
)
func main(){
fmt.Println("hello")
conn,err :=goSqlHelper.MysqlOpen("user:password#tcp(127.0.0.1:3306)/dbname")
checkErr(err)
row,err := conn.QueryRow("select * from table where col1 = ? and col2 = ?","123","abc")
checkErr(err)
if *row==nil {
fmt.Println("no found row")
}else{
fmt.Printf("%+v",row)
}
}
func checkErr(err error){
if err!=nil {
panic(err)
}
}
output:
&map[col1:abc col2:123]
I am using go-sqlite3 to retrieve the number of rows with a column of a certain value:
query := "select count(notebook) from pages where notebook="
result, err := db.Query(fmt.Sprint(query, id))
Where id is passed to the function running the query.
How can I retrieve the count value from result?
This should work:
// Output will be stored here.
var output string
id := "1234"
// Prepare your query
query, err := db.Prepare("select count(notebook) from pages where notebook = ?")
if err != nil {
fmt.Printf("%s", err)
}
defer query.Close()
// Execute query using 'id' and place value into 'output'
err = query.QueryRow(id).Scan(&output)
// Catch errors
switch {
case err == sql.ErrNoRows:
fmt.Printf("No notebook with that ID.")
case err != nil:
fmt.Printf("%s", err)
default:
fmt.Printf("Counted %s notebooks\n", output)
}