golang sql/database prepared statement in transaction - go

While I was reading the example of "Prepared" statement in "transaction" in golang SQL/database example. One of the line says, "danger", yet the code example was provided without an alternative.
I wanted to have more clear explanation on below query, as not much information was provided on Wiki page at - http://go-database-sql.org/prepared.html
tx, err := db.Begin()
if err != nil {
log.Fatal(err)
}
defer tx.Rollback()
stmt, err := tx.Prepare("INSERT INTO foo VALUES (?)")
if err != nil {
log.Fatal(err)
}
defer stmt.Close() // danger!
for i := 0; i < 10; i++ {
_, err = stmt.Exec(i)
if err != nil {
log.Fatal(err)
}
}
err = tx.Commit()
if err != nil {
log.Fatal(err)
}
// stmt.Close() runs here!
If you see in defer stmt.Close() it mentions, it's dangerous and yet not commented out for users to remove it.
Though I see no issue in above code as "defer" is going to run the code at the end but do they mean, above code is wrong and it should be replaced with below code or other better alternatives code.
tx, err := db.Begin()
if err != nil {
log.Fatal(err)
}
defer tx.Rollback()
stmt, err := tx.Prepare("INSERT INTO foo VALUES (?)")
if err != nil {
log.Fatal(err)
}
// Commented out below line.
// defer stmt.Close()
for i := 0; i < 10; i++ {
_, err = stmt.Exec(i)
if err != nil {
log.Fatal(err)
}
}
err = tx.Commit()
if err != nil {
log.Fatal(err)
}
// Comment removed from below line to close the stmt
stmt.Close()
I see no difference in both of the code above, yet, I need expert advice on above if there is any difference or If I am missing something.

a defer statement is a good way to make sure something runs no matter how you exit the function.
In this particular case, it seems to not matter, since all the error handlers use log.Fatal. If you replace the log.Fatals with return statements, and remove the defers, you now have to cleanup in many places:
tx, err := db.Begin()
if err != nil {
return nil,err
}
stmt, err := tx.Prepare("INSERT INTO foo VALUES (?)")
if err != nil {
tx.Rollback()
return nil,err
}
defer
for i := 0; i < 10; i++ {
_, err = stmt.Exec(i)
if err != nil {
tx.Rollback()
return nil,err
}
}
err = tx.Commit()
if err != nil {
stmt.Close()
tx.Rollback()
return nil,err
}
stmt.Close()
return someValue, nil
If you use defer, it is harder to forget one place you need to clean something up.

Related

How to write clean code without all these cascading error Christmas trees? [duplicate]

This question already has answers here:
Go Error Handling Techniques [closed]
(11 answers)
Closed 2 years ago.
I wrote a function that should do a simple thing:
look up for a specific address in a table and return the ID, if
already existing
if not, create a new record for this particular address
return the ID of this newly created record
As RDMS I use mysql here. I put everything in a transaction to avoid race conditions in my concurrent go-routines that makes calls to this function.
However, the tons of constant checks for err makes the code ugly and full test coverage hard to get.
Is there anything I can improve here in terms of better code quality?
func getAddressId(db *sql.DB, address string) (int64, error) {
tx, err := db.Begin()
if err != nil {
tx.Rollback()
return 0, err
}
stmt, err := tx.Prepare("SELECT id FROM address WHERE `address`=?")
if err != nil {
tx.Rollback()
return 0, err
}
defer stmt.Close()
var result sql.NullInt64
err = stmt.QueryRow(address).Scan(&result)
if err != nil && err != sql.ErrNoRows {
tx.Rollback()
return 0, err
}
if result.Valid {
tx.Commit()
return result.Int64, nil
}
stmt, err = tx.Prepare("INSERT INTO address (address) VALUES (?)")
if err != nil {
tx.Rollback()
return 0, err
}
var res sql.Result = nil
res, err = stmt.Exec(address)
if err != nil {
tx.Rollback()
return 0, err
}
tx.Commit()
var id int64 = 0
id, err = res.LastInsertId()
return id, err
}
First, and most importantly, there's very little wrong with the above code. There are a few pieces I'd adjust (and will below), but generally it is very clear, straightforward, and (almost) hard to get wrong. There is nothing ugly about that.
Second, see Error Handling and Go for thoughts on error handling Go, though I won't be using those techniques here because they're not necessary.
Now there is one thing that's a bit bad, which is that it's easy to forget to call tx.Rollback() or tx.Commit() in the right places. In my opinion, that's reasonable to fix (but it's really more style than substance). The below isn't tested.
// Name your return values so that we can use bare returns.
func getAddressId(db *sql.DB, address string) (id int64, err error) {
tx, err := db.Begin()
if err != nil {
return // This is a bare return. No need to write "0, err" everywhere.
}
// From this point on, if we exit with an error, then rollback, otherwise commit.
defer func() {
if err != nil {
tx.Rollback()
} else {
tx.Commit()
}
}()
stmt, err := tx.Prepare("SELECT id FROM address WHERE `address`=?")
if err != nil {
return
}
defer stmt.Close() // I'm not sure this is correct, because you reuse stmt
// This is purely style, but you can tighten up `err = ...; if err` logic like this:
var result sql.NullInt64
if err = stmt.QueryRow(address).Scan(&result); err != nil && err != sql.ErrNoRows {
return
}
if result.Valid {
id = result.Int64
return
}
if stmt, err = tx.Prepare("INSERT INTO address (address) VALUES (?)"); err != nil {
return
}
res, err := stmt.Exec(address)
if err != nil {
return
}
id = res.LastInsertId()
}
That said, I think this function is doing way too much, and if you break it up, it becomes easier to understand. For example (again, untested):
func getExistingAddressId(tx *sql.Tx, address string) (id int64, err error) {
stmt, err := tx.Prepare("SELECT id FROM address WHERE `address`=?")
if err != nil {
return
}
// I believe you need to close both statements, and splitting it up makes that clearer
defer stmt.Close()
var result sql.NullInt64
if err = stmt.QueryRow(address).Scan(&result); err != nil && err != sql.ErrNoRows {
return
}
// This is probably over-complicated. If !Valid, then .Int64 is 0.
if result.Valid {
return result.Int64, nil
}
return 0, nil
}
func insertNewAddress(tx *sql.Tx, address string) (id int64, err error) {
stmt, err := tx.Prepare("INSERT INTO address (address) VALUES (?)")
if err != nil {
return
}
defer stmt.Close()
res, err := stmt.Exec(address)
if err != nil {
return
}
return res.LastInsertId()
}
func getAddressId(db *sql.DB, address string) (id int64, err error) {
tx, err := db.Begin()
if err != nil {
return
}
defer func() {
if err != nil {
tx.Rollback()
} else {
tx.Commit()
}
}()
if id, err = getExistingAddressId(tx, address); err != nil || id != 0 {
return
}
return insertNewAddress(tx, address)
}
Using named return values like this is a matter of style, and you could certainly not do it that way and it would be just as clear. But the point (a) defer is a powerful way to avoid duplicating logic that must always run and (b) if a function becomes a mess of error handling, it probably is doing too much.
As a side note, I strongly suspect you could get rid of the Prepare calls here, would would simplify things significantly. You only use the Statements one time. If you cached that Statements and reused them, then it would make sense to Prepare them. If you do that, then the code simplifies to:
func getExistingAddressId(tx *sql.Tx, address string) (int64, error) {
var result sql.NullInt64
if err := tx.QueryRow("SELECT id FROM address WHERE `address`=?", address).
Scan(&result); err != nil && err != sql.ErrNoRows {
return 0, err
}
return result.Int64, nil
}
func insertNewAddress(tx *sql.Tx, address string) (int64, error) {
res, err := tx.Exec("INSERT INTO address (address) VALUES (?)", address)
if err != nil {
return 0, err
}
return res.LastInsertId()
}
func getAddressId(db *sql.DB, address string) (id int64, err error) {
tx, err := db.Begin()
if err != nil {
return 0, err
}
defer func() {
if err != nil {
tx.Rollback()
} else {
tx.Commit()
}
}()
if id, err = getExistingAddressId(tx, address); err != nil || id != 0 {
return
}
return insertNewAddress(tx, address)
}
Rather than trying to simplify Go syntax, this simplifies the operation, which as a side effect makes the syntax simpler.
A small subtlety that may go overlooked if you're not very familiar with named return values. In return insertNewAddress(...), the return value of the function call gets assigned to id and err before the defer runs, so the if err != nil check will correctly reflect the returned value. This can be a bit tricky, so you may prefer to write this all more explicitly, especially now that the function is so much shorter.
func getAddressId(db *sql.DB, address string) (int64, error) {
tx, err := db.Begin()
if err != nil {
return 0, err
}
var id Int64
id, err = getExistingAddressId(tx, address)
if err == nil && id == 0 {
id, err = insertNewAddress(tx, address)
}
if err != nil {
tx.Rollback()
return 0, err
}
tx.Commit()
return id, nil
}
And now the code is very straightforward, with no tricks, which IMO is Go at its best.

How to gently defer execution of a function that might return an error?

Most cleanup functions, especially those related to the IO operations, return an error, and normally we'd prefer to defer their execution in case if we'd not forget to call them when we're done with acquired resources. For example, at some point in the code we might write something like this:
var r *SomeResource
var err error
if r, err = Open(/* parameters */); err != nil {
return nil, err
}
defer r.Close() // This might return an error
It seems that if Close function returns an error, it'll be ignored. How can we gently process the returned error from such a function?
Using defer with a func() {}() like so.
var r *SomeResource
var err error
if r, err = Open(/* parameters */); err != nil {
return nil, err
}
defer func() {
if err = r.Close(); err != nil {
fmt.Printf("ERROR: %v", err)
}
}()
Fail gracefully with an error. Report the first error. Don't overwrite earlier errors. For example,
package main
import (
"fmt"
"os"
)
func demo() (name string, err error) {
filename := `test.file`
f, err := os.Open(filename)
if err != nil {
return "", err
}
defer func() {
e := f.Close()
if e != nil {
if err == nil {
err = e
}
}
}()
// do someting with the file
name = f.Name()
fi, err := f.Stat()
if err != nil {
return name, err
}
if fi.Size() == 0 {
err = fmt.Errorf("%s: empty file", filename)
return name, err
}
return name, err
}
func main() {
name, err := demo()
fmt.Println(name, err)
}
We can handle this in ways like:
way-1:
func myFn() error {
var err error
if r, err = Open(/* parameters */); err != nil {
return nil, err
}
defer func() {
if cErr = r.Close(); cErr != nil {
err = cErr
}
}()
return err
}
way-2:
func myFn() error {
var err error
if r, err = Open(/* parameters */); err != nil {
return nil, err
}
defer func() {
if cErr = r.Close(); cErr != nil {
// we can log the error
// or
// whatever we want to do
}
}()
return err
}
I have also find a nice blog on this topic, i mean handling error when defer func returns an error. Check here https://blog.learngoprogramming.com/5-gotchas-of-defer-in-go-golang-part-iii-36a1ab3d6ef1.

Why defer a Rollback?

I have started using Go for a web-service and have some database interaction (surprise!!!) and I have found this example:
tx, err := db.Begin()
if err != nil {
log.Fatal(err)
}
defer tx.Rollback()
stmt, err := tx.Prepare("INSERT INTO foo VALUES (?)")
if err != nil {
log.Fatal(err)
}
defer stmt.Close() // danger!
for i := 0; i < 10; i++ {
_, err = stmt.Exec(i)
if err != nil {
log.Fatal(err)
}
}
err = tx.Commit()
if err != nil {
log.Fatal(err)
}
// stmt.Close() runs here!
From http://go-database-sql.org/prepared.html
The example is well formulated an easy to understand. However, it leaves me with an unanswered question. Why defer the transaction Rollback call?
Why not just do the following:
err := tx.Commit()
if err != nil {
log.Error(err)
tx.Rollback()
}
Would defer tx.Rollback() not always attempt a rollback? Even if tx.Commit() was a success, or have I misunderstood something about defer?
The important thing is that if you defer tx.Rollback() you are sure that it will be called even if you do an early return and the "trick" is that calling tx.Rollback() on a committed transaction will not actually do the rollback, because once a transaction is committed, it's committed and there is no way to roll it back :)
So this is a neat trick on how to keep the code simple.
The example is a little bit misleading. It uses log.Fatal(err) for error handling. You wouldn't normally do that, and instead return err. So the deferred rollback is there to ensure that the transaction is rolled back in case of an early return.

golang scp file using crypto/ssh

I'm trying to download a remote file over ssh
The following approach works fine on shell
ssh hostname "tar cz /opt/local/folder" > folder.tar.gz
However the same approach on golang giving some difference in output artifact size. For example the same folders with pure shell produce artifact gz file 179B and same with go script 178B.
I assume that something has been missed from io.Reader or session got closed earlier. Kindly ask you guys to help.
Here is the example of my script:
func executeCmd(cmd, hostname string, config *ssh.ClientConfig, path string) error {
conn, _ := ssh.Dial("tcp", hostname+":22", config)
session, err := conn.NewSession()
if err != nil {
panic("Failed to create session: " + err.Error())
}
r, _ := session.StdoutPipe()
scanner := bufio.NewScanner(r)
go func() {
defer session.Close()
name := fmt.Sprintf("%s/backup_folder_%v.tar.gz", path, time.Now().Unix())
file, err := os.OpenFile(name, os.O_APPEND|os.O_WRONLY|os.O_CREATE, 0644)
if err != nil {
panic(err)
}
defer file.Close()
for scanner.Scan() {
fmt.Println(scanner.Bytes())
if err := scanner.Err(); err != nil {
fmt.Println(err)
}
if _, err = file.Write(scanner.Bytes()); err != nil {
log.Fatal(err)
}
}
}()
if err := session.Run(cmd); err != nil {
fmt.Println(err.Error())
panic("Failed to run: " + err.Error())
}
return nil
}
Thanks!
bufio.Scanner is for newline delimited text. According to the documentation, the scanner will remove the newline characters, stripping any 10s out of your binary file.
You don't need a goroutine to do the copy, because you can use session.Start to start the process asynchronously.
You probably don't need to use bufio either. You should be using io.Copy to copy the file, which has an internal buffer already on top of any buffering already done in the ssh client itself. If an additional buffer is needed for performance, wrap the session output in a bufio.Reader
Finally, you return an error value, so use it rather than panic'ing on regular error conditions.
conn, err := ssh.Dial("tcp", hostname+":22", config)
if err != nil {
return err
}
session, err := conn.NewSession()
if err != nil {
return err
}
defer session.Close()
r, err := session.StdoutPipe()
if err != nil {
return err
}
name := fmt.Sprintf("%s/backup_folder_%v.tar.gz", path, time.Now().Unix())
file, err := os.OpenFile(name, os.O_APPEND|os.O_WRONLY|os.O_CREATE, 0644)
if err != nil {
return err
}
defer file.Close()
if err := session.Start(cmd); err != nil {
return err
}
n, err := io.Copy(file, r)
if err != nil {
return err
}
if err := session.Wait(); err != nil {
return err
}
return nil
You can try doing something like this:
r, _ := session.StdoutPipe()
reader := bufio.NewReader(r)
go func() {
defer session.Close()
// open file etc
// 10 is the number of bytes you'd like to copy in one write operation
p := make([]byte, 10)
for {
n, err := reader.Read(p)
if err == io.EOF {
break
}
if err != nil {
log.Fatal("err", err)
}
if _, err = file.Write(p[:n]); err != nil {
log.Fatal(err)
}
}
}()
Make sure your goroutines are synchronized properly so output is completeky written to the file.

How to implement "tar cvfz xxx.tar.gz " in golang?

I would like to decompress tar-gz file by golang.
err := DecompressTarGz('xxx.tar.gz', '/Users/foobarbuzz/')
Use compress/gzip in combination with archive/tar or use os/exec to call tar and gzip directly if you don't like to implement all of that in Go.
I was looking for an answer to this one too. Implemented something that should work.
func Decompress(targetdir string, reader io.ReadCloser) error {
gzReader, err := gzip.NewReader(reader)
if err != nil {
return err
}
defer gzReader.Close()
tarReader := tar.NewReader(gzReader)
for {
header, err := tarReader.Next()
if err == io.EOF {
break
} else if err != nil {
return err
}
target := path.Join(targetdir, header.Name)
switch header.Typeflag {
case tar.TypeDir:
err = os.MkdirAll(target, os.FileMode(header.Mode))
if err != nil {
return err
}
setAttrs(target, header)
break
case tar.TypeReg:
w, err := os.Create(target)
if err != nil {
return err
}
_, err = io.Copy(w, tarReader)
if err != nil {
return err
}
w.Close()
setAttrs(target, header)
break
default:
log.Printf("unsupported type: %v", header.Typeflag)
break
}
}
return nil
}
func setAttrs(target string, header *tar.Header) {
os.Chmod(target, os.FileMode(header.Mode))
os.Chtimes(target, header.AccessTime, header.ModTime)
}

Resources