i have function:
func write() {
defer func() {
serverConn.Close()
}()
for message := range msgChan {
w, err := serverConn.NextWriter(websocket.TextMessage)
if err != nil {
return
}
bmessage, err := json.Marshal(message)
if err != nil {
return
}
_, err = w.Write(bmessage)
if err != nil {
fmt.Println(err)
}
if err := w.Close(); err != nil {
return
}
}
}
And i got
panic: concurrent write to websocket connection
I'm wondering how is this possible? in ws writes only this function, which is running in 1 instance. and second question: what is the point of using NextWriter instead of just conn.WriteMessage()? Is it possible that with a large number of messages NextWriter accumulate and can try to write at the same time?
This question already has answers here:
Go Error Handling Techniques [closed]
(11 answers)
Closed 2 years ago.
I wrote a function that should do a simple thing:
look up for a specific address in a table and return the ID, if
already existing
if not, create a new record for this particular address
return the ID of this newly created record
As RDMS I use mysql here. I put everything in a transaction to avoid race conditions in my concurrent go-routines that makes calls to this function.
However, the tons of constant checks for err makes the code ugly and full test coverage hard to get.
Is there anything I can improve here in terms of better code quality?
func getAddressId(db *sql.DB, address string) (int64, error) {
tx, err := db.Begin()
if err != nil {
tx.Rollback()
return 0, err
}
stmt, err := tx.Prepare("SELECT id FROM address WHERE `address`=?")
if err != nil {
tx.Rollback()
return 0, err
}
defer stmt.Close()
var result sql.NullInt64
err = stmt.QueryRow(address).Scan(&result)
if err != nil && err != sql.ErrNoRows {
tx.Rollback()
return 0, err
}
if result.Valid {
tx.Commit()
return result.Int64, nil
}
stmt, err = tx.Prepare("INSERT INTO address (address) VALUES (?)")
if err != nil {
tx.Rollback()
return 0, err
}
var res sql.Result = nil
res, err = stmt.Exec(address)
if err != nil {
tx.Rollback()
return 0, err
}
tx.Commit()
var id int64 = 0
id, err = res.LastInsertId()
return id, err
}
First, and most importantly, there's very little wrong with the above code. There are a few pieces I'd adjust (and will below), but generally it is very clear, straightforward, and (almost) hard to get wrong. There is nothing ugly about that.
Second, see Error Handling and Go for thoughts on error handling Go, though I won't be using those techniques here because they're not necessary.
Now there is one thing that's a bit bad, which is that it's easy to forget to call tx.Rollback() or tx.Commit() in the right places. In my opinion, that's reasonable to fix (but it's really more style than substance). The below isn't tested.
// Name your return values so that we can use bare returns.
func getAddressId(db *sql.DB, address string) (id int64, err error) {
tx, err := db.Begin()
if err != nil {
return // This is a bare return. No need to write "0, err" everywhere.
}
// From this point on, if we exit with an error, then rollback, otherwise commit.
defer func() {
if err != nil {
tx.Rollback()
} else {
tx.Commit()
}
}()
stmt, err := tx.Prepare("SELECT id FROM address WHERE `address`=?")
if err != nil {
return
}
defer stmt.Close() // I'm not sure this is correct, because you reuse stmt
// This is purely style, but you can tighten up `err = ...; if err` logic like this:
var result sql.NullInt64
if err = stmt.QueryRow(address).Scan(&result); err != nil && err != sql.ErrNoRows {
return
}
if result.Valid {
id = result.Int64
return
}
if stmt, err = tx.Prepare("INSERT INTO address (address) VALUES (?)"); err != nil {
return
}
res, err := stmt.Exec(address)
if err != nil {
return
}
id = res.LastInsertId()
}
That said, I think this function is doing way too much, and if you break it up, it becomes easier to understand. For example (again, untested):
func getExistingAddressId(tx *sql.Tx, address string) (id int64, err error) {
stmt, err := tx.Prepare("SELECT id FROM address WHERE `address`=?")
if err != nil {
return
}
// I believe you need to close both statements, and splitting it up makes that clearer
defer stmt.Close()
var result sql.NullInt64
if err = stmt.QueryRow(address).Scan(&result); err != nil && err != sql.ErrNoRows {
return
}
// This is probably over-complicated. If !Valid, then .Int64 is 0.
if result.Valid {
return result.Int64, nil
}
return 0, nil
}
func insertNewAddress(tx *sql.Tx, address string) (id int64, err error) {
stmt, err := tx.Prepare("INSERT INTO address (address) VALUES (?)")
if err != nil {
return
}
defer stmt.Close()
res, err := stmt.Exec(address)
if err != nil {
return
}
return res.LastInsertId()
}
func getAddressId(db *sql.DB, address string) (id int64, err error) {
tx, err := db.Begin()
if err != nil {
return
}
defer func() {
if err != nil {
tx.Rollback()
} else {
tx.Commit()
}
}()
if id, err = getExistingAddressId(tx, address); err != nil || id != 0 {
return
}
return insertNewAddress(tx, address)
}
Using named return values like this is a matter of style, and you could certainly not do it that way and it would be just as clear. But the point (a) defer is a powerful way to avoid duplicating logic that must always run and (b) if a function becomes a mess of error handling, it probably is doing too much.
As a side note, I strongly suspect you could get rid of the Prepare calls here, would would simplify things significantly. You only use the Statements one time. If you cached that Statements and reused them, then it would make sense to Prepare them. If you do that, then the code simplifies to:
func getExistingAddressId(tx *sql.Tx, address string) (int64, error) {
var result sql.NullInt64
if err := tx.QueryRow("SELECT id FROM address WHERE `address`=?", address).
Scan(&result); err != nil && err != sql.ErrNoRows {
return 0, err
}
return result.Int64, nil
}
func insertNewAddress(tx *sql.Tx, address string) (int64, error) {
res, err := tx.Exec("INSERT INTO address (address) VALUES (?)", address)
if err != nil {
return 0, err
}
return res.LastInsertId()
}
func getAddressId(db *sql.DB, address string) (id int64, err error) {
tx, err := db.Begin()
if err != nil {
return 0, err
}
defer func() {
if err != nil {
tx.Rollback()
} else {
tx.Commit()
}
}()
if id, err = getExistingAddressId(tx, address); err != nil || id != 0 {
return
}
return insertNewAddress(tx, address)
}
Rather than trying to simplify Go syntax, this simplifies the operation, which as a side effect makes the syntax simpler.
A small subtlety that may go overlooked if you're not very familiar with named return values. In return insertNewAddress(...), the return value of the function call gets assigned to id and err before the defer runs, so the if err != nil check will correctly reflect the returned value. This can be a bit tricky, so you may prefer to write this all more explicitly, especially now that the function is so much shorter.
func getAddressId(db *sql.DB, address string) (int64, error) {
tx, err := db.Begin()
if err != nil {
return 0, err
}
var id Int64
id, err = getExistingAddressId(tx, address)
if err == nil && id == 0 {
id, err = insertNewAddress(tx, address)
}
if err != nil {
tx.Rollback()
return 0, err
}
tx.Commit()
return id, nil
}
And now the code is very straightforward, with no tricks, which IMO is Go at its best.
Most cleanup functions, especially those related to the IO operations, return an error, and normally we'd prefer to defer their execution in case if we'd not forget to call them when we're done with acquired resources. For example, at some point in the code we might write something like this:
var r *SomeResource
var err error
if r, err = Open(/* parameters */); err != nil {
return nil, err
}
defer r.Close() // This might return an error
It seems that if Close function returns an error, it'll be ignored. How can we gently process the returned error from such a function?
Using defer with a func() {}() like so.
var r *SomeResource
var err error
if r, err = Open(/* parameters */); err != nil {
return nil, err
}
defer func() {
if err = r.Close(); err != nil {
fmt.Printf("ERROR: %v", err)
}
}()
Fail gracefully with an error. Report the first error. Don't overwrite earlier errors. For example,
package main
import (
"fmt"
"os"
)
func demo() (name string, err error) {
filename := `test.file`
f, err := os.Open(filename)
if err != nil {
return "", err
}
defer func() {
e := f.Close()
if e != nil {
if err == nil {
err = e
}
}
}()
// do someting with the file
name = f.Name()
fi, err := f.Stat()
if err != nil {
return name, err
}
if fi.Size() == 0 {
err = fmt.Errorf("%s: empty file", filename)
return name, err
}
return name, err
}
func main() {
name, err := demo()
fmt.Println(name, err)
}
We can handle this in ways like:
way-1:
func myFn() error {
var err error
if r, err = Open(/* parameters */); err != nil {
return nil, err
}
defer func() {
if cErr = r.Close(); cErr != nil {
err = cErr
}
}()
return err
}
way-2:
func myFn() error {
var err error
if r, err = Open(/* parameters */); err != nil {
return nil, err
}
defer func() {
if cErr = r.Close(); cErr != nil {
// we can log the error
// or
// whatever we want to do
}
}()
return err
}
I have also find a nice blog on this topic, i mean handling error when defer func returns an error. Check here https://blog.learngoprogramming.com/5-gotchas-of-defer-in-go-golang-part-iii-36a1ab3d6ef1.
I am creating a utility that needs to be aware of all the datasets/tables that exist in my BigQuery project. My current code for getting this information is as follows (using Go API):
func populateExistingTableMap(service *bigquery.Service, cloudCtx context.Context, projectId string) (map[string]map[string]bool, error) {
tableMap := map[string]map[string]bool{}
call := service.Datasets.List(projectId)
//call.Fields("datasets/datasetReference")
if err := call.Pages(cloudCtx, func(page *bigquery.DatasetList) error {
for _, v := range page.Datasets {
if tableMap[v.DatasetReference.DatasetId] == nil {
tableMap[v.DatasetReference.DatasetId] = map[string]bool{}
}
table_call := service.Tables.List(projectId, v.DatasetReference.DatasetId)
//table_call.Fields("tables/tableReference")
if err := table_call.Pages(cloudCtx, func(page *bigquery.TableList) error {
for _, t := range page.Tables {
tableMap[v.DatasetReference.DatasetId][t.TableReference.TableId] = true
}
return nil
}); err != nil {
return errors.New("Error Parsing Table")
}
}
return nil
}); err != nil {
return tableMap, err
}
return tableMap, nil
}
For a project with about 5000 datasets, each with up to 10 tables, this code takes almost 15 minutes to return. Is there a faster way to iterate through the names of all existing datasets/tables? I have tried using the Fields method to return only the fields I need (you can see those lines commented out above), but that results in only 50 (exactly 50) of my datasets being returned.
Any ideas?
Here is an updated version of my code, with concurrency, that reduced the processing time from about 15 minutes to 3 minutes.
func populateExistingTableMap(service *bigquery.Service, cloudCtx context.Context, projectId string) (map[string]map[string]bool, error) {
tableMap = map[string]map[string]bool{}
call := service.Datasets.List(projectId)
//call.Fields("datasets/datasetReference")
if err := call.Pages(cloudCtx, func(page *bigquery.DatasetList) error {
var wg sync.WaitGroup
wg.Add(len(page.Datasets))
for _, v := range page.Datasets {
if tableMap[v.DatasetReference.DatasetId] == nil {
tableMap[v.DatasetReference.DatasetId] = map[string]bool{}
}
go func(service *bigquery.Service, datasetID string, projectId string) {
defer wg.Done()
table_call := service.Tables.List(projectId, datasetID)
//table_call.Fields("tables/tableReference")
if err := table_call.Pages(cloudCtx, func(page *bigquery.TableList) error {
for _, t := range page.Tables {
tableMap[datasetID][t.TableReference.TableId] = true
}
return nil // NOTE: returning a non-nil error stops pagination.
}); err != nil {
// TODO: Handle error.
fmt.Println(err)
}
}(service, v.DatasetReference.DatasetId, projectId)
}
wg.Wait()
return nil // NOTE: returning a non-nil error stops pagination.
}); err != nil {
return tableMap, err
// TODO: Handle error.
}
return tableMap, nil
}
Currently using db.Update() to update the key-value in boltdb.
err := db.Update(func(tx *bolt.Tx) error {
b, err := tx.CreateBucket([]byte("widgets"))
if err != nil {
return err
}
if err := b.Put([]byte("foo"), []byte("bar")); err != nil {
return err
}
return nil
})
How to use db.Batch() operations using go routines?
Just call db.Batch() from your goroutines. Batch() was created to be used this way. There is an example in documentation.