I have implemented a CSV export endpoint that is grabbing data from a database, row by row, and writing each line to BodyWriter. This must happen line by line, because the implementation needs to be conscious of memory consumption...
Here is a naive pseudo implementation:
rows, err := db.Query(q)
if err != nil {
w.WriteHeader(http.StatusInternalServerError) // <-- location 0
return
}
for rows.Next() {
if err := rows.Err(); err != nil {
// <-- location 1
return
}
r := result{}
if err := rows.Scan(&r); err != nil {
// <-- location 2
return
}
stringSlice := convertToStringSlice(r)
err := w.Write([]byte(line))
if err != nil {
// <-- location 3
return
}
}
return // <-- location 4
In location 0 - there is only one call to BodyWriter.WriteHeader so no problem.
In location 4 - I've already implicitly called BodyWriter.WriteHeader by calling BodyWriter.Write. Once I return, the BodyWriter is probably released and that is how (I assume) how the connection is closed (client gets EOF?).
But what if an error occurs after I've already written a few lines, in position 1/2/3? How do I differentiate this from the situation when we return in location 4?
I want to somehow notify the client that something went wrong, but the 200 status was already sent...
It also seems that Golang standard http library manages the connection internally and does not expose an easy way to manually close the TCP connection.
What is the best way to handle this situation?
use the context package
so if the frontend close the request you are not endup reading data and processing data from your database for no use
func test3Handler(w http.ResponseWriter, r *http.Request) {
rows, err := db.QueryContext(r.Context(), q)
}
the *http.Request have a context that is active while the request is active so if it gets an error that 'the context is closed' which means that the frontend dropped the request in the middle of the request
you can't update the `w.WriteHeader(200)` or call `w.WriteHeader(200)` after writing to the body it needs to be called before writing to the body,
you will need to send the error in that row, also even it didn'
for rows.Next() {
res := result{}
if err := rows.Scan(&res); err != nil {
// <-- location 2
fmt.Fprintf(w, "cant read row from the db error %v\n", err)
return
}
stringSlice, err := convertToStringSlice(r)
if err != nil {
fmt.Fprintf(w, "cant process row %+v, error %v\n", res, stringSlice)
return
}
fmt.Fprintln(w, stringSlice)
}
if you do want to return a 500 on a scan error, scan all rows to memory and if there is an error return an 500 else return all rows
func csvHandler(w http.ResponseWriter, r *http.Request) {
rows, err := db.QueryContext(r.Context(), q)
if err != nil {
w.WriteHeader(http.StatusInternalServerError)
return
}
var list []*result
for rows.Next() {
res := &result{}
if err := rows.Scan(res); err != nil {
// <-- location 2
w.WriteHeader(http.StatusInternalServerError)
return
}
list = append(list, res)
}
csvEnc := csv.NewWriter(w)
for _, res := range list {
err = csvEnc.Write(convertToStringSlice(res)...)
}
if err != nil {
// <-- location 3
// handle response writing error
return
}
return
}
this solution is using more memory than the previous example
I want to execute some redis commands atomically (HDel, SADD, HSet etc). I see the Watch feature in the go-redis to implement transactions , however since I am not going to modify the value of a key i.e use SET,GET etc , does it make sense to use Watch to execute it as transaction or just wrapping the commands in a TxPipeline would be good enough?
Approach 1 : Using Watch
func sampleTransaction() error{
transactionFunc := func(tx *redis.Tx) error {
// Get the current value or zero.
_, err := tx.TxPipelined(context.Background(), func(pipe redis.Pipeliner) error {
_, Err := tx.SAdd(context.Background(), "redis-set-key", "value1").Result()
if Err != nil {
return Err
}
_, deleteErr := tx.HDel(context.Background(), "redis-hash-key", "value1").Result()
if deleteErr != nil {
return deleteErr
}
return nil
})
return err
}
retries:=10
// Retry if the key has been changed.
for i := 0; i < retries; i++ {
fmt.Println("tries", i)
err := redisClient.Watch(context.Background(), transactionFunc())
if err == nil {
// Success.
return nil
}
if err == redis.TxFailedErr {
continue
}
return err
}
}
Approach 2: Just wrapping in TxPipelined
func sampleTransaction() error {
_, err:= tx.TxPipelined(context.Background(), func(pipe redis.Pipeliner) error {
_, Err := tx.SAdd(context.Background(), "redis-set-key", "value1").Result()
if Err != nil {
return Err
}
_, deleteErr := tx.HDel(context.Background(), "redis-hash-key", "value1").Result()
if deleteErr != nil {
return deleteErr
}
return nil
})
return err
}
As far as I know, pipelines do not guarantee atomicity. If you need atomicity, use lua.
https://pkg.go.dev/github.com/mediocregopher/radix.v3#NewEvalScript
I am trying to make a functionality which would work in the following manner:
As soon as the service function is called, it uses the Fetch function to get records from a service (which come in the form of byte array), JSON unmarshal the byte array, populate the struct and then send the struct to a DB function to save to database.
Now, since this needs to be a continuous job, I have added two if conditions such that, if the records received are of length 0, then we use the retry function to retry pulling the records, else we just write to the database.
I have been trying to debug the retry function for a while now, but it is just not working, and basically stops after the first retry (even though I specify the attempts as 100). What can I do to make sure, it keeps retrying pulling the records ?
The code is as Follows:
// RETRY FUNCTION
func retry(attempts int, sleep time.Duration, f func() error) (err error) {
for i := 0; ; i++ {
err = f()
if err == nil {
return
}
if i >= (attempts - 1) {
break
}
time.Sleep(sleep)
sleep *= 2
log.Println("retrying after error:", err)
}
return fmt.Errorf("after %d attempts, last error: %s", attempts, err) }
//Save Data function
type Records struct {
Messages [][]byte
}
func (s *Service) SaveData(records Records, lastSentPlace uint) error {
//lastSentPlace is sent as 0 to begin with.
for i := lastSentPlace; i <= records.Place-1; i++ {
var msg Records
msg.Unmarshal(records.Messages[i])
order := MyStruct{
Fruit: msg.Fruit,
Burger: msg.Burger,
Fries: msg.Fries,
}
err := s.db.UpdateOrder(context.TODO(), nil , order)
if err != nil {
logging.Error("Error occured...")
}
}return nil}
//Service function (This runs as a batch, which is why we need retrying)
func (s *Service) MyServiceFunction(ctx context.Context, place uint, length uint) (err error) {
var lastSentPlace = place
records, err := s.Poll(context.Background(), place, length)
if err != nil {
logging.Info(err)
}
// if no records found then retry.
if len(records.Messages) == 0 {
err = retry(100, 2*time.Minute, func() (err error) {
records, err := s.Poll(context.Background(), place, length)
// if data received, write to DB
if len(records.Messages) != 0 {
err = s.SaveData(records, lastSentPlace)
}
return
})
// if data is not received, or if err is not null, retry
if err != nil || len(records.Messages) == 0 {
log.Println(err)
return
}
// if data received on first try, then no need to retry, write to db
} else if len(records.Messages) >0 {
err = s.SaveData(records, lastSentPlace)
if err != nil {
return err
}
}
return nil }
I think, the issue is with the way I am trying to implement the retry function, I have been trying to debug this for a while, but being new to the language, I am really stuck. What I wanted to do was, implement a backoff if no records are found. Any help is greatly appreciated.
Thanks !!!
I make a simpler retry.
Use simpler logic for loop to ensure correctness.
We sleep before executing a retry, so use i > 0 as the condition for the sleeping.
Here's the code:
func retry(attempts int, sleep time.Duration, f func() error) (err error) {
for i := 0; i < attempts; i++ {
if i > 0 {
log.Println("retrying after error:", err)
time.Sleep(sleep)
sleep *= 2
}
err = f()
if err == nil {
return nil
}
}
return fmt.Errorf("after %d attempts, last error: %s", attempts, err)
}
I know this is an old question, but came across it when searching for retries and used it as the base of a solution.
This version can accept a func with 2 return values and uses generics in golang 1.18 to make that possible. I tried it in 1.17, but couldn't figure out a way to make the method generic.
This could be extended to any number of return values of any type. I have used any here, but that could be limited to a list of types.
func retry[T any](attempts int, sleep int, f func() (T, error)) (result T, err error) {
for i := 0; i < attempts; i++ {
if i > 0 {
log.Println("retrying after error:", err)
time.Sleep(time.Duration(sleep) * time.Second)
sleep *= 2
}
result, err = f()
if err == nil {
return result, nil
}
}
return result, fmt.Errorf("after %d attempts, last error: %s", attempts, err)
}
Usage example:
var config Configuration
something, err := retry(config.RetryAttempts, config.RetrySleep, func() (Something, error) { return GetSomething(config.Parameter) })
func GetSomething(parameter string) (something Something, err error) {
// Do something flakey here that might need a retry...
return something, error
}
Hope that helps someone with the same use case as me.
The function you are calling is using a context. So it is important that you handle that context.
If you don't know what a context is and how to use it, I would recomend that post: https://blog.golang.org/context
Your retry function should also handle the context. Just to get you on the track I give you a simple implementation.
func retryMyServiceFunction(ctx context.Context, place uint, length uint, sleep time.Duration) {
for {
select {
case ctx.Done():
return
default:
err := MyServiceFunction(ctx, place, length)
if err != nil {
log.Println("handle error here!", err)
time.Sleep(sleep)
} else {
return
}
}
}
}
I don't like the sleep part. So you should analyse the returned error. Also you have to think about timeouts. When you let your service sleep to long there could be a timeout.
There is a library for the retry mechanism.
https://github.com/avast/retry-go
url := "http://example.com"
var body []byte
err := retry.Do(
func() error {
resp, err := http.Get(url)
if err != nil {
return err
}
defer resp.Body.Close()
body, err = ioutil.ReadAll(resp.Body)
if err != nil {
return err
}
return nil
},
)
fmt.Println(body)
In the GoPlayground in the comments of the accepted answer, there are some things I would consider adding. Using continue and break in the for loop would make the loop even simpler by not using the if i > 0 { statement. Furthermore I would use early return in all the functions to directly return on an error. And last I would consistently use errors to check if a function failed or not, checking the validity of a value should be inside the executed function itself.
This would be my little attempt:
package main
import (
"errors"
"fmt"
"log"
"time"
)
func main() {
var complicatedFunctionPassing bool = false
var attempts int = 5
// if complicatedFunctionPassing is true retry just makes one try
// if complicatedFunctionPassing is false retry makes ... attempts
err := retry(attempts, time.Second, func() (err error) {
if !complicatedFunctionPassing {
return errors.New("somthing went wrong in the important function")
}
log.Println("Complicated function passed")
return nil
})
if err != nil {
log.Printf("failed after %d attempts with error: %s", attempts, err.Error())
}
}
func retry(attempts int, sleep time.Duration, f func() error) (err error) {
for i := 0; i < attempts; i++ {
fmt.Println("This is attempt number", i+1)
// calling the important function
err = f()
if err != nil {
log.Printf("error occured after attempt number %d: %s", i+1, err.Error())
log.Println("sleeping for: ", sleep.String())
time.Sleep(sleep)
sleep *= 2
continue
}
break
}
return err
}
You can try it out here:
https://go.dev/play/p/Ag8ObCb980U
So the docs here state specifically to use iterator.Done:
Next loads the next row into dst. Its return value is iterator.Done if there are no more results. Once Next returns iterator.Done, all subsequent calls will return iterator.Done.
However if I attempt to use Done it generates a compiler error. Indeed, Done is not defined on the RowIterator docs here.
My code (almost identical to the docs):
it, err := job.Read(ctx)
if err != nil {
fmt.Println(err)
}
for {
var rec MyType
err := it.Next(&rec)
// the docs say to use Done, but it provides an error
if err == it.Done {
break
}
if err != nil {
fmt.Println(err)
}
rows = append(rows, rec)
}
When I try to build it, I get:
./test.go:94:15: it.Done undefined (type *"cloud.google.com/go/bigquery".RowIterator has no field or method Done)
What am I missing?
iterator.Done is a variable defined in the iterator package. So replace it.Done with iterator.Done. This is shown in this example:
package main
import (
"cloud.google.com/go/bigquery"
"context"
"fmt"
"google.golang.org/api/iterator"
)
func main() {
ctx := context.Background()
client, err := bigquery.NewClient(ctx, "project-id")
if err != nil {
// TODO: Handle error.
}
q := client.Query("select name, num from t1")
it, err := q.Read(ctx)
if err != nil {
// TODO: Handle error.
}
for {
var row []bigquery.Value
err := it.Next(&row)
if err == iterator.Done {
break
}
if err != nil {
// TODO: Handle error.
}
fmt.Println(row)
}
}
You have confused between the iterator returned from the Job.Read call i.e. the RowIterator and the generic Google API iterator in https://pkg.go.dev/google.golang.org/api/iterator
You should check the return value from the latter i.e. err == iterator.Done to check if the the iteration is complete. The sample codes under the documentation has useful examples explained - https://github.com/GoogleCloudPlatform/golang-samples/tree/master/bigquery
im looking for the most efficient way to invoke couple of method
together.
Basically what im trying to to is invoke those method together and if something went wrong return error else return the struct Type.
This code is working but i can't get the struct type or error and im not sure if its the correct way.
go func()(struct,err) {
struct,err= sm.MethodA()//return struct type or error
err = sm.MethodB()//return error or nill
return struct,err
}()
In Go, it's idiomatic to return the two values and check for nil against the error
For example:
func myFunc(sm SomeStruct) (MyStruct, error) {
s, err := sm.MethodA()
if err != nil {
return nil, err
}
if err := sm.MethodB(); err != nil {
return nil, err
}
return s, nil
}
One thing to note, is that you're running your function in a goroutine. Any return value inside that goroutine won't be returned to your main goroutine.
In order to get the return values for that go routine you must use channels that will wait for the values.
In your case
errChan := make(chan error)
retChan := make(chan SomeStructType)
go func() {
myVal, err := sm.MethodA()
if err != nil {
errChan <- err
return
}
if err := sm.MethodB(); err != nil {
errChan <- err
return
}
retChan <- myVal
}()
select {
case err := <-errChan:
fmt.Println(err)
case val := <-retChan:
fmt.Printf("My value: %v\n", val)
}
You can mess around with it here to make more sense out of it:
http://play.golang.org/p/TtfFIZerhk