Combinitorial Explosion You have lots of code that does almost the same thing.. but with tiny variations in data or behavior. This can be difficult to refactor-- perhaps using generics or an interpreter? - Jeff Atwood via Coding Horror
In this case it is not lots of code, but it is still bugging me. I have a shared problem, that is when trying to connect to an IP, if it fails, I should retry with the next IP.
I have one function which generates a producer for NSQ:
//Since we are in a critical system, we try with each IP until we get a producer
var err error
for i, success := 0, false; i < len(ips) && !success; i++ {
publisher, err = nsq.NewProducer(ips[i], nsq.NewConfig())
if err == nil {
success = true
}
}
The other function that almost shares the same code is one which takes a NSQ consumer and connects it:
var err error
for i, success := 0, false; i < len(ips) && !success; i++ {
err = consumer.ConnectToNSQD(ips[i])
if err == nil {
success = true
}
}
I would like to get rid of this almost repeated code without sacrificing legibility. Ideas?
You have it backwards. Your solution should follow the shape of the problem, not the shape of a particular solution. There's nothing in the solution that's worth refactoring. It's just going to add pointless complexity.
For example,
package main
import "github.com/nsqio/go-nsq"
// NewProducer is nsq.NewProducer with retries of an address list.
func NewProducer(addrs []string, config *nsq.Config) (producer *nsq.Producer, err error) {
if len(addrs) == 0 {
addrs = append(addrs, "")
}
for _, addr := range addrs {
producer, err = nsq.NewProducer(addr, config)
if err == nil {
break
}
}
return producer, err
}
// ConnectToNSQD is nsq.ConnectToNSQD with retries of an address list.
func ConnectToNSQD(c *nsq.Consumer, addrs []string) (err error) {
if len(addrs) == 0 {
addrs = append(addrs, "")
}
for _, addr := range addrs {
err = c.ConnectToNSQD(addr)
if err == nil {
break
}
}
return err
}
func main() {}
Maybe something like this?
var publisher *nsq.Producer
connectToWorkingIP(ips, func(ip string) error {
var err error
publisher, err = nsq.NewProducer(ip, nsq.NewConfig())
return err
})
connectToWorkingIP(ips, func(ip string) error {
return consumer.ConnectToNSQD(ip)
})
func connectToWorkingIP(ips []string, f func(string) error) {
for i, success := 0, false; i < len(ips) && !success; i++ {
err := f(ips[i])
if err == nil {
success = true
}
}
}
Related
I am trying to make a functionality which would work in the following manner:
As soon as the service function is called, it uses the Fetch function to get records from a service (which come in the form of byte array), JSON unmarshal the byte array, populate the struct and then send the struct to a DB function to save to database.
Now, since this needs to be a continuous job, I have added two if conditions such that, if the records received are of length 0, then we use the retry function to retry pulling the records, else we just write to the database.
I have been trying to debug the retry function for a while now, but it is just not working, and basically stops after the first retry (even though I specify the attempts as 100). What can I do to make sure, it keeps retrying pulling the records ?
The code is as Follows:
// RETRY FUNCTION
func retry(attempts int, sleep time.Duration, f func() error) (err error) {
for i := 0; ; i++ {
err = f()
if err == nil {
return
}
if i >= (attempts - 1) {
break
}
time.Sleep(sleep)
sleep *= 2
log.Println("retrying after error:", err)
}
return fmt.Errorf("after %d attempts, last error: %s", attempts, err) }
//Save Data function
type Records struct {
Messages [][]byte
}
func (s *Service) SaveData(records Records, lastSentPlace uint) error {
//lastSentPlace is sent as 0 to begin with.
for i := lastSentPlace; i <= records.Place-1; i++ {
var msg Records
msg.Unmarshal(records.Messages[i])
order := MyStruct{
Fruit: msg.Fruit,
Burger: msg.Burger,
Fries: msg.Fries,
}
err := s.db.UpdateOrder(context.TODO(), nil , order)
if err != nil {
logging.Error("Error occured...")
}
}return nil}
//Service function (This runs as a batch, which is why we need retrying)
func (s *Service) MyServiceFunction(ctx context.Context, place uint, length uint) (err error) {
var lastSentPlace = place
records, err := s.Poll(context.Background(), place, length)
if err != nil {
logging.Info(err)
}
// if no records found then retry.
if len(records.Messages) == 0 {
err = retry(100, 2*time.Minute, func() (err error) {
records, err := s.Poll(context.Background(), place, length)
// if data received, write to DB
if len(records.Messages) != 0 {
err = s.SaveData(records, lastSentPlace)
}
return
})
// if data is not received, or if err is not null, retry
if err != nil || len(records.Messages) == 0 {
log.Println(err)
return
}
// if data received on first try, then no need to retry, write to db
} else if len(records.Messages) >0 {
err = s.SaveData(records, lastSentPlace)
if err != nil {
return err
}
}
return nil }
I think, the issue is with the way I am trying to implement the retry function, I have been trying to debug this for a while, but being new to the language, I am really stuck. What I wanted to do was, implement a backoff if no records are found. Any help is greatly appreciated.
Thanks !!!
I make a simpler retry.
Use simpler logic for loop to ensure correctness.
We sleep before executing a retry, so use i > 0 as the condition for the sleeping.
Here's the code:
func retry(attempts int, sleep time.Duration, f func() error) (err error) {
for i := 0; i < attempts; i++ {
if i > 0 {
log.Println("retrying after error:", err)
time.Sleep(sleep)
sleep *= 2
}
err = f()
if err == nil {
return nil
}
}
return fmt.Errorf("after %d attempts, last error: %s", attempts, err)
}
I know this is an old question, but came across it when searching for retries and used it as the base of a solution.
This version can accept a func with 2 return values and uses generics in golang 1.18 to make that possible. I tried it in 1.17, but couldn't figure out a way to make the method generic.
This could be extended to any number of return values of any type. I have used any here, but that could be limited to a list of types.
func retry[T any](attempts int, sleep int, f func() (T, error)) (result T, err error) {
for i := 0; i < attempts; i++ {
if i > 0 {
log.Println("retrying after error:", err)
time.Sleep(time.Duration(sleep) * time.Second)
sleep *= 2
}
result, err = f()
if err == nil {
return result, nil
}
}
return result, fmt.Errorf("after %d attempts, last error: %s", attempts, err)
}
Usage example:
var config Configuration
something, err := retry(config.RetryAttempts, config.RetrySleep, func() (Something, error) { return GetSomething(config.Parameter) })
func GetSomething(parameter string) (something Something, err error) {
// Do something flakey here that might need a retry...
return something, error
}
Hope that helps someone with the same use case as me.
The function you are calling is using a context. So it is important that you handle that context.
If you don't know what a context is and how to use it, I would recomend that post: https://blog.golang.org/context
Your retry function should also handle the context. Just to get you on the track I give you a simple implementation.
func retryMyServiceFunction(ctx context.Context, place uint, length uint, sleep time.Duration) {
for {
select {
case ctx.Done():
return
default:
err := MyServiceFunction(ctx, place, length)
if err != nil {
log.Println("handle error here!", err)
time.Sleep(sleep)
} else {
return
}
}
}
}
I don't like the sleep part. So you should analyse the returned error. Also you have to think about timeouts. When you let your service sleep to long there could be a timeout.
There is a library for the retry mechanism.
https://github.com/avast/retry-go
url := "http://example.com"
var body []byte
err := retry.Do(
func() error {
resp, err := http.Get(url)
if err != nil {
return err
}
defer resp.Body.Close()
body, err = ioutil.ReadAll(resp.Body)
if err != nil {
return err
}
return nil
},
)
fmt.Println(body)
In the GoPlayground in the comments of the accepted answer, there are some things I would consider adding. Using continue and break in the for loop would make the loop even simpler by not using the if i > 0 { statement. Furthermore I would use early return in all the functions to directly return on an error. And last I would consistently use errors to check if a function failed or not, checking the validity of a value should be inside the executed function itself.
This would be my little attempt:
package main
import (
"errors"
"fmt"
"log"
"time"
)
func main() {
var complicatedFunctionPassing bool = false
var attempts int = 5
// if complicatedFunctionPassing is true retry just makes one try
// if complicatedFunctionPassing is false retry makes ... attempts
err := retry(attempts, time.Second, func() (err error) {
if !complicatedFunctionPassing {
return errors.New("somthing went wrong in the important function")
}
log.Println("Complicated function passed")
return nil
})
if err != nil {
log.Printf("failed after %d attempts with error: %s", attempts, err.Error())
}
}
func retry(attempts int, sleep time.Duration, f func() error) (err error) {
for i := 0; i < attempts; i++ {
fmt.Println("This is attempt number", i+1)
// calling the important function
err = f()
if err != nil {
log.Printf("error occured after attempt number %d: %s", i+1, err.Error())
log.Println("sleeping for: ", sleep.String())
time.Sleep(sleep)
sleep *= 2
continue
}
break
}
return err
}
You can try it out here:
https://go.dev/play/p/Ag8ObCb980U
I am trying to achieve some sort of multi-thread processing over here.
func (m *Map) Parse(mapData Node) error {
wg := &sync.WaitGroup{}
for _, node := range mapData.child {
wg.Add(1)
go parseChild(node, m, wg)
}
wg.Wait()
close(errors)
return nil
}
func parseChild(node Node, m *Map, wg *sync.WaitGroup) {
defer wg.Done()
var nodeType uint8
if err := binary.Read(node.data, binary.LittleEndian, &nodeType); err != nil {
errors <- err
}
if nodeType == OTBMNodeTowns {
for _, town := range node.child {
var nodeType uint8
if err := binary.Read(town.data, binary.LittleEndian, &nodeType); err != nil {
errors <- err
return
}
if nodeType != OTBMNodeTown {
errors <- fmt.Errorf("Parsing map towns: expected %v got %v", OTBMNodeTown, nodeType)
return
}
currentTown := Town{}
if err := binary.Read(town.data, binary.LittleEndian, ¤tTown.ID); err != nil {
errors <- err
return
} else if currentTown.Name, err = town.ReadString(); err != nil {
errors <- err
return
} else if currentTown.TemplePosition, err = town.ReadPosition(); err != nil {
errors <- err
return
}
m.Towns = append(m.Towns, currentTown)
errors <- fmt.Errorf("This should be called: %v, nodeType)
return
}
}
}
But my goroutine never sends anything to the errors channel. Seems to be that the main thread is not waiting for the goroutines to even finish
I have no idea what I am missing here. Im waiting for all routines to finish using wg.Wait but doesnt seem to be working as I think it should
And yes. the slice is populated with atleast 3 results. This is the errrors channel
var (
errors = make(chan error, 0)
)
func init() {
go errChannel()
}
func errChannel() {
for {
select {
case err := <-errors:
log.Println(err)
}
}
}
I am creating a utility that needs to be aware of all the datasets/tables that exist in my BigQuery project. My current code for getting this information is as follows (using Go API):
func populateExistingTableMap(service *bigquery.Service, cloudCtx context.Context, projectId string) (map[string]map[string]bool, error) {
tableMap := map[string]map[string]bool{}
call := service.Datasets.List(projectId)
//call.Fields("datasets/datasetReference")
if err := call.Pages(cloudCtx, func(page *bigquery.DatasetList) error {
for _, v := range page.Datasets {
if tableMap[v.DatasetReference.DatasetId] == nil {
tableMap[v.DatasetReference.DatasetId] = map[string]bool{}
}
table_call := service.Tables.List(projectId, v.DatasetReference.DatasetId)
//table_call.Fields("tables/tableReference")
if err := table_call.Pages(cloudCtx, func(page *bigquery.TableList) error {
for _, t := range page.Tables {
tableMap[v.DatasetReference.DatasetId][t.TableReference.TableId] = true
}
return nil
}); err != nil {
return errors.New("Error Parsing Table")
}
}
return nil
}); err != nil {
return tableMap, err
}
return tableMap, nil
}
For a project with about 5000 datasets, each with up to 10 tables, this code takes almost 15 minutes to return. Is there a faster way to iterate through the names of all existing datasets/tables? I have tried using the Fields method to return only the fields I need (you can see those lines commented out above), but that results in only 50 (exactly 50) of my datasets being returned.
Any ideas?
Here is an updated version of my code, with concurrency, that reduced the processing time from about 15 minutes to 3 minutes.
func populateExistingTableMap(service *bigquery.Service, cloudCtx context.Context, projectId string) (map[string]map[string]bool, error) {
tableMap = map[string]map[string]bool{}
call := service.Datasets.List(projectId)
//call.Fields("datasets/datasetReference")
if err := call.Pages(cloudCtx, func(page *bigquery.DatasetList) error {
var wg sync.WaitGroup
wg.Add(len(page.Datasets))
for _, v := range page.Datasets {
if tableMap[v.DatasetReference.DatasetId] == nil {
tableMap[v.DatasetReference.DatasetId] = map[string]bool{}
}
go func(service *bigquery.Service, datasetID string, projectId string) {
defer wg.Done()
table_call := service.Tables.List(projectId, datasetID)
//table_call.Fields("tables/tableReference")
if err := table_call.Pages(cloudCtx, func(page *bigquery.TableList) error {
for _, t := range page.Tables {
tableMap[datasetID][t.TableReference.TableId] = true
}
return nil // NOTE: returning a non-nil error stops pagination.
}); err != nil {
// TODO: Handle error.
fmt.Println(err)
}
}(service, v.DatasetReference.DatasetId, projectId)
}
wg.Wait()
return nil // NOTE: returning a non-nil error stops pagination.
}); err != nil {
return tableMap, err
// TODO: Handle error.
}
return tableMap, nil
}
I have a loop that is apparently causing a data race its near the bottom of this function and I will have it marked:
func (p *PartialParty) SendReadyCheck(party PartialParty) {
msg, err := json.Marshal(&ReadyCheckMsg{"ReadyCheck", ""})
if err != nil {
log.Println(err)
}
for _, member := range party.Members {
member.Conn.send <- msg
}
counter := 0
loopBreaker := true
for {
select {
case <-p.Accept:
counter++
resp, err := json.Marshal(&ReadyCheckMsg{"ReadyAccepted", ""})
if err != nil {
log.Println(err)
}
for _, member := range party.Members {
member.Conn.send <- resp
}
if counter == 2 {
// Create a new party with all members
partyid := PartyID(feeds.NewUUID().String())
db := common.Db()
newParty := &Party{
Active: true,
Members: p.Members,
Broadcast: make(chan []byte),
PartyID: partyid,
}
// Insert the new party into the database
_, err := db.Exec("INSERT INTO party SET party_id = ?, active = ?", partyid.String(), true)
if err != nil {
log.Println(err)
}
// Go through the members and update the database
var wg sync.WaitGroup
for _, member := range party.Members {
wg.Add(1)
m := member
go func() {
_, err := db.Exec("UPDATE party_members SET active = ? WHERE steamid = ?", false, m.SteamID)
if err != nil {
log.Println(err)
}
_, err = db.Exec("INSERT INTO party_members SET belongs_to =?, active = ?, steamid = ?", partyid.String(), true, m.SteamID)
if err != nil {
log.Println(err)
}
wg.Done()
}()
}
// Wait for all the database stuff to finish
wg.Wait()
PHub.AddNewParty(newParty)
loopBreaker = false
break
}
case conn := <-p.Decline:
if conn.Ready {
break
}
conn.Ready = false
conn.InQueue = false
conn.CurrentParty = ""
resp, err := json.Marshal(&ReadyCheckMsg{"ReadyCheckDeclined", ""})
if err != nil {
log.Println(err)
}
p.Accepting = true
identifier := conn.Identifier
if _, ok := party.Members[identifier]; ok {
delete(p.Members, identifier)
}
for _, m := range party.Members {
member := m
member.Conn.send <- resp
}
log.Println("Here")
loopBreaker = false
break
case <-time.After(30 * time.Second):
if counter == 2 {
return
}
p.Accepting = true
failedMsg, err := json.Marshal(&ReadyCheckMsg{"FailedToReady", ""})
if err != nil {
log.Println(err)
}
somebodyDeclinedMsg, err := json.Marshal(&ReadyCheckMsg{"ReadyCheckDeclined", ""})
if err != nil {
log.Println(err)
}
>>>> for _, member := range party.Members { ***<<<< This Line***
m := member
if !m.Conn.Ready {
m.Conn.Ready = false
m.Conn.InQueue = false
m.Conn.CurrentParty = ""
m.Conn.send <- failedMsg
} else {
m.Conn.Ready = false
m.Conn.send <- somebodyDeclinedMsg
}
}
loopBreaker = false
break
}
if !loopBreaker {
break
}
}
}
It is apparently conflicting with this:
// AddNewMember will add a new user to the party
func (p *PartyHub) AddNewMember(member *Member, partyid PartyID) {
p.Lock()
defer p.Unlock()
>>> p.PartialPartys[partyid].Members[member.Conn.Identifier] = member
}
type PartialParty struct {
Accepting bool
Members map[Identifier]*Member
Accept chan *Connection
Decline chan *Connection
PartyID PartyID
sync.Mutex
}
Right now it is impossible to AddNewMember if the goroutine SendReadyCheck is running ``because it is protected by an if statement that checks if the goroutine is running, so I'm not sure why they are saying they are racing each other. Any help on clearing this up would be great. I've tried setting a variable inside the loop to try to get away from it but it doesn't seem to cause it
Right now it is impossible to AddNewMember if the goroutine SendReadyCheck is running ``because it is protected by an if statement that checks if the goroutine is running
You didn't actually show that part of the code, but presumably it's not impossible. What if SendReadyCheck starts running after the if test but before AddNewMember does its modification?
Trying to understand the mentality of Go. I wrote the following function which looks for *.txt files of a folder that have a date in the filename, get the latest date and return that date.
func getLatestDate(path string) (time.Time, error) {
if fns, e := filepath.Glob(filepath.Join(path, "*.txt")); e == nil {
re, _ := regexp.Compile(`_([0-9]{8}).txt$`)
max := ""
for _, fn := range fns {
if ms := re.FindStringSubmatch(fn); ms != nil {
if ms[1] > max {
max = ms[1]
}
}
}
date, _ := time.Parse("20060102", max)
return date, nil
} else {
return time.Time{}, e
}
}
What would be the more idiomatic version of this function, if there is one?
Here is my take
Use MustCompile to compile a static regexp. This will panic if it doesn't compile and saves an error check
Hoist compiling the regexp out of the function - you only need it compiled once. Note that I've called it with a lowercase initial letter so it won't be visible outside the package.
Use an early return when checking errors - this saves indentation and is idiomatic go
Use named return parameters for those early returns - saves defining nil values for types and typing in general (not to everyone's taste though)
return time.Parse directly which checks the errors (you weren't before)
The code
var dateRe = regexp.MustCompile(`_([0-9]{8}).txt$`)
func getLatestDate(path string) (date time.Time, err error) {
fns, err := filepath.Glob(filepath.Join(path, "*.txt"))
if err != nil {
return
}
max := ""
for _, fn := range fns {
if ms := dateRe.FindStringSubmatch(fn); ms != nil {
if ms[1] > max {
max = ms[1]
}
}
}
return time.Parse("20060102", max)
}
Here's how I would have written it. Don't ignore errors, use guard clauses for error handling, and don't recompile regexps inside a loop.
var datePat = regexp.MustCompile(`_([0-9]{8}).txt$`)
func getLatestDate(path string) (time.Time, error) {
fns, err := filepath.Glob(filepath.Join(path, "*.txt"))
if err != nil {
return time.Time{}, err
}
max := time.Time{}
for _, fn := range fns {
if ms := re.FindStringSubmatch(fn); ms != nil {
if t, err := time.Parse("20060102", ms[1]); err == nil && t.After(max) {
max = t
}
}
}
return max, nil
}