I am working on a ETL type application and a large majority of the error handling I do is just retrying API requests until they succeed (they randomly fail on occasion due to connection etc.).
Therefore I have noticed a lot of code duplication that looks a lot like
for err != nil {
a,b,err = myfunc(c, d, e)
}
return a, b
So basically just keep doing the function until the error goes away (I put a sleep and other error checking as necessary as well to avoid rate limiting).
What I would like to do is simplify this to just one function that takes an arbitrary function, finds the error type in its output (if it has one) and runs it recursively until err!=nil. My problem is that although go seems to let you use any (interface{}) as an input it is not variadic on function definitions e.g.
(type func(a int) (int, int, error)) as the type func(...any) []any
I am wondering if this is impossible to do in go and if so any suggestions to get around it/get similar functionality more idiomatically.
Trying to test with something like this but the compiler does not like it.
func main() {
Deal(SometimesFail, 10)
}
func Deal(f func(...any) []any, inputs ...any) []any {
outputs := f(inputs)
for _, val := range outputs {
err, ok := val.(error)
if ok {
for err != nil {
outputs = Deal(f, inputs...)
}
return outputs
}
continue
}
return outputs
}
func SometimesFail(a int) (int, int, error) {
random := rand.Intn(a)
if random%2 == 0 {
return random, random, nil
} else {
return random, random, errors.New("error")
}
}
I guess what I could do to get around this is create a type for each function out/input scheme and allow the generic function to take any of these. This would keep the code duplication at a minimum while still achieving the goal.
The following: func(any), func(any, any), func(...any) are all different types and you can't assign one type to another. There is no single function type that would include all of them.
One way to work around this is to decouple function invocation (which must know the exact type of the function) from the retrial logic:
type result struct {
vals []any
err error
}
func main() {
vals := repeatUntilSuccess(func(inputs ...any) result {
val, err := failingRandomly(10)
return result{[]any{val}, err}
})
fmt.Println(vals)
}
func repeatUntilSuccess(fn func(...any) result) []any {
res := fn()
for res.err != nil {
res = fn()
}
return res.vals
}
func failingRandomly(i int) (int, error) {
random := rand.Intn(i)
if random%2 == 0 {
return random, nil
} else {
return random, errors.New("bad luck")
}
}
Related
package main
import (
"errors"
"fmt"
)
type myError struct{ err error }
func (e myError) Error() string { return e.err.Error() }
func new(msg string, args ...any) error {
return myError{fmt.Errorf(msg, args...)}
}
func (e myError) Unwrap() error { return e.err }
func (e myError) As(target any) bool { return errors.As(e.err, target) }
func (e myError) Is(target error) bool { return errors.Is(e.err, target) }
func isMyError(err error) bool {
target := new("")
return errors.Is(err, target)
}
func asMyError(err error) (error, bool) {
var target myError
ok := errors.As(err, &target)
return target, ok
}
func main() {
err := fmt.Errorf("wrap: %w", new("I am a myError"))
fmt.Println(isMyError(err))
fmt.Println(asMyError(err))
err = fmt.Errorf("wrap: %w", errors.New("I am not a myError"))
fmt.Println(isMyError(err))
fmt.Println(asMyError(err))
}
I expected
true
I am a myError true
false
I am not a myError false
but I got
false
I am a myError true
false
%!v(PANIC=Error method: runtime error: invalid memory address or nil pointer dereference) false
I tried to add
func (e myError) Unwrap() error { return e.err }
func (e myError) As(target any) bool { return errors.As(e.err, target) }
func (e myError) Is(target error) bool { return errors.Is(e.err, target) }
I tried
func asMyError(err error) (error, bool) {
target := &myError{} // was 'var target myError' before
ok := errors.As(err, &target)
return target, ok
}
I tried
func new(msg string, args ...any) error {
return &myError{fmt.Errorf(msg, args...)} // The change is the character '&'
}
but none of these changed anything. I also tried
func asMyError(err error) (error, bool) {
target := new("") // // was 'var target myError' or 'target := &myError{}' before
ok := errors.As(err, &target)
return target, ok
}
and then I got
false
wrap: I am a myError true
false
wrap: I am not a myError true
, which I guess makes sense but again does not solve my problem. I have a hard time to wrap my head this problem. Can you give me a hand?
So the point of errors.Is and errors.As is that errors can be wrapped, and these functions allow you to extract what the underlying cause of a given error was. This essentially relies on certain errors having specific error values. Take this as an example:
const (
ConnectionFailed = "connection error"
ConnectionTimedOut = "connection timed out"
)
type PkgErr struct {
Msg string
}
func (p PkgErr) Error() string {
return p.Msg
}
func getError(msg string) error {
return PkgErr{
Msg: msg,
}
}
func DoStuff() (bool, error) {
// stuff
err := getError(ConnectionFailed)
return false, fmt.Errorf("unable to do stuff: %w", err)
}
Then, in the caller, you can do something like this:
_, err := pkg.DoStuff()
var pErr pkg.PkgErr
if errors.As(err, &pErr) {
fmt.Printf("DoStuff failed with error %s, underlying error is: %s\n", err, pErr)
}
Or, if you only want to handle connection timeouts, but connection errors should instantly fail, you could do something like this:
accept := pkg.PkgErr{
Msg: pkg.ConnectionTimedOut,
}
if err := pkg.DoStuff(); err != nil {
if !errors.Is(err, accept) {
panic("Fatal: " + err.Error())
}
// handle timeout
}
There is, essentially, nothing you need to implement for the unwrap/is/as part. The idea is that you get a "generic" error back, and you want to unwrap the underlying error values that you know about, and you can/want to handle. If anything, at this point, the custom error type is more of a nuisance than an added value. The common way of using this wrapping/errors.Is thing is by just having your errors as variables:
var (
ErrConnectionFailed = errors.New("connection error")
ErrConnectionTimedOut = errors.New("connection timed out")
)
// then return something like this:
return fmt.Errorf("failed to do X: %w", ErrConnectionFailed)
Then in the caller, you can determine why something went wrong by doing:
if error.Is(err, pkg.ErrConnectionFailed) {
panic("connection is borked")
} else if error.Is(err, pkg.ErrConnectionTimedOut) {
// handle connection time-out, perhaps retry...
}
An example of how this is used can be found in the SQL packages. The driver package has an error variable defined like driver.ErrBadCon, but errors from DB connections can come from various places, so when interacting with a resource like this, you can quickly work out what went wrong by doing something like:
if err := foo.DoStuff(); err != nil {
if errors.Is(err, driver.ErrBadCon) {
panic("bad connection")
}
}
I myself haven't really used the errors.As all that much. IMO, it feels a bit wrong to return an error, and pass it further up the call stack to be handled depending on what the error exactly is, or even: extract an underlying error (often removing data), to pass it back up. I suppose it could be used in cases where error messages could contain sensitive information you don't want to send back to a client or something:
// dealing with credentials:
var ErrInvalidData = errors.New("data invalid")
type SanitizedErr struct {
e error
}
func (s SanitizedErr) Error() string { return s.e.Error() }
func Authenticate(user, pass string) error {
// do stuff
if !valid {
return fmt.Errorf("user %s, pass %s invalid: %w", user, pass, SanitizedErr{
e: ErrInvalidData,
})
}
}
Now, if this function returns an error, to prevent the user/pass data to be logged or sent back in any way shape or form, you can extract the generic error message by doing this:
var sanitized pkg.SanitizedErr
_ = errors.As(err, &sanitized)
// return error
return sanitized
All in all though, this has been a part of the language for quite some time, and I've not seen it used all that much. If you want your custom error types to implement an Unwrap function of sorts, though, the way to do this is really quite easy. Taking this sanitized error type as an example:
func (s SanitizedErr) Unwrap() error {
return s.e
}
That's all. The thing to keep in mind is that, at first glance, the Is and As functions work recursively, so the more custom types that you use that implement this Unwrap function, the longer the actual unwrapping will take. That's not even accounting for situations where you might end up with something like this:
boom := SanitizedErr{}
boom.e = boom
Now the Unwrap method will simply return the same error over and over again, which is just a recipe for disaster. The value you get from this is, IMO, quite minimal anyway.
Due to error handling in Go, I often end up with multiple values functions. So far, the way I have managed this has been very messy and I am looking for best practices to write cleaner code.
Let's say I have the following function:
type Item struct {
Value int
Name string
}
func Get(value int) (Item, error) {
// some code
return item, nil
}
How can I assign a new variable to item.Value elegantly. Before introducing the error handling, my function just returned item and I could simply do this:
val := Get(1).Value
Now I do this:
item, _ := Get(1)
val := item.Value
Isn't there a way to access directly the first returned variable?
In case of a multi-value return function you can't refer to fields or methods of a specific value of the result when calling the function.
And if one of them is an error, it's there for a reason (which is the function might fail) and you should not bypass it because if you do, your subsequent code might also fail miserably (e.g. resulting in runtime panic).
However there might be situations where you know the code will not fail in any circumstances. In these cases you can provide a helper function (or method) which will discard the error (or raise a runtime panic if it still occurs).
This can be the case if you provide the input values for a function from code, and you know they work.
Great examples of this are the template and regexp packages: if you provide a valid template or regexp at compile time, you can be sure they can always be parsed without errors at runtime. For this reason the template package provides the Must(t *Template, err error) *Template function and the regexp package provides the MustCompile(str string) *Regexp function: they don't return errors because their intended use is where the input is guaranteed to be valid.
Examples:
// "text" is a valid template, parsing it will not fail
var t = template.Must(template.New("name").Parse("text"))
// `^[a-z]+\[[0-9]+\]$` is a valid regexp, always compiles
var validID = regexp.MustCompile(`^[a-z]+\[[0-9]+\]$`)
Back to your case
IF you can be certain Get() will not produce error for certain input values, you can create a helper Must() function which would not return the error but raise a runtime panic if it still occurs:
func Must(i Item, err error) Item {
if err != nil {
panic(err)
}
return i
}
But you should not use this in all cases, just when you're sure it succeeds. Usage:
val := Must(Get(1)).Value
Go 1.18 generics update: Go 1.18 adds generics support, it is now possible to write a generic Must() function:
func Must[T any](v T, err error) T {
if err != nil {
panic(err)
}
return v
}
This is available in github.com/icza/gog, as gog.Must() (disclosure: I'm the author).
Alternative / Simplification
You can even simplify it further if you incorporate the Get() call into your helper function, let's call it MustGet:
func MustGet(value int) Item {
i, err := Get(value)
if err != nil {
panic(err)
}
return i
}
Usage:
val := MustGet(1).Value
See some interesting / related questions:
How to pass multiple return values to a variadic function?
Return map like 'ok' in Golang on normal functions
Yes, there is.
Surprising, huh? You can get a specific value from a multiple return using a simple mute function:
package main
import "fmt"
import "strings"
func µ(a ...interface{}) []interface{} {
return a
}
type A struct {
B string
C func()(string)
}
func main() {
a := A {
B:strings.TrimSpace(µ(E())[1].(string)),
C:µ(G())[0].(func()(string)),
}
fmt.Printf ("%s says %s\n", a.B, a.C())
}
func E() (bool, string) {
return false, "F"
}
func G() (func()(string), bool) {
return func() string { return "Hello" }, true
}
https://play.golang.org/p/IwqmoKwVm-
Notice how you select the value number just like you would from a slice/array and then the type to get the actual value.
You can read more about the science behind that from this article. Credits to the author.
No, but that is a good thing since you should always handle your errors.
There are techniques that you can employ to defer error handling, see Errors are values by Rob Pike.
ew := &errWriter{w: fd}
ew.write(p0[a:b])
ew.write(p1[c:d])
ew.write(p2[e:f])
// and so on
if ew.err != nil {
return ew.err
}
In this example from the blog post he illustrates how you could create an errWriter type that defers error handling till you are done calling write.
No, you cannot directly access the first value.
I suppose a hack for this would be to return an array of values instead of "item" and "err", and then just do
item, _ := Get(1)[0]
but I would not recommend this.
How about this way?
package main
import (
"fmt"
"errors"
)
type Item struct {
Value int
Name string
}
var items []Item = []Item{{Value:0, Name:"zero"},
{Value:1, Name:"one"},
{Value:2, Name:"two"}}
func main() {
var err error
v := Get(3, &err).Value
if err != nil {
fmt.Println(err)
return
}
fmt.Println(v)
}
func Get(value int, err *error) Item {
if value > (len(items) - 1) {
*err = errors.New("error")
return Item{}
} else {
return items[value]
}
}
Here's a generic helper function with assumption checking:
func assumeNoError(value interface{}, err error) interface{} {
if err != nil {
panic("error encountered when none assumed:" + err.Error())
}
return value
}
Since this returns as an interface{}, you'll generally need to cast it back to your function's return type.
For example, the OP's example called Get(1), which returns (Item, error).
item := assumeNoError(Get(1)).(Item)
The trick that makes this possible: Multi-values returned from one function call can be passed in as multi-variable arguments to another function.
As a special case, if the return values of a function or method g are equal in number and individually assignable to the parameters of another function or method f, then the call f(g(parameters_of_g)) will invoke f after binding the return values of g to the parameters of f in order.
This answer borrows heavily from existing answers, but none had provided a simple, generic solution of this form.
I am parsing a form and have written a number of functions func parseAndValidateX(val string) (T, err) where T is any type.
Now I would like to write a closure func catchError(T, Error) T, so that I can do something like:
errors []Error
func catchError(val T, err Error) T {
if err != nil {
//append err to errors
}
return val
}
data = MyStruct {
Age = catchError(parseAndValidateAge("5")) // Age is a int
DistanceFromHome = catchError(parseAndValidatePi("3.14")) // DistanceFromHome is a float
Location = catchError(parseAndValidatePi("3.14,2.0")) //Location is a custom Struct
}
if len(errors) > 0 {
// o, oh
}
Is this possible in Go? How can this be done easily/idiomatically?
Nope; you cannot do that since Go has no parametric polymorphism for user-defined functions. All you can do is take and return interface{} and add type assertions at the call sites.
Everything in Go is type-safe if you don't use the unsafe package, so you don't have to worry about that. A type assertion will fail at runtime instead of at compile-time, though.
If you are willing to violate DRY, though:
type errorList []Error
func (l *errorList) addIfNotNil(err Error) {
if err != nil {
*l = append(*l, err)
}
}
func (l *errorList) int(x int, err Error) int {
l.addIfNotNil(err)
return x
}
func (l *errorList) float32(x float32, err Error) float32 {
l.addIfNotNil(err)
return x
}
list := errorList([]Error{})
data := MyStruct{
Age: list.int(parseAndValidateAge("5")),
DistanceFromHome: list.float32(parseAndValidatePi("3.14")),
}
Is it possible to create a wrapper for arbitrary function in Go that would take the same arguments and return the same value?
I'm not talking about the wrapper that would look exactly the same, it may look differently, but it should solve the problem.
For example the problem might be to create a wrapper of arbitrary function that first looks for the result of the function call in cache and only in case of cache miss executes the wrapped function.
Here's a solution using reflect.MakeFunc. This particular solution assumes that your transformation function knows what to do with every different type of function. Watch this in action: http://play.golang.org/p/7ZM4Hlcqjr
package main
import (
"fmt"
"reflect"
)
type genericFunction func(args []reflect.Value) (results []reflect.Value)
// A transformation takes a function f,
// and returns a genericFunction which should do whatever
// (ie, cache, call f directly, etc)
type transformation func(f interface{}) genericFunction
// Given a transformation, makeTransformation returns
// a function which you can apply directly to your target
// function, and it will return the transformed function
// (although in interface form, so you'll have to make
// a type assertion).
func makeTransformation(t transformation) func(interface{}) interface{} {
return func(f interface{}) interface{} {
// g is the genericFunction that transformation
// produced. It will work fine, except that it
// takes reflect.Value arguments and returns
// reflect.Value return values, which is cumbersome.
// Thus, we do some reflection magic to turn it
// into a fully-fledged function with the proper
// type signature.
g := t(f)
// typ is the type of f, and so it will also
// be the type that of the function that we
// create from the transformation (that is,
// it's essentially also the type of g, except
// that g technically takes reflect.Value
// arguments, so we need to do the magic described
// in the comment above).
typ := reflect.TypeOf(f)
// v now represents the actual function we want,
// except that it's stored in a reflect.Value,
// so we need to get it out as an interface value.
v := reflect.MakeFunc(typ, g)
return v.Interface()
}
}
func main() {
mult := func(i int) int { return i * 2 }
timesTwo := func(f interface{}) genericFunction {
return func(args []reflect.Value) (results []reflect.Value) {
// We know we'll be getting an int as the only argument,
// so this type assertion will always succeed.
arg := args[0].Interface().(int)
ff := f.(func(int) int)
result := ff(arg * 2)
return []reflect.Value{reflect.ValueOf(result)}
}
}
trans := makeTransformation(timesTwo)
// Since mult multiplies its argument by 2,
// and timesTwo transforms functions to multiply
// their arguments by 2, f will multiply its
// arguments by 4.
f := trans(mult).(func(int) int)
fmt.Println(f(1))
}
The answer based on #joshlf13 idea and answer, but seems more simple to me.
http://play.golang.org/p/v3zdMGfKy9
package main
import (
"fmt"
"reflect"
)
type (
// Type of function being wrapped
sumFuncT func(int, int) (int)
// Type of the wrapper function
wrappedSumFuncT func(sumFuncT, int, int) (int)
)
// Wrapper of any function
// First element of array is the function being wrapped
// Other elements are arguments to the function
func genericWrapper(in []reflect.Value) []reflect.Value {
// this is the place to do something useful in the wrapper
return in[0].Call(in[1:])
}
// Creates wrapper function and sets it to the passed pointer to function
func createWrapperFunction(function interface {}) {
fn := reflect.ValueOf(function).Elem()
v := reflect.MakeFunc(reflect.TypeOf(function).Elem(), genericWrapper)
fn.Set(v)
}
func main() {
var wrappedSumFunc wrappedSumFuncT
createWrapperFunction(&wrappedSumFunc)
// The function being wrapped itself
sumFunc := func (a int, b int) int {
return a + b
}
result := wrappedSumFunc(sumFunc, 1, 3)
fmt.Printf("Result is %v", result)
}
The best I've come up with is to take a function def and return an interface, which will need type assertion afterwards:
func Wrapper(metaParams string, f func() (interface{}, string, error)) (interface{}, error) {
// your wrapper code
res, metaResults, err := f()
// your wrapper code
return res, err
}
Then to use this also takes a little work to function like a wrapper:
resInterface, err := Wrapper("data for wrapper", func() (interface{}, string, error) {
res, err := YourActualFuntion(whatever, params, needed)
metaResults := "more data for wrapper"
return res, metaResults, err
}) // note f() is not called here! Pass the func, not its results
if err != nil {
// handle it
}
res, ok := resInterface.(actualType)
if !ok {
// handle it
}
The upside is this is somewhat generic, can handle anything with 1 return type + error, and doesn't require reflection.
The downside is this takes a lot of work to use as it's not a simple wrapper or decorator.
Building on previous answers and using Go's new generic capabilities, I believe this can be implemented quite elegantly (playground link):
package main
import (
"fmt"
"reflect"
)
// Creates wrapper function and sets it to the passed pointer to function
func wrapFunction[T any](function T) T {
v := reflect.MakeFunc(reflect.TypeOf(function), func(in []reflect.Value) []reflect.Value {
// This is the place to intercept your call.
fmt.Println("Params are:", in)
f := reflect.ValueOf(function)
return f.Call(in)
})
return v.Interface().(T)
}
func main() {
// The function being wrapped itself
sum := func(a int, b int) int {
return a + b
}
wrapped := wrapFunction(sum)
fmt.Printf("Result is %v", wrapped(1, 3))
}
Like this?
var cache = make(map[string]string)
func doStuff(key string) {
//do-something-that-takes-a-long-time
cache[key] = value
return value
}
fun DoStuff(key string) {
if v, ok := cache[key]; ok {
return v
}
return doStuff(key)
}
Is there a more built-in wrapper to make a function that returns (X, error) successfully execute or abort, like regexp.MustCompile?
I'm talking about something like this, but more "built-in".
There is not. The best you'll get is something like this:
func Must(fn func() (interface{}, error)) interface{} {
v, err := fn()
if err != nil {
log.Fatalln(err)
}
return v
}
Then to use it:
Must(func() (interface{}, error) {
return template.ParseGlob(pattern)
}).(*template.Template)
Assuming that template.ParseGlob(pattern) is the call you wanted to wrap.
Go does not have parametric polymorphism, so this kind of code will end up requiring type assertions to restore the original type and so (in my opinion) is more effort than it's worth. The tidiest, idiomatic error handling you'll get for long chains of potential failure is simply to check for an error, and return it. Defer your cleanup handlers:
func MyFunc() (err error) {
a, err := blah1()
if err != nil {
return
}
defer a.Close()
b, err := blah2(a)
if err != nil {
return
}
defer b.Close()
// ad nauseam
}
Long and tedious, but at least it's explicit and easy to follow. Here are two modules I wrote that are crying out for parametric polymorphism that might give you some ideas for dealing without it:
bitbucket.org/anacrolix/dms/futures
bitbucket.org/anacrolix/dms/cache
Since Go 1.18 we can define typed Must instead of interface{}:
func Must[T any](obj T, err error) T {
if err != nil {
panic(err)
}
return obj
}
How to use: https://go.dev/play/p/ajQAjfro0HG
func success() (int, error) {
return 0, nil
}
func fail() (int, error) {
return -1, fmt.Errorf("Failed")
}
func main() {
n1 := Must(success())
fmt.Println(n1)
var n2 int = Must(fail())
fmt.Println(n2)
}
Must fails inside main, when fail() returns non-nil error
You can even define Mustn for more than 1 return parameter, e.g.
func Must2[T1 any, T2 any](obj1 T1, obj2 T2, err error) (T1, T2) {
if err != nil {
panic(err)
}
return obj1, obj2
}
I don't think a built-in mechanism would make sense since you could very well handle a non-nil error in various ways, as does the examples in the template package itself: see "text/template/examplefiles_test.go", illustrating 2 different usage of 'err':
// Here starts the example proper.
// T0.tmpl is the first name matched, so it becomes the starting template,
// the value returned by ParseGlob.
tmpl := template.Must(template.ParseGlob(pattern))
err := tmpl.Execute(os.Stdout, nil)
if err != nil {
log.Fatalf("template execution: %s", err)
}
// Output:
// T0 invokes T1: (T1 invokes T2: (This is T2))
In the particular case of the helper function (*Template) Must(), transforming an error into an exception (panic) isn't always the right course for all go programs (as debated in this thread), and to cover all the possible way to handle an error would mean to create a lot of "built-in" mechanisms.
I have encountered the same problem myself and decided to develop the following solution: https://github.com/boramalper/must
Example:
database := must.MV(sql.Open("sqlite3", "...")).(*sql.DB)
defer must.M(database.Close())
must.M(database.Ping())
// Use MustValVoid (or MVV shortly) if you don't care about
// the return value.
must.MVV(database.Exec(`
PRAGMA foreign_key_check;
...
`))
I am not sure why all the answers here are using the log package, when the
source itself uses panic:
func MustCompile(str string) *Regexp {
regexp, err := Compile(str)
if err != nil {
panic(`regexp: Compile(` + quote(str) + `): ` + err.Error())
}
return regexp
}
My recommendation would be instead of a generic Must wrapper, just implement
Must variants as needed in your code.
https://github.com/golang/go/blob/go1.16.5/src/regexp/regexp.go#L305-L314