If the .Read() method of a net.Conn returns an error, does this imply that future reads also will fail with an error? Or are there recoverable errors? If so, how do I know whether/when to retry reads?
In general, you're not going to have any errors from a conn.Read operation that can be retried. Most uses of the io.Reader interface will assume that all errors are final.
Any net package errors that are assured to be retry-able will conform to the net.Error interface, and expose a Temporary method.
This is most often used in an Accept loop, like this paraphrased example from the http package
for {
rw, e := l.Accept()
if e != nil {
if ne, ok := e.(net.Error); ok && ne.Temporary() {
if tempDelay == 0 {
tempDelay = 5 * time.Millisecond
} else {
tempDelay *= 2
}
if max := 1 * time.Second; tempDelay > max {
tempDelay = max
}
time.Sleep(tempDelay)
continue
}
return e
}
}
Any other possible cases need to be handled on an individual basis, with knowledge of the protocol and situation at hand.
Timeout is the only recoverable error on read from a net.TCPConn and that error will only be returned when a read deadline is set on the connection.
Use Error.Temporary() to check for errors that may resolve on retry and Error.Timeout() to check for timeouts:
n, err := c.Read(buf)
// process buf[:n] bytes
if e.(net.Error); ok && e.Timeout() && e.Temporary() {
// handle recoverable read deadline expiration
} else if err != nil {
// handle other errors
}
See the specific error type of the net package https://golang.org/pkg/net/#OpError
It provides a specific Temporary() method to figure out if it is a non-terminal error.
To manually figure out which error is Temporary you have to go through each defined error within the net, os and some other internal packages.
To programmatically check for a temporary error, you might declare your own local temporary interface{ Temporary() bool }, or you can rely on this interface provided by the net package https://golang.org/pkg/net/#Error
The OpError.Temporary method test if its internal error implements the net.temporary interface (https://golang.org/src/net/net.go?s=16056:16090#L501), and if so returns the result of the call to Temporary() of the internal error, for example https://golang.org/src/net/net.go?s=19122:19157#L605.
I am unsure which read retry you are thinking about, however, internal fd.read methods implements retries for eagain
https://golang.org/src/internal/poll/fd_unix.go#L165
Related
Everywhere I look, the "way" to "wrap" errors in Go is to use fmt.Errof with the %w verb
https://go.dev/blog/go1.13-errors
However, fmt.Errorf does not recursively wrap errors. There is no way to use it to wrap three previously defined errors (Err1, Err2, and Err3) and then check the result by using Is() and get true for each those three errors.
FINAL EDIT:
Thanks to #mkopriva's answer and comments below it, I now have a straightforward way to implement this (although, I am still curious if there is some standard type which does this). In the absence of an example, my attempts at creating one failed. The piece I was missing was adding an Is and As method to my type. Because the custom type needs to contain an error and a pointer to the next error, the custom Is and As methods allows us to compare the error contained in the custom type, rather than the custom type itself.
Here is a working example: https://go.dev/play/p/6BYGgIb728k
Highlights from the above link
type errorChain struct {
err error
next *errorChain
}
//These two functions were the missing ingredient
//Defined this way allows for full functionality even if
//The wrapped errors are also chains or other custom types
func (c errorChain) Is(err error) bool { return errors.Is(c.err, err) }
func (c errorChain) As(target any) bool { return errors.As(c.err, target) }
//Omitting Error and Unwrap methods for brevity
func Wrap(errs ...error) error {
out := errorChain{err: errs[0]}
n := &out
for _, err := range errs[1:] {
n.next = &errorChain{err: err}
n = n.next
}
return out
}
var Err0 = errors.New("error 0")
var Err1 = errors.New("error 1")
var Err2 = errors.New("error 2")
var Err3 = errors.New("error 3")
func main() {
//Check basic Is functionality
errs := Wrap(Err1, Err2, Err3)
fmt.Println(errs) //error 1: error 2: error 3
fmt.Println(errors.Is(errs, Err0)) //false
fmt.Println(errors.Is(errs, Err2)) //true
}
While the Go source specifically mentions the ability to define an Is method, the example does not implement it in a way that can solve my issue and the discussion do not make it immediately clear that it would be needed to utilize the recursive nature of errors.Is.
AND NOW BACK TO THE ORIGINAL POST:
Is there something built into Go where this does work?
I played around with making one of my own (several attempts), but ran into undesirable issues. These issues stem from the fact that errors in Go appear to be compared by address. i.e. if Err1 and Err2 point to the same thing, they are the same.
This causes me issues. I can naively get errors.Is and errors.As to work recursively with a custom error type. It is straightforward.
Make a type that implements the error interface (has an Error() string method)
The type must have a member that represents the wrapped error which is a pointer to its own type.
Implement an Unwrap() error method that returns the wrapped error.
Implement some method which wraps one error with another
It seems good. But there is trouble.
Since errors are pointers, if I make something like myWrappedError = Wrap(Err1, Err2) (in this case assume Err1 is being wrapped by Err2). Not only will errors.Is(myWrappedError, Err1) and errors.Is(myWrappedError, Err2) return true, but so will errors.Is(Err2, Err1)
Should the need arise to make myOtherWrappedError = Wrap(Err3, Err2) and later call errors.Is(myWrappedError, Err1) it will now return false! Making myOtherWrappedError changes myWrappedError.
I tried several approaches, but always ran into related issues.
Is this possible? Is there a Go library which does this?
NOTE: I am more interested in the presumably already existing right way to do this rather than the specific thing that is wrong with my basic attempt
Edit 3: As suggested by one of the answers, the issue in my first code is obviously that I modify global errors. I am aware, but failed to adequately communicate. Below, I will include other broken code which uses no pointers and modifies no globals.
Edit 4: slight modification to make it work more, but it is still broken
See https://go.dev/play/p/bSytCysbujX
type errorGroup struct {
err error
wrappedErr error
}
//...implemention Unwrap and Error excluded for brevity
func Wrap(inside error, outside error) error {
return &errorGroup{outside, inside}
}
var Err1 = errorGroup{errors.New("error 1"), nil}
var Err2 = errorGroup{errors.New("error 2"), nil}
var Err3 = errorGroup{errors.New("error 3"), nil}
func main() {
errs := Wrap(Err1, Err2)
errs = Wrap(errs, Err3)
fmt.Println(errs)//error 3: error 2: error 1
fmt.Println(errors.Is(errs, Err1)) //true
fmt.Println(errors.Is(errs, Err2)) //false <--- a bigger problem
fmt.Println(errors.Is(errs, Err3)) //false <--- a bigger problem
}
Edit 2: playground version shortened
See https://go.dev/play/p/swFPajbMcXA for an example of this.
EDIT 1: A trimmed version of my code focusing on the important parts:
type errorGroup struct {
err error
wrappedErr *errorGroup
}
//...implemention Unwrap and Error excluded for brevity
func Wrap(errs ...*errorGroup) (r *errorGroup) {
r = &errorGroup{}
for _, err := range errs {
err.wrappedErr = r
r = err
}
return
}
var Err0 = &errorGroup{errors.New("error 0"), nil}
var Err1 = &errorGroup{errors.New("error 1"), nil}
var Err2 = &errorGroup{errors.New("error 2"), nil}
var Err3 = &errorGroup{errors.New("error 3"), nil}
func main() {
errs := Wrap(Err1, Err2, Err3)//error 3: error 2: error 1
fmt.Println(errors.Is(errs, Err1)) //true
//Creating another wrapped error using the Err1, Err2, or Err3 breaks the previous wrap, errs.
_ = Wrap(Err0, Err2, Err3)
fmt.Println(errors.Is(errs, Err1)) //false <--- the problem
}
You can use something like this:
type errorChain struct {
err error
next *errorChain
}
func Wrap(errs ...error) error {
out := errorChain{err: errs[0]}
n := &out
for _, err := range errs[1:] {
n.next = &errorChain{err: err}
n = n.next
}
return out
}
func (c errorChain) Is(err error) bool {
return c.err == err
}
func (c errorChain) Unwrap() error {
if c.next != nil {
return c.next
}
return nil
}
https://go.dev/play/p/6oUGefSxhvF
Your code modifies package-global error values, so it is inherently broken. This defect has nothing to do with Go's error handling mechanics.
Per the documentation you linked, there are two error-handling helpers: Is, and As. Is lets you recursively unwrap an error, looking for a specific error value, which is necessarily a package global for this to be useful. As, on the other hand, lets you recursively unwrap an error looking for any wrapped error value of a given type.
How does wrapping work? You wrap error A in a new error value B. A Wrap() helper would necessarily return a new value, as fmt.Errorf does in the examples in the linked documentation. A Wrap helper should never modify the value of the error being wrapped. That value should be considered immutable. In fact, in any normal implementation, the value would be of type error, so that you can wrap any error, rather than just wrapping concentric values of your custom error type in each other; and, in that case, you have no access to the fields of the wrapped error to modify them anyway. Essentially, Wrap should be roughly:
func Wrap(err error) error {
return &errGroup{err}
}
And that's it. That's not very useful, because your implementation of errGroup doesn't really do anything - it provides no details about the error that occurred, it's just a container for other errors. For it to have value, it should have a string error message, or methods like some other error types' IsNotFound, or something that makes it more useful than just using error and fmt.Errorf.
Based on the usage in your example code, it also looks like you're presuming the use case is to say "I want to wrap A in B in C", which I've never seen in the wild and I cannot think of any scenario where that would be needed. The purpose of wrapping is to say "I've recieved error A, I'm going to wrap it in error B to add context, and return it". The caller might wrap that error in error C, and so on, which is what makes recursive wrapping valuable.
For example: https://go.dev/play/p/XeoONx19dgX
Instead of chaining/wrapping, you will "soon" (Go 1.20, as seen in Go 1.20-rc1 in Dec. 2022) be able to return a slice/tree of errors.
(In the meantime, mdobak/go-xerrors is a good alternative)
The release note explains:
Wrapping multiple errors
Go 1.20 expands support for error wrapping to permit an error to wrap
multiple other errors.
An error e can wrap more than one error by providing an Unwrap method
that returns a []error.
The errors.Is and errors.As functions have been updated to inspect
multiply wrapped errors.
The fmt.Errorf function now supports multiple occurrences of the %w
format verb, which will cause it to return an error that wraps all of
those error operands.
The new function errors.Join returns an error wrapping a list of
errors.
That comes from:
proposal: errors: add support for wrapping multiple errors
Background
Since Go 1.13, an error may wrap another by providing an Unwrap method returning the wrapped error.
The errors.Is and errors.As functions operate on chains of wrapped errors.
A common request is for a way to combine a list of errors into a single error.
Proposal
An error wraps multiple errors if its type has the method
Unwrap() []error
Reusing the name Unwrap avoids ambiguity with the existing singular Unwrap method.
Returning a 0-length list from Unwrap means the error doesn't wrap anything.
Callers must not modify the list returned by Unwrap.
The list returned by Unwrap must not contain any nil errors.
We replace the term "error chain" with "error tree".
The errors.Is and errors.As functions are updated to unwrap multiple errors.
Is reports a match if any error in the tree matches.
As finds the first matching error in a inorder preorder traversal of the tree.
The errors.Join function provides a simple implementation of a multierr.
It does not flatten errors.
// Join returns an error that wraps the given errors.
// Any nil error values are discarded.
// The error formats as the text of the given errors, separated by newlines.
// Join returns nil if errs contains no non-nil values.
func Join(errs ...error) error
The fmt.Errorf function permits multiple instances of the %w formatting verb.
The errors.Unwrap function is unaffected: It returns nil when called on an error with an Unwrap() []error method.
Why should this be in the standard library?
This proposal adds something which cannot be provided outside the standard library: Direct support for error trees in errors.Is and errors.As.
Existing combining errors operate by providing Is and As methods which inspect the contained errors, requiring each implementation to duplicate this logic, possibly in incompatible ways.
This is best handled in errors.Is and errors.As, for the same reason those functions handle singular unwrapping.
In addition, this proposal provides a common method for the ecosystem to use to represent combined errors, permitting interoperation between third-party implementations.
So far (Sept. 2022) this proposal seems a likely accept has been accepted!
CL 432575 starts the implementation.
There arr several approaches but there is one thing that you should keep in mind: if you have multiple errors, you may need to handle it as a slice of errors
For instance, imagine you need to check if all errors are the same, or there is at least one error of certain type you can use the snippet below.
You can extend this concept or use some existing library to handle multierrors
type Errors []error
func (errs Errors) String() string {
…
}
func (errs Errors) Any(target error) bool{
for _, err := range errs {
if errors.Is(err,target) {
return true
}
}
return false
}
func (errs Errors) All(target error) bool{
if len(errs) == 0 { return false }
for _, err := range errs {
if !errors.Is(err,target) {
return false
}
}
return true
}
I am trying to mock the below method using gomock
func (w *writer) Publish(vacancies []model.Vacancy) error {
...
if _, err = w.conn.WriteMessages(msg); err != nil {
return fmt.Errorf("failed to write message: %w", err)
}
Interface:
type Producer interface {
Publish(vacancies []model.Vacancy) error
Close() error
}
SuiteTest:
func (p *ProducerTestSuite) SetupTest() {
p.mockCtrl = gomock.NewController(p.T())
p.producer = NewMockProducer(p.mockCtrl)
writer, err := producer.NewWriter(context.Background(), scheduler.KafkaConf{Addr: "localhost:9092", Topic: "test"})
p.Require().NoError(err)
p.writer = writer
}
...
func (p *ProducerTestSuite) TestProducer_Publish() {
p.producer.EXPECT().Publish([]model.Vacancy{}).Return(nil)
p.Require().NoError(p.writer.Publish([]model.Vacancy{}))
}
mockgen:
//go:generate mockgen -package producer_test -destination mock_test.go -source ../kafka.go
When I try run test, I got this message:
=== RUN TestSuite/TestProducer_Publish
controller.go:137: missing call(s) to *producer_test.MockProducer.Publish(is equal to [] ([]storage.Vacancy)) /Users/...
controller.go:137: aborting test due to missing call(s)
Where I wrong?
It appears as if you are not calling the same thing that you are expecting on. Your expect is watching p.producer.Publish(), but your test code calls p.writer.Publish(). I cannot see any code here that would lead writer to call anything in producer.
The following code would behave as you expect:
func (p *ProducerTestSuite) TestProducer_Publish() {
p.producer.EXPECT().Publish([]model.Vacancy{}).Return(nil)
p.Require().NoError(p.producer.Publish([]model.Vacancy{}))
}
However, this test does not seem to actually exercise the unit that the test name indicates it should. Perhaps you are misunderstanding mocking ?
the Expect() means these must be a call to this method with the specified parameter, otherwise it will be failed, the missing call means your set a Expect() but didn't call it.
This answer is late, but it might helpful.
To require the function to be called once:
mockService.EXPECT().DoSomething().Return(nil, nil)
To allow the function to be called zero or more times:
mockService.EXPECT().DoSomething().Return(nil, nil).AnyTimes()
I also resolved a similar issue by adding a:
EXPECT().<interface-method>().Return(...).AnyTimes()
It seems that if one sets an EXPECT() gomock assumes that it must be called at least once. Adding AnyTimes() allows it to be called 0 times.
I wondering what is the best way to handle error form multiple level abstraction in go. Every time if I must add a new level abstraction to program, I am forced to transfer error code from level less to level high. Thereby is duplicate communitaces in log file or I must remmember to delete communicate form level low and transfer him to level higher. Below simply example. I skipped creating each object to more shortly and celar code, but I think You understand my problem
type ObjectOne struct{
someValue int
}
func (o* ObjectOne)CheckValue()error{
if o.someValue == 0 {
SomeLogger.Printf("Value is 0 error program") // communicate form first level abstraction to logger
return errors.New("Internal value in object is 0")
}
return nil
}
type ObjectTwoHigherLevel struct{
objectOne ObjectOne
}
func (oT* ObjectTwoHigherLevel)CheckObjectOneIsReady() error{
if err := oT.objectOne.CheckValue() ; err != nil{
SomeLogger.Printf("Value in objectOne is not correct for objectTwo %s" , err) // second communicate
return err
}
return nil
}
type ObjectThreeHiggerLevel struct{
oT ObjectTwoHigherLevel
}
func (oTh* ObjectThreeHiggerLevel)CheckObjectTwoIsReady()error{
if err := oTh.oT.CheckObjectOneIsReady() ; err != nil{
SomeLogger.Printf("Value in objectTwo is not correct for objectThree %s" , err)
return err
}
return nil
}
In result in log file I get duplicate posts
Value is 0 error program
Value in objectOne is not correct for objectTwo Internal value in object is 0
Value in objectTwo is not correct for objectThree Internal value in object is 0
In turn if I only transfer some err to higher level without additional log I lost information what happend in each level.
How this solve ? How privent duplicate communicates ? Or My way is the good and the only ?
Problem is more frustrating if I create a few object which search something in database on a few abstraction level then I get also few lines form this same task in logFile.
EDIT: This answer pre-dates Go 1.13 which provides something similar to the presented technique. Please check The Go Blog: Working with Errors in Go 1.13.
You should either handle an error, or not handle it but delegate it to a higher level (to the caller). Handling the error and returning it is bad practice as if the caller also does the same, the error might get handled several times.
Handling an error means inspecting it and making a decision based on that, which may be you simply log it, but that also counts as "handling" it.
If you choose to not handle but delegate it to a higher level, that may be perfectly fine, but don't just return the error value you got, as it may be meaningless to the caller without context.
Annotating errors
A really nice and recommended way of delegation is Annotating errors. This means you create and return a new error value, but the old one is also wrapped in the returned value. The wrapper provides the context for the wrapped error.
There is a public library for annotating errors: github.com/pkg/errors; and its godoc: errors
It basically has 2 functions: 1 for wrapping an existing error:
func Wrap(cause error, message string) error
And one for extracting a wrapped error:
func Cause(err error) error
Using these, this is how your error handling may look like:
func (o *ObjectOne) CheckValue() error {
if o.someValue == 0 {
return errors.New("Object1 illegal state: value is 0")
}
return nil
}
And the second level:
func (oT *ObjectTwoHigherLevel) CheckObjectOneIsReady() error {
if err := oT.objectOne.CheckValue(); err != nil {
return errors.Wrap(err, "Object2 illegal state: Object1 is invalid")
}
return nil
}
And the third level: call only the 2nd level check:
func (oTh *ObjectThreeHiggerLevel) CheckObjectTwoIsReady() error {
if err := oTh.ObjectTwoHigherLevel.CheckObjectOneIsReady(); err != nil {
return errors.Wrap(err, "Object3 illegal state: Object2 is invalid")
}
return nil
}
Note that since the CheckXX() methods do not handle the errors, they don't log anything. They are delegating annotated errors.
If someone using ObjectThreeHiggerLevel decides to handle the error:
o3 := &ObjectThreeHiggerLevel{}
if err := o3.CheckObjectTwoIsReady(); err != nil {
fmt.Println(err)
}
The following nice output will be presented:
Object3 illegal state: Object2 is invalid: Object2 illegal state: Object1 is invalid: Object1 illegal state: value is 0
There is no pollution of multiple logs, and all the details and context are preserved because we used errors.Wrap() which produces an error value which formats to a string which preserves the wrapped errors, recursively: the error stack.
You can read more about this technique in blog post:
Dave Cheney: Don’t just check errors, handle them gracefully
"Extending" errors
If you like things simpler and / or you don't want to hassle with external libraries and you're fine with not being able to extract the original error (the exact error value, not the error string which you can), then you may simply extend the error with the context and return this new, extended error.
Extending an error is easiest done by using fmt.Errorf() which allows you to create a "nice" formatted error message, and it returns you a value of type error so you can directly return that.
Using fmt.Errorf(), this is how your error handling may look like:
func (o *ObjectOne) CheckValue() error {
if o.someValue == 0 {
return fmt.Errorf("Object1 illegal state: value is %d", o.someValue)
}
return nil
}
And the second level:
func (oT *ObjectTwoHigherLevel) CheckObjectOneIsReady() error {
if err := oT.objectOne.CheckValue(); err != nil {
return fmt.Errorf("Object2 illegal state: %v", err)
}
return nil
}
And the third level: call only the 2nd level check:
func (oTh *ObjectThreeHiggerLevel) CheckObjectTwoIsReady() error {
if err := oTh.ObjectTwoHigherLevel.CheckObjectOneIsReady(); err != nil {
return fmt.Errorf("Object3 illegal state: %v", err)
}
return nil
}
And the following error message would be presented at ObjectThreeHiggerLevel should it decide to "handle" it:
o3 := &ObjectThreeHiggerLevel{}
if err := o3.CheckObjectTwoIsReady(); err != nil {
fmt.Println(err)
}
The following nice output will be presented:
Object3 illegal state: Object2 illegal state: Object1 illegal state: value is 0
Be sure to also read blog post: Error handling and Go
There are various libraries that embed stack traces in Go errors. Simply create your error with one of those, and it will bubble up with the full stack context you can later inspect or log.
One such library:
https://github.com/go-errors/errors
And there are a few others I forgot.
I wondering what is the best way to handle error form multiple level abstraction in go. Every time if I must add a new level abstraction to program, I am forced to transfer error code from level less to level high. Thereby is duplicate communitaces in log file or I must remmember to delete communicate form level low and transfer him to level higher. Below simply example. I skipped creating each object to more shortly and celar code, but I think You understand my problem
type ObjectOne struct{
someValue int
}
func (o* ObjectOne)CheckValue()error{
if o.someValue == 0 {
SomeLogger.Printf("Value is 0 error program") // communicate form first level abstraction to logger
return errors.New("Internal value in object is 0")
}
return nil
}
type ObjectTwoHigherLevel struct{
objectOne ObjectOne
}
func (oT* ObjectTwoHigherLevel)CheckObjectOneIsReady() error{
if err := oT.objectOne.CheckValue() ; err != nil{
SomeLogger.Printf("Value in objectOne is not correct for objectTwo %s" , err) // second communicate
return err
}
return nil
}
type ObjectThreeHiggerLevel struct{
oT ObjectTwoHigherLevel
}
func (oTh* ObjectThreeHiggerLevel)CheckObjectTwoIsReady()error{
if err := oTh.oT.CheckObjectOneIsReady() ; err != nil{
SomeLogger.Printf("Value in objectTwo is not correct for objectThree %s" , err)
return err
}
return nil
}
In result in log file I get duplicate posts
Value is 0 error program
Value in objectOne is not correct for objectTwo Internal value in object is 0
Value in objectTwo is not correct for objectThree Internal value in object is 0
In turn if I only transfer some err to higher level without additional log I lost information what happend in each level.
How this solve ? How privent duplicate communicates ? Or My way is the good and the only ?
Problem is more frustrating if I create a few object which search something in database on a few abstraction level then I get also few lines form this same task in logFile.
EDIT: This answer pre-dates Go 1.13 which provides something similar to the presented technique. Please check The Go Blog: Working with Errors in Go 1.13.
You should either handle an error, or not handle it but delegate it to a higher level (to the caller). Handling the error and returning it is bad practice as if the caller also does the same, the error might get handled several times.
Handling an error means inspecting it and making a decision based on that, which may be you simply log it, but that also counts as "handling" it.
If you choose to not handle but delegate it to a higher level, that may be perfectly fine, but don't just return the error value you got, as it may be meaningless to the caller without context.
Annotating errors
A really nice and recommended way of delegation is Annotating errors. This means you create and return a new error value, but the old one is also wrapped in the returned value. The wrapper provides the context for the wrapped error.
There is a public library for annotating errors: github.com/pkg/errors; and its godoc: errors
It basically has 2 functions: 1 for wrapping an existing error:
func Wrap(cause error, message string) error
And one for extracting a wrapped error:
func Cause(err error) error
Using these, this is how your error handling may look like:
func (o *ObjectOne) CheckValue() error {
if o.someValue == 0 {
return errors.New("Object1 illegal state: value is 0")
}
return nil
}
And the second level:
func (oT *ObjectTwoHigherLevel) CheckObjectOneIsReady() error {
if err := oT.objectOne.CheckValue(); err != nil {
return errors.Wrap(err, "Object2 illegal state: Object1 is invalid")
}
return nil
}
And the third level: call only the 2nd level check:
func (oTh *ObjectThreeHiggerLevel) CheckObjectTwoIsReady() error {
if err := oTh.ObjectTwoHigherLevel.CheckObjectOneIsReady(); err != nil {
return errors.Wrap(err, "Object3 illegal state: Object2 is invalid")
}
return nil
}
Note that since the CheckXX() methods do not handle the errors, they don't log anything. They are delegating annotated errors.
If someone using ObjectThreeHiggerLevel decides to handle the error:
o3 := &ObjectThreeHiggerLevel{}
if err := o3.CheckObjectTwoIsReady(); err != nil {
fmt.Println(err)
}
The following nice output will be presented:
Object3 illegal state: Object2 is invalid: Object2 illegal state: Object1 is invalid: Object1 illegal state: value is 0
There is no pollution of multiple logs, and all the details and context are preserved because we used errors.Wrap() which produces an error value which formats to a string which preserves the wrapped errors, recursively: the error stack.
You can read more about this technique in blog post:
Dave Cheney: Don’t just check errors, handle them gracefully
"Extending" errors
If you like things simpler and / or you don't want to hassle with external libraries and you're fine with not being able to extract the original error (the exact error value, not the error string which you can), then you may simply extend the error with the context and return this new, extended error.
Extending an error is easiest done by using fmt.Errorf() which allows you to create a "nice" formatted error message, and it returns you a value of type error so you can directly return that.
Using fmt.Errorf(), this is how your error handling may look like:
func (o *ObjectOne) CheckValue() error {
if o.someValue == 0 {
return fmt.Errorf("Object1 illegal state: value is %d", o.someValue)
}
return nil
}
And the second level:
func (oT *ObjectTwoHigherLevel) CheckObjectOneIsReady() error {
if err := oT.objectOne.CheckValue(); err != nil {
return fmt.Errorf("Object2 illegal state: %v", err)
}
return nil
}
And the third level: call only the 2nd level check:
func (oTh *ObjectThreeHiggerLevel) CheckObjectTwoIsReady() error {
if err := oTh.ObjectTwoHigherLevel.CheckObjectOneIsReady(); err != nil {
return fmt.Errorf("Object3 illegal state: %v", err)
}
return nil
}
And the following error message would be presented at ObjectThreeHiggerLevel should it decide to "handle" it:
o3 := &ObjectThreeHiggerLevel{}
if err := o3.CheckObjectTwoIsReady(); err != nil {
fmt.Println(err)
}
The following nice output will be presented:
Object3 illegal state: Object2 illegal state: Object1 illegal state: value is 0
Be sure to also read blog post: Error handling and Go
There are various libraries that embed stack traces in Go errors. Simply create your error with one of those, and it will bubble up with the full stack context you can later inspect or log.
One such library:
https://github.com/go-errors/errors
And there are a few others I forgot.
I'm trying to develop a simple job queue server with some worker that query it but I encountered a problem with my net/http server. I'm surely doing something bad but after ~3 minutes my server start displaying :
http: Accept error: accept tcp [::]:4200: accept4: too many open files; retrying in 1s
For information it receive 10 request per second in my test case.
Here's two files to reproduce this error :
// server.go
package main
import (
"net/http"
)
func main() {
http.HandleFunc("/get", func(rw http.ResponseWriter, r *http.Request) {
http.Error(rw, "Try again", http.StatusInternalServerError)
})
http.ListenAndServe(":4200", nil)
}
// worker.go
package main
import (
"net/http"
"time"
)
func main() {
for {
res, _ := http.Get("http://localhost:4200/get")
defer res.Body.Close()
if res.StatusCode == http.StatusInternalServerError {
time.Sleep(100 * time.Millisecond)
continue
}
return
}
}
I already done some search about this error and I found some interesting response but none of these fixed my issue.
The first response I saw was to correctly close the Body in the http.Get response, as you can see I did it.
The second response was to change the file descriptor ulimit of my system but as I will not control where my app will run I prefer to not use this solution (But for information it's set at 1024 on my system)
Can someone explain me why this problem happen and how I can fix it in my code ?
Thanks a lot for your time
EDIT :
EDIT 2 : In comment Martin says that I'm not closing the Body, I tried to close it (without defer so) and it fixed the issue. Thanks Martin ! I was thinking that continue will execute my defer, I was wrong.
I found a post explaining the root problem in a lot more detail.
Nathan Smith even explains how to control timeouts on the TCP level, if needed.
Below is a summary of everything I could find on this particular problem, as well as the best practices to avoid this problem in future.
Problem
When a response is received regardless of whether response-body is required or not, the connection is kept alive until the response-body stream is closed. So, as mentioned in this thread, always close the response-body. Even if you do not need to use/read the body content:
func Ping(url string) (bool) {
// simple GET request on given URL
res, err := http.Get(url)
if err != nil {
// if unable to GET given URL, then ping must fail
return false
}
// always close the response-body, even if content is not required
defer res.Body.Close()
// is the page status okay?
return res.StatusCode == http.StatusOK
}
Best Practice
As mentioned by Nathan Smith never use the http.DefaultClient in production systems, this includes calls like http.Get as it uses http.DefaultClient at its base.
Another reason to avoid http.DefaultClient is that it is a Singleton (package level variable), meaning that the garbage collector will not try to clean it up, which will leave idling subsequent streams/sockets alive.
Instead create your own instance of http.Client and remember to always specify a sane Timeout:
func Ping(url string) (bool) {
// create a new instance of http client struct, with a timeout of 2sec
client := http.Client{ Timeout: time.Second * 2 }
// simple GET request on given URL
res, err := client.Get(url)
if err != nil {
// if unable to GET given URL, then ping must fail
return false
}
// always close the response-body, even if content is not required
defer res.Body.Close()
// is the page status okay?
return res.StatusCode == http.StatusOK
}
Safety Net
The safety net is for that newbie on the team, who does not know the shortfalls of http.DefaultClient usage. Or even that very useful, but not so active, open-source library that is still riddled with http.DefaultClient calls.
Since http.DefaultClient is a Singleton we can easily change the Timeout setting, just to ensure that legacy code does not cause idle connections to remain open.
I find it best to set this on the package main file in the init function:
package main
import (
"net/http"
"time"
)
func init() {
/*
Safety net for 'too many open files' issue on legacy code.
Set a sane timeout duration for the http.DefaultClient, to ensure idle connections are terminated.
Reference: https://stackoverflow.com/questions/37454236/net-http-server-too-many-open-files-error
*/
http.DefaultClient.Timeout = time.Minute * 10
}
As Martin say in comment I don't really closed the Body after the Get request. I used defer res.Body.Close() but it's not executed since I'm staying in the for loop. So continue dont't trigger defer
Please note that in some cases the setting in /etc/sysctl.conf
net.ipv4.tcp_tw_recycle = 1
Could cause this error because TCP connections remain open.
A temporary solution, just increase the number of open files:
ulimit -Sn 10000