If I have a program like this:
package main
import "strconv"
func main() {
a, err := strconv.Atoi("100")
println(a)
}
I get this result, as expected:
.\file.go:5:7: err declared but not used
However this program:
package main
import "strconv"
func main() {
a, err := strconv.Atoi("100")
if err != nil {
panic(err)
}
b, err := strconv.Atoi("100")
println(a, b)
}
Compiles without error, even though I never checked the second err value. Why does this happen? Also, can I change some option, so that these mistakes result in compile time errors or warnings?
This is because in the second case you are re-using an existing err variable, so it is being used. Despite the := instantiate & assign operator, a new err variable is not instantiated.
If you named the errors differently, such as this:
func main() {
a, err := strconv.Atoi("100")
if err != nil {
panic(err)
}
b, err2 := strconv.Atoi("100")
println(a, b)
}
Then you would see a compile error for both cases.
If you don't want to change your code but also still want to be notified of this issue, you will need to rely on a go linter instead of the go compiler. Go has a very robust ecosystem of linters so I won't recommend one in particular, but at my organization I would see an error like this from our linter if I were to write such code:
scratch/test.go:10:2: ineffectual assignment to err (ineffassign)
b, err := strconv.Atoi("100")
I have around 220'000 image files (.png) to create. I run into this error message when trying to create the 1'081th file:
panic: open /media/Snaps/pics/image1081_0.png: too many open files
I've added the defer w.Close() line but it did not change the error.
i := 1
for i <= 223129 {
(some other code to prepare the data and create the chart)
img := vgimg.New(450, 600)
dc := draw.New(img)
canvases := table.Align(plots, dc)
plots[0][0].Draw(canvases[0][0])
plots[1][0].Draw(canvases[1][0])
plots[2][0].Draw(canvases[2][0])
testFile := "/media/Snaps/pics/image"+strconv.Itoa(i+60)+"_"+gain_loss+".png"
w, err := os.Create(testFile)
if err != nil {
panic(err)
}
defer w.Close()
png := vgimg.PngCanvas{Canvas: img}
if _, err := png.WriteTo(w); err != nil {
panic(err)
}
//move to next image
i = i + 1
}
Surely this limit can be worked around ? Maybe I'm not closing the files properly ?
The Go Programming Language Specification
Defer statements
A "defer" statement invokes a function whose execution is deferred to
the moment the surrounding function returns, either because the
surrounding function executed a return statement, reached the end of
its function body, or because the corresponding goroutine is
panicking.
DeferStmt = "defer" Expression .
The expression must be a function or method call; it cannot be
parenthesized. Calls of built-in functions are restricted as for
expression statements.
Each time a "defer" statement executes, the function value and
parameters to the call are evaluated as usual and saved anew but the
actual function is not invoked. Instead, deferred functions are
invoked immediately before the surrounding function returns, in the
reverse order they were deferred. If a deferred function value
evaluates to nil, execution panics when the function is invoked, not
when the "defer" statement is executed.
In other words, if you are processing files in a loop, put the processing for a single file in a separate function to pair the Open with the defer Close(). This avoids the "too many open files" error.
For example, use a file processing structure like this to guarantee each file is closed immediately after use.
package main
import (
"fmt"
"io/ioutil"
"os"
)
// process single file
func processFile(name string) error {
f, err := os.Open(name)
if err != nil {
return err
}
defer f.Close()
fi, err := f.Stat()
if err != nil {
return err
}
fmt.Println(fi.Name(), fi.Size())
return nil
}
func main() {
wd, err := os.Getwd()
if err != nil {
fmt.Fprintln(os.Stderr, err)
return
}
fis, err := ioutil.ReadDir(wd)
if err != nil {
fmt.Fprintln(os.Stderr, err)
return
}
// process all files
for _, fi := range fis {
processFile(fi.Name())
if err != nil {
fmt.Fprintln(os.Stderr, err)
}
}
}
Playground: https://play.golang.org/p/FrBWqlMOzaS
Output:
dev 1644
etc 1644
tmp 548
usr 822
Deferred statements are not executed until the surrounding function returns, that is why your files stay open until after the for-loop.
To fix this you can simply insert an anonymous function call inside the loop:
for ... {
func() {
w, err := os.Create(testFile)
if err != nil {
panic(err)
}
defer w.Close()
...
}()
}
That way, after each iteration of the loop, the current file is closed.
ok, i got it, changed defer w.Close() to w.Close() and moved it after
png := vgimg.PngCanvas{Canvas: img}
if _, err := png.WriteTo(w); err != nil {
panic(err)
}
I'm now above 10'000 images and running...
I have a bunch of *io.PipeWriter created and would like to make a multiwriter based on all those pipewriters in a function. So I call a function something like
func copyToWriters(reader *bufio.Reader, errs chan error, writers []*io.PipeWriter) {
for _, writer := range writers {
defer writer.Close()
}
mw := io.MultiWriter(writers)
_, err := io.Copy(mw, reader)
if err != nil {
errs <- err
}
}
I call the method with arguments copyToWriters(reader, errs, []*io.PipeWriter{w1, w2})
But it says
cannot use writers (type []*io.PipeWriter) as type []io.Writer in argument to io.MultiWriter. But
if I change io.MultiWriter(writers) to io.MultiWriter(writers[0],writers[1]) it works. How can I make the existing function work by not having to pass writers separately.
Unfortunately, Golang's type system does not allow casting []*io.PipeWriter to []io.Writer even when *io.PipeWriter implements io.Writer since it requires O(n) operation (reference).
The best you can do is create another []io.Writer and copy the pipe writers into it
ws := make([]io.Writer, len(writers))
for i, w := range writers {
ws[i] = w
}
mw := io.MultiWriter(ws...)
And reason why you nead ..., read the document
I declare some variables (offsetI and limitI) outside of a conditional statement. Inside the conditional statement I am trying to assign them values, then use those values for a query after the conditional statement.
var (
number, size, offset, limit string
offsetI, limitI uint64
)
// Get the string values for number, size, offset, and limit
// ...
if size != "" {
// Parse the number value
numberI, err := strconv.ParseUint(number, 10, 64)
if err != nil {...}
// Parse the size value
limitI, err = strconv.ParseUint(size, 10, 64)
if err != nil {...}
// Calculate the offset
offsetI = numberI * limitI
} else {
// Parse the limit value
limitI, err := strconv.ParseUint(limit, 10, 64) // limitI declared and not used
if err != nil {...}
// Parse the offset value
offsetI, err = strconv.ParseUint(offset, 10, 64)
if err != nil {...}
}
// Make the query using offsetI and limitI
result, err := s.GetAllPaginated(offsetI, limitI)
if err != nil {...}
I am not intending to re-declare the limitI variable in the scope of the else statement, but I need to use the := operator for declaring a new err variable.
The only thing I could come up with was to separately declare another err variable, so I could use a regular assignment statement:
} else {
var err error // New
// Regular assignment statement now
limitI, err = strconv.ParseUint(limit, 10, 64)
if err != nil {...}
I would like to be able to do this without having to declare an additional error variable.
The extra var error is awkward, but it's a common way to address this situation. The spec on scoping says (emphasis mine):
The scope of a constant or variable identifier declared inside a function begins at the end of the ConstSpec or VarSpec (ShortVarDecl for short variable declarations) and ends at the end of the innermost containing block.
So in your case, that short variable declaration is declaring a different limitI scoped to the "innermost containing block." Since it only "lives" until the next closing brace, it isn't used.
In your specific case, an option might be to declare err outside the if/else, since it's used in both inner scopes, so you can use use = instead of := with those functions returning errors. Then there's no "inner limitI" declared and you have no unused variable issue.
"Shadowing" situations like this can also produce unexpected behavior rather than an error. go vet -shadow tries to detect "[v]ariables that may have been unintentionally shadowed" and, different but related, gordonklaus/ineffasign generalizes the "unused variable" check to detect useless assignments even if they weren't declarations.
The only thing I could come up with was to separately declare another err variable
This is being studied in the context of a future Go 2 (2019? 2020).
See "Error Handling — Problem Overview" (Russ Cox - August 27, 2018)
A possible proposed new syntax would avoid having to redeclare err:
Example:
func CopyFile(src, dst string) error {
handle err {
return fmt.Errorf("copy %s %s: %v", src, dst, err)
}
r := check os.Open(src)
defer r.Close()
w := check os.Create(dst)
handle err {
w.Close()
os.Remove(dst) // (only if a check fails)
}
check io.Copy(w, r)
check w.Close()
return nil
}
Note that go vet -shadow is no longer available since Go 1.12 (February 2019):
The experimental -shadow option is no longer available with go vet.
Checking for variable shadowing may now be done using
go install golang.org/x/tools/go/analysis/passes/shadow/cmd/shadow
go vet -vettool=$(which shadow)
Is there a way to clean up this (IMO) horrific-looking code?
aJson, err1 := json.Marshal(a)
bJson, err2 := json.Marshal(b)
cJson, err3 := json.Marshal(c)
dJson, err4 := json.Marshal(d)
eJson, err5 := json.Marshal(e)
fJson, err6 := json.Marshal(f)
gJson, err4 := json.Marshal(g)
if err1 != nil {
return err1
} else if err2 != nil {
return err2
} else if err3 != nil {
return err3
} else if err4 != nil {
return err4
} else if err5 != nil {
return err5
} else if err5 != nil {
return err5
} else if err6 != nil {
return err6
}
Specifically, I'm talking about the error handling. It would be nice to be able to handle all the errors in one go.
var err error
f := func(dest *D, src S) bool {
*dest, err = json.Marshal(src)
return err == nil
} // EDIT: removed ()
f(&aJson, a) &&
f(&bJson, b) &&
f(&cJson, c) &&
f(&dJson, d) &&
f(&eJson, e) &&
f(&fJson, f) &&
f(&gJson, g)
return err
Put the result in a slice instead of variables, put the intial values in another slice to iterate and return during the iteration if there's an error.
var result [][]byte
for _, item := range []interface{}{a, b, c, d, e, f, g} {
res, err := json.Marshal(item)
if err != nil {
return err
}
result = append(result, res)
}
You could even reuse an array instead of having two slices.
var values, err = [...]interface{}{a, b, c, d, e, f, g}, error(nil)
for i, item := range values {
if values[i], err = json.Marshal(item); err != nil {
return err
}
}
Of course, this'll require a type assertion to use the results.
define a function.
func marshalMany(vals ...interface{}) ([][]byte, error) {
out := make([][]byte, 0, len(vals))
for i := range vals {
b, err := json.Marshal(vals[i])
if err != nil {
return nil, err
}
out = append(out, b)
}
return out, nil
}
you didn't say anything about how you'd like your error handling to work. Fail one, fail all? First to fail? Collect successes or toss them?
I believe the other answers here are correct for your specific problem, but more generally, panic can be used to shorten error handling while still being a well-behaving library. (i.e., not panicing across package boundaries.)
Consider:
func mustMarshal(v interface{}) []byte {
bs, err := json.Marshal(v)
if err != nil {
panic(err)
}
return bs
}
func encodeAll() (err error) {
defer func() {
if r := recover(); r != nil {
var ok bool
if err, ok = r.(error); ok {
return
}
panic(r)
}
}()
ea := mustMarshal(a)
eb := mustMarshal(b)
ec := mustMarshal(c)
return nil
}
This code uses mustMarshal to panic whenever there is a problem marshaling a value. But the encodeAll function will recover from the panic and return it as a normal error value. The client in this case is never exposed to the panic.
But this comes with a warning: using this approach everywhere is not idiomatic. It can also be worse since it doesn't lend itself well to handling each individual error specially, but more or less treating each error the same. But it has its uses when there are tons of errors to handle. As an example, I use this kind of approach in a web application, where a top-level handler can catch different kinds of errors and display them appropriately to the user (or a log file) depending on the kind of error.
It makes for terser code when there is a lot of error handling, but at the loss of idiomatic Go and handling each error specially. Another down-side is that it could prevent something that should panic from actually panicing. (But this can be trivially solved by using your own error type.)
You can use go-multierror by Hashicorp.
var merr error
if err := step1(); err != nil {
merr = multierror.Append(merr, err)
}
if err := step2(); err != nil {
merr = multierror.Append(merr, err)
}
return merr
You can create a reusable method to handle multiple errors, this implementation will only show the last error but you could return every error msg combined by modifying the following code:
func hasError(errs ...error) error {
for i, _ := range errs {
if errs[i] != nil {
return errs[i]
}
}
return nil
}
aJson, err := json.Marshal(a)
bJson, err1 := json.Marshal(b)
cJson, err2 := json.Marshal(c)
if error := hasError(err, err1, err2); error != nil {
return error
}
Another perspective on this is, instead of asking "how" to handle the abhorrent verbosity, whether we actually "should". This advice is heavily dependent on context, so be careful.
In order to decide whether handling the json.Marshal error is worth it, we can inspect its implementation to see when errors are returned. In order to return errors to the caller and preserve code terseness, json.Marshal uses panic and recover internally in a manner akin to exceptions. It defines an internal helper method which, when called, panics with the given error value. By looking at each call of this function, we learn that json.Marshal errors in the given scenarios:
calling MarshalJSON or MarshalText on a value/field of a type which implements json.Marshaler or encoding.TextMarshaler returns an error—in other words, a custom marshaling method fails;
the input is/contains a cyclic (self-referencing) structure;
the input is/contains a value of an unsupported type (complex, chan, func);
the input is/contains a floating-point number which is NaN or Infinity (these are not allowed by the spec, see section 2.4);
the input is/contains a json.Number string that is an incorrect number representation (for example, "foo" instead of "123").
Now, a usual scenario for marshaling data is creating an API response, for example. In that case, you will 100% have data types that satisfy all of the marshaler's constraints and valid values, given that the server itself generates them. In the situation user-provided input is used, the data should be validated anyway beforehand, so it should still not cause issues with the marshaler. Furthermore, we can see that, apart from the custom marshaler errors, all the other errors occur at runtime because Go's type system cannot enforce the required conditions by itself. With all these points given, here comes the question: given our control over the data types and values, do we need to handle json.Marshal's error at all?
Probably no. For a type like
type Person struct {
Name string
Age int
}
it is now obvious that json.Marshal cannot fail. It is trickier when the type looks like
type Foo struct {
Data any
}
(any is a new Go 1.18 alias for interface{}) because there is no compile-time guarantee that Foo.Data will hold a value of a valid type—but I'd still argue that if Foo is meant to be serialized as a response, Foo.Data will also be serializable. Infinity or NaN floats remain an issue, but, given the JSON standard limitation, if you want to serialize these two special values you cannot use JSON numbers anyway, so you'll have to look for another solution, which means that you'll end up avoiding the error anyway.
To conclude, my point is that you can probably do:
aJson, _ := json.Marshal(a)
bJson, _ := json.Marshal(b)
cJson, _ := json.Marshal(c)
dJson, _ := json.Marshal(d)
eJson, _ := json.Marshal(e)
fJson, _ := json.Marshal(f)
gJson, _ := json.Marshal(g)
and live fine with it. If you want to be pedantic, you can use a helper such as:
func must[T any](v T, err error) T {
if err != nil {
panic(err)
}
return v
}
(note the Go 1.18 generics usage) and do
aJson := must(json.Marshal(a))
bJson := must(json.Marshal(b))
cJson := must(json.Marshal(c))
dJson := must(json.Marshal(d))
eJson := must(json.Marshal(e))
fJson := must(json.Marshal(f))
gJson := must(json.Marshal(g))
This will work nice when you have something like an HTTP server, where each request is wrapped in a middleware that recovers from panics and responds to the client with status 500. It's also where you would care about these unexpected errors—when you don't want the program/service to crash at all. For one-time scripts you'll probably want to have the operation halted and a stack trace dumped.
If you're unsure of how your types will be changed in the future, you don't trust your tests, data may not be in your full control, the codebase is too big to trace the data or whatever other reason which causes uncertainty over the correctness of your data, it is better to handle the error. Pay attention to the context you're in!
P.S.: Pragmatically ignoring errors should be generally sought after. For example, the Write* methods on bytes.Buffer, strings.Builder never return errors; fmt.Fprintf, with a valid format string and a writer that doesn't return errors, also returns no errors; bufio.Writer aswell doesn't, if the underlying writer doesn't return. You will find some types implement interfaces with methods that return errors but don't actually return any. In these cases, if you know the concrete type, handling errors is unnecessarily verbose and redundant. What do you prefer,
var sb strings.Builder
if _, err := sb.WriteString("hello "); err != nil {
return err
}
if _, err := sb.WriteString("world!"); err != nil {
return err
}
or
var sb strings.Builder
sb.WriteString("hello ")
sb.WriteString("world!")
(of course, ignoring that it could be a single WriteString call)?
The given examples write to an in-memory buffer, which unless the machine is out of memory, an error which you cannot handle in Go, cannot ever fail. Other such situations will surface in your code—blindly handling errors adds little to no value! Caution is key—if an implementation changes and does return errors, you may be in trouble. Standard library or well-established packages are good candidates for eliding error checking, if possible.