"too many open files" with os.Create - go

Question

I have around 220'000 image files (.png) to create. I run into this error message when trying to create the 1'081th file:
panic: open /media/Snaps/pics/image1081_0.png: too many open files
I've added the defer w.Close() line but it did not change the error.
i := 1
for i <= 223129 {
(some other code to prepare the data and create the chart)
img := vgimg.New(450, 600)
dc := draw.New(img)
canvases := table.Align(plots, dc)
plots[0][0].Draw(canvases[0][0])
plots[1][0].Draw(canvases[1][0])
plots[2][0].Draw(canvases[2][0])
testFile := "/media/Snaps/pics/image"+strconv.Itoa(i+60)+"_"+gain_loss+".png"
w, err := os.Create(testFile)
if err != nil {
panic(err)
}
defer w.Close()
png := vgimg.PngCanvas{Canvas: img}
if _, err := png.WriteTo(w); err != nil {
panic(err)
}
//move to next image
i = i + 1
}
Surely this limit can be worked around ? Maybe I'm not closing the files properly ?

Answer 1

The Go Programming Language Specification
Defer statements
A "defer" statement invokes a function whose execution is deferred to
the moment the surrounding function returns, either because the
surrounding function executed a return statement, reached the end of
its function body, or because the corresponding goroutine is
panicking.
DeferStmt = "defer" Expression .
The expression must be a function or method call; it cannot be
parenthesized. Calls of built-in functions are restricted as for
expression statements.
Each time a "defer" statement executes, the function value and
parameters to the call are evaluated as usual and saved anew but the
actual function is not invoked. Instead, deferred functions are
invoked immediately before the surrounding function returns, in the
reverse order they were deferred. If a deferred function value
evaluates to nil, execution panics when the function is invoked, not
when the "defer" statement is executed.
In other words, if you are processing files in a loop, put the processing for a single file in a separate function to pair the Open with the defer Close(). This avoids the "too many open files" error.
For example, use a file processing structure like this to guarantee each file is closed immediately after use.
package main
import (
"fmt"
"io/ioutil"
"os"
)
// process single file
func processFile(name string) error {
f, err := os.Open(name)
if err != nil {
return err
}
defer f.Close()
fi, err := f.Stat()
if err != nil {
return err
}
fmt.Println(fi.Name(), fi.Size())
return nil
}
func main() {
wd, err := os.Getwd()
if err != nil {
fmt.Fprintln(os.Stderr, err)
return
}
fis, err := ioutil.ReadDir(wd)
if err != nil {
fmt.Fprintln(os.Stderr, err)
return
}
// process all files
for _, fi := range fis {
processFile(fi.Name())
if err != nil {
fmt.Fprintln(os.Stderr, err)
}
}
}
Playground: https://play.golang.org/p/FrBWqlMOzaS
Output:
dev 1644
etc 1644
tmp 548
usr 822

Answer 2

Deferred statements are not executed until the surrounding function returns, that is why your files stay open until after the for-loop.
To fix this you can simply insert an anonymous function call inside the loop:
for ... {
func() {
w, err := os.Create(testFile)
if err != nil {
panic(err)
}
defer w.Close()
...
}()
}
That way, after each iteration of the loop, the current file is closed.

Answer 3

ok, i got it, changed defer w.Close() to w.Close() and moved it after
png := vgimg.PngCanvas{Canvas: img}
if _, err := png.WriteTo(w); err != nil {
panic(err)
}
I'm now above 10'000 images and running...