List directory in Go - go

I've been trying to figure out how to simply list the files and folders in a single directory in Go.
I've found filepath.Walk, but it goes into sub-directories automatically, which I don't want. All of my other searches haven't turned anything better up.
I'm sure that this functionality exists, but it's been really hard to find. Let me know if anyone knows where I should look. Thanks.

You can try using the ReadDir function in the os package. Per the docs:
ReadDir reads the named directory, returning all its directory entries sorted by filename.
The resulting slice contains os.DirEntry types, which provide the methods listed here. Here is a basic example that lists the name of everything in the current directory (folders are included but not specially marked - you can check if an item is a folder by using the IsDir() method):
package main
import (
"fmt"
"os"
"log"
)
func main() {
entries, err := os.ReadDir("./")
if err != nil {
log.Fatal(err)
}
for _, e := range entries {
fmt.Println(e.Name())
}
}

We can get a list of files inside a folder on the file system using various golang standard library functions.
filepath.Walk
ioutil.ReadDir
os.File.Readdir
package main
import (
"fmt"
"io/ioutil"
"log"
"os"
"path/filepath"
)
func main() {
var (
root string
files []string
err error
)
root := "/home/manigandan/golang/samples"
// filepath.Walk
files, err = FilePathWalkDir(root)
if err != nil {
panic(err)
}
// ioutil.ReadDir
files, err = IOReadDir(root)
if err != nil {
panic(err)
}
//os.File.Readdir
files, err = OSReadDir(root)
if err != nil {
panic(err)
}
for _, file := range files {
fmt.Println(file)
}
}
Using filepath.Walk
The path/filepath package provides a handy way to scan all the files
in a directory, it will automatically scan each sub-directories in the
directory.
func FilePathWalkDir(root string) ([]string, error) {
var files []string
err := filepath.Walk(root, func(path string, info os.FileInfo, err error) error {
if !info.IsDir() {
files = append(files, path)
}
return nil
})
return files, err
}
Using ioutil.ReadDir
ioutil.ReadDir reads the directory named by dirname and returns a
list of directory entries sorted by filename.
func IOReadDir(root string) ([]string, error) {
var files []string
fileInfo, err := ioutil.ReadDir(root)
if err != nil {
return files, err
}
for _, file := range fileInfo {
files = append(files, file.Name())
}
return files, nil
}
Using os.File.Readdir
Readdir reads the contents of the directory associated with file and
returns a slice of up to n FileInfo values, as would be returned by
Lstat, in directory order. Subsequent calls on the same file will
yield further FileInfos.
func OSReadDir(root string) ([]string, error) {
var files []string
f, err := os.Open(root)
if err != nil {
return files, err
}
fileInfo, err := f.Readdir(-1)
f.Close()
if err != nil {
return files, err
}
for _, file := range fileInfo {
files = append(files, file.Name())
}
return files, nil
}
Benchmark results.
Get more details on this Blog Post

Even simpler, use path/filepath:
package main
import (
"fmt"
"log"
"path/filepath"
)
func main() {
files, err := filepath.Glob("*")
if err != nil {
log.Fatal(err)
}
fmt.Println(files) // contains a list of all files in the current directory
}

Starting with Go 1.16, you can use the os.ReadDir function.
func ReadDir(name string) ([]DirEntry, error)
It reads a given directory and returns a DirEntry slice that contains the directory entries sorted by filename.
It's an optimistic function, so that, when an error occurs while reading the directory entries, it tries to return you a slice with the filenames up to the point before the error.
package main
import (
"fmt"
"log"
"os"
)
func main() {
files, err := os.ReadDir(".")
if err != nil {
log.Fatal(err)
}
for _, file := range files {
fmt.Println(file.Name())
}
}
Of interest: Go 1.17 (Q3 2021) includes fs.FileInfoToDirEntry():
func FileInfoToDirEntry(info FileInfo) DirEntry
FileInfoToDirEntry returns a DirEntry that returns information from info.
If info is nil, FileInfoToDirEntry returns nil.
Background
Go 1.16 (Q1 2021) will propose, with CL 243908 and CL 243914 , the ReadDir function, based on the FS interface:
// An FS provides access to a hierarchical file system.
//
// The FS interface is the minimum implementation required of the file system.
// A file system may implement additional interfaces,
// such as fsutil.ReadFileFS, to provide additional or optimized functionality.
// See io/fsutil for details.
type FS interface {
// Open opens the named file.
//
// When Open returns an error, it should be of type *PathError
// with the Op field set to "open", the Path field set to name,
// and the Err field describing the problem.
//
// Open should reject attempts to open names that do not satisfy
// ValidPath(name), returning a *PathError with Err set to
// ErrInvalid or ErrNotExist.
Open(name string) (File, error)
}
That allows for "os: add ReadDir method for lightweight directory reading":
See commit a4ede9f:
// ReadDir reads the contents of the directory associated with the file f
// and returns a slice of DirEntry values in directory order.
// Subsequent calls on the same file will yield later DirEntry records in the directory.
//
// If n > 0, ReadDir returns at most n DirEntry records.
// In this case, if ReadDir returns an empty slice, it will return an error explaining why.
// At the end of a directory, the error is io.EOF.
//
// If n <= 0, ReadDir returns all the DirEntry records remaining in the directory.
// When it succeeds, it returns a nil error (not io.EOF).
func (f *File) ReadDir(n int) ([]DirEntry, error)
// A DirEntry is an entry read from a directory (using the ReadDir method).
type DirEntry interface {
// Name returns the name of the file (or subdirectory) described by the entry.
// This name is only the final element of the path, not the entire path.
// For example, Name would return "hello.go" not "/home/gopher/hello.go".
Name() string
// IsDir reports whether the entry describes a subdirectory.
IsDir() bool
// Type returns the type bits for the entry.
// The type bits are a subset of the usual FileMode bits, those returned by the FileMode.Type method.
Type() os.FileMode
// Info returns the FileInfo for the file or subdirectory described by the entry.
// The returned FileInfo may be from the time of the original directory read
// or from the time of the call to Info. If the file has been removed or renamed
// since the directory read, Info may return an error satisfying errors.Is(err, ErrNotExist).
// If the entry denotes a symbolic link, Info reports the information about the link itself,
// not the link's target.
Info() (FileInfo, error)
}
src/os/os_test.go#testReadDir() illustrates its usage:
file, err := Open(dir)
if err != nil {
t.Fatalf("open %q failed: %v", dir, err)
}
defer file.Close()
s, err2 := file.ReadDir(-1)
if err2 != nil {
t.Fatalf("ReadDir %q failed: %v", dir, err2)
}
Ben Hoyt points out in the comments to Go 1.16 os.ReadDir:
os.ReadDir(path string) ([]os.DirEntry, error), which you'll be able to call directly without the Open dance.
So you can probably shorten this to just os.ReadDir, as that's the concrete function most people will call.
See commit 3d913a9 (Dec. 2020):
os: add ReadFile, WriteFile, CreateTemp (was TempFile), MkdirTemp (was TempDir) from io/ioutil
io/ioutil was a poorly defined collection of helpers.
Proposal #40025 moved out the generic I/O helpers to io.
This CL for proposal #42026 moves the OS-specific helpers to os,
making the entire io/ioutil package deprecated.
os.ReadDir returns []DirEntry, in contrast to ioutil.ReadDir's []FileInfo.
(Providing a helper that returns []DirEntry is one of the primary motivations for this change.)

ioutil.ReadDir is a good find, but if you click and look at the source you see that it calls the method Readdir of os.File. If you are okay with the directory order and don't need the list sorted, then this Readdir method is all you need.

From your description, what you probably want is os.Readdirnames.
func (f *File) Readdirnames(n int) (names []string, err error)
Readdirnames reads the contents of the directory associated with file and returns a slice of up to n names of files in the directory, in directory order. Subsequent calls on the same file will yield further names.
...
If n <= 0, Readdirnames returns all the names from the directory in a single slice.
Snippet:
file, err := os.Open(path)
if err != nil {
return err
}
defer file.Close()
names, err := file.Readdirnames(0)
if err != nil {
return err
}
fmt.Println(names)
Credit to SquattingSlavInTracksuit's comment; I'd have suggested promoting their comment to an answer if I could.

A complete example of printing all the files in a directory recursively using Readdirnames
package main
import (
"fmt"
"os"
)
func main() {
path := "/path/to/your/directory"
err := readDir(path)
if err != nil {
panic(err)
}
}
func readDir(path string) error {
file, err := os.Open(path)
if err != nil {
return err
}
defer file.Close()
names, _ := file.Readdirnames(0)
for _, name := range names {
filePath := fmt.Sprintf("%v/%v", path, name)
file, err := os.Open(filePath)
if err != nil {
return err
}
defer file.Close()
fileInfo, err := file.Stat()
if err != nil {
return err
}
fmt.Println(filePath)
if fileInfo.IsDir() {
readDir(filePath)
}
}
return nil
}

Related

List FTP files with goftp

I am trying to write a simple Go program which connects to an FTP server, list the files in a specified directory and pulls them.
The code is this:
package main
import (
"bytes"
"fmt"
"github.com/secsy/goftp"
"io/ioutil"
"log"
"os"
"path"
"time"
)
func main() {
config := goftp.Config{
User: "anonymous",
Password: "root#local.me",
ConnectionsPerHost: 21,
Timeout: 10 * time.Second,
Logger: os.Stderr,
}
// Connecting to the server
client, dailErr := goftp.DialConfig(config, "ftp.example.com")
if dailErr != nil {
log.Fatal(dailErr)
panic(dailErr)
}
// setting the search directory
dir := "/downloads/"
files, err := client.ReadDir(dir)
if err != nil {
for _, file := range files {
if file.IsDir() {
path.Join(dir, file.Name())
} else {
fmt.Println("the file is %s", file.Name())
}
}
}
// this section works , I am setting a file name and I can pull it
// if I mark the search part
ret_file := "example.PDF"
fmt.Println("Retrieving file: ", ret_file)
buf := new(bytes.Buffer)
fullPathFile := dir + ret_file
rferr := client.Retrieve(fullPathFile, buf)
if rferr != nil {
panic(rferr)
}
fmt.Println("writing data to file", ret_file)
fmt.Println("Opening file", ret_file, "for writing")
w, _ := ioutil.ReadAll(buf)
ferr := ioutil.WriteFile(ret_file, w, 0644)
if ferr != nil {
log.Fatal(ferr)
panic(ferr)
} else {
fmt.Println("Writing", ret_file, " completed")
}
}
For some reason I am getting an error on the ReadDir function.
I need to grab the files names so I can download them.
You're attempting to loop through files when ReadDir() returns an error. That will never work, as any time an error is returned files is nil.
This is pretty standard behavior and can be confirmed by reading the implementation of ReadDir().
I'm guessing you may have used the the example from the project used to demonstrate ReadDir() as a starting point. Within the example, the error handling is involved because it's deciding whether or not to continue walking the directory tree. However, note that when ReadDir() returns an error that doesn't result in stopping the program, the subsequent for loop is a no-op, since files is nil.
Here's a small program that demonstrates successfully using the results of Readdir() in a straightforward manner:
package main
import (
"fmt"
"github.com/secsy/goftp"
)
const (
ftpServerURL = "ftp.us.debian.org"
ftpServerPath = "/debian/"
)
func main() {
client, err := goftp.Dial(ftpServerURL)
if err != nil {
panic(err)
}
files, err := client.ReadDir(ftpServerPath)
if err != nil {
panic(err)
}
for _, file := range files {
fmt.Println(file.Name())
}
}
It outputs (which matches the current listing at http://ftp.us.debian.org/debian/):
$ go run goftp-test.go
README
README.CD-manufacture
README.html
README.mirrors.html
README.mirrors.txt
dists
doc
extrafiles
indices
ls-lR.gz
pool
project
tools
zzz-dists

How to get the number of files in a directory in beego [duplicate]

I've been trying to figure out how to simply list the files and folders in a single directory in Go.
I've found filepath.Walk, but it goes into sub-directories automatically, which I don't want. All of my other searches haven't turned anything better up.
I'm sure that this functionality exists, but it's been really hard to find. Let me know if anyone knows where I should look. Thanks.
You can try using the ReadDir function in the os package. Per the docs:
ReadDir reads the named directory, returning all its directory entries sorted by filename.
The resulting slice contains os.DirEntry types, which provide the methods listed here. Here is a basic example that lists the name of everything in the current directory (folders are included but not specially marked - you can check if an item is a folder by using the IsDir() method):
package main
import (
"fmt"
"os"
"log"
)
func main() {
entries, err := os.ReadDir("./")
if err != nil {
log.Fatal(err)
}
for _, e := range entries {
fmt.Println(e.Name())
}
}
We can get a list of files inside a folder on the file system using various golang standard library functions.
filepath.Walk
ioutil.ReadDir
os.File.Readdir
package main
import (
"fmt"
"io/ioutil"
"log"
"os"
"path/filepath"
)
func main() {
var (
root string
files []string
err error
)
root := "/home/manigandan/golang/samples"
// filepath.Walk
files, err = FilePathWalkDir(root)
if err != nil {
panic(err)
}
// ioutil.ReadDir
files, err = IOReadDir(root)
if err != nil {
panic(err)
}
//os.File.Readdir
files, err = OSReadDir(root)
if err != nil {
panic(err)
}
for _, file := range files {
fmt.Println(file)
}
}
Using filepath.Walk
The path/filepath package provides a handy way to scan all the files
in a directory, it will automatically scan each sub-directories in the
directory.
func FilePathWalkDir(root string) ([]string, error) {
var files []string
err := filepath.Walk(root, func(path string, info os.FileInfo, err error) error {
if !info.IsDir() {
files = append(files, path)
}
return nil
})
return files, err
}
Using ioutil.ReadDir
ioutil.ReadDir reads the directory named by dirname and returns a
list of directory entries sorted by filename.
func IOReadDir(root string) ([]string, error) {
var files []string
fileInfo, err := ioutil.ReadDir(root)
if err != nil {
return files, err
}
for _, file := range fileInfo {
files = append(files, file.Name())
}
return files, nil
}
Using os.File.Readdir
Readdir reads the contents of the directory associated with file and
returns a slice of up to n FileInfo values, as would be returned by
Lstat, in directory order. Subsequent calls on the same file will
yield further FileInfos.
func OSReadDir(root string) ([]string, error) {
var files []string
f, err := os.Open(root)
if err != nil {
return files, err
}
fileInfo, err := f.Readdir(-1)
f.Close()
if err != nil {
return files, err
}
for _, file := range fileInfo {
files = append(files, file.Name())
}
return files, nil
}
Benchmark results.
Get more details on this Blog Post
Even simpler, use path/filepath:
package main
import (
"fmt"
"log"
"path/filepath"
)
func main() {
files, err := filepath.Glob("*")
if err != nil {
log.Fatal(err)
}
fmt.Println(files) // contains a list of all files in the current directory
}
Starting with Go 1.16, you can use the os.ReadDir function.
func ReadDir(name string) ([]DirEntry, error)
It reads a given directory and returns a DirEntry slice that contains the directory entries sorted by filename.
It's an optimistic function, so that, when an error occurs while reading the directory entries, it tries to return you a slice with the filenames up to the point before the error.
package main
import (
"fmt"
"log"
"os"
)
func main() {
files, err := os.ReadDir(".")
if err != nil {
log.Fatal(err)
}
for _, file := range files {
fmt.Println(file.Name())
}
}
Of interest: Go 1.17 (Q3 2021) includes fs.FileInfoToDirEntry():
func FileInfoToDirEntry(info FileInfo) DirEntry
FileInfoToDirEntry returns a DirEntry that returns information from info.
If info is nil, FileInfoToDirEntry returns nil.
Background
Go 1.16 (Q1 2021) will propose, with CL 243908 and CL 243914 , the ReadDir function, based on the FS interface:
// An FS provides access to a hierarchical file system.
//
// The FS interface is the minimum implementation required of the file system.
// A file system may implement additional interfaces,
// such as fsutil.ReadFileFS, to provide additional or optimized functionality.
// See io/fsutil for details.
type FS interface {
// Open opens the named file.
//
// When Open returns an error, it should be of type *PathError
// with the Op field set to "open", the Path field set to name,
// and the Err field describing the problem.
//
// Open should reject attempts to open names that do not satisfy
// ValidPath(name), returning a *PathError with Err set to
// ErrInvalid or ErrNotExist.
Open(name string) (File, error)
}
That allows for "os: add ReadDir method for lightweight directory reading":
See commit a4ede9f:
// ReadDir reads the contents of the directory associated with the file f
// and returns a slice of DirEntry values in directory order.
// Subsequent calls on the same file will yield later DirEntry records in the directory.
//
// If n > 0, ReadDir returns at most n DirEntry records.
// In this case, if ReadDir returns an empty slice, it will return an error explaining why.
// At the end of a directory, the error is io.EOF.
//
// If n <= 0, ReadDir returns all the DirEntry records remaining in the directory.
// When it succeeds, it returns a nil error (not io.EOF).
func (f *File) ReadDir(n int) ([]DirEntry, error)
// A DirEntry is an entry read from a directory (using the ReadDir method).
type DirEntry interface {
// Name returns the name of the file (or subdirectory) described by the entry.
// This name is only the final element of the path, not the entire path.
// For example, Name would return "hello.go" not "/home/gopher/hello.go".
Name() string
// IsDir reports whether the entry describes a subdirectory.
IsDir() bool
// Type returns the type bits for the entry.
// The type bits are a subset of the usual FileMode bits, those returned by the FileMode.Type method.
Type() os.FileMode
// Info returns the FileInfo for the file or subdirectory described by the entry.
// The returned FileInfo may be from the time of the original directory read
// or from the time of the call to Info. If the file has been removed or renamed
// since the directory read, Info may return an error satisfying errors.Is(err, ErrNotExist).
// If the entry denotes a symbolic link, Info reports the information about the link itself,
// not the link's target.
Info() (FileInfo, error)
}
src/os/os_test.go#testReadDir() illustrates its usage:
file, err := Open(dir)
if err != nil {
t.Fatalf("open %q failed: %v", dir, err)
}
defer file.Close()
s, err2 := file.ReadDir(-1)
if err2 != nil {
t.Fatalf("ReadDir %q failed: %v", dir, err2)
}
Ben Hoyt points out in the comments to Go 1.16 os.ReadDir:
os.ReadDir(path string) ([]os.DirEntry, error), which you'll be able to call directly without the Open dance.
So you can probably shorten this to just os.ReadDir, as that's the concrete function most people will call.
See commit 3d913a9 (Dec. 2020):
os: add ReadFile, WriteFile, CreateTemp (was TempFile), MkdirTemp (was TempDir) from io/ioutil
io/ioutil was a poorly defined collection of helpers.
Proposal #40025 moved out the generic I/O helpers to io.
This CL for proposal #42026 moves the OS-specific helpers to os,
making the entire io/ioutil package deprecated.
os.ReadDir returns []DirEntry, in contrast to ioutil.ReadDir's []FileInfo.
(Providing a helper that returns []DirEntry is one of the primary motivations for this change.)
ioutil.ReadDir is a good find, but if you click and look at the source you see that it calls the method Readdir of os.File. If you are okay with the directory order and don't need the list sorted, then this Readdir method is all you need.
From your description, what you probably want is os.Readdirnames.
func (f *File) Readdirnames(n int) (names []string, err error)
Readdirnames reads the contents of the directory associated with file and returns a slice of up to n names of files in the directory, in directory order. Subsequent calls on the same file will yield further names.
...
If n <= 0, Readdirnames returns all the names from the directory in a single slice.
Snippet:
file, err := os.Open(path)
if err != nil {
return err
}
defer file.Close()
names, err := file.Readdirnames(0)
if err != nil {
return err
}
fmt.Println(names)
Credit to SquattingSlavInTracksuit's comment; I'd have suggested promoting their comment to an answer if I could.
A complete example of printing all the files in a directory recursively using Readdirnames
package main
import (
"fmt"
"os"
)
func main() {
path := "/path/to/your/directory"
err := readDir(path)
if err != nil {
panic(err)
}
}
func readDir(path string) error {
file, err := os.Open(path)
if err != nil {
return err
}
defer file.Close()
names, _ := file.Readdirnames(0)
for _, name := range names {
filePath := fmt.Sprintf("%v/%v", path, name)
file, err := os.Open(filePath)
if err != nil {
return err
}
defer file.Close()
fileInfo, err := file.Stat()
if err != nil {
return err
}
fmt.Println(filePath)
if fileInfo.IsDir() {
readDir(filePath)
}
}
return nil
}

Handling nested zip files with archive/zip

I'm struggling to handle nested zip files in Go (where a zip file contains another zip file). I'm trying to recurse a zip file and list all of the files it contains.
archive/zip gives you two methods for handling a zip file:
zip.NewReader
zip.OpenReader
OpenReader opens a file on disk. NewReader accepts an io.ReaderAt and a file size. As you iterate through the zipped files with either of these, you get out a zip.File for each file inside the zip. To get the file contents of file f, you call f.Open which gives you a zip.ReadCloser. To open a nested zip file, I'd need to use NewReader, but zip.File and zip.ReadCloser do not satisfy the io.ReaderAt interface.
zip.File has a private field zipr which is an io.ReaderAt and zip.ReadCloser has a private field f which is an os.File which should satisfy the requirements for NewReader.
My question: is there any way to open a nested zip file without first writing the contents to a file on disk, or reading the whole thing into memory.
It looks like everything that is needed is available in zip.File, but isn't exported. I'm hoping I missed something.
How about an io.ReaderAt from an io.Reader that reinitializes if you decided to go backwards: (this code is largely untested, but hopefully you get the idea)
package main
import (
"io"
"io/ioutil"
"os"
"strings"
)
type inefficientReaderAt struct {
rdr io.ReadCloser
cur int64
initer func() (io.ReadCloser, error)
}
func newInefficentReaderAt(initer func() (io.ReadCloser, error)) *inefficientReaderAt {
return &inefficientReaderAt{
initer: initer,
}
}
func (r *inefficientReaderAt) Read(p []byte) (n int, err error) {
n, err = r.rdr.Read(p)
r.cur += int64(n)
return n, err
}
func (r *inefficientReaderAt) ReadAt(p []byte, off int64) (n int, err error) {
// reset on rewind
if off < r.cur || r.rdr == nil {
r.cur = 0
r.rdr, err = r.initer()
if err != nil {
return 0, err
}
}
if off > r.cur {
sz, err := io.CopyN(ioutil.Discard, r.rdr, off-r.cur)
n = int(sz)
if err != nil {
return n, err
}
}
return r.Read(p)
}
func main() {
r := newInefficentReaderAt(func() (io.ReadCloser, error) {
return ioutil.NopCloser(strings.NewReader("ABCDEFG")), nil
})
io.Copy(os.Stdout, io.NewSectionReader(r, 0, 3))
io.Copy(os.Stdout, io.NewSectionReader(r, 1, 3))
}
If you mostly move forwards this probably works ok. Especially if you use a buffered reader.
I should note that this violates the io.ReaderAt guarantees: https://godoc.org/io#ReaderFrom , namely it doesn't allow parallel calls to ReadAt, and doesn't block on full reads, so this may not even work properly
I ran into the exact same need and came up with the following approach, not sure if its any help to you:
// NewZipFromReader ...
func NewZipFromReader(file io.ReadCloser, size int64) (*zip.Reader, error) {
in := file.(io.Reader)
if _, ok := in.(io.ReaderAt); ok != true {
buffer, err := ioutil.ReadAll(in)
if err != nil {
return nil, err
}
in = bytes.NewReader(buffer)
size = int64(len(buffer))
}
reader, err := zip.NewReader(in.(io.ReaderAt), size)
if err != nil {
return nil, err
}
return reader, nil
}
So if file doesn't implement io.ReaderAt it reads the whole contents into a buffer.
It's probably not safe to handle ZIP bombs, and will defenitely fail with OOM for files larger than RAM.

How to create nested directories using Mkdir in Golang?

I am trying to create a set of nested directories from a Go executable such as 'dir1/dir2/dir3'. I have succeeded in creating a single directory with this line:
os.Mkdir("." + string(filepath.Separator) + c.Args().First(),0777);
However, I have no idea how to approach creating a predetermined nested set of directories inside of that directory.
os.Mkdir is used to create a single directory. To create a folder path, instead try using:
os.MkdirAll(folderPath, os.ModePerm)
Go documentation
func MkdirAll(path string, perm FileMode) error
MkdirAll creates a directory named path, along with any necessary parents, and returns nil, or else returns an error. The permission bits perm are used for all directories that MkdirAll creates. If path is already a directory, MkdirAll does nothing and returns nil.
Edit:
Updated to correctly use os.ModePerm instead.
For concatenation of file paths, use package path/filepath as described in #Chris' answer.
This way you don't have to use any magic numbers:
os.MkdirAll(newPath, os.ModePerm)
Also, rather than using + to create paths, you can use:
import "path/filepath"
path := filepath.Join(someRootPath, someSubPath)
The above uses the correct separators automatically on each platform for you.
If the issue is to create all the necessary parent directories, you could consider using os.MkDirAll()
MkdirAll creates a directory named path, along with any necessary parents, and returns nil, or else returns an error.
The path_test.go is a good illustration on how to use it:
func TestMkdirAll(t *testing.T) {
tmpDir := TempDir()
path := tmpDir + "/_TestMkdirAll_/dir/./dir2"
err := MkdirAll(path, 0777)
if err != nil {
t.Fatalf("MkdirAll %q: %s", path, err)
}
defer RemoveAll(tmpDir + "/_TestMkdirAll_")
...
}
(Make sure to specify a sensible permission value, as mentioned in this answer)
This is one alternative for achieving the same but it avoids race condition caused by having two distinct "check ..and.. create" operations.
package main
import (
"fmt"
"os"
)
func main() {
if err := ensureDir("/test-dir"); err != nil {
fmt.Println("Directory creation failed with error: " + err.Error())
os.Exit(1)
}
// Proceed forward
}
func ensureDir(dirName string) error {
err := os.MkdirAll(dirName, os.ModeDir)
if err == nil || os.IsExist(err) {
return nil
} else {
return err
}
}
An utility method like the following can be used to solve this.
import (
"os"
"path/filepath"
"log"
)
func ensureDir(fileName string) {
dirName := filepath.Dir(fileName)
if _, serr := os.Stat(dirName); serr != nil {
merr := os.MkdirAll(dirName, os.ModePerm)
if merr != nil {
panic(merr)
}
}
}
func main() {
_, cerr := os.Create("a/b/c/d.txt")
if cerr != nil {
log.Fatal("error creating a/b/c", cerr)
}
log.Println("created file in a sub-directory.")
}

Getting count of files in directory using Go

How might I get the count of items returned by io/ioutil.ReadDir()?
I have this code, which works, but I have to think isn't the RightWay(tm) in Go.
package main
import "io/ioutil"
import "fmt"
func main() {
files,_ := ioutil.ReadDir("/Users/dgolliher/Dropbox/INBOX")
var count int
for _, f := range files {
fmt.Println(f.Name())
count++
}
fmt.Println(count)
}
Lines 8-12 seem like way too much to go through to just count the results of ReadDir, but I can't find the correct syntax to get the count without iterating over the range. Help?
Found the answer in http://blog.golang.org/go-slices-usage-and-internals
package main
import "io/ioutil"
import "fmt"
func main() {
files,_ := ioutil.ReadDir("/Users/dgolliher/Dropbox/INBOX")
fmt.Println(len(files))
}
ReadDir returns a list of directory entries sorted by filename, so it is not just files. Here is a little function for those wanting to get a count of files only (and not dirs):
func fileCount(path string) (int, error){
i := 0
files, err := ioutil.ReadDir(path)
if err != nil {
return 0, err
}
for _, file := range files {
if !file.IsDir() {
i++
}
}
return i, nil
}
Starting with Go 1.16 (Feb 2021), a better option is os.ReadDir:
package main
import "os"
func main() {
d, e := os.ReadDir(".")
if e != nil {
panic(e)
}
println(len(d))
}
os.ReadDir returns fs.DirEntry instead of fs.FileInfo, which means that
Size and ModTime methods are omitted, making the process more efficient if
you just need an entry count.
https://golang.org/pkg/os#ReadDir
If you wanna get all files (not recursive) you can use len(files). If you need to just get the files without folders and hidden files just loop over them and increase a counter. And please don’t ignore errors
By looking at the code of ioutil.ReadDir
func ReadDir(dirname string) ([]fs.FileInfo, error) {
f, err := os.Open(dirname)
if err != nil {
return nil, err
}
list, err := f.Readdir(-1)
f.Close()
if err != nil {
return nil, err
}
sort.Slice(list, func(i, j int) bool { return list[i].Name() < list[j].Name() })
return list, nil
}
you would realize that it calls os.File.Readdir() then sorts the files.
In case of counting it, you don't need to sort, so you are better off calling os.File.Readdir() directly.
You can simply copy and paste this function then remove the sort.
But I did find out that f.Readdirnames(-1) is much faster than f.Readdir(-1).
Running time is almost half for /usr/bin/ with 2808 items (16ms vs 35ms).
So to summerize it in an example:
package main
import (
"fmt"
"os"
)
func main() {
f, err := os.Open(os.Args[1])
if err != nil {
panic(err)
}
list, err := f.Readdirnames(-1)
f.Close()
if err != nil {
panic(err)
}
fmt.Println(len(list))
}

Resources