How to check in Go if folder is empty? I can check like:
files, err := ioutil.ReadDir(*folderName)
if err != nil {
return nil, err
}
// here check len of files
But it kinda looks to me that there should be more a elegant solution.
Whether a directory is empty or not is not stored in the file-system level as properties like its name, creation time or its size (in case of files).
That being said you can't just obtain this information from an os.FileInfo. The easiest way is to query the children (content) of the directory.
ioutil.ReadDir() is quite a bad choice as that first reads all the contents of the specified directory and then sorts them by name, and then returns the slice. The fastest way is as Dave C mentioned: query the children of the directory using File.Readdir() or (preferably) File.Readdirnames() .
Both File.Readdir() and File.Readdirnames() take a parameter which is used to limit the number of returned values. It is enough to query only 1 child. As Readdirnames() returns only names, it is faster because no further calls are required to obtain (and construct) FileInfo structs.
Note that if the directory is empty, io.EOF is returned as an error (and not an empty or nil slice) so we don't even need the returned names slice.
The final code could look like this:
func IsEmpty(name string) (bool, error) {
f, err := os.Open(name)
if err != nil {
return false, err
}
defer f.Close()
_, err = f.Readdirnames(1) // Or f.Readdir(1)
if err == io.EOF {
return true, nil
}
return false, err // Either not empty or error, suits both cases
}
Related
i have a file. it has some ip
1.1.1.0/24
1.1.2.0/24
2.2.1.0/24
2.2.2.0/24
i read this file to slice, and used *(*string)(unsafe.Pointer(&b)) to parse []byte to string, but is doesn't work
func TestInitIpRangeFromFile(t *testing.T) {
filepath := "/tmp/test"
file, err := os.Open(filepath)
if err != nil {
t.Errorf("failed to open ip range file:%s, err:%s", filepath, err)
}
reader := bufio.NewReader(file)
ranges := make([]string, 0)
for {
ip, _, err := reader.ReadLine()
if err != nil {
if err == io.EOF {
break
}
logger.Fatalf("failed to read ip range file, err:%s", err)
}
t.Logf("ip:%s", *(*string)(unsafe.Pointer(&ip)))
ranges = append(ranges, *(*string)(unsafe.Pointer(&ip)))
}
t.Logf("%v", ranges)
}
result:
task_test.go:71: ip:1.1.1.0/24
task_test.go:71: ip:1.1.2.0/24
task_test.go:71: ip:2.2.1.0/24
task_test.go:71: ip:2.2.2.0/24
task_test.go:75: [2.2.2.0/24 1.1.2.0/24 2.2.1.0/24 2.2.2.0/24]
why 1.1.1.0/24 changed to 2.2.2.0/24 ?
change
*(*string)(unsafe.Pointer(&ip))
to string(ip) it works
So, while reinterpreting a slice-header as a string-header the way you did is absolutely bonkers and has no guarantee whatsoever of working correctly, it's only indirectly the cause of your problem.
The real problem is that you're retaining a pointer to the return value of bufio/Reader.ReadLine(), but the docs for that method say "The returned buffer is only valid until the next call to ReadLine." Which means that the reader is free to reuse that memory later on, and that's what's happening.
When you do the cast in the proper way, string(ip), Go copies the contents of the buffer into the newly-created string, which remains valid in the future. But when you type-pun the slice into a string, you keep the exact same pointer, which stops working as soon as the reader refills its buffer.
If you decided to do the pointer trickery as a performance hack to avoid copying and allocation... too bad. The reader interface is going to force you to copy the data out anyway, and since it does, you should just use string().
I have a 7z archive of a number of .txt files. I am trying to list all the files in the archive and upload them to an s3 bucket. But I'm having trouble with extracting .7z archives on Go. To do this, I found a package github.com/gen2brain/go-unarr (imported as extractor) and this is what I have so far
content, err := ioutil.ReadFile("sample_archive.7z")
if err != nil {
fmt.Printf("err: %+v", err)
}
a, err := extractor.NewArchiveFromMemory(content)
if err != nil {
fmt.Printf("err: %+v", err)
}
lst, _ := a.List()
fmt.Printf("lst: %+v", last)
This prints a list of all the files in the archive. But this has two issues.
It reads files from local using ioutil and the input of NewArchiveFromMemory must be of type []byte. But I can't read from local and will have to use a file from memory of type os.file. So I will either have to find a different method or convert the os.file to []byte. There's another method NewArchiveFromReader(r io.Reader). But this is returning an error saying Bad File Descriptor.
file, err := os.OpenFile(
path,
os.O_WRONLY|os.O_TRUNC|os.O_CREATE,
0666,
)
a, err := extractor.NewArchiveFromReader(file)
if err != nil {
fmt.Printf("ERROR: %+v", err)
}
lst, _ := a.List()
fmt.Printf("files: %+v\n", lst)
I am able to get the list of the files in the archive. And using Extract(destinaltion_path string), I can also extract it to a local directory. But I want the extracted files also in os.file format ( ie. a list of os.file since there will be multiple files ).
How can I change my current code to achieve both the above targets? Is there any other library to do this?
os.File implements the io.Reader interface (because it has a Read([]byte) (int, error) method defined), so you can use NewArchiveFromReader(file) without any conversions needed. You can read up on Go interfaces for more background on why that works.
If you're okay with extracting to a local directory, you can do that and then read the files back in (warning, may contain typos):
func extractAndOpenAll(*extractor.Archive) ([]*os.File, error) {
err := a.Extract("/tmp/path") // consider using ioutil.TempDir()
if err != nil {
return nil, err
}
filestats, err := ioutil.ReadDir("/tmp/path")
if err != nil {
return nil, err
}
# warning: all these file handles must be closed by the caller,
# which is why even the error case here returns the list of files.
# if you forget, your process might leak file handles.
files := make([]*os.File, 0)
for _, fs := range(filestats) {
file, err := os.Open(fs.Name())
if err != nil {
return files, err
}
files = append(files, file)
}
return files, nil
}
It is possible to use the archived files without writing back to disk (https://github.com/gen2brain/go-unarr#read-all-entries-from-archive), but whether or not you should do that instead depends on what your next step is.
I've been trying to figure out how to simply list the files and folders in a single directory in Go.
I've found filepath.Walk, but it goes into sub-directories automatically, which I don't want. All of my other searches haven't turned anything better up.
I'm sure that this functionality exists, but it's been really hard to find. Let me know if anyone knows where I should look. Thanks.
You can try using the ReadDir function in the os package. Per the docs:
ReadDir reads the named directory, returning all its directory entries sorted by filename.
The resulting slice contains os.DirEntry types, which provide the methods listed here. Here is a basic example that lists the name of everything in the current directory (folders are included but not specially marked - you can check if an item is a folder by using the IsDir() method):
package main
import (
"fmt"
"os"
"log"
)
func main() {
entries, err := os.ReadDir("./")
if err != nil {
log.Fatal(err)
}
for _, e := range entries {
fmt.Println(e.Name())
}
}
We can get a list of files inside a folder on the file system using various golang standard library functions.
filepath.Walk
ioutil.ReadDir
os.File.Readdir
package main
import (
"fmt"
"io/ioutil"
"log"
"os"
"path/filepath"
)
func main() {
var (
root string
files []string
err error
)
root := "/home/manigandan/golang/samples"
// filepath.Walk
files, err = FilePathWalkDir(root)
if err != nil {
panic(err)
}
// ioutil.ReadDir
files, err = IOReadDir(root)
if err != nil {
panic(err)
}
//os.File.Readdir
files, err = OSReadDir(root)
if err != nil {
panic(err)
}
for _, file := range files {
fmt.Println(file)
}
}
Using filepath.Walk
The path/filepath package provides a handy way to scan all the files
in a directory, it will automatically scan each sub-directories in the
directory.
func FilePathWalkDir(root string) ([]string, error) {
var files []string
err := filepath.Walk(root, func(path string, info os.FileInfo, err error) error {
if !info.IsDir() {
files = append(files, path)
}
return nil
})
return files, err
}
Using ioutil.ReadDir
ioutil.ReadDir reads the directory named by dirname and returns a
list of directory entries sorted by filename.
func IOReadDir(root string) ([]string, error) {
var files []string
fileInfo, err := ioutil.ReadDir(root)
if err != nil {
return files, err
}
for _, file := range fileInfo {
files = append(files, file.Name())
}
return files, nil
}
Using os.File.Readdir
Readdir reads the contents of the directory associated with file and
returns a slice of up to n FileInfo values, as would be returned by
Lstat, in directory order. Subsequent calls on the same file will
yield further FileInfos.
func OSReadDir(root string) ([]string, error) {
var files []string
f, err := os.Open(root)
if err != nil {
return files, err
}
fileInfo, err := f.Readdir(-1)
f.Close()
if err != nil {
return files, err
}
for _, file := range fileInfo {
files = append(files, file.Name())
}
return files, nil
}
Benchmark results.
Get more details on this Blog Post
Even simpler, use path/filepath:
package main
import (
"fmt"
"log"
"path/filepath"
)
func main() {
files, err := filepath.Glob("*")
if err != nil {
log.Fatal(err)
}
fmt.Println(files) // contains a list of all files in the current directory
}
Starting with Go 1.16, you can use the os.ReadDir function.
func ReadDir(name string) ([]DirEntry, error)
It reads a given directory and returns a DirEntry slice that contains the directory entries sorted by filename.
It's an optimistic function, so that, when an error occurs while reading the directory entries, it tries to return you a slice with the filenames up to the point before the error.
package main
import (
"fmt"
"log"
"os"
)
func main() {
files, err := os.ReadDir(".")
if err != nil {
log.Fatal(err)
}
for _, file := range files {
fmt.Println(file.Name())
}
}
Of interest: Go 1.17 (Q3 2021) includes fs.FileInfoToDirEntry():
func FileInfoToDirEntry(info FileInfo) DirEntry
FileInfoToDirEntry returns a DirEntry that returns information from info.
If info is nil, FileInfoToDirEntry returns nil.
Background
Go 1.16 (Q1 2021) will propose, with CL 243908 and CL 243914 , the ReadDir function, based on the FS interface:
// An FS provides access to a hierarchical file system.
//
// The FS interface is the minimum implementation required of the file system.
// A file system may implement additional interfaces,
// such as fsutil.ReadFileFS, to provide additional or optimized functionality.
// See io/fsutil for details.
type FS interface {
// Open opens the named file.
//
// When Open returns an error, it should be of type *PathError
// with the Op field set to "open", the Path field set to name,
// and the Err field describing the problem.
//
// Open should reject attempts to open names that do not satisfy
// ValidPath(name), returning a *PathError with Err set to
// ErrInvalid or ErrNotExist.
Open(name string) (File, error)
}
That allows for "os: add ReadDir method for lightweight directory reading":
See commit a4ede9f:
// ReadDir reads the contents of the directory associated with the file f
// and returns a slice of DirEntry values in directory order.
// Subsequent calls on the same file will yield later DirEntry records in the directory.
//
// If n > 0, ReadDir returns at most n DirEntry records.
// In this case, if ReadDir returns an empty slice, it will return an error explaining why.
// At the end of a directory, the error is io.EOF.
//
// If n <= 0, ReadDir returns all the DirEntry records remaining in the directory.
// When it succeeds, it returns a nil error (not io.EOF).
func (f *File) ReadDir(n int) ([]DirEntry, error)
// A DirEntry is an entry read from a directory (using the ReadDir method).
type DirEntry interface {
// Name returns the name of the file (or subdirectory) described by the entry.
// This name is only the final element of the path, not the entire path.
// For example, Name would return "hello.go" not "/home/gopher/hello.go".
Name() string
// IsDir reports whether the entry describes a subdirectory.
IsDir() bool
// Type returns the type bits for the entry.
// The type bits are a subset of the usual FileMode bits, those returned by the FileMode.Type method.
Type() os.FileMode
// Info returns the FileInfo for the file or subdirectory described by the entry.
// The returned FileInfo may be from the time of the original directory read
// or from the time of the call to Info. If the file has been removed or renamed
// since the directory read, Info may return an error satisfying errors.Is(err, ErrNotExist).
// If the entry denotes a symbolic link, Info reports the information about the link itself,
// not the link's target.
Info() (FileInfo, error)
}
src/os/os_test.go#testReadDir() illustrates its usage:
file, err := Open(dir)
if err != nil {
t.Fatalf("open %q failed: %v", dir, err)
}
defer file.Close()
s, err2 := file.ReadDir(-1)
if err2 != nil {
t.Fatalf("ReadDir %q failed: %v", dir, err2)
}
Ben Hoyt points out in the comments to Go 1.16 os.ReadDir:
os.ReadDir(path string) ([]os.DirEntry, error), which you'll be able to call directly without the Open dance.
So you can probably shorten this to just os.ReadDir, as that's the concrete function most people will call.
See commit 3d913a9 (Dec. 2020):
os: add ReadFile, WriteFile, CreateTemp (was TempFile), MkdirTemp (was TempDir) from io/ioutil
io/ioutil was a poorly defined collection of helpers.
Proposal #40025 moved out the generic I/O helpers to io.
This CL for proposal #42026 moves the OS-specific helpers to os,
making the entire io/ioutil package deprecated.
os.ReadDir returns []DirEntry, in contrast to ioutil.ReadDir's []FileInfo.
(Providing a helper that returns []DirEntry is one of the primary motivations for this change.)
ioutil.ReadDir is a good find, but if you click and look at the source you see that it calls the method Readdir of os.File. If you are okay with the directory order and don't need the list sorted, then this Readdir method is all you need.
From your description, what you probably want is os.Readdirnames.
func (f *File) Readdirnames(n int) (names []string, err error)
Readdirnames reads the contents of the directory associated with file and returns a slice of up to n names of files in the directory, in directory order. Subsequent calls on the same file will yield further names.
...
If n <= 0, Readdirnames returns all the names from the directory in a single slice.
Snippet:
file, err := os.Open(path)
if err != nil {
return err
}
defer file.Close()
names, err := file.Readdirnames(0)
if err != nil {
return err
}
fmt.Println(names)
Credit to SquattingSlavInTracksuit's comment; I'd have suggested promoting their comment to an answer if I could.
A complete example of printing all the files in a directory recursively using Readdirnames
package main
import (
"fmt"
"os"
)
func main() {
path := "/path/to/your/directory"
err := readDir(path)
if err != nil {
panic(err)
}
}
func readDir(path string) error {
file, err := os.Open(path)
if err != nil {
return err
}
defer file.Close()
names, _ := file.Readdirnames(0)
for _, name := range names {
filePath := fmt.Sprintf("%v/%v", path, name)
file, err := os.Open(filePath)
if err != nil {
return err
}
defer file.Close()
fileInfo, err := file.Stat()
if err != nil {
return err
}
fmt.Println(filePath)
if fileInfo.IsDir() {
readDir(filePath)
}
}
return nil
}
I have a really puzzling problem with the filepath.Walk function. It only seems to find directories with that are DOT files. Such as .AndroidStudio or .arduino. It does not file any files or directories other than those if I set the root to something like /home/charles
//Watches ...Recursively walk the filesystem, entrypoint to file watching
func Watches(tops []string) {
dirSet := make(map[string]bool)
for _, top := range tops {
err := filepath.Walk(top, func(path string, f os.FileInfo, err error) error {
if err != nil {
log.Println(err)
return err
}
log.Println("File: ", path)
if f.IsDir() {
//Maps can only have one key that matches, duplicates will be overwritten
dirSet[path] = true
}
return nil
})
if err != nil {
log.Println(err)
}
}
}
Package filepath
import "path/filepath"
func Walk
The files are walked in lexical order,
type WalkFunc
If there was a problem walking to the file or directory named by path,
the incoming error will describe the problem and the function can
decide how to handle that error (and Walk will not descend into that
directory). If an error is returned, processing stops. The sole
exception is when the function returns the special value SkipDir. If
the function returns SkipDir when invoked on a directory, Walk skips
the directory's contents entirely. If the function returns SkipDir
when invoked on a non-directory file, Walk skips the remaining files
in the containing directory.
In Walk, dot (Unicode Full Stop '.' U+002E) directory files are near first in lexical order.
In your WalkFunc you return an error: "If an error is returned, processing stops." For example,
if err != nil {
log.Println(err)
return err
}
Output:
open /home/peter/.cache/dconf: permission denied
To ignore an error return nil. For example,
if err != nil {
log.Println(err)
return nil
}
I have a slightly funky issue in golang. Essentially I have a slice of strings which represent file paths. I then run a cat against those filepaths to combine the files before sorting, deduping, etc.
here is the section of code (where 'applicableReductions' is the string slice):
applicableReductions := []string{}
for _, fqFromListName := range fqFromListNames {
filePath := GetFilePath()
//BROKE CODE GOES HERE
}
applicableReductions = append(applicableReductions, filePath)
fileOut, err := os.Create(toListWriteTmpFilePath)
if err != nil {
return err
}
cat := exec.Command("cat", applicableReductions...)
catStdOut, err := cat.StdoutPipe()
if err != nil {
return err
}
go func(cat *exec.Cmd) error {
if err := cat.Start(); err != nil {
return fmt.Errorf("File reduction error (cat) : %s", err)
}
return nil
}(cat)
// Init Writer & write file
writer := bufio.NewWriter(fileOut)
defer writer.Flush()
_, err = io.Copy(writer, catStdOut)
if err != nil {
return err
}
if err = cat.Wait(); err != nil {
return err
}
fDiff.StandardiseData(fileOut, toListUpdateFolderPath, list.Name)
The above works fine. The problem comes when I try to append a new ele to the array. I have a seperate function which creates a new file from db content which is then added to the applicableReductions slice.
func RetrieveDomainsFromDB(collection *Collection, listName, outputPath string) error {
domains, err := domainReviews.GetDomainsForList(listName)
if err != nil {
return err
}
if len(domains) < 1 {
return ErrNoDomainReviewsForList
}
fh, err := os.OpenFile(outputPath, os.O_RDWR, 0774)
if err != nil {
fh, err = os.Create(outputPath)
if err != nil {
return err
}
}
defer fh.Close()
_, err = fh.WriteString(strings.Join(domains, "\n"))
if err != nil {
return err
}
return nil
}
If I call the above function and append the filePath to the applicableReduction slice, it is in there but doesnt get called by cat.
To clarify, when I put the following where it says BROKE CODE GOES HERE:
if dbSource {
err = r.RetrieveDomainsFromDB(collection, ToListName, filePath)
if err != nil {
return err
continue
}
}
The filepath can be seen when doing fmt.Println(applicableReductions) but the content of the files contents are not seen in the cat output file.
I thought perhaps a delay in the file being written so i tried adding a time.wait, tis didnt help. However the solution I found was to sort the slice, e.g this code above the call to exec cat solves the problem but I dont know why:
sort.Strings(applicableReductions)
I have confirmed all files present on both successful and unsucessful runs the only difference is without the sort, the content of the final appended file is missing
An explanation from a go-pro out there would be very much appreciated, let me know if you need more info, debug - happy to oblige to understand
UPDATE
It has been suggested that this is the same issue as here: Golang append an item to a slice, I think I understand the issue there and I'm not saying this isnt the same but I cannot see the same thing happenning - the slice in question is not touched from outside the main function (e.g. no editing of the slice in RetrieveDomainsFromDB function), I create the slice before a loop, append to it within a loop and then use it after the loop - Ive added an example at the top to show how the slice is built - please could someone clarify where this slice is being copied if this is the case
UPDATE AND CLOSE
Please close question - the issue was unrelated to the use of a string slice. Turns out that I was reading from the final output file before bufio-writer had been flushed (at end of function before defer flush kicked in on function return)
I think the sorting was just re-arranging the problem so I didnt notice it persisted or possibly giving some time for the buffer to flush. Either way sorted now with a manual call to flush.
Thanks for all help provided