i need some help in golan to write a program that can search through a given directory and its subdirectories to look for a particular word word in each of them.
this what i have so far to list the directories and save them as an array. now i want to check each of them to see if it has children, if yes, i should open it until i reach the last level of the tree.
package main
import (
"fmt"
"os"
)
func main() {
d, err := os.Open("/Perkins")
// fmt.Println(d.Readdirnames(-1))
y, err:=d.Readdirnames(-1) //
fmt.Println(y)
for i:=0; i<len(y); i++{
if y[i]!=" "{
Folders:=y[i]
temp,err:=os.Open("/"Folders) //how do i out the array element as a path?
fmt.Println (temp)
fmt.Println(err)
}
}
Note: "/"Folders wouldn't work: "/" + Folders
In your case, this should work better:
temp,err:=os.Open("/Perkins/" + Folders)
(even though 'Folders' is not a good name, 'subfolder' would be more appropriate)
A more efficient way, as commented by chendesheng (see answer), would be (as in this class) to use path/filepath/#Walk:
package main
import (
"fmt"
"os"
"path/filepath"
)
func main() {
filepath.Walk("/Perkins", func(path string, info os.FileInfo, err error) error {
fmt.Println(path)
return nil
})
}
That will list all the file, but you can associate it with a function which will filter those: see this example:
matched, err := filepath.Match("*.mp3", fi.Name())
You can then ignore the files you don't want and proceed only for the ones matching your pattern.
I think you can use filepath.Walk
Walk walks the file tree rooted at root, calling walkFn for each file
or directory in the tree, including root. All errors that arise
visiting files and directories are filtered by walkFn. The files are
walked in lexical order, which makes the output deterministic but
means that for very large directories Walk can be inefficient. Walk
does not follow symbolic links.
Maybe you need this?
http://golang.org/pkg/io/ioutil/#ReadDir
And then check is type from http://golang.org/pkg/os/#FileInfo and do recursive func if it folder
Related
Is there a function in go standard library that lets me do this
a = 'www.my.com/your/stuff'
b = 'www.my.com/your/stuff/123/4'
function(b,a) // /123/4
or
function(URL(b),URL(a)) // /123/4
The following is probably defined in this case
function(a,b) // error ? or ../../
I'm aware that I can use path package for this. But it cannot work in many cases where there is query param, file extension etc.
Basically I'm looking for a path.resolve counterpart for URL
It turns out that the path/filepath package can do this for you. If you ignore the fact that these are URLs and instead treat them like paths, you can use filepath.Rel():
package main
import (
"fmt"
"path/filepath"
)
func main() {
base := "www.my.com/your/stuff"
target := "www.my.com/your/stuff/123/4"
rel, _ := filepath.Rel(base, target)
fmt.Println(rel) // prints "123/4"
}
Playground: https://play.golang.org/p/nnF9zfFAFfc
If you want to treat these paths as actual URLs, you should probably use the net/url package to first parse the path as a URL, then extract the path and use filepath.Rel() on that. This allows you to properly deal with things like queries in the URL string, which would trip up filepath, like so:
package main
import (
"fmt"
"path/filepath"
"net/url"
)
func main() {
url1, _ := url.Parse("http://www.my.com/your/stuff")
url2, _ := url.Parse("http://www.my.com/your/stuff/123/4?query=test")
base := url1.Path
target := url2.Path
rel, _ := filepath.Rel(base, target)
fmt.Println(base) // "/your/stuff"
fmt.Println(target) // "/your/stuff/123/4"
fmt.Println(rel) // "123/4"
}
Playground: https://play.golang.org/p/gnZfk0t8GOZ
As a bonus, filepath.Rel() is smart enough to handle relative paths in the other direction, too:
rel, _ = filepath.Rel(target, base) // rel is now "../.."
Is there an idiomatic way to read a file from the system starting from a (file scheme) url and not a path?
I tried this first:
fileUrlStr := "file:///path/to/file.json"
jsonBuffer, _ := ioutil.ReadFile(fileUrlStr)
This is my current (mostly working version) but I'm concerned there are some gotchas that I'm missing, so I'm hoping there's a more tried and true way to do it:
fileUrlStr := "file:///path/to/file.json"
fileUrl, _ := url.Parse(fileUrlStr)
jsonBuffer, _ := ioutil.ReadFile(fileUrl.Path)
(Bonus if I can support both file:///Users/jdoe/temp.json and file:///c:/WINDOWS/clock.json without having to add code-paths accounting for them)
Using net/url, the solution that you were using, is the correct one.
It's properly deals with hostnames and paths across platforms and also gives you a chance to check the url scheme is the file scheme.
package main
import (
"fmt"
"net/url"
)
func main() {
for _, path := range []string{
"file:///path/to/file.json",
"file:///c:/WINDOWS/clock.json",
"file://localhost/path/to/file.json",
"file://localhost/c:/WINDOWS/clock.avi",
// A case that you probably don't need to handle given the rarity,
// but is a known legacy win32 issue when translating \\remotehost\share\dir\file.txt
"file:////remotehost/share/dir/file.txt",
} {
u, _ := url.ParseRequestURI(path)
fmt.Printf("url:%v\nscheme:%v host:%v Path:%v\n\n", u, u.Scheme, u.Host, u.Path)
}
}
Go provides a directory iteration functionality out of the box, with filepath.Walk in the path/filepath package.
However, filepath.Walk walks the file tree in lexical order. How can I walks the file tree in the order of last-modified date? Thx
PS (after accepting the answer) I think the Go filepath.Walk function should provide a way for people to provide the sorting themselves, like the following answer, in which accepting type ByModTime is all it take for people to sort the files themselves themselves.
I think, you should implement it by yourself, because filepath.Walk doesn't allow you to set order.
Look at Walk method. It calls walk, which is relying on file names from readDirNames. So basically, you should make your own Walk method with another readDirNames logic.
Here's how you get files in the order of last-modified date (note, that I'm ignoring errors):
package main
import (
"fmt"
"os"
"sort"
)
type ByModTime []os.FileInfo
func (fis ByModTime) Len() int {
return len(fis)
}
func (fis ByModTime) Swap(i, j int) {
fis[i], fis[j] = fis[j], fis[i]
}
func (fis ByModTime) Less(i, j int) bool {
return fis[i].ModTime().Before(fis[j].ModTime())
}
func main() {
f, _ := os.Open("/")
fis, _ := f.Readdir(-1)
f.Close()
sort.Sort(ByModTime(fis))
for _, fi := range fis {
fmt.Println(fi.Name())
}
}
I have the following structure:
project/
docs/
index.html
root.html
I'm trying to iterate through this project structure so that I can read the contents of each file to process them. So I want to say "search through the directory project", then it will search through all the files, and only the first level of directories and their files, so if there was another directory with a file inside of docs/, it would ignore it.
Currently, I've tried to accomplish this with the "path/filepath" library:
func traverse(path string, file os.FileInfo, err error) error {
if file, err := os.Open(file.Name()); err == nil {
defer file.Close()
if fileStat, err := file.Stat(); err == nil {
switch mode := fileStat.Mode(); {
case mode.IsDir():
fmt.Println("it be a directory! lets traverse", file.Name())
filepath.Walk(file.Name(), traverse)
case mode.IsRegular():
fmt.Println("the thingy ", file.Name(), " is a file")
}
} else {
return errors.New("failed to check status")
}
}
return errors.New("failed 2 open file/dir?")
}
Then I call it from here:
if err := filepath.Walk("project/", traverse); err != nil {
setupErr("%s", err)
}
Note that I run this executable relative to my test directory, so it's finding the directory okay. My problem is actually when I run it, I get the following:
it be a directory! lets traverse project
it be a directory! lets traverse project
# ^ printed about 20 more times ^
failed 2 open file/dir?
I think my recursion is a little off, and it's not changing into the directory perhaps? Any ideas, if you need any more information just ask and I'll update.
First, it looks like what you want do to contradicts with the code you have. You wrote:
So I want to say "search through the directory project", then it will search through all the files, and only the first level of directories and their files, so if there was another directory with a file inside of docs/, it would ignore it.
Does it mean that you want to iterate only two levels of directories (current and one below) and ignore the rest?
If so then you do not need a recursion, just a simple loop that executes search function over the files within the current directory and for every its subdirectory.
The code that you have walks over the filesystem directory subtree.
Basically, filepath.Walk that you use should do it for you. So you either implement recursive walking or use Walk, but not both.
Second, the recursion is implemented incorrectly in your code. It missing iterating over the directories.
So the code that prints the file names in the current directory and its subdirectories (but not further) is:
package main
import (
"fmt"
"io/ioutil"
)
func main() {
items, _ := ioutil.ReadDir(".")
for _, item := range items {
if item.IsDir() {
subitems, _ := ioutil.ReadDir(item.Name())
for _, subitem := range subitems {
if !subitem.IsDir() {
// handle file there
fmt.Println(item.Name() + "/" + subitem.Name())
}
}
} else {
// handle file there
fmt.Println(item.Name())
}
}
}
Walk walks the file tree rooted at root, calling walkFn for each file
or directory in the tree, including root. All errors that arise
visiting files and directories are filtered by walkFn. The files are
walked in lexical order, which makes the output deterministic but
means that for very large directories Walk can be inefficient. Walk
does not follow symbolic links.
When in a source file $PWD/dir/src.go I use
os.Open("myfile.txt")
it looks for myfile.txt in $PWD (which looks normal).
Is there way to tell Go to look for myfile.txt in the same directory as src.go ? I need something like __FILE__ in Ruby.
Go is not an interpreted language so looking for a file in the same location as the source file doesn't make any sense. The go binary is compiled and the source file doesn't need to be present for the binary to run. Because of that Go doesn't come with an equivalent to FILE. The runtime.Caller function returns the file name at the time the binary was compiled.
I think perhaps if we understood why you actually wanted this functionality we could advise you better.
A possible substitute skeleton:
func __FILE__() (fn string) {
_, fn, _, _ = runtime.Caller(0)
return
}
Details here.
Use package osext
It's providing function ExecutableFolder() that returns an absolute path to folder where the currently running program executable reside (useful for cron jobs). It's cross platform.
Online documentation
package main
import (
"github.com/kardianos/osext"
"fmt"
"log"
)
func main() {
folderPath, err := osext.ExecutableFolder()
if err != nil {
log.Fatal(err)
}
fmt.Println(folderPath)
}
You can also get full executable path (similar to __FILE__):
package main
import (
"github.com/kardianos/osext"
"fmt"
)
func main() {
exeAbsolutePath, _ := osext.Executable()
fmt.Println(exeAbsolutePath)
}