Find a file by regex in Golang, given the regex and path - go

I'm trying to implement cache busting, via the following:
on the front end of things, I insert, via JS plugin, content hash to the asset filenames (e.g. bundle.1947shkey7.js)
in the HTML file, use some template tag to create the asset (e.g. <script>) tags for me
from the Golang side, have template func that will extract the directory path dirpath, create a regex from the filename filenameRegex, and use dirpath,filenameRegex to find the file and form the tag for it.
Most of this is already working, except I have no idea how best to do that regex-based find.
The regex for the file is something like [name].\\w+.[ext], for reference. This works and I've tested it.
However, how to use that regex and the dirpath to get the actual file's path? I know we can do something like
filepath.Walk(dirpath, func(path string, info os.FileInfo, err error) error {
// logic to check info.IsDir(), info.Name(), and written actual filename
})
but there's one problem with this: the complexity. If I understand this correctly, this is going to execute for every file in directory dirname!
Is this the only way to find the asset filename via regex or is there a much simpler way?

You can use something like this, where you can customize the callback to your
needs:
package main
import (
"io/fs"
"path/filepath"
)
func glob(root string, fn func(string)bool) []string {
var files []string
filepath.WalkDir(root, func(s string, d fs.DirEntry, e error) error {
if fn(s) {
files = append(files, s)
}
return nil
})
return files
}
func main() {
files := glob(".", func(s string) bool {
return filepath.Ext(s) == ".js"
})
for _, file := range files {
println(file)
}
}

Related

Failing to understand go embed

Goal is to embed a directory tree in my binary, then copy it to a user directory later. It consists mostly of text files. I am attempting to display contents of the files in that subdirectory within the project, named .config (hence the use of go:embed all).
As I understand Go embedding, the following should work outside its own directory, but when executed, lists the name of the first
file in the directory tree but cannot open it or display its contents. Within its own project directory it works fine.
//go:embed all:.config
var configFiles embed.FS
func main() {
ls(configFiles)
}
func ls(files embed.FS) error {
fs.WalkDir(files, ".", func(path string, d fs.DirEntry, err error) error {
if err != nil {
return err
}
if !d.IsDir() {
show(path) // Display contents of file
}
return nil
})
return nil
}
(Complete code at https://go.dev/play/p/A0HzD0rbvX- but it doesn't work because there's no .config directory)
The program walks the embedded file system, but opens files using the operating system. Fix by opening the file in the file system.
Pass the file system to show:
if !d.IsDir() {
show(files, path) // <-- pass files here
}
Open the file using the file system:
func show(files fs.FS, filename string) { // <-- add arg here
f, err := files.Open(filename) // <-- FS.Open

How to find a sibling file when the os.Getwd() is different in different environments

myprogram/
|
|-main.go
|-dir1/
|-data/
|-datafile.json
|-runner.go
|-runner_test.go
In runner.go, I have a simple function that reads the datafile.json. Something like
func GetPayload() (string, err) {
dBytes, dErr := ioutil.ReadFile("dir1/data/datafile.json")
if dErr != nil { return nil, dErr}
return dBytes, nil
}
I'm using Go in a Lambda with a structure similar to above. When the Lambda runs in its actual environment, it starts at main.go, and then invokes GetPayload() from runner.go. However, I have a test in a simple worker node machine in runner_test.go that also hits GetPayload() .
During "normal" execution (from main.go) - this works OK. However, when GetPayload() is invoked from runner_test.go, it errors, saying
open dir1/data/datafile.json no such file or directory
This makes sense, because during the test, the working directory is the directory that houses runner_test.go, which is data/, so there is no dir1 as a child of it. I've been trying to play with using os.Getwd() and getting the paths from there like:
pwd, _ := os.Getwd()
dBytes, dErr := ioutil.ReadFile(pwd + "dir1/data/datafile.json")
But again, that won't work, because for runner_test.go pwd is user/myname/myprogram/dir1, but from main.go, it turns up as user/myname/myprogram.
Any idea how I can find and open datafile.json from within GetPayload() in any environment? I could pass an optional parameter to GetPayload() but if possible, it'd be great to avoid that.
If the file is static (meaning that it doesn't need to change after you build the program), you can embed it into the built program. This means you no longer have to worry about run-time file paths.
import (
"embed"
)
//go:embed data/*
var dataFiles embed.FS
func GetPayload() (string, err) {
dBytes, dErr := dataFiles.ReadFile(dataFiles, "data/datafile.json")
if dErr != nil { return nil, dErr}
return dBytes, nil
}
Now the files in your data/ directory are embedded in this variable dataFiles which acts as a read-only file system.
For more info:
Read more about embed in the package documentation overview.
Read my answer about "when to use embed"
For data files that your program needs during runtime, either use a fixed directory and refer to that, or accept a command line argument or some sort of configuration that tells you where the file is.
When running unit tests, the wd is the directory containing the test file. One convention is to use a testdata/ directory under the directory containing the test, and put all data files there. That way you can refer to that file from the test by using testdata/datafile.json.
You can use a copy of the file you need during runtime as your test file, or you can use a symlink from the runtime data file to the test file under the testdata/ dir.
For data files that your program needs during runtime, either use a fixed
directory and refer to that
Someone made this suggestion, which I agree with. To that end, you can use
something like this:
package main
import (
"os"
"path/filepath"
)
func main() {
d, err := os.UserCacheDir()
if err != nil {
panic(err)
}
d = filepath.Join(d, "file.json")
f, err := os.Open(d)
if err != nil {
panic(err)
}
defer f.Close()
os.Stdout.ReadFrom(f)
}
https://golang.org/pkg/os#UserCacheDir
https://golang.org/pkg/os#UserConfigDir

API to get the module name

Is there an API to get the module name of a project which uses go 1.11 module system?
so I need to get abc.com/a/m from the module definition module abc.com/a/m in go.mod file.
As of this writing, I am not aware of any exposed APIs for that. However, looking at go mod sources, there is a function that can be quite useful in Go mod source file
// ModulePath returns the module path from the gomod file text.
// If it cannot find a module path, it returns an empty string.
// It is tolerant of unrelated problems in the go.mod file.
func ModulePath(mod []byte) string {
//...
}
func main() {
src := `
module github.com/you/hello
require rsc.io/quote v1.5.2
`
mod := ModulePath([]byte(src))
fmt.Println(mod)
}
Which outputs github.com/you/hello
Try this?
package main
import (
"fmt"
"io/ioutil"
"os"
modfile "golang.org/x/mod/modfile"
)
const (
RED = "\033[91m"
RESET = "\033[0m"
)
func main() {
modName := GetModuleName()
fmt.Fprintf(os.Stdout, "modName=%+v\n", modName)
}
func exitf(beforeExitFunc func(), code int, format string, args ...interface{}) {
beforeExitFunc()
fmt.Fprintf(os.Stderr, RED+format+RESET, args...)
os.Exit(code)
}
func GetModuleName() string {
goModBytes, err := ioutil.ReadFile("go.mod")
if err != nil {
exitf(func() {}, 1, "%+v\n", err)
}
modName := modfile.ModulePath(goModBytes)
fmt.Fprintf(os.Stdout, "modName=%+v\n", modName)
return modName
}
If your starting point is a go.mod file and you are asking how to parse it, I would suggest starting with go mod edit -json, which outputs a particular go.mod file in JSON format. Here is the documentation:
https://golang.org/cmd/go/#hdr-Edit_go_mod_from_tools_or_scripts
Alternatively, you could use rogpeppe/go-internal/modfile, which is a go package that can parse a go.mod file, and which is used by rogpeppe/gohack and some other tools from the broader community.
Issue #28101 I think tracks adding a new API to the Go standard library to parse go.mod files.
Here is a snippet of the documentation for go mod edit -json:
The -json flag prints the final go.mod file in JSON format instead of
writing it back to go.mod. The JSON output corresponds to these Go
types:
type Module struct {
Path string
Version string
}
type GoMod struct {
Module Module
Go string
Require []Require
Exclude []Module
Replace []Replace
}
type Require struct {
Path string
Version string
Indirect bool
}
Here is an example snippet of JSON output from go mod edit -json that shows the actual module path (aka module name), which was your original question:
{
"Module": {
"Path": "rsc.io/quote"
},
In this case, the module name is rsc.io/quote.
As of Go 1.12 (for those finding this via a search who are using modules but not necessarily the older version the OP mentioned), the runtime/debug package includes functionality for getting information about the build, including the module name. For example:
import (
"fmt"
"runtime/debug"
)
func main() {
info, _ := debug.ReadBuildInfo()
fmt.Printf("info: %+v", info.Main.Path)
}
You can run this example on the playground: https://go.dev/play/p/5oGbCRxSnjM
For more information, see the documentation for "runtime/debug".BuildInfo

check if given path is a subdirectory of another in golang

Say we have two paths:
c:\foo\bar\baz and c:\foo\bar
Is there any package/method that will help me determine if one is a subdirectory of another? I am looking at a cross-platform option.
You could try and use path.filepath.Rel():
func Rel(basepath, targpath string) (string, error)
Rel returns a relative path that is lexically equivalent to targpath when joined to basepath with an intervening separator.
That is, Join(basepath, Rel(basepath, targpath)) is equivalent to targpath itself
That means Rel("c:\foo\bar", "c:\foo\bar\baz") should be baz, meaning a subpath completely included in c:\foo\bar\baz, and without any '../'.
The same would apply for unix paths.
That would make c:\foo\bar\baz a subdirectory of c:\foo\bar.
I haven't found a reliable solution for all types of paths, but the best you can get is by using filepath.Rel as VonC suggested.
It works if both filepaths are either absolute or relative (mixing is not allowed) and works on both Windows and Linux:
func SubElem(parent, sub string) (bool, error) {
up := ".." + string(os.PathSeparator)
// path-comparisons using filepath.Abs don't work reliably according to docs (no unique representation).
rel, err := filepath.Rel(parent, sub)
if err != nil {
return false, err
}
if !strings.HasPrefix(rel, up) && rel != ".." {
return true, nil
}
return false, nil
}
Absolute windows paths that start with a drive letter will require an additional check though.
You can use the function path.filepath.Match()
Match reports whether name matches the shell file name pattern.
For example:
pattern := "C:\foo\bar" + string(filepath.Separator) + "*"
matched, err := filepath.Match(pattern, "C:\foo\bar\baz")
Where matched should be true.
If you first canonicalize both paths by calling filepath.EvalSymlinks() and filepath.Abs() on them, you can simply append a '/' to each one, since the UNIX kernel itself forbids a '/' within a path component. At this point you can simply use strings.HasPrefix() on the two paths, in either order.
Try this code. This checks if either is a sub-directory of the other. Try changing values of both base and path and the results should be valid.
package main
import (
"fmt"
"path/filepath"
"strings"
)
func main() {
base := "/b/c/"
path := "/a/b/c/d"
if len(base) > len(path) {
base, path = path, base
}
rel, err := filepath.Rel(base, path)
fmt.Printf("Base %q: Path %q: Rel %q Err %v\n", base, path, rel, err)
if err != nil {
fmt.Println("PROCEED")
return
}
if strings.Contains(rel, "..") {
fmt.Println("PROCEED")
return
}
fmt.Println("DENY")
}

Open a file in the same directory as the .go source file in Go

When in a source file $PWD/dir/src.go I use
os.Open("myfile.txt")
it looks for myfile.txt in $PWD (which looks normal).
Is there way to tell Go to look for myfile.txt in the same directory as src.go ? I need something like __FILE__ in Ruby.
Go is not an interpreted language so looking for a file in the same location as the source file doesn't make any sense. The go binary is compiled and the source file doesn't need to be present for the binary to run. Because of that Go doesn't come with an equivalent to FILE. The runtime.Caller function returns the file name at the time the binary was compiled.
I think perhaps if we understood why you actually wanted this functionality we could advise you better.
A possible substitute skeleton:
func __FILE__() (fn string) {
_, fn, _, _ = runtime.Caller(0)
return
}
Details here.
Use package osext
It's providing function ExecutableFolder() that returns an absolute path to folder where the currently running program executable reside (useful for cron jobs). It's cross platform.
Online documentation
package main
import (
"github.com/kardianos/osext"
"fmt"
"log"
)
func main() {
folderPath, err := osext.ExecutableFolder()
if err != nil {
log.Fatal(err)
}
fmt.Println(folderPath)
}
You can also get full executable path (similar to __FILE__):
package main
import (
"github.com/kardianos/osext"
"fmt"
)
func main() {
exeAbsolutePath, _ := osext.Executable()
fmt.Println(exeAbsolutePath)
}

Resources