Say we have two paths:
c:\foo\bar\baz and c:\foo\bar
Is there any package/method that will help me determine if one is a subdirectory of another? I am looking at a cross-platform option.
You could try and use path.filepath.Rel():
func Rel(basepath, targpath string) (string, error)
Rel returns a relative path that is lexically equivalent to targpath when joined to basepath with an intervening separator.
That is, Join(basepath, Rel(basepath, targpath)) is equivalent to targpath itself
That means Rel("c:\foo\bar", "c:\foo\bar\baz") should be baz, meaning a subpath completely included in c:\foo\bar\baz, and without any '../'.
The same would apply for unix paths.
That would make c:\foo\bar\baz a subdirectory of c:\foo\bar.
I haven't found a reliable solution for all types of paths, but the best you can get is by using filepath.Rel as VonC suggested.
It works if both filepaths are either absolute or relative (mixing is not allowed) and works on both Windows and Linux:
func SubElem(parent, sub string) (bool, error) {
up := ".." + string(os.PathSeparator)
// path-comparisons using filepath.Abs don't work reliably according to docs (no unique representation).
rel, err := filepath.Rel(parent, sub)
if err != nil {
return false, err
}
if !strings.HasPrefix(rel, up) && rel != ".." {
return true, nil
}
return false, nil
}
Absolute windows paths that start with a drive letter will require an additional check though.
You can use the function path.filepath.Match()
Match reports whether name matches the shell file name pattern.
For example:
pattern := "C:\foo\bar" + string(filepath.Separator) + "*"
matched, err := filepath.Match(pattern, "C:\foo\bar\baz")
Where matched should be true.
If you first canonicalize both paths by calling filepath.EvalSymlinks() and filepath.Abs() on them, you can simply append a '/' to each one, since the UNIX kernel itself forbids a '/' within a path component. At this point you can simply use strings.HasPrefix() on the two paths, in either order.
Try this code. This checks if either is a sub-directory of the other. Try changing values of both base and path and the results should be valid.
package main
import (
"fmt"
"path/filepath"
"strings"
)
func main() {
base := "/b/c/"
path := "/a/b/c/d"
if len(base) > len(path) {
base, path = path, base
}
rel, err := filepath.Rel(base, path)
fmt.Printf("Base %q: Path %q: Rel %q Err %v\n", base, path, rel, err)
if err != nil {
fmt.Println("PROCEED")
return
}
if strings.Contains(rel, "..") {
fmt.Println("PROCEED")
return
}
fmt.Println("DENY")
}
Related
Context: I'm trying to resolve this issue.
In other words, there's a NormalizeJsonString() for JSON strings (see this for more context:
// Takes a value containing JSON string and passes it through
// the JSON parser to normalize it, returns either a parsing
// error or normalized JSON string.
func NormalizeJsonString(jsonString interface{}) (string, error) {
that allows to have the following code:
return structure.NormalizeJsonString(old) == structure.NormalizeJsonString(new)
but it doesn't work for strings that are proto files (all proto files are guaranteed to have just one message definition). For example, I could see:
syntax = "proto3";
- package bar.proto;
+ package bar.proto;
option java_outer_classname = "FooProto";
message Foo {
...
- int64 xyz = 3;
+ int64 xyz = 3;
Is there NormalizeProtoString available in some Go SDKs? I found MessageDifferencer but it's in C++ only. Another option I considered was to replace all new lines / group of whitespaces with a single whitespace but it's a little bit hacky.
To do this in a semantic fashion, the proto definitions should really be parsed. Naively stripping and/or replacing whitespace may get you somewhere, but likely will have gotchas.
As far as I'm aware the latest official Go protobuf package don't have anything to handle parsing protobuf definitions - the protoc compiler handles that side of affairs, and this is written in C++
There would be options to execute the protoc compiler to get hold of the descriptor set output (e.g. protoc --descriptor_set_out=...), however I'm guessing this would also be slightly haphazard considering it requires one to have protoc available - and version differences could potentially cause problems too.
Assuming that is no go, one further option is to use a 3rd party parser written in Go - github.com/yoheimuta/go-protoparser seems to handle things quite well. One slight issue when making comparisons is that the parser records meta information about source line + column positions for each type; however it is relatively easy to make a comparison and ignore these, by using github.com/google/go-cmp
For example:
package main
import (
"fmt"
"log"
"os"
"github.com/google/go-cmp/cmp"
"github.com/google/go-cmp/cmp/cmpopts"
"github.com/yoheimuta/go-protoparser/v4"
"github.com/yoheimuta/go-protoparser/v4/parser"
"github.com/yoheimuta/go-protoparser/v4/parser/meta"
)
func main() {
if err := run(); err != nil {
log.Fatal(err)
}
}
func run() error {
proto1, err := parseFile("example1.proto")
if err != nil {
return err
}
proto2, err := parseFile("example2.proto")
if err != nil {
return err
}
equal := cmp.Equal(proto1, proto2, cmpopts.IgnoreTypes(meta.Meta{}))
fmt.Printf("equal: %t", equal)
return nil
}
func parseFile(path string) (*parser.Proto, error) {
f, err := os.Open(path)
if err != nil {
return nil, err
}
defer f.Close()
return protoparser.Parse(f)
}
outputs:
equal: true
for the example you provided.
myprogram/
|
|-main.go
|-dir1/
|-data/
|-datafile.json
|-runner.go
|-runner_test.go
In runner.go, I have a simple function that reads the datafile.json. Something like
func GetPayload() (string, err) {
dBytes, dErr := ioutil.ReadFile("dir1/data/datafile.json")
if dErr != nil { return nil, dErr}
return dBytes, nil
}
I'm using Go in a Lambda with a structure similar to above. When the Lambda runs in its actual environment, it starts at main.go, and then invokes GetPayload() from runner.go. However, I have a test in a simple worker node machine in runner_test.go that also hits GetPayload() .
During "normal" execution (from main.go) - this works OK. However, when GetPayload() is invoked from runner_test.go, it errors, saying
open dir1/data/datafile.json no such file or directory
This makes sense, because during the test, the working directory is the directory that houses runner_test.go, which is data/, so there is no dir1 as a child of it. I've been trying to play with using os.Getwd() and getting the paths from there like:
pwd, _ := os.Getwd()
dBytes, dErr := ioutil.ReadFile(pwd + "dir1/data/datafile.json")
But again, that won't work, because for runner_test.go pwd is user/myname/myprogram/dir1, but from main.go, it turns up as user/myname/myprogram.
Any idea how I can find and open datafile.json from within GetPayload() in any environment? I could pass an optional parameter to GetPayload() but if possible, it'd be great to avoid that.
If the file is static (meaning that it doesn't need to change after you build the program), you can embed it into the built program. This means you no longer have to worry about run-time file paths.
import (
"embed"
)
//go:embed data/*
var dataFiles embed.FS
func GetPayload() (string, err) {
dBytes, dErr := dataFiles.ReadFile(dataFiles, "data/datafile.json")
if dErr != nil { return nil, dErr}
return dBytes, nil
}
Now the files in your data/ directory are embedded in this variable dataFiles which acts as a read-only file system.
For more info:
Read more about embed in the package documentation overview.
Read my answer about "when to use embed"
For data files that your program needs during runtime, either use a fixed directory and refer to that, or accept a command line argument or some sort of configuration that tells you where the file is.
When running unit tests, the wd is the directory containing the test file. One convention is to use a testdata/ directory under the directory containing the test, and put all data files there. That way you can refer to that file from the test by using testdata/datafile.json.
You can use a copy of the file you need during runtime as your test file, or you can use a symlink from the runtime data file to the test file under the testdata/ dir.
For data files that your program needs during runtime, either use a fixed
directory and refer to that
Someone made this suggestion, which I agree with. To that end, you can use
something like this:
package main
import (
"os"
"path/filepath"
)
func main() {
d, err := os.UserCacheDir()
if err != nil {
panic(err)
}
d = filepath.Join(d, "file.json")
f, err := os.Open(d)
if err != nil {
panic(err)
}
defer f.Close()
os.Stdout.ReadFrom(f)
}
https://golang.org/pkg/os#UserCacheDir
https://golang.org/pkg/os#UserConfigDir
os.Chtimes always to follow symlinks and change the real files timestamp.
Is there a method to change the symlinks timestamp in?
Just like touch -h does.
Not sure it's possible, at least from the syscall package.
Looking at the source-code for say syscall.Chtimes:
func Chtimes(name string, atime time.Time, mtime time.Time) error {
var utimes [2]syscall.Timespec
utimes[0] = syscall.NsecToTimespec(atime.UnixNano())
utimes[1] = syscall.NsecToTimespec(mtime.UnixNano())
if e := syscall.UtimesNano(fixLongPath(name), utimes[0:]); e != nil {
return &PathError{"chtimes", name, e}
}
return nil
}
duplicating this code - and removing the fixLongPath call which I assumed followed the symlinks - still affects the target file, not the source symlink.
Even trying this operation on a symlink which points to a non-existent file, returns a runtime error no such file or directory.
A CGO pkg - could, but that seems overkill.
If you use linux, you can use golang.org/x/sys/unix package, which provides Lutimes
import "golang.org/x/sys/unix"
unix.Lutimes(symlink, nil)
You can check if a symlink exists and if so, remove it and create another one.
if _, err := os.Lstat(symlinkPath); err == nil {
os.Remove(symlinkPath)
}
err := os.Symlink(filePath, symlinkPath)
if err != nil {
fmt.Println(err)
}
I'm trying to scan some folders and sort them to find the highest version number. {"10.1","9.6","7.2"} and then build a path. However, What I get has [] brackets in the path and I need to get rid of those.
Here's what I'm getting:
C:\Program Files\PostgreSQL\[10.1]\bin\psql.exe
root := "C:/Program Files/PostgreSQL"
files, err := ioutil.ReadDir(root)
if err != nil {
return "", err
}
folders := []float64{}
for _, f := range files {
if f.IsDir() {
if converted, err := strconv.ParseFloat(f.Name(),64); err == nil {
folders = append(folders, converted)
}
}
}
sort.Float64s(folders)
log.Println(folders[len(folders)-1:])
highestVersion := fmt.Sprintf("%v",folders[len(folders)-1:])
execPath = filepath.Join(root, highestVersion, "bin", "psql.exe")
log.Println(execPath)
The issues are on this line:
highestVersion := fmt.Sprintf("%v",folders[len(folders)-1:])
The %v format specifier, as some people have mentioned, is shorthand for "value". Now let's look at your value:
folders[len(folders)-1:]
What you are saying here is, "take a slice from folders starting at len(folders-1)". Your variable is a slice that only contains the last item in folders.
Slices are printed by using brackets around the values, and since you have one value, it prints the value surrounded by square brackets.
If you want to print just the float contained in that location, you should remove the colon as specified in a comment. I would recommend printing using the fmt verb %f or %g, depending on your use case.
More information can be found in the pkg/fmt docs about what verbs are available to printf and other related functions.
One possible approach would be to use a regular expression to ensure that each path has the expected format and to extract the version number as a float via a submatch (matching group), then sort the path strings by their floating point version number value and return the highest one.
For example:
func main() {
paths := []string{
`C:\Program Files\PostgreSQL\[1.2]\bin\psql.exe`,
`C:\Program Files\PostgreSQL\[9.6]\bin\psql.exe`,
`C:\Program Files\PostgreSQL\[10.1]\bin\psql.exe`,
`C:\Program Files\PostgreSQL\[7.2]\bin\psql.exe`,
}
sort.Slice(paths, func(i, j int) bool {
return parseVersion(paths[i]) >= parseVersion(paths[j])
})
fmt.Printf("OK: highest version path = %s\n", paths[0])
// OK: highest version path = C:\Program Files\PostgreSQL\[10.1]\bin\psql.exe
}
var re = regexp.MustCompile(`C:\\Program Files\\PostgreSQL\\\[(\d+\.\d+)\]\\bin\\psql.exe`)
func parseVersion(s string) float32 {
match := re.FindStringSubmatch(s)
if match == nil {
panic(fmt.Errorf("invalid path %q", s))
}
version, err := strconv.ParseFloat(match[1], 32)
if err != nil {
panic(err)
}
return float32(version)
}
Of course, you could modify the path regular expression to match different location patterns if that matters for your use case.
I am creating Go command-line app and I need to generate some stuff in the current directory (the directory which the user execute the commands from)
to get the pwd I need to use
os.Getwd()
but this give me path like
/Users/s05333/go/src/appcmd
and I need path like this
/Users/s05333/go/src/
which option I've in this case?
Omit the last string after the / or there is better way in Go?
Take a look at the filepath package, particularly filepath.Dir:
wd,err := os.Getwd()
if err != nil {
panic(err)
}
parent := filepath.Dir(wd)
Per the docs:
Dir returns all but the last element of path, typically the path's directory.
Another option is the path package:
package main
import "path"
func main() {
s := "/Users/s05333/go/src/appcmd"
t := path.Dir(s)
println(t == "/Users/s05333/go/src")
}
https://golang.org/pkg/path#Dir