I have found that there are lots of similar function in the package path and the package path/filepath. I have tried several common paths like /var/log/something but did not find any differences. When should one use path directly and when should one use filepath instead?
What is the difference?
While functionally similar, path and path/filepath offer differing implementations. Filepath depends on the os package to choose the target runtime's file separators and other differing components when dealing with path strings.
You can look as the os source to see that there are differing implementations for various utility functions. This allows operating system specific details to be abstracted away by the library and helps achieve portability. The path/filepath dependency graph illustrates how the package depends upon the os package. You can compare this with the path dependency graph. I would encourage you to go into the filepath and path source code to observe this relationship.
When do I use each?
You should use filepath when working with files. This ensures your paths will be matched with actual files regardless of the underlying runtime. The path library should be used within models or when paths may be serialized or communicated with other programs. This ensures that a single formatting scheme is used regardless of what platform the programming is running on. Having a consistent format makes reasoning about models more generic and easier to understand.
I find path/filepath useful with Windows, as it handles Slash or Backslash,
while path only handles Slash:
package main
import (
"path"
"path/filepath"
)
func main() {
{ // example 1
s := filepath.Dir(`C:\go\bin`)
println(s == `C:\go`)
}
{ // example 2
s := filepath.Dir("C:/go/bin")
println(s == `C:\go`)
}
{ // example 3
s := path.Dir("C:/go/bin")
println(s == "C:/go")
}
{ // example 4
s := path.Dir(`C:\go\bin`)
println(s == ".")
}
}
https://pkg.go.dev/path
Package path implements utility routines for manipulating slash-separated paths.
The path package should only be used for paths separated by forward slashes, such as the paths in URLs. This package does not deal with Windows paths with drive letters or backslashes; to manipulate operating system paths, use the path/filepath package.
https://pkg.go.dev/filepath
Package filepath implements utility routines for manipulating filename paths in a way compatible with the target operating system-defined file paths.
The filepath package uses either forward slashes or backslashes, depending on the operating system. To process paths such as URLs that always use forward slashes regardless of the operating system, see the path package.
Related
The current documentation of both functions reads very similar to each another:
System.IOUtils.TPath.IsUNCRooted
System.IOUtils.TPath.IsUNCPath
Both are static members of the same class, with one of them decorated inline, so I wouldn't think they are separate implementations with equivalent functionality you often find across various Delphi classes (although examples within a common class do exist in Embarcadero's standard library).
Specifically, I can't come up with a case where a path is a valid UNC path but not a rooted UNC path. So what does IsUNCRooted even mean?
IsUNCRooted only checks if the parameter starts with an UNC sequence, while IsUNCPath also checks the rest for valid path names. So a valid UNCPath is indeed also UNC rooted, but not always the other way round.
I am developing a go package, which is a little bit complex and thus I want to organize the source code into multiple directories.
However, I don't want the users of the package to have to use too long imports. Anyways, the internal structure of the package isn't their concern.
Thus, my package structure looks so:
subDir1
subSubDir1
subSubDir2
subDir2
subSubDir3
...and so on. All of them have their exported calls.
I would like to avoid that my users have to import
import (
"mypackage/subDir1"
"mypackage/subDir1/subSubDir2"
)
...and so on.
I only want, if they want to use an exported function from my package, they should have access all of them by simply importing mypackage.
I tried that I declare package mypackage in all of the .go files. Thus, I had source files in different directories, but with the same package declaration.
In this case, the problem what I've confronted was that I simply couldn't import multiple directories from the same package. It said:
./src1.go:6:15: error: redefinition of ‘mypackage’
"mypackage/mysubdir1"
^
./src1.go:4:10: note: previous definition of ‘mypackage’ was here
"mypackage"
^
./src1.go:5:15: error: redefinition of ‘mypackage’
"mypackage/mysubdir2"
^
./src1.go:4:10: note: previous definition of ‘mypackage’ was here
"mypackage"
^
Is it somehow possible?
You should not do this in any case, as the language spec allows a compiler implementation to reject such constructs. Quoting from Spec: Package clause:
A set of files sharing the same PackageName form the implementation of a package. An implementation may require that all source files for a package inhabit the same directory.
Instead "structure" your file names to mimic the folder structure; e.g. instead of files of
foo/foo1.go
foo/bar/bar1.go
foo/bar/bar2.go
You could simply use:
foo/foo1.go
foo/bar-bar1.go
foo/bar-bar2.go
Also if your package is so big that you would need multiple folders to even "host" the files of the package implementation, you should really consider not implementing it as a single package, but break it into multiple packages.
Also note that Go 1.5 introduced internal packages. If you create a special internal subfolder inside your package folder, you may create any number of subpackages inside that (even using multiple levels). Your package will be able to import and use them (or to be more precise all packages rooted at your package folder), but no one else outside will be able to do so, it would be a compile time error.
E.g. you may create a foo package, have a foo/foo.go file, and foo/internal/bar package. foo will be able to import foo/internal/bar, but e.g. boo won't. Also foo/baz will also be able to import and use foo/internal/bar because it's rooted at foo/.
So you may use internal packages to break down your big package into smaller ones, effectively grouping your source files into multiple folders. Only thing you have to pay attention to is to put everything your package wants to export into the package and not into the internal packages (as those are not importable / visible from the "outside").
Inside your package source code, you have to differentiate your source directories by renamed imports. You can declare the same package mypackage in all of your source files (even if they are in different directories).
However, while you import them, you should give an induvidual names to the directories. In your source src1.go, import the other directories on this way:
import (
"mypackage"
submodule1 "mypackage/mySubDir"
)
And you will be able to reach the API defined in "mypackage" as mypackage.AnyThing(), and the API defined in mySubDir as submodule1.AnyThing().
The external world (i.e. the users of your package) will see all exported entities in myPackage.AnyThing().
Avoid namespace collisions. And use better understable, intuitive naming as in the example.
Yes, this is doable without any problems, just invoke the Go compiler by hand, that is not via the go tool.
But the best advice is: Don't do that. It's ugly and unnecessarily complicated. Just design your package properly.
Addendum (because the real intention of this answer seems to get lost sometimes, maybe because irony is too subtle): Don't do that!! This is an incredible stupid idea! Stop fighting the tools! Everybody will rightfully hate you if you do that! Nobody will understand your code or be able to compile it! Just because something is doable in theory doesn't mean this is a sensible idea in any way. Not even for "learning purpose"! You probably even don't know how to invoke the Go compiler by hand and if you figure it out it will be a major pita.
I have some questions on package naming for external Go libraries.
I am interested if using generic names like "text" is considered a good practice? Having in mind that I cannot declare a "nested package" and that the library I am building deals with text processing, is it ok to have the package named "text" or should I stick to the library name as a package name too?
I am building a set of libraries (different projects) and I want to combine them under the same package. Is this also problematic? I am new to the Go community and am still not sure if package pollution is a problem or not (I do not see a problem as long as I import few packages in my code).
The reference on that naming topic is "blog: Package names"
It includes:
Avoid unnecessary package name collisions.
While packages in different directories may have the same name, packages that are frequently used together should have distinct names. This reduces confusion and the need for local renaming in client code. For the same reason, avoid using the same name as popular standard packages like io or http.
Check also your package publishing practice, as it will help disambiguate your "text" package from others.
As illustrated in "An Introduction to Programming in Go / Packages":
math is the name of a package that is part of Go's standard distribution, but since Go packages can be hierarchical we are safe to use the same name for our package. (The real math package is just math, ours is golang-book/chapter11/math)
When we import our math library we use its full name (import "golang-book/chapter11/math"), but inside of the math.go file we only use the last part of the name (package math).
We also only use the short name math when we reference functions from our library. If we wanted to use both libraries in the same program Go allows us to use an alias:
import m "golang-book/chapter11/math"
func main() {
xs := []float64{1,2,3,4}
avg := m.Average(xs)
fmt.Println(avg)
}
m is the alias.
As mentioned in the comments by elithrar, Dave Cheney has some additional tips:
In other languages it is quite common to ensure your package has a unique namespace by prefixing it with your company name, say com.sun.misc.Unsafe.
If everyone only writes packages corresponding to domains that they control, then there is little possibility of a collision.
In Go, the convention is to include the location of the source code in the package’s import path, ie
$GOPATH/src/github.com/golang/glog
This is not required by the language, it is just a feature of go get.
I am trying to add some support for D programming language to my vim config. For autocompletion I need to detect packages that are included. This is not exactly hard to do in simple case:
import std.stdio;
import std.conv;
My config:
set include=^\\s*import
set includeexpr=substitute(v:fname,'\\.','/','g')
Works great.
However, imports can have more complicated format, for example:
package import std.container, std.stdio = io, std.conv;
I was not able to find a simple way to parse this with include and includeexpr.
Also there is a second problem: import can have different access modifiers, like public and private. VIM scans included files recursively, import statements from included files are parsed too. But I need to distinguish between the file I am working with now and files which are scanned automatically: in current file all imports should be detected, but in other files only public import statements should add more files to the search.
Thanks for help.
Update
It's a shame if this can not be done without full parsers. Essentially, I only need two things:
ability to return an array from includeexpr instead of one file name
ability to distinguish between includes in current and other files
I think only way to do it reliably is to use complete parser and semantic analyzer. D Completion Daemon (https://github.com/Hackerpilot/DCD/tree/master/editors/vim ) has vim plugin and is not very resource-hungry.
Vim's include mechanism and 'includeexpr' are heavily influenced by the C programming language and only work for single files. You cannot return a list of filenames, so it won't be possible to support D's complex include mechanism with Vim. Use an IDE that is fully tailored to support the programming language, not a general-purpose text editor.
I'm using io/ioutil to read a small text file:
fileBytes, err := ioutil.ReadFile("/absolute/path/to/file.txt")
And that works fine, but this isn't exactly portable. In my case, the files I want to open are in my GOPATH, for example:
/Users/matt/Dev/go/src/github.com/mholt/mypackage/data/file.txt
Since the data folder rides right alongside the source code, I'd love to just specify the relative path:
data/file.txt
But then I get this error:
panic: open data/file.txt: no such file or directory
How can I open files using their relative path, especially if they live alongside my Go code?
(Note that my question is specifically about opening files relative to the GOPATH. Opening files using any relative path in Go is as easy as giving the relative path instead of an absolute path; files are opened relative to the compiled binary's working directory. In my case, I want to open files relative to where the binary was compiled. In hindsight, this is a bad design decision.)
Hmm... the path/filepath package has Abs() which does what I need (so far) though it's a bit inconvenient:
absPath, _ := filepath.Abs("../mypackage/data/file.txt")
Then I use absPath to load the file and it works fine.
Note that, in my case, the data files are in a package separate from the main package from which I'm running the program. If it was all in the same package, I'd remove the leading ../mypackage/. Since this path is obviously relative, different programs will have different structures and need this adjusted accordingly.
If there's a better way to use external resources with a Go program and keep it portable, feel free to contribute another answer.
this seems to work pretty well:
import "os"
import "io/ioutil"
pwd, _ := os.Getwd()
txt, _ := ioutil.ReadFile(pwd+"/path/to/file.txt")
I wrote gobundle to solve exactly this problem. It generates Go source code from data files, which you then compile into your binary. You can then access the file data through a VFS-like layer. It's completely portable, supports adding entire file trees, compression, etc.
The downside is that you need an intermediate step to build the Go files from the source data. I usually use make for this.
Here's how you'd iterate over all files in a bundle, reading the bytes:
for _, name := range bundle.Files() {
r, _ := bundle.Open(name)
b, _ := ioutil.ReadAll(r)
fmt.Printf("file %s has length %d\n", name, len(b))
}
You can see a real example of its use in my GeoIP package. The Makefile generates the code, and geoip.go uses the VFS.
Starting from Go 1.16, you can use the embed package. This allows you to embed the files in the running go program.
Given the file structure:
-- main.go
-- data
\- file.txt
You can reference the file using a go directive
package main
import (
"embed"
"fmt"
)
//go:embed data/file.txt
var content embed.FS
func main() {
text, _ := content.ReadFile("data/file.txt")
fmt.Println(string(text))
}
This program will run successfully regardless of where the program is executed. This is useful in case the file could be called from multiple different locations, for instance, from a test directory.
I think Alec Thomas has provided The Answer, but in my experience it isn't foolproof. One problem I had with compiling resources into the binary is that compiling may require a lot of memory depending on the size of your assets. If they're small, then it's probably nothing to worry about. In my particular scenario, a 1MB font file was causing compilation to require somewhere around 1GB of memory to compile. It was a problem because I wanted it to be go gettable on a Raspberry Pi. This was with Go 1.0; things may have improved in Go 1.1.
So in that particular case, I opt to just use the go/build package to find the source directory of the program based on the import path. Of course, this requires that your targets have a GOPATH set up and that the source is available. So it isn't an ideal solution in all cases.