I am writing a Go command-line tool that generates some files based on templates.
The templates are located in the Git repository alongside the code for the command-line tool itself.
I wanted to allow the following:
The binary, wherever it is called from, should always find the templates directory.
The templates directory can be overriden by the user if need be.
Since this is a Go application, I went with something like:
templateRoot := filepath.Join(
os.Getenv("GOPATH"),
"src/github.com/myuser/myproject/templates",
)
But being rather new to Go, I wonder if this approach is reliable enough: is it guaranteed that my application template will always be accessible at that path ?
What if someone vendorize my application into their own project ? Does that even make sense for a command-line tool ?
Because of 2., I obviously can't/won't use go-bindata because I want to allow for the templates to be overriden if need be.
In summary: what is a good strategy to reliably refer to non-go, static files in a Go command-line tool ?
GOPATH is used for building the application. While you could look for GOPATH and check relative locations to each GOPATH entry at runtime, you can't be sure it will exist (unless of course you make it a prerequisite for running your application).
go get itself is a convenience for developers to fetch and build a go package. It relies on having a GOPATH (though there's a default now in go1.8), and GOBIN in your PATH. Many programs require extra steps not covered by the simple go tool, and have scripts or Makefiles to do the build. If you're targeting users that aren't developers, you need to provide a way to install into standard system paths anyway.
Do what any regular program would do, and use some well-known path to locate the template files. You can certainly add some logic in your program to check for a hierarchy of locations: relative to $GOPATH, relative to the binary, working directory, $HOME, etc; just provide a list of locations to the user that your program will look for templates.
Related
coming from a java and grails background, (and having written millions of lines of C 30 years ago), I cant see how go can be usable with a fixed gopath on windows.
installing go creates this structure
c:\users\me\go\scr
\pkg
\bin
As you will want to have many go projects it seems they have to be mixed together in the same src/kpg/bin dirs, polluting each other. e.g.
/src/project1/hello.go
/project2/hello.go
/pkg/xx
/bin/hello.exe
Which hello.go will hello.exe run?
Unless I am missing something fundamental, this seems crazy - all completely separate projects are expected to share the same package and bin dirs. It means you dont know which versions of which packages, and which exe files belong to which project, and there is presumably plenty of scope for conflicts. I would have expected a /src, /pkg and /bin for each separate go app (like java or grails or even C), and go path is completely redundant, it can be relative to the current project root (like in grails).
To make matters works, for work, we have to use a different directry, e.g.
d:\work\project3
\project4
\package5
\go_utility6
\go_utility7
So now we have a total of 6 separate directories where go progams live. It is not feasible to change the path every time you switch to working on a different project. I guess the other option is to add the 6 paths to the GOPATH. Presumably, all 7 go projects write to the same pkg and bin dir, which is going to be a disaster.
Is there a tenable solution to this situation (on windows at least)?
If we need to add a PATH to GOPATH for every project, what should the file structure be under each project?
E.g. uner xxx\go_utility6, which is a stand alone command line go app, what should the structure be? does there need to be a src dir in there somewhere? does this dir need gopath to point to it? does it need its own pkg, or should it use the c:\users\me\pkg dir?
UPDATE: When I posted this Go did not have modules support and we built and used a tool called vg. These days the recommended way to go is to use go modules.
I use vg for that, it takes care of keeping separate GOPATH paths per project and it switches automatically when you cd a project.
Your example "which hello.exe" should be used honestly makes not much sense. Two tools with the same name?
Even if both are, let's say, an api, your devops will be happier with more meaningful names.
The bin folder is used for 3rd party tools you install, you so not have to install your project binaries. Except they are tools, but then the name should be meaningful again.
You can get more information about the project structure here: https://golang.org/doc/code.html
Since go 1.8 supports a vendor folder below project folders, it is possible to break the original structure. (imho vendors were not maintainable before 1.8, yes that was crazy)
You might want to use a tool like direnv, which would support your desire to change GOPATH per project.
https://github.com/direnv/direnv
It also has some built in function for adding the current path to the GOPATH.
https://github.com/direnv/direnv/blob/master/stdlib.sh#L355:1
For example GoLang also supports handling multiple GOPATHs and per project GOPATHs. So direnv should also work properly.
In my company we have one go folder right next to our other projects.
Under go/src are our projects. No problem so far, since vendors are in the projects' vendor folders and committed.
The so far best dependency manager I would recommend for go is:
https://github.com/golang/dep
I hope that input helps.
With Go 1.11 Go Modules were introduced. You can use Go Modules to have Go projects outside the GOPATH directory.
Here is an example of how to configure a project using GoModules.
I'm playing with golang and made a tool for password generation. Initially it was intended to be used as a command line tool. I later separated the generation logic into a separate package (still same github repository) and left the main function in the root of the project. Now I also want to add a simple web frontend (nothing fancy), but I don't know how to structure the packages.
Am I supposed to put both the command line entry point as well as the web UI into their own packages in the same project (which leaves the root empty). Or maybe I should move the actual generation library to the root and the UIs in separate packages. I guess the other option is to have the UIs in separate projects on github, but they are only going to be used for this library, so it does not seem like a good idea.
I remember seeing in some projects packages named cmd for example, but never have I encountered one, with multiple front ends. Is there a go(-gettable-)way for doing this?
I agree that there's not much point in making separate projects/repositories if they're only going to be used for this library. I would just have a cmd directory with a subdirectory for each executable you're building.
Something like this:
github.com/user/project
generation
cmd
cmdline
main.go
web
main.go
The main.go files can use the functionality that you've broken out into your "generation" package.
The name of the executables produced by go build will be the name of the parent directory, so cmdline and web in this example (you would want to choose better names).
Note: you don't actually have a package cmdline or web. The files in those directories would all be in [their own separate] package main.
I'm on OS-X (Mavericks, if that matters), and I'm making a bash script that will use resources from a folder called "templates". I'm trying to figure out where I should put it (the templates folder). I'd like to make it so the user doesn't need to modify their path when they install it, so I'd rather not do it the way the terminal mysql command does it (it lives in a folder in /usr/local/mysql/bin). I really want to be able to put them into usr/bin, but I don't know if it's "polite" to put folders in there (I don't see any in there).
Right now I'm leaning towards putting the scripts in usr/bin and having the templates in usr/lib. Is that how this type of thing is normally done, or is there another way? I'd like to follow a convention, assuming there is one. I'd also like it to apply to as many Unix platforms as possible (I'd like to put in a directory where bash scripts live that's consistent across as many Unix platforms as possible). Thanks.
If you follow the Filesystem Hierarchy Standard (FHS), your executable goes in /usr/local/bin, while read-only template files go in /usr/local/share/YOURAPP/. To quote the FHS:
/usr/local/share
The requirements for the contents of this directory are the same as /usr/share. […]
and:
The /usr/share hierarchy is for all read-only architecture independent data files.
(Emphasis added)
If the system admin is meant to customize the template files to take effect system-wide, then they would simply go in /etc/YOURAPP/templates (or something like that).
If the template files are customized on a per-user basis, then the modified copies of the templates (copied from /usr/local/share/YOURAPP/templates) need to be saved in the user's directory, under $HOME/.config/YOURAPP/templates or something like that (thanks to technosaurus for the correction).
You mentioned that you want to install the templates in a directory alongside your executable. That is not the standard approach on UNIX, at least going by the FHS. If you really want to go this route, there is a sort of convention of installing your app to /opt/YOURAPP/, using whatever organization you want inside that folder.
In all cases, it is not good practice to install executables directly to /usr/bin, as that directory is considered to be under the exclusive control of the OS/distribution. If you want to install there, the accepted way to do that is to create a package for the package manager of every supported OS/distribution.
I have a go app which relies heavily on static resources like images and jars. I want to install that go executable in different platforms like linux, mac and windows.
I first thought of bundling the resources using https://github.com/jteeuwen/go-bindata, but since the files(~100) have size ~ 20MB or so, it takes a really long time to build the executable. I thought having a single executable is an easy way for people to download the executable and run it. But seems like that is not an effective way.
I then thought of writing a installation package for each of the platform like creating a .rpm or .deb packages? So these packages contain all the resources and puts it into some platform specific pre defined locations and the go executable can reference them. But the only thing is that I have to handle that in the go code. I have to see if it is windows then load the files from say c:\go-installs or if it is linux then load the files from say /usr/local/share/go-installs. I want the go code to be as platform agnostic as it can be.
Or is there some other strategy for this?
Thanks
Possibly does not qualify as real answer but still…
As to your point №2, one way to handle this is to exploit Go's way to do conditional compilation: you might create a set of files like res_linux.go, res_windows.go etc and put a set of the same variables in each, pointing to different locations, like
var InstallsPath = `C:\go-installs`
in res_windows.go and
var InstallsPath = `/usr/share/myapp`
in res_linux.go and so on. Then in the rest of the program just reference the res.InstallsPath variable and use the path/filepath package to construct full pathnames of actual resources.
Of course, another way to go is to do a runtime switch on runtime.GOOS variable—possibly in an init() function in one of the source files.
Pack everything in a zip archive and read your resource files from it using archive/zip. This way you'll have to distribute just two files—almost "xcopy deployment".
Note that while on Windows you could just have your executable extract the directory from the pathname of itself (os.Args[0]) and assume the resource file is located in the same directory, on POSIX platforms (GNU/Linux and *BSD etc) the resource file should still be located under /usr/share/myapp or a similar place dictated by FHS (or particular distro's rules), so some logic to locate that file will still be required.
All in all, if this is supposed to be a piece of FOSS, I'd go with the first variant to let the downstream packagers tweak the pathnames. If this is a proprietary (or just niche) software the second idea appears to be rather OK as you'll play the role of downstream packagers yourself.
My application uses json configuration files and other resources. Where should I place them in my project hierarchy?
I could not find the answer in http://golang.org/doc/code.html (How to Write Go Code)
Upd:
The question is not about automatic distribution of resources with application but much simpler: Where should I keep my resources in project hierarchy? Is there some standard place anyone expects them to be?
There is no single correct answer, nor are there any strong conventions assumed or enforced by any Go tooling at this time.
Typically I start by assuming that the files I need are located in the same directory from where the program will be run. For instance, suppose I need conf.json for myprog.go; then both of those files live together in the same directory and it works to just run something like
go build -o myprog && ./myprog
When I deploy the code, the myprog binary and conf.json live together on the server. The run/supervisor script needs to cd to that directory and then run the program.
This also works when you have a lot of resources; for instance, if you have a webserver with JS, CSS, and images, you just assume they're relative to cwd in the code and deploy the resource directories along with the server binary.
Another alternative to assuming a cwd is to have a -conf flag which the user can use to specify a configuration file. I typically use this for distributing command-line tools and open-source server applications that require a single configuration file. You could even use an -assets flag or something to point to a whole tree of resource files, if you wanted.
Finally, one more approach is to not have any resource files. go-bindata is a useful tool that I've used for this purpose -- it just encodes some data as bytes into a Go source file. That way it's all baked into your binary. I think this method is most useful when the resource data will rarely or never change, and is pretty small. (Otherwise you're going to be shipping around huge binaries.) One (kind of silly) example of when I've used go-bindata in the past was for baking a favicon into a really simple server which didn't otherwise require any extra files besides the server binary.
For static resource it might be the most convenient solution to include them in the binary similar to resources in Java. Newer Go version, at least 1.18 are providing the //go:embed directive to include content:
import (
_ "embed"
)
//go:embed myfile.txt
var myfile string
You can now use myfile in your code. E.g. IntelliJ provides also support for this.
There are also other options to include the content, e.g. as binary or dynamically, see the link.