I am beginner in go and facing difficulty in understanding go/pkg folder.As suggested by documentation it contains pkg/mod and pkg/windowsamd_64. pkg/windowsamd_64 for storing compiled files. What happens if I have a file importing some external github modules and do go build on that.
Will it go first to pkg/mod (but modules are compiled in
pkg/windowsamd_64) to search for external modules
Will it go first to pkg/windowsamd_64 (then what will be use of
pkg/mod) to search for modules
Will it go to {gopath}/src and do something from there
pkg/mod is just a folder ,why do we call it cache as it will keep on
filling or better when does it populate?
The go command has two different modes of locating packages: module mode (introduced in Go 1.11) and GOPATH mode (much older). Module mode is the default as of Go 1.16, and if you are new to Go you will probably want to work exclusively in that mode. (There isn't much point to working in GOPATH mode unless you have a large legacy codebase that requires it.)
pkg/mod stores cached source code for use in module mode. The source code for a given version of a module is downloaded automatically when you build a package from that module (for example, as a dependency of some other package).
GOPATH/src stores source code for use in GOPATH mode. You can also choose to work in that directory in module mode, but that's a completely optional/aesthetic choice and shouldn't change the behavior of anything in module mode.
pkg/windows_amd64 stores installed packages in GOPATH mode. However, installed packages aren't very useful anyway because Go has a separate build cache (even in GOPATH mode). So you can mostly ignore pkg/windows_amd64 completely.
Related
I'm trying to understand how the source code for third-party dependencies is or is not compiled into my Go binary. I'm building in a Docker container, so I can see precisely what's fetched for my build without interference from other builds.
After my go build completes I see source code files for several dependencies under go/pkg/mod/$module#$version directories. The Module cache documentation tells me that these directories contain "extracted contents of a module .zip file. This serves as a module root directory for a downloaded module." My best guess is that the presence of extracted source code for these dependencies indicates that "yes, these dependencies are definitely compiled into your binary."
I also see many more dependencies pulled into go/pkg/mod/cache/download/$module directories. The Module cache documentation tells me that this directory contains "files downloaded from module proxies and files derived from version control systems," which I don't fully understand. As far as I can see, these files do not include any extracted source code, though there are several .zip files that I assume contain the source. For the most part these files seem to be .mod files that just contain text representing some sort of dependency graph.
My question is: if a third-party dependency has module files under go/pkg/mod/cache/download but no source code under go/pkg/mod/$module#$version, does that mean that dependency's code was NOT compiled into my Go binary?
I don't understand why the Go build pulls in all these module files but only has extracted source code for some of the third-party modules. Perhaps Go preemptively parses and pulls module information for the full transitive set of modules referenced from the modules my first-party code imports, but perhaps many of those modules don't end up being needed for my binary's compile + build process and therefore don't get extracted. If that's not true and the answer to my question is no, then I don't understand how or why my binary can link in those dependencies without go build fetching their source code.
As mentioned in "Compile and install packages and dependencies"
Compiled packages are cached automatically.
GOPATH and Module includes:
When using modules, GOPATH is no longer used for resolving imports.
However, it is still used to store downloaded source code (in GOPATH/pkg/mod) and compiled commands (in GOPATH/bin).
So if you see sources in pkg/mod which are not in pkg/mod/cache, try a go mod tidy
add missing and remove unused modules
From there, you should have the same modules between sources (pkg/mod) and compiled modules (pkg/mod/cache)
Based on the OP's comment
I need to know exactly what's included in the binary for compliance reasons.
I would recommend a completely different approach: dumping the list of symbols contained in the binary and then correlating it with whatever information is needed.
The command
go tool nm -type /path/to/the/executable/image/file
would dump the symbols — names of the functions — whose code was taken from both the standard library packages, 3rd-party and/or vendored packages and internal packages, compiled and linked into the binary, and print to its standard output stream a sequence of lines
address type name
which you can then process programmatically.
Another approach you might employ is to use go list with various flags to query the program's source code about the packages and/or modules which will be used when building: whatever that command outputs describing the full dependency graph of the source code is whatever go build will use when building — provided the source code is not changed between these calls.
Yet another possibility is to build the program using go build -x, save the debug trace it produces on its standard error stream and parsing it for exact module names the command reported as used during building.
My project consists of multiple go modules which I need to work on simultaneously. This is made easy by using the replace directive in the go.mod file.
Further, in order to prevent this change from getting accidentally checked in, the go tools (starting with version 1.14) offer a -modfile switch which lets me put the replace directive in a go.local.mod file instead. This is super convenient.
Unfortunately, I am unable to get Goland to pick up this go.local.mod instead of go.mod.
I have tried setting the GOMOD environment variable under Preferences -> Go -> Go Modules to point to my go.local.mod file. This did not do what I expected it to (which is to use the go.local.mod file instead of go.mod)
Not surprisingly, this did not work since GOMOD is supposed to be readonly (as pointed out to me in the comments).
For now I can go back to putting the replace directive in the go.mod file and using commit hooks to prevent accidental check-in (and Goland has some tooling to perform this check as well) However, it would be super convenient if Goland can recognize this new -modfile switch that go has added. Figured I would ask to make sure I'm not missing something obvious here.
Thanks!
At the moment, 2020.1.3 stable release, this is not supported. See the related issue for future updates on this.
coming from a java and grails background, (and having written millions of lines of C 30 years ago), I cant see how go can be usable with a fixed gopath on windows.
installing go creates this structure
c:\users\me\go\scr
\pkg
\bin
As you will want to have many go projects it seems they have to be mixed together in the same src/kpg/bin dirs, polluting each other. e.g.
/src/project1/hello.go
/project2/hello.go
/pkg/xx
/bin/hello.exe
Which hello.go will hello.exe run?
Unless I am missing something fundamental, this seems crazy - all completely separate projects are expected to share the same package and bin dirs. It means you dont know which versions of which packages, and which exe files belong to which project, and there is presumably plenty of scope for conflicts. I would have expected a /src, /pkg and /bin for each separate go app (like java or grails or even C), and go path is completely redundant, it can be relative to the current project root (like in grails).
To make matters works, for work, we have to use a different directry, e.g.
d:\work\project3
\project4
\package5
\go_utility6
\go_utility7
So now we have a total of 6 separate directories where go progams live. It is not feasible to change the path every time you switch to working on a different project. I guess the other option is to add the 6 paths to the GOPATH. Presumably, all 7 go projects write to the same pkg and bin dir, which is going to be a disaster.
Is there a tenable solution to this situation (on windows at least)?
If we need to add a PATH to GOPATH for every project, what should the file structure be under each project?
E.g. uner xxx\go_utility6, which is a stand alone command line go app, what should the structure be? does there need to be a src dir in there somewhere? does this dir need gopath to point to it? does it need its own pkg, or should it use the c:\users\me\pkg dir?
UPDATE: When I posted this Go did not have modules support and we built and used a tool called vg. These days the recommended way to go is to use go modules.
I use vg for that, it takes care of keeping separate GOPATH paths per project and it switches automatically when you cd a project.
Your example "which hello.exe" should be used honestly makes not much sense. Two tools with the same name?
Even if both are, let's say, an api, your devops will be happier with more meaningful names.
The bin folder is used for 3rd party tools you install, you so not have to install your project binaries. Except they are tools, but then the name should be meaningful again.
You can get more information about the project structure here: https://golang.org/doc/code.html
Since go 1.8 supports a vendor folder below project folders, it is possible to break the original structure. (imho vendors were not maintainable before 1.8, yes that was crazy)
You might want to use a tool like direnv, which would support your desire to change GOPATH per project.
https://github.com/direnv/direnv
It also has some built in function for adding the current path to the GOPATH.
https://github.com/direnv/direnv/blob/master/stdlib.sh#L355:1
For example GoLang also supports handling multiple GOPATHs and per project GOPATHs. So direnv should also work properly.
In my company we have one go folder right next to our other projects.
Under go/src are our projects. No problem so far, since vendors are in the projects' vendor folders and committed.
The so far best dependency manager I would recommend for go is:
https://github.com/golang/dep
I hope that input helps.
With Go 1.11 Go Modules were introduced. You can use Go Modules to have Go projects outside the GOPATH directory.
Here is an example of how to configure a project using GoModules.
I am writing a Go command-line tool that generates some files based on templates.
The templates are located in the Git repository alongside the code for the command-line tool itself.
I wanted to allow the following:
The binary, wherever it is called from, should always find the templates directory.
The templates directory can be overriden by the user if need be.
Since this is a Go application, I went with something like:
templateRoot := filepath.Join(
os.Getenv("GOPATH"),
"src/github.com/myuser/myproject/templates",
)
But being rather new to Go, I wonder if this approach is reliable enough: is it guaranteed that my application template will always be accessible at that path ?
What if someone vendorize my application into their own project ? Does that even make sense for a command-line tool ?
Because of 2., I obviously can't/won't use go-bindata because I want to allow for the templates to be overriden if need be.
In summary: what is a good strategy to reliably refer to non-go, static files in a Go command-line tool ?
GOPATH is used for building the application. While you could look for GOPATH and check relative locations to each GOPATH entry at runtime, you can't be sure it will exist (unless of course you make it a prerequisite for running your application).
go get itself is a convenience for developers to fetch and build a go package. It relies on having a GOPATH (though there's a default now in go1.8), and GOBIN in your PATH. Many programs require extra steps not covered by the simple go tool, and have scripts or Makefiles to do the build. If you're targeting users that aren't developers, you need to provide a way to install into standard system paths anyway.
Do what any regular program would do, and use some well-known path to locate the template files. You can certainly add some logic in your program to check for a hierarchy of locations: relative to $GOPATH, relative to the binary, working directory, $HOME, etc; just provide a list of locations to the user that your program will look for templates.
I can specify dependencies to be downloaded by go get after checking out my project by importing them. I can even force the download of packages that are not used in the code by importing them for side effects:
import _ "github.com/jteeuwen/go-bindata"
Furthermore, on the shell I can apparently install a program with go get by using an ellipsis after the path:
$ go get github.com/jteeuwen/go-bindata/...
However, it seems I cannot combine both techniques:
import _ "github.com/jteeuwen/go-bindata/..."
$ go get
main.go:9:8: open c:\gopath\src\github.com\jteeuwen\go-bindata\...: Access denied
I would like to tell go get that for building (actually go generateing) this project, go-bindata has to be installed. Is it possible to specify install-dependencies?
To answer your question: No.
But you could vendor go-bindata into your project which would make it available after after go geting your project.
But maybe there is a slight confusion about when and why to run go generate: The intended use for go generate (as I understand it) is for package or command authors to generate code during the development phase. Code which is checked in and processed "normally" by go {build,install,get}. So you run go generate, check in the generated stuff and users of your package go get it and do not run go generate. They don't need to, the code comes in the proper checked in version during geting.
For more complicated builds which a end-user has to perform: Use Makefiles or similar tools as such stuff is out of the scope of go get.
In one of my projects I use Godep. According to its page:
This tool assumes you are working in a standard Go workspace, as
described in http://golang.org/doc/code.html. We expect godep to build
on Go 1.4* or newer, but you can use it on any project that works with
Go 1 or newer.
You'll have your dependencies in a JSON file, just like Node, Bower, etc... It's vert simple to use.
In your case, assuming you already have go geted the package, run:
godep save
This will generate the JSON file with all your other dependencies and save to a folder in your project. Thanks to it I was capable of cross compiling my project.