Using "go get" to download binaries without adding them to go.mod - go

I'm using Go modules in my project and in my build system (e.g. Travis CI) I'm downloading a command-line utility (written in Go) with go get to assist with my build process, e.g.:
go get github.com/mitchellh/gox
However, this go get causes the file to be added to my go.mod file. This is contaminating the build environment, causing it to be "dirty" (since there are changes to some files tracked in git, in this case go.mod and go.sum), and I use git describe --always --dirty --tag to describe my build, which shows up as "dirty".
Is there a way to "go get" a binary just to download it, without adding it to the go.mod/go.sum?
I've tried setting GOPATH to somewhere else, even then, go get updates the go.mod/go.sum to add this as an // indirect dependency.
dir="$(mktemp -d)"; \
env GOPATH="$dir" go get github.com/mitchellh/gox && \
mv "$dir/bin/gox" "$(go env GOPATH)"/bin/gox

Go 1.16 onwards
Go 1.16 (released February 2021) includes a change that makes it possible to install a binary without affecting go.mod.
Issue 40276 tracks the proposal:
cmd/go: 'go install' should install executables in module mode outside a module
This was implemented in CL 254365. As part of this change, you can run e.g.:
go install golang.org/x/tools/cmd/goimports#latest
to install a binary without affecting go.mod.
To install a specific version, replace #latest with e.g. #v0.1.5.

Hopefully in Go 1.14 there will be a new flag for go get that does exactly what you are asking. This is tracked in issue #30515 "cmd/go: offer a consistent global install command".
Prior to that, you have a few different options.
Go 1.12 and 1.13: change directory
If you are using Go 1.12 or later, the simplest solution is probably to move outside your current module to a directory without a go.mod prior to doing the go get, such as:
$ cd /tmp
$ go get github.com/foo/bar#v1.2.3
$ cd - # return to prior directory
Go 1.11, 1.12, 1.13+: gobin
gobin is a module-aware command to install or run binaries that provides additional flexibility, including the ability to install without altering your current module's go.mod. See the gobin README and FAQ for more details.
Go 1.11: temporary module
If you are using Go 1.11 with modules, the first step is probably to upgrade to Go 1.12 or 1.13 given there are many improvements in modules. If you are required to use Go 1.11 and want to use the #version syntax without updating your current module's go.mod, then one approach is to create a temporary module:
cd $(mktemp -d) && go mod init tempmod && go get github.com/foo/bar#v1.2.3
This is because in Go 1.11, you can't use the #version syntax unless you are in a module, which was relaxed in Go 1.12. This approach has been automated by a simple shell script by #rogpeppe.
Additional Details
In general, the go command in module-module always determines what module it is "in", which is based on the current working directory when you invoke the go command. (You could make an analogy to how make without any args will look for a makefile in the current working directory, or how historically go build without any args will build the current working directory, etc.).
With modules, go get looks for a go.mod file in the current working directory or any of its parents, and go get will use the constraints listed in any go.mod as part of solving for versions, as well as update the go.mod if needed based on doing the go get. That is why your go.mod file is updated if you run go get from within an existing module.
On the other hand, starting with Go 1.12, if you are in a directory that is not part of any module (that is, the directory does not have a go.mod, nor do any of its parents), then there is no go.mod to update, but the go command is still able to operate in module mode and use the #version syntax.
From the Go 1.12 release notes:
When GO111MODULE is set to on, the go command now supports module-aware operations outside of a module directory, provided that those operations do not need to resolve import paths relative to the current directory or explicitly edit the go.mod file. Commands such as go get, go list, and go mod download behave as if in a module with initially-empty requirements. In this mode, go env GOMOD reports the system's null device (/dev/null or NUL).

Per go help build:
The -mod build flag provides additional control over updating and use of go.mod.
If invoked with -mod=readonly, the go command is disallowed from the implicit automatic updating of go.mod

Related

How to prevent go get from updating go.mod file

TL;DR: Is there any way I can forcefully prevent go get from altering the go.mod file?
When I do a go get of certain packages, e.g.:
$ go get github.com/AsynkronIT/protoactor-go/protobuf/protoc-gen-gograin
It will printout that it has updated dependencies (which are defined in my go.mod file):
go get: upgraded github.com/AsynkronIT/protoactor-go v0.0.0-20200815184336-b225d28383f2 => v0.0.0-20210405044454-10bc19881ad6
# (...) Note, this happens for other packages, not just `AsynkronIT/protoactor-go`.
This causes the go.mod file to change during a CI build, and affects subsequent build stages where, while building something, it’ll try to use an updated version of the dependency, which might introduce breaking changes, instead of the version defined on the go.mod file initially.
I’ve tried using -mod=readonly or making sure the -u flag is not used but it will still update the go.mod file, e.g.:
$ GOFLAGS=-mod=readonly go get github.com/AsynkronIT/protoactor-go/protobuf/protoc-gen-gograin
go get: upgraded github.com/AsynkronIT/protoactor-go v0.0.0-20200815184336-b225d28383f2 => v0.0.0-20210405044454-10bc19881ad6
# (...)
I've also tried finding similar issues, like this one, or this other one, but haven't yet find an alternative to prevent go get commands from altering the go.mod.
The current workaround I'm using to stop this behavior is to do a git checkout -- go.mod right after certain go get … steps to reset any changes done by go get, and hence, avoiding breaking changes with certain dependencies newer versions.
I'm using go version 1.16.3.
For Go 1.16 and after, you can use go install to install binaries without affecting go.mod
go install github.com/AsynkronIT/protoactor-go/protobuf/protoc-gen-gograin

How can I install a specific version of golint for use globally?

I'm trying to create a docker image for use in a build pipeline that has various tools pre-installed for building and testing go projects. One tool we need is golint but I'm struggling to see how to install a specific version of it. The reason I want to lock down the version is to avoid accidental / unwanted / unintended breakages at a later date.
For a start, looking here the versions are not exactly in an easy-to-type format.
Also when I try to use the following command
go get -u golang.org/x/lint/golint#v0.0.0-20181217174547-8f45f776aaf1
I get an error
go: cannot use path#version syntax in GOPATH mode
Googling has so far yielded very few relevant results...
Is what I'm trying to do possible? Many thanks!
You need to be in go module mode to get code of a specific version, since in addition to downloading the code, the version is recorded in the go module file.
The easiest way to do this would be to create an empty directory, run go mod init, which will create a go.mod file.
Then, you can run go get golang.org/x/lint/golint#v0.0.0-20181217174547-8f45f776aaf1, which will add golint at that version to your go.mod file. You can then run go install golang.org/x/lint/golint from within that directory, which will install golint at the version specified into your $GOBIN directory (which defaults to $GOPATH/bin).

Why listing go modules dependencies fail when the module is in my GOPATH?

I am using this project https://github.com/sttts/beerdb. When I clone in into /tmp/, if I run:
go list -m all
I get a list of all the dependencies. If the same project is in my
$GOPATH/src/github.com/stts/beerdb, and then I run the same command I get:
go list -m: not using modules
If in both cases I am in the path where go.mod and go.sum are present, why I get a different behaviour? My go version is go1.12
Go 1.12 only has preliminary support for modules (introduces in Go 1.11). GOPATH is the default mode, but when GO111MODULE env var is set to on, the go command now supports module-aware operations outside of a module directory. For details, see Go 1.12 Release notes: Modules.
Recommended to switch to Go 1.13 which added full module support. In Go 1.13 module-aware mode is active by default whenever a go.mod file is found in, or in a parent of, the current directory.

`go install` adds record in `go.mod`

I'm a bit confused about how Go modules work on installing binaries using go install.
I tried to install (https://github.com/joho/godotenv) binary by executing go install github.com/joho/godotenv/cmd/godotenv and I found out that it adds a record in the go.mod.
I'm lost as we don't use this package in the code and after running go mod tidy it gets deleted (as it is not it the code).
Can someone explain is it expected behaviour of go modules?
Secondly, how I could avoid adding it to the go.mod as we only need to install and execute the binary?
Thanks.
Go version: go version go1.13.4 darwin/amd64
Command go: The go.mod file:
The go command automatically updates go.mod each time it uses the module graph, to make sure go.mod always accurately reflects reality and is properly formatted.
The go tool will update go.mod automatically when it detects dependencies are inaccurate when performing a build.
When you install github.com/joho/godotenv/cmd/godotenv from your module, this install requires at least the package in question being built / installed (and also its dependencies, transitively).
You may safely run go mod tidy to undo the recording of this "one-time" dependency.
In general if you want to disallow the go tool to update the go.mod file, you may use the -mod=readonly flag, but that would fail go install ("can't load package: package xxx: import lookup disabled by -mod=readonly"). You can read more about this here: Go Wiki: Go modules: Can I control when go.mod gets updated and when the go tools use the network to satisfy dependencies?
Alternatively, if you want to avoid this, build / install your tools outside of your module. You may use a "dummy" module for this.

go mod vendor without update to latest

I’m trying to figure out if it’s possible to run go mod vendor without the go tool updating my go.mod file.
I specifically go get package/subpackage#commit and commit my go.mod with the correct version.
Then I run go mod vendor and it automatically bumps the version of the package that I just specifically set.
I’ve looked at this page to no avail: https://github.com/golang/go/wiki/Modules#how-do-i-use-vendoring-with-modules-is-vendoring-going-away
I need to use vendor because I run a script that edits some of the vendored deps., I’m looking at the following build flow:
GO111MODULE=on go get package/subpackge#commit
GO111MODULE=on go mod vendor
./Script/patch_vendors.sh --write
GO111MODULE=off go build
My other option is modifying the copied source wherever go mod vendor donwloads it to, but
not sure how to approach that.
Thanks in advance
Per https://tip.golang.org/cmd/go/#hdr-Maintaining_module_requirements:
The go command itself automatically updates the go.mod file to maintain a standard formatting and the accuracy of require statements.
Any go command that finds an unfamiliar import will look up the module containing that import and add the latest version of that module to go.mod automatically. […]
Any go command can determine that a module requirement is missing and must be added […].
The go mod vendor command copies in all of the transitive imports of your packages and their tests, so it will automatically update the go.mod file to ensure that all of the imported packages are present.
So the problem here is likely that the commit you've selected for package/subpackage fails to provide some package that appears in the transitive imports of your program. If that is correct, you should find that go list all, go test all, and go mod tidy all make that same edit to your module's requirements.

Resources