Go update all modules - go

Using this module as an example (using a specific commit so others will see
what I see):
git clone git://github.com/walles/moar
Set-Location moar
git checkout d24acdbf
I would like a way to tell Go to "update everything". Assume that the module
will work with the newest version of everything. Below are five ways I found to
do this, assume each is run on a clean clone. This results in a go.mod of 19
lines:
go get -u
This results in a go.mod of 14 lines:
go get -u
go mod tidy
This results in a go.mod of 13 lines:
go mod tidy
If I just manually delete everything in require and run go mod tidy, I get
12 lines. If I just manually delete everything in require and run go get -u, I get 11 lines. My question is, why are these methods producing different
results, and what is the "right way" to do what I am trying to do?

tl;dr;
this is what you want:
go get -u
go mod tidy
and to recursively update packages in any subdirectories:
go get -u ./...
The inconsistencies you are seeing is due to the inherent organic nature of software.
Using your example, commit d24acdbf of git://github.com/walles/moar most likely was checked in by the maintainer without running go mod tidy (explaining the longer 19 lines). If the maintainer had, then you would see the 13 line version you see at the end.
go get -u on it's own is more aggressive in pulling in dependencies. Also, the mere fact of updating dependencies to their latest (compatible) version, may in & of itself pull in new direct/indirect dependencies. These dependencies may grow even further if you tried this tomorrow (the latest version of some sub-dependency adds new functionality, so it needs new dependencies). So there may be a valid reason the repo maintainer fixes at a particular (non-latest) version.
go mod tidy cleans up this aggressive dependency analysis.
P.S. It's a common misconception that dependencies will shrink after go mod tidy: tracking go.sum, in some cases this file will grow after a tidy (though, not in this case)

Run go get -u && go mod tidy 1
More details:
go get -u (same as go get -u .) updates the package in the current directory, hence the module that provides that package, and its dependencies to the newer minor or patch releases when available. In typical projects, running this in the module root is enough, as it likely imports everything else.
go get -u ./... will expand to all packages rooted in the current directory, which effectively also updates everything (all modules that provide those packages).
Following from the above, go get -u ./foo/... will update everything that is rooted in ./foo
go get -u all updates everything including test dependencies; from Package List and Patterns
When using modules, all expands to all packages in the main module and their dependencies, including dependencies needed by tests of any of those.
go get will also add to the go.mod file the require directives for dependencies that were just updated.
go mod tidy makes sure go.mod matches the source code in the module. In your project it results in 12 lines because those are the bare minimum to match the source code.
go mod tidy will prune go.sum and go.mod by removing the unnecessary checksums and transitive dependencies (e.g. // indirect), that were added to by go get -u due to newer semver available. It may also add missing entries to go.sum.
Note that starting from Go 1.17, newly-added indirect dependencies in go.mod are arranged in a separate require block.
1: updates dependencies' newest minor/patch versions, go.mod, go.sum

There are some hidden dragons, here is what I recommend:
go get -u ./... walks all packages in your project. This is the command you want to use.
go get -t -u ./... walks all packages in your project and also downloads tests files of these dependencies. Probably you don’t need that.
go get -u updates in the current directory only. Useful for small single-package projects, just use the first version.
go get -u specific.com/package updates just one (or more separated by space) packages (and dependencies).
go get -u specific.com/package#version the same but to a specific version.
go get -u all updates modules from the build list from go.mod. This is useful for listing (go list -m -u all) but not too useful for updates.

Related

How to prevent go get from updating go.mod file

TL;DR: Is there any way I can forcefully prevent go get from altering the go.mod file?
When I do a go get of certain packages, e.g.:
$ go get github.com/AsynkronIT/protoactor-go/protobuf/protoc-gen-gograin
It will printout that it has updated dependencies (which are defined in my go.mod file):
go get: upgraded github.com/AsynkronIT/protoactor-go v0.0.0-20200815184336-b225d28383f2 => v0.0.0-20210405044454-10bc19881ad6
# (...) Note, this happens for other packages, not just `AsynkronIT/protoactor-go`.
This causes the go.mod file to change during a CI build, and affects subsequent build stages where, while building something, it’ll try to use an updated version of the dependency, which might introduce breaking changes, instead of the version defined on the go.mod file initially.
I’ve tried using -mod=readonly or making sure the -u flag is not used but it will still update the go.mod file, e.g.:
$ GOFLAGS=-mod=readonly go get github.com/AsynkronIT/protoactor-go/protobuf/protoc-gen-gograin
go get: upgraded github.com/AsynkronIT/protoactor-go v0.0.0-20200815184336-b225d28383f2 => v0.0.0-20210405044454-10bc19881ad6
# (...)
I've also tried finding similar issues, like this one, or this other one, but haven't yet find an alternative to prevent go get commands from altering the go.mod.
The current workaround I'm using to stop this behavior is to do a git checkout -- go.mod right after certain go get … steps to reset any changes done by go get, and hence, avoiding breaking changes with certain dependencies newer versions.
I'm using go version 1.16.3.
For Go 1.16 and after, you can use go install to install binaries without affecting go.mod
go install github.com/AsynkronIT/protoactor-go/protobuf/protoc-gen-gograin

modules: how to force version of dependent lib

Overall go modules works just like the docs say. But, in one case, a dependent lib's latest tag is v1.3.0 which go mod tidy finds. However I need v1.1.0. No matter what I do (hand edit go.mod, go get -u) 'go build' merely reverts the version right back to v1.3.0 which breaks the build. Indeed, the whole point of modules is setting the dependencies: so how? Go get should set it permanently but it doesn't stay set. Using go1.13.5
No matter what I do (hand edit go.mod, go get -u) ...
After you modify the version of the dependency in the go.mod file, then what you need to do is perform go mod tidy command (from your explanation I assume you haven't done it).
Example:
# modify the dependency version on go.mod, then
go mod tidy
If you use vendoring then you also need to run the go mod vendor command to update the downloaded dependencies inside vendor/ folder.
# modify the dependency version on go.mod, then
go mod tidy
go mod vendor

Using "go get" to download binaries without adding them to go.mod

I'm using Go modules in my project and in my build system (e.g. Travis CI) I'm downloading a command-line utility (written in Go) with go get to assist with my build process, e.g.:
go get github.com/mitchellh/gox
However, this go get causes the file to be added to my go.mod file. This is contaminating the build environment, causing it to be "dirty" (since there are changes to some files tracked in git, in this case go.mod and go.sum), and I use git describe --always --dirty --tag to describe my build, which shows up as "dirty".
Is there a way to "go get" a binary just to download it, without adding it to the go.mod/go.sum?
I've tried setting GOPATH to somewhere else, even then, go get updates the go.mod/go.sum to add this as an // indirect dependency.
dir="$(mktemp -d)"; \
env GOPATH="$dir" go get github.com/mitchellh/gox && \
mv "$dir/bin/gox" "$(go env GOPATH)"/bin/gox
Go 1.16 onwards
Go 1.16 (released February 2021) includes a change that makes it possible to install a binary without affecting go.mod.
Issue 40276 tracks the proposal:
cmd/go: 'go install' should install executables in module mode outside a module
This was implemented in CL 254365. As part of this change, you can run e.g.:
go install golang.org/x/tools/cmd/goimports#latest
to install a binary without affecting go.mod.
To install a specific version, replace #latest with e.g. #v0.1.5.
Hopefully in Go 1.14 there will be a new flag for go get that does exactly what you are asking. This is tracked in issue #30515 "cmd/go: offer a consistent global install command".
Prior to that, you have a few different options.
Go 1.12 and 1.13: change directory
If you are using Go 1.12 or later, the simplest solution is probably to move outside your current module to a directory without a go.mod prior to doing the go get, such as:
$ cd /tmp
$ go get github.com/foo/bar#v1.2.3
$ cd - # return to prior directory
Go 1.11, 1.12, 1.13+: gobin
gobin is a module-aware command to install or run binaries that provides additional flexibility, including the ability to install without altering your current module's go.mod. See the gobin README and FAQ for more details.
Go 1.11: temporary module
If you are using Go 1.11 with modules, the first step is probably to upgrade to Go 1.12 or 1.13 given there are many improvements in modules. If you are required to use Go 1.11 and want to use the #version syntax without updating your current module's go.mod, then one approach is to create a temporary module:
cd $(mktemp -d) && go mod init tempmod && go get github.com/foo/bar#v1.2.3
This is because in Go 1.11, you can't use the #version syntax unless you are in a module, which was relaxed in Go 1.12. This approach has been automated by a simple shell script by #rogpeppe.
Additional Details
In general, the go command in module-module always determines what module it is "in", which is based on the current working directory when you invoke the go command. (You could make an analogy to how make without any args will look for a makefile in the current working directory, or how historically go build without any args will build the current working directory, etc.).
With modules, go get looks for a go.mod file in the current working directory or any of its parents, and go get will use the constraints listed in any go.mod as part of solving for versions, as well as update the go.mod if needed based on doing the go get. That is why your go.mod file is updated if you run go get from within an existing module.
On the other hand, starting with Go 1.12, if you are in a directory that is not part of any module (that is, the directory does not have a go.mod, nor do any of its parents), then there is no go.mod to update, but the go command is still able to operate in module mode and use the #version syntax.
From the Go 1.12 release notes:
When GO111MODULE is set to on, the go command now supports module-aware operations outside of a module directory, provided that those operations do not need to resolve import paths relative to the current directory or explicitly edit the go.mod file. Commands such as go get, go list, and go mod download behave as if in a module with initially-empty requirements. In this mode, go env GOMOD reports the system's null device (/dev/null or NUL).
Per go help build:
The -mod build flag provides additional control over updating and use of go.mod.
If invoked with -mod=readonly, the go command is disallowed from the implicit automatic updating of go.mod

go mod vendor without update to latest

I’m trying to figure out if it’s possible to run go mod vendor without the go tool updating my go.mod file.
I specifically go get package/subpackage#commit and commit my go.mod with the correct version.
Then I run go mod vendor and it automatically bumps the version of the package that I just specifically set.
I’ve looked at this page to no avail: https://github.com/golang/go/wiki/Modules#how-do-i-use-vendoring-with-modules-is-vendoring-going-away
I need to use vendor because I run a script that edits some of the vendored deps., I’m looking at the following build flow:
GO111MODULE=on go get package/subpackge#commit
GO111MODULE=on go mod vendor
./Script/patch_vendors.sh --write
GO111MODULE=off go build
My other option is modifying the copied source wherever go mod vendor donwloads it to, but
not sure how to approach that.
Thanks in advance
Per https://tip.golang.org/cmd/go/#hdr-Maintaining_module_requirements:
The go command itself automatically updates the go.mod file to maintain a standard formatting and the accuracy of require statements.
Any go command that finds an unfamiliar import will look up the module containing that import and add the latest version of that module to go.mod automatically. […]
Any go command can determine that a module requirement is missing and must be added […].
The go mod vendor command copies in all of the transitive imports of your packages and their tests, so it will automatically update the go.mod file to ensure that all of the imported packages are present.
So the problem here is likely that the commit you've selected for package/subpackage fails to provide some package that appears in the transitive imports of your program. If that is correct, you should find that go list all, go test all, and go mod tidy all make that same edit to your module's requirements.

How do go modules work with installable commands?

I've recently started with Go 1.11 and love the modules. Apart from runtime dependencies I need to work with go modules during the build, e.g. during go generate.
How can I install a specific build dependency (e.g. github.com/aprice/embed/cmd/embed) and run that specific tool from which folder? Is go get the right tool for doing so?
If you get an error
I was not seeing the dependency that I wanted added to the go.mod and I was getting this error:
internal/tools/tools.go:6:5: import "github.com/UnnoTed/fileb0x" is a program, not an importable package
(fileb0x is the thing I'm trying to add)
I'm not 100% clear on the sequence of events that fixed it, but I did all of these things:
Using a "tools" package
I made a tools directory:
mkdir -p internal/tools
I put the tools package inside of it (as mentioned above):
internal/tools/tools.go:
// +build tools
package tools
import (
_ "github.com/UnnoTed/fileb0x"
)
Note that the tag is mostly not important. You could use foo:
// +build foo
However, you cannot use ignore. That's a special predefined tag.
// +build ignore
// NO NO NO NO NO
// `ignore` is a special keyword which (surprise) will cause
// the file to be ignore, even for dependencies
Updating go.mod
The best way is probably to run go mod tidy:
go mod tidy
However, before I did that I ran a number of commands trying to figure out which one would cause it to go into go.mod:
go install github.com/UnnoTed/fileb0x # didn't seem to do the trick
go get
go generate ./...
go build ./...
go install ./...
go mod vendor
Later I did a git reset and rm -rf ~/go/pkg/mod; mkdir ~/go/pkg/mod and found that go mod tidy did well enough on its own.
vendoring
In order to actually take advantage of the modules cache in a project you need to copy-in the source code
go mod vendor
That will grab all dependencies from go.mod
You also need to change nearly all of your go commands to use -mod=vendor in any Makefiles, Dockerfiles or other scripts.
go fmt -mod=vendor ./... # has a bug slated to be fixed in go1.15
go generate -mod=vendor ./...
go build -mod=vendor ./...
That includes go build, go get, go install, and any go run called by go generate (and even the go generate itself)
//go:generate go run -mod=vendor github.com/UnnoTed/fileb0x b0x.toml
package main
// ...
https://github.com/golang/go/issues/25922 proved helpful for me, especially
when using build-only dependencies with modules the main point is version selection (not installing these!)
To avoid installing you can modify your //go:generate directive to something like:
//go:generate go run golang.org/x/tools/cmd/stringer ARGS
There is also the best practices repo: https://github.com/go-modules-by-example/index/blob/master/010_tools/README.md
The convention is to add a file named "tools.go" that is guarded by a build constraint and imports all required tools:
// +build tools
package tools
import (
_ "github.com/aprice/embed/cmd/embed"
)
https://github.com/golang/go/issues/25922#issuecomment-412992431
The tools are then installed as usual in one of
$GOBIN
$GOPATH/bin
$HOME/go/bin
You may also want to follow https://github.com/golang/go/issues/27653, which discusses future explicit support for tools.
tools.go is a great solution if you're building an app or service. But if you're building a library, tools.go still leaks dependencies to things consuming your library (your tools are still there as indirect dependencies, and go mod tidy will pull them in since it considers every possible target). That's not the end of the world since those modules never end up in the actual built binaries of the consumer, but it's still messy.
https://github.com/myitcv/gobin/issues/44 is probably the most promising approach to fixing this long term, but short term I've used a combination of the "internal module" approach explained there along with https://github.com/izumin5210/gex.
First, I install gex globally:
GO111MODULE=off go get github.com/izumin5210/gex/cmd/gex
Then before actually using gex I create a structure like this:
myproject/
\
- go.mod: module github.com/ysamlan/myproject
\
internal/
\
tools/
- go.mod: module github.com/ysamlan/myproject/tools
To install a build-only tool I just cd internal/tools and run gex --add (sometool), which puts that tool in internal/tools/bin. CI scripts and other folks that want to build my stuff locally just need to run cd internal/tools && gex --build to reliably and reproducibly populate the tool binaries, but the top-level go.mod is unchanged.
The key piece there is creating that internal/tools/go.mod file with a different module path than the one the root project uses, and then only running gex from that directory.

Resources