TL;DR: Is there any way I can forcefully prevent go get from altering the go.mod file?
When I do a go get of certain packages, e.g.:
$ go get github.com/AsynkronIT/protoactor-go/protobuf/protoc-gen-gograin
It will printout that it has updated dependencies (which are defined in my go.mod file):
go get: upgraded github.com/AsynkronIT/protoactor-go v0.0.0-20200815184336-b225d28383f2 => v0.0.0-20210405044454-10bc19881ad6
# (...) Note, this happens for other packages, not just `AsynkronIT/protoactor-go`.
This causes the go.mod file to change during a CI build, and affects subsequent build stages where, while building something, it’ll try to use an updated version of the dependency, which might introduce breaking changes, instead of the version defined on the go.mod file initially.
I’ve tried using -mod=readonly or making sure the -u flag is not used but it will still update the go.mod file, e.g.:
$ GOFLAGS=-mod=readonly go get github.com/AsynkronIT/protoactor-go/protobuf/protoc-gen-gograin
go get: upgraded github.com/AsynkronIT/protoactor-go v0.0.0-20200815184336-b225d28383f2 => v0.0.0-20210405044454-10bc19881ad6
# (...)
I've also tried finding similar issues, like this one, or this other one, but haven't yet find an alternative to prevent go get commands from altering the go.mod.
The current workaround I'm using to stop this behavior is to do a git checkout -- go.mod right after certain go get … steps to reset any changes done by go get, and hence, avoiding breaking changes with certain dependencies newer versions.
I'm using go version 1.16.3.
For Go 1.16 and after, you can use go install to install binaries without affecting go.mod
go install github.com/AsynkronIT/protoactor-go/protobuf/protoc-gen-gograin
Related
Using this module as an example (using a specific commit so others will see
what I see):
git clone git://github.com/walles/moar
Set-Location moar
git checkout d24acdbf
I would like a way to tell Go to "update everything". Assume that the module
will work with the newest version of everything. Below are five ways I found to
do this, assume each is run on a clean clone. This results in a go.mod of 19
lines:
go get -u
This results in a go.mod of 14 lines:
go get -u
go mod tidy
This results in a go.mod of 13 lines:
go mod tidy
If I just manually delete everything in require and run go mod tidy, I get
12 lines. If I just manually delete everything in require and run go get -u, I get 11 lines. My question is, why are these methods producing different
results, and what is the "right way" to do what I am trying to do?
tl;dr;
this is what you want:
go get -u
go mod tidy
and to recursively update packages in any subdirectories:
go get -u ./...
The inconsistencies you are seeing is due to the inherent organic nature of software.
Using your example, commit d24acdbf of git://github.com/walles/moar most likely was checked in by the maintainer without running go mod tidy (explaining the longer 19 lines). If the maintainer had, then you would see the 13 line version you see at the end.
go get -u on it's own is more aggressive in pulling in dependencies. Also, the mere fact of updating dependencies to their latest (compatible) version, may in & of itself pull in new direct/indirect dependencies. These dependencies may grow even further if you tried this tomorrow (the latest version of some sub-dependency adds new functionality, so it needs new dependencies). So there may be a valid reason the repo maintainer fixes at a particular (non-latest) version.
go mod tidy cleans up this aggressive dependency analysis.
P.S. It's a common misconception that dependencies will shrink after go mod tidy: tracking go.sum, in some cases this file will grow after a tidy (though, not in this case)
Run go get -u && go mod tidy 1
More details:
go get -u (same as go get -u .) updates the package in the current directory, hence the module that provides that package, and its dependencies to the newer minor or patch releases when available. In typical projects, running this in the module root is enough, as it likely imports everything else.
go get -u ./... will expand to all packages rooted in the current directory, which effectively also updates everything (all modules that provide those packages).
Following from the above, go get -u ./foo/... will update everything that is rooted in ./foo
go get -u all updates everything including test dependencies; from Package List and Patterns
When using modules, all expands to all packages in the main module and their dependencies, including dependencies needed by tests of any of those.
go get will also add to the go.mod file the require directives for dependencies that were just updated.
go mod tidy makes sure go.mod matches the source code in the module. In your project it results in 12 lines because those are the bare minimum to match the source code.
go mod tidy will prune go.sum and go.mod by removing the unnecessary checksums and transitive dependencies (e.g. // indirect), that were added to by go get -u due to newer semver available. It may also add missing entries to go.sum.
Note that starting from Go 1.17, newly-added indirect dependencies in go.mod are arranged in a separate require block.
1: updates dependencies' newest minor/patch versions, go.mod, go.sum
There are some hidden dragons, here is what I recommend:
go get -u ./... walks all packages in your project. This is the command you want to use.
go get -t -u ./... walks all packages in your project and also downloads tests files of these dependencies. Probably you don’t need that.
go get -u updates in the current directory only. Useful for small single-package projects, just use the first version.
go get -u specific.com/package updates just one (or more separated by space) packages (and dependencies).
go get -u specific.com/package#version the same but to a specific version.
go get -u all updates modules from the build list from go.mod. This is useful for listing (go list -m -u all) but not too useful for updates.
I'm trying to create a docker image for use in a build pipeline that has various tools pre-installed for building and testing go projects. One tool we need is golint but I'm struggling to see how to install a specific version of it. The reason I want to lock down the version is to avoid accidental / unwanted / unintended breakages at a later date.
For a start, looking here the versions are not exactly in an easy-to-type format.
Also when I try to use the following command
go get -u golang.org/x/lint/golint#v0.0.0-20181217174547-8f45f776aaf1
I get an error
go: cannot use path#version syntax in GOPATH mode
Googling has so far yielded very few relevant results...
Is what I'm trying to do possible? Many thanks!
You need to be in go module mode to get code of a specific version, since in addition to downloading the code, the version is recorded in the go module file.
The easiest way to do this would be to create an empty directory, run go mod init, which will create a go.mod file.
Then, you can run go get golang.org/x/lint/golint#v0.0.0-20181217174547-8f45f776aaf1, which will add golint at that version to your go.mod file. You can then run go install golang.org/x/lint/golint from within that directory, which will install golint at the version specified into your $GOBIN directory (which defaults to $GOPATH/bin).
I'm a bit confused about how Go modules work on installing binaries using go install.
I tried to install (https://github.com/joho/godotenv) binary by executing go install github.com/joho/godotenv/cmd/godotenv and I found out that it adds a record in the go.mod.
I'm lost as we don't use this package in the code and after running go mod tidy it gets deleted (as it is not it the code).
Can someone explain is it expected behaviour of go modules?
Secondly, how I could avoid adding it to the go.mod as we only need to install and execute the binary?
Thanks.
Go version: go version go1.13.4 darwin/amd64
Command go: The go.mod file:
The go command automatically updates go.mod each time it uses the module graph, to make sure go.mod always accurately reflects reality and is properly formatted.
The go tool will update go.mod automatically when it detects dependencies are inaccurate when performing a build.
When you install github.com/joho/godotenv/cmd/godotenv from your module, this install requires at least the package in question being built / installed (and also its dependencies, transitively).
You may safely run go mod tidy to undo the recording of this "one-time" dependency.
In general if you want to disallow the go tool to update the go.mod file, you may use the -mod=readonly flag, but that would fail go install ("can't load package: package xxx: import lookup disabled by -mod=readonly"). You can read more about this here: Go Wiki: Go modules: Can I control when go.mod gets updated and when the go tools use the network to satisfy dependencies?
Alternatively, if you want to avoid this, build / install your tools outside of your module. You may use a "dummy" module for this.
I'm using Go modules in my project and in my build system (e.g. Travis CI) I'm downloading a command-line utility (written in Go) with go get to assist with my build process, e.g.:
go get github.com/mitchellh/gox
However, this go get causes the file to be added to my go.mod file. This is contaminating the build environment, causing it to be "dirty" (since there are changes to some files tracked in git, in this case go.mod and go.sum), and I use git describe --always --dirty --tag to describe my build, which shows up as "dirty".
Is there a way to "go get" a binary just to download it, without adding it to the go.mod/go.sum?
I've tried setting GOPATH to somewhere else, even then, go get updates the go.mod/go.sum to add this as an // indirect dependency.
dir="$(mktemp -d)"; \
env GOPATH="$dir" go get github.com/mitchellh/gox && \
mv "$dir/bin/gox" "$(go env GOPATH)"/bin/gox
Go 1.16 onwards
Go 1.16 (released February 2021) includes a change that makes it possible to install a binary without affecting go.mod.
Issue 40276 tracks the proposal:
cmd/go: 'go install' should install executables in module mode outside a module
This was implemented in CL 254365. As part of this change, you can run e.g.:
go install golang.org/x/tools/cmd/goimports#latest
to install a binary without affecting go.mod.
To install a specific version, replace #latest with e.g. #v0.1.5.
Hopefully in Go 1.14 there will be a new flag for go get that does exactly what you are asking. This is tracked in issue #30515 "cmd/go: offer a consistent global install command".
Prior to that, you have a few different options.
Go 1.12 and 1.13: change directory
If you are using Go 1.12 or later, the simplest solution is probably to move outside your current module to a directory without a go.mod prior to doing the go get, such as:
$ cd /tmp
$ go get github.com/foo/bar#v1.2.3
$ cd - # return to prior directory
Go 1.11, 1.12, 1.13+: gobin
gobin is a module-aware command to install or run binaries that provides additional flexibility, including the ability to install without altering your current module's go.mod. See the gobin README and FAQ for more details.
Go 1.11: temporary module
If you are using Go 1.11 with modules, the first step is probably to upgrade to Go 1.12 or 1.13 given there are many improvements in modules. If you are required to use Go 1.11 and want to use the #version syntax without updating your current module's go.mod, then one approach is to create a temporary module:
cd $(mktemp -d) && go mod init tempmod && go get github.com/foo/bar#v1.2.3
This is because in Go 1.11, you can't use the #version syntax unless you are in a module, which was relaxed in Go 1.12. This approach has been automated by a simple shell script by #rogpeppe.
Additional Details
In general, the go command in module-module always determines what module it is "in", which is based on the current working directory when you invoke the go command. (You could make an analogy to how make without any args will look for a makefile in the current working directory, or how historically go build without any args will build the current working directory, etc.).
With modules, go get looks for a go.mod file in the current working directory or any of its parents, and go get will use the constraints listed in any go.mod as part of solving for versions, as well as update the go.mod if needed based on doing the go get. That is why your go.mod file is updated if you run go get from within an existing module.
On the other hand, starting with Go 1.12, if you are in a directory that is not part of any module (that is, the directory does not have a go.mod, nor do any of its parents), then there is no go.mod to update, but the go command is still able to operate in module mode and use the #version syntax.
From the Go 1.12 release notes:
When GO111MODULE is set to on, the go command now supports module-aware operations outside of a module directory, provided that those operations do not need to resolve import paths relative to the current directory or explicitly edit the go.mod file. Commands such as go get, go list, and go mod download behave as if in a module with initially-empty requirements. In this mode, go env GOMOD reports the system's null device (/dev/null or NUL).
Per go help build:
The -mod build flag provides additional control over updating and use of go.mod.
If invoked with -mod=readonly, the go command is disallowed from the implicit automatic updating of go.mod
I’m trying to figure out if it’s possible to run go mod vendor without the go tool updating my go.mod file.
I specifically go get package/subpackage#commit and commit my go.mod with the correct version.
Then I run go mod vendor and it automatically bumps the version of the package that I just specifically set.
I’ve looked at this page to no avail: https://github.com/golang/go/wiki/Modules#how-do-i-use-vendoring-with-modules-is-vendoring-going-away
I need to use vendor because I run a script that edits some of the vendored deps., I’m looking at the following build flow:
GO111MODULE=on go get package/subpackge#commit
GO111MODULE=on go mod vendor
./Script/patch_vendors.sh --write
GO111MODULE=off go build
My other option is modifying the copied source wherever go mod vendor donwloads it to, but
not sure how to approach that.
Thanks in advance
Per https://tip.golang.org/cmd/go/#hdr-Maintaining_module_requirements:
The go command itself automatically updates the go.mod file to maintain a standard formatting and the accuracy of require statements.
Any go command that finds an unfamiliar import will look up the module containing that import and add the latest version of that module to go.mod automatically. […]
Any go command can determine that a module requirement is missing and must be added […].
The go mod vendor command copies in all of the transitive imports of your packages and their tests, so it will automatically update the go.mod file to ensure that all of the imported packages are present.
So the problem here is likely that the commit you've selected for package/subpackage fails to provide some package that appears in the transitive imports of your program. If that is correct, you should find that go list all, go test all, and go mod tidy all make that same edit to your module's requirements.