I'm currently using Dep and would like to start using Go modules.
How do I migrate?
Migrating from Dep to Go Modules is very easy.
Run go version and make sure you're using Go version 1.11 or later.
Move your code outside of GOPATH or set export GO111MODULE=on.
go mod init [module path]: This will import dependencies from Gopkg.lock.
go mod tidy: This will remove unnecessary imports, and add indirect ones.
(Optional) Delete your vendor folder (rm -rf vendor/ or move to trash)
go build: Do a test build to see if it works.
rm -f Gopkg.lock Gopkg.toml: Delete the obsolete files used for Dep.
Go has imported my dependencies from Dep by reading the Gopkg.lock file and also created a go.mod file.
If you want to keep your vendor folder:
Run go mod vendor to copy your dependencies into the vendor folder.
Run go build -mod=vendor to ensure go build uses your vendor folder.
To add to #Nicholas answer's:
Here is from the offical golang documenation:
To create a go.mod for an existing project:
Navigate to the root of the module's source tree outside of GOPATH:
$ export GO111MODULE=on # manually active module mode
$ cd $GOPATH/src/<project path> # e.g., cd $GOPATH/src/you/hello
Create the initial module definition and write it to the go.mod file:
$ go mod init
This step converts from any existing dep Gopkg.lock file or from any of the other nine total supported dependency formats, adding require statements to match the existing configuration.
Build the module. When executed from the root directory of a module, the ./... pattern matches all the packages within the current module. go build will automatically add missing or unconverted dependencies as needed to satisfy imports for this particular build invocation:
$ go build ./...
Test the module as configured to ensure that it works with the selected versions:
$ go test ./...
(Optional) Run the tests for your module plus the tests for all direct and indirect dependencies to check for incompatibilities:
$ go test all
Another way to upgrade to modules.
Remove the Gopkg.toml and Gopkg.lock
rm Gopkg.*
Initialise the Go modules
GO111MODULE=on go mod init
Run go mod tidy to pull all the indirect modules and remove unused modules
GO111MODULE=on go mod tidy
Run build to ensure everything works fine
go build
Tip in case you face few modules not found then manually update the modules tag in go.mod file.
Related
I am trying to fetch a Go project and copy the dependencies under the vendor/ directory so I have the complete source code of the project and its dependencies in my project. However, even after doing that, deleting the packages under $GOPATH/pkg/mod and rebuilding cause the Go compiler to re-fetch all dependencies, which takes a considerable of time.
This is what I did:
# Fetch the project, e.g. influx/telegraf
go get -d github.com/influxdata/telegraf
# CD into the project
cd $GOPATH/src/influxdata/telegraf
# Fetch the modules under vendor/ directory
go mod vendor
After calling the last command, Go will fetch all the dependencies under pkg/mod. Not sure why it is doing that, but I assume it is because it needs to build the project normally, and then move the fetched dependencies under the vendor/ folder. After that, I can build successfully. However, to make sure I no longer require those dependencies, I deleted the pkg/mod directory completely and rebuilt the project. Go compiler, for some reason, fetched the packages again.
Is there anything I am doing wrong?
Thanks!
The vendor folder is not used automatically in all cases.
To make sure dependencies are loaded from the main module's vendor folder, pass -mod=vendor to the go tool.
The vendor folder if present is only used automatically (if not specified otherwise with -mod=mod) if the go version specified by go.mod file is 1.14 or higher.
These are detailed in Command go: Maintaining module requirements:
If invoked with -mod=vendor, the go command loads packages from the main module's vendor directory instead of downloading modules to and loading packages from the module cache. The go command assumes the vendor directory holds correct copies of dependencies, and it does not compute the set of required module versions from go.mod files. However, the go command does check that vendor/modules.txt (generated by 'go mod vendor') contains metadata consistent with go.mod.
If invoked with -mod=mod, the go command loads modules from the module cache even if there is a vendor directory present.
If the go command is not invoked with a -mod flag and the vendor directory is present and the "go" version in go.mod is 1.14 or higher, the go command will act as if it were invoked with -mod=vendor.
Setup:
Our project is using golang 1.12.14
We are using go build -mod=vendor
Issue: When new dependencies are added to go.mod the vendor folder isn't updated and people are committing code and forgetting to run go mod vendor to update the folder. My understanding is that since -mod=vendor specifies to use packages from the vendor folder, the go.mod file will have discrepancies from what we are actually using when building the project.
Question: Should go mod vendor be added to a pre-commit hook?
As of Go 1.14, the go command automatically checks for consistency between the vendor directory and the go.mod file whenever the vendor directory is used. In addition, it uses the vendor directory by default if the module specifies go 1.14 or higher (see https://tip.golang.org/doc/go1.14#go-command).
As of today, the oldest supported version of the Go toolchain is Go 1.15.13.
So if you upgrade to a supported version of the Go toolchain, it should not be necessary to run go mod vendor as a pre-commit hook. The go command itself will flag inconsistencies whenever the vendor directory is used.
I've installed a package using go modules (go get in Go 1.13) and now I want to remove it. In the documentation there is nothing about this and in go get docu neither.
Removing the package from go.mod manually doesn't solve the issue so it remains in go.sum.
How should I remove a package in a clean way?
Found it https://go.dev/blog/using-go-modules#removing-unused-dependencies
go mod tidy
So basically, once the package is not being imported in any package you can perform a go mod tidy and it will safely remove the unused dependencies.
And if you are vendoring the dependencies, then run the command below to make the module changes be applied in the vendor folder:
go mod vendor
#jesugmz answer doesn't say, what if you wanna remove a currently using package in go modules.
So, if you're using go modules (you have a go.mod file in your project) and you want to remove a currently using package, check $GOPATH/pkg/mod/ directory and simply remove the package named package#version.
For example, if you have github.com/some/project package installed, you should run the following command:
rm -rf $(go env GOPATH)/pkg/mod/github.com/some/project#v1.0.0
You can find the using package version in go.mod file.
I'm using Go modules in my project and in my build system (e.g. Travis CI) I'm downloading a command-line utility (written in Go) with go get to assist with my build process, e.g.:
go get github.com/mitchellh/gox
However, this go get causes the file to be added to my go.mod file. This is contaminating the build environment, causing it to be "dirty" (since there are changes to some files tracked in git, in this case go.mod and go.sum), and I use git describe --always --dirty --tag to describe my build, which shows up as "dirty".
Is there a way to "go get" a binary just to download it, without adding it to the go.mod/go.sum?
I've tried setting GOPATH to somewhere else, even then, go get updates the go.mod/go.sum to add this as an // indirect dependency.
dir="$(mktemp -d)"; \
env GOPATH="$dir" go get github.com/mitchellh/gox && \
mv "$dir/bin/gox" "$(go env GOPATH)"/bin/gox
Go 1.16 onwards
Go 1.16 (released February 2021) includes a change that makes it possible to install a binary without affecting go.mod.
Issue 40276 tracks the proposal:
cmd/go: 'go install' should install executables in module mode outside a module
This was implemented in CL 254365. As part of this change, you can run e.g.:
go install golang.org/x/tools/cmd/goimports#latest
to install a binary without affecting go.mod.
To install a specific version, replace #latest with e.g. #v0.1.5.
Hopefully in Go 1.14 there will be a new flag for go get that does exactly what you are asking. This is tracked in issue #30515 "cmd/go: offer a consistent global install command".
Prior to that, you have a few different options.
Go 1.12 and 1.13: change directory
If you are using Go 1.12 or later, the simplest solution is probably to move outside your current module to a directory without a go.mod prior to doing the go get, such as:
$ cd /tmp
$ go get github.com/foo/bar#v1.2.3
$ cd - # return to prior directory
Go 1.11, 1.12, 1.13+: gobin
gobin is a module-aware command to install or run binaries that provides additional flexibility, including the ability to install without altering your current module's go.mod. See the gobin README and FAQ for more details.
Go 1.11: temporary module
If you are using Go 1.11 with modules, the first step is probably to upgrade to Go 1.12 or 1.13 given there are many improvements in modules. If you are required to use Go 1.11 and want to use the #version syntax without updating your current module's go.mod, then one approach is to create a temporary module:
cd $(mktemp -d) && go mod init tempmod && go get github.com/foo/bar#v1.2.3
This is because in Go 1.11, you can't use the #version syntax unless you are in a module, which was relaxed in Go 1.12. This approach has been automated by a simple shell script by #rogpeppe.
Additional Details
In general, the go command in module-module always determines what module it is "in", which is based on the current working directory when you invoke the go command. (You could make an analogy to how make without any args will look for a makefile in the current working directory, or how historically go build without any args will build the current working directory, etc.).
With modules, go get looks for a go.mod file in the current working directory or any of its parents, and go get will use the constraints listed in any go.mod as part of solving for versions, as well as update the go.mod if needed based on doing the go get. That is why your go.mod file is updated if you run go get from within an existing module.
On the other hand, starting with Go 1.12, if you are in a directory that is not part of any module (that is, the directory does not have a go.mod, nor do any of its parents), then there is no go.mod to update, but the go command is still able to operate in module mode and use the #version syntax.
From the Go 1.12 release notes:
When GO111MODULE is set to on, the go command now supports module-aware operations outside of a module directory, provided that those operations do not need to resolve import paths relative to the current directory or explicitly edit the go.mod file. Commands such as go get, go list, and go mod download behave as if in a module with initially-empty requirements. In this mode, go env GOMOD reports the system's null device (/dev/null or NUL).
Per go help build:
The -mod build flag provides additional control over updating and use of go.mod.
If invoked with -mod=readonly, the go command is disallowed from the implicit automatic updating of go.mod
I am trying to convert one Go project to the new modules standard, but I am having trouble with one dependency.
That repository has vendored a library (in this case golang.org/x/net/html) to add some functionality to the library; meaning that it has modified the vendored dependency (added a method to a struct).
So the problem is that after doing go mod init, when i execute go get ./... the system complains about a call to that added functionality, because it ignores the modified vendor and tries to import the upstream library instead.
Is there any way to let go mod know that it should use the vendored dependency?
You can try replace directive. Something along the lines of
First
cd ./vendor/golang.org/x/net/html
go mod init
Then in your root go.mod
module your/super/module
replace golang.org/x/net/html => ./vendor/golang.org/x/net/html
require golang.org/x/net/html v0.0.0
Note that both require and replace are required.
After go1.15 you can build with -mod=readonly to avoid imports only from the vendor directory.
example:
go build -mod=readonly -o ./build/project -ldflags "-s -w" main.go