I am trying to convert one Go project to the new modules standard, but I am having trouble with one dependency.
That repository has vendored a library (in this case golang.org/x/net/html) to add some functionality to the library; meaning that it has modified the vendored dependency (added a method to a struct).
So the problem is that after doing go mod init, when i execute go get ./... the system complains about a call to that added functionality, because it ignores the modified vendor and tries to import the upstream library instead.
Is there any way to let go mod know that it should use the vendored dependency?
You can try replace directive. Something along the lines of
First
cd ./vendor/golang.org/x/net/html
go mod init
Then in your root go.mod
module your/super/module
replace golang.org/x/net/html => ./vendor/golang.org/x/net/html
require golang.org/x/net/html v0.0.0
Note that both require and replace are required.
After go1.15 you can build with -mod=readonly to avoid imports only from the vendor directory.
example:
go build -mod=readonly -o ./build/project -ldflags "-s -w" main.go
Related
Overall go modules works just like the docs say. But, in one case, a dependent lib's latest tag is v1.3.0 which go mod tidy finds. However I need v1.1.0. No matter what I do (hand edit go.mod, go get -u) 'go build' merely reverts the version right back to v1.3.0 which breaks the build. Indeed, the whole point of modules is setting the dependencies: so how? Go get should set it permanently but it doesn't stay set. Using go1.13.5
No matter what I do (hand edit go.mod, go get -u) ...
After you modify the version of the dependency in the go.mod file, then what you need to do is perform go mod tidy command (from your explanation I assume you haven't done it).
Example:
# modify the dependency version on go.mod, then
go mod tidy
If you use vendoring then you also need to run the go mod vendor command to update the downloaded dependencies inside vendor/ folder.
# modify the dependency version on go.mod, then
go mod tidy
go mod vendor
Converting an existing project with a /vendor directory to use Go Modules (go version 1.12). I do go mod init to generate the go.mod file. Then I do go get -u ./... to populate the go.mod file. During this time, it tries to locate a package on github that no longer exists. It is vendored in my /vendor directory.
Until I can upgrade my code to use a different package, how can I continue the conversion to using modules? That is, I want to keep some things vendored (I also have some modified code under /vendor), while other things are handled by go modules.
You can't mix the vendor directory behavior and modules, each method of dependency resolution precludes the other. You can re-publish the missing package yourself somewhere that go mod can locate it, or you can redirect it directly to the existing vendored source in your module.
To redirect the source of a module, use the replace directive in the go.mod file
replace missing/package v0.0.1 => ./vendor/missing/package
I'm currently using Dep and would like to start using Go modules.
How do I migrate?
Migrating from Dep to Go Modules is very easy.
Run go version and make sure you're using Go version 1.11 or later.
Move your code outside of GOPATH or set export GO111MODULE=on.
go mod init [module path]: This will import dependencies from Gopkg.lock.
go mod tidy: This will remove unnecessary imports, and add indirect ones.
(Optional) Delete your vendor folder (rm -rf vendor/ or move to trash)
go build: Do a test build to see if it works.
rm -f Gopkg.lock Gopkg.toml: Delete the obsolete files used for Dep.
Go has imported my dependencies from Dep by reading the Gopkg.lock file and also created a go.mod file.
If you want to keep your vendor folder:
Run go mod vendor to copy your dependencies into the vendor folder.
Run go build -mod=vendor to ensure go build uses your vendor folder.
To add to #Nicholas answer's:
Here is from the offical golang documenation:
To create a go.mod for an existing project:
Navigate to the root of the module's source tree outside of GOPATH:
$ export GO111MODULE=on # manually active module mode
$ cd $GOPATH/src/<project path> # e.g., cd $GOPATH/src/you/hello
Create the initial module definition and write it to the go.mod file:
$ go mod init
This step converts from any existing dep Gopkg.lock file or from any of the other nine total supported dependency formats, adding require statements to match the existing configuration.
Build the module. When executed from the root directory of a module, the ./... pattern matches all the packages within the current module. go build will automatically add missing or unconverted dependencies as needed to satisfy imports for this particular build invocation:
$ go build ./...
Test the module as configured to ensure that it works with the selected versions:
$ go test ./...
(Optional) Run the tests for your module plus the tests for all direct and indirect dependencies to check for incompatibilities:
$ go test all
Another way to upgrade to modules.
Remove the Gopkg.toml and Gopkg.lock
rm Gopkg.*
Initialise the Go modules
GO111MODULE=on go mod init
Run go mod tidy to pull all the indirect modules and remove unused modules
GO111MODULE=on go mod tidy
Run build to ensure everything works fine
go build
Tip in case you face few modules not found then manually update the modules tag in go.mod file.
I am looking to create reproducible builds with go.
For individual projects we are using glide.
So for example I use:
glide get github.com/stretchr/testify
to fix the version of the "testify" package.
This does not work for tools however.
For example:
glide install github.com/tebeka/go2xunit
returns success but does not actually install go2xunit
so I have to use:
go get github.com/tebeka/go2xunit
which installs go2xunit to $GOPATH/bin.
Q How can I fix the version of tools like go2xunit?
I also note that glide says use dep instead and dep says golang has diverged from its implementation and will probably end up using something based on vgo. There are a plethora of dependency management tools for go perhaps one of the less well known ones supports this?
In case its relevant I'm using go 1.7.4 as provided by Debian9.
The solution for go1.11 using go modules is to create a fake tools package.
You create a tools.go file like the following:
// +build tools
package tools
import (
_ "github.com/tebeka/go2xunit"
)
+build tools is a magic comment which prevents the package being built.
>go mod init tools
Will create a go.mod file for the fake tools package
>go install github.com/tebeka/go2xunit
Will install go2xunit and update go.mod as follows.
module tools
require github.com/tebeka/go2xunit v1.4.8 // indirect
Now if you run go install github.com/tebeka/go2xunit in the future (for a clean build say) its version will be fixed to v1.4 by the go.mod
For versions of go before 1.11 the tool to use is retool.
It works like this:
bootstrap:
go get github.com/twitchtv/retool
add tool:
retool add github.com/jteeuwen/go-bindata/go-bindata origin/master
use tool:
retool do go-bindata -pkg testdata -o ./testdata/testdata.go ./testdata/data.json
Adding support for this may be on the roadmap to target go 1.12 (https://github.com/golang/go/issues/27653)
I did this very similarly, but just different enough that I think it's worth sharing again:
If you get an error
I was not seeing the dependency that I wanted added to the go.mod and I was getting this error:
tools/tools.go:6:5: import "github.com/UnnoTed/fileb0x" is a program, not an importable package
(fileb0x is the thing I'm trying to add)
I'm not 100% clear on the sequence of events that fixed it, but I did all of these things:
Using a "tools" package
I made a tools directory:
mkdir -p tools
I put the tools package inside of it (as mentioned above):
// +build tools
package tools
import (
_ "github.com/UnnoTed/fileb0x"
)
Note that the tag is mostly not important. You could use foo:
// +build foo
However, you cannot use ignore. That's a special predefined tag.
// +build ignore
// NO NO NO NO NO
// `ignore` is a special keyword which (surprise) will cause
// the file to be ignore, even for dependencies
Updating go.mod
The best way is probably to run go mod tidy:
go mod tidy
However, before I did that I ran a number of commands trying to figure out which one would cause it to go into go.mod:
go install github.com/UnnoTed/fileb0x # didn't seem to do the trick
go get
go generate ./...
go build ./...
go install ./...
go mod vendor
Later I did a git reset and rm -rf ~/go/pkg/mod; mkdir ~/go/pkg/mod and found that go mod tidy did well enough on its own.
vendoring
In order to actually take advantage of the modules cache in a project you need to copy-in the source code
go mod vendor
That will grab all dependencies from go.mod
You also need to change nearly all of your go commands to use -mod=vendor in any Makefiles, Dockerfiles or other scripts.
go fmt -mod=vendor ./... # has a bug which should be fixed in go1.15
go generate -mod=vendor ./...
go build -mod=vendor ./...
That includes go build, go get, go install, and any go run called by go generate (and even the go generate itself)
//go:generate go run -mod=vendor github.com/UnnoTed/fileb0x b0x.toml
package main
// ...
I've recently started with Go 1.11 and love the modules. Apart from runtime dependencies I need to work with go modules during the build, e.g. during go generate.
How can I install a specific build dependency (e.g. github.com/aprice/embed/cmd/embed) and run that specific tool from which folder? Is go get the right tool for doing so?
If you get an error
I was not seeing the dependency that I wanted added to the go.mod and I was getting this error:
internal/tools/tools.go:6:5: import "github.com/UnnoTed/fileb0x" is a program, not an importable package
(fileb0x is the thing I'm trying to add)
I'm not 100% clear on the sequence of events that fixed it, but I did all of these things:
Using a "tools" package
I made a tools directory:
mkdir -p internal/tools
I put the tools package inside of it (as mentioned above):
internal/tools/tools.go:
// +build tools
package tools
import (
_ "github.com/UnnoTed/fileb0x"
)
Note that the tag is mostly not important. You could use foo:
// +build foo
However, you cannot use ignore. That's a special predefined tag.
// +build ignore
// NO NO NO NO NO
// `ignore` is a special keyword which (surprise) will cause
// the file to be ignore, even for dependencies
Updating go.mod
The best way is probably to run go mod tidy:
go mod tidy
However, before I did that I ran a number of commands trying to figure out which one would cause it to go into go.mod:
go install github.com/UnnoTed/fileb0x # didn't seem to do the trick
go get
go generate ./...
go build ./...
go install ./...
go mod vendor
Later I did a git reset and rm -rf ~/go/pkg/mod; mkdir ~/go/pkg/mod and found that go mod tidy did well enough on its own.
vendoring
In order to actually take advantage of the modules cache in a project you need to copy-in the source code
go mod vendor
That will grab all dependencies from go.mod
You also need to change nearly all of your go commands to use -mod=vendor in any Makefiles, Dockerfiles or other scripts.
go fmt -mod=vendor ./... # has a bug slated to be fixed in go1.15
go generate -mod=vendor ./...
go build -mod=vendor ./...
That includes go build, go get, go install, and any go run called by go generate (and even the go generate itself)
//go:generate go run -mod=vendor github.com/UnnoTed/fileb0x b0x.toml
package main
// ...
https://github.com/golang/go/issues/25922 proved helpful for me, especially
when using build-only dependencies with modules the main point is version selection (not installing these!)
To avoid installing you can modify your //go:generate directive to something like:
//go:generate go run golang.org/x/tools/cmd/stringer ARGS
There is also the best practices repo: https://github.com/go-modules-by-example/index/blob/master/010_tools/README.md
The convention is to add a file named "tools.go" that is guarded by a build constraint and imports all required tools:
// +build tools
package tools
import (
_ "github.com/aprice/embed/cmd/embed"
)
https://github.com/golang/go/issues/25922#issuecomment-412992431
The tools are then installed as usual in one of
$GOBIN
$GOPATH/bin
$HOME/go/bin
You may also want to follow https://github.com/golang/go/issues/27653, which discusses future explicit support for tools.
tools.go is a great solution if you're building an app or service. But if you're building a library, tools.go still leaks dependencies to things consuming your library (your tools are still there as indirect dependencies, and go mod tidy will pull them in since it considers every possible target). That's not the end of the world since those modules never end up in the actual built binaries of the consumer, but it's still messy.
https://github.com/myitcv/gobin/issues/44 is probably the most promising approach to fixing this long term, but short term I've used a combination of the "internal module" approach explained there along with https://github.com/izumin5210/gex.
First, I install gex globally:
GO111MODULE=off go get github.com/izumin5210/gex/cmd/gex
Then before actually using gex I create a structure like this:
myproject/
\
- go.mod: module github.com/ysamlan/myproject
\
internal/
\
tools/
- go.mod: module github.com/ysamlan/myproject/tools
To install a build-only tool I just cd internal/tools and run gex --add (sometool), which puts that tool in internal/tools/bin. CI scripts and other folks that want to build my stuff locally just need to run cd internal/tools && gex --build to reliably and reproducibly populate the tool binaries, but the top-level go.mod is unchanged.
The key piece there is creating that internal/tools/go.mod file with a different module path than the one the root project uses, and then only running gex from that directory.