Migrate to Go modules while keep on using distribution packages - go

Question

I have a GOPATH based project that I currently build like this on Fedora:
sudo dnf install golang-etcd-bbolt-devel golang-x-sys-devel golang-x-text-devel
GOPATH=$HOME/go:/usr/share/gocode go build
My project (gonzofilter) implements a command line utility and thus source files are located in the main package (i.e. they have a package main declaration).
With Fedora 34 and beyond it seems that Go removed support for building GOPATH style projects and one really has to use Go modules:
go build
go: cannot find main module; see 'go help modules'
That Go blog post kind of covers my case (-> 'Without a dependency manager'), but it doesn't explicitly mention how to deal with main package projects or with distribution provided dependencies.
So, how do I migrate such a project?
How do I have to tell Go/go mod tidy to look for my dependencies under /usr/share/gocode?
Edit: To be precise: Fedora 34 comes with Go 1.16 which 'just' changed the GO111MODULE default from auto to on. Thus, one still can restore the old behavior by setting GO111MODULE=auto.
However, Golang developers already announced that they want to drop support for GOPATH style projects in Go 1.17:
We plan to drop support for GOPATH mode in Go 1.17. In other words, Go 1.17 will ignore GO111MODULE. If you have projects that do not build in module-aware mode, now is the time to migrate.
Update 2023-02-19: As of Go 1.19.5 (Fedora 37), GO111MODULE=off still works for building GOPATH style (i.e. non-modularized) projects. Example:
export GOPATH=$HOME/go:/usr/share/gocode GOPROXY=off GO111MODULE=off
go build helloworld.go
Apparently, the Go team adjusted their GO11MODULE deprecation plan, without targeting any new future release for removal, yet.

Answer 1

You can explicity define using replace keyword in your generated mod file to refer to your local modules.
replace packagename => /usr/share/gcode

Answer 2

Ideally, distributions should package installed dependencies in a layout that is compatible with the GOPROXY protocol. Then you would be able to set GOPROXY appropriately and run go mod tidy to use the installed dependencies.
However, to my knowledge, no distributions actually provide a GOPROXY tree at this point. You may need a workaround.
The next best alternative is to wire up the replacements yourself using replace directives. go mod tidy already knows to prefer the versions found in replace directives over other versions, so it should suffice to do something like:
go mod init github.com/gsauthof/gonzofilter
go mod edit -replace go.etcd.io/bbolt=/usr/share/gocode/src/go.etcd.io/bbolt
go mod edit -replace golang.org/x/sys=/usr/share/gocode/src/golang.org/x/sys
go mod edit -replace golang.org/x/text=/usr/share/gocode/src/golang.org/x/text
go mod tidy
However, note that the replace directive requires the target to include an explicit go.mod file, so this will only work if either your dependencies already have explicit go.mod files or your distro packager has added them as a patch to support this use-case.
Note that with those replace directives in place, your build would not be reproducible or repeatable over time: if you change the distro-installed versions of your dependencies, then the meaning of go build will silently change to use those different (and possibly incompatible!) versions as well.
So I would recommend that you not do that, and instead use go get and/or go mod tidy to fetch the specific module versions you want from upstream or from a public module proxy (such as proxy.golang.org).
If you are concerned about upstream tampering, note that the Go project's official binary distribution of the go command — and the go command built from pristine source — by default automatically verifies checksums for downloaded modules against an auditable public database at https://sum.golang.org; however, the Fedora distribution of the go command disables use of the checksum database by default.

Answer 3

One way to migrate such a GOPATH project to a Go modules aware one is the following:
First, manually create the go.mod with the direct dependencies:
module github.com/gsauthof/gonzofilter
go 1.15
require (
go.etcd.io/bbolt v1.3.5
golang.org/x/sys v0.0.0-20210423185535-09eb48e85fd7
golang.org/x/text v0.3.5
)
I got the minimal versions from what's currently on my system:
$ rpm -q golang-etcd-bbolt-devel golang-x-sys-devel golang-x-text-devel golang-bin
golang-etcd-bbolt-devel-1.3.5-2.fc33.noarch
golang-x-sys-devel-0-0.39.20210123git9b0068b.fc33.noarch
golang-x-text-devel-0.3.5-1.fc33.noarch
golang-bin-1.15.8-1.fc33.x86_64
The problem now is that one can't simply tell the module-aware Go tools to look for those modules under the filesystem path /usr/share/gocode/src where they are all located.
There is the replace directive one can add to our go.mod file to set the lookup path for single dependencies, e.g. like this:
replace (
go.etcd.io/bbolt => /usr/share/gocode/src/go.etcd.io/bbolt
golang.org/x/sys => /usr/share/gocode/src/golang.org/x/sys
golang.org/x/text => /usr/share/gocode/src/golang.org/x/text
)
However, a go build still doesn't use indirect dependencies from /usr/share/gocode, e.g. the ones that are provided by the packages golang-x-text-devel depends on. In this example, go build does find /usr/share/gocode/src/golang.org/x/text/go.mod but that file doesn't include any replace directive.
Thus, this fails:
$ GOPROXY=off go build -mod=readonly
go: golang.org/x/text#v0.3.6 requires
golang.org/x/tools#v0.0.0-20180917221912-90fa682c2a6e: module lookup disabled
by GOPROXY=off
To resolve this we have to add replace directive lines also for all indirect dependencies.
Doing this manually is tedious and error prone, of course.
Thus, we can automate this with a small shell one-liner:
We start with this go.mod file:
module github.com/gsauthof/gonzofilter
go 1.15
require (
go.etcd.io/bbolt v1.3.5
golang.org/x/sys v0.0.0-20210423185535-09eb48e85fd7
golang.org/x/text v0.3.6
)
replace (
//replace-this
)
The following fixpoint iteration then adds dependencies until the build succeeds:
while true; do
GOPROXY=off go build -mod readonly 2> t
if [ $? -eq 1 ] && grep 'module lookup disabled\|missing go.sum entry' t >/dev/null; then
x=$(grep 'disabled\|missing' t | tr ' \t' '\n' | grep '#' | head -n1 | cut -d'#' -f1 )
echo "Adding $x"
sed -i "s#^\(//replace-this\)#\t$x => /usr/share/gocode/src/$x\n\1#" go.mod
continue
fi
break
done
Which results in the following require directive for this example project:
replace (
go.etcd.io/bbolt => /usr/share/gocode/src/go.etcd.io/bbolt
golang.org/x/sys => /usr/share/gocode/src/golang.org/x/sys
golang.org/x/text => /usr/share/gocode/src/golang.org/x/text
golang.org/x/tools => /usr/share/gocode/src/golang.org/x/tools
github.com/yuin/goldmark => /usr/share/gocode/src/github.com/yuin/goldmark
golang.org/x/mod => /usr/share/gocode/src/golang.org/x/mod
golang.org/x/net => /usr/share/gocode/src/golang.org/x/net
golang.org/x/sync => /usr/share/gocode/src/golang.org/x/sync
golang.org/x/xerrors => /usr/share/gocode/src/golang.org/x/xerrors
golang.org/x/crypto => /usr/share/gocode/src/golang.org/x/crypto
golang.org/x/term => /usr/share/gocode/src/golang.org/x/term
//replace-this
)

Answer 4

Another way to point Go to the system-wide available dependencies is to exploit Go's module vendoring support: That means by symbolically linking the module file paths, creating a minimal vendor/modules.txt and compiling with -mod vendor.
So with -mod vendor Go build doesn't try to download any required modules, instead it looks them up in vendor/ directory.
One might be tempted to just create a symbolic link that points from vendor/ to /usr/share/gocode/src (where all the distributions golang-...-devel packages are installed), but this doesn't work because -mod vendor also requires yet another modules description file, i.e. vendor/modules.txt.
So, besides some more specific symlinks, we have to create a proper modules.txt when using the vendoring feature:
First, to create all the sublevel symlinks:
awk -vvd=wendor 'BEGIN {
PROCINFO["sorted_in"]="#ind_str_asc";
system("mkdir -p "vd)
}
func push_mod(pkg) {
if (!seen[pkg]) {
ARGV[ARGC++]="/usr/share/gocode/src/"pkg"/go.mod";
seen[pkg]=1;
}
sub("/.+$", "", pkg);
xs[pkg]=1;
}
/^require[^(]+$/ {
push_mod($2);
next
}
/^require/ {
inb=1;
next
}
/^\)/ {
inb=0;
next
}
inb {
push_mod($1);
}
END {
for (i in xs) {
print i;
system("ln -sfn /usr/share/gocode/src/"i" "vd"/"i)
}
}' go.mod
Then, to create the vendor/modules.txt:
awk -vvd=wendor 'BEGIN {
fn=vd"/modules.txt"
}
/^require/ {
inb=1;
next
}
/^\)/ {
inb=0;
next
}
inb {
printf("# %s %s\n## explicit\n%s\n", $1, $2, $1) > fn
}' go.mod
After those commands, the vendor directory looks like this for our example project:
$ ls -l vendor
total 16
lrwxrwxrwx. 1 juser juser 32 2021-04-25 11:12 github.com -> /usr/share/gocode/src/github.com
lrwxrwxrwx. 1 juser juser 32 2021-04-25 11:12 go.etcd.io -> /usr/share/gocode/src/go.etcd.io
lrwxrwxrwx. 1 juser juser 32 2021-04-25 11:12 golang.org -> /usr/share/gocode/src/golang.org
-rw-r--r--. 1 juser juser 195 2021-04-25 11:13 modules.txt
$ cat vendor/modules.txt
# go.etcd.io/bbolt v1.3.5
## explicit
go.etcd.io/bbolt
# golang.org/x/sys v0.0.0-20210423185535-09eb48e85fd7
## explicit
golang.org/x/sys
# golang.org/x/text v0.3.5
## explicit
golang.org/x/text
Whereas go.mod is quite minimal, i.e. it doesn't contain any replace directive:
module github.com/gsauthof/gonzofilter
go 1.15
require (
go.etcd.io/bbolt v1.3.5
golang.org/x/sys v0.0.0-20210423185535-09eb48e85fd7
golang.org/x/text v0.3.5
)
And this is sufficient to let
GOPROXY=off go build -mod vendor
succeed.
Note that indirect dependencies such as golang.org/x/tools aren't listed in the modules.txt, however, they are still available via vendor/ because we transitively traversed all go.mod files starting with the required modules when creating symlinks to the top level directories.
The discrete charme of this scheme is that one doesn't have to mess with replace directives in the go.mod file. That means go.mod can be kept quite generic, one basically just have to add the -mod vendor switch to the build command.