I am given a repository hosted on a private site, and it looks like this:
a-private-git.com/myrepo/
go/
go.mod
go.sum
bar.go
The repository is located at a-private-git.com/myrepo and the module file is: (Note that the module itself is actually located at a-private-git.com/myrepo/go)
module a-private-git.com/myrepo
Is it possible to include a specific commit/branch of this repository? Or do I need the provider of the repo to change it?
I've tried:
go get a-private-git.com/myrepo#v0.0.0-<commit time>-<commit hash>
> the module is added to go.mod, but when I build I get:
> no required module provides package a-private-git.com/myrepo; to add it:
> go get a-private-git.com/myrepo
go get a-private-git.com/myrepo/go#v0.0.0-<commit time>-<commit hash>
> module declares its path as: a-private-git.com/myrepo
> but was required as: a-private-git.com/myrepo/go
(Based on https://go.dev/doc/modules/managing-source)
go get a-private-git.com/myrepo/go#go/v0.0.0-<commit time>-<commit hash>
> invalid version: unknown revision go/v0.0.0-<commit time>-<commit hash>
The easieset way would be to simply 'go get' the commit hash you want:
go get a-private-git.com/myrepo#commit_hash
Alternatively you can use a branch name to get its latest commit:
go get a-private-git.com/myrepo#branch_name
Reviewing numerous articles, running into this issue:
Created a private package, 'mypackage' in my git account
In git, I have MyCompany/mypackage
In another project, I want to import the package:
...
import (
"github.com/MyCompany/mypackage'
)
When I run:
go mod tidy
I get the following error:
github.com/MyCompany/mypackage: module github.com/MyCompany/mypackage#latest found (v0.0.0-20220126203606-a88fea44b771), but does not contain package github.com/MyCompany/mypackage
Not sure how to resolve this.
In my .gitconfig I have:
[url "ssh://git#github.com/"]
insteadOf = https://github.com/
go env
shows these settings:
GOPRIVATE="github.com/MyCompany/*"
GONOPROXY="github.com/MyCompany/*"
GONOSUMDB="github.com/MyCompany/*"
What other configurations am I missing?
The go.mod file looks like:
module github.com/MyCompany/mypackage
go 1.17
require (
github.com/go-redis/redis/v8 v8.11.4
github.com/pkg/errors v0.9.1
)
require (
github.com/cespare/xxhash/v2 v2.1.2 // indirect
github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f // indirect
)
Added a semver tag (git tag v1.0.0, git push --tags) then was able to download
I am creating a new private module which will ultimately be stored in bitbucket.
The structure of this module is the following:
package1Directory
package2Directory
package3Directory
go.mod
go.sum
The go.mod file look like this:
module newModule
go 1.17
require (
//..,, bunch of external modules
)
As still in early stages of dev, I do not wish to push it to bitbucket yet. But I still want to use it in other local project for testing purpose.
On this local project I have the following code:
On my different go files:
import ("bitbucket.org/myRepo/myModules/newModule")
//Do something with newModule
On my go.mod file:
require (
//Commented until actually pushed and versioned
//bitbucket.org/myRepo/myModules/newModule
...
)
//Specify to use local version instead of remote version
replace bitbucket.org/myRepo/myModules/newModule => ../myModules/newModule/
However this does not work as I hoped.
When running the command go mod tidy, I am getting the following error:
bitbucket.org/myRepo/myModules/newModule: module bitbucket.org/myRepo/myModules/newModule#latest found (v0.0.0-00010101000000-000000000000, replaced by ../myModules/newModule/), but does not contain package bitbucket.org/myRepo/myModules/newModule
I am not sure what I am doing wrong, if someone can direct me on the right step, it would be much appreciated !
As prompted by the error message
... but does not contain package bitbucket.org/myRepo/myModules/newModule
You should edit you go.mod of module bitbucket.org/myRepo/myModules/newModule as (notice the first line of declaration of the module):
module bitbucket.org/myRepo/myModules/newModule
go 1.17
require (
//..,, bunch of external modules
)
And you must add version tag in format like v1.2.3, in require and replace clauses, in the go.mod of the other modules importing your newModule:
require bitbucket.org/myRepo/myModules/newModule v0.0.0
replace bitbucket.org/myRepo/myModules/newModule v0.0.0 => ../myModules/newModule/
Refer to https://go.dev/doc/modules/managing-dependencies#local_directory.
I have a GOPATH based project that I currently build like this on Fedora:
sudo dnf install golang-etcd-bbolt-devel golang-x-sys-devel golang-x-text-devel
GOPATH=$HOME/go:/usr/share/gocode go build
My project (gonzofilter) implements a command line utility and thus source files are located in the main package (i.e. they have a package main declaration).
With Fedora 34 and beyond it seems that Go removed support for building GOPATH style projects and one really has to use Go modules:
go build
go: cannot find main module; see 'go help modules'
That Go blog post kind of covers my case (-> 'Without a dependency manager'), but it doesn't explicitly mention how to deal with main package projects or with distribution provided dependencies.
So, how do I migrate such a project?
How do I have to tell Go/go mod tidy to look for my dependencies under /usr/share/gocode?
Edit: To be precise: Fedora 34 comes with Go 1.16 which 'just' changed the GO111MODULE default from auto to on. Thus, one still can restore the old behavior by setting GO111MODULE=auto.
However, Golang developers already announced that they want to drop support for GOPATH style projects in Go 1.17:
We plan to drop support for GOPATH mode in Go 1.17. In other words, Go 1.17 will ignore GO111MODULE. If you have projects that do not build in module-aware mode, now is the time to migrate.
Update 2023-02-19: As of Go 1.19.5 (Fedora 37), GO111MODULE=off still works for building GOPATH style (i.e. non-modularized) projects. Example:
export GOPATH=$HOME/go:/usr/share/gocode GOPROXY=off GO111MODULE=off
go build helloworld.go
Apparently, the Go team adjusted their GO11MODULE deprecation plan, without targeting any new future release for removal, yet.
You can explicity define using replace keyword in your generated mod file to refer to your local modules.
replace packagename => /usr/share/gcode
Ideally, distributions should package installed dependencies in a layout that is compatible with the GOPROXY protocol. Then you would be able to set GOPROXY appropriately and run go mod tidy to use the installed dependencies.
However, to my knowledge, no distributions actually provide a GOPROXY tree at this point. You may need a workaround.
The next best alternative is to wire up the replacements yourself using replace directives. go mod tidy already knows to prefer the versions found in replace directives over other versions, so it should suffice to do something like:
go mod init github.com/gsauthof/gonzofilter
go mod edit -replace go.etcd.io/bbolt=/usr/share/gocode/src/go.etcd.io/bbolt
go mod edit -replace golang.org/x/sys=/usr/share/gocode/src/golang.org/x/sys
go mod edit -replace golang.org/x/text=/usr/share/gocode/src/golang.org/x/text
go mod tidy
However, note that the replace directive requires the target to include an explicit go.mod file, so this will only work if either your dependencies already have explicit go.mod files or your distro packager has added them as a patch to support this use-case.
Note that with those replace directives in place, your build would not be reproducible or repeatable over time: if you change the distro-installed versions of your dependencies, then the meaning of go build will silently change to use those different (and possibly incompatible!) versions as well.
So I would recommend that you not do that, and instead use go get and/or go mod tidy to fetch the specific module versions you want from upstream or from a public module proxy (such as proxy.golang.org).
If you are concerned about upstream tampering, note that the Go project's official binary distribution of the go command — and the go command built from pristine source — by default automatically verifies checksums for downloaded modules against an auditable public database at https://sum.golang.org; however, the Fedora distribution of the go command disables use of the checksum database by default.
One way to migrate such a GOPATH project to a Go modules aware one is the following:
First, manually create the go.mod with the direct dependencies:
module github.com/gsauthof/gonzofilter
go 1.15
require (
go.etcd.io/bbolt v1.3.5
golang.org/x/sys v0.0.0-20210423185535-09eb48e85fd7
golang.org/x/text v0.3.5
)
I got the minimal versions from what's currently on my system:
$ rpm -q golang-etcd-bbolt-devel golang-x-sys-devel golang-x-text-devel golang-bin
golang-etcd-bbolt-devel-1.3.5-2.fc33.noarch
golang-x-sys-devel-0-0.39.20210123git9b0068b.fc33.noarch
golang-x-text-devel-0.3.5-1.fc33.noarch
golang-bin-1.15.8-1.fc33.x86_64
The problem now is that one can't simply tell the module-aware Go tools to look for those modules under the filesystem path /usr/share/gocode/src where they are all located.
There is the replace directive one can add to our go.mod file to set the lookup path for single dependencies, e.g. like this:
replace (
go.etcd.io/bbolt => /usr/share/gocode/src/go.etcd.io/bbolt
golang.org/x/sys => /usr/share/gocode/src/golang.org/x/sys
golang.org/x/text => /usr/share/gocode/src/golang.org/x/text
)
However, a go build still doesn't use indirect dependencies from /usr/share/gocode, e.g. the ones that are provided by the packages golang-x-text-devel depends on. In this example, go build does find /usr/share/gocode/src/golang.org/x/text/go.mod but that file doesn't include any replace directive.
Thus, this fails:
$ GOPROXY=off go build -mod=readonly
go: golang.org/x/text#v0.3.6 requires
golang.org/x/tools#v0.0.0-20180917221912-90fa682c2a6e: module lookup disabled
by GOPROXY=off
To resolve this we have to add replace directive lines also for all indirect dependencies.
Doing this manually is tedious and error prone, of course.
Thus, we can automate this with a small shell one-liner:
We start with this go.mod file:
module github.com/gsauthof/gonzofilter
go 1.15
require (
go.etcd.io/bbolt v1.3.5
golang.org/x/sys v0.0.0-20210423185535-09eb48e85fd7
golang.org/x/text v0.3.6
)
replace (
//replace-this
)
The following fixpoint iteration then adds dependencies until the build succeeds:
while true; do
GOPROXY=off go build -mod readonly 2> t
if [ $? -eq 1 ] && grep 'module lookup disabled\|missing go.sum entry' t >/dev/null; then
x=$(grep 'disabled\|missing' t | tr ' \t' '\n' | grep '#' | head -n1 | cut -d'#' -f1 )
echo "Adding $x"
sed -i "s#^\(//replace-this\)#\t$x => /usr/share/gocode/src/$x\n\1#" go.mod
continue
fi
break
done
Which results in the following require directive for this example project:
replace (
go.etcd.io/bbolt => /usr/share/gocode/src/go.etcd.io/bbolt
golang.org/x/sys => /usr/share/gocode/src/golang.org/x/sys
golang.org/x/text => /usr/share/gocode/src/golang.org/x/text
golang.org/x/tools => /usr/share/gocode/src/golang.org/x/tools
github.com/yuin/goldmark => /usr/share/gocode/src/github.com/yuin/goldmark
golang.org/x/mod => /usr/share/gocode/src/golang.org/x/mod
golang.org/x/net => /usr/share/gocode/src/golang.org/x/net
golang.org/x/sync => /usr/share/gocode/src/golang.org/x/sync
golang.org/x/xerrors => /usr/share/gocode/src/golang.org/x/xerrors
golang.org/x/crypto => /usr/share/gocode/src/golang.org/x/crypto
golang.org/x/term => /usr/share/gocode/src/golang.org/x/term
//replace-this
)
Another way to point Go to the system-wide available dependencies is to exploit Go's module vendoring support: That means by symbolically linking the module file paths, creating a minimal vendor/modules.txt and compiling with -mod vendor.
So with -mod vendor Go build doesn't try to download any required modules, instead it looks them up in vendor/ directory.
One might be tempted to just create a symbolic link that points from vendor/ to /usr/share/gocode/src (where all the distributions golang-...-devel packages are installed), but this doesn't work because -mod vendor also requires yet another modules description file, i.e. vendor/modules.txt.
So, besides some more specific symlinks, we have to create a proper modules.txt when using the vendoring feature:
First, to create all the sublevel symlinks:
awk -vvd=wendor 'BEGIN {
PROCINFO["sorted_in"]="#ind_str_asc";
system("mkdir -p "vd)
}
func push_mod(pkg) {
if (!seen[pkg]) {
ARGV[ARGC++]="/usr/share/gocode/src/"pkg"/go.mod";
seen[pkg]=1;
}
sub("/.+$", "", pkg);
xs[pkg]=1;
}
/^require[^(]+$/ {
push_mod($2);
next
}
/^require/ {
inb=1;
next
}
/^\)/ {
inb=0;
next
}
inb {
push_mod($1);
}
END {
for (i in xs) {
print i;
system("ln -sfn /usr/share/gocode/src/"i" "vd"/"i)
}
}' go.mod
Then, to create the vendor/modules.txt:
awk -vvd=wendor 'BEGIN {
fn=vd"/modules.txt"
}
/^require/ {
inb=1;
next
}
/^\)/ {
inb=0;
next
}
inb {
printf("# %s %s\n## explicit\n%s\n", $1, $2, $1) > fn
}' go.mod
After those commands, the vendor directory looks like this for our example project:
$ ls -l vendor
total 16
lrwxrwxrwx. 1 juser juser 32 2021-04-25 11:12 github.com -> /usr/share/gocode/src/github.com
lrwxrwxrwx. 1 juser juser 32 2021-04-25 11:12 go.etcd.io -> /usr/share/gocode/src/go.etcd.io
lrwxrwxrwx. 1 juser juser 32 2021-04-25 11:12 golang.org -> /usr/share/gocode/src/golang.org
-rw-r--r--. 1 juser juser 195 2021-04-25 11:13 modules.txt
$ cat vendor/modules.txt
# go.etcd.io/bbolt v1.3.5
## explicit
go.etcd.io/bbolt
# golang.org/x/sys v0.0.0-20210423185535-09eb48e85fd7
## explicit
golang.org/x/sys
# golang.org/x/text v0.3.5
## explicit
golang.org/x/text
Whereas go.mod is quite minimal, i.e. it doesn't contain any replace directive:
module github.com/gsauthof/gonzofilter
go 1.15
require (
go.etcd.io/bbolt v1.3.5
golang.org/x/sys v0.0.0-20210423185535-09eb48e85fd7
golang.org/x/text v0.3.5
)
And this is sufficient to let
GOPROXY=off go build -mod vendor
succeed.
Note that indirect dependencies such as golang.org/x/tools aren't listed in the modules.txt, however, they are still available via vendor/ because we transitively traversed all go.mod files starting with the required modules when creating symlinks to the top level directories.
The discrete charme of this scheme is that one doesn't have to mess with replace directives in the go.mod file. That means go.mod can be kept quite generic, one basically just have to add the -mod vendor switch to the build command.
I have local 2 packages that I need to import, that are at different levels within root a
Note: a the root folder name and the file name - a/a.go
src structure is:
--src
--a/
a.go
go.mod
--b/
b.go
go.mod
I need to import package b into package a
github.com/x/y/src/a/a.go
github.com/x/y/src/a/b/b.go
a is a package that uses other packages that i've written, like custom logging tools etc
a.go:
package A
import (
b "github.com/x/y/src/a/b"
When I run:
go mod tidy
I get error:
go: github.com/x/y/src/a/a#v0.0.0 requires
github.com/x/y/src/a/b#v0.0.0: reading github.com/x/y/src/a/b/src/a/b/go.mod
at revision src/a/b/v0.0.0: unknown revision src/a/b/v0.0.0
Why is src/a/b being duplicated in the path it's reading from?!?!?!
Here's a's go.mod file, it's at the root directory and I need to import packages in folders one level down, so b is ./b/b.go and a is ./a.go :
module A
go 1.13
replace (
github.com/x/y/src/a/b => ./b // look inside folder b for b.go from path where a is located
)
require (
github.com/x/y/src/a/b v0.0.0
)
b's go.mod:
module a/b
go 1.13
require golang.org/x/sys v0.0.0-20210124154548-22da62e12c0c
Can someone tell be why go mod tidy is duplicating the path to b:
github.com/x/y/src/a/b/src/a/b/go.mod
It should be:
github.com/x/y/src/a/b/go.mod