Following the release of Go 1.11, I have been trying to move my repositories to Go modules, by adding a go.mod file at their root.
One of my root libraries my.host/root is in its version 17.0.1, so I wrote in its go.mod file:
module my.host/root/v17
I tagged that version v17.0.1 as documented in the Go modules manual.
When I try to make a new Go project that uses my root library, like:
package main
import root "my.host/root/v17"
func main() {
root.DoSomething()
}
And try to compile it, I get the following error:
go: my.host/root#v0.0.0-20180828034419-6bc78016491a: go.mod has post-v0 module path "my.host/root/v17" at revision 6bc78016491a
I am at loss figuring out why this happens. I explicitly added v17.0.1 in the go.mod file, yet every attempt at go build replaces the entry with a v0.0.0-20180828034419-6bc78016491a version which then fails because at that commit, the go.mod file module entry of my root library indeed ends with a v17, as it should.
For the record, this commit is the same as the tagged v17.0.1 version.
What am I doing wrong here? How can I debug this situation?
I had make two mistakes:
My initial v17.0.0 tag would point to a commit where go.mod did not contain the v17 import path suffix. As a result, it seems Go tooling considers the whole v17 major version as a v0/v1 instead, even if later v17 tags point to a commit with a correct go.mod directive, hence the commit ID "translation".
In my dependent projects, in the go.mod file, I mistakenly specified
require my.host/root v17.0.1 instead of
require my.host/root/v17 v17.0.1.
After fixing both those issues, everything seems back to normal and it works perfectly. I wish the documentation had been clearer about this but I guess this is a good opportunity to make a contribution!
The error I got was: github.com/emicklei/go-restful#v0.0.0-20180531035034-3658237ded10: go.mod has post-v0 module path "github.com/emicklei/go-restful/v2" at revision 3658237ded10
Appending github.com/emicklei/go-restful with v2 like so: github.com/emicklei/go-restful/v2 in my go.mod file fixed it for me.
Related
I ran into a repo that seems to be a Go module, but there's no go.mod file in it: github.com/confluentinc/confluent-kafka-go.
Is it ok for a go module to have no go.mod file with dependencies, or the authors of that library just didn't migrate to modules yet?
Dependency modules do not need to have explicit go.mod files.
The “main module” in module mode — that is, the module containing the working directory for the go command — must have a go.mod file, so that the go command can figure out the import paths for the packages within that module (based on its module path), and so that it has a place to record its dependencies once resolved.
In addition, any modules slotted in using replace directives must have go.mod files (in order to reduce confusion due to typos or other errors in replacement paths).
However, in general a module that lacks an explicit go.mod file is valid and fine to use. Its effective module path is the path by which it was required, which can be a bit confusing if the same repository ends up in use via multiple paths. Since a module with no go.mod file necessarily doesn't specify its own dependencies, consumers of that module will have to fill in those dependencies themselves (go mod tidy will mark them as // indirect in the consumer's go.mod file).
SHORT SUMMARY OF THE DISCUSSION:
The answer is "No"!
This project contains a set of go packages, but it is not a Go module as it doesn't contain go.mod file (although, it used to be a multi-module repo (Go) previously).
go get can run in both ways: module-aware mode and legacy GOPATH mode (as of Go 1.16).
To read more about this, refer to docs by using the go command:
$ go help gopath-get
and
$ go help module-get
It'd tell about how go get works in both cases.
Also, I noticed that it can download any repository and would treat it as a Go package, even if it contains an arbitrary Python project.
I did a simple test to demonstrate the same:
$ go get github.com/mongoengine/mongoengine
And it surprisingly worked.
Modules are defined by their go.mod file. Without a go.mod file, it is not a module.
See this from the Go Modules Reference
A module is a collection of packages that are released, versioned, and distributed together. Modules may be downloaded directly from version control repositories or from module proxy servers.
A module is identified by a module path, which is declared in a go.mod file, together with information about the module's dependencies. The module root directory is the directory that contains the go.mod file.
And
A module is defined by a UTF-8 encoded text file named go.mod in its root directory.
I have my projects that have many packages which import each other and import outside packages. When I make a change to one of my low lever packages, and then push it to git it is fine and works in that section. When I go get it for use in another project that was working perfectly I now get this go get this error:
module declares its path as: github.com/xdg-go/scram
but was required as: github.com/xdg/scram
None of my code uses either of those directly. It looks like it automatically updated some lower external packages and broke things the used to then old import.
How do I either find out the package that is importing the wrong name or stop all auto-updates?
The go.mod file at github.com/xdg/scram declares itself as github.com/xdg-go/scram:
module github.com/xdg-go/scram
go 1.11
require (
github.com/xdg-go/stringprep v1.0.2
golang.org/x/crypto v0.0.0-20210322153248-0c34fe9e7dc2
)
The go.mod file should be updated to reflect the correct import path.
Unfortunately if this module is for you an indirect dependency, the best fix possible is to update whatever project you import that is directly importing it.
When that is not an option, a solution to this error is to clone the problematic repository locally and use the replace directive in your go.mod file:
module mymodule
replace github.com/xdg/stringprep => ../strprep
go 1.16.2
require (
github.com/divjotarora/mgo v0.0.0-20190308170442-1d451d2a3149
)
where ../strprep is where the code of the required module exists in your local machine, relative to the go.mod file of your project.
The downside of this of course is that you have to replicate this palliative fix wherever you plan to go get your modules.
Note also:
divjotarora/mgo is just a random example of a project that imports one of those packages using their old import path.
I'm using xdg/stringprep as an example because I can't find modules that import xdg/scram instead, but apparently it suffers from the same issue
Beside, you can use:
go mod why <package> to find out why a certain package is listed as a dependency of your project
go mod graph to show the full dependency graph. The output is in <package> <requirement> format
When I run go mod tidy, it breaks because a package imported by my project imports another package using path github.com/coreos/bbolt, but when it fetches the package from this path its go.mod says its path is go.etcd.io/bbolt.
The problem is that both the importing package and the imported package are 3rd party packages. I know I could edit the go module cache to fix it, but it would be a real hell fixing it when new versions of these packages become available.
Partial echoed messages are shown below:
github.com/coreos/etcd/client tested by
github.com/coreos/etcd/client.test imports
github.com/coreos/etcd/integration imports
github.com/coreos/etcd/etcdserver imports
github.com/coreos/etcd/mvcc/backend imports
github.com/coreos/bbolt: github.com/coreos/bbolt#v1.3.5: parsing go.mod:
module declares its path as: go.etcd.io/bbolt
but was required as: github.com/coreos/bbolt
So, how can I fix or work around this situation?
You can fix this solution by using the replace directive
Simply add:
replace github.com/coreos/bbolt v1.3.5 => go.etcd.io/bbolt v1.3.5
at the end of your go.mod file
The mismatched path implies that your dependency (github.com/coreos/etcd/mvcc/backend) is written against an old version of the bbolt repository — one that predates commit e65d4d.
I notice that the current go.mod file in the github.com/etcd-io/etcd repo specifies its module path as go.etcd.io/etcd/v3.
So the most robust fix for you is probably to update to that path, which you can do by changing your import statements to refer to the new canonical import path and running go mod tidy to update your dependencies accordingly:
sed -i s,github.com/coreos/etcd,go.etcd.io/etcd/v3,g $(find . -name '*.go')
go mod tidy
Barring that, you could explicitly choose a version of github.com/coreos/bbolt that matches the older import path. I notice that the highest version for that module listed at https://beta.pkg.go.dev/github.com/etcd-io/bbolt?tab=versions is v1.3.3, and v1.3.4 does seem to add a go.mod file with the updated path. So as a fallback, you could try:
go get -d github.com/coreos/bbolt#v1.3.3
The downside to that approach is that v1.3.3 is the end of the line: you wouldn't be able to pull in bug-fixes after that point, because those fixes are all at the go.etcd.io path.
You could also use the replace directive with the command line, as example:
go mod edit -replace github.com/pselle/bar=/Users/pselle/Projects/bar
More info in this article
What I essentially have is a mono-repo, which doesn't have a go.mod at the root level. There are multiple directories inside this mono-repo, with each of them having their own go.mod files. I'll refer to them as sub-modules.
Now, I've figured out a way to be able to access the sub-modules independently (versioned) in a completely different codebase. The issue I'm facing now is, to dis-allow the import of the entire mono-repo, using:
go get link.to/mono-repo#commit_id ------> A
and only allow import using :
go get link.to/mono-repo/sub_mod1#v0.x.y
go get link.to/mono-repo/sub_mod2#v0.y.z
The command A is able to fetch the entire repo, and then can be used to access the inner modules. Is there any way to stop this from being allowed?
I tried a few things, like:
Added non-compilable code in a file noCompile.go at the root level. On go get..., the compilation error is printed, but use of the inner modules still works fine.
Added an init() function in the same noCompile.go file, which just calls panic(). This init function is not being executed, as the root directory is never directly accessed, only the inner modules are.
Is there ANY way to achieve what I'm intending to?
Any directory that contains its own go.mod file is excluded from the module in the parent directory. So if you go get link.to/mono-repo#commit_id, that should not contain the packages link.to/mono-repo/sub_mod1 or link.to/mono-repo/sub_mod2 (assuming that they exist and have their own go.mod files).
I suspect that you were observing imports like import "link.to/mono-repo/sub_mod2/some/package/here" to resolve not because of that initial go get, but rather because the go command was resolving the missing imports (and adding the missing dependencies) automatically; see https://golang.org/ref/mod#go-mod-file-updates.
As of Go 1.16 (released today!), most go commands no longer implicitly modify the go.mod file, so the fact that the module at the repo root does not include the contents of nested modules should hopefully be clearer.
I am now using golang 1.13 and use go module.
However, when I import a package (for example, a) that is not opted in go module, in go.sum file there is still two lines. Go module tells us that "Each known module version results in two lines in the go.sum file. The first line gives the hash of the module version's file tree. The second line appends "/go.mod" to the version and gives the hash of only the module version's (possibly synthesized) go.mod file. The go.mod-only hash allows downloading and authenticating a module version's go.mod file, which is needed to compute the dependency graph, without also downloading all the module's source code."
(https://tip.golang.org/cmd/go/#hdr-Module_downloading_and_verification).
But this package is not a module, and so it does not have a go.mod file? For example, if I import package call "github.com/example/a" that is not a module, in go.sum file, it still has these two lines:
github.com/example/a v0.0.0-20190627063042-31896c4e4162 h1:rSqi2vQEpS+GAFKrLvmxzWW3OGlLI4hANnEf/ib/ofo=
github.com/example/a v0.0.0-20190627063042-31896c4e4162/go.mod h1:tcpxll8wcruwpPpWBbjAsWc1JbLHld/v9F+3rgLIr4c=
My question is, how the second line generated?
go.sum contains the expected cryptographic checksums
of the content of specific module versions. Each time a dependency is
used, its checksum is added to go.sum if missing or else required to match
the existing entry in go.sum.
Every package/module is dependency and every dependency mean to be maintained with checksums in go.sum so whether it is package or module it would be maintained.
The sources would be downloaded in $GOPATH/src directory accordingly.
TRY -
Cause in into the go.sum file are written EVERY dependencies sum hash. The one related to your go.mod file, and the one imported from the modules that you have imported. Try run go mod tidy in order to reduce the imported modules, your go.mod file will contains some //indirect import, that are the one that your imported modules use internally.
Maybe the Golang source code can explain the reason:
func (r *codeRepo) legacyGoMod(rev, dir string) []byte {
// We used to try to build a go.mod reflecting pre-existing
// package management metadata files, but the conversion
// was inherently imperfect (because those files don't have
// exactly the same semantics as go.mod) and, when done
// for dependencies in the middle of a build, impossible to
// correct. So we stopped.
// Return a fake go.mod that simply declares the module path.
return []byte(fmt.Sprintf("module %s\n", modfile.AutoQuote(r.modPath)))
}
code from here: https://github.com/golang/go/blob/acac535c3ca571beeb168c953d6d672f61387ef1/src/cmd/go/internal/modfetch/coderepo.go#L857
↓ VSCode open /usr/local/go/src:
locate to cmd/go/internal/modfetch/coderepo.go, add a breakpoint to func legacyGoMod
locate to cmd/go/internal/modfetch/coderepo_test.go, press F5
wait for a while, stop at the breakpoint