I'm trying to compare the behavior of go mod tidy (and the resulting content of go.sum) to the output of go list -m all.
Reading the docs, I understand go.sum contains the whole list of dependent modules declared in go.mod and in dependencies' go.mod files, go list -m all shows the modules really loaded during the execution.
As an example, an application including logrus and prometheus like this:
go.mod
module mytest
go 1.14
require (
github.com/prometheus/common v0.4.0
github.com/sirupsen/logrus v1.8.1
)
main.go
package main
import "github.com/sirupsen/logrus"
import "github.com/prometheus/common/version"
func main() {
logrus.Info("Hello World")
logrus.Infof("Prometheus info: %v", version.Info())
}
After go mod tidy, go.sum shows both logrus v1.8.1, requested by the go.mod, and 1.2.0, dependency of prometheus v0.4.0; go list -m all shows only v1.8.1.
go.sum
[...]
github.com/sirupsen/logrus v1.2.0/go.mod h1:LxeOpSwHxABJmUn/MG1IvRgCAasNZTLOkJPxbbu5VWo=
github.com/sirupsen/logrus v1.8.1 h1:dJKuHgqk1NNQlqoA6BTlM1Wf9DOH3NBjQyu0h9+AZZE=
github.com/sirupsen/logrus v1.8.1/go.mod h1:yWOB1SBYBC5VeMP7gHvWumXLIWorT60ONWic61uBYv0=
[...]
output of go list
[...]
github.com/sirupsen/logrus v1.8.1
[...]
Is it correct to say the modules really used by the application are listed by go list -m all?
The underlying problem is that a static code analysis detects insecure module versions listed in go.sum, but actually these versions don't show up in go list -m all, hence they shouldn't be really used by the application, but only downloaded during build phase to select the proper minimal version.
Some reference:
https://go.dev/ref/mod#go-mod-tidy
go mod tidy acts as if all build tags are enabled, so it will consider
platform-specific source files and files that require custom build
tags, even if those source files wouldn’t normally be built.
https://go.dev/ref/mod#go-sum-files
The go.sum file may contain hashes for multiple versions of a module.
The go command may need to load go.mod files from multiple versions of
a dependency in order to perform minimal version selection. go.sum may
also contain hashes for module versions that aren’t needed anymore
(for example, after an upgrade).
https://github.com/golang/go/wiki/Modules#is-gosum-a-lock-file-why-does-gosum-include-information-for-module-versions-i-am-no-longer-using
[...]In addition, your module's go.sum records checksums for all
direct and indirect dependencies used in a build (and hence your
go.sum will frequently have more modules listed than your go.mod).
https://github.com/golang/go/wiki/Modules#version-selection
The minimal version selection algorithm is used to select the versions
of all modules used in a build. For each module in a build, the
version selected by minimal version selection is always the
semantically highest of the versions explicitly listed by a require
directive in the main module or one of its dependencies.
As an example, if your module depends on module A which has a
require D v1.0.0, and your module also depends on module B which
has a require D v1.1.1, then minimal version selection would choose
v1.1.1 of D to include in the build (given it is the highest listed
require version). [...] To see a list of the selected module versions
(including indirect dependencies), use go list -m all.
Yes, it correct to say the modules really "used" by the application are listed by go list -m all (as per documentation you provided the link of). By "used", it means the package selected at build time for the compilation of the go code of your application.
We had a similar issue with a static analysis tool and we had to change the configuration to use the output of go list -m all (dumped in a file) instead of go.sum.
Related
I've used openapi-generator to generate multiple SDK separately and put it in the same parent dir:
sdks
--- bar-api
------ go.mod
--- foo-api
------ go.mod
I'd like to be able to import bar-api in my client code. Initially there's bar-api's go.mod was generated as:
module github.com/coolrepo/bar-api
go 1.13
but I couldn't fix the imports in my client code:
bar "github.com/cool-repo/bar-api"
so what I did was:
sdks
--- bar-api
------ go.mod
---go.mod
i.e., created sdks/go.mod:
module github.com/coolrepo
go 1.13
and manually edited bar-api's go.mod to:
module github.com/coolrepo/bar-api
require (
...
)
replace github.com/coolrepo => ./..
go 1.15
Is there a better way to fix it? This one seems to work but looks kinda hacky with this replace.
The “better way to fix it” is to put all of the generated Go packages inside a single module, instead of splitting up the repo into multiple modules.
rm bar-api/go.mod
go mod tidy
You only really need to split a repo into separate modules if you need to be able to tag a release for one set of packages independently from another set of packages in the same repo. For generated service APIs, that probably isn't worth the development overhead of juggling versions for multiple modules.
I ran into a repo that seems to be a Go module, but there's no go.mod file in it: github.com/confluentinc/confluent-kafka-go.
Is it ok for a go module to have no go.mod file with dependencies, or the authors of that library just didn't migrate to modules yet?
Dependency modules do not need to have explicit go.mod files.
The “main module” in module mode — that is, the module containing the working directory for the go command — must have a go.mod file, so that the go command can figure out the import paths for the packages within that module (based on its module path), and so that it has a place to record its dependencies once resolved.
In addition, any modules slotted in using replace directives must have go.mod files (in order to reduce confusion due to typos or other errors in replacement paths).
However, in general a module that lacks an explicit go.mod file is valid and fine to use. Its effective module path is the path by which it was required, which can be a bit confusing if the same repository ends up in use via multiple paths. Since a module with no go.mod file necessarily doesn't specify its own dependencies, consumers of that module will have to fill in those dependencies themselves (go mod tidy will mark them as // indirect in the consumer's go.mod file).
SHORT SUMMARY OF THE DISCUSSION:
The answer is "No"!
This project contains a set of go packages, but it is not a Go module as it doesn't contain go.mod file (although, it used to be a multi-module repo (Go) previously).
go get can run in both ways: module-aware mode and legacy GOPATH mode (as of Go 1.16).
To read more about this, refer to docs by using the go command:
$ go help gopath-get
and
$ go help module-get
It'd tell about how go get works in both cases.
Also, I noticed that it can download any repository and would treat it as a Go package, even if it contains an arbitrary Python project.
I did a simple test to demonstrate the same:
$ go get github.com/mongoengine/mongoengine
And it surprisingly worked.
Modules are defined by their go.mod file. Without a go.mod file, it is not a module.
See this from the Go Modules Reference
A module is a collection of packages that are released, versioned, and distributed together. Modules may be downloaded directly from version control repositories or from module proxy servers.
A module is identified by a module path, which is declared in a go.mod file, together with information about the module's dependencies. The module root directory is the directory that contains the go.mod file.
And
A module is defined by a UTF-8 encoded text file named go.mod in its root directory.
I just inherited a Go project that has a go.mod file missing a declared dependency, but the dependency is in the the go.sum file:
...
cloud.google.com/go/storage v?.?.? <- this is the missing entry in go.mod
...
these are the entries in go.sum file:
...
cloud.google.com/go/storage v1.0.0/go.mod h1:<some hash>
cloud.google.com/go/storage v1.5.0/go.mod h1:<some hash>
cloud.google.com/go/storage v1.6.0/go.mod h1:<some hash>
cloud.google.com/go/storage v1.8.0/go.mod h1:<some hash>
cloud.google.com/go/storage v1.10.0 h1:<some hash>
cloud.google.com/go/storage v1.10.0/go.mod h1:<some hash>
...
My questions are:
Why are there 5 versions in the go.sum file?
If there are other libraries that depend on these specific versions do all 5 get compiled into the binary?
Which version of the lib will be linked to my application code since the dependency is not declared?
I tried to find an explanation in the Go documentation but could not locate, any help appreciated.
These dependencies are in all likelihood transitive dependencies, that is dependencies of the packages you depend on (or those that they depend on, etc). The go.sum contains lines for all dependencies of your module, direct or otherwise, in order for builds to be reproducible.
From the Go blog:
In addition to go.mod, the go command maintains a file named go.sum containing the expected cryptographic hashes of the content of specific module versions
...
The go command uses the go.sum file to ensure that future downloads of these modules retrieve the same bits as the first download, to ensure the modules your project depends on do not change unexpectedly, whether for malicious, accidental, or other reasons. Both go.mod and go.sum should be checked into version control.
The version of the package that does get included depends on the go.mod file of the package that you depend on. You may depend on several packages and each may depend on a different version of the dependency.
Whether they end up in your build depends on whether the dependency that includes them is compiled into your binary. An example where that inclusion may not happen are test files/packages, which will usually depend on testing libraries and their dependencies. These are never included in your average go build executable.
You can check the list of packages that will be included in your build like this:
go list -m all
You should just in case run go mod tidy to remove any dependencies that aren't actually needed anymore.
I am now using golang 1.13 and use go module.
However, when I import a package (for example, a) that is not opted in go module, in go.sum file there is still two lines. Go module tells us that "Each known module version results in two lines in the go.sum file. The first line gives the hash of the module version's file tree. The second line appends "/go.mod" to the version and gives the hash of only the module version's (possibly synthesized) go.mod file. The go.mod-only hash allows downloading and authenticating a module version's go.mod file, which is needed to compute the dependency graph, without also downloading all the module's source code."
(https://tip.golang.org/cmd/go/#hdr-Module_downloading_and_verification).
But this package is not a module, and so it does not have a go.mod file? For example, if I import package call "github.com/example/a" that is not a module, in go.sum file, it still has these two lines:
github.com/example/a v0.0.0-20190627063042-31896c4e4162 h1:rSqi2vQEpS+GAFKrLvmxzWW3OGlLI4hANnEf/ib/ofo=
github.com/example/a v0.0.0-20190627063042-31896c4e4162/go.mod h1:tcpxll8wcruwpPpWBbjAsWc1JbLHld/v9F+3rgLIr4c=
My question is, how the second line generated?
go.sum contains the expected cryptographic checksums
of the content of specific module versions. Each time a dependency is
used, its checksum is added to go.sum if missing or else required to match
the existing entry in go.sum.
Every package/module is dependency and every dependency mean to be maintained with checksums in go.sum so whether it is package or module it would be maintained.
The sources would be downloaded in $GOPATH/src directory accordingly.
TRY -
Cause in into the go.sum file are written EVERY dependencies sum hash. The one related to your go.mod file, and the one imported from the modules that you have imported. Try run go mod tidy in order to reduce the imported modules, your go.mod file will contains some //indirect import, that are the one that your imported modules use internally.
Maybe the Golang source code can explain the reason:
func (r *codeRepo) legacyGoMod(rev, dir string) []byte {
// We used to try to build a go.mod reflecting pre-existing
// package management metadata files, but the conversion
// was inherently imperfect (because those files don't have
// exactly the same semantics as go.mod) and, when done
// for dependencies in the middle of a build, impossible to
// correct. So we stopped.
// Return a fake go.mod that simply declares the module path.
return []byte(fmt.Sprintf("module %s\n", modfile.AutoQuote(r.modPath)))
}
code from here: https://github.com/golang/go/blob/acac535c3ca571beeb168c953d6d672f61387ef1/src/cmd/go/internal/modfetch/coderepo.go#L857
↓ VSCode open /usr/local/go/src:
locate to cmd/go/internal/modfetch/coderepo.go, add a breakpoint to func legacyGoMod
locate to cmd/go/internal/modfetch/coderepo_test.go, press F5
wait for a while, stop at the breakpoint
Following the release of Go 1.11, I have been trying to move my repositories to Go modules, by adding a go.mod file at their root.
One of my root libraries my.host/root is in its version 17.0.1, so I wrote in its go.mod file:
module my.host/root/v17
I tagged that version v17.0.1 as documented in the Go modules manual.
When I try to make a new Go project that uses my root library, like:
package main
import root "my.host/root/v17"
func main() {
root.DoSomething()
}
And try to compile it, I get the following error:
go: my.host/root#v0.0.0-20180828034419-6bc78016491a: go.mod has post-v0 module path "my.host/root/v17" at revision 6bc78016491a
I am at loss figuring out why this happens. I explicitly added v17.0.1 in the go.mod file, yet every attempt at go build replaces the entry with a v0.0.0-20180828034419-6bc78016491a version which then fails because at that commit, the go.mod file module entry of my root library indeed ends with a v17, as it should.
For the record, this commit is the same as the tagged v17.0.1 version.
What am I doing wrong here? How can I debug this situation?
I had make two mistakes:
My initial v17.0.0 tag would point to a commit where go.mod did not contain the v17 import path suffix. As a result, it seems Go tooling considers the whole v17 major version as a v0/v1 instead, even if later v17 tags point to a commit with a correct go.mod directive, hence the commit ID "translation".
In my dependent projects, in the go.mod file, I mistakenly specified
require my.host/root v17.0.1 instead of
require my.host/root/v17 v17.0.1.
After fixing both those issues, everything seems back to normal and it works perfectly. I wish the documentation had been clearer about this but I guess this is a good opportunity to make a contribution!
The error I got was: github.com/emicklei/go-restful#v0.0.0-20180531035034-3658237ded10: go.mod has post-v0 module path "github.com/emicklei/go-restful/v2" at revision 3658237ded10
Appending github.com/emicklei/go-restful with v2 like so: github.com/emicklei/go-restful/v2 in my go.mod file fixed it for me.