I have a dependency in my go.sum which has a vulnerability. I'm trying to determine if we are affected.
The go.mod has the up to date version, but the go.sum lists many other earlier vulnerable versions. I run go mod why https://github.com/prometheus/client_golang and go mod why -m https://github.com/prometheus/client_golang and go mod graph | grep client_golang and see slightly different outputs, the last being the most verbose.
I also see this message "main module does not need module". Well, yes I understand that the main module doesn't need it but clearly it's a dependency. Does that mean something else I'm not understanding?
Related
I'm trying to compare the behavior of go mod tidy (and the resulting content of go.sum) to the output of go list -m all.
Reading the docs, I understand go.sum contains the whole list of dependent modules declared in go.mod and in dependencies' go.mod files, go list -m all shows the modules really loaded during the execution.
As an example, an application including logrus and prometheus like this:
go.mod
module mytest
go 1.14
require (
github.com/prometheus/common v0.4.0
github.com/sirupsen/logrus v1.8.1
)
main.go
package main
import "github.com/sirupsen/logrus"
import "github.com/prometheus/common/version"
func main() {
logrus.Info("Hello World")
logrus.Infof("Prometheus info: %v", version.Info())
}
After go mod tidy, go.sum shows both logrus v1.8.1, requested by the go.mod, and 1.2.0, dependency of prometheus v0.4.0; go list -m all shows only v1.8.1.
go.sum
[...]
github.com/sirupsen/logrus v1.2.0/go.mod h1:LxeOpSwHxABJmUn/MG1IvRgCAasNZTLOkJPxbbu5VWo=
github.com/sirupsen/logrus v1.8.1 h1:dJKuHgqk1NNQlqoA6BTlM1Wf9DOH3NBjQyu0h9+AZZE=
github.com/sirupsen/logrus v1.8.1/go.mod h1:yWOB1SBYBC5VeMP7gHvWumXLIWorT60ONWic61uBYv0=
[...]
output of go list
[...]
github.com/sirupsen/logrus v1.8.1
[...]
Is it correct to say the modules really used by the application are listed by go list -m all?
The underlying problem is that a static code analysis detects insecure module versions listed in go.sum, but actually these versions don't show up in go list -m all, hence they shouldn't be really used by the application, but only downloaded during build phase to select the proper minimal version.
Some reference:
https://go.dev/ref/mod#go-mod-tidy
go mod tidy acts as if all build tags are enabled, so it will consider
platform-specific source files and files that require custom build
tags, even if those source files wouldn’t normally be built.
https://go.dev/ref/mod#go-sum-files
The go.sum file may contain hashes for multiple versions of a module.
The go command may need to load go.mod files from multiple versions of
a dependency in order to perform minimal version selection. go.sum may
also contain hashes for module versions that aren’t needed anymore
(for example, after an upgrade).
https://github.com/golang/go/wiki/Modules#is-gosum-a-lock-file-why-does-gosum-include-information-for-module-versions-i-am-no-longer-using
[...]In addition, your module's go.sum records checksums for all
direct and indirect dependencies used in a build (and hence your
go.sum will frequently have more modules listed than your go.mod).
https://github.com/golang/go/wiki/Modules#version-selection
The minimal version selection algorithm is used to select the versions
of all modules used in a build. For each module in a build, the
version selected by minimal version selection is always the
semantically highest of the versions explicitly listed by a require
directive in the main module or one of its dependencies.
As an example, if your module depends on module A which has a
require D v1.0.0, and your module also depends on module B which
has a require D v1.1.1, then minimal version selection would choose
v1.1.1 of D to include in the build (given it is the highest listed
require version). [...] To see a list of the selected module versions
(including indirect dependencies), use go list -m all.
Yes, it correct to say the modules really "used" by the application are listed by go list -m all (as per documentation you provided the link of). By "used", it means the package selected at build time for the compilation of the go code of your application.
We had a similar issue with a static analysis tool and we had to change the configuration to use the output of go list -m all (dumped in a file) instead of go.sum.
I ran into a repo that seems to be a Go module, but there's no go.mod file in it: github.com/confluentinc/confluent-kafka-go.
Is it ok for a go module to have no go.mod file with dependencies, or the authors of that library just didn't migrate to modules yet?
Dependency modules do not need to have explicit go.mod files.
The “main module” in module mode — that is, the module containing the working directory for the go command — must have a go.mod file, so that the go command can figure out the import paths for the packages within that module (based on its module path), and so that it has a place to record its dependencies once resolved.
In addition, any modules slotted in using replace directives must have go.mod files (in order to reduce confusion due to typos or other errors in replacement paths).
However, in general a module that lacks an explicit go.mod file is valid and fine to use. Its effective module path is the path by which it was required, which can be a bit confusing if the same repository ends up in use via multiple paths. Since a module with no go.mod file necessarily doesn't specify its own dependencies, consumers of that module will have to fill in those dependencies themselves (go mod tidy will mark them as // indirect in the consumer's go.mod file).
SHORT SUMMARY OF THE DISCUSSION:
The answer is "No"!
This project contains a set of go packages, but it is not a Go module as it doesn't contain go.mod file (although, it used to be a multi-module repo (Go) previously).
go get can run in both ways: module-aware mode and legacy GOPATH mode (as of Go 1.16).
To read more about this, refer to docs by using the go command:
$ go help gopath-get
and
$ go help module-get
It'd tell about how go get works in both cases.
Also, I noticed that it can download any repository and would treat it as a Go package, even if it contains an arbitrary Python project.
I did a simple test to demonstrate the same:
$ go get github.com/mongoengine/mongoengine
And it surprisingly worked.
Modules are defined by their go.mod file. Without a go.mod file, it is not a module.
See this from the Go Modules Reference
A module is a collection of packages that are released, versioned, and distributed together. Modules may be downloaded directly from version control repositories or from module proxy servers.
A module is identified by a module path, which is declared in a go.mod file, together with information about the module's dependencies. The module root directory is the directory that contains the go.mod file.
And
A module is defined by a UTF-8 encoded text file named go.mod in its root directory.
I just inherited a Go project that has a go.mod file missing a declared dependency, but the dependency is in the the go.sum file:
...
cloud.google.com/go/storage v?.?.? <- this is the missing entry in go.mod
...
these are the entries in go.sum file:
...
cloud.google.com/go/storage v1.0.0/go.mod h1:<some hash>
cloud.google.com/go/storage v1.5.0/go.mod h1:<some hash>
cloud.google.com/go/storage v1.6.0/go.mod h1:<some hash>
cloud.google.com/go/storage v1.8.0/go.mod h1:<some hash>
cloud.google.com/go/storage v1.10.0 h1:<some hash>
cloud.google.com/go/storage v1.10.0/go.mod h1:<some hash>
...
My questions are:
Why are there 5 versions in the go.sum file?
If there are other libraries that depend on these specific versions do all 5 get compiled into the binary?
Which version of the lib will be linked to my application code since the dependency is not declared?
I tried to find an explanation in the Go documentation but could not locate, any help appreciated.
These dependencies are in all likelihood transitive dependencies, that is dependencies of the packages you depend on (or those that they depend on, etc). The go.sum contains lines for all dependencies of your module, direct or otherwise, in order for builds to be reproducible.
From the Go blog:
In addition to go.mod, the go command maintains a file named go.sum containing the expected cryptographic hashes of the content of specific module versions
...
The go command uses the go.sum file to ensure that future downloads of these modules retrieve the same bits as the first download, to ensure the modules your project depends on do not change unexpectedly, whether for malicious, accidental, or other reasons. Both go.mod and go.sum should be checked into version control.
The version of the package that does get included depends on the go.mod file of the package that you depend on. You may depend on several packages and each may depend on a different version of the dependency.
Whether they end up in your build depends on whether the dependency that includes them is compiled into your binary. An example where that inclusion may not happen are test files/packages, which will usually depend on testing libraries and their dependencies. These are never included in your average go build executable.
You can check the list of packages that will be included in your build like this:
go list -m all
You should just in case run go mod tidy to remove any dependencies that aren't actually needed anymore.
Here's a snippet from a go.sum file for a project I maintain.
github.com/golang/protobuf v1.2.0/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
github.com/golang/protobuf v1.3.1/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
github.com/golang/protobuf v1.3.5 h1:F768QJ1E9tib+q5Sc8MkdJi1RxLTbRcTf8LJV56aRls=
github.com/golang/protobuf v1.3.5/go.mod h1:6O5/vntMXwX2lRkT1hjjk0nAC1IDOTvTlVgjlRvqsdk=
As far as I can tell, this project, and the other projects it depends on, only use v1.3.5.
Is there a way to determine why v1.3.1 and v1.2.0 are in the go.sum file? For example, can I run go mod why ... with something in the place of ... to determine why those lines are present?
I understand they are not being used by the project when I invoke import "github.com/golang/protobuf", but I would like to understand the toolchain a little better.
If you use go mod graph, you can see the full dependency list, including the versions your project is not using. Some sample lines from the go mod graph output.
github.com/acme/project github.com/getsentry/sentry-go#v0.5.2-0.20200226112222-4dddaaad5cc5
...
github.com/getsentry/sentry-go#v0.5.2-0.20200226112222-4dddaaad5cc5 github.com/onsi/gomega#v1.7.1
...
github.com/onsi/gomega#v1.7.1 github.com/golang/protobuf#v1.2.0
I believe, this happens due to cyclic interdependency that protobuf has.
The issue has been addressed here already: https://github.com/golang/protobuf/issues/1204
However, they are not going to do anything about it, cause it is not a technical problem, the projects can be built.
Following the release of Go 1.11, I have been trying to move my repositories to Go modules, by adding a go.mod file at their root.
One of my root libraries my.host/root is in its version 17.0.1, so I wrote in its go.mod file:
module my.host/root/v17
I tagged that version v17.0.1 as documented in the Go modules manual.
When I try to make a new Go project that uses my root library, like:
package main
import root "my.host/root/v17"
func main() {
root.DoSomething()
}
And try to compile it, I get the following error:
go: my.host/root#v0.0.0-20180828034419-6bc78016491a: go.mod has post-v0 module path "my.host/root/v17" at revision 6bc78016491a
I am at loss figuring out why this happens. I explicitly added v17.0.1 in the go.mod file, yet every attempt at go build replaces the entry with a v0.0.0-20180828034419-6bc78016491a version which then fails because at that commit, the go.mod file module entry of my root library indeed ends with a v17, as it should.
For the record, this commit is the same as the tagged v17.0.1 version.
What am I doing wrong here? How can I debug this situation?
I had make two mistakes:
My initial v17.0.0 tag would point to a commit where go.mod did not contain the v17 import path suffix. As a result, it seems Go tooling considers the whole v17 major version as a v0/v1 instead, even if later v17 tags point to a commit with a correct go.mod directive, hence the commit ID "translation".
In my dependent projects, in the go.mod file, I mistakenly specified
require my.host/root v17.0.1 instead of
require my.host/root/v17 v17.0.1.
After fixing both those issues, everything seems back to normal and it works perfectly. I wish the documentation had been clearer about this but I guess this is a good opportunity to make a contribution!
The error I got was: github.com/emicklei/go-restful#v0.0.0-20180531035034-3658237ded10: go.mod has post-v0 module path "github.com/emicklei/go-restful/v2" at revision 3658237ded10
Appending github.com/emicklei/go-restful with v2 like so: github.com/emicklei/go-restful/v2 in my go.mod file fixed it for me.