I try to get all the dependencies my project uses.
I looked at the go.mod file, though this only contains dependencies/modules that I added, and not the dependencies from my dependencies.
Looking at the go.sum file, this looked more promising, though then I noticed it contains multiple duplicates. Even though I only use one version of the dependency. As an example:
github.com/golang/protobuf v1.5.0/go.mod h1:FsONVRAS9T7sI+LIUmWTfcYkHO4aIWwzhcaSAoJOfIk=
github.com/golang/protobuf v1.5.2 h1:ROPKBNFfQgOUMifHyP+KYbvpjbdoFNs+aK7DXlji0Tw=
github.com/golang/protobuf v1.5.2/go.mod h1:XVQd3VNwM+JqD3oG2Ue2ip4fOMUkwXdXDdiuN0vRsmY=
google.golang.org/protobuf v1.27.1/go.mod h1:9q0QmTI4eRPtz6boOQmLYwt+qCgq0jsYwAQnmE0givc=
How can I get a list containing only all actively used dependencies/modules?
You are right about these 2 files, and technically you can clean up the go.sum by adding replace statements to the go.mod file. Though, it is not reliable that you (and your team) will always keep this clean.
In my opinion, the best way to get the list you are looking for is go list -m all (or go list --json -m all to get a json response). It will list you all the dependencies recursively in your project. But filters out the unused ones.
The modules and their requirements of a Go project can be created with the go mod graph command. The output lists the dependencies recursively.
tzhang:~/github.com/golang/example$ go mod graph
golang.org/x/example golang.org/x/tools#v0.0.0-20210112183307-1e6ecd4bf1b0
golang.org/x/tools#v0.0.0-20210112183307-1e6ecd4bf1b0 github.com/yuin/goldmark#v1.2.1
golang.org/x/tools#v0.0.0-20210112183307-1e6ecd4bf1b0 golang.org/x/mod#v0.3.0
golang.org/x/tools#v0.0.0-20210112183307-1e6ecd4bf1b0 golang.org/x/net#v0.0.0-20201021035429-f5854403a974
golang.org/x/tools#v0.0.0-20210112183307-1e6ecd4bf1b0 golang.org/x/sync#v0.0.0-20201020160332-67f06af15bc9
golang.org/x/tools#v0.0.0-20210112183307-1e6ecd4bf1b0 golang.org/x/xerrors#v0.0.0-20200804184101-5ec99f83aff1
golang.org/x/mod#v0.3.0 golang.org/x/crypto#v0.0.0-20191011191535-87dc89f01550
golang.org/x/mod#v0.3.0 golang.org/x/tools#v0.0.0-20191119224855-298f0cb1881e
golang.org/x/mod#v0.3.0 golang.org/x/xerrors#v0.0.0-20191011141410-1b5146add898
golang.org/x/net#v0.0.0-20201021035429-f5854403a974 golang.org/x/crypto#v0.0.0-20200622213623-75b288015ac9
golang.org/x/net#v0.0.0-20201021035429-f5854403a974 golang.org/x/sys#v0.0.0-20200930185726-fdedc70b468f
golang.org/x/net#v0.0.0-20201021035429-f5854403a974 golang.org/x/text#v0.3.3
golang.org/x/crypto#v0.0.0-20191011191535-87dc89f01550 golang.org/x/net#v0.0.0-20190404232315-eb5bcb51f2a3
golang.org/x/crypto#v0.0.0-20191011191535-87dc89f01550 golang.org/x/sys#v0.0.0-20190412213103-97732733099d
golang.org/x/tools#v0.0.0-20191119224855-298f0cb1881e golang.org/x/net#v0.0.0-20190620200207-3b0461eec859
golang.org/x/tools#v0.0.0-20191119224855-298f0cb1881e golang.org/x/sync#v0.0.0-20190423024810-112230192c58
golang.org/x/tools#v0.0.0-20191119224855-298f0cb1881e golang.org/x/xerrors#v0.0.0-20190717185122-a985d3407aa7
golang.org/x/crypto#v0.0.0-20200622213623-75b288015ac9 golang.org/x/net#v0.0.0-20190404232315-eb5bcb51f2a3
golang.org/x/crypto#v0.0.0-20200622213623-75b288015ac9 golang.org/x/sys#v0.0.0-20190412213103-97732733099d
golang.org/x/text#v0.3.3 golang.org/x/tools#v0.0.0-20180917221912-90fa682c2a6e
golang.org/x/net#v0.0.0-20190404232315-eb5bcb51f2a3 golang.org/x/crypto#v0.0.0-20190308221718-c2843e01d9a2
golang.org/x/net#v0.0.0-20190404232315-eb5bcb51f2a3 golang.org/x/text#v0.3.0
golang.org/x/net#v0.0.0-20190620200207-3b0461eec859 golang.org/x/crypto#v0.0.0-20190308221718-c2843e01d9a2
golang.org/x/net#v0.0.0-20190620200207-3b0461eec859 golang.org/x/sys#v0.0.0-20190215142949-d0b11bdaac8a
golang.org/x/net#v0.0.0-20190620200207-3b0461eec859 golang.org/x/text#v0.3.0
golang.org/x/crypto#v0.0.0-20190308221718-c2843e01d9a2 golang.org/x/sys#v0.0.0-20190215142949-d0b11bdaac8a
Related
I'm trying to compare the behavior of go mod tidy (and the resulting content of go.sum) to the output of go list -m all.
Reading the docs, I understand go.sum contains the whole list of dependent modules declared in go.mod and in dependencies' go.mod files, go list -m all shows the modules really loaded during the execution.
As an example, an application including logrus and prometheus like this:
go.mod
module mytest
go 1.14
require (
github.com/prometheus/common v0.4.0
github.com/sirupsen/logrus v1.8.1
)
main.go
package main
import "github.com/sirupsen/logrus"
import "github.com/prometheus/common/version"
func main() {
logrus.Info("Hello World")
logrus.Infof("Prometheus info: %v", version.Info())
}
After go mod tidy, go.sum shows both logrus v1.8.1, requested by the go.mod, and 1.2.0, dependency of prometheus v0.4.0; go list -m all shows only v1.8.1.
go.sum
[...]
github.com/sirupsen/logrus v1.2.0/go.mod h1:LxeOpSwHxABJmUn/MG1IvRgCAasNZTLOkJPxbbu5VWo=
github.com/sirupsen/logrus v1.8.1 h1:dJKuHgqk1NNQlqoA6BTlM1Wf9DOH3NBjQyu0h9+AZZE=
github.com/sirupsen/logrus v1.8.1/go.mod h1:yWOB1SBYBC5VeMP7gHvWumXLIWorT60ONWic61uBYv0=
[...]
output of go list
[...]
github.com/sirupsen/logrus v1.8.1
[...]
Is it correct to say the modules really used by the application are listed by go list -m all?
The underlying problem is that a static code analysis detects insecure module versions listed in go.sum, but actually these versions don't show up in go list -m all, hence they shouldn't be really used by the application, but only downloaded during build phase to select the proper minimal version.
Some reference:
https://go.dev/ref/mod#go-mod-tidy
go mod tidy acts as if all build tags are enabled, so it will consider
platform-specific source files and files that require custom build
tags, even if those source files wouldn’t normally be built.
https://go.dev/ref/mod#go-sum-files
The go.sum file may contain hashes for multiple versions of a module.
The go command may need to load go.mod files from multiple versions of
a dependency in order to perform minimal version selection. go.sum may
also contain hashes for module versions that aren’t needed anymore
(for example, after an upgrade).
https://github.com/golang/go/wiki/Modules#is-gosum-a-lock-file-why-does-gosum-include-information-for-module-versions-i-am-no-longer-using
[...]In addition, your module's go.sum records checksums for all
direct and indirect dependencies used in a build (and hence your
go.sum will frequently have more modules listed than your go.mod).
https://github.com/golang/go/wiki/Modules#version-selection
The minimal version selection algorithm is used to select the versions
of all modules used in a build. For each module in a build, the
version selected by minimal version selection is always the
semantically highest of the versions explicitly listed by a require
directive in the main module or one of its dependencies.
As an example, if your module depends on module A which has a
require D v1.0.0, and your module also depends on module B which
has a require D v1.1.1, then minimal version selection would choose
v1.1.1 of D to include in the build (given it is the highest listed
require version). [...] To see a list of the selected module versions
(including indirect dependencies), use go list -m all.
Yes, it correct to say the modules really "used" by the application are listed by go list -m all (as per documentation you provided the link of). By "used", it means the package selected at build time for the compilation of the go code of your application.
We had a similar issue with a static analysis tool and we had to change the configuration to use the output of go list -m all (dumped in a file) instead of go.sum.
I just inherited a Go project that has a go.mod file missing a declared dependency, but the dependency is in the the go.sum file:
...
cloud.google.com/go/storage v?.?.? <- this is the missing entry in go.mod
...
these are the entries in go.sum file:
...
cloud.google.com/go/storage v1.0.0/go.mod h1:<some hash>
cloud.google.com/go/storage v1.5.0/go.mod h1:<some hash>
cloud.google.com/go/storage v1.6.0/go.mod h1:<some hash>
cloud.google.com/go/storage v1.8.0/go.mod h1:<some hash>
cloud.google.com/go/storage v1.10.0 h1:<some hash>
cloud.google.com/go/storage v1.10.0/go.mod h1:<some hash>
...
My questions are:
Why are there 5 versions in the go.sum file?
If there are other libraries that depend on these specific versions do all 5 get compiled into the binary?
Which version of the lib will be linked to my application code since the dependency is not declared?
I tried to find an explanation in the Go documentation but could not locate, any help appreciated.
These dependencies are in all likelihood transitive dependencies, that is dependencies of the packages you depend on (or those that they depend on, etc). The go.sum contains lines for all dependencies of your module, direct or otherwise, in order for builds to be reproducible.
From the Go blog:
In addition to go.mod, the go command maintains a file named go.sum containing the expected cryptographic hashes of the content of specific module versions
...
The go command uses the go.sum file to ensure that future downloads of these modules retrieve the same bits as the first download, to ensure the modules your project depends on do not change unexpectedly, whether for malicious, accidental, or other reasons. Both go.mod and go.sum should be checked into version control.
The version of the package that does get included depends on the go.mod file of the package that you depend on. You may depend on several packages and each may depend on a different version of the dependency.
Whether they end up in your build depends on whether the dependency that includes them is compiled into your binary. An example where that inclusion may not happen are test files/packages, which will usually depend on testing libraries and their dependencies. These are never included in your average go build executable.
You can check the list of packages that will be included in your build like this:
go list -m all
You should just in case run go mod tidy to remove any dependencies that aren't actually needed anymore.
Here's a snippet from a go.sum file for a project I maintain.
github.com/golang/protobuf v1.2.0/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
github.com/golang/protobuf v1.3.1/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
github.com/golang/protobuf v1.3.5 h1:F768QJ1E9tib+q5Sc8MkdJi1RxLTbRcTf8LJV56aRls=
github.com/golang/protobuf v1.3.5/go.mod h1:6O5/vntMXwX2lRkT1hjjk0nAC1IDOTvTlVgjlRvqsdk=
As far as I can tell, this project, and the other projects it depends on, only use v1.3.5.
Is there a way to determine why v1.3.1 and v1.2.0 are in the go.sum file? For example, can I run go mod why ... with something in the place of ... to determine why those lines are present?
I understand they are not being used by the project when I invoke import "github.com/golang/protobuf", but I would like to understand the toolchain a little better.
If you use go mod graph, you can see the full dependency list, including the versions your project is not using. Some sample lines from the go mod graph output.
github.com/acme/project github.com/getsentry/sentry-go#v0.5.2-0.20200226112222-4dddaaad5cc5
...
github.com/getsentry/sentry-go#v0.5.2-0.20200226112222-4dddaaad5cc5 github.com/onsi/gomega#v1.7.1
...
github.com/onsi/gomega#v1.7.1 github.com/golang/protobuf#v1.2.0
I believe, this happens due to cyclic interdependency that protobuf has.
The issue has been addressed here already: https://github.com/golang/protobuf/issues/1204
However, they are not going to do anything about it, cause it is not a technical problem, the projects can be built.
For example, given a package A that depends on package B and package C, where package C also depends on package D - is there a way to output this information? (Using a vendoring tool or otherwise)
The vendor.yaml output by govend doesn't include transitive dependency information - neither does the Gopkg.toml file
output by dep, from what I can see. The go.mod file produced by Golang 1.11's mod and does annotate some dependencies as // indirect - but it does not annotate dependencies with any information about which dependency they were pulled in via.
Did you try https://github.com/KyleBanks/depth?
It does provide a decent dependency tree at first look I tried.
Following the release of Go 1.11, I have been trying to move my repositories to Go modules, by adding a go.mod file at their root.
One of my root libraries my.host/root is in its version 17.0.1, so I wrote in its go.mod file:
module my.host/root/v17
I tagged that version v17.0.1 as documented in the Go modules manual.
When I try to make a new Go project that uses my root library, like:
package main
import root "my.host/root/v17"
func main() {
root.DoSomething()
}
And try to compile it, I get the following error:
go: my.host/root#v0.0.0-20180828034419-6bc78016491a: go.mod has post-v0 module path "my.host/root/v17" at revision 6bc78016491a
I am at loss figuring out why this happens. I explicitly added v17.0.1 in the go.mod file, yet every attempt at go build replaces the entry with a v0.0.0-20180828034419-6bc78016491a version which then fails because at that commit, the go.mod file module entry of my root library indeed ends with a v17, as it should.
For the record, this commit is the same as the tagged v17.0.1 version.
What am I doing wrong here? How can I debug this situation?
I had make two mistakes:
My initial v17.0.0 tag would point to a commit where go.mod did not contain the v17 import path suffix. As a result, it seems Go tooling considers the whole v17 major version as a v0/v1 instead, even if later v17 tags point to a commit with a correct go.mod directive, hence the commit ID "translation".
In my dependent projects, in the go.mod file, I mistakenly specified
require my.host/root v17.0.1 instead of
require my.host/root/v17 v17.0.1.
After fixing both those issues, everything seems back to normal and it works perfectly. I wish the documentation had been clearer about this but I guess this is a good opportunity to make a contribution!
The error I got was: github.com/emicklei/go-restful#v0.0.0-20180531035034-3658237ded10: go.mod has post-v0 module path "github.com/emicklei/go-restful/v2" at revision 3658237ded10
Appending github.com/emicklei/go-restful with v2 like so: github.com/emicklei/go-restful/v2 in my go.mod file fixed it for me.