I have a question about dependencies in golang.
My application defines a go.mod like this:
module my.host.com/myapp
require (
ext1.com/module1 v0.0.1
)
go 1.14
The dependency relationship is：
ext1.com/module1 v0.0.1 depends on ext3.com/module3 v0.0.3
A security scan detects ext3.com/module3 v0.0.3 is insecure and must be updated to v0.0.4.
Is there a way to "force" myapp to get only module3 v0.0.4, overriding the directives defined in module1 v0.0.1 go.mod?
Let's say ext1.com/module1 v0.0.1 is already at the latest version, so upgrading it doesn't work.
Would "replace" work?
module my.host.com/myapp
require (
ext1.com/module1 v0.0.1
)
replace ext3.com/module3 v0.0.3 => ext3.com/module3 v0.0.4
go 1.14
Thanks in advance!
Run go get -u ext3.com/module3#v0.0.4.
This upgrades the module to at least the v0.0.4
Given the dependency main -> B -> C, when main requires a higher version of C than that required by B, the higher version is selected, with // indirect.
See this https://go.dev/ref/mod#go-mod-file-require
If the go directive specifies go 1.16 or lower, the go command adds an indirect requirement when the selected version of a module is higher than what is already implied (transitively) by the main module’s other dependencies. That may occur because of an explicit upgrade (go get -u ./...)
I quote this part because your go.mod has go 1.14
Related
I'm trying to compare the behavior of go mod tidy (and the resulting content of go.sum) to the output of go list -m all.
Reading the docs, I understand go.sum contains the whole list of dependent modules declared in go.mod and in dependencies' go.mod files, go list -m all shows the modules really loaded during the execution.
As an example, an application including logrus and prometheus like this:
go.mod
module mytest
go 1.14
require (
github.com/prometheus/common v0.4.0
github.com/sirupsen/logrus v1.8.1
)
main.go
package main
import "github.com/sirupsen/logrus"
import "github.com/prometheus/common/version"
func main() {
logrus.Info("Hello World")
logrus.Infof("Prometheus info: %v", version.Info())
}
After go mod tidy, go.sum shows both logrus v1.8.1, requested by the go.mod, and 1.2.0, dependency of prometheus v0.4.0; go list -m all shows only v1.8.1.
go.sum
[...]
github.com/sirupsen/logrus v1.2.0/go.mod h1:LxeOpSwHxABJmUn/MG1IvRgCAasNZTLOkJPxbbu5VWo=
github.com/sirupsen/logrus v1.8.1 h1:dJKuHgqk1NNQlqoA6BTlM1Wf9DOH3NBjQyu0h9+AZZE=
github.com/sirupsen/logrus v1.8.1/go.mod h1:yWOB1SBYBC5VeMP7gHvWumXLIWorT60ONWic61uBYv0=
[...]
output of go list
[...]
github.com/sirupsen/logrus v1.8.1
[...]
Is it correct to say the modules really used by the application are listed by go list -m all?
The underlying problem is that a static code analysis detects insecure module versions listed in go.sum, but actually these versions don't show up in go list -m all, hence they shouldn't be really used by the application, but only downloaded during build phase to select the proper minimal version.
Some reference:
https://go.dev/ref/mod#go-mod-tidy
go mod tidy acts as if all build tags are enabled, so it will consider
platform-specific source files and files that require custom build
tags, even if those source files wouldn’t normally be built.
https://go.dev/ref/mod#go-sum-files
The go.sum file may contain hashes for multiple versions of a module.
The go command may need to load go.mod files from multiple versions of
a dependency in order to perform minimal version selection. go.sum may
also contain hashes for module versions that aren’t needed anymore
(for example, after an upgrade).
https://github.com/golang/go/wiki/Modules#is-gosum-a-lock-file-why-does-gosum-include-information-for-module-versions-i-am-no-longer-using
[...]In addition, your module's go.sum records checksums for all
direct and indirect dependencies used in a build (and hence your
go.sum will frequently have more modules listed than your go.mod).
https://github.com/golang/go/wiki/Modules#version-selection
The minimal version selection algorithm is used to select the versions
of all modules used in a build. For each module in a build, the
version selected by minimal version selection is always the
semantically highest of the versions explicitly listed by a require
directive in the main module or one of its dependencies.
As an example, if your module depends on module A which has a
require D v1.0.0, and your module also depends on module B which
has a require D v1.1.1, then minimal version selection would choose
v1.1.1 of D to include in the build (given it is the highest listed
require version). [...] To see a list of the selected module versions
(including indirect dependencies), use go list -m all.
Yes, it correct to say the modules really "used" by the application are listed by go list -m all (as per documentation you provided the link of). By "used", it means the package selected at build time for the compilation of the go code of your application.
We had a similar issue with a static analysis tool and we had to change the configuration to use the output of go list -m all (dumped in a file) instead of go.sum.
E.g., github security alery signaled:
Upgrade github.com/opencontainers/runc to version 1.0.0-rc95 or later. For example:
How can I do it in golang with minimal code changes? This package is not being declared in go.mod (seems like it's indirect dependency).
When I run go get -u, I can see a whole bunch of undirect dependencies added to my go.mod which is a bit overkill:
require (
cloud.google.com/go v0.N.0 // indirect <--------
Is there a way to update directy that dependency?
Pretty sure go get -u github.com/opencontainers/runc should do it.
I have a GOPATH based project that I currently build like this on Fedora:
sudo dnf install golang-etcd-bbolt-devel golang-x-sys-devel golang-x-text-devel
GOPATH=$HOME/go:/usr/share/gocode go build
My project (gonzofilter) implements a command line utility and thus source files are located in the main package (i.e. they have a package main declaration).
With Fedora 34 and beyond it seems that Go removed support for building GOPATH style projects and one really has to use Go modules:
go build
go: cannot find main module; see 'go help modules'
That Go blog post kind of covers my case (-> 'Without a dependency manager'), but it doesn't explicitly mention how to deal with main package projects or with distribution provided dependencies.
So, how do I migrate such a project?
How do I have to tell Go/go mod tidy to look for my dependencies under /usr/share/gocode?
Edit: To be precise: Fedora 34 comes with Go 1.16 which 'just' changed the GO111MODULE default from auto to on. Thus, one still can restore the old behavior by setting GO111MODULE=auto.
However, Golang developers already announced that they want to drop support for GOPATH style projects in Go 1.17:
We plan to drop support for GOPATH mode in Go 1.17. In other words, Go 1.17 will ignore GO111MODULE. If you have projects that do not build in module-aware mode, now is the time to migrate.
Update 2023-02-19: As of Go 1.19.5 (Fedora 37), GO111MODULE=off still works for building GOPATH style (i.e. non-modularized) projects. Example:
export GOPATH=$HOME/go:/usr/share/gocode GOPROXY=off GO111MODULE=off
go build helloworld.go
Apparently, the Go team adjusted their GO11MODULE deprecation plan, without targeting any new future release for removal, yet.
You can explicity define using replace keyword in your generated mod file to refer to your local modules.
replace packagename => /usr/share/gcode
Ideally, distributions should package installed dependencies in a layout that is compatible with the GOPROXY protocol. Then you would be able to set GOPROXY appropriately and run go mod tidy to use the installed dependencies.
However, to my knowledge, no distributions actually provide a GOPROXY tree at this point. You may need a workaround.
The next best alternative is to wire up the replacements yourself using replace directives. go mod tidy already knows to prefer the versions found in replace directives over other versions, so it should suffice to do something like:
go mod init github.com/gsauthof/gonzofilter
go mod edit -replace go.etcd.io/bbolt=/usr/share/gocode/src/go.etcd.io/bbolt
go mod edit -replace golang.org/x/sys=/usr/share/gocode/src/golang.org/x/sys
go mod edit -replace golang.org/x/text=/usr/share/gocode/src/golang.org/x/text
go mod tidy
However, note that the replace directive requires the target to include an explicit go.mod file, so this will only work if either your dependencies already have explicit go.mod files or your distro packager has added them as a patch to support this use-case.
Note that with those replace directives in place, your build would not be reproducible or repeatable over time: if you change the distro-installed versions of your dependencies, then the meaning of go build will silently change to use those different (and possibly incompatible!) versions as well.
So I would recommend that you not do that, and instead use go get and/or go mod tidy to fetch the specific module versions you want from upstream or from a public module proxy (such as proxy.golang.org).
If you are concerned about upstream tampering, note that the Go project's official binary distribution of the go command — and the go command built from pristine source — by default automatically verifies checksums for downloaded modules against an auditable public database at https://sum.golang.org; however, the Fedora distribution of the go command disables use of the checksum database by default.
One way to migrate such a GOPATH project to a Go modules aware one is the following:
First, manually create the go.mod with the direct dependencies:
module github.com/gsauthof/gonzofilter
go 1.15
require (
go.etcd.io/bbolt v1.3.5
golang.org/x/sys v0.0.0-20210423185535-09eb48e85fd7
golang.org/x/text v0.3.5
)
I got the minimal versions from what's currently on my system:
$ rpm -q golang-etcd-bbolt-devel golang-x-sys-devel golang-x-text-devel golang-bin
golang-etcd-bbolt-devel-1.3.5-2.fc33.noarch
golang-x-sys-devel-0-0.39.20210123git9b0068b.fc33.noarch
golang-x-text-devel-0.3.5-1.fc33.noarch
golang-bin-1.15.8-1.fc33.x86_64
The problem now is that one can't simply tell the module-aware Go tools to look for those modules under the filesystem path /usr/share/gocode/src where they are all located.
There is the replace directive one can add to our go.mod file to set the lookup path for single dependencies, e.g. like this:
replace (
go.etcd.io/bbolt => /usr/share/gocode/src/go.etcd.io/bbolt
golang.org/x/sys => /usr/share/gocode/src/golang.org/x/sys
golang.org/x/text => /usr/share/gocode/src/golang.org/x/text
)
However, a go build still doesn't use indirect dependencies from /usr/share/gocode, e.g. the ones that are provided by the packages golang-x-text-devel depends on. In this example, go build does find /usr/share/gocode/src/golang.org/x/text/go.mod but that file doesn't include any replace directive.
Thus, this fails:
$ GOPROXY=off go build -mod=readonly
go: golang.org/x/text#v0.3.6 requires
golang.org/x/tools#v0.0.0-20180917221912-90fa682c2a6e: module lookup disabled
by GOPROXY=off
To resolve this we have to add replace directive lines also for all indirect dependencies.
Doing this manually is tedious and error prone, of course.
Thus, we can automate this with a small shell one-liner:
We start with this go.mod file:
module github.com/gsauthof/gonzofilter
go 1.15
require (
go.etcd.io/bbolt v1.3.5
golang.org/x/sys v0.0.0-20210423185535-09eb48e85fd7
golang.org/x/text v0.3.6
)
replace (
//replace-this
)
The following fixpoint iteration then adds dependencies until the build succeeds:
while true; do
GOPROXY=off go build -mod readonly 2> t
if [ $? -eq 1 ] && grep 'module lookup disabled\|missing go.sum entry' t >/dev/null; then
x=$(grep 'disabled\|missing' t | tr ' \t' '\n' | grep '#' | head -n1 | cut -d'#' -f1 )
echo "Adding $x"
sed -i "s#^\(//replace-this\)#\t$x => /usr/share/gocode/src/$x\n\1#" go.mod
continue
fi
break
done
Which results in the following require directive for this example project:
replace (
go.etcd.io/bbolt => /usr/share/gocode/src/go.etcd.io/bbolt
golang.org/x/sys => /usr/share/gocode/src/golang.org/x/sys
golang.org/x/text => /usr/share/gocode/src/golang.org/x/text
golang.org/x/tools => /usr/share/gocode/src/golang.org/x/tools
github.com/yuin/goldmark => /usr/share/gocode/src/github.com/yuin/goldmark
golang.org/x/mod => /usr/share/gocode/src/golang.org/x/mod
golang.org/x/net => /usr/share/gocode/src/golang.org/x/net
golang.org/x/sync => /usr/share/gocode/src/golang.org/x/sync
golang.org/x/xerrors => /usr/share/gocode/src/golang.org/x/xerrors
golang.org/x/crypto => /usr/share/gocode/src/golang.org/x/crypto
golang.org/x/term => /usr/share/gocode/src/golang.org/x/term
//replace-this
)
Another way to point Go to the system-wide available dependencies is to exploit Go's module vendoring support: That means by symbolically linking the module file paths, creating a minimal vendor/modules.txt and compiling with -mod vendor.
So with -mod vendor Go build doesn't try to download any required modules, instead it looks them up in vendor/ directory.
One might be tempted to just create a symbolic link that points from vendor/ to /usr/share/gocode/src (where all the distributions golang-...-devel packages are installed), but this doesn't work because -mod vendor also requires yet another modules description file, i.e. vendor/modules.txt.
So, besides some more specific symlinks, we have to create a proper modules.txt when using the vendoring feature:
First, to create all the sublevel symlinks:
awk -vvd=wendor 'BEGIN {
PROCINFO["sorted_in"]="#ind_str_asc";
system("mkdir -p "vd)
}
func push_mod(pkg) {
if (!seen[pkg]) {
ARGV[ARGC++]="/usr/share/gocode/src/"pkg"/go.mod";
seen[pkg]=1;
}
sub("/.+$", "", pkg);
xs[pkg]=1;
}
/^require[^(]+$/ {
push_mod($2);
next
}
/^require/ {
inb=1;
next
}
/^\)/ {
inb=0;
next
}
inb {
push_mod($1);
}
END {
for (i in xs) {
print i;
system("ln -sfn /usr/share/gocode/src/"i" "vd"/"i)
}
}' go.mod
Then, to create the vendor/modules.txt:
awk -vvd=wendor 'BEGIN {
fn=vd"/modules.txt"
}
/^require/ {
inb=1;
next
}
/^\)/ {
inb=0;
next
}
inb {
printf("# %s %s\n## explicit\n%s\n", $1, $2, $1) > fn
}' go.mod
After those commands, the vendor directory looks like this for our example project:
$ ls -l vendor
total 16
lrwxrwxrwx. 1 juser juser 32 2021-04-25 11:12 github.com -> /usr/share/gocode/src/github.com
lrwxrwxrwx. 1 juser juser 32 2021-04-25 11:12 go.etcd.io -> /usr/share/gocode/src/go.etcd.io
lrwxrwxrwx. 1 juser juser 32 2021-04-25 11:12 golang.org -> /usr/share/gocode/src/golang.org
-rw-r--r--. 1 juser juser 195 2021-04-25 11:13 modules.txt
$ cat vendor/modules.txt
# go.etcd.io/bbolt v1.3.5
## explicit
go.etcd.io/bbolt
# golang.org/x/sys v0.0.0-20210423185535-09eb48e85fd7
## explicit
golang.org/x/sys
# golang.org/x/text v0.3.5
## explicit
golang.org/x/text
Whereas go.mod is quite minimal, i.e. it doesn't contain any replace directive:
module github.com/gsauthof/gonzofilter
go 1.15
require (
go.etcd.io/bbolt v1.3.5
golang.org/x/sys v0.0.0-20210423185535-09eb48e85fd7
golang.org/x/text v0.3.5
)
And this is sufficient to let
GOPROXY=off go build -mod vendor
succeed.
Note that indirect dependencies such as golang.org/x/tools aren't listed in the modules.txt, however, they are still available via vendor/ because we transitively traversed all go.mod files starting with the required modules when creating symlinks to the top level directories.
The discrete charme of this scheme is that one doesn't have to mess with replace directives in the go.mod file. That means go.mod can be kept quite generic, one basically just have to add the -mod vendor switch to the build command.
I just inherited a Go project that has a go.mod file missing a declared dependency, but the dependency is in the the go.sum file:
...
cloud.google.com/go/storage v?.?.? <- this is the missing entry in go.mod
...
these are the entries in go.sum file:
...
cloud.google.com/go/storage v1.0.0/go.mod h1:<some hash>
cloud.google.com/go/storage v1.5.0/go.mod h1:<some hash>
cloud.google.com/go/storage v1.6.0/go.mod h1:<some hash>
cloud.google.com/go/storage v1.8.0/go.mod h1:<some hash>
cloud.google.com/go/storage v1.10.0 h1:<some hash>
cloud.google.com/go/storage v1.10.0/go.mod h1:<some hash>
...
My questions are:
Why are there 5 versions in the go.sum file?
If there are other libraries that depend on these specific versions do all 5 get compiled into the binary?
Which version of the lib will be linked to my application code since the dependency is not declared?
I tried to find an explanation in the Go documentation but could not locate, any help appreciated.
These dependencies are in all likelihood transitive dependencies, that is dependencies of the packages you depend on (or those that they depend on, etc). The go.sum contains lines for all dependencies of your module, direct or otherwise, in order for builds to be reproducible.
From the Go blog:
In addition to go.mod, the go command maintains a file named go.sum containing the expected cryptographic hashes of the content of specific module versions
...
The go command uses the go.sum file to ensure that future downloads of these modules retrieve the same bits as the first download, to ensure the modules your project depends on do not change unexpectedly, whether for malicious, accidental, or other reasons. Both go.mod and go.sum should be checked into version control.
The version of the package that does get included depends on the go.mod file of the package that you depend on. You may depend on several packages and each may depend on a different version of the dependency.
Whether they end up in your build depends on whether the dependency that includes them is compiled into your binary. An example where that inclusion may not happen are test files/packages, which will usually depend on testing libraries and their dependencies. These are never included in your average go build executable.
You can check the list of packages that will be included in your build like this:
go list -m all
You should just in case run go mod tidy to remove any dependencies that aren't actually needed anymore.
I read everywhere that packages have to have a tag of v0 or v1. Why can't a tag be v2020 or something other than v0 or v1. I have tried this personal and is get the following error with I use v2020.
Scotts-Mac-mini:seeding syacko$ go mod tidy
go: errors parsing go.mod:
/Users/syacko/workspace/sotesoft/src/utils/seeding/go.mod:10: require gitlab.com/soteapps/packages: version "v2020.2.0" invalid: module contains a go.mod file, so major version must be compatible: should be v0 or v1, not v2020
Scotts-Mac-mini:seeding syacko$
It's a convention, a convenience for everyone. Go modules chose to use the widely accepted Semantic Versioning v2.
Go Modules Wiki:
What happens if I create a go.mod but do not apply semver tags to my repository?
semver is a foundation of the modules system. In order to provide the best experience for consumers, module authors are encouraged to apply semver VCS tags (e.g., v0.1.0 or v1.2.3-rc.1), but semver VCS tags are not strictly required:
Modules are required to follow the semver specification in order for the go command to behave as documented. This includes following the semver specification regarding how and when breaking changes are allowed.
Modules that do not have semver VCS tags are recorded by consumers using a semver version in the form of a pseudo-version. Typically this will be a v0 major version, unless the module author constructed a v2+ module following the "Major Subdirectory" approach.
Therefore, modules that do not apply semver VCS tags and have not created a "Major Subdirectory" are effectively declaring themselves to be in the semver v0 major version series, and a module-based consumer will treat them as having a semver v0 major version.
An interesting and relevant blog post from Dave Cheney that predates Go modules: Gophers, please tag your releases
What do we want? Version management for Go packages! When do we want it? Yesterday!
[...] We want our Go build tool of choice to fetch the latest stable version when you start using the package in your project.[...]
But as it stands, today, in 2016, there is no way for a human, or a tool, to look at an arbitrary git (or mercurial, or bzr, etc) repository of Go code and ask questions like:
What versions of this project have been released?
What is the latest stable release of this software?
If I have version 1.2.3, is there a bugfix or security update that I should apply?
The reason for this is Go projects (repositories of Go packages) do not have versions, at least not in the way that our friends in other languages use that word. Go projects do not have versions because there is no formalised release process.
[...] I recommend that Go projects adopt SemVer 2.0.0. It’s a sound standard, it is well understood by many, not just Go programmers, and semantic versioning will let people write tools to build a dependency management ecosystem on top of a minimal release process.
After re-reading github.com/golang/go/wiki/…, I see the issue is not with the v2020.y.z, it is with the directory structure not matching the version number. path abc/def/v2020 v2020.y.z should work. Thank you for the various links. A good working example is github.com/jackc/pgx/v4
Here are the steps for the item you are going version:
When you get to the point where you want to change the version to V2 or higher, these are the steps that need to be followed.
Sample go.mod file before version change
module gitlab.com/soteapps/packages
go 1.14
require (
github.com/aws/aws-sdk-go v1.32.4
github.com/jackc/pgconn v1.5.0
github.com/jackc/pgx/v4 v4.6.0
golang.org/x/net v0.0.0-20200602114024-627f9648deb9 // indirect
golang.org/x/text v0.3.3 // indirect
)
Sample go.mod file after version change
module gitlab.com/soteapps/packages/v2100
go 1.14
require (
github.com/aws/aws-sdk-go v1.32.4
github.com/jackc/pgconn v1.5.0
github.com/jackc/pgx/v4 v4.6.0
golang.org/x/net v0.0.0-20200602114024-627f9648deb9 // indirect
golang.org/x/text v0.3.3 // indirect
)
Sample Golang file import before change
package sDatabase
import (
"context"
"encoding/json"
"fmt"
"strings"
"github.com/jackc/pgx/v4"
"github.com/jackc/pgx/v4/pgxpool"
"gitlab.com/soteapps/packages/sError"
"gitlab.com/soteapps/packages/sLogger"
)
const (
Sample Golang file import after change
package sDatabase
import (
"context"
"encoding/json"
"fmt"
"strings"
"github.com/jackc/pgx/v4"
"github.com/jackc/pgx/v4/pgxpool"
"gitlab.com/soteapps/packages/v2020/sError"
"gitlab.com/soteapps/packages/v2020/sLogger"
)
const (
Item having a version v2+ number:
Edit the go.mod module line to include the major version number at the end of the existing module path. So, the above module line module gitlab.com/soteapps/packages would change too module gitlab.com/soteapps/packages/v2100.
All import references to gitlab.com/soteapps/packages in all the *.go files in the project must be updated to gitlab.com/soteapps/packages/{version}. In our example, this would the following, gitlab.com/soteapps/packages/v2100.
Commit the change to source control.
If you are using master, then commit to master.
After you commit to master, create a branch from master with the version name. v2100 in the above example.
If you are using branch, then commit to a branch with the version name. v2100 in the above example.
The last step is to create a tag using semantic versioning format (vX.Y.Z) that point to the version branch. v2100.1.0 would be the tag for this example.