I have a problem with the usage of commands go mod vendor and go mod download in a go module.
When I'm using go mod vendor, the generated go.sum has extra-lines. E.g, when I'm using go mod download, the go.sum contains:
github.com/cespare/xxhash/v2 v2.1.1/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
However if I remove the go.sum and run go mod vendor, I have 2 lines:
github.com/cespare/xxhash/v2 v2.1.1 h1:6MnRN8NT7+YBpUIWxHtefFZOKTAPgGjpQSxqLNn0+qY=
github.com/cespare/xxhash/v2 v2.1.1/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
This is problematic because if I commit the go.sum after running go mod vendor, any further go mod download (which may be used by someone else or in CI) will not work because of mismatching hashes which lead to a security issue.
Is there something I'm doing wrong in the usage of go mod vendor and go mod download? Why would they produce different hashes in the go.sum?
Here is the kind of error which instantly happens:
verifying xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx#v0.1.0: checksum mismatch
downloaded: h1:tIKKCv/bUyBNvVsB6YLo0Ds9ZFdGJ0FKkFun22nwvCI=
go.sum: h1:qta5K5jjJg+TnsD76tcFK7Bjf402WP9MIbPsJGU11Ms=
SECURITY ERROR
This download does NOT match an earlier download recorded in go.sum.
The bits may have been replaced on the origin server, or an attacker may
have intercepted the download attempt.
(Note: I replaced the name of the package by xxx because it carries my company name)
The problem was actually that the git repository was using git-lfs, but git-lfs was not available on either client or server side, which lead to incoherent hashes when calculated with and without git-lfs.
The fact that go mod download did not add go.sum entries for the downloaded modules was arguably a bug, and it will be fixed in the upcoming Go 1.16 release (see https://golang.org/issue/41341).
Related
Question
Can go mod tidy generate meaningful commit versions for local git repositories?
Background
I recently noticed that go mod tidy, when run from a local directory, generates a version using the "zero time" value from Golang, like so:
require example.com/db_api v0.0.0-00010101000000-000000000000
This seems reasonable for a local repository with no commits and no version tags... however, when I tried to add commits and tags to a local directory, I noticed that go mod tidy still relied on the zero timestamp as the identifier for the require statement.
Example
Let's start with a dependency:
➜ db_api git:(heads/v1.0.0) tree
.
├── go.mod
└── vars
└── constants.go
And a main repo:
➜ webapp git:(master) ✗ tree
.
├── go.mod
└── main.go
Now, if I add a create replace directive like so (such that the webapp/main.go depends on the constants.go file in the db_api directory...
replace example.com/db_api => ../db_api
Then the result of go mod tidy will always be the above "zero timestamp", as opposed to v1.0.0.
In contrast, running go mod tidy against a generic "GitHub" repo (i.e. cobra), results in a "tag" based version pin:
require (
example.com/db_api v0.0.0-00010101000000-000000000000
github.com/spf13/cobra v1.2.1
)
It seems to me like, the logic associated with remote git repositories (to get the latest semver tag), should be identical even for remote repositories. However, based on my experiments in here, it appears like local repositories aren't inspected for tags.
Hence, the original question... Can go mod tidy guess versioned data from a local repo in the same manner as it would for a remote repository, or is a local git repo fundamentally different in the way it is pinned in a go module?
Can go mod tidy generate meaningful commit versions for local git repositories?
No
Hence, the original question... Can go mod tidy guess versioned data from a local repo in the same manner as it would for a remote repository,
No
or is a local git repo fundamentally different in the way it is pinned in a go module?
Yes.
replace in go.mod means "use those files I dumped there as the module". The fact that you might manage these files with git or rsync or whatnot doesn't matter the slightest.
I'm new to golang. I'm currently trying to use the godror driver to read from an Oracle db. I ran go get github.com/godror/godror in my project's root directory and am importing the library like so:
_ "github.com/godror/godror"
But, I'm getting the error
cannot find package "." in:
/test_repo/vendor/github.com/godror/godror"
I believe my PATH is set up properly, as the "go" command properly returns the expected "Go is a tool for managing Go source code..." response.
I can't exactly replicate your issue nor have I seen such a weird error - but regardless, if you were following the current go modules pattern you wouldn't have this issue to begin with.
You shouldn't run go get anymore to download modules to use for your programs. Instead, in the root directory of every go project, you'll run go mod init [modulename], which will create a go.mod file for you. After running go mod tidy, it will download all the dependencies and generate a go.sum file containing the dependency hashes for you as well. Next, running go build will generate a binary that you can run. At this point, if you make changes to any source file(s), running go build every subsequent time afterwards will make a new, updated binary in the same directory.
Say you are using Go 1.13 and set up a project initialized for Go Modules.
$ mkdir my-project
$ cd my-project
$ git mod init github.com/bmuschko/my-project
Instead of using the default Google proxy to download dependencies, you set GoCenter or a different proxy.
$ export GOPROXY=https://gocenter.io
Once you download dependencies, it doesn't seem that Go keeps track of the originating proxy. In the end, there's no way to know or verify that a dependency came from the Google proxy, a custom proxy or directly from the source code repository. In theory, the checksums could be different depending on the originating proxy if you switch between them even if you pull the same version.
$ go get github.com/spf13/cobra
Does Go store this information somewhere in the cache? I couldn't find this information. Any advice would be appreciated.
The originating proxy should not matter and is not recorded: if you have downloaded the module from anywhere, then the bytes in your module cache should match the checksum found in either your go.sum file or the global checksum database.
(The go command fetches checksums for any new module dependencies from the database before downloading the module or adding the checksum to your go.sum file.)
For my project I am trying to get all the dependencies and sub dependencies of my project. I need to specific version of each of these dependencies. Not only do I need the dependencies of my project, but the dependencies of the dependencies and so on until the root.
For my project, go list -m all works for everything except indirect dependencies that have not opted into using go.mod files. Right now my workflow is taking an initial batch of repositories, downloading them from git then using “GO111MODULE=on go build ./…”. and “GO111MODULE=on go list -m -json all” to get the list of dependencies. I do not check for go.mod as all of the repositories I am scanning are using go.mod files.
For the list of dependencies that come out of this initial list I have some questions, for files without go.mod files, I used this as a reference: “https://blog.golang.org/using-go-modules”
-Path = Received from go list -m all, it can be GitHub, gopkg, or whatever is used to dl the go package.
Without go.mod
-“GO111MODULE=on go mod init <PATH from parent go.mod>”
-“GO111MODULE=on go build ./…”
-“GO111MODULE=on go mod tidy”
-“GO111MODULE=on go list -m -json all”
-From there I get a list of the dependencies of this module.
With go.mod
-“GO111MODULE=on go build ./…”
-“GO111MODULE=on go mod tidy”
-“GO111MODULE=on go list -m -json all”
Should I be running go build on each dependency that has a go.mod file? For ones without a go.mod file, I understand this should be done, as how else will we populate the go.mod file with the dependencies. But for files with a go.mod file, will I be pulling extra stuff that is not necessarily being used by my project with go build, like test files and other files that might not be used when I am simply importing that project? I understand that its better to get more unused dependencies rather than missing some, but it is getting a bit overwhelming with how massive the amount of dependencies is.
I can try to analyze go.sum file (when you execute go list -u, go.sum was created)
The go command uses the go.sum file to ensure that future downloads of these modules retrieve the same bits as the first download, to ensure the modules your project depends on do not change unexpectedly, whether for malicious, accidental, or other reasons. Both go.mod and go.sum should be checked into version control. (Using Go Modules - Adding a dependency)
go.sum file lists down the checksum (and version tag) of direct and indirect dependency required by the module.
% cat go.sum
...
github.com/bmizerany/perks v0.0.0-20141205001514-d9a9656a3a4b/go.mod h1:ac9efd0D1fsDb3EJvhqgXRbFx7bs2wqZ10HQPeU8U/Q=
github.com/c2h5oh/datasize v0.0.0-20171227191756-4eba002a5eae/go.mod h1:S/7n9copUssQ56c7aAgHqftWO4LTf4xY6CGWt8Bc+3M=
github.com/cespare/xxhash v1.1.0/go.mod h1:XrSqR1VqqWfGrhpAt58auRo0WTKS1nRRg3ghfAqPWnc=
github.com/client9/misspell v0.3.4/go.mod h1:qj6jICC3Q7zFZvVWo7KLAzC3yx5G7kyvSDkc90ppPyw=
github.com/cockroachdb/apd v1.1.0/go.mod h1:8Sl8LxpKi29FqWXR16WEFZRNSz3SoPzUzeMeY4+DwBQ=
...
The phrase “all the dependencies” is unfortunately ambiguous.
go list -m lists all modules whose selected versions are determined by your go.mod file. That is one possible definition of "all the dependencies", but it is a much broader set of modules than I think most people intend when they talk about the “dependencies” of a module.
In practice, go list -test all (without the -m flag) is the broadest set of dependencies I would generally care about: it includes all of the packages listed in import statements within your module (i.e. everything you need in order to run go test ./... within your module), plus all of the packages transitively needed to run go test on those packages.
(In particular, go list -test all is also the set of packages that will be resolved when you run go mod tidy.)
I have a program with source code hosted on GitHub that uses Go Modules introduced in go 1.11.
go.mod file describes my dependencies, but go.sum file seems to be a lockfile. Should I be adding go.sum to my repository or should I gitignore it?
https://github.com/golang/go/wiki/Modules#releasing-modules-all-versions:
Ensure your go.sum file is committed along with your go.mod file.
(Building on a previous answer.)
Yes, commit go.sum.
Ensure your go.sum file is committed along with your go.mod file. See FAQ below for more details and rationale.
From the FAQ:
Should I commit my 'go.sum' file as well as my 'go.mod' file?
Typically your module's go.sum file should be committed along with
your go.mod file.
go.sum contains the expected cryptographic checksums of the content of specific module versions.
If someone clones your repository and downloads your dependencies using the go command, they will receive an error if there is any
mismatch between their downloaded copies of your dependencies and the
corresponding entries in your go.sum.
In addition, go mod verify checks that the on-disk cached copies of module downloads still match the entries in go.sum.
Note that go.sum is not a lock file as used in some alternative dependency management systems. (go.mod provides enough information
for reproducible builds).
See very brief rationale here from
Filippo Valsorda on why you should check in your go.sum. See the
"Module downloading and
verification"
section of the tip documentation for more details. See possible future
extensions being discussed for example in
#24117 and
#25530.