I have multiple dependencies for an application. Is it possible to have some dependencies vendored(application vendored code local that I can add debugging when deployed) and some that are downloaded via go.mod/sum. When I attempt to do this the dependencies vendored and in the modules.txt get pulled down regardless. Am I missing a step? Do I need to update the imports, go.mod/sum additionally to prevent this?
vendor/modules.txt
# github.com/sendgrid/sendgrid-go v2.0.0+incompatible
github.com/sendgrid/sendgrid-go
# github.com/sendgrid/smtpapi-go v0.4.0
github.com/sendgrid/smtpapi-go
Related
I'd like to use Gazelle to manage my Go dependencies (and their dependencies) in Bazel. Running bazel run //:gazelle update-repos firebase.google.com/go adds a properly configured go_repository to my WORKSPACE file:
go_repository(
name = "com_google_firebase_go",
importpath = "firebase.google.com/go",
sum = "h1:3TdYC3DDi6aHn20qoRkxwGqNgdjtblwVAyRLQwGn/+4=",
version = "v3.13.0+incompatible",
)
However, this does not work out of the box. Running bazel build #com_google_firebase_go//:go_default_library returns an error:
ERROR: /private/var/tmp/_bazel_spencerconnaughton/9b09d78e8f2190e9af61aa37bcab571e/external/com_google_firebase_go/BUILD.bazel:3:11: no such package '#org_golang_google_api//option': The repository '#org_golang_google_api' could not be resolved and referenced by '#com_google_firebase_go//:go'
ERROR: Analysis of target '#com_google_firebase_go//:go_default_library' failed; build aborted: Analysis failed
INFO: Elapsed time: 0.596s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (23 packages loaded, 133 targets configured)
Is there a way to tell gazelle to load the #org_golang_google_api transitive dependency and others without needing to run update-repos for each one?
I have been struggling with this as well, but it seems that you can simply solve it by using go mod :)
If you use the go mod and go get commands to generate go.mod files, the transative dependencies are included automatically. You can then use this go.mod file in your bazel-gazelle command ;)
let's say your project is called github.com/somesampleproject:
go mod init github.com/somesampleproject
Then, use go get to add your dependency to the go.mod file:
go get firebase.google.com/go
go mod actually handles the transitive dependencies as well, as described in the documentation here. So your go.mod file should contain all of your required dependencies now :)
I quickly tried this out with gazelle in one of our projects (as described in the gazelle documentation)
bazel run //:gazelle -- update-repos -from_file=\<insert-subfolder-here>/go.mod
now our WORKSPACE file includes the firebase dependency, but also the dependencies that firebase has itself.
For bonus points you could use tools to validate your go.mod file to make sure you have no dependencies with known security bugs, a quick google search returned snyke, nancy and built-in support in ide's such as vs code apparently ;)
I work in a closed environment in which machines do not have access to internet. I'm wondering if there is a way for me to load the dependency modules for development with Go. One way I am thinking is to get the package files, if there is any. However, I couldn't find any such file for the dependencies I am using.
There are several things you could do:
You can use replace directives in your go.mod file to redirect packages to local directories where you have copies of the dependency packages.
You can use go mod vendor (see docs) to "vendor" dependency packages locally.
You can use a local proxy and set the GOPROXY env var.
Say you are using Go 1.13 and set up a project initialized for Go Modules.
$ mkdir my-project
$ cd my-project
$ git mod init github.com/bmuschko/my-project
Instead of using the default Google proxy to download dependencies, you set GoCenter or a different proxy.
$ export GOPROXY=https://gocenter.io
Once you download dependencies, it doesn't seem that Go keeps track of the originating proxy. In the end, there's no way to know or verify that a dependency came from the Google proxy, a custom proxy or directly from the source code repository. In theory, the checksums could be different depending on the originating proxy if you switch between them even if you pull the same version.
$ go get github.com/spf13/cobra
Does Go store this information somewhere in the cache? I couldn't find this information. Any advice would be appreciated.
The originating proxy should not matter and is not recorded: if you have downloaded the module from anywhere, then the bytes in your module cache should match the checksum found in either your go.sum file or the global checksum database.
(The go command fetches checksums for any new module dependencies from the database before downloading the module or adding the checksum to your go.sum file.)
For my project I am trying to get all the dependencies and sub dependencies of my project. I need to specific version of each of these dependencies. Not only do I need the dependencies of my project, but the dependencies of the dependencies and so on until the root.
For my project, go list -m all works for everything except indirect dependencies that have not opted into using go.mod files. Right now my workflow is taking an initial batch of repositories, downloading them from git then using “GO111MODULE=on go build ./…”. and “GO111MODULE=on go list -m -json all” to get the list of dependencies. I do not check for go.mod as all of the repositories I am scanning are using go.mod files.
For the list of dependencies that come out of this initial list I have some questions, for files without go.mod files, I used this as a reference: “https://blog.golang.org/using-go-modules”
-Path = Received from go list -m all, it can be GitHub, gopkg, or whatever is used to dl the go package.
Without go.mod
-“GO111MODULE=on go mod init <PATH from parent go.mod>”
-“GO111MODULE=on go build ./…”
-“GO111MODULE=on go mod tidy”
-“GO111MODULE=on go list -m -json all”
-From there I get a list of the dependencies of this module.
With go.mod
-“GO111MODULE=on go build ./…”
-“GO111MODULE=on go mod tidy”
-“GO111MODULE=on go list -m -json all”
Should I be running go build on each dependency that has a go.mod file? For ones without a go.mod file, I understand this should be done, as how else will we populate the go.mod file with the dependencies. But for files with a go.mod file, will I be pulling extra stuff that is not necessarily being used by my project with go build, like test files and other files that might not be used when I am simply importing that project? I understand that its better to get more unused dependencies rather than missing some, but it is getting a bit overwhelming with how massive the amount of dependencies is.
I can try to analyze go.sum file (when you execute go list -u, go.sum was created)
The go command uses the go.sum file to ensure that future downloads of these modules retrieve the same bits as the first download, to ensure the modules your project depends on do not change unexpectedly, whether for malicious, accidental, or other reasons. Both go.mod and go.sum should be checked into version control. (Using Go Modules - Adding a dependency)
go.sum file lists down the checksum (and version tag) of direct and indirect dependency required by the module.
% cat go.sum
...
github.com/bmizerany/perks v0.0.0-20141205001514-d9a9656a3a4b/go.mod h1:ac9efd0D1fsDb3EJvhqgXRbFx7bs2wqZ10HQPeU8U/Q=
github.com/c2h5oh/datasize v0.0.0-20171227191756-4eba002a5eae/go.mod h1:S/7n9copUssQ56c7aAgHqftWO4LTf4xY6CGWt8Bc+3M=
github.com/cespare/xxhash v1.1.0/go.mod h1:XrSqR1VqqWfGrhpAt58auRo0WTKS1nRRg3ghfAqPWnc=
github.com/client9/misspell v0.3.4/go.mod h1:qj6jICC3Q7zFZvVWo7KLAzC3yx5G7kyvSDkc90ppPyw=
github.com/cockroachdb/apd v1.1.0/go.mod h1:8Sl8LxpKi29FqWXR16WEFZRNSz3SoPzUzeMeY4+DwBQ=
...
The phrase “all the dependencies” is unfortunately ambiguous.
go list -m lists all modules whose selected versions are determined by your go.mod file. That is one possible definition of "all the dependencies", but it is a much broader set of modules than I think most people intend when they talk about the “dependencies” of a module.
In practice, go list -test all (without the -m flag) is the broadest set of dependencies I would generally care about: it includes all of the packages listed in import statements within your module (i.e. everything you need in order to run go test ./... within your module), plus all of the packages transitively needed to run go test on those packages.
(In particular, go list -test all is also the set of packages that will be resolved when you run go mod tidy.)
I have a program with source code hosted on GitHub that uses Go Modules introduced in go 1.11.
go.mod file describes my dependencies, but go.sum file seems to be a lockfile. Should I be adding go.sum to my repository or should I gitignore it?
https://github.com/golang/go/wiki/Modules#releasing-modules-all-versions:
Ensure your go.sum file is committed along with your go.mod file.
(Building on a previous answer.)
Yes, commit go.sum.
Ensure your go.sum file is committed along with your go.mod file. See FAQ below for more details and rationale.
From the FAQ:
Should I commit my 'go.sum' file as well as my 'go.mod' file?
Typically your module's go.sum file should be committed along with
your go.mod file.
go.sum contains the expected cryptographic checksums of the content of specific module versions.
If someone clones your repository and downloads your dependencies using the go command, they will receive an error if there is any
mismatch between their downloaded copies of your dependencies and the
corresponding entries in your go.sum.
In addition, go mod verify checks that the on-disk cached copies of module downloads still match the entries in go.sum.
Note that go.sum is not a lock file as used in some alternative dependency management systems. (go.mod provides enough information
for reproducible builds).
See very brief rationale here from
Filippo Valsorda on why you should check in your go.sum. See the
"Module downloading and
verification"
section of the tip documentation for more details. See possible future
extensions being discussed for example in
#24117 and
#25530.