Is it possible to automatically load transitive dependencies with Gazelle? - go

I'd like to use Gazelle to manage my Go dependencies (and their dependencies) in Bazel. Running bazel run //:gazelle update-repos firebase.google.com/go adds a properly configured go_repository to my WORKSPACE file:
go_repository(
name = "com_google_firebase_go",
importpath = "firebase.google.com/go",
sum = "h1:3TdYC3DDi6aHn20qoRkxwGqNgdjtblwVAyRLQwGn/+4=",
version = "v3.13.0+incompatible",
)
However, this does not work out of the box. Running bazel build #com_google_firebase_go//:go_default_library returns an error:
ERROR: /private/var/tmp/_bazel_spencerconnaughton/9b09d78e8f2190e9af61aa37bcab571e/external/com_google_firebase_go/BUILD.bazel:3:11: no such package '#org_golang_google_api//option': The repository '#org_golang_google_api' could not be resolved and referenced by '#com_google_firebase_go//:go'
ERROR: Analysis of target '#com_google_firebase_go//:go_default_library' failed; build aborted: Analysis failed
INFO: Elapsed time: 0.596s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (23 packages loaded, 133 targets configured)
Is there a way to tell gazelle to load the #org_golang_google_api transitive dependency and others without needing to run update-repos for each one?

I have been struggling with this as well, but it seems that you can simply solve it by using go mod :)
If you use the go mod and go get commands to generate go.mod files, the transative dependencies are included automatically. You can then use this go.mod file in your bazel-gazelle command ;)
let's say your project is called github.com/somesampleproject:
go mod init github.com/somesampleproject
Then, use go get to add your dependency to the go.mod file:
go get firebase.google.com/go
go mod actually handles the transitive dependencies as well, as described in the documentation here. So your go.mod file should contain all of your required dependencies now :)
I quickly tried this out with gazelle in one of our projects (as described in the gazelle documentation)
bazel run //:gazelle -- update-repos -from_file=\<insert-subfolder-here>/go.mod
now our WORKSPACE file includes the firebase dependency, but also the dependencies that firebase has itself.
For bonus points you could use tools to validate your go.mod file to make sure you have no dependencies with known security bugs, a quick google search returned snyke, nancy and built-in support in ide's such as vs code apparently ;)

Related

go build -buildvcs does not insert vcs.revision buildinfo

This question poses problems that are distinct from those discussed here: How do you read debug VCS version info from a Go 1.18 binary?
I am having trouble getting go build -buildvcs=true to insert correct version information when building executables out of a git repo that has the following structure:
go version go1.18.4 linux/amd64
in project:
go.mod
cmd/exe1/main.go
cmd/exe2/main.go
pkg/pkg1/...
pkg/pkg2/...
(1) If I cd project;go build -buildvcs=true -o /tmp/exe1 cmd/exe1/main.go then the BuildInfo included in the exe includes Deps entries for all the dependencies of all the packages, but there is no embedded Setting with key vcs.revision, and the Dep entry for the module named in go.mod is (devel). I guess this latter issue is related to how to specify versions for modules, which I have not yet looked into, and therefore I assume it's using a default value.
(2) If I cd project/cmd/exe1;go build -buildvcs=true -o /tmp/exe1 (leaving out any relative path specifying what to build) then the BuildInfo included in the exe does NOT include Deps entries but DOES include the vcs.revision
Questions:
Is there any way to get both Deps and vcs.revision into BuildInfo?
Is this directory structure ok? The documentation for this stuff is not in "reference" format, and many important details are spread out throughout all of the tutorials and howtos. Quite frustrating to get to the bottom of these behaviors.
It seems to me go should embed a vcs.revision whenever generating an executable, but before I open a bug, I wanted to get community feedback on whether this is expected behavior when specifying a relative target on the command line. I've seen that that can be an issue in general, with go build.
Any pointers to the right place to read a comprehensive guide about this would be great.

Go mod vendored dependencies being downloaded not ignored

I have multiple dependencies for an application. Is it possible to have some dependencies vendored(application vendored code local that I can add debugging when deployed) and some that are downloaded via go.mod/sum. When I attempt to do this the dependencies vendored and in the modules.txt get pulled down regardless. Am I missing a step? Do I need to update the imports, go.mod/sum additionally to prevent this?
vendor/modules.txt
# github.com/sendgrid/sendgrid-go v2.0.0+incompatible
github.com/sendgrid/sendgrid-go
# github.com/sendgrid/smtpapi-go v0.4.0
github.com/sendgrid/smtpapi-go

How to fix envoyproxy's validate rules proto dependency in Go repository with bazel build tool

I am trying to use bazel and gazelle tools to build a large Golang repository with pre-generated proto files. There are issues in using validate rules from envoyproxy in proto definitions and the bazel tool fails to build repository with this dependency. To reproduce the issue, I have created a github repository with a sample gRPC service. Without envoyproxy's validate proto dependency, the bazel tool succesfully builds the go packages. But introducing this dependency throws error.
Commands used to build the repository:
Generate proto files before building repository.
protoc -I=. -I ~/go/pkg/mod/github.com/envoyproxy/protoc-gen-validate#v0.6.1 --go_out=./../../../ --validate_out="lang=go:./../../../" --go-grpc_out=./../../../ ./protos/pinger/service.proto
Use gazelle to generate dependencies
bazel run //:gazelle -- update-repos -from_file=go.mod -to_macro=deps.bzl%go_dependencies -build_file_proto_mode=disable_global -prune
Use gazelle to generate BUILD.bazel files
bazel run //:gazelle
Build pinger go binary
bazel build //cmd/pinger
The last build command throws an error
go/src/github.com/ihtkas/bazel-gazelle-ex/api/pinger/BUILD.bazel:3:11:
error loading package
'#com_github_envoyproxy_protoc_gen_validate//validate': Unable to find
package for #com_google_protobuf//:protobuf.bzl: The repository
'#com_google_protobuf' could not be resolved. and referenced by
'//api/pinger:pinger' ERROR: Analysis of target '//cmd/pinger:pinger'
failed; build aborted: Analysis failed
There is a github issue for the same. There are multiple suggestions and none of them seems to work for me. Explaining all the attempts will make this question useless. I need help in finding one solution which will solve the issue and build the repository successfully with bazel tool with pre-generated proto files. Based on suggestion, I will experiment and share the results here or in the same repository.
The issue can be fixed by following three steps.
-> Use specific commit from envoyproxy/protoc-gen-validate with the bazel naming convention fix
go get github.com/envoyproxy/protoc-gen-validate#4f41f10dde19ec4dd9dd8cf6f7537139f52fb58b
-> Include dependency for "com_google_protobuf" in bazel WORKSPACE
http_archive(
name = "com_google_protobuf",
sha256 = "d0f5f605d0d656007ce6c8b5a82df3037e1d8fe8b121ed42e536f569dec16113",
strip_prefix = "protobuf-3.14.0",
urls = [
"https://mirror.bazel.build/github.com/protocolbuffers/protobuf/archive/v3.14.0.tar.gz",
"https://github.com/protocolbuffers/protobuf/archive/v3.14.0.tar.gz",
],
)
load("#com_google_protobuf//:protobuf_deps.bzl", "protobuf_deps")
protobuf_deps()
-> Update io_bazel_rules_go to v0.28.0
All the fixes are included in a PR and merged to a branch "working" in the repository for reference.

Getting all dependencies of a Go project

For my project I am trying to get all the dependencies and sub dependencies of my project. I need to specific version of each of these dependencies. Not only do I need the dependencies of my project, but the dependencies of the dependencies and so on until the root.
For my project, go list -m all works for everything except indirect dependencies that have not opted into using go.mod files. Right now my workflow is taking an initial batch of repositories, downloading them from git then using “GO111MODULE=on go build ./…”. and “GO111MODULE=on go list -m -json all” to get the list of dependencies. I do not check for go.mod as all of the repositories I am scanning are using go.mod files.
For the list of dependencies that come out of this initial list I have some questions, for files without go.mod files, I used this as a reference: “https://blog.golang.org/using-go-modules”
-Path = Received from go list -m all, it can be GitHub, gopkg, or whatever is used to dl the go package.
Without go.mod
-“GO111MODULE=on go mod init <PATH from parent go.mod>”
-“GO111MODULE=on go build ./…”
-“GO111MODULE=on go mod tidy”
-“GO111MODULE=on go list -m -json all”
-From there I get a list of the dependencies of this module.
With go.mod
-“GO111MODULE=on go build ./…”
-“GO111MODULE=on go mod tidy”
-“GO111MODULE=on go list -m -json all”
Should I be running go build on each dependency that has a go.mod file? For ones without a go.mod file, I understand this should be done, as how else will we populate the go.mod file with the dependencies. But for files with a go.mod file, will I be pulling extra stuff that is not necessarily being used by my project with go build, like test files and other files that might not be used when I am simply importing that project? I understand that its better to get more unused dependencies rather than missing some, but it is getting a bit overwhelming with how massive the amount of dependencies is.
I can try to analyze go.sum file (when you execute go list -u, go.sum was created)
The go command uses the go.sum file to ensure that future downloads of these modules retrieve the same bits as the first download, to ensure the modules your project depends on do not change unexpectedly, whether for malicious, accidental, or other reasons. Both go.mod and go.sum should be checked into version control. (Using Go Modules - Adding a dependency)
go.sum file lists down the checksum (and version tag) of direct and indirect dependency required by the module.
% cat go.sum
...
github.com/bmizerany/perks v0.0.0-20141205001514-d9a9656a3a4b/go.mod h1:ac9efd0D1fsDb3EJvhqgXRbFx7bs2wqZ10HQPeU8U/Q=
github.com/c2h5oh/datasize v0.0.0-20171227191756-4eba002a5eae/go.mod h1:S/7n9copUssQ56c7aAgHqftWO4LTf4xY6CGWt8Bc+3M=
github.com/cespare/xxhash v1.1.0/go.mod h1:XrSqR1VqqWfGrhpAt58auRo0WTKS1nRRg3ghfAqPWnc=
github.com/client9/misspell v0.3.4/go.mod h1:qj6jICC3Q7zFZvVWo7KLAzC3yx5G7kyvSDkc90ppPyw=
github.com/cockroachdb/apd v1.1.0/go.mod h1:8Sl8LxpKi29FqWXR16WEFZRNSz3SoPzUzeMeY4+DwBQ=
...
The phrase “all the dependencies” is unfortunately ambiguous.
go list -m lists all modules whose selected versions are determined by your go.mod file. That is one possible definition of "all the dependencies", but it is a much broader set of modules than I think most people intend when they talk about the “dependencies” of a module.
In practice, go list -test all (without the -m flag) is the broadest set of dependencies I would generally care about: it includes all of the packages listed in import statements within your module (i.e. everything you need in order to run go test ./... within your module), plus all of the packages transitively needed to run go test on those packages.
(In particular, go list -test all is also the set of packages that will be resolved when you run go mod tidy.)

Gitlab CI - Auto DevOps job failed, Unable to select a buildpack for a Go project?

My Gitlab CI Auto DevOps job failed with
Status: Downloaded newer image for gliderlabs/herokuish:latest
-----> Unable to select a buildpack
ERROR: Job failed: exit code 1
I've gone through
Auto DevOps and
Getting started with Auto DevOps
and am still not sure where I should put the buildpack.
Mine should be heroku-buildpack-go, which I've lost track where I get that from.
My repo consist only a single-character README.md, and the "Hello, playground" main.go.
Conclusion:
Thanks to David's comprehensive explanation, I was able to get DevOps started with correct buildpack:
From this I would conclude that your single .go file at the root of the directory tree does not meet the activation criteria for auto-building Go projects. I'd suggest picking one of the dependency managers in the requirements above and modifying your project to support it.
FTA, I just did touch go.mod then git add & git push and the AutoDevops started building my GO project indeed.
However it seems to me that Gitlab AutoDevops is not able to build any GO projects very easily, as I get the following error (with project variable TRACE=true):
...
!! The go.mod file for this project does not specify a Go version
!!
!! Defaulting to go1.11.1
!!
!! For more details see: https://devcenter.heroku.com/articles/go-apps-with-modules#build-configuration
!!
-----> Installing go1.11.1
-----> Fetching go1.11.1.linux-amd64.tar.gz... done
!! Installing package '.' (default)
!!
!! To install a different package spec add a comment in the following form to your `go.mod` file:
!! // +heroku install ./cmd/...
!!
!! For more details see: https://devcenter.heroku.com/articles/go-apps-with-modules#build-configuration
!!
-----> Running: go install -v -tags heroku .
go: cannot determine module path for source directory /tmp/build (outside GOPATH, no import comments)
ERROR: Job failed: exit code 1
The easier solution is to use .gitlab-ci.yml file instead, documented in
https://blog.boatswain.io/post/build-go-project-with-gitlab-ci/
(and followed up at Gitlab CI - Start Shared Runner for normal repos).
From the AutoDevops documentation:
Auto Build creates a build of the application in one of two ways:
If there is a Dockerfile, it will use docker build to create a Docker image.
Otherwise, it will use Herokuish and Heroku buildpacks to automatically detect and build the application into a Docker image.
Then looking at the build activation criteria, as per the Heroku Go buildpack documentation:
This buildpack will detect your repository as Go if you are using either:
go modules
dep
govendor
glide
GB
Godep
Or more specifically for godep, govendor or GB:
The Heroku Go buildpack is used when an application meets one of the following requirements:
has a Godeps/Godeps.json file, identifying the application as being managed by godep;
has a vendor/vendor.json file, identifying the application as being managed by govendor;
has a src directory, which has sub directories, contains one or more .go files, identifying the application as being managed by gb.
From this I would conclude that your single .go file at the root of the directory tree does not meet the activation criteria for auto-building Go projects. I'd suggest picking one of the dependency managers in the requirements above and modifying your project to support it. After that AutoDevops should start building your project.
If you are still having issues after that, this debugging note might help:
After making sure that the project meets the buildpack requirements;
if it still fails, setting a project variable TRACE=true will enable verbose logging which​ may help to continue troubleshooting.

Resources