I'm using rules_protobuf to build Python language bindings for my helloworld.proto file. My helloworld.proto imports wrappers.proto.
syntax = "proto3";
package main;
import "google/protobuf/wrappers.proto";
My BUILD file
load("#org_pubref_rules_protobuf//python:rules.bzl", "py_proto_compile")
# Wrapper around proto_compile.
# https://github.com/pubref/rules_protobuf/blob/master/protobuf/internal/proto_compile.bzl
py_proto_compile(
name = "py",
with_grpc = True,
protos = ["helloworld.proto"],
imports = ["/usr/local/home/username/myproject/include"]
)
The wrappers.proto file is located in the directory
/usr/local/home/username/myproject/include
The Bazel rule py_proto_compile is defined by rules_protobuf and documented in the README.md. imports is defined as:
Name: imports
Type: string_list`
Description: Optional paths to be passed as -I arguments to the protoc tool.
Default: []
My BUILD rules works, however I've hard-coded the location of of wrappers.proto with:
imports = ["/usr/local/home/username/myproject/include"]
Bazel doesn't appear to have any predefined Make variables that reference my WORKSPACE root. Ideally, I'd like to do something like this:
imports = ["$WORKSPACE_ROOT"/include"]
NEW UPDATE:
There is also a more elaborate way of doing it. Check this out. Have a look at the lines where it says:
# Grab a reference to the root of the users project
project_dir = ctx.path(ctx.attr.file_in_project).dirname
OLD ANSWER:
You can use __workspace_dir__ to get the path to the current working directory (i.e. the directory containing the WORKSPACE file).
Source: https://github.com/tensorflow/serving/blob/4d0a571ff9c15b937f58d3d5e97a5310b5decf2b/tensorflow_serving/workspace.bzl
Related
I have a .proto protobuf definition file in a dir and I'm building a go library from it with Bazel like so (BUILD.bazel file below generated using gazelle):
load("#rules_proto//proto:defs.bzl", "proto_library")
load("#io_bazel_rules_go//go:def.bzl", "go_library")
load("#io_bazel_rules_go//proto:def.bzl", "go_proto_library")
proto_library(
name = "events_proto",
srcs = ["events.proto"],
visibility = ["//visibility:public"],
deps = ["#com_google_protobuf//:timestamp_proto"],
)
go_proto_library(
name = "proto_go_proto",
importpath = "github.com/acme/icoyote/proto",
proto = ":events_proto",
visibility = ["//visibility:public"],
)
go_library(
name = "proto",
embed = [":proto_go_proto"],
importpath = "github.com/acme/icoyote/proto",
visibility = ["//visibility:public"],
)
Some other code depends on //icoyote/proto:proto, and when I run go mod tidy in my module, it complains that it can't find the package github.com/acme/icoyote/proto:
go: finding module for package github.com/acme/icoyote/proto
github.com/acme/icoyote/cmd/icoyote imports
github.com/acme/icoyote/proto: no matching versions for query "latest"
Any IDE that doesn't have Bazel integration (e.g. VSCode, GoLand/IntelliJ without the Bazel plugin) complains as well
What do I do?
This is happening of course because because Bazel does generate .go files using protoc under the covers for the go_proto_library rule in the BUILD file, but only writes them out to a temp dir under bazel-bin to be used by the go_library rule, and go mod tidy doesn't seem look into bazel-bin (probably because it's a symlink but also if it did, the path of those files relative to the location of go.mod is all wrong)
One option is to manually generate the go files by calling protoc on your own, and remove the proto_library and go_proto_library rules in the BUILD file, then change the go_library rule to build your generated files. This is suboptimal because you have to manually rerun protoc every time you make changes to the .proto file (and if you put it into a //go:generate directive, you have to rerun go generate).
Instead, we can do the following:
Add a file empty.go to the dir that contains the .proto file. It should look like this:
package proto
Then tell gazelle to ignore empty.go (so it doesn't try to add a go_library rule to the BUILD file when you run gazelle --fix). We do that by adding the following to the BUILD file:
# gazelle:exclude empty.go
That's enough to make go mod tidy shut up.
This will also make the IDE stop complaining about the import, although you'll still get errors when referring to anything that's supposed to be in that package. If you don't want to abandon your IDE for an excellent GoLand or IntelliJ IDEA with a Bazel plugin, you might have to resort to the manual protoc method. Perhaps there's a way to create a symlink to wherever Bazel writes out the generated .go files under bazel-bin and force go mod tidy to follow it, but I haven't tried that. If you do and it works, do share!
Not so familiar with Golang, it's probably a stupid mistake I made... But still, I can't for the life of me figure it out.
So, I got a proto3 file (let's call it file.proto), whose header is as follows:
syntax = "proto3";
package [package_name];
option go_package = "github.com/[user]/[repository]";
And I use protoc:
protoc --go_out=$GOPATH/src --go-grpc_out=$GOPATH/src file.proto
So far so good, I end up with two generated files (file.pb.go and file_grpc.pb.go) inside /go/src/github.com/[user]/[repository]/, and they are defined inside the package [package_name].
Then, the code I'm trying to build has the following import:
import (
"github.com/[user]/[repository]/[package_name]"
)
And I naively thought it would work. However, it produces the following error when running go mod tidy:
go: downloading github.com/[user]/[repository] v0.0.0-20211105185458-d7aab96b7629
go: finding module for package github.com/[user]/[repository]/[package_name]
example/xxx imports
github.com/[user]/[repository]/[package_name]: module github.com/[user]/[repository]#latest found (v0.0.0-20211105185458-d7aab96b7629), but does not contain package github.com/[user]/[repository]/[package_name]
Any idea what I'm doing wrong here? Go version is go1.19 linux/amd64 within Docker (golang:1.19-alpine).
Note: I also tried to only import github.com/[user]/[repository], same issue obviously.
UPDATE:
OK so what I do is that I get the proto file from the git repository that only contains the proto file:
wget https://raw.githubusercontent.com/[user]/[repository]/file.proto
Then I generate go files from that file with protoc:
protoc --go_out=. --go-grpc_out=. file.proto
Right now, in current directory, it looks like:
- directory
| - process.go
| - file.proto
| - github.com
| - [user]
| - [repository]
| - file.pb.go
| - file_grpc.pb.go
In that same directory, I run:
go mod init xxx
go mod tidy
CGO_ENABLED=0 go build process.go
The import directive in process.go is as follows:
import (
"xxx/github.com/[user]/[repository]"
)
Now it looks like it finds it, but still getting a gRPC error, which is weird because nothing changed. I still have to figure out if it comes from the issue above or not. Thanks!
Your question is really a number of questions in one; I'll try to provide some info that will help. The initial issue you had was because
At least one file with the .go extension must be present in a directory for it to be considered a package.
This makes sense because importing github.com/[user]/[repository] would be fairly pointless if that repository does not contain any .go files (i.e. the go compiler could not really do anything with the files).
Your options are:
Copy the output from protoc directly into your project folder and change the package declarations to match your package. If you do this there is no need for any imports.
Copy (or set go_out argument to protoc) the output from protoc into a subfolder of your project. The import path will then be the value of the module declaration in your go.mod plus the path from the folder that the go.mod is in (this is what you have done).
Store the files in a repo (on github or somewhere else). This does not need to be the same repo as your .proto files if you "want it to be agnostic" (note that 2 & 3 can be combined if the generated files will only be used within one code base or the repo is accessible to all users).
Option 1 is simple but its often beneficial to keep the generated code separate (makes it clear what you should not edit and improves editor autocomplete etc).
Option 2 is OK (especially if protoc writes the files directly and you set go_package appropriately). However issues may arise when the generated files will be used in multiple modules (e.g. as part of your customers code) and your repo is private. They will need to change go_package before running protoc (or search/replace the package declarations) and importing other .proto files may not work well.
Option 3 is probably the best approach in most situations because this works with the go tooling. You can create github.com/[user]/goproto (or similar) and put all of your generated code in there. To use this your customers just need to import github.com/[user]/goproto (no need to run protoc etc).
Go Modules/package intro
The go spec does not detail the format of import paths, leaving it up to the implementation:
The interpretation of the ImportPath is implementation-dependent but it is typically a substring of the full file name of the compiled package and may be relative to a repository of installed packages.
As you are using go modules (pretty much the default now) the implementations rules for resolving package paths (synonym of import path) can be summarised as:
Each package within a module is a collection of source files in the same directory that are compiled together. A package path is the module path joined with the subdirectory containing the package (relative to the module root). For example, the module "golang.org/x/net" contains a package in the directory "html". That package’s path is "golang.org/x/net/html".
So if your "module path" (generally the top line in a go.mod) is set to xxx (go mod init xxx) then you would import the package in subfolder github.com/[user]/[repository] with import xxx/github.com/[user]/[repository] (as you have found). If you got rid of the intervening folders and put the files into the [repository] subfolder (directly off your main folder) then it would be import xxx/[repository]
You will note in the examples above that the module names I used are paths to repo (as opposed to the xxx you used in go mod init xxx). This is intentional because it allows the go tooling to find the package when you import it from a different module. For example if you had used go mod init github.com/[user]/[repository] and option go_package = "github.com/[user]/[repository]/myproto";" then the generated files should go into the myproto folder in your project and you import them with import github.com/[user]/[repository]/myproto.
While you do not have to follow this approach I'd highly recommend it (it will save you from a lot of pain!). It can take a while to understand the go way of doing this, but once you do, it works well and makes it very clear where a package is hosted.
I am trying to browse code for gVisor with VScode.
However, unlike other Go projects, gVisor is built with bazel, and the source code is not located under $GOROOT or $GOPATH. Maybe this prevents gVisor packages from being searched by VSCode go extension. Go-to-definition is not working in most cases, except for cases where the definition can be found under the same directory.
How can I setup VSCode with bazel Go projects? Especially gVisor. Thanks!
The linked gVisor rule now proxies a more canonical implementation
This setup worked for me:
In your workspace's root BUILD file you can add the following build rule
# in BUILD.bazel
load("#io_bazel_rules_go//go:def.bzl", "go_path")
go_path(
name = "gopath",
mode = "link",
deps = [
"//my/binary/here",
"//any/other/binaries/you/want/linked",
],
)
(if you don't yet have a gazelle import for the bazelbuild/go_rules, you would need to import it for bazel)
# in WORKSPACE
http_archive(
name = "io_bazel_rules_go",
sha256 = "8e968b5fcea1d2d64071872b12737bbb5514524ee5f0a4f54f5920266c261acb",
urls = [
"https://mirror.bazel.build/github.com/bazelbuild/rules_go/releases/download/v0.28.0/rules_go-v0.28.0.zip",
"https://github.com/bazelbuild/rules_go/releases/download/v0.28.0/rules_go-v0.28.0.zip",
],
)
Build this command to create a symlinked folder in your blaze-out at bazel-bin/gopath that contains links to each of your dependencies. You'll have to do this any time you add a new dependency. You will see a line for each symlink created.
bazel build :gopath
(assuming you're using VSCode with the Golang extension) Set your workspace settings for the go extension to point to this gopath. Note you'll need to have it be a worspace trusted extension in order for this to work.
// In .vscode/settings.json
{
"go.gopath": "/YOUR ABSOLUTE PATH TO YOUR WORKSPACE//bazel-bin/gopath"
}
Restart VSCode
Enjoy!
NOTE: if you have a go.mod file in your root dir, this will not work.
gVisor recently added a gopath BUILD rule that creates a canonical GOPATH tree from the source.
You may be able to use that the edit more effectively from VScode.
In my WORKSPACE, I have defined the go_repostitory for importpath golang.org/x/net.
go_repository(
name = "org_golang_x_net",
commit = "5ccada7d0a7ba9aeb5d3aca8d3501b4c2a509fec",
importpath = "golang.org/x/net",
)
But when I run gazelle, the dependency added to BUILD.Bazel is not #org_golang_x_net//context:go_default_library. Instead, it is the following: //golang.org/x/net/context:go_default_library
Why does gazelle ignore the defined go_repository? Is there a way for it to consider deps defined in WORKSPACE?
I'm guessing from your previous question this is because you're using an empty prefix.
Gazelle resolves Go imports to Bazel labels in several steps:
If there is a go_library in your workspace with a matching importpath, Gazelle will use the name of that library.
If the import path is below the top-level prefix for your repository, the import will be considered "local", so you'll get a label like //golang.org/x/net/content:go_default_library.
Since you're using an empty prefix at the top level, this catch will catch every unknown import.
This should be improved. Gazelle's resolver should be aware of prefixes defined in different directories (currently, those are only used to determine the importpath attribute on generated rules). I've filed bazelbuild/bazel-gazelle#101 to fix that.
If the import is not considered local, Gazelle will generate a label for an external repository or for the vendor directory, depending on the external mode.
For now, you may want to add a directive to your top-level build file like:
# gazelle:prefix __do_not_match__
This will basically disable the second case, since no import will start with that. You'll still have an empty prefix in your src directory (set with # gazelle:prefix in src/BUILD.bazel), so your libraries will still have the correct importpath directives. You might want to set that to more specific subdirectories though.
I just started with Go, and i love it! I have tried to make the structure of my project a bit more manageable, instead of having everything in my main.go
So now i have a structure like this.
src/
-> main.go
-> routes.go
-> handlers/
--> user_handlers.go
But when i try to build this with the following command
go build -v -o ./bin/my_bin ./src/...
I get this error
cannot use -o with multiple packages
But if i make it a flat structure like this
src/
-> main.go
-> routes.go
-> user_handlers.go
It works just fine, all my files got "package main" in the top of them.
What am i doing wrong?
The package name must match the directory name. Moving a source file to a new directory requires that you also change the package name.
foo/foo.go // package foo
foo/bar/bar.go // package bar
foo/bar/qux.go // package bar
The PATH is not relevant in terms of the package name.
Package foo: /some/path/some/where/foo
This allows multiple "foo" packages to be created and imported provided your import specifies the desired location of "foo"
P.S. The convention for package names is lowercase, no punctuation (e.g., no _'s)
It tells you what you did wrong, you can't separate a single package over multiple folders.
You need to set and properly use $GOPATH and properly import your routes/ folder in routes.go.
A simple example of it is:
// routes.go
// the . means you can call imported functions without prefixing them with the package name
import . "full-path-to-routes/-relative-to-$GOPATH"
From https://golang.org/doc/code.html:
The GOPATH environment variable specifies the location of your workspace. It is likely the only environment variable you'll need to set when developing Go code.
To get started, create a workspace directory and set GOPATH accordingly. Your workspace can be located wherever you like, but we'll use $HOME/go in this document. Note that this must not be the same path as your Go installation.
I highly recommend reading Effective Go.