tl;dr A repo formerly configured to use GOPATH is now configured for Modules. All's good and better. However, protoc correctly (!) generates Golang code for protobufs defined within the repo in a github.com/path/to/repo/protos structure when I'd now prefer these to be generated in my sources, outside of GOPATH. I'm moving them to resolve this. Is there a better solution?
I have a GitHub repo. For the sake of discussion, let's call it github.com/acme/toolbox. In a subdirectory, I have protobuf files that include:
package acme.toolbox.v1;
option go_package = "github.com/acme/toolbox/protos";
When I was GOPATH'ing, all was well and protoc would generate Golang bindings in $GOPATH/src/github.com/acme/toolbox/protos and my code, importing pb "github.com/acme/toolbox/protos", would work.
Moving to Go Modules hasn't been pain-free but, the benefits outweigh the cost and I'm future-proofing myself and the code.
My issue is that I don't see how I can get protoc to generate the Golang bindings into my arbitrarily and outside of GOPATH located clone.
I'm moving the files after they're generated but this feels... inelegant:
cd ${TOOLBOX}
protoc \
--proto_path=./protos \
--go_out=plugins=grpc:/go/src
./protos/*.proto
mv ${GOPATH}/src/github.com/acme/toolbox/protos/*.go ${TOOLBOX}/protos
Is there a better solution?
The main point of the go_package option is to define what the go package name will be. With that said, it can behave differently depending on what you set it too.
If option go_package is defined to be a valid go package name (e.g. protos), protoc will generate the files in the folder defined by --go_out with that package name. If option go_package is instead a path (e.g. github.com/acme/toolbox/protos), protoc will create the folder structure defined relative to --go_out and place the files there with the package name being the same as the last folder name.
Unless I am mistaken in what you are wanting to do, you can change go_package to be:
option go_package = "protos";
and change your protoc invocation to be:
protoc \
--proto_path=./protos \
--go_out=plugins=grpc:${TOOLBOX}/protos
./protos/*.proto
Doing that, the generated files will be placed in ${TOOLBOX}/protos with the go package package protos.
Related
I successfully used rules_go to build a gRPC service:
go_proto_library(
name = "processor_go_proto",
compilers = ["#io_bazel_rules_go//proto:go_grpc"],
importpath = "/path/to/proto/package",
proto = ":processor_proto",
deps = ["//services/shared/proto/common:common_go_proto"],
)
However, I'm not sure how to import the resulting file in VSCode. The generated file is nested under bazel_bin and under the original proto file path; so to import this, it seems like I would need to write out the entire path (including the bazel_bin part) to the generated Go file. To my understanding, there doesn't seem to be a way to instruct VSCode to look under certain folders that only contain Go packages/files; everything seems to need a go.mod file. This makes it quite difficult to develop in.
For clarity, my directory structure looks something like this:
WORKSPACE
bazel-bin
- path
- to
- generated_Go_file.go
go.mod
go.sum
proto
- path
- to
- gRPC_proto.proto
main.go
main.go should use the generated_Go_file.go.
Is there a way around this?
I don't use Bazel and so cannot help with the Bazel configuration. It's likely there is a way to specify the generated code location so that you can revise this to reflect you preference.
The outline you provide of the generated code, is workable though and a common pattern. Often the generated proto|gRPC code is placed in a module's gen subdirectory.
This is somewhat similar to vendoring where your code incorporates what may often be a 3rd-party's stubs (client|server) into your code. The stubs must reflect the proto(s) package(s) and, when these are 3rd-party, using gen or bazel-bin provide a way to keep potentially multiple namespaces discrete.
You're correct that the import for main.go, could (!) be prefixed with the module name from go.mod (first line) followed by the folder path to the generated code. This is standard go packaging and treats the generated code in a similar way to vendored modules.
Another approach is to use|place the generated code in a different module.
For code generated from 3rd-party protos, this may be preferable and the generated code may be provided by the 3rd-party in a module that you can go get or add to your go.mod.
An example of this approach is Google Well-Known Types. The proto (sources) are bundled with protoc (lib directory) and, when protoc compiles sources that references any of these, the Go code that is generated includes imports that reference a Google-hosted location of the generated code (!) for these types (google.golang.org/protobuf/types/known).
Alternatively, you can replicate this behavior without having to use an external repo. The bazel-bin folder must be outside of the current module. Each distinct module in bazel-bin, would need its own go.mod file. You would include in a require block in your code's go.mod file references to the modules' (one or more) locations. You don't need to publish the module to a external repo but can simply require ( name => path/to/module ) to provide a local reference.
Not so familiar with Golang, it's probably a stupid mistake I made... But still, I can't for the life of me figure it out.
So, I got a proto3 file (let's call it file.proto), whose header is as follows:
syntax = "proto3";
package [package_name];
option go_package = "github.com/[user]/[repository]";
And I use protoc:
protoc --go_out=$GOPATH/src --go-grpc_out=$GOPATH/src file.proto
So far so good, I end up with two generated files (file.pb.go and file_grpc.pb.go) inside /go/src/github.com/[user]/[repository]/, and they are defined inside the package [package_name].
Then, the code I'm trying to build has the following import:
import (
"github.com/[user]/[repository]/[package_name]"
)
And I naively thought it would work. However, it produces the following error when running go mod tidy:
go: downloading github.com/[user]/[repository] v0.0.0-20211105185458-d7aab96b7629
go: finding module for package github.com/[user]/[repository]/[package_name]
example/xxx imports
github.com/[user]/[repository]/[package_name]: module github.com/[user]/[repository]#latest found (v0.0.0-20211105185458-d7aab96b7629), but does not contain package github.com/[user]/[repository]/[package_name]
Any idea what I'm doing wrong here? Go version is go1.19 linux/amd64 within Docker (golang:1.19-alpine).
Note: I also tried to only import github.com/[user]/[repository], same issue obviously.
UPDATE:
OK so what I do is that I get the proto file from the git repository that only contains the proto file:
wget https://raw.githubusercontent.com/[user]/[repository]/file.proto
Then I generate go files from that file with protoc:
protoc --go_out=. --go-grpc_out=. file.proto
Right now, in current directory, it looks like:
- directory
| - process.go
| - file.proto
| - github.com
| - [user]
| - [repository]
| - file.pb.go
| - file_grpc.pb.go
In that same directory, I run:
go mod init xxx
go mod tidy
CGO_ENABLED=0 go build process.go
The import directive in process.go is as follows:
import (
"xxx/github.com/[user]/[repository]"
)
Now it looks like it finds it, but still getting a gRPC error, which is weird because nothing changed. I still have to figure out if it comes from the issue above or not. Thanks!
Your question is really a number of questions in one; I'll try to provide some info that will help. The initial issue you had was because
At least one file with the .go extension must be present in a directory for it to be considered a package.
This makes sense because importing github.com/[user]/[repository] would be fairly pointless if that repository does not contain any .go files (i.e. the go compiler could not really do anything with the files).
Your options are:
Copy the output from protoc directly into your project folder and change the package declarations to match your package. If you do this there is no need for any imports.
Copy (or set go_out argument to protoc) the output from protoc into a subfolder of your project. The import path will then be the value of the module declaration in your go.mod plus the path from the folder that the go.mod is in (this is what you have done).
Store the files in a repo (on github or somewhere else). This does not need to be the same repo as your .proto files if you "want it to be agnostic" (note that 2 & 3 can be combined if the generated files will only be used within one code base or the repo is accessible to all users).
Option 1 is simple but its often beneficial to keep the generated code separate (makes it clear what you should not edit and improves editor autocomplete etc).
Option 2 is OK (especially if protoc writes the files directly and you set go_package appropriately). However issues may arise when the generated files will be used in multiple modules (e.g. as part of your customers code) and your repo is private. They will need to change go_package before running protoc (or search/replace the package declarations) and importing other .proto files may not work well.
Option 3 is probably the best approach in most situations because this works with the go tooling. You can create github.com/[user]/goproto (or similar) and put all of your generated code in there. To use this your customers just need to import github.com/[user]/goproto (no need to run protoc etc).
Go Modules/package intro
The go spec does not detail the format of import paths, leaving it up to the implementation:
The interpretation of the ImportPath is implementation-dependent but it is typically a substring of the full file name of the compiled package and may be relative to a repository of installed packages.
As you are using go modules (pretty much the default now) the implementations rules for resolving package paths (synonym of import path) can be summarised as:
Each package within a module is a collection of source files in the same directory that are compiled together. A package path is the module path joined with the subdirectory containing the package (relative to the module root). For example, the module "golang.org/x/net" contains a package in the directory "html". That package’s path is "golang.org/x/net/html".
So if your "module path" (generally the top line in a go.mod) is set to xxx (go mod init xxx) then you would import the package in subfolder github.com/[user]/[repository] with import xxx/github.com/[user]/[repository] (as you have found). If you got rid of the intervening folders and put the files into the [repository] subfolder (directly off your main folder) then it would be import xxx/[repository]
You will note in the examples above that the module names I used are paths to repo (as opposed to the xxx you used in go mod init xxx). This is intentional because it allows the go tooling to find the package when you import it from a different module. For example if you had used go mod init github.com/[user]/[repository] and option go_package = "github.com/[user]/[repository]/myproto";" then the generated files should go into the myproto folder in your project and you import them with import github.com/[user]/[repository]/myproto.
While you do not have to follow this approach I'd highly recommend it (it will save you from a lot of pain!). It can take a while to understand the go way of doing this, but once you do, it works well and makes it very clear where a package is hosted.
I'm trying to make my Go project using hexagonal architecture as described here.
In my project I'm using a gRPC communication generated with protoc from .proto file.
The directories structure:
|- grpc.proto
|-internal
|-core
|-domain
|-services
|- grpcprotocol
And my grpc.proto file has go_package option that points to a specific directory in my Go project
syntax = "proto3";
option go_package = "github.com/myuser/myrepo/internal/core/services/grpcprotocol";
...
Using protoc --go_out=internal/core/domain --go_opt=paths=source_relative --go-grpc_out=internal/core/services/grpcprotocol --go-grpc_opt=paths=source_relative ports.proto I'm able to generate both grpc.pb.go file in internal/core/domain directory and grpc_grpc.pb.go file inside internal/core/services/grpcprotocol directory.
However, grpc.pb.go has a go package named grpcprotocol while it should have a package named domain (I also use other types defined there in separate Go files).
Also grpc_grpc.pb.go file has code that uses types defined in grpc.pb.go without imports (it treats it as if it was defined in the same package).
Is it possible to split those two files into separate Go packages and enforce code that's in grpc_grpc.pb.go to import types from domain directory instead of treating them as defined in the same package?
Your best solution here is too separate the code that you want in grpcprotocol and the one you want into domain, into separate files. Such as:
domain.proto
syntax = "proto3";
package domain;
option go_package = "github.com/myuser/myrepo/internal/core/domain";
//...
grpc.proto
syntax = "proto3";
package grpcprotocol;
option go_package = "github.com/myuser/myrepo/internal/core/services/grpcprotocol";
//...
Then you could import your domain.proto in your grpc.proto, by simply writing import "domain.proto";, provide a --proto_path if domain.proto and grpc.proto are not in the same directory. And finally to reference an object from domain.proto in grpc.proto you can write:
domain.AnObject
After that you can take advantage of the --go_opt=module and --go-grpc_opt=module, to strip the module name in go_package and generate the code at the right place. Like:
protoc --go_out=. --go_opt=module=github.com/myuser/myrepo --go-grpc_out=. --go-grpc_opt=module=github.com/myuser/myrepo *.proto
What this will do is that, it will remove github.com/myuser/myrepo from each go_package and basically start from the root of your module. That's why you can do a --go_out=. and --go-grpc_out=..
Hope that helps, let me know how I can further improve my answer.
Notes
Protobuf package and go_package are not the same. The former is only used for protobuf in order to give context and it extends the qualified name. The go_package is used during go code generation.
The package in the proto file is optional, it makes things more clear and nobody can misuse you proto file without specifying the fully qualified name (little bit safer, if named properly).
I encounter an issue with Go Module management and a generation of protobuffers (using go1.16, protoc-gen-go#latest).
I have this project structure:
subproj
├── go.mod (module company.tld/proj/subproj)
├── subproj.go (entry point : package main)
├── proto (folder containing .proto files)
├── packageFolder
| └── file1.go (package packageFolder)
└── Makefile (used to generate *.pb.go and build subproj binary)
The proto folder is used by other projects (obviously...) (via git submodule).
Protos are like the following:
syntax = "proto3"
option csharp_namespace = "Proj.Proto";
option go_package = "company.tld/proj/projpb";
package entity.proj
...
because of different version of messages, few protobuffer files need to be in another "namespace":
option go_package = "company.tld/proj/projpb/other";
package entity.proj.other
In my Makefile, I tried to generate the right *.pb.go at the right place:
# Proto sources
PROTO= $(wildcard ${PROTODIR}/*.proto)
PBGO= $(PROTO:.proto=.pb.go)
MODULE_NAME=company.tld/proj
GO_OPT_FLAG= --go_opt=module=${MODULE_NAME}
GRPC_OPT_FLAG= --go-grpc_opt=module=${MODULE_NAME}
#GO_OPT_FLAG= --go_opt=paths=import
#GRPC_OPT_FLAG= --go-grpc_opt=paths=import
.PHONY: clean install proto
## Builds the project
build: proto
go build ${LDFLAGS} -o ${BINARY}
$(PROTOBUF_GO_PLUGIN):
go install google.golang.org/protobuf/cmd/protoc-gen-go#latest
$(GRPC_GO_PLUGIN):
go install google.golang.org/grpc/cmd/protoc-gen-go-grpc#latest
%.pb.go: %.proto | $(PROTOBUF_GO_PLUGIN) $(GRPC_GO_PLUGIN)
protoc --proto_path=${PROTODIR} --go_out=. ${GO_OPT_FLAG} --go-grpc_out=. ${GRPC_OPT_FLAG} $<
proto: $(PBGO)
So, depending on the option used with the protoc compiler:
→ With --go_opt=paths=import
A folder tree company.tld/proj/projpb is created by protoc at the project's root. Each object is in a package called projpb or other, in the subpackage other.
Generated Proto objects, that include the other namespace-d objects, have the import path import other "company.tld/proj/projpb/other" (which is brought by the go_package option, but which is wrong because it is not an existing module - go mod tidy/vendor is complaining that it cannot find it).
Normal project files need the following import path to reach the Generated Proto objects :
import pb "company.tld/proj/subproj/company.tld/proj/projpb"
which seems odd and not the proper way to do.
→ With --go_opt=module=company.tld/proj
A folder projpb is created by protoc at the project's root and each generated .pb.go has the package projpb or other, in the subpackage other.
Generated Proto objects, that include the other namespace-d objects, still have the import path import other "company.tld/proj/projpb/other" (which is still brought by the go_package option and is still wrong because this is still a non-existing module - these are generated files... why would I want to create a module of these ?).
The cool thing is that with this go_opt, accessing generated types looks much more normal with
import pb "company.tld/proj/subproj/projpb".
Finally, I tried
using local import path on the go_package option in the .proto files (that is refused on build time, because there would be an import other "./projpb/other" in generated protobuffer object)
to use the replace instruction in the go.mod file like this :
replace (
company.tld/proj/projpb => ./projpb
company.tld/proj/projpb/other => ./projpb/other
)
(but go mod tidy/vendor is complaining that it cannot find the go.mod file inside the generated folder ./projpb)
Has someone encountered a similar problem? Or am I missing a command option to tell to Go, «I generate protobuffer objects in a package, or package in a package, and I simply want to use them. They are not a module, so please, provide the right import paths to the generated object and let me use them in my code».
[Update 01]
I gave a try to the go_opt=paths=source_relative (inspired by this ticket).
I created the folder in the Makefile, protoc generates files inside.
Notes:
generated protos use the full path, specified with the go_package option, to relate to one another.
As long as go_package option needs a full path, Go (go mod tidy/vendor) will want to search for a go.mod file inside the created folder, containing generated protos.
What is the correct way to tell Go that I am not looking for a Module, yet still satisfy the go_package option's full path constraint in the protobuffer file ?
After changing a numerous amount of time the go_package option in the proto files, changing the go_opt on the protoc compiler command, the only way I found to compile my project with my generated protobuffers, respecting every Go constraints, is by creating a go.mod file on-the-fly...
final proto «header» (respects the full puth in the go_package option)
syntax = "proto3";
option csharp_namespace = "Proj.Proto";
option go_package = "company.tld/proj/projpb";
// or for subpackages...
option csharp_namespace = "Proj.Proto.Other";
option go_package = "company.tld/proj/projpb/other";
my Makefile (creates a go.mod file for the generated proto files)
# Proto sources
PROTO= $(shell find ${PROTODIR} -type f -name '*.proto')
PBGO= $(PROTO:.proto=.pb.go)
DEST_DIR=.
MODULE_NAME=company.tld/proj
GO_OPT_FLAG= --go_opt=module=${MODULE_NAME}
GRPC_OPT_FLAG= --go-grpc_opt=module=${MODULE_NAME}
PROTO_PKG_DIR=projpb
PROTO_MODULE_NAME=${MODULE_NAME}/${PROTO_PKG_DIR}
PROTO_GOMOD_FILE=${PROTO_PKG_DIR}/go.mod
.PHONY: clean install proto gomod
build: proto gomod
go build ${LDFLAGS} -o ${BINARY}
%.pb.go: %.proto | $(PROTOBUF_GO_PLUGIN) $(GRPC_GO_PLUGIN) $(DEST_DIR)
${PROTOC} --proto_path=${PROTODIR} --go_out=${DEST_DIR} ${GO_OPT_FLAG} --go-grpc_out=${DEST_DIR} ${GRPC_OPT_FLAG} $<
proto: $(PBGO)
gomod: ${PROTO_GOMOD_FILE}
${PROTO_GOMOD_FILE}:
cd ${PROTO_PKG_DIR} && go mod init ${PROTO_MODULE_NAME} && cd ..
my main go.mod file (redirects the on-the-fly-created module to a local folder inside the project's scope)
module company.tld/proj/subproj
go 1.16
require (
// ...
company.tld/proj/projpb v0.0.0
)
replace company.tld/proj/projpb v0.0.0 => ./projpb
Thanks to the replace instruction, go mod tidy/vendor is happy and do not try to search the module in a remote repository.
Generated *.pb.go files have the right import path : company.tld/proj/projpb (and company.tld/proj/projpb/other for the subpackages).
And the import statement to use generated protos are working fine in the main project.
I hoped there was a simpler and more prettier solution, but alas...
Sorry for the any and thanks to those who gave it a thought !
I am fairly certain this is operator error and I am at the point I am not thinking clearly.
Here is the setup:
$GOPATH/src/github.com/<company>/<service a>/proto/a.proto
$GOPATH/src/github.com/<company>/<service b>/proto/b.proto
etc.
Now in the proto file I am using imports similar to go (perhaps the issue) such that a.proto has:
import "github.com/<company>/<service b>/b.proto"
I have possibly two separate issues.
I cannot get the import to compile properly using go:generate protoc
I cannot get the output a.pb.go file to be placed in the $GOPATH/src/github.com/<company>/<service a>/proto/ path.
I have attempted multiple configurations probably not in the correct combination.
Using option go_package = "github.com/<company>/<service b>/proto" in each .proto file
Multiple variations of go generate;
go:generate protoc --proto_path=.:$GOPATH/src --go_out=$GOPATH/src a.proto
go:generate protoc --proto_path=.:$GOPATH/src --go_out=. a.proto
go:generate protoc --go_out=import_prefix=github.com/<company>/:. api.proto
I clearly have a poor understanding on how protoc looks at import paths and file outputs. Anyone point me in the direction of what I am doing wrong?
Thanks!
Update #1
In a.proto;
option go_package = "github.com/<company>/<service a>/proto";
import "github.com/<company>/<service b>/proto/b.proto";
and the go generate;
//go:generate protoc --proto_path=$GOPATH/src --go_out=$GOPATH/src/github.com/<company>/<service a>/proto a.proto
Which is called from a go file in the proto directory with the a.proto.
I received the error;
a.proto: File does not reside within any path specified using --proto_path (or -I). You must specify a --proto_path ch encompasses this file. Note that the proto_path must be an exact prefix of the .proto file names -- protoc is too dumb to figure out when two paths (e.g. absolute and relative) are equivalent (it's harder than you think).
I have confirmed $GOPATH is to the expected location.
Solution
Thanks to Shivam Jindal for pointing me in the correct direction. The import is exactly as described in his solution. The output was a problem of my misusing both --go_out and option go_package. I used the go_package to specify the output location and --go_out to specify the root similar to --proto_path. Now everything works.
option go_package = "github.com/<company>/<service a>/proto";
and
//go:generate protoc --proto_path=$GOPATH/src/ --go_out=$GOPATH/src/ $GOPATH/src/github.com/<company>/<service a>/proto/a.proto
Thanks!
Firstly, option go_package is not meant for other dependency import at all, it's the Go package name where the new proto bindings for Go (a.pb.go file) will be placed.
Now coming to the output file location, I can see you are using go-generate. Firstly it depends from which directory you are invoking that if the path used in --go_out= is relative path. I would say use absolute paths. If you want to put the output file in that location you mentioned, use --go_out=$GOPATH/src/github.com/<company>/<service a>/proto/ in go-generate.
To correctly import the other file b.proto in your a.proto use the fully qualified import path as you have done. Just that use --proto_path $GOPATH/src in go-generate. Also please update the question with the errors you are seeing in case it does not work.
Please see this for more information on import paths.