I am trying to have protoc-generated server interface and client implementation in separate packages
The header part of my .proto files is the following:
syntax = "proto3";
option go_package = "github.com/<username>/<myservice>/pkg/grpc";
And I am using this command to generate .go files:
protoc --go_out=. --go_opt=paths=source_relative\
--go-grpc_out=. --go-grpc_opt=paths=source_relative\
pkg/grpc/*.proto
It generates pkg/grpc/<name>.pb.go files containing models and pkg/grpc/<name>_grpc.pb.go files containing server interface and client implementation (picture)
But I want the server to go to, say internal/pkg/grpc/, while the client and the models remaining inside pkg/grpc/, and the server correctly importing the models.
Versions:
protoc version is libprotoc 3.19.0
protoc-gen-go-grpc version is protoc-gen-go-grpc 1.1.0
protoc-gen-go version is protoc-gen-go v1.27.1
I am new to golang and protobuf, so if whatever I am asking happens to be bad practice, feel free to point me to the idiomatic one
It seems there isn't an option to do this. The plugin protoc-gen-go-grpc writes the output service code to the same file with _grpc.pb.go suffix, where "service" includes both client and server code.
You can only define different output paths per plugin:
protoc-gen-go supports --go_out and --go_opt flags
protoc-gen-go-grpc supports --go-grpc_out and --go-grpc_opt flags
Related
I'm trying to make my Go project using hexagonal architecture as described here.
In my project I'm using a gRPC communication generated with protoc from .proto file.
The directories structure:
|- grpc.proto
|-internal
|-core
|-domain
|-services
|- grpcprotocol
And my grpc.proto file has go_package option that points to a specific directory in my Go project
syntax = "proto3";
option go_package = "github.com/myuser/myrepo/internal/core/services/grpcprotocol";
...
Using protoc --go_out=internal/core/domain --go_opt=paths=source_relative --go-grpc_out=internal/core/services/grpcprotocol --go-grpc_opt=paths=source_relative ports.proto I'm able to generate both grpc.pb.go file in internal/core/domain directory and grpc_grpc.pb.go file inside internal/core/services/grpcprotocol directory.
However, grpc.pb.go has a go package named grpcprotocol while it should have a package named domain (I also use other types defined there in separate Go files).
Also grpc_grpc.pb.go file has code that uses types defined in grpc.pb.go without imports (it treats it as if it was defined in the same package).
Is it possible to split those two files into separate Go packages and enforce code that's in grpc_grpc.pb.go to import types from domain directory instead of treating them as defined in the same package?
Your best solution here is too separate the code that you want in grpcprotocol and the one you want into domain, into separate files. Such as:
domain.proto
syntax = "proto3";
package domain;
option go_package = "github.com/myuser/myrepo/internal/core/domain";
//...
grpc.proto
syntax = "proto3";
package grpcprotocol;
option go_package = "github.com/myuser/myrepo/internal/core/services/grpcprotocol";
//...
Then you could import your domain.proto in your grpc.proto, by simply writing import "domain.proto";, provide a --proto_path if domain.proto and grpc.proto are not in the same directory. And finally to reference an object from domain.proto in grpc.proto you can write:
domain.AnObject
After that you can take advantage of the --go_opt=module and --go-grpc_opt=module, to strip the module name in go_package and generate the code at the right place. Like:
protoc --go_out=. --go_opt=module=github.com/myuser/myrepo --go-grpc_out=. --go-grpc_opt=module=github.com/myuser/myrepo *.proto
What this will do is that, it will remove github.com/myuser/myrepo from each go_package and basically start from the root of your module. That's why you can do a --go_out=. and --go-grpc_out=..
Hope that helps, let me know how I can further improve my answer.
Notes
Protobuf package and go_package are not the same. The former is only used for protobuf in order to give context and it extends the qualified name. The go_package is used during go code generation.
The package in the proto file is optional, it makes things more clear and nobody can misuse you proto file without specifying the fully qualified name (little bit safer, if named properly).
There is gRPC api https://github.com/LTD-Beget/antivirus/tree/master/proto/v1
I want to generate files for GO from this API, but there is no way to add go_package option.
In old version of protoc was cmd option as import_path, and it can can be generated for example by next command
root#d887087e9fa2:/shared/api# protoc -I/usr/local/include -I/shared/api --go_out=import_path=antivirus_proto_v1,M/shared/api/antivirus/proto/v1/antivirus.proto=antivirus_proto_v1,M/shared/api/antivirus/proto/v1/structures.proto=antivirus_proto_v1,plugins=grpc:internal/grpc /shared/api/antivirus/proto/v1/antivirus.proto /shared/api/antivirus/proto/v1/structures.proto
But with new version of protoc
protoc-gen-go: no such flag -import_path
--go_out: protoc-gen-go: Plugin failed with status code 1.
follow the guidelines
In order to generate Go code, the Go package's import path must be provided for every .proto file (including those transitively depended upon by the .proto files being generated). There are two ways to specify the Go import path:
by declaring it within the .proto file, or
by declaring it on the command line when invoking protoc.
but when i declaring it on cmd
root#d887087e9fa2:/shared/api# protoc -I/usr/local/include -I/shared/api --go_out=generated --go-grpc_out=generated --go_opt=M/shared/api/antivirus/proto/v1/antivirus.proto=antivirus/antivirus --go_opt=M/shared/api/antivirus/proto/v1/structures.proto=antivirus/structures /shared/api/antivirus/proto/v1/antivirus.proto /shared/api/antivirus/proto/v1/structures.proto
protoc-gen-go: unable to determine Go import path for "antivirus/proto/v1/structures.proto"
Please specify either:
• a "go_package" option in the .proto source file, or
• a "M" argument on the command line.
See https://developers.google.com/protocol-buffers/docs/reference/go-generated#package for more information.
what am i missing?
I am trying to import annotations into my proto file. I copied the "google" folder to "project / proto/google".
import "google/api/annotations.proto";
the path google / api / annotations.proto - the project is located.
I am tried use the go mode init and vendor, it didn`t help
In the proto file, only google/protobuf offers autocomplete and that's it, I don't see such a folder in GOPATH
What do I do to make it work?
Edit 1-
enter image description here
I use
protoc -I $GOPATH/src/github.com/jeka2708/test-grpc/proto/ --go_out=. --go_opt=paths=source_relative --go-grpc_out=. --go-grpc_opt=paths=source_relative proto/login.proto
google/api/annotations.proto is not part of the base standard of include files that comes with the protoc compiler distribution (see latest here).
To incorporate it in your protoc build - simply download the version you need (latest, you'll also need it's included dependency http.proto) - and place it in a directory who's suffix matches the import path i.e.
/SOME/DIRECTORY/PATH/google/api/annotations.proto
/SOME/DIRECTORY/PATH/google/api/http.proto
The prefix path could be the system default include path (/usr/include) e.g.
/usr/include/google/api/annotations.proto
/usr/include/google/api/http.proto
or you can place in some project directory:
/home/user/my_proj/google/api/annotations.proto
/home/user/my_proj/google/api/http.proto
and the protoc compiler will default to looking in the current working directory - or you can specify custom include paths via the command-line options:
Usage: ./protoc [OPTION] PROTO_FILES
Parse PROTO_FILES and generate output based on the options given:
-IPATH, --proto_path=PATH Specify the directory in which to search for
imports. May be specified multiple times;
directories will be searched in order. If not
given, the current working directory is used.
If not found in any of the these directories,
the --descriptor_set_in descriptors will be
checked for required proto file.
I have 2 proto compiled under different go packages but when I register them in a a server and run it, I get :
panic: proto: file "common.proto" is already registered
previously from: "github.com/soft/test-platform.go/common"
currently from: "github.com/soft/proto-asterix/asterix"
Here is common.proto in test-platform repository (in /api folder) :
syntax = "proto3";
package soft.testplatform.common; // here I do defint a UNIQUE package name !
option java_multiple_files = true;
option go_package = "github.com/soft/test-platform.go/common"; // Here I do define a unique go package name !
message MyMessage{
string commandId = 1;
}
As you can see, the package definition for go_package and package do not collide with package from github.com/soft/proto-asterix/asterix. Only the .proto filenames are similar (common.proto).
I generate go files with protoc an protoc-gen-go plugin using the following command :
protoc \
--proto_path=../test-platform/api/ \
--go_out=./common --go_opt=paths=source_relative \
../test-platform/api/common.proto
As per documentation here https://developers.google.com/protocol-buffers/docs/reference/go/faq#fix-namespace-conflict the package and filename should be appended to check for a registration conflict but it does not seem to be the case here.
Has anyone encountered such behavior ? Do I miss something obvious to resolve this package name collision ?
Here is what I tried :
Adding/removing package instruction to common.proto file
Change protoc command to use an absolute (and not relative) proto_path
Protoc version : libprotoc 3.15.7
Protoc go plugin version : protoc-gen-go v1.26.0
The accepted answer is not correct anymore. This commit reverted the "bugfix" as it was different from other gRPC implementations.
My only way to fix this was to rename the file/folder.
Thanks to #blackgreen suggestion, indeed, this was a bug fixed by https://go-review.googlesource.com/c/protobuf/+/301953/
While the next release of protoc-gen-go is out, here is a quick fix for your projects :
Use the fixed protoc-gen-go :
go install google.golang.org/protobuf/cmd/protoc-gen-go#febffdd
Change your imports in your go.mod to match
google.golang.org/protobuf v1.26.1-0.20210525005349-febffdd88e85
You should be good to go !
tl;dr A repo formerly configured to use GOPATH is now configured for Modules. All's good and better. However, protoc correctly (!) generates Golang code for protobufs defined within the repo in a github.com/path/to/repo/protos structure when I'd now prefer these to be generated in my sources, outside of GOPATH. I'm moving them to resolve this. Is there a better solution?
I have a GitHub repo. For the sake of discussion, let's call it github.com/acme/toolbox. In a subdirectory, I have protobuf files that include:
package acme.toolbox.v1;
option go_package = "github.com/acme/toolbox/protos";
When I was GOPATH'ing, all was well and protoc would generate Golang bindings in $GOPATH/src/github.com/acme/toolbox/protos and my code, importing pb "github.com/acme/toolbox/protos", would work.
Moving to Go Modules hasn't been pain-free but, the benefits outweigh the cost and I'm future-proofing myself and the code.
My issue is that I don't see how I can get protoc to generate the Golang bindings into my arbitrarily and outside of GOPATH located clone.
I'm moving the files after they're generated but this feels... inelegant:
cd ${TOOLBOX}
protoc \
--proto_path=./protos \
--go_out=plugins=grpc:/go/src
./protos/*.proto
mv ${GOPATH}/src/github.com/acme/toolbox/protos/*.go ${TOOLBOX}/protos
Is there a better solution?
The main point of the go_package option is to define what the go package name will be. With that said, it can behave differently depending on what you set it too.
If option go_package is defined to be a valid go package name (e.g. protos), protoc will generate the files in the folder defined by --go_out with that package name. If option go_package is instead a path (e.g. github.com/acme/toolbox/protos), protoc will create the folder structure defined relative to --go_out and place the files there with the package name being the same as the last folder name.
Unless I am mistaken in what you are wanting to do, you can change go_package to be:
option go_package = "protos";
and change your protoc invocation to be:
protoc \
--proto_path=./protos \
--go_out=plugins=grpc:${TOOLBOX}/protos
./protos/*.proto
Doing that, the generated files will be placed in ${TOOLBOX}/protos with the go package package protos.