Service compiling successfully, but message structs not generating - gRPC/Go - protocol-buffers

I am using gRPC/protobufs as the protocol to communicate between my client and server, both written in go. I'm able to run the command show below to generate the cards.pb.go (server) and cards_grpc.pb.go (client) files without any problem. The server file is working perfectly, without any issues. The client file, however, does not seem to have access to the message items that I have defined within my cards.proto file. My services, as well as my client code, require the defined message struct in order to call the service methods, but I'm not sure what I'm missing.
Here is the command I'm running:
protoc -I="./protos" \
--go_out=plugins=grpc:./server \
--go-grpc_out=./client \
protos/*.proto
Here is my project file structure:
|-- client
|-- protos (generated protobufs for client)
|-- cards_grpc.pb.go (this compiled successfully, but structs representing my messages cannot be found)
|-- protos (This is where the proto files are defined)
|-- cards.proto
|-- server
|-- protos (generated protobufs for server)
|-- cards.pb.go (this is working perfectly, has compiled services and messages)
Note: I have defined option go_package = "./protos"; in my cards.proto file, which is why the generated files have outputted into */protos/*.pb.go locations

So you are not generating any protobuf related code for the client code here, only gRPC one. In order to generate the structure that you are looking for, use the following command:
protoc -I./protos \
--go_out=./server \
--go-grpc_out=./server \
--go_out=./client \
--go-grpc_out=./client \
protos/*.proto
The --go_out generates the go code for protobuf and the --go-grpc_out generates the go code for gRPC.
Another thing, --go_out=plugins=grpc are not supported in go anymore. You should use the --go-grpc_out.
More recommendations
I highly recommend to share the proto directory with both the client and the server (if possible), this limits the potential error due to unsynchronised Proto files.
So you would have something like:
|-- client
|-- protos
|-- cards.proto
|-- cards_grpc.pb.go
|-- cards.pb.go
|-- server
and then both access the files needed.
Second, if you are working with Go modules, I recommend that you use the go_package as following:
option go_package = "${YOUR_MODULE}/protos"
and then generate the code like this:
protoc -Iprotos \
--go_opt=module=${YOUR_MODULE} --go_out=. \
--go-grpc_opt=module=${YOUR_MODULE} --go-grpc_out=. \
protos/*.proto
Notice the . for the --go_out and --go-grpc_out. This maps the root of your project to the module name and this will generate the code inside your protos directory by removing the Module name to the go_package option. Then you will be able to access this generated code in your code like so:
import (
pb "${YOUR_MODULE}/protos"
)
Clarification
Just to be clear about the go_package, you need to understand one thing: the protobuf package and the go_package are not the same thing, the former defines package only usable in .proto files, the latter defines the package ... inside your go files. An example:
For Protobuf package
file1.proto
//...
package protos;
message Test {}
//...
file2.proto
//...
//no package definition
message Test2 {
protos.Test a_test = 1;
}
//...
For go_package
go.mod
module my_module
file1.proto (at location: ${ROOT}/protos)
//...
option go_package = "my_module/protos"
message Test {}
//...
generation
protoc -I./protos \
--go_out=./server \
--go-grpc_out=./server \
--go_out=./client \
--go-grpc_out=./client \
protos/file1.proto
main.go
package main
import (
pb "my_module/proto"
)
func main() {
var test pb.Test;
}

Related

Golang - package name as double underscore

In my protobuf generated go code, the package is:
package __
What does the double underscore mean, does it means the same as folder name ?
Is there a document for this, I searched but didn't found any. And the code can compile without error.
Yes it means the same directory. Let's look at the following code.
Directory Structure
.
├── go.mod
├── greet
│ └── greet.go
└── main.go
Content in greet.go
package __
import "fmt"
func Hello(name string) {
fmt.Printf("Hello %s\n", name)
}
Content in main.go
package main
import greet "playground/greet"
func main() {
greet.Hello("Eric")
}
Current Directory
$ pwd
/Users/thedatageek/Codes/go-playground
Unfortunately I also couldn't find any docs for go.
But it seems that its kinda good thing. You really don't need to name the package. You just name the directory and the package name would be automatically the same.
Note: This is definitely not the grpc or protobuf thing. It is however a customary that if you have generated proto stub from a proto file and if you add some additional utility file you may put those into a dir and then import it directly via directory name. For example the following github repos
https://github.com/Ash110/gRPC-Logger
https://github.com/dist1ll/cache-prototype
https://github.com/kamensotirov99/int-gateway
https://github.com/rachaelyychen/go-gee
https://github.com/suvvm/ToadOCREngine
https://github.com/denyami/drawing-api

How to use a package of generated protobuf inside a go module?

I encounter an issue with Go Module management and a generation of protobuffers (using go1.16, protoc-gen-go#latest).
I have this project structure:
subproj
├── go.mod (module company.tld/proj/subproj)
├── subproj.go (entry point : package main)
├── proto (folder containing .proto files)
├── packageFolder
| └── file1.go (package packageFolder)
└── Makefile (used to generate *.pb.go and build subproj binary)
The proto folder is used by other projects (obviously...) (via git submodule).
Protos are like the following:
syntax = "proto3"
option csharp_namespace = "Proj.Proto";
option go_package = "company.tld/proj/projpb";
package entity.proj
...
because of different version of messages, few protobuffer files need to be in another "namespace":
option go_package = "company.tld/proj/projpb/other";
package entity.proj.other
In my Makefile, I tried to generate the right *.pb.go at the right place:
# Proto sources
PROTO= $(wildcard ${PROTODIR}/*.proto)
PBGO= $(PROTO:.proto=.pb.go)
MODULE_NAME=company.tld/proj
GO_OPT_FLAG= --go_opt=module=${MODULE_NAME}
GRPC_OPT_FLAG= --go-grpc_opt=module=${MODULE_NAME}
#GO_OPT_FLAG= --go_opt=paths=import
#GRPC_OPT_FLAG= --go-grpc_opt=paths=import
.PHONY: clean install proto
## Builds the project
build: proto
go build ${LDFLAGS} -o ${BINARY}
$(PROTOBUF_GO_PLUGIN):
go install google.golang.org/protobuf/cmd/protoc-gen-go#latest
$(GRPC_GO_PLUGIN):
go install google.golang.org/grpc/cmd/protoc-gen-go-grpc#latest
%.pb.go: %.proto | $(PROTOBUF_GO_PLUGIN) $(GRPC_GO_PLUGIN)
protoc --proto_path=${PROTODIR} --go_out=. ${GO_OPT_FLAG} --go-grpc_out=. ${GRPC_OPT_FLAG} $<
proto: $(PBGO)
So, depending on the option used with the protoc compiler:
→ With --go_opt=paths=import
A folder tree company.tld/proj/projpb is created by protoc at the project's root. Each object is in a package called projpb or other, in the subpackage other.
Generated Proto objects, that include the other namespace-d objects, have the import path import other "company.tld/proj/projpb/other" (which is brought by the go_package option, but which is wrong because it is not an existing module - go mod tidy/vendor is complaining that it cannot find it).
Normal project files need the following import path to reach the Generated Proto objects :
import pb "company.tld/proj/subproj/company.tld/proj/projpb"
which seems odd and not the proper way to do.
→ With --go_opt=module=company.tld/proj
A folder projpb is created by protoc at the project's root and each generated .pb.go has the package projpb or other, in the subpackage other.
Generated Proto objects, that include the other namespace-d objects, still have the import path import other "company.tld/proj/projpb/other" (which is still brought by the go_package option and is still wrong because this is still a non-existing module - these are generated files... why would I want to create a module of these ?).
The cool thing is that with this go_opt, accessing generated types looks much more normal with
import pb "company.tld/proj/subproj/projpb".
Finally, I tried
using local import path on the go_package option in the .proto files (that is refused on build time, because there would be an import other "./projpb/other" in generated protobuffer object)
to use the replace instruction in the go.mod file like this :
replace (
company.tld/proj/projpb => ./projpb
company.tld/proj/projpb/other => ./projpb/other
)
(but go mod tidy/vendor is complaining that it cannot find the go.mod file inside the generated folder ./projpb)
Has someone encountered a similar problem? Or am I missing a command option to tell to Go, «I generate protobuffer objects in a package, or package in a package, and I simply want to use them. They are not a module, so please, provide the right import paths to the generated object and let me use them in my code».
[Update 01]
I gave a try to the go_opt=paths=source_relative (inspired by this ticket).
I created the folder in the Makefile, protoc generates files inside.
Notes:
generated protos use the full path, specified with the go_package option, to relate to one another.
As long as go_package option needs a full path, Go (go mod tidy/vendor) will want to search for a go.mod file inside the created folder, containing generated protos.
What is the correct way to tell Go that I am not looking for a Module, yet still satisfy the go_package option's full path constraint in the protobuffer file ?
After changing a numerous amount of time the go_package option in the proto files, changing the go_opt on the protoc compiler command, the only way I found to compile my project with my generated protobuffers, respecting every Go constraints, is by creating a go.mod file on-the-fly...
final proto «header» (respects the full puth in the go_package option)
syntax = "proto3";
option csharp_namespace = "Proj.Proto";
option go_package = "company.tld/proj/projpb";
// or for subpackages...
option csharp_namespace = "Proj.Proto.Other";
option go_package = "company.tld/proj/projpb/other";
my Makefile (creates a go.mod file for the generated proto files)
# Proto sources
PROTO= $(shell find ${PROTODIR} -type f -name '*.proto')
PBGO= $(PROTO:.proto=.pb.go)
DEST_DIR=.
MODULE_NAME=company.tld/proj
GO_OPT_FLAG= --go_opt=module=${MODULE_NAME}
GRPC_OPT_FLAG= --go-grpc_opt=module=${MODULE_NAME}
PROTO_PKG_DIR=projpb
PROTO_MODULE_NAME=${MODULE_NAME}/${PROTO_PKG_DIR}
PROTO_GOMOD_FILE=${PROTO_PKG_DIR}/go.mod
.PHONY: clean install proto gomod
build: proto gomod
go build ${LDFLAGS} -o ${BINARY}
%.pb.go: %.proto | $(PROTOBUF_GO_PLUGIN) $(GRPC_GO_PLUGIN) $(DEST_DIR)
${PROTOC} --proto_path=${PROTODIR} --go_out=${DEST_DIR} ${GO_OPT_FLAG} --go-grpc_out=${DEST_DIR} ${GRPC_OPT_FLAG} $<
proto: $(PBGO)
gomod: ${PROTO_GOMOD_FILE}
${PROTO_GOMOD_FILE}:
cd ${PROTO_PKG_DIR} && go mod init ${PROTO_MODULE_NAME} && cd ..
my main go.mod file (redirects the on-the-fly-created module to a local folder inside the project's scope)
module company.tld/proj/subproj
go 1.16
require (
// ...
company.tld/proj/projpb v0.0.0
)
replace company.tld/proj/projpb v0.0.0 => ./projpb
Thanks to the replace instruction, go mod tidy/vendor is happy and do not try to search the module in a remote repository.
Generated *.pb.go files have the right import path : company.tld/proj/projpb (and company.tld/proj/projpb/other for the subpackages).
And the import statement to use generated protos are working fine in the main project.
I hoped there was a simpler and more prettier solution, but alas...
Sorry for the any and thanks to those who gave it a thought !

protoc Go Package Command Local Overwrite?

Following the Quick Start gRPC Go guide on the official gRPC website it has a step which asks the user to recompile the updated .proto file using this command:
$ protoc --go_out=. --go_opt=paths=source_relative --go-grpc_out=. --go-grpc_opt=paths=source_relative helloworld/helloworld.proto
I'm a little confused as to how the newly compiled protobuf files are consumed by the "human written" Go code.
In the example "human written" Go code they reference the protobuf code using the following import:
pb "google.golang.org/grpc/examples/helloworld/helloworld"
protoc does not update this package but instead updates the helloworld/helloworld.proto in the same directory that the command is run in. How does the protoc command ensure that the "human written" Go code consumes the newly compiled protobuf code?
I find this one of the more confusing aspects of Protobufs (and gRPC).
I think the issue being solved is that Protobufs need to:
Permit namespacing to scope services|messages to e.g. DNS domains
Support a myriad programming languages (which implement namespacing differently).
protochas options that permit (re)mapping of e.g. a protobuf's package v1/api to a language-specific namespace using protobuf options, e.g. go_package). See Go Generated Code for Golang.
This is slightly more complex when using Go Modules but, in summary, what you're experiencing is probably some (unexpected) combination of the above where, the sample code assumes one module name and protoc is building on the assumption of a different one.
TL;DR update the code's module reference to reflect the correctly generated pb path. If the generate code is in the wrong place, you can simply move it to the correct subdirectory (path) but it's better to update your protoc command to generate the files to the correct directory.
Example
something.proto:
syntax = "proto3";
package v1alpha1;
option go_package = "github.com/me/my-protos;v1alpha1";
NOTE go_package aliases the proto package v1alpha1 to what I want to reference as github.com/me/my-protos in Golang.
Then I generate with:
MODULE="github.com/me/my-protos"
protoc \
--proto_path=. \
--go_out=./api/v1alpha1 \
--go_opt=module=${MODULE} \
--go-grpc_out=./api/v1alpha1 \
--go-grpc_opt=module=${MODULE} \
./something.proto
NOTE This example generate gRPC code too. It avoids (!) protoc creating a path github.com/me/my-protos for the generated code because I'm generating the source in that repo. I just want the relative path ./api/v1alpha where the files will be created.
Which yields:
my-protos
├── something.proto
├── api
│   └── v1alpha1
│   ├── something_grpc.pb.go
│   └── something.pb.go
├── go.mod
├── go.sum
└── README.md
And I can import with:
import (
pb "github.com/me/my-protos/api/v1alpha1"
)
NOTE From a different repo, I can now access my protos repo github.com/me/my-project/api/v1alpha1 combining the repro and the generated location to give my desired path.

Resolve Go package naming conflict when compiling protobuf definitions with different packages

I'm using protoc to create some DTOs. The definitions are in the following structure:
/protobuf
|-- common.proto
|-- /api
|-- /service
|-- csvdownload.proto
My csvdownload.proto looks like this:
syntax = "proto3";
package protobuf.api.service;
import "common.proto";
option go_package = ".;service"; // golang
message CsvExportRequest {
Common.Currency exportCurrency = 2;
Common.Decimal rounding = 3;
}
and the stub of common.proto looks like this:
syntax = "proto3";
package protobuf;
option go_package = ".;gopb"; // golang
I'm attempt to compile csvdownload.proto by running the following command from the /protobuf directory:
protoc --go_out=gopb --go_opt=paths=source_relative .\api\service\csvdownload.proto
However, I'm getting the following error:
protoc-gen-go: Go package "." has inconsistent names gopb (common.proto) and service (api/service/csvdownload.proto)
I assume this to mean that the code cannot be generated because common.proto and csvdownload.proto have declared different packages but I'm not sure that should make a difference and, from my understanding of how Protobuf works, it shouldn't hinder my ability to compile csvdownload.proto. What am I doing wrong here?
Any help in dealing with this issue would be greatly appreciated.
You must change your directory structure to something like this:
/protobuf
|-- /common
|-- common.proto
|-- /api
|-- /service
|-- csvdownload.proto
also, I suggest you add a real package address for common.proto
syntax = "proto3";
package protobuf;
option go_package = "myProject.com/proto/common"; // golang
then you can import this common to your other protos like this.
import "common/common.proto";
the generating code is(it is the full request you want(I think))
protoc --proto_path=../base_directory/protobuf/api/service/ --proto_path=../base_directory/protobuf/ --go_out=plugins=grpc:./the/generated/path/you/want/directory ../base_directory/protobuf/api/service/csvdownload.proto

protobuf golang import .proto and .pb.proto from different directories

I have a library called myProtos which looks like this
.
|-- proto
|---- hello.proto
|
|-- generated
└---- hello.pb.go
I have a .proto file outside called example.proto that should import hello.proto
So the top of the file looks like this:
syntax = "proto3";
package example;
import "path/to/myProtos/proto/hello.proto"
Now, when I compile example.proto I get an import error on example.pb.go because it has the import line import "path/to/myProtos/proto/hello.pb.go"
I tried adding both import paths, but I get an 'import but not used error'. I also tried doing relative imports and passing both directories as flags to protoc, which worked, but I need the import path in the go file to be absolute.
How can I tell protoc that on the go file the path is different?
Is there a better 'best practice' in this case?
What worked for me is to define option go_package = "github.com/<your-account>/<your-cool-project>/<sub-dirs>
Let's assume you have the folder structure your stated:
.
|-- proto
|---- hello.proto
|
|-- generated
└---- hello.pb.go
In your case you would add option go_package = "<path>/<in>/<GOPATH>/generated" to hello.proto.
What is important is that you then have to run
protoc -I. --go_out=$GOPATH ./*.proto
Generally, I would generate the go files alongside the proto files to keep the import paths the same for proto and go files. But this might be a matter of taste. In that case you would then simply set option go_package = "<path>/<in>/<GOPATH>/proto" to hello.proto.
In both cases a relative import of the .proto file should now resolve to the proper import in the generated Go code and the .pb.go files should also be put into the proper Go package folders.
Use package generated; inside your hello.proto file.
Then, protoc -I proto/ proto/*.proto --go_out=generated will generate a hello.pb.go inside generated folder by the package name of generated.
The package inside the proto files tells the protobuf generator which package to use inside the generated file.

Resources