When I generate c++ output files using protoc with only one "main" proto file without any imports, similar to the protobuf examples everything is generated fine and works when using it in my project.
But when I work with any form of imports, c++ output code is faulty. I compile my .proto with:
protoc --cpp_out=../grpc --grpc_out=../grpc --plugin=protoc-gen-grpc=/usr/local/bin/grpc_cpp_plugin sample.proto
This is my sample code that works:
syntax = "proto3";
service SampleService
{
rpc Login(LoginRequest) returns (LoginResponse) {};
}
message LoginRequest {}
message LoginResponse {}
When I try to outsource a single message, protoc still compiles without errors but is generating faulty cc and h files.
E.g. adding
import "messages.proto
with content
syntax = "proto3";
message LoginRequest
{
}
My c++ code wont compile anymore and contains faulty stuff.
My sample.pb.h contains and unresolved inclusion
#include "messages.pb.h
And my sample.pb.cc contains two unresolved functions
::protobuf_messages_2eproto::InitDefaults();
::protobuf_messages_2eproto::AddDescripttors();
Edit:
It seems I have to compile each .proto I already import too. But then I'm confused.
I want to have a "main" .proto which includes a service.proto and a messages.proto which contains imports to all my messages, which are also .proto-files . From time to time I want to create new messages so I can include them easily instead of one big unreadable file.
How do i do this? I don't want to have 200 output cc/h files for my little project. How is that intended?
My folder structure:
*src-Folder*
--main.proto (imports service.proto and messages.proto)
--service.proto (imports messages.proto)
--messages.proto (imports /messages/*
--*messages-Folder*
-----messagepair1
-----messagepair2
...
Edit2:
I wrote myself a workarround until there is a better solution against that file cluttering. I edited my .proto files and removed every description then the messages itself (imports, syntax, package...) use "cat" to concatenate each .proto file into one single file and compile this with protoc as posted above.
Not nice but I dont want 50-100 cc/h files later in my project. There have to be a better resolution.
Related
I have two .proto files (a.proto and b.proto) having the same contents in them:
syntax = "proto3";
message env {
string broker = 1;
}
When I execute the compiler (I want to generate Java source files) and specify both files on the command line
protoc.exe --java_out=. a.proto b.proto
I get error messages:
b.proto:4:10: "env.broker" is already defined in file "a.proto".
b.proto:3:9: "env" is already defined in file "a.proto".
I'd expect the compiler to generate two Java classes (A and B), each having a nested class Env. This is how I understand the docs. But this does not happen.
What am I doing wrong?
Thank you for any hints.
b.proto:4:10: "env.broker" is already defined in file "a.proto".
b.proto:3:9: "env" is already defined in file "a.proto".
Most protobuf libraries put all messages with same package name into same namespace. The Java library is special in generating the outer class, which is needed because Java does not allow multiple top-level classes per file.
Protoc is doing you a service checking the rule here. If you went ahead and used the same message name in multiple files without specifying a package, you would make it very difficult for other people to use your .proto files with different programming languages.
From here:
https://developers.google.com/protocol-buffers/docs/reference/go-generated#package
It is written:
There is no correlation between the Go import path and the package
specifier in the .proto file. The latter is only relevant to the
protobuf namespace, while the former is only relevant to the Go
namespace. Also, there is no correlation between the Go import path
and the .proto import path.
So, if I understand correctly - the "package" in your .proto file is just for importing other .proto's whereas the "option go_package" is what determines the package name generated into your .go files.
My question is why was this done? If you have the following in your .proto file:
syntax = "proto3";
package cool.app.v1;
option go_package = "github.com/some-user/cool-app/pkg/app/proto";
message Test{}
Then in other proto files you can reference Test using cool.app.v1.Test. But, then in the generated golang file - you end up with the following package (that last string after the last "/"):
package proto
Why wouldn't protoc just use the package specified in the .proto file? Are there other languages generated by protoc that require the same "option" override for the package name?
I successfully used rules_go to build a gRPC service:
go_proto_library(
name = "processor_go_proto",
compilers = ["#io_bazel_rules_go//proto:go_grpc"],
importpath = "/path/to/proto/package",
proto = ":processor_proto",
deps = ["//services/shared/proto/common:common_go_proto"],
)
However, I'm not sure how to import the resulting file in VSCode. The generated file is nested under bazel_bin and under the original proto file path; so to import this, it seems like I would need to write out the entire path (including the bazel_bin part) to the generated Go file. To my understanding, there doesn't seem to be a way to instruct VSCode to look under certain folders that only contain Go packages/files; everything seems to need a go.mod file. This makes it quite difficult to develop in.
For clarity, my directory structure looks something like this:
WORKSPACE
bazel-bin
- path
- to
- generated_Go_file.go
go.mod
go.sum
proto
- path
- to
- gRPC_proto.proto
main.go
main.go should use the generated_Go_file.go.
Is there a way around this?
I don't use Bazel and so cannot help with the Bazel configuration. It's likely there is a way to specify the generated code location so that you can revise this to reflect you preference.
The outline you provide of the generated code, is workable though and a common pattern. Often the generated proto|gRPC code is placed in a module's gen subdirectory.
This is somewhat similar to vendoring where your code incorporates what may often be a 3rd-party's stubs (client|server) into your code. The stubs must reflect the proto(s) package(s) and, when these are 3rd-party, using gen or bazel-bin provide a way to keep potentially multiple namespaces discrete.
You're correct that the import for main.go, could (!) be prefixed with the module name from go.mod (first line) followed by the folder path to the generated code. This is standard go packaging and treats the generated code in a similar way to vendored modules.
Another approach is to use|place the generated code in a different module.
For code generated from 3rd-party protos, this may be preferable and the generated code may be provided by the 3rd-party in a module that you can go get or add to your go.mod.
An example of this approach is Google Well-Known Types. The proto (sources) are bundled with protoc (lib directory) and, when protoc compiles sources that references any of these, the Go code that is generated includes imports that reference a Google-hosted location of the generated code (!) for these types (google.golang.org/protobuf/types/known).
Alternatively, you can replicate this behavior without having to use an external repo. The bazel-bin folder must be outside of the current module. Each distinct module in bazel-bin, would need its own go.mod file. You would include in a require block in your code's go.mod file references to the modules' (one or more) locations. You don't need to publish the module to a external repo but can simply require ( name => path/to/module ) to provide a local reference.
I'm trying to make my Go project using hexagonal architecture as described here.
In my project I'm using a gRPC communication generated with protoc from .proto file.
The directories structure:
|- grpc.proto
|-internal
|-core
|-domain
|-services
|- grpcprotocol
And my grpc.proto file has go_package option that points to a specific directory in my Go project
syntax = "proto3";
option go_package = "github.com/myuser/myrepo/internal/core/services/grpcprotocol";
...
Using protoc --go_out=internal/core/domain --go_opt=paths=source_relative --go-grpc_out=internal/core/services/grpcprotocol --go-grpc_opt=paths=source_relative ports.proto I'm able to generate both grpc.pb.go file in internal/core/domain directory and grpc_grpc.pb.go file inside internal/core/services/grpcprotocol directory.
However, grpc.pb.go has a go package named grpcprotocol while it should have a package named domain (I also use other types defined there in separate Go files).
Also grpc_grpc.pb.go file has code that uses types defined in grpc.pb.go without imports (it treats it as if it was defined in the same package).
Is it possible to split those two files into separate Go packages and enforce code that's in grpc_grpc.pb.go to import types from domain directory instead of treating them as defined in the same package?
Your best solution here is too separate the code that you want in grpcprotocol and the one you want into domain, into separate files. Such as:
domain.proto
syntax = "proto3";
package domain;
option go_package = "github.com/myuser/myrepo/internal/core/domain";
//...
grpc.proto
syntax = "proto3";
package grpcprotocol;
option go_package = "github.com/myuser/myrepo/internal/core/services/grpcprotocol";
//...
Then you could import your domain.proto in your grpc.proto, by simply writing import "domain.proto";, provide a --proto_path if domain.proto and grpc.proto are not in the same directory. And finally to reference an object from domain.proto in grpc.proto you can write:
domain.AnObject
After that you can take advantage of the --go_opt=module and --go-grpc_opt=module, to strip the module name in go_package and generate the code at the right place. Like:
protoc --go_out=. --go_opt=module=github.com/myuser/myrepo --go-grpc_out=. --go-grpc_opt=module=github.com/myuser/myrepo *.proto
What this will do is that, it will remove github.com/myuser/myrepo from each go_package and basically start from the root of your module. That's why you can do a --go_out=. and --go-grpc_out=..
Hope that helps, let me know how I can further improve my answer.
Notes
Protobuf package and go_package are not the same. The former is only used for protobuf in order to give context and it extends the qualified name. The go_package is used during go code generation.
The package in the proto file is optional, it makes things more clear and nobody can misuse you proto file without specifying the fully qualified name (little bit safer, if named properly).
I am running gosec on my project but it gives following error due to which our pipeline is failing
Golang errors in file: [/home/usr/exmaple/project.go]:
[line 22 : column 8] - could not import C (no metadata for C)
While go build command successfully builds the project means there is no golang error.
This is not an issue.
We found out that our C code is present in .c and .h files in package directory, where .h file containing only function prototypes. And we are including only .h file in go files (cgo).
go build and go test commands are working because both of these commands when see .c and .h in package directory, compiles and link them.
But gosec is a SAST tool, it doesn't compile .c and .h files present in package directory due to this gosec scan was getting only function prototypes not definition and was throwing Golang error could not import C (no metadata for C).
Including the whole code in comments (CGO) worked. Now gosec works fine.
This seems to be a bug encoutered by other people as well. See this issue or this one. You should probably report an issue to gosec as well for this. Or maybe about the go/packages package in the Go repo directly.