Is there a good way to integrate protobuf compiles with the go build command?
goprotobuf "provides Go support, in the form of a library and protocol compiler plugin, for Google's protocol buffers".
The README at the goprotobuf library has some good info.
From https://code.google.com/p/goprotobuf/source/browse/README#106 :
Consider file test.proto, containing
package example;
enum FOO { X = 17; };
message Test {
required string label = 1;
optional int32 type = 2 [default=77];
repeated int64 reps = 3;
optional group OptionalGroup = 4 {
required string RequiredField = 5;
}
}
To build a package from test.proto and some other Go files, write a Makefile like this:
include $(GOROOT)/src/Make.$(GOARCH)
TARG=path/to/example
GOFILES=\
test.pb.go\
other.go
include $(GOROOT)/src/Make.pkg
include $(GOROOT)/src/pkg/code.google.com/p/goprotobuf/Make.protobuf
Related
I need to use Protocol Buffers to serialize JSON messages received from the Google Drive drives:list method and write them to the BigQuery Storage Write API (GRPC). This is working for all field types except timestamp. I cannot for the life of me generate go classes that include timestamps. To begin, I'm following this document, although have also tried everything I can find online including here on stackoverflow to no avail.
On MacOS 12.6, protoc is installed from this zip to /usr/local/bin and the contents of include from the zip are installed to /usr/local/include.
This is the drives.proto file I need to create a class for:
syntax = "proto3";
option go_package = "./driveBuffers";
import "google/protobuf/timestamp.proto";
message Drive {
string id =1;
string name =2;
string colorRgb = 3;
string backgroundImageLink =4;
bool hidden = 5;
string orgUnitId = 6;
timestamp createdTime = 7;
message restrictions {
bool adminManagedRestrictions = 1;
bool domainUsersOnly = 2;
bool copyRequiresWriterPermission = 3;
bool driveMembersOnly = 4;
}
}
If I remove the field with type timestamp, the tool creates a file named ./driveBuffers/drives.pb.go. With the timestamp type, this error is thrown:
% protoc --go_out=. -I ./ -I /usr/local/include/ drives.proto
drives.proto:11:3: "timestamp" is not defined.
Thank you.
You should refer the type as google.protobuf.Timestamp. As example:
string orgUnitId = 6;
google.protobuf.Timestamp createdTime = 7;
I have a message of the following type
message Foo {
string bar = 1;
float baz = 2;
}
Is there any problem in converting it to the following for use in Go ?
message Foo {
string bar = 1;
optional float baz = 2;
}
Is the preferred way to deprecate and create a new field in the proto in this case as well ?
depends on how deeply integrated that particular message is in your code base - meaning
are you storing the marshaled binary representation somewhere like your database
are different parts of your code base using different versions of the message you are modifying - e.g. older versions of your android/ios apps and such
point being if you use a message structure to unmarshal encoded data, that was not generated with the very same message structure, into - bad things will happen.
The docs recommend adding a new element to circumvent such scenarios entirely. If that is not something you want to do, take the above points into consideration.
The optional will make the field a pointer type. So in Go generated code, optional float will become *float32, which of course is not float32.
To deprecate a field, use [deprecated = true] field option:
message Foo {
string bar = 1;
float baz = 2 [deprecated = true];
}
If in subsequent releases of your protobuf schema you actually remove the field altogether from the message, you might want to add reserved 2, where 2 is the number of the field you removed.
message Foo {
string bar = 1;
reserved 2;
}
This helps preventing other people or future you from adding a new field in position 2. This is relevant in case you have outdated clients which still expect a float in position 2.
PS: optional fieds in Proto3 are supported from version 3.15
I suggest you to use the FloatValue type defined in the google.protobuf package. As example:
syntax = "proto3";
import "google/protobuf/wrappers.proto";
message Foo {
string bar = 1;
google.protobuf.FloatValue baz = 2;
}
Will generate a pb files with the content:
type Foo struct {
state protoimpl.MessageState
sizeCache protoimpl.SizeCache
unknownFields protoimpl.UnknownFields
Bar string `protobuf:"bytes,1,opt,name=bar,proto3" json:"bar,omitempty"`
Baz *wrapperspb.FloatValue `protobuf:"bytes,2,opt,name=baz,proto3" json:"baz,omitempty"`
}
You can use as follow:
f := Foo{
Bar: "Bar",
Baz: &wrapperspb.FloatValue{Value: float32(3)},
}
var floatValue float32
if f.Baz != nil {
floatValue = f.Baz.GetValue()
}
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 2 years ago.
Improve this question
What's best practice for separating the declaration of messages used in services in (Go-specific) Protocol Buffers? The context is a large application with multiple gRPC services. Some messages are used in multiple services. The thought was to divide the definitions of message and services, like this (simplified a bit):
airline/pb/airline_messages.proto:
syntax = "proto3";
option go_package = "github.com/example.com/example-repo/airline/pb";
message Airline {
string code = 1;
string name = 2;
string country = 3;
}
airline/pb/airline_services.proto:
syntax = "proto3";
option go_package = "github.com/example.com/example-repo/airline/pb";
import "airline/pb/airline_messages.proto"
service AirlineService {
rpc GetAirline(string) returns (Airline) {}
rpc GetAirlines(GetAirlinesRequest) returns (repeated Airline) {}
}
message GetAirlinesRequest {
int max = 1;
string country = 2;
string pattern = 3;
string sortby = 4;
}
I'm calling protoc like this:
protoc --go_out=. \
--go_opt=paths=source_relative \
--go-grpc_out=. \
--go-grpc_opt=paths=source_relative \
--proto_path=. \
--proto_path=../../ airline_services.proto
This doesn't work because Airline isn't defined. Invoking it like this:
protoc ... airline_services.proto airline_messages.proto
generates an error message the Airline is multiply defined. Doing this:
protoc ... airline_messages.proto
protoc ... airline_services.proto
just overwrites the Go files and is equivalent to just compiling airline_services.proto.
What's best practice for being able to reuse message definitions in multiple services?
Not sure if it helps or if you are using a different version of the Go generator.
I have multiple *.proto files in one directory. Some of them import others:
A part of device.proto importing status.proto:
syntax = "proto3";
package somePackage;
import "status.proto";
option go_package = "goPackageName";
service Device {
rpc GetStatus (GetStatusRequest) returns (Status) {}
}
message GetStatusRequest {
// ...
}
with status.proto containing the Status message:
syntax = "proto3";
package somePackage;
option go_package = "goPackageName";
message Status {
int32 customer = 1;
string device_id = 2;
// ...
}
I generate the Go code with:
protoc --go_out=plugins=grpc:. --go_opt=paths=source_relative *.proto
Note: I'm still using the old Go protobuf github.com/golang/protobuf.
The problem:
I have 2 files in the root directory.
I use Makefile to generate Go code from .proto files.
But the language field in the Video struct is a value not a pointer to the value.
And the subtitles field in the Video struct is an array of values not an array of pointers to the value.
The question is:
How can I make protoc generate a pointer to the value?
video.pb.go
type Video struct {
state protoimpl.MessageState
sizeCache protoimpl.SizeCache
unknownFields protoimpl.UnknownFields
Id string `protobuf:"bytes,1,opt,name=id,proto3" json:"id,omitempty"`
Title string `protobuf:"bytes,2,opt,name=title,proto3" json:"title,omitempty"`
// I need *language.ISO639_1 below
Languages language.ISO639_1 `protobuf:"varint,3,opt,name=languages,proto3,enum=language.ISO639_1" json:"languages,omitempty"`
// I need []*language.ISO639_1 below
Subtitles []language.ISO639_1 `protobuf:"varint,4,rep,packed,name=subtitles,proto3,enum=language.ISO639_1" json:"subtitles,omitempty"`
}
Makefile
gen:
# Video
protoc -I. --go_out=plugins=grpc,paths=source_relative:video video.proto
# Language
protoc -I. --go_out=plugins=grpc,paths=source_relative:language language.proto
language.proto
syntax = "proto3";
package language;
option go_package = "example.com/group/repo/language;language";
enum ISO639_1 {
UNKNOWN = 0;
zh = 1;
}
video.proto
syntax = "proto3";
package video;
import "language.proto";
option go_package = "example.com/group/repo/video;video";
message Video {
string id = 1;
string title = 2;
language.ISO639_1 language = 3;
repeated language.ISO639_1 subtitles = 4;
}
protoc version: libprotoc 3.11.4
Starting in proto3 version 3.12, the field presence feature is supported experimentally, meaning you can again use the optional keyword, similarly to proto2.
You can achieve this by passing a flag --experimental_allow_proto3_optional to protoc when generating the pb.go files (ensure you're running new enough version of protoc and protoc-gen-go to support this experimental feature).
So given this pseudo-.proto file:
enum ISO639_1 {
UNKNOWN = 0;
zh = 1;
}
message Video {
string id = 1;
string title = 2;
optional ISO639_1 language = 3;
repeated ISO639_1 subtitles = 4;
}
You should get generated struct with:
type Video struct {
state protoimpl.MessageState
sizeCache protoimpl.SizeCache
unknownFields protoimpl.UnknownFields
Language *ISO639_1 `protobuf:"varint,3,opt,name=language,proto3,enum=grpctrace.ISO639_1,oneof" json:"language,omitempty"`
Subtitles []ISO639_1 `protobuf:"varint,4,rep,packed,name=subtitles,proto3,enum=grpctrace.ISO639_1" json:"subtitles,omitempty"`
}
While this solves your issue with the language, I'm not sure you'll be able to work around the repeated field, to get a pointer value. Maybe this thread might be helpful - https://stackoverflow.com/a/25637833/13183366
I am providing following proto file to flatbuffer compiler to generate .fbs file.
File: test.proto
message A {
optional int32 afoo = 1;
message B {
optional int32 bfoo_ = 1;
}
optional B bfoo= 2;
}
message C {
optional int32 abar = 1;
message B {
optional int32 bbar_ = 1;
}
optional B bbar = 2;
}
After this i ran: flatc --proto test.proto, which will generate .fbs file
File: test.fbs
// Generated from test.proto
namespace ;
table A {
afoo:int;
bfoo:_A.B;
}
namespace _A;
table B {
bfoo_:int;
}
namespace ;
table C {
abar:int;
bbar:_C.B;
}
namespace _C;
table B {
bbar_:int;
}
Point to note here is "B" has been pulled out to global namespace
After that i ran flat -cpp test.proto, which eventually generate incorrect header file. Contains two definitions of B
struct B FLATBUFFERS_FINAL_CLASS : private flatbuffers::Table
This appears to be a bug in FlatBuffers, in the generated code they're all in the same namespace. Report a bug here: https://github.com/google/flatbuffers/issues
Meanwhile, un-nest the .proto manually, and it should work.
EDIT: This was recently fixed: https://github.com/google/flatbuffers/commit/20c0082ee5bfeeecaa443c001a89934e9448ffa4