The problem:
I have 2 files in the root directory.
I use Makefile to generate Go code from .proto files.
But the language field in the Video struct is a value not a pointer to the value.
And the subtitles field in the Video struct is an array of values not an array of pointers to the value.
The question is:
How can I make protoc generate a pointer to the value?
video.pb.go
type Video struct {
state protoimpl.MessageState
sizeCache protoimpl.SizeCache
unknownFields protoimpl.UnknownFields
Id string `protobuf:"bytes,1,opt,name=id,proto3" json:"id,omitempty"`
Title string `protobuf:"bytes,2,opt,name=title,proto3" json:"title,omitempty"`
// I need *language.ISO639_1 below
Languages language.ISO639_1 `protobuf:"varint,3,opt,name=languages,proto3,enum=language.ISO639_1" json:"languages,omitempty"`
// I need []*language.ISO639_1 below
Subtitles []language.ISO639_1 `protobuf:"varint,4,rep,packed,name=subtitles,proto3,enum=language.ISO639_1" json:"subtitles,omitempty"`
}
Makefile
gen:
# Video
protoc -I. --go_out=plugins=grpc,paths=source_relative:video video.proto
# Language
protoc -I. --go_out=plugins=grpc,paths=source_relative:language language.proto
language.proto
syntax = "proto3";
package language;
option go_package = "example.com/group/repo/language;language";
enum ISO639_1 {
UNKNOWN = 0;
zh = 1;
}
video.proto
syntax = "proto3";
package video;
import "language.proto";
option go_package = "example.com/group/repo/video;video";
message Video {
string id = 1;
string title = 2;
language.ISO639_1 language = 3;
repeated language.ISO639_1 subtitles = 4;
}
protoc version: libprotoc 3.11.4
Starting in proto3 version 3.12, the field presence feature is supported experimentally, meaning you can again use the optional keyword, similarly to proto2.
You can achieve this by passing a flag --experimental_allow_proto3_optional to protoc when generating the pb.go files (ensure you're running new enough version of protoc and protoc-gen-go to support this experimental feature).
So given this pseudo-.proto file:
enum ISO639_1 {
UNKNOWN = 0;
zh = 1;
}
message Video {
string id = 1;
string title = 2;
optional ISO639_1 language = 3;
repeated ISO639_1 subtitles = 4;
}
You should get generated struct with:
type Video struct {
state protoimpl.MessageState
sizeCache protoimpl.SizeCache
unknownFields protoimpl.UnknownFields
Language *ISO639_1 `protobuf:"varint,3,opt,name=language,proto3,enum=grpctrace.ISO639_1,oneof" json:"language,omitempty"`
Subtitles []ISO639_1 `protobuf:"varint,4,rep,packed,name=subtitles,proto3,enum=grpctrace.ISO639_1" json:"subtitles,omitempty"`
}
While this solves your issue with the language, I'm not sure you'll be able to work around the repeated field, to get a pointer value. Maybe this thread might be helpful - https://stackoverflow.com/a/25637833/13183366
Related
I need to use Protocol Buffers to serialize JSON messages received from the Google Drive drives:list method and write them to the BigQuery Storage Write API (GRPC). This is working for all field types except timestamp. I cannot for the life of me generate go classes that include timestamps. To begin, I'm following this document, although have also tried everything I can find online including here on stackoverflow to no avail.
On MacOS 12.6, protoc is installed from this zip to /usr/local/bin and the contents of include from the zip are installed to /usr/local/include.
This is the drives.proto file I need to create a class for:
syntax = "proto3";
option go_package = "./driveBuffers";
import "google/protobuf/timestamp.proto";
message Drive {
string id =1;
string name =2;
string colorRgb = 3;
string backgroundImageLink =4;
bool hidden = 5;
string orgUnitId = 6;
timestamp createdTime = 7;
message restrictions {
bool adminManagedRestrictions = 1;
bool domainUsersOnly = 2;
bool copyRequiresWriterPermission = 3;
bool driveMembersOnly = 4;
}
}
If I remove the field with type timestamp, the tool creates a file named ./driveBuffers/drives.pb.go. With the timestamp type, this error is thrown:
% protoc --go_out=. -I ./ -I /usr/local/include/ drives.proto
drives.proto:11:3: "timestamp" is not defined.
Thank you.
You should refer the type as google.protobuf.Timestamp. As example:
string orgUnitId = 6;
google.protobuf.Timestamp createdTime = 7;
I have a message of the following type
message Foo {
string bar = 1;
float baz = 2;
}
Is there any problem in converting it to the following for use in Go ?
message Foo {
string bar = 1;
optional float baz = 2;
}
Is the preferred way to deprecate and create a new field in the proto in this case as well ?
depends on how deeply integrated that particular message is in your code base - meaning
are you storing the marshaled binary representation somewhere like your database
are different parts of your code base using different versions of the message you are modifying - e.g. older versions of your android/ios apps and such
point being if you use a message structure to unmarshal encoded data, that was not generated with the very same message structure, into - bad things will happen.
The docs recommend adding a new element to circumvent such scenarios entirely. If that is not something you want to do, take the above points into consideration.
The optional will make the field a pointer type. So in Go generated code, optional float will become *float32, which of course is not float32.
To deprecate a field, use [deprecated = true] field option:
message Foo {
string bar = 1;
float baz = 2 [deprecated = true];
}
If in subsequent releases of your protobuf schema you actually remove the field altogether from the message, you might want to add reserved 2, where 2 is the number of the field you removed.
message Foo {
string bar = 1;
reserved 2;
}
This helps preventing other people or future you from adding a new field in position 2. This is relevant in case you have outdated clients which still expect a float in position 2.
PS: optional fieds in Proto3 are supported from version 3.15
I suggest you to use the FloatValue type defined in the google.protobuf package. As example:
syntax = "proto3";
import "google/protobuf/wrappers.proto";
message Foo {
string bar = 1;
google.protobuf.FloatValue baz = 2;
}
Will generate a pb files with the content:
type Foo struct {
state protoimpl.MessageState
sizeCache protoimpl.SizeCache
unknownFields protoimpl.UnknownFields
Bar string `protobuf:"bytes,1,opt,name=bar,proto3" json:"bar,omitempty"`
Baz *wrapperspb.FloatValue `protobuf:"bytes,2,opt,name=baz,proto3" json:"baz,omitempty"`
}
You can use as follow:
f := Foo{
Bar: "Bar",
Baz: &wrapperspb.FloatValue{Value: float32(3)},
}
var floatValue float32
if f.Baz != nil {
floatValue = f.Baz.GetValue()
}
In the following example:
try (ParquetWriter<Example> writer =
new ProtoParquetWriter<>(
new Path("file:/tmp/foo.parquet"),
Example.class,
SNAPPY,
DEFAULT_BLOCK_SIZE,
DEFAULT_PAGE_SIZE)) {
writer.write(
Example.newBuilder()
.setTs(System.currentTimeMillis())
.setTenantId("tenant")
.setSomeFlag(false)
.setSomeInt(1)
.setOtherInt(0)
.build());
}
}
And example .proto file:
syntax = "proto3";
package com.example;
message Example {
uint64 ts = 1;
string tenantId = 2;
bool someFlag = 3;
int32 someInt = 4;
int32 otherInt = 2;
}
The resulting parquet file won't have the fields someFlag and otherInt because they are false and 0 respectively.
Is there a way to make it write it anyway or should I handle this on the reader side?
In proto3, presence tracking was not enabled historically, and the only presence rule was around zero defaults. Fortunately this changed recently in new versions of protoc. The optional keyword can now be used in from of fields in proto3 to enable this. So: add optional, and any compliant implementation should do what you want. The defaults are still zero/false/etc, but if they are explicitly set: they are serialized.
syntax = "proto3";
package com.example;
message Example {
optional uint64 ts = 1;
optional string tenantId = 2;
optional bool someFlag = 3;
optional int32 someInt = 4;
optional int32 otherInt = 2; // [sic]
}
Also, the second 2 should be a 5
I get an error when I try to compile a proto file to convert to .java.
Could you point out what I'm missing ?
protoc --proto_path=src\main\resources\proto --java_out=src\main\java src\main\resources\proto\PayloadProtocol.proto
PayloadProtocol.proto:32:14: "DataContainer" is not defined.
PayloadProtocol.proto: warning: Import BackendCommunicationService.proto but not used.
Payload.proto
import "BackendCommunicationService.proto";
package com.fleetboard.tp.payload.protocol.protobuf;
option java_package = "com.fleetboard.tp.proto.protocol";
message TPMessage {
required int32 serviceId = 1; // telematic service (TS) id, who owns this message
required int32 functionId = 2; // function id refers to the Java class for the payload
optional uint64 requestId = 3; // Identifier to associate the request to a response
optional TPPayload payload = 4; // serialized representation of a TP message
optional uint64 durability = 5; // life time of message - used from backend
optional DataContainer dataPayload = 6;**
}
BackendCommunicationService.proto
package com.fleetboard.tp.backend.protobuf;
option java_package = "com.fleetboard.tp.proto.backend";
message DataContainer {
required DeviceApplication application = 1; // The container's recipient (MT) or sender (MO)
required string fileName = 2; // File name (no path), length up to 255
required uint64 fileTime = 3; // File time as ms since 1970-01-01 00:00 UTC
}
Fully qualify the name in the importing file:
com.fleetboard.tp.backend.protobuf.DataContainer
or
.com.fleetboard.tp.backend.protobuf.DataContainer
(the . ensures it starts at the root)
You could also try using just the disjoint part, but I don't know if it will work:
backend.protobuf.DataContainer
(since both have the com.fleetboard.tp. prefix)
Is there a good way to integrate protobuf compiles with the go build command?
goprotobuf "provides Go support, in the form of a library and protocol compiler plugin, for Google's protocol buffers".
The README at the goprotobuf library has some good info.
From https://code.google.com/p/goprotobuf/source/browse/README#106 :
Consider file test.proto, containing
package example;
enum FOO { X = 17; };
message Test {
required string label = 1;
optional int32 type = 2 [default=77];
repeated int64 reps = 3;
optional group OptionalGroup = 4 {
required string RequiredField = 5;
}
}
To build a package from test.proto and some other Go files, write a Makefile like this:
include $(GOROOT)/src/Make.$(GOARCH)
TARG=path/to/example
GOFILES=\
test.pb.go\
other.go
include $(GOROOT)/src/Make.pkg
include $(GOROOT)/src/pkg/code.google.com/p/goprotobuf/Make.protobuf