I need to use Protocol Buffers to serialize JSON messages received from the Google Drive drives:list method and write them to the BigQuery Storage Write API (GRPC). This is working for all field types except timestamp. I cannot for the life of me generate go classes that include timestamps. To begin, I'm following this document, although have also tried everything I can find online including here on stackoverflow to no avail.
On MacOS 12.6, protoc is installed from this zip to /usr/local/bin and the contents of include from the zip are installed to /usr/local/include.
This is the drives.proto file I need to create a class for:
syntax = "proto3";
option go_package = "./driveBuffers";
import "google/protobuf/timestamp.proto";
message Drive {
string id =1;
string name =2;
string colorRgb = 3;
string backgroundImageLink =4;
bool hidden = 5;
string orgUnitId = 6;
timestamp createdTime = 7;
message restrictions {
bool adminManagedRestrictions = 1;
bool domainUsersOnly = 2;
bool copyRequiresWriterPermission = 3;
bool driveMembersOnly = 4;
}
}
If I remove the field with type timestamp, the tool creates a file named ./driveBuffers/drives.pb.go. With the timestamp type, this error is thrown:
% protoc --go_out=. -I ./ -I /usr/local/include/ drives.proto
drives.proto:11:3: "timestamp" is not defined.
Thank you.
You should refer the type as google.protobuf.Timestamp. As example:
string orgUnitId = 6;
google.protobuf.Timestamp createdTime = 7;
Related
syntax = "proto3";
package model;
import "google/protobuf/Empty.proto";
message User {
string id = 1;
string name = 2;
string email = 3;
string alamat = 4;
string password = 5;
}
message UserList {
repeated User list = 1;
}
message userId {
string id = 1;
}
message UserUpdate {
string id = 1;
User user = 2;
}
service Users {
rpc getUserList(google.protobuf.Empty) returns (UserList) {}
rpc getUserById(userId) returns (User) {}
rpc inserUser(User) returns (google.protobuf.Empty) {}
rpc updateUser(UserUpdate) returns (google.protobuf.Empty) {}
rpc deleteUser(userId) returns (google.protobuf.Empty) {}
}
above is my proto file. I get error google/protobuf/Empty.proto: File not found.
when trying to compile the proto file above. can someone help me ?
First of all, the correct import is import "google/protobuf/empty.proto";
second, for generating a proto file run this code:
protoc --proto_path={proto_directory_address} --proto_path={proto_directory_name} --go-grpc_out={generated_directory_path} --go_out={generated_directory_path} {proto_directory_address}/{proto_file_name}.proto
hi there / i had the same issue for a long time .. this process worked for me i hope it dose for you too :
navigate to this directory using your cmd(command line) :
cd .local/include
this directory normally should contain some folder named "google" copy this folder and paste it to this directory :
/usr/local/include
and now try the protoc engine again to generate your project and if the error still exists then try the rest of process :
navigate to that specific directory and check if its been copied or not . if it is then try to navigate to the folder from where you are (which it should be /usr/local/include) if the error says you have no permission to get in the folder
use this command to get the permission
$ sudo chmod o+r -R ./google
and then try to get permission to get in protobuf folder in the same directory using above command again
when its all done . check the protoc generator again /// hope works for you because it dose for me
In the following example:
try (ParquetWriter<Example> writer =
new ProtoParquetWriter<>(
new Path("file:/tmp/foo.parquet"),
Example.class,
SNAPPY,
DEFAULT_BLOCK_SIZE,
DEFAULT_PAGE_SIZE)) {
writer.write(
Example.newBuilder()
.setTs(System.currentTimeMillis())
.setTenantId("tenant")
.setSomeFlag(false)
.setSomeInt(1)
.setOtherInt(0)
.build());
}
}
And example .proto file:
syntax = "proto3";
package com.example;
message Example {
uint64 ts = 1;
string tenantId = 2;
bool someFlag = 3;
int32 someInt = 4;
int32 otherInt = 2;
}
The resulting parquet file won't have the fields someFlag and otherInt because they are false and 0 respectively.
Is there a way to make it write it anyway or should I handle this on the reader side?
In proto3, presence tracking was not enabled historically, and the only presence rule was around zero defaults. Fortunately this changed recently in new versions of protoc. The optional keyword can now be used in from of fields in proto3 to enable this. So: add optional, and any compliant implementation should do what you want. The defaults are still zero/false/etc, but if they are explicitly set: they are serialized.
syntax = "proto3";
package com.example;
message Example {
optional uint64 ts = 1;
optional string tenantId = 2;
optional bool someFlag = 3;
optional int32 someInt = 4;
optional int32 otherInt = 2; // [sic]
}
Also, the second 2 should be a 5
The problem:
I have 2 files in the root directory.
I use Makefile to generate Go code from .proto files.
But the language field in the Video struct is a value not a pointer to the value.
And the subtitles field in the Video struct is an array of values not an array of pointers to the value.
The question is:
How can I make protoc generate a pointer to the value?
video.pb.go
type Video struct {
state protoimpl.MessageState
sizeCache protoimpl.SizeCache
unknownFields protoimpl.UnknownFields
Id string `protobuf:"bytes,1,opt,name=id,proto3" json:"id,omitempty"`
Title string `protobuf:"bytes,2,opt,name=title,proto3" json:"title,omitempty"`
// I need *language.ISO639_1 below
Languages language.ISO639_1 `protobuf:"varint,3,opt,name=languages,proto3,enum=language.ISO639_1" json:"languages,omitempty"`
// I need []*language.ISO639_1 below
Subtitles []language.ISO639_1 `protobuf:"varint,4,rep,packed,name=subtitles,proto3,enum=language.ISO639_1" json:"subtitles,omitempty"`
}
Makefile
gen:
# Video
protoc -I. --go_out=plugins=grpc,paths=source_relative:video video.proto
# Language
protoc -I. --go_out=plugins=grpc,paths=source_relative:language language.proto
language.proto
syntax = "proto3";
package language;
option go_package = "example.com/group/repo/language;language";
enum ISO639_1 {
UNKNOWN = 0;
zh = 1;
}
video.proto
syntax = "proto3";
package video;
import "language.proto";
option go_package = "example.com/group/repo/video;video";
message Video {
string id = 1;
string title = 2;
language.ISO639_1 language = 3;
repeated language.ISO639_1 subtitles = 4;
}
protoc version: libprotoc 3.11.4
Starting in proto3 version 3.12, the field presence feature is supported experimentally, meaning you can again use the optional keyword, similarly to proto2.
You can achieve this by passing a flag --experimental_allow_proto3_optional to protoc when generating the pb.go files (ensure you're running new enough version of protoc and protoc-gen-go to support this experimental feature).
So given this pseudo-.proto file:
enum ISO639_1 {
UNKNOWN = 0;
zh = 1;
}
message Video {
string id = 1;
string title = 2;
optional ISO639_1 language = 3;
repeated ISO639_1 subtitles = 4;
}
You should get generated struct with:
type Video struct {
state protoimpl.MessageState
sizeCache protoimpl.SizeCache
unknownFields protoimpl.UnknownFields
Language *ISO639_1 `protobuf:"varint,3,opt,name=language,proto3,enum=grpctrace.ISO639_1,oneof" json:"language,omitempty"`
Subtitles []ISO639_1 `protobuf:"varint,4,rep,packed,name=subtitles,proto3,enum=grpctrace.ISO639_1" json:"subtitles,omitempty"`
}
While this solves your issue with the language, I'm not sure you'll be able to work around the repeated field, to get a pointer value. Maybe this thread might be helpful - https://stackoverflow.com/a/25637833/13183366
I get an error when I try to compile a proto file to convert to .java.
Could you point out what I'm missing ?
protoc --proto_path=src\main\resources\proto --java_out=src\main\java src\main\resources\proto\PayloadProtocol.proto
PayloadProtocol.proto:32:14: "DataContainer" is not defined.
PayloadProtocol.proto: warning: Import BackendCommunicationService.proto but not used.
Payload.proto
import "BackendCommunicationService.proto";
package com.fleetboard.tp.payload.protocol.protobuf;
option java_package = "com.fleetboard.tp.proto.protocol";
message TPMessage {
required int32 serviceId = 1; // telematic service (TS) id, who owns this message
required int32 functionId = 2; // function id refers to the Java class for the payload
optional uint64 requestId = 3; // Identifier to associate the request to a response
optional TPPayload payload = 4; // serialized representation of a TP message
optional uint64 durability = 5; // life time of message - used from backend
optional DataContainer dataPayload = 6;**
}
BackendCommunicationService.proto
package com.fleetboard.tp.backend.protobuf;
option java_package = "com.fleetboard.tp.proto.backend";
message DataContainer {
required DeviceApplication application = 1; // The container's recipient (MT) or sender (MO)
required string fileName = 2; // File name (no path), length up to 255
required uint64 fileTime = 3; // File time as ms since 1970-01-01 00:00 UTC
}
Fully qualify the name in the importing file:
com.fleetboard.tp.backend.protobuf.DataContainer
or
.com.fleetboard.tp.backend.protobuf.DataContainer
(the . ensures it starts at the root)
You could also try using just the disjoint part, but I don't know if it will work:
backend.protobuf.DataContainer
(since both have the com.fleetboard.tp. prefix)
Is there a good way to integrate protobuf compiles with the go build command?
goprotobuf "provides Go support, in the form of a library and protocol compiler plugin, for Google's protocol buffers".
The README at the goprotobuf library has some good info.
From https://code.google.com/p/goprotobuf/source/browse/README#106 :
Consider file test.proto, containing
package example;
enum FOO { X = 17; };
message Test {
required string label = 1;
optional int32 type = 2 [default=77];
repeated int64 reps = 3;
optional group OptionalGroup = 4 {
required string RequiredField = 5;
}
}
To build a package from test.proto and some other Go files, write a Makefile like this:
include $(GOROOT)/src/Make.$(GOARCH)
TARG=path/to/example
GOFILES=\
test.pb.go\
other.go
include $(GOROOT)/src/Make.pkg
include $(GOROOT)/src/pkg/code.google.com/p/goprotobuf/Make.protobuf