Protobuf: how do I get right options order of a method? - go

I am writing protoc plugin in Go which should generate documentation for our GRPC services and currently struggle in attempt to know right order of options.
First, how the protobuf looks like
syntax = "proto3";
option go_package = "sample";
package sample
import "common/extensions.proto"
message SimpleMessage {
// Id represents the message identifier.
string id = 1;
int64 num = 2;
}
message Response {
int32 code = 1;
}
enum ErrorCodes {
RESERVED = 0;
OK = 200
ERROR = 6000
PANIC = 6001
}
service EchoService {
rpc Echo (SimpleMessage) returns (Response) {
// common.grpc_status is an extension defined somewhere
// these are list of possible return statuses
option (common.grpc_status) = {
status: "OK"
status: "ERROR"
status: "PANIC" // Every status string will must be one of ErrorCodes items
};
option (common.middlewares) = {
middleware: "csrf"
middleware: "session"
}
}
};
As you see, there're two options here. The problem is protoc doesn't bind position directly to tokens. It leaves this information in a special sections where it can be restored via using so called "paths". And these paths are rely on order, while options are hidden and can only be retrieved using proto.GetExtension function which doesn't report option index either.
I need this token location information to report errors. Is there any way to get option index or something equivalent?
I am thinking about using standalone parser just to retrieve the right order, but this feels somewhat awkward. Hope there's a better way.

Related

How to create a protocol buffer for a client server in ballerina?

We wish to build a repository of functions that a developer can assemble to build a complex program.
There can exist several versions of a function, each with its metadata. This, function metadata includes the developer's full name and email address, the language the function is implemented in and a set of keywords related to the functionality fulfilled by the function.
The versions of a function can be represented as a directed acyclic graph.
To manage the repository, we use a remote invocation based gRPC that allows a client to interact with a server and execute the following operations:
add_new_fn: to add either a brand new function or a new version to an existing function;
add_fns: to add multiple functions streamed by the client (note that multiple versions of a function are not allowed);
delete_fn: to delete a function (this might require reordering the versions of the function);
show_fn: to view a specific version of a function;
show_all_fns: to view all versions of a function (the versions are streamed back by the server)
show_all_with_criteria: to view all latest versions of functions implemented in a given language or related to a set of keywords (bi-directional streaming).
Assuming you just want the API defined:
syntax = "proto3";
package foo;
import "google/protobuf/any.proto";
// Convention is to name RPC method messages after the method
// Provides flexibility in evolving them distinctly too
service FnRepository {
rpc AddFn(AddFnPackageRequest) returns (AddFnPackageResponse) {}
rpc AddFns(stream
AddFnPackageRequest) returns (stream AddFnPackageResponse) {}
rpc DeleteFn(DelFnPackageRequest) returns (DelFnPackageResponse) {}
...
}
// Requests containing messages provides extensibility
message AddFnPackageRequest {
FnPackage fn_package = 1;
}
// Unique ID returned here
// Would likely be used to delete
message AddFnPackageResponse {
string id = 1;
}
message Developer {
string name = 1;
string email = 2;
}
// Enum assumes a predefined list of languages
message Fn {
string name = 1;
Signature signature = 2;
enum Language {
GOLANG = 0;
RUST = 1;
}
Language language = 3;
}
// Conceptually cleaner to define subtypes for e.g. Developer
// This does disconnect e.g. Fn from Developer
message FnPackage {
Fn fn = 1;
Developer developer = 2;
Metadata metadata = 3;
}
message Metadata {
Developer developer = 1;
repeated string keywords = 2;
}
// Any is a way to provide polymorphism
// Since Fn signatures are arbitrary types
message Signature {
google.protobuf.Any params = 1;
google.protobuf.Any return = 2;
}
...
This is written entirely here and so there's no warranty that it will actually compile 😃

Build proto3 gRPC API with overloaded endpoints

I am fairly new to API building, so this may be a broader question than I originally posed.
I am creating an API in Golang (using protobuf 3 and gRPC) that has two similar endpoints:
GET /project/genres
GET /project/{id}
The problem is that when I run curl localhost:8080/project/genres, the pattern matching results in the /project/{id} endpoint getting called with genres as the id. Is there some simple way around this, or do I have to build something into the server code to call the proper function based on the type?
I also tried flipping the ordering of these definitions, just in case the pattern matching had some order of operations that I didn't know about, but this didn't make a difference.
Here are the definitions in my proto file:
message EmptyRequest { }
message ProjectRequest {
string id = 1;
}
message GenreResponse {
int32 id = 1;
string name = 2;
}
message ProjectResponse {
int32 id = 1;
string name = 2;
}
service ProjectService {
rpc GetProject(ProjectRequest) returns (ProjectResponse) {
option (google.api.http) = {
get: "/v1/project/{id}"
};
}
rpc GetGenres(EmptyRequest) returns (GenreResponse) {
option (google.api.http) = {
get: "/v1/project/genres"
};
}
}
I was able to specify a check in the url path template to get around this issue:
rpc GetGenres(EmptyRequest) returns (GenreResponse) {
option (google.api.http) = {
get: "/v1/project/{id=genres}"
};
}
This seems to have fixed the problem. I don't know if there are other solutions, or if this is the right way to do this, but I'm happy to accept other answers if something better comes in.

Is change of a message type to a similar will break backward compatibility?

I want to preserve my application from future issues with backward compatibility. Now I have this version of test.proto:
syntax = "proto3";
service TestApi {
rpc DeleteFoo(DeleteFooIn) returns (BoolResult) {}
rpc DeleteBar(DeleteBarIn) returns (BoolResult) {}
}
message DeleteFooIn {
int32 id = 1;
}
message DeleteBarIn {
int32 id = 1;
}
message BoolResult {
bool result = 1;
}
I'm interested in a case when I will want to change result message of DeleteBar() to a message like "DeleteBarOut":
syntax = "proto3";
service TestApi {
rpc DeleteFoo(DeleteFooIn) returns (BoolResult) {}
rpc DeleteBar(DeleteBarIn) returns (DeleteBarOut) {}
}
message DeleteFooIn {
int32 id = 1;
}
message DeleteBarIn {
int32 id = 1;
}
message DeleteBarOut {
reserved 1;
string time = 2;
}
message BoolResult {
bool result = 1;
}
The question is about backward compatibility on-wire with the old .proto. Can I change the name of the result message from "BoolResult" to "DeleteBarOut"?
Or I should save the old name of the message and edit fields list of "BoolResult"? But then how can I save DeleteFoo() from any changes in this solution?
When making a breaking change to an API like this, it is a common practice to support both versions while transitioning. In order to do this, you would need to add a version field to the request message and then in your request handler, route the message to different backends based on which version is specified. Once there is no more traffic going to the v1 backend you can make a hard cutover to v2 and stop supporting v1.
Unfortunately, if you just change the RPC definition without versioning, it is impossible to avoid a version incompatibility between the server and the client. The other option of course is to add a new RPC endpoint rather than modifying an existing one.
In general if you are making breaking API changes you're going to have an unpleasant time.

InvalidProtocolBufferException in Java client when deserializing protobuf data from C++ server

I have a protobuf message like this:
message Update {
Path path = 1; // The path (key) for the update.
Value value = 2 [deprecated=true]; // The value (value) for the update.
TypedValue val = 3; // The explicitly typed update value.
}
// TypedValue is used to encode a value being sent between the client and
// target (originated by either entity).
message TypedValue {
oneof value {
string string_val = 1; // String value.
int64 int_val = 2; // Integer
....
google.protobuf.Any any_val = 9; // protobuf.Any encoded bytes.
....
}
}
On the server side (C++), we are setting this field as follows (LLDP is the outer class and Interfaces is inside that):
openconfig_lldp::Lldp out;
GetLldpProto(&out);
update->mutable_val()->mutable_any_val()->PackFrom(out.interfaces());
On the client side (Java), we are extracting this field like this:
OpenconfigLldp.Lldp.Interfaces interfaces = update.getVal().getAnyVal().unpack(OpenconfigLldp.Lldp.Interfaces.class);
This is throwing a InvalidProtocolBufferException exception. When I dump the "update" in my Java client, I see this:
path {
elem {
name: "lldp"
}
elem {
name: "interfaces"
}
}
val {
any_val {
type_url: "type.googleapis.com/openconfig_lldp.Lldp.Interfaces"
value: "\212\207\237\334\v\374\001\022\371\001\262\211\267l\031\342\367\304\260\002\v\n\tEth 1/1/1\242\340\247\230\017\002\b\001\352\316\234\250\017\324\001\262\217\304\272\017/\022-\302\340\317\247\001\'\202\225\377\302\001\b\n\006Neigh1\342\253\214\353\001\v\n\tEth 1/1/1\242\364\301\261\a\002\b\n\262\217\304\272\017/\022-\302\340\317\247\001\'\202\225\377\302\001\b\n\006Neigh2\342\253\214\353\001\v\n\tEth 1/1/2\242\364\301\261\a\002\b\n\262\217\304\272\017/\022-\302\340\317\247\001\'\202\225\377\302\001\b\n\006Neigh3\342\253\214\353\001\v\n\tEth 1/1/3\242\364\301\261\a\002\b\n\262\217\304\272\017/\022-\302\340\317\247\001\'\202\225\377\302\001\b\n\006Neigh4\342\253\214\353\001\v\n\tEth 1/1/4\242\364\301\261\a\002\b\n\212\207\237\334\v\374\001\022\371\001\262\211\267l\031\342\367\304\260\002\v\n\tEth 1/1/2\242\340\247\230\017\002\b\001\352\316\234\250\017\324\001\262\217\304\272\017/\022-\302\340\317\247\001\'\202\225\377\302\001\b\n\006Neigh1\342\253\214\353\001\v\n\tEth 1/1/1\242\364\301\261\a\002\b\n\262\217\304\272\017/\022-\302\340\317\247\001\'\202\225\377\302\001\b\n\006Neigh2\342\253\214\353\001\v\n\tEth 1/1/2\242\364\301\261\a\002\b\n\262\217\304\272\017/\022-\302\340\317\247\001\'\202\225\377\302\001\b\n\006Neigh3\342\253\214\353\001\v\n\tEth 1/1/3\242\364\301\261\a\002\b\n\262\217\304\272\017/\022-\302\340\317\247\001\'\202\225\377\302\001\b\n\006Neigh4\342\253\214\353\001\v\n\tEth 1/1/4\242\364\301\261\a\002\b\n\212\207\237\334\v\374\001\022\371\001\262\211\267l\031\342\367\304\260\002\v\n\tEth 1/1/3\242\340\247\230\017\002\b\001\352\316\234\250\017\324\001\262\217\304\272\017/\022-\302\340\317\247\001\'\202\225\377\302\001\b\n\006Neigh1\342\253\214\353\001\v\n\tEth 1/1/1\242\364\301\261\a\002\b\n\262\217\304\272\017/\022-\302\340\317\247\001\'\202\225\377\302\001\b\n\006Neigh2\342\253\214\353\001\v\n\tEth 1/1/2\242\364\301\261\a\002\b\n\262\217\304\272\017/\022-\302\340\317\247\001\'\202\225\377\302\001\b\n\006Neigh3\342\253\214\353\001\v\n\tEth 1/1/3\242\364\301\261\a\002\b\n\262\217\304\272\017/\022-\302\340\317\247\001\'\202\225\377\302\001\b\n\006Neigh4\342\253\214\353\001\v\n\tEth 1/1/4\242\364\301\261\a\002\b\n\212\207\237\334\v\374\001\022\371\001\262\211\267l\031\342\367\304\260\002\v\n\tEth 1/1/4\242\340\247\230\017\002\b\001\352\316\234\250\017\324\001\262\217\304\272\017/\022-\302\340\317\247\001\'\202\225\377\302\001\b\n\006Neigh1\342\253\214\353\001\v\n\tEth 1/1/1\242\364\301\261\a\002\b\n\262\217\304\272\017/\022-\302\340\317\247\001\'\202\225\377\302\001\b\n\006Neigh2\342\253\214\353\001\v\n\tEth 1/1/2\242\364\301\261\a\002\b\n\262\217\304\272\017/\022-\302\340\317\247\001\'\202\225\377\302\001\b\n\006Neigh3\342\253\214\353\001\v\n\tEth 1/1/3\242\364\301\261\a\002\b\n\262\217\304\272\017/\022-\302\340\317\247\001\'\202\225\377\302\001\b\n\006Neigh4\342\253\214\353\001\v\n\tEth 1/1/4\242\364\301\261\a\002\b\n"
}
}
The type_url seems correct to me. What am I doing wrong here?
Thanks for your time.
EDIT #1:
I looked at the exception string. It is "Type of the Any message does not match the given class."
The same proto file is used for C++ and Java, but I see "openconfig_lldp.Lldp.Interfaces" in C++, where as, it is "OpenconfigLldp.Lldp.Interfaces" in Java. Need to find out why..
EDIT #2:
The same .proto file is used. In this case, it is:
openconfig_lldp.proto
---------------------
syntax = "proto3";
package openconfig.openconfig_lldp;
message Lldp {
message Config {
....
....
}
....
....
}
In case of Java, I see the parent class as OpenconfigLldp in a package called openconfig_lldp.
package openconfig.openconfig_lldp;
public final class OpenconfigLldp {
private OpenconfigLldp() {}
....
....
/**
* Protobuf type {#code openconfig.openconfig_lldp.Lldp}
*/
public static final class Lldp extends com.google.protobuf.GeneratedMessageV3 implements
// ##protoc_insertion_point(message_implements:openconfig.openconfig_lldp.Lldp)
....
....
}
In C++, I don't see any class called "OpenconfigLldp" generated. Instead it is just "Lldp"
So, the type_url in the Any.protobuf is a mismatch. C++ side puts it as
type_url: "type.googleapis.com/openconfig_lldp.Lldp.Interfaces"
While in the Java side I use:
OpenconfigLldp.Lldp.Interfaces interfaces = update.getVal().getAnyVal().unpack(OpenconfigLldp.Lldp.Interfaces.class);
Anyone has thoughts on why there is a wrapper class in Java protoc output?
EDIT #3
Apparently looks like it is because of the "outer_class_name". In the Java code, I have an outer class "OpenconfigLldp".
The type_url format is:
type.googleapis.com/packagename.messagename
So, C++ code sets this to openconfig_lldp.Lldp.Interfaces.
But, this maps to OpenconfigLldp.Lldp.Interfaces in Java.
How could I work around this?
FINAL EDIT and FINAL QUESTION
After some digging around, this is what I found out.
By default, type_url is:
type_url: "type.googleapis.com/openconfig_lldp.Lldp.Interfaces"
On the Java side, I looked at the Any implementation. It tries to compare this with:
openconfig.openconfig_lldp.Lldp.Interfaces
I found this out by printing:
Lldp.Interfaces defaultInstance = (Lldp.Interfaces)Internal.getDefaultInstance(Lldp.Interfaces.class);
logger.info("full descriptor name: " + defaultInstance.getDescriptorForType().getFullName());
So, I hacked the C++ side to send:
update->mutable_val()->mutable_any_val()->set_type_url(std::string("type.googleapis.com/openconfig.openconfig_lldp.Lldp.Interfaces"));
So, I think I know what is happening here!
Thanks for reading through all the edits.
I am not sure if I understand correctly what's going wrong -- ideally the qualified names would be in protocol buffer namespaces -- language specific mappings shouldn't matter.
If the question is still open, I'd recommend to move the core of it to the top, preserving the edits as "what I have done so far".
Perhaps this is some kind of bug that could be worked around with on of these options:
java_multiple_files
java_outer_classname
More details about these options can be found here:
https://developers.google.com/protocol-buffers/docs/proto3

Can protobuf service method return primitive type?

I'm trying to use Google protobuf and i 'm having the next descriptions:
message.proto file:
message Request {
required int32 id = 1;
optional string value = 2;
}
service.proto file:
import "message.proto";
service Service {
rpc request (Request) returns (bool);
}
I'm trying to generate c++ sources and getting an error:
$ protoc service.proto --cpp_out=/tmp/proto/build
service.proto:4:40: Expected message type.
Do i have to return user-defined types only? Are primitive (like bool or string) supported? Can i use primitive types as service method argument (instead of Request in my example)?
No, you cannot use a primitive type as either the request or response. You must use a message type.
This is important because a message type can be extended later, in case you decide you want to add a new parameter or return some additional data.
If you want to return a primitive type, wrap it in a message and return it:
message Name {
string name = 1;
}
In case you don't want to return anything, void I mean, you can just create an empty message:
message Void {}
message Name {
string name = 1;
}
..
service MyService{
rpc MyFunc(Name) returns (Void);
}
You can return scalar datatypes like bool, int, etc by making use of wrappers.proto
service.proto file:
import "message.proto";
import "google/protobuf/wrappers.proto";
service Service {
rpc request (Request) returns (.google.protobuf.BoolValue);
}

Resources