Suppose I have the following proto file.
syntax = "proto3";
package your.service.v1;
import "google/api/annotations.proto";
message FirstMessage {
string value = 1;
}
message SecondMessage {
string value = 1;
}
service YourService {
rpc Echo1(FirstMessage) returns (FirstMessage) {}
rpc Echo2(SecondMessage) returns (SecondMessage) {
option (my_option) = {
expose_for_web: true
};
}
}
I want to iterate over rpcs of my service and if the rpc contains an option (in this case Echo2 method) keep it and otherwise remove it.
The reason I want to do this is that I have an large project with lots of protos. I want to compile it to a lightweight Typescript code. I want to only compile the rpcs which have an specific annotation.
Is there any tool or library which I could use to modify and rewrite proto files?
Related
Protobuf provides service keyword that defines rpc-interface of one application.
I also want to use concept of entity which means that is part of service (one service contains multiple entities). Each entity type has own unique identifier that gives possibility to address different entities in service.
I would like to use proto like this
message UserReq {
string username = 1;
string password = 2;
}
message RegReq {
uint8 result_code = 1;
}
message RemoteEntityInterface
{
MyEntity entity = 1;
}
message GiveItemResult
{
uint8 result_code = 1;
}
service MyService {
rpc RegisterUser (UserReq) returns (RegReq) {}
rpc Login(UserReq) returns (RemoteEntityInterface) {}
}
entity MyEntity
{
rpc GiveItem (GiveItemReq) returns (GiveItemResult) {}
}
As you can see in example, I used unknown for protobuf keyword entity, this keyword means that MyService can return the interface to some remote object (MyEntity) by using Login remote method.
What are the ways to do this? (maybe write plugin or known way to modify source code of protobuf). Or maybe there are more flexible solutions than protobuf?
I also would like to use multiple parameters per one rpc; adding java-like attributes to rpc; service and entity; and data-model for entity (variables/fields) to add real-time replication support from entity to another service.
I think it is very flexible for services in game-development.
The only official way to extend .proto syntax is to define custom options.
For example, you could have something like:
extend google.protobuf.ServiceOptions {
optional bool is_entity = 123456;
}
service MyEntity
{
option (is_entity) = true;
rpc GiveItem (GiveItemReq) returns (GiveItemResult) {}
}
The default code generator will not do anything special with this option, but you can access it from your own code and from a protoc plugin if you write one.
I am trying to create a custom extension that I can use in a custom protoc compiler I am writing.
I am trying to compiled this protofile:
syntax = "proto3";
import "google/protobuf/descriptor.proto";
extend google.protobuf.MessageOptions {
string my_option = 5123456;
}
message TestMessageRequest {
option (my_option) = "Hello world!";
uint64 value1 = 2;
string value2 = 3;
}
The CodeGeneratorRequest that my protoc plugin gets doesn't seem to have this option anywhere. Here is the entire request: Google Drive Link.
Where can I get the value "Hello world!" for my TestMessageRequest message?
The option is there in the descriptor of TestMessageRequest:
"ProtoFile": [
{
"Name": "google/protobuf/descriptor.proto",
....
"HasOptions": true,
}
but because it is an extension field, it does not get converted into JSON.
You should be able to access it the same way that extension fields are accessed in whatever protobuf library you are using. There are examples for a few languages here: Python Protocol Buffer field options
I have a protobuf message like this:
message Update {
Path path = 1; // The path (key) for the update.
Value value = 2 [deprecated=true]; // The value (value) for the update.
TypedValue val = 3; // The explicitly typed update value.
}
// TypedValue is used to encode a value being sent between the client and
// target (originated by either entity).
message TypedValue {
oneof value {
string string_val = 1; // String value.
int64 int_val = 2; // Integer
....
google.protobuf.Any any_val = 9; // protobuf.Any encoded bytes.
....
}
}
On the server side (C++), we are setting this field as follows (LLDP is the outer class and Interfaces is inside that):
openconfig_lldp::Lldp out;
GetLldpProto(&out);
update->mutable_val()->mutable_any_val()->PackFrom(out.interfaces());
On the client side (Java), we are extracting this field like this:
OpenconfigLldp.Lldp.Interfaces interfaces = update.getVal().getAnyVal().unpack(OpenconfigLldp.Lldp.Interfaces.class);
This is throwing a InvalidProtocolBufferException exception. When I dump the "update" in my Java client, I see this:
path {
elem {
name: "lldp"
}
elem {
name: "interfaces"
}
}
val {
any_val {
type_url: "type.googleapis.com/openconfig_lldp.Lldp.Interfaces"
value: "\212\207\237\334\v\374\001\022\371\001\262\211\267l\031\342\367\304\260\002\v\n\tEth 1/1/1\242\340\247\230\017\002\b\001\352\316\234\250\017\324\001\262\217\304\272\017/\022-\302\340\317\247\001\'\202\225\377\302\001\b\n\006Neigh1\342\253\214\353\001\v\n\tEth 1/1/1\242\364\301\261\a\002\b\n\262\217\304\272\017/\022-\302\340\317\247\001\'\202\225\377\302\001\b\n\006Neigh2\342\253\214\353\001\v\n\tEth 1/1/2\242\364\301\261\a\002\b\n\262\217\304\272\017/\022-\302\340\317\247\001\'\202\225\377\302\001\b\n\006Neigh3\342\253\214\353\001\v\n\tEth 1/1/3\242\364\301\261\a\002\b\n\262\217\304\272\017/\022-\302\340\317\247\001\'\202\225\377\302\001\b\n\006Neigh4\342\253\214\353\001\v\n\tEth 1/1/4\242\364\301\261\a\002\b\n\212\207\237\334\v\374\001\022\371\001\262\211\267l\031\342\367\304\260\002\v\n\tEth 1/1/2\242\340\247\230\017\002\b\001\352\316\234\250\017\324\001\262\217\304\272\017/\022-\302\340\317\247\001\'\202\225\377\302\001\b\n\006Neigh1\342\253\214\353\001\v\n\tEth 1/1/1\242\364\301\261\a\002\b\n\262\217\304\272\017/\022-\302\340\317\247\001\'\202\225\377\302\001\b\n\006Neigh2\342\253\214\353\001\v\n\tEth 1/1/2\242\364\301\261\a\002\b\n\262\217\304\272\017/\022-\302\340\317\247\001\'\202\225\377\302\001\b\n\006Neigh3\342\253\214\353\001\v\n\tEth 1/1/3\242\364\301\261\a\002\b\n\262\217\304\272\017/\022-\302\340\317\247\001\'\202\225\377\302\001\b\n\006Neigh4\342\253\214\353\001\v\n\tEth 1/1/4\242\364\301\261\a\002\b\n\212\207\237\334\v\374\001\022\371\001\262\211\267l\031\342\367\304\260\002\v\n\tEth 1/1/3\242\340\247\230\017\002\b\001\352\316\234\250\017\324\001\262\217\304\272\017/\022-\302\340\317\247\001\'\202\225\377\302\001\b\n\006Neigh1\342\253\214\353\001\v\n\tEth 1/1/1\242\364\301\261\a\002\b\n\262\217\304\272\017/\022-\302\340\317\247\001\'\202\225\377\302\001\b\n\006Neigh2\342\253\214\353\001\v\n\tEth 1/1/2\242\364\301\261\a\002\b\n\262\217\304\272\017/\022-\302\340\317\247\001\'\202\225\377\302\001\b\n\006Neigh3\342\253\214\353\001\v\n\tEth 1/1/3\242\364\301\261\a\002\b\n\262\217\304\272\017/\022-\302\340\317\247\001\'\202\225\377\302\001\b\n\006Neigh4\342\253\214\353\001\v\n\tEth 1/1/4\242\364\301\261\a\002\b\n\212\207\237\334\v\374\001\022\371\001\262\211\267l\031\342\367\304\260\002\v\n\tEth 1/1/4\242\340\247\230\017\002\b\001\352\316\234\250\017\324\001\262\217\304\272\017/\022-\302\340\317\247\001\'\202\225\377\302\001\b\n\006Neigh1\342\253\214\353\001\v\n\tEth 1/1/1\242\364\301\261\a\002\b\n\262\217\304\272\017/\022-\302\340\317\247\001\'\202\225\377\302\001\b\n\006Neigh2\342\253\214\353\001\v\n\tEth 1/1/2\242\364\301\261\a\002\b\n\262\217\304\272\017/\022-\302\340\317\247\001\'\202\225\377\302\001\b\n\006Neigh3\342\253\214\353\001\v\n\tEth 1/1/3\242\364\301\261\a\002\b\n\262\217\304\272\017/\022-\302\340\317\247\001\'\202\225\377\302\001\b\n\006Neigh4\342\253\214\353\001\v\n\tEth 1/1/4\242\364\301\261\a\002\b\n"
}
}
The type_url seems correct to me. What am I doing wrong here?
Thanks for your time.
EDIT #1:
I looked at the exception string. It is "Type of the Any message does not match the given class."
The same proto file is used for C++ and Java, but I see "openconfig_lldp.Lldp.Interfaces" in C++, where as, it is "OpenconfigLldp.Lldp.Interfaces" in Java. Need to find out why..
EDIT #2:
The same .proto file is used. In this case, it is:
openconfig_lldp.proto
---------------------
syntax = "proto3";
package openconfig.openconfig_lldp;
message Lldp {
message Config {
....
....
}
....
....
}
In case of Java, I see the parent class as OpenconfigLldp in a package called openconfig_lldp.
package openconfig.openconfig_lldp;
public final class OpenconfigLldp {
private OpenconfigLldp() {}
....
....
/**
* Protobuf type {#code openconfig.openconfig_lldp.Lldp}
*/
public static final class Lldp extends com.google.protobuf.GeneratedMessageV3 implements
// ##protoc_insertion_point(message_implements:openconfig.openconfig_lldp.Lldp)
....
....
}
In C++, I don't see any class called "OpenconfigLldp" generated. Instead it is just "Lldp"
So, the type_url in the Any.protobuf is a mismatch. C++ side puts it as
type_url: "type.googleapis.com/openconfig_lldp.Lldp.Interfaces"
While in the Java side I use:
OpenconfigLldp.Lldp.Interfaces interfaces = update.getVal().getAnyVal().unpack(OpenconfigLldp.Lldp.Interfaces.class);
Anyone has thoughts on why there is a wrapper class in Java protoc output?
EDIT #3
Apparently looks like it is because of the "outer_class_name". In the Java code, I have an outer class "OpenconfigLldp".
The type_url format is:
type.googleapis.com/packagename.messagename
So, C++ code sets this to openconfig_lldp.Lldp.Interfaces.
But, this maps to OpenconfigLldp.Lldp.Interfaces in Java.
How could I work around this?
FINAL EDIT and FINAL QUESTION
After some digging around, this is what I found out.
By default, type_url is:
type_url: "type.googleapis.com/openconfig_lldp.Lldp.Interfaces"
On the Java side, I looked at the Any implementation. It tries to compare this with:
openconfig.openconfig_lldp.Lldp.Interfaces
I found this out by printing:
Lldp.Interfaces defaultInstance = (Lldp.Interfaces)Internal.getDefaultInstance(Lldp.Interfaces.class);
logger.info("full descriptor name: " + defaultInstance.getDescriptorForType().getFullName());
So, I hacked the C++ side to send:
update->mutable_val()->mutable_any_val()->set_type_url(std::string("type.googleapis.com/openconfig.openconfig_lldp.Lldp.Interfaces"));
So, I think I know what is happening here!
Thanks for reading through all the edits.
I am not sure if I understand correctly what's going wrong -- ideally the qualified names would be in protocol buffer namespaces -- language specific mappings shouldn't matter.
If the question is still open, I'd recommend to move the core of it to the top, preserving the edits as "what I have done so far".
Perhaps this is some kind of bug that could be worked around with on of these options:
java_multiple_files
java_outer_classname
More details about these options can be found here:
https://developers.google.com/protocol-buffers/docs/proto3
I am writing protoc plugin in Go which should generate documentation for our GRPC services and currently struggle in attempt to know right order of options.
First, how the protobuf looks like
syntax = "proto3";
option go_package = "sample";
package sample
import "common/extensions.proto"
message SimpleMessage {
// Id represents the message identifier.
string id = 1;
int64 num = 2;
}
message Response {
int32 code = 1;
}
enum ErrorCodes {
RESERVED = 0;
OK = 200
ERROR = 6000
PANIC = 6001
}
service EchoService {
rpc Echo (SimpleMessage) returns (Response) {
// common.grpc_status is an extension defined somewhere
// these are list of possible return statuses
option (common.grpc_status) = {
status: "OK"
status: "ERROR"
status: "PANIC" // Every status string will must be one of ErrorCodes items
};
option (common.middlewares) = {
middleware: "csrf"
middleware: "session"
}
}
};
As you see, there're two options here. The problem is protoc doesn't bind position directly to tokens. It leaves this information in a special sections where it can be restored via using so called "paths". And these paths are rely on order, while options are hidden and can only be retrieved using proto.GetExtension function which doesn't report option index either.
I need this token location information to report errors. Is there any way to get option index or something equivalent?
I am thinking about using standalone parser just to retrieve the right order, but this feels somewhat awkward. Hope there's a better way.
I'm trying to use Google protobuf and i 'm having the next descriptions:
message.proto file:
message Request {
required int32 id = 1;
optional string value = 2;
}
service.proto file:
import "message.proto";
service Service {
rpc request (Request) returns (bool);
}
I'm trying to generate c++ sources and getting an error:
$ protoc service.proto --cpp_out=/tmp/proto/build
service.proto:4:40: Expected message type.
Do i have to return user-defined types only? Are primitive (like bool or string) supported? Can i use primitive types as service method argument (instead of Request in my example)?
No, you cannot use a primitive type as either the request or response. You must use a message type.
This is important because a message type can be extended later, in case you decide you want to add a new parameter or return some additional data.
If you want to return a primitive type, wrap it in a message and return it:
message Name {
string name = 1;
}
In case you don't want to return anything, void I mean, you can just create an empty message:
message Void {}
message Name {
string name = 1;
}
..
service MyService{
rpc MyFunc(Name) returns (Void);
}
You can return scalar datatypes like bool, int, etc by making use of wrappers.proto
service.proto file:
import "message.proto";
import "google/protobuf/wrappers.proto";
service Service {
rpc request (Request) returns (.google.protobuf.BoolValue);
}