I have a protobuf message like this:
message Update {
Path path = 1; // The path (key) for the update.
Value value = 2 [deprecated=true]; // The value (value) for the update.
TypedValue val = 3; // The explicitly typed update value.
}
// TypedValue is used to encode a value being sent between the client and
// target (originated by either entity).
message TypedValue {
oneof value {
string string_val = 1; // String value.
int64 int_val = 2; // Integer
....
google.protobuf.Any any_val = 9; // protobuf.Any encoded bytes.
....
}
}
On the server side (C++), we are setting this field as follows (LLDP is the outer class and Interfaces is inside that):
openconfig_lldp::Lldp out;
GetLldpProto(&out);
update->mutable_val()->mutable_any_val()->PackFrom(out.interfaces());
On the client side (Java), we are extracting this field like this:
OpenconfigLldp.Lldp.Interfaces interfaces = update.getVal().getAnyVal().unpack(OpenconfigLldp.Lldp.Interfaces.class);
This is throwing a InvalidProtocolBufferException exception. When I dump the "update" in my Java client, I see this:
path {
elem {
name: "lldp"
}
elem {
name: "interfaces"
}
}
val {
any_val {
type_url: "type.googleapis.com/openconfig_lldp.Lldp.Interfaces"
value: "\212\207\237\334\v\374\001\022\371\001\262\211\267l\031\342\367\304\260\002\v\n\tEth 1/1/1\242\340\247\230\017\002\b\001\352\316\234\250\017\324\001\262\217\304\272\017/\022-\302\340\317\247\001\'\202\225\377\302\001\b\n\006Neigh1\342\253\214\353\001\v\n\tEth 1/1/1\242\364\301\261\a\002\b\n\262\217\304\272\017/\022-\302\340\317\247\001\'\202\225\377\302\001\b\n\006Neigh2\342\253\214\353\001\v\n\tEth 1/1/2\242\364\301\261\a\002\b\n\262\217\304\272\017/\022-\302\340\317\247\001\'\202\225\377\302\001\b\n\006Neigh3\342\253\214\353\001\v\n\tEth 1/1/3\242\364\301\261\a\002\b\n\262\217\304\272\017/\022-\302\340\317\247\001\'\202\225\377\302\001\b\n\006Neigh4\342\253\214\353\001\v\n\tEth 1/1/4\242\364\301\261\a\002\b\n\212\207\237\334\v\374\001\022\371\001\262\211\267l\031\342\367\304\260\002\v\n\tEth 1/1/2\242\340\247\230\017\002\b\001\352\316\234\250\017\324\001\262\217\304\272\017/\022-\302\340\317\247\001\'\202\225\377\302\001\b\n\006Neigh1\342\253\214\353\001\v\n\tEth 1/1/1\242\364\301\261\a\002\b\n\262\217\304\272\017/\022-\302\340\317\247\001\'\202\225\377\302\001\b\n\006Neigh2\342\253\214\353\001\v\n\tEth 1/1/2\242\364\301\261\a\002\b\n\262\217\304\272\017/\022-\302\340\317\247\001\'\202\225\377\302\001\b\n\006Neigh3\342\253\214\353\001\v\n\tEth 1/1/3\242\364\301\261\a\002\b\n\262\217\304\272\017/\022-\302\340\317\247\001\'\202\225\377\302\001\b\n\006Neigh4\342\253\214\353\001\v\n\tEth 1/1/4\242\364\301\261\a\002\b\n\212\207\237\334\v\374\001\022\371\001\262\211\267l\031\342\367\304\260\002\v\n\tEth 1/1/3\242\340\247\230\017\002\b\001\352\316\234\250\017\324\001\262\217\304\272\017/\022-\302\340\317\247\001\'\202\225\377\302\001\b\n\006Neigh1\342\253\214\353\001\v\n\tEth 1/1/1\242\364\301\261\a\002\b\n\262\217\304\272\017/\022-\302\340\317\247\001\'\202\225\377\302\001\b\n\006Neigh2\342\253\214\353\001\v\n\tEth 1/1/2\242\364\301\261\a\002\b\n\262\217\304\272\017/\022-\302\340\317\247\001\'\202\225\377\302\001\b\n\006Neigh3\342\253\214\353\001\v\n\tEth 1/1/3\242\364\301\261\a\002\b\n\262\217\304\272\017/\022-\302\340\317\247\001\'\202\225\377\302\001\b\n\006Neigh4\342\253\214\353\001\v\n\tEth 1/1/4\242\364\301\261\a\002\b\n\212\207\237\334\v\374\001\022\371\001\262\211\267l\031\342\367\304\260\002\v\n\tEth 1/1/4\242\340\247\230\017\002\b\001\352\316\234\250\017\324\001\262\217\304\272\017/\022-\302\340\317\247\001\'\202\225\377\302\001\b\n\006Neigh1\342\253\214\353\001\v\n\tEth 1/1/1\242\364\301\261\a\002\b\n\262\217\304\272\017/\022-\302\340\317\247\001\'\202\225\377\302\001\b\n\006Neigh2\342\253\214\353\001\v\n\tEth 1/1/2\242\364\301\261\a\002\b\n\262\217\304\272\017/\022-\302\340\317\247\001\'\202\225\377\302\001\b\n\006Neigh3\342\253\214\353\001\v\n\tEth 1/1/3\242\364\301\261\a\002\b\n\262\217\304\272\017/\022-\302\340\317\247\001\'\202\225\377\302\001\b\n\006Neigh4\342\253\214\353\001\v\n\tEth 1/1/4\242\364\301\261\a\002\b\n"
}
}
The type_url seems correct to me. What am I doing wrong here?
Thanks for your time.
EDIT #1:
I looked at the exception string. It is "Type of the Any message does not match the given class."
The same proto file is used for C++ and Java, but I see "openconfig_lldp.Lldp.Interfaces" in C++, where as, it is "OpenconfigLldp.Lldp.Interfaces" in Java. Need to find out why..
EDIT #2:
The same .proto file is used. In this case, it is:
openconfig_lldp.proto
---------------------
syntax = "proto3";
package openconfig.openconfig_lldp;
message Lldp {
message Config {
....
....
}
....
....
}
In case of Java, I see the parent class as OpenconfigLldp in a package called openconfig_lldp.
package openconfig.openconfig_lldp;
public final class OpenconfigLldp {
private OpenconfigLldp() {}
....
....
/**
* Protobuf type {#code openconfig.openconfig_lldp.Lldp}
*/
public static final class Lldp extends com.google.protobuf.GeneratedMessageV3 implements
// ##protoc_insertion_point(message_implements:openconfig.openconfig_lldp.Lldp)
....
....
}
In C++, I don't see any class called "OpenconfigLldp" generated. Instead it is just "Lldp"
So, the type_url in the Any.protobuf is a mismatch. C++ side puts it as
type_url: "type.googleapis.com/openconfig_lldp.Lldp.Interfaces"
While in the Java side I use:
OpenconfigLldp.Lldp.Interfaces interfaces = update.getVal().getAnyVal().unpack(OpenconfigLldp.Lldp.Interfaces.class);
Anyone has thoughts on why there is a wrapper class in Java protoc output?
EDIT #3
Apparently looks like it is because of the "outer_class_name". In the Java code, I have an outer class "OpenconfigLldp".
The type_url format is:
type.googleapis.com/packagename.messagename
So, C++ code sets this to openconfig_lldp.Lldp.Interfaces.
But, this maps to OpenconfigLldp.Lldp.Interfaces in Java.
How could I work around this?
FINAL EDIT and FINAL QUESTION
After some digging around, this is what I found out.
By default, type_url is:
type_url: "type.googleapis.com/openconfig_lldp.Lldp.Interfaces"
On the Java side, I looked at the Any implementation. It tries to compare this with:
openconfig.openconfig_lldp.Lldp.Interfaces
I found this out by printing:
Lldp.Interfaces defaultInstance = (Lldp.Interfaces)Internal.getDefaultInstance(Lldp.Interfaces.class);
logger.info("full descriptor name: " + defaultInstance.getDescriptorForType().getFullName());
So, I hacked the C++ side to send:
update->mutable_val()->mutable_any_val()->set_type_url(std::string("type.googleapis.com/openconfig.openconfig_lldp.Lldp.Interfaces"));
So, I think I know what is happening here!
Thanks for reading through all the edits.
I am not sure if I understand correctly what's going wrong -- ideally the qualified names would be in protocol buffer namespaces -- language specific mappings shouldn't matter.
If the question is still open, I'd recommend to move the core of it to the top, preserving the edits as "what I have done so far".
Perhaps this is some kind of bug that could be worked around with on of these options:
java_multiple_files
java_outer_classname
More details about these options can be found here:
https://developers.google.com/protocol-buffers/docs/proto3
Related
Protobuf provides service keyword that defines rpc-interface of one application.
I also want to use concept of entity which means that is part of service (one service contains multiple entities). Each entity type has own unique identifier that gives possibility to address different entities in service.
I would like to use proto like this
message UserReq {
string username = 1;
string password = 2;
}
message RegReq {
uint8 result_code = 1;
}
message RemoteEntityInterface
{
MyEntity entity = 1;
}
message GiveItemResult
{
uint8 result_code = 1;
}
service MyService {
rpc RegisterUser (UserReq) returns (RegReq) {}
rpc Login(UserReq) returns (RemoteEntityInterface) {}
}
entity MyEntity
{
rpc GiveItem (GiveItemReq) returns (GiveItemResult) {}
}
As you can see in example, I used unknown for protobuf keyword entity, this keyword means that MyService can return the interface to some remote object (MyEntity) by using Login remote method.
What are the ways to do this? (maybe write plugin or known way to modify source code of protobuf). Or maybe there are more flexible solutions than protobuf?
I also would like to use multiple parameters per one rpc; adding java-like attributes to rpc; service and entity; and data-model for entity (variables/fields) to add real-time replication support from entity to another service.
I think it is very flexible for services in game-development.
The only official way to extend .proto syntax is to define custom options.
For example, you could have something like:
extend google.protobuf.ServiceOptions {
optional bool is_entity = 123456;
}
service MyEntity
{
option (is_entity) = true;
rpc GiveItem (GiveItemReq) returns (GiveItemResult) {}
}
The default code generator will not do anything special with this option, but you can access it from your own code and from a protoc plugin if you write one.
Suppose I have the following proto file.
syntax = "proto3";
package your.service.v1;
import "google/api/annotations.proto";
message FirstMessage {
string value = 1;
}
message SecondMessage {
string value = 1;
}
service YourService {
rpc Echo1(FirstMessage) returns (FirstMessage) {}
rpc Echo2(SecondMessage) returns (SecondMessage) {
option (my_option) = {
expose_for_web: true
};
}
}
I want to iterate over rpcs of my service and if the rpc contains an option (in this case Echo2 method) keep it and otherwise remove it.
The reason I want to do this is that I have an large project with lots of protos. I want to compile it to a lightweight Typescript code. I want to only compile the rpcs which have an specific annotation.
Is there any tool or library which I could use to modify and rewrite proto files?
I am fairly new to API building, so this may be a broader question than I originally posed.
I am creating an API in Golang (using protobuf 3 and gRPC) that has two similar endpoints:
GET /project/genres
GET /project/{id}
The problem is that when I run curl localhost:8080/project/genres, the pattern matching results in the /project/{id} endpoint getting called with genres as the id. Is there some simple way around this, or do I have to build something into the server code to call the proper function based on the type?
I also tried flipping the ordering of these definitions, just in case the pattern matching had some order of operations that I didn't know about, but this didn't make a difference.
Here are the definitions in my proto file:
message EmptyRequest { }
message ProjectRequest {
string id = 1;
}
message GenreResponse {
int32 id = 1;
string name = 2;
}
message ProjectResponse {
int32 id = 1;
string name = 2;
}
service ProjectService {
rpc GetProject(ProjectRequest) returns (ProjectResponse) {
option (google.api.http) = {
get: "/v1/project/{id}"
};
}
rpc GetGenres(EmptyRequest) returns (GenreResponse) {
option (google.api.http) = {
get: "/v1/project/genres"
};
}
}
I was able to specify a check in the url path template to get around this issue:
rpc GetGenres(EmptyRequest) returns (GenreResponse) {
option (google.api.http) = {
get: "/v1/project/{id=genres}"
};
}
This seems to have fixed the problem. I don't know if there are other solutions, or if this is the right way to do this, but I'm happy to accept other answers if something better comes in.
I'm wondering what would be the best way to ignore/discard the unknown enum values in GraphQL/Apollo server.
Let's say my GraphQL schema defines array of enums "enum Service { Supermarket, TicketSales }" and it works fine now, but later on other service I'm using is adding some new values (e.g. Playground) and my client just doesn't support it and I would just like to ignore it and return the supported values without error.
What would be the best way to do this in GraphQL. My first idea was to make directive that would read the supported values from schema and ignore everything else, but after googling around I didn't find any good examples how to do it. Can you point me a direction where to go about this?
If your resolver function will accept arbitrary strings, then you can use a custom scalar type, or just String.
"""
The type of a service. `Supermarket` means..., and
`TicketSales` means...; any other value is ignored.
"""
scalar Service
GraphQL generally places responsibility on the client to conform to the server's expectations, rather than making the server try to support any request. There are a couple of places you can reasonably expect an enum value like this to appear:
enum Service { Supermarket, TicketSales }
type Query {
inAReturnValue: Service!
asAQueryParam(service: Service!): Node
}
type Mutation {
asAMutationInput(service: Service!): Node
}
In particular it may not make sense to tell the server "make the type of this object be a playground" if the server just doesn't understand that. Conversely, if the server knows about "playground", it could return it in cases the client may not expect. Having an enum here makes it explicit what the server knows about. The server has said what it supports and it's the client's responsibility to cooperate.
Note that it's possible for the client to find out if the server supports playgrounds, if it's an enum value, and this might help it inform its behavior.
query GetServiceTypes {
__type(name: "Service") {
enumValues { name }
}
}
After playing around I found something that I can use to get around my original problem, so I will post it here in case somebody else is wondering the same thing.
So my original problem was in short that I'm receiving several different "available services" kind of string arrays from another services and I was thinking to map them to enum for better typescript support etc. But the problem was that if I get some unknown value from another service, my graphql will fail.
So my original idea was to fix it with directive which I after all got working:
# In schema
directive #mapUnknownTo(value: String) on ENUM
enum SomeAttribute #mapUnknownTo(value: "__UNKNOWN__") {
SomeAttribute1
AnotherAttribute
SomethingElse
__UNKNOWN__
}
And the directive implementation is:
import { SchemaDirectiveVisitor } from 'graphql-tools';
import { GraphQLEnumType } from 'graphql';
export class MapUnknownToDirective extends SchemaDirectiveVisitor {
visitEnum(type: GraphQLEnumType) {
const { value = '__UNKNOWN__' } = this.args;
const valueMap = type.getValues().reduce((map, v) => map.set(v.value, v.name), new Map<string, string>());
type.serialize = (v: string): string => valueMap.get(v) || value;
}
}
So this will map all the values not defined in schema into some custom value, which is not exactly what I originally wanted, but at least it's not giving an error, so it's okay-ish.
I'm still not 100% sure if directives are way to go on cases like this, but at least it's one possible solution.
I am writing protoc plugin in Go which should generate documentation for our GRPC services and currently struggle in attempt to know right order of options.
First, how the protobuf looks like
syntax = "proto3";
option go_package = "sample";
package sample
import "common/extensions.proto"
message SimpleMessage {
// Id represents the message identifier.
string id = 1;
int64 num = 2;
}
message Response {
int32 code = 1;
}
enum ErrorCodes {
RESERVED = 0;
OK = 200
ERROR = 6000
PANIC = 6001
}
service EchoService {
rpc Echo (SimpleMessage) returns (Response) {
// common.grpc_status is an extension defined somewhere
// these are list of possible return statuses
option (common.grpc_status) = {
status: "OK"
status: "ERROR"
status: "PANIC" // Every status string will must be one of ErrorCodes items
};
option (common.middlewares) = {
middleware: "csrf"
middleware: "session"
}
}
};
As you see, there're two options here. The problem is protoc doesn't bind position directly to tokens. It leaves this information in a special sections where it can be restored via using so called "paths". And these paths are rely on order, while options are hidden and can only be retrieved using proto.GetExtension function which doesn't report option index either.
I need this token location information to report errors. Is there any way to get option index or something equivalent?
I am thinking about using standalone parser just to retrieve the right order, but this feels somewhat awkward. Hope there's a better way.