Performant gRPC rich error handling in Go - go

Google docs propose the following model (https://cloud.google.com/apis/design/errors#error_model) for sending rich errors in gRPC but it seems that the error string is sent to the user every time. What I want to do is to send a code and then map it to a string when it reaches the client.
What I want to know is whatever the proto3 language supports writing data so that I would use it client-side, without defining a custom structure for the purposes of mapping error codes to error messages.

In your proto definition, define a simple enum with any extra error codes:
enum extraStatusCode {
UNKNOWN = 0; // not set/used
TOO_MANY_FOOS = 1;
NOT_ENOUGH_BARS = 2;
}
And include it as a top-level field in any returned message:
message User {
string uid = 1;
string email = 2;
// ...
extraStatusCode = 15;
}
if a message is sent with a non-zero extraStatusCode - then an edge case was encountered.

Related

Can I delete a field from a protobuf message?

I'm implementing a policy enforcement point between a client and a server that receives messages from the server, and, if the client doesn't have adequate authorization to see some parts of the message deletes those parts before sending them to the client.
message {
string not_sensitive = 1;
optional string sensitive = 2;
}
pseudo code
from_server >> my_msg;
if (!authorized) {
my_msg.delete("sensitive");
}
to_client << my_msg;
Yes.
As I understand current v3 pb schema language, all fields are optional. But regardless of that, a field marked optional in v2 is something that need not be there. So expanding your pseudo code to, say, C++ (see here), one can see that the generated class would end up with a has_sensitive() method and a clear_sensitive() method. Calling the latter and then serialising the object would result in wire format data that omitted the sensitive field.

Safeness of changing proto field number

For example if I changed from:
message Request {
int foo = 1;
}
to
message Request {
int bar = 1;
int foo = 2;
}
Is it safe to change foo from 1 to 2? Docs say not to: These numbers are used to identify your fields in the message binary format, and should not be changed once your message type is in use., but I'd like to know why.
If you have a serialized version of the message generated with the first version, you will no be able to deserialize with the second version of the message.
If you use protobuf to generate a model to store in a DB or to be published in a broker like Apache Kafka, you need to follow the convention. If you use proto buffer to generate model and service for online usage, you should do not break anything (if you will regenerate all the models)
See also the reserved keyword in order to do not reuse an old numer. Further reading also here

Best way to model gRPC messages

I would like to model messages for bidirectional streaming. In both directions I can expect different types of messages and I am unsure as to what the better practice would be. The two ideas as of now:
message MyMessage {
MessageType type = 1;
string payload = 2;
}
In this case I would have an enum that defines which type of message that is and a JSON payload that will be serialized and deserialized into models both client and sever side. The second approach is:
message MyMessage {
oneof type {
A typeA = 1;
B typeB = 2;
C typeC = 3;
}
}
In the second example a oneof is defined such that only one of the message types can be set. Both sides a switch must be made on each of the cases (A, B, C or None).
If you know all of the possible types ahead of time, using oneof would be the way to go here as you have described.
The major reason for using protocol buffers is the schema definition. With the schema definition, you get types, code generation, safe schema evolution, efficient encoding, etc. With the oneof approach, you will get these benefits for your nested payload.
I would not recommend using string payload since using a string for the actual payload removes many of the benefits of using protocol buffers. Also, even though you don't need a switch statement to deserialize the payload, you'll likely need a switch statement at some point to make use of the data (unless you're just forwarding the data on to some other system).
Alternative option
If you don't know the possible types ahead of time, you can use the Any type to embed an arbitrary protocol buffer message into your message.
import "google/protobuf/any.proto";
message MyMessage {
google.protobuf.Any payload = 1;
}
The Any message is basically your string payload approach, but using bytes instead of string.
message Any {
string type_url = 1;
bytes value = 2;
}
Using Any has the following advantages over string payload:
Encourages the use of protocol buffer to define the dynamic payload contents
Tooling in the protocol buffer library for each language for packing and unpacking protocol buffer messages into the Any type
For more information on Any, see:
https://developers.google.com/protocol-buffers/docs/proto3#any
https://github.com/protocolbuffers/protobuf/blob/master/src/google/protobuf/any.proto

Rule of thumb for designing protocol messages

I have a very simple music player, and I'd like to make it into a music server.
I plan in using gRPC to communicate between the clients and the server.
However, I'm not sure how I should design the protocol messages to handle the playback.
I envision two types of design :
A message for each type of query. This method defines clearly all possible actions, but seems to create a lot of redundant code.
message Play{
bool flag = 1; // False means Pause
}
message Stop{
bool flag = 1;
}
A unique message, with a key containing the action. This approach seems more flexible, but also more prone to errors. I could use an enum object to limits the possible actions though.
message Playback{
enum Action {
PLAY = 0;
STOP = 1;
}
Action action = 1;
}
Basically, I guess that what's I'm asking here is whether I should define an action by the type of the message or by its content.
Is there a rule of thumb or a design pattern to apply here ?
I would recommend to use the oneof construct here:
syntax = "proto3";
message Play {
}
message Stop {
}
message Command {
oneof command {
Play play = 1;
Stop stop = 2;
...
}
}
Empty messages are fine when there are no parameters that you need to pass, and this also leaves open an easy way to extend the messages in the future, for example changing Play to:
message Play {
string filename = 1;
}
would allow including an optional filename with the request, while retaining compatibility with the old version.

Protobuf: Nesting a message of arbitrary type

in short, is there a way to define a protobuf Message that contains another Message of arbitrary type? Something like:
message OuterMsg {
required int32 type = 1;
required Message nestedMsg = 2; //Any sort of message can go here
}
I suspect that there's a way to do this because in the various protobuf-implementations, compiled messages extend from a common Message base class.
Otherwise I guess I have to create a common base Message for all sorts of messages like this:
message BaseNestedMessage {
extensions 1 to max;
}
and then do
message OuterMessage {
required int32 type = 1;
required BaseNestedMessage nestedMsg = 2;
}
Is this the only way to achieve this?
The most popular way to do is to make optional fields for each message type:
message UnionMessage
{
optional MsgType1 msg1 = 1;
optional MsgType2 msg2 = 2;
optional MsgType3 msg3 = 3;
}
This technique is also described in the official Google documentation, and is well-supported across implementations:
https://developers.google.com/protocol-buffers/docs/techniques#union
Not directly, basically; protocol buffers very much wants to know the structure in advance, and the type of the message is not included on the wire. The common Message base-class is an implementation detail for providing common plumbing code - the protocol buffers specification does not include inheritance.
There are, therefore, limited options:
use different field-numbers per message-type
serialize the message separately, and include it as a bytes type, and convey the "what is this?" information separately (presumably a discriminator / enumeration)
I should also note that some implementations may provide more support for this; protobuf-net (C# / .NET) supports (separately) both inheritance and dynamic message-types (i.e. what you have above), but that is primarily intended for use only from that one library to that one library. Because this is all in addition to the specification (remaining 100% valid in terms of the wire format), it may be unnecessarily messy to interpret such data from other implementations.
Alternatively to multiple optional fields, the oneof keyword can be used since v2.6 of Protocol Buffers.
message UnionMessage {
oneof data {
string a = 1;
bytes b = 2;
int32 c = 3;
}
}

Resources