I have a protobuf message like this:
message MyMessage{
string foo = 1;
int toBeRemovedBar = 2 [deprecated = true];
string zag = 3;
}
toBeRemovedBar attribute is no longer required and needs to be removed.
As per the guidelines here and here, I could reserve either the field name or the number.
What is the best practice here?
Reserve either the field number or name
Reserve both field number and name
Reserving field number would prevent only the reuse of the field number. Reuse of the field name could not be prevented. Similar argument holds good for reserving only the field name.
So, the right way to remove an attribute is to reserve both field name and number as given below?
message MyMessage{
string foo = 1;
reserved 2;
reserved "toBeRemovedBar";
string zag = 3;
}
You should reserve the field number. This will prevent new fields from being added with the same tag, which would result in a wire incompatibility for any users running an old build that has the legacy field compiled in. Reusing the name is fine from a wire-compatibility standpoint, although doing so could cause some confusion.
It seems good to me.
In this way you prevent both the field number and the name from being used in future. I did the same in a work project because I had to be sure that both the field number and the name wouldn't be used again.
Related
If a field stores a specific value and is interpreted in a specific manner, is it possible to change this interpretation in a backwards compatible way?
Let's say I have a field that stores values of different data types.
The most generic case is to store it as a byte array and let the apps encode and decode it to the correct data type.
Common cases for data types are integers and strings, so support for those types is present.
Using a oneof structure this looks as follows:
message Foo
{
...
oneof value
{
uint32 integer = 1;
string text = 2;
bytes data = 3;
}
}
Applications that want to store an ip prefix in the value field, have to use the generic data field and do the encoding and decoding correctly.
If I now want to add support for ip prefixes to the Foo message itself so the apps don't have to deal with the encoding and decoding anymore, I could add a new field to the oneof structure with an IpPrefix datatype:
message Foo
{
...
oneof value
{
uint32 integer = 1;
string text = 2;
bytes data = 3;
IpPrefix ip_prefix = 4;
}
}
Even though this makes life easier for the apps, I believe it breaks backwards compatibility.
If a sending app has support for the new field, it will put its ip prefix value in the ip_prefix field.
But if a receiving app does not have support for this new field yet, it will ignore the field.
It will look for the ip prefix value in the data field, as it always did, but it won't find it there.
So the receiving app can no longer correctly read the ip prefix value anymore.
Is there a way to make this scenario somehow backwards compatible?
PS: I realize this is a somewhat vague and perhaps unrealistic example. The exact case I need it for is for the representation of RADIUS attributes in a protobuf message. These attributes are in essence a byte array that is sent over the network, but the bytes in the array have meaning and could be stored as different fields in the protobuf message. A basic attribute exists of a Type field and a Value field where the value field can be a string, integer, ip address... From time to time new datatypes (even complex ones) are added and I would like to be able to add new datatypes in a backwards compatible way.
There are two ways to go about this:
1. Enforce an update schedule, readers before writers
Add the new type of field to the .proto definition, but document that it should not be used except for testing and reception. Document that all readers of the message must support both the old and the new field by a specific milestone/date, after which the writers can start using it. Eventually you can deprecate the old field and new readers don't need to support it anymore.
2. Have both fields during the transition period
message Foo
{
...
oneof value
{
uint32 integer = 1;
string text = 2;
bytes data = 3;
}
IpPrefix ip_prefix = 4;
}
Document that writers should set both data and ip_prefix during the transition period. The readers can start using ip_prefix as soon as writers have added support, and can optionally fall back to data.
Later, you can deprecate data and move ip_prefix to inside the oneof without breaking compatibility.
I want to understand if the messages bellow are compatible from the perspective of protobuf and serialization/deserialization.
message HelloReply {
string message = 1;
string personalized_message = 2;
}
message HelloReply {
string personalized_message = 2;
string message = 1;
}
Does the order matter for compatibility in any situation?
The textual order is largely irrelevant, although it may impact some code generation tooling - but most languages don't care about declaration order, so even that: won't matter. The fields are still defined semantically equivalent - the numbers match the existing meaning (name) and type. It is the number that is the determining feature in identifying a field.
At the protocol level:
parsers must allow fields in any order
serializers should (but not must) write fields in ascending numerical field order
The Code
Consider the following protobuf message declaration:
syntax = "proto3";
enum Airport {
TLV = 0;
JFK = 1;
...
}
message Flight {
...
Airport origin_airport = 11;
...
}
The Problem
Due to some business requirements, we have to set the Airport to be a free string rather than choosing from a closed enumerated list. I know that I can add and remove fields at will, as long as I don't reuse the same number. However, I'm not sure if I can use the same name, a-la:
message Flight {
...
reserved 11; // The old enum field number
string origin_airport = 18; // Unused so far
...
}
My Question
Upon updaing a protobuf3 field type, can the field name be preserved, as long as its number changes?
If you aren't using the JSON variant, then names aren't used at all in the payload, so yes technically it is perfectly legal to reuse names; however: this might lead to unnecessary problems with existing code - depending on existing code and language / framework specific rules, and could cause confusion. Since that is avoidable, I would advocate using a name like origin_airport_code, or similar.
(The point I'm making here: any code that used the old field probably needs attention; I can see some scenarios where the existing code might still compile after the change, but mean something different, and therefore introduce a bug that would have been avoided if you'd changed the name and forced every usage to be visited)
I've got a situation where the other end of the grpc communication is not in sync with their releases. My higher ups, would like me to therefore add a field that is going to work if the other side does or doesn't fill it out, for a short time period (like two weeks)
I believe I can do this by adding it to the end of the proto message such that the indices for the other fields do not change. From, what I've Googled, the optional field is not avail prior to version 3.15, so I have to use a work around.
The workaround that was described to me was to use oneof. However, I am not 100% sure what that looks like. All examples show the oneof field by itself. Are the indices that belong to the oneof values indendent of the indices that belong to the rest of the message?
message TestMessage {
string somefield = 1;
int someotherfield = 2;
oneof mynewoptionalfield
{
string mynewfield = ???? Does this have to be 3 or is it 1?
int ifihadanother = ???? Does this need to be 4 or 2?
}
}
Questions:
What are the indices I use where the ??? marks are
Is this the proper work around to use when the other side isn't going to recompile and deploy with the changes to the protofile?
How do I then check if the field was filled in my C++ code?
Your use-case is exactly what protobufs were designed to handle. All you need to do is: add a new field to the message. In the easiest case, the client application code simply doesn't look at the new field until the server roll-out is complete and so doesn't notice sometimes it is present and other times missing.
You are correct that you should not change the indices (field ids) of the pre-existing fields. Although I'll note that you can add your new field anywhere within the message; the order the fields are written in does not matter for protobuf.
So you'd just add another field like:
message TestMessage {
string somefield = 1;
int someotherfield = 2;
string mynewfield = 3;
}
You don't have to use 3 as the id. You could use 4, or 10, or 10000. But small numbers are more efficient for protobuf and it is typical to just choose the "next" id. On-the-wire protobuf uses the id to identify the field, so it is important you don't change the id later.
In protobuf 3, all fields are "optional" in the protobuf 2 sense; there are no "required" fields. However, protobuf 2 also provided "field presence" for all fields. Protobuf 3 only provided field presence for oneofs and messages... until the recent re-introduction of the "optional" keyword.
In protobuf 3 if you call textMessage.getMynewfield() it will always return a non-null string. If the string was not sent, it will use the empty string (""). For integers 0 is returned and for messages the "default message" (all defaults) is returned. This is plenty for many use-cases, and may be enough for you.
But let's say you need to distinguish between "" and <notsent>. That's what field presence provides. Messages in protobuf 3 have "has" methods that return true if a value is present. But primitives don't have that presence information. One option is to "box" the primitive with standard wrappers that make the primitive a message. Another option available in newer versions of protobuf is the optional keyword. Both options will provide a method like textMessage.hasMynewfield().
message TestMessage {
string somefield = 1;
int someotherfield = 2;
google.protobuf.StringValue mynewfield = 3;
// -or-
optional string mynewfield = 3;
}
With protocol buffer, does changing field name of a message still let it compatible backward? I couldn't find any cite about that.
Eg: original message
message Person {
required string name = 1;
required int32 id = 2;
optional string email = 3;
}
Change to:
message Person {
required string full_name = 1;
required int32 id = 2;
optional string email = 3;
}
Changing a field name will not affect the protobuf encoding or compatibility between applications that use proto definitions which differ only by field names.
The binary protobuf encoding is based on tag numbers, so that is what you need to preserve.
You can even change a field type to some extent (check the type table at https://developers.google.com/protocol-buffers/docs/encoding#structure) providing its wire type stays the same, but that requires additional considerations whether, for example, changing uint32 to uint64 is safe from the point of view of your application code and, for some definition of 'better', is better than simply defining a new field.
Changing a field name will affect json representation, if you use that feature.