I have a protobuf message defined something like this
message Foo
{
oneof test //oneof field
{
int32 a = 1;
MM b = 2;
}
}
message MM
{
string str =1;
}
how i reflect of oneof filed in protobuf
For the most part, you handle a message with oneof the same way you would a message without:
message Foo
{
int32 a = 1;
MM b = 2;
}
Oneof is largely transparent to reflection, and doesn't affect the wire format. Its effect is that the generated setter code automatically clears other members of a oneof whenever one is set.
Now, if you care about oneofs for some reason, there's Descriptor::oneof_decl that allows you to enumerate them, Descriptor::FindOneofByName, and FieldDescriptor::containing_oneof (if you are working your way from the field up). With OneofDescriptor in hand, you can find its name and enumerate its fields, and that's pretty much it.
Related
I'm designing a protobuf to represent an event, where each event can hold extra fields.
There are a lot of possible extra fields (~100), but only a small portion of them will be used in each message (~3)
Each extra field will be used only once, but multiple of them can exist, therefore I would like to have a concept of an anyof message type, but unfortunately, there is no such thing in protobuf.
So to try and mock this behavior, and as mentioned in this discussion I thought I can put all my extra fields in a oneof, wrap it with a message, and use this message as repeated in my event:
message ExtraField {
oneof extra_field_value {
string extraData1 = 1;
uint64 extraData2 = 2;
....
SomeOtherMessage extraData100 = 100;
}
}
message MyEvent {
uint64 timestamp = 1;
string event_name = 2;
string some_other_data = 3;
...
repeated ExtraField extra_fields = 8;
}
Even though this solution is more explicit for my understanding, it isn't the most memory effective, and the repeated message with oneof implementation allows to add the same extra field more than once (unwanted behavior)
I can also just write all the extra fields as-is in an inner message, but most of them will be empty all the time
message ExtraFields {
string extraData1 = 1;
uint64 extraData2 = 2;
....
SomeOtherMessage extraData100 = 100;
}
message MyEvent {
uint64 timestamp = 1;
string event_name = 2;
string some_other_data = 3;
...
extraFields extra_fields = 8;
}
If I understand correctly, using empty fields in a message isn't going to make my serialized data larger, and therefore the second protobuf design is the preferred practice
Am I correct?
Is there another protobuf design for my needs?
I have a message of the following type
message Foo {
string bar = 1;
float baz = 2;
}
Is there any problem in converting it to the following for use in Go ?
message Foo {
string bar = 1;
optional float baz = 2;
}
Is the preferred way to deprecate and create a new field in the proto in this case as well ?
depends on how deeply integrated that particular message is in your code base - meaning
are you storing the marshaled binary representation somewhere like your database
are different parts of your code base using different versions of the message you are modifying - e.g. older versions of your android/ios apps and such
point being if you use a message structure to unmarshal encoded data, that was not generated with the very same message structure, into - bad things will happen.
The docs recommend adding a new element to circumvent such scenarios entirely. If that is not something you want to do, take the above points into consideration.
The optional will make the field a pointer type. So in Go generated code, optional float will become *float32, which of course is not float32.
To deprecate a field, use [deprecated = true] field option:
message Foo {
string bar = 1;
float baz = 2 [deprecated = true];
}
If in subsequent releases of your protobuf schema you actually remove the field altogether from the message, you might want to add reserved 2, where 2 is the number of the field you removed.
message Foo {
string bar = 1;
reserved 2;
}
This helps preventing other people or future you from adding a new field in position 2. This is relevant in case you have outdated clients which still expect a float in position 2.
PS: optional fieds in Proto3 are supported from version 3.15
I suggest you to use the FloatValue type defined in the google.protobuf package. As example:
syntax = "proto3";
import "google/protobuf/wrappers.proto";
message Foo {
string bar = 1;
google.protobuf.FloatValue baz = 2;
}
Will generate a pb files with the content:
type Foo struct {
state protoimpl.MessageState
sizeCache protoimpl.SizeCache
unknownFields protoimpl.UnknownFields
Bar string `protobuf:"bytes,1,opt,name=bar,proto3" json:"bar,omitempty"`
Baz *wrapperspb.FloatValue `protobuf:"bytes,2,opt,name=baz,proto3" json:"baz,omitempty"`
}
You can use as follow:
f := Foo{
Bar: "Bar",
Baz: &wrapperspb.FloatValue{Value: float32(3)},
}
var floatValue float32
if f.Baz != nil {
floatValue = f.Baz.GetValue()
}
What's the difference between using an Enum and a oneof kind in protobuf3? As far as I can tell, an Enum restricts the field to be one of a predefined set of values, but so does the oneof kind.
Enums are named numbers. You define the names in the enum definition and assign them a value. An enum should always have the value zero it it's set.
enum State {
A = 0;
B = 1;
C = 2;
}
Next you can use this enum in any of your message
message Update {
State currentState = 1;
State previousState = 2;
}
A oneof is something very different. It allows you to send different types but only allocate limited memory for them. This as you can only set one of those types at a time. This is similar to an union in C/C++ or a std::variant as of C++17.
Take this example in which we have a message, a integer and double defined in our oneof.
// The message in our oneof
message someMsg {
// Multiple fields
}
// The message holding our oneof
message msgWithOneof {
oneof theOneof {
someMsg msg = 1;
int32 counter = 2;
double value = 3;
}
// Feel free to add more fields her of before the oneof
}
You are only able to set msg, counter or value at one time. If you set another this will clear the other field.
Assuming an C/C++ implementation the largest field will determine the amount of memory allocated. Say someMsg is the largest, setting a integer or double will not be a problem as they fit in the same space. If you not use a oneof the total memory allocated would be the sum of sizeof(someMsg) + sizeof(int32) + sizeof(double).
There is some overhead to keep track which field has been set. In the google C++ implementation this is a bit in the presence variable. This is similar to fields which are marked optional.
What I am looking for is a function that returns the message of a repeated field
I know there is Reflection::AddMessage which has the return type that I want but I do not want to add a message, just return an object of that message.
Here is an example of what I am trying to do let's say I have in the .proto file a message:
message Bar{
uint32 t x = 1;
uint64 t y = 2;
}
message Foo{
repeated Bar myMessage = 1;
}
I am using reflection to iterate through the Foo message and I want to be able to do something like this:
Message* Msg = createMessage(refl->FooMsg, FieldDesc)
I know there is also GetRepeatedMessage but that requires index.
First of all when the protobuf compiler generates the code for compiling you get an accessor function in the interface. The are functions mutable_nameOf_message() which returns the entire repeated field which is a std::vector in c++, or mutable_nameOf_message( index ) which gives you the specified element.
Now if you do not want to use Bar then you d'not need too.
message ArrayOfBar
{
repeated Bar arrayOfBar = 0;
message Bar{
uint32 t x = 1;
uint64 t y = 2;
}
}
If thats what you have hade in mind you could also be do something like this.
std::vector<Bar> arrayOfBars;
But that idea needs refinement because of the internal specifics of the Protobuf. Some unwanted behavior might occur with something like that.
File my_protocol.proto:
Message MyProtocol {
required int32 protocolId = 1;
required **unknownType** protocolBody = 2;
}
I want to parse protocolBody according to protocolId.
For example,
protocolId = 10001 represents a login event, then the protocolBody should be a login message and can be parsed by login.proto.
I don't know which type should be set on the unknownType.
Protocol buffers parses the whole message in the one go.
You could have a message
Message MyProtocolGetId {
required int32 protocolId = 1;
}
Parse that then reparse with the correct message id. Alternatively the ProtocolBody could be stored as bytes and passed seperately
You can fake it, but it will be awkward. You have to use optional fields, and indicate which one if present using a flag. Make sure to double check that the field is set in your receiver code:
Message OneOfManyTypes {
required int32 kind = 1;
optional MessageType1 ty1 = 2;
optional MessageType2 ty2 = 3;
...
}