File my_protocol.proto:
Message MyProtocol {
required int32 protocolId = 1;
required **unknownType** protocolBody = 2;
}
I want to parse protocolBody according to protocolId.
For example,
protocolId = 10001 represents a login event, then the protocolBody should be a login message and can be parsed by login.proto.
I don't know which type should be set on the unknownType.
Protocol buffers parses the whole message in the one go.
You could have a message
Message MyProtocolGetId {
required int32 protocolId = 1;
}
Parse that then reparse with the correct message id. Alternatively the ProtocolBody could be stored as bytes and passed seperately
You can fake it, but it will be awkward. You have to use optional fields, and indicate which one if present using a flag. Make sure to double check that the field is set in your receiver code:
Message OneOfManyTypes {
required int32 kind = 1;
optional MessageType1 ty1 = 2;
optional MessageType2 ty2 = 3;
...
}
Related
I'm designing a protobuf to represent an event, where each event can hold extra fields.
There are a lot of possible extra fields (~100), but only a small portion of them will be used in each message (~3)
Each extra field will be used only once, but multiple of them can exist, therefore I would like to have a concept of an anyof message type, but unfortunately, there is no such thing in protobuf.
So to try and mock this behavior, and as mentioned in this discussion I thought I can put all my extra fields in a oneof, wrap it with a message, and use this message as repeated in my event:
message ExtraField {
oneof extra_field_value {
string extraData1 = 1;
uint64 extraData2 = 2;
....
SomeOtherMessage extraData100 = 100;
}
}
message MyEvent {
uint64 timestamp = 1;
string event_name = 2;
string some_other_data = 3;
...
repeated ExtraField extra_fields = 8;
}
Even though this solution is more explicit for my understanding, it isn't the most memory effective, and the repeated message with oneof implementation allows to add the same extra field more than once (unwanted behavior)
I can also just write all the extra fields as-is in an inner message, but most of them will be empty all the time
message ExtraFields {
string extraData1 = 1;
uint64 extraData2 = 2;
....
SomeOtherMessage extraData100 = 100;
}
message MyEvent {
uint64 timestamp = 1;
string event_name = 2;
string some_other_data = 3;
...
extraFields extra_fields = 8;
}
If I understand correctly, using empty fields in a message isn't going to make my serialized data larger, and therefore the second protobuf design is the preferred practice
Am I correct?
Is there another protobuf design for my needs?
I have a protobuf message defined something like this
message Foo
{
oneof test //oneof field
{
int32 a = 1;
MM b = 2;
}
}
message MM
{
string str =1;
}
how i reflect of oneof filed in protobuf
For the most part, you handle a message with oneof the same way you would a message without:
message Foo
{
int32 a = 1;
MM b = 2;
}
Oneof is largely transparent to reflection, and doesn't affect the wire format. Its effect is that the generated setter code automatically clears other members of a oneof whenever one is set.
Now, if you care about oneofs for some reason, there's Descriptor::oneof_decl that allows you to enumerate them, Descriptor::FindOneofByName, and FieldDescriptor::containing_oneof (if you are working your way from the field up). With OneofDescriptor in hand, you can find its name and enumerate its fields, and that's pretty much it.
What I am looking for is a function that returns the message of a repeated field
I know there is Reflection::AddMessage which has the return type that I want but I do not want to add a message, just return an object of that message.
Here is an example of what I am trying to do let's say I have in the .proto file a message:
message Bar{
uint32 t x = 1;
uint64 t y = 2;
}
message Foo{
repeated Bar myMessage = 1;
}
I am using reflection to iterate through the Foo message and I want to be able to do something like this:
Message* Msg = createMessage(refl->FooMsg, FieldDesc)
I know there is also GetRepeatedMessage but that requires index.
First of all when the protobuf compiler generates the code for compiling you get an accessor function in the interface. The are functions mutable_nameOf_message() which returns the entire repeated field which is a std::vector in c++, or mutable_nameOf_message( index ) which gives you the specified element.
Now if you do not want to use Bar then you d'not need too.
message ArrayOfBar
{
repeated Bar arrayOfBar = 0;
message Bar{
uint32 t x = 1;
uint64 t y = 2;
}
}
If thats what you have hade in mind you could also be do something like this.
std::vector<Bar> arrayOfBars;
But that idea needs refinement because of the internal specifics of the Protobuf. Some unwanted behavior might occur with something like that.
I get an error when I try to compile a proto file to convert to .java.
Could you point out what I'm missing ?
protoc --proto_path=src\main\resources\proto --java_out=src\main\java src\main\resources\proto\PayloadProtocol.proto
PayloadProtocol.proto:32:14: "DataContainer" is not defined.
PayloadProtocol.proto: warning: Import BackendCommunicationService.proto but not used.
Payload.proto
import "BackendCommunicationService.proto";
package com.fleetboard.tp.payload.protocol.protobuf;
option java_package = "com.fleetboard.tp.proto.protocol";
message TPMessage {
required int32 serviceId = 1; // telematic service (TS) id, who owns this message
required int32 functionId = 2; // function id refers to the Java class for the payload
optional uint64 requestId = 3; // Identifier to associate the request to a response
optional TPPayload payload = 4; // serialized representation of a TP message
optional uint64 durability = 5; // life time of message - used from backend
optional DataContainer dataPayload = 6;**
}
BackendCommunicationService.proto
package com.fleetboard.tp.backend.protobuf;
option java_package = "com.fleetboard.tp.proto.backend";
message DataContainer {
required DeviceApplication application = 1; // The container's recipient (MT) or sender (MO)
required string fileName = 2; // File name (no path), length up to 255
required uint64 fileTime = 3; // File time as ms since 1970-01-01 00:00 UTC
}
Fully qualify the name in the importing file:
com.fleetboard.tp.backend.protobuf.DataContainer
or
.com.fleetboard.tp.backend.protobuf.DataContainer
(the . ensures it starts at the root)
You could also try using just the disjoint part, but I don't know if it will work:
backend.protobuf.DataContainer
(since both have the com.fleetboard.tp. prefix)
I am writing adapter class (library function) which will take different kind of PB messages as the input in the form of std::Map and serialize this map write in to the file then vice versa.
Example:
message user_defined_type
{
optional int Val1 = 1;
optional string Val2 = 2;
}
message Store
{
optional int Key = 1;
optional user_defined_type Value = 2;
}
The client will create std::Map and stores the above message (i.e., std::map XYZ). The library takes the std::Map as input and does serializing the message and store it in to the file. But the library don't have/know the Proto message definitions.
To achieve the above came up with an approach, the library will have intermediate proto message which has both the fields are byte type
message MAP
{
optional byte KeyField = 1;
optional byte ValueField = 2;
}
Such that the KeyField takes has value of Store::Key and ValueField has the value of Store::user_defined_type so the serialization and de-serialization will be generic for all type of messages.
In C# using the protobuf.serializer.serialize I can serialize/de-serialize to the designated type but in C++ don't know how to make it, any help/pointer much appreciated.
If I understand correctly, the challenge is that your library needs to know how to parse ValueField (and perhaps KeyField) but the library itself does not know their types; only the caller of the library does.
The way to solve this is to have the caller pass in a "prototype" instance of their message type, from which you can spawn additional instances:
map<string, Message*> parse(string data, const Message& prototype) {
map<string, Message*> result;
MapProto proto;
proto.ParseFromString(data);
for (int i = 0; i < proto.entry_size(); i++) {
Message* value = prototype->New();
value->ParseFromString(proto.entry(i).value());
result[proto.entry(i).key()] = value;
}
return result;
}
The caller would call this function like:
map<string, Message*> items = parse(data, MyValueType::default_instance());
Then, each message in the returned map will be an instance of MyValueType (the caller can use static_cast to cast the pointer to that type). The trick is that we had the caller pass in the default instance of their type, and we called its New() method to construct additional instances of the same type.