I have a message and I would like to package it into an any repeated google proto type::
Is there a way to encode an repeated any message type?
Can I even use repeated tag with google.protobuf.any?
message Onesensor{
string name=1
string type=2
int32_t reading=3
}
/** Any Message **/
message RepeatedAny{
repeated google.protobuf.any sensors = 1;
}
I am looking for an example, currently using nanopb to encode.
Sure, it is just a regular message.
https://github.com/nanopb/nanopb/tree/master/tests/any_type shows how to encode a single Any message, encoding many is like encoding any array. You'll have a choice between allocating statically, allocating dynamically or using callbacks. Or you can just encode a single subfield at a time into output stream, because concatenating encoded concatenates arrays in protobuf format.
I think I found my issue, I cannot use(repeated tag on the google.protobuf.any, as I would like to append the RepeatedAny messages in the final binary):
message Onesensor{
string name=1
string type=2
int32_t reading=3
}
message RepeatedAny{
repeated google.protobuf.any sensors = 1;
}
Instead I should use something like this:
message Onesensor{
string name=1
string type=2
int32_t reading=3
}
message SensorAny{
google.protobuf.any sensor = 1;
}
message RepeatedAny{
repeated SensorAny sensors = 1;
}
I should not use the repeated tag on the google.protobuf.any, I should be using it on a message that contains the google.protobuf.any instead, so that the protobinary can contain the format (sensors1), (sensors2).....(sensorsN), one or more SensorAny messages.
Below is the sample code, if someone finds this question in the future for nanopb:
/* First encode the SensorAny message by setting the value of the first field,
The first field of this message is of type google.protobuf.any, so it should have
1. sensor.type_url
2. sensor.value
*/
void* pBufAny = calloc(1, sBufSize);
pb_ostream_t ostream_any = pb_ostream_from_buffer(pBufAny, sBufSize);
SensorAny SensorAnyProto = SensorAny_init_default;
SensorAnyProto.has_message = true;
SensorAnyProto.sensor.type_url.arg = "type.googleapis.com/SensorAny.proto";
SensorAnyProto.sensor.type_url.funcs.encode = Proto_encode_string;
ProtoEncodeBufferInfo_t BufInfo = {
.Buffer = pBuf, /* I have already filled and encoded Onesensor message previously as pBuf */
.BufferSize = ostream.bytes_written,
};
SensorAnyProto.sensor.value.funcs.encode = Proto_encode_buffer;
SensorAnyProto.sensor.value.arg = &BufInfo;
pb_encode(&ostream_any, SensorAny_fields, &SensorAnyProto);
free(pBuf);
// Now Use the above encoded Any message buffer pBufAny to set the first repeated field in RepeatedAny
RepeatedAny SensorAnyRepeated = RepeatedAny_init_default;
ProtoEncodeBufferInfo_t AnyBufInfo = {
.Buffer = pBufAny,
.BufferSize = ostream_any.bytes_written,
};
AnyRepeated.sensors.arg=&AnyBufInfo;
AnyRepeated.sensors.funcs.encode = Proto_encode_buffer;
void* pBufAnyRepeated = calloc(1, sBufSize);
pb_ostream_t ostream_repeated = pb_ostream_from_buffer(pBufAnyRepeated, sBufSize);
!pb_encode(&ostream_repeated, RepeatedAny_fields, &AnyRepeated);
free(pBufAny);
Related
I'm designing a protobuf to represent an event, where each event can hold extra fields.
There are a lot of possible extra fields (~100), but only a small portion of them will be used in each message (~3)
Each extra field will be used only once, but multiple of them can exist, therefore I would like to have a concept of an anyof message type, but unfortunately, there is no such thing in protobuf.
So to try and mock this behavior, and as mentioned in this discussion I thought I can put all my extra fields in a oneof, wrap it with a message, and use this message as repeated in my event:
message ExtraField {
oneof extra_field_value {
string extraData1 = 1;
uint64 extraData2 = 2;
....
SomeOtherMessage extraData100 = 100;
}
}
message MyEvent {
uint64 timestamp = 1;
string event_name = 2;
string some_other_data = 3;
...
repeated ExtraField extra_fields = 8;
}
Even though this solution is more explicit for my understanding, it isn't the most memory effective, and the repeated message with oneof implementation allows to add the same extra field more than once (unwanted behavior)
I can also just write all the extra fields as-is in an inner message, but most of them will be empty all the time
message ExtraFields {
string extraData1 = 1;
uint64 extraData2 = 2;
....
SomeOtherMessage extraData100 = 100;
}
message MyEvent {
uint64 timestamp = 1;
string event_name = 2;
string some_other_data = 3;
...
extraFields extra_fields = 8;
}
If I understand correctly, using empty fields in a message isn't going to make my serialized data larger, and therefore the second protobuf design is the preferred practice
Am I correct?
Is there another protobuf design for my needs?
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 2 years ago.
Improve this question
Recently started exploring nanopb - apology if my questions sound sily. I faced some issues while assigning and retrieval of strings and integers when I modified simple example of nanopb. Let me give my steps before my questions -
I defined simple.proto file
message Device{
optional string devid =1;
optional string mac = 2;
optional string cpu=3 [default = "x86"] ;
optional bool isSecured=4;
optional int32 uptime = 5 [default = 1234];
}
Also defined simple.option
Device.devid max_size:64
Device.cpu max_size:64
Then I compiled as usual : protoc -osimple.pb simple.proto
Here is my code :
3a) Using same string encode decode utility function as in How to encode a string when it is a pb_callback_t type
//string encode decode to pb
bool encode_string(pb_ostream_t* stream, const pb_field_t* field, void* const* arg)
{
const char* str = (const char*)(*arg);
if (!pb_encode_tag_for_field(stream, field))
return false;
return pb_encode_string(stream, (uint8_t*)str, strlen(str));
}
bool print_string(pb_istream_t *stream, const pb_field_t *field, void **arg)
{
uint8_t buffer[1024] = {0};
/* We could read block-by-block to avoid the large buffer... */
if (stream->bytes_left > sizeof(buffer) - 1)
return false;
if (!pb_read(stream, buffer, stream->bytes_left))
return false;
/* Print the string, in format comparable with protoc --decode.
* Format comes from the arg defined in main(). */
printf((char*)*arg, buffer);
return true;
}
3b) here is my main code snippet based on simple.c of example -
/* This is the buffer where we will store our message. */
uint8_t buffer[128];
size_t message_length;
bool status;
/* Encode our message */
{
/* Allocate space on the stack to store the message data, check out the contents of simple.pb.h
* good to always initialize your structures so that no garbage data from RAM in there. */
//Device message = Device_init_zero; //init zero for empty
Device message = Device_init_default; //init default for default
/* Create a stream that will write to our buffer. */
pb_ostream_t stream = pb_ostream_from_buffer(buffer, sizeof(buffer));
/* Fill in the data */
message.devid.arg = "device1";
message.devid.funcs.encode = &encode_string;
//strcpy(message.devid,"device1"); // no easier way like this ?
message.isSecured = true; // should be 1 if printed with %d
message.uptime=9876; // should change, why it is not working ?
/* Now we are ready to encode the message! */
// encode stream to buffer , also get the buffer length
status = pb_encode(&stream, Device_fields, &message);
message_length = stream.bytes_written;
/* Then just check for any errors.. */
if (!status)
{
printf("Encoding failed: %s\n", PB_GET_ERROR(&stream));
return 1;
}
}
/* Now we could transmit the message over network, store it in a file etc. */
/* just decode it immediately. */
{
/* Allocate space for the decoded message. */
Device message = Device_init_zero;
/* Create a stream that reads from the buffer. */
pb_istream_t stream = pb_istream_from_buffer(buffer, message_length);
message.devid.funcs.decode = &print_string;
message.devid.arg = "before decode Device - devid: %s \n"; //works here
message.cpu.funcs.decode = &print_string;
message.cpu.arg = "before decode Device -cpu: %s \n"; //where in op?
printf("before decode isSecured %d\n",message.isSecured); // doesn't work
printf("before decode uptime %d\n",message.uptime); //doesn't work
/* Now ready to decode the message. */
// decode stream buffer into message
status = pb_decode(&stream, Device_fields, &message);
/* Check for errors... */
if (!status)
{
printf("Decoding failed: %s\n", PB_GET_ERROR(&stream));
return 1;
}
/* Print the data contained in the message. */
message.devid.funcs.decode = &print_string;
message.devid.arg = "after decode Devic - devid: %s \n"; // doesn't work here
printf(" after decode isSecured %d\n",message.isSecured); // prints default 0
printf(" after decode uptime %d\n",(int)message.uptime); //prints default assigned in proto
}
The output after build and run :
$ ./simple
before decode isSecured 0
before decode uptime 0
before decode Device - devid: device1
after decode isSecured 0
after decode uptime 1234
My queries ( also added my inline comments in code) :
In original simple.c message.lucky_number=13 assignment works but here message.uptime assignment is not working , it is taking default value. Similarly assigning boolean value to message.isSecured is not working. Please tell where is my fault.
I used Device_init_default before pb_encode as some have default values and Device_init_zero before pb_decode call as it will populate after decode. Is my approach correct ?
Is there any simpler way to assign string value using strcpy and printing it in C way by printf("%s",strvar) apart from the encode_string and decode_string util ?
The string is printed only before pb_decode call but uptime default value is printed after pb_decode call. Also boolean value assignment is not working. Why ? What is my mistake ?
I saw encode string and int functions in https://github.com/nanopb/nanopb/blob/master/tests/callbacks/encode_callbacks.c
How to encode and decode float and boolean ?
Thanks in anticipation
In original simple.c message.lucky_number=13 assignment works but here message.uptime assignment is not working , it is taking default value. Similarly assigning boolean value to message.isSecured is not working. Please tell where is my fault.
If you look into the generated .pb.h file, you'll find that every optional field has a boolean has_field. You'll have to set that to true also, to signify that the field is present.
I used Device_init_default before pb_encode as some have default values and Device_init_zero before pb_decode call as it will populate after decode. Is my approach correct ?
That's fine.
Is there any simpler way to assign string value using strcpy and printing it in C way by printf("%s",strvar) apart from the encode_string and decode_string util ?
Because you have already set a max_size for your string fields, they should be generated as a char array instead of callbacks. You can try passing -v switch like: ../generator/nanopb_generator.py -v simple.pb to see more verbose messages that can point out why the option is not applying. Perhaps the file name is incorrect or the message name is incorrect.
The string is printed only before pb_decode call but uptime default value is printed after pb_decode call. Also boolean value assignment is not working. Why ? What is my mistake ?
I saw encode string and int functions in https://github.com/nanopb/nanopb/blob/master/tests/callbacks/encode_callbacks.c How to encode and decode float and boolean ?
Well, usually you wouldn't have to resort to callbacks. But if you decide you need them, you can encode booleans with pb_encode_varint() and floats with pb_encode_fixed32(). For studying callbacks, the protobuf encoding documentation and this test case can be helpful.
You may find the network_server example useful to study.
Also, the Stack Overflow format works best when you have only a single question per post. That way the questions and answers remain focused and easy to follow.
I have a protobuf schema with a bunch of repeated structures. Something like
syntax = "proto3";
package My
message TopLevel
{
string swVersion = 3;
string reportMac = 4;
string reportSsid = 6
}
message Temperature
{
required uint64 ts = 1;
required uint32 source = 3;
repeated sint32 readings = 4;
}
message MyMessage
{
required TopLevel topLevel = 1;
repeated Temperature temperature = 2;
}
I compile with pbtools and get the structures and functions for Temperature and readings. However I am having a hard time figuring out how to add "Temperature" entries dynamically.
Or am I out of luck and pbtools requires telling it ahead of time how many entries I have. One problem is data is encoded as it is generated and I do not know how many of what I will have for each report.
I attached the generated code.
pbtools requires the length before adding any items.
because there is no extend in proto3, so I combine the base message with google.protobuf.Any type message, but it's binary length is too long
.proto file
message TradeMessage {
google.protobuf.Any message = 1;
string code = 2;
}
message Connect {
int32 seq = 1;
string appid = 2;
string clientid = 3;
string ver = 4;
}
...
.java file
TradeProtocol.Connect inner = TradeProtocol.Connect.newBuilder()
.setSeq(1)
.setAppid("test")
.build();
TradeProtocol.TradeMessage packet = TradeProtocol.TradeMessage.newBuilder()
.setMessage(Any.pack(inner))
.setCode(2)
.build();
service send packet to client, client can decode all message to base TradeMessage, the problem is the inner's length is 8 bytes, while packet's length is 56 bytes. the same function implement use proto2's extend just ten more bytes, so is there any way to implement extend function in proto3 or reduce the packet's length ? thanks
One alternative is to use oneof:
message Connect {
int32 seq = 1;
string appid = 2;
string clientid = 3;
string ver = 4;
}
message TradeMessage {
string code = 1;
oneof inner {
Connect inner_connect = 2;
SomeOtherMessage inner_other = 3;
...
}
}
The encoded size will still be larger than with extend, but only by 1-2 bytes.
File my_protocol.proto:
Message MyProtocol {
required int32 protocolId = 1;
required **unknownType** protocolBody = 2;
}
I want to parse protocolBody according to protocolId.
For example,
protocolId = 10001 represents a login event, then the protocolBody should be a login message and can be parsed by login.proto.
I don't know which type should be set on the unknownType.
Protocol buffers parses the whole message in the one go.
You could have a message
Message MyProtocolGetId {
required int32 protocolId = 1;
}
Parse that then reparse with the correct message id. Alternatively the ProtocolBody could be stored as bytes and passed seperately
You can fake it, but it will be awkward. You have to use optional fields, and indicate which one if present using a flag. Make sure to double check that the field is set in your receiver code:
Message OneOfManyTypes {
required int32 kind = 1;
optional MessageType1 ty1 = 2;
optional MessageType2 ty2 = 3;
...
}