I am trying to use the protobuf-c library to send messages in between sockets in C.
Steps to reproduce
Create a .proto file with the messages:
syntax = "proto3";
message MyMsg {}
Compile it with protoc c_out=. rpc.proto I get my rpc.pb-c.h/c files ready to use.
Include the header, try to pack an empty message.
int main () {
MyMsg msg = MY_MSG__INIT;
size_t buflen = my_msg__get_packed_size(&msg);
printf("buflen: %zu\n", buflen);
}
prints 0.
Further details
Even if I ignore this and I allocate a larger buffer for packing the msg, the my_msg__pack(buf), function will not write any bytes into the buffer.
If I define a non-empty message, I do get some bytes into the buffer. Still, it seems to me the generated code only packs the body of the message, not the header, indicating the type of the message.
Question
Is there some other API I need to use in order to pack the whole message?
Since your message is defined as an empty one, e.g. no fields, the length of the serialized message is, of course, 0. This is true, even if you define a message with some fields, but NOT set any field of the message.
message MyMsg {
int32 i = 1; // if you don't call MyMsg::set_i(int), the length of the serialized message is still 0.
}
not the header, indicating the type of the message.
Protobuf does NOT encode meta data, e.g. message type, into the serialized value. Instead, the protobuf definition file is the meta data. That's why you must have a protobuf file, if you want to parse the serialized data.
Also if you have 2 messages with exactly the same definition, you can even parse the serialized data of one message to build another message.
message A {
}
message B {
// have the same fields definition with A
}
A a;
serialized_data = serialize(a);
B b;
parse(b, serialized_data);
Related
I have a message and I would like to package it into an any repeated google proto type::
Is there a way to encode an repeated any message type?
Can I even use repeated tag with google.protobuf.any?
message Onesensor{
string name=1
string type=2
int32_t reading=3
}
/** Any Message **/
message RepeatedAny{
repeated google.protobuf.any sensors = 1;
}
I am looking for an example, currently using nanopb to encode.
Sure, it is just a regular message.
https://github.com/nanopb/nanopb/tree/master/tests/any_type shows how to encode a single Any message, encoding many is like encoding any array. You'll have a choice between allocating statically, allocating dynamically or using callbacks. Or you can just encode a single subfield at a time into output stream, because concatenating encoded concatenates arrays in protobuf format.
I think I found my issue, I cannot use(repeated tag on the google.protobuf.any, as I would like to append the RepeatedAny messages in the final binary):
message Onesensor{
string name=1
string type=2
int32_t reading=3
}
message RepeatedAny{
repeated google.protobuf.any sensors = 1;
}
Instead I should use something like this:
message Onesensor{
string name=1
string type=2
int32_t reading=3
}
message SensorAny{
google.protobuf.any sensor = 1;
}
message RepeatedAny{
repeated SensorAny sensors = 1;
}
I should not use the repeated tag on the google.protobuf.any, I should be using it on a message that contains the google.protobuf.any instead, so that the protobinary can contain the format (sensors1), (sensors2).....(sensorsN), one or more SensorAny messages.
Below is the sample code, if someone finds this question in the future for nanopb:
/* First encode the SensorAny message by setting the value of the first field,
The first field of this message is of type google.protobuf.any, so it should have
1. sensor.type_url
2. sensor.value
*/
void* pBufAny = calloc(1, sBufSize);
pb_ostream_t ostream_any = pb_ostream_from_buffer(pBufAny, sBufSize);
SensorAny SensorAnyProto = SensorAny_init_default;
SensorAnyProto.has_message = true;
SensorAnyProto.sensor.type_url.arg = "type.googleapis.com/SensorAny.proto";
SensorAnyProto.sensor.type_url.funcs.encode = Proto_encode_string;
ProtoEncodeBufferInfo_t BufInfo = {
.Buffer = pBuf, /* I have already filled and encoded Onesensor message previously as pBuf */
.BufferSize = ostream.bytes_written,
};
SensorAnyProto.sensor.value.funcs.encode = Proto_encode_buffer;
SensorAnyProto.sensor.value.arg = &BufInfo;
pb_encode(&ostream_any, SensorAny_fields, &SensorAnyProto);
free(pBuf);
// Now Use the above encoded Any message buffer pBufAny to set the first repeated field in RepeatedAny
RepeatedAny SensorAnyRepeated = RepeatedAny_init_default;
ProtoEncodeBufferInfo_t AnyBufInfo = {
.Buffer = pBufAny,
.BufferSize = ostream_any.bytes_written,
};
AnyRepeated.sensors.arg=&AnyBufInfo;
AnyRepeated.sensors.funcs.encode = Proto_encode_buffer;
void* pBufAnyRepeated = calloc(1, sBufSize);
pb_ostream_t ostream_repeated = pb_ostream_from_buffer(pBufAnyRepeated, sBufSize);
!pb_encode(&ostream_repeated, RepeatedAny_fields, &AnyRepeated);
free(pBufAny);
I have a protobuf message defined something like this
message Foo
{
oneof test //oneof field
{
int32 a = 1;
MM b = 2;
}
}
message MM
{
string str =1;
}
how i reflect of oneof filed in protobuf
For the most part, you handle a message with oneof the same way you would a message without:
message Foo
{
int32 a = 1;
MM b = 2;
}
Oneof is largely transparent to reflection, and doesn't affect the wire format. Its effect is that the generated setter code automatically clears other members of a oneof whenever one is set.
Now, if you care about oneofs for some reason, there's Descriptor::oneof_decl that allows you to enumerate them, Descriptor::FindOneofByName, and FieldDescriptor::containing_oneof (if you are working your way from the field up). With OneofDescriptor in hand, you can find its name and enumerate its fields, and that's pretty much it.
I’m developing a system for our application to get data from an external device. As soon as I send it a specific message, it sends back short messages to us 10x/second (so about 1 message per 100 milliseconds). I’m using Boost for this communication.
The process is rather simple: I create the socket, send the message, giving it a handler for the message receive:
// Header file:
...
std::unique_ptr<boost::asio::io_service> _theIOService;
std::unique_ptr<boost::asio::ip::tcp::socket> _theSocket;
int size_of_the_data = 100;
std::vector<char> _raw_buffer = std::vector<char>(size_of_the_data);
boost::asio::mutable_buffers_1 _data_buffer = boost::asio::buffer(_raw_buffer, size_of_the_data);
...
// Implementation file:
...
void DeviceDataListener::initiateTransfer() {
// create and connect the socket up here
...
// send the message
boost::system::error_code error;
boost::asio::write(*_theSocket,
boost::asio::buffer(beginMessage),
boost::asio::transfer_all(), error);
// start the receive
auto handler = boost::bind(&SCUDataListener::dataHandler, this, _1, _2);
_theSocket->async_receive( _data_buffer, handler );
std::thread run_thread([&]{ _theIOService->run(); });
...
}
void DeviceDataListener::dataHandler (
const boost::system::error_code& error, // Result of operation.
std::size_t bytes_transferred // Number of bytes received.
) {
int foo = bytes_transferred;
// this line crashes application
char* pData = static_cast<char*>(_data_buffer.data());
}
It works, my handler gets called immediately, as it should. The problem is, I can’t get the data out of _data_buffer. This:
auto it = _data_buffer.begin();
causes a crash, even though _data_buffer is valid. This:
const char* pData = static_cast<char*>(_data_buffer.data());
won’t compile. The error is “Method 'data' could not be resolved”. The mutable_buffer_1 API says data() is a completely valid method that returns the beginning of the memory range.
Inspecting via a debugger, I can see that there is no error and I can see data as a member of _data_buffer and the memory address it contains does contain the data we’re expecting. The thing is, I can’t get to it via code. Does anyone know how to get to the data in a Boost mutable_buffers_1?
We’re using Eclipse CDT, C++11 and gcc running on Linux.
“Method 'data' could not be resolved”.
this error may be true, but it depends on what version of Boost you use. data() is member of mutable_buffer since >= 1.66 version. Because mutable_buffer is the base class for mutable_buffers_1 your code should compile if you use at least 1.66 version of Boost.
If your version is < 1.66 you should use
char* p1 = boost::asio::buffer_cast<char*>(_data_buffer);
to get the pointer to data in the buffer.
_data_buffer.begin();
you should not use begin() method, it returns pointer to mutable_buffer_1 itself. This method is used by internal functions of asio-boost library, for instance to copy sequence of buffers, then begin() points the particular buffer to be copied.
What I am looking for is a function that returns the message of a repeated field
I know there is Reflection::AddMessage which has the return type that I want but I do not want to add a message, just return an object of that message.
Here is an example of what I am trying to do let's say I have in the .proto file a message:
message Bar{
uint32 t x = 1;
uint64 t y = 2;
}
message Foo{
repeated Bar myMessage = 1;
}
I am using reflection to iterate through the Foo message and I want to be able to do something like this:
Message* Msg = createMessage(refl->FooMsg, FieldDesc)
I know there is also GetRepeatedMessage but that requires index.
First of all when the protobuf compiler generates the code for compiling you get an accessor function in the interface. The are functions mutable_nameOf_message() which returns the entire repeated field which is a std::vector in c++, or mutable_nameOf_message( index ) which gives you the specified element.
Now if you do not want to use Bar then you d'not need too.
message ArrayOfBar
{
repeated Bar arrayOfBar = 0;
message Bar{
uint32 t x = 1;
uint64 t y = 2;
}
}
If thats what you have hade in mind you could also be do something like this.
std::vector<Bar> arrayOfBars;
But that idea needs refinement because of the internal specifics of the Protobuf. Some unwanted behavior might occur with something like that.
File my_protocol.proto:
Message MyProtocol {
required int32 protocolId = 1;
required **unknownType** protocolBody = 2;
}
I want to parse protocolBody according to protocolId.
For example,
protocolId = 10001 represents a login event, then the protocolBody should be a login message and can be parsed by login.proto.
I don't know which type should be set on the unknownType.
Protocol buffers parses the whole message in the one go.
You could have a message
Message MyProtocolGetId {
required int32 protocolId = 1;
}
Parse that then reparse with the correct message id. Alternatively the ProtocolBody could be stored as bytes and passed seperately
You can fake it, but it will be awkward. You have to use optional fields, and indicate which one if present using a flag. Make sure to double check that the field is set in your receiver code:
Message OneOfManyTypes {
required int32 kind = 1;
optional MessageType1 ty1 = 2;
optional MessageType2 ty2 = 3;
...
}