enncoding and decoding issue in nanopb [closed] - protocol-buffers

Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 2 years ago.
Improve this question
Recently started exploring nanopb - apology if my questions sound sily. I faced some issues while assigning and retrieval of strings and integers when I modified simple example of nanopb. Let me give my steps before my questions -
I defined simple.proto file
message Device{
optional string devid =1;
optional string mac = 2;
optional string cpu=3 [default = "x86"] ;
optional bool isSecured=4;
optional int32 uptime = 5 [default = 1234];
}
Also defined simple.option
Device.devid max_size:64
Device.cpu max_size:64
Then I compiled as usual : protoc -osimple.pb simple.proto
Here is my code :
3a) Using same string encode decode utility function as in How to encode a string when it is a pb_callback_t type
//string encode decode to pb
bool encode_string(pb_ostream_t* stream, const pb_field_t* field, void* const* arg)
{
const char* str = (const char*)(*arg);
if (!pb_encode_tag_for_field(stream, field))
return false;
return pb_encode_string(stream, (uint8_t*)str, strlen(str));
}
bool print_string(pb_istream_t *stream, const pb_field_t *field, void **arg)
{
uint8_t buffer[1024] = {0};
/* We could read block-by-block to avoid the large buffer... */
if (stream->bytes_left > sizeof(buffer) - 1)
return false;
if (!pb_read(stream, buffer, stream->bytes_left))
return false;
/* Print the string, in format comparable with protoc --decode.
* Format comes from the arg defined in main(). */
printf((char*)*arg, buffer);
return true;
}
3b) here is my main code snippet based on simple.c of example -
/* This is the buffer where we will store our message. */
uint8_t buffer[128];
size_t message_length;
bool status;
/* Encode our message */
{
/* Allocate space on the stack to store the message data, check out the contents of simple.pb.h
* good to always initialize your structures so that no garbage data from RAM in there. */
//Device message = Device_init_zero; //init zero for empty
Device message = Device_init_default; //init default for default
/* Create a stream that will write to our buffer. */
pb_ostream_t stream = pb_ostream_from_buffer(buffer, sizeof(buffer));
/* Fill in the data */
message.devid.arg = "device1";
message.devid.funcs.encode = &encode_string;
//strcpy(message.devid,"device1"); // no easier way like this ?
message.isSecured = true; // should be 1 if printed with %d
message.uptime=9876; // should change, why it is not working ?
/* Now we are ready to encode the message! */
// encode stream to buffer , also get the buffer length
status = pb_encode(&stream, Device_fields, &message);
message_length = stream.bytes_written;
/* Then just check for any errors.. */
if (!status)
{
printf("Encoding failed: %s\n", PB_GET_ERROR(&stream));
return 1;
}
}
/* Now we could transmit the message over network, store it in a file etc. */
/* just decode it immediately. */
{
/* Allocate space for the decoded message. */
Device message = Device_init_zero;
/* Create a stream that reads from the buffer. */
pb_istream_t stream = pb_istream_from_buffer(buffer, message_length);
message.devid.funcs.decode = &print_string;
message.devid.arg = "before decode Device - devid: %s \n"; //works here
message.cpu.funcs.decode = &print_string;
message.cpu.arg = "before decode Device -cpu: %s \n"; //where in op?
printf("before decode isSecured %d\n",message.isSecured); // doesn't work
printf("before decode uptime %d\n",message.uptime); //doesn't work
/* Now ready to decode the message. */
// decode stream buffer into message
status = pb_decode(&stream, Device_fields, &message);
/* Check for errors... */
if (!status)
{
printf("Decoding failed: %s\n", PB_GET_ERROR(&stream));
return 1;
}
/* Print the data contained in the message. */
message.devid.funcs.decode = &print_string;
message.devid.arg = "after decode Devic - devid: %s \n"; // doesn't work here
printf(" after decode isSecured %d\n",message.isSecured); // prints default 0
printf(" after decode uptime %d\n",(int)message.uptime); //prints default assigned in proto
}
The output after build and run :
$ ./simple
before decode isSecured 0
before decode uptime 0
before decode Device - devid: device1
after decode isSecured 0
after decode uptime 1234
My queries ( also added my inline comments in code) :
In original simple.c message.lucky_number=13 assignment works but here message.uptime assignment is not working , it is taking default value. Similarly assigning boolean value to message.isSecured is not working. Please tell where is my fault.
I used Device_init_default before pb_encode as some have default values and Device_init_zero before pb_decode call as it will populate after decode. Is my approach correct ?
Is there any simpler way to assign string value using strcpy and printing it in C way by printf("%s",strvar) apart from the encode_string and decode_string util ?
The string is printed only before pb_decode call but uptime default value is printed after pb_decode call. Also boolean value assignment is not working. Why ? What is my mistake ?
I saw encode string and int functions in https://github.com/nanopb/nanopb/blob/master/tests/callbacks/encode_callbacks.c
How to encode and decode float and boolean ?
Thanks in anticipation

In original simple.c message.lucky_number=13 assignment works but here message.uptime assignment is not working , it is taking default value. Similarly assigning boolean value to message.isSecured is not working. Please tell where is my fault.
If you look into the generated .pb.h file, you'll find that every optional field has a boolean has_field. You'll have to set that to true also, to signify that the field is present.
I used Device_init_default before pb_encode as some have default values and Device_init_zero before pb_decode call as it will populate after decode. Is my approach correct ?
That's fine.
Is there any simpler way to assign string value using strcpy and printing it in C way by printf("%s",strvar) apart from the encode_string and decode_string util ?
Because you have already set a max_size for your string fields, they should be generated as a char array instead of callbacks. You can try passing -v switch like: ../generator/nanopb_generator.py -v simple.pb to see more verbose messages that can point out why the option is not applying. Perhaps the file name is incorrect or the message name is incorrect.
The string is printed only before pb_decode call but uptime default value is printed after pb_decode call. Also boolean value assignment is not working. Why ? What is my mistake ?
I saw encode string and int functions in https://github.com/nanopb/nanopb/blob/master/tests/callbacks/encode_callbacks.c How to encode and decode float and boolean ?
Well, usually you wouldn't have to resort to callbacks. But if you decide you need them, you can encode booleans with pb_encode_varint() and floats with pb_encode_fixed32(). For studying callbacks, the protobuf encoding documentation and this test case can be helpful.
You may find the network_server example useful to study.
Also, the Stack Overflow format works best when you have only a single question per post. That way the questions and answers remain focused and easy to follow.

Related

How to encode a repeated google.protobuf.any?

I have a message and I would like to package it into an any repeated google proto type::
Is there a way to encode an repeated any message type?
Can I even use repeated tag with google.protobuf.any?
message Onesensor{
string name=1
string type=2
int32_t reading=3
}
/** Any Message **/
message RepeatedAny{
repeated google.protobuf.any sensors = 1;
}
I am looking for an example, currently using nanopb to encode.
Sure, it is just a regular message.
https://github.com/nanopb/nanopb/tree/master/tests/any_type shows how to encode a single Any message, encoding many is like encoding any array. You'll have a choice between allocating statically, allocating dynamically or using callbacks. Or you can just encode a single subfield at a time into output stream, because concatenating encoded concatenates arrays in protobuf format.
I think I found my issue, I cannot use(repeated tag on the google.protobuf.any, as I would like to append the RepeatedAny messages in the final binary):
message Onesensor{
string name=1
string type=2
int32_t reading=3
}
message RepeatedAny{
repeated google.protobuf.any sensors = 1;
}
Instead I should use something like this:
message Onesensor{
string name=1
string type=2
int32_t reading=3
}
message SensorAny{
google.protobuf.any sensor = 1;
}
message RepeatedAny{
repeated SensorAny sensors = 1;
}
I should not use the repeated tag on the google.protobuf.any, I should be using it on a message that contains the google.protobuf.any instead, so that the protobinary can contain the format (sensors1), (sensors2).....(sensorsN), one or more SensorAny messages.
Below is the sample code, if someone finds this question in the future for nanopb:
/* First encode the SensorAny message by setting the value of the first field,
The first field of this message is of type google.protobuf.any, so it should have
1. sensor.type_url
2. sensor.value
*/
void* pBufAny = calloc(1, sBufSize);
pb_ostream_t ostream_any = pb_ostream_from_buffer(pBufAny, sBufSize);
SensorAny SensorAnyProto = SensorAny_init_default;
SensorAnyProto.has_message = true;
SensorAnyProto.sensor.type_url.arg = "type.googleapis.com/SensorAny.proto";
SensorAnyProto.sensor.type_url.funcs.encode = Proto_encode_string;
ProtoEncodeBufferInfo_t BufInfo = {
.Buffer = pBuf, /* I have already filled and encoded Onesensor message previously as pBuf */
.BufferSize = ostream.bytes_written,
};
SensorAnyProto.sensor.value.funcs.encode = Proto_encode_buffer;
SensorAnyProto.sensor.value.arg = &BufInfo;
pb_encode(&ostream_any, SensorAny_fields, &SensorAnyProto);
free(pBuf);
// Now Use the above encoded Any message buffer pBufAny to set the first repeated field in RepeatedAny
RepeatedAny SensorAnyRepeated = RepeatedAny_init_default;
ProtoEncodeBufferInfo_t AnyBufInfo = {
.Buffer = pBufAny,
.BufferSize = ostream_any.bytes_written,
};
AnyRepeated.sensors.arg=&AnyBufInfo;
AnyRepeated.sensors.funcs.encode = Proto_encode_buffer;
void* pBufAnyRepeated = calloc(1, sBufSize);
pb_ostream_t ostream_repeated = pb_ostream_from_buffer(pBufAnyRepeated, sBufSize);
!pb_encode(&ostream_repeated, RepeatedAny_fields, &AnyRepeated);
free(pBufAny);

Decoding of char array data

I have a mesh from which I need to read the vertex positions of but I can just get a buffer with that data, which I seemingly can get as an utf-8 char array.
Currently I'm getting the data from the buffer into the array I metioned and wirte it into a char* but i can't get the decoding correctly or so it seems.
The following code reads the dara from the buffer:
char* GetDataFromIBuffer(Windows::Storage::Streams::IBuffer^ container)
{
unsigned int bufferLength = container->Length;
auto dataReader = Windows::Storage::Streams::DataReader::FromBuffer(container);
Platform::Array<unsigned char>^ managedBytes =
ref new Platform::Array<unsigned char>(bufferLength);
dataReader->ReadBytes(managedBytes);
char * bytes = new char[bufferLength];
for (unsigned int i = 0; i < bufferLength; i++)
{
if (managedBytes[i] == '\0')
{
bytes[i] = '0';
}
else
{
bytes[i] = managedBytes[i];
}
}
}
I can see the data in debug mode but i need a method to make it readable and write it into a file, where i can copy the mesh data and draw the mesh in a seperate program.
The following image shows the array data which can be seen in the array:
debug mode
Be careful not to mix up text encoding and data types.
char is a type often used for buffers because it has the size of a byte, but that doesn't mean that the data contained in the buffer is text.
Your debug view seem to confirm that the data inside your buffer is not text, because when interpreted as text, it gives weird characters such as 'ΓΏ', '^', etc...
UTF-8 is a way to encode unicode text, so it has nothing to do with binary data.
You need to find a way to cast your buffer data info the internal type of the data, it should be documented where you got that data (maybe it's just an array of floats ?)

How to detect snprintf failure?

I am using snprintf to format string using user-defined format (also given as string). The code looks like this:
void DataPoint::valueReceived( QVariant value ) {
// Get the formating QVariant, which is only considered valid if it's string
QVariant format = this->property("format");
if( format.isValid() && format.type()==QMetaType::QString && !format.isNull() ) {
// Convert QString to std string
const std::string formatStr = format.toString().toStdString();
LOGMTRTTIINFO(pointName<<"="<<value.toString().toUtf8().constData()<<"=>"<<formatStr<<"["<<formatStr.length()<<'\n');
// The attempt to catch exceptions caused by invalid formating string
try {
if( value.type() == QMetaType::QString ) {
// Treat value as string (values are allways ASCII)
const std::string array = value.toString().toStdString();
const char* data = (char*)array.c_str();
// Assume no more than 10 characters are added during formating.
char* result = (char*)calloc(array.length()+10, sizeof(char));
snprintf(result, array.length()+10, formatStr.c_str(), data);
value = result;
}
// If not string, then it's a number.
else {
double data = value.toDouble();
char* result = (char*)calloc(30, sizeof(char));
// Even 15 characters is already longer than largest number you can make any sense of
snprintf(result, 30, formatStr.c_str(), data);
LOGMTRTTIINFO(pointName<<"="<<data<<"=>"<<formatStr<<"["<<formatStr.length()<<"]=>"<<result<<'\n');
value = result;
}
} catch(...) {
LOGMTRTTIERR("Format error in "<<pointName<<'\n');
}
}
ui->value->setText(value.toString());
}
As you can see I assumed there will be some exception. But there's not, invalid formatting string results in gibberish. This is what I get if I try to format double using %s:
So is there a way to detect that invalid formatting option was selected, such as formatting number as string or vice-versa? And what if totally invalid formatting string is given?
You ask if it's possible to detect format/argument mismatch at run-time, right? Then the short and only answer is no.
To expand on that "no" it's because Variable-argument functions (functions using the ellipsis ...) have no kind of type-safety. The compiler will convert some types of arguments to others (e.g. char or short will be converted to int, float will be converted to double), and if you use a literal string for the format some compilers will be able to parse the string and check the arguments you pass.
However since you pass a variable string, that can change at run-time, the compiler have no possibility for any kind of compile-time checking, and the function must trust that the format string passed is using the correct formatting for the arguments passed. If it's not then you have undefined behavior.
It should be noted that snprintf might not actually fail when being passed mismatching format specifier and argument value.
For example if using the %d format to print an int value, but then passing a double value, the snprintf would happily extract sizeof(int) bytes from the double value, and interpret it as an int value. The value printed will be quite unexpected, but there won't be a "failure" as such. Only undefined behavior (as mentioned above).
Thus it's not really possible to detect such errors or problems at all. At least not through the code. This is something that needs proper testing and code-review to catch.
What happens when snprintf fails? When snprintf fails, POSIX requires that errno is set:
If an output error was encountered, these functions shall return a negative value and set errno to indicate the error.
Also you can find some relevant information regarding how to handle snprintf failures Here.

Extract trailing int from string containing other characters

I have a problem in regards of extracting signed int from string in c++.
Assuming that i have a string of images1234, how can i extract the 1234 from the string without knowing the position of the last non numeric character in C++.
FYI, i have try stringstream as well as lexical_cast as suggested by others through the post but stringstream returns 0 while lexical_cast stopped working.
int main()
{
string virtuallive("Images1234");
//stringstream output(virtuallive.c_str());
//int i = stoi(virtuallive);
//stringstream output(virtuallive);
int i;
i = boost::lexical_cast<int>(virtuallive.c_str());
//output >> i;
cout << i << endl;
return 0;
}
How can i extract the 1234 from the string without knowing the position of the last non numeric character in C++?
You can't. But the position is not hard to find:
auto last_non_numeric = input.find_last_not_of("1234567890");
char* endp = &input[0];
if (last_non_numeric != std::string::npos)
endp += last_non_numeric + 1;
if (*endp) { /* FAILURE, no number on the end */ }
auto i = strtol(endp, &endp, 10);
if (*endp) {/* weird FAILURE, maybe the number was really HUGE and couldn't convert */}
Another possibility would be to put the string into a stringstream, then read the number from the stream (after imbuing the stream with a locale that classifies everything except digits as white space).
// First the desired facet:
struct digits_only: std::ctype<char> {
digits_only(): std::ctype<char>(get_table()) {}
static std::ctype_base::mask const* get_table() {
// everything is white-space:
static std::vector<std::ctype_base::mask>
rc(std::ctype<char>::table_size,std::ctype_base::space);
// except digits, which are digits
std::fill(&rc['0'], &rc['9'], std::ctype_base::digit);
// and '.', which we'll call punctuation:
rc['.'] = std::ctype_base::punct;
return &rc[0];
}
};
Then the code to read the data:
std::istringstream virtuallive("Images1234");
virtuallive.imbue(locale(locale(), new digits_only);
int number;
// Since we classify the letters as white space, the stream will ignore them.
// We can just read the number as if nothing else were there:
virtuallive >> number;
This technique is useful primarily when the stream contains a substantial amount of data, and you want all the data in that stream to be interpreted in the same way (e.g., only read numbers, regardless of what else it might contain).

C++ Ensuring that user input value is int only [duplicate]

This question already has answers here:
How to test whether stringstream operator>> has parsed a bad type and skip it
(5 answers)
Closed 8 years ago.
I am a little new to C++ and would really appreciate any input or suggestions! So with our intro course projects I have been looking for a way to ensure that when the prog. is asking for int values it correctly responds! That is it states its invalid in cases of both a double as well as string being entered! So if cin >> intVariable ... intVariable will not accept cin entry of "abdf" or 20.01.
So to achieve this I wrote the following function...It works but I am looking for your thoughts on how this process can be further improved!
void getIntegerOnly(int& intVariable, string coutStatement)
{
bool isInteger; // Check if value entered by user is int form or not
string tmpValue; // Variable to store temp value enetered by user
cout << coutStatement; // Output the msg for the cin statement
do
{
cin >> tmpValue; // Ask user to input their value
try // Use try to catch any exception caused by what user enetered
{
/* Ex. if user enters 20.01 then the if statement converts the
string to a form of int anf float to compare. that is int value
will be 20 and float will be 20.01. And if values do not match
then user input is not integer else it is. Keep looping untill
user enters a proper int value. Exception is 20 = 20.00 */
if (stoi(tmpValue) != stof(tmpValue))
{
isInteger = false; // Set to false!
clear_response(); // Clear response to state invalid
}
else
{
isInteger = true; //Set to true!
clear_cin(); // Clear cin to ignore all text and space in cin!
}
}
catch (...) // If the exception is trigured!
{
isInteger = false; // Set to false!
clear_response(); // Clear response to state invalid
}
} while (!isInteger); //Request user to input untill int clause met
//Store the int value to the variable passed by reference
intVariable = stoi(tmpValue);
}
This is simply an example of getting users age and age is greater than zero when running a Win32 console based application! Thank you for the feedback :)
One way would be something like the following:
std::string str;
std::cin >> str;
bool are_digits = std::all_of(
str.begin(), str.end(),
[](char c) { return isdigit(static_cast<unsigned char>(c)); }
);
return are_digits ? std::stoi(str) : throw std::invalid_argument{"Invalid input"};
and catch the exceptions on the calling side (stoi can also throw std::out_of_range).
You can leverage the second parameter of stoi().
string tmpValue;
size_t readChars;
stoi(tmpValue, &readChars);
if(readChars == tmpValue.length())
{
// input was integer
}
EDIT: this will not work for strings containing "." (for example integers passed in scientific notation).
This is not my work, but the answer to this question is what you want. Pass the string to it as a reference. It will return true is your string is an integer.
How do I check if a C++ string is an int?

Resources