In a Google protocol buffer, I'm going to use a field to store values that will be integers in [0,255]. From http://code.google.com/apis/protocolbuffers/docs/proto.html#scalar, it looks like the uint32 will be the appropriate value type to use. Despite the field being able to hold up to 32-bit integers, those extra bits will not be wasted in my case due to the variable length encoding. (Correct me if I'm wrong up to here.)
My question is: how should I indicate that the reader of a serialized message can assume that the largest value in that field will be 255? Just a comment in the protocol buffer specification? Is there any other way?
In .proto there is no such specification; you must simply document it (and presumably cast it appropriately at the consuming code).
Aside: if you happen to be using the C# protobuf-net implementation, then you can do this by working outside a .proto definition (protobuf-net allows code-first):
[ProtoMember(3)] // <=== field number
public byte SomeValue {get;set;}
This is then obviously constrained to 0-255, but is encoded on the wire as you expect (like a uint32). It also does a checked conversion when deserializing, to sanity-check the values.
In .proto, the above is closest to:
optional uint32 someValue = 3;
Related
I use golang "github.com/olivere/elastic/v7" package
xx_time is uint32 in my code
xx_time is long in es mapping
After update by script
ctx._source.xx_time = Math.max(ctx._source.xx_time, params.xx_time)
I get the same doc again get error
json: cannot unmarshal number 1.604394299E9 into Go struct field RiskInnerMachine.xx_time of type uint32
Get 1.604394299E9 not 1604394299 ,get 0.0 not 0.
I want to ask some question:
Q1: Does Math.max change my data type ? I even can't find any official document talk about which type the function return and how to use the function.
Q2: Why the data can save in es ,if Q1 is true ,even the data is not a long data ?
Q3: if Q1 is false, why I get the float data and other fields aren't affected ?
My solution is
ctx._source.xx_time = (long)Math.max(ctx._source.xx_time, params.xx_time)
I have to change the type manually. Make me pain.
Q1. The Painless Math functions are described here, they are actually Java methods. Even though Java provides several overloaded Math.max() methods for long, float and double, Painless only provides the one for double, probably because all other types (i.e. long and float) can be upcast to double.
So yes, if you provide long or float values, Math.max will always return double. You can cast that value back to long as you do, which is not too much of a hurdle, I guess.
Q2. If you have long in your mapping, it's ok, because ES will coerce the double value into a long (enabled by default). And since the value actually represents and contains a long value, there's no information lost (i.e. 1604394299.0 coerced into 1604394299 doesn't remove any meaningful information).
Q3. -
I can't find a way to layout structure to a byte array programming by GO.
I'm programming extension for GoBGP service. I want to set community for the route via GoBGP API. More about BGP and communities.
My situation:
I'm trying to send the 32bit number. First, 16bits are interpreted as AS number, other 16bits- communities. API has an interface for uint32 to set all this data. I want to have a simple structure like:
type routeCommunity struct {
AS uint16
Community uint16
}
I'm struggling to pass this structure data to a uint32 variable. I would like to layout it to a byte array and then I could just make uint32 from it.
In C++ I would have a pointer of structure pointing to a char array and continued to work with that structure and in the end, I would send char array.
How to make it happen in Golang??
You can pass your structure to a uint32 variable using "unsafe" package:
func (rc *routeCommunity) bits() uint32 {
return *(*uint32)(unsafe.Pointer(rc))
}
https://play.golang.org/p/AQeeoYUn7o0
UPD:
Keep in mind the bits position depends on an order of the structure fields.
Low two bytes of the variable contain the first field value and high two bytes contain the second one.
I use charles and got a protobuf http message from other iOS applications. Now I want to genereate the same http packet but the output is not the same.
My protobuf file:
message TTCreateConversationBody
{
repeated uint32 imUid1 = 2;
}
I'm using objective-c:
TTCreateConversationBody *body = [TTCreateConversationBody new];
GPBUInt32Array *arr = [[GPBUInt32Array alloc] initWithCapacity:2];
[arr addValue:123123];
[arr addValue:9999999];
body.imUid1Array = arr;
and my output, charles decode it as a length-delimited string:
it's raw data and mine:
8A-26-10-08-01-10-AE-F7-81-80-9F-03-10-D4-E4-82-F0-D2-01
8A-26-10-08-01-12-0C-F9-F6-C3-9D-FA-02-AE-F7-81-80-9F-03
What's the correct protobuf file format?
They're actually both valid... ish.
This comes down to "packed" fields; without "packed", your two integers are encoded as
[header, varint][value][header, varint][value]
[10][AE-F7-81-80-9F-03][10][D4-E4-82-F0-D2-01]
where-as with "packed", it becomes
[header, string][length][value][value]
[12][0C][F9-F6-C3-9D-FA-02][AE-F7-81-80-9F-03]
note: the actual values look very different in the two runs... I'm assuming that is accidental.
To quote from the specification:
Protocol buffer parsers must be able to parse repeated fields that were compiled as packed as if they were not packed, and vice versa. This permits adding [packed=true] to existing fields in a forward- and backward-compatible way.
So: serializers should write the layout that is defined by whether your data is "packed" or not, but decoders must be able to handle it either way. Some libraries, when encountering data that should be "packed": determine which layout will actually be shorter, and make the final decision based on that. In reality, this can be approximated to "use packed encoding whenever there's at least two items".
I've recently encountered all sorts of wrappers in Google's protobuf package. I'm struggling to imagine the use case. Can anyone shed the light: what problem were these intended to solve?
Here's one of the documentation links: https://developers.google.com/protocol-buffers/docs/reference/csharp/class/google/protobuf/well-known-types/string-value (it says nothing about what can this be used for).
One thing that will be different in behavior between this, and simple string type is that this field will be written less efficiently (a couple extra bytes, plus a redundant memory allocation). For other wrappers, the story is even worse, since the repeated variants of those fields will be written inefficiently (official Google's Protobuf serializer doesn't support packed encoding for non-numeric types).
Neither seems to be desirable. So, what's this all about?
There's a few reasons, mostly to do with where these are used - see struct.proto.
StringValue can be null, string often can't be in a language interfacing with protobufs. e.g. in Go strings are always set; the "zero value" for a string is "", the empty string, so it's impossible to distinguish between "this value is intentionally set to empty string" and "there was no value present". StringValue can be null and so solves this problem. It's especially important when they're used in a StructValue, which may represent arbitrary JSON: to do so it needs to distinguish between a JSON key which was set to empty string (StringValue with an empty string) or a JSON key which wasn't set at all (null StringValue).
Also if you look at struct.proto, you'll see that these aren't fully fledged message types in the proto - they're all generated from message Value, which has a oneof kind { number_value, string_value, bool_value... etc. By using a oneof struct.proto can represent a variety of different values in one field. Again this makes sense considering what struct.proto is designed to handle - arbitrary JSON - you don't know what type of value a given JSON key has ahead of time.
In addition to George's answer, you can't use a Protobuf primitive as the parameter or return value of a gRPC procedure.
I want to get the name of the field with which it was declared from GENERIC representation. I have a BIT_FIELD_REF tree and it's DECL_NAME is zero. For example,
struct {
int a;
unsigned b:1;
} s;
...
if (s.b)
...
For s.b I'll get a BIT_FIELD_REF and there's no obvious way to get the “b”, which is the original name of the field. How to do it?
Try call debug_c_tree (tree_var) or call debug_tree (tree_var) from GDB, and see if that knows the name. If it does, reverse engineer the pretty-printer.
What exactly did I do: investigating stuff in tree-dump.c I ended up understanding that names of the bit fields where they're known were coming from struct's DIEs and were hard to track.
Then I decided to get the name from the BIT_FIELD_REF argument 0 (reference to structure) type, which is RECORD_TYPE and it stores all the fields' sizes and offsets.
Problem was to understand that BIT_FIELD_REF doesn't reference the bits itself: it is used like BIT_FIELD_REF & INTEGER_CST, where constant acts like mask. After understanding this, I quickly computed the offsets and got the name from the type.