I have a field that is of type fixed64 in a .proto file.
I want to read it as an int64 field:
score := int64(pb_obj.Score)
When I try to compile the line agove I get the error message cannot convert pb_obj.Score (type *uint64) to type int64. I tried converting the a uint64 as well, and got an almost identical message.
pb_obj.Score's type seems to be *uint64 (pointer to uint64), not uint64. You just need to access to the value the pointer is referencing:
score := int64(*pb_obj.Score)
(See the * prefix as the difference)
Based on the compile error you're working with a uint64 pointer and not a uint64 value. You may get what you want by referencing the value directly using the * operator. I've never worked with protobuf, so I could be off but that should get you moving. Here's a nice reference that may help golang pointers
Related
I use golang "github.com/olivere/elastic/v7" package
xx_time is uint32 in my code
xx_time is long in es mapping
After update by script
ctx._source.xx_time = Math.max(ctx._source.xx_time, params.xx_time)
I get the same doc again get error
json: cannot unmarshal number 1.604394299E9 into Go struct field RiskInnerMachine.xx_time of type uint32
Get 1.604394299E9 not 1604394299 ,get 0.0 not 0.
I want to ask some question:
Q1: Does Math.max change my data type ? I even can't find any official document talk about which type the function return and how to use the function.
Q2: Why the data can save in es ,if Q1 is true ,even the data is not a long data ?
Q3: if Q1 is false, why I get the float data and other fields aren't affected ?
My solution is
ctx._source.xx_time = (long)Math.max(ctx._source.xx_time, params.xx_time)
I have to change the type manually. Make me pain.
Q1. The Painless Math functions are described here, they are actually Java methods. Even though Java provides several overloaded Math.max() methods for long, float and double, Painless only provides the one for double, probably because all other types (i.e. long and float) can be upcast to double.
So yes, if you provide long or float values, Math.max will always return double. You can cast that value back to long as you do, which is not too much of a hurdle, I guess.
Q2. If you have long in your mapping, it's ok, because ES will coerce the double value into a long (enabled by default). And since the value actually represents and contains a long value, there's no information lost (i.e. 1604394299.0 coerced into 1604394299 doesn't remove any meaningful information).
Q3. -
I'm struggling to figure out a reason for this behavior, or maybe this is suppose to happen and I just wasn't aware.
For background, I'm using proto3, and am doing this in Go1.15, and I do know that packed is the default in proto3, and I'm relatively new to protobufs.
I defined the following message in a proto file:
message Response {
repeated uint32 points = 1 [packed=true];
}
Which will generate the following code using protoc-gen-go v1.25.0.
type Response struct {
state protoimpl.MessageState
sizeCache protoimpl.SizeCache
unknownFields protoimpl.UnknownFields
Points []uint32 `protobuf:"varint,3,rep,packed,name=points,json=points,proto3" json:"points,omitempty"`
}
I go to use the new struct, and it doesn't behave like I would normally expect a struct to behave. Here's some things I wrote, along with what was printed out.
newResponse := pb.Response{Points: []uint32{2,4,6,8}}
fmt.Println(newResponse)
//{{{} [] [] <nil>} 0 [] [2 4 6 8] --> I expect this
refToNewResponse := &newResponse
fmt.Println(refToNewResponse)
// points:2 points:4 points:6 points:8 --> not what I expected
Now you might be thinking, it's just formatting big deal.
But I expect a list... not numbers that each individually have a label. I've seen and used other protobufs... and when I see the response that they return, it doesn't look like this, it's one label to a list like:
points: [2 4 6 8]
I do need to use the reference version of this because I eventually want to expand and use a list of Responses which the generated code will spit out a slice of pointer Responses, but I can't understand why it's separating and labeling each element in the slice.
I'm hoping someone can point out something I'm doing or not doing that is causing this... thank you in advance.
This is indeed just formatting. Nothing has changed in the underlying data structure. You requested a repeated uint32 Points and it's literally printing them that way.
The marshaler in the protobuf implementation can really output whatever it likes, there is no reference version of the human-readable representation of a protobuf.
If you really must have a custom format for the .String() output, you can try a different proto library such as gogoprotobuf, or try various extensions. But ultimately, it's just human-readable output.
Note:
this has nothing to do with packed=true (which is indeed the default).
if you're confused about printing the pointer vs the basic type, it's because the String() method has a pointer receiver. See this question
The struct in the .pb.go file generated by .proto file has three additional fields and some other things.like this:
When converting this struct to json, if one field is empty, the field will not appear in json. Now I know it can be done using jsonpb.Marshaler.
m := jsonpb.Marshaler{EmitDefaults: true}
Now, I coverting struct to map[string]interface{}, put it in
InfluxDB. I have to convert struct to map[string]interface{}.The function NewPoint needs. like this:
I use structs.Map(value) function in go ,The transformed map has three additional fields, and running the program causes errors，like this:
{"error":"unable to parse 'txt,severity=1 CurrentValue=\"1002\",MetricAlias=\"CPU\",XXX_sizecache=0i,XXX_unrecognized= 1552551101': missing field value"}
When I remove these three fields, the program runs OK.These three fields are automatically generated, and I have a lot of structs.
What should I do?Thank you!
Protobuf generator adds some additional fields with names starting from XXX that are intended for optimizations. You can't change this behavior of protoc-gen-go.
The problem is in the way you convert struct to map[sting]interface{}. It's hard to figure out from which package exactly structs.Map comes from. Seems like it goes from here: https://github.com/fatih/structs/blob/master/structs.go#L89 - this code uses reflect to iterate through all fields of the structure and push them to map[sting]interface{}. You just need to write your own slightly modified version of FillMap routine that will omit XXX fields.
I'm writing a small application that receives message in BSON format from network(its not MongoDB) and have to save fields in files on local machine. I'm using gopkg.in/mgo.v2/bson for message unmarshaling and it works fine.
Almost everything works except one. There "userdefined" binary field in message and I have to save it to separate file. I tried to use:
var pwr = msg["pwr"].([]byte)
but got an "error panic: interface conversion: interface is bson.Binary, not []uint8".
Can some one point me an example how to convert bson.Binary to []byte, so I can save it to file.
This does what you want:
pwr := bson.Binary(msg["pwr"].(bson.Binary)).Data
But assumes msg["pwr"] can't be anything other than a bson.Binary... if that's not an invariant you should do the type assertion first (handle the possible type mismatch case when it happens) and then cast to get the Data field.
In a Google protocol buffer, I'm going to use a field to store values that will be integers in [0,255]. From http://code.google.com/apis/protocolbuffers/docs/proto.html#scalar, it looks like the uint32 will be the appropriate value type to use. Despite the field being able to hold up to 32-bit integers, those extra bits will not be wasted in my case due to the variable length encoding. (Correct me if I'm wrong up to here.)
My question is: how should I indicate that the reader of a serialized message can assume that the largest value in that field will be 255? Just a comment in the protocol buffer specification? Is there any other way?
In .proto there is no such specification; you must simply document it (and presumably cast it appropriately at the consuming code).
Aside: if you happen to be using the C# protobuf-net implementation, then you can do this by working outside a .proto definition (protobuf-net allows code-first):
[ProtoMember(3)] // <=== field number
public byte SomeValue {get;set;}
This is then obviously constrained to 0-255, but is encoded on the wire as you expect (like a uint32). It also does a checked conversion when deserializing, to sanity-check the values.
In .proto, the above is closest to:
optional uint32 someValue = 3;