For testing, I often see go code read byte slices, which are parsed into structs using yaml, for example here:
https://github.com/kubernetes/kubernetes/blob/master/pkg/util/strategicpatch/patch_test.go#L74m
I just got bitten by not exporting my field names, resulting in an empty list which I iterated over in my test cases, thus assuming that all tests were passing (in hindsight, that should have been a red flag :)). There are other errors which are silently ignored by yaml unmarshaling, such as a key being misspelled and not matching a struct field exactly.
Is there a way to ensure that all the data in the byte slice was actually parsed into the struct returned by yaml.Unmarshal? If not, how do others handle this situation?
go-yaml/yaml
For anyone searching for a solution to this problem, the yaml.v2 library has an UnmarshalStrict method that returns an error if there are keys in the yaml document that have no corresponding fields in the go struct.
import yaml "gopkg.in/yaml.v2"
err := yaml.UnmarshalStrict(data, destinationStruct)
BurntSushi/toml
It's not part of the question, but I'd just like to document how to achieve something similar in toml:
You can find if there were any keys in the toml file that could not be decoded by using the metadata returned by the toml.decode function.
import "github.com/BurntSushi/toml"
metadata, err := toml.Decode(data, destinationStruct)
undecodedKeys := metadata.Undecoded()
Note that metadata.Undecoded() also returns keys that have not been decoded because of a Primitive value. You can read more about it here.
Json
The default go json library does not support this currently, but there is a proposal ready to be merged. It seems that it will be a part of go 1.10.
Related
Suppose we are debugging some Go code, and somewhere in an external dependency we encounter this line:
return json.Marshal(foo)
We want to set a breakpoint and use IntelliJ's "Evaluate Expression" to inspect the JSON being produced. However, this doesn't work:
If we evaluate the expression json.Marshal(foo), we only get to see the byte array.
Evaluating string(json.Marshal(foo)) doesn't work because json.Marshal returns two values, the byte array and an error.
There is no way in Go to access one of the return values directly.
So how can I use "Evaluate Expression" to achieve my goal of just printing the produced JSON string when I'm not able to change the underlying source code?
you can print the returned bytes as a string
bytes, err := json.Marshal(foo)
// check error here
fmt.Println(string(bytes))
update based on comments
You can't change the byte slice in the debugger to a string without changing the source code.
I'm struggling to figure out a reason for this behavior, or maybe this is suppose to happen and I just wasn't aware.
For background, I'm using proto3, and am doing this in Go1.15, and I do know that packed is the default in proto3, and I'm relatively new to protobufs.
I defined the following message in a proto file:
message Response {
repeated uint32 points = 1 [packed=true];
}
Which will generate the following code using protoc-gen-go v1.25.0.
type Response struct {
state protoimpl.MessageState
sizeCache protoimpl.SizeCache
unknownFields protoimpl.UnknownFields
Points []uint32 `protobuf:"varint,3,rep,packed,name=points,json=points,proto3" json:"points,omitempty"`
}
I go to use the new struct, and it doesn't behave like I would normally expect a struct to behave. Here's some things I wrote, along with what was printed out.
newResponse := pb.Response{Points: []uint32{2,4,6,8}}
fmt.Println(newResponse)
//{{{} [] [] <nil>} 0 [] [2 4 6 8] --> I expect this
refToNewResponse := &newResponse
fmt.Println(refToNewResponse)
// points:2 points:4 points:6 points:8 --> not what I expected
Now you might be thinking, it's just formatting big deal.
But I expect a list... not numbers that each individually have a label. I've seen and used other protobufs... and when I see the response that they return, it doesn't look like this, it's one label to a list like:
points: [2 4 6 8]
I do need to use the reference version of this because I eventually want to expand and use a list of Responses which the generated code will spit out a slice of pointer Responses, but I can't understand why it's separating and labeling each element in the slice.
I'm hoping someone can point out something I'm doing or not doing that is causing this... thank you in advance.
This is indeed just formatting. Nothing has changed in the underlying data structure. You requested a repeated uint32 Points and it's literally printing them that way.
The marshaler in the protobuf implementation can really output whatever it likes, there is no reference version of the human-readable representation of a protobuf.
If you really must have a custom format for the .String() output, you can try a different proto library such as gogoprotobuf, or try various extensions. But ultimately, it's just human-readable output.
Note:
this has nothing to do with packed=true (which is indeed the default).
if you're confused about printing the pointer vs the basic type, it's because the String() method has a pointer receiver. See this question
The struct in the .pb.go file generated by .proto file has three additional fields and some other things.like this:
When converting this struct to json, if one field is empty, the field will not appear in json. Now I know it can be done using jsonpb.Marshaler.
m := jsonpb.Marshaler{EmitDefaults: true}
Now, I coverting struct to map[string]interface{}, put it in
InfluxDB. I have to convert struct to map[string]interface{}.The function NewPoint needs. like this:
I use structs.Map(value) function in go ,The transformed map has three additional fields, and running the program causes errors，like this:
{"error":"unable to parse 'txt,severity=1 CurrentValue=\"1002\",MetricAlias=\"CPU\",XXX_sizecache=0i,XXX_unrecognized= 1552551101': missing field value"}
When I remove these three fields, the program runs OK.These three fields are automatically generated, and I have a lot of structs.
What should I do?Thank you!
Protobuf generator adds some additional fields with names starting from XXX that are intended for optimizations. You can't change this behavior of protoc-gen-go.
The problem is in the way you convert struct to map[sting]interface{}. It's hard to figure out from which package exactly structs.Map comes from. Seems like it goes from here: https://github.com/fatih/structs/blob/master/structs.go#L89 - this code uses reflect to iterate through all fields of the structure and push them to map[sting]interface{}. You just need to write your own slightly modified version of FillMap routine that will omit XXX fields.
I've recently encountered all sorts of wrappers in Google's protobuf package. I'm struggling to imagine the use case. Can anyone shed the light: what problem were these intended to solve?
Here's one of the documentation links: https://developers.google.com/protocol-buffers/docs/reference/csharp/class/google/protobuf/well-known-types/string-value (it says nothing about what can this be used for).
One thing that will be different in behavior between this, and simple string type is that this field will be written less efficiently (a couple extra bytes, plus a redundant memory allocation). For other wrappers, the story is even worse, since the repeated variants of those fields will be written inefficiently (official Google's Protobuf serializer doesn't support packed encoding for non-numeric types).
Neither seems to be desirable. So, what's this all about?
There's a few reasons, mostly to do with where these are used - see struct.proto.
StringValue can be null, string often can't be in a language interfacing with protobufs. e.g. in Go strings are always set; the "zero value" for a string is "", the empty string, so it's impossible to distinguish between "this value is intentionally set to empty string" and "there was no value present". StringValue can be null and so solves this problem. It's especially important when they're used in a StructValue, which may represent arbitrary JSON: to do so it needs to distinguish between a JSON key which was set to empty string (StringValue with an empty string) or a JSON key which wasn't set at all (null StringValue).
Also if you look at struct.proto, you'll see that these aren't fully fledged message types in the proto - they're all generated from message Value, which has a oneof kind { number_value, string_value, bool_value... etc. By using a oneof struct.proto can represent a variety of different values in one field. Again this makes sense considering what struct.proto is designed to handle - arbitrary JSON - you don't know what type of value a given JSON key has ahead of time.
In addition to George's answer, you can't use a Protobuf primitive as the parameter or return value of a gRPC procedure.
As you could probably tell from the below code I am working on a project which creates csv reports from data in mongoDB. After getting the data I need in, I need to structure the data into something more sensible then how it exists in the db, which is fairly horrendous (not my doing) and near impossible to print the way I need it. The structure that makes the most sense to me is a slice (for each document of data) of maps of the name of the data to a structure holding the data for that name. Then I would simply have to loop through the document and stuff values into the structs where they belong.
My implementation of this is
type mongo_essential_data_t struct {
caution string
citation string
caution_note string
}
mongo_rows_struct := make([]map[string]mongo_essential_data_t, len(mongodata_rows))
//setting the values goes like this
mongo_rows_struct[i][data_name].caution_note = fmt.Sprint(k)
//"i" being the document, "k" being the data I want to store
This doesn't work however. When doing "go run" it returns ./answerstest.go:140: cannot assign to mongo_rows_struct[i][data_name].caution_note. I am new to Go and not sure why I am not allowed to do this. I'm sure this is an invalid way to reference that particular data location, if it is even possible to reference it in Go. What is another way to accomplish this setting line? If it is too much work to accomplish this the way I want, I am willing to use a different type of data structure and am open to suggestions.
This is a known issue of Golang, known as issue 3117. You can use a temporary variable to get around it:
var tmp = mongo_rows_struct[i][data_name]
tmp.caution_note = fmt.Sprint(k)
mongo_rows_struct[i][data_name] = tmp
as per my understanding, when you write:
mongo_rows_struct[i][data_name]
compiler will generate code, which will return copy of mongo_essential_data_t struct(since struct in go is value type, not reference type), and
mongo_rows_struct[i][data_name].caution_note = fmt.Sprint(k)
will write new value to that copy. And after that copy will be discarded. Obviously, its not what you expect. So Go compiler generate error to prevent this misunderstanding.
In order to solve this problem you can:
1. Change definition of your data type to
[]map[string]*mongo_essential_data_t
2. Explicitly create copy of your struct, make changes in that copy and write it back to the map
data := mongo_rows_struct[i][data_name]
data.caution_note = fmt.Sprint(k)
mongo_rows_struct[i][data_name] = data
Of course, first solution is preferable because you will avoid unnecessary copying of data