Given the following proto message
message MyMsg {
string my_field = 1;
string your_field=2;
}
...the protoreflect package can be used to get a descriptor for each field
protoMessage := myMsg.ProtoReflect()
messageDescriptor := protoMessage.Descriptor() // protobuf type information
fieldDescriptors := messageDescriptor.Fields() // list of field declarations
Getting a field descriptor for a specific field is trivial
fieldDescriptor := fieldDescriptors.ByTextName("my_field") // describes a field
Can this be achieved without hard-coding the field name "my_field"? I guess it would be nice to use the generated code to refer to the field I'm interested in. Something like (not working code)
fieldDescriptor := fieldDescriptors.ByTextName(pb.MyMsg.MyField) // describes a field
This way, if the field name changes, it will be caught at compile time, or even from static analysis by an IDE.
The FieldDescriptors type has three methods for getting a field descriptor by name:
ByName(s Name)
ByJSONName(s string)
ByTextName(s string)
ByJSONName and ByTextName both require hard-coded field names (as strings), and by ByName accepts a Name which is type'd to a string. The upshot is, I don't see anything in the protoreflect package that points to a solution.
Context
Field masks are the recommended way to support partial resource updates. It's trivial to iterate over the masks provided in a field mask
protoMessage := myMsg.ProtoReflect()
messageDescriptor := protoMessage.Descriptor() // protobuf type information
fieldDescriptors := messageDescriptor.Fields() // list of field declarations
// iterate over the field paths in the field mask
for _, p := range mask.GetPaths() {
// find the field descriptor for the field path
fieldDescriptor := fieldDescriptors.ByTextName(p)
if fieldDescriptor == nil {
// field descriptor cannot be found for the field path
return
}
// great, the field path points to a field, let's use it
switch p {
case "my_field":
// the client wants to update MyMsg.my_field
case "your_field":
// the client wants to update MyMsg.your_field
}
}
The problem is, in order to actually update the correct field in MyMsg, it's necessary to hard-code the field name in the switch statement.
The interface has ByNumber too
I assume (I've not used the method) that you could give this 1 from your example.
And Get, of course so that you can enumerate all of the FieldDescriptors.
I think it's not unreasonable to want to reflect (!) the generated (struct) types and enumerate across them but it feels like turtles-all-the-way-down.
It may help solicit other answers if you could present the problem that you're unable to solve without the desired method?
Related
I am working with go, specifically QT bindings. However, I do not understand the use of leading underscores in the struct below. I am aware of the use of underscores in general but not this specific example.
type CustomLabel struct {
core.QObject
_ func() `constructor:"init"`
_ string `property:"text"`
}
Does it relate to the struct tags?
Those are called blank-fields because the blank identifier is used as the field name.
They cannot be referred to (just like any variable that has the blank identifier as its name) but they take part in the struct's memory layout. Usually and practically they are used as padding, to align subsequent fields to byte-positions (or memory-positions) that match layout of the data coming from (or going to) another system. The gain is that so these struct values (or rather their memory space) can be dumped or read simply and efficiently in one step.
#mkopriva's answer details what the specific use case from the question is for.
A word of warning: these blank fields as "type-annotations" should be used sparingly, as they add unnecessary overhead to all (!) values of such struct. These fields cannot be referred to, but they still require memory. If you add a blank field whose size is 8 bytes (e.g. int64), if you create a million elements, those 8 bytes will count a million times. As such, this is a "flawed" use of blank fields: the intention is to add meta info to the type itself (not to its instances), yet the cost is that all elements will require increased memory.
You might say then to use a type whose size is 0, such as struct{}. It's better, as if used in the right position (e.g. being the first field, for reasoning see Struct has different size if the field order is different and also Why position of `[0]byte` in the struct matters?), they won't change the struct's size. Still, code that use reflection to iterate over the struct's fields will still have to loop over these too, so it makes such code less efficient (typically all marshaling / unmarshaling process). Also, since now we can't use an arbitrary type, we lose the advantage of carrying a type information.
This last statement (about when using struct{} we lose the carried type information) can be circumvented. struct{} is not the only type with 0 size, all arrays with 0 length also have zero size (regardless of the actual element type). So we can retain the type information by using a 0-sized array of the type we'd like to incorporate, such as:
type CustomLabel struct {
_ [0]func() `constructor:"init"`
_ [0]string `property:"text"`
}
Now this CustomLabel type looks much better performance-wise as the type in question: its size is still 0. And it is still possible to access the array's element type using Type.Elem() like in this example:
type CustomLabel struct {
_ [0]func() `constructor:"init"`
_ [0]string `property:"text"`
}
func main() {
f := reflect.ValueOf(CustomLabel{}).Type().Field(0)
fmt.Println(f.Tag)
fmt.Println(f.Type)
fmt.Println(f.Type.Elem())
}
Output (try it on the Go Playground):
constructor:"init"
[0]func()
func()
For an overview of struct tags, read related question: What are the use(s) for tags in Go?
You can think of it as meta info of the type, it's not accessible through an instance of that type but can be accessed using reflect or go/ast. This gives the interested package/program some directives as to what to do with that type. For example based on those tags it could generate code using go:generate.
Considering that one of the tags says constructor:"init" and the field's type is func() it's highly probable that this is used with go:generate to generate an constructor function or initializer method named init for the type CustomLabel.
Here's an example of using reflect to get the "meta" info (although as I've already mentioned, the specific qt example is probably meant to be handled by go:generate).
type CustomLabel struct {
_ func() `constructor:"init"`
_ string `property:"text"`
}
fmt.Println(reflect.ValueOf(CustomLabel{}).Type().Field(0).Tag)
// constructor:"init"
fmt.Println(reflect.ValueOf(CustomLabel{}).Type().Field(0).Type)
// func()
https://play.golang.org/p/47yWG4U0uit
Is it (practically) possible to change the type name of a protobuf message type (or enum) without breaking communications?
Obviously the using code would need to be adpated to re-compile. The question is if old clients that use the same structure, but the old names, would continue to work?
Example, base on the real file:
test.proto:
syntax = "proto3";
package test;
// ...
message TestMsgA {
message TestMsgB { // should be called TestMsgZZZ going forward
// ...
enum TestMsgBEnum { // should be called TestMsgZZZEnum going forward
// ...
}
TestMsgBEnum foo = 1;
// ...
}
repeated TestMsgB bar = 1;
// ...
}
Does the on-the-wire format of the protobuf payload change in any way if type or enum names are changed?
If you're talking about the binary format, then no: names don't matter and will not impact your ability to load data; For enums, only the integer value is stored in the payload. For fields, only the field-number is stored.
Obviously if you swap two names, confusion could happen, but: it should load as long as the structure matches.
If you're talking about the JSON format, then it may matter.
I am new to golang and I have one issue which I think community can help me to solve it.
I have one data structure like below
type ParentIDInfo struct {
PCOrderID string `json:"PCorderId",omitempty"`
TableVarieties TableVarietyDC `json:"tableVariety",omitempty"`
ProduceID string `json:"PRID",omitempty"`
}
type PCDCOrderAsset struct {
PcID string `json:"PCID",omitempty"`
DcID string `json:"DCID",omitempty"`
RequiredDate string `json:"requiredDate",omitempty"`
Qty uint64 `json:"QTY",omitempty"`
OrderID string `json:"ORDERID",omitempty"`
Status string `json:"STATUS",omitempty"`
Produce string `json:"Produce",omitempty"`
Variety string `json:"VARIETY",omitempty"`
Transports []TransportaionPCDC `json:"Transportaion",omitempty"`
ParentInfo []ParentIDInfo `json:"ParentInfo",omitempty"`
So I have issue to access the PCOrderID which inside the []ParentIDInfo . I have tried below however I getting error as "pcdcorder.ParentInfo.PCOrderID undefined (type []ParentIDInfo has no field or method PCOrderID)"
keyfarmercas = append(keyfarmercas, pcdcorder.ParentInfo.PCOrderID)
Any help will be very good
Thanks in advance
PCDCOrderAsset.ParentInfo is not a struct, it does not have a PCOrderID field. It's a slice (of element type ParentIDInfo), so its elements do, e.g. pcdcorder.ParentInfo[0].PCOrderID.
Whether this is what you want we can't tell. pcdcorder.ParentInfo[0].PCOrderID gives you the PCOrderID field of the first element of the slice. Based on your question this may or may not be what you want. You may want to append all IDs (one from each element). Also note that if the slice is empty (its length is 0), then pcdcorder.ParentInfo[0] would result in a runtime panic. You could avoid that by first checking its length and only index it if its not empty.
In case you'd want to add ids of all elements, you could use a for loop to do that, e.g.:
for i := range pcdorder.ParentInfo {
keyfarmercas = append(keyfarmercas, pcdcorder.ParentInfo[i].PCOrderID)
}
I am very new to Go and am trying to get my head around all the different types and how to use them. I have an interface with the following (which was originally in a json file):
[map[item:electricity transform:{fuelType}] map[transform:{fuelType} item:gas]]
and I have the following struct
type urlTransform struct {
item string
transform string
}
I have no idea how to get the interface data into the struct; I'm sure this is really stupid, but I have been trying all day. Any help would be greatly appreciated.
Decode the JSON directly to types you want instead of decoding to an interface{}.
Declare types that match the structure of your JSON data. Use structs for JSON objects and slices for JSON arrays:
type transform struct {
// not enough information in question to fill this in.
}
type urlTransform struct {
Item string
Transform transform
}
var transforms []urlTransform
The field names must be exported (start with uppercase letter).
Unmarshal the JSON to the declared value:
err := json.Unmarshal(data, &transforms)
or
err := json.NewDecoder(reader).Decode(&transforms)
From your response : [map[item:electricity transform:{fuelType}] map[transform:{fuelType} item:gas]].
As you can see here this is a an array that has map in it.
One way to get the value from this is :
values := yourResponse[0].(map[string]interface{}). // convert first index to map that has interface value.
transform := urlTransform{}
transform.Item = values["item"].(string) // convert the item value to string
transform.Transform = values["transform"].(string)
//and so on...
as you can see from the code above I'm getting the the value using map. And convert the value to the appropriate type in this case is string.
You can convert it to appropriate type like int or bool or other type. but this approach is painful as you need to get the value one bye one and assign it your your field struct.
Whats the correct way in go to distinguish between when a value in a struct was never set, or is just empty, for example, given the following:
type Organisation struct {
Category string
Code string
Name string
}
I need to know (for example) if the category was never set, or was saved as blank by the user, should I be doing this:
type Organisation struct {
Category *string
Code *string
Name *string
}
I also need to ensure I correctly persist either null or an empty string to the database
I'm still learning GO so it is entirely possible my question needs more info.
The zero value for a string is an empty string, and you can't distinguish between the two.
If you are using the database/sql package, and need to distinguish between NULL and empty strings, consider using the sql.NullString type. It is a simple struct that keeps track of the NULL state:
type NullString struct {
String string
Valid bool // Valid is true if String is not NULL
}
You can scan into this type and use it as a query parameter, and the package will handle the NULL state for you.
Google's protocol buffers (https://code.google.com/p/goprotobuf/) use pointers to describe optional fields.
The generated objects provide GetFoo methods which take the pain away from testing for nil (a.GetFoo() returns an empty string if a.Foo is nil, otherwise it returns *a.Foo).
It introduces a nuisance when you want to write literal structs (in tests, for example), because &"something" is not valid syntax to generate a pointer to a string, so you need a helper function (see, for example, the source code of the protocol buffer library for proto.String).
// String is a helper routine that allocates a new string value
// to store v and returns a pointer to it.
func String(v string) *string {
return &v
}
Overall, using pointers to represent optional fields is not without drawbacks, but it's certainly a viable design choice.
The standard database/sql package provides a NullString struct (members are just String string and Valid bool). To take care of some of the repetitive work of persistence, you could look at an object-relational manager like gorp.
I looked into whether there was some way to distinguish two kinds of empty string just out of curiosity, and couldn't find one. With []bytes, []byte{} == []byte(nil) currently returns false, but I'm not sure if the spec guarantees that to always remain true. In any case, it seems like the most practical thing to do is to go with the flow and use NullString.