My proto file look like this
message DeviceOption {
string ApID = 1;
string Other = 2;
}
After run protoc, the DeviceOption struct generated like this:
type DeviceOption struct {
state protoimpl.MessageState
sizeCache protoimpl.SizeCache
unknownFields protoimpl.UnknownFields
ApID string `protobuf:"bytes,1,opt,name=ApID,proto3" json:"ApID,omitempty"`
Other string `protobuf:"bytes,2,opt,name=Other,proto3" json:"Other,omitempty"`
}
Now, in server I wanna parse all available fields using https://pkg.go.dev/reflect. My code is like:
v := reflect.ValueOf(pb.DeviceOption{
ApID: "random1",
Other: "random2",
})
for i := 0; i < v.NumField(); i++ {
// Do my thing
}
v.NumField() returns 5, which means it includes all the fields, including state, sizeCache, unknownFields which we don't want. We just want ApID and Other.
Is there any other way to let reflect return only data (exported) fields, and ignore metadata (unexported) fields?
The method NumFields correctly returns the number of fields either exported or unexported.
Go 1.17 and above
Use StructField.IsExported
for _, f := range reflect.VisibleFields(v.Type()) {
if f.IsExported() {
// do your thing
}
}
Up to Go 1.16
To know which ones are exported, check if field PkgPath is empty.
One could also use CanSet, but the implementation of StructField.IsExported in Go 1.17 is literally f.PkgPath == "".
The documentation on the PkgPath field states:
PkgPath is the package path that qualifies a lower case (unexported) field name. It is empty for upper case (exported) field names. See https://golang.org/ref/spec#Uniqueness_of_identifiers
typ := v.Type()
for i := 0; i < typ.NumField(); i++ {
if f := typ.Field(i); f.PkgPath == "" {
// do your thing
}
}
It's most likely not a good idea to use reflect on the struct types generated by the protobuf compiler. Their exact structure should be considered an implementation detail. Some versions of the compiler used to generate exported fields named XXX_NoUnkeyedLiteral, XXX_unrecognized and XXX_sizecache, which would have been picked up as "data" fields if you only consider the structure of the Go type. I don't think there's any specific guarantee that might not happen again.
The google.golang.org/protobuf/reflect/protoreflect package offers methods to dynamically inspect messages. Given a message m, you can do the following to range over all fields set on it:
proto.MessageReflect(m).Range(
func (field protoreflect.MessageDescriptor, value protoreflect.Value) bool {
// return `true` to continue, `false` to break
...
})
Or you can use proto.MessageReflect(m).Descriptor().Fields() to get an interface representing a list of declared fields in the message.
See also: A new Go API for Protocol Buffers
I have a "key1:value1; key2:value2" like string (string with key:value pattern concated by ;).
Now I wish to parse this string to a Go struct:
type CustomStruct struct {
KeyName1 string `name:"key1" somevalidation:"xxx"`
KeyName2 int `name:"key2" somevalidation:"yyy"`
}
In the above example, the struct tag defines the name of the key in the string and can provide some validation for its corresponding value (it can set a default value if validation fails). For instance, KeyName2 is an int value, so I wish the somevalidation can check whether the KeyName2 satisfy, let's say, greater than 30 and less equal than 100.
And in another senario, I can define another struct CustomStruct2 for string like key3:value3; key4:value4;
How can I archive this kind of requirement efficiently and elegantly?
I'll assume that you can parse the data to a map[string]interface{}.
Use the reflect package to set the fields. Here's the basic function:
// set sets fields in struct pointed to by pv to values in data.
func set(pv interface{}, data map[string]interface{}) {
// pv is assumed to be pointer to a struct
s := reflect.ValueOf(pv).Elem()
// Loop through fields
t := s.Type()
for i := 0; i < t.NumField(); i++ {
// Set field if there's a data value for the field.
f := t.Field(i)
if d, ok := data[f.Tag.Get("name")]; ok {
s.Field(i).Set(reflect.ValueOf(d))
}
}
}
This code assumes that the values in the data map are assignable to the corresponding field in the struct and that the first argument is a pointer to a struct. The code will panic if these assumptions are not true. You can protect against this by checking types and assignability using the reflect package.
playground example
To avoid too much spaghetti style code, I'd like to set certain properties of a golang protobuf object by name. So lets say there's some kind of .proto definition like
syntax = "proto3";
package foo;
message User {
uint64 uid = 1;
string name = 2;
}
and code accessing it like
o := foo.User{Name: "John Doe"}
o.Uid = 40
exists. I'd like to be able to set Uid without the dotted notation. Reflection constructs like
r := reflect.ValueOf(o)
f := reflect.Indirect(r).FieldByName("Uid")
f.SetUint(42)
seem to fail, because Uid is not addressable. I found several posts about struct fields not being addressable, but it's still unclear (to me as a "golang newbie") what has to be done in this context to make it work.
You need to pass a pointer to o to reflect.ValueOf. By simply passing the value, the fields of the struct will not be addressable.
r := reflect.ValueOf(&o)
f := reflect.Indirect(r).FieldByName("Uid")
f.SetUint(42)
If at all possible, however, I would try to implement this without reflection. Either use a map in your protocol buffer definition, or a switch in your Go code:
switch name {
case "Uid":
o.Uid = uintValue
case "Name"
o.Name = stringValue
}
The non-reflect code is less likely to fail at runtime.
Is there a good way in golang to pass all fields of some struct instance c?
I'm looking for some syntactic sugar functionality, so that instead of doing this:
method(&c.field1, &c.field2, &c.field3, &c.field4, &c.field5, ...)
I could do this:
method(FieldsPointers(c)...)
I'm rather new to golang and still learning the basics, if there is no good way to do what I want for a good reason, I'd appreciate an explanation as to why.
Besides all sql specified tools, if you want to access to pointers of a struct, you can use reflect. Be warned that the package is tricky and rob pike said it is not for everyone.
reflect.Value has methods NumField which returns the numbber of fields in the struct and Field(int) which accepts the index of a field and return the field itself.
But as you want to set a value to it, it is more complicated than just calling the two methods. Let me show you in code:
func Scan(x interface{}) {
v := reflect.ValueOf(x).Elem()
for i := 0; i < v.NumField(); i++ {
switch f := v.Field(i); f.Kind() {
case reflect.Int:
nv := 37
f.Set(reflect.ValueOf(nv))
case reflect.Bool:
nv := true
f.Set(reflect.ValueOf(nv))
}
}
}
First, you need to pass a pointer of the struct into Scan, since you are modifying data and the value must be settable. That is why we are calling .Elem(), to dereference the pointer.
Second, reflect.Value.Set must use a same type to set. You cannot set uint32 to a int64 like normal assignment.
Playground: https://play.golang.org/p/grvXAc1Px8g
I have a listener which receives protobuf messages. However it doesn't know which type of message comes in when. So I tried to unmarshal into an interface{} so I can later type cast:
var data interface{}
err := proto.Unmarshal(message, data)
if err != nil {
log.Fatal("unmarshaling error: ", err)
}
log.Printf("%v\n", data)
However this code doesn't compile:
cannot use data (type interface {}) as type proto.Message in argument to proto.Unmarshal:
interface {} does not implement proto.Message (missing ProtoMessage method)
How can I unmarshal and later type cast an "unknown" protobuf message in go?
First, two words about the OP's question, as presented by them:
proto.Unmarshal can't unmarshal into an interface{}. The method signature is obvious, you must pass a proto.Message argument, which is an interface implemented by concrete protobuffer types.
When handling a raw protobuffer []byte payload that didn't come in an Any, ideally you have at least something (a string, a number, etc...) coming together with the byte slice, that you can use to map to the concrete protobuf message.
You can then switch on that and instantiate the appropriate protobuf concrete type, and only then pass that argument to Unmarshal:
var message proto.Message
switch atLeastSomething {
case "foo":
message = &mypb.Foo{}
case "bar":
message = &mypb.Bar{}
}
_ = proto.Unmarshal(data, message)
Now, what if the byte payload is truly unknown?
As a foreword, consider that this should seldom happen in practice. The schema used to generate the protobuffer types in your language of choice represents a contract, and by accepting protobuffer payloads you are, for some definitions of it, fulfilling that contract.
Anyway, if for some reason you must deal with a completely unknown, mysterious, protobuffer payload in wire format, you can extract some information from it with the protowire package.
Be aware that the wire representation of a protobuf message is ambiguous. A big source of uncertainty is the "length-delimited" type (2) being used for strings, bytes, repeated fields and... sub-messages (reference).
You can retrieve the payload content, but you are bound to have weak semantics.
The code
With that said, this is what a parser for unknown proto messages may look like. The idea is to leverage protowire.ConsumeField to read through the original byte slice.
The data model could be like this:
type Field struct {
Tag Tag
Val Val
}
type Tag struct {
Num int32
Type protowire.Type
}
type Val struct {
Payload interface{}
Length int
}
And the parser:
func parseUnknown(b []byte) []Field {
fields := make([]Field, 0)
for len(b) > 0 {
n, t, fieldlen := protowire.ConsumeField(b)
if fieldlen < 1 {
return nil
}
field := Field{
Tag: Tag{Num: int32(n), Type: t },
}
_, _, taglen := protowire.ConsumeTag(b[:fieldlen])
if taglen < 1 {
return nil
}
var (
v interface{}
vlen int
)
switch t {
case protowire.VarintType:
v, vlen = protowire.ConsumeVarint(b[taglen:fieldlen])
case protowire.Fixed64Type:
v, vlen = protowire.ConsumeFixed64(b[taglen:fieldlen])
case protowire.BytesType:
v, vlen = protowire.ConsumeBytes(b[taglen:fieldlen])
sub := parseUnknown(v.([]byte))
if sub != nil {
v = sub
}
case protowire.StartGroupType:
v, vlen = protowire.ConsumeGroup(n, b[taglen:fieldlen])
sub := parseUnknown(v.([]byte))
if sub != nil {
v = sub
}
case protowire.Fixed32Type:
v, vlen = protowire.ConsumeFixed32(b[taglen:fieldlen])
}
if vlen < 1 {
return nil
}
field.Val = Val{Payload: v, Length: vlen - taglen}
// fmt.Printf("%#v\n", field)
fields = append(fields, field)
b = b[fieldlen:]
}
return fields
}
Sample input and output
Given a proto schema like:
message Foo {
string a = 1;
string b = 2;
Bar bar = 3;
}
message Bar {
string c = 1;
}
initialized in Go as:
&test.Foo{A: "A", B: "B", Bar: &test.Bar{C: "C"}}
And by adding a fmt.Printf("%#v\n", field) statement at the end of the loop in the above code, it will output the following:
main.Field{Tag:main.Tag{Num:1, Type:2}, Val:main.Val{Payload:[]uint8{0x41}, Length:1}}
main.Field{Tag:main.Tag{Num:2, Type:2}, Val:main.Val{Payload:[]uint8{0x42}, Length:1}}
main.Field{Tag:main.Tag{Num:1, Type:2}, Val:main.Val{Payload:[]uint8{0x43}, Length:1}}
main.Field{Tag:main.Tag{Num:3, Type:2}, Val:main.Val{Payload:[]main.Field{main.Field{Tag:main.Tag{Num:1, Type:2}, Val:main.Val{Payload:[]uint8{0x43}, Length:1}}}, Length:3}}
About sub-messages
As you can see from the above the idea to deal with a protowire.BytesType that may or may not be a message field is to attempt to parse it, recursively. If it succeeds, we keep the resulting msg and store it in the field value, if it fails, we store the bytes as-is, which then may be a proto string or bytes. BTW, if I'm reading correctly, this seems what Marc Gravell does in the Protogen code.
About repeated fields
The code above doesn't deal with repeated fields explicitly, but after the parsing is done, repeated fields will have the same value for Field.Tag.Num. From that, packing the fields into a slice/array should be trivial.
About maps
The code above also doesn't deal with proto maps. I suspect that maps are semantically equivalent to a repeated k/v pair, e.g.:
message Pair {
string key = 1; // or whatever key type
string val = 2; // or whatever val type
}
If my assumption is correct, then maps can be parsed with the given code as sub-messages.
About oneofs
I haven't yet tested this, but I expect that information about the union type are completely lost. The byte payload will contain only the value that was actually set.
But what about Any?
The Any proto type doesn't fit in the picture. Contrary to what it may look like, Any is not analogous to, say, map[string]interface{} for JSON objects. And the reason is simple: Any is a proto message with a very well defined structure, namely (in Go):
type Any struct {
// unexported fields
TypeUrl string // struct tags omitted
Value []byte // struct tags omitted
}
So it is more similar to the implementation of a Go interface{} in that it holds some actual data and that data's type information.
It can hold itself arbitrary proto payloads (with their type information!) but it can not be used to decode unknown messages, because Any has exactly those two fields, type url and a byte payload.
To wrap up, this answer doesn't provide a full-blown production-grade solution, but it shows how to decode arbitrary payloads while preserving as much original semantics as possible. Hopefully it will point you in the right direction.
As you've seen, and the commenters have pointed out, you can't use proto.Unmarshal to interface{} because, the method expects a type Message which implements an interface MessageV1.
Protobuf messages are typed and correspond to method invocations ("comes in") and the implementation cannot take generic types of protobuf but specific protobufs:
func (s *server) M(ctx context.Context, _ *pb.Foo) (*pb.Bar, error)
The solution is to envelope your generic types as Any within a specific type perhaps Envelope:
message Envelope {
google.protobuf.Any content = 1;
...
}
The content is then transmitted as a []byte (see Golang anypb.Any) and the implementation (anypb) includes methods to pack|unpack these.
The 'trick' with Any is that messages include a [TypeURL] that uniquely identifies the message so that the receiver knows how to e.g. Unmarshal it.