I'm using Protobuf for Golang.
Protobuf generates message types where type pointer implements proto.Message().
e.g.
func (*SomeMessage) Message() {}
The protobuf lib have methods like Marshal(proto.Message)
Now to my actual issue.
message := SomeMessage {}
SendMessage(&message)
func SendMessage(message interface{}) {
switch msg := message.(type) {
case proto.Message:
//send across the wire or whatever
default:
//non proto message, panic or whatever
}
}
The above works fine.
However, If I don't pass the message as a pointer, then the code in SendMessage will not match, as the interface is only implemented on the SomeMessage pointer, not on the value.
What I would like to do is:
message := SomeMessage {}
SendMessage(message) //pass by value
//there are more stuff going on in my real code, but just trying to show the relevant parts
func SendMessage(message interface{}) {
//match both pointer and value as proto.Message
//and then turn the value into a pointer so that
//other funcs or protobuf can consume it
message = MagicallyTurnBoxedValueIntoBoxedStruct(message)
switch msg := message.(type) {
case proto.Message:
//send across the wire or whatever
default:
//non proto message, panic or whatever
}
}
preferably I'd like to be able to pass both as pointer and as value.
The reason why I want to pass by value, is that this can act as a poor mans isolation when passing messages across goroutines/threads etc.
(in lack of immutability)
All of this could probably be avoided if the protobuf generator generated allowed values to be treated as proto.Message() too.
Or if there was some nicer way to do immutable messages.
It's not super important,if its possible, cool, if its not, meh :-)
[EDIT]
If I have the reflect.Type of the message and the reflect.Type of the pointer type of the message.
Is it somehow possible to create an instance of the pointer type pointing to the value using "reflect" ?
Normally, you can't take the address of a value which means you can't simply convert the interface{} to a pointer to satisfy Protobuf's requirement.
That said, you can dynamically create a new pointer then copy the value in to that then pass the newly allocated pointer to protobuf.
Here's an example on Play
The value -> pointer conversion is:
func mkPointer(i interface{}) interface{} {
val := reflect.ValueOf(i)
if val.Kind() == reflect.Ptr {
return i
}
if val.CanAddr() {
return val.Addr().Interface()
}
nv := reflect.New(reflect.TypeOf(i))
nv.Elem().Set(val)
return nv.Interface()
}
We first see if it's a pointer, if so, just return the value.
Then we check to see if it's addressable and return that.
Lastly, we make a new instance of the type and copy the contents to that and return it.
Since this this copies the data, it may not be practical for your purposes. It will all depend on size of message and expected rate of calling with a value (as that will generate more garbage).
Related
hi a have this func for get type of value, but i try this and never can get reflect.struct:
type Test struct {
Code int
Name string
}
func main(){
test := getTest()
data, err := getBytes(slice...)
sanitizedFile := bytes.Split(data, []byte("\r\n"))
err = Unmarshal(sanitizedFile[0], &test)
}
func getTest() interface{} {
return Test{}
}
With this code i don't can get the reflecet.struct from v params in Unmarshall func
func Unmarshal(data []byte, v interface{}) error {
rv := reflect.ValueOf(v)
if rv.Kind() == reflect.Ptr {
rvElem := rv.Elem()
switch rvElem.Kind() {
case reflect.Struct:
// implement me
}
}
return ErrInvalid
}
I would like to know if I can somehow find out if an interface is of type struct or access the values of that struct.
I think the real problem here is illustrated by this quote:
I would like to know if I can somehow find out if an interface is of type struct or access the values of that struct.
An interface value isn't "of type struct". Never! An interface value can contain a value whose type is some structure, but it is not a value of that type. It just contains one. This is similar to the way that a box1 you get from Amazon can contain a corkscrew, but the box is not a corkscrew, ever.
Given a non-nil value of type interface I for some interface type I, you know that you have a value that implements the methods of I. Since {} is the empty set of methods, all types implement it, so given a (still non-nil) value of type interface{}, you have a value that implements no methods. That's not at all useful by itself: it means you can invoke no methods, which means you can't do anything method-like.
But just because you can't do anything method-y doesn't mean you can't do anything at all. Any interface value, regardless of the interface type, can have a type-assertion used on it:
iv := somethingThatReturnsAnInterface()
cv := iv.(struct S) // assert that iv contains a `struct S`
If iv does in fact contain a struct S value—if that's what's inside the box once you open it—then this type-assertion doesn't panic, and cv winds up with the concrete value of type struct S. If panic is undesirable, we can use the cv, ok := iv.(struct S) form, or a type switch. All of these—including the version that panics—work by checking the type of the value inside the interface.
What this—or, more precisely, the way the Go language is defined—tells us is that the interface "box" really holds two things:
a concrete type, and
a concrete value.
Well, that is, unless it holds a <nil, nil> pair, in which case iv == nil is true. Note that the iv == nil test actually tests both parts.
If Go had a syntax for this, we could write something like iv.type and iv.value to get at the two separate parts. But we can't do that. We have to use type assertions, type-switch, or reflect. So, going back to this:
I would like to know if I can somehow find out if an interface is of type struct
we can see that the question itself is just a little malformed. We don't want to know if an interface value has this type. We want to know if a non-nil interface's held value is of this type, as if we could inspect iv.type and iv.value directly.
If you have a limited set of possible types, you can use the type-switch construct, and enumerate all your allowed possiblities:
switch cv := iv.(type) {
case struct S:
// work with cv, which is a struct S
case *struct S:
// work with cv, which is a *struct S
// add more cases as appropriate
}
If you need more generality, instead of doing the above, we end up using the reflect package:
tv := reflect.TypeOf(iv)
or:
vv := reflect.ValueOf(iv)
The latter is actually the more useful form, since vv captures both the iv.type pseudo-field and the iv.value pseudo-field.
As mkopriva notes in a comment, test, in your sample code, has type interface{}, so &test has type *interface{}. In most cases this is not a good idea: you just want to pass the interface{} value directly.
To allow the called function to set the object to a new value, you will want to pass a pointer to the object as the interface value. You do not want to pass a pointer to the interface while having the interface hold the struct "in the box" as it were. You need a reflect.Value on which you can invoke Set(), and to get one, you will need to follow an elem on the reflect.Value that is a pointer to the struct (not one that is a pointer to the interface).
There's a more complete example here on the Go Playground.
1This is partly an allusion to "boxed values" in certain other programming languages (see What is boxing and unboxing and what are the trade offs?), but partly literal. Don't mistake Go's interfaces for Java's boxed values, though: they are not the same at all.
Maybe what you need is type assertion?
t, ok := v.(myStruct)
https://tour.golang.org/methods/15
In any case this code prints "struct":
type tt struct {}
var x tt
var z interface{}
z = x
v := reflect.ValueOf(z).Kind()
fmt.Printf("%v\n", v)
And see this for setting the value of a struct field using reflection:
Using reflect, how do you set the value of a struct field?
I have a function which returns a slice of pointers of some interface. I want to change the type later in the code to the implementation type but nothing works, I still get an invalid type assertion.
Example
func Test(c Parsable)([]*Parsable, error) {
// generate slice by factory method on Parsable inteface and return slice
}
var implParsable ImplParsable
results, err := Test(implParsable)
data := results[0].(ImplParsable) // I tried this in many variations but nothing works
resultSets[0] is a pointer to an interface, so you need to dereference that pointer to get the interface value, which you can do inline since slice values are addressable.
data := (*resultSets[0]).(ImplParsable)
ERROR: type CustomStruct is not an expression.
type CustomStruct struct {
}
func getTypeName(t interface{}) string {
rt := reflect.TypeOf(t).Elem()
return rt.Name()
}
getTypeName(CustomStruct)
How can I pass struct type to function without type instance?
This will work
getTypeName((*CustomStruct)(nil))
But I wonder if there is more simple version..
You can't. You can only pass a value, and CustomStruct is not a value but a type. Using a type identifier is a compile-time error.
Usually when a "type" is to be passed, you pass a reflect.Type value which describes the type. This is what you "create" inside your getTypeName(), but then the getTypeName() will have little left to do:
func getTypeName(t reflect.Type) string {
return t.Name()
}
// Calling it:
getTypeName(reflect.TypeOf(CustomStruct{}))
(Also don't forget that this returns an empty string for anonymous types such as []int.)
Another way is to pass a "typed" nil pointer value as you did, but again, you can just as well use a typed nil value to create the reflect.Type too, without creating a value of the type in question, like this:
t := reflect.TypeOf((*CustomStruct)(nil)).Elem()
fmt.Println(t.Name()) // Prints CustomStruct
Lets resurrect this!
The generics proposal for Go got approved, and that's coming, eventually. When this question was first asked, this probably made more sense as a question, but for anyone looking to implement a generics pattern now, I think I've got an alright API for it.
For now, you can't interact with abstract types, but you can interact with methods on the abstract type, and reflect allows you to examine function signatures. For a method, the 0th is the receiver.
type Example struct {int}
type Generic struct{reflect.Type}
func (p Example) Type() {}
func Reflect(generic interface{}) Generic {
real := reflect.TypeOf(generic)
if real.Kind() != reflect.Func || real.NumIn() < 1 {
panic("reflect.Type.In(n) panics if not a func and if n out of bounds")
}
return Generic{real.In(0)}
}
func (g Generic) Make() interface{} {
return reflect.Zero(g.Type).Interface()
}
func main() {
tOfp := Reflect(Example.Type)
fmt.Printf("Name of the type: %v\n", tOfp.Name())
fmt.Printf("Real (initial)value: %v\n", tOfp.Make())
}
Some quick notes:
The structure of "Example" doesn't matter, rather only that it has a method with a non-pointer receiver.
The definition of a type called "Generic" as a struct is to accomplish what I believed OP's actual intent to be.
The above definition of "Generic" is a struct instead of an interface so that it can have its own method set. Defining "Generic" as an interface, and using a methodset specific to each operand-type used with it would make tons of sense.
If you weren't aware, actual generics are coming in Go 1.18. My example above has no linter or compile protection, and will panic at runtime if used incorrectly. It does work, and will let you reason over abstract types while you wait for a native implementation.
Happy Coding!
From Go version 1.18 a new feature Generics has been introduced. In most of the case instead of passing types to function, we can use generics. Then we will also get compile time error instead of runtime error and it's more efficient than reflect also.
Example Code
func HttpGet[T](url, body) T {
var resp T
return T
}
resp := HttpGet[ResponseType]("dummy.example", nil)
I have a listener which receives protobuf messages. However it doesn't know which type of message comes in when. So I tried to unmarshal into an interface{} so I can later type cast:
var data interface{}
err := proto.Unmarshal(message, data)
if err != nil {
log.Fatal("unmarshaling error: ", err)
}
log.Printf("%v\n", data)
However this code doesn't compile:
cannot use data (type interface {}) as type proto.Message in argument to proto.Unmarshal:
interface {} does not implement proto.Message (missing ProtoMessage method)
How can I unmarshal and later type cast an "unknown" protobuf message in go?
First, two words about the OP's question, as presented by them:
proto.Unmarshal can't unmarshal into an interface{}. The method signature is obvious, you must pass a proto.Message argument, which is an interface implemented by concrete protobuffer types.
When handling a raw protobuffer []byte payload that didn't come in an Any, ideally you have at least something (a string, a number, etc...) coming together with the byte slice, that you can use to map to the concrete protobuf message.
You can then switch on that and instantiate the appropriate protobuf concrete type, and only then pass that argument to Unmarshal:
var message proto.Message
switch atLeastSomething {
case "foo":
message = &mypb.Foo{}
case "bar":
message = &mypb.Bar{}
}
_ = proto.Unmarshal(data, message)
Now, what if the byte payload is truly unknown?
As a foreword, consider that this should seldom happen in practice. The schema used to generate the protobuffer types in your language of choice represents a contract, and by accepting protobuffer payloads you are, for some definitions of it, fulfilling that contract.
Anyway, if for some reason you must deal with a completely unknown, mysterious, protobuffer payload in wire format, you can extract some information from it with the protowire package.
Be aware that the wire representation of a protobuf message is ambiguous. A big source of uncertainty is the "length-delimited" type (2) being used for strings, bytes, repeated fields and... sub-messages (reference).
You can retrieve the payload content, but you are bound to have weak semantics.
The code
With that said, this is what a parser for unknown proto messages may look like. The idea is to leverage protowire.ConsumeField to read through the original byte slice.
The data model could be like this:
type Field struct {
Tag Tag
Val Val
}
type Tag struct {
Num int32
Type protowire.Type
}
type Val struct {
Payload interface{}
Length int
}
And the parser:
func parseUnknown(b []byte) []Field {
fields := make([]Field, 0)
for len(b) > 0 {
n, t, fieldlen := protowire.ConsumeField(b)
if fieldlen < 1 {
return nil
}
field := Field{
Tag: Tag{Num: int32(n), Type: t },
}
_, _, taglen := protowire.ConsumeTag(b[:fieldlen])
if taglen < 1 {
return nil
}
var (
v interface{}
vlen int
)
switch t {
case protowire.VarintType:
v, vlen = protowire.ConsumeVarint(b[taglen:fieldlen])
case protowire.Fixed64Type:
v, vlen = protowire.ConsumeFixed64(b[taglen:fieldlen])
case protowire.BytesType:
v, vlen = protowire.ConsumeBytes(b[taglen:fieldlen])
sub := parseUnknown(v.([]byte))
if sub != nil {
v = sub
}
case protowire.StartGroupType:
v, vlen = protowire.ConsumeGroup(n, b[taglen:fieldlen])
sub := parseUnknown(v.([]byte))
if sub != nil {
v = sub
}
case protowire.Fixed32Type:
v, vlen = protowire.ConsumeFixed32(b[taglen:fieldlen])
}
if vlen < 1 {
return nil
}
field.Val = Val{Payload: v, Length: vlen - taglen}
// fmt.Printf("%#v\n", field)
fields = append(fields, field)
b = b[fieldlen:]
}
return fields
}
Sample input and output
Given a proto schema like:
message Foo {
string a = 1;
string b = 2;
Bar bar = 3;
}
message Bar {
string c = 1;
}
initialized in Go as:
&test.Foo{A: "A", B: "B", Bar: &test.Bar{C: "C"}}
And by adding a fmt.Printf("%#v\n", field) statement at the end of the loop in the above code, it will output the following:
main.Field{Tag:main.Tag{Num:1, Type:2}, Val:main.Val{Payload:[]uint8{0x41}, Length:1}}
main.Field{Tag:main.Tag{Num:2, Type:2}, Val:main.Val{Payload:[]uint8{0x42}, Length:1}}
main.Field{Tag:main.Tag{Num:1, Type:2}, Val:main.Val{Payload:[]uint8{0x43}, Length:1}}
main.Field{Tag:main.Tag{Num:3, Type:2}, Val:main.Val{Payload:[]main.Field{main.Field{Tag:main.Tag{Num:1, Type:2}, Val:main.Val{Payload:[]uint8{0x43}, Length:1}}}, Length:3}}
About sub-messages
As you can see from the above the idea to deal with a protowire.BytesType that may or may not be a message field is to attempt to parse it, recursively. If it succeeds, we keep the resulting msg and store it in the field value, if it fails, we store the bytes as-is, which then may be a proto string or bytes. BTW, if I'm reading correctly, this seems what Marc Gravell does in the Protogen code.
About repeated fields
The code above doesn't deal with repeated fields explicitly, but after the parsing is done, repeated fields will have the same value for Field.Tag.Num. From that, packing the fields into a slice/array should be trivial.
About maps
The code above also doesn't deal with proto maps. I suspect that maps are semantically equivalent to a repeated k/v pair, e.g.:
message Pair {
string key = 1; // or whatever key type
string val = 2; // or whatever val type
}
If my assumption is correct, then maps can be parsed with the given code as sub-messages.
About oneofs
I haven't yet tested this, but I expect that information about the union type are completely lost. The byte payload will contain only the value that was actually set.
But what about Any?
The Any proto type doesn't fit in the picture. Contrary to what it may look like, Any is not analogous to, say, map[string]interface{} for JSON objects. And the reason is simple: Any is a proto message with a very well defined structure, namely (in Go):
type Any struct {
// unexported fields
TypeUrl string // struct tags omitted
Value []byte // struct tags omitted
}
So it is more similar to the implementation of a Go interface{} in that it holds some actual data and that data's type information.
It can hold itself arbitrary proto payloads (with their type information!) but it can not be used to decode unknown messages, because Any has exactly those two fields, type url and a byte payload.
To wrap up, this answer doesn't provide a full-blown production-grade solution, but it shows how to decode arbitrary payloads while preserving as much original semantics as possible. Hopefully it will point you in the right direction.
As you've seen, and the commenters have pointed out, you can't use proto.Unmarshal to interface{} because, the method expects a type Message which implements an interface MessageV1.
Protobuf messages are typed and correspond to method invocations ("comes in") and the implementation cannot take generic types of protobuf but specific protobufs:
func (s *server) M(ctx context.Context, _ *pb.Foo) (*pb.Bar, error)
The solution is to envelope your generic types as Any within a specific type perhaps Envelope:
message Envelope {
google.protobuf.Any content = 1;
...
}
The content is then transmitted as a []byte (see Golang anypb.Any) and the implementation (anypb) includes methods to pack|unpack these.
The 'trick' with Any is that messages include a [TypeURL] that uniquely identifies the message so that the receiver knows how to e.g. Unmarshal it.
I am trying to understand the code that is used at my company. I am new to go lang, and I have already gone through the tutorial on their official website. However, I am having a hard time wrapping my head around empty interfaces, i.e. interface{}. From various sources online, I figured out that the empty interface can hold any type. But, I am having a hard time figuring out the codebase, especially some of the functions. I will not be posting the entire thing here, but just the minimal functions in which it has been used. Please bear with me!
Function (I am trying to understand):
func (this *RequestHandler) CreateAppHandler(rw http.ResponseWriter, r *http.Request) *foo.ResponseError {
var data *views.Data = &views.Data{Attributes: &domain.Application{}}
var request *views.Request = &views.Request{Data: data}
if err := json.NewDecoder(r.Body).Decode(request); err != nil {
logrus.Error(err)
return foo.NewResponsePropogateError(foo.STATUS_400, err)
}
requestApp := request.Data.Attributes.(*domain.Application)
requestApp.CreatedBy = user
Setting some context, RequestHandler is a struct defined in the same package as this code. domain and views are seperate packages. Application is a struct in the package domain. The following two structs are part of the package views:
type Data struct {
Id string `json:"id"`
Type string `json:"type"`
Attributes interface{} `json:"attributes"`
}
type Request struct {
Data *Data `json:"data"`
}
The following are part of the package json:
func NewDecoder(r io.Reader) *Decoder {
return &Decoder{r: r}
}
func (dec *Decoder) Decode(v interface{}) error {
if dec.err != nil {
return dec.err
}
if err := dec.tokenPrepareForDecode(); err != nil {
return err
}
if !dec.tokenValueAllowed() {
return &SyntaxError{msg: "not at beginning of value"}
}
// Read whole value into buffer.
n, err := dec.readValue()
if err != nil {
return err
}
dec.d.init(dec.buf[dec.scanp : dec.scanp+n])
dec.scanp += n
// Don't save err from unmarshal into dec.err:
// the connection is still usable since we read a complete JSON
// object from it before the error happened.
err = dec.d.unmarshal(v)
// fixup token streaming state
dec.tokenValueEnd()
return err
}
type Decoder struct {
r io.Reader
buf []byte
d decodeState
scanp int // start of unread data in buf
scan scanner
err error
tokenState int
tokenStack []int
}
Now, I understood that, in the struct Data in package views, Application is being set as a type for the empty interface. After that, a pointer to Request in the same package is created which points to the variable data.
I have the following doubts:
What exactly does this keyword mean in Go? What is the purpose of writing this * RequestHandler?
Initialization of a structure in Go can be done while assigning it to a variable by specifying the values of all it's members. However, here, for the struct Data, only the empty interface value is assigned and the values for the other two fields are not assigned?
What is the advantage of assigning the Application struct to an empty interface? Does it mean I can use the struct members using the interface variable directly?
Can someone help me figure out the meaning of this statement? json.NewDecoder(r.Body).Decode(request)?
While I know this is too much, but I am having a hard time figuring out the meaning of interfaces in Go. Please help!
this is not a keyword in go; any variable name can be used there. That is called the receiver. A function declared in that way must be called like thing.func(params), where "thing" is an expression of the type of the receiver. Within the function, the receiver is set to the value of thing.
A struct literal does not have to contain values for all the fields (or any of them). Any fields not explicitly set will have the zero value for their types.
As you said, an empty interface can take on a value of any type. To use a value of type interface{}, you would use type assertion or a type switch to determine the type of the value, or you could use reflection to use the value without having to have code for the specific type.
What specifically about that statement do you not understand? json is the name of a package in which the function NewDecoder is declared. That function is called, and then the Decode function (which is implemented by the type of the return value of NewDecoder) is called on that return value.
You may want to take a look at Effective Go and/or The Go Programming Language Specification for more information.