I want to write a mockData method which can accept several types of parameter and return correspond objects based on its json data. The code as below:
func MockData(jsonPath string,v interface{})(interface{},error){
var ret interface{}
data,_ := ioutil.ReadFile(jsonPath)
switch v.(type) {
case Req:
ret = Req{}
fmt.Printf("\n===before Unmarshal==%T===\n",ret)
err = json.Unmarshal(data,&ret)
if err!=nil{...}
fmt.Printf("======after unmarshal===%T\n",ret)
case ...
default:
fmt.Printf("error===not match")
}
return ret,err
}
However, it panics when I use it. The code as below:
func main(){
reqJsonPath := /xx/yy/req.json
obj,err:=test.MockData(jsonFile,Req{})
if err!=nil{...}
require := obj.(Req) //panic cant []interface{} to Req
}
and the output of MockData is:
===before Unmarshal==Req===
======after unmarshal===[]interface{}
the type of object changed after unmarshal. and some more strange is that if I replace:
ret = Req{}
with
ret = &Req{}
the output will be same as below:
===before Unmarshal==*Req===
======after unmarshal===*Req
To reproduce the problem more conveniently I give the Require struct as below:
type Req []*Ele
type Ele struct {
ID int
Level int
}
summary:
Can I achieve expected function which produces different types of objects based on its json and type?
Why does the type of object changed after unmarshal, and why it not changed after I add &?
Can I achieve expected function which produces different types of objects based on its json and type?
func MockData(filename string, v interface{}) (interface{}, error) {
data, _ := ioutil.ReadFile(filename)
switch t := v.(type) {
case Req:
// t at this point is a Req{}
err := json.Unmarshal(data, &t)
return t, err
}
return nil, errors.New("unknown type")
}
I don't really know your motivation why you you need to pass an actual struct rather than a pointer. Check this demonstration
Why does the type of object changed after unmarshal, and why it not changed after I add &?
When you unmarshal using &ret where ret is an interface, you are getting the address of the interface. Hence, json.Unmarshal() will see that the backing data is a interface rather than a pointer to a struct. The default data type that json.Unmarshal() will use is map[string]interface{} for objects and []interface{} for arrays.
Now if you unmarshal using ret where ret is &Req{}, json.Unmarshal() will check that the backing data is a struct, hence it can do it's unmarshaling using the struct's fields.
Edit:
You seem to be confused by pointer to an interface which is different to an interface which has a pointer. Try this code and you'll see the difference.
var x interface{} = Req{}
var y interface{} = &x
var z interface{} = &Req{}
fmt.Printf("%T\n", y)
fmt.Printf("%T\n", z)
Remember that interfaces are just normal values and they also take memory. Now if you take an address of that memory, you get the pointer to the interface rather than the pointer to the data the interface is referring to.
Can I achieve expected function which produces different types of objects based on its json and type?
Yes, but you'll have to convert it back using a type assertion at the calling end ie
MyFoo:=MockData("foo.json", Foo{}).(Foo)
(or have multiple return ret.(Foo) return ret.(Bar) in the func)
Why does the type of object changed after unmarshal, and why it not changed after I add &?
There are some helpful comments in the top of the Unmarshal source
namely
// To unmarshal JSON into a pointer, Unmarshal first handles the case of
// the JSON being the JSON literal null. In that case, Unmarshal sets
// the pointer to nil. Otherwise, Unmarshal unmarshals the JSON into
// the value pointed at by the pointer. If the pointer is nil, Unmarshal
// allocates a new value for it to point to.
and
// To unmarshal JSON into an interface value,
// Unmarshal stores one of these in the interface value:
//
// bool, for JSON booleans
// float64, for JSON numbers
// string, for JSON strings
// []interface{}, for JSON arrays
// map[string]interface{}, for JSON objects
// nil for JSON null
So in the first case you are unmarshalling into an interface value (ret is declared as an interface{})
In the second case there is a pointer to a struct so that's what you get
Related
I'm trying to test around an SQL query wherein one of the arguments is a gosnowflake.Array (essentially a wrapper to a slice) using the go-sqlmock package. Normally, something like this requires me to create a value converter, which I have included:
func (opt arrayConverterOption[T]) ConvertValue(v any) (driver.Value, error) {
casted, ok := v.(*[]T)
if ok {
Expect(*casted).Should(HaveLen(len(opt.Expected)))
for i, c := range *casted {
Expect(c).Should(Equal(opt.Expected[i]))
}
} else {
fmt.Printf("Type: %T\n", v)
return v, nil
}
return "TEST_RESULT", nil
}
Now, this function is called for every argument submitted to the query. I use it to test the correctness of the values in the slice or pass the argument through if it isn't one. The problem I'm having is that, when I create a arrayConverterOption[string] and give it a gosnowflake.Array(["A", "B", "C"]) as an argument, the type assertion fails because gosnowflake.Array returns an internal dynamic type, *stringArray, which is defined as a *[]string.
So you can see my dilemma here. On the one hand, I can't convert v because it's an interface{} and I can't alias v because the inner type is not *[]string, but *stringArray. So then, what should I do here?
I didn't find a way to do this without resulting to reflection. However, with reflction I did manage it:
var casted []T
var ok bool
value := reflect.ValueOf(v)
if value.Kind() == reflect.Pointer {
if inner := value.Elem(); inner.Kind() == reflect.Slice {
r := inner.Convert(reflect.TypeOf([]T{})).Interface()
casted, ok = r.([]T)
}
}
So, this code checks specifically for anything that is a pointer to a slice, which my dynamic type is. Then it uses reflection to convert the inner object to the slice type I was expecting. After that, I call Interface() on the result to get the interface{} from the reflected value and then cast it to a []T. This succeeds. If it doesn't then I'm not working with one of those dynamically typed slices and I can handle the type normally.
I have a variable that needs to be either a string or map[string]string (will be deserializing from JSON). So I declare it as interface{}. How can I check that the value is map[string]string?
This question How to check interface is a map[string]string in golang almost answers my question. But the accepted answer only works if the variable is declared as a map[string]string not if the variable is interface{}.
package main
import (
"fmt"
)
func main() {
var myMap interface{}
myMap = map[string]interface{}{
"foo": "bar",
}
_, ok := myMap.(map[string]string)
if !ok {
fmt.Println("This will be printed")
}
}
See https://play.golang.org/p/mA-CVk7bdb9
I can use two type assertions though. One on the map and one on the map value.
package main
import (
"fmt"
)
func main() {
var myMap interface{}
myMap = map[string]interface{}{
"foo": "bar",
}
valueMap, ok := myMap.(map[string]interface{})
if !ok {
fmt.Println("will not be printed")
}
for _, v := range valueMap {
if _, ok := v.(string); !ok {
fmt.Println("will not be printed")
}
}
}
See https://play.golang.org/p/hCl8eBcKSqE
Question: is there a better way?
If you declare a variable as type interface{}, it is type interface{}. It is not, ever, some map[keytype]valuetype value. But a variable of type interface{} can hold a value that has some other concrete type. When it does so, it does so—that's all there is to it. It still is type interface{}, but it holds a value of some other type.
An interface value has two parts
The key distinction here is between what an interface{} variable is, and what it holds. Any interface variable actually has two slots inside it: one to hold what type is stored in it, and one to hold what value is stored in it. Any time you—or anyone—assign a value to the variable, the compiler fills in both slots: the type, from the type of the value you used, and the value, from the value you used.1 The interface variable compares equal to nil if it has nil in both slots; and that's also the default zero value.
Hence, your runtime test:
valueMap, ok := myMap.(map[string]interface{})
is a sensible thing to do: if myMap holds a value that has type map[string]interface, ok gets set to true and valueMap contains the value (which has that type). If myMap holds a value with some other type, ok gets set to false and valueMap gets set to the zero-value of type map[string]interface{}. In other words, at runtime, the code checks the type-slot first, then either copies the value-slot across to valueMap and sets ok to true, or sets valueMap to nil and sets ok to false.
If and when ok has been set to true, each valueMap[k] value is type interface{}. As before, for myMap itself, each of these interface{} variables can—but do not have to—hold a value of type string, and you must use some sort of "what is the actual type-and-value" run-time test to tease them apart.
When you use json.Unmarshal to stuff decoded JSON into a variable of type interface{}, it is capable of deserializing any of these documented JSON types. The list then tells you what type gets stuffed into the interface variable:
bool, for JSON booleans
float64, for JSON numbers
string, for JSON strings
[]interface{}, for JSON arrays
map[string]interface{}, for JSON objects
nil for JSON null
So after doing json.Unmarshal into a variable of type interface{}, you should check what type got put into the type-slot of the variable. You can do this with an assertion and an ok boolean, or you can, if you prefer, use a type switch to decode it:
var i interface
if err := json.Unmarshal(data, &i); err != nil {
panic(err)
}
switch v := i.(type) {
case string:
... code ...
case map[string]interface{}:
... code ...
... add some or all of the types listed ...
}
The thing is, no matter what you do in code here, you did have json.Unmarshal put something into an interface{}, and interface{} is the type of i. You must test at runtime what type and value pair the interface holds.
Your other option is to inspect your JSON strings manually and decide what type of variable to provide to json.Unmarshal. That gives you less code to write after the Unmarshal, but more code to write before it.
There's a more complete example here, on the Go playground, of using type switches to inspect the result from a json.Unmarshal. It's deliberately incomplete but, I think, has enough input and output cases to let you work out how to handle everything, given the quote above about what json.Unmarshal writes into a variable of type interface{}.
1Of course, if you assign one interface{} from some other interface{}:
var i1, i2 interface{}
... set i1 from some actual value ...
// more code, then:
i2 = i1
the compiler just copies both slots from i1 into i2. The two-separate-slots thing becomes clearer when you do:
var f float64
... code that sets f to, say, 1.5 ...
i2 = f
for instance, as that writes float64 into the type-slot, and the value 1.5 into the value-slot. The compiler knows that f is float64 so the type-setting just means "stick a constant in it". The compiler doesn't necessarily know the value of f so the value-setting is a copy of whatever the actual value is.
I have a function which returns a slice of pointers of some interface. I want to change the type later in the code to the implementation type but nothing works, I still get an invalid type assertion.
Example
func Test(c Parsable)([]*Parsable, error) {
// generate slice by factory method on Parsable inteface and return slice
}
var implParsable ImplParsable
results, err := Test(implParsable)
data := results[0].(ImplParsable) // I tried this in many variations but nothing works
resultSets[0] is a pointer to an interface, so you need to dereference that pointer to get the interface value, which you can do inline since slice values are addressable.
data := (*resultSets[0]).(ImplParsable)
I have a listener which receives protobuf messages. However it doesn't know which type of message comes in when. So I tried to unmarshal into an interface{} so I can later type cast:
var data interface{}
err := proto.Unmarshal(message, data)
if err != nil {
log.Fatal("unmarshaling error: ", err)
}
log.Printf("%v\n", data)
However this code doesn't compile:
cannot use data (type interface {}) as type proto.Message in argument to proto.Unmarshal:
interface {} does not implement proto.Message (missing ProtoMessage method)
How can I unmarshal and later type cast an "unknown" protobuf message in go?
First, two words about the OP's question, as presented by them:
proto.Unmarshal can't unmarshal into an interface{}. The method signature is obvious, you must pass a proto.Message argument, which is an interface implemented by concrete protobuffer types.
When handling a raw protobuffer []byte payload that didn't come in an Any, ideally you have at least something (a string, a number, etc...) coming together with the byte slice, that you can use to map to the concrete protobuf message.
You can then switch on that and instantiate the appropriate protobuf concrete type, and only then pass that argument to Unmarshal:
var message proto.Message
switch atLeastSomething {
case "foo":
message = &mypb.Foo{}
case "bar":
message = &mypb.Bar{}
}
_ = proto.Unmarshal(data, message)
Now, what if the byte payload is truly unknown?
As a foreword, consider that this should seldom happen in practice. The schema used to generate the protobuffer types in your language of choice represents a contract, and by accepting protobuffer payloads you are, for some definitions of it, fulfilling that contract.
Anyway, if for some reason you must deal with a completely unknown, mysterious, protobuffer payload in wire format, you can extract some information from it with the protowire package.
Be aware that the wire representation of a protobuf message is ambiguous. A big source of uncertainty is the "length-delimited" type (2) being used for strings, bytes, repeated fields and... sub-messages (reference).
You can retrieve the payload content, but you are bound to have weak semantics.
The code
With that said, this is what a parser for unknown proto messages may look like. The idea is to leverage protowire.ConsumeField to read through the original byte slice.
The data model could be like this:
type Field struct {
Tag Tag
Val Val
}
type Tag struct {
Num int32
Type protowire.Type
}
type Val struct {
Payload interface{}
Length int
}
And the parser:
func parseUnknown(b []byte) []Field {
fields := make([]Field, 0)
for len(b) > 0 {
n, t, fieldlen := protowire.ConsumeField(b)
if fieldlen < 1 {
return nil
}
field := Field{
Tag: Tag{Num: int32(n), Type: t },
}
_, _, taglen := protowire.ConsumeTag(b[:fieldlen])
if taglen < 1 {
return nil
}
var (
v interface{}
vlen int
)
switch t {
case protowire.VarintType:
v, vlen = protowire.ConsumeVarint(b[taglen:fieldlen])
case protowire.Fixed64Type:
v, vlen = protowire.ConsumeFixed64(b[taglen:fieldlen])
case protowire.BytesType:
v, vlen = protowire.ConsumeBytes(b[taglen:fieldlen])
sub := parseUnknown(v.([]byte))
if sub != nil {
v = sub
}
case protowire.StartGroupType:
v, vlen = protowire.ConsumeGroup(n, b[taglen:fieldlen])
sub := parseUnknown(v.([]byte))
if sub != nil {
v = sub
}
case protowire.Fixed32Type:
v, vlen = protowire.ConsumeFixed32(b[taglen:fieldlen])
}
if vlen < 1 {
return nil
}
field.Val = Val{Payload: v, Length: vlen - taglen}
// fmt.Printf("%#v\n", field)
fields = append(fields, field)
b = b[fieldlen:]
}
return fields
}
Sample input and output
Given a proto schema like:
message Foo {
string a = 1;
string b = 2;
Bar bar = 3;
}
message Bar {
string c = 1;
}
initialized in Go as:
&test.Foo{A: "A", B: "B", Bar: &test.Bar{C: "C"}}
And by adding a fmt.Printf("%#v\n", field) statement at the end of the loop in the above code, it will output the following:
main.Field{Tag:main.Tag{Num:1, Type:2}, Val:main.Val{Payload:[]uint8{0x41}, Length:1}}
main.Field{Tag:main.Tag{Num:2, Type:2}, Val:main.Val{Payload:[]uint8{0x42}, Length:1}}
main.Field{Tag:main.Tag{Num:1, Type:2}, Val:main.Val{Payload:[]uint8{0x43}, Length:1}}
main.Field{Tag:main.Tag{Num:3, Type:2}, Val:main.Val{Payload:[]main.Field{main.Field{Tag:main.Tag{Num:1, Type:2}, Val:main.Val{Payload:[]uint8{0x43}, Length:1}}}, Length:3}}
About sub-messages
As you can see from the above the idea to deal with a protowire.BytesType that may or may not be a message field is to attempt to parse it, recursively. If it succeeds, we keep the resulting msg and store it in the field value, if it fails, we store the bytes as-is, which then may be a proto string or bytes. BTW, if I'm reading correctly, this seems what Marc Gravell does in the Protogen code.
About repeated fields
The code above doesn't deal with repeated fields explicitly, but after the parsing is done, repeated fields will have the same value for Field.Tag.Num. From that, packing the fields into a slice/array should be trivial.
About maps
The code above also doesn't deal with proto maps. I suspect that maps are semantically equivalent to a repeated k/v pair, e.g.:
message Pair {
string key = 1; // or whatever key type
string val = 2; // or whatever val type
}
If my assumption is correct, then maps can be parsed with the given code as sub-messages.
About oneofs
I haven't yet tested this, but I expect that information about the union type are completely lost. The byte payload will contain only the value that was actually set.
But what about Any?
The Any proto type doesn't fit in the picture. Contrary to what it may look like, Any is not analogous to, say, map[string]interface{} for JSON objects. And the reason is simple: Any is a proto message with a very well defined structure, namely (in Go):
type Any struct {
// unexported fields
TypeUrl string // struct tags omitted
Value []byte // struct tags omitted
}
So it is more similar to the implementation of a Go interface{} in that it holds some actual data and that data's type information.
It can hold itself arbitrary proto payloads (with their type information!) but it can not be used to decode unknown messages, because Any has exactly those two fields, type url and a byte payload.
To wrap up, this answer doesn't provide a full-blown production-grade solution, but it shows how to decode arbitrary payloads while preserving as much original semantics as possible. Hopefully it will point you in the right direction.
As you've seen, and the commenters have pointed out, you can't use proto.Unmarshal to interface{} because, the method expects a type Message which implements an interface MessageV1.
Protobuf messages are typed and correspond to method invocations ("comes in") and the implementation cannot take generic types of protobuf but specific protobufs:
func (s *server) M(ctx context.Context, _ *pb.Foo) (*pb.Bar, error)
The solution is to envelope your generic types as Any within a specific type perhaps Envelope:
message Envelope {
google.protobuf.Any content = 1;
...
}
The content is then transmitted as a []byte (see Golang anypb.Any) and the implementation (anypb) includes methods to pack|unpack these.
The 'trick' with Any is that messages include a [TypeURL] that uniquely identifies the message so that the receiver knows how to e.g. Unmarshal it.
I am trying to understand the code that is used at my company. I am new to go lang, and I have already gone through the tutorial on their official website. However, I am having a hard time wrapping my head around empty interfaces, i.e. interface{}. From various sources online, I figured out that the empty interface can hold any type. But, I am having a hard time figuring out the codebase, especially some of the functions. I will not be posting the entire thing here, but just the minimal functions in which it has been used. Please bear with me!
Function (I am trying to understand):
func (this *RequestHandler) CreateAppHandler(rw http.ResponseWriter, r *http.Request) *foo.ResponseError {
var data *views.Data = &views.Data{Attributes: &domain.Application{}}
var request *views.Request = &views.Request{Data: data}
if err := json.NewDecoder(r.Body).Decode(request); err != nil {
logrus.Error(err)
return foo.NewResponsePropogateError(foo.STATUS_400, err)
}
requestApp := request.Data.Attributes.(*domain.Application)
requestApp.CreatedBy = user
Setting some context, RequestHandler is a struct defined in the same package as this code. domain and views are seperate packages. Application is a struct in the package domain. The following two structs are part of the package views:
type Data struct {
Id string `json:"id"`
Type string `json:"type"`
Attributes interface{} `json:"attributes"`
}
type Request struct {
Data *Data `json:"data"`
}
The following are part of the package json:
func NewDecoder(r io.Reader) *Decoder {
return &Decoder{r: r}
}
func (dec *Decoder) Decode(v interface{}) error {
if dec.err != nil {
return dec.err
}
if err := dec.tokenPrepareForDecode(); err != nil {
return err
}
if !dec.tokenValueAllowed() {
return &SyntaxError{msg: "not at beginning of value"}
}
// Read whole value into buffer.
n, err := dec.readValue()
if err != nil {
return err
}
dec.d.init(dec.buf[dec.scanp : dec.scanp+n])
dec.scanp += n
// Don't save err from unmarshal into dec.err:
// the connection is still usable since we read a complete JSON
// object from it before the error happened.
err = dec.d.unmarshal(v)
// fixup token streaming state
dec.tokenValueEnd()
return err
}
type Decoder struct {
r io.Reader
buf []byte
d decodeState
scanp int // start of unread data in buf
scan scanner
err error
tokenState int
tokenStack []int
}
Now, I understood that, in the struct Data in package views, Application is being set as a type for the empty interface. After that, a pointer to Request in the same package is created which points to the variable data.
I have the following doubts:
What exactly does this keyword mean in Go? What is the purpose of writing this * RequestHandler?
Initialization of a structure in Go can be done while assigning it to a variable by specifying the values of all it's members. However, here, for the struct Data, only the empty interface value is assigned and the values for the other two fields are not assigned?
What is the advantage of assigning the Application struct to an empty interface? Does it mean I can use the struct members using the interface variable directly?
Can someone help me figure out the meaning of this statement? json.NewDecoder(r.Body).Decode(request)?
While I know this is too much, but I am having a hard time figuring out the meaning of interfaces in Go. Please help!
this is not a keyword in go; any variable name can be used there. That is called the receiver. A function declared in that way must be called like thing.func(params), where "thing" is an expression of the type of the receiver. Within the function, the receiver is set to the value of thing.
A struct literal does not have to contain values for all the fields (or any of them). Any fields not explicitly set will have the zero value for their types.
As you said, an empty interface can take on a value of any type. To use a value of type interface{}, you would use type assertion or a type switch to determine the type of the value, or you could use reflection to use the value without having to have code for the specific type.
What specifically about that statement do you not understand? json is the name of a package in which the function NewDecoder is declared. That function is called, and then the Decode function (which is implemented by the type of the return value of NewDecoder) is called on that return value.
You may want to take a look at Effective Go and/or The Go Programming Language Specification for more information.