I am trying to understand the code that is used at my company. I am new to go lang, and I have already gone through the tutorial on their official website. However, I am having a hard time wrapping my head around empty interfaces, i.e. interface{}. From various sources online, I figured out that the empty interface can hold any type. But, I am having a hard time figuring out the codebase, especially some of the functions. I will not be posting the entire thing here, but just the minimal functions in which it has been used. Please bear with me!
Function (I am trying to understand):
func (this *RequestHandler) CreateAppHandler(rw http.ResponseWriter, r *http.Request) *foo.ResponseError {
var data *views.Data = &views.Data{Attributes: &domain.Application{}}
var request *views.Request = &views.Request{Data: data}
if err := json.NewDecoder(r.Body).Decode(request); err != nil {
logrus.Error(err)
return foo.NewResponsePropogateError(foo.STATUS_400, err)
}
requestApp := request.Data.Attributes.(*domain.Application)
requestApp.CreatedBy = user
Setting some context, RequestHandler is a struct defined in the same package as this code. domain and views are seperate packages. Application is a struct in the package domain. The following two structs are part of the package views:
type Data struct {
Id string `json:"id"`
Type string `json:"type"`
Attributes interface{} `json:"attributes"`
}
type Request struct {
Data *Data `json:"data"`
}
The following are part of the package json:
func NewDecoder(r io.Reader) *Decoder {
return &Decoder{r: r}
}
func (dec *Decoder) Decode(v interface{}) error {
if dec.err != nil {
return dec.err
}
if err := dec.tokenPrepareForDecode(); err != nil {
return err
}
if !dec.tokenValueAllowed() {
return &SyntaxError{msg: "not at beginning of value"}
}
// Read whole value into buffer.
n, err := dec.readValue()
if err != nil {
return err
}
dec.d.init(dec.buf[dec.scanp : dec.scanp+n])
dec.scanp += n
// Don't save err from unmarshal into dec.err:
// the connection is still usable since we read a complete JSON
// object from it before the error happened.
err = dec.d.unmarshal(v)
// fixup token streaming state
dec.tokenValueEnd()
return err
}
type Decoder struct {
r io.Reader
buf []byte
d decodeState
scanp int // start of unread data in buf
scan scanner
err error
tokenState int
tokenStack []int
}
Now, I understood that, in the struct Data in package views, Application is being set as a type for the empty interface. After that, a pointer to Request in the same package is created which points to the variable data.
I have the following doubts:
What exactly does this keyword mean in Go? What is the purpose of writing this * RequestHandler?
Initialization of a structure in Go can be done while assigning it to a variable by specifying the values of all it's members. However, here, for the struct Data, only the empty interface value is assigned and the values for the other two fields are not assigned?
What is the advantage of assigning the Application struct to an empty interface? Does it mean I can use the struct members using the interface variable directly?
Can someone help me figure out the meaning of this statement? json.NewDecoder(r.Body).Decode(request)?
While I know this is too much, but I am having a hard time figuring out the meaning of interfaces in Go. Please help!
this is not a keyword in go; any variable name can be used there. That is called the receiver. A function declared in that way must be called like thing.func(params), where "thing" is an expression of the type of the receiver. Within the function, the receiver is set to the value of thing.
A struct literal does not have to contain values for all the fields (or any of them). Any fields not explicitly set will have the zero value for their types.
As you said, an empty interface can take on a value of any type. To use a value of type interface{}, you would use type assertion or a type switch to determine the type of the value, or you could use reflection to use the value without having to have code for the specific type.
What specifically about that statement do you not understand? json is the name of a package in which the function NewDecoder is declared. That function is called, and then the Decode function (which is implemented by the type of the return value of NewDecoder) is called on that return value.
You may want to take a look at Effective Go and/or The Go Programming Language Specification for more information.
Related
I'm trying to write in Go custom cache for Google DataStore (more precisely - a wrapper around one of existing cache libraries). At cache initialisation, it should accept any custom type of struct (with appropriately-defined datastore fields), which then would be the basis for all items stored. The idea is that cache can be created/initialised for various types which reflect the structure of a particular DataStore entry (CustomEntry)
Approach 1 - store reflect.Type and use it. Problem encountered - can't iterate over a slice of a custom type
type CustomEntry struct {
Data struct {
name string `datastore:"name,noindex"`
address []string `datastore:"address,noindex"`
} `datastore:"data,noindex"`
}
func (cache *MyCache) CacheData(dataQuery string, dataType reflect.Type) {
slice := reflect.MakeSlice(reflect.SliceOf(dataType), 10, 10)
if keys, err := DataStoreClient.GetAll(cache.ctx, datastore.NewQuery(dataQuery), &slice); err != nil {
//handle error
} else {
for i, dataEntry:= range slice {
// ERROR: Cannot range over 'slice' (type Value)
cache.Set(keys[i].Name, dataEntry)
}
}
//usage: Cache.CacheData("Person", reflect.TypeOf(CustomEntry{})
Approach 2 - accept an array of interfaces as arguments. Problem encountered = []CustomEntry is not []interface{}
func (cache *MyCache) CacheData(dataQuery string, dataType []interface{}) {
if keys, err := DataStoreClient.GetAll(cache.ctx, datastore.NewQuery(dataQuery), &dataType); err != nil {
//handle error
} else {
for i, dataEntry:= range slice {
// this seems to work fine
cache.Set(keys[i].Name, dataEntry)
}
}
//usage:
var dataType []CustomEntry
Cache.CacheData("Person", data)
// ERROR: Cannot use 'data' (type []CustomEntry) as type []interface{}
Any suggestions would be highly appreciated.
I have found a solution and thought it might be worth sharing in case anyone else has a similar problem.
The easiest way is to initiate a slice of structs which the DataStore is expected to receive, and then to pass a pointer to it as an argument (interface{}) into the desired function. DataStore, similarly to a few unmarshaling functions (I have tried with JSON package) will be able to successfully append the data to it.
Trying to dynamically create the slice within the function, given a certain Type, which would be then accepted by a function (such as DataStore client) might be quite difficult (I have not managed to find a way to do it). Similarly, passing a slice of interfaces (to allow for easy iteration) only complicates things.
Secondly, in order to iterate over the data (e.g. to store it in cache), it is necessary to:
(1) retrieve the underlying value of the interface (i.e. the pointer itself) - this can be achieved using reflect.ValueOf(pointerInterface),
(2) dereference the pointer so that we obtain access to the underlying, iterable slice of structs - this can be done by invoking .Elem(),
(3) iterate over the underlying slice using .Index(i) method (range will not accept an interface, even if the underlying type is iterable).
Naturally, adding a number of switch-case statements might be appropriate to ensure that any errors are caught rather than cause a runtime panic.
Hence the following code provides a working solution to the above problem:
In main:
var data []customEntry
c.CacheData("Person",&data)
And the function itself:
func (cache *MyCache) CacheData(dataQuery string, data interface{}) error {
if keys, err := DataStoreClient.GetAll(cache.ctx, datastore.NewQuery(dataQuery), data); err != nil {
return err
} else {
s := reflect.ValueOf(data).Elem()
for i := 0; i < s.Len(); i++ {
cache.Set(keys[i].Name, s.Index(i), 1)
}
}
}
I want to write a mockData method which can accept several types of parameter and return correspond objects based on its json data. The code as below:
func MockData(jsonPath string,v interface{})(interface{},error){
var ret interface{}
data,_ := ioutil.ReadFile(jsonPath)
switch v.(type) {
case Req:
ret = Req{}
fmt.Printf("\n===before Unmarshal==%T===\n",ret)
err = json.Unmarshal(data,&ret)
if err!=nil{...}
fmt.Printf("======after unmarshal===%T\n",ret)
case ...
default:
fmt.Printf("error===not match")
}
return ret,err
}
However, it panics when I use it. The code as below:
func main(){
reqJsonPath := /xx/yy/req.json
obj,err:=test.MockData(jsonFile,Req{})
if err!=nil{...}
require := obj.(Req) //panic cant []interface{} to Req
}
and the output of MockData is:
===before Unmarshal==Req===
======after unmarshal===[]interface{}
the type of object changed after unmarshal. and some more strange is that if I replace:
ret = Req{}
with
ret = &Req{}
the output will be same as below:
===before Unmarshal==*Req===
======after unmarshal===*Req
To reproduce the problem more conveniently I give the Require struct as below:
type Req []*Ele
type Ele struct {
ID int
Level int
}
summary:
Can I achieve expected function which produces different types of objects based on its json and type?
Why does the type of object changed after unmarshal, and why it not changed after I add &?
Can I achieve expected function which produces different types of objects based on its json and type?
func MockData(filename string, v interface{}) (interface{}, error) {
data, _ := ioutil.ReadFile(filename)
switch t := v.(type) {
case Req:
// t at this point is a Req{}
err := json.Unmarshal(data, &t)
return t, err
}
return nil, errors.New("unknown type")
}
I don't really know your motivation why you you need to pass an actual struct rather than a pointer. Check this demonstration
Why does the type of object changed after unmarshal, and why it not changed after I add &?
When you unmarshal using &ret where ret is an interface, you are getting the address of the interface. Hence, json.Unmarshal() will see that the backing data is a interface rather than a pointer to a struct. The default data type that json.Unmarshal() will use is map[string]interface{} for objects and []interface{} for arrays.
Now if you unmarshal using ret where ret is &Req{}, json.Unmarshal() will check that the backing data is a struct, hence it can do it's unmarshaling using the struct's fields.
Edit:
You seem to be confused by pointer to an interface which is different to an interface which has a pointer. Try this code and you'll see the difference.
var x interface{} = Req{}
var y interface{} = &x
var z interface{} = &Req{}
fmt.Printf("%T\n", y)
fmt.Printf("%T\n", z)
Remember that interfaces are just normal values and they also take memory. Now if you take an address of that memory, you get the pointer to the interface rather than the pointer to the data the interface is referring to.
Can I achieve expected function which produces different types of objects based on its json and type?
Yes, but you'll have to convert it back using a type assertion at the calling end ie
MyFoo:=MockData("foo.json", Foo{}).(Foo)
(or have multiple return ret.(Foo) return ret.(Bar) in the func)
Why does the type of object changed after unmarshal, and why it not changed after I add &?
There are some helpful comments in the top of the Unmarshal source
namely
// To unmarshal JSON into a pointer, Unmarshal first handles the case of
// the JSON being the JSON literal null. In that case, Unmarshal sets
// the pointer to nil. Otherwise, Unmarshal unmarshals the JSON into
// the value pointed at by the pointer. If the pointer is nil, Unmarshal
// allocates a new value for it to point to.
and
// To unmarshal JSON into an interface value,
// Unmarshal stores one of these in the interface value:
//
// bool, for JSON booleans
// float64, for JSON numbers
// string, for JSON strings
// []interface{}, for JSON arrays
// map[string]interface{}, for JSON objects
// nil for JSON null
So in the first case you are unmarshalling into an interface value (ret is declared as an interface{})
In the second case there is a pointer to a struct so that's what you get
ERROR: type CustomStruct is not an expression.
type CustomStruct struct {
}
func getTypeName(t interface{}) string {
rt := reflect.TypeOf(t).Elem()
return rt.Name()
}
getTypeName(CustomStruct)
How can I pass struct type to function without type instance?
This will work
getTypeName((*CustomStruct)(nil))
But I wonder if there is more simple version..
You can't. You can only pass a value, and CustomStruct is not a value but a type. Using a type identifier is a compile-time error.
Usually when a "type" is to be passed, you pass a reflect.Type value which describes the type. This is what you "create" inside your getTypeName(), but then the getTypeName() will have little left to do:
func getTypeName(t reflect.Type) string {
return t.Name()
}
// Calling it:
getTypeName(reflect.TypeOf(CustomStruct{}))
(Also don't forget that this returns an empty string for anonymous types such as []int.)
Another way is to pass a "typed" nil pointer value as you did, but again, you can just as well use a typed nil value to create the reflect.Type too, without creating a value of the type in question, like this:
t := reflect.TypeOf((*CustomStruct)(nil)).Elem()
fmt.Println(t.Name()) // Prints CustomStruct
Lets resurrect this!
The generics proposal for Go got approved, and that's coming, eventually. When this question was first asked, this probably made more sense as a question, but for anyone looking to implement a generics pattern now, I think I've got an alright API for it.
For now, you can't interact with abstract types, but you can interact with methods on the abstract type, and reflect allows you to examine function signatures. For a method, the 0th is the receiver.
type Example struct {int}
type Generic struct{reflect.Type}
func (p Example) Type() {}
func Reflect(generic interface{}) Generic {
real := reflect.TypeOf(generic)
if real.Kind() != reflect.Func || real.NumIn() < 1 {
panic("reflect.Type.In(n) panics if not a func and if n out of bounds")
}
return Generic{real.In(0)}
}
func (g Generic) Make() interface{} {
return reflect.Zero(g.Type).Interface()
}
func main() {
tOfp := Reflect(Example.Type)
fmt.Printf("Name of the type: %v\n", tOfp.Name())
fmt.Printf("Real (initial)value: %v\n", tOfp.Make())
}
Some quick notes:
The structure of "Example" doesn't matter, rather only that it has a method with a non-pointer receiver.
The definition of a type called "Generic" as a struct is to accomplish what I believed OP's actual intent to be.
The above definition of "Generic" is a struct instead of an interface so that it can have its own method set. Defining "Generic" as an interface, and using a methodset specific to each operand-type used with it would make tons of sense.
If you weren't aware, actual generics are coming in Go 1.18. My example above has no linter or compile protection, and will panic at runtime if used incorrectly. It does work, and will let you reason over abstract types while you wait for a native implementation.
Happy Coding!
From Go version 1.18 a new feature Generics has been introduced. In most of the case instead of passing types to function, we can use generics. Then we will also get compile time error instead of runtime error and it's more efficient than reflect also.
Example Code
func HttpGet[T](url, body) T {
var resp T
return T
}
resp := HttpGet[ResponseType]("dummy.example", nil)
I'm using gob to serialize structs to disk. The struct in question contains an interface field, so the concrete type needs to be registered using gob.Register(...).
The wrinkle here is that the library doing the gob-ing should be ignorant of the concrete type in use. I wanted the serialization to be possible even when callers have defined their own implementations of the interface.
I can successfully encode the data by registering the type on the fly (see trivial example below), but upon trying to re-read that data, gob refuses to accept the un-registered type. Its frustrating, because it feels like all the data is there - why isn't gob just unpacking that as a main.UpperCaseTransformation struct if it's labelled as such?
package main
import (
"encoding/gob"
"fmt"
"os"
"strings"
)
type Transformation interface {
Transform(s string) string
}
type TextTransformation struct {
BaseString string
Transformation Transformation
}
type UpperCaseTransformation struct{}
func (UpperCaseTransformation) Transform(s string) string {
return strings.ToUpper(s)
}
func panicOnError(err error) {
if err != nil {
panic(err)
}
}
// Execute this twice to see the problem (it will tidy up files)
func main() {
file := os.TempDir() + "/so-example"
if _, err := os.Stat(file); os.IsNotExist(err) {
tt := TextTransformation{"Hello, World!", UpperCaseTransformation{}}
// Note: didn't need to refer to concrete type explicitly
gob.Register(tt.Transformation)
f, err := os.Create(file)
panicOnError(err)
defer f.Close()
enc := gob.NewEncoder(f)
err = enc.Encode(tt)
panicOnError(err)
fmt.Println("Run complete, run again for error.")
} else {
f, err := os.Open(file)
panicOnError(err)
defer os.Remove(f.Name())
defer f.Close()
var newTT TextTransformation
dec := gob.NewDecoder(f)
// Errors with: `gob: name not registered for interface: "main.UpperCaseTransformation"'
err = dec.Decode(&newTT)
panicOnError(err)
}
}
My work-around would be to require implementers of the interface to register their type with gob. But I don't like how that reveals my serialization choices to the callers.
Is there any route forward that avoids this?
Philosophical argumentation
The encoding/gob package cannot (or rather should not) make that decision on its own. Since the gob package creates a serialized form independent of / detached from the app, there is no guarantee that values of interface types will exist in the decoder; and even if they do (matched by the concrete type name), there is no guarantee that they represent the same type (or the same implementation of a given type).
By calling gob.Register() (or gob.RegisterName()) you make that intent clear, you give green light to the gob package to use that type. This also ensures that the type does exist, else you would not be able to pass a value of it when registering.
Technical requirement
There's also a technical point of view that dictates this requirement (that you must register prior): you cannot obtain the reflect.Type type descriptor of a type given by its string name. Not just you, the encoding/gob package can't do it either.
So by requiring you to call gob.Register() prior, the gob package will receive a value of the type in question, and therefore it can (and it will) access and store its reflect.Type descriptor internally, and so when a value of this type is detected, it is capable of creating a new value of this type (e.g. using reflect.New()) in order to store the value being decoded into it.
The reason why you can't "lookup" types by name is that they may not end up in your binary (they may get "optimized out") unless you explicitly refer to them. For details see Call all functions with special prefix or suffix in Golang; and Splitting client/server code. When registering your custom types (by passing values of them), you are making an explicit reference to them and thus ensuring that they won't get excluded from the binaries.
I am trying to create a function that takes a []byte and an interface{} (standing for the struct) and returns an interface{} as the struct type passed into the func.
Something like this:
package main
import (
"encoding/json"
)
func UnmarshalFromJSONArray(sms []byte,tt string) (interface{}) {
var ts = new(tt)
err := json.Unmarshal(sms,&ts)
if(err != nil) {
fmt.Println(err)
}
return sms
}
So that method would run something like this:
// let's say a struct has the following definition:
type MyStructType struct {
Id int
Name string
Desc string
}
// we can some how get its fully qualified class name (this may require reflection?) or pass it into the UnMarshal method direction some how.
mst := "package.MyStructType",
// and then assume a byte array ba that has JSON format for
ba := []byte(`{"Id":"3","Name":"Jack","Desc":"the man"}`)
stct := UnmarshalFromJSONArray(ba,mst)
MyStructureType out := stct
// leaving "stct" being the unmarshalled byte array which can be used like any other struct of type "MyStructureType"
The key being that I never need to know what the fields of MyStructureType are before unmarshalling. All I need are the name of the struct and some way to instance one and then populate it with JSON byte array data that matches its fields. Hopefully that is possible (it is trivial in java using reflection). So I want to basically unmarshal an anonymous struct type by it's name without needing to know what fields it has.
Any suggestions?
The short answer is that this is impossible. There is no string to type translator in Go. You can make a map of strings to reflect.Type's, but you would need to know the possible options ahead of time or you need to provide the caller with a way to register types (perhaps in init).
Assuming you have found a way to resolve the string to its reflect.Type, you can simply call reflect.New(typ).Interface() to get the pointer you need to pass to json.Unmarshal().
The best answer is to avoid trying this all together. Writing idiomatic Java in Go isn't really possible. If I knew more about your problem, I could give you a more idiomatic Go solution.