I need to parse a Go source code file, find a specific type (by name) and use it in my program.
I already managed to find the type I need using the go/ast package but I don't know how to "load" it into my program so that I can use it.
Question: What's the best way to extract and use a type from an external source code file and use it on runtime?
I can't think of anything except an ugly method to basically copy the file, modify it by injecting a "main" function with my encoding stuff which sends the result to stdOut, execute the it, collect the encoded data from stdout, delete the modified file.
Use case: Analyse go source code and encode the types in a specific format (e.g. json schema)
Edit:
Here is some code. The question is how to encode type allTypes (zero value) and then print it.
package main
import (
"fmt"
"go/ast"
"go/parser"
"go/token"
"encoding/json"
)
var src string = `
package mypack
type allTypes struct{
Brands Brands
Colours Colours
}
type Brands struct{
Sony string
Apple string
}
type Colours struct{
Red string
Green string
}
`
type sometype struct{
Nothing int
}
func main() {
// src is the input for which we want to inspect the AST.
// Create the AST by parsing src.
fset := token.NewFileSet() // positions are relative to fset
f, err := parser.ParseFile(fset, "src.go", src, 0)
if err != nil {
panic(err)
}
// Inspect the AST and find our function
var tp ast.TypeSpec
ast.Inspect(f, func(n ast.Node) bool {
switch x := n.(type) {
case *ast.TypeSpec:
if x.Name.Name == "allTypes"{
tp = *x
}
}
return true
})
fmt.Printf("We found the type: it is %v", tp)
// Encode the zero value of sometype
x := sometype{}
b, _ := json.Marshal(&x)
fmt.Printf("\n Zero value of someType (json) %s", b)
//Next/Question: How to encode the zero value of "allTypes" ???
}
Also on playground
If I understand you are asking for dynamic type loading ala Java's Class.forName(String className). The short answer is Go doesn't support this.
The correct way, as Nick Johnson pointed out, is to parse the tree using ast, and then "generate" the JSON yourself. You will not be able to "load" the type and use JSON.Marshal. It is also worth noting that any type which supports the json.Marshaler interface may generate custom JSON. You also need to ignore, but mark optional "omitempty" behavior. This really prevents you from using the compile it and hack through "stdout" behavior as well.
If you need to extract type information at runtime, you need to use the reflect package. This is the way Go's encoding/json and other similar packages work.
If you want to operate on the types defined in a Go source file, you can use the go.parser package to read and parse a source file into an AST, then traverse the AST for the elements you want to examine.
It seems there is no way to load the type information extracted from the source code(please correct me if I'm wrong). The only solution is to create/generate a package (main), inject it with all the types extracted from the target source code file, build/compile it, execute it and collect the encoded data from stdout.
Related
I want to create a data type which acts like a stack. I want to add and remove entries at the "top", as well as being able to print it out. In this example an XPath type for traversing an xml document and keeping track of the current path.
So I created a type xPath []string, and write the appropriate functions, ie: push() pop() and String().
My problem here is that the type loses its state, which baffles me a bit since I thought slices were reference types. Also if I try changing my functions into pointer receivers I run into several compile errors. At this point just to get by the problem, I simply changed []string into a struct with a single []string field. Though it still bathers me that I can't make it work with just a slice as the underlying type.
What is the correct way to do this?
package main
import (
"fmt"
"strings"
)
type xPath []string
func (xp xPath) push(entry string) {
xp = append(xp, entry)
}
func (xp xPath) String() string {
sb := strings.Builder{}
sb.WriteString("/")
sb.WriteString(strings.Join(xp, "/"))
return sb.String()
}
func main() {
xp := xPath{}
xp.push("rss")
xp.push("channel")
xp.push("items")
fmt.Println(xp)
// Output: /
// Wanted: /rss/channel/items
}
Your push function is doing nothing.
Correct push function:
func (xp *xPath) push(entry string) {
*xp = append(*xp, entry)
}
Slices are reference types in cases where you want to change their values (e.g. using indexes).
On the other hand, if you want to reassign them and replace the whole slice, you should use pointers.
Also about the stack, the are some better approaches:
have a look at this question.
I would like, at any stage in a program, to perform an operation on all instances of a struct defined in a given package. In order to do that, I have contemplated developping a function that gather those instances. But I haven't found anything yet.
To clarify my goal (following '#Muffin Top' and '#Volker' answer), I currently do this registration manualy into a pointer store present in each package. There are two problems with this approach, the registration is a mundane task and it is error prone.
To make my case clear, consider a package foo with one struct Foo. Foo has one string field Name. The package also defines two instances of Foo declared as Foo1 & Foo2
package foo
type Foo struct {
Name string
}
var Foo1, Foo2 = Foo{Name: "Foo1"}, Foo{Name: "Foo2"}
The GetInstancesOfPkg I am trying to implement would be called like in the below example. The operation to perform is just to print the Name field.
This is a working example but with a mockup version of GetInstancesOfPkg.
package main
import (
"log"
"reflect"
"github.com/thomaspeugeot/test-heapdump/foo"
)
func main() {
s := GetInstancesOfPkg("github.com/thomaspeugeot/test-heapdump/foo")
for _, instance := range s {
// cast instance on foo.Foo
if f, ok := instance.(*foo.Foo); !ok {
log.Panic("Unknown Struct " + reflect.TypeOf(instance).Name())
} else {
log.Printf("Instance of Foo has name %s ", f.Name)
}
}
}
// GetInstancesOfPkg return a slice of instances of Struct defined in the package pkgPath
// --- this is the mockup verion ---
func GetInstancesOfPkg(pkgPath string) (interfaceSlice []interface{}) {
var listOfFooInstances = []*foo.Foo{&foo.Foo1, &foo.Foo2}
interfaceSlice = make([]interface{}, len(listOfFooInstances))
for i, d := range listOfFooInstances {
interfaceSlice[i] = d
}
return
}
For the solution, I know there is no such function in the reflect standard library or any other library I know of. However, I was contemplating :
an algorithm that parses the entire program memory, identify each variable, get its type and gather those that match the struct type of the package. Since delve and heapdump do parse the entire memory and that delve can guess the variable type, I have a hunch that my desired function could be implemented. I have been trying to see how delve is doing or how heapcoredump is developped but this is a too big piece of knowledge to swallow for the desired goal.
an alternative would be to add a pre-compilation step that modifies the abstract syntax tree (AST) by adding the registration each time there is a a variable initialisation.
I am not even sure if this level of reflexion is possible with go (or if it is portable ?)
Performance is not an issue, the program could be compiled with wathever flag necessary or the entire memory could be parsed.
Is there a way in go to dynamicaly gather instances of a given struct type?
No.
You must redesign, e.g. by allocating the instances through a function which memorizes them.
I'm using gob to serialize structs to disk. The struct in question contains an interface field, so the concrete type needs to be registered using gob.Register(...).
The wrinkle here is that the library doing the gob-ing should be ignorant of the concrete type in use. I wanted the serialization to be possible even when callers have defined their own implementations of the interface.
I can successfully encode the data by registering the type on the fly (see trivial example below), but upon trying to re-read that data, gob refuses to accept the un-registered type. Its frustrating, because it feels like all the data is there - why isn't gob just unpacking that as a main.UpperCaseTransformation struct if it's labelled as such?
package main
import (
"encoding/gob"
"fmt"
"os"
"strings"
)
type Transformation interface {
Transform(s string) string
}
type TextTransformation struct {
BaseString string
Transformation Transformation
}
type UpperCaseTransformation struct{}
func (UpperCaseTransformation) Transform(s string) string {
return strings.ToUpper(s)
}
func panicOnError(err error) {
if err != nil {
panic(err)
}
}
// Execute this twice to see the problem (it will tidy up files)
func main() {
file := os.TempDir() + "/so-example"
if _, err := os.Stat(file); os.IsNotExist(err) {
tt := TextTransformation{"Hello, World!", UpperCaseTransformation{}}
// Note: didn't need to refer to concrete type explicitly
gob.Register(tt.Transformation)
f, err := os.Create(file)
panicOnError(err)
defer f.Close()
enc := gob.NewEncoder(f)
err = enc.Encode(tt)
panicOnError(err)
fmt.Println("Run complete, run again for error.")
} else {
f, err := os.Open(file)
panicOnError(err)
defer os.Remove(f.Name())
defer f.Close()
var newTT TextTransformation
dec := gob.NewDecoder(f)
// Errors with: `gob: name not registered for interface: "main.UpperCaseTransformation"'
err = dec.Decode(&newTT)
panicOnError(err)
}
}
My work-around would be to require implementers of the interface to register their type with gob. But I don't like how that reveals my serialization choices to the callers.
Is there any route forward that avoids this?
Philosophical argumentation
The encoding/gob package cannot (or rather should not) make that decision on its own. Since the gob package creates a serialized form independent of / detached from the app, there is no guarantee that values of interface types will exist in the decoder; and even if they do (matched by the concrete type name), there is no guarantee that they represent the same type (or the same implementation of a given type).
By calling gob.Register() (or gob.RegisterName()) you make that intent clear, you give green light to the gob package to use that type. This also ensures that the type does exist, else you would not be able to pass a value of it when registering.
Technical requirement
There's also a technical point of view that dictates this requirement (that you must register prior): you cannot obtain the reflect.Type type descriptor of a type given by its string name. Not just you, the encoding/gob package can't do it either.
So by requiring you to call gob.Register() prior, the gob package will receive a value of the type in question, and therefore it can (and it will) access and store its reflect.Type descriptor internally, and so when a value of this type is detected, it is capable of creating a new value of this type (e.g. using reflect.New()) in order to store the value being decoded into it.
The reason why you can't "lookup" types by name is that they may not end up in your binary (they may get "optimized out") unless you explicitly refer to them. For details see Call all functions with special prefix or suffix in Golang; and Splitting client/server code. When registering your custom types (by passing values of them), you are making an explicit reference to them and thus ensuring that they won't get excluded from the binaries.
package main
import (
"fmt"
"encoding/json"
"reflect"
)
type GeneralConfig map[string]interface{}
var data string = `
{
"key":"value",
"important_key":
{"foo":"bar"}
}`
func main() {
jsonData := &GeneralConfig{}
json.Unmarshal([]byte(data), jsonData)
fmt.Println(reflect.TypeOf(jsonData)) //main.GeneralConfig
jsonTemp := (*jsonData)["important_key"]
fmt.Println(reflect.TypeOf(jsonTemp)) //map[string]interface {}
//newGeneralConfig := GeneralConfig(jsonTemp)
//cannot convert jsonTemp (type interface {}) to type GeneralConfig:
//need type assertion
newGeneralConfig := jsonTemp.(GeneralConfig)
//fmt.Println(reflect.TypeOf(newGeneralConfig))
//panic: interface conversion: interface {} is map[string]interface {},
//not main.GeneralConfig
}
Available at the playground
I understand that I can use a nested struct in lieu of GeneralConfig, but that would require me knowing the exact structure of the payload, ie it wouldn't work for different keys (I would be locked into "important_key").
Is there a golang workaround for when I don't know what the value of "important_key" is? I say golang, because if possible, one could require all "important_keys" to have a constant parent key, which could resolve this issue.
To summarize, given an arbitrary json object, there must be a way that I can traverse its keys, and if a value is a custom type, convert the value to that type. Right now it seems that if I use type conversion, it tells me that the type is interface{} and I need to use type assertion; however, if I use type assertion, it tells me that interface{} is map[string]interface{} not main.GeneralConfig.
I agree the comments about trying to utilise the expected structure of the incoming JSON in order to write well-defined Structs, but I'll attempt to answer the question anyway.
The thing to take away from what you're seeing printed versus the error messages that you're seeing is that the compiler knows less about the type than the runtime because the runtime can look at the actual value. To bring the compiler up-to-speed we must (i) assert (*jsonData)["important_key"] is a map[string]interface{} -- the compiler only knows it to be an interface{} -- and then (ii) type-cast that to a GeneralConfig type. See:
package main
import (
"fmt"
"encoding/json"
)
type GeneralConfig map[string]interface{}
func main() {
jsonStruct := new(GeneralConfig)
json.Unmarshal([]byte(`{"parent_key": {"foo": "bar"}}`), jsonStruct)
fmt.Printf("%#v\n", jsonStruct)
// => &main.GeneralConfig{"parent_key":map[string]interface {}{"foo":"bar"}}
nestedStruct := (*jsonStruct)["parent_key"]
fmt.Printf("%#v\n", nestedStruct)
// => map[string]interface {}{"foo":"bar"}
// Whilst this shows the runtime knows its actual type is
// map[string]interface, the compiler only knows it to be an interface{}.
// First we assert for the compiler that it is indeed a
// map[string]interface{} we are working with. You can imagine the issues
// that might arrise if we has passed in `{"parent_key": 123}`.
mapConfig, ok := nestedStruct.(map[string]interface{})
if !ok {
// TODO: Error-handling.
}
// Now that the compiler can be sure mapConfig is a map[string]interface{}
// we can type-cast it to GeneralConfig:
config := GeneralConfig(mapConfig)
fmt.Printf("%#v\n", config)
// => main.GeneralConfig{"foo":"bar"}
}
You are looking for json.RawMessage.
You can delay unmarshalling based upon some other value and then force it to unmarshal to a specific type.
This is not a good idea, but might be closer to what you are looking for.
http://play.golang.org/p/PWwAUDySE0
This is a standard "workaround" if get what you're after. When handling unknown data you can implement this pattern (modified from your example) of switching on the type recursively to get to the concrete values in an unknown body of json data.
package main
import (
"encoding/json"
"fmt"
"reflect"
)
var data = `
{
"key":"value",
"important_key":
{"foo":"bar"}
}`
func main() {
var jsonData interface{}
json.Unmarshal([]byte(data), &jsonData)
fmt.Println(reflect.TypeOf(jsonData))
parseArbitraryJSON(jsonData.(map[string]interface{}))
}
func parseArbitraryJSON(data map[string]interface{}) {
for k, v := range data {
switch a := v.(type) {
case string:
fmt.Printf("%v:%v\n", k, a)
case map[string]interface{}:
fmt.Printf("%v:%v\n", k, a)
parseArbitraryJSON(a)
}
}
}
The resulting output is:
map[string]interface {}
key:value
important_key:map[foo:bar]
foo:bar
This example only accounts for the base data being a string type but you can switch on any type that you expect to receive, and like any switch you can group your cases, so you can treat all numbers similarly for example.
I am trying to create a function that takes a []byte and an interface{} (standing for the struct) and returns an interface{} as the struct type passed into the func.
Something like this:
package main
import (
"encoding/json"
)
func UnmarshalFromJSONArray(sms []byte,tt string) (interface{}) {
var ts = new(tt)
err := json.Unmarshal(sms,&ts)
if(err != nil) {
fmt.Println(err)
}
return sms
}
So that method would run something like this:
// let's say a struct has the following definition:
type MyStructType struct {
Id int
Name string
Desc string
}
// we can some how get its fully qualified class name (this may require reflection?) or pass it into the UnMarshal method direction some how.
mst := "package.MyStructType",
// and then assume a byte array ba that has JSON format for
ba := []byte(`{"Id":"3","Name":"Jack","Desc":"the man"}`)
stct := UnmarshalFromJSONArray(ba,mst)
MyStructureType out := stct
// leaving "stct" being the unmarshalled byte array which can be used like any other struct of type "MyStructureType"
The key being that I never need to know what the fields of MyStructureType are before unmarshalling. All I need are the name of the struct and some way to instance one and then populate it with JSON byte array data that matches its fields. Hopefully that is possible (it is trivial in java using reflection). So I want to basically unmarshal an anonymous struct type by it's name without needing to know what fields it has.
Any suggestions?
The short answer is that this is impossible. There is no string to type translator in Go. You can make a map of strings to reflect.Type's, but you would need to know the possible options ahead of time or you need to provide the caller with a way to register types (perhaps in init).
Assuming you have found a way to resolve the string to its reflect.Type, you can simply call reflect.New(typ).Interface() to get the pointer you need to pass to json.Unmarshal().
The best answer is to avoid trying this all together. Writing idiomatic Java in Go isn't really possible. If I knew more about your problem, I could give you a more idiomatic Go solution.