I'm trying to get simple YAML parsing in go working with gopkg.in/yaml.v2.
From the documentation:
Maps and pointers (to a struct, string, int, etc) are accepted as
out values. If an internal pointer within a struct is not
initialized, the yaml package will initialize it if necessary
for unmarshalling the provided data. The out parameter must not be nil.
The type of the decoded values should be compatible with the
respective values in out. If one or more values cannot be decoded
due to a type mismatches, decoding continues partially until the
end of the YAML content, and a *yaml.TypeError is returned with
details for all missed values.
Note the important bits here: "pointer within struct initialized if nec'y", and "yaml.TypeError returned if a value can't be decoded".
Now:
package main
import (
"fmt"
"gopkg.in/yaml.v2"
)
type YamlTypeA struct {
D string `yaml: "d"`
E string `yaml: "e"`
}
type YamlTypeB struct {
F string `yaml: "f"`
}
type YamlTypeC struct {
TypeA *YamlTypeA `yaml: "a"`
TypeB []YamlTypeB `yaml: "b"`
}
func main() {
var yamlObj YamlTypeC
text := []byte(`
---
a:
d: foo
e: bar
b: [{f: "fails"},
{f: "completely"}]
`)
err := yaml.Unmarshal(text,&yamlObj)
if err != nil {
fmt.Println(err.Error())
return
}
fmt.Println("OK")
fmt.Printf("TypeA.D: %s\n", yamlObj.TypeA.D)
fmt.Printf("I have %d TypeB\n", len(yamlObj.TypeB))
}
yields
OK
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x0 pc=0x4014b3]
which appears to violate the promises made in the documentation jointly. If I make the nested YamlTypeA immediate instead of a pointer, the result is that the output value is not touched, still no error:
OK
TypeA.D:
I have 0 TypeB
What gives here? Is it just broken / poorly documented? How can you get nested struct values to parse out of YAML? (Using maps of maps is cumbersome and not at all a solution here.)
You have spaces in your struct tags, so they're being ignored:
type YamlTypeA struct {
D string `yaml:"d"`
E string `yaml:"e"`
}
type YamlTypeB struct {
F string `yaml:"f"`
}
type YamlTypeC struct {
TypeA *YamlTypeA `yaml:"a"`
TypeB []YamlTypeB `yaml:"b"`
}
Related
I am confused whether to use a & with go when declaring a variable and init with a struct
say we have a struct wrapper
type HttpResult struct {
Status int32 `json:"status"`
Msg string `json:"msg"`
Data interface{} `json:"data,omitempty"`
}
and a struct defining the user model
type OmUser struct {
Id primitive.ObjectID `json:"id" bson:"_id,omitempty"`
Name string `json:"name"`
Password string `json:"password"`
Email string `json:"email"`
}
And the following declaring seems give the same result:
myOmUser := OmUser{ //note no & sign here
Name: "Tony",
Password: "mypass",
Email: "tony#foo.com"
}
httpResult := &HttpResult{
Status: 0,
Msg: "ok",
Data: myOmUser,
}
js, _ := json.Marshal(httpResult)
fmt.Println(js)
Or
myOmUser := &OmUser{ //note the & sign
Name: "Tony",
Password: "mypass",
Email: "tony#foo.com"
}
httpResult := &HttpResult{
Status: 0,
Msg: "ok",
Data: myOmUser,
}
js, _ := json.Marshal(httpResult)
fmt.Println(js)
so, when to use & and why?
In your particular example it doesn't make a difference.
But when we look at an example of using json.Unmarshal() it makes a bit more sense:
jsonBlob := []byte(`{"id": "1", "name": "bob", "password": "pass", "email", "hi#me.com"}`)
var newOmUser OmUser
err := json.Unmarshal(jsonBlob, &newOmUser)
if err != nil {
panic(err)
}
Here we declare the variable before hand, and then we use the & to pass a pointer to that variable into the Unmarshal function.
That means that the Unmarshal function can reach out and update that variable, even though it's declared outside of the function.
Without the &, the Unmarshal function would get a copy of the newOmUser variable, and it would leave the original newOmUser variable that we declared empty.
When it comes to pointers, my general rule of thumb is:
Don't use them unless you have to.
If you need to use any unmarshalling functions, you'll need them. There are lots of other functions that make use of them.
Here's a quick exercise that helps me understand a little more about pointers:
func uppercase(s string) {
s = strings.ToUpper(s)
fmt.Println(s)
}
// Same as the uppercase() function, but works with a pointer.
func uppercasePointer(s *string) {
*s = strings.ToUpper(*s)
fmt.Println(*s)
}
name := "bob"
uppercase(name) // prints 'BOB'
fmt.Println(name) // prints 'bob' - our variable was not changed
name2 := "bobpointer"
uppercasePointer(&name2) // prints 'BOBPOINTER'
fmt.Println(name2) // prints 'BOBPOINTER' - our variable was changed
When we call the uppercase(name) function, go makes a copy of the name variable and sends it to the uppercase function.
Whatever the function does to that copy that it received stays in the function. The original variable that we declared outside the function is not changed.
When we call the uppercasePointer(&name2) function, we are sending a pointer to the name2 variable we declared.
The function can use that pointer to reach out and update the name2 variable that we declared earlier.
At first, you might not see the point of pointers, but as you continue to use go, you will see that they help us solve some complex problems.
Empty interface type in Go can hold values of any type. Tour here.
So in your HttpResult.Data is an empty interface type. So you can assigne any type to it.
The difference between defining a variable with & is getting a pointer of that type. Tour here.
Those are obviously two types with two different functionalities in Go. But you can assign both to empty interface type variable because its accepting values of any type.
package main
import (
"fmt"
"reflect"
)
type OmUser struct {
}
func main() {
myOmUser := OmUser{}
myOmUser2 := &OmUser{}
fmt.Println(reflect.TypeOf(myOmUser)) //main.OmUser
fmt.Println(reflect.TypeOf(myOmUser2)) //*main.OmUser
}
For more details about &, read Go doc address operators
For an
operand x of type T, the address operation &x generates a pointer of
type *T to x. The operand must be addressable, that is, either a
variable, pointer indirection, or slice indexing operation; or a field
selector of an addressable struct operand; or an array indexing
operation of an addressable array. As an exception to the
addressability requirement, x may also be a (possibly parenthesized)
composite literal. If the evaluation of x would cause a run-time
panic, then the evaluation of &x does too.
fooA := &Foo{}
fooA has type *Foo.
fooB := Foo{}
fooB has type Foo.
https://tour.golang.org/moretypes/1
In practice, this means if you had a func that accepted type *Foo you could do either of the following...
func someFunc(f *Foo) {
// ...
}
fooA := &Foo{}
someFunc(fooA)
fooB := Foo{}
someFunc(&fooB)
So realistically, create whichever you need to be honest.
Defined struct type:
type ValidateTemplateQuery struct {
TemplateURL *string `json:"template_url" valid:"optional"`
TemplateBody *string `json:"template_body" valid:"optional"`
}
Trying to initiate with mock values:
action := &ValidateTemplateQuery{
TemplateURL: *("TemplateURLValue"),
TemplateBody: *("TemplateBodyValue"),
}
Error
getting "panic: runtime error: invalid memory address or nil pointer dereference"
trying to initiate with mock values
Use the address of variables. For example,
package main
import (
"fmt"
)
func main() {
type ValidateTemplateQuery struct {
TemplateURL *string `json:"template_url" valid:"optional"`
TemplateBody *string `json:"template_body" valid:"optional"`
}
url := "TemplateURLValue"
body := "TemplateBodyValue"
action := &ValidateTemplateQuery{
TemplateURL: &url,
TemplateBody: &body,
}
fmt.Println(action, *action.TemplateURL, *action.TemplateBody)
}
Playground: https://play.golang.org/p/LZKV2LZ4KiE
Output:
&{0x40c138 0x40c140} TemplateURLValue TemplateBodyValue
The Go Programming Language Specification
Address operators
For an operand x of type T, the address operation &x generates a
pointer of type *T to x. The operand must be addressable, that is,
either a variable, pointer indirection, or slice indexing operation;
or a field selector of an addressable struct operand; or an array
indexing operation of an addressable array. As an exception to the
addressability requirement, x may also be a (possibly parenthesized)
composite literal. If the evaluation of x would cause a run-time
panic, then the evaluation of &x does too.
I'd like to iterate over the fields in a struct and prompt for string values to string fields, doing this recursively for fields that are pointers to structs.
Currently this is what I've tried, but I get an error when trying to set this value in the pointer's string field.
package main
import (
"fmt"
"reflect"
)
type Table struct {
PK *Field
}
type Field struct {
Name string
}
func main() {
PopulateStruct(&Table{})
}
func PopulateStruct(a interface{}) interface {} {
typeOf := reflect.TypeOf(a)
valueOf := reflect.ValueOf(a)
for i := 0; i < typeOf.Elem().NumField(); i++ {
switch typeOf.Elem().Field(i).Type.Kind() {
case reflect.String:
fmt.Print(typeOf.Elem().Field(i).Name)
var s string
fmt.Scanf("%s", &s)
valueOf.Elem().Field(i).SetString(s)
case reflect.Ptr:
ptr := reflect.New(valueOf.Elem().Field(i).Type())
PopulateStruct(ptr.Elem().Interface())
valueOf.Elem().Field(i).Set(ptr)
}
}
}
Expecting the return value to include an initialised struct with the pointers string field set.
Getting an error when setting the pointer's string field.
panic: reflect: call of reflect.Value.Field on zero Value
I dropped your code as-is into the Go Playground and it doesn't build because PopulateStruct is declared as returning interface{} but does not actually return anything. Removing the declared return type produces the panic you mention.
This is because at entry to the outer PopulateStruct call, you have a valid pointer, pointing to a zero-valued Table. A zero-valued Table has one element: a nil pointer in it of type *Field. Your loop therefore runs once and finds a reflect.Ptr, as you expected. Adding more diagnostic print messages helps see what's happening:
fmt.Printf("PopulateStruct: I have typeOf=%v, valueOf=%v\n", typeOf, valueOf)
for i := 0; i < typeOf.Elem().NumField(); i++ {
switch typeOf.Elem().Field(i).Type.Kind() {
// ... snipped some code ...
case reflect.Ptr:
ptr := reflect.New(valueOf.Elem().Field(i).Type())
fmt.Println("after allocating ptr, we have:", ptr.Type(), ptr,
"but its Elem is:", ptr.Elem().Type(), ptr.Elem())
This prints:
PopulateStruct: I have typeOf=*main.Table, valueOf=&{<nil>}
after allocating ptr, we have: **main.Field 0x40c138 but its Elem is: *main.Field <nil>
Given the way PopulateStruct itself is constructed, we must actually allocate a real Field instance now, before calling PopulateStruct. We can do this with:
p2 := ptr.Elem()
ptr.Elem().Set(reflect.New(p2.Type().Elem()))
(code borrowed from json.Unmarshal). Now we can fill in this Field, which has one field named Name of type String.
The overall strategy here is not that great, in my opinion: filling-in probably should take a generic pointer, not specifically a pointer-to-struct pointer. You can then emulate the indirect function in the json unmarshaller. However, the addition of these two lines—creating the target object and making the allocated pointer point to it—suffices to make your existing code run.
(Alternatively, you could just create and return a whole instance from scratch, in which case all you need is the type—but I'm assuming you have a pattern in which only some fields are nil.)
Here's the complete Go Playground example. I made a few other changes as there's nothing to scan from when using the playground.
Here is a piece of code play.google.org that runs without any problem:
package main
import (
"fmt"
)
func PrintAnonymous(v struct {
i int
s string
}) {
fmt.Printf("%d: %s\n", v.i, v.s)
}
func PrintAnonymous2(v struct{}) {
fmt.Println("Whatever")
}
func main() {
value := struct {
i int
s string
}{
0, "Hello, world!",
}
PrintAnonymous(value)
PrintAnonymous2(struct{}{})
}
However, if the PrintAnonymous() function exists in another package (let's say, temp), the code will not work:
cannot use value (type struct { i int; s string })
as type struct { i int; s string } in argument to temp.PrintAnonymous
My question are:
Is there a way to call a (public) function with anonymous struct as a parameter (a.k.a. PrintAnonymous() above)?
A function with empty struct as a parameter (a.k.a. PrintAnonymous2() above) can be called even if it exists in another package. Is this a special case?
Well, I know that I can always name the struct to solve the problem, I'm just curious about this, and wonder why it seems that this is not allowed.
The fields of your anonymous struct type are unexported. This means you cannot create values of this struct and specify values for the fields from another package. Spec: Composite literals:
It is an error to specify an element for a non-exported field of a struct belonging to a different package.
If you change the struct definition to export the fields, then it will work because all fields can be assigned to by other packages (see Siu Ching Pong -Asuka Kenji-'s answer).
This is the case with the empty struct (with no fields) too: the empty struct has no fields, thus it has no unexported fields, so you're allowed to pass a value of that.
You can call the function with unmodified struct (with unexported fields) via reflection though. You can obtain the reflect.Type of the PrintAnonymous() function, and you can use Type.In() to get the reflect.Type of its first parameter. That is the anonymous struct we want to pass a value for. And you can construct a value of that type using reflect.New(). This will be a reflect.Value, wrapping a pointer to the zero value of the anonymous struct. Sorry, you can't have a struct value with fields having non-zero values (for reason mentioned above).
This is how it could look like:
v := reflect.ValueOf(somepackage.PrintAnonymous)
paramt := v.Type().In(0)
v.Call([]reflect.Value{reflect.New(paramt).Elem()})
This will print:
0:
0 is zero value for int, and "" empty string for string.
For deeper inside into the type system and structs with unexported fields, see related questions:
Identify non builtin-types using reflect
How to clone a structure with unexported field?
Interestingly (this is a bug, see linked issue below), using reflection, you can use a value of your own anonymous struct type (with matching, unexported fields), and in this case we can use values other than the zero value of the struct fields:
value := struct {
i int
s string
}{
1, "Hello, world!",
}
v.Call([]reflect.Value{reflect.ValueOf(value)})
Above runs (without panic):
1: Hello, world!
The reason why this is allowed is due to a bug in the compiler. See the example code below:
s := struct{ i int }{2}
t := reflect.TypeOf(s)
fmt.Printf("Name: %q, PkgPath: %q\n", t.Name(), t.PkgPath())
fmt.Printf("Name: %q, PkgPath: %q\n", t.Field(0).Name, t.Field(0).PkgPath)
t2 := reflect.TypeOf(subplay.PrintAnonymous).In(0)
fmt.Printf("Name: %q, PkgPath: %q\n", t2.Name(), t2.PkgPath())
fmt.Printf("Name: %q, PkgPath: %q\n", t2.Field(0).Name, t2.Field(0).PkgPath)
Output is:
Name: "", PkgPath: ""
Name: "i", PkgPath: "main"
Name: "", PkgPath: ""
Name: "i", PkgPath: "main"
As you can see the unexported field i in both anonymous struct types (in main package and in somepackage as parameter to PrintAnonymous() function) –falsely– report the same package, and thus their type will be equal:
fmt.Println(t == t2) // Prints true
Note: I consider this a flaw: if this is allowed using reflection, then this should be possible without using reflection too. If without reflection the compile-time error is justified, then using reflection should result in runtime panic. I opened an issue for this, you can follow it here: issue #16616. Fix currently aims Go 1.8.
Oh! I just realized that the field names are in lowercase, and thus not public! Changing the first letter of the field names to uppercase solves the problem:
package main
import (
"temp"
)
func main() {
value := struct {
I int
S string
}{
0, "Hello, world!",
}
temp.PrintAnonymous(value)
temp.PrintAnonymous2(struct{}{})
}
package temp
import (
"fmt"
)
func PrintAnonymous(v struct{I int; S string}) {
fmt.Printf("%d: %s\n", v.I, v.S)
}
func PrintAnonymous2(v struct{}) {
fmt.Println("Whatever")
}
I'm running into a slight architectural problem with Golang right now that's causing me to copy/paste a bit more code than I'd prefer. I feel like there must be a solution, so please let me know if this is perhaps possible:
When I pass things through an interface {}-typed function parameter, I start getting errors such as "expected struct or slice", etc. ... even though what I passed was previously a struct or a slice. I realize that I could manually convert these to another type after receiving them in that function, but then that become tedious in instances such as this:
local interface type *interface {} can only be decoded from remote
interface type; received concrete type
... In this case, the receiving function seems like it'd need to be hard-coded to convert all interface {} items back to their respective original types in order to work properly, because the receiving function needs to know the exact type in order to process the item correctly.
Is there a way to dynamically re-type Golang interface {} typed variables back to their original type? Something like this, How to I convert reflect.New's return value back to the original type ... maybe?
EDIT: To clarify, basically, I'm passing &out to a function and it needs to be its original type by the time it reaches another inner function call.
Example code:
// NOTE: This is sort of pseudo-Golang code, not meant to be compiled or taken too seriously.
func PrepareTwoDifferentThings(keyA string, keyB string) {
var somethingA TypeA;
var somethingB TypeB;
loadFromCache(keyA, &somethingA, nil);
loadFromCache(keyB, &somethingB, nil);
fmt.Printf("Somethings: %v, %v", somethingA, somethingB);
}
func loadFromCache(key string, isNew, out interface {}, saveNewData interface {}) {
if err := cache.load(key, &out); err!=nil { // NOTE: Current issue is that this expects "&out" to be `TypeA`/`TypeB` not "interface {}", but I don't want to copy and paste this whole function's worth of code or whatever.
panic("oh no!");
}
if (saveNewData!=nil) {
cache.save(key, saveNewData); // This doesn't seem to care if "saveNewData" is "interface {}" when saving, but later cache fetches above using the "load()" method to an "interface {}"-typed `&out` parameter throw an exception that the "interface {}" type on `&out` does not match the original when it was saved here (`TypeA`/`TypeB`).
}
}
To change the type of an interface into its rightful type, you can use type assertions:
package main
import r "reflect"
type A struct {
Name string
}
func main() {
// No pointer
aa := A{"name"}
var ii interface{} = aa
bb := ii.(A)
// main.A
// Pointer
a := &A{"name"}
var i interface{} = a
b := *i.(*A)
// main.A
c := i.(*A)
// *main.A
d := r.Indirect(r.ValueOf(i)).Interface().(A)
// main.A
}
Playground 1
When using type assertions, you have to know the underlying type of your interface. In Go, there is no way to use type assertion with a dynamic type. reflect.Type is not a type, it's an interface representing a type. So no, you can't use it this way.
If you have several type possibilities, the solution is the type switch:
package main
import "fmt"
type TypeA struct {
A string
}
type TypeB struct {
B string
}
func doSomethingA(t TypeA) {
fmt.Println(t.A)
}
func doSomethingB(t TypeB) {
fmt.Println(t.B)
}
func doSomething(t interface{}) {
switch t := t.(type) {
case TypeA:
doSomethingA(t)
case TypeB:
doSomethingB(t)
default:
panic("Unrecognized type")
}
}
func main() {
a := TypeA{"I am A"}
b := TypeB{"I am B"}
doSomething(a)
// I am A
doSomething(b)
// I am B
}
Playground 2
It turns out that using JSON instead of Gob for serialization avoids the error that I was encountering entirely. Other functions can handle passing into interfaces, etc.