Why is the unmarshaled YAML data empty? - go

I have a problem unmarshaling a simple slice of YAML data:
package main
import (
"fmt"
"gopkg.in/yaml.v2"
)
type myDataStruct struct {
HTTP []struct {
Name string
Target string
}
}
func main() {
yamlData := `
HTTP:
- name: one
target: http://wazaa
- name: two
target: http://wazii
`
var myData myDataStruct
err := yaml.Unmarshal([]byte(yamlData), &myData)
if err != nil {
fmt.Printf("error: %v", err)
return
}
fmt.Print(myData)
}
Playground: https://play.golang.org/p/Srb2DJVVZqN
The result is {[]} and my limited knowledge of Go does not help to understand why?

If you don't specify the mapping between labels used in the YAML source and Go struct fields, by default they will only be matched if only changing the first letter to lower matches.
E.g. the struct field Name will match name, but not NAME.
Specify the mapping for the HTTP field:
type myDataStruct struct {
HTTP []struct {
Name string
Target string
} `yaml:"HTTP"`
}
With this change it works and outputs (try it on the Go Playground):
{[{one http://wazaa} {two http://wazii}]}
It's good practice to provide mappings for all fields, so it'll continue to work if you rename the fields:
type myDataStruct struct {
HTTP []struct {
Name string `yaml:"name"`
Target string `yaml:"target"`
} `yaml:"HTTP"`
}
Although in your case the default matching works for the Name and Target fields without providing the mappings.

Related

How do I properly parse a yaml file that can't Marshall to []string?

So I'm looking to parse a yaml file for my job, and being fairly new to Golang I am a bit stuck... Im used to being on python all the time so some concepts are still dragging in my head for Go. But I have the following issue ...
So I have this yaml file I am trying to parse and Im running the following ...
I created the go struct using yaml-to-go. So I have this ..
type Application struct {
Environments []struct {
Name string `yaml:"name"`
RulesEngineTargets []string `yaml:"rulesEngineTargets,omitempty"`
Zones []struct {
Name string `yaml:"name"`
CloudURI string `yaml:"cloudURI"`
Deployments []struct {
Name string `yaml:"name"`
VipTemplate string `yaml:"vipTemplate"`
} `yaml:"deployments"`
} `yaml:"zones"`
} `yaml:"environments"`
Apps []struct {
Name string `yaml:"name"`
Path string `yaml:"path"`
Route string `yaml:"route"`
NumDeploys int `yaml:"numDeploys"`
EcpVersion string `yaml:"ecpVersion,omitempty"`
ExcludeZones []string `yaml:"excludeZones,omitempty"`
EpaasFile string `yaml:"epaasFile,omitempty"`
Version struct {
Path string `yaml:"path"`
JSONPath string `yaml:"jsonPath"`
} `yaml:"version,omitempty"`
Service string `yaml:"service,omitempty"`
} `yaml:"apps"`
}
And I keep getting this error
line 231: cannot unmarshal !!str `m3-TEST1...` into []string
line 239: cannot unmarshal !!str `m3-TEST2...` into []string
line 247: cannot unmarshal !!str `m3-TEST2...` into []string
line 260: cannot unmarshal !!str `m3-TEST1...` into []string
The error seems to be coming from the excludedZones field in the yaml ..
rulesEngineTargets: [m2-rules-eng.test.com, m4-rules-eng.test2.test.com]
zones:
- name: m3-TEST1-zone1
cloudURI: m3-Test1.com
deployments:
- name: m3-TEST2-zone1-green
vipTemplate: "{{.route}}-green1.test.com"
- name: m3-TEST1-zone1-blue
vipTemplate: "{{.route}}-blue1.test.com"
What am I missing in my code to make this work right? Im fairly lost, is it something wrong with the parsing on the '-' character?
Here is my full code
package main
import (
"fmt"
"gopkg.in/yaml.v3"
"log"
"os"
)
type Application struct {
Environments []struct {
Name string `yaml:"name"`
RulesEngineTargets []string `yaml:"rulesEngineTargets,omitempty"`
Zones []struct {
Name string `yaml:"name"`
CloudURI string `yaml:"cloudURI"`
Deployments []struct {
Name string `yaml:"name"`
VipTemplate string `yaml:"vipTemplate"`
} `yaml:"deployments"`
} `yaml:"zones"`
} `yaml:"environments"`
Apps []struct {
Name string `yaml:"name"`
Path string `yaml:"path"`
Route string `yaml:"route"`
NumDeploys int `yaml:"numDeploys"`
EcpVersion string `yaml:"ecpVersion,omitempty"`
ExcludeZones []string `yaml:"excludeZones,omitempty"`
EpaasFile string `yaml:"epaasFile,omitempty"`
Version struct {
Path string `yaml:"path"`
JSONPath string `yaml:"jsonPath"`
} `yaml:"version,omitempty"`
Service string `yaml:"service,omitempty"`
} `yaml:"apps"`
}
func main() {
file, err := os.ReadFile("r42.yaml")
if err != nil {
log.Println("action failed: ", err)
return
}
var apps Application
err = yaml.Unmarshal(file, &apps)
if err != nil {
log.Println("action failed: ", err)
return
}
fmt.Printf("Here are the Results: %+v\\n", apps)
//for i := 0; i < len(apps.Apps); i++ {
// fmt.Println(apps.Apps[i].Name)
// fmt.Println(apps.Apps[i].Route)
}

Merge structs with some overlapping fields

I saw several questions asking on how to merge unique structs and how to merge identical structs.
But how would I merge structs that have some overlap? and which fields get taken & when?
e.g.:
type structOne struct {
id string `json:id`
date string `json:date`
desc string `json:desc`
}
and
type structTwo struct {
id string `json:id`
date string `json:date`
name string `json:name`
}
how would I merge it such that I get
{
id string `json:id`
date string `json:date`
desc string `json:desc`
name string `json:name`
}
also, what happens if in this case the two id's are the same (assuming a join over id's) but the names are different?
In javascript, doing something like Object.assign(structOne, structTwo).
Go is a strongly typed language, unlike javascript you can't merge two struct into one combined struct because all type are determined at compile-time. You have two solution here :
Using embedded struct:
One great solution is to use embedded struct because you don't have to merge anything anymore.
package main
import (
"bytes"
"encoding/json"
"fmt"
"log"
)
// Shared field
type common struct {
ID string `json:id`
Date string `json:date`
}
type merged struct {
// Common field is embedded
common
Name string `json:name`
Desc string `json:desc`
}
func main() {
buf := bytes.Buffer{}
buf.WriteString("{ \"id\": \"1\", \"date\": \"27/07/2020\", \"desc\": \"the decription...\" }")
merged := &merged{}
err := json.Unmarshal(buf.Bytes(), merged)
if err != nil {
log.Fatal(err)
}
// Look how you can easily access field from
// embedded struct
fmt.Println("ID:", merged.ID)
fmt.Println("Date:", merged.Date)
fmt.Println("Name:", merged.Name)
fmt.Println("Desc:", merged.Desc)
// Output:
// ID: 1
// Date: 27/07/2020
// Name:
// Desc: the decription...
}
If you want to read more about struct embedding:
golangbyexample.com
travix.io
Using Maps
Another solution is to use maps but you will loose the benefits of struct and methods. This example is not the simplest but there is some great example in the other responses.
In this example I'm using Mergo. Mergo is library that can merge structs and map. Here it is used for creating maps object in the Map methods but you can totally write your own methods.
package main
import (
"fmt"
"log"
"github.com/imdario/mergo"
)
type tOne struct {
ID string
Date string
Desc string
}
// Map build a map object from the struct tOne
func (t1 tOne) Map() map[string]interface{} {
m := make(map[string]interface{}, 3)
if err := mergo.Map(&m, t1); err != nil {
log.Fatal(err)
}
return m
}
type tTwo struct {
ID string
Date string
Name string
}
// Map build a map object from the struct tTwo
func (t2 tTwo) Map() map[string]interface{} {
m := make(map[string]interface{}, 3)
if err := mergo.Map(&m, t2); err != nil {
log.Fatal(err)
}
return m
}
func main() {
dst := tOne{
ID: "destination",
Date: "26/07/2020",
Desc: "destination object",
}.Map()
src := tTwo{
ID: "src",
Date: "26/07/1010",
Name: "source name",
}.Map()
if err := mergo.Merge(&dst, src); err != nil {
log.Fatal(err)
}
fmt.Printf("Destination:\n%+v", dst)
// Output:
// Destination:
// map[date:26/07/2020 desc:destination object iD:destination name:object name
}
Go structs and JavaScript objects are very different. Go structs do not have dynamic fields.
If you want dynamic key/value sets that you can easily iterate over and merge, and are very JSON friendly, why not a map[string]interface{}?
$ go run t.go
map[a:1 b:4]
map[a:1 b:4 c:3]
$ cat t.go
package main
import(
"fmt"
)
type MyObj map[string]interface{}
func (mo MyObj)Merge(omo MyObj){
for k, v := range omo {
mo[k] = v
}
}
func main() {
a := MyObj{"a": 1, "b": 4}
b := MyObj{"b": 2, "c": 3}
b.Merge(a)
fmt.Printf("%+v\n%+v\n", a, b)
}
you can use github.com/fatih/structs to convert your struct to map. Then iterate over that map and choose which fields need to copy over. I have a snippet of code which illustrates this solution.
func MergeStruct (a structOne,b structTwo) map[string]interface{}{
a1:=structs.Map(a)
b1:=structs.Map(b)
/* values of structTwo over writes values of structOne */
var myMap=make(map[string]interface{})
for val,key:=range(a1){
myMap[key]=val
}
for val,key:=range(b1){
myMap[key]=val
}
return myMap
}
You can use the reflect package to do this. Try iterating through the two structs and then you can either use another struct type to store the values or maybe use a map.
Check out this question to find out how you can iterate over a struct.
Check out this question to find out how to get the name of the fields.
Remember to use exported field names for the reflect package to work.
Here is an example which works.

Parsing JSON using struct

I'm trying to parse JSON using Go. Can anyone tell me why my code is not working as expected?
package main
import (
"encoding/json"
"fmt"
)
type Message struct {
Name string
Body string
Time int64
}
type Person struct {
M Message
}
func get_content() {
body := []byte(`{"person":{"Name":"Alice","Body":"Hello","Time":1294706395881547000}}`)
var data Person
err := json.Unmarshal(body, &data)
if err != nil {
panic(err.Error())
}
fmt.Printf("%v",data.M.Name)
}
func main() {
get_content()
}
I'm expecting it to print the Name.
Go playground Code
There are two problems in the code.
The first one is what #umar-hayat mentioned above -> you are unmarshalling into the data object and you should be aiming at the data.M field.
The second problem is that your JSON's structure doesn't match your struct's structure. Your Person has a single field called M. If we want to represent this as JSON it would look like this:
{
"M": {
"Name": "Joe",
"Body": "Hi",
"time": 2600
}
}
Instead, you have a field called person in your JSON which cannot be matched to any field in your struct. The fact that it's similar to the name of the struct's type doesn't help in any way, I'm afraid.
So, you can either change your JSON and your target:
body := []byte(`{"Name":"Alice","Body":"Hello","Time":1294706395881547000}`)
var data Person
err := json.Unmarshal(body, &data.M)
Or just your JSON:
body := []byte(`{"M":{"Name":"Alice","Body":"Hello","Time":1294706395881547000}}`)
var data Person
err := json.Unmarshal(body, &data)
But it's essential that the names of the fields in your JSON match the names of the fields in your struct. Or, as mentioned by Konstantinos, you can use tags in order to specify particular names with which your struct's fields will be represented in the JSON.
You might find this helpful: https://gobyexample.com/json
Here is how to Unmarshel JSON to the struct. you can check it on Go Playground here:
package main
import (
"encoding/json"
"fmt"
)
type Message struct {
Name string
Body string
Time int64
}
type Person struct {
M Message
}
func get_content() {
body := []byte(`{"Name":"Alice","Body":"Hello","Time":1294706395881547000}`)
var data Person
err := json.Unmarshal(body, &data.M)
if err != nil {
panic(err.Error())
}
fmt.Printf(data.M.Name)
}
func main() {
get_content()
}
Replace data with data.M in below line.
err := json.Unmarshal(body, &data)
As long as you intent to map Json keys on structs whose fields have different names you should add tags:
type Message struct {
Name string `json:"Name"`
Body string `json:"Body"`
Time int64 `json:"Time"`
}
type Person struct {
M Message `json:"person"`
}
You can find more information here
In addition this answer explains in an nutshell the purpose of tags in go.

Unmarshal a YAML to a struct with unexpected fields in Go

I encountered an issue while trying to unmarshal a struct with an unexported field using github.com/go-yaml/yaml. The struct looks like:
type Example struct {
ExportedField string `yaml:"exported-field"`
OneMoreExported string `yaml:"one-more-exported"`
unexportedField map[string]*AnotherExample `yaml:"unexported-field"`
}
type AnotherExample struct {
Name string `yaml:"name"`
}
And I'd like to unmarshal such YAML as
exported-field: lorem ipsum
one-more-exported: dolor set
unexported-field:
something:
name: anything
What I tried is a custom unmarshaler:
func (example *Example) UnmarshalYAML(unmarshal func(interface{}) error) error {
type Alias Example
tmp := struct {
UnexportedField map[string]*AnotherExample `yaml:"unexported-field"`
*Alias
}{
Alias: (*Alias)(example),
}
if err := unmarshal(&tmp); err != nil {
return err
}
if tmp.UnexportedField != nil {
example.unexportedField = tmp.UnexportedField
}
example.CopyJobNonNil(Example(*tmp.Alias)) // Copies all the non-nil fields from the passed Example instance
return nil
}
tmp after calling unmarshal() doesn't contain any fields but unexportedField — other fields seems to be omitted.
The reproduced issue on Go Playground (although it isn't working due to the dependencies): https://play.golang.org/p/XZg7tEPGXna
Because most Go unmarshalling packages (including the encoding/* packages) use the reflect package to get at struct fields, and reflect can't access unexported struct fields, the unmarshaler can't parse into unexported fields.
That said, there is still a way to do it. You can unmarshall the YAML into an an unexported type with public fields, which can then be embedded into an exported type. Getters and setters in the same package can just use the embedded struct.
For example:
// Define the types...
type innerData struct {
ExportedField string
unexportedField string
}
type Data struct {
innerData
}
// and then...
d := Data{}
DoSomeUnmarshalling(yamlbytes, &d.innerData)
// and then you can use getters/setters/whatever on `Data`
func (d *Data) GetUnexported() string {
return d.innerData.unexportedField;
}
(warning: completely untested)
See JSON and dealing with unexported fields for reference.

Copy one struct to another where structs have same members and different types

I have two struct having the same members, I want to copy one struct to another, see the pseudo code below:
type Common struct {
Gender int
From string
To string
}
type Foo struct {
Id string
Name string
Extra Common
}
type Bar struct {
Id string
Name string
Extra Common
}
Then I have foo of struct Foo, and bar of struct Bar, Is there any way to copy bar from foo?
Use a conversion to change the type. The following code uses a conversion to copy a value of type Foo to a value of type Bar:
foo := Foo{Id: "123", Name: "Joe"}
bar := Bar(foo)
playground example
The conversion only works when the underlying types are identical except for struct tags.
https://github.com/jinzhu/copier (same author of gorm) is also quite a good one, I have nested structs and all I do is:
copier.Copy(&employees, &user)
works great
If you would like to copy or clone to a different struct, I would suggest using deepcopier
It provides nice features like skipping, custom mapping, and forcing. below is an example from github:
Install:
go get -u github.com/ulule/deepcopier
Example:
package main
import (
"fmt"
"github.com/ulule/deepcopier"
)
// Model
type User struct {
// Basic string field
Name string
// Deepcopier supports https://golang.org/pkg/database/sql/driver/#Valuer
Email sql.NullString
}
func (u *User) MethodThatTakesContext(ctx map[string]interface{}) string {
// do whatever you want
return "hello from this method"
}
// Resource
type UserResource struct {
//copy from field "Name"
DisplayName string `deepcopier:"field:Name"`
//this will be skipped in copy
SkipMe string `deepcopier:"skip"`
//this should call method named MethodThatTakesContext
MethodThatTakesContext string `deepcopier:"context"`
Email string `deepcopier:"force"`
}
func main() {
user := &User{
Name: "gilles",
Email: sql.NullString{
Valid: true,
String: "gilles#example.com",
},
}
resource := &UserResource{}
deepcopier.Copy(user).To(resource)
//copied from User's Name field
fmt.Println(resource.DisplayName)//output: gilles
fmt.Println(resource.Email) //output: gilles#example.com
fmt.Println(resource.MethodThatTakesContext) //output: hello from this method
}
Also, some other way you could achieve this is by encoding the source object to JSON and then decode it back to the destination object.
This is another possible answer
type Common struct {
Gender int
From string
To string
}
type Foo struct {
Id string
Name string
Extra Common
}
type Bar struct {
Id string
Name string
Extra Common
}
foo:=Foo{
Id:"123",
Name:"damitha",
Extra: struct {
Gender int
From string
To string
}{Gender:1 , From:"xx", To:"yy" },
}
bar:=*(*Bar)(unsafe.Pointer(&foo))
fmt.Printf("%+v\n",bar)
If you would like to copy or clone to a different struct, I would suggest using deepcopier
It provides nice features like skipping, custom mapping, and forcing.
You can achieve nested struct copy following way.
Install:
go get -u github.com/ulule/deepcopier
Example:
package main
import (
"fmt"
"strconv"
"github.com/ulule/deepcopier"
)
//FieldStruct - Field Struct
type FieldStruct struct {
Name string `deepcopier:"field:TargetName"`
Type string `deepcopier:"field:TargetType"`
}
//SourceStruct - Source Struct
type SourceStruct struct {
Name string `deepcopier:"field:TargetName"`
Age int `deepcopier:"field:TargetAge"`
StringArray []string `deepcopier:"field:TargetStringArray"`
StringToInt string `deepcopier:"context"`
Field FieldStruct
Fields []FieldStruct
}
//TargetFieldStruct - Field Struct
type TargetFieldStruct struct {
TargetName string
TargetType string
}
//TargetStruct - Target Struct
type TargetStruct struct {
TargetName string
TargetAge int
TargetStringArray []string
TargetInt int
TargetField TargetFieldStruct
TargetFields []TargetFieldStruct
}
//write methods
//TargetInt - StringToInt
func (s *SourceStruct) TargetInt() int {
i, _ := strconv.Atoi(s.StringToInt)
return i
}
func main() {
s := &SourceStruct{
Name: "Name",
Age: 12,
StringArray: []string{"1", "2"},
StringToInt: "123",
Field: FieldStruct{
Name: "Field",
Type: "String",
},
Fields: []FieldStruct{
FieldStruct{
Name: "Field1",
Type: "String1",
},
FieldStruct{
Name: "Field2",
Type: "String2",
},
},
}
t := &TargetStruct{}
//coping data into inner struct
deepcopier.Copy(&t.TargetField).From(&s.Field)
// copied array of Struct
for i := range s.Fields {
// init a struct
t.TargetFields = append(t.TargetFields, TargetFieldStruct{})
// coping the data
deepcopier.Copy(&t.TargetFields[i]).From(&s.Fields[i])
}
//Top level copy
deepcopier.Copy(t).From(s)
fmt.Println(t)
}
Output:
&{Name 12 [1 2] 123 {Field String} [{Field1 String1} {Field2 String2}]}

Resources