Go Yaml Interpretation Example - go

This Go Yaml Interpretation Example,
https://gist.github.com/suntong001/74c85d15b19ef4b14b0e, can unmarshal a simple YAML like this:
foo: 1
bar:
- one
- two
Now I want to interpretation an array of the exact same data structure, what's the correct way to do?
Below is my code so far, and it is not working as expected. Please help.
package main
import (
"fmt"
"log"
"gopkg.in/yaml.v2"
)
type Config struct {
Foo string
Bar []string
}
type Configs struct {
Cfgs []Config
}
var data = `
- foo: 1
bar:
- one
- two
- three
- foo: 2
bar:
- one1
- two2
- three3
`
func main() {
var config Configs
/*
filename := os.Args[1]
source, err := ioutil.ReadFile(filename)
if err != nil {
panic(err)
}
*/
source := []byte(data)
err := yaml.Unmarshal(source, &config)
if err != nil {
log.Fatalf("error: %v", err)
}
//fmt.Printf("Value: %#v\n", config.Bar[0])
fmt.Printf("--- config:\n%v\n\n", config)
}
UPDATE:
To make it work, Jfly pointed out, simply replace my var config Configs with var config []Config, and it does the trick. Now I think if change my data definition to the following, my above code would work.
foobars:
- foo: 1
bar:
- one
- two
- three
- foo: 2
bar:
- one1
- two2
- three3
Nops, it doesn't. Please help.

The contents of the example yaml file are sequence of objects, so do it like this:
package main
import (
"fmt"
"log"
"gopkg.in/yaml.v2"
)
type Config struct {
Foo string
Bar []string
}
var data = `
- foo: 1
bar:
- one
- two
- three
- foo: 2
bar:
- one1
- two2
- three3
`
func main() {
var config []Config
/*
filename := os.Args[1]
source, err := ioutil.ReadFile(filename)
if err != nil {
panic(err)
}
*/
source := []byte(data)
err := yaml.Unmarshal(source, &config)
if err != nil {
log.Fatalf("error: %v", err)
}
//fmt.Printf("Value: %#v\n", config.Bar[0])
fmt.Printf("--- config:\n%v\n\n", config)
}
As to your updated question, your code almost works, just give a hint to the yaml parser like this:
type Configs struct {
Cfgs []Config `foobars`
}

Related

How to unmarshal a yaml array?

I have next yaml, if I validate it in online yaml website, it said it's valid:
- {"foo": "1", "bar": "2"}
Then, I write a code to parse the value 1 and 2 from this yaml as next:
test.go:
package main
import "gopkg.in/yaml.v2"
import "fmt"
type Config struct {
Foo string
Bar string
}
type Configs struct {
Cfgs []Config `foobar`
}
func main() {
//var data = `
// foobar:
// - foo: 1
// bar: 2
//`
var data = `
- foo: 1
bar: 2
`
source := []byte("foobar:" + data)
var configs Configs
err := yaml.Unmarshal(source, &configs)
if err != nil {
fmt.Println("error: %v", err)
}
fmt.Printf("--- configs:\n%v\n\n", configs)
fmt.Println(configs.Cfgs[0].Foo)
fmt.Println(configs.Cfgs[0].Bar)
}
It works ok as next:
shubuntu1#shubuntu1:~/20210810$ go run test.go
--- configs:
{[{1 2}]}
1
2
What's the problem?
You could see I made a workaround here as next to add special foobar key before the original yaml, then I could use type Configs struct to unmarshal it:
From
- foo: 1
bar: 2
to
foobar:
- foo: 1
bar: 2
So, if I don't use the workaround to add a prefix foobar:, how could I directly parse - {"foo": 1, "bar": 2}?
Since your YAML data is an array, unmarshal it to an array of Config structure.
package main
import (
"fmt"
"gopkg.in/yaml.v2"
)
type Config struct {
Foo string
Bar string
}
func main() {
var configs []Config
var data = `
- foo: 1
bar: 2
`
err := yaml.Unmarshal([]byte(data), &configs)
if err != nil {
panic(err)
}
fmt.Println(configs)
}
Output:
[{1 2}]
Try on - Go Playground
Since the question has already been answered, I thought I might add something to the discussion:
var data = `
- foo: 1
bar: 2
`
The data variable here, the way you wrote it, will include the indentation at the start of each line.
The extra indentation here WILL be passed into yaml.Unmarshal() alongside all the actual yaml data, which can mess things up since gofmt formats your code to use TAB as indentation and TAB indentation is forbidden in yaml (https://stackoverflow.com/a/19976827/7509248).
If you use tab as indentation in yaml, an error like this will be thrown when trying to unmarshal it:
yaml: line 1: found a tab character that violates indentation
Preferably, you want to load data from a separate file to avoid such issue.
package main
import (
"fmt"
"io/ioutil"
"gopkg.in/yaml.v2"
)
type Config struct {
Foo string
Bar string
}
func main() {
var configs []Config
source, err := ioutil.ReadFile("config.yaml")
if err != nil {
fmt.Printf("failed reading config file: %v\n", err)
}
err = yaml.Unmarshal(source, &configs)
if err != nil {
fmt.Printf("error: %v\n", err)
}
fmt.Printf("config:\n%+v\n", configs)
}
config.yaml:
- foo: 1
bar: 2
output:
config:
[{Foo:1 Bar:2}]

Go: How can I create an instance of type schema.ResourceData with values for unit testing?

I want to write a unit test for a method similar to this:
package bingo
import "github.com/hashicorp/terraform-plugin-sdk/helper/schema"
func doStuff(d *schema.ResourceData) error {
foo := d.Get("foo").(string)
// ... operations using d
return nil
}
Is there a way to create an instance of type schema.ResourceData with values in it?
I tried creating a "blank" instance of ResourceData and populate it via .Set(...). But it doesn't work, since no schema is present:
package bingo
import "github.com/hashicorp/terraform-plugin-sdk/helper/schema"
func TestDoStuff(t *testing.T) {
d := schema.ResourceData{}
err := d.Set("foo", "bar")
if err != nil {
t.Errorf("failed to set value: %s", err)
// > failed to set value: Invalid address to set: []string{"foo"}
}
// test doStuff()
}
schema.TestResourceDataRaw does exactly that:
package bingo
import "github.com/hashicorp/terraform-plugin-sdk/helper/schema"
var testSchema = map[string]*schema.Schema{
"foo": {Type: schema.TypeString}
}
func TestDoStuff(t *testing.T) {
d := schema.TestResourceDataRaw(t, testSchema, map[string]interface{
"foo": "bar",
})
err := doStuff(d)
if err != nil {
t.Fail()
}
}
Link to the documentation
Thanks to #mkopriva for pointing out the builtin helper method.

Golang yaml.v2 marshals an array as a sequence

Given the following YAML:
array.test: ["val1", "val2", "val3"]
I Unmarshal it using gopkg.in/yaml.v2 into a map[string]interface{}. Then I get a single key whose value is an array of 3 values.
When I then Marshal it again to YAML, the resulting YAML looks like this:
array.test:
- val1
- val2
- val3
The array was actually marshaled as a sequence instead of an array.
This is the entire GoLang code:
func main(){
data := `array.test: ["val1", "val2", "val3"]`
conf := make(map[string]interface{})
yaml.Unmarshal([]byte(data), conf)
data2, _ := yaml.Marshal(conf)
fmt.Printf("%s\n", string(data2))
}
How can I overcome this issue?
This one helped me in the same case as you.
package main
import (
"fmt"
"log"
"gopkg.in/yaml.v2"
)
var data = `
a: Easy!
b:
c: 2
d.test: ["d1", "d2"]
`
// Note: struct fields must be public in order for unmarshal to
// correctly populate the data.
type T struct {
A string
B struct {
RenamedC int `yaml:"c"`
DTest []string `yaml:"d.test,flow"`
}
}
func main() {
// if we use struct containing yaml encoding for yaml formated string
t := T{}
err := yaml.Unmarshal([]byte(data), &t)
if err != nil {
log.Fatalf("error: %v", err)
}
fmt.Printf("--- t after unmarshal:\n%v\n\n", t)
d, err := yaml.Marshal(&t)
if err != nil {
log.Fatalf("error: %v", err)
}
fmt.Printf("--- t after marshal:\n%s\n\n", string(d))
}
Ref: https://github.com/go-yaml/yaml
Use flow in struct field tag format, to indicate you desire this behavior. But, of course, this requires unmarshaling to a struct, not to a map.
Flow tag allows you to choose the representation of an array in yaml
package main
import (
"fmt"
"gopkg.in/yaml.v2"
)
type Conf struct {
Test []string `yaml:"array.test,flow"`
}
func main(){
data := `array.test: ["val1", "val2", "val3"]`
var conf Conf
yaml.Unmarshal([]byte(data), &conf)
data2, _ := yaml.Marshal(conf)
fmt.Printf("%s\n", string(data2))
}

Read multiple yamls in a file

How does one parse multiple yamls in a file similar to how kubectl does it?
example.yaml
---
a: Easy!
b:
c: 0
d: [1, 2]
---
a: Peasy!
b:
c: 1000
d: [3, 4]
There's a difference in behavior between gopkg.in/yaml.v2 and gopkg.in/yaml.v3:
V2: https://play.golang.org/p/XScWhdPHukO
V3: https://play.golang.org/p/OfFY4qH5wW2
Both implementations produce an incorrect result IMHO but V3 is apparently slightly worse.
There's a workaround. If you slightly change code in the accepted answer, it works correctly and in the same fashion with both versions of yaml package: https://play.golang.org/p/r4ogBVcRLCb
Current gopkg.in/yaml.v3 Deocder produces quite correct result, you just need to pay attention on creating new structure for each document and check it was parsed correctly (with nil check), and handle EOF error correctly:
package main
import "fmt"
import "gopkg.in/yaml.v3"
import "os"
import "errors"
import "io"
type Spec struct {
Name string `yaml:"name"`
}
func main() {
f, err := os.Open("spec.yaml")
if err != nil {
panic(err)
}
d := yaml.NewDecoder(f)
for {
// create new spec here
spec := new(Spec)
// pass a reference to spec reference
err := d.Decode(&spec)
// check it was parsed
if spec == nil {
continue
}
// break the loop in case of EOF
if errors.Is(err, io.EOF) {
break
}
if err != nil {
panic(err)
}
fmt.Printf("name is '%s'\n", spec.Name)
}
}
Test file spec.yaml:
---
name: "doc first"
---
name: "second"
---
---
name: "skip 3, now 4"
---
Solution I found using gopkg.in/yaml.v2:
package main
import (
"bytes"
"fmt"
"io/ioutil"
"log"
"path/filepath"
"gopkg.in/yaml.v2"
)
type T struct {
A string
B struct {
RenamedC int `yaml:"c"`
D []int `yaml:",flow"`
}
}
func main() {
filename, _ := filepath.Abs("./example.yaml")
yamlFile, err := ioutil.ReadFile(filename)
if err != nil {
panic(err)
}
r := bytes.NewReader(yamlFile)
dec := yaml.NewDecoder(r)
var t T
for dec.Decode(&t) == nil {
fmt.Printf("a :%v\nb :%v\n", t.A, t.B)
}
}

How do I use Viper to get a value from a nested YAML structure?

How do I write the code below to get a string from my nested yaml struct?
Here is my yaml:
element:
- one:
url: http://test
nested: 123
- two:
url: http://test
nested: 123
weather:
- test:
zipcode: 12345
- ca:
zipcode: 90210
Here is example code
viper.SetConfigName("main_config")
viper.AddConfigPath(".")
err := viper.ReadInConfig()
if err != nil {
panic(err)
}
testvar := viper.GetString("element.one.url")
My problem:
I get a blank string when I print this. According to the docs, this is how you get a nested element. I suspect its not working because the elements are lists. Do I need to do a struct? I am not sure how to make one, especially if it needs to be nested.
You can unmarshal a nested configuration file.
main.go
package main
import (
"fmt"
"github.com/spf13/viper"
)
type NestedURL struct {
URL string `mapstructure:"url"`
Nested int `mapstructure:"nested"`
}
type ZipCode struct {
Zipcode string `mapstructure:"zipcode"`
}
type Config struct {
Element [] map[string]NestedURL `mapstructure:"element"`
Weather [] map[string]ZipCode `mapstructure:"weather"`
}
func main() {
viper.SetConfigName("config")
viper.AddConfigPath(".")
if err := viper.ReadInConfig(); err != nil {
return
}
var config Config
if err := viper.Unmarshal(&config); err != nil {
fmt.Println(err)
return
}
fmt.Println(config)
}
config.yml
element:
- one:
url: http://test
nested: 123
- two:
url: http://test
nested: 123
weather:
- test:
zipcode: 12345
- ca:
zipcode: 90210
There are different Get methods available in viper library and your YML structure is of type []map[string]string, so to parse your YML configuration file you have to use viper.Get method. So you have to parse your file something like this..
Note: You can use struct as well to un-marshal the data. Please refer this post mapping-nested-config-yaml-to-struct
package main
import (
"fmt"
"github.com/spf13/viper"
)
func main() {
viper.SetConfigName("config")
viper.AddConfigPath(".")
err := viper.ReadInConfig()
if err != nil {
panic(err)
}
testvar := viper.Get("element")
fmt.Println(testvar)
elementsMap := testvar.([]interface{})
for k, vmap := range elementsMap {
fmt.Print("Key: ", k)
fmt.Println(" Value: ", vmap)
eachElementsMap := vmap.(map[interface{}]interface{})
for k, vEachValMap := range eachElementsMap {
fmt.Printf("%v: %v \n", k, vEachValMap)
vEachValDataMap := vEachValMap.(map[interface{}]interface{})
for k, v := range vEachValDataMap {
fmt.Printf("%v: %v \n", k, v)
}
}
}
}
// Output:
/*
Key: 0 Value: map[one:map[url:http://test nested:123]]
one: map[url:http://test nested:123]
url: http://test
nested: 123
Key: 1 Value: map[two:map[url:http://test nested:123]]
two: map[url:http://test nested:123]
url: http://test
nested: 123
*/
You can use Unmarshal or UnmarshalKey to parse all or part of your data and fill a struct. It is very similar to unmarshaling a json.
In your case, code will be like this:
package main
import (
"fmt"
"github.com/spf13/viper"
)
// here we define schema of data, just like what we might do when we parse json
type Element struct {
Url string `mapstructure:"url"`
Nested int `mapstructure:"nested"`
}
func main() {
viper.SetConfigName("config")
viper.AddConfigPath(".")
err := viper.ReadInConfig()
if err != nil {
panic(err)
}
// data in `element` key is a map of string to Element. We define a variable to store data into it.
elementParsed := make(map[string]*Element)
// read the key `element` in the yaml file, and parse it's data and put it in `elementParsed` variable
err = viper.UnmarshalKey("element", &elementParsed)
if err != nil {
panic(err)
}
fmt.Println(elementParsed["one"].Url) // will print: http://test
fmt.Println(elementParsed["one"].Nested) // will print: 123
}

Resources