I am trying to parse a yaml file into go using the "gopkg.in/yaml.v3" package.
The problem I haven't been able to solve is parsing a file starting with a -. For example:
---
- type: people
info:
- { name: John, last: Doe }
...
So when I try to parse this
package main
import (
"fmt"
"io/ioutil"
"log"
"gopkg.in/yaml.v3"
)
type YamlFile struct {
type string `yaml:"type"`
}
func main() {
d := YamlFile{}
src, err := ioutil.ReadFile("test1.yaml")
if err != nil {
log.Println(err)
}
err = yaml.Unmarshal(src, &d)
if err != nil {
log.Printf("error: %v", err)
}
fmt.Println(d)
}
output: error: yaml: unmarshal errors: line 2: cannot unmarshal !!seq into main.YamlFile
The above code works when the - is removed from the file
---
type: people
info:
- { name: John, last: Doe }
...
However I cannot reformat the file so I need to know what I am doing wrong trying to parse with the - in place. Thanks for any pointers in the right direction.
The - indicates that it's a list/array. Therefore you must unmarshal into a slice or array in Go.
Change d := YamlFile{} to d := []YamlFile{}, and you will no longer get that error.
But also note that you'll always get an empty result with the struct you've defined, because it has no exported fields.
Try instead:
type YamlFile struct {
Type string `yaml:"type"`
}
See it on the playground.
Related
I have next yaml, if I validate it in online yaml website, it said it's valid:
- {"foo": "1", "bar": "2"}
Then, I write a code to parse the value 1 and 2 from this yaml as next:
test.go:
package main
import "gopkg.in/yaml.v2"
import "fmt"
type Config struct {
Foo string
Bar string
}
type Configs struct {
Cfgs []Config `foobar`
}
func main() {
//var data = `
// foobar:
// - foo: 1
// bar: 2
//`
var data = `
- foo: 1
bar: 2
`
source := []byte("foobar:" + data)
var configs Configs
err := yaml.Unmarshal(source, &configs)
if err != nil {
fmt.Println("error: %v", err)
}
fmt.Printf("--- configs:\n%v\n\n", configs)
fmt.Println(configs.Cfgs[0].Foo)
fmt.Println(configs.Cfgs[0].Bar)
}
It works ok as next:
shubuntu1#shubuntu1:~/20210810$ go run test.go
--- configs:
{[{1 2}]}
1
2
What's the problem?
You could see I made a workaround here as next to add special foobar key before the original yaml, then I could use type Configs struct to unmarshal it:
From
- foo: 1
bar: 2
to
foobar:
- foo: 1
bar: 2
So, if I don't use the workaround to add a prefix foobar:, how could I directly parse - {"foo": 1, "bar": 2}?
Since your YAML data is an array, unmarshal it to an array of Config structure.
package main
import (
"fmt"
"gopkg.in/yaml.v2"
)
type Config struct {
Foo string
Bar string
}
func main() {
var configs []Config
var data = `
- foo: 1
bar: 2
`
err := yaml.Unmarshal([]byte(data), &configs)
if err != nil {
panic(err)
}
fmt.Println(configs)
}
Output:
[{1 2}]
Try on - Go Playground
Since the question has already been answered, I thought I might add something to the discussion:
var data = `
- foo: 1
bar: 2
`
The data variable here, the way you wrote it, will include the indentation at the start of each line.
The extra indentation here WILL be passed into yaml.Unmarshal() alongside all the actual yaml data, which can mess things up since gofmt formats your code to use TAB as indentation and TAB indentation is forbidden in yaml (https://stackoverflow.com/a/19976827/7509248).
If you use tab as indentation in yaml, an error like this will be thrown when trying to unmarshal it:
yaml: line 1: found a tab character that violates indentation
Preferably, you want to load data from a separate file to avoid such issue.
package main
import (
"fmt"
"io/ioutil"
"gopkg.in/yaml.v2"
)
type Config struct {
Foo string
Bar string
}
func main() {
var configs []Config
source, err := ioutil.ReadFile("config.yaml")
if err != nil {
fmt.Printf("failed reading config file: %v\n", err)
}
err = yaml.Unmarshal(source, &configs)
if err != nil {
fmt.Printf("error: %v\n", err)
}
fmt.Printf("config:\n%+v\n", configs)
}
config.yaml:
- foo: 1
bar: 2
output:
config:
[{Foo:1 Bar:2}]
I'm getting time to learn Go, and I'm having a problem dealing with yaml file.
this is my yaml file
---
endpoints:
service1:
url: "https://service1.com"
frequency: 2
interval: 1
service2:
url: "https://service2.com"
frequency: 3
interval: 2
My go code
package main
import (
"fmt"
"io/ioutil"
"reflect"
"gopkg.in/yaml.v3"
)
// Config define estrutura do arquivo de configuração
type Config struct {
Endpoint map[string]interface{} `yaml:"endpoints"`
}
func main() {
yamlFile, err := ioutil.ReadFile("config.yml")
if err != nil {
fmt.Printf("Error reading YAML file: %s\n", err)
return
}
var yamlConfig Config
err = yaml.Unmarshal(yamlFile, &yamlConfig)
if err != nil {
fmt.Printf("Error parsing YAML file: %s\n", err)
}
for k := range yamlConfig.Endpoint {
nm := reflect.ValueOf(yamlConfig.Endpoint[k])
for _, key := range nm.MapKeys() {
strct := nm.MapIndex(key)
fmt.Println(key.Interface(), strct.Interface())
}
}
}
// PingEndpoint acessa os endpoint informados
func PingEndpoint(url string, frequency, interval int) {
// do something
}
Is there a better way to define the Config structure without using interface? And is really necessary use reflect do get properties of a service1 or exist a better whey to do that?
In general, an interface{} is used in this situation if you don't know the structure. In this case, the structure appears to be fixed:
type Service struct {
URL string `yaml:"url"`
Frequency int `yaml:"frequency"`
Interval int `yaml:"interval"`
}
type Config struct {
Endpoint map[string]Service `yaml:"endpoints"`
}
For your second question, you no longer need to deal with unknown fields after you do this, but even if you had interface{}, you can use type assertions (type yaml library unmarshals yaml into a map[interface{}]interface{}):
for k := range yamlConfig.Endpoint {
if mp, ok:=yamlConfig.Endpoint[k].(map[interface{}]interface{}); ok {
for key, value:=range mp {
}
}
}
How do I write the code below to get a string from my nested yaml struct?
Here is my yaml:
element:
- one:
url: http://test
nested: 123
- two:
url: http://test
nested: 123
weather:
- test:
zipcode: 12345
- ca:
zipcode: 90210
Here is example code
viper.SetConfigName("main_config")
viper.AddConfigPath(".")
err := viper.ReadInConfig()
if err != nil {
panic(err)
}
testvar := viper.GetString("element.one.url")
My problem:
I get a blank string when I print this. According to the docs, this is how you get a nested element. I suspect its not working because the elements are lists. Do I need to do a struct? I am not sure how to make one, especially if it needs to be nested.
You can unmarshal a nested configuration file.
main.go
package main
import (
"fmt"
"github.com/spf13/viper"
)
type NestedURL struct {
URL string `mapstructure:"url"`
Nested int `mapstructure:"nested"`
}
type ZipCode struct {
Zipcode string `mapstructure:"zipcode"`
}
type Config struct {
Element [] map[string]NestedURL `mapstructure:"element"`
Weather [] map[string]ZipCode `mapstructure:"weather"`
}
func main() {
viper.SetConfigName("config")
viper.AddConfigPath(".")
if err := viper.ReadInConfig(); err != nil {
return
}
var config Config
if err := viper.Unmarshal(&config); err != nil {
fmt.Println(err)
return
}
fmt.Println(config)
}
config.yml
element:
- one:
url: http://test
nested: 123
- two:
url: http://test
nested: 123
weather:
- test:
zipcode: 12345
- ca:
zipcode: 90210
There are different Get methods available in viper library and your YML structure is of type []map[string]string, so to parse your YML configuration file you have to use viper.Get method. So you have to parse your file something like this..
Note: You can use struct as well to un-marshal the data. Please refer this post mapping-nested-config-yaml-to-struct
package main
import (
"fmt"
"github.com/spf13/viper"
)
func main() {
viper.SetConfigName("config")
viper.AddConfigPath(".")
err := viper.ReadInConfig()
if err != nil {
panic(err)
}
testvar := viper.Get("element")
fmt.Println(testvar)
elementsMap := testvar.([]interface{})
for k, vmap := range elementsMap {
fmt.Print("Key: ", k)
fmt.Println(" Value: ", vmap)
eachElementsMap := vmap.(map[interface{}]interface{})
for k, vEachValMap := range eachElementsMap {
fmt.Printf("%v: %v \n", k, vEachValMap)
vEachValDataMap := vEachValMap.(map[interface{}]interface{})
for k, v := range vEachValDataMap {
fmt.Printf("%v: %v \n", k, v)
}
}
}
}
// Output:
/*
Key: 0 Value: map[one:map[url:http://test nested:123]]
one: map[url:http://test nested:123]
url: http://test
nested: 123
Key: 1 Value: map[two:map[url:http://test nested:123]]
two: map[url:http://test nested:123]
url: http://test
nested: 123
*/
You can use Unmarshal or UnmarshalKey to parse all or part of your data and fill a struct. It is very similar to unmarshaling a json.
In your case, code will be like this:
package main
import (
"fmt"
"github.com/spf13/viper"
)
// here we define schema of data, just like what we might do when we parse json
type Element struct {
Url string `mapstructure:"url"`
Nested int `mapstructure:"nested"`
}
func main() {
viper.SetConfigName("config")
viper.AddConfigPath(".")
err := viper.ReadInConfig()
if err != nil {
panic(err)
}
// data in `element` key is a map of string to Element. We define a variable to store data into it.
elementParsed := make(map[string]*Element)
// read the key `element` in the yaml file, and parse it's data and put it in `elementParsed` variable
err = viper.UnmarshalKey("element", &elementParsed)
if err != nil {
panic(err)
}
fmt.Println(elementParsed["one"].Url) // will print: http://test
fmt.Println(elementParsed["one"].Nested) // will print: 123
}
I am currently unable to unmarshall the data correctly from a map into a structure. Following is a code snippet (Brief Code at playground):
Request you to kindly provide the reason of getting default values on unmarshlling back the data.
package main
import (
"fmt"
"encoding/json"
"os"
)
func main() {
fmt.Println("Hello, playground")
type PDPOffer struct {
cart_value int `json:"cart_value"`
discount_amount_default int `json:"discount_amount_default"`
max_discount string `json:"max_discount"`
}
a:= map[string]interface{} {
"cart_value" : 1,
"max_discount" : 2,
}
var pdf PDPOffer
b, err := json.Marshal(a)
if err != nil {
fmt.Println("error:", err)
}
os.Stdout.Write(b)//working
err1 := json.Unmarshal(b, &pdf)
if err1 != nil {
fmt.Println("error:", err)
}
fmt.Printf("%+v", pdf)//displaying just the defualt values????????
}
json.Marshal and json.Unmarshal can only work on exported struct fields. Your fields aren't exported and aren't visible to the json code.
The reason for your unmarshal not working is that you need to expose fields of the struct and for doing that you need to start field name with Capital letters. Something has below:
type PDPOffer struct {
Cart_value int `json:"cart_value"`
Discount_amount_default int `json:"discount_amount_default"`
Max_discount string `json:"max_discount"`
}
Also, you were trying to marshal an int value into a string for "max_discount", you need to store it as a string in your map that you're marshaling:
a := map[string]interface{}{
"cart_value": 1,
"max_discount": "2",
}
The error handling had a bug checking for err1 != nil then printing err which was hiding the message error: json: cannot unmarshal number into Go value of type string
Working example with all the fixes: http://play.golang.org/p/L8VC-531nS
I am fairly new to Golang, please excuse my newbyness.
I am currently using yaml.v2 package (https://github.com/go-yaml/yaml) to unmarshal YAML data into structs.
Consider the following example code:
package main
import (
"fmt"
"gopkg.in/yaml.v2"
"log"
)
type Container struct {
First string
Second struct {
Nested1 string
Nested2 string
Nested3 string
Nested4 int
}
}
var data = `
first: first value
second:
nested1: GET
nested2: /bin/bash
nested3: /usr/local/bin/customscript
nested4: 8080
first: second value
second:
nested1: POST
nested2: /bin/ksh
nested3: /usr/local/bin/customscript2
nested4: 8081
`
func main() {
container := Container{}
err := yaml.Unmarshal([]byte(data), &container)
if err != nil {
log.Fatalf("error: %v", err)
}
fmt.Printf("---values found:\n%+v\n\n", container)
}
The result:
---values found: {First:second value Second:{Nested1:POST Nested2:/bin/ksh Nested3:/usr/local/bin/customscript2 Nested4:8081}}
This is as expected, the unmarshal function finds one occurrence of the YAML data.
What I would like to do is write a simple while/each/for loop that loops through the data variable and marshals all the occurrences into seperate Container structs. How could I achieve this?
A simple change to accomplish what you want is to have the data in the yaml be items in an array, then Unmarshal into a slice of Container
var data = `
- first: first value
second:
nested1: GET
nested2: /bin/bash
nested3: /usr/local/bin/customscript
nested4: 8080
- first: second value
second:
nested1: POST
nested2: /bin/ksh
nested3: /usr/local/bin/customscript2
nested4: 8081
`
func main() {
container := []Container{}
err := yaml.Unmarshal([]byte(data), &container)
if err != nil {
log.Fatalf("error: %v", err)
}
fmt.Printf("---values found:\n%+v\n\n", container)
}
---values found:
[{First:first value Second:{Nested1:GET Nested2:/bin/bash Nested3:/usr/local/bin/customscript Nested4:8080}} {First:second value Second:{Nested1:POST Nested2:/bin/ksh Nested3:/usr/local/bin/customscript2 Nested4:8081}}]