How to unmarshal YAML in Go - go

I have a hard time to unmarshal this piece of YAML in Go. The error I'm getting is cannot unmarshal !!seq into map[string][]map[string][]string. I've tried all kind of maps with no success (map[string]string ; []map[string]string and so on)
import (
"gopkg.in/yaml.v1"
"io/ioutil"
)
type AppYAML struct {
Runtime string `yaml:"runtime,omitempty"`
Handlers map[string][]map[string][]string `yaml:"handlers,omitempty"`
Env_Variables map[string]string `yaml:"env_variables,omitempty"`
}
func main() {
s := `
runtime: go
handlers:
- url: /.*
runtime: _go_app
secure: always
env_variables:
something: 'test'
`
var a AppYAML
if err = yaml.Unmarshal([]byte(s), &a); err != nil {
log.Error(err)
return
}
}

Change type declaration to this:
type AppYAML struct {
Runtime string `yaml:"runtime,omitempty"`
Handlers []map[string]string `yaml:"handlers,omitempty"`
Env_Variables map[string]string `yaml:"env_variables,omitempty"`
}

Related

Unmarshal yaml map dict key to struct property

I really searched a while here, but didn't found an adequate answer:
I am trying to unmarshall yaml dict keys onto a property of a struct rather than the key of a map.
Given this yaml
commands:
php:
service: php
bin: /bin/php
node:
service: node
bin: /bin/node
I am able to unmarshall this into a struct like this:
type Config struct {
Commands map[string]struct {
Service string
Bin string
}
}
But how am I able to unmarshall it into a struct like this:
type Config struct {
Commands []struct {
Name string // <-- this should be key from the yaml (i.e. php or node)
Service string
Bin string
}
}
Thx in advance for the help
You can write a custom unmarshaler, like this (on Go playground):
package main
import (
"fmt"
"gopkg.in/yaml.v3"
)
var input []byte = []byte(`
commands:
php:
service: php
bin: /bin/php
node:
service: node
bin: /bin/node
`)
type Command struct {
Service string
Bin string
}
type NamedCommand struct {
Command
Name string
}
type NamedCommands []NamedCommand
type Config struct {
Commands NamedCommands
}
func (p *NamedCommands) UnmarshalYAML(value *yaml.Node) error {
if value.Kind != yaml.MappingNode {
return fmt.Errorf("`commands` must contain YAML mapping, has %v", value.Kind)
}
*p = make([]NamedCommand, len(value.Content)/2)
for i := 0; i < len(value.Content); i += 2 {
var res = &(*p)[i/2]
if err := value.Content[i].Decode(&res.Name); err != nil {
return err
}
if err := value.Content[i+1].Decode(&res.Command); err != nil {
return err
}
}
return nil
}
func main() {
var f Config
var err error
if err = yaml.Unmarshal(input, &f); err != nil {
panic(err)
}
for _, cmd := range f.Commands {
fmt.Printf("%+v\n", cmd)
}
}
I have split the command data into Command and NamedCommand to make the code simpler, since you can just call Decode giving the embedded Command struct for the values. If everything was in the same struct, you'd need to manually map keys to struct fields.

Getting value from yaml object in file

I'm getting time to learn Go, and I'm having a problem dealing with yaml file.
this is my yaml file
---
endpoints:
service1:
url: "https://service1.com"
frequency: 2
interval: 1
service2:
url: "https://service2.com"
frequency: 3
interval: 2
My go code
package main
import (
"fmt"
"io/ioutil"
"reflect"
"gopkg.in/yaml.v3"
)
// Config define estrutura do arquivo de configuração
type Config struct {
Endpoint map[string]interface{} `yaml:"endpoints"`
}
func main() {
yamlFile, err := ioutil.ReadFile("config.yml")
if err != nil {
fmt.Printf("Error reading YAML file: %s\n", err)
return
}
var yamlConfig Config
err = yaml.Unmarshal(yamlFile, &yamlConfig)
if err != nil {
fmt.Printf("Error parsing YAML file: %s\n", err)
}
for k := range yamlConfig.Endpoint {
nm := reflect.ValueOf(yamlConfig.Endpoint[k])
for _, key := range nm.MapKeys() {
strct := nm.MapIndex(key)
fmt.Println(key.Interface(), strct.Interface())
}
}
}
// PingEndpoint acessa os endpoint informados
func PingEndpoint(url string, frequency, interval int) {
// do something
}
Is there a better way to define the Config structure without using interface? And is really necessary use reflect do get properties of a service1 or exist a better whey to do that?
In general, an interface{} is used in this situation if you don't know the structure. In this case, the structure appears to be fixed:
type Service struct {
URL string `yaml:"url"`
Frequency int `yaml:"frequency"`
Interval int `yaml:"interval"`
}
type Config struct {
Endpoint map[string]Service `yaml:"endpoints"`
}
For your second question, you no longer need to deal with unknown fields after you do this, but even if you had interface{}, you can use type assertions (type yaml library unmarshals yaml into a map[interface{}]interface{}):
for k := range yamlConfig.Endpoint {
if mp, ok:=yamlConfig.Endpoint[k].(map[interface{}]interface{}); ok {
for key, value:=range mp {
}
}
}

Unable to unmarshal array field in golang

I am trying to unmasrhal the following flux HelmRelease file.
apiVersion: helm.fluxcd.io/v1
kind: HelmRelease
metadata:
annotations:
fluxcd.io/automated: 'false'
fluxcd.io/tag.ats: glob:*
name: ats
namespace: myns
spec:
chart:
git: git#github.com:reponame/project.git
path: charts/path1/path1/myapp
ref: master
releaseName: foobar
values:
allowAllEgress: true
recycleApp: true
hooks:
slackChannel: https://hooks.slack.com/services/something/somethingelse/
Here are my models
type HelmReleaseValues struct {
AllowAllEgress bool `yaml:"allowAllEgress"`
RecycleApp bool `yaml:"recycleApp"`
Hooks `yaml:"hooks"`
}
type Hooks struct {
SlackChannel string `yaml:"slackChannel"`
}
type Values struct {
HelmReleaseValues `yaml:"values"`
ReleaseName string `yaml:"releaseName"`
Chart `yaml:"chart"`
}
type Spec struct {
Values `yaml:"spec"`
}
The problem is that the fields allowAllEgress and recycleApp are getting unmarshalled.
However the Hooks field in my struct turns out to be empty.
What am I doing wrong in the struct modelling / tagging?
edit: here is my code
package main
import (
"fmt"
"io/ioutil"
"os"
"github.com/davecgh/go-spew/spew"
"gopkg.in/yaml.v3"
)
const ExitCodeCmdErr = 1
func main() {
rawYaml := parseHelmReleaseFile("myfile.yaml")
spew.Dump(rawYaml)
}
func parseHelmReleaseFile(fileName string) Spec {
var v Spec
yamlFile, err := ioutil.ReadFile(fileName)
if err != nil {
fmt.Printf("yaml file err #%v ", err)
os.Exit(ExitCodeCmdErr)
}
err = yaml.Unmarshal(yamlFile, &v)
if err != nil {
fmt.Printf("Unmarshal: %v", err)
os.Exit(ExitCodeCmdErr)
}
return v
}
I am running the program and grepping for the output (the actual helm release file is huge)
▶ go clean && gb .
~/Desktop/yamltutorial
./foobar | grep -i hooks -A 3
--
Hooks: (main.Hooks) {
SlackChannel: (string) ""
}
},
You did not have Chart struct
type Chart struct {
Git string `yaml:"git"`
Path string `yaml:"path"`
Ref string `yaml:"ref"`
}
Added that and got the following output
{Values:{HelmReleaseValues:{AllowAllEgress:true RecycleApp:true Hooks:{SlackChannel:https://hooks.slack.com/services/something/somethingelse/}} ReleaseName:foobar Chart:{Git:git#github.com:reponame/project.git Path:charts/path1/path1/myapp Ref:master}}}
Playground file with complete code.
https://play.golang.org/p/vCnjApr6gI9

How do I use Viper to get a value from a nested YAML structure?

How do I write the code below to get a string from my nested yaml struct?
Here is my yaml:
element:
- one:
url: http://test
nested: 123
- two:
url: http://test
nested: 123
weather:
- test:
zipcode: 12345
- ca:
zipcode: 90210
Here is example code
viper.SetConfigName("main_config")
viper.AddConfigPath(".")
err := viper.ReadInConfig()
if err != nil {
panic(err)
}
testvar := viper.GetString("element.one.url")
My problem:
I get a blank string when I print this. According to the docs, this is how you get a nested element. I suspect its not working because the elements are lists. Do I need to do a struct? I am not sure how to make one, especially if it needs to be nested.
You can unmarshal a nested configuration file.
main.go
package main
import (
"fmt"
"github.com/spf13/viper"
)
type NestedURL struct {
URL string `mapstructure:"url"`
Nested int `mapstructure:"nested"`
}
type ZipCode struct {
Zipcode string `mapstructure:"zipcode"`
}
type Config struct {
Element [] map[string]NestedURL `mapstructure:"element"`
Weather [] map[string]ZipCode `mapstructure:"weather"`
}
func main() {
viper.SetConfigName("config")
viper.AddConfigPath(".")
if err := viper.ReadInConfig(); err != nil {
return
}
var config Config
if err := viper.Unmarshal(&config); err != nil {
fmt.Println(err)
return
}
fmt.Println(config)
}
config.yml
element:
- one:
url: http://test
nested: 123
- two:
url: http://test
nested: 123
weather:
- test:
zipcode: 12345
- ca:
zipcode: 90210
There are different Get methods available in viper library and your YML structure is of type []map[string]string, so to parse your YML configuration file you have to use viper.Get method. So you have to parse your file something like this..
Note: You can use struct as well to un-marshal the data. Please refer this post mapping-nested-config-yaml-to-struct
package main
import (
"fmt"
"github.com/spf13/viper"
)
func main() {
viper.SetConfigName("config")
viper.AddConfigPath(".")
err := viper.ReadInConfig()
if err != nil {
panic(err)
}
testvar := viper.Get("element")
fmt.Println(testvar)
elementsMap := testvar.([]interface{})
for k, vmap := range elementsMap {
fmt.Print("Key: ", k)
fmt.Println(" Value: ", vmap)
eachElementsMap := vmap.(map[interface{}]interface{})
for k, vEachValMap := range eachElementsMap {
fmt.Printf("%v: %v \n", k, vEachValMap)
vEachValDataMap := vEachValMap.(map[interface{}]interface{})
for k, v := range vEachValDataMap {
fmt.Printf("%v: %v \n", k, v)
}
}
}
}
// Output:
/*
Key: 0 Value: map[one:map[url:http://test nested:123]]
one: map[url:http://test nested:123]
url: http://test
nested: 123
Key: 1 Value: map[two:map[url:http://test nested:123]]
two: map[url:http://test nested:123]
url: http://test
nested: 123
*/
You can use Unmarshal or UnmarshalKey to parse all or part of your data and fill a struct. It is very similar to unmarshaling a json.
In your case, code will be like this:
package main
import (
"fmt"
"github.com/spf13/viper"
)
// here we define schema of data, just like what we might do when we parse json
type Element struct {
Url string `mapstructure:"url"`
Nested int `mapstructure:"nested"`
}
func main() {
viper.SetConfigName("config")
viper.AddConfigPath(".")
err := viper.ReadInConfig()
if err != nil {
panic(err)
}
// data in `element` key is a map of string to Element. We define a variable to store data into it.
elementParsed := make(map[string]*Element)
// read the key `element` in the yaml file, and parse it's data and put it in `elementParsed` variable
err = viper.UnmarshalKey("element", &elementParsed)
if err != nil {
panic(err)
}
fmt.Println(elementParsed["one"].Url) // will print: http://test
fmt.Println(elementParsed["one"].Nested) // will print: 123
}

Parse yaml return empty object

I have the following yaml which I need to parse to struct.
In the builds property I got empty value while debug, what am I missing here?
I use "gopkg.in/yaml.v2"
- name: srv
type: java
path: srv
builds:
- name: db
properties:
JBR_CONFIG_RESOURCE_CONFIG: '[META-INF/context.xml:
{"service_name" : "~{h-container}"}]'
TEST2: aaaa
The struct is
type Runs struct {
Name string
Type string
Path string `yaml:"path,omitempty"`
Builds []Builds `yaml:”builds,omitempty"`
}
type Builds struct {
Name string `yaml:"name,omitempty"`
Properties Properties `yaml:"properties,omitempty"`
}
type Properties map[string]string
Properly formated yaml is the first thing that you should consider.
If u wanna have one Runs you should have your yaml formated something like that
name: srv
builds:
-
name: db
properties:
JBR_CONFIG_RESOURCE_CONFIG: "[META-INF/context.xml:
{\"service_name\" : \"~{h-container}\"}]"
TEST2: aaaa
path: srv
type: java
But then i u wanna have more of this object you need to group them in one parameter. It can look like this
runs:
-
name: srv
builds:
-
name: db
properties:
JBR_CONFIG_RESOURCE_CONFIG: "[META-INF/context.xml:
{\"service_name\" : \"~{h-container}\"}]"
TEST2: aaaa
path: srv
type: java
-
name: srv2
builds:
-
name: db2
properties:
JBR_CONFIG_RESOURCE_CONFIG: "[META-INF/context.xml:
{\"service_name\" : \"~{h-container}\"}]"
TEST2: aaaa2
path: srv2
type: java2
And then in your code could look like this
package main
import (
"fmt"
"gopkg.in/yaml.v2"
"io/ioutil"
"log"
"os"
)
type Document struct {
Runs []Runs `yaml:"runs,omitempty"`
}
type Runs struct {
Name string `yaml:"name,omitempty"`
Type string `yaml:"type,omitempty"`
Path string `yaml:"path,omitempty"`
Builds []Builds `yaml:"builds,omitempty"`
}
type Builds struct {
Name string `yaml:"name,omitempty"`
Properties map[string]string `yaml:"properties,omitempty"`
}
func main() {
var document Document
reader, err := os.Open("demo.yml")
if err != nil {
log.Fatal(err)
}
buf, _ := ioutil.ReadAll(reader)
yaml.Unmarshal(buf, &document)
if err := yaml.Unmarshal(buf, &document); err != nil {
fmt.Print(err)
os.Exit(1)
}
fmt.Println(document)
}
Make sure your yaml file is formatted correctly. Check with this tool.
The following piece of code worked fine.
package main
import (
"fmt"
"gopkg.in/yaml.v2"
"log"
)
type Runs struct {
Name string
Type string
Path string `yaml:"path,omitempty"`
Builds []Builds `yaml:”builds,omitempty"`
}
type Builds struct {
Name string `yaml:"name,omitempty"`
Properties Properties `yaml:"properties,omitempty"`
}
type Properties map[string]string
func main() {
data := `builds:
-
name: db
properties:
JBR_CONFIG_RESOURCE_CONFIG: "[META-INF/context.xml: {\"service_name\" : \"~{h-container}\"}]"
TEST2: aaaa
name: srv
path: srv
type: java
`
runs := Runs{}
err := yaml.Unmarshal([]byte(data), &runs)
if err != nil {
log.Fatalf("error: %v", err)
}
fmt.Printf("--- t:\n%v\n\n", runs)
d, err := yaml.Marshal(&runs)
if err != nil {
log.Fatalf("error: %v", err)
}
fmt.Printf("--- t dump:\n%s\n\n", string(d))
}
I hope this helps!

Resources