I am trying to parse a yaml file into go using the "gopkg.in/yaml.v3" package.
The problem I haven't been able to solve is parsing a file starting with a -. For example:
---
- type: people
info:
- { name: John, last: Doe }
...
So when I try to parse this
package main
import (
"fmt"
"io/ioutil"
"log"
"gopkg.in/yaml.v3"
)
type YamlFile struct {
type string `yaml:"type"`
}
func main() {
d := YamlFile{}
src, err := ioutil.ReadFile("test1.yaml")
if err != nil {
log.Println(err)
}
err = yaml.Unmarshal(src, &d)
if err != nil {
log.Printf("error: %v", err)
}
fmt.Println(d)
}
output: error: yaml: unmarshal errors: line 2: cannot unmarshal !!seq into main.YamlFile
The above code works when the - is removed from the file
---
type: people
info:
- { name: John, last: Doe }
...
However I cannot reformat the file so I need to know what I am doing wrong trying to parse with the - in place. Thanks for any pointers in the right direction.
The - indicates that it's a list/array. Therefore you must unmarshal into a slice or array in Go.
Change d := YamlFile{} to d := []YamlFile{}, and you will no longer get that error.
But also note that you'll always get an empty result with the struct you've defined, because it has no exported fields.
Try instead:
type YamlFile struct {
Type string `yaml:"type"`
}
See it on the playground.
I am trying to unmasrhal the following flux HelmRelease file.
apiVersion: helm.fluxcd.io/v1
kind: HelmRelease
metadata:
annotations:
fluxcd.io/automated: 'false'
fluxcd.io/tag.ats: glob:*
name: ats
namespace: myns
spec:
chart:
git: git#github.com:reponame/project.git
path: charts/path1/path1/myapp
ref: master
releaseName: foobar
values:
allowAllEgress: true
recycleApp: true
hooks:
slackChannel: https://hooks.slack.com/services/something/somethingelse/
Here are my models
type HelmReleaseValues struct {
AllowAllEgress bool `yaml:"allowAllEgress"`
RecycleApp bool `yaml:"recycleApp"`
Hooks `yaml:"hooks"`
}
type Hooks struct {
SlackChannel string `yaml:"slackChannel"`
}
type Values struct {
HelmReleaseValues `yaml:"values"`
ReleaseName string `yaml:"releaseName"`
Chart `yaml:"chart"`
}
type Spec struct {
Values `yaml:"spec"`
}
The problem is that the fields allowAllEgress and recycleApp are getting unmarshalled.
However the Hooks field in my struct turns out to be empty.
What am I doing wrong in the struct modelling / tagging?
edit: here is my code
package main
import (
"fmt"
"io/ioutil"
"os"
"github.com/davecgh/go-spew/spew"
"gopkg.in/yaml.v3"
)
const ExitCodeCmdErr = 1
func main() {
rawYaml := parseHelmReleaseFile("myfile.yaml")
spew.Dump(rawYaml)
}
func parseHelmReleaseFile(fileName string) Spec {
var v Spec
yamlFile, err := ioutil.ReadFile(fileName)
if err != nil {
fmt.Printf("yaml file err #%v ", err)
os.Exit(ExitCodeCmdErr)
}
err = yaml.Unmarshal(yamlFile, &v)
if err != nil {
fmt.Printf("Unmarshal: %v", err)
os.Exit(ExitCodeCmdErr)
}
return v
}
I am running the program and grepping for the output (the actual helm release file is huge)
▶ go clean && gb .
~/Desktop/yamltutorial
./foobar | grep -i hooks -A 3
--
Hooks: (main.Hooks) {
SlackChannel: (string) ""
}
},
You did not have Chart struct
type Chart struct {
Git string `yaml:"git"`
Path string `yaml:"path"`
Ref string `yaml:"ref"`
}
Added that and got the following output
{Values:{HelmReleaseValues:{AllowAllEgress:true RecycleApp:true Hooks:{SlackChannel:https://hooks.slack.com/services/something/somethingelse/}} ReleaseName:foobar Chart:{Git:git#github.com:reponame/project.git Path:charts/path1/path1/myapp Ref:master}}}
Playground file with complete code.
https://play.golang.org/p/vCnjApr6gI9
I've the following yaml file which I need to parse (the parse is working as expected) and need to provide a datafrom the yaml file content that should be exposed by the following decoupled functions
I need to provide the following functions (here is example of some of those functions, need more with the same pattern...)
getApps()
getServices()
GetApp(appname)
GetServiceForApp(appname)
This is the code (which works...)
var DMZ = []byte(`
applications:
- name: app1
type: php
src: /app1
host: us
use:
- redis
- mysql
- name: app2
type: rust
src: /app2
host: eu
use:
- mongo
- mysql
- name: app3
type: golang
src: /app3
host: us
use:
- postgress
- mysql
services:
- name: mongo
type: db
host: us
- name: mysql
type: db
host: eu
- name: postgress
type: db
host: us
- name: redis
type: db
host: us
`)
This is the structs
type DMZ struct {
Applications []*Applications `yaml:"applications,omitempty"`
Services []*Services `yaml:"services,omitempty"`
}
type Applications struct {
Name string
Type string
Src string `yaml:"src,omitempty"`
use []Use `yaml:"use,omitempty"`
}
type Services struct {
Name string
Type string
Host string `yaml:"host,omitempty"`
}
type Use struct {
Name string `yaml:"name,omitempty"`
host string `yaml:"host,omitempty"`
Type string `yaml:"type,omitempty"`
}
// Parse file
func Parse(yamlContent []byte) (out DMZ, err error) {
dmz := DMZ{}
err = yaml.Unmarshal([]byte(yamlContent), &dmz)
if err != nil {
logs.Error("Yaml file is not valid, Error: " + err.Error())
}
return dmz, err
}
As the Parse function is a per-requite to all the needed functions (which I listed above) I wonder how is the best to create them ,create simple function that every time call to the parse function and then do the logic (not a problem) but I wonder if there is better approach which follows the clean code principles for Golang , with 'interface / dependency injections' ?
UPDATE:
I want to avoid doing things like following, assume that I need to call to those function from different packages or even different GitHub repository how it's best to do it with Golang clean code.
func getApps(){
dmz := Parse()
....
}
func getServices(){
dmz := Parse()
....
}
func getApp(appname string){
dmz := Parse()
....
}
func GetServiceForApp(appname string){
dmz := Parse()
....
}
And I need more functions with the same pattern ...
What I need some Clean Code solution using interface/dependency injection like a best practice code example in Golang
If something is not clear please let me know :)
Parse the value inside pointer type struct value which will store the value inside the struct pointed by a pointer. Then Create method receives on All the methods from which you want to get the value of an app or service inside the struct.
Using pointer receiver in all methods you will be able to access the original struct which is updated when parsing the yaml.
Create a pointer to empty instance of the struct and pass it as a method receiver. Then access that struct in unmarshal to update yaml data in original struct. Access the updated struct in every function by accessing the value of original struct using pointer receiver to method.
package main
import (
"fmt"
"log"
yaml "gopkg.in/yaml.v2"
)
var dmz = []byte(`
applications:
- name: app1
type: php
src: /app1
host: us
use:
- redis
- mysql
- name: app2
type: rust
src: /app2
host: eu
use:
- mongo
- mysql
- name: app3
type: golang
src: /app3
host: us
use:
- postgress
- mysql
services:
- name: mongo
type: db
host: us
- name: mysql
type: db
host: eu
- name: postgress
type: db
host: us
- name: redis
type: db
host: us
`)
type DMZ struct {
Applications []*Applications `yaml:"applications,omitempty"`
Services []*Services `yaml:"services,omitempty"`
}
type Applications struct {
Name string
Type string
Src string `yaml:"src,omitempty"`
use []Use `yaml:"use,omitempty"`
}
type Services struct {
Name string
Type string
Host string `yaml:"host,omitempty"`
}
type Use struct {
Name string `yaml:"name,omitempty"`
host string `yaml:"host,omitempty"`
Type string `yaml:"type,omitempty"`
}
func main() {
dm := &DMZ{}
result, err := dm.Parse(dmz)
dm.getApp("app1")
if err != nil {
fmt.Println(err)
}
fmt.Println(result)
fmt.Println(dm.getApp("app2"))
fmt.Println(dm.GetServiceForApp("mongo"))
}
func (dmz *DMZ) getApps() []*Applications {
return dmz.Applications
}
func (dmz *DMZ) getServices() []*Services {
return dmz.Services
}
func (dmz *DMZ) getApp(appname string) *Applications {
for _, value := range dmz.Applications {
if appname == value.Name {
fmt.Println((*value).Name)
return value
}
}
return nil
}
func (dmz *DMZ) GetServiceForApp(appname string) *Services {
for _, value := range dmz.Services {
if appname == value.Name {
return value
}
}
return nil
}
// Parse file
func (dmz *DMZ) Parse(yamlContent []byte) (out *DMZ, err error) {
err = yaml.Unmarshal([]byte(yamlContent), &dmz)
if err != nil {
log.Fatal("Yaml file is not valid, Error: " + err.Error())
}
return dmz, err
}
Working code on Playground
If you want your code to be more clean then you can also skip the struct returned from parse function. Since we are passing a pointer type receiver and updating the original struct as:
package main
import (
"fmt"
"log"
yaml "gopkg.in/yaml.v2"
)
var dmz = []byte(`
applications:
- name: app1
type: php
src: /app1
host: us
use:
- redis
- mysql
- name: app2
type: rust
src: /app2
host: eu
use:
- mongo
- mysql
- name: app3
type: golang
src: /app3
host: us
use:
- postgress
- mysql
services:
- name: mongo
type: db
host: us
- name: mysql
type: db
host: eu
- name: postgress
type: db
host: us
- name: redis
type: db
host: us
`)
type DMZ struct {
Applications []*Applications `yaml:"applications,omitempty"`
Services []*Services `yaml:"services,omitempty"`
}
type Applications struct {
Name string
Type string
Src string `yaml:"src,omitempty"`
use []Use `yaml:"use,omitempty"`
}
type Services struct {
Name string
Type string
Host string `yaml:"host,omitempty"`
}
type Use struct {
Name string `yaml:"name,omitempty"`
host string `yaml:"host,omitempty"`
Type string `yaml:"type,omitempty"`
}
func main() {
dm := &DMZ{}
dm.Parse(dmz)
fmt.Println(dm.getApp("app2"))
fmt.Println(dm.GetServiceForApp("mongo"))
}
func (dmz *DMZ) getApps() []*Applications {
return dmz.Applications
}
func (dmz *DMZ) getServices() []*Services {
return dmz.Services
}
func (dmz *DMZ) getApp(appname string) *Applications {
for _, value := range dmz.Applications {
if appname == value.Name {
fmt.Println((*value).Name)
return value
}
}
return nil
}
func (dmz *DMZ) GetServiceForApp(appname string) *Services {
for _, value := range dmz.Services {
if appname == value.Name {
return value
}
}
return nil
}
// Parse file
func (dmz *DMZ) Parse(yamlContent []byte) {
if err := yaml.Unmarshal([]byte(yamlContent), &dmz); err != nil {
log.Fatal("Yaml file is not valid, Error: " + err.Error())
}
}
As we can notice that Parse function will become more clean since we are not returning anything from it we are just updating the original struct using method receiver which is a much better way to achieve what you have been trying to achieve.
You can also choose to implement an interface by defining the methods in struct as receiver for which you are trying to implement the interface as:
package main
import (
"fmt"
"log"
yaml "gopkg.in/yaml.v2"
)
type DI interface {
GetApps() []*Applications
GetServices() *Services
}
var dmz = []byte(`
applications:
- name: app1
type: php
src: /app1
host: us
use:
- redis
- mysql
- name: app2
type: rust
src: /app2
host: eu
use:
- mongo
- mysql
- name: app3
type: golang
src: /app3
host: us
use:
- postgress
- mysql
services:
- name: mongo
type: db
host: us
- name: mysql
type: db
host: eu
- name: postgress
type: db
host: us
- name: redis
type: db
host: us
`)
type DMZ struct {
Applications []*Applications `yaml:"applications,omitempty"`
Services []*Services `yaml:"services,omitempty"`
}
type Applications struct {
Name string
Type string
Src string `yaml:"src,omitempty"`
use []Use `yaml:"use,omitempty"`
}
type Services struct {
Name string
Type string
Host string `yaml:"host,omitempty"`
}
type Use struct {
Name string `yaml:"name,omitempty"`
host string `yaml:"host,omitempty"`
Type string `yaml:"type,omitempty"`
}
func main() {
dm := &DMZ{}
dm.Parse(dmz)
fmt.Println(dm.getApp("app2"))
fmt.Println(dm.GetServiceForApp("mongo"))
}
func (dmz *DMZ) GetApps() []*Applications {
return dmz.Applications
}
func (dmz *DMZ) GetServices() []*Services {
return dmz.Services
}
func (dmz *DMZ) getApp(appname string) *Applications {
for _, value := range dmz.Applications {
if appname == value.Name {
fmt.Println((*value).Name)
return value
}
}
return nil
}
func (dmz *DMZ) GetServiceForApp(appname string) *Services {
for _, value := range dmz.Services {
if appname == value.Name {
return value
}
}
return nil
}
// Parse file
func (dmz *DMZ) Parse(yamlContent []byte) {
if err := yaml.Unmarshal([]byte(yamlContent), &dmz); err != nil {
log.Fatal("Yaml file is not valid, Error: " + err.Error())
}
}
Note:
Generics are convenient but they come at a cost in complexity in the
type system and run-time. We haven't yet found a design that gives
value proportionate to the complexity, although we continue to think
about it. Meanwhile, Go's built-in maps and slices, plus the ability
to use the empty interface to construct containers (with explicit
unboxing) mean in many cases it is possible to write code that does
what generics would enable, if less smoothly.
You can define an interface and provide implementations in the struct
type DMZI interface {
GetApps() []Application
GetService() []Service
GetApp(name string) (Application, error)
GetServiceForApp(name string) ([]string, error)
}
type DMZ struct {
Application []Application `yaml:"applications,omitempty"`
Service []Service `yaml:"services,omitempty"`
}
func (dmz DMZ) GetApps() []Application {
return dmz.Application
}
func (dmz DMZ) GetService() []Service {
return dmz.Service
}
func (dmz DMZ) GetApp(name string) (Application, error) {
for _, app := range dmz.Application {
if app.Name == name {
return app, nil
}
}
return Application{}, fmt.Errorf("Did not find application with name %s", name)
}
func (dmz DMZ) GetServiceForApp(name string) ([]string, error) {
app, err := dmz.GetApp(name)
if err != nil {
return []string{}, err
}
return app.Use, nil
}
type Application struct {
Name string
Type string
Src string `yaml:"src,omitempty"`
Use []string `yaml:"use,omitempty"`
}
type Service struct {
Name string
Type string
Host string `yaml:"host,omitempty"`
}
// Parse file
func Parse(yamlContent []byte) (out DMZI, err error) {
dmz := DMZ{}
err = yaml.Unmarshal([]byte(yamlContent), &dmz)
if err != nil {
fmt.Println("Yaml file is not valid, Error: " + err.Error())
}
return dmz, err
}
So, you can call the methods on the returned interface e.g.
fmt.Printf("Apps : %+v\n", dmz.GetApps())
fmt.Printf("Service : %+v\n", dmz.GetService())
UPDATE
main method as requested in the comment
func main() {
dmz, err := Parse([]byte(ymlStr))
if err != nil {
panic(err)
}
fmt.Printf("Apps : %+v\n", dmz.GetApps())
fmt.Printf("Service : %+v\n", dmz.GetService())
}
Will print
Apps : [{Name:app1 Type:php Src:/app1 Use:[redis mysql]} {Name:app2 Type:rust Src:/app2 Use:[mongo mysql]} {Name:app3 Type:golang Src:/app3 Use:[postgress mysql]}]
Service : [{Name:mongo Type:db Host:us} {Name:mysql Type:db Host:eu} {Name:postgress Type:db Host:us} {Name:redis Type:db Host:us}]
I've changed your code a little and done what I think you are asking for, which is to always call "Parse" whenever you create a new instance of DMZ. This is how you would do this using Dargo. Look at the bottom to see how the code would bind DMZ into a ServiceLocator and how to get new instances of DMZ
type DMZ struct {
Applications []*Applications `yaml:"applications,omitempty"`
Services []*Services `yaml:"services,omitempty"`
}
// This method is called every time DMZ is created via the Dargo API
func (dmz *DMZ) DargoInitialize(ioc.Descriptor) error {
// get ur bytes from... wherever
Parse(dmz, []byte{})
}
type Applications struct {
Name string
Type string
Src string `yaml:"src,omitempty"`
use []Use `yaml:"use,omitempty"`
}
type Services struct {
Name string
Type string
Host string `yaml:"host,omitempty"`
}
type Use struct {
Name string `yaml:"name,omitempty"`
host string `yaml:"host,omitempty"`
Type string `yaml:"type,omitempty"`
}
// Parse file
func Parse(dmz *DMZ, yamlContent []byte) (out *DMZ, err error) {
err = yaml.Unmarshal([]byte(yamlContent), &dmz)
if err != nil {
logs.Error("Yaml file is not valid, Error: " + err.Error())
}
return dmz, err
}
func useDargo() error {
locator, err := ioc.CreateAndBind("example", func(binder ioc.Binder) error {
binder.Bind("DMZ", &DMZ{}).InScope(ioc.PerLookup)
})
if err != nil {
return err
}
// This is how you would get your instances of DMZ
locator.GetDService("DMZ")
}
I've the following program in which I need to parse yaml
with the following structure
https://codebeautify.org/yaml-validator/cbabd352
this is valid yaml and I use byte to make it more simple
maybe the indentation was changed during the copy paste to the question but you can see in the link that the yaml is valid
The yaml have an api_version
and runners, for each runner (key which is name) I've a list of commands
and I need to print those commands for function1 and function4 ,what am I doing wrong here ?
package main
import (
"fmt"
"log"
"gopkg.in/yaml.v2"
)
var runContent = []byte(`
api_ver: 1
runners:
- name: function1
type:
- command: spawn child process
- command: build
- command: gulp
- name: function2
type:
- command: run function 1
- name: function3
type:
- command: ruby build
- name: function4
type:
- command: go build
`)
type Result struct {
Version string `yaml:"api_ver"`
Runners []Runners `yaml:"runners"`
}
type Runners struct {
Name string `yaml:"name"`
Type []Command `yaml:"type"`
}
type Command struct {
Command string `yaml:"command"`
}
func main() {
var runners []Result
err := yaml.Unmarshal(runContent, &runners)
if err != nil {
log.Fatalf("Error : %v", err)
}
fmt.Printf("%+v", runners[0])
}
The error which I got cannot unmarshal !!map into []main.Result
I cannot change the yaml and it should be exactly like this
https://codebeautify.org/yaml-validator/cbabd352
This is the code
https://play.golang.org/p/zidjOA6-gc7
The yaml that you have provided contains error in token. Validate the yaml used in your code here https://codebeautify.org/yaml-validator/cbaabb32
After that Create a variable of struct type result not an array. Because the yaml that you are using is creating a struct with Runners array and api_version.
This
var runners []Result
should be
var runners Result
Since because the struct is not a slice. To fetch the list of command for a name of function used in yaml. You need to loop over the runners array to find the name of function and get the value of commands for that function.
Below is the working code:
package main
import (
"fmt"
"log"
"gopkg.in/yaml.v2"
)
var runContent = []byte(`
api_ver: 1
runners:
- name: function1
type:
- command: spawn child process
- command: build
- command: gulp
- name: function2
type:
- command: run function 1
- name: function3
type:
- command: ruby build
- name: function4
type:
- command: go build
`)
type Result struct {
Version string `yaml:"api_ver"`
Runners []Runners `yaml:"runners"`
}
type Runners struct {
Name string `yaml:"name"`
Type []Command `yaml:"type"`
}
type Command struct {
Command string `yaml:"command"`
}
func main() {
var runners Result
// parse mta yaml
err := yaml.Unmarshal(runContent, &runners)
if err != nil {
log.Fatalf("Error : %v", err)
}
commandList := getCommandList("function1", runners.Runners)
fmt.Printf("%+v", commandList)
}
func getCommandList(name string, runners []Runners) []Command {
var commandList []Command
for _, value := range runners {
if value.Name == name {
commandList = value.Type
}
}
return commandList
}
Playground example
I have a hard time to unmarshal this piece of YAML in Go. The error I'm getting is cannot unmarshal !!seq into map[string][]map[string][]string. I've tried all kind of maps with no success (map[string]string ; []map[string]string and so on)
import (
"gopkg.in/yaml.v1"
"io/ioutil"
)
type AppYAML struct {
Runtime string `yaml:"runtime,omitempty"`
Handlers map[string][]map[string][]string `yaml:"handlers,omitempty"`
Env_Variables map[string]string `yaml:"env_variables,omitempty"`
}
func main() {
s := `
runtime: go
handlers:
- url: /.*
runtime: _go_app
secure: always
env_variables:
something: 'test'
`
var a AppYAML
if err = yaml.Unmarshal([]byte(s), &a); err != nil {
log.Error(err)
return
}
}
Change type declaration to this:
type AppYAML struct {
Runtime string `yaml:"runtime,omitempty"`
Handlers []map[string]string `yaml:"handlers,omitempty"`
Env_Variables map[string]string `yaml:"env_variables,omitempty"`
}