I’ve a yaml like following which I need to parse using go.
When I tried to run the code with the parse I got an error.
Below is the code:
var runContent= []byte(`
- runners:
- name: function1
type: func1
- command: spawn child process
- command: build
- command: gulp
- name: function1
type: func2
- command: run function 1
- name: function3
type: func3
- command: ruby build
- name: function4
type: func4
- command: go build
`)
These are the types:
type Runners struct {
runners string `yaml:"runners"`
name string `yaml:”name”`
Type: string `yaml: ”type”`
command [] Command
}
type Command struct {
command string `yaml: ”command”`
}
runners := Runners{}
err = yaml.Unmarshal(runContent, &runners)
if err != nil {
log.Fatalf("Error : %v", err)
}
When I try to parse it I got an error invalid map , what could be missing here ?
The code you have posted contains multiple errors including the struct field Type. The yaml provided in your code is not valid. This will lead to err when unmarshalling the yaml into struct.
On unmarshalling yaml in go, It is required that:
The type of the decoded values should be compatible with the
respective values in out. If one or more values cannot be decoded due
to a type mismatches, decoding continues partially until the end of
the YAML content, and a *yaml.TypeError is returned with details for
all missed values.
Along with that:
Struct fields are only unmarshalled if they are exported (have an
upper case first letter), and are unmarshalled using the field name
lowercased as the default key.
Also there is an error in defining the yaml tags, which contains space. Custom keys may be defined via the "yaml" name in the field tag: the content preceding the first comma is used as the key.
type Runners struct {
runners string `yaml:"runners"` // fields should be exportable
name string `yaml:”name”`
Type: string `yaml: ”type”` // tags name should not have space in them.
command [] Command
}
To make the struct exportable convert the struct and fields into uppercase starting letter and remove space in yaml tag names:
type Runners struct {
Runners string `yaml:"runners"`
Name string `yaml:"name"`
Type string `yaml:"type"`
Command []Command
}
type Command struct {
Command string `yaml:"command"`
}
Modify the code as below to make it work.
package main
import (
"fmt"
"log"
"gopkg.in/yaml.v2"
)
var runContent = []byte(`
- runners:
- name: function1
type:
- command: spawn child process
- command: build
- command: gulp
- name: function1
type:
- command: run function 1
- name: function3
type:
- command: ruby build
- name: function4
type:
- command: go build
`)
type Runners []struct {
Runners []struct {
Type []struct {
Command string `yaml:"command"`
} `yaml:"type"`
Name string `yaml:"name"`
} `yaml:"runners"`
}
func main() {
runners := Runners{}
// parse mta yaml
err := yaml.Unmarshal(runContent, &runners)
if err != nil {
log.Fatalf("Error : %v", err)
}
fmt.Println(runners)
}
Playground example
Validate your yaml online here https://codebeautify.org/yaml-validator/cb92c85b
Related
I've the following program in which I need to parse yaml
with the following structure
https://codebeautify.org/yaml-validator/cbabd352
this is valid yaml and I use byte to make it more simple
maybe the indentation was changed during the copy paste to the question but you can see in the link that the yaml is valid
The yaml have an api_version
and runners, for each runner (key which is name) I've a list of commands
and I need to print those commands for function1 and function4 ,what am I doing wrong here ?
package main
import (
"fmt"
"log"
"gopkg.in/yaml.v2"
)
var runContent = []byte(`
api_ver: 1
runners:
- name: function1
type:
- command: spawn child process
- command: build
- command: gulp
- name: function2
type:
- command: run function 1
- name: function3
type:
- command: ruby build
- name: function4
type:
- command: go build
`)
type Result struct {
Version string `yaml:"api_ver"`
Runners []Runners `yaml:"runners"`
}
type Runners struct {
Name string `yaml:"name"`
Type []Command `yaml:"type"`
}
type Command struct {
Command string `yaml:"command"`
}
func main() {
var runners []Result
err := yaml.Unmarshal(runContent, &runners)
if err != nil {
log.Fatalf("Error : %v", err)
}
fmt.Printf("%+v", runners[0])
}
The error which I got cannot unmarshal !!map into []main.Result
I cannot change the yaml and it should be exactly like this
https://codebeautify.org/yaml-validator/cbabd352
This is the code
https://play.golang.org/p/zidjOA6-gc7
The yaml that you have provided contains error in token. Validate the yaml used in your code here https://codebeautify.org/yaml-validator/cbaabb32
After that Create a variable of struct type result not an array. Because the yaml that you are using is creating a struct with Runners array and api_version.
This
var runners []Result
should be
var runners Result
Since because the struct is not a slice. To fetch the list of command for a name of function used in yaml. You need to loop over the runners array to find the name of function and get the value of commands for that function.
Below is the working code:
package main
import (
"fmt"
"log"
"gopkg.in/yaml.v2"
)
var runContent = []byte(`
api_ver: 1
runners:
- name: function1
type:
- command: spawn child process
- command: build
- command: gulp
- name: function2
type:
- command: run function 1
- name: function3
type:
- command: ruby build
- name: function4
type:
- command: go build
`)
type Result struct {
Version string `yaml:"api_ver"`
Runners []Runners `yaml:"runners"`
}
type Runners struct {
Name string `yaml:"name"`
Type []Command `yaml:"type"`
}
type Command struct {
Command string `yaml:"command"`
}
func main() {
var runners Result
// parse mta yaml
err := yaml.Unmarshal(runContent, &runners)
if err != nil {
log.Fatalf("Error : %v", err)
}
commandList := getCommandList("function1", runners.Runners)
fmt.Printf("%+v", commandList)
}
func getCommandList(name string, runners []Runners) []Command {
var commandList []Command
for _, value := range runners {
if value.Name == name {
commandList = value.Type
}
}
return commandList
}
Playground example
I'm using the following code to parse yaml and should get output as runners object and the function buildshould change the data structure and provide output according to below struct
type Exec struct {
NameVal string
Executer []string
}
This is what I have tried but I'm not sure how to replace
the hard-code values inside the function runner from the value I'm getting inside the yaml
return []Exec{
{"#mytest",
[]string{"spawn child process", "build", "gulp"}},
}
with the data from the parsed runner
This is all what I have tried any idea how it could be done?
package main
import (
"log"
"gopkg.in/yaml.v2"
)
var runContent = []byte(`
api_ver: 1
runners:
- name: function1
data: mytest
type:
- command: spawn child process
- command: build
- command: gulp
- name: function2
data: mytest2
type:
- command: webpack
- name: function3
data: mytest3
type:
- command: ruby build
- name: function4
type:
- command: go build
`)
type Result struct {
Version string `yaml:"api_ver"`
Runners []Runners `yaml:"runners"`
}
type Runners struct {
Name string `yaml:"name"`
Type []Command `yaml:"type"`
}
type Command struct {
Command string `yaml:"command"`
}
func main() {
var runners Result
err := yaml.Unmarshal(runContent, &runners)
if err != nil {
log.Fatalf("Error : %v", err)
}
//Here Im calling to the function with the parsed structured data which need to return the list of Exec
build("function1", runners)
}
type Exec struct {
NameVal string
Executer []string
}
func build(name string, runners Result) []Exec {
for _, runner := range runners.Runners {
if name == runner.Name {
return []Exec{
// this just for example, nameVal and Command
{"# mytest",
[]string{"spawn child process", "build", "gulp"}},
}
}
}
}
Assign the name of runners object to the struct Exec field for name and append the command list to the []string type field with the commands of the function that matched the name as:
func build(name string, runners Result) []Exec {
exec := make([]Exec, len(runners.Runners))
for i, runner := range runners.Runners {
if name == runner.Name {
exec[i].NameVal = runner.Name
for _, cmd := range runner.Type {
exec[i].Executer = append(exec[i].Executer, cmd.Command)
}
fmt.Printf("%+v", exec)
return exec
}
}
return exec
}
Working code on Playground
I have a program which parses yamls file to an objects (structs).
I use the following repo to do it
https://github.com/go-yaml/yaml
for example in the file I have:
dependency :
- name: ui
type: runner
cwd: /ui
install:
- name: api
group: test
And I use the following struct for it
type Dependency struct {
Name string
Type string
CWD string
Install []Install
//here I have the issue
Requires ?
}
type Install struct {
Name string
Group string
}
Now I have two few issue with a bit complex struct.
This is the entry which could be inside the Dependency struct and this is how it look in the yaml file
requires:
- name: db
- type: mongo
but it also can be
requires:
- name: db
- name: rst
- name: test
- name: test2
Since it have multiple name properties how should I got build this struct
In addition I've field in the yaml
_type-version: "1.0.0"
when I put it inside struct like following I got error since I use -
type TypeVer struct{
_Type-version string
}
How to overcome this?
The yaml package actually allows you to remap the the name for the properties, you can use this to handle your _type-version Property.
And your initial question: Just define Requires the same as Install:
package main
import (
"fmt"
"log"
"github.com/go-yaml/yaml"
)
type File struct {
TypeVersion string `yaml:"_type-version"`
Dependency []Dependency
}
type Dependency struct {
Name string
Type string
CWD string
Install []Install
Requires []Requires
}
type Install struct {
Name string
Group string
}
type Requires struct {
Name string
Type string
}
var data = `
_type-version: "1.0.0"
dependency:
- name: ui
type: runner
cwd: /ui
install:
- name: api
group: test
requires:
- name: db
- type: mongo
- name: rst
- name: test
- name: test2
`
func main() {
f := File{}
err := yaml.Unmarshal([]byte(data), &f)
if err != nil {
log.Fatalf("error: %v", err)
}
fmt.Printf("--- t:\n%v\n\n", f)
d, err := yaml.Marshal(&f)
if err != nil {
log.Fatalf("error: %v", err)
}
fmt.Printf("--- t dump:\n%s\n\n", string(d))
}
I’ve multiple yaml files which need to be parsed and have exactly the same structure
schema: "1.0.0"
id: test
version: "1.2.3"
dependency :
- name: ui
type: runner
cwd: /ui
install:
- name: api
group: test
properties:
name: app
url: appUrl
- name: backend
type: mongoDb
path: be
install:
- name: db
type: mongo
provides:
- name: api
properties:
url: url
The schema section is mandatory for all the yaml which the app should
get always
The dependency tag can contain 1..n entries with mandatory fields
,name, type, cwd
The dependency tag can (or not) contain install section with name
and properties which is mandatory
The dependency tag can (or not) contain provides section with name
and which is mandatory
The install can have properties and the provides also can have
properties
I am using a yaml parser to parse the files but my question is how in Golang I can build struct that when I parse the doc it will automatically fill the main struct and will include the sub structs (such as dependency/ install sections )
I have tried something like
type main struct {
schema struct {
schema string
id int
version string
}
dependency struct {
name string
type string
cwd string
install struct {
name string
}
}
In the install section, it can be group or type or both and it can have also properties section, so I'm not sure how to build
some generic /extendable struct which I use to parse the document (the document have close list of properties, what I put in the example describe the most options)
I use this lib to parse the doc
yaml parser
Your struct definition should be something like this
type Yaml struct {
Schema string
ID string
Version string
Dependency []Dependency
}
type Dependency struct {
Name string
Type string
CWD string
Install []Install
Provides []Provide
}
type Install struct {
Name string
Group string
Type string
Properties Properties
}
type Properties struct {
Name string
URL string
}
type Provide struct {
Name string
Properties Properties
}
Here is full sample code:
package main
import (
"fmt"
"io/ioutil"
"log"
"gopkg.in/yaml.v2"
)
var data = `
schema: "1.0.0"
id: test
version: "1.2.3"
dependency :
- name: ui
type: runner
cwd: /ui
install:
- name: api
group: test
properties:
name: app
url: appUrl
- name: backend
type: mongoDb
path: be
install:
- name: db
type: mongo
provides:
- name: api
properties:
url: url
`
type Yaml struct {
Schema string
ID string
Version string
Dependency []Dependency
}
type Dependency struct {
Name string
Type string
CWD string
Install []Install
Provides []Provide
}
type Install struct {
Name string
Group string
Type string
Properties Properties
}
type Properties struct {
Name string
URL string
}
type Provide struct {
Name string
Properties Properties
}
func main() {
y := Yaml{}
err := yaml.Unmarshal([]byte(data), &y)
if err != nil {
log.Fatalf("error: %v", err)
}
fmt.Printf("%+v\n", y)
}
Output
{Schema:1.0.0 ID:test Version:1.2.3 Dependency:[{Name:ui Type:runner CWD:/ui Install:[{Name:api Group:test Type: Properties:{Name:appURL:appUrl}}] Provides:[]} {Name:backend Type:mongoDb CWD: Install:[{Name:db Group: Type:mongo Properties:{Name: URL:}}] Provides:[{Name:api Properties:{Name: URL:url}}]}]}
If you want to read from a yaml file, in the main func just replace:
err := yaml.Unmarshal([]byte(data), &y)
with
yamlFile, err := ioutil.ReadFile("yaml_sample.yaml")
if err != nil {
log.Printf("yamlFile.Get err #%v ", err)
}
err = yaml.Unmarshal(yamlFile, &y)
Having a problem parsing this sort of yaml file. Using "yaml.v2"
info: "abc"
data:
source: http://intra
destination: /tmp
run:
- id: "A1"
exe: "run.a1"
output: "output.A1"
- id: "A2"
exe: "run.a2"
output: "output.A2"
I would like to get all the values of the YAML file so I have a basic struct like this
type Config struct {
Info string
Data struct {
Source string `yaml:"source"`
Destination string `yaml:"destination"`
}
}
This works
But, I am not sure how to setup the struct for "run". The extra layer is confusing me.
type Run struct {
...
}
the OP's example of YAML is invalid. When value of run is list of dictionary it should be something like this:
info: "abc"
data:
source: http://intra
destination: /tmp
run:
- id: "A1"
exe: "run.a1"
output: "output.A1"
- id: "A2"
exe: "run.a2"
output: "output.A2"
And here's the corresponding data struture, and example for decoding YAML into golang's structure.
package main
import (
"fmt"
"io/ioutil"
"os"
yaml "gopkg.in/yaml.v2"
)
type Config struct {
Info string
Data struct {
Source string
Destination string
}
Run []struct {
Id string
Exe string
Output string
}
}
func main() {
var conf Config
reader, _ := os.Open("example.yaml")
buf, _ := ioutil.ReadAll(reader)
yaml.Unmarshal(buf, &conf)
fmt.Printf("%+v\n", conf)
}
running this will output (added some indent for readability):
{Info:abc
Data:{Source:http://intra Destination:/tmp}
Run:[{Id:A1 Exe:run.a1 Output:output.A1}
{Id:A2 Exe:run.a2 Output:output.A2}]