I’ve multiple yaml files which need to be parsed and have exactly the same structure
schema: "1.0.0"
id: test
version: "1.2.3"
dependency :
- name: ui
type: runner
cwd: /ui
install:
- name: api
group: test
properties:
name: app
url: appUrl
- name: backend
type: mongoDb
path: be
install:
- name: db
type: mongo
provides:
- name: api
properties:
url: url
The schema section is mandatory for all the yaml which the app should
get always
The dependency tag can contain 1..n entries with mandatory fields
,name, type, cwd
The dependency tag can (or not) contain install section with name
and properties which is mandatory
The dependency tag can (or not) contain provides section with name
and which is mandatory
The install can have properties and the provides also can have
properties
I am using a yaml parser to parse the files but my question is how in Golang I can build struct that when I parse the doc it will automatically fill the main struct and will include the sub structs (such as dependency/ install sections )
I have tried something like
type main struct {
schema struct {
schema string
id int
version string
}
dependency struct {
name string
type string
cwd string
install struct {
name string
}
}
In the install section, it can be group or type or both and it can have also properties section, so I'm not sure how to build
some generic /extendable struct which I use to parse the document (the document have close list of properties, what I put in the example describe the most options)
I use this lib to parse the doc
yaml parser
Your struct definition should be something like this
type Yaml struct {
Schema string
ID string
Version string
Dependency []Dependency
}
type Dependency struct {
Name string
Type string
CWD string
Install []Install
Provides []Provide
}
type Install struct {
Name string
Group string
Type string
Properties Properties
}
type Properties struct {
Name string
URL string
}
type Provide struct {
Name string
Properties Properties
}
Here is full sample code:
package main
import (
"fmt"
"io/ioutil"
"log"
"gopkg.in/yaml.v2"
)
var data = `
schema: "1.0.0"
id: test
version: "1.2.3"
dependency :
- name: ui
type: runner
cwd: /ui
install:
- name: api
group: test
properties:
name: app
url: appUrl
- name: backend
type: mongoDb
path: be
install:
- name: db
type: mongo
provides:
- name: api
properties:
url: url
`
type Yaml struct {
Schema string
ID string
Version string
Dependency []Dependency
}
type Dependency struct {
Name string
Type string
CWD string
Install []Install
Provides []Provide
}
type Install struct {
Name string
Group string
Type string
Properties Properties
}
type Properties struct {
Name string
URL string
}
type Provide struct {
Name string
Properties Properties
}
func main() {
y := Yaml{}
err := yaml.Unmarshal([]byte(data), &y)
if err != nil {
log.Fatalf("error: %v", err)
}
fmt.Printf("%+v\n", y)
}
Output
{Schema:1.0.0 ID:test Version:1.2.3 Dependency:[{Name:ui Type:runner CWD:/ui Install:[{Name:api Group:test Type: Properties:{Name:appURL:appUrl}}] Provides:[]} {Name:backend Type:mongoDb CWD: Install:[{Name:db Group: Type:mongo Properties:{Name: URL:}}] Provides:[{Name:api Properties:{Name: URL:url}}]}]}
If you want to read from a yaml file, in the main func just replace:
err := yaml.Unmarshal([]byte(data), &y)
with
yamlFile, err := ioutil.ReadFile("yaml_sample.yaml")
if err != nil {
log.Printf("yamlFile.Get err #%v ", err)
}
err = yaml.Unmarshal(yamlFile, &y)
Related
I'm new in go, I can't find a way to unmarshal a yaml using "gopkg.in/yaml.v2"
I suppose the error is in the way I define the struct.
I need to parse a kubernetes job yaml and edit in go to generate an update yaml.
The structure is almost static but have two lists in which the keys could have different things inside.
I reduced the yaml to one listes (volumes) to simplify the example.
apiVersion: batch/v1
kind: Job
metadata:
name: jobname
namespace: namespace
spec:
ttlSecondsAfterFinished: 86400
template:
spec:
containers:
- name: container-name
image: containerimage:tag
command:
- php
- cli/migrations.php
- up
restartPolicy: Never
volumes:
- name: filestore
persistentVolumeClaim:
claimName: data-pvc
readOnly: false
- name: stackdriver
secret:
secretName: stackdriver-prod
backoffLimit: 1
those are my structs definitions:
type PersistentVolumeClaims struct {
ClaimName string `yaml:"claimName,omitempty"`
ReadOnly bool `yaml:"readOnly,omitempty"`
}
type Secrets struct {
SecretName string `yaml:"secretName,omitempty"`
}
type Names struct {
Name string `yaml:"name"`
PersistentVolumeClaim PersistentVolumeClaims `yaml:"persistentVolumeClaim,omitempty"`
Secret Secrets `yaml:"secret,omitempty"`
}
type Jobs struct {
ApiVersion string `yaml:"apiVersion"`
Kind string `yaml:"kind"`
Metadata struct {
Name string `yaml:"name"`
Namespace string `yaml:"namespace"`
}
Spec struct {
TtlSecondsAfterFinished int `yaml:"ttlSecondsAfterFinished"`
Template struct {
Spec struct {
Containers []struct {
Name string
Image string `yaml:"image"`
Command []string `yaml:"command"`
VolumeMounts []struct {
Name string
SubPath string `yaml:"subPath"`
MountPath string `yaml:"mountPath"`
ReadOnly bool `yaml:"readOnly"`
}
RestartPolicy string `yaml:"restartPolicy"`
}
Volumes map[string][]Names
}
BackoffLimit int `yaml:"backoffLimit"`
}
}
}
I tried different structures but I don't get the solution.
Any help will be appreciated.
--- SOLVED
I have redone the tool using the official go-client https://github.com/kubernetes/client-go as suggested by Jonas. Now everything work!
I need to parse a kubernetes job yaml and edit in go to generate an update yaml. The structure is almost static but have two lists in wich the keys could have different things inside.
It sound like your application is running in a cluster, want to retrieve a Job, modify it and then update it.
I would recommend the official Kubernetes client-go for this. It has libraries for Kubernetes resources like Job. See the example using a Deployment
You can use the official structs from the Kubernetes APIs. If you still want to declare the structs: Your yaml tags are incomplete. You do not have tags for nested structs:
type Jobs struct {
Metadata struct {
Name string `yaml:"name"`
Namespace string `yaml:"namespace"`
} `yaml:"metadata"` // Here is one
Spec struct {
TtlSecondsAfterFinished int `yaml:"ttlSecondsAfterFinished"`
Template struct {
Spec struct {
Containers []struct {
VolumeMounts []struct {
Name string
SubPath string `yaml:"subPath"`
MountPath string `yaml:"mountPath"`
ReadOnly bool `yaml:"readOnly"`
} `yaml:"volumeMounts"` // Here is another
} `yaml:"containers"` // Here is another
Volumes map[string][]Names `yaml:"volumes"` // This is missin as well
} `yaml:"spec"` //
} `yaml:"template"` //
} `yaml:"spec"` //
}
Suppose that your data is in the file example.yaml; you can unmarshall into a struct using gopkg.in/yaml.v2 like
package main
import (
"fmt"
"log"
"io/ioutil"
"gopkg.in/yaml.v2"
)
type item struct {
ItemA string `yaml:"item_a"`
ItemB string `yaml:"item_b"`
}
func read(filename string) (item, error) {
var output item
content, err := ioutil.ReadFile(filename)
if err != nil {
return item{}, err
}
err = yaml.Unmarshal(content, output)
if err != nil {
return item{}, err
}
return output, nil
}
func main() {
output, err := read("example.yaml")
if err != nil {
log.Fatal(err)
}
fmt.Println("output: ", output)
}
I've the following program in which I need to parse yaml
with the following structure
https://codebeautify.org/yaml-validator/cbabd352
this is valid yaml and I use byte to make it more simple
maybe the indentation was changed during the copy paste to the question but you can see in the link that the yaml is valid
The yaml have an api_version
and runners, for each runner (key which is name) I've a list of commands
and I need to print those commands for function1 and function4 ,what am I doing wrong here ?
package main
import (
"fmt"
"log"
"gopkg.in/yaml.v2"
)
var runContent = []byte(`
api_ver: 1
runners:
- name: function1
type:
- command: spawn child process
- command: build
- command: gulp
- name: function2
type:
- command: run function 1
- name: function3
type:
- command: ruby build
- name: function4
type:
- command: go build
`)
type Result struct {
Version string `yaml:"api_ver"`
Runners []Runners `yaml:"runners"`
}
type Runners struct {
Name string `yaml:"name"`
Type []Command `yaml:"type"`
}
type Command struct {
Command string `yaml:"command"`
}
func main() {
var runners []Result
err := yaml.Unmarshal(runContent, &runners)
if err != nil {
log.Fatalf("Error : %v", err)
}
fmt.Printf("%+v", runners[0])
}
The error which I got cannot unmarshal !!map into []main.Result
I cannot change the yaml and it should be exactly like this
https://codebeautify.org/yaml-validator/cbabd352
This is the code
https://play.golang.org/p/zidjOA6-gc7
The yaml that you have provided contains error in token. Validate the yaml used in your code here https://codebeautify.org/yaml-validator/cbaabb32
After that Create a variable of struct type result not an array. Because the yaml that you are using is creating a struct with Runners array and api_version.
This
var runners []Result
should be
var runners Result
Since because the struct is not a slice. To fetch the list of command for a name of function used in yaml. You need to loop over the runners array to find the name of function and get the value of commands for that function.
Below is the working code:
package main
import (
"fmt"
"log"
"gopkg.in/yaml.v2"
)
var runContent = []byte(`
api_ver: 1
runners:
- name: function1
type:
- command: spawn child process
- command: build
- command: gulp
- name: function2
type:
- command: run function 1
- name: function3
type:
- command: ruby build
- name: function4
type:
- command: go build
`)
type Result struct {
Version string `yaml:"api_ver"`
Runners []Runners `yaml:"runners"`
}
type Runners struct {
Name string `yaml:"name"`
Type []Command `yaml:"type"`
}
type Command struct {
Command string `yaml:"command"`
}
func main() {
var runners Result
// parse mta yaml
err := yaml.Unmarshal(runContent, &runners)
if err != nil {
log.Fatalf("Error : %v", err)
}
commandList := getCommandList("function1", runners.Runners)
fmt.Printf("%+v", commandList)
}
func getCommandList(name string, runners []Runners) []Command {
var commandList []Command
for _, value := range runners {
if value.Name == name {
commandList = value.Type
}
}
return commandList
}
Playground example
I’ve a yaml like following which I need to parse using go.
When I tried to run the code with the parse I got an error.
Below is the code:
var runContent= []byte(`
- runners:
- name: function1
type: func1
- command: spawn child process
- command: build
- command: gulp
- name: function1
type: func2
- command: run function 1
- name: function3
type: func3
- command: ruby build
- name: function4
type: func4
- command: go build
`)
These are the types:
type Runners struct {
runners string `yaml:"runners"`
name string `yaml:”name”`
Type: string `yaml: ”type”`
command [] Command
}
type Command struct {
command string `yaml: ”command”`
}
runners := Runners{}
err = yaml.Unmarshal(runContent, &runners)
if err != nil {
log.Fatalf("Error : %v", err)
}
When I try to parse it I got an error invalid map , what could be missing here ?
The code you have posted contains multiple errors including the struct field Type. The yaml provided in your code is not valid. This will lead to err when unmarshalling the yaml into struct.
On unmarshalling yaml in go, It is required that:
The type of the decoded values should be compatible with the
respective values in out. If one or more values cannot be decoded due
to a type mismatches, decoding continues partially until the end of
the YAML content, and a *yaml.TypeError is returned with details for
all missed values.
Along with that:
Struct fields are only unmarshalled if they are exported (have an
upper case first letter), and are unmarshalled using the field name
lowercased as the default key.
Also there is an error in defining the yaml tags, which contains space. Custom keys may be defined via the "yaml" name in the field tag: the content preceding the first comma is used as the key.
type Runners struct {
runners string `yaml:"runners"` // fields should be exportable
name string `yaml:”name”`
Type: string `yaml: ”type”` // tags name should not have space in them.
command [] Command
}
To make the struct exportable convert the struct and fields into uppercase starting letter and remove space in yaml tag names:
type Runners struct {
Runners string `yaml:"runners"`
Name string `yaml:"name"`
Type string `yaml:"type"`
Command []Command
}
type Command struct {
Command string `yaml:"command"`
}
Modify the code as below to make it work.
package main
import (
"fmt"
"log"
"gopkg.in/yaml.v2"
)
var runContent = []byte(`
- runners:
- name: function1
type:
- command: spawn child process
- command: build
- command: gulp
- name: function1
type:
- command: run function 1
- name: function3
type:
- command: ruby build
- name: function4
type:
- command: go build
`)
type Runners []struct {
Runners []struct {
Type []struct {
Command string `yaml:"command"`
} `yaml:"type"`
Name string `yaml:"name"`
} `yaml:"runners"`
}
func main() {
runners := Runners{}
// parse mta yaml
err := yaml.Unmarshal(runContent, &runners)
if err != nil {
log.Fatalf("Error : %v", err)
}
fmt.Println(runners)
}
Playground example
Validate your yaml online here https://codebeautify.org/yaml-validator/cb92c85b
I have a program which parses yamls file to an objects (structs).
I use the following repo to do it
https://github.com/go-yaml/yaml
for example in the file I have:
dependency :
- name: ui
type: runner
cwd: /ui
install:
- name: api
group: test
And I use the following struct for it
type Dependency struct {
Name string
Type string
CWD string
Install []Install
//here I have the issue
Requires ?
}
type Install struct {
Name string
Group string
}
Now I have two few issue with a bit complex struct.
This is the entry which could be inside the Dependency struct and this is how it look in the yaml file
requires:
- name: db
- type: mongo
but it also can be
requires:
- name: db
- name: rst
- name: test
- name: test2
Since it have multiple name properties how should I got build this struct
In addition I've field in the yaml
_type-version: "1.0.0"
when I put it inside struct like following I got error since I use -
type TypeVer struct{
_Type-version string
}
How to overcome this?
The yaml package actually allows you to remap the the name for the properties, you can use this to handle your _type-version Property.
And your initial question: Just define Requires the same as Install:
package main
import (
"fmt"
"log"
"github.com/go-yaml/yaml"
)
type File struct {
TypeVersion string `yaml:"_type-version"`
Dependency []Dependency
}
type Dependency struct {
Name string
Type string
CWD string
Install []Install
Requires []Requires
}
type Install struct {
Name string
Group string
}
type Requires struct {
Name string
Type string
}
var data = `
_type-version: "1.0.0"
dependency:
- name: ui
type: runner
cwd: /ui
install:
- name: api
group: test
requires:
- name: db
- type: mongo
- name: rst
- name: test
- name: test2
`
func main() {
f := File{}
err := yaml.Unmarshal([]byte(data), &f)
if err != nil {
log.Fatalf("error: %v", err)
}
fmt.Printf("--- t:\n%v\n\n", f)
d, err := yaml.Marshal(&f)
if err != nil {
log.Fatalf("error: %v", err)
}
fmt.Printf("--- t dump:\n%s\n\n", string(d))
}
Having a problem parsing this sort of yaml file. Using "yaml.v2"
info: "abc"
data:
source: http://intra
destination: /tmp
run:
- id: "A1"
exe: "run.a1"
output: "output.A1"
- id: "A2"
exe: "run.a2"
output: "output.A2"
I would like to get all the values of the YAML file so I have a basic struct like this
type Config struct {
Info string
Data struct {
Source string `yaml:"source"`
Destination string `yaml:"destination"`
}
}
This works
But, I am not sure how to setup the struct for "run". The extra layer is confusing me.
type Run struct {
...
}
the OP's example of YAML is invalid. When value of run is list of dictionary it should be something like this:
info: "abc"
data:
source: http://intra
destination: /tmp
run:
- id: "A1"
exe: "run.a1"
output: "output.A1"
- id: "A2"
exe: "run.a2"
output: "output.A2"
And here's the corresponding data struture, and example for decoding YAML into golang's structure.
package main
import (
"fmt"
"io/ioutil"
"os"
yaml "gopkg.in/yaml.v2"
)
type Config struct {
Info string
Data struct {
Source string
Destination string
}
Run []struct {
Id string
Exe string
Output string
}
}
func main() {
var conf Config
reader, _ := os.Open("example.yaml")
buf, _ := ioutil.ReadAll(reader)
yaml.Unmarshal(buf, &conf)
fmt.Printf("%+v\n", conf)
}
running this will output (added some indent for readability):
{Info:abc
Data:{Source:http://intra Destination:/tmp}
Run:[{Id:A1 Exe:run.a1 Output:output.A1}
{Id:A2 Exe:run.a2 Output:output.A2}]