Parse a dynamic yaml file - go

I have a yaml file that is currently written as:
keys:
- key: secret/dog
values:
- username: shiba
- password: inu
- key: secret/cat
values:
- dbhost: localhost
- words: meow
However, this yaml file often changes, so new entries could be added with different values each time:
keys:
- key: secret/dog
values:
- username: shiba
- password: inu
- key: secret/cat
values:
- dbhost: localhost
- words: meow
- key: secret/mouse
values:
- color: white
- key: secret/clouds
values:
- type: fluffy
I know from go using the gopkg.in/yaml.v2 package, i can parse through the yaml file, if all the values are the same, for example:
type Secrets struct {
Keys []struct {
Key string `json:"key"`
Values []struct {
Username string `json:"username"`
Password string `json:"password"`
} `json:"values"`
} `json:"keys"`
}
func main() {
var secret Secrets
reader, err := os.Open("demo.yml")
if err != nil {
log.Fatal(err)
}
buf, _ := ioutil.ReadAll(reader)
yaml.Unmarshal(buf, &secret)
fmt.Printf("%+v\n", secret.Keys[1].Key)
}
In the example above, it would only work for the secret/dog key, but not the others.
How can I do this in Go, when new values are added to my yaml file often?
Thanks

If you don't now exact struct you should probably make your struct looking like this
type Secrets struct {
Keys []struct {
Key string `json:"key"`
Values []map[string]string `json:"values"`
} `json:"keys"`
}
It will parse your whole yaml and get all values, but it will be an array so you loosing type hinting on object. Other way arround would be advance endoding https://blog.gopheracademy.com/advent-2016/advanced-encoding-decoding/ but you will need to add new object every time new key/value pair will appear.

Related

How to clear struct values except certain fields with struct's method

I have a struct. I want to clear all fields except some public fields, e.g. Name, Gender, how to implement the function through method?
In my real code, I have many fields in the struct, so reset those sensitive fields manually is not my option.
type Agent struct {
Name string
Gender string
Secret1 string
Secret2 string
}
func (a *Agent) HideSecret() {
fmt.Println("Hidding secret...")
new := &Agent{
Name: a.Name,
Gender: a.Gender,
}
a = new
}
I tried a few combination of * and &, but it seems not working...
Please help.
James := Agent{
Name: "James Bond",
Gender: "M",
Secret1: "1234",
Secret2: "abcd",
}
fmt.Printf("[Before] Secret: %s, %s\n", James.Secret1, James.Secret2)
James.HideSecret()
fmt.Printf("[After] Secret: %s, %s\n", James.Secret1, James.Secret2) // not working
The golang playground is here: https://go.dev/play/p/ukJf2Fa0fPI
The receiver is a pointer. You have to update the object that the pointer points to:
func (a *Agent) HideSecret() {
fmt.Println("Hidding secret...")
cleaned := Agent{
Name: a.Name,
Gender: a.Gender,
}
*a=cleaned
}
If you want to just clear the fields, this is an easy solution. it saves some memory
func (a *Agent) HideSecret() {
fmt.Println("Hidding secret...")
a.Secret1 = ""
a.Secret2 = ""
}

How to iterate over yml sections with structs?

I use viper. I'm trying to get info from structs with yml-config.
type Config struct {
Account User `mapstructure:"user"`
}
type User struct {
Name string `mapstructure:"name"`
Contacts []Contact `mapstructure:"contact"`
}
type Contact struct {
Type string `mapstructure:"type"`
Value string `mapstructure:"value"`
}
func Init() *Config {
conf := new(Config)
viper.SetConfigType("yaml")
viper.ReadInConfig()
...
viper.Unmarshal(conf)
return conf
}
...
config := Init()
...
for _, contact := range config.Account.Contacts {
type := contact.type
vlaue := contact.value
}
And config.yml
user:
name: John
contacts:
email:
type: email
value: test#test.com
skype:
type: skype
value: skypeacc
Can I get structure items like this? I could not get contact data like that. Is it possible?
If I'm getting right what you want to achieve, and based on the for loop you provided;
What you actually need is a YAML sequence, which is an array. So your final YAML file should look like;
user:
name: John
contacts:
- type: email
value: test#test.com
- type: skype
value: skypeacc
- type: email
value: joe#example.com
Also, you have a typo in your tag on Contacts slice. It should match the YAML key;
type User struct {
Name string `mapstructure:"name"`
Contacts []Contact `mapstructure:"contacts"`
}
If you wish to keep the original YAML file structure you have to provide a tag (and the corresponding struct field) for each YAML key, so looping over it wouldn't be possible out of the box, because email and skype are parsed as struct fields. An example of the struct for the original YAML file would be;
type Config struct {
Account User `mapstructure:"user"`
}
type User struct {
Name string `mapstructure:"name"`
Contacts Contacts `mapstructure:"contacts"`
}
type Contacts struct {
Email Contact `mapstructure:"email"`
Skype Contact `mapstructure:"skype"`
}
type Contact struct {
Type string `mapstructure:"type"`
Value string `mapstructure:"value"`
}
I think the only significant problem is that in your data structures you've declared Contacts as a list, but in your YAML file it's a dictionary. If you structure your input file like this:
user:
name: John
contacts:
- type: email
value: test#test.com
- type: skype
value: skypeacc
Then you can read it like this:
package main
import (
"fmt"
"github.com/spf13/viper"
)
type Config struct {
User User
}
type User struct {
Name string
Contacts []Contact
}
type Contact struct {
Type string
Value string
}
func main() {
var cfg Config
viper.SetConfigName("config")
viper.AddConfigPath(".")
err := viper.ReadInConfig()
if err != nil {
panic(err)
}
viper.Unmarshal(&cfg)
fmt.Println("user: ", cfg.User.Name)
for _, contact := range cfg.User.Contacts {
fmt.Println(" ", contact.Type, ": ", contact.Value)
}
}
The above code is runnable as-is; you should just be able to drop it
into a file and build it. When I run the above example, I get as
output:
user: John
email : test#test.com
skype : skypeacc

How do I read this YAML?

I have a YAML configuration file like below
development: true
databases:
- label: masterdata
properties:
host: localhost
port: 54321
user: admin
password: nimda
dialect: postgres
dbname: business
- label: transactional
properties:
host: localhost
port: 54321
user: webuser
password: insecure
dialect: postgres
dbname: web
And structure like below
type ApplicationConfiguration struct {
development bool `yaml:"development"`
databases []Properties `yaml:"databases"`
}
type Properties struct {
Label string `yaml:"label"`
Values map[string]string `yaml:"properties"`
}
Using library gopkg.in/yaml.v2
Attempt to read this
config := ApplicationConfiguration{}
b, _ := ioutil.ReadFile(configPath)
marshalError := yaml.Unmarshal(b, &config)
But the value of databases in config is always nil and development always false, which means it not being unmarshalled at all. How can I read this configuration, may be I am using wrong struct or yaml needs change or a different API?
try to unmarshel it using the following struct:
type ApplicationConfiguration struct {
Development bool `yaml:"development"`
Databases []struct {
Label string `yaml:"label"`
Properties struct {
Host string `yaml:"host"`
Port int `yaml:"port"`
User string `yaml:"user"`
Password string `yaml:"password"`
Dialect string `yaml:"dialect"`
Dbname string `yaml:"dbname"`
} `yaml:"properties"`
} `yaml:"databases"`
if you need to unmarshal your props into map, try the followings:
using map[string]interface{}
type Properties struct {
Label string yaml:"label"
Values map[string]interface{} yaml:"properties"
}
marshal it into a struct and then using the following example:
type databases interface{}
var myMap map[string]interface{}
using struct and process it after:
if v.Kind() == reflect.Map {
for _, key := range v.MapKeys() {
strct := v.MapIndex(key)
fmt.Println(key.Interface(), strct.Interface())
}}

Error while parsing yaml in golang

I’ve a yaml like following which I need to parse using go.
When I tried to run the code with the parse I got an error.
Below is the code:
var runContent= []byte(`
- runners:
- name: function1
type: func1
- command: spawn child process
- command: build
- command: gulp
- name: function1
type: func2
- command: run function 1
- name: function3
type: func3
- command: ruby build
- name: function4
type: func4
- command: go build
`)
These are the types:
type Runners struct {
runners string `yaml:"runners"`
name string `yaml:”name”`
Type: string `yaml: ”type”`
command [] Command
}
type Command struct {
command string `yaml: ”command”`
}
runners := Runners{}
err = yaml.Unmarshal(runContent, &runners)
if err != nil {
log.Fatalf("Error : %v", err)
}
When I try to parse it I got an error invalid map , what could be missing here ?
The code you have posted contains multiple errors including the struct field Type. The yaml provided in your code is not valid. This will lead to err when unmarshalling the yaml into struct.
On unmarshalling yaml in go, It is required that:
The type of the decoded values should be compatible with the
respective values in out. If one or more values cannot be decoded due
to a type mismatches, decoding continues partially until the end of
the YAML content, and a *yaml.TypeError is returned with details for
all missed values.
Along with that:
Struct fields are only unmarshalled if they are exported (have an
upper case first letter), and are unmarshalled using the field name
lowercased as the default key.
Also there is an error in defining the yaml tags, which contains space. Custom keys may be defined via the "yaml" name in the field tag: the content preceding the first comma is used as the key.
type Runners struct {
runners string `yaml:"runners"` // fields should be exportable
name string `yaml:”name”`
Type: string `yaml: ”type”` // tags name should not have space in them.
command [] Command
}
To make the struct exportable convert the struct and fields into uppercase starting letter and remove space in yaml tag names:
type Runners struct {
Runners string `yaml:"runners"`
Name string `yaml:"name"`
Type string `yaml:"type"`
Command []Command
}
type Command struct {
Command string `yaml:"command"`
}
Modify the code as below to make it work.
package main
import (
"fmt"
"log"
"gopkg.in/yaml.v2"
)
var runContent = []byte(`
- runners:
- name: function1
type:
- command: spawn child process
- command: build
- command: gulp
- name: function1
type:
- command: run function 1
- name: function3
type:
- command: ruby build
- name: function4
type:
- command: go build
`)
type Runners []struct {
Runners []struct {
Type []struct {
Command string `yaml:"command"`
} `yaml:"type"`
Name string `yaml:"name"`
} `yaml:"runners"`
}
func main() {
runners := Runners{}
// parse mta yaml
err := yaml.Unmarshal(runContent, &runners)
if err != nil {
log.Fatalf("Error : %v", err)
}
fmt.Println(runners)
}
Playground example
Validate your yaml online here https://codebeautify.org/yaml-validator/cb92c85b

Only return specific fields from struct

I have a pair of structs like so:
type Datacenter struct {
Name string
Key string
Hosts []Host
}
type Host struct {
Name string
Port int
}
And then an example configuration file:
datacenters:
- name: dc1
key: test
hosts:
- name: dc1-host
port: 8200
- name: dc2
key: dc-test
hosts:
- name: dc2-host
port: 8200
I'm using viper to read the configuration file, here's the function:
func getDatacenters() []config.Datacenter {
err := viper.UnmarshalKey("datacenters", &datacenters)
if err != nil {
log.Error("Unable to read hosts key in config file: %s", err)
}
return datacenters
}
What I'd like to be able do now is specify an optional parameter, datacenter and if that's specified, to only return the keys from that datacenter. If the parameter is not specified, I'd like it to unmarshal and return the whole thing.
Is this possible?
EDIT: I should add, all I do with this so far is range over them:
for _, d := range datacenters {
for _, h := range d.Hosts {
}
}
So it may be there's a better way.

Resources