YAML forming via Golang - go

Please help to fo create correct YAML structure.
I need to receive something like this :
groups:
- name: Group1
targets:
- host1
- host2
- name: Group2
targets:
- host1
- host2
And I have wrote the next code which is working but not correctly :
type YamlConfig struct {
Groups struct {
Name string `yaml:"name"`
Targets []string `yaml:"targets"`
} `yaml:"groups"`
}
var config YamlConfig
var hosts []string = []string{"host1", "host2"}
for host := range hosts {
config.Groups.Name = "Group"+strconv.Itoa(host)
config.Groups.Targets = hosts
}
y, err := yaml.Marshal(config)
if err != nil {
fmt.Printf("Marshal: %v", err)
}
fmt.Println(string(y))
But this example is forming only this struct :
groups:
name: Group1
targets:
- host1
- host2
Please help to get the first result on correct way

You need to change your structure a little bit. As Group is a list, you need an array in go struct. Then when populating the data, create a new group and append to config.
package main
import (
"fmt"
"strconv"
"gopkg.in/yaml.v2"
)
func main() {
type Group struct {
Name string `yaml:"name"`
Targets []string `yaml:"targets"`
}
type YamlConfig struct {
Groups []Group `yaml:"groups"`
}
var config YamlConfig
var hosts []string = []string{"host1", "host2"}
for host := range hosts {
var group Group
group.Name = "Group" + strconv.Itoa(host)
group.Targets = hosts
config.Groups = append(config.Groups, group)
}
y, err := yaml.Marshal(config)
if err != nil {
fmt.Printf("Marshal: %v", err)
}
fmt.Println(string(y))
}

You also need an array/slice of Group so that your field "Groups" can have type []group or []Group. Do that with its own struct.
Something like:
type group struct {
Name string `yaml:"name"`
Targets []string `yaml:"targets"`
}
type YamlConfig struct {
Groups []group `yaml:"groups"`
}

Related

looping over struct and accessing array in golang

My issue is
I want to be able to loop over each server and within that loop, loop over UsageData
My current loop gives me an error about ranging over assets not sure why
I cant access assets.Server.UsageData
Here is my code : https://go.dev/play/p/ttNVW5_Q4Ys
package main
import (
"encoding/json"
"fmt"
)
type Asset struct {
Server struct {
Host string `json:"host"`
Port string `json:"port"`
} `json:"server"`
Postgres struct {
Host string `json:"host"`
User string `json:"user"`
Password string `json:"password"`
DB string `json:"db"`
UsageData []struct {
Region string `json:"Region"`
Mbps int `json:"Mpbs"`
} `json:"UsageData"`
} `json:"database"`
}
func main() {
jsonConfig := []byte(`[
{
"server":{
"host":"serverA",
"port":"8080"},
"database":{
"host":"serverA",
"user":"db_user",
"password":"supersecret",
"db":"A_db",
"UsageData":[{"Region":"US","Mbps":100}, {"Region":"EU","Mbps":140}]
}
},
{
"server":{
"host":"serverB",
"port":"8383"},
"database":{
"host":"serverB",
"user":"db_user2",
"password":"acbd123",
"db":"B_db",
"UsageData":[{"Region":"US","Mbps":58}, {"Region":"EU","Mbps":250}]
}
}
]`)
var assets []Asset
err := json.Unmarshal(jsonConfig, &assets)
if err != nil {
panic(err)
}
fmt.Printf("Assets: %+v\n", assets)
//fmt.Printf("Config: %+v\n", assets.Server.Host)
//fmt.Printf("Config: %+v\n", assets.database.UsageData)
//fmt.Printf("Config: %+v\n", assets.Server.UsageData)
for _, asset := range assets {
fmt.Printf("%v\n", asset)
//for _, Usage := range assets.UsageData {
// fmt.Printf("%v\n",Usage)
//}
}
}
** Code with the correct answer, I was calling the nested struct incorrectly**
https://go.dev/play/p/tEbA405WWbC
Provided jsonConfig is not technically incorrect but keys SHOULD be unique (see question).
It seems that encoding/json will override value with last occurrence.
So you have 2 options:
change jsonConfig to [{asset1}, {asset2}] (see fixed playground)
implement custom unmarshaler
I definitely recommend first option.

I am trying to unmarshall the following yaml into the below structs, how do I convert the List field into a map field and access it using struct?

The YAML file:
namespaces:
- namespace: default
aliasname: k8s
components:
- component: comp1
replicas: 1
port: 8080
- component: comp2
replicas: 1
port: 9999
- namespace: ns2
components:
- component: comp1
replicas: 1
From the YAML file above, I want to create structs like the following:
type Namespaces struct {
NamespaceName string `yaml:"namespace"`
Aliasname string `yaml:"aliasname,omitempty"`
ListOfComponents []Components `yaml:"components"`
ComponentMap map[string]Components
}
type Components struct {
ComponentName string `yaml:"component"`
NumReplicas int `yaml:"replicas"`
Port int `yaml:"port"`
}
type Config struct {
ListOfNamespaces []Namespaces `yaml:"namespaces"`
NamespaceMap map[string]Namespaces
}
The fields Namespacemap and Componentmap should be able to be retrieved when accessing the config and namespace object respectively. I created a method to convert the list of the namespaces and components into maps, but when I call config.Namespacemap or Namespace.ComponentMap, it returns an empty map.
Basically I would like to know: How do we add extra fields to type structs? I would like to access new variables like a map from the config struct.
Update:
Thanks blami for guiding me, but when I try to write the same for Components, it doesn't give me the whole namespacemap with the componentmap included:
type Components struct {
ComponentName string `yaml:"component"`
NumReplicas int `yaml:"replicas"`
Port int `yaml:"port"`
}
type Namespaces struct {
NamespaceName string `yaml:"namespace"`
Aliasname string `yaml:"aliasname"`
ComponentMap map[string]Components `yaml:"components"`
}
func (n *Namespaces) UnmarshalYAML(unmarshal func(interface{}) error) error {
type origNamespace struct {
ListOfComponents []Components `yaml:"components"`
}
var on origNamespace
err1 := unmarshal(&on)
if err1 != nil {
return err1
}
n.ComponentMap = make(map[string]Components)
for _, i := range on.ListOfComponents {
n.ComponentMap[i.ComponentName] = i
}
return nil
}
When I run the config.NamespaceMap it gives the following
map[:{NamespaceName: K8sNamespace: ComponentMap:map[comp1:{ComponentName:comp1 NumShards:0 NumReplicas:1 Port:0 EpochLength:0}]}]
If you want to do such transformation you will need to write a customized UnmarshalYAML() receiver on types Config and Namespace. Here is basic working example of doing so (only for "namespaces"):
type Config struct {
Namespaces map[string]Namespace `yaml:"namespaces"`
}
// Replace go-yaml built-in unmarshaller with custom that will transform list to map.
// Note this code is meant only as demonstration!
func (c *Config) UnmarshalYAML(unmarshal func(interface{}) error) error {
type origConfig struct {
Namespaces []Namespace `yaml:"namespaces"`
}
var o origConfig
err := unmarshal(&o)
if err != nil {
return err
}
// Assign namespaces in list to map; namespaces with same name will be overwritten
c.Namespaces = make(map[string]Namespace)
for _, n := range o.Namespaces {
c.Namespaces[n.Namespace] = n
}
return nil
}
// You can use "Config" type as usual
func main() {
var config Config
err := yaml.Unmarshal(<your_yaml>, &config)
if err != nil {
log.Panic(err)
}
fmt.Println(config.Namespaces)
// map[default:{default k8s} ns2:{ns2 }]
fmt.Printf("%v\n", config.Namespaces["default"])
// {default k8s}
}
As noted in code example this will cause some problems (e.g. what to do if namespace names are same?).
Playground link: https://go.dev/play/p/IKg8kmRnknq

Why is the unmarshaled YAML data empty?

I have a problem unmarshaling a simple slice of YAML data:
package main
import (
"fmt"
"gopkg.in/yaml.v2"
)
type myDataStruct struct {
HTTP []struct {
Name string
Target string
}
}
func main() {
yamlData := `
HTTP:
- name: one
target: http://wazaa
- name: two
target: http://wazii
`
var myData myDataStruct
err := yaml.Unmarshal([]byte(yamlData), &myData)
if err != nil {
fmt.Printf("error: %v", err)
return
}
fmt.Print(myData)
}
Playground: https://play.golang.org/p/Srb2DJVVZqN
The result is {[]} and my limited knowledge of Go does not help to understand why?
If you don't specify the mapping between labels used in the YAML source and Go struct fields, by default they will only be matched if only changing the first letter to lower matches.
E.g. the struct field Name will match name, but not NAME.
Specify the mapping for the HTTP field:
type myDataStruct struct {
HTTP []struct {
Name string
Target string
} `yaml:"HTTP"`
}
With this change it works and outputs (try it on the Go Playground):
{[{one http://wazaa} {two http://wazii}]}
It's good practice to provide mappings for all fields, so it'll continue to work if you rename the fields:
type myDataStruct struct {
HTTP []struct {
Name string `yaml:"name"`
Target string `yaml:"target"`
} `yaml:"HTTP"`
}
Although in your case the default matching works for the Name and Target fields without providing the mappings.

Create function that require other repeatbale logic as pre-requiste (clean code)

I've the following yaml file which I need to parse (the parse is working as expected) and need to provide a datafrom the yaml file content that should be exposed by the following decoupled functions
I need to provide the following functions (here is example of some of those functions, need more with the same pattern...)
getApps()
getServices()
GetApp(appname)
GetServiceForApp(appname)
This is the code (which works...)
var DMZ = []byte(`
applications:
- name: app1
type: php
src: /app1
host: us
use:
- redis
- mysql
- name: app2
type: rust
src: /app2
host: eu
use:
- mongo
- mysql
- name: app3
type: golang
src: /app3
host: us
use:
- postgress
- mysql
services:
- name: mongo
type: db
host: us
- name: mysql
type: db
host: eu
- name: postgress
type: db
host: us
- name: redis
type: db
host: us
`)
This is the structs
type DMZ struct {
Applications []*Applications `yaml:"applications,omitempty"`
Services []*Services `yaml:"services,omitempty"`
}
type Applications struct {
Name string
Type string
Src string `yaml:"src,omitempty"`
use []Use `yaml:"use,omitempty"`
}
type Services struct {
Name string
Type string
Host string `yaml:"host,omitempty"`
}
type Use struct {
Name string `yaml:"name,omitempty"`
host string `yaml:"host,omitempty"`
Type string `yaml:"type,omitempty"`
}
// Parse file
func Parse(yamlContent []byte) (out DMZ, err error) {
dmz := DMZ{}
err = yaml.Unmarshal([]byte(yamlContent), &dmz)
if err != nil {
logs.Error("Yaml file is not valid, Error: " + err.Error())
}
return dmz, err
}
As the Parse function is a per-requite to all the needed functions (which I listed above) I wonder how is the best to create them ,create simple function that every time call to the parse function and then do the logic (not a problem) but I wonder if there is better approach which follows the clean code principles for Golang , with 'interface / dependency injections' ?
UPDATE:
I want to avoid doing things like following, assume that I need to call to those function from different packages or even different GitHub repository how it's best to do it with Golang clean code.
func getApps(){
dmz := Parse()
....
}
func getServices(){
dmz := Parse()
....
}
func getApp(appname string){
dmz := Parse()
....
}
func GetServiceForApp(appname string){
dmz := Parse()
....
}
And I need more functions with the same pattern ...
What I need some Clean Code solution using interface/dependency injection like a best practice code example in Golang
If something is not clear please let me know :)
Parse the value inside pointer type struct value which will store the value inside the struct pointed by a pointer. Then Create method receives on All the methods from which you want to get the value of an app or service inside the struct.
Using pointer receiver in all methods you will be able to access the original struct which is updated when parsing the yaml.
Create a pointer to empty instance of the struct and pass it as a method receiver. Then access that struct in unmarshal to update yaml data in original struct. Access the updated struct in every function by accessing the value of original struct using pointer receiver to method.
package main
import (
"fmt"
"log"
yaml "gopkg.in/yaml.v2"
)
var dmz = []byte(`
applications:
- name: app1
type: php
src: /app1
host: us
use:
- redis
- mysql
- name: app2
type: rust
src: /app2
host: eu
use:
- mongo
- mysql
- name: app3
type: golang
src: /app3
host: us
use:
- postgress
- mysql
services:
- name: mongo
type: db
host: us
- name: mysql
type: db
host: eu
- name: postgress
type: db
host: us
- name: redis
type: db
host: us
`)
type DMZ struct {
Applications []*Applications `yaml:"applications,omitempty"`
Services []*Services `yaml:"services,omitempty"`
}
type Applications struct {
Name string
Type string
Src string `yaml:"src,omitempty"`
use []Use `yaml:"use,omitempty"`
}
type Services struct {
Name string
Type string
Host string `yaml:"host,omitempty"`
}
type Use struct {
Name string `yaml:"name,omitempty"`
host string `yaml:"host,omitempty"`
Type string `yaml:"type,omitempty"`
}
func main() {
dm := &DMZ{}
result, err := dm.Parse(dmz)
dm.getApp("app1")
if err != nil {
fmt.Println(err)
}
fmt.Println(result)
fmt.Println(dm.getApp("app2"))
fmt.Println(dm.GetServiceForApp("mongo"))
}
func (dmz *DMZ) getApps() []*Applications {
return dmz.Applications
}
func (dmz *DMZ) getServices() []*Services {
return dmz.Services
}
func (dmz *DMZ) getApp(appname string) *Applications {
for _, value := range dmz.Applications {
if appname == value.Name {
fmt.Println((*value).Name)
return value
}
}
return nil
}
func (dmz *DMZ) GetServiceForApp(appname string) *Services {
for _, value := range dmz.Services {
if appname == value.Name {
return value
}
}
return nil
}
// Parse file
func (dmz *DMZ) Parse(yamlContent []byte) (out *DMZ, err error) {
err = yaml.Unmarshal([]byte(yamlContent), &dmz)
if err != nil {
log.Fatal("Yaml file is not valid, Error: " + err.Error())
}
return dmz, err
}
Working code on Playground
If you want your code to be more clean then you can also skip the struct returned from parse function. Since we are passing a pointer type receiver and updating the original struct as:
package main
import (
"fmt"
"log"
yaml "gopkg.in/yaml.v2"
)
var dmz = []byte(`
applications:
- name: app1
type: php
src: /app1
host: us
use:
- redis
- mysql
- name: app2
type: rust
src: /app2
host: eu
use:
- mongo
- mysql
- name: app3
type: golang
src: /app3
host: us
use:
- postgress
- mysql
services:
- name: mongo
type: db
host: us
- name: mysql
type: db
host: eu
- name: postgress
type: db
host: us
- name: redis
type: db
host: us
`)
type DMZ struct {
Applications []*Applications `yaml:"applications,omitempty"`
Services []*Services `yaml:"services,omitempty"`
}
type Applications struct {
Name string
Type string
Src string `yaml:"src,omitempty"`
use []Use `yaml:"use,omitempty"`
}
type Services struct {
Name string
Type string
Host string `yaml:"host,omitempty"`
}
type Use struct {
Name string `yaml:"name,omitempty"`
host string `yaml:"host,omitempty"`
Type string `yaml:"type,omitempty"`
}
func main() {
dm := &DMZ{}
dm.Parse(dmz)
fmt.Println(dm.getApp("app2"))
fmt.Println(dm.GetServiceForApp("mongo"))
}
func (dmz *DMZ) getApps() []*Applications {
return dmz.Applications
}
func (dmz *DMZ) getServices() []*Services {
return dmz.Services
}
func (dmz *DMZ) getApp(appname string) *Applications {
for _, value := range dmz.Applications {
if appname == value.Name {
fmt.Println((*value).Name)
return value
}
}
return nil
}
func (dmz *DMZ) GetServiceForApp(appname string) *Services {
for _, value := range dmz.Services {
if appname == value.Name {
return value
}
}
return nil
}
// Parse file
func (dmz *DMZ) Parse(yamlContent []byte) {
if err := yaml.Unmarshal([]byte(yamlContent), &dmz); err != nil {
log.Fatal("Yaml file is not valid, Error: " + err.Error())
}
}
As we can notice that Parse function will become more clean since we are not returning anything from it we are just updating the original struct using method receiver which is a much better way to achieve what you have been trying to achieve.
You can also choose to implement an interface by defining the methods in struct as receiver for which you are trying to implement the interface as:
package main
import (
"fmt"
"log"
yaml "gopkg.in/yaml.v2"
)
type DI interface {
GetApps() []*Applications
GetServices() *Services
}
var dmz = []byte(`
applications:
- name: app1
type: php
src: /app1
host: us
use:
- redis
- mysql
- name: app2
type: rust
src: /app2
host: eu
use:
- mongo
- mysql
- name: app3
type: golang
src: /app3
host: us
use:
- postgress
- mysql
services:
- name: mongo
type: db
host: us
- name: mysql
type: db
host: eu
- name: postgress
type: db
host: us
- name: redis
type: db
host: us
`)
type DMZ struct {
Applications []*Applications `yaml:"applications,omitempty"`
Services []*Services `yaml:"services,omitempty"`
}
type Applications struct {
Name string
Type string
Src string `yaml:"src,omitempty"`
use []Use `yaml:"use,omitempty"`
}
type Services struct {
Name string
Type string
Host string `yaml:"host,omitempty"`
}
type Use struct {
Name string `yaml:"name,omitempty"`
host string `yaml:"host,omitempty"`
Type string `yaml:"type,omitempty"`
}
func main() {
dm := &DMZ{}
dm.Parse(dmz)
fmt.Println(dm.getApp("app2"))
fmt.Println(dm.GetServiceForApp("mongo"))
}
func (dmz *DMZ) GetApps() []*Applications {
return dmz.Applications
}
func (dmz *DMZ) GetServices() []*Services {
return dmz.Services
}
func (dmz *DMZ) getApp(appname string) *Applications {
for _, value := range dmz.Applications {
if appname == value.Name {
fmt.Println((*value).Name)
return value
}
}
return nil
}
func (dmz *DMZ) GetServiceForApp(appname string) *Services {
for _, value := range dmz.Services {
if appname == value.Name {
return value
}
}
return nil
}
// Parse file
func (dmz *DMZ) Parse(yamlContent []byte) {
if err := yaml.Unmarshal([]byte(yamlContent), &dmz); err != nil {
log.Fatal("Yaml file is not valid, Error: " + err.Error())
}
}
Note:
Generics are convenient but they come at a cost in complexity in the
type system and run-time. We haven't yet found a design that gives
value proportionate to the complexity, although we continue to think
about it. Meanwhile, Go's built-in maps and slices, plus the ability
to use the empty interface to construct containers (with explicit
unboxing) mean in many cases it is possible to write code that does
what generics would enable, if less smoothly.
You can define an interface and provide implementations in the struct
type DMZI interface {
GetApps() []Application
GetService() []Service
GetApp(name string) (Application, error)
GetServiceForApp(name string) ([]string, error)
}
type DMZ struct {
Application []Application `yaml:"applications,omitempty"`
Service []Service `yaml:"services,omitempty"`
}
func (dmz DMZ) GetApps() []Application {
return dmz.Application
}
func (dmz DMZ) GetService() []Service {
return dmz.Service
}
func (dmz DMZ) GetApp(name string) (Application, error) {
for _, app := range dmz.Application {
if app.Name == name {
return app, nil
}
}
return Application{}, fmt.Errorf("Did not find application with name %s", name)
}
func (dmz DMZ) GetServiceForApp(name string) ([]string, error) {
app, err := dmz.GetApp(name)
if err != nil {
return []string{}, err
}
return app.Use, nil
}
type Application struct {
Name string
Type string
Src string `yaml:"src,omitempty"`
Use []string `yaml:"use,omitempty"`
}
type Service struct {
Name string
Type string
Host string `yaml:"host,omitempty"`
}
// Parse file
func Parse(yamlContent []byte) (out DMZI, err error) {
dmz := DMZ{}
err = yaml.Unmarshal([]byte(yamlContent), &dmz)
if err != nil {
fmt.Println("Yaml file is not valid, Error: " + err.Error())
}
return dmz, err
}
So, you can call the methods on the returned interface e.g.
fmt.Printf("Apps : %+v\n", dmz.GetApps())
fmt.Printf("Service : %+v\n", dmz.GetService())
UPDATE
main method as requested in the comment
func main() {
dmz, err := Parse([]byte(ymlStr))
if err != nil {
panic(err)
}
fmt.Printf("Apps : %+v\n", dmz.GetApps())
fmt.Printf("Service : %+v\n", dmz.GetService())
}
Will print
Apps : [{Name:app1 Type:php Src:/app1 Use:[redis mysql]} {Name:app2 Type:rust Src:/app2 Use:[mongo mysql]} {Name:app3 Type:golang Src:/app3 Use:[postgress mysql]}]
Service : [{Name:mongo Type:db Host:us} {Name:mysql Type:db Host:eu} {Name:postgress Type:db Host:us} {Name:redis Type:db Host:us}]
I've changed your code a little and done what I think you are asking for, which is to always call "Parse" whenever you create a new instance of DMZ. This is how you would do this using Dargo. Look at the bottom to see how the code would bind DMZ into a ServiceLocator and how to get new instances of DMZ
type DMZ struct {
Applications []*Applications `yaml:"applications,omitempty"`
Services []*Services `yaml:"services,omitempty"`
}
// This method is called every time DMZ is created via the Dargo API
func (dmz *DMZ) DargoInitialize(ioc.Descriptor) error {
// get ur bytes from... wherever
Parse(dmz, []byte{})
}
type Applications struct {
Name string
Type string
Src string `yaml:"src,omitempty"`
use []Use `yaml:"use,omitempty"`
}
type Services struct {
Name string
Type string
Host string `yaml:"host,omitempty"`
}
type Use struct {
Name string `yaml:"name,omitempty"`
host string `yaml:"host,omitempty"`
Type string `yaml:"type,omitempty"`
}
// Parse file
func Parse(dmz *DMZ, yamlContent []byte) (out *DMZ, err error) {
err = yaml.Unmarshal([]byte(yamlContent), &dmz)
if err != nil {
logs.Error("Yaml file is not valid, Error: " + err.Error())
}
return dmz, err
}
func useDargo() error {
locator, err := ioc.CreateAndBind("example", func(binder ioc.Binder) error {
binder.Bind("DMZ", &DMZ{}).InScope(ioc.PerLookup)
})
if err != nil {
return err
}
// This is how you would get your instances of DMZ
locator.GetDService("DMZ")
}

Only return specific fields from struct

I have a pair of structs like so:
type Datacenter struct {
Name string
Key string
Hosts []Host
}
type Host struct {
Name string
Port int
}
And then an example configuration file:
datacenters:
- name: dc1
key: test
hosts:
- name: dc1-host
port: 8200
- name: dc2
key: dc-test
hosts:
- name: dc2-host
port: 8200
I'm using viper to read the configuration file, here's the function:
func getDatacenters() []config.Datacenter {
err := viper.UnmarshalKey("datacenters", &datacenters)
if err != nil {
log.Error("Unable to read hosts key in config file: %s", err)
}
return datacenters
}
What I'd like to be able do now is specify an optional parameter, datacenter and if that's specified, to only return the keys from that datacenter. If the parameter is not specified, I'd like it to unmarshal and return the whole thing.
Is this possible?
EDIT: I should add, all I do with this so far is range over them:
for _, d := range datacenters {
for _, h := range d.Hosts {
}
}
So it may be there's a better way.

Resources