The YAML file:
namespaces:
- namespace: default
aliasname: k8s
components:
- component: comp1
replicas: 1
port: 8080
- component: comp2
replicas: 1
port: 9999
- namespace: ns2
components:
- component: comp1
replicas: 1
From the YAML file above, I want to create structs like the following:
type Namespaces struct {
NamespaceName string `yaml:"namespace"`
Aliasname string `yaml:"aliasname,omitempty"`
ListOfComponents []Components `yaml:"components"`
ComponentMap map[string]Components
}
type Components struct {
ComponentName string `yaml:"component"`
NumReplicas int `yaml:"replicas"`
Port int `yaml:"port"`
}
type Config struct {
ListOfNamespaces []Namespaces `yaml:"namespaces"`
NamespaceMap map[string]Namespaces
}
The fields Namespacemap and Componentmap should be able to be retrieved when accessing the config and namespace object respectively. I created a method to convert the list of the namespaces and components into maps, but when I call config.Namespacemap or Namespace.ComponentMap, it returns an empty map.
Basically I would like to know: How do we add extra fields to type structs? I would like to access new variables like a map from the config struct.
Update:
Thanks blami for guiding me, but when I try to write the same for Components, it doesn't give me the whole namespacemap with the componentmap included:
type Components struct {
ComponentName string `yaml:"component"`
NumReplicas int `yaml:"replicas"`
Port int `yaml:"port"`
}
type Namespaces struct {
NamespaceName string `yaml:"namespace"`
Aliasname string `yaml:"aliasname"`
ComponentMap map[string]Components `yaml:"components"`
}
func (n *Namespaces) UnmarshalYAML(unmarshal func(interface{}) error) error {
type origNamespace struct {
ListOfComponents []Components `yaml:"components"`
}
var on origNamespace
err1 := unmarshal(&on)
if err1 != nil {
return err1
}
n.ComponentMap = make(map[string]Components)
for _, i := range on.ListOfComponents {
n.ComponentMap[i.ComponentName] = i
}
return nil
}
When I run the config.NamespaceMap it gives the following
map[:{NamespaceName: K8sNamespace: ComponentMap:map[comp1:{ComponentName:comp1 NumShards:0 NumReplicas:1 Port:0 EpochLength:0}]}]
If you want to do such transformation you will need to write a customized UnmarshalYAML() receiver on types Config and Namespace. Here is basic working example of doing so (only for "namespaces"):
type Config struct {
Namespaces map[string]Namespace `yaml:"namespaces"`
}
// Replace go-yaml built-in unmarshaller with custom that will transform list to map.
// Note this code is meant only as demonstration!
func (c *Config) UnmarshalYAML(unmarshal func(interface{}) error) error {
type origConfig struct {
Namespaces []Namespace `yaml:"namespaces"`
}
var o origConfig
err := unmarshal(&o)
if err != nil {
return err
}
// Assign namespaces in list to map; namespaces with same name will be overwritten
c.Namespaces = make(map[string]Namespace)
for _, n := range o.Namespaces {
c.Namespaces[n.Namespace] = n
}
return nil
}
// You can use "Config" type as usual
func main() {
var config Config
err := yaml.Unmarshal(<your_yaml>, &config)
if err != nil {
log.Panic(err)
}
fmt.Println(config.Namespaces)
// map[default:{default k8s} ns2:{ns2 }]
fmt.Printf("%v\n", config.Namespaces["default"])
// {default k8s}
}
As noted in code example this will cause some problems (e.g. what to do if namespace names are same?).
Playground link: https://go.dev/play/p/IKg8kmRnknq
I've the following yaml file which I need to parse (the parse is working as expected) and need to provide a datafrom the yaml file content that should be exposed by the following decoupled functions
I need to provide the following functions (here is example of some of those functions, need more with the same pattern...)
getApps()
getServices()
GetApp(appname)
GetServiceForApp(appname)
This is the code (which works...)
var DMZ = []byte(`
applications:
- name: app1
type: php
src: /app1
host: us
use:
- redis
- mysql
- name: app2
type: rust
src: /app2
host: eu
use:
- mongo
- mysql
- name: app3
type: golang
src: /app3
host: us
use:
- postgress
- mysql
services:
- name: mongo
type: db
host: us
- name: mysql
type: db
host: eu
- name: postgress
type: db
host: us
- name: redis
type: db
host: us
`)
This is the structs
type DMZ struct {
Applications []*Applications `yaml:"applications,omitempty"`
Services []*Services `yaml:"services,omitempty"`
}
type Applications struct {
Name string
Type string
Src string `yaml:"src,omitempty"`
use []Use `yaml:"use,omitempty"`
}
type Services struct {
Name string
Type string
Host string `yaml:"host,omitempty"`
}
type Use struct {
Name string `yaml:"name,omitempty"`
host string `yaml:"host,omitempty"`
Type string `yaml:"type,omitempty"`
}
// Parse file
func Parse(yamlContent []byte) (out DMZ, err error) {
dmz := DMZ{}
err = yaml.Unmarshal([]byte(yamlContent), &dmz)
if err != nil {
logs.Error("Yaml file is not valid, Error: " + err.Error())
}
return dmz, err
}
As the Parse function is a per-requite to all the needed functions (which I listed above) I wonder how is the best to create them ,create simple function that every time call to the parse function and then do the logic (not a problem) but I wonder if there is better approach which follows the clean code principles for Golang , with 'interface / dependency injections' ?
UPDATE:
I want to avoid doing things like following, assume that I need to call to those function from different packages or even different GitHub repository how it's best to do it with Golang clean code.
func getApps(){
dmz := Parse()
....
}
func getServices(){
dmz := Parse()
....
}
func getApp(appname string){
dmz := Parse()
....
}
func GetServiceForApp(appname string){
dmz := Parse()
....
}
And I need more functions with the same pattern ...
What I need some Clean Code solution using interface/dependency injection like a best practice code example in Golang
If something is not clear please let me know :)
Parse the value inside pointer type struct value which will store the value inside the struct pointed by a pointer. Then Create method receives on All the methods from which you want to get the value of an app or service inside the struct.
Using pointer receiver in all methods you will be able to access the original struct which is updated when parsing the yaml.
Create a pointer to empty instance of the struct and pass it as a method receiver. Then access that struct in unmarshal to update yaml data in original struct. Access the updated struct in every function by accessing the value of original struct using pointer receiver to method.
package main
import (
"fmt"
"log"
yaml "gopkg.in/yaml.v2"
)
var dmz = []byte(`
applications:
- name: app1
type: php
src: /app1
host: us
use:
- redis
- mysql
- name: app2
type: rust
src: /app2
host: eu
use:
- mongo
- mysql
- name: app3
type: golang
src: /app3
host: us
use:
- postgress
- mysql
services:
- name: mongo
type: db
host: us
- name: mysql
type: db
host: eu
- name: postgress
type: db
host: us
- name: redis
type: db
host: us
`)
type DMZ struct {
Applications []*Applications `yaml:"applications,omitempty"`
Services []*Services `yaml:"services,omitempty"`
}
type Applications struct {
Name string
Type string
Src string `yaml:"src,omitempty"`
use []Use `yaml:"use,omitempty"`
}
type Services struct {
Name string
Type string
Host string `yaml:"host,omitempty"`
}
type Use struct {
Name string `yaml:"name,omitempty"`
host string `yaml:"host,omitempty"`
Type string `yaml:"type,omitempty"`
}
func main() {
dm := &DMZ{}
result, err := dm.Parse(dmz)
dm.getApp("app1")
if err != nil {
fmt.Println(err)
}
fmt.Println(result)
fmt.Println(dm.getApp("app2"))
fmt.Println(dm.GetServiceForApp("mongo"))
}
func (dmz *DMZ) getApps() []*Applications {
return dmz.Applications
}
func (dmz *DMZ) getServices() []*Services {
return dmz.Services
}
func (dmz *DMZ) getApp(appname string) *Applications {
for _, value := range dmz.Applications {
if appname == value.Name {
fmt.Println((*value).Name)
return value
}
}
return nil
}
func (dmz *DMZ) GetServiceForApp(appname string) *Services {
for _, value := range dmz.Services {
if appname == value.Name {
return value
}
}
return nil
}
// Parse file
func (dmz *DMZ) Parse(yamlContent []byte) (out *DMZ, err error) {
err = yaml.Unmarshal([]byte(yamlContent), &dmz)
if err != nil {
log.Fatal("Yaml file is not valid, Error: " + err.Error())
}
return dmz, err
}
Working code on Playground
If you want your code to be more clean then you can also skip the struct returned from parse function. Since we are passing a pointer type receiver and updating the original struct as:
package main
import (
"fmt"
"log"
yaml "gopkg.in/yaml.v2"
)
var dmz = []byte(`
applications:
- name: app1
type: php
src: /app1
host: us
use:
- redis
- mysql
- name: app2
type: rust
src: /app2
host: eu
use:
- mongo
- mysql
- name: app3
type: golang
src: /app3
host: us
use:
- postgress
- mysql
services:
- name: mongo
type: db
host: us
- name: mysql
type: db
host: eu
- name: postgress
type: db
host: us
- name: redis
type: db
host: us
`)
type DMZ struct {
Applications []*Applications `yaml:"applications,omitempty"`
Services []*Services `yaml:"services,omitempty"`
}
type Applications struct {
Name string
Type string
Src string `yaml:"src,omitempty"`
use []Use `yaml:"use,omitempty"`
}
type Services struct {
Name string
Type string
Host string `yaml:"host,omitempty"`
}
type Use struct {
Name string `yaml:"name,omitempty"`
host string `yaml:"host,omitempty"`
Type string `yaml:"type,omitempty"`
}
func main() {
dm := &DMZ{}
dm.Parse(dmz)
fmt.Println(dm.getApp("app2"))
fmt.Println(dm.GetServiceForApp("mongo"))
}
func (dmz *DMZ) getApps() []*Applications {
return dmz.Applications
}
func (dmz *DMZ) getServices() []*Services {
return dmz.Services
}
func (dmz *DMZ) getApp(appname string) *Applications {
for _, value := range dmz.Applications {
if appname == value.Name {
fmt.Println((*value).Name)
return value
}
}
return nil
}
func (dmz *DMZ) GetServiceForApp(appname string) *Services {
for _, value := range dmz.Services {
if appname == value.Name {
return value
}
}
return nil
}
// Parse file
func (dmz *DMZ) Parse(yamlContent []byte) {
if err := yaml.Unmarshal([]byte(yamlContent), &dmz); err != nil {
log.Fatal("Yaml file is not valid, Error: " + err.Error())
}
}
As we can notice that Parse function will become more clean since we are not returning anything from it we are just updating the original struct using method receiver which is a much better way to achieve what you have been trying to achieve.
You can also choose to implement an interface by defining the methods in struct as receiver for which you are trying to implement the interface as:
package main
import (
"fmt"
"log"
yaml "gopkg.in/yaml.v2"
)
type DI interface {
GetApps() []*Applications
GetServices() *Services
}
var dmz = []byte(`
applications:
- name: app1
type: php
src: /app1
host: us
use:
- redis
- mysql
- name: app2
type: rust
src: /app2
host: eu
use:
- mongo
- mysql
- name: app3
type: golang
src: /app3
host: us
use:
- postgress
- mysql
services:
- name: mongo
type: db
host: us
- name: mysql
type: db
host: eu
- name: postgress
type: db
host: us
- name: redis
type: db
host: us
`)
type DMZ struct {
Applications []*Applications `yaml:"applications,omitempty"`
Services []*Services `yaml:"services,omitempty"`
}
type Applications struct {
Name string
Type string
Src string `yaml:"src,omitempty"`
use []Use `yaml:"use,omitempty"`
}
type Services struct {
Name string
Type string
Host string `yaml:"host,omitempty"`
}
type Use struct {
Name string `yaml:"name,omitempty"`
host string `yaml:"host,omitempty"`
Type string `yaml:"type,omitempty"`
}
func main() {
dm := &DMZ{}
dm.Parse(dmz)
fmt.Println(dm.getApp("app2"))
fmt.Println(dm.GetServiceForApp("mongo"))
}
func (dmz *DMZ) GetApps() []*Applications {
return dmz.Applications
}
func (dmz *DMZ) GetServices() []*Services {
return dmz.Services
}
func (dmz *DMZ) getApp(appname string) *Applications {
for _, value := range dmz.Applications {
if appname == value.Name {
fmt.Println((*value).Name)
return value
}
}
return nil
}
func (dmz *DMZ) GetServiceForApp(appname string) *Services {
for _, value := range dmz.Services {
if appname == value.Name {
return value
}
}
return nil
}
// Parse file
func (dmz *DMZ) Parse(yamlContent []byte) {
if err := yaml.Unmarshal([]byte(yamlContent), &dmz); err != nil {
log.Fatal("Yaml file is not valid, Error: " + err.Error())
}
}
Note:
Generics are convenient but they come at a cost in complexity in the
type system and run-time. We haven't yet found a design that gives
value proportionate to the complexity, although we continue to think
about it. Meanwhile, Go's built-in maps and slices, plus the ability
to use the empty interface to construct containers (with explicit
unboxing) mean in many cases it is possible to write code that does
what generics would enable, if less smoothly.
You can define an interface and provide implementations in the struct
type DMZI interface {
GetApps() []Application
GetService() []Service
GetApp(name string) (Application, error)
GetServiceForApp(name string) ([]string, error)
}
type DMZ struct {
Application []Application `yaml:"applications,omitempty"`
Service []Service `yaml:"services,omitempty"`
}
func (dmz DMZ) GetApps() []Application {
return dmz.Application
}
func (dmz DMZ) GetService() []Service {
return dmz.Service
}
func (dmz DMZ) GetApp(name string) (Application, error) {
for _, app := range dmz.Application {
if app.Name == name {
return app, nil
}
}
return Application{}, fmt.Errorf("Did not find application with name %s", name)
}
func (dmz DMZ) GetServiceForApp(name string) ([]string, error) {
app, err := dmz.GetApp(name)
if err != nil {
return []string{}, err
}
return app.Use, nil
}
type Application struct {
Name string
Type string
Src string `yaml:"src,omitempty"`
Use []string `yaml:"use,omitempty"`
}
type Service struct {
Name string
Type string
Host string `yaml:"host,omitempty"`
}
// Parse file
func Parse(yamlContent []byte) (out DMZI, err error) {
dmz := DMZ{}
err = yaml.Unmarshal([]byte(yamlContent), &dmz)
if err != nil {
fmt.Println("Yaml file is not valid, Error: " + err.Error())
}
return dmz, err
}
So, you can call the methods on the returned interface e.g.
fmt.Printf("Apps : %+v\n", dmz.GetApps())
fmt.Printf("Service : %+v\n", dmz.GetService())
UPDATE
main method as requested in the comment
func main() {
dmz, err := Parse([]byte(ymlStr))
if err != nil {
panic(err)
}
fmt.Printf("Apps : %+v\n", dmz.GetApps())
fmt.Printf("Service : %+v\n", dmz.GetService())
}
Will print
Apps : [{Name:app1 Type:php Src:/app1 Use:[redis mysql]} {Name:app2 Type:rust Src:/app2 Use:[mongo mysql]} {Name:app3 Type:golang Src:/app3 Use:[postgress mysql]}]
Service : [{Name:mongo Type:db Host:us} {Name:mysql Type:db Host:eu} {Name:postgress Type:db Host:us} {Name:redis Type:db Host:us}]
I've changed your code a little and done what I think you are asking for, which is to always call "Parse" whenever you create a new instance of DMZ. This is how you would do this using Dargo. Look at the bottom to see how the code would bind DMZ into a ServiceLocator and how to get new instances of DMZ
type DMZ struct {
Applications []*Applications `yaml:"applications,omitempty"`
Services []*Services `yaml:"services,omitempty"`
}
// This method is called every time DMZ is created via the Dargo API
func (dmz *DMZ) DargoInitialize(ioc.Descriptor) error {
// get ur bytes from... wherever
Parse(dmz, []byte{})
}
type Applications struct {
Name string
Type string
Src string `yaml:"src,omitempty"`
use []Use `yaml:"use,omitempty"`
}
type Services struct {
Name string
Type string
Host string `yaml:"host,omitempty"`
}
type Use struct {
Name string `yaml:"name,omitempty"`
host string `yaml:"host,omitempty"`
Type string `yaml:"type,omitempty"`
}
// Parse file
func Parse(dmz *DMZ, yamlContent []byte) (out *DMZ, err error) {
err = yaml.Unmarshal([]byte(yamlContent), &dmz)
if err != nil {
logs.Error("Yaml file is not valid, Error: " + err.Error())
}
return dmz, err
}
func useDargo() error {
locator, err := ioc.CreateAndBind("example", func(binder ioc.Binder) error {
binder.Bind("DMZ", &DMZ{}).InScope(ioc.PerLookup)
})
if err != nil {
return err
}
// This is how you would get your instances of DMZ
locator.GetDService("DMZ")
}
I try create package config in my example project but something doesn't work as I expected, I have folder structure:
config/config.go // package config
main.go // package main
and I want use config in my main file:
func main() {
conf := config.GetConf()
db := dbConn{
schemas: map[string]*sql.DB{},
url: fmt.Sprintf("tcp(%s)", conf.db['dev']),
username: db.user,
password: db.password,
}
db.create()
}
my config file:
type Config struct {
db map[string]string
user string
password string
}
func GetConf() *Config {
config := Config{
db: map[string]string{
"dev": "database.url",
},
user: "root",
password: "pass",
}
return &config
}
compiler return error: conf.db undefined (cannot refer to unexported field or method db)
It is a compile-time error to refer to unexported identifiers from other packages (other than the declaring package).
Export the identifiers (start them with an uppercase letter), and it will work.
type Config struct {
DB map[string]string
User string
Password string
}
func GetConf() *Config {
config := Config{
DB: map[string]string{
"dev": "database.url",
},
User: "root",
Password: "pass",
}
return &config
}
And in your main:
func main() {
conf := config.GetConf()
db := dbConn{
schemas: map[string]*sql.DB{},
url: fmt.Sprintf("tcp(%s)", conf.DB['dev']),
username: conf.User,
password: conf.Password,
}
db.create()
}
I have a yaml file that is currently written as:
keys:
- key: secret/dog
values:
- username: shiba
- password: inu
- key: secret/cat
values:
- dbhost: localhost
- words: meow
However, this yaml file often changes, so new entries could be added with different values each time:
keys:
- key: secret/dog
values:
- username: shiba
- password: inu
- key: secret/cat
values:
- dbhost: localhost
- words: meow
- key: secret/mouse
values:
- color: white
- key: secret/clouds
values:
- type: fluffy
I know from go using the gopkg.in/yaml.v2 package, i can parse through the yaml file, if all the values are the same, for example:
type Secrets struct {
Keys []struct {
Key string `json:"key"`
Values []struct {
Username string `json:"username"`
Password string `json:"password"`
} `json:"values"`
} `json:"keys"`
}
func main() {
var secret Secrets
reader, err := os.Open("demo.yml")
if err != nil {
log.Fatal(err)
}
buf, _ := ioutil.ReadAll(reader)
yaml.Unmarshal(buf, &secret)
fmt.Printf("%+v\n", secret.Keys[1].Key)
}
In the example above, it would only work for the secret/dog key, but not the others.
How can I do this in Go, when new values are added to my yaml file often?
Thanks
If you don't now exact struct you should probably make your struct looking like this
type Secrets struct {
Keys []struct {
Key string `json:"key"`
Values []map[string]string `json:"values"`
} `json:"keys"`
}
It will parse your whole yaml and get all values, but it will be an array so you loosing type hinting on object. Other way arround would be advance endoding https://blog.gopheracademy.com/advent-2016/advanced-encoding-decoding/ but you will need to add new object every time new key/value pair will appear.
I have a pair of structs like so:
type Datacenter struct {
Name string
Key string
Hosts []Host
}
type Host struct {
Name string
Port int
}
And then an example configuration file:
datacenters:
- name: dc1
key: test
hosts:
- name: dc1-host
port: 8200
- name: dc2
key: dc-test
hosts:
- name: dc2-host
port: 8200
I'm using viper to read the configuration file, here's the function:
func getDatacenters() []config.Datacenter {
err := viper.UnmarshalKey("datacenters", &datacenters)
if err != nil {
log.Error("Unable to read hosts key in config file: %s", err)
}
return datacenters
}
What I'd like to be able do now is specify an optional parameter, datacenter and if that's specified, to only return the keys from that datacenter. If the parameter is not specified, I'd like it to unmarshal and return the whole thing.
Is this possible?
EDIT: I should add, all I do with this so far is range over them:
for _, d := range datacenters {
for _, h := range d.Hosts {
}
}
So it may be there's a better way.