Access runtime config instance created in main func from another file - go

I have a runtime config instance that I need in other parts of the app, but it can only be created in main(). Ideally I would like to avoid using global variables.
// main.go
type RuntimeConfig struct {
db *DatabaseInstance
app_name string
... etc ...
}
func main() {
dbInstance = ConnectToDB(...args) // returns *DatabaseInstance
runtimeConfig := *Config{
dbInstance,
"My app",
... etc ...
}
}
// elsewhere.go
func SomeUtilityFuncThatNeedsRuntime(i int) int {
runtime := GetRuntime() // imaginary, magical runtime getter
db := runtime.DatabaseInstance
appName := runtime.appName
db.Save(appName, db, ...)
return i + 1
}
Currently it's impossible to create anonymous util functions that could really benefit from having access to certain config variables. If the variables were basic types (like a string or int), I would probably just hard-code them in. However, a field like dbInstance requires a specific instance of a connected database.

This looks to me like a use case for the singleton pattern: your RuntimeConfig is a structure that should be initialized, exactly one instance of it should exist, and it should be possible to access it.

Create configuration package with private variable and public functions (pseudo code):
package configuration
type Configuration struct {
}
var config *Configuration = nil
func GetConfig() *Configuration {
return config
}
func configLoad(filePath string) error {
config = new(Configuration)
// load your config from file, fill config structure
return nil
}
func NewConfig(flags models.ConfigFlags) (*Configuration, error) {
err := configLoad(flags.Flagconfiguration) // Path of config file.
if err != nil {
return nil, err
}
return config, nil
}
Then in your main.go initialize config:
func main() {
config, err := configuration.NewConfig(FlagsParameters)
// use this config variable in main package
}
In other packages use:
config := configuration.Config()
As an alternative, you can implement singleton pattern (but I like it less)
type Configuration struct {
}
var config *Configuration
var once sync.Once
func GetConfig() *Configuration {
once.Do(func() {
// init your config here. This code will executed once and thread safe
})
return config
}

Related

How to setup a golang app config according to the environment using yaml file

Im trying to setup a config struct to use through my application.
Currently I load a yaml file and decode it in my config struct.
config.yml
database_url: postgres://postgres:#localhost:5432/database_dev
config.go
import (
"os"
"gopkg.in/yaml.v2"
)
type AppConfig struct {
DatabaseUrl string `yaml:"database_url"`
}
func LoadConfig() *AppConfig {
appConfig := &AppConfig{}
file, _ := os.Open("config.yml")
defer f.Close()
decoder := yaml.NewDecoder(file)
decoder.Decode(config)
return appConfig
}
It works really fine, but now I need to setup different configuration according with the environment (test, local, production, etc.).
I thought that I could use a nested yaml file to declare the environments variables.
config.yml
dev:
database_url: postgres://postgres:#localhost:5432/database_dev
test:
database_url: postgres://postgres:#localhost:5432/database_test
I would like to receive the environment as a parameter in my LoadConfig function, and get the correct configuration.
But I have no idea how to.
config.go
type configFile struct {
Dev struct { AppConfig }`yaml:"dev"`
Test struct { AppConfig }`yaml:"test"`
}
func LoadConfig(env string) *AppConfig {
appConfig := &AppConfig{}
configFile := &configFile{}
file, _ := os.Open("config.yml")
defer f.Close()
decoder := yaml.NewDecoder(file)
decoder.Decode(configFile)
// How to get the correct struct here ?
// config = configFile["env"]
// It doesn't works
// invalid operation: cannot index configFile (variable of type *configFile)
return appConfig
}
Any suggestion is welcome.
If the list of environments is arbitrary, then you don't want a struct for the top level; you want a map[string]AppConfig. That would look something like this:
package main
import (
"fmt"
"os"
"gopkg.in/yaml.v2"
)
type (
AppConfig struct {
DatabaseUrl string `yaml:"database_url"`
}
ConfigFile map[string]*AppConfig
)
func LoadConfig(env string) (*AppConfig, error) {
configFile := ConfigFile{}
file, _ := os.Open("config.yml")
defer file.Close()
decoder := yaml.NewDecoder(file)
// Always check for errors!
if err := decoder.Decode(&configFile); err != nil {
return nil, err
}
appConfig, ok := configFile[env]
if !ok {
return nil, fmt.Errorf("no such environment: %s", env)
}
return appConfig, nil
}
func main() {
appConfig, err := LoadConfig(os.Args[1])
if err != nil {
panic(err)
}
fmt.Printf("config: %+v\n", appConfig)
}
Assuming we have the config.yml from your question, we can run the above example with different environments and see the desired output:
$ ./example test
config: &{DatabaseUrl:postgres://postgres:#localhost:5432/database_test}
$ ./example dev
config: &{DatabaseUrl:postgres://postgres:#localhost:5432/database_dev}
I think furthermore setting up a config file ,you should create a YML file and put this the server port...
Unfortunately I am inexperienced Therefore, I may not have given the correct answer.
I think one way is that we can different config files for different environments. For example, in dev environment we have config_dev.yaml, in production, we have config_prod.yaml.
Then we specify environment variable in the bootstrap scripts. For example we run the app by export MyEnv=dev && ./my_app. In the code, we check MyEnv to decide which config file we would use. In this case, we found MyEnv is dev so we use config_dev.yaml.

Use Viper to load configuration in Golang application

I am new to golang, and coming from java background. I am looking for some best practices for configuring the application.
I can simply load the configuration to viper and then just import to any packages and access the value using viper.Get, as the lib uses the pointer to viper so that It will always be accessing the same instance.
An alternative is to create a new viper instance cfg:=viper.New() and load the configuration then pass the cfg to all the package that needs it.
I found it cumbersome to pass the config to all the dependencies. I also have the same question for logrus, can i use the same pattern or I need to create a new instance and pass it along to all the dependencies.
Here are the code to illustrate my questions (viper seems to have all the configuration value across the packages)
main.go ($PROJECTHOME/cmd/main.go)
func main() {
loadConfig()
fmt.Printf("in main package %s", viper.Get("clientConfig"))
r := chi.NewRouter()
r.Use(middleware.RequestID)
r.Use(middleware.RealIP)
r.Use(middleware.Logger)
r.Use(middleware.Recoverer)
routes.AddUserResource(r, cfg)
http.ListenAndServe(":"+port, r)
}
func loadConfig() {
viper.SetConfigType("yaml")
viper.SetConfigName("config.local")
viper.AddConfigPath("config")
viper.AddConfigPath("../config")
err := viper.ReadInConfig()
if err != nil {
panic(fmt.Errorf("fatal error config file: %w", err))
}
}
other package ($PROJECTHOME/api/client/userClient.go)
package client
....
func RetrieveUser(config domain.ServicConfig, id string) (domain.User, bool) {
fmt.Printf("in http client package %s", viper.Get("clientConfig"))
for _, user := range users {
if user.Id == id {
return user, true
}
}
return domain.User{}, false
}

How to design classes for X number of config files which needs to be read individually in memory?

I am working with lot of config files. I need to read all those individual config file in their own struct and then make one giant Config struct which holds all other individual config struct in it.
Let's suppose if I am working with 3 config files.
ClientConfig deals with one config file.
DataMapConfig deals with second config file.
ProcessDataConfig deals with third config file.
I created separate class for each of those individual config file and have separate Readxxxxx method in them to read their own individual config and return struct back. Below is my config.go file which is called via Init method from main function after passing path and logger.
config.go
package config
import (
"encoding/json"
"fmt"
"io/ioutil"
"github.com/david/internal/utilities"
)
type Config struct {
ClientMapConfigs ClientConfig
DataMapConfigs DataMapConfig
ProcessDataConfigs ProcessDataConfig
}
func Init(path string, logger log.Logger) (*Config, error) {
var err error
clientConfig, err := ReadClientMapConfig(path, logger)
dataMapConfig, err := ReadDataMapConfig(path, logger)
processDataConfig, err := ReadProcessDataConfig(path, logger)
if err != nil {
return nil, err
}
return &Config{
ClientMapConfigs: *clientConfig,
DataMapConfigs: *dataMapConfig,
ProcessDataConfigs: *processDataConfig,
}, nil
}
clientconfig.go
package config
import (
"encoding/json"
"fmt"
"io/ioutil"
"github.com/david/internal/utilities"
)
type ClientConfig struct {
.....
.....
}
const (
ClientConfigFile = "clientConfigMap.json"
)
func ReadClientMapConfig(path string, logger log.Logger) (*ClientConfig, error) {
files, err := utilities.FindFiles(path, ClientConfigFile)
// read all the files
// do some validation on all those files
// deserialize them into ClientConfig struct
// return clientconfig object back
}
datamapconfig.go
Similar style I have for datamapconfig too. Exactly replica of clientconfig.go file but operating on different config file name and will return DataMapConfig struct back.
processdataConfig.go
Same thing as clientconfig.go file. Only difference is it will operate on different config file and return ProcessDataConfig struct back.
Problem Statement
I am looking for ideas where this above design can be improved? Is there any better way to do this in golang? Can we use interface or anything else which can improve the above design?
If I have let's say 10 different files instead of 3, then do I need to keep doing above same thing for remaining 7 files? If yes, then the code will look ugly. Any suggestions or ideas will greatly help me.
Update
Everything looks good but I have few questions as I am confuse on how can I achieve those with your current suggestion. On majority of my configs, your suggestion is perfect but there are two cases on two different configs where I am confuse on how to do it.
Case 1 After deserializing json into original struct which matches json format, I make another different struct after massaging that data and then I return that struct back.
Case 2 All my configs have one file but there are few configs which have multiple files in them and the number isn't fixed. So I pass regex file name and then I find all the files starting with that regex and then loop over all those files one by one. After deserializing each json file, I start populating another object and keep populating it until all files have been deserialized and then make a new struct with those objects and then return it.
Example of above scenarios:
Sample case 1
package config
import (
"encoding/json"
"fmt"
"io/ioutil"
"github.com/david/internal/utilities"
)
type CustomerManifest struct {
CustomerManifest map[int64]Site
}
type CustomerConfigs struct {
CustomerConfigurations []Site `json:"customerConfigurations"`
}
type Site struct {
....
....
}
const (
CustomerConfigFile = "abc.json"
)
func ReadCustomerConfig(path string, logger log.Logger) (*CustomerManifest, error) {
// I try to find all the files with my below utility method.
// Work with single file name and also with regex name
files, err := utilities.FindFiles(path, CustomerConfigFile)
if err != nil {
return nil, err
}
var customerConfig CustomerConfigs
// there is only file for this config so loop will run once
for _, file := range files {
body, err := ioutil.ReadFile(file)
if err != nil {
return nil, err
}
err = json.Unmarshal(body, &customerConfig)
if err != nil {
return nil, err
}
}
customerConfigIndex := BuildIndex(customerConfig, logger)
return &CustomerManifest{CustomerManifest: customerConfigIndex}, nil
}
func BuildIndex(customerConfig CustomerConfigs, logger log.Logger) map[int64]Site {
...
...
}
As you can see above in sample case 1, I am making CustomerManifest struct from CustomerConfigs struct and then return it instead of returning CustomerConfigs directly.
Sample case 2
package config
import (
"encoding/json"
"fmt"
"io/ioutil"
"github.com/david/internal/utilities"
)
type StateManifest struct {
NotionTemplates NotionTemplates
NotionIndex map[int64]NotionTemplates
}
type NotionMapConfigs struct {
NotionTemplates []NotionTemplates `json:"notionTemplates"`
...
}
const (
// there are many files starting with "state-", it's not fixed number
StateConfigFile = "state-*.json"
)
func ReadStateConfig(path string, logger log.Logger) (*StateManifest, error) {
// I try to find all the files with my below utility method.
// Work with single file name and also with regex name
files, err := utilities.FindFiles(path, StateConfigFile)
if err != nil {
return nil, err
}
var defaultTemp NotionTemplates
var idx = map[int64]NotionTemplates{}
// there are lot of config files for this config so loop will run multiple times
for _, file := range files {
var notionMapConfig NotionMapConfigs
body, err := ioutil.ReadFile(file)
if err != nil {
return nil, err
}
err = json.Unmarshal(body, &notionMapConfig)
if err != nil {
return nil, err
}
for _, tt := range notionMapConfig.NotionTemplates {
if tt.IsProcess {
defaultTemp = tt
} else if tt.ClientId > 0 {
idx[tt.ClientId] = tt
}
}
}
stateManifest := StateManifest{
NotionTemplates: defaultTemp,
NotionIndex: idx,
}
return &stateManifest, nil
}
As you can see above in my both the cases, I am making another different struct after deserializing is done and then I return that struct back but as of now in your current suggestion I think I won't be able to do this generically because for each config I do different type of massaging and then return those struct back. Is there any way to achieve above functionality with your current suggestion? Basically for each config if I want to do some massaging, then I should be able to do it and return new modified struct back but for some cases if I don't want to do any massaging then I can return direct deserialize json struct back. Can this be done generically?
Since there are config which has multiple files in them so that is why I was using my utilities.FindFiles method to give me all files basis on file name or regex name and then I loop over all those files to either return original struct back or return new struct back after massaging original struct data.
You can use a common function to load all the configuration files.
Assume you have config structures:
type Config1 struct {...}
type Config2 struct {...}
type Config3 struct {...}
You define configuration validators for those who need it:
func (c Config1) Validate() error {...}
func (c Config2) Validate() error {...}
Note that these implement a Validatable interface:
type Validatable interface {
Validate() error
}
There is one config type that includes all these configurations:
type Config struct {
C1 Config1
C2 Config2
C3 Config3
...
}
Then, you can define a simple configuration loader function:
func LoadConfig(fname string, out interface{}) error {
data, err:=ioutil.ReadFile(fname)
if err!=nil {
return err
}
if err:=json.Unmarshal(data,out); err!=nil {
return err
}
// Validate the config if necessary
if validator, ok:=out.(Validatable); ok {
if err:=validator.Validate(); err!=nil {
return err
}
}
return nil
}
Then, you can load the files:
var c Config
if err:=LoadConfig("file1",&c.C1); err!=nil {
return err
}
if err:=LoadConfig("file2",&c.C2); err!=nil {
return err
}
...
If there are multiple files loading different parts of the same struct, you can do:
LoadConfig("file1",&c.C3)
LoadConfig("file2",&c.C3)
...
You can simplify this further by defining a slice:
type cfgInfo struct {
fileName string
getCfg func(*Config) interface{}
}
var configs=[]cfgInfo {
{
fileName: "file1",
getCfg: func(c *Config) interface{} {return &c.C1},
},
{
fileName: "file2",
getCfg: func(c *Config) interface{} {return &c.C2},
},
{
fileName: "file3",
getCfg: func(c *Config) interface{} {return &c.C3},
},
...
}
func loadConfigs(cfg *Config) error {
for _,f:=range configs {
if err:=loadConfig(f.fileName,f.getCfg(cfg)); err!=nil {
return err
}
}
return nil
}
Then, loadConfigs would load all the configuration files into cfg.
func main() {
var cfg Config
if err:=loadConfigs(&cfg); err!=nil {
panic(err)
}
...
}
Any configuration that doesn't match this pattern can be dealt with using LoadConfig:
var customConfig1 CustomConfigStruct1
if err:=LoadConfig("customConfigFile1",&customConfig1); err!=nil {
panic(err)
}
cfg.CustomConfig1 = processCustomConfig1(customConfig1)
var customConfig2 CustomConfigStruct2
if err:=LoadConfig("customConfigFile2",&customConfig2); err!=nil {
panic(err)
}
cfg.CustomConfig2 = processCustomConfig2(customConfig2)

Go use init method inside multiple function

I’ve the following go code which works, im creating a VTS property which is used in some files under the same package 
File A is creating VTS which should be used in all of the functions below (in different files under the same package) 
File A
package foo
var VTS = initSettings()
func initSettings() *cli.EnvSettings {
conf := cli.New()
conf.RepositoryCache = "/tmp"
return conf
}
In file B im using it like 
package foo
func Get(url string, conf *action.Configuration) (*chart.Chart, error) {
cmd := action.NewInstall(conf)
    // Here see the last parameters 
chartLocation, err := cmd.ChartPathOptions.LocateChart(url, VTS)
return loader.Load(chartLocation)
}
File C 
package foo
func Upgrade(ns, name, url string, vals map[string]interface{}, conf *action.Configuration) (*release.Release, error) {
… 
if url == "" {
ch = rel.Chart
} else {
cp, err := client.ChartPathOptions.LocateChart(url, VTS)
if err != nil {
return nil, err
}
ch, err = loader.Load(cp)
}
And in additional files under the same package.
Is there a cleaner way to initiate the VTS and use it in different files instead of package variable ?
I've tried something like
func Settings() *cli.EnvSettings {
cfg := cli.New()
cfg.RepositoryCache = "/tmp"
return cfg
}
and pass it as param but I got error
func GetChart(url string, Settings func(), cfg *action.Configuration) (*chart.Chart, error) {
Just add a *cli.EnvSettings as an additional parameter toGet() and Upgrade(), and then have the caller pass VTS as an argument.
File A
package foo
func initSettings() *cli.EnvSettings {
conf := cli.New()
conf.RepositoryCache = "/tmp"
return conf
}
File B
package foo
func Get(url string, conf *action.Configuration, vts *cli.EnvSettings) (*chart.Chart, error) {
cmd := action.NewInstall(conf)
// Here see the last parameters
chartLocation, err := cmd.ChartPathOptions.LocateChart(url, vts)
return loader.Load(chartLocation)
}
File C
package foo
func Upgrade(ns, name, url string, vals map[string]interface{}, conf *action.Configuration, vts *cli.EnvSettings) (*release.Release, error) {
…
if url == "" {
ch = rel.Chart
} else {
cp, err := client.ChartPathOptions.LocateChart(url, vts)
if err != nil {
return nil, err
}
ch, err = loader.Load(cp)
}
File D: Some other file of a higher level package
...
vts := foo.initSettings()
foo.Get(myUrl, myConf, vts)
Of course, if you want to call foo.Get() or foo.Update() from several other files and packages throughout your project, and if you want all of those calls to use the same *cli.EnvSettings object, you'll likely have to construct VTS from a higher level and pass it around through more functions (i.e. continue the pattern).
In general, this is a form of dependency injection, where foo.Get() and foo.Update() are the "clients" and VTS is the "service". One big advantage of function parameters over package variables is testability. It is difficult to test how foo.Get() behaves with different *cli.EnvSettings objects if said *cli.EnvSettings object is a global / package variable. It's far easier for the tests to decide what *cli.EnvSettings objects to use, and then pass them to foo.Get().
One disadvantage of this pattern is that you can end up with functions with many parameters if they require many injected services, and they can become a bit unwieldy. However, if a function or object truly depends on many services that are truly independent, then there's really no work-around to this anyways. It's usually better to have a function with many parameters than a function that is very difficult to test.

How to dry up database code

I have a database package that contains the following code.
package database
import (
"log"
"github.com/jinzhu/gorm"
// required by gorm
_ "github.com/mattn/go-sqlite3"
)
type Podcast struct {
ID int `sql:"index"`
Title string
RssURL string `sql:"unique_index"`
Paused bool
Episodes []Episode
}
type Episode struct {
ID int `sql:"index"`
PodcastID int
Title string
EnclosureURL string `sql:"unique_index"`
Downloaded bool
GUID string `sql:"unique_index"`
PubDate string
}
func DBSession() (db gorm.DB) {
sqliteSession, err := gorm.Open("sqlite3", cache.db)
if err != nil {
log.Fatal(err)
}
return sqliteSession
}
Followed by a bunch of methods that all start with the following code.
FindSomethingByID(id int) {
db := DBSession()
db.LogMode(false)
// code
}
FindSomethingElse {
db := DBSession()
db.LogMode(false)
// code
}
Calling DBSession and setting LogMode in each func seems bad. I just don't know how to do it better. Could someone help?
Calling gorm.Open inside every function isn't very efficient: Open opens a new connection pool, and should be called just once (see the database/sql docs, which gorm wraps).
A simple improvement is to establish a global gorm.DB, initialise it in init() it from all of your functions - e.g.
package database
var db gorm.DB
func init() {
var err error
// Note we use an = and not a := as our variables
// are already initialised
db, err = gorm.Open("sqlite3", "cache.db")
if err != nil {
log.Fatal(err)
}
// Turn off logging globally
db.LogMode(false)
}
FindSomethingByID(id int) {
err := db.Query("...")
// code
}
This is a quick win and reduces the repetition.
In a larger application it typically makes sense to pass dependencies (like DB pools, config params, etc.) more explicitly by wrapping them in types and creating custom handlers.
You also might initialise the connection in your package main and pass the *gorm.DB to your database package via a func New(db *gorm.DB) function that sets a private, package-level variable.
The most obvious simplification would be to move the db.LogMode(false) call into the DBSession() function, and give DBSession() a shorter name like DB():
func DB() (db gorm.DB) {
sqliteSession, err := gorm.Open("sqlite3", cache.db)
if err != nil {
log.Fatal(err)
}
sqliteSession.LogMode(false)
return sqliteSession
}
And using it:
FindSomethingByID(id int) {
db := DB()
// code
}
Now there's only 1 line in each of your functions using the db session, one simple function call. You can't really make it any shorter if you always need a new db session.

Resources