I am new to golang, and coming from java background. I am looking for some best practices for configuring the application.
I can simply load the configuration to viper and then just import to any packages and access the value using viper.Get, as the lib uses the pointer to viper so that It will always be accessing the same instance.
An alternative is to create a new viper instance cfg:=viper.New() and load the configuration then pass the cfg to all the package that needs it.
I found it cumbersome to pass the config to all the dependencies. I also have the same question for logrus, can i use the same pattern or I need to create a new instance and pass it along to all the dependencies.
Here are the code to illustrate my questions (viper seems to have all the configuration value across the packages)
main.go ($PROJECTHOME/cmd/main.go)
func main() {
loadConfig()
fmt.Printf("in main package %s", viper.Get("clientConfig"))
r := chi.NewRouter()
r.Use(middleware.RequestID)
r.Use(middleware.RealIP)
r.Use(middleware.Logger)
r.Use(middleware.Recoverer)
routes.AddUserResource(r, cfg)
http.ListenAndServe(":"+port, r)
}
func loadConfig() {
viper.SetConfigType("yaml")
viper.SetConfigName("config.local")
viper.AddConfigPath("config")
viper.AddConfigPath("../config")
err := viper.ReadInConfig()
if err != nil {
panic(fmt.Errorf("fatal error config file: %w", err))
}
}
other package ($PROJECTHOME/api/client/userClient.go)
package client
....
func RetrieveUser(config domain.ServicConfig, id string) (domain.User, bool) {
fmt.Printf("in http client package %s", viper.Get("clientConfig"))
for _, user := range users {
if user.Id == id {
return user, true
}
}
return domain.User{}, false
}
Related
Im trying to setup a config struct to use through my application.
Currently I load a yaml file and decode it in my config struct.
config.yml
database_url: postgres://postgres:#localhost:5432/database_dev
config.go
import (
"os"
"gopkg.in/yaml.v2"
)
type AppConfig struct {
DatabaseUrl string `yaml:"database_url"`
}
func LoadConfig() *AppConfig {
appConfig := &AppConfig{}
file, _ := os.Open("config.yml")
defer f.Close()
decoder := yaml.NewDecoder(file)
decoder.Decode(config)
return appConfig
}
It works really fine, but now I need to setup different configuration according with the environment (test, local, production, etc.).
I thought that I could use a nested yaml file to declare the environments variables.
config.yml
dev:
database_url: postgres://postgres:#localhost:5432/database_dev
test:
database_url: postgres://postgres:#localhost:5432/database_test
I would like to receive the environment as a parameter in my LoadConfig function, and get the correct configuration.
But I have no idea how to.
config.go
type configFile struct {
Dev struct { AppConfig }`yaml:"dev"`
Test struct { AppConfig }`yaml:"test"`
}
func LoadConfig(env string) *AppConfig {
appConfig := &AppConfig{}
configFile := &configFile{}
file, _ := os.Open("config.yml")
defer f.Close()
decoder := yaml.NewDecoder(file)
decoder.Decode(configFile)
// How to get the correct struct here ?
// config = configFile["env"]
// It doesn't works
// invalid operation: cannot index configFile (variable of type *configFile)
return appConfig
}
Any suggestion is welcome.
If the list of environments is arbitrary, then you don't want a struct for the top level; you want a map[string]AppConfig. That would look something like this:
package main
import (
"fmt"
"os"
"gopkg.in/yaml.v2"
)
type (
AppConfig struct {
DatabaseUrl string `yaml:"database_url"`
}
ConfigFile map[string]*AppConfig
)
func LoadConfig(env string) (*AppConfig, error) {
configFile := ConfigFile{}
file, _ := os.Open("config.yml")
defer file.Close()
decoder := yaml.NewDecoder(file)
// Always check for errors!
if err := decoder.Decode(&configFile); err != nil {
return nil, err
}
appConfig, ok := configFile[env]
if !ok {
return nil, fmt.Errorf("no such environment: %s", env)
}
return appConfig, nil
}
func main() {
appConfig, err := LoadConfig(os.Args[1])
if err != nil {
panic(err)
}
fmt.Printf("config: %+v\n", appConfig)
}
Assuming we have the config.yml from your question, we can run the above example with different environments and see the desired output:
$ ./example test
config: &{DatabaseUrl:postgres://postgres:#localhost:5432/database_test}
$ ./example dev
config: &{DatabaseUrl:postgres://postgres:#localhost:5432/database_dev}
I think furthermore setting up a config file ,you should create a YML file and put this the server port...
Unfortunately I am inexperienced Therefore, I may not have given the correct answer.
I think one way is that we can different config files for different environments. For example, in dev environment we have config_dev.yaml, in production, we have config_prod.yaml.
Then we specify environment variable in the bootstrap scripts. For example we run the app by export MyEnv=dev && ./my_app. In the code, we check MyEnv to decide which config file we would use. In this case, we found MyEnv is dev so we use config_dev.yaml.
I don't really understand how viper works.
This is my code:
configuration.go
var Config *Configuration
type ServerConfiguration struct {
Port string
}
type Configuration struct {
Server ServerConfiguration
}
func Init() {
var configuration *Configuration
viper.SetConfigFile(".env")
viper.AutomaticEnv()
if err := viper.ReadInConfig(); err != nil {
log.Fatalf("Error reading config file, %s", err)
}
err := viper.Unmarshal(&configuration)
if err != nil {
log.Fatalf("Unable to decode into struct, %v", err)
}
Config = configuration
}
func GetConfig() *Configuration {
return Config
}
.env
SERVER_PORT=:4747
The problem is that Unmarshal does not work
When I use for example configuration.Server.Port it's empty
spf13/viper predominantly uses mapstructure package to convert between one native Go type to another i.e. when un-marshaling. The package internally uses map[string]interface{} type to store your config (see viper.go - L1327). After that depending on the config type (your case being env), viper calls the right parsing package to store your config values. For envfile type, it uses subosito/gotenv to put in the above said map type (see viper.go - L1501)
The crux of your problem is how to make viper unmarshal this config in a map to a struct of your choice. This is where the mapstructure package comes in, to unmarshal the map into a nested structure of you have defined. At this point you are left with two options
Unmarshal the config as a map[string]interface{} type, and later put in the appropriate structure using mapstructure
Use the DecodeHookFunc as a 2nd argument to your method to unmarshal your config (See viper.go - L904)
For the sake of simplicity reasons you can do one, which according to a trivial example I've reproduced with your example can be done below
package main
import (
"fmt"
"github.com/mitchellh/mapstructure"
"github.com/spf13/viper"
)
type ServerConfiguration struct {
Port string `mapstructure:"server_port"`
}
type Configuration struct {
Server ServerConfiguration `mapstructure:",squash"`
}
func main() {
var result map[string]interface{}
var config Configuration
viper.SetConfigFile(".env")
if err := viper.ReadInConfig(); err != nil {
fmt.Printf("Error reading config file, %s", err)
}
err := viper.Unmarshal(&result)
if err != nil {
fmt.Printf("Unable to decode into map, %v", err)
}
decErr := mapstructure.Decode(result, &config)
if decErr != nil {
fmt.Println("error decoding")
}
fmt.Printf("config:%+v\n", config)
}
You can make this working example customized depending on your actual use case. More information about the mapstructure squash tags for embedded structure can be found here
New Gopher here, coming from Java land.
Let's say I have a some generic storage interface:
package repositories
type Repository interface {
Get(key string) string
Save(key string) string
}
I support multiple different backends (Redis, Boltdb, etc) by implementing this interface in separate packages. However, each implementation has unique configuration values that need to be passed in. So I define a constructor in each package, something like:
package redis
type Config struct {
...
}
func New(config *Config) *RedisRepository {
...
}
and
package bolt
type Config struct {
...
}
func New(config *Config) *BoltRepository {
...
}
main.go reads a json configuration file that looks something like:
type AppConfig struct {
DatabaseName string,
BoltConfig *bolt.Config,
RedisConfig *redis.Config,
}
Based on the value of DatabaseName, the app will instantiate the desired repository. What is the best way to do this? Where do I do it? Right now I'm doing some kind of horrible factoryfactory method which seems very much like a Go anti-pattern.
in my main.go, I have a function that reads the above reflected configuration values, selecting the proper configuration (either BoltConfig or RedisConfig) based on the value of DatabaseName:
func newRepo(c reflect.Value, repoName string) (repositories.Repository, error) {
t := strings.Title(repoName)
repoConfig := c.FieldByName(t).Interface()
repoFactory, err := repositories.RepoFactory(t)
if err != nil {
return nil, err
}
return repoFactory(repoConfig)
}
and in my repositories package, I have a factory that looks for the repository type and returns a factory function that produces an instantiated repository:
func RepoFactory(provider string) (RepoProviderFunc, error) {
r, ok := repositoryProviders[provider]
if !ok {
return nil, fmt.Errorf("repository does not exist for provider: %s", r)
}
return r, nil
}
type RepoProviderFunc func(config interface{}) (Repository, error)
var ErrBadConfigType = errors.New("wrong configuration type")
var repositoryProviders = map[string]RepoProviderFunc{
redis: func(config interface{}) (Repository, error) {
c, ok := config.(*redis.Config)
if !ok {
return nil, ErrBadConfigType
}
return redis.New(c)
},
bolt: func(config interface{}) (Repository, error) {
c, ok := config.(*bolt.Config)
if !ok {
return nil, ErrBadConfigType
}
return bolt.New(c)
},
}
bringing it all together, my main.go looks like:
cfg := &AppConfig{}
err = json.Unmarshal(data, cfg)
if err != nil {
log.Fatalln(err)
}
c := reflect.ValueOf(*cfg)
repo, err := newRepo(c, cfg.DatabaseName)
if err != nil {
log.Fatalln(err)
}
And yes, the second I was done typing this code I recoiled at the horror I had brought into this world. Can someone please help me escape this factory hell? What's a better way to do this type of thing -i.e selecting an interface implementation at runtime.
Do you need dynamic registration? It seems like the list of backends is already baked into your server because of the AppConfig type, so you may be better just writing the simplest possible factory code:
func getRepo(cfg *AppConfig) (Repository, error) {
switch cfg.DatabaseName {
case "bolt":
return bolt.New(cfg.BoltConfig), nil
case "redis":
return redis.New(cfg.RedisConfig), nil
}
return nil, fmt.Errorf("unknown database: %q", cfg.DatabaseName)
}
func main() {
...
var cfg AppConfig
if err := json.Unmarshal(data, &cfg); err != nil {
log.Fatalf("failed to parse config: %s", err)
}
repo, err := getRepo(&cfg)
if err != nil {
log.Fatalln("repo construction failed: %s", err)
}
...
}
Sure, you can replace this with generic reflection-based code. But while that saves a few lines of duplicated code and removes the need to update getRepo if you add a new backend, it introduces a whole mess of confusing abstraction, and you're going to have to edit code anyway if you introduce a new backend (for example, extending your AppConfig type), so saving a couple of lines in getRepo is hardly a saving.
It might make sense to move getRepo and AppConfig into a repos package if this code is used by more than one program.
I have a runtime config instance that I need in other parts of the app, but it can only be created in main(). Ideally I would like to avoid using global variables.
// main.go
type RuntimeConfig struct {
db *DatabaseInstance
app_name string
... etc ...
}
func main() {
dbInstance = ConnectToDB(...args) // returns *DatabaseInstance
runtimeConfig := *Config{
dbInstance,
"My app",
... etc ...
}
}
// elsewhere.go
func SomeUtilityFuncThatNeedsRuntime(i int) int {
runtime := GetRuntime() // imaginary, magical runtime getter
db := runtime.DatabaseInstance
appName := runtime.appName
db.Save(appName, db, ...)
return i + 1
}
Currently it's impossible to create anonymous util functions that could really benefit from having access to certain config variables. If the variables were basic types (like a string or int), I would probably just hard-code them in. However, a field like dbInstance requires a specific instance of a connected database.
This looks to me like a use case for the singleton pattern: your RuntimeConfig is a structure that should be initialized, exactly one instance of it should exist, and it should be possible to access it.
Create configuration package with private variable and public functions (pseudo code):
package configuration
type Configuration struct {
}
var config *Configuration = nil
func GetConfig() *Configuration {
return config
}
func configLoad(filePath string) error {
config = new(Configuration)
// load your config from file, fill config structure
return nil
}
func NewConfig(flags models.ConfigFlags) (*Configuration, error) {
err := configLoad(flags.Flagconfiguration) // Path of config file.
if err != nil {
return nil, err
}
return config, nil
}
Then in your main.go initialize config:
func main() {
config, err := configuration.NewConfig(FlagsParameters)
// use this config variable in main package
}
In other packages use:
config := configuration.Config()
As an alternative, you can implement singleton pattern (but I like it less)
type Configuration struct {
}
var config *Configuration
var once sync.Once
func GetConfig() *Configuration {
once.Do(func() {
// init your config here. This code will executed once and thread safe
})
return config
}
I am using a mysql database and have many different Functions/Methods that interact with the database. For every Function I offcourse have to supply the Database Credentials such as
ReadAll.go
func ReadAll() {
db, err := sql.Open("mysql",
"user:password#tcp(127.0.0.1:3306)/hello")
if err != nil {
log.Fatal(err)
}
defer db.Close()
}
The part of "mysql",
"user:password#tcp(127.0.0.1:3306)/hello" never changes and I am supplying that to every Function that interacts with DB. I was wondering how can I for instance create a new File say DataBase.go put those credentials into some global variable and then reference when I need those strings ? That way if I have to change the credentials I only have to change them in 1 place.
I want to do something like
Database.go
const GlobalDB := "mysql","user:password#tcp(127.0.0.1:3306)/hello"
then
ReadAll.go
func ReadAll() {
db, err := sql.Open(GlobalDB)
if err != nil {
log.Fatal(err)
}
defer db.Close()
}
I am brand new to Golang but trying to figure this out.
I would probably do this by opening a session to the database once, then pass this session around to any function or method that may need it. This has a few potential problems:
You may need to lock access to it, so you don't co-mingle multiple queries on the same session (but it may be that your DB library ensures this, FWIW "database/sql" is concurrency-safe and recommends NOT opening short-lived database connections)
You can't safely close the session as it may still be in use elsewhere.
Another way would be to have a function that returns a DB sesssion, so instead of doing:
db, err := sql.Open("mysql", "user:password#tcp(127.0.0.1:3306)/hello")
You do the following:
func dbSession() (sql.DB, error) {
return sql.Open("mysql", "credentials")
}
func ReadAll() {
db, err := dbSession()
if err != nil {
log.Fatal(err)
}
defer db.Close()
}
And if you want even more flexibility, you can have a struct that contains the data you need, then build your DB connection parameters from that.
type dbData struct {
DBType, DBName, User, Host, Password string
}
var DBData dbData
func dbSession() (*sql.DB, error) {
return sql.Open(DBData.DBType, fmt.Sprintf("%s:%s#tcp(%s)/%s", DBData.User, DBData.Password, DBData.Host, DBData.DBName)
}
Also note the following in the documentation from sql.Open:
The returned DB is safe for concurrent use by multiple goroutines and
maintains its own pool of idle connections. Thus, the Open function
should be called just once. It is rarely necessary to close a DB.
you can easily create a new File with your credentials. Just have the file be in the main package main.
package main
var myDBConnectionString := "mysql://...."
This will be included when you compile your source.
The problem is, that you have to recompile your code everytime you have to connect to another database. Think about a development System vs. production System. The database credentials should differ in those systems, right? :)
To fix this, it is quit common to have a config file. So you can change the credentials with out re compiling your code.
I've got an other idea - just connect to the db once, and access this resource globally.
package main
import (
"fmt"
)
var myDb = "example"
func main() {
fmt.Println("Hello, playground")
doSomthingWithDatabase()
}
func doSomthingWithDatabase() {
fmt.Println("We can access a global variable here, see", myDb)
}
https://play.golang.org/p/npZ6Z49ink
For the configuration handling you can look here
https://blog.gopheracademy.com/advent-2014/reading-config-files-the-go-way/
hiboot-data provides out of the box starter that meet your requirement, the starter is github.com/hidevopsio/hiboot-data/starter/gorm, or you can implement your own starter by using hiboot framework, then you can inject then anywhere to decouple from the creation of the database configuration.
package service
import (
"errors"
"hidevops.io/hiboot-data/examples/gorm/entity"
"hidevops.io/hiboot-data/starter/gorm"
"hidevops.io/hiboot/pkg/app"
"hidevops.io/hiboot/pkg/utils/idgen"
)
type UserService interface {
AddUser(user *entity.User) (err error)
GetUser(id uint64) (user *entity.User, err error)
GetAll() (user *[]entity.User, err error)
DeleteUser(id uint64) (err error)
}
type UserServiceImpl struct {
// add UserService, it means that the instance of UserServiceImpl can be found by UserService
UserService
repository gorm.Repository
}
func init() {
// register UserServiceImpl
app.Component(newUserService)
}
// will inject BoltRepository that configured in github.com/hidevopsio/hiboot/pkg/starter/data/bolt
func newUserService(repository gorm.Repository) UserService {
repository.AutoMigrate(&entity.User{})
return &UserServiceImpl{
repository: repository,
}
}
func (s *UserServiceImpl) AddUser(user *entity.User) (err error) {
if user == nil {
return errors.New("user is not allowed nil")
}
if user.Id == 0 {
user.Id, _ = idgen.Next()
}
err = s.repository.Create(user).Error()
return
}
func (s *UserServiceImpl) GetUser(id uint64) (user *entity.User, err error) {
user = &entity.User{}
err = s.repository.Where("id = ?", id).First(user).Error()
return
}
func (s *UserServiceImpl) GetAll() (users *[]entity.User, err error) {
users = &[]entity.User{}
err = s.repository.Find(users).Error()
return
}
func (s *UserServiceImpl) DeleteUser(id uint64) (err error) {
err = s.repository.Where("id = ?", id).Delete(entity.User{}).Error()
return
}