Using viper to read config from envfile - go

I don't really understand how viper works.
This is my code:
configuration.go
var Config *Configuration
type ServerConfiguration struct {
Port string
}
type Configuration struct {
Server ServerConfiguration
}
func Init() {
var configuration *Configuration
viper.SetConfigFile(".env")
viper.AutomaticEnv()
if err := viper.ReadInConfig(); err != nil {
log.Fatalf("Error reading config file, %s", err)
}
err := viper.Unmarshal(&configuration)
if err != nil {
log.Fatalf("Unable to decode into struct, %v", err)
}
Config = configuration
}
func GetConfig() *Configuration {
return Config
}
.env
SERVER_PORT=:4747
The problem is that Unmarshal does not work
When I use for example configuration.Server.Port it's empty

spf13/viper predominantly uses mapstructure package to convert between one native Go type to another i.e. when un-marshaling. The package internally uses map[string]interface{} type to store your config (see viper.go - L1327). After that depending on the config type (your case being env), viper calls the right parsing package to store your config values. For envfile type, it uses subosito/gotenv to put in the above said map type (see viper.go - L1501)
The crux of your problem is how to make viper unmarshal this config in a map to a struct of your choice. This is where the mapstructure package comes in, to unmarshal the map into a nested structure of you have defined. At this point you are left with two options
Unmarshal the config as a map[string]interface{} type, and later put in the appropriate structure using mapstructure
Use the DecodeHookFunc as a 2nd argument to your method to unmarshal your config (See viper.go - L904)
For the sake of simplicity reasons you can do one, which according to a trivial example I've reproduced with your example can be done below
package main
import (
"fmt"
"github.com/mitchellh/mapstructure"
"github.com/spf13/viper"
)
type ServerConfiguration struct {
Port string `mapstructure:"server_port"`
}
type Configuration struct {
Server ServerConfiguration `mapstructure:",squash"`
}
func main() {
var result map[string]interface{}
var config Configuration
viper.SetConfigFile(".env")
if err := viper.ReadInConfig(); err != nil {
fmt.Printf("Error reading config file, %s", err)
}
err := viper.Unmarshal(&result)
if err != nil {
fmt.Printf("Unable to decode into map, %v", err)
}
decErr := mapstructure.Decode(result, &config)
if decErr != nil {
fmt.Println("error decoding")
}
fmt.Printf("config:%+v\n", config)
}
You can make this working example customized depending on your actual use case. More information about the mapstructure squash tags for embedded structure can be found here

Related

Use Viper to load configuration in Golang application

I am new to golang, and coming from java background. I am looking for some best practices for configuring the application.
I can simply load the configuration to viper and then just import to any packages and access the value using viper.Get, as the lib uses the pointer to viper so that It will always be accessing the same instance.
An alternative is to create a new viper instance cfg:=viper.New() and load the configuration then pass the cfg to all the package that needs it.
I found it cumbersome to pass the config to all the dependencies. I also have the same question for logrus, can i use the same pattern or I need to create a new instance and pass it along to all the dependencies.
Here are the code to illustrate my questions (viper seems to have all the configuration value across the packages)
main.go ($PROJECTHOME/cmd/main.go)
func main() {
loadConfig()
fmt.Printf("in main package %s", viper.Get("clientConfig"))
r := chi.NewRouter()
r.Use(middleware.RequestID)
r.Use(middleware.RealIP)
r.Use(middleware.Logger)
r.Use(middleware.Recoverer)
routes.AddUserResource(r, cfg)
http.ListenAndServe(":"+port, r)
}
func loadConfig() {
viper.SetConfigType("yaml")
viper.SetConfigName("config.local")
viper.AddConfigPath("config")
viper.AddConfigPath("../config")
err := viper.ReadInConfig()
if err != nil {
panic(fmt.Errorf("fatal error config file: %w", err))
}
}
other package ($PROJECTHOME/api/client/userClient.go)
package client
....
func RetrieveUser(config domain.ServicConfig, id string) (domain.User, bool) {
fmt.Printf("in http client package %s", viper.Get("clientConfig"))
for _, user := range users {
if user.Id == id {
return user, true
}
}
return domain.User{}, false
}

How to design classes for X number of config files which needs to be read individually in memory?

I am working with lot of config files. I need to read all those individual config file in their own struct and then make one giant Config struct which holds all other individual config struct in it.
Let's suppose if I am working with 3 config files.
ClientConfig deals with one config file.
DataMapConfig deals with second config file.
ProcessDataConfig deals with third config file.
I created separate class for each of those individual config file and have separate Readxxxxx method in them to read their own individual config and return struct back. Below is my config.go file which is called via Init method from main function after passing path and logger.
config.go
package config
import (
"encoding/json"
"fmt"
"io/ioutil"
"github.com/david/internal/utilities"
)
type Config struct {
ClientMapConfigs ClientConfig
DataMapConfigs DataMapConfig
ProcessDataConfigs ProcessDataConfig
}
func Init(path string, logger log.Logger) (*Config, error) {
var err error
clientConfig, err := ReadClientMapConfig(path, logger)
dataMapConfig, err := ReadDataMapConfig(path, logger)
processDataConfig, err := ReadProcessDataConfig(path, logger)
if err != nil {
return nil, err
}
return &Config{
ClientMapConfigs: *clientConfig,
DataMapConfigs: *dataMapConfig,
ProcessDataConfigs: *processDataConfig,
}, nil
}
clientconfig.go
package config
import (
"encoding/json"
"fmt"
"io/ioutil"
"github.com/david/internal/utilities"
)
type ClientConfig struct {
.....
.....
}
const (
ClientConfigFile = "clientConfigMap.json"
)
func ReadClientMapConfig(path string, logger log.Logger) (*ClientConfig, error) {
files, err := utilities.FindFiles(path, ClientConfigFile)
// read all the files
// do some validation on all those files
// deserialize them into ClientConfig struct
// return clientconfig object back
}
datamapconfig.go
Similar style I have for datamapconfig too. Exactly replica of clientconfig.go file but operating on different config file name and will return DataMapConfig struct back.
processdataConfig.go
Same thing as clientconfig.go file. Only difference is it will operate on different config file and return ProcessDataConfig struct back.
Problem Statement
I am looking for ideas where this above design can be improved? Is there any better way to do this in golang? Can we use interface or anything else which can improve the above design?
If I have let's say 10 different files instead of 3, then do I need to keep doing above same thing for remaining 7 files? If yes, then the code will look ugly. Any suggestions or ideas will greatly help me.
Update
Everything looks good but I have few questions as I am confuse on how can I achieve those with your current suggestion. On majority of my configs, your suggestion is perfect but there are two cases on two different configs where I am confuse on how to do it.
Case 1 After deserializing json into original struct which matches json format, I make another different struct after massaging that data and then I return that struct back.
Case 2 All my configs have one file but there are few configs which have multiple files in them and the number isn't fixed. So I pass regex file name and then I find all the files starting with that regex and then loop over all those files one by one. After deserializing each json file, I start populating another object and keep populating it until all files have been deserialized and then make a new struct with those objects and then return it.
Example of above scenarios:
Sample case 1
package config
import (
"encoding/json"
"fmt"
"io/ioutil"
"github.com/david/internal/utilities"
)
type CustomerManifest struct {
CustomerManifest map[int64]Site
}
type CustomerConfigs struct {
CustomerConfigurations []Site `json:"customerConfigurations"`
}
type Site struct {
....
....
}
const (
CustomerConfigFile = "abc.json"
)
func ReadCustomerConfig(path string, logger log.Logger) (*CustomerManifest, error) {
// I try to find all the files with my below utility method.
// Work with single file name and also with regex name
files, err := utilities.FindFiles(path, CustomerConfigFile)
if err != nil {
return nil, err
}
var customerConfig CustomerConfigs
// there is only file for this config so loop will run once
for _, file := range files {
body, err := ioutil.ReadFile(file)
if err != nil {
return nil, err
}
err = json.Unmarshal(body, &customerConfig)
if err != nil {
return nil, err
}
}
customerConfigIndex := BuildIndex(customerConfig, logger)
return &CustomerManifest{CustomerManifest: customerConfigIndex}, nil
}
func BuildIndex(customerConfig CustomerConfigs, logger log.Logger) map[int64]Site {
...
...
}
As you can see above in sample case 1, I am making CustomerManifest struct from CustomerConfigs struct and then return it instead of returning CustomerConfigs directly.
Sample case 2
package config
import (
"encoding/json"
"fmt"
"io/ioutil"
"github.com/david/internal/utilities"
)
type StateManifest struct {
NotionTemplates NotionTemplates
NotionIndex map[int64]NotionTemplates
}
type NotionMapConfigs struct {
NotionTemplates []NotionTemplates `json:"notionTemplates"`
...
}
const (
// there are many files starting with "state-", it's not fixed number
StateConfigFile = "state-*.json"
)
func ReadStateConfig(path string, logger log.Logger) (*StateManifest, error) {
// I try to find all the files with my below utility method.
// Work with single file name and also with regex name
files, err := utilities.FindFiles(path, StateConfigFile)
if err != nil {
return nil, err
}
var defaultTemp NotionTemplates
var idx = map[int64]NotionTemplates{}
// there are lot of config files for this config so loop will run multiple times
for _, file := range files {
var notionMapConfig NotionMapConfigs
body, err := ioutil.ReadFile(file)
if err != nil {
return nil, err
}
err = json.Unmarshal(body, &notionMapConfig)
if err != nil {
return nil, err
}
for _, tt := range notionMapConfig.NotionTemplates {
if tt.IsProcess {
defaultTemp = tt
} else if tt.ClientId > 0 {
idx[tt.ClientId] = tt
}
}
}
stateManifest := StateManifest{
NotionTemplates: defaultTemp,
NotionIndex: idx,
}
return &stateManifest, nil
}
As you can see above in my both the cases, I am making another different struct after deserializing is done and then I return that struct back but as of now in your current suggestion I think I won't be able to do this generically because for each config I do different type of massaging and then return those struct back. Is there any way to achieve above functionality with your current suggestion? Basically for each config if I want to do some massaging, then I should be able to do it and return new modified struct back but for some cases if I don't want to do any massaging then I can return direct deserialize json struct back. Can this be done generically?
Since there are config which has multiple files in them so that is why I was using my utilities.FindFiles method to give me all files basis on file name or regex name and then I loop over all those files to either return original struct back or return new struct back after massaging original struct data.
You can use a common function to load all the configuration files.
Assume you have config structures:
type Config1 struct {...}
type Config2 struct {...}
type Config3 struct {...}
You define configuration validators for those who need it:
func (c Config1) Validate() error {...}
func (c Config2) Validate() error {...}
Note that these implement a Validatable interface:
type Validatable interface {
Validate() error
}
There is one config type that includes all these configurations:
type Config struct {
C1 Config1
C2 Config2
C3 Config3
...
}
Then, you can define a simple configuration loader function:
func LoadConfig(fname string, out interface{}) error {
data, err:=ioutil.ReadFile(fname)
if err!=nil {
return err
}
if err:=json.Unmarshal(data,out); err!=nil {
return err
}
// Validate the config if necessary
if validator, ok:=out.(Validatable); ok {
if err:=validator.Validate(); err!=nil {
return err
}
}
return nil
}
Then, you can load the files:
var c Config
if err:=LoadConfig("file1",&c.C1); err!=nil {
return err
}
if err:=LoadConfig("file2",&c.C2); err!=nil {
return err
}
...
If there are multiple files loading different parts of the same struct, you can do:
LoadConfig("file1",&c.C3)
LoadConfig("file2",&c.C3)
...
You can simplify this further by defining a slice:
type cfgInfo struct {
fileName string
getCfg func(*Config) interface{}
}
var configs=[]cfgInfo {
{
fileName: "file1",
getCfg: func(c *Config) interface{} {return &c.C1},
},
{
fileName: "file2",
getCfg: func(c *Config) interface{} {return &c.C2},
},
{
fileName: "file3",
getCfg: func(c *Config) interface{} {return &c.C3},
},
...
}
func loadConfigs(cfg *Config) error {
for _,f:=range configs {
if err:=loadConfig(f.fileName,f.getCfg(cfg)); err!=nil {
return err
}
}
return nil
}
Then, loadConfigs would load all the configuration files into cfg.
func main() {
var cfg Config
if err:=loadConfigs(&cfg); err!=nil {
panic(err)
}
...
}
Any configuration that doesn't match this pattern can be dealt with using LoadConfig:
var customConfig1 CustomConfigStruct1
if err:=LoadConfig("customConfigFile1",&customConfig1); err!=nil {
panic(err)
}
cfg.CustomConfig1 = processCustomConfig1(customConfig1)
var customConfig2 CustomConfigStruct2
if err:=LoadConfig("customConfigFile2",&customConfig2); err!=nil {
panic(err)
}
cfg.CustomConfig2 = processCustomConfig2(customConfig2)

Getting value from yaml object in file

I'm getting time to learn Go, and I'm having a problem dealing with yaml file.
this is my yaml file
---
endpoints:
service1:
url: "https://service1.com"
frequency: 2
interval: 1
service2:
url: "https://service2.com"
frequency: 3
interval: 2
My go code
package main
import (
"fmt"
"io/ioutil"
"reflect"
"gopkg.in/yaml.v3"
)
// Config define estrutura do arquivo de configuraĆ§Ć£o
type Config struct {
Endpoint map[string]interface{} `yaml:"endpoints"`
}
func main() {
yamlFile, err := ioutil.ReadFile("config.yml")
if err != nil {
fmt.Printf("Error reading YAML file: %s\n", err)
return
}
var yamlConfig Config
err = yaml.Unmarshal(yamlFile, &yamlConfig)
if err != nil {
fmt.Printf("Error parsing YAML file: %s\n", err)
}
for k := range yamlConfig.Endpoint {
nm := reflect.ValueOf(yamlConfig.Endpoint[k])
for _, key := range nm.MapKeys() {
strct := nm.MapIndex(key)
fmt.Println(key.Interface(), strct.Interface())
}
}
}
// PingEndpoint acessa os endpoint informados
func PingEndpoint(url string, frequency, interval int) {
// do something
}
Is there a better way to define the Config structure without using interface? And is really necessary use reflect do get properties of a service1 or exist a better whey to do that?
In general, an interface{} is used in this situation if you don't know the structure. In this case, the structure appears to be fixed:
type Service struct {
URL string `yaml:"url"`
Frequency int `yaml:"frequency"`
Interval int `yaml:"interval"`
}
type Config struct {
Endpoint map[string]Service `yaml:"endpoints"`
}
For your second question, you no longer need to deal with unknown fields after you do this, but even if you had interface{}, you can use type assertions (type yaml library unmarshals yaml into a map[interface{}]interface{}):
for k := range yamlConfig.Endpoint {
if mp, ok:=yamlConfig.Endpoint[k].(map[interface{}]interface{}); ok {
for key, value:=range mp {
}
}
}

Unmarshal hcl to struct using viper

Trying to Unmarshal a hcl config file to a struct, using viper, this error is returned: 1 error(s) decoding:\n\n* 'NATS' expected a map, got 'slice'. What is missing?
The code:
func lab() {
var c conf
// config file
viper.SetConfigName("draft")
viper.AddConfigPath(".")
viper.SetConfigType("hcl")
if err := viper.ReadInConfig(); err != nil {
log.Error(err)
return
}
log.Info(viper.Get("NATS")) // gives [map[port:10041 username:cl1 password:__Psw__4433__ http_port:10044]]
if err := viper.Unmarshal(&c); err != nil {
log.Error(err)
return
}
log.Infow("got conf", "conf", c)
}
type conf struct {
NATS struct {
HTTPPort int
Port int
Username string
Password string
}
}
And the config file (draft.hcl inside current directory):
NATS {
HTTPPort = 10044
Port = 10041
Username = "cl1"
Password = "__Psw__4433__"
}
Edit
Have checked this struct with hcl package and it gets marshaled/unmarshalled correctly. Also this works correctly with yaml and viper.
There is a difference between these two where log.Info(viper.Get("NATS")) is called. While the hcl version returns a slice of maps, the yaml version returns a map: map[password:__psw__4433__ httpport:10044 port:10041 username:cl1].
Your conf struct is not matching the HCL. When converted to json the HCL looks like below
{
"NATS": [
{
"HTTPPort": 10044,
"Password": "__Psw__4433__",
"Port": 10041,
"Username": "cl1"
}
]
}
So the Conf Struct should look like this
type Conf struct {
NATS []struct{
HTTPPort int
Port int
Username string
Password string
}
}
Modified code
package main
import (
"log"
"github.com/spf13/viper"
"fmt"
)
type Conf struct {
NATS []struct{
HTTPPort int
Port int
Username string
Password string
}
}
func main() {
var c Conf
// config file
viper.SetConfigName("draft")
viper.AddConfigPath(".")
viper.SetConfigType("hcl")
if err := viper.ReadInConfig(); err != nil {
log.Fatal(err)
}
fmt.Println(viper.Get("NATS")) // gives [map[port:10041 username:cl1 password:__Psw__4433__ http_port:10044]]
if err := viper.Unmarshal(&c); err != nil {
log.Fatal(err)
}
fmt.Println(c.NATS[0].Username)
}
I know this question is more than two years old now, but I came across the same issue recently.
I'm using viper to be able to load different configuration files into a Go struct, allowing configuration in JSON, YAML, TOML, HCL, just pick your favourite :)
HCL file format does wrap a map into a slice because it allows redefining a section like:
section = {
key1 = "value"
}
section = {
key2 = "value"
}
which is something that is not supported by the other formats.
And here's how I fixed it:
My solution implies each new block will override any previous definition of the same key, and keep all the others. You can do some merging magic but I didn't need to.
You need to make a hook to convert a slice of maps into a map:
// sliceOfMapsToMapHookFunc merges a slice of maps to a map
func sliceOfMapsToMapHookFunc() mapstructure.DecodeHookFunc {
return func(from reflect.Type, to reflect.Type, data interface{}) (interface{}, error) {
if from.Kind() == reflect.Slice && from.Elem().Kind() == reflect.Map && (to.Kind() == reflect.Struct || to.Kind() == reflect.Map) {
source, ok := data.([]map[string]interface{})
if !ok {
return data, nil
}
if len(source) == 0 {
return data, nil
}
if len(source) == 1 {
return source[0], nil
}
// flatten the slice into one map
convert := make(map[string]interface{})
for _, mapItem := range source {
for key, value := range mapItem {
convert[key] = value
}
}
return convert, nil
}
return data, nil
}
}
then you need to create a DecodeHook:
configOption := viper.DecodeHook(mapstructure.ComposeDecodeHookFunc(
sliceOfMapsToMapHookFunc(),
mapstructure.StringToTimeDurationHookFunc(),
mapstructure.StringToSliceHookFunc(","),
))
the two other hooks are the default ones so you might want to keep them
then you pass the option to the Unmarshal method
viper.Unmarshal(&c, configOption)
With this method you don't need a slice around your structs or your maps. Also that makes it compatible with the other configuration file formats

Instantiating an interface implementation based on a dynamic configuration value

New Gopher here, coming from Java land.
Let's say I have a some generic storage interface:
package repositories
type Repository interface {
Get(key string) string
Save(key string) string
}
I support multiple different backends (Redis, Boltdb, etc) by implementing this interface in separate packages. However, each implementation has unique configuration values that need to be passed in. So I define a constructor in each package, something like:
package redis
type Config struct {
...
}
func New(config *Config) *RedisRepository {
...
}
and
package bolt
type Config struct {
...
}
func New(config *Config) *BoltRepository {
...
}
main.go reads a json configuration file that looks something like:
type AppConfig struct {
DatabaseName string,
BoltConfig *bolt.Config,
RedisConfig *redis.Config,
}
Based on the value of DatabaseName, the app will instantiate the desired repository. What is the best way to do this? Where do I do it? Right now I'm doing some kind of horrible factoryfactory method which seems very much like a Go anti-pattern.
in my main.go, I have a function that reads the above reflected configuration values, selecting the proper configuration (either BoltConfig or RedisConfig) based on the value of DatabaseName:
func newRepo(c reflect.Value, repoName string) (repositories.Repository, error) {
t := strings.Title(repoName)
repoConfig := c.FieldByName(t).Interface()
repoFactory, err := repositories.RepoFactory(t)
if err != nil {
return nil, err
}
return repoFactory(repoConfig)
}
and in my repositories package, I have a factory that looks for the repository type and returns a factory function that produces an instantiated repository:
func RepoFactory(provider string) (RepoProviderFunc, error) {
r, ok := repositoryProviders[provider]
if !ok {
return nil, fmt.Errorf("repository does not exist for provider: %s", r)
}
return r, nil
}
type RepoProviderFunc func(config interface{}) (Repository, error)
var ErrBadConfigType = errors.New("wrong configuration type")
var repositoryProviders = map[string]RepoProviderFunc{
redis: func(config interface{}) (Repository, error) {
c, ok := config.(*redis.Config)
if !ok {
return nil, ErrBadConfigType
}
return redis.New(c)
},
bolt: func(config interface{}) (Repository, error) {
c, ok := config.(*bolt.Config)
if !ok {
return nil, ErrBadConfigType
}
return bolt.New(c)
},
}
bringing it all together, my main.go looks like:
cfg := &AppConfig{}
err = json.Unmarshal(data, cfg)
if err != nil {
log.Fatalln(err)
}
c := reflect.ValueOf(*cfg)
repo, err := newRepo(c, cfg.DatabaseName)
if err != nil {
log.Fatalln(err)
}
And yes, the second I was done typing this code I recoiled at the horror I had brought into this world. Can someone please help me escape this factory hell? What's a better way to do this type of thing -i.e selecting an interface implementation at runtime.
Do you need dynamic registration? It seems like the list of backends is already baked into your server because of the AppConfig type, so you may be better just writing the simplest possible factory code:
func getRepo(cfg *AppConfig) (Repository, error) {
switch cfg.DatabaseName {
case "bolt":
return bolt.New(cfg.BoltConfig), nil
case "redis":
return redis.New(cfg.RedisConfig), nil
}
return nil, fmt.Errorf("unknown database: %q", cfg.DatabaseName)
}
func main() {
...
var cfg AppConfig
if err := json.Unmarshal(data, &cfg); err != nil {
log.Fatalf("failed to parse config: %s", err)
}
repo, err := getRepo(&cfg)
if err != nil {
log.Fatalln("repo construction failed: %s", err)
}
...
}
Sure, you can replace this with generic reflection-based code. But while that saves a few lines of duplicated code and removes the need to update getRepo if you add a new backend, it introduces a whole mess of confusing abstraction, and you're going to have to edit code anyway if you introduce a new backend (for example, extending your AppConfig type), so saving a couple of lines in getRepo is hardly a saving.
It might make sense to move getRepo and AppConfig into a repos package if this code is used by more than one program.

Resources