Forgive me, I come from a c# background!
I have following struct in Go. We populate this struct by reading in the config from a file, which works well. However Im trying to find out a way to tell if a particular property in the struct, when passed in through the config file is null. As in, explicitly not set at all.
I've struggled for about 3 hours on this. I can do it for type strings etc, but I can't find out how to do it generically, across all types?
package main
import (
"encoding/json"
"fmt"
"os"
"reflect"
)
// Config type for configuration
type Config struct {
BatchSize int `json:"batchSize"`
BatchTime int `json:"batchTime"`
DataFolder string `json:"dataFolder"`
TempFolder string `json:"tempFolder"`
//Kafka configuration
Brokers []string `json:"streamBrokers"`
TopicJoined string `json:"streamTopicJoined"`
TopicRemoved string `json:"streamTopicRemoved"`
Group string `json:"streamGroup"`
ClientName string `json:"streamClientName"`
// Stats configuration
StatsPrefix string `json:"statsPrefix"`
//AWS S3 configuration
AccessKey string `json:"amazonAccessKey"`
SecretKey string `json:"amazonSecretKey"`
Region string `json:"amazonRegion"`
Endpoint string `json:"amazonEndpoint"`
S3Bucket string `json:"amazonS3Bucket"`
S3UploadBufferSize int32 `json:"amazonS3UploadBufferSize"`
S3UploadConcurrentSize int32 `json:"amazonS3UploadConcurrentSize"`
S3UploadRetries int32 `json:"amazonS3UploadRetries"`
S3UploadRetryTime int32 `json:"amazonS3UploadRetryTime"`
//Logging
StatsdHost string `json:"statsdHost"`
StatsdPort int `json:"statsdPort"`
StatsdRate float64 `json:"statsdRate"`
//Test Publishing
TestMode bool `json:"testMode"`
TestCount int `json:"testCount"`
}
// LoadConfig load config from file
func LoadConfig(configFile string) *Config {
if _, err := os.Stat(configFile); os.IsNotExist(err) {
panic(err)
}
if config, err := loadFromFile(configFile); nil != err {
panic(err)
} else {
fmt.Println("OneDrive", os.Getenv("OneDrive"))
msValuePtr := reflect.ValueOf(config)
msValue := msValuePtr.Elem()
typeOfT := msValue.Type()
for i := 0; i < msValue.NumField(); i++ {
field := msValue.Field(i)
// TODO: Check if field is null, regardless of type and the value from OS env variables...
}
return config
}
}
func loadFromFile(path string) (*Config, error) {
var config Config
file, err := os.Open(path)
if err != nil {
return nil, fmt.Errorf("could not open config path %q: %v", path, err)
}
defer file.Close()
decoder := json.NewDecoder(file)
err = decoder.Decode(&config)
if err != nil {
return nil, fmt.Errorf("could not parse config path %q: %v", path, err)
}
return &config, nil
}
In go, a value's default value is its zero value. You may want to make all of your types pointers (eg: *string rather than string) since the zero value of a pointer is nil. Unmarshaling your config file into a struct would preserve nil values for the keys that are missing / have null values.
Note that since slices (eg: []string) are reference types, they act as pointers and are nullable (meaning you wouldn't need to declare the type as *[]string).
I've used this library in the past to help with merging config / setting required keys (and many others exist):
https://github.com/jinzhu/configor
Example of encoding/decoding json - https://play.golang.org/p/DU_5Tuvm5-
Related
I really searched a while here, but didn't found an adequate answer:
I am trying to unmarshall yaml dict keys onto a property of a struct rather than the key of a map.
Given this yaml
commands:
php:
service: php
bin: /bin/php
node:
service: node
bin: /bin/node
I am able to unmarshall this into a struct like this:
type Config struct {
Commands map[string]struct {
Service string
Bin string
}
}
But how am I able to unmarshall it into a struct like this:
type Config struct {
Commands []struct {
Name string // <-- this should be key from the yaml (i.e. php or node)
Service string
Bin string
}
}
Thx in advance for the help
You can write a custom unmarshaler, like this (on Go playground):
package main
import (
"fmt"
"gopkg.in/yaml.v3"
)
var input []byte = []byte(`
commands:
php:
service: php
bin: /bin/php
node:
service: node
bin: /bin/node
`)
type Command struct {
Service string
Bin string
}
type NamedCommand struct {
Command
Name string
}
type NamedCommands []NamedCommand
type Config struct {
Commands NamedCommands
}
func (p *NamedCommands) UnmarshalYAML(value *yaml.Node) error {
if value.Kind != yaml.MappingNode {
return fmt.Errorf("`commands` must contain YAML mapping, has %v", value.Kind)
}
*p = make([]NamedCommand, len(value.Content)/2)
for i := 0; i < len(value.Content); i += 2 {
var res = &(*p)[i/2]
if err := value.Content[i].Decode(&res.Name); err != nil {
return err
}
if err := value.Content[i+1].Decode(&res.Command); err != nil {
return err
}
}
return nil
}
func main() {
var f Config
var err error
if err = yaml.Unmarshal(input, &f); err != nil {
panic(err)
}
for _, cmd := range f.Commands {
fmt.Printf("%+v\n", cmd)
}
}
I have split the command data into Command and NamedCommand to make the code simpler, since you can just call Decode giving the embedded Command struct for the values. If everything was in the same struct, you'd need to manually map keys to struct fields.
I am working with lot of config files. I need to read all those individual config file in their own struct and then make one giant Config struct which holds all other individual config struct in it.
Let's suppose if I am working with 3 config files.
ClientConfig deals with one config file.
DataMapConfig deals with second config file.
ProcessDataConfig deals with third config file.
I created separate class for each of those individual config file and have separate Readxxxxx method in them to read their own individual config and return struct back. Below is my config.go file which is called via Init method from main function after passing path and logger.
config.go
package config
import (
"encoding/json"
"fmt"
"io/ioutil"
"github.com/david/internal/utilities"
)
type Config struct {
ClientMapConfigs ClientConfig
DataMapConfigs DataMapConfig
ProcessDataConfigs ProcessDataConfig
}
func Init(path string, logger log.Logger) (*Config, error) {
var err error
clientConfig, err := ReadClientMapConfig(path, logger)
dataMapConfig, err := ReadDataMapConfig(path, logger)
processDataConfig, err := ReadProcessDataConfig(path, logger)
if err != nil {
return nil, err
}
return &Config{
ClientMapConfigs: *clientConfig,
DataMapConfigs: *dataMapConfig,
ProcessDataConfigs: *processDataConfig,
}, nil
}
clientconfig.go
package config
import (
"encoding/json"
"fmt"
"io/ioutil"
"github.com/david/internal/utilities"
)
type ClientConfig struct {
.....
.....
}
const (
ClientConfigFile = "clientConfigMap.json"
)
func ReadClientMapConfig(path string, logger log.Logger) (*ClientConfig, error) {
files, err := utilities.FindFiles(path, ClientConfigFile)
// read all the files
// do some validation on all those files
// deserialize them into ClientConfig struct
// return clientconfig object back
}
datamapconfig.go
Similar style I have for datamapconfig too. Exactly replica of clientconfig.go file but operating on different config file name and will return DataMapConfig struct back.
processdataConfig.go
Same thing as clientconfig.go file. Only difference is it will operate on different config file and return ProcessDataConfig struct back.
Problem Statement
I am looking for ideas where this above design can be improved? Is there any better way to do this in golang? Can we use interface or anything else which can improve the above design?
If I have let's say 10 different files instead of 3, then do I need to keep doing above same thing for remaining 7 files? If yes, then the code will look ugly. Any suggestions or ideas will greatly help me.
Update
Everything looks good but I have few questions as I am confuse on how can I achieve those with your current suggestion. On majority of my configs, your suggestion is perfect but there are two cases on two different configs where I am confuse on how to do it.
Case 1 After deserializing json into original struct which matches json format, I make another different struct after massaging that data and then I return that struct back.
Case 2 All my configs have one file but there are few configs which have multiple files in them and the number isn't fixed. So I pass regex file name and then I find all the files starting with that regex and then loop over all those files one by one. After deserializing each json file, I start populating another object and keep populating it until all files have been deserialized and then make a new struct with those objects and then return it.
Example of above scenarios:
Sample case 1
package config
import (
"encoding/json"
"fmt"
"io/ioutil"
"github.com/david/internal/utilities"
)
type CustomerManifest struct {
CustomerManifest map[int64]Site
}
type CustomerConfigs struct {
CustomerConfigurations []Site `json:"customerConfigurations"`
}
type Site struct {
....
....
}
const (
CustomerConfigFile = "abc.json"
)
func ReadCustomerConfig(path string, logger log.Logger) (*CustomerManifest, error) {
// I try to find all the files with my below utility method.
// Work with single file name and also with regex name
files, err := utilities.FindFiles(path, CustomerConfigFile)
if err != nil {
return nil, err
}
var customerConfig CustomerConfigs
// there is only file for this config so loop will run once
for _, file := range files {
body, err := ioutil.ReadFile(file)
if err != nil {
return nil, err
}
err = json.Unmarshal(body, &customerConfig)
if err != nil {
return nil, err
}
}
customerConfigIndex := BuildIndex(customerConfig, logger)
return &CustomerManifest{CustomerManifest: customerConfigIndex}, nil
}
func BuildIndex(customerConfig CustomerConfigs, logger log.Logger) map[int64]Site {
...
...
}
As you can see above in sample case 1, I am making CustomerManifest struct from CustomerConfigs struct and then return it instead of returning CustomerConfigs directly.
Sample case 2
package config
import (
"encoding/json"
"fmt"
"io/ioutil"
"github.com/david/internal/utilities"
)
type StateManifest struct {
NotionTemplates NotionTemplates
NotionIndex map[int64]NotionTemplates
}
type NotionMapConfigs struct {
NotionTemplates []NotionTemplates `json:"notionTemplates"`
...
}
const (
// there are many files starting with "state-", it's not fixed number
StateConfigFile = "state-*.json"
)
func ReadStateConfig(path string, logger log.Logger) (*StateManifest, error) {
// I try to find all the files with my below utility method.
// Work with single file name and also with regex name
files, err := utilities.FindFiles(path, StateConfigFile)
if err != nil {
return nil, err
}
var defaultTemp NotionTemplates
var idx = map[int64]NotionTemplates{}
// there are lot of config files for this config so loop will run multiple times
for _, file := range files {
var notionMapConfig NotionMapConfigs
body, err := ioutil.ReadFile(file)
if err != nil {
return nil, err
}
err = json.Unmarshal(body, ¬ionMapConfig)
if err != nil {
return nil, err
}
for _, tt := range notionMapConfig.NotionTemplates {
if tt.IsProcess {
defaultTemp = tt
} else if tt.ClientId > 0 {
idx[tt.ClientId] = tt
}
}
}
stateManifest := StateManifest{
NotionTemplates: defaultTemp,
NotionIndex: idx,
}
return &stateManifest, nil
}
As you can see above in my both the cases, I am making another different struct after deserializing is done and then I return that struct back but as of now in your current suggestion I think I won't be able to do this generically because for each config I do different type of massaging and then return those struct back. Is there any way to achieve above functionality with your current suggestion? Basically for each config if I want to do some massaging, then I should be able to do it and return new modified struct back but for some cases if I don't want to do any massaging then I can return direct deserialize json struct back. Can this be done generically?
Since there are config which has multiple files in them so that is why I was using my utilities.FindFiles method to give me all files basis on file name or regex name and then I loop over all those files to either return original struct back or return new struct back after massaging original struct data.
You can use a common function to load all the configuration files.
Assume you have config structures:
type Config1 struct {...}
type Config2 struct {...}
type Config3 struct {...}
You define configuration validators for those who need it:
func (c Config1) Validate() error {...}
func (c Config2) Validate() error {...}
Note that these implement a Validatable interface:
type Validatable interface {
Validate() error
}
There is one config type that includes all these configurations:
type Config struct {
C1 Config1
C2 Config2
C3 Config3
...
}
Then, you can define a simple configuration loader function:
func LoadConfig(fname string, out interface{}) error {
data, err:=ioutil.ReadFile(fname)
if err!=nil {
return err
}
if err:=json.Unmarshal(data,out); err!=nil {
return err
}
// Validate the config if necessary
if validator, ok:=out.(Validatable); ok {
if err:=validator.Validate(); err!=nil {
return err
}
}
return nil
}
Then, you can load the files:
var c Config
if err:=LoadConfig("file1",&c.C1); err!=nil {
return err
}
if err:=LoadConfig("file2",&c.C2); err!=nil {
return err
}
...
If there are multiple files loading different parts of the same struct, you can do:
LoadConfig("file1",&c.C3)
LoadConfig("file2",&c.C3)
...
You can simplify this further by defining a slice:
type cfgInfo struct {
fileName string
getCfg func(*Config) interface{}
}
var configs=[]cfgInfo {
{
fileName: "file1",
getCfg: func(c *Config) interface{} {return &c.C1},
},
{
fileName: "file2",
getCfg: func(c *Config) interface{} {return &c.C2},
},
{
fileName: "file3",
getCfg: func(c *Config) interface{} {return &c.C3},
},
...
}
func loadConfigs(cfg *Config) error {
for _,f:=range configs {
if err:=loadConfig(f.fileName,f.getCfg(cfg)); err!=nil {
return err
}
}
return nil
}
Then, loadConfigs would load all the configuration files into cfg.
func main() {
var cfg Config
if err:=loadConfigs(&cfg); err!=nil {
panic(err)
}
...
}
Any configuration that doesn't match this pattern can be dealt with using LoadConfig:
var customConfig1 CustomConfigStruct1
if err:=LoadConfig("customConfigFile1",&customConfig1); err!=nil {
panic(err)
}
cfg.CustomConfig1 = processCustomConfig1(customConfig1)
var customConfig2 CustomConfigStruct2
if err:=LoadConfig("customConfigFile2",&customConfig2); err!=nil {
panic(err)
}
cfg.CustomConfig2 = processCustomConfig2(customConfig2)
I am using the REST proxy instance of Kafka for producing and consuming messages.Using the API to get new messages but I am not able to convert those messages to a struct model in Go. For example:
// Get records
req, err := http.NewRequest(http.MethodGet, fmt.Sprintf(FETCH_CONSUMER, URL, GROUP, CONSUMER), nil)
if err != nil {
panic(err)
}
req.Header.Add("Accept", CONTENT_TYPE)
respRecords, err := client.Do(req)
if err != nil {
panic(err)
}
defer respRecords.Body.Close()
fmt.Printf("Response %s\n", respRecords.Status)
fmt.Println(respRecords.Body)
recordsBodyResp := bufio.NewScanner(respRecords.Body)
for recordsBodyResp.Scan() {
fmt.Printf("<--Body %s\n", recordsBodyResp.Text())
}
The value returned is in the following format:
[{"topic":"backward","key":null,"value":{"AdoptionID":"abcd123","IPAddress":"8.8.8.8","Port":"80","Status":"requested"},"partition":0,"offset":7}]
Since it's an array of objects, I want to extract the value portion of the key "value" into a struct.
That is where I am stuck.
You can create a struct like:
type AutoGenerated []struct {
Topic string `json:"topic"`
Key interface{} `json:"key"`
Value Value `json:"value"`
Partition int `json:"partition"`
Offset int `json:"offset"`
}
type Value struct {
AdoptionID string `json:"AdoptionID"`
IPAddress string `json:"IPAddress"`
Port string `json:"Port"`
Status string `json:"Status"`
}
And Unmarshal in that struct.
See this sample code:
package main
import (
"fmt"
"encoding/json"
)
func main() {
type Value struct {
AdoptionID string `json:"AdoptionID"`
IPAddress string `json:"IPAddress"`
Port string `json:"Port"`
Status string `json:"Status"`
}
type AutoGenerated []struct {
Topic string `json:"topic"`
Key interface{} `json:"key"`
Value Value `json:"value"`
Partition int `json:"partition"`
Offset int `json:"offset"`
}
byt := []byte(`[{"topic":"backward","key":null,"value":{"AdoptionID":"abcd123","IPAddress":"8.8.8.8","Port":"80","Status":"requested"},"partition":0,"offset":7}]`)
var dat AutoGenerated
if err := json.Unmarshal(byt, &dat); err != nil {
panic(err)
}
fmt.Printf("%#v", dat)
}
I'm trying to build a generic function which will parse input (in JSON) into a specified structure. The structure may vary at run-time, based on parameters which are passed to the function. I'm currently trying to achieve this by passing an object of the right type and using reflect.New() to create a new output object of the same type.
I'm then parsing the JSON into this object, and scanning the fields.
If I create the object and specify the type in code, everything works. If I pass an object and try to create a replica, I get an "invalid indirect" error a few steps down (see code).
import (
"fmt"
"reflect"
"encoding/json"
"strings"
)
type Test struct {
FirstName *string `json:"FirstName"`
LastName *string `json:"LastName"`
}
func genericParser(incomingData *strings.Reader, inputStructure interface{}) (interface{}, error) {
//******* Use the line below and things work *******
//parsedInput := new(Test)
//******* Use vvv the line below and things don't work *******
parsedInput := reflect.New(reflect.TypeOf(inputStructure))
decoder := json.NewDecoder(incomingData)
err := decoder.Decode(&parsedInput)
if err != nil {
//parsing error
return nil, err
}
//******* This is the line that generates the error "invalid indirect of parsedInput (type reflect.Value)" *******
contentValues := reflect.ValueOf(*parsedInput)
for i := 0; i < contentValues.NumField(); i++ {
//do stuff with each field
fmt.Printf("Field name was: %s\n", reflect.TypeOf(parsedInput).Elem().Field(i).Name)
}
return parsedInput, nil
}
func main() {
inputData := strings.NewReader("{\"FirstName\":\"John\", \"LastName\":\"Smith\"}")
exampleObject := new(Test)
processedData, err := genericParser(inputData, exampleObject)
if err != nil {
fmt.Println("Parsing error")
} else {
fmt.Printf("Success: %v", processedData)
}
}
If I can't create a replica of the object, then a way of updating / returning the one supplied would be feasible. The key thing is that this function must be completely agnostic to the different structures available.
reflect.New isn't a direct analog to new, as it can't return a specific type, it only can return a reflect.Value. This means that you are attempting to unmarshal into a *reflect.Value, which obviously isn't going to work (even if it did, your code would have passed in **Type, which isn't what you want either).
Use parsedInput.Interface() to get the underlying value after creating the new value to unmarshal into. You then don't need to reflect on the same value a second time, as that would be a reflect.Value of a reflect.Value, which again isn't going to do anything useful.
Finally, you need to use parsedInput.Interface() before you return, otherwise you are returning the reflect.Value rather than the value of the input type.
For example:
func genericParser(incomingData io.Reader, inputStructure interface{}) (interface{}, error) {
parsedInput := reflect.New(reflect.TypeOf(inputStructure).Elem())
decoder := json.NewDecoder(incomingData)
err := decoder.Decode(parsedInput.Interface())
if err != nil {
return nil, err
}
for i := 0; i < parsedInput.Elem().NumField(); i++ {
fmt.Printf("Field name was: %s\n", parsedInput.Type().Elem().Field(i).Name)
}
return parsedInput.Interface(), nil
}
https://play.golang.org/p/CzDrj6sgQNt
Trying to Unmarshal a hcl config file to a struct, using viper, this error is returned: 1 error(s) decoding:\n\n* 'NATS' expected a map, got 'slice'. What is missing?
The code:
func lab() {
var c conf
// config file
viper.SetConfigName("draft")
viper.AddConfigPath(".")
viper.SetConfigType("hcl")
if err := viper.ReadInConfig(); err != nil {
log.Error(err)
return
}
log.Info(viper.Get("NATS")) // gives [map[port:10041 username:cl1 password:__Psw__4433__ http_port:10044]]
if err := viper.Unmarshal(&c); err != nil {
log.Error(err)
return
}
log.Infow("got conf", "conf", c)
}
type conf struct {
NATS struct {
HTTPPort int
Port int
Username string
Password string
}
}
And the config file (draft.hcl inside current directory):
NATS {
HTTPPort = 10044
Port = 10041
Username = "cl1"
Password = "__Psw__4433__"
}
Edit
Have checked this struct with hcl package and it gets marshaled/unmarshalled correctly. Also this works correctly with yaml and viper.
There is a difference between these two where log.Info(viper.Get("NATS")) is called. While the hcl version returns a slice of maps, the yaml version returns a map: map[password:__psw__4433__ httpport:10044 port:10041 username:cl1].
Your conf struct is not matching the HCL. When converted to json the HCL looks like below
{
"NATS": [
{
"HTTPPort": 10044,
"Password": "__Psw__4433__",
"Port": 10041,
"Username": "cl1"
}
]
}
So the Conf Struct should look like this
type Conf struct {
NATS []struct{
HTTPPort int
Port int
Username string
Password string
}
}
Modified code
package main
import (
"log"
"github.com/spf13/viper"
"fmt"
)
type Conf struct {
NATS []struct{
HTTPPort int
Port int
Username string
Password string
}
}
func main() {
var c Conf
// config file
viper.SetConfigName("draft")
viper.AddConfigPath(".")
viper.SetConfigType("hcl")
if err := viper.ReadInConfig(); err != nil {
log.Fatal(err)
}
fmt.Println(viper.Get("NATS")) // gives [map[port:10041 username:cl1 password:__Psw__4433__ http_port:10044]]
if err := viper.Unmarshal(&c); err != nil {
log.Fatal(err)
}
fmt.Println(c.NATS[0].Username)
}
I know this question is more than two years old now, but I came across the same issue recently.
I'm using viper to be able to load different configuration files into a Go struct, allowing configuration in JSON, YAML, TOML, HCL, just pick your favourite :)
HCL file format does wrap a map into a slice because it allows redefining a section like:
section = {
key1 = "value"
}
section = {
key2 = "value"
}
which is something that is not supported by the other formats.
And here's how I fixed it:
My solution implies each new block will override any previous definition of the same key, and keep all the others. You can do some merging magic but I didn't need to.
You need to make a hook to convert a slice of maps into a map:
// sliceOfMapsToMapHookFunc merges a slice of maps to a map
func sliceOfMapsToMapHookFunc() mapstructure.DecodeHookFunc {
return func(from reflect.Type, to reflect.Type, data interface{}) (interface{}, error) {
if from.Kind() == reflect.Slice && from.Elem().Kind() == reflect.Map && (to.Kind() == reflect.Struct || to.Kind() == reflect.Map) {
source, ok := data.([]map[string]interface{})
if !ok {
return data, nil
}
if len(source) == 0 {
return data, nil
}
if len(source) == 1 {
return source[0], nil
}
// flatten the slice into one map
convert := make(map[string]interface{})
for _, mapItem := range source {
for key, value := range mapItem {
convert[key] = value
}
}
return convert, nil
}
return data, nil
}
}
then you need to create a DecodeHook:
configOption := viper.DecodeHook(mapstructure.ComposeDecodeHookFunc(
sliceOfMapsToMapHookFunc(),
mapstructure.StringToTimeDurationHookFunc(),
mapstructure.StringToSliceHookFunc(","),
))
the two other hooks are the default ones so you might want to keep them
then you pass the option to the Unmarshal method
viper.Unmarshal(&c, configOption)
With this method you don't need a slice around your structs or your maps. Also that makes it compatible with the other configuration file formats