I'm a little confused about the use of sync.Mutex in Go. I understand the basic concept (calling Lock() will prevent other goroutines from executing the code between it and Unlock()), but I'm not sure if I need it here. I've seen a fair few C++ answers for this but in each example they all seem to be modifying and accessing a variable directly.
This is my code from a package called configuration, which I will use throughout the application to get (surprisingly) configuration and settings information.
package config
import (
"encoding/json"
"fmt"
"os"
"sync"
log "github.com/sirupsen/logrus"
)
/*
ConfigurationError is an implementation of the error interface describing errors that occurred while dealing with this
package.
*/
type ConfigurationError string
/*
Error prints the error message for this ConfigurationError. It also implements the error interface.
*/
func (ce ConfigurationError) Error() string {
return fmt.Sprintf("Configuration error: %s", string(ce))
}
/*
configuration is a data struct that holds all of the required information for setting up the program. It is unexported
to prevent other packages creating more instances. Other packages that need settings information should call Current()
to access a copy of the unexported programConfig package variable.
*/
type configuration struct {
sync.RWMutex
LogLevel log.Level `json:"logLevel,omitempty"` //Logging
LogLocation string `json:"logLocation,omitempty"` //-
HttpPort int `json:"port,omitempty"` //Web
LoginUri string `json:"loginUri"` //-
WebBaseUri string `json:"webBaseUri"` //-
StaticBaseUri string `json:"staticBaseUri"` //-
ApiBaseUri string `json:"apiBaseUri"` //-
StaticContentLocalPath string `json:"staticContentLocalPath"` //-
MaxSimultaneousReports int `json:"maxSimultaneousReports"` //Reporting
}
var programConfig configuration
/*
Current returns a copy of the currently loaded program configuration.
*/
func Current() configuration {
programConfig.RLock()
defer programConfig.RUnlock()
return programConfig
}
/*
Load attempts to load a JSON settings file containing a representation of the Configuration struct. It will then set
the value of the package-level config struct to the loaded values. Some settings changes will require a restart of the
server application.
filepath - the full path of the settings file including a leading slash or drive name (depending on the OS).
*/
func Load(filepath string) error {
//Open the file for reading.
settingsFile, err := os.Open(filepath)
if err != nil {
return ConfigurationError(err.Error())
}
defer settingsFile.Close()
//Decode JSON into package level var.
decoder := json.NewDecoder(settingsFile)
newSettings := configuration{}
err = decoder.Decode(&newSettings)
if err != nil {
return ConfigurationError(err.Error())
}
programConfig.Lock() //I'm not 100% sure this is the correct use of a mutex for this situation, so check up on that.
programConfig = newSettings
programConfig.Unlock()
return nil
}
As you can see, I've used mutex in two places.
In Current(). Do I need this here if the function is not returning a pointer but a copy of the programConfig variable? The only way the underlying package variable will be modified is through the Load() function.
In the Load() function. This can be called at any time by any goroutine, although it rarely will be.
Given that, am I using them correctly and why do I need one when reading a copy of the data (if so)?
When you read data which can be written at the same time you need a mutex. Otherwise it might happen that you read while it's written and get half of the old data and half of the new data.
So your example seems to be just fine. Because you are probably reading the config very often but writing it rarely your usage of a RWLock makes sense. This means that multiple threads can read at the same time as long as it's not written.
What in your code looks dangerous is:
programConfig.Lock()
programConfig = newSettings
programConfig.Unlock()
Because programConfig contains the Mutex you are doing the Lock and Unlock on different instances which will lead to deadlocks. You should add the Mutex to the "parent" of the instance. In this case probably the package.
Related
I'm surprised to see the lack of (in any) modules for a configuration module in Go which is thread safe for concurrent reads/writes.
I'm surprised there is no easy method to basically have something like https://github.com/spf13/viper, but thread safe.. where the Get/Set holds a Lock.
what's the right Go way to handle this without bloating code?
I normally use: https://github.com/spf13/viper however if the program for example as a GUI configuration is changeable during runtime, this package doesn't work.
I started doing the following
var config struct {
lock sync.RWMutex
myString string
myInt int
}
func main() {
config.lock.RLock()
_ = config.myString // any read
config.lock.RUnlock()
}
however this becomes very very tedious when accessing members to each time Lock/Unlock every single access for a read or a write and code becomes bloated and repetitive.
I typically use a configuration struct that is retrieved and updated atomically. This allows multiple fields to be updated in a single "transaction" and is easy to implement/use.
With Go1.18 and earlier this can be implemented with atomic.Value and wrap with some simple helper functions to convert the type from interface{} to your *config.
It is even simpler with atomic.Pointer in Go 1.19:
package main
import (
"sync/atomic"
)
type config struct {
str string
num int
}
var cfg atomic.Pointer[config]
func main() {
// Store initial configuration.
cfg.Store(&config{str: "foo", num: 42})
// Clone and modify multiple fields.
newCfg := *cfg.Load()
newCfg.str = "bar"
newCfg.num++
cfg.Store(&newCfg)
// Retrieve a value.
_ = cfg.Load().str
}
With Go1.18 and earlier you can use atomic.Value and wrap them with some simple helper functions to convert the type from interface{} to your *config.
I have recently started working with golang so having difficulties understanding on how to achieve same thing which I am able to do it easily in Java or C#. I am trying to make an object of configmanager class and when the first time configmanager class is called, it should initialize all my configs and store it in memory in some struct object. And then I should have access to configmanager object and should be able to access all those configs from my main function using some getters maybe?
Below is my configmanager go class. It is very simple for now to make it easier to understand.
package config
import (
"encoding/json"
"fmt"
"io/ioutil"
"github.com/david/internal/utilities"
)
type ClientMetrics struct {
ClientMetrics []ClientMetric `json:"clientMetrics"`
}
type CustomerData struct {
Process []string `json:"Process"`
Mat []string `json:"Mat"`
}
type ClientMetric struct {
ClientID int `json:"clientId"`
CustomerData CustomerData `json:"customerData,omitempty"`
LegCustomer []int `json:"legCustomer"`
OtherIds []int `json:"otherIds,omitempty"`
CatID int `json:"catId,omitempty"`
}
func Init(root string) (err error) {
files, err := utilities.FindFiles(root, "process-data.json")
if err != nil {
return fmt.Errorf("cannot find process-data.json file: %v", err)
}
for _, file := range files {
body, err := ioutil.ReadFile(file)
if err != nil {
return fmt.Errorf("unable to read file: %v", err)
}
var auto ClientMetrics
json.Unmarshal(body, &auto)
}
return nil
}
And here is I use it in my main function - This is just basic code just to demonstrate what I am doing but this isn't production ready code.
package main
import (
"github.com/david/internal/config"
)
func main() {
root := "/home/Configurations"
config.Init(root)
//
}
In my above Init function, I find process-data.json file if it is there on the disk and then load it in memory by deserializing it into ClientMetrics object. Everything works fine as shown above.
Problem Statement
Since I am coming from Java and C# background, so my confusion is how can I make an object of configmanager class and how should I initialize all my configs during the first time when I call configmanager and also have access to ClientMetrics struct using some getters. In Java and C#, I used to have constructor where I initialize all these things and then have some getters to access the config. How should I do the same thing in golang.
My main confusion is do we have constuctors in go and how should I make getters to access struct object in my main function? I am just looking for better design in go and how should I do my above code in golang?
Update
I think I wasn't able to explain properly. I have X (let's suppose 5 for now) different json files in a folder and each of those json files needs to have their own struct because they are totally different json files. My configmanager file will be responsible to load all those 5 json files into their own struct and then I should be able to access all those 5 structs and their fields from the object of configmanager. All this should happen during the initialization of configmanager class when it is called for the first time.
Here is just an example where I have bunch of json files in their own corresponding folder (folderX). I have three categories of files (clientMap-*.json, dataMap-*.json, process-data.json) as shown below.
Products
├── folder1
│ ├── clientMap-123.json
│ ├── clientMap-987.json
│ ├── clientMap-7161.json
├── folder2
│ ├── dataMap-18271.json
│ ├── dataMap-12921.json
│ ├── dataMap-98121.json
└── folder3
├── process-data.json
Now I need to read all these files (clientMap-*.json, dataMap-*.json, process-data.json) in their own struct. And I should able to use configmanager class to get corresponding struct and their fields too after unmarshalling.
For example: Read clientMap-*.json files.
files, err := utilities.FindFiles(root, "clientMap-*.json")
// load all clientMap-*.json files in its own struct after unmarshalling
Similarly for dataMap-*.json files
files, err := utilities.FindFiles(root, "dataMap-*.json")
// load all dataMap-*.json files in its own struct after unmarshalling
Also for process-data.json files
files, err := utilities.FindFiles(root, "process-data.json")
// load all process-data.json files in its own struct after unmarshalling
My FindFiles method will find all the files even with regex like above. files variable is an array containing list of all files matching particular pattern. Now I can create ConfigManager struct type holding all other structs for my config but I am trying to find a solution which is easily extensible so that in future if I add any other json file category it should be able to extend easily. What is the correct way to solve this?
This question is hard to answer because you kind of forgot to ask one in a definitive way, so I will start my answer by extracting a question based on what you wrote. I believe we can do so from this:
Problem Statement
[...] my confusion is how can I make an object of configmanager class and how should I initialize all my configs during the first time when I call configmanager and also have access to ClientMetrics struct using some getters
I believe the real question here is "How do I separate the concern of reading and unmarshalling the files from the concern of storing the results for my program to use?".
It's common to separate concerns by breaking things up into multiple functions/methods, which you have already done to some extent. Storage however is strictly a matter of types, so we need a type capable of holding the results. I'll use this opportunity to omit the word Manager from the type's name, because all it does is to provide useless abstraction. This type does not manage the config. It is the config, in that it contains all of the config.
type Config struct {
ClientMapConfigs []ClientMapConfig
DataMapConfigs []DataMapConfig
ProcessDataConfigs []ProcessDataConfig
}
Notice the fields start with an upper-case letter, making them public. This communicates that there could be nonsense in there, as nothing is protected from writes, which aligns with the fact that we read this data from files. A correct program must validate this data before using it. You could then communicate the validity of the validated data in the variable name.
func main() {
validConfig := getValidConfig("path/to/dir")
// ...
}
func getValidConfig(configDirectoryPath string) *Config {
config, err := NewConfigFromConfigDirectory(configDirectoryPath)
if err != nil {
log.Printf("Failed to read config from dir '%s': %v\n", configDirectoryPath, err)
os.Exit(1)
}
if err = ValidateConfig(config); err != nil {
log.Printf("Config from dir '%s' failed to validate: %v\n", configDirectoryPath, err)
os.Exit(1)
}
}
func NewConfigFromConfigDirectory(configDirectoryPath string) *Config {
// <- read individual configs and add to slices here
return &Config{ // This is the closest to a "constructor" that Go has.
ClientMapConfigs: clientMapConfigs,
DataMapConfigs: dataMapConfigs,
ProcessDataConfigs: processDataConfigs,
}
}
Notice that there is no imminent need for the functions validating and reading the config to have a receiver, i.e. be a method of a struct. They're fine as standalone functions until your requirements change in ways which demand the introduction of statefulness for either logic.
Also I use exit code 1 for the error-cases here, because Golang uses code 2 when the program terminates due to a panic. The former can be thought of as a problem with the environment, while the later indicates a problem within the program itself. It's a useful distinction to make, and aligns with the semantics of Exception versus RuntimeException which you may know from Java.
I can have let's say 10 different configs (files) and each of those configs can have their own structs since it's a different configs so I need to have separate struct for them
That looks like dynamic JSON struct unmarshalling, which was presented in 2015 by John Asmuth in decoding with mixed structures
You can run the following example here.
type Dog struct {
Name string
Frisbees int
}
type Cat struct {
Name string
Strings int
}
type RawAnimal struct {
Type string
Cat
Dog
}
type Animal RawAnimal
func (a *Animal) UnmarshalJSON(data []byte) error {
if err := json.Unmarshal(data, (*RawAnimal)(a)); err != nil {
return err
}
switch a.Type {
case "cat":
return json.Unmarshal(data, &a.Cat)
case "dog":
return json.Unmarshal(data, &a.Dog)
}
return fmt.Errorf("unknown type %q", a.Type)
}
From there, your ConfigManager will instantiate the right Config structure, depending on the raw JSON read.
I think the trouble is that you're looking at Go from a Java/C# perspective and thus struggling to make sense of the features. If you have the time, then I would suggest that you go thru some Go tutorials or books before you start coding (this one is pretty good: https://www.amazon.com/Programming-Language-Addison-Wesley-Professional-Computing/dp/0134190440)
To answer your question directly, what you need to do is to create a function which returns a pointer to an object of the struct (see here for a simple example: https://gobyexample.com/structs)
Taking ClientMetric as an example:
func NewClientMetric(ClientID int, CustomerData CustomerData, LegCustomer []int, OtherIds []int, CatID int) (*ClientMetric, error) {
//validate your inputs
//in case of errors, create and error message in the variable err and then: return nil, err
//other code which would go into a typical constructor would go here
return &ClientMetric{ClientID,CustomerData, LegCustomer, OtherIds, CatID}, nil
}
In this case, the function NewClientMetric is the constructor and it returns a pointer/reference to the newly created object. It also returns an error object, which is the same as saying that the constructor throws exceptions. Just as you would need to use try/catch in Java, you would need to check to handle the err variable returned by this function.
You would need to make similar functions for each of your types.
As for getters & setters, generally speaking, that should be avoided in Go. You can access the attributes of a struct directly. A function (like a getter) is only useful if you're going to do something to the attribute before returning it. Something like this:
type Customer struct {
FirstName string
LastName string
}
func (this *Customer) GetFullName() string {
return this.FirstName + " " + this.LastName
}
and these can then be accessed like this:
var customer *Customer
customer = &Customer{"John","Smith")
fmt.Println(customer.FirstName)
fmt.Println(customer.LastName)
fmt.Println(customer.GetFullName())
Please note that attributes, functions and methods which begin with a capital letter are public, the others are private. If FirstName was written as firstName, it would not be accessible outside the package in which it was declared.
Please do note that I'm not checking for errors if the pointer is null/nil, but that is always recommended.
I've heard many times that you should avoid global variables.
In my example I declared a global myTypes variable only to avoid declaring that variable over and over again in a function call or something similar.
Is this how it should be done? Is there a better way? A more testable way?
var myTypes = map[string]string{
"type1": "tpl1",
"type2": "tpl2",
}
func AFunc(someType string) string {
fmt.Sprintf("this is your type %s", myTypes[someType])
}
func main() {
AFunc("type1")
}
Not all global variables are bad. In your case:
The global variable is in the main package and therefore only accessible by a single program. This is ok.
The global variable is initialized once and not modified thereafter. This is ok.
On the other hand, whenever a global variable is modified during the program’s execution, the program becomes more difficult to understand. Therefore it should be avoided.
In a package that is meant to be reusable, global variables should be avoided since then two users of that package might influence each other. Imagine if the json package had a global variable var Indent bool. Such a variable is better to be hidden inside a data structure like JsonFormatter that gets created anew each time someone wants to format some JSON.
One usual way is to use Method Value
Consider a struct type T with two methods, Mv, whose receiver is of type T, and Mp, whose receiver is of type *T.
type T struct {
a int
}
func (tv T) Mv(a int) int { return 0 } // value receiver
func (tp *T) Mp(f float32) float32 { return 1 } // pointer receiver
var t T
The expression
T.Mv
yields a function equivalent to Mv but with an explicit receiver as its first argument; it has signature
func(tv T, a int) int
You can see an example of Method Value in this thread
// TODO: Get rid of the global variable.
var foo service
func handleFoo(w http.ResponseWriter, req *http.Request) {
// code that uses foo
}
func main() {
foo = initFoo()
http.HandleFunc("/foo", handleFoo)
}
One way to get rid of that global variable is to use method values:
type fooHandler struct {
foo service
}
func (h fooHandler) handle(w http.ResponseWriter, req *http.Request) {
// code that uses h.foo
}
func main() {
foo := initFoo()
http.HandleFunc("/foo", fooHandler{foo}.handle)
}
A new official approach for your global values is introduced in Go 1.7 with context.Context#Values.
Use context Values only for request-scoped data that transits processes and APIs, not for passing optional parameters to functions.
See "How to correctly use context.Context in Go 1.7"
Finally, in addition of being hard to test, global values can prevent vendoring.
See "To vendor or not to vendor, that is a question"
Many Go’s libaries have exported package variables. Those variables can be viewed as certain global states of a certain package.
Prior vendoring era, we can go get each imported package once and the global state of each imported package can be shared within all other imported packages.
Some devs may take it as granted and simply manipulate those global states at will.
However, with vendoring each imported package may have its own view of global states. Now a dev may found it impossible to change other package’s view of global state
You're not modifying myTypes, just reading it, so it's not a variable at all, it's a constant, and would be defined as such if Go supported it (and make sure you don't mutate it -- unfortunately Go doesn't allow you to enforce "constness" like other languages do). Global constants are mostly fine.
If you were to modify myTypes, e.g. by providing a function to add new types at runtime, then yes, it's a bad idea to retain myTypes as global state. You might just get away with it as long as you do it only in your "main program" which you're sure will never be a package to be imported by other code, but you don't know where this code might end up / get copied to (or even just used from multiple places in the same package), so why risk it. If this becomes a package that gets imported by other packages, things may work fine as long as there's not more than one such "client" package is active at runtime, but as soon as somebody links together several such packages into one binary, all those user packages will stomp over each other's data in the global myTypes. If a client package expects to only see the myTypes that it put in earlier, this will break if there's another client with different expectations. So packages that work fine when used individually may break when used together, with no way to fix this except changing the shared code. So just don't do it. It's a shame that Google themselves use global state in some of their own public stuff, e.g. in the standard "flag" package and things like "glog" which uses it and thus inherits the problem. Don't do it.
I'm working with a new go service and I have a SetupLogger utility function that creates a new instance of go-kit's logging struct log.Logger.
Is this method safe to invoke from code that's handling requests inside separate go-routines?
package utils
import (
"fmt"
"github.com/go-kit/kit/log"
"io"
"os"
"path/filepath"
)
// If the environment-specified directory for writing log files exists, open the existing log file
// if it already exists or create a log file if no log file exists.
// If the environment-specified directory for writing log files does not exist, configure the logger
// to log to process stdout.
// Returns an instance of go-kit logger
func SetupLogger() log.Logger {
var logWriter io.Writer
var err error
LOG_FILE_DIR := os.Getenv("CRAFT_API_LOG_FILE_DIR")
LOG_FILE_NAME := os.Getenv("CRAFT_API_LOG_FILE_NAME")
fullLogFilePath := filepath.Join(
LOG_FILE_DIR,
LOG_FILE_NAME,
)
if dirExists, _ := Exists(&ExistsOsCheckerStruct{}, LOG_FILE_DIR); dirExists {
if logFileExists, _ := Exists(&ExistsOsCheckerStruct{}, fullLogFilePath); !logFileExists {
os.Create(fullLogFilePath)
}
logWriter, err = os.OpenFile(fullLogFilePath, os.O_RDWR|os.O_CREATE|os.O_APPEND, 0666)
if err != nil {
fmt.Println("Could not open log file. ", err)
}
} else {
logWriter = os.Stdout
}
return log.NewContext(log.NewJSONLogger(logWriter)).With(
"timestamp", log.DefaultTimestampUTC,
"caller", log.DefaultCaller,
)
}
First recommendation: Use the -race flag to go build and go test. It will almost always be able to tell you if you have a race condition. Although it might not in this case since you could end up calling your os.Create() and your os.OpenFile() simultaneously.
So, second recommendation is to avoid, if at all possible, the "If it exists/matches/has permissions Then open/delete/whatever" pattern.
That pattern leads to the TOCTTOU (Time of Check To Time Of Use) bug, which is often a security bug and can at the very least lead to data loss.
To avoid it either wrap the check and use into the same mutex, or use atomic operations, such as an OpenFile call that creates the file or returns an error if it already existed (although to be technical, its locked inside the OS kernel. Just like how atomic CPU ops are locked in the hardware bus.).
In your case here I am not quite sure why you have two Open calls since it looks like just one would do the job.
Since your setting up of your Logger only involves library instantiation, creating a log file if it doesn't exist, opening the log file and no writing involved there will be no problem calling it from different go-routines since the shared data is not getting tampered with.
Side note:
Design wise it makes sense (assuming Logger is writing to the same file) to pass around the only instantiated instance of Logger for logging which would prevent two go-routines calling your setup function at the same time.
In my Go program, there are configuration values that I want to be constant for the duration of program execution, but that I want to be able to change at the deployment site. As far as I can tell, there's no way to achieve this with the const keyword, since (again, as far as I can tell) its value must be a constant specified at compile time. This means that the only way to achieve what I want would be to declare normal variables and initialize them during the package's init function. It's not that that won't work, but rather that there will now be nothing to prevent these pseudo-constant's values from changing.
My two questions are:
Am I missing something about how const works?
Assuming I'm not, what's the preferred way to handle this? A public function that returns a private variable that I never expose, never changing it? Just hoping people don't alter the variables, since they're really configuration settings?
Create a file "config.go" and create the vars you want to expose.
Don't export them (make them all lower case). Instead create public (upper case) funcs that give access to each item.
package config
var x = 10
func X() int {
return x
}
When you want those variables you simply import ./config and use them in code as follows:
if config.X()
Obviously, you can set the variables in the package init.
The following code is almost the same as the second method of #Christopher, except that it is not a module, it located in the main package.
package main
import (
"os"
)
type Config struct {
debug bool
key string
proxyNumber int
}
func (c *Config) Debug() bool {
return c.debug
}
func (c *Config) Key() string {
return c.key
}
func (c *Config) ProxyNumber() int {
return c.proxyNumber
}
const (
CONFIG_NAME = "config.ini"
)
var config *Config
func init() {
DefaultConfig()
if Exists(CONFIG_NAME) {
//try to save the config file
}else {
//try to load from the config file
}
}
func DefaultConfig() {
config = &Config{debug:true, key:"abcde",
proxyNumber:5,
}
}
//Exist: check the file exist
func Exists(path string) bool {
_, err := os.Stat(path)
if err == nil { return true }
if os.IsNotExist(err) { return false }
return false
}
you can use config to load and save the config file.
This is a very good question because it delves into what I suspect may be an omission from Go - immutable state.
From the language reference, "constant expressions may contain only constant operands and are evaluated at compile-time."
You cannot make vars constant, which is a shame. Joe's answer proposes encapsulation as a solution, which will work well - but it's verbose, tedious and might introduce errors.
By comparison, many impure functional languages combine mutable variables with single-assignment immutable values. For example, Scala has the keywords 'val' and 'var'; the meaning of Scala's 'var' is quite similar to Go's 'var'. Immutability is a useful tool in the toolbox because referentially-transparent side-effect-free functions can be written, alongside stateful mutable procedural code. Both have their place. Immutability is also an valuable tool for concurrency because there is no worry about possible race conditions if immutable values are shared between goroutines.
So in my opinion, amongst its many strengths, this is one of Go's shortcomings. It would presumably not be hard to support vals as well as vars, the difference being that the compiler checks that each val is assigned exactly once.
Until that feature is added, you have encapsulation as your only option.
You can do something like this:
package main
import (
"fmt"
"strconv"
)
var a string
func main() {
myvar, err := strconv.Atoi(a)
if err != nil {
fmt.Println(err)
}
fmt.Println(myvar)
}
and compile the program with
go build -ldflags '-X main.a 10' test.go
That way you can define a constant during compile time.
Just use standard go flags with iniflags. Standard go flags allow setting arbitrary config variables at program start via passing command-line flags, while iniflags "magically" add support for reading config variables from ini files.
You can wrap the variable in a function that returns its value:
func genConst(x int) func() int {
return func() int {
return x
}
}
var Constx = genConst(5)
var x1 = Constx()
This way the value cannot be changed by accident, even in the file where it's defined