Golang logrus - how to do a centralized configuration? - go

I am using logrus in a Go app. I believe this question is applicable to any other logging package (which doesn't offer external file based configuration) as well.
logrus provides functions to setup various configuration, e.g. SetOutput, SetLevel etc.
Like any other application I need to do logging from multiple source files/packages, it seems you need to setup these options in each file with logrus.
Is there any way to setup these options once somewhere in a central place to be shared all over the application. That way if I have to make logging level change I can do it in one place and applies to all the components of the app.

You don't need to set these options in each file with Logrus.
You can import Logrus as log:
import log "github.com/Sirupsen/logrus"
Then functions like log.SetOutput() are just functions and modify the global logger and apply to any file that includes this import.
You can create a package global log variable:
var log = logrus.New()
Then functions like log.SetOutput() are methods and modify your package global. This is awkward IMO if you have multiple packages in your program, because each of them has a different logger with different settings (but maybe that's good for some use cases). I also don't like this approach because it confuses goimports (which will want to insert log into your imports list).
Or you can create your own wrapper (which is what I do). I have my own log package with its own logger var:
var logger = logrus.New()
Then I make top-level functions to wrap Logrus:
func Info(args ...interface{}) {
logger.Info(args...)
}
func Debug(args ...interface{}) {
logger.Debug(args...)
}
This is slightly tedious, but allows me to add functions specific to my program:
func WithConn(conn net.Conn) *logrus.Entry {
var addr string = "unknown"
if conn != nil {
addr = conn.RemoteAddr().String()
}
return logger.WithField("addr", addr)
}
func WithRequest(req *http.Request) *logrus.Entry {
return logger.WithFields(RequestFields(req))
}
So I can then do things like:
log.WithConn(c).Info("Connected")
(I plan in the future to wrap logrus.Entry into my own type so that I can chain these better; currently I can't call log.WithConn(c).WithRequest(r).Error(...) because I can't add WithRequest() to logrus.Entry.)

This is the solution that I arrived at for my application that allows adding of fields to the logging context. It does have a small performance impact due to the copying of context base fields.
package logging
import (
log "github.com/Sirupsen/logrus"
)
func NewContextLogger(c log.Fields) func(f log.Fields) *log.Entry {
return func(f log.Fields) *log.Entry {
for k, v := range c {
f[k] = v
}
return log.WithFields(f)
}
}
package main
import (
"logging"
)
func main {
app.Logger = logging.NewContextLogger(log.Fields{
"module": "app",
"id": event.Id,
})
app.Logger(log.Fields{
"startTime": event.StartTime,
"endTime": event.EndTime,
"title": event.Name,
}).Info("Starting process")
}

Related

Golang fasthttp router custom logger

I'm playing with fasthttp and it's router, I have no issues with basic things, I have a working server and a router, that is the easy part.
The issue is with the logger, I would like to be able to customize that one, but it does not seem possible with the ctx.Logger() as it only takes a Printf argument and the format is not what I'm looking for.
Does anyone knows where in the documentation I can find a working example of what I want to do?
Example of code I currently have:
package server
import (
"github.com/fasthttp/router"
"github.com/valyala/fasthttp"
)
// Router will manage the routes of our API server
func Router() *router.Router {
r := router.New()
r.GET("/", index)
return r
}
func index(ctx *fasthttp.RequestCtx) {
ctx.Logger().Printf("/")
ctx.WriteString("Welcome!")
}
As I'm still trying my hand with the web servers and I still don't understand some things with it and Go also. So An example would be welcome.
For example I would like to be able to do something like that using a logger define in the main package:
package server
import (
"github.com/fasthttp/router"
"github.com/valyala/fasthttp"
"go.uber.org/zap"
)
// Router will manage the routes of our API server
func Router(loger *zap.Logger) *router.Router {
r := router.New()
r.GET("/", index)
return r
}
func index(ctx *fasthttp.RequestCtx) {
ctx.Logger().Printf("/") // Here should print in the zap format of my choice.
ctx.WriteString("Welcome!")
}
If you look at the source code, it's apparent that all you have is the ability to write standard Go-formatted strings:
func (cl *ctxLogger) Printf(format string, args ...interface{}) {
msg := fmt.Sprintf(format, args...)
ctxLoggerLock.Lock()
cl.logger.Printf("%.3f %s - %s", time.Since(cl.ctx.ConnTime()).Seconds(), cl.ctx.String(), msg)
ctxLoggerLock.Unlock()
}
The logger simply adds some additional information from the context. So further cutomisation beyond the standard Go formatting does not seem possible. I'm not sure what "zap format of my choice" is, so I can't say if there's a workaround or even if standard Go formatting options will serve for you here.

Initialize internal package global from parent package global?

I am trying to use a global variable defined in parent package in the internal package. There are no errors but none of the logs from internal package are printed. My assumption on order of init is: parent.go global variables and then init() from internal package; which seems right when tested with fmt.Println(). But looks like the call to ReplaceGlobals is dropped during compilation. Is there a way to fix it so that I can use global variable from internal package?
Here is the zap package I am using: zap. The API in question is discussed here. Here is the code snippet.
~/parent/parent.go
package main
import (
"log"
"go.uber.org/zap"
"parent/internal"
)
var (
logger = GetMyLogger()
undo = zap.ReplaceGlobals(logger.Desugar())
)
func GetMyLogger() *zap.SugaredLogger {
xx, err := zap.NewProduction()
if err != nil {
log.Fatal("error")
}
sugar := xx.Sugar()
return sugar
}
func main() {
logger.Infof("logger in main")
internal.New()
}
Then in ~/parent/internal/internal.go
package internal
import (
"go.uber.org/zap"
)
var logger *zap.SugaredLogger
func init() {
logger = zap.L().Sugar()
logger.Infof("init() from internal: %v\n", logger)
logger = zap.S()
logger.Infof("init() from internal: %v\n", logger)
}
func New() {
logger.Infof("New() from internal")
}
Package initialization order is well defined, see Spec: Package initialization. Basically you can be sure that if you refer to a package, that package's initialization will be done first, recursively.
Since your internal package does not refer to your main, you have no guarantee if main is initialized before anything is executed from internal.
An easy solution in your case is to move the logger to your internal package, and have main refer / use that.
If you have more packages involved and you want to use the same logger from other packages, you may have a designated package holding the logger, and have everyone refer / use that.

Different packages with different config props - Functional option

I have an application which needs configuration and I’ve created a configuration struct and I’m entering the configuration as a parameter to the function. The problem is that the configuration struct becomes bigger (like monolith) and bigger and I move the config to different functions in my app and which doesn’t need all the fields, just few of them. My question is if there is better approach to implement it in Go.
After struggling to find good way I’ve found this article (which a bit old but hopefully still relevant) and I wonder how and if I can use it to solve my problem.
Functional options instead of config struct
https://dave.cheney.net/2014/10/17/functional-options-for-friendly-apis
I need to inject some configuration properties to my application in
For example for function run (which is entry point ) I need to inject the log level and some other env variable like port host
For function build I need to “inject” the build flavor and build type etc.
Any example for my content will be very helpful
How to structure it in the code ?
How to implement it?
update
I need some E2E example how can I use the functional approach for different configs in the same package and other packages
It sounds like you're looking for an alternative to passing around the same configuration monolith structure to every package and every function. There are many solutions to this problem (more than I'm going to list here), and which one is right for you requires more knowledge of your code and your goals than we have, so it's probably best if you decide. And it sounds like you're wondering whether Dave Cheney's post on functional options provides a solution and how to apply it.
If your application's configuration is static in that it's not likely to change (mutate) through different threads of execution, and you don't need to create multiple instances with different configurations in the same main, then one option is package level variables and package initialization. If you object to exported package variables, you can use unexported package variables and control access via exported functions. Say run and build are two different packages:
// package main
import(
"github.com/profilename/appname/build"
"github.com/profilename/appname/run"
)
func main() {
// do something to get configuration values
build.Initialize(buildFlavor, buildType)
// any post-build-initialize-pre-run-initialize stuff
run.Initialize(logLevel, port, host)
// other processing
build.PreBuild("title") // other build functions may rely on configuration
build.Build()
// other stuff
run.ReadFiles(f1, f2)
run.Validate(preferredBackupPort) // port availability, chance to log.Fatal out
run.Run()
// cleanup
}
// package run
var Host string
var LogLevel, Port int
init() {
Host = `localhost`
Port = 8888
Loglevel = 1
}
func Initialize(logLevel, port int, host string) {
// validation, panic on failure
LogLevel = logLevel
Host = host
Port = port
}
func Run() {
// do something with LogLevel, Host, Port
}
But that doesn't solve the problem addressed in Dave Cheney's post. What if the user is running this without host, port, or buildType (or other configuration variables), because he doesn't need those features? What if the user wants to run multiple instances with different configurations?
Dave's approach is primarily intended for situations where you will not use package-level variables for configuration. Indeed, it is meant to enable several instances of a thing where each instance can have a different configuration. Your optional configuration parameters become a single variadic parameter where the type is a function that modifies a pointer to the thing being configured. For you, that could be
// package run
type Runner struct {
Port int
// rest of runner configuration
}
func NewRunner(options ...func(*Runner)) (runner *Runner, err error) {
// any setup
for _, option := range options {
err = option(runner)
if err != nil {
// do something
}
}
return runner, err
}
// package main
func main() {
// do something to get configuration values
port := func(runner *Runner) {
runner.Port = configuredPort
}
// other configuration if applicable
runner := run.NewRunner(port)
// ...
In a way, Dave's approach appears targeted at packages that will be used as very flexible libraries, and will provide application interfaces that users might wish to create several instances of. It allows for main definitions that launch multiple instances with different configurations. In that post he doesn't go into detail on how to process configuration input in the main or on a configuration package.
Note that the way the port is set in the resulting code above is not very different from this:
// package run
type Runner struct {
Port int
// rest of runner configuration
}
// package main, func main()
runner := new(run.Runner)
runner.Port = configuredPort
which is more traditional, probably easier for most developers to read and understand, and a perfectly fine approach if it suits your needs. (And you could make runner.port unexported and add a func (r *Runner) SetPort(p int) { r.port = p } method if you wanted.) It is also a design that has the potential, depending on implementation, to deal with mutating configuration, multiple threads of execution (you'll need channels or the sync package to deal with mutation there), and multiple instances.
Where the function options design Dave proposed becomes much more powerful than that approach is when you have many more statements related to the setting of the option that you want to place in main rather than in run -- those will make up the function body.
UPDATE Here's a runnable example using Dave's functional options approach, in two files. Be sure to update the import path to match wherever you put the run package.
Package run:
package run
import(
"fmt"
"log"
)
const(
DefaultPort = 8888
DefaultHost = `localhost`
DefaultLogLevel = 1
)
type Runner struct {
Port int
Host string
LogLevel int
}
func NewRunner(options ...func(*Runner) error) (runner *Runner) {
// any setup
// set defaults
runner = &Runner{DefaultPort, DefaultHost, DefaultLogLevel}
for _, option := range options {
err := option(runner)
if err != nil {
log.Fatalf("Failed to set NewRunner option: %s\n", err)
}
}
return runner
}
func (r *Runner) Run() {
fmt.Println(r)
}
func (r *Runner) String() string {
return fmt.Sprintf("Runner Configuration:\n%16s %22d\n%16s %22s\n%16s %22d",
`Port`, r.Port, `Host`, r.Host, `LogLevel`, r.LogLevel)
}
Package main:
package main
import(
"errors"
"flag"
"github.com/jrefior/run" // update this path for your filesystem
)
func main() {
// do something to get configuration values
portFlag := flag.Int("p", 0, "Override default listen port")
logLevelFlag := flag.Int("l", 0, "Override default log level")
flag.Parse()
// put your runner options here
runnerOpts := make([]func(*run.Runner) error, 0)
// with flags, we're not sure if port was set by flag, so test
if *portFlag > 0 {
runnerOpts = append(runnerOpts, func(runner *run.Runner) error {
if *portFlag < 1024 {
return errors.New("Ports below 1024 are privileged")
}
runner.Port = *portFlag
return nil
})
}
if *logLevelFlag > 0 {
runnerOpts = append(runnerOpts, func(runner *run.Runner) error {
if *logLevelFlag > 8 {
return errors.New("The maximum log level is 8")
}
runner.LogLevel = *logLevelFlag
return nil
})
}
// other configuration if applicable
runner := run.NewRunner(runnerOpts...)
runner.Run()
}
Example usage:
$ ./program -p 8987
Runner Configuration:
Port 8987
Host localhost
LogLevel 1
I use this to define per package Config Structs which are easier to manage and are loaded at the app start.
Define your config struct like this
type Config struct {
Conf1 package1.Configuration `group:"conf1" namespace:"conf1"`
Conf2 package2.Configuration `group:"conf2" namespace:"conf2"`
Conf3 Config3 `group:"conf3" namespace:"conf3"`
GeneralSetting string `long:"Setting" description:"setting" env:"SETTING" required:"true"`
}
type Config3 struct {
setting string
}
And use "github.com/jessevdk/go-flags" to pass either --config3.setting=stringValue cmd arguments, or ENV variables export CONFIG3_SETTING=stringValue:
type Configuration interface {}
const DefaultFlags flags.Options = flags.HelpFlag | flags.PassDoubleDash
func Parse(cfg Configuration) []string {
args, _ := flags.NewParser(cfg, DefaultFlags).Parse()
return args
}
And your main should look something like this:
func main() {
// Parse the configuration.
var cfg Config
Parse(&cfg)
service := NewService(cfg.Conf3.Setting)
}

Share object from in local package

I'm new to Go and I've created an application which is working as expected.
My application structure is like following:
myproj
Gopkg.toml
Gopkg.lock
src
server
main.go
utils
file1.go
logger.go
handler
handler1.go
handler2.go
Now inside the main.go file I've created a logger like the following:
File server-> main.go
import (
"handler"
"utils"
"github.com/sirupsen/logrus"
)
var logger *logrus.Logger
fun init(){
logger = utils.InitLogs()
}
func main(){
logger.info("my message")
…
handler.run()
}
Everything is working as expected!
Now I want to use the logger inside the handler1&2 files (from diff package inside my local project)
To do this, I did the following steps.
Inside the handler init the logger (exactly as I did in the main file) and it’s working
File handler-> handler1.go
import (
"utils"
"github.com/sirupsen/logrus"
)
var logger *logrus.Logger
fun init(){
//Init the logger again
logger = utils.InitLogs()
}
func run(){
//here Im not using logger otherwise maybe I can move it as parameter…
}
func build(){
//Here Im using the logger
logger.info("Hi")
}
While this is working but I've created two instance of logger, first on main and second on handler1 which Im not sure is best
My questions are:
I want to reuse the logger inside the handler class/module, is a better way to do it in Go?
This is diff package (inside my local proj) and I'm not sure that I'm that if Im not re-use the same logger object from the main method, this is the right way to do it...
(lower prio question) Is my project structure okay for Go? I used this as reference which is similar...also this which is a bit different
If you want to share your logger instance across your project, one way is to just export it as a global variable from your utils package as Armin suggested in his answer:
package utils
...
var Logger *logrus.Logger
func init() {
Logger = InitLogs()
}
I'd bet you can then get away with not having to export InitLogs() by changing it to initLogs()
Then, elsewhere in your code, you can import utils and use that logger instance:
import "utils"
...
func something() {
utils.Logger.Info("Hi")
}
Alternatively, if it makes sense to keep all your configuration in one place, you can declare your logger pointer as a field of a config struct and initialize it along with the rest of your program configuration (if you have any).
For example, say you have the following in your utils package:
package utils
...
type MyAppConfig struct {
// whatever config parameters your app needs,
// like DB connections, etc.
Logger *logrus.Logger
}
// pass in whatever configuration parameters you need,
// like DB URL, etc.
func InitConfig() (*MyAppConfig, error) {
// set up other configuration
config := &MyAppConfig{
// other config
Logger: InitLogs(),
}
return config, nil
}
func (c *MyAppConfig) DoSomethingImportant() {
c.Logger.Info("Hello")
}
Meanwhile, you can use this anywhere else, like in your CLI interface:
package main
import "utils"
...
func main() {
// input, or CLI parameters...
// pass in other CLI parameters, if any
// (of course you'd have to change the function signature in
// the previous file above)
config, err := utils.NewConfig()
if err != nil {
// handle error
}
config.DoSomethingImportant()
// or since Logger is exported:
config.Logger.Info("hello")
}
If you find you end up with lots of global things like the Logger you need to configure, the config struct is the more scalable approach IMO. As an added bonus, using a config type makes it easier to do dependency injection in unit tests than having the global variables used directly.
On the other hand, if you only have to worry about the one global Logger, then the config struct might be overkill, especially if it's not something you'll need to instrument when designing tests. It's a subjective call depending on your circumstances.
To answer your second question...
Project Layout
If /server has a main package with a command, move it under cmd/server.
Also, the way you currently have it, it would be a bit confusing for others to depend on your project. Since your repository appears to begin at src/, here's what it will look like when someone else tries to import your utils package:
File structure:
<top of someone else's GOPATH>/
src/
myproject/src/utils/
...
To import:
import "myproject/src/utils"
... and actually, since you import "utils" in your code right now, I don't think your code will compile in someone else's GOPATH because they don't have a $GOPATH/src/utils.
Solution: Your repository should not include src/. The idea is that you setup your $GOPATH and you put different repositories/packages inside of it. For example (assuming you host your code on Github):
<top of gopath>/
src/
github.com/you/myproject/ <---- your repo starts here
cmd/server/
main.go
utils
file1.go
logger.go
handler
handler1.go
handler2.go
This would make your packages importable to you and to others like so:
import "github.com/you/myproject/utils"
Also, this project structure allow others to include your project inside their $GOPATH or in their vendor/ interchangeably.
It seems you want to use the exact logger from main.go. You should just export it and use it in other packages:
server/main.go
var Logger *logrus.Logger // Notice capital L
func main() {
Logger = utils.InitLogs() // Init logger once
...
}
handler/handler1.go
import "server"
func run(){
//here Im not using logger otherwise maybe I can move it as parameter…
}
func build(){
// No need to initialize logger as its done in main func.
//Here Im using the logger
server.Logger.info("in xyz")
}
And your project structure is not standard. See this.

How to access flags outside of main package?

We parse flags in main.go which is in main package, of course. Then we have another package where we want to read some flag's value.
flags.Args() work fine, it will return all non-flag values.
But I cannot figure out to how read already parsed value for a flag in a package other than main.
Is it possible?
Thanks
Amer
I had the same requirement recently and I wanted a solution that avoided calling flag.Parse repeatedly in init functions.
Perusing the flag package I found Lookup(name string) which returns a Flag which has a Value. Every built in Value implements the flag.Getter interface. The call chain looks like this:
flag.Lookup("httplog").Value.(flag.Getter).Get().(bool)
If you mistype the flag name or use the wrong type you get a runtime error. I wrapped the lookup in a function that I call directly where needed since the lookup and get methods are fast and the function is not called often. So the main package declares the flag.
// main.go
package main
import "flag"
var httplog = flag.Bool("httplog", false, "Log every HTTP request and response.")
func main() {
flag.Parse()
// ...
}
And the utility package, which is decoupled from main except for the flag name, reads the flag value.
// httpdiag.go
package utility
import "flag"
func logging() bool {
return flag.Lookup("httplog").Value.(flag.Getter).Get().(bool)
}
You can define the var storing the flag in the separate package, as an exported variable, then call the flag parsing in the main package to use that variable, like this:
mypackage/const.go
var (
MyExportedVar string
)
mainpackage/main.go
func init() {
flag.StringVar(&mypackage.MyExportedVar, "flagName", "defaultValue", "usage")
flag.Parse()
}
This way, everybody can access that flag, including that package itself.
Note:
this only works for exported variables.
You can define the flag in that package and call flag.Parse() in func init(), flag.Parse can be called multiple times.
However, if you want to access the flag's value from multiple packages you have to expose it or create an exposed function to check it.
for example:
// pkgA
var A = flag.Bool("a", false, "why a?")
func init() {
flag.Parse()
}
// main package
func main() {
flag.Parse()
if *pkgA.A {
// stuff
}
}
Also you can use FlagSet.Parse(os.Args) if you want to reparse the args.
When you parse your flags, parse them into global variables which start with an initial Capital so they are public, eg
package main
var Species = flag.String("species", "gopher", "the species we are studying")
func main() {
flag.Parse()
}
Then in your other package you can refer to them as
package other
import "/path/to/package/main"
func Whatever() {
fmt.Println(main.Species)
}

Resources