I'm used to work with java and log4j, in which I could dynamically change the log level for a single class without any additional (coding) effort.
Now, when I work with golang (zap for logging) – I seek the same functionality, but cannot find it.
Is there an easy way to change log level dynamically for a file or a package or a part of the code?
zap supports log.Logger wrapping via constructors like NewStdLog, so if you want to keep using zap then the following technique will work:
If you don't want to use a third-party loggers, the following technique is quite simple, using go's standard library log.Logger:
Define some default app loggers:
var (
pkgname = "mypkgname"
Info = log.New(os.Stdout, "[INFO:"+pkgname+"] ", log.LstdFlags)
Debug = log.New(ioutil.Discard, "[DBUG:"+pkgname+"] ", log.LstdFlags|log.Lshortfile)
Trace = log.New(ioutil.Discard, "[TRCE:"+pkgname+"] ", log.LstdFlags|log.Lmicroseconds|log.Lshortfile)
)
so by default Info will write to Stdout - but Debug & Trace will go be "off" by default.
One can then, safely (i.e. goroutine-safe) turn these log.Loggers on/off either at start time:
func init() {
if v, ok := os.LookupEnv("DEBUG"); ok && v != "" {
Info.Println("ENV VAR 'DEBUG' set: enabling debug-level logging")
Debug.SetOutput(os.Stderr)
}
}
or during runtime:
func httpServiceHandler(r *req) {
if r.TraceingOn {
Trace.SetOutput(os.Stderr)
} else {
Trace.SetOutput(ioutil.Discard)
}
}
Logging at pkg level is just like any other log method:
Debug.Printf("http request: %+v", r)
And if there's an expensive log event that you don't want to generate - if say the logger is set to discard - you can safely check the state of the logger like so:
if Trace.Writer() != ioutil.Discard {
// do expensive logging here
// e.g. bs, _ = ioutil.ReadAll(resp.Body); Trace.Println("RESP: RAW-http response:", string(bs))
}
Related
I have a piece of code which is used to load configuration from file and parse it into a struct, I use this configuration quite often and hence I pass it around in the method parameters. Now I as my method parameters are increasing, I am looking at dependency injection and have settle with wire.
Now I have created a provider to load the configuration and an injector to provide the config struct. However each time I call the injection my file is read again, I want that the file is read once and the injection provided as many times as required without any additional loading.
Here is my provider:
// ProvideConfig config provider ...
func ProvideConfig() *config.FileConfig {
var cfp string
flag.StringVar(&cfp, "config", "config.json", "absolute path")
flag.Parse()
return config.Loadconfig(cfp)
}
Injector:
// GetConfig injector ...
func GetConfig() ConfigResource {
wire.Build(ProvideConfig, NewConfigResource)
return ConfigResource{}
}
Now when I call:
injection.GetConfig()
I see that ProvideConfig is called always. I can have a check in the provide config method the determine if the config is already loaded, I am not sure if there is a better way, something like a single instance loader which is built into the wire. I tried looking into the docs but could not find anything relevant.
As far as I'm aware, there's no built in way in wire to specify that a provider is a singleton / should only be called once.
This is accomplished in the usual way in Go, by using sync.Once. Your provider function can be a closure that does the expensive operation only once using sync.Once.Do. This is idiomatic in Go, and doesn't require any special provision from every library that wants to provide "single" loading.
Here's an example without wire:
type Value struct {
id int
msg string
}
type ValueProvider func() *Value
// consumer takes a function that provides a new *Value and consumes
// the *Value provided by it.
func consumer(vp ValueProvider) {
v := vp()
fmt.Printf("Consuming %+v\n", *v)
}
// MakeSingleLoader returns a ValueProvider that creates a value once using an
// expensive operation, and then keeps returning the same value.
func MakeSingleLoader() ValueProvider {
var v *Value
var once sync.Once
return func() *Value {
once.Do(func() {
v = ExpensiveOperation()
})
return v
}
}
// ExpensiveOperation emulates an expensive operation that can take a while
// to run.
func ExpensiveOperation() *Value {
return &Value{id: 1, msg: "hello"}
}
func main() {
sl := MakeSingleLoader()
consumer(sl)
consumer(sl)
consumer(sl)
}
If you're OK with the "singleton" value being a global, this code can be simplified a bit. Otherwise, it only calls ExpensiveOperation once, and keeps the value cached in a local inaccessible outside MakeSingleLoader.
Currently running a go service on GCP however in the logs viewer every message is treated as an error.
Is there a generally advised way of logging to stderr and stdout depending on the log level. Ie errors to stderr and anything else to stdout.
I'm currently using the logrus package and have come across this implementation. Other ways i see achieving this while still using the same package is to pass the logger to each package that needs it or to create a global log object, neither of which i am too keen on.
https://github.com/microsoft/fabrikate/pull/252/commits/bd24d62d7c2b851ad6e7b36653eb0a6dc364474b#diff-ed0770fdbf87b0c6d536e33a99a8df9c
You can use Stackdriver library package for GoLang:
go get -u cloud.google.com/go/logging
Then you can use StandardLogger:
// Sample stdlogging writes log.Logger logs to the Stackdriver Logging.
package main
import (
"context"
"log"
"cloud.google.com/go/logging"
)
func main() {
ctx := context.Background()
// Sets your Google Cloud Platform project ID.
projectID := "YOUR_PROJECT_ID"
// Creates a client.
client, err := logging.NewClient(ctx, projectID)
if err != nil {
log.Fatalf("Failed to create client: %v", err)
}
defer client.Close()
// Sets the name of the log to write to.
logName := "my-log"
logger := client.Logger(logName).StandardLogger(logging.Info)
// Logs "hello world", log entry is visible at
// Stackdriver Logs.
logger.Println("hello world")
}
Here you can find documentation on Google Cloud website
Update:
Alternatively you could give a try GCP formatter for logrus
This will not tie your app to Google Cloud Platform. However, it does not mean that on another platform you will not need to change your code to format output.
Using StackDriver library is the recommended solution for Google Cloud.
We use https://github.com/rs/zerolog with the following method called in our Init() method to setup the logging options at global level:
package main
import (
"github.com/rs/zerolog"
"github.com/rs/zerolog/log"
"os"
)
// initializeLogging sets up the logging configuration for the service.
// Invoke this method in your Init() method.
func initializeLogging() {
// Set logging options for production development
if os.Getenv("ENV") != "DEV" {
// change the level field name to ensure these are parsed correctly in Stackdriver
zerolog.LevelFieldName = "severity"
// UNIX Time is faster and smaller than most timestamps
zerolog.TimeFieldFormat = zerolog.TimeFormatUnix
} else {
// Set logging options for local development
log.Logger = log.Output(zerolog.ConsoleWriter{Out: os.Stderr})
zerolog.SetGlobalLevel(zerolog.DebugLevel)
}
// Example log
log.Info().Msg("This is how you log at Info level")
}
I recommend the https://github.com/apsystole/log. You can swap either log and logrus imports for it. It is a small zero-dependency module, hence much lighter than logrus.
I am using logrus library for structured logging in my Go project.
I have configured my logrus as below:
// Global variable for logging
var gLog = &Logger{moduleName: ModuleName, logrus: logrus.New()}
type Logger struct {
moduleName string
logrus *logrus.Logger
}
func SetupGlobalLogger(logPrefix string, logMode string) error {
if logMode == "file" {
logFilePath := fmt.Sprintf("var/%s.log", vite.Environment())
file, err := os.OpenFile(logFilePath, logFileFlags, logFilePermission)
if err != nil {
return err
}
gLog.logrus.SetOutput(file)
// redirect logs written using standard log library to same place as logrus
log.SetOutput(gLog.logrus.Writer())
log.Println(vite.MarkInfo, "redirect log to file:", logFilePath)
}
return nil
}
In this Go project, there are several places where standard log library statements like log.Println() are used.
I want to redirect those log messages to logrus.
For that I am using following statement in above code.
log.SetOutput(gLog.logrus.Writer())
My question is: Is this thread safe?
If one thread/go-routine is executing log.Println() while another executing gLog.logrus.Info() or something on logrus, will that be fine?
Is Logrus Thread-safe?
Yes:
Thread safety
By default, Logger is protected by a mutex for concurrent writes.
Except when it isn't.
Most of the non-thread-safe cases are bugs which will occur only in rare situations. Whether any of them matter to you depends, of course, on your use case.
I have an application which needs configuration and I’ve created a configuration struct and I’m entering the configuration as a parameter to the function. The problem is that the configuration struct becomes bigger (like monolith) and bigger and I move the config to different functions in my app and which doesn’t need all the fields, just few of them. My question is if there is better approach to implement it in Go.
After struggling to find good way I’ve found this article (which a bit old but hopefully still relevant) and I wonder how and if I can use it to solve my problem.
Functional options instead of config struct
https://dave.cheney.net/2014/10/17/functional-options-for-friendly-apis
I need to inject some configuration properties to my application in
For example for function run (which is entry point ) I need to inject the log level and some other env variable like port host
For function build I need to “inject” the build flavor and build type etc.
Any example for my content will be very helpful
How to structure it in the code ?
How to implement it?
update
I need some E2E example how can I use the functional approach for different configs in the same package and other packages
It sounds like you're looking for an alternative to passing around the same configuration monolith structure to every package and every function. There are many solutions to this problem (more than I'm going to list here), and which one is right for you requires more knowledge of your code and your goals than we have, so it's probably best if you decide. And it sounds like you're wondering whether Dave Cheney's post on functional options provides a solution and how to apply it.
If your application's configuration is static in that it's not likely to change (mutate) through different threads of execution, and you don't need to create multiple instances with different configurations in the same main, then one option is package level variables and package initialization. If you object to exported package variables, you can use unexported package variables and control access via exported functions. Say run and build are two different packages:
// package main
import(
"github.com/profilename/appname/build"
"github.com/profilename/appname/run"
)
func main() {
// do something to get configuration values
build.Initialize(buildFlavor, buildType)
// any post-build-initialize-pre-run-initialize stuff
run.Initialize(logLevel, port, host)
// other processing
build.PreBuild("title") // other build functions may rely on configuration
build.Build()
// other stuff
run.ReadFiles(f1, f2)
run.Validate(preferredBackupPort) // port availability, chance to log.Fatal out
run.Run()
// cleanup
}
// package run
var Host string
var LogLevel, Port int
init() {
Host = `localhost`
Port = 8888
Loglevel = 1
}
func Initialize(logLevel, port int, host string) {
// validation, panic on failure
LogLevel = logLevel
Host = host
Port = port
}
func Run() {
// do something with LogLevel, Host, Port
}
But that doesn't solve the problem addressed in Dave Cheney's post. What if the user is running this without host, port, or buildType (or other configuration variables), because he doesn't need those features? What if the user wants to run multiple instances with different configurations?
Dave's approach is primarily intended for situations where you will not use package-level variables for configuration. Indeed, it is meant to enable several instances of a thing where each instance can have a different configuration. Your optional configuration parameters become a single variadic parameter where the type is a function that modifies a pointer to the thing being configured. For you, that could be
// package run
type Runner struct {
Port int
// rest of runner configuration
}
func NewRunner(options ...func(*Runner)) (runner *Runner, err error) {
// any setup
for _, option := range options {
err = option(runner)
if err != nil {
// do something
}
}
return runner, err
}
// package main
func main() {
// do something to get configuration values
port := func(runner *Runner) {
runner.Port = configuredPort
}
// other configuration if applicable
runner := run.NewRunner(port)
// ...
In a way, Dave's approach appears targeted at packages that will be used as very flexible libraries, and will provide application interfaces that users might wish to create several instances of. It allows for main definitions that launch multiple instances with different configurations. In that post he doesn't go into detail on how to process configuration input in the main or on a configuration package.
Note that the way the port is set in the resulting code above is not very different from this:
// package run
type Runner struct {
Port int
// rest of runner configuration
}
// package main, func main()
runner := new(run.Runner)
runner.Port = configuredPort
which is more traditional, probably easier for most developers to read and understand, and a perfectly fine approach if it suits your needs. (And you could make runner.port unexported and add a func (r *Runner) SetPort(p int) { r.port = p } method if you wanted.) It is also a design that has the potential, depending on implementation, to deal with mutating configuration, multiple threads of execution (you'll need channels or the sync package to deal with mutation there), and multiple instances.
Where the function options design Dave proposed becomes much more powerful than that approach is when you have many more statements related to the setting of the option that you want to place in main rather than in run -- those will make up the function body.
UPDATE Here's a runnable example using Dave's functional options approach, in two files. Be sure to update the import path to match wherever you put the run package.
Package run:
package run
import(
"fmt"
"log"
)
const(
DefaultPort = 8888
DefaultHost = `localhost`
DefaultLogLevel = 1
)
type Runner struct {
Port int
Host string
LogLevel int
}
func NewRunner(options ...func(*Runner) error) (runner *Runner) {
// any setup
// set defaults
runner = &Runner{DefaultPort, DefaultHost, DefaultLogLevel}
for _, option := range options {
err := option(runner)
if err != nil {
log.Fatalf("Failed to set NewRunner option: %s\n", err)
}
}
return runner
}
func (r *Runner) Run() {
fmt.Println(r)
}
func (r *Runner) String() string {
return fmt.Sprintf("Runner Configuration:\n%16s %22d\n%16s %22s\n%16s %22d",
`Port`, r.Port, `Host`, r.Host, `LogLevel`, r.LogLevel)
}
Package main:
package main
import(
"errors"
"flag"
"github.com/jrefior/run" // update this path for your filesystem
)
func main() {
// do something to get configuration values
portFlag := flag.Int("p", 0, "Override default listen port")
logLevelFlag := flag.Int("l", 0, "Override default log level")
flag.Parse()
// put your runner options here
runnerOpts := make([]func(*run.Runner) error, 0)
// with flags, we're not sure if port was set by flag, so test
if *portFlag > 0 {
runnerOpts = append(runnerOpts, func(runner *run.Runner) error {
if *portFlag < 1024 {
return errors.New("Ports below 1024 are privileged")
}
runner.Port = *portFlag
return nil
})
}
if *logLevelFlag > 0 {
runnerOpts = append(runnerOpts, func(runner *run.Runner) error {
if *logLevelFlag > 8 {
return errors.New("The maximum log level is 8")
}
runner.LogLevel = *logLevelFlag
return nil
})
}
// other configuration if applicable
runner := run.NewRunner(runnerOpts...)
runner.Run()
}
Example usage:
$ ./program -p 8987
Runner Configuration:
Port 8987
Host localhost
LogLevel 1
I use this to define per package Config Structs which are easier to manage and are loaded at the app start.
Define your config struct like this
type Config struct {
Conf1 package1.Configuration `group:"conf1" namespace:"conf1"`
Conf2 package2.Configuration `group:"conf2" namespace:"conf2"`
Conf3 Config3 `group:"conf3" namespace:"conf3"`
GeneralSetting string `long:"Setting" description:"setting" env:"SETTING" required:"true"`
}
type Config3 struct {
setting string
}
And use "github.com/jessevdk/go-flags" to pass either --config3.setting=stringValue cmd arguments, or ENV variables export CONFIG3_SETTING=stringValue:
type Configuration interface {}
const DefaultFlags flags.Options = flags.HelpFlag | flags.PassDoubleDash
func Parse(cfg Configuration) []string {
args, _ := flags.NewParser(cfg, DefaultFlags).Parse()
return args
}
And your main should look something like this:
func main() {
// Parse the configuration.
var cfg Config
Parse(&cfg)
service := NewService(cfg.Conf3.Setting)
}
I am using logrus in a Go app. I believe this question is applicable to any other logging package (which doesn't offer external file based configuration) as well.
logrus provides functions to setup various configuration, e.g. SetOutput, SetLevel etc.
Like any other application I need to do logging from multiple source files/packages, it seems you need to setup these options in each file with logrus.
Is there any way to setup these options once somewhere in a central place to be shared all over the application. That way if I have to make logging level change I can do it in one place and applies to all the components of the app.
You don't need to set these options in each file with Logrus.
You can import Logrus as log:
import log "github.com/Sirupsen/logrus"
Then functions like log.SetOutput() are just functions and modify the global logger and apply to any file that includes this import.
You can create a package global log variable:
var log = logrus.New()
Then functions like log.SetOutput() are methods and modify your package global. This is awkward IMO if you have multiple packages in your program, because each of them has a different logger with different settings (but maybe that's good for some use cases). I also don't like this approach because it confuses goimports (which will want to insert log into your imports list).
Or you can create your own wrapper (which is what I do). I have my own log package with its own logger var:
var logger = logrus.New()
Then I make top-level functions to wrap Logrus:
func Info(args ...interface{}) {
logger.Info(args...)
}
func Debug(args ...interface{}) {
logger.Debug(args...)
}
This is slightly tedious, but allows me to add functions specific to my program:
func WithConn(conn net.Conn) *logrus.Entry {
var addr string = "unknown"
if conn != nil {
addr = conn.RemoteAddr().String()
}
return logger.WithField("addr", addr)
}
func WithRequest(req *http.Request) *logrus.Entry {
return logger.WithFields(RequestFields(req))
}
So I can then do things like:
log.WithConn(c).Info("Connected")
(I plan in the future to wrap logrus.Entry into my own type so that I can chain these better; currently I can't call log.WithConn(c).WithRequest(r).Error(...) because I can't add WithRequest() to logrus.Entry.)
This is the solution that I arrived at for my application that allows adding of fields to the logging context. It does have a small performance impact due to the copying of context base fields.
package logging
import (
log "github.com/Sirupsen/logrus"
)
func NewContextLogger(c log.Fields) func(f log.Fields) *log.Entry {
return func(f log.Fields) *log.Entry {
for k, v := range c {
f[k] = v
}
return log.WithFields(f)
}
}
package main
import (
"logging"
)
func main {
app.Logger = logging.NewContextLogger(log.Fields{
"module": "app",
"id": event.Id,
})
app.Logger(log.Fields{
"startTime": event.StartTime,
"endTime": event.EndTime,
"title": event.Name,
}).Info("Starting process")
}