I am writing an application in which i need to record logs in two different files. For Example weblogs.go and debuglogs.go. I tried using the log4go but my requirement is i need the logger to be created in main file and be accessible in sub directory as major of decoding and logging is done in sub file. Can anybody please help with that?
Here's one way to do it, using the standard log package:
package main
import (
"io"
"log"
"os"
)
func main() {
f1, err := os.Create("/tmp/file1")
if err != nil {
panic(err)
}
defer f1.Close()
f2, err := os.Create("/tmp/file2")
if err != nil {
panic(err)
}
defer f2.Close()
w := io.MultiWriter(os.Stdout, f1, f2)
logger := log.New(w, "logger", log.LstdFlags)
myfunc(logger)
}
func myfunc(logger *log.Logger) {
logger.Print("Hello, log file!!")
}
Notes:
io.MultiWriter is used to combine several writers together. Here, it creates a writer w - a write to w will go to os.Stdout as well as two files
log.New lets us create a new log.Logger object with a custom writer
The log.Logger object can be passed around to functions and used by them to log things
This is how you can manage logs for debugging and prod envoirments with multi log files:
First, make a type for managing multiple loggers you can make a separate file for this like logging.go
type LOGGER struct {
debug *log.Logger
prod *log.Logger
.
.
.
}
For getting the LOGGER pointer anywhere in your project the best approach will be to get it through the singleton pattern like
var lock = &sync.Mutex{}
var loggers *LOGGER
func GetLoggerInstance() *LOGGER {
if singleInstance == nil {
lock.Lock()
defer lock.Unlock()
if loggers == nil {
fmt.Println("Creating LOGGER instance now.")
loggers = &LOGGER{}
} else {
fmt.Println("LOGGER instance already created.")
}
} else {
fmt.Println("LOGGER instance already created.")
}
return loggers
}
Now setup logger for debug env in any pkg or directory level better is to setup it at the root level
logger := GetLoggerInstance()
f, err := os.OpenFile("path/to/debug/log/file", os.O_RDWR|os.O_CREATE|os.O_APPEND, 0666)
if err != nil {
fmt.Println("debug log file not created", err.Error())
}
loggers.debug = log.New(f, "[DEBUG]", log.Ldate|log.Ltime|log.Lmicroseconds|log.LUTC)
loggers.debug.Println("This is debug log message")
With the same approach, you can also create prod or as many as you want.
logger := GetLoggerInstance()
f, err = os.OpenFile("path/to/prod/log/file", os.O_RDWR|os.O_CREATE|os.O_APPEND, 0666)
if err != nil {
fmt.Println("prod log file not created", err.Error())
}
loggers.prod = log.New(f, "[PROD]", log.Ldate|log.Ltime|log.Lmicroseconds|log.LUTC)
loggers.prod.Println("This is prod log message")
Related
My goal is to create a logger that I can use to output to stdout (info logs) and stderr (error logs) as well as respective files (info.log) and (errors.log) all at the same time.
I am currently using loggers in Go like the following:
package main
import (
"log"
"os"
"io"
)
func main() {
// Current method
infoLog := log.New("./data/info.log", "INFO\t", log.Ldate|log.Ltime)
errorLog := log.New("./data/errors.log", "ERROR\t", log.Ldate|log.Ltime|log.Lshortfile)
infoLog.Println("Hello INFO!")
errorLog.Println("Hello ERROR!")
// I've read about using io.MultiWriter like the following in order to write to stdout/stderr as well as a flat file at the same time
f, err := os.OpenFile("./data/info.log", os.O_RDWR|os.O_CREATE|os.O_APPEND, 0666)
if err != nil {
log.Fatalf("error opening file: %v", err)
}
defer f.Close()
wrt := io.MultiWriter(os.Stdout, f)
log.SetOutput(wrt)
log.Println("Hello World!")
}
I'd like to basically take the formatting of infoLog and errorLog (like in my first logging solution) like log.Ldate|log.Ltime and apply it to the io.MultiWriter. Is there a way to do this?
For logging info and error in file with standard format that other logging services such as filebeat, logstash and etc can understand it I suggest you to use github.com/sirupsen/logrus.
This package has SetFormater function to define format of output log. For example:
logrus.SetFormatter(&logrus.JSONFormatter{
FieldMap: logrus.FieldMap{
logrus.FieldKeyTime: "#timestamp",
logrus.FieldKeyMsg: "message",
logrus.FieldKeyFunc: "func",
logrus.FieldKeyFile: "file",
},
})
and you can define file for logging. For example:
file, err := os.OpenFile("payment_logs.log", os.O_RDWR|os.O_CREATE|os.O_APPEND, 0666)
And set log is very easy and standard:
logger.Info("This is an info message")
logger.Warn("This is a warning message")
logger.Error("This is an error message")
I have this method used in a lambda:
import (
"os"
"go.uber.org/zap"
"go.uber.org/zap/zapcore"
)
func InitLogger() *zap.Logger {
config := zap.NewProductionEncoderConfig()
config.EncodeTime = zapcore.RFC3339TimeEncoder
consoleEncoder := zapcore.NewJSONEncoder(config)
core := zapcore.NewTee(zapcore.NewCore(consoleEncoder, zapcore.AddSync(os.Stdout), zapcore.InfoLevel))
return zap.New(core).With()
}
And in my lambda Handler i have:
var (
log *zap.Logger
)
func init() {
log = u.InitLogger()
}
func handler(r events.APIGatewayProxyRequest) (*events.APIGatewayProxyResponse, error) {
out, err := exec.Command("uuidgen").Output()
uuid := strings.ReplaceAll(string(out), "\n", "")
if err != nil {
log.Error(err.Error())
}
log.Info("PRINT_1", zap.Any("uuid", uuid), zap.Any("Request", r.Body))
}
I have a question, is possible add the UUID to all logs without adding one by one?, because in each log that I need print something, I need add zap.Any("uuid", uuid)
The problem is that I need pass as parameter to all methods the UUID to print it in the log info, or error.
You will have to slightly re-arrange your code since you're only creating the UUID in the handler, which implies it's request-specific whilst the logger is global...
But the gist, specific to the library, is that you've got to create a child logger (which you are, in fact, already doing: you just need to pass the fields there). Any subsequent log writes to the child logger will include those fields.
For example:
func main() {
logger := InitLogger(zap.String("foo", "bar"))
logger.Info("First message with our `foo` key")
logger.Info("Second message with our `foo` key")
}
func InitLogger(fields ...zap.Field) *zap.Logger {
config := zap.NewProductionEncoderConfig()
config.EncodeTime = zapcore.RFC3339TimeEncoder
consoleEncoder := zapcore.NewJSONEncoder(config)
core := zapcore.NewTee(zapcore.NewCore(consoleEncoder, zapcore.AddSync(os.Stdout), zapcore.InfoLevel))
return zap.New(core).With(fields...)
}
Output:
{"level":"info","ts":"2022-11-24T18:30:45+01:00","msg":"First message with our `foo` key","foo":"bar"}
{"level":"info","ts":"2022-11-24T18:30:45+01:00","msg":"Second message with our `foo` key","foo":"bar"}
Based off the configurations for zap.NewDevelopmentConfig() and zap.NewProductionConfig(), I've assumed that zap writes logs to stderr. However, I can't seem to capture the output in unit tests.
I have the following captureOutput func:
func captureOutput(f func()) string {
r, w, err := os.Pipe()
if err != nil {
panic(err)
}
stdout := os.Stdout
os.Stdout = w
defer func() {
os.Stdout = stdout
}()
stderr := os.Stderr
os.Stderr = w
defer func() {
os.Stderr = stderr
}()
f()
w.Close()
var buf bytes.Buffer
io.Copy(&buf, r)
return buf.String()
}
It fails to capture zap output but does manage to grab output from fmt.Println(...):
func TestZapCapture(t *testing.T) {
auditor, _ := zap.NewProduction()
output := captureOutput(func() {
auditor.Info("hi")
})
assert.NotEmpty(t, output)
//fails to captures output
}
func TestFmtCapture(t *testing.T) {
output := captureOutput(func() {
fmt.Println("hi")
})
assert.NotEmpty(t, output)
//successfully captures output
}
I'm aware of using the zap observer for situations like this but my real use case is to test a highly modified zap logger so testing a new zap.Core would defeat the purpose. Whats the best way to capture that output?
Test that messages are logged at all
Use zapcore.NewTee. In your unit tests, you instantiate a logger whose core is comprised of your own highly modified core and the observed core tee'd together. The observed core will receive the log entries, so you can assert that single fields are what you expect (level, message, fields, etc.)
func main() {
// some arbitrary custom core logger
mycore := zapcore.NewCore(
zapcore.NewJSONEncoder(zap.NewProductionEncoderConfig()),
os.Stderr,
zapcore.InfoLevel,
)
// test core
observed, logs := observer.New(zapcore.InfoLevel)
// new logger with the two cores tee'd together
logger := zap.New(zapcore.NewTee(mycore, observed))
logger.Error("foo")
entry := logs.All()[0]
fmt.Println(entry.Message == "foo") // true
fmt.Println(entry.Level == zapcore.ErrorLevel) // true
}
Test the final log format
In this case you want to pipe the logger output to arbitrary writers. You can achieve this with zap.CombineWriteSyncers and inject this as a dependency of your custom core.
// this would be placed in your main code
func NewCustomLogger(pipeTo io.Writer) zapcore.Core {
return zapcore.NewCore(
zapcore.NewJSONEncoder(zap.NewProductionEncoderConfig()),
zap.CombineWriteSyncers(os.Stderr, zapcore.AddSync(pipeTo)),
zapcore.InfoLevel,
)
}
func TestLogger(t *testing.T) {
b := &bytes.Buffer{}
// call the constructor from your test code with the arbitrary writer
mycore := NewCustomLogger(b)
logger := zap.New(mycore)
logger.Error("foo")
fmt.Println(b.String()) // {"level":"error","ts":1639813360.853494,"msg":"foo"}
}
I am using go.uber.org/zap/zapcore for logging in my Go app.
package logger
import (
"go.uber.org/zap"
"go.uber.org/zap/zapcore"
"log"
)
var l *zap.Logger
func Get() *zap.Logger {
return l
}
func Init() {
conf := zap.NewProductionConfig()
logger, err := conf.Build()
if err != nil {
log.Fatal("Init logger failed", err)
}
l = logger
}
I also have Sentry project and use github.com/getsentry/raven-go.
I want to send logs at error level and above to Sentry.
For example when logging at info level with logger.Info() I want to just log them as usual, but in case of error or fatal logs I need send these messages to Sentry. How can I achieve that?
The answer is you should use zap wrapper for adding hooks then you have to use the function of logger which is called WithOptions
sentryOptions := zap.WrapCore(func(core zapcore.Core) zapcore.Core {
return zapcore.RegisterHooks(core, func(entry zapcore.Entry) error {
// your logic here
})
})
logger.WithOptions(sentryOptions)
The following will capture the message and send it to the sentry when an error level is detected, with customized error line number and message.
err := sentry.Init(sentry.ClientOptions{Dsn: "http://~~~~~"})
if err != nil {
log.fatal("Sentry Error Setup ::", err.Error())
}
logger, _ := zap.NewDevelopment(zap.Hooks(func(entry zapcore.Entry) error {
if entry.Level == zapcore.ErrorLevel {
defer sentry.Flush(2 * time.Second)
sentry.CaptureMessage(fmt.Sprintf("%s, Line No: %d :: %s", entry.Caller.File, entry.Caller.Line, entry.Message))
}
return nil
}))
sugar := logger.Sugar()
I'm trying to build a customized zap logger with 1) customized *zap.Config and 2) lumberjack, but can't find proper example to apply both configurations.
Since config.Build does not accept WriteSync as an input. Do you know how to achieve this?
func genBaseLoggerZap() Logger {
ex, err := os.Executable()
if err != nil {
Fatalf("Failed to get os.Executable, err: %v", err)
}
zlManager.outputPath = path.Join(filepath.Dir(ex), zlManager.outputPath)
// Want to add sync here..
zapcore.AddSync(&lumberjack.Logger{
Filename: zlManager.outputPath + "123",
MaxSize: 500,
MaxBackups: 10,
MaxAge: 28,
})
return genLoggerZap(BaseLogger, genDefaultConfig())
}
// genLoggerZap creates a zapLogger with given ModuleID and Config.
func genLoggerZap(mi ModuleID, cfg *zap.Config) Logger {
logger, err := cfg.Build()
if err != nil {
Fatalf("Failed to generate zap logger, err: %v", err)
}
newLogger := &zapLogger{mi, cfg, logger.Sugar()}
newLogger.register()
return newLogger
}
You can add custom log destinations using the zap.RegisterSink function and the Config.OutputPaths field. RegisterSink maps URL schemes to Sink constructors, and OutputPaths configures log destinations (encoded as URLs).
Conveniently, *lumberjack.Logger implements almost all of the zap.Sink interface already. Only the Sync method is missing, which can be easily added with a thin wrapper type.
package main
import (
"net/url"
"go.uber.org/zap"
lumberjack "gopkg.in/natefinch/lumberjack.v2"
)
type lumberjackSink struct {
*lumberjack.Logger
}
// Sync implements zap.Sink. The remaining methods are implemented
// by the embedded *lumberjack.Logger.
func (lumberjackSink) Sync() error { return nil }
func main() {
zap.RegisterSink("lumberjack", func(u *url.URL) (zap.Sink, error) {
return lumberjackSink{
Logger: &lumberjack.Logger{
Filename: u.Opaque,
// Use query parameters or hardcoded values for remaining
// fields.
},
}, nil
})
config := zap.NewProductionConfig()
// Add a URL with the "lumberjack" scheme.
config.OutputPaths = append(config.OutputPaths, "lumberjack:foo.log")
log, _ := config.Build()
log.Info("test", zap.String("foo", "bar"))
}