Uber Zap logger function name in logs - go

How to get function name printed in logs from Uber Zap logging ?
This is the PR request with which they seemed to have added the functionality to output function names in log.
I am using golang version 1.15 and go.uber.org/zap v1.16.0
This is my code:
package main
import (
"go.uber.org/zap"
)
var logger *zap.Logger
func main() {
logger := NewLogger()
logger.Info("Test msg Main")
TestFunc(logger)
}
func TestFunc(logger *zap.Logger) {
logger.Info("Test msg TestFunc")
}
func NewLogger() *zap.Logger {
config := zap.NewDevelopmentConfig()
opts := []zap.Option{
zap.AddCallerSkip(1), // traverse call depth for more useful log lines
zap.AddCaller(),
}
logger, _ = config.Build(opts...)
return logger
}
This is the output I get with/without the addition of AddCaller() option
2021-03-01T15:00:02.927-0800 INFO runtime/proc.go:204 Test msg Main
2021-03-01T15:00:02.927-0800 INFO cmd/main.go:12 Test msg TestFunc
I am expecting something like
2021-03-01T15:00:02.927-0800 INFO runtime/proc.go:204 main Test msg Main
2021-03-01T15:00:02.927-0800 INFO cmd/main.go:12 TestFunc Test msg TestFunc

By default, the provided encoder presets (NewDevelopmentEncoderConfig used by NewDevelopmentConfig and NewProductionEncoderConfig used by NewProductionConfig) do not enable function name logging.
To enable function name, you need to enable caller (true by default) and set a non-empty value for config.EncoderConfig.FunctionKey.
Source: EncoderConfig
type EncoderConfig struct {
// Set the keys used for each log entry. If any key is empty, that portion
// of the entry is omitted.
...
CallerKey string `json:"callerKey" yaml:"callerKey"`
FunctionKey string `json:"functionKey" yaml:"functionKey"` // this needs to be set
StacktraceKey string `json:"stacktraceKey" yaml:"stacktraceKey"`
...
}
Example Console Logger:
func main() {
config := zap.NewDevelopmentConfig()
// if you're using console encoding, the FunctionKey value can be any
// non-empty string because console encoding does not print the key.
config.EncoderConfig.FunctionKey = "F"
logger, _ := config.Build()
logger.Info("Test Logging")
// Output: 2021-03-03T11:41:47.728+0800 INFO example/main.go:11 main.main Test Logging
}
Example JSON Logger:
func main() {
config := zap.NewProductionConfig()
// the FunctionKey value matters because it will become the JSON field
config.EncoderConfig.FunctionKey = "func"
logger, _ := config.Build()
log(logger)
// Output: {"level":"info","ts":1614743088.538128,"caller":"example/main.go:15","func":"main.log","msg":"Test Logging"}
}
func log(logger *zap.Logger) {
logger.Info("Test Logging")
}

Related

Zap logger add UUID to all logs in golang

I have this method used in a lambda:
import (
"os"
"go.uber.org/zap"
"go.uber.org/zap/zapcore"
)
func InitLogger() *zap.Logger {
config := zap.NewProductionEncoderConfig()
config.EncodeTime = zapcore.RFC3339TimeEncoder
consoleEncoder := zapcore.NewJSONEncoder(config)
core := zapcore.NewTee(zapcore.NewCore(consoleEncoder, zapcore.AddSync(os.Stdout), zapcore.InfoLevel))
return zap.New(core).With()
}
And in my lambda Handler i have:
var (
log *zap.Logger
)
func init() {
log = u.InitLogger()
}
func handler(r events.APIGatewayProxyRequest) (*events.APIGatewayProxyResponse, error) {
out, err := exec.Command("uuidgen").Output()
uuid := strings.ReplaceAll(string(out), "\n", "")
if err != nil {
log.Error(err.Error())
}
log.Info("PRINT_1", zap.Any("uuid", uuid), zap.Any("Request", r.Body))
}
I have a question, is possible add the UUID to all logs without adding one by one?, because in each log that I need print something, I need add zap.Any("uuid", uuid)
The problem is that I need pass as parameter to all methods the UUID to print it in the log info, or error.
You will have to slightly re-arrange your code since you're only creating the UUID in the handler, which implies it's request-specific whilst the logger is global...
But the gist, specific to the library, is that you've got to create a child logger (which you are, in fact, already doing: you just need to pass the fields there). Any subsequent log writes to the child logger will include those fields.
For example:
func main() {
logger := InitLogger(zap.String("foo", "bar"))
logger.Info("First message with our `foo` key")
logger.Info("Second message with our `foo` key")
}
func InitLogger(fields ...zap.Field) *zap.Logger {
config := zap.NewProductionEncoderConfig()
config.EncodeTime = zapcore.RFC3339TimeEncoder
consoleEncoder := zapcore.NewJSONEncoder(config)
core := zapcore.NewTee(zapcore.NewCore(consoleEncoder, zapcore.AddSync(os.Stdout), zapcore.InfoLevel))
return zap.New(core).With(fields...)
}
Output:
{"level":"info","ts":"2022-11-24T18:30:45+01:00","msg":"First message with our `foo` key","foo":"bar"}
{"level":"info","ts":"2022-11-24T18:30:45+01:00","msg":"Second message with our `foo` key","foo":"bar"}

Why custom encoding is lost after calling logger.With in Uber Zap?

(based on this question: Uber Zap Logger: how to prepend every log entry with a string)
I replaced the Encoder of my uber-zap logger with a custom one to prepend every log entry with a SystemD-friendly error level (<LEVEL>), but now after I use the logger with additional fields (With(fields ...Field)), the custom prepending is gone:
package main
import (
"os"
"go.uber.org/zap"
"go.uber.org/zap/buffer"
"go.uber.org/zap/zapcore"
)
func getConfig() zap.Config {
// your current config options
return zap.NewProductionConfig()
}
type prependEncoder struct {
// embed a zapcore encoder
// this makes prependEncoder implement the interface without extra work
zapcore.Encoder
// zap buffer pool
pool buffer.Pool
}
// EncodeEntry implementing only EncodeEntry
func (e *prependEncoder) EncodeEntry(entry zapcore.Entry, fields []zapcore.Field) (*buffer.Buffer, error) {
// new log buffer
buf := e.pool.Get()
// prepend the JournalD prefix based on the entry level
buf.AppendString(e.toJournaldPrefix(entry.Level))
buf.AppendString(" ")
// calling the embedded encoder's EncodeEntry to keep the original encoding format
consolebuf, err := e.Encoder.EncodeEntry(entry, fields)
if err != nil {
return nil, err
}
// just write the output into your own buffer
_, err = buf.Write(consolebuf.Bytes())
if err != nil {
return nil, err
}
return buf, nil
}
// some mapper function
func (e *prependEncoder) toJournaldPrefix(lvl zapcore.Level) string {
switch lvl {
case zapcore.DebugLevel:
return "<7>"
case zapcore.InfoLevel:
return "<6>"
case zapcore.WarnLevel:
return "<4>"
}
return ""
}
func main() {
cfg := getConfig()
// constructing our prependEncoder with a ConsoleEncoder using your original configs
enc := &prependEncoder{
Encoder: zapcore.NewConsoleEncoder(cfg.EncoderConfig),
pool: buffer.NewPool(),
}
logger := zap.New(
zapcore.NewCore(
enc,
os.Stdout,
zapcore.DebugLevel,
),
// this mimics the behavior of NewProductionConfig.Build
zap.ErrorOutput(os.Stderr),
)
logger.Info("this is info")
logger.Debug("this is debug")
logger.Warn("this is warn")
logger = logger.With(zap.String("foo", "bar"))
logger.With(zap.String("foo", "bar")).Info("this does not have the prefix :(")
}
The output I get is:
<6> 1.640656130756576e+09 info this is info
<7> 1.640656130756611e+09 debug this is debug
<4> 1.640656130756615e+09 warn this is warn
1.6406561307566311e+09 info this does not have the prefix :( {"foo": "bar"}
What am I doing wrong?
You have to also implement Clone() from the zapcore.Encoder interface. If you wish to keep the parent logger unaltered, you have to construct an actual clone — possibly with the same config, so you might want to store it as a field:
type prependEncoder struct {
zapcore.Encoder
pool buffer.Pool
cfg zapcore.EncoderConfig
}
func (e *prependEncoder) Clone() zapcore.Encoder {
return &prependEncoder{
// cloning the encoder with the base config
Encoder: zapcore.NewConsoleEncoder(e.cfg),
pool: buffer.NewPool(),
cfg: e.cfg,
}
}
If you don't implement it, the method that runs is the next shallowest one when calling logger.Clone(), which is the Clone() declared on the embedded zapcore.Encoder. That one then doesn't have your custom EncodeEntry anymore.
Now running the following:
logger.Info("this is info")
logger.Debug("this is debug")
logger.Warn("this is warn")
child := logger.With(zap.String("foo", "bar"))
logger.Warn("original")
child.Info("new one")
Outputs:
<6> INFO this is info
<7> DEBUG this is debug
<4> WARN this is warn
cloning...
<4> WARN original
<6> INFO new one {"foo": "bar"}

How to mock zap logger from ctrl "sigs.k8s.io/controller-runtime"?

package logger
import (
"bytes"
. "github.com/onsi/ginkgo"
. "github.com/onsi/gomega"
ctrl "sigs.k8s.io/controller-runtime"
)
var _ = Describe("Logger", func() {
It("Test Default Log Level", func() {
buf := &bytes.Buffer{}
testLog := ctrl.Log.WithName("setup")
SetLogger()
testLog.Info("This is a test")
Expect(buf.String(),"This is a test")
})
})
And this is the SetLogger function, which is used also in production:
package logger
import (
ctrl "sigs.k8s.io/controller-runtime"
"sigs.k8s.io/controller-runtime/pkg/log/zap"
...
)
func SetLogger() {
opts := zap.Options{
Development: developmentFlag,
StacktraceLevel: stacktraceLevel,
Level: isLevelEnabler,
Encoder: logFmtEncoder,
}
ctrl.SetLogger(zap.New(zap.UseFlagOptions(&opts)))
}
How I can change the output of the testLog.Info to buffer?
If you are only interested in testing the log message, you can use a hook.
In particular zap.Hooks function constructs a zap.Option from a variable number of hooks. A hook is just a func(entry zapcore.Entry) error which you can use to intercept the entry and write its message to the buffer.
To set this zap.Option into your sigs.k8s.io logger, you set it to the ZapOpts field:
opts := k8szap.Options{
// ...
ZapOpts: []zap.Option{
zap.Hooks(func(entry zapcore.Entry) error {
buf.WriteString(entry.Message)
return nil
}),
},
}
So since you need access to the buffer, you can pass it as argument to the SetLogger function:
func SetLogger(buf *bytes.Buffer) {
opts := zap.Options{
Development: developmentFlag,
StacktraceLevel: stacktraceLevel,
Level: isLevelEnabler,
Encoder: logFmtEncoder,
// here 'zap' selector is 'go.uber.org/zap'
ZapOpts: []zap.Option{
zap.Hooks(func(entry zapcore.Entry) error {
buf.WriteString(entry.Message)
return nil
}),
},
}
// here I call 'k8szap' selector the package 'sigs.k8s.io/controller-runtime/pkg/log/zap'
ctrl.SetLogger(k8szap.New(k8szap.UseFlagOptions(&opts)))
}
And then in your test function:
It("Test Default Log Level", func() {
buf := &bytes.Buffer{}
testLog := ctrl.Log.WithName("setup")
// pass buffer to SetLogger
SetLogger(buf)
testLog.Info("This is a test")
Expect(buf.String(), "This is a test")
})
Minimal example (it may timeout when downloading the packages in the playground): https://play.golang.org/p/oBN3SHFKVC8

Is it possible to wrap logrus.Logger functions without losing the line number prefix?

When using the wrapped logrus function/logger, the logger prefixes all log lines with the file name and line number of the logger function call, for example:
INFO[0000]logging.go:39 myfolder/logging.Info()
If I wrap the log function like this, for instance:
package logging
import (
"fmt"
"github.com/sirupsen/logrus"
"os"
"path"
"runtime"
)
var (
log *logrus.Logger
)
func init() {
log = logrus.New()
log.SetReportCaller(true)
log.Formatter = &logrus.TextFormatter{
CallerPrettyfier: func(f *runtime.Frame) (string, string) {
filename := path.Base(f.File)
return fmt.Sprintf("%s()", f.Function), fmt.Sprintf("%s:%d", filename, f.Line)
},
}
}
func Info(args ...interface{}) {
log.Info(args...)
}
Every line emitted by this function is going to be prefixed with the line number of the logging function call. That is as expected, but the desired behavior is for each line to be prefixed with the line number of the line where Info is called.
The Desired output should be :
INFO[0000]myfile.go:39 myfolder/myfile.myfunction()
Is there any way around it?
It is not possible to do it in the logrus. I had a similar requirement and ended up doing the following which worked for us.
package mylog
import (
"fmt"
"github.com/Sirupsen/logrus"
"runtime"
"strings"
)
var logger = logrus.New()
func SetLogFormatter(formatter logrus.Formatter) {
logger.Formatter = formatter
}
// Info logs a message at level Info on the standard logger.
func Info(args ...interface{}) {
if logger.Level >= logrus.InfoLevel {
entry := logger.WithFields(logrus.Fields{})
entry.Data["file"] = fileInfo(2)
entry.Info(args...)
}
}
func fileInfo(skip int) string {
_, file, line, ok := runtime.Caller(skip)
if !ok {
file = "<???>"
line = 1
} else {
slash := strings.LastIndex(file, "/")
if slash >= 0 {
file = file[slash+1:]
}
}
return fmt.Sprintf("%s:%d", file, line)
}
See if this or some variation of this works for your use case. I have removed the application-specific code from the code snippet above.
Hoping that I am not misunderstanding so you want the "actual" path and line number where the logger was called. The code (json format as an example) below should give you want you want. If you want to add more info, such as the function name etc., just modify caller() method.
logrus.SetReportCaller(true)
// ...
logrus.SetFormatter(&logrus.JSONFormatter{
CallerPrettyfier: caller(),
FieldMap: logrus.FieldMap{
logrus.FieldKeyFile: "caller",
},
})
// caller returns string presentation of log caller which is formatted as
// `/path/to/file.go:line_number`. e.g. `/internal/app/api.go:25`
func caller() func(*runtime.Frame) (function string, file string) {
return func(f *runtime.Frame) (function string, file string) {
p, _ := os.Getwd()
return "", fmt.Sprintf("%s:%d", strings.TrimPrefix(f.File, p), f.Line)
}
}
{
"caller": "/internal/controller/create.go:21",
"level": "info",
"msg": "i am a dummy log",
"time": "2020-08-30T19:17:48+01:00"
}

How to get file and function name for loggers

I'm using logrus OS which works as expected, now we have a requirement to add to the logger output the file and the function which from where you put the logger call,
we need it to be something like
File log-ut-usage
func main(){
logs := lts.InitLogger("test","1","debug")
logs.Debugf("test 123")
....
}
This is the required output
{"file":"log-ut-usage/main.go:21","function":"main","level":"warn","test 123":"ddd","timestamp":"2019-10-02T09:21:39.309559Z"}
currently we got the file and function of the
file logger.go
func InitLog(label string) LoggerI {
loggerImpl = &logrus.Logger{
Out: os.Stdout,
Level: level,
ReportCaller: true,
Formatter: &logrus.JSONFormatter{
TimestampFormat: timestampFormat,
CallerPrettyfier: func(f *runtime.Frame) (string, string) {
s := strings.Split(f.Function, ".")
funcname := s[len(s)-1]
_, filename := path.Split(f.File)
return funcname, filename
},
},
}
This is the (unwanted) output
{"file":"logger.go","func":"InitLog","level":"debug","msg":"test 123","time":"2019-10-02 12:21:39"}
I dont want to get the file logger.go where we coded the json formater, I want to get the file that with the usage of the logger .
You can wrap your logger with file, function and line information and then use that.
Here's an example (live):
package main
import (
"os"
"runtime"
"strconv"
"strings"
log "github.com/sirupsen/logrus"
)
func init() {
log.SetFormatter(&log.JSONFormatter{})
log.SetOutput(os.Stdout)
}
func logger() *log.Entry {
pc, file, line, ok := runtime.Caller(1)
if !ok {
panic("Could not get context info for logger!")
}
filename := file[strings.LastIndex(file, "/")+1:] + ":" + strconv.Itoa(line)
funcname := runtime.FuncForPC(pc).Name()
fn := funcname[strings.LastIndex(funcname, ".")+1:]
return log.WithField("file", filename).WithField("function", fn)
}
func test() {
logger().Info("Testing...")
}
func main() {
logger().Info("Testing...")
test()
}
Output:
{"file":"prog.go:34","function":"main","level":"info","msg":"Testing...","time":"2009-11-10T23:00:00Z"}
{"file":"prog.go:30","function":"test","level":"info","msg":"Testing...","time":"2009-11-10T23:00:00Z"}
Have you tried to use debug.Stack() to fetch the file name which calls InitLog?
https://play.golang.org/p/g6yLGsiuEEn
goroutine 1 [running]:
runtime/debug.Stack(0x15d6b0, 0x3, 0x68360, 0x1580)
/usr/local/go/src/runtime/debug/stack.go:24 +0xc0
main.fun2()
/tmp/sandbox834348417/prog.go:20 +0x20
main.fun1(...)
/tmp/sandbox834348417/prog.go:15
main.main()
/tmp/sandbox834348417/prog.go:10 +0x20
Hope it works.

Resources