Why custom encoding is lost after calling logger.With in Uber Zap? - go

(based on this question: Uber Zap Logger: how to prepend every log entry with a string)
I replaced the Encoder of my uber-zap logger with a custom one to prepend every log entry with a SystemD-friendly error level (<LEVEL>), but now after I use the logger with additional fields (With(fields ...Field)), the custom prepending is gone:
package main
import (
"os"
"go.uber.org/zap"
"go.uber.org/zap/buffer"
"go.uber.org/zap/zapcore"
)
func getConfig() zap.Config {
// your current config options
return zap.NewProductionConfig()
}
type prependEncoder struct {
// embed a zapcore encoder
// this makes prependEncoder implement the interface without extra work
zapcore.Encoder
// zap buffer pool
pool buffer.Pool
}
// EncodeEntry implementing only EncodeEntry
func (e *prependEncoder) EncodeEntry(entry zapcore.Entry, fields []zapcore.Field) (*buffer.Buffer, error) {
// new log buffer
buf := e.pool.Get()
// prepend the JournalD prefix based on the entry level
buf.AppendString(e.toJournaldPrefix(entry.Level))
buf.AppendString(" ")
// calling the embedded encoder's EncodeEntry to keep the original encoding format
consolebuf, err := e.Encoder.EncodeEntry(entry, fields)
if err != nil {
return nil, err
}
// just write the output into your own buffer
_, err = buf.Write(consolebuf.Bytes())
if err != nil {
return nil, err
}
return buf, nil
}
// some mapper function
func (e *prependEncoder) toJournaldPrefix(lvl zapcore.Level) string {
switch lvl {
case zapcore.DebugLevel:
return "<7>"
case zapcore.InfoLevel:
return "<6>"
case zapcore.WarnLevel:
return "<4>"
}
return ""
}
func main() {
cfg := getConfig()
// constructing our prependEncoder with a ConsoleEncoder using your original configs
enc := &prependEncoder{
Encoder: zapcore.NewConsoleEncoder(cfg.EncoderConfig),
pool: buffer.NewPool(),
}
logger := zap.New(
zapcore.NewCore(
enc,
os.Stdout,
zapcore.DebugLevel,
),
// this mimics the behavior of NewProductionConfig.Build
zap.ErrorOutput(os.Stderr),
)
logger.Info("this is info")
logger.Debug("this is debug")
logger.Warn("this is warn")
logger = logger.With(zap.String("foo", "bar"))
logger.With(zap.String("foo", "bar")).Info("this does not have the prefix :(")
}
The output I get is:
<6> 1.640656130756576e+09 info this is info
<7> 1.640656130756611e+09 debug this is debug
<4> 1.640656130756615e+09 warn this is warn
1.6406561307566311e+09 info this does not have the prefix :( {"foo": "bar"}
What am I doing wrong?

You have to also implement Clone() from the zapcore.Encoder interface. If you wish to keep the parent logger unaltered, you have to construct an actual clone — possibly with the same config, so you might want to store it as a field:
type prependEncoder struct {
zapcore.Encoder
pool buffer.Pool
cfg zapcore.EncoderConfig
}
func (e *prependEncoder) Clone() zapcore.Encoder {
return &prependEncoder{
// cloning the encoder with the base config
Encoder: zapcore.NewConsoleEncoder(e.cfg),
pool: buffer.NewPool(),
cfg: e.cfg,
}
}
If you don't implement it, the method that runs is the next shallowest one when calling logger.Clone(), which is the Clone() declared on the embedded zapcore.Encoder. That one then doesn't have your custom EncodeEntry anymore.
Now running the following:
logger.Info("this is info")
logger.Debug("this is debug")
logger.Warn("this is warn")
child := logger.With(zap.String("foo", "bar"))
logger.Warn("original")
child.Info("new one")
Outputs:
<6> INFO this is info
<7> DEBUG this is debug
<4> WARN this is warn
cloning...
<4> WARN original
<6> INFO new one {"foo": "bar"}

Related

Zap logger add UUID to all logs in golang

I have this method used in a lambda:
import (
"os"
"go.uber.org/zap"
"go.uber.org/zap/zapcore"
)
func InitLogger() *zap.Logger {
config := zap.NewProductionEncoderConfig()
config.EncodeTime = zapcore.RFC3339TimeEncoder
consoleEncoder := zapcore.NewJSONEncoder(config)
core := zapcore.NewTee(zapcore.NewCore(consoleEncoder, zapcore.AddSync(os.Stdout), zapcore.InfoLevel))
return zap.New(core).With()
}
And in my lambda Handler i have:
var (
log *zap.Logger
)
func init() {
log = u.InitLogger()
}
func handler(r events.APIGatewayProxyRequest) (*events.APIGatewayProxyResponse, error) {
out, err := exec.Command("uuidgen").Output()
uuid := strings.ReplaceAll(string(out), "\n", "")
if err != nil {
log.Error(err.Error())
}
log.Info("PRINT_1", zap.Any("uuid", uuid), zap.Any("Request", r.Body))
}
I have a question, is possible add the UUID to all logs without adding one by one?, because in each log that I need print something, I need add zap.Any("uuid", uuid)
The problem is that I need pass as parameter to all methods the UUID to print it in the log info, or error.
You will have to slightly re-arrange your code since you're only creating the UUID in the handler, which implies it's request-specific whilst the logger is global...
But the gist, specific to the library, is that you've got to create a child logger (which you are, in fact, already doing: you just need to pass the fields there). Any subsequent log writes to the child logger will include those fields.
For example:
func main() {
logger := InitLogger(zap.String("foo", "bar"))
logger.Info("First message with our `foo` key")
logger.Info("Second message with our `foo` key")
}
func InitLogger(fields ...zap.Field) *zap.Logger {
config := zap.NewProductionEncoderConfig()
config.EncodeTime = zapcore.RFC3339TimeEncoder
consoleEncoder := zapcore.NewJSONEncoder(config)
core := zapcore.NewTee(zapcore.NewCore(consoleEncoder, zapcore.AddSync(os.Stdout), zapcore.InfoLevel))
return zap.New(core).With(fields...)
}
Output:
{"level":"info","ts":"2022-11-24T18:30:45+01:00","msg":"First message with our `foo` key","foo":"bar"}
{"level":"info","ts":"2022-11-24T18:30:45+01:00","msg":"Second message with our `foo` key","foo":"bar"}

How to design classes for X number of config files which needs to be read individually in memory?

I am working with lot of config files. I need to read all those individual config file in their own struct and then make one giant Config struct which holds all other individual config struct in it.
Let's suppose if I am working with 3 config files.
ClientConfig deals with one config file.
DataMapConfig deals with second config file.
ProcessDataConfig deals with third config file.
I created separate class for each of those individual config file and have separate Readxxxxx method in them to read their own individual config and return struct back. Below is my config.go file which is called via Init method from main function after passing path and logger.
config.go
package config
import (
"encoding/json"
"fmt"
"io/ioutil"
"github.com/david/internal/utilities"
)
type Config struct {
ClientMapConfigs ClientConfig
DataMapConfigs DataMapConfig
ProcessDataConfigs ProcessDataConfig
}
func Init(path string, logger log.Logger) (*Config, error) {
var err error
clientConfig, err := ReadClientMapConfig(path, logger)
dataMapConfig, err := ReadDataMapConfig(path, logger)
processDataConfig, err := ReadProcessDataConfig(path, logger)
if err != nil {
return nil, err
}
return &Config{
ClientMapConfigs: *clientConfig,
DataMapConfigs: *dataMapConfig,
ProcessDataConfigs: *processDataConfig,
}, nil
}
clientconfig.go
package config
import (
"encoding/json"
"fmt"
"io/ioutil"
"github.com/david/internal/utilities"
)
type ClientConfig struct {
.....
.....
}
const (
ClientConfigFile = "clientConfigMap.json"
)
func ReadClientMapConfig(path string, logger log.Logger) (*ClientConfig, error) {
files, err := utilities.FindFiles(path, ClientConfigFile)
// read all the files
// do some validation on all those files
// deserialize them into ClientConfig struct
// return clientconfig object back
}
datamapconfig.go
Similar style I have for datamapconfig too. Exactly replica of clientconfig.go file but operating on different config file name and will return DataMapConfig struct back.
processdataConfig.go
Same thing as clientconfig.go file. Only difference is it will operate on different config file and return ProcessDataConfig struct back.
Problem Statement
I am looking for ideas where this above design can be improved? Is there any better way to do this in golang? Can we use interface or anything else which can improve the above design?
If I have let's say 10 different files instead of 3, then do I need to keep doing above same thing for remaining 7 files? If yes, then the code will look ugly. Any suggestions or ideas will greatly help me.
Update
Everything looks good but I have few questions as I am confuse on how can I achieve those with your current suggestion. On majority of my configs, your suggestion is perfect but there are two cases on two different configs where I am confuse on how to do it.
Case 1 After deserializing json into original struct which matches json format, I make another different struct after massaging that data and then I return that struct back.
Case 2 All my configs have one file but there are few configs which have multiple files in them and the number isn't fixed. So I pass regex file name and then I find all the files starting with that regex and then loop over all those files one by one. After deserializing each json file, I start populating another object and keep populating it until all files have been deserialized and then make a new struct with those objects and then return it.
Example of above scenarios:
Sample case 1
package config
import (
"encoding/json"
"fmt"
"io/ioutil"
"github.com/david/internal/utilities"
)
type CustomerManifest struct {
CustomerManifest map[int64]Site
}
type CustomerConfigs struct {
CustomerConfigurations []Site `json:"customerConfigurations"`
}
type Site struct {
....
....
}
const (
CustomerConfigFile = "abc.json"
)
func ReadCustomerConfig(path string, logger log.Logger) (*CustomerManifest, error) {
// I try to find all the files with my below utility method.
// Work with single file name and also with regex name
files, err := utilities.FindFiles(path, CustomerConfigFile)
if err != nil {
return nil, err
}
var customerConfig CustomerConfigs
// there is only file for this config so loop will run once
for _, file := range files {
body, err := ioutil.ReadFile(file)
if err != nil {
return nil, err
}
err = json.Unmarshal(body, &customerConfig)
if err != nil {
return nil, err
}
}
customerConfigIndex := BuildIndex(customerConfig, logger)
return &CustomerManifest{CustomerManifest: customerConfigIndex}, nil
}
func BuildIndex(customerConfig CustomerConfigs, logger log.Logger) map[int64]Site {
...
...
}
As you can see above in sample case 1, I am making CustomerManifest struct from CustomerConfigs struct and then return it instead of returning CustomerConfigs directly.
Sample case 2
package config
import (
"encoding/json"
"fmt"
"io/ioutil"
"github.com/david/internal/utilities"
)
type StateManifest struct {
NotionTemplates NotionTemplates
NotionIndex map[int64]NotionTemplates
}
type NotionMapConfigs struct {
NotionTemplates []NotionTemplates `json:"notionTemplates"`
...
}
const (
// there are many files starting with "state-", it's not fixed number
StateConfigFile = "state-*.json"
)
func ReadStateConfig(path string, logger log.Logger) (*StateManifest, error) {
// I try to find all the files with my below utility method.
// Work with single file name and also with regex name
files, err := utilities.FindFiles(path, StateConfigFile)
if err != nil {
return nil, err
}
var defaultTemp NotionTemplates
var idx = map[int64]NotionTemplates{}
// there are lot of config files for this config so loop will run multiple times
for _, file := range files {
var notionMapConfig NotionMapConfigs
body, err := ioutil.ReadFile(file)
if err != nil {
return nil, err
}
err = json.Unmarshal(body, &notionMapConfig)
if err != nil {
return nil, err
}
for _, tt := range notionMapConfig.NotionTemplates {
if tt.IsProcess {
defaultTemp = tt
} else if tt.ClientId > 0 {
idx[tt.ClientId] = tt
}
}
}
stateManifest := StateManifest{
NotionTemplates: defaultTemp,
NotionIndex: idx,
}
return &stateManifest, nil
}
As you can see above in my both the cases, I am making another different struct after deserializing is done and then I return that struct back but as of now in your current suggestion I think I won't be able to do this generically because for each config I do different type of massaging and then return those struct back. Is there any way to achieve above functionality with your current suggestion? Basically for each config if I want to do some massaging, then I should be able to do it and return new modified struct back but for some cases if I don't want to do any massaging then I can return direct deserialize json struct back. Can this be done generically?
Since there are config which has multiple files in them so that is why I was using my utilities.FindFiles method to give me all files basis on file name or regex name and then I loop over all those files to either return original struct back or return new struct back after massaging original struct data.
You can use a common function to load all the configuration files.
Assume you have config structures:
type Config1 struct {...}
type Config2 struct {...}
type Config3 struct {...}
You define configuration validators for those who need it:
func (c Config1) Validate() error {...}
func (c Config2) Validate() error {...}
Note that these implement a Validatable interface:
type Validatable interface {
Validate() error
}
There is one config type that includes all these configurations:
type Config struct {
C1 Config1
C2 Config2
C3 Config3
...
}
Then, you can define a simple configuration loader function:
func LoadConfig(fname string, out interface{}) error {
data, err:=ioutil.ReadFile(fname)
if err!=nil {
return err
}
if err:=json.Unmarshal(data,out); err!=nil {
return err
}
// Validate the config if necessary
if validator, ok:=out.(Validatable); ok {
if err:=validator.Validate(); err!=nil {
return err
}
}
return nil
}
Then, you can load the files:
var c Config
if err:=LoadConfig("file1",&c.C1); err!=nil {
return err
}
if err:=LoadConfig("file2",&c.C2); err!=nil {
return err
}
...
If there are multiple files loading different parts of the same struct, you can do:
LoadConfig("file1",&c.C3)
LoadConfig("file2",&c.C3)
...
You can simplify this further by defining a slice:
type cfgInfo struct {
fileName string
getCfg func(*Config) interface{}
}
var configs=[]cfgInfo {
{
fileName: "file1",
getCfg: func(c *Config) interface{} {return &c.C1},
},
{
fileName: "file2",
getCfg: func(c *Config) interface{} {return &c.C2},
},
{
fileName: "file3",
getCfg: func(c *Config) interface{} {return &c.C3},
},
...
}
func loadConfigs(cfg *Config) error {
for _,f:=range configs {
if err:=loadConfig(f.fileName,f.getCfg(cfg)); err!=nil {
return err
}
}
return nil
}
Then, loadConfigs would load all the configuration files into cfg.
func main() {
var cfg Config
if err:=loadConfigs(&cfg); err!=nil {
panic(err)
}
...
}
Any configuration that doesn't match this pattern can be dealt with using LoadConfig:
var customConfig1 CustomConfigStruct1
if err:=LoadConfig("customConfigFile1",&customConfig1); err!=nil {
panic(err)
}
cfg.CustomConfig1 = processCustomConfig1(customConfig1)
var customConfig2 CustomConfigStruct2
if err:=LoadConfig("customConfigFile2",&customConfig2); err!=nil {
panic(err)
}
cfg.CustomConfig2 = processCustomConfig2(customConfig2)

How can I duplicate entry keys and show it in the same log with Uber Zap?

I would like to duplicate entry keys like caller with another key name method and show both in the log...
{"level":"error", "caller: "testing/testing.go:1193", "method": "testing/testing.go:1193", "message": "foo"}
Any ideas?
You can't change the fields of a zapcore.Entry. You may change how it is marshalled, but honestly adding ghost fields to a struct is a bad hack. What you can do is use a custom encoder, and append to []zapcore.Field a new string item with a copy of the caller. In particular, the default output of the JSON encoder is obtained from Caller.TrimmedPath():
type duplicateCallerEncoder struct {
zapcore.Encoder
}
func (e *duplicateCallerEncoder) Clone() zapcore.Encoder {
return &duplicateCallerEncoder{Encoder: e.Encoder.Clone()}
}
func (e *duplicateCallerEncoder) EncodeEntry(entry zapcore.Entry, fields []zapcore.Field) (*buffer.Buffer, error) {
// appending to the fields list
fields = append(fields, zap.String("method", entry.Caller.TrimmedPath()))
return e.Encoder.EncodeEntry(entry, fields)
}
Note that the above implements Encoder.Clone(). See this for details: Why custom encoding is lost after calling logger.With in Uber Zap?
And then you can use it by either constructing a new Zap core, or by registering the custom encoder. The registered constructor embeds a JSONEncoder into your custom encoder, which is the default encoder for the production logger:
func init() {
// name is whatever you like
err := zap.RegisterEncoder("duplicate-caller", func(config zapcore.EncoderConfig) (zapcore.Encoder, error) {
return &duplicateCallerEncoder{Encoder: zapcore.NewJSONEncoder(config)}, nil
})
// it's reasonable to panic here, since the program can't initialize
if err != nil {
panic(err)
}
}
func main() {
cfg := zap.NewProductionConfig()
cfg.Encoding = "duplicate-caller"
logger, _ := cfg.Build()
logger.Info("this is info")
}
The above replicates the initialization of a production logger with your custom config.
For such a simple config, I prefer the init() approach with zap.RegisterEncoder. It makes it faster to refactor code, if needed, and/or if you place this in some other package to begin with. You can of course do the registration in main(); or if you need additional customization, then you may use zap.New(zapcore.NewCore(myCustomEncoder, /* other args */))
You can see the full program in this playground: https://go.dev/play/p/YLDXbdZ-qZP
It outputs:
{"level":"info","ts":1257894000,"caller":"sandbox3965111040/prog.go:24","msg":"this is info","method":"sandbox3965111040/prog.go:24"}

Uber Zap logger function name in logs

How to get function name printed in logs from Uber Zap logging ?
This is the PR request with which they seemed to have added the functionality to output function names in log.
I am using golang version 1.15 and go.uber.org/zap v1.16.0
This is my code:
package main
import (
"go.uber.org/zap"
)
var logger *zap.Logger
func main() {
logger := NewLogger()
logger.Info("Test msg Main")
TestFunc(logger)
}
func TestFunc(logger *zap.Logger) {
logger.Info("Test msg TestFunc")
}
func NewLogger() *zap.Logger {
config := zap.NewDevelopmentConfig()
opts := []zap.Option{
zap.AddCallerSkip(1), // traverse call depth for more useful log lines
zap.AddCaller(),
}
logger, _ = config.Build(opts...)
return logger
}
This is the output I get with/without the addition of AddCaller() option
2021-03-01T15:00:02.927-0800 INFO runtime/proc.go:204 Test msg Main
2021-03-01T15:00:02.927-0800 INFO cmd/main.go:12 Test msg TestFunc
I am expecting something like
2021-03-01T15:00:02.927-0800 INFO runtime/proc.go:204 main Test msg Main
2021-03-01T15:00:02.927-0800 INFO cmd/main.go:12 TestFunc Test msg TestFunc
By default, the provided encoder presets (NewDevelopmentEncoderConfig used by NewDevelopmentConfig and NewProductionEncoderConfig used by NewProductionConfig) do not enable function name logging.
To enable function name, you need to enable caller (true by default) and set a non-empty value for config.EncoderConfig.FunctionKey.
Source: EncoderConfig
type EncoderConfig struct {
// Set the keys used for each log entry. If any key is empty, that portion
// of the entry is omitted.
...
CallerKey string `json:"callerKey" yaml:"callerKey"`
FunctionKey string `json:"functionKey" yaml:"functionKey"` // this needs to be set
StacktraceKey string `json:"stacktraceKey" yaml:"stacktraceKey"`
...
}
Example Console Logger:
func main() {
config := zap.NewDevelopmentConfig()
// if you're using console encoding, the FunctionKey value can be any
// non-empty string because console encoding does not print the key.
config.EncoderConfig.FunctionKey = "F"
logger, _ := config.Build()
logger.Info("Test Logging")
// Output: 2021-03-03T11:41:47.728+0800 INFO example/main.go:11 main.main Test Logging
}
Example JSON Logger:
func main() {
config := zap.NewProductionConfig()
// the FunctionKey value matters because it will become the JSON field
config.EncoderConfig.FunctionKey = "func"
logger, _ := config.Build()
log(logger)
// Output: {"level":"info","ts":1614743088.538128,"caller":"example/main.go:15","func":"main.log","msg":"Test Logging"}
}
func log(logger *zap.Logger) {
logger.Info("Test Logging")
}

zap logging with 1) customized config and 2) lumberjack

I'm trying to build a customized zap logger with 1) customized *zap.Config and 2) lumberjack, but can't find proper example to apply both configurations.
Since config.Build does not accept WriteSync as an input. Do you know how to achieve this?
func genBaseLoggerZap() Logger {
ex, err := os.Executable()
if err != nil {
Fatalf("Failed to get os.Executable, err: %v", err)
}
zlManager.outputPath = path.Join(filepath.Dir(ex), zlManager.outputPath)
// Want to add sync here..
zapcore.AddSync(&lumberjack.Logger{
Filename: zlManager.outputPath + "123",
MaxSize: 500,
MaxBackups: 10,
MaxAge: 28,
})
return genLoggerZap(BaseLogger, genDefaultConfig())
}
// genLoggerZap creates a zapLogger with given ModuleID and Config.
func genLoggerZap(mi ModuleID, cfg *zap.Config) Logger {
logger, err := cfg.Build()
if err != nil {
Fatalf("Failed to generate zap logger, err: %v", err)
}
newLogger := &zapLogger{mi, cfg, logger.Sugar()}
newLogger.register()
return newLogger
}
You can add custom log destinations using the zap.RegisterSink function and the Config.OutputPaths field. RegisterSink maps URL schemes to Sink constructors, and OutputPaths configures log destinations (encoded as URLs).
Conveniently, *lumberjack.Logger implements almost all of the zap.Sink interface already. Only the Sync method is missing, which can be easily added with a thin wrapper type.
package main
import (
"net/url"
"go.uber.org/zap"
lumberjack "gopkg.in/natefinch/lumberjack.v2"
)
type lumberjackSink struct {
*lumberjack.Logger
}
// Sync implements zap.Sink. The remaining methods are implemented
// by the embedded *lumberjack.Logger.
func (lumberjackSink) Sync() error { return nil }
func main() {
zap.RegisterSink("lumberjack", func(u *url.URL) (zap.Sink, error) {
return lumberjackSink{
Logger: &lumberjack.Logger{
Filename: u.Opaque,
// Use query parameters or hardcoded values for remaining
// fields.
},
}, nil
})
config := zap.NewProductionConfig()
// Add a URL with the "lumberjack" scheme.
config.OutputPaths = append(config.OutputPaths, "lumberjack:foo.log")
log, _ := config.Build()
log.Info("test", zap.String("foo", "bar"))
}

Resources