My goal is to create a logger that I can use to output to stdout (info logs) and stderr (error logs) as well as respective files (info.log) and (errors.log) all at the same time.
I am currently using loggers in Go like the following:
package main
import (
"log"
"os"
"io"
)
func main() {
// Current method
infoLog := log.New("./data/info.log", "INFO\t", log.Ldate|log.Ltime)
errorLog := log.New("./data/errors.log", "ERROR\t", log.Ldate|log.Ltime|log.Lshortfile)
infoLog.Println("Hello INFO!")
errorLog.Println("Hello ERROR!")
// I've read about using io.MultiWriter like the following in order to write to stdout/stderr as well as a flat file at the same time
f, err := os.OpenFile("./data/info.log", os.O_RDWR|os.O_CREATE|os.O_APPEND, 0666)
if err != nil {
log.Fatalf("error opening file: %v", err)
}
defer f.Close()
wrt := io.MultiWriter(os.Stdout, f)
log.SetOutput(wrt)
log.Println("Hello World!")
}
I'd like to basically take the formatting of infoLog and errorLog (like in my first logging solution) like log.Ldate|log.Ltime and apply it to the io.MultiWriter. Is there a way to do this?
For logging info and error in file with standard format that other logging services such as filebeat, logstash and etc can understand it I suggest you to use github.com/sirupsen/logrus.
This package has SetFormater function to define format of output log. For example:
logrus.SetFormatter(&logrus.JSONFormatter{
FieldMap: logrus.FieldMap{
logrus.FieldKeyTime: "#timestamp",
logrus.FieldKeyMsg: "message",
logrus.FieldKeyFunc: "func",
logrus.FieldKeyFile: "file",
},
})
and you can define file for logging. For example:
file, err := os.OpenFile("payment_logs.log", os.O_RDWR|os.O_CREATE|os.O_APPEND, 0666)
And set log is very easy and standard:
logger.Info("This is an info message")
logger.Warn("This is a warning message")
logger.Error("This is an error message")
Related
I am very new at Golang AWS SDK V2, I had similar code work without AWS results, but this one is getting me issues since the types are different. Also, I have search and none of the examples is with the code pipeline aws-sdk-v2 with the type of JSON I have to unmarshal.
I hope some of you can help me.
— This below is main.go I have the structs in another file called un-marshal.go that I created with the result of getting the same output I need from awscli tool and passing it by https://mholt.github.io/json-to-go/
package main
import (
"context"
"fmt"
"log"
"encoding/json"
"github.com/aws/aws-sdk-go-v2/config"
"github.com/aws/aws-sdk-go-v2/service/codepipeline"
)
func main() {
pipeline_name := "frontend"
// Load the Shared AWS Configuration (~/.aws/config)
cfg, err := config.LoadDefaultConfig(context.TODO(), config.WithRegion("eu-central-1"))
if err != nil {
log.Fatal(err)
}
if err != nil {
log.Fatalf("unable to load SDK config, %v", err)
}
client := codepipeline.NewFromConfig(cfg)
pipeJson, err := client.GetPipelineState(context.TODO(), &codepipeline.GetPipelineStateInput{
Name: &pipeline_name,
})
if err != nil {
log.Println("Error getting Pipeline")
}
var cookie PipeLineResult
json.Unmarshal(pipeJson, &cookie)
fmt.Println("The name of the pipeline is: %s",cookie.PipelineName)
}
The error I am getting is:
/main.go:39:17: cannot use pipeJson (variable of type *codepipeline.GetPipelineStateOutput) as type []byte in argument to json.Unmarshal
Here I am lost because it is a new type, and not sure if I should convert, how to convert or work with the native type etc.
Thanks in advance.
I am using go.uber.org/zap/zapcore for logging in my Go app.
package logger
import (
"go.uber.org/zap"
"go.uber.org/zap/zapcore"
"log"
)
var l *zap.Logger
func Get() *zap.Logger {
return l
}
func Init() {
conf := zap.NewProductionConfig()
logger, err := conf.Build()
if err != nil {
log.Fatal("Init logger failed", err)
}
l = logger
}
I also have Sentry project and use github.com/getsentry/raven-go.
I want to send logs at error level and above to Sentry.
For example when logging at info level with logger.Info() I want to just log them as usual, but in case of error or fatal logs I need send these messages to Sentry. How can I achieve that?
The answer is you should use zap wrapper for adding hooks then you have to use the function of logger which is called WithOptions
sentryOptions := zap.WrapCore(func(core zapcore.Core) zapcore.Core {
return zapcore.RegisterHooks(core, func(entry zapcore.Entry) error {
// your logic here
})
})
logger.WithOptions(sentryOptions)
The following will capture the message and send it to the sentry when an error level is detected, with customized error line number and message.
err := sentry.Init(sentry.ClientOptions{Dsn: "http://~~~~~"})
if err != nil {
log.fatal("Sentry Error Setup ::", err.Error())
}
logger, _ := zap.NewDevelopment(zap.Hooks(func(entry zapcore.Entry) error {
if entry.Level == zapcore.ErrorLevel {
defer sentry.Flush(2 * time.Second)
sentry.CaptureMessage(fmt.Sprintf("%s, Line No: %d :: %s", entry.Caller.File, entry.Caller.Line, entry.Message))
}
return nil
}))
sugar := logger.Sugar()
I am using github.com/sirupsen/logrus for logging in my golang scripts, however I want to get the filename and the line number which is logging the message. I am able to get that using the below code:
package main
import (
"fmt"
"os"
"runtime"
"strings"
"github.com/sirupsen/logrus"
)
func GetLogger() (*logrus.Logger, *os.File) {
log := logrus.New()
log.SetReportCaller(true)
file, err := os.OpenFile("info.log", os.O_CREATE|os.O_APPEND, 0644)
if err != nil {
log.Fatal(err)
}
log.Out = file
log.Formatter = &logrus.TextFormatter{
CallerPrettyfier: func(f *runtime.Frame) (string, string) {
repopath := fmt.Sprintf("%s/src/github.com/bob", os.Getenv("GOPATH"))
filename := strings.Replace(f.File, repopath, "", -1)
return fmt.Sprintf("%s()", f.Function), fmt.Sprintf("%s:%d", filename, f.Line)
},
}
return log, file
}
However this gives log in the below format:
time="2020-04-02T11:43:19+05:30" level=info msg=Hello func="main.main()" file="D:/.../main.go:13"
But I want the log in format as below:
Apr 02 00:00:00 INFO main.go:20 : Hello this is a log line
How can a custom formatter be written to get this?
This option is included in the library itself since the end of 2018.
Just set 'SetReportCaller' to true.
Here's an example:
package main
import (
log "github.com/sirupsen/logrus"
)
func main() {
// Add this line for logging filename and line number!
log.SetReportCaller(true)
log.Println("hello world")
}
The output:
INFO[0000]/home/trex/go/src/awesomeProject/main.go:11 main.main() hello world
FYI
log.SetReportCaller(true)
log.SetFormatter(&log.JSONFormatter{
CallerPrettyfier: func(frame *runtime.Frame) (function string, file string) {
fileName := path.Base(frame.File) + ":" + strconv.Itoa(frame.Line)
//return frame.Function, fileName
return "", fileName
},
})
output:
{"file":"inspParse.go:290","level":"info","msg":"(3, 24), ","time":"2021-08-30T16:41:38+08:00"}
you can take advantage of codes below
package main
import (
"bytes"
"fmt"
"github.com/sirupsen/logrus"
"io"
"os"
"strings"
)
type MyFormatter struct {}
var levelList = [] string{
"PANIC",
"FATAL",
"ERROR",
"WARN",
"INFO",
"DEBUG",
"TRACE",
}
func (mf *MyFormatter) Format(entry *logrus.Entry) ([]byte, error){
var b *bytes.Buffer
if entry.Buffer != nil {
b = entry.Buffer
} else {
b = &bytes.Buffer{}
}
level := levelList[int(entry.Level)]
strList := strings.Split(entry.Caller.File, "/")
fileName := strList[len(strList)-1]
b.WriteString(fmt.Sprintf("%s - %s - [line:%d] - %s - %s\n",
entry.Time.Format("2006-01-02 15:04:05,678"), fileName,
entry.Caller.Line, level, entry.Message))
return b.Bytes(), nil
}
func MakeLogger(filename string, display bool) *logrus.Logger {
f, err := os.OpenFile(filename, os.O_CREATE|os.O_RDWR|os.O_APPEND, 0644)
if err != nil {
panic(err.Error())
}
logger := logrus.New()
if display {
logger.SetOutput(io.MultiWriter(os.Stdout, f))
} else {
logger.SetOutput(io.MultiWriter(f))
}
logger.SetReportCaller(true)
logger.SetFormatter(&MyFormatter{})
return logger
}
func main() {
logger := MakeLogger("/tmp/test.log", true)
logger.Info("hello world!")
}
Result:
/tmp/test.log
2021-11-24 00:49:10,678 - main.go - [line:58] - INFO - hello world!
The package you're using github.com/sirupsen/logrus produces structured log output: that is key/value pairs. It looks like you want just a plain text logger.
The standard logger import "log", produces output quite like what you want: log.New(out, "INFO", .Ldate|log.Ltime|log.Lshortfile). (See https://play.golang.org/p/LKitIwjPuVH on the playground)
Here's example output:
INFO 2009/11/10 23:00:00 prog.go:10: hello
In go1.14, the extra flag log.Lmsgprefix moves the INFO to before the message, if that's preferable (and you can wait).
If the standard library logger doesn't do what you want (and you're not prepared to live with it), why not just copy it and edit it, essentially making your own log package? It's around 400 lines of straightforward code, and by the time you remove the parts you don't want, it'll be a lot less.
The source is here: https://golang.org/src/log/log.go
Showing file, function name, and line number with logrus:
func Error(err error, msg ...interface{}) {
if pc, file, line, ok := runtime.Caller(1); ok {
file = file[strings.LastIndex(file, "/")+1:]
funcName := runtime.FuncForPC(pc).Name()
logrus.WithFields(
logrus.Fields{
"err": err,
"src": fmt.Sprintf("%s:%s:%d", file, funcName, line),
}).Error(msg...)
}
}
Call this in every log event.
I am writing an application in which i need to record logs in two different files. For Example weblogs.go and debuglogs.go. I tried using the log4go but my requirement is i need the logger to be created in main file and be accessible in sub directory as major of decoding and logging is done in sub file. Can anybody please help with that?
Here's one way to do it, using the standard log package:
package main
import (
"io"
"log"
"os"
)
func main() {
f1, err := os.Create("/tmp/file1")
if err != nil {
panic(err)
}
defer f1.Close()
f2, err := os.Create("/tmp/file2")
if err != nil {
panic(err)
}
defer f2.Close()
w := io.MultiWriter(os.Stdout, f1, f2)
logger := log.New(w, "logger", log.LstdFlags)
myfunc(logger)
}
func myfunc(logger *log.Logger) {
logger.Print("Hello, log file!!")
}
Notes:
io.MultiWriter is used to combine several writers together. Here, it creates a writer w - a write to w will go to os.Stdout as well as two files
log.New lets us create a new log.Logger object with a custom writer
The log.Logger object can be passed around to functions and used by them to log things
This is how you can manage logs for debugging and prod envoirments with multi log files:
First, make a type for managing multiple loggers you can make a separate file for this like logging.go
type LOGGER struct {
debug *log.Logger
prod *log.Logger
.
.
.
}
For getting the LOGGER pointer anywhere in your project the best approach will be to get it through the singleton pattern like
var lock = &sync.Mutex{}
var loggers *LOGGER
func GetLoggerInstance() *LOGGER {
if singleInstance == nil {
lock.Lock()
defer lock.Unlock()
if loggers == nil {
fmt.Println("Creating LOGGER instance now.")
loggers = &LOGGER{}
} else {
fmt.Println("LOGGER instance already created.")
}
} else {
fmt.Println("LOGGER instance already created.")
}
return loggers
}
Now setup logger for debug env in any pkg or directory level better is to setup it at the root level
logger := GetLoggerInstance()
f, err := os.OpenFile("path/to/debug/log/file", os.O_RDWR|os.O_CREATE|os.O_APPEND, 0666)
if err != nil {
fmt.Println("debug log file not created", err.Error())
}
loggers.debug = log.New(f, "[DEBUG]", log.Ldate|log.Ltime|log.Lmicroseconds|log.LUTC)
loggers.debug.Println("This is debug log message")
With the same approach, you can also create prod or as many as you want.
logger := GetLoggerInstance()
f, err = os.OpenFile("path/to/prod/log/file", os.O_RDWR|os.O_CREATE|os.O_APPEND, 0666)
if err != nil {
fmt.Println("prod log file not created", err.Error())
}
loggers.prod = log.New(f, "[PROD]", log.Ldate|log.Ltime|log.Lmicroseconds|log.LUTC)
loggers.prod.Println("This is prod log message")
I've been exploring configuration management in Go using the viper package, which I learned about in another question. I'm having trouble understanding how to initialize a new configuration. What I want to do is to find system configurations if they exist, then user configuration, and then, if all else fails, to create (and then use) a new user configuration from defaults. The next time I run the app, I'd expect to find the configuration that I previously created. I'm using it like this:
import (
"fmt"
"github.com/spf13/viper"
"os"
"os/user"
"path/filepath"
)
usr, err := user.Current()
appMode := "test" // or "production" or...
configDir := filepath.Join(usr.HomeDir, ".myapp")
config := viper.New()
config.SetConfigName(appMode)
config.SetConfigType("json")
config.SetDefault("group1.key1", "value1.1")
config.SetDefault("group1.key2", 1234)
config.SetDefault("group2.key1", "value2.1")
config.SetDefault("group2.key2", 5678)
config.AddConfigPath("/usr/share/myapp/config")
config.AddConfigPath("/usr/local/share/myapp/config")
config.AddConfigPath(configDir)
if err := config.ReadInConfig(); err != nil {
filename := filepath(configDir, fmt.Sprintf("%s.json", appMode))
_, err := os.Create(filename)
if err != nil {
panic(fmt.Stringf("Failed to create %s", filename))
}
}
if err := config.ReadInConfig(); err != nil {
panic("Created %s, but Viper failed to read it: %s",
filename, err)
}
At this point, I would like for ~/.myapp/test.json to be created for me with the following:
{
"group1": {
"key1": "value1.1",
"key2": 1234
},
"group2": {
"key1": "value2.1",
"key2": 5678
}
}
The result is that the file is empty, and the second attempt to read the file also fails with the message "open : no such file or directory" even though it exists. If I hand-edit the file, Viper finds it and parses it. Obviously I can create the JSON file programatically, but this seems like such an obvious use case that I must be missing something here.