Currently running a go service on GCP however in the logs viewer every message is treated as an error.
Is there a generally advised way of logging to stderr and stdout depending on the log level. Ie errors to stderr and anything else to stdout.
I'm currently using the logrus package and have come across this implementation. Other ways i see achieving this while still using the same package is to pass the logger to each package that needs it or to create a global log object, neither of which i am too keen on.
https://github.com/microsoft/fabrikate/pull/252/commits/bd24d62d7c2b851ad6e7b36653eb0a6dc364474b#diff-ed0770fdbf87b0c6d536e33a99a8df9c
You can use Stackdriver library package for GoLang:
go get -u cloud.google.com/go/logging
Then you can use StandardLogger:
// Sample stdlogging writes log.Logger logs to the Stackdriver Logging.
package main
import (
"context"
"log"
"cloud.google.com/go/logging"
)
func main() {
ctx := context.Background()
// Sets your Google Cloud Platform project ID.
projectID := "YOUR_PROJECT_ID"
// Creates a client.
client, err := logging.NewClient(ctx, projectID)
if err != nil {
log.Fatalf("Failed to create client: %v", err)
}
defer client.Close()
// Sets the name of the log to write to.
logName := "my-log"
logger := client.Logger(logName).StandardLogger(logging.Info)
// Logs "hello world", log entry is visible at
// Stackdriver Logs.
logger.Println("hello world")
}
Here you can find documentation on Google Cloud website
Update:
Alternatively you could give a try GCP formatter for logrus
This will not tie your app to Google Cloud Platform. However, it does not mean that on another platform you will not need to change your code to format output.
Using StackDriver library is the recommended solution for Google Cloud.
We use https://github.com/rs/zerolog with the following method called in our Init() method to setup the logging options at global level:
package main
import (
"github.com/rs/zerolog"
"github.com/rs/zerolog/log"
"os"
)
// initializeLogging sets up the logging configuration for the service.
// Invoke this method in your Init() method.
func initializeLogging() {
// Set logging options for production development
if os.Getenv("ENV") != "DEV" {
// change the level field name to ensure these are parsed correctly in Stackdriver
zerolog.LevelFieldName = "severity"
// UNIX Time is faster and smaller than most timestamps
zerolog.TimeFieldFormat = zerolog.TimeFormatUnix
} else {
// Set logging options for local development
log.Logger = log.Output(zerolog.ConsoleWriter{Out: os.Stderr})
zerolog.SetGlobalLevel(zerolog.DebugLevel)
}
// Example log
log.Info().Msg("This is how you log at Info level")
}
I recommend the https://github.com/apsystole/log. You can swap either log and logrus imports for it. It is a small zero-dependency module, hence much lighter than logrus.
Related
I'm playing with fasthttp and it's router, I have no issues with basic things, I have a working server and a router, that is the easy part.
The issue is with the logger, I would like to be able to customize that one, but it does not seem possible with the ctx.Logger() as it only takes a Printf argument and the format is not what I'm looking for.
Does anyone knows where in the documentation I can find a working example of what I want to do?
Example of code I currently have:
package server
import (
"github.com/fasthttp/router"
"github.com/valyala/fasthttp"
)
// Router will manage the routes of our API server
func Router() *router.Router {
r := router.New()
r.GET("/", index)
return r
}
func index(ctx *fasthttp.RequestCtx) {
ctx.Logger().Printf("/")
ctx.WriteString("Welcome!")
}
As I'm still trying my hand with the web servers and I still don't understand some things with it and Go also. So An example would be welcome.
For example I would like to be able to do something like that using a logger define in the main package:
package server
import (
"github.com/fasthttp/router"
"github.com/valyala/fasthttp"
"go.uber.org/zap"
)
// Router will manage the routes of our API server
func Router(loger *zap.Logger) *router.Router {
r := router.New()
r.GET("/", index)
return r
}
func index(ctx *fasthttp.RequestCtx) {
ctx.Logger().Printf("/") // Here should print in the zap format of my choice.
ctx.WriteString("Welcome!")
}
If you look at the source code, it's apparent that all you have is the ability to write standard Go-formatted strings:
func (cl *ctxLogger) Printf(format string, args ...interface{}) {
msg := fmt.Sprintf(format, args...)
ctxLoggerLock.Lock()
cl.logger.Printf("%.3f %s - %s", time.Since(cl.ctx.ConnTime()).Seconds(), cl.ctx.String(), msg)
ctxLoggerLock.Unlock()
}
The logger simply adds some additional information from the context. So further cutomisation beyond the standard Go formatting does not seem possible. I'm not sure what "zap format of my choice" is, so I can't say if there's a workaround or even if standard Go formatting options will serve for you here.
I am using logrus library for structured logging in my Go project.
I have configured my logrus as below:
// Global variable for logging
var gLog = &Logger{moduleName: ModuleName, logrus: logrus.New()}
type Logger struct {
moduleName string
logrus *logrus.Logger
}
func SetupGlobalLogger(logPrefix string, logMode string) error {
if logMode == "file" {
logFilePath := fmt.Sprintf("var/%s.log", vite.Environment())
file, err := os.OpenFile(logFilePath, logFileFlags, logFilePermission)
if err != nil {
return err
}
gLog.logrus.SetOutput(file)
// redirect logs written using standard log library to same place as logrus
log.SetOutput(gLog.logrus.Writer())
log.Println(vite.MarkInfo, "redirect log to file:", logFilePath)
}
return nil
}
In this Go project, there are several places where standard log library statements like log.Println() are used.
I want to redirect those log messages to logrus.
For that I am using following statement in above code.
log.SetOutput(gLog.logrus.Writer())
My question is: Is this thread safe?
If one thread/go-routine is executing log.Println() while another executing gLog.logrus.Info() or something on logrus, will that be fine?
Is Logrus Thread-safe?
Yes:
Thread safety
By default, Logger is protected by a mutex for concurrent writes.
Except when it isn't.
Most of the non-thread-safe cases are bugs which will occur only in rare situations. Whether any of them matter to you depends, of course, on your use case.
I am trying to configure a RabbitMQ broker using the go-micro framework. I have noticed that the broker interface in go-micro has a broker.SubscriberOptions struct which allows configuring the parameters I am looking for (AutoAck, Queue name and so on) however I am unable to figure out how to pass this when starting a broker.
This is how a simple rabbit go-micro setup would look like
package main
import (
"log"
"github.com/micro/go-micro/server"
"github.com/micro/go-plugins/broker/rabbitmq"
micro "github.com/micro/go-micro"
)
func main() {
// Create a new service. Optionally include some options here.
service := micro.NewService(
micro.Name("go-micro-rabbit"),
micro.Broker(rabbitmq.NewBroker()),
)
// Init will parse the command line flags.
service.Init()
// Register handler
proto.RegisterGreeterHandler(service.Server(), new(Greeter))
micro.RegisterSubscriber("micro-exchange", service.Server(), myFunc, server.SubscriberQueue("my-queue"))
// Run the server
if err := service.Run(); err != nil {
log.Fatal(err)
}
}
The micro.RegisterSubscriber method takes in a list of server.SubscriberOption but does not allow me to set the broker.SubscriberOptions and the rabbitmq.NewBroker allows setting broker.Options but once again, not broker.SubscriberOptions
I have dug in the code of go-micro but have been unable to figure out how the broker.Subscribe method (Which exposes the correct struct) is called or by who.
Is this possible at all? Is it maybe something not yet fully fleshed out in the API?
I'm trying to insert a simple record as in GoDoc. But this returns,
rpc error: code = 7 desc = "User can't access project: tidy-groove"
When I searched for grpc codes, it says..
PermissionDenied Code = 7
// Unauthenticated indicates the request does not have valid
// authentication credentials for the operation.
I've enabled Big table in my console and created a cluster and a service account and recieved the json. What I'm doing wrong here?
package main
import (
"fmt"
"golang.org/x/net/context"
"golang.org/x/oauth2/google"
"google.golang.org/cloud"
"google.golang.org/cloud/bigtable"
"io/ioutil"
)
func main() {
fmt.Println("Start!")
put()
}
func getClient() *bigtable.Client {
jsonKey, err := ioutil.ReadFile("TestProject-7854ea9op741.json")
if err != nil {
fmt.Println(err.Error())
}
config, err := google.JWTConfigFromJSON(
jsonKey,
bigtable.Scope,
) // or bigtable.AdminScope, etc.
if err != nil {
fmt.Println(err.Error())
}
ctx := context.Background()
client, err := bigtable.NewClient(ctx, "tidy-groove", "asia-east1-b", "test1-bigtable", cloud.WithTokenSource(config.TokenSource(ctx)))
if err != nil {
fmt.Println(err.Error())
}
return client
}
func put() {
ctx := context.Background()
client := getClient()
tbl := client.Open("table1")
mut := bigtable.NewMutation()
mut.Set("links", "maps.google.com", bigtable.Now(), []byte("1"))
mut.Set("links", "golang.org", bigtable.Now(), []byte("1"))
err := tbl.Apply(ctx, "com.google.cloud", mut)
if err != nil {
fmt.Println(err.Error())
}
}
I've solved the problem. It's nothing wrong with the code, but config json itself. So anyone who out there want to authenticate and came here by google search... This code is correct and working perfectly. What I've done wrong is follows.
First I made a service account and got the json. But google warned me that im not an owner of project hence it wont be added to accept list but anyway it let me download the json.
Then I deleted that key from console and requested project owner to create a key for me.
There he has created another key with the same name I given.. And since he's the owner no error/warning msgs displayed and successfully json file was downloaded.
When I tried with that... my question begun. That's when i posted this question.
After that with no solutions. I asked owner to delete that key and create another key but with a different name..
Then it worked! It seems if you try to create a key with non-owner account and then again create with same name ( after deleting original of course ) has no effect. Hope this helps everyone out there :)
Take a look at: helloworld.go or search.go which uses GOOGLE_APPLICATION_CREDENTIALS environment variable.
For most environments, you no longer even need to set GOOGLE_APPLICATION_CREDENTIALS. Google Cloud Platform, Managed VMs or Google App Engine all have the right thing set for you. Your desktop environment will also be correct if you've used gcloud init or it's predecessor gcloud auth login followed by gcloud config set project <projectID>.
I am using logrus in a Go app. I believe this question is applicable to any other logging package (which doesn't offer external file based configuration) as well.
logrus provides functions to setup various configuration, e.g. SetOutput, SetLevel etc.
Like any other application I need to do logging from multiple source files/packages, it seems you need to setup these options in each file with logrus.
Is there any way to setup these options once somewhere in a central place to be shared all over the application. That way if I have to make logging level change I can do it in one place and applies to all the components of the app.
You don't need to set these options in each file with Logrus.
You can import Logrus as log:
import log "github.com/Sirupsen/logrus"
Then functions like log.SetOutput() are just functions and modify the global logger and apply to any file that includes this import.
You can create a package global log variable:
var log = logrus.New()
Then functions like log.SetOutput() are methods and modify your package global. This is awkward IMO if you have multiple packages in your program, because each of them has a different logger with different settings (but maybe that's good for some use cases). I also don't like this approach because it confuses goimports (which will want to insert log into your imports list).
Or you can create your own wrapper (which is what I do). I have my own log package with its own logger var:
var logger = logrus.New()
Then I make top-level functions to wrap Logrus:
func Info(args ...interface{}) {
logger.Info(args...)
}
func Debug(args ...interface{}) {
logger.Debug(args...)
}
This is slightly tedious, but allows me to add functions specific to my program:
func WithConn(conn net.Conn) *logrus.Entry {
var addr string = "unknown"
if conn != nil {
addr = conn.RemoteAddr().String()
}
return logger.WithField("addr", addr)
}
func WithRequest(req *http.Request) *logrus.Entry {
return logger.WithFields(RequestFields(req))
}
So I can then do things like:
log.WithConn(c).Info("Connected")
(I plan in the future to wrap logrus.Entry into my own type so that I can chain these better; currently I can't call log.WithConn(c).WithRequest(r).Error(...) because I can't add WithRequest() to logrus.Entry.)
This is the solution that I arrived at for my application that allows adding of fields to the logging context. It does have a small performance impact due to the copying of context base fields.
package logging
import (
log "github.com/Sirupsen/logrus"
)
func NewContextLogger(c log.Fields) func(f log.Fields) *log.Entry {
return func(f log.Fields) *log.Entry {
for k, v := range c {
f[k] = v
}
return log.WithFields(f)
}
}
package main
import (
"logging"
)
func main {
app.Logger = logging.NewContextLogger(log.Fields{
"module": "app",
"id": event.Id,
})
app.Logger(log.Fields{
"startTime": event.StartTime,
"endTime": event.EndTime,
"title": event.Name,
}).Info("Starting process")
}