I deployed a go application in google cloud using kubernetes which automatically logs to google stackdriver. Oddly, all log statements are being tagged with severity "ERROR"
For example:
log.Println("This should have log level info")
will be tagged as an error.
Their docs say "Severities: By default, logs written to the standard output are on the INFO level and logs written to the standard error are on the ERROR level."
Anyone know what could be wrong with my setup?
Take a look at this logging package: github.com/teltech/logger, with an accompanying blog post. It will output your logs in a JSON format, including the severity, that is readable by the Stackdriver Fluentd agent.
Related
My goal is to include my stacktrace and log message into a single log message for my Spring Boot applications. The problem I'm running into is each line of the stacktrace is a separate log message. I want to be able to search my logs for a log level of ERROR and find the log message with the stacktrace. I've found two solutions but not sure which to use.
I can use Logback to put them all in one line but would like to keep the new lines for a pretty format. Also the guide I found might override defaults that I want to keep. https://fabianlee.org/2018/03/09/java-collapsing-multiline-stack-traces-into-a-single-log-event-using-spring-backed-by-logback-or-log4j2/
I could also use ECS and concatenate it there, but it could affect other logs (though I think we only have Java apps). https://docs.aws.amazon.com/AmazonECS/latest/developerguide/firelens-concatanate-multiline.html
Which would be the best way to do it? Also is there a better way to do it in Spring compared to the guide that I found?
Normally, in docker/k8s, it's recommended that directly ouput the logs to the stdout.
Then we can use kubectl logs or docker logs to see the logs.
like: time=123 action=write msg=hello world , as a tty, it might be colorized for human friendliness.
However, if we want to export the logs to a log processing center, like EFK (elasticsearch-fluentd-kibana), we need a json-format log file.
like: {"time"=123,"action"="write","msg"="hello world"}
What I want?
Is there a log method that can take into account both human friendliness and json format?
I'm looking for a way that if I use docker logs, then I can get human-readable logs, and at the same time, the log collector can still get the logs in json-format
Conclusion
Thanks for the answer below. I have got 2 methods:
different log format in different env:
1.1 use text-format in developing: docker logs will print colorized and human readable logs.
1.2 use json-format in production: EFK can process json-format well.
log collector's format convertion
2.1 we use text-format, but in log collector like fluentd we can define some scripts to translate text-format kv pair to json-format kv pair.
Kubernetes has such option of structured logging for its system components.
Klog library allows to use --logging-format=json flag that enables to change the format of logs to JSON output - more information about it here and here.
yes you can do that with Flunetd, below are the basic action items that you need to take to finalize this setup
Configure docker container to log to stdout (you can use any format you like)
Configure FluentD to tail docker files from /var/lib/docker/containers/*/*-json.log
parse logs with Flunetd and change the format to JSON logs
output the logs to Elasticsearch.
This article shows exactly how to do this setup also this one explain how to parse Key-Value logs
We currently save all Kubernetes logs on a central log analysis tool. We use fluentbit to ship the logs.
Although we are able to retrieve and analyze the logs sent to stdout by the containers, we are unable to find the information shown in pods when we describe them (kubectl describe pod somepod). We want to save this information too, as it shows the Exit Code and Reason for a Terminated pod. Like exit codes 137 and OOMKilled.
Having this information in prometheus would also be valid. In prometheus we can see that some of the information is present like kube_pod_container_status_terminated_reason but the "exit code" is missing.
How could this be performed?
I am trying send info/error logs to StackDriver Logging on GCP from Cloud Function written in Go, however all logs dosen't have log level assignment.
I have created function from https://github.com/GoogleCloudPlatform/golang-samples/blob/master/functions/helloworld/hello_logging.go
to demonstrate problem.
Cloud Support here! What you are trying to do is not possible, as specified in the documentation:
Logs to stdout or stderr do not have an associated log level.
Internal system messages have the DEBUG log level.
What you probably need is to use Logging API, specifically the Log Levels section.
If this is not working for you either you could try using node.js instead of go as it currently emits logs to INFO and ERROR levels using console.log() and console.error() respectively.
I created a package to do exactly this:
github.com/ncruces/go-gcp/glog
It works on App Engine, Kubernetes Engine, Cloud Run, and Cloud Functions. Supports setting logging levels, request/tracing metadata, and structured logging.
Usage:
func HelloWorld(w http.ResponseWriter, r *http.Request) {
glog.Infoln("Hello logs")
glog.Errorln("Hello logs")
// or, to set request metadata:
logger := glog.ForRequest(r)
logger.Infoln("Hello logs")
}
you can use external library which allow you to set the log level. Here an example of what I use.
Use logrus for setting the minimal log level (you can improve this code by providing log level in env var) and joonix for fluentd formatter.(line 25)
A point of attention. Line 11, I rename the logrus package log
log "github.com/sirupsen/logrus"
Thus, don't use log standard library, but logrus library. It's sometime boring... you can simply replace log by logrus for avoiding all confusion.
Joonix is a fluentD formatter for transforming the logs into an ingestable format for Stackdriver.
I, too, took a stab at creating a package for this. If you use logrus (https://github.com/sirupsen/logrus), then this may be helpful:
https://github.com/tekkamanendless/gcfhook
I'm using this in production, and it's working just fine.
I recommend the https://github.com/apsystole/log. It's also log- and logrus-compatible, but a small zero-dependency module unlike the libraries used in two existing answers which bring 400+ modules as their dependencies (gasp... I am simply looking at go mod graph).
Unable to get stack driver to recognize syslog levels. Everything appears as an error despite specifying DEBUG
logwriter, e := syslog.New(syslog.LOG_DEBUG, "myprog")
if e == nil {
log.SetOutput(logwriter)
}
log.Print("log me")
I am aware of format requirements
if I stdout correct format as json payload stackdriver will magically pick it up and it works for me.
But why can't stackdriver recognize syslog input even if I syslog the json payload?
Syslog is a different protocol.
Try the golang driver.
This is tagged with k8s... if you need your kubernetes logs then just use the exporter. If you're just trying to get pod logs, you can send to STDOUT and the below will forward for you.