How to get logrus to print stack of pkg/errors - go

I'm using github.com/sirupsen/logrus and github.com/pkg/errors. When I hand an error wrapped or created from pkg/errors, all I see in the log out is the error message. I want to see the stack trace.
From this issue, https://github.com/sirupsen/logrus/issues/506, I infer that logrus has some native method for working with pkg/errors.
How can I do this?

The comment on your Logrus issue is incorrect (and incidentally, appears to come from someone with no affiliation with Logrus, and who has made no contributions to Logrus, so not actually from "the Logrus team").
It is easy to extract the stack trace in a pkg/errors error, as documented:
type stackTracer interface {
StackTrace() errors.StackTrace
}
This means that the easiest way to log the stack trace with logrus would be simply:
if stackErr, ok := err.(stackTracer); ok {
log.WithField("stacktrace", fmt.Sprintf("%+v", stackErr.StackTrace()))
}
As of today, when my a pull request of mine was merged with pkg/errors, this is now even easier, if you're using JSON logging:
if stackErr, ok := err.(stackTracer); ok {
log.WithField("stacktrace", stackErr.StackTrace())
}
This will produce a log format similar to "%+v", but without newlines or tabs, with one log entry per string, for easy marshaling into a JSON array.
Of course, both of these options force you to use the format defined by pkg/errors, which isn't always ideal. So instead, you can iterate through the stack trace, and produce your own formatting, possibly producing a format easily marshalable to JSON.
if err, ok := err.(stackTracer); ok {
for _, f := range err.StackTrace() {
fmt.Printf("%+s:%d\n", f, f) // Or your own formatting
}
}
Rather than printing each frame, you can coerce it into any format you like.

The inference is wrong. Logrus does not actually know how to handle the error.
Update the Logrus team official said that this is NOT a supported feature, https://github.com/sirupsen/logrus/issues/895#issuecomment-457656556.
A Java-ish Response
In order to universally work with error handlers in this way, I composed a new version of Entry, which is from Logrus. As the example shows, create a new Entry with what ever common fields you want (below the example is a logger set in a handler that keeps track of the caller id. Pass PgkError through your layers as you work the Entry. When you need to log specific errors, like call variables experiencing the error, start with the PkgError.WithError(...) then add your details.
This is a starting point. If you want to use this generally, implement all of the Entity interface on PkgErrorEntry. Continue to delegate to the internal entry, but return a new PkgErrorEntry. Such a change would make the value true drop in replacement for Entry.
package main
import (
"fmt"
"github.com/sirupsen/logrus"
"strings"
unwrappedErrors "errors"
"github.com/pkg/errors"
)
// PkgErrorEntry enables stack frame extraction directly into the log fields.
type PkgErrorEntry struct {
*logrus.Entry
// Depth defines how much of the stacktrace you want.
Depth int
}
// This is dirty pkg/errors.
type stackTracer interface {
StackTrace() errors.StackTrace
}
func (e *PkgErrorEntry) WithError(err error) *logrus.Entry {
out := e.Entry
common := func(pError stackTracer) {
st := pError.StackTrace()
depth := 3
if e.Depth != 0 {
depth = e.Depth
}
valued := fmt.Sprintf("%+v", st[0:depth])
valued = strings.Replace(valued, "\t", "", -1)
stack := strings.Split(valued, "\n")
out = out.WithField("stack", stack[2:])
}
if err2, ok := err.(stackTracer); ok {
common(err2)
}
if err2, ok := errors.Cause(err).(stackTracer); ok {
common(err2)
}
return out.WithError(err)
}
func someWhereElse() error {
return unwrappedErrors.New("Ouch")
}
func level1() error {
return level2()
}
func level2() error {
return errors.WithStack(unwrappedErrors.New("All wrapped up"))
}
func main() {
baseLog := logrus.New()
baseLog.SetFormatter(&logrus.JSONFormatter{})
errorHandling := PkgErrorEntry{Entry: baseLog.WithField("callerid", "1000")}
errorHandling.Info("Hello")
err := errors.New("Hi")
errorHandling.WithError(err).Error("That should have a stack.")
err = someWhereElse()
errorHandling.WithError(err).Info("Less painful error")
err = level1()
errorHandling.WithError(err).Warn("Should have multiple layers of stack")
}
A Gopher-ish way
See https://www.reddit.com/r/golang/comments/ajby88/how_to_get_stack_traces_in_logrus/ for more detail.
Ben Johnson wrote about making errors part of your domain. An abbreviated version is that you should put tracer attributes onto a custom error. When code directly under your control errors or when an error from a 3rd party library occurs, the code immediately dealing with the error should put a unique value into the custom error. This value will print as part of the custom error's Error() string implementation.
When developers get the log file, they will be able to grep the code base for that unique value. Ben says "Finally, we need to be able to provide all this information plus a logical stack trace to our operator so they can debug issues. Go already provides a simple method, error.Error(), to print error information so we can utilize that."
Here's Ben's example
// attachRole inserts a role record for a user in the database
func (s *UserService) attachRole(ctx context.Context, id int, role string) error {
const op = "attachRole"
if _, err := s.db.Exec(`INSERT roles...`); err != nil {
return &myapp.Error{Op: op, Err: err}
}
return nil
}
An issue I have with the grep-able code is that it's easy for the value to diverge from the original context. For example, say the name of the function was changed from attachRole to something else and the function was longer. It possible that the op value can diverge from the function name. Regardless, this appears to satisfy the general need of tracing to a problem, while treating errors a first class citizens.
Go2 might throw a curve at this into more the Java-ish response. Stay tuned.
https://go.googlesource.com/proposal/+/refs/changes/97/159497/3/design/XXXXX-error-values.md

Use custom hook to extract stacktrace
import (
"fmt"
"github.com/pkg/errors"
"github.com/sirupsen/logrus"
)
type StacktraceHook struct {
}
func (h *StacktraceHook) Levels() []logrus.Level {
return logrus.AllLevels
}
func (h *StacktraceHook) Fire(e *logrus.Entry) error {
if v, found := e.Data[logrus.ErrorKey]; found {
if err, iserr := v.(error); iserr {
type stackTracer interface {
StackTrace() errors.StackTrace
}
if st, isst := err.(stackTracer); isst {
stack := fmt.Sprintf("%+v", st.StackTrace())
e.Data["stacktrace"] = stack
}
}
}
return nil
}
func main() {
logrus.SetFormatter(&logrus.TextFormatter{DisableQuote: true})
logrus.AddHook(&StacktraceHook{})
logrus.WithError(errors.New("Foo")).Error("Wrong")
}
Output
time=2009-11-10T23:00:00Z level=error msg=Wrong error=Foo stacktrace=
main.main
/tmp/sandbox1710078453/prog.go:36
runtime.main
/usr/local/go-faketime/src/runtime/proc.go:250
runtime.goexit
/usr/local/go-faketime/src/runtime/asm_amd64.s:1594

Related

Formatting errors that wrap multiple errors

In Go 1.20, we'll be able to wrap multiple errors. The following is a simple example where two errors, err1 and err2, each wrap two lower level errors:
package main
import (
"errors"
"fmt"
)
func main() {
err1_1 := errors.New("err1.1")
err1_2 := errors.New("err1.2")
err1 := fmt.Errorf("err1: %w", errors.Join(err1_1, err1_2))
err2_1 := errors.New("err2.1")
err2_2 := errors.New("err2.2")
err2 := fmt.Errorf("err2: %w", errors.Join(err2_1, err2_2))
fmt.Printf("%s\n", errors.Join(err1, err2))
}
The output from the above looks as follows:
err1: err1.1
err1.2
err2: err2.1
err2.2
I have to admit that I am not a fan of how this looks because there's no indication that err1.2 is an inner error of err1 and similar for err2.2 with respect to err2. The following is one option that I find would make it clearer:
err1: err1.1
err1.2
err2: err2.1
err2.2
So my question is, do we have any error formatting options in Go 1.20 that I have overlooked? I have come across this proposal related to error formatting for Go2, but I assume we won't see that any time soon.
Edit
As far as I can see, unwrapping the errors myself is not a reliable option because errors may be constructed in different ways. For example, an error may be wrapped as follows:
type CustomError struct {
someValue string
wrappedErr error
}
func (err *CustomError) Error() string {
return fmt.Sprintf("custom error %s: %s", someValue, wrappedErr )
}
func (err *CustomError) Unwrap() error {
return err.wrappedErr
}
Now consider the situation where wrappedErr is a joined error. I can see that it's possible to unwrap CustomError and detect that wrappedErr is a joined/multi error by checking if it implements interface { Unwrap() []error }. I think the issue is when one tries to reconstruct the error message. I can't see how one can get the text from CustomError.Error() while replacing the value of wrapperErr to a custom formatted joined error. Furthermore, while a joined error may have been constructed using errors.Join it could also have been constructed with a custom type that implements interface { Unwrap() []error } where again the formatting of the error message is an unknown.
I guess this issue is the whole purpose of the proposal that I linked above.

unserialize php in goland

I have a file with serialized array in PHP.
The content of the file locks like this
a:2:{i:250;s:7:"my_catz";s:7:"abcd.jp";a:2:{s:11:"category_id";i:250;s:13:"category_name";s:7:"my_catz";}}
The array unserialized is this
(
[250] => my_catz
[abcd.jp] => Array
(
[category_id] => 250
[category_name] => my_catz
)
)
Now, i want to get the content of the file in GO, unserialize it convert it to an array.
In GO i can get the content of the file using
dat, err := os.ReadFile("/etc/squid3/compiled-categories.db")
if err != nil {
if e.Debug {
log.Printf("error reading /etc/squid3/compiled-categories.db: ", err)
}
}
And unserialized it using github.com/techoner/gophp library
package categorization
import (
"fmt"
"os"
"github.com/techoner/gophp"
"log"
"errors"
)
type Data struct {
Website string
Debug bool
}
func (e Data) CheckPersonalCategories() (int,string) {
if e.Debug {
log.Printf("Checking Personal Categories")
}
if _, err := os.Stat("/etc/squid3/compiled-categories.db"); errors.Is(err, os.ErrNotExist) {
if e.Debug {
log.Printf("/etc/squid3/compiled-categories.db not exit: ", err)
}
return 0,""
}
dat, err := os.ReadFile("/etc/squid3/compiled-categories.db")
if err != nil {
if e.Debug {
log.Printf("error reading /etc/squid3/compiled-categories.db: ", err)
}
}
out, _ := gophp.Unserialize(dat)
fmt.Println(out["abcd.jp"])
return 0,""
}
But I can't access to the array, for example, when I try access to array key using out["abcd.jp"] i get this error message
invalid operation: out["abcd.jp"] (type interface {} does not support indexing)
The result of out is
map[250:my_catz abcd.jp:map[category_id:250 category_name:my_catz]]
Seams that is unserializing
Don't make assumptions about what is and isn't succeeding in your code. Error responses are the only reliable way to know whether a function succeeded. In this case the assumption may hold, but ignoring errors is always a mistake. Invest time in catching errors and at least panic them - don't instead waste your time ignoring errors and then trying to debug unreliable code.
invalid operation: out["abcd.jp"] (type interface {} does not support indexing)
The package you're using unfortunately doesn't provide any documentation so you have to read the source to understand that gophp.Unserialize returns (interface{}, error). This makes sense; php can serialize any value, so Unserialize must be able to return any value.
out is therefore an interface{} whose underlying value depends on the data. To turn an interface{} into a particular value requires a type assertion. In this case, we think the underlying data should be map[string]interface{}. So we need to do a type assertion:
mout, ok := out.(map[string]interface{})
Before we get to the working code, one more point I'd like you to think about. Look at the code below: I started it from your code, but the resemblance is very slight. I took out almost all the code because it was completely irrelevant to your question. I added the input data to the code to make a minimal reproduction of your code (as I asked you to do and you declined to do). This is a very good use of your time for 2 reasons: first, it makes it a lot easier to get answers (both because it shows sufficient effort on your part and because it simplifies the description of the problem), and second, because it's excellent practice for debugging. I make minimal reproductions of code flows all the time to better understand how to do things.
You'll notice you can run this code now without any additional effort. That's the right way to provide a minimal reproducible example - not with a chunk of mostly irrelevant code which still can't be executed by anybody.
The Go Plaground is a great way to demonstrate go-specific code that others can execute and investigate. You can also see the code below at https://go.dev/play/p/QfCl08Gx53e
package main
import (
"fmt"
"github.com/techoner/gophp"
)
type Data struct {
Website string
Debug bool
}
func main() {
var dat = []byte(`a:2:{i:250;s:7:"my_catz";s:7:"abcd.jp";a:2:{s:11:"category_id";i:250;s:13:"category_name";s:7:"my_catz";}}`)
out, err := gophp.Unserialize(dat)
if err != nil {
panic(err)
}
if mout, ok := out.(map[string]interface{}); ok {
fmt.Println(mout["abcd.jp"])
}
}

How to compare Go errors

I have an error value which when printed on console gives me Token is expired
How can I compare it with a specific error value? I tried this but it did not work:
if err == errors.New("Token is expired") {
log.Printf("Unauthorised: %s\n", err)
}
Declaring an error, and comparing it with '==' (as in err == myPkg.ErrTokenExpired) is no longer the best practice with Go 1.13 (Q3 2019)
The release notes mentions:
Go 1.13 contains support for error wrapping, as first proposed in the Error Values proposal and discussed on the associated issue.
An error e can wrap another error w by providing an Unwrap method that returns w.
Both e and w are available to programs, allowing e to provide additional context to w or to reinterpret it while still allowing programs to make decisions based on w.
To support wrapping, fmt.Errorf now has a %w verb for creating wrapped errors, and three new functions in the errors package ( errors.Unwrap, errors.Is and errors.As) simplify unwrapping and inspecting wrapped errors.
So the Error Value FAQ explains:
You need to be prepared that errors you get may be wrapped.
If you currently compare errors using ==, use errors.Is instead.
Example:
if err == io.ErrUnexpectedEOF
becomes
if errors.Is(err, io.ErrUnexpectedEOF)
Checks of the form if err != nil need not be changed.
Comparisons to io.EOF need not be changed, because io.EOF should never be wrapped.
If you check for an error type using a type assertion or type switch, use errors.As instead. Example:
if e, ok := err.(*os.PathError); ok
becomes
var e *os.PathError
if errors.As(err, &e)
Also use this pattern to check whether an error implements an interface. (This is one of those rare cases when a pointer to an interface is appropriate.)
Rewrite a type switch as a sequence of if-elses.
This answer is for Go 1.12 and earlier releases.
Define an error value in a library
package fruits
var NoMorePumpkins = errors.New("No more pumpkins")
Do not create errors with errors.New anywhere in the code but return the predefined value whenever error occurs and then you can do the following:
package shop
if err == fruits.NoMorePumpkins {
...
}
See io package errors for reference.
This can be improved by adding methods to hide the check implementation and make the client code more immune to changes in fruits package.
package fruits
func IsNoMorePumpkins(err error) bool {
return err == NoMorePumpkins
}
See os package errors for reference.
Try
err.Error() == "Token is expired"
Or create your own error by implementing the error interface.
It's idiomatic for packages to export error variables that they use so others can compare against them.
E.g. If an error would came from a package named myPkg and was defined as:
var ErrTokenExpired error = errors.New("Token is expired")
You could compare the errors directly as:
if err == myPkg.ErrTokenExpired {
log.Printf("Unauthorised: %s\n", err)
}
If the errors come from a third party package and that doesn't use exported error variables then what you can do is simply to compare against the string you get from err.Error() but be careful with this approach as changing an Error string might not be released in a major version and would break your business logic.
The error type is an interface type. An error variable represents any value that can describe itself as a string. Here is the interface's declaration:
type error interface {
Error() string
}
The most commonly-used error implementation is the errors package's unexported errorString type:
// errorString is a trivial implementation of error.
type errorString struct {
s string
}
func (e *errorString) Error() string {
return e.s
}
See this working code output (The Go Playground):
package main
import (
"errors"
"fmt"
"io"
)
func main() {
err1 := fmt.Errorf("Error")
err2 := errors.New("Error")
err3 := io.EOF
fmt.Println(err1) //Error
fmt.Printf("%#v\n", err1) // &errors.errorString{s:"Error"}
fmt.Printf("%#v\n", err2) // &errors.errorString{s:"Error"}
fmt.Printf("%#v\n", err3) // &errors.errorString{s:"EOF"}
}
output:
Error
&errors.errorString{s:"Error"}
&errors.errorString{s:"Error"}
&errors.errorString{s:"EOF"}
Also see: Comparison operators
Comparison operators compare two operands and yield an untyped boolean
value. In any comparison, the first operand must be assignable to the
type of the second operand, or vice versa.
The equality operators == and != apply to operands that are
comparable.
Pointer values are comparable. Two pointer values are equal if they
point to the same variable or if both have value nil. Pointers to
distinct zero-size variables may or may not be equal.
Interface values are comparable. Two interface values are equal if
they have identical dynamic types and equal dynamic values or if both
have value nil.
A value x of non-interface type X and a value t of interface type T
are comparable when values of type X are comparable and X implements
T. They are equal if t's dynamic type is identical to X and t's
dynamic value is equal to x.
Struct values are comparable if all their fields are comparable. Two
struct values are equal if their corresponding non-blank fields are
equal.
So:
1- You may use Error(), like this working code (The Go Playground):
package main
import (
"errors"
"fmt"
)
func main() {
err1 := errors.New("Token is expired")
err2 := errors.New("Token is expired")
if err1.Error() == err2.Error() {
fmt.Println(err1.Error() == err2.Error()) // true
}
}
output:
true
2- Also you may compare it with nil, like this working code (The Go Playground):
package main
import (
"errors"
"fmt"
)
func main() {
err1 := errors.New("Token is expired")
err2 := errors.New("Token is expired")
if err1 != nil {
fmt.Println(err1 == err2) // false
}
}
output:
false
3- Also you may compare it with exact same error, like this working code
(The Go Playground):
package main
import (
"fmt"
"io"
)
func main() {
err1 := io.EOF
if err1 == io.EOF {
fmt.Println("err1 is : ", err1)
}
}
output:
err1 is : EOF
ref: https://blog.golang.org/error-handling-and-go
It's being discouraged to compare errors by strings. Instead you should compare errors by value.
package main
import "errors"
var NotFound = errors.New("not found")
func main() {
if err := doSomething(); errors.Is(err, NotFound) {
println(err)
}
}
func doSomething() error {
return NotFound
}
It is especially useful if you are library author and would like to export errors so users can act differently on different type of errors. Standard library does it as well.
Problem with this approach is that exported values can be changed by anyone as Go doesn't support immutable values. Nothing prevents you, though, to use string as an error and make it const.
package main
type CustomError string
func (ce CustomError) Error() string {
return string(ce)
}
const NotFound CustomError = "not found"
func main() {
if err := doSomething(); errors.Is(err, NotFound) {
println(err)
}
}
func doSomething() error {
return NotFound
}
It is more verbose but safer approach.
You should first consider comparing errors by value, as described in other solutions with:
if errors.Is(err1, err2) {
// do sth
}
However in some cases the error returned from a function is a bit complex, e.g. an error is being wrapped multiple times, with a context being added to it in each function call like fmt.Errorf("some context: %w", err), and you may simply just want to compare the error message of two errors. In such cases you can do this:
// SameErrorMessage checks whether two errors have the same messages.
func SameErrorMessage(err, target error) bool {
if target == nil || err == nil {
return err == target
}
return err.Error() == target.Error()
}
func main() {
...
if SameErrorMessage(err1, err2) {
// do sth
}
}
Note that if you simply use
if err1.Error() == err2.Error() {
// do sth
}
You might face nil pointer dereference runtime error if either of err1 or err2 be nil.
To add to #wst 's answer, in some cases, the errors.Is(err, NotFound) approach may not work for reasons I am trying to figure out too. If someone knows, please let me know in the comments.
But I found a better approach to use it in the following way which was working for me:
if NotFound.Is(err) {
// do something
}
Where var NotFound = errors.New("not found") is an exported common error declared.
In my case, the solution was
if models.GetUnAuthenticatedError().Is(err) {
// Do something
}
I want to post one case where errors.Is could work well for custom errors with non-comparable values.
type CustomError struct {
Meta map[string]interface{}
Message string
}
func (c CustomError) Error() string {
return c.Message
}
var (
ErrorA = CustomError{Message: "msg", Meta: map[string]interface{}{"key": "value"}}
)
func DoSomething() error {
return ErrorA
}
func main() {
err := DoSomething()
if errors.Is(err, ErrorA) {
fmt.Println("error is errorA")
} else {
fmt.Println("error is NOT errorA")
}
}
Output
error is NOT errorA
Playground
The reason is errors.Is checks whether the target is comparable or not
func Is(err, target error) bool {
if target == nil {
return err == target
}
isComparable := reflectlite.TypeOf(target).Comparable()
The comparable type in Go are
booleans, numbers, strings, pointers, channels, arrays of comparable types, structs whose fields are all comparable types
Since the Meta map[string]interface{} of CustomError is NOT comparable, so errors.Is checks failed.
One workaround is declare the ErrorA = &CustomError{Message: "msg", Meta: map[string]interface{}{"key": "value"}} as pointer.

How to get the stack trace pointing to actual error reason

Let's say I have some code like this:
value, err := some3rdpartylib.DoSomething()
if err != nil {
panic(err)
}
In case err != nil I will get something like this:
panic: some error explanation here
goroutine 1 [running]:
main.main()
/tmp/blabla/main.go:6 +0x80
This stack trace is completely legit but sometimes these error messages may not clarify what happened and so I'd like to dig deeper into the source code of the 3rd party library to investigate what exactly causes this error to be returned. However when my code panics like this, there is no way to get the actual place that returned this error.
A little bit more clarification: as I'm coming from JVM world where exceptions are thrown I can completely trace what exactly line of code thrown the exception and thus can easily find the place and see what gone wrong. Go stack trace ends exactly where my code panics and thus not too useful in my case.
I've created a playground here and ideally I'd like to be able to trace the error to the place it was actually returned from, not panic. (e.g. to line 17, return "", errors.New("some error explanation here"))
Is this even possible?
I think that there is an easier way to achieve this. You can try wrapping errors using the golang "default" third party library error package:
You need to define the interface to be implemented by your error :
type stackTracer interface {
StackTrace() errors.StackTrace
}
Then use it when wrapping/processing an error :
err, ok := errors.(stackTracer) // ok is false if errors doesn't implement stackTracer
stack := err.StackTrace()
fmt.Println(stack) // here you'll have your stack trace
Shortly: this is not possible.
Since errors are values, they are not treated in any special way. Due to this, when function (normally) returns, stack is no more available (ie. another function call may overwrite memory used by returning-error function' stack).
There is a tool called trace which was introduced with go1.5, but for now, there is no comprehensive tutorial available neither any of those I found says that this kind of feature will be included.
As others have pointed out tracing errors in go isn't trivial. There are projects like juju/errgo, that allow you to wrap errors and then trace these errors back. For that to work tough, you must use them consistently throughout your project and that won't help you with errors in 3rd party libraries or with errors that get handled and never get returned.
Because this is such a common issue and I really got annoyed with this, I wrote a small debug utility that will add debug code to go files that logs every returned error (value that implements error) and the function in which it was returned to STDOUT (if you need more advanced logging just hack the logger in the project, it's really simple).
Installation
go get github.com/gellweiler/errgotrace
Usage
To debug all files in the current directory:
$ find . -name '*.go' -print0 | xargs -0 errgotrace -w
To remove the added debug code from the go files:
$ find . -name '*.go' -print0 | xargs -0 errgotrace -w -r
Then just simply compile & run your code or your test cases.
Sample Output
[...]
2017/12/13 00:54:39 [ERRGOTRACE] parser.*Parser.objectKey: EOF token found
2017/12/13 00:54:39 [ERRGOTRACE] parser.*Parser.objectItem: EOF token found
2017/12/13 00:54:39 [ERRGOTRACE] parser.*Parser.objectKey: EOF token found
2017/12/13 00:54:39 [ERRGOTRACE] parser.*Parser.objectItem: EOF token found
2017/12/13 00:54:39 [ERRGOTRACE] parser.*Parser.objectKey: At 3:4: nested object expected: LBRACE got: ASSIGN
2017/12/13 00:54:39 [ERRGOTRACE] parser.*Parser.objectItem: At 3:4: nested object expected: LBRACE got: ASSIGN
2017/12/13 00:54:39 [ERRGOTRACE] parser.*Parser.objectList: At 3:4: nested object expected: LBRACE got: ASSIGN
2017/12/13 00:54:39 [ERRGOTRACE] parser.*Parser.Parse: At 2:31: literal not terminated
2017/12/13 00:54:39 [ERRGOTRACE] parser.Parse: At 2:31: literal not terminated
2017/12/13 00:54:39 [ERRGOTRACE] hcl.parse: At 2:31: literal not terminated
2017/12/13 00:54:39 [ERRGOTRACE] hcl.ParseBytes: At 2:31: literal not terminated
2017/12/13 00:54:39 [ERRGOTRACE] formula.parse: parsing failed
[...]
As you can see from this output, it's really easy to tell in which function the error originally occurred. Once you know that, you can use a debugger to get more context.
Take a look at https://github.com/ztrue/tracerr
I created this package in order to have both stack trace and source fragments to be able to debug faster and log errors with much more details.
Here is a code example:
package main
import (
"io/ioutil"
"github.com/ztrue/tracerr"
)
func main() {
if err := read(); err != nil {
tracerr.PrintSourceColor(err)
}
}
func read() error {
return readNonExistent()
}
func readNonExistent() error {
_, err := ioutil.ReadFile("/tmp/non_existent_file")
// Add stack trace to existing error, no matter if it's nil.
return tracerr.Wrap(err)
}
And here is the output:
package main
import (
"errors"
"fmt"
)
func main() {
value, err := DoSomething()
if err != nil {
panic(err)
}
fmt.Println(value)
}
func DoSomething() (string, error) {
return "", errors.New("some error explanation here")
}
The problem is the standard package errors does not attach the stack trace at the point it occurs. You can use github.com/pkg/errors to do that:
package main
import (
"github.com/pkg/errors"
"fmt"
)
func main() {
value, err := DoSomething()
if err != nil {
fmt.Printf("%+v", err)
}
fmt.Println(value)
}
func DoSomething() (string, error) {
return "", errors.New("some error explanation here")
}
$ go run stacktrace.go
some error explanation here
main.DoSomething
/Users/quanta/go/src/github.com/quantonganh/errors/stacktrace.go:18
main.main
/Users/quanta/go/src/github.com/quantonganh/errors/stacktrace.go:10
runtime.main
/usr/local/Cellar/go/1.15.2/libexec/src/runtime/proc.go:204
runtime.goexit
/usr/local/Cellar/go/1.15.2/libexec/src/runtime/asm_amd64.s:1374
More details: https://dave.cheney.net/2016/06/12/stack-traces-and-the-errors-package
Take a look at: https://github.com/efimovalex/stackerr
Is this the thing you are looking for?
package main
import "github.com/efimovalex/stackerr"
import "fmt"
func f1() *stackerr.Err {
err := stackerr.Error("message")
return err.Stack()
}
func f2() *stackerr.Err {
err := f1()
return err.Stack()
}
type t1 struct{}
func (t *t1) f3() *stackerr.Err {
err := f2()
return err.Stack()
}
func main() {
ts := t1{}
err := ts.f3()
err.Log()
}
Result:
2017/08/31 12:13:47 Error Stacktrace:
-> github.com/efimovalex/stackerr/example/main.go:25 (main.main)
-> github.com/efimovalex/stackerr/example/main.go:19 (main.(*t1).f3)
-> github.com/efimovalex/stackerr/example/main.go:12 (main.f2)
-> github.com/efimovalex/stackerr/example/main.go:7 (main.f1)
As I know, stackrerror is the simplest stack display package. You can use all the native log library records or output the call stack yourself. For example:
package main
import "github.com/lingdor/stackerror"
func act1()error {
return stackerror.New("here Error")
}
func main(){
err:=act1()
fmt.println(err.Error()) //panic(err) and log.Info(err) are ok
}
output :
*stackError.stackError : here Error
at main.act1( /Users/user/go/testMain/src/main/main.go:17 )
at main.main( /Users/user/go/testMain/src/main/main.go:22 )
at runtime.main( /usr/local/Cellar/go/1.13.4/libexec/src/runtime/proc.go:203 )
You can use the built-in Recover function to handle panic and print the stack trace.
From https://blog.golang.org/defer-panic-and-recover
Recover is a built-in function that regains control of a panicking
goroutine. Recover is only useful inside deferred functions. During
normal execution, a call to recover will return nil and have no other
effect. If the current goroutine is panicking, a call to recover will
capture the value given to panic and resume normal execution.
I have modified your example to use recover and eris. Eris provides a better way to handle, trace, and log errors in Go.
package main
import (
"github.com/rotisserie/eris"
"fmt"
)
func main() {
value, err := DoSomething()
defer func() {
if r := recover(); r!= nil {
fmt.Println(fmt.Sprintf("%+v", r))
}
}()
if err != nil {
panic(err)
}
fmt.Println(value)
}
func DoSomething() (string, error) {
return "", eris.New("some error explanation here")
}
The output is:
some error explanation here
main.DoSomething: /tmp/sandbox147128055/prog.go: 23
main.main: /tmp/sandbox147128055/prog.go: 9
runtime.main: /usr/local/go/src/runtime/proc.go: 203
runtime.goexit: /usr/local/go/src/runtime/asm_amd64p32.s: 523
See it in action here https://play.golang.org/p/jgkaR42ub5q

how to find, "invalid character ',' looking for beginning of value" error message

I have a short Go program that runs the go list -json command for several packages, stores the output of each run of the command in a json.RawMessage, appends each json.RawMessage into a slice of json.RawMessages, and then returns the result to the server after concatenating each of the json.RawMessages together and compacting the json. However, there is an error message that gets produced when I run json.Compact that I can't locate the source of. Googling this error message reveals that most people who seem to encounter it--whether it's for an invalid , or some other character--have a hard time finding the source of it.
invalid character ',' looking for beginning of value
The code with comments is available to view here on play.golang.org (although it won't run there) and also below.
Question: can you explain the source of this error and how to prevent it?
(Note, some of the packages were included just for testing purposes)
package main
import (
"expvar"
"encoding/json"
"bytes"
"fmt"
"github.com/go-martini/martini"
"github.com/zenazn/goji"
"github.com/zenazn/goji/web"
"go/build"
"log"
"math/rand"
"net/http"
_ "net/http/pprof"
"os/exec"
)
type myType struct {
J []json.RawMessage
}
var pack map[string]string
type GoList struct {
Imports []string
}
type Import struct {
Dir string
ImportPath string
Name string
Target string
Standard bool
Root string
GoFiles []string
Imports []string
Deps []string
}
const contentTypeJSON = "application/json"
func main() {
http.HandleFunc("/importgraph", func(w http.ResponseWriter, r *http.Request) { importGraph(w, r) })
http.HandleFunc("/", handler)
http.ListenAndServe(":8080", nil)
}
func handler(w http.ResponseWriter, r *http.Request) {
fmt.Println("Inside handler")
fmt.Fprintf(w, "Hello world from my Go program!")
}
func importGraph(w http.ResponseWriter, r *http.Request) {
pack = make(map[string]string)
var t myType
cmd := exec.Command("go", "list", "-json")
stdout, err := cmd.Output()
if err != nil {
println(err.Error())
return
}
var list GoList
err = json.Unmarshal(stdout, &list)
for _, d := range list.Imports {
//get the imports for each of the packages listed by go list -json
t.imports(d)
}
var buff bytes.Buffer
//concatenate the separate json.RawMessages together into json
buff.WriteByte('[')
for i, j := range t.J {
if i != 0 {
buff.WriteByte(',')
}
buff.Write([]byte(j))
}
buff.WriteByte(']')
var buffer bytes.Buffer
if err := json.Compact(&buffer, buff.Bytes()); err != nil {
println(err.Error()) //error message: invalid character ',' looking for beginning of value
return
}
w.Header().Set("Content-Type", contentTypeJSON)
w.Write(buffer.Bytes())
}
func (myObj *myType) imports(pk string) error {
cmd := exec.Command("go", "list", "-json", pk)
stdout, _ := cmd.Output()
pack[pk] = pk
var deplist Import
json.Unmarshal(stdout, &deplist)
var newj json.RawMessage
json.Unmarshal(stdout, &newj)
myObj.J = append(myObj.J, newj)
for _, imp := range deplist.Imports {
if _, ok := pack[imp]; !ok {
myObj.imports(imp) //recursive call to get the imports of the imports etc
}
}
return nil
}
First, as has been commented, are you sure you can't use
the go/build package directly rather than running go list?
I Wouldn't use println (or fmt.Println) inside HTTP handlers. It's much better to use log.Println and/or get the error into the ResponseWriter. Also, it's a good idea to wrap your ListenAndServe call with log.Fatal.
When printing/logging error values you can just use err, no need to have err.Error().
Further, when you actually want to do something more detailed than just reporting/logging the error message you can look at it's type and other info. For example, log.Printf("verbose error info: %#v", err) gives:
&json.SyntaxError{msg:"invalid character ',' looking for beginning of value", Offset:0}
I tried this because I know the json package returns various error types with additional info and I was hoping the offset value would be of help. If it had been then something like this might have been helpful:
if err := json.Compact(…) {
if err != nil {
log.Println("json.Compact:", err)
if serr, ok := err.(*json.SyntaxError); ok {
log.Println("Occurred at offset:", serr.Offset)
// … something to show the data in buff around that offset …
}
}
}
But offset zero isn't helpful :(
So although this doesn't identify you problem hopefully
it can be of some help to your further investigation.
Edit:
So after adding:
log.Println("Write file:", ioutil.WriteFile("data.json", buff.Bytes(), 0600))
to the above error handling block I then ran a JSON validator on the resultant file and it identified this piece:
"XTestImports": [
"io",
"log",
"net"
]
},,{
"Dir": "/usr/local/go/src/mime",
"ImportPath": "mime",
"Name": "mime",
Note the double ,,.
That should tell you whete the error in your code is.
But if not, you need to skip empty entries, either when processing t.J or when you build it. The later is better and just involves:
if len(newj) > 0 {
myObj.J = append(myObj.J, newj)
}
(where btw you don't check for errors from json.Unmarshal so it's not clear if that is supposed to ever be empty or if it's empty due to a preceeding error. Never ignore error returns!)
I also encountered the same error message in a Go program, but the error message was within the HTTP response error, in HTML format when my HTTP response parser expected JSON.
For me, the solution was to change my request to include setting the Content-Type header to application/json. How you do this depends on which http client library you happen to be using; if you have access to the http.Header core type, you can set the header with .Set(...).
I realize the scope of this fix for me may not apply to the original question, but I came here first after googling and thought this would help others, since the message was not particularly obvious at first glance. The hint is that the invalid < character is the first HTML character in the error/response, which is likely the result of the request type not being set to application/json, thus the server responds with a non JSON response.
For me the issue was I was trying to parse the already parsed JSON.
I was also facing this error "invalid character 'N' looking for beginning of value".
This error was coming while "unmarshalling the non-json response into a json". I was expecting a json response, so wrote go code to unmarshal it into a json. But, due to URL change, the response that I was getting was a text ie. "404 Not found" error, which cannot be unmarshalled into a json.
"invalid character 'N' looking for beginning of value"
So, to summarise, this error appears when we are trying to unmarshal a non-json response (text/html/xml) into json.
Reason for this eerie error message is :
// When unmarshaling quoted strings, invalid UTF-8 or
// invalid UTF-16 surrogate pairs are not treated as an error.
// Instead, they are replaced by the Unicode replacement
// character U+FFFD.
https://golang.org/src/encoding/json/decode.go
In my case I saved my json as string then parsed it by :
stringData = JSON.parse(myJsonString)
I also had the same error another time using gin-context-ShouldBind() (https://godoc.org/github.com/gin-gonic/gin#Context.ShouldBind) and mapping my json to go object:
error was because it needs a json as string, so I used : JSON.stringify(jsonObject) when sending my request from front-end part.
And in case someone has the same problem as me, I needed to call JSON.stringify on my post data.
I encountered a similar problem with my error message being:
invalid character 'I' looking for beginning of value
In my case, i was trying to decode BSON using json.Unmarshal. Json doesn't recognize the ISODate type, which caused this error.
I had a similar issue. For me I omitted the first letter of my authorization token. So instead of
"eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJlbWFpbCI6InJhcGhhZWxuZ0BlbWFpbC5jb20iLCJleHAiOjE2MTM5NTQzMjB9.yPGC937VNAF8Qg05Z1x3fZ3zu_MUs-cA_Iag5-4RcJE"
I used this
"yJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJlbWFpbCI6InJhcGhhZWxuZ0BlbWFpbC5jb20iLCJleHAiOjE2MTM5NTQzMjB9.yPGC937VNAF8Qg05Z1x3fZ3zu_MUs-cA_Iag5-4RcJE"

Resources