How to obtain the stack trace of the last (ideally of all) go routine (the app has multiple go routines) which panicked and recovered and logged only a not much descriptive error message? I don't know which routine recovered. Also, please keep in mind that I will not alter the code of any imported package. This panic happened in some of the imported packages which creates multiple go routines so I need a way to grab the stack trace of the last recovered routine in order to find where it panic.
The short answer is: Not possible but there are exceptions.
Golang has a few stack control methods and types.
You can control the stack levels with runtime/debug/SetTraceback
func SetTraceback(level string)
SetTraceback sets the amount of detail printed by the runtime
inthe traceback it prints before exiting due to an
unrecovered panic or an internal runtime error.
The level argument takes the same values as the GOTRACEBACK
environment variable. For example, SetTraceback("all") ensure
that the program prints all goroutines when it crashes.
See the package runtime documentation for details. If
SetTraceback is called with a level lower than that of the
environment variable, the call is ignored.
You can also print the stack strace with runtime/debug/Stack
func Stack() []byte
Stack returns a formatted stack trace of the goroutine that calls it. It calls runtime.Stack with a large enough buffer to capture the entire trace.
Also you need to understand how the Built-in funct recover works.
The recover built-in function allows a program to manage behavior of a
panicking goroutine. Executing a call to recover inside a deferred
function (but not any function called by it) stops the panicking sequence
by restoring normal execution and retrieves the error value passed to the
call of panic. If recover is called outside the deferred function it will
not stop a panicking sequence. In this case, or when the goroutine is not
panicking, or if the argument supplied to panic was nil, recover returns
nil. Thus the return value from recover reports whether the goroutine is
panicking.
func recover() interface{}
Working Example
This example assumes that the package does not call recover (detailed in another section).
Golang Playground Link
package main
import (
"log"
"errors"
"runtime/debug"
"time"
)
func f2() {
panic(errors.New("oops")) // line 11
}
func f1() {
f2() // line 15
}
func main() {
defer func() {
if e := recover(); e != nil {
log.Printf("%s: %s", e, debug.Stack()) // line 20
}
}()
go f1() // line 25
time.Sleep(time.Second * 1)
}
If package calls recover
If the code is recovering from the panic you need to use a debugger or remove the recover to understand what is going on as seen on the example below which demonstrate that recovered panics can not be "recovered" again.
Golang Playground Link
package main
import (
"log"
"errors"
"runtime/debug"
"time"
)
func f2() {
panic(errors.New("oops")) // line 11
}
func f1() {
defer func() {
if e := recover(); e != nil {
log.Printf("internal %s: %s", e, debug.Stack()) // line 20
}
}()
f2() // line 15
}
func main() {
defer func() {
if e := recover(); e != nil {
log.Printf("external %s: %s", e, debug.Stack()) // line 20
} else {
log.Println("Nothing to print")
}
}()
go f1() // line 25
time.Sleep(time.Second * 1)
}
Lesser then two evils
Debug with Delve Or temporaly edit the package so it logs the full message (once understood you can revert the changes).
Also if you find the problem let the package author know so it can be fixed.
Related
I was referred to this question: Program recovered from panic does not exit as expected
It works fine but it relies on knowing where the panic occurs in order to place the deferred function.
My code is as follows.
package main
import "fmt"
func main() {
defer recoverPanic()
f1()
f2()
f3()
}
func f1() {
fmt.Println("f1")
}
func f2() {
defer f3() //<--- don't want to defer f3 here because I might not know f2 will panic, panic could occuer elsewhere
fmt.Println("f2")
panic("f2")
}
func f3() {
fmt.Println("f3")
}
func recoverPanic() {
if r := recover(); r != nil {
fmt.Printf("Cause of panic ==>> %q\n", r)
}
}
Having the deferred function call f3() in the panicking function works, output below.
f1
f2
f3
Cause of panic ==>> "f2"
What if you have an application where you don't know where a panic occurs, do I need to put a defer in every function that might panic?
Commenting out the defer f3() gives me the following output.
f1
f2
Cause of panic ==>> "f2"
f3 never runs.
My question is how to continue execution of the program without having a deferred function call in every function that might panic?
You can't resume function execution after a panic. Panic is used when the current line of execution cannot continue correctly. Arbitrarily resuming execution after a panic (if it were possible) is begging immediately for another panic, because the state is already incorrect and just blazing ahead won't fix that.
For example, let's say a function panics when it tries to read out of bounds on a slice. How can it continue? What would it even mean to continue? Should it just read the out of bounds memory location and get garbage data? Continue with a zero value? Take a different value from the slice?
You must handle error cases; either by explicitly recovering, or preemptively checking / correcting conditions that will result in panic. At least in the standard library, functions that may spur a panic will say so in their documentation with an explanation of which conditions will result in panic.
If you commonly need to safely call void functions and recover from any panics, you can make a simple wrapper function for that.
func try(f func()) {
defer func() {
if err := recover(); err != nil {
fmt.Println("caught panic:", err)
}
}()
f()
}
Then
func main() {
try(f1)
try(f2)
try(f3)
}
What if an imported library spawns a goroutine that can panic? In this case, there is nothing a developer can do to stop the program from exiting.
Like in this code, calling a bad library with deferred recover does not help as the library is spawning a goroutine that panics, and it cannot be caught by the main's recover.
Do I understand it correct that the only remedy is to be very careful when choosing dependencies for one's project and hope that the authors do not do something similar?
package main
import (
"fmt"
"time"
)
func main() {
defer func() {
r := recover()
if r != nil {
fmt.Println("panic caught")
}
}()
badLibrary()
}
func badLibrary() {
go recklessFunction()
time.Sleep(time.Second)
}
func recklessFunction() {
panic("something went wrong")
}
You're right, you can't do anything about that. Neither can you if a dependency calls os.Exit() for example.
Launching goroutines as a library is often bad practice anyway, let the client (user of the library) choose if concurrent execution (a goroutine) is wanted, adding the go keyword is trivial. There are exceptions of course.
I have came across the following code that manages recover from panic using a wrapper. I understand that panic don't propagate across goroutines and they have to be managed independently, so it needs to be handled in the goroutine where the panic occurred.
package main
import (
"fmt"
"time"
)
func main() {
go Wrap(test)()
time.Sleep(time.Second)
fmt.Println("HELLO")
}
func test() {
panic("PANIC")
}
func Wrap(f func()) func() {
return func() {
defer func() {
if r := recover(); r != nil {
fmt.Printf("RECOVERED - %v\r\n", r)
}
}()
f()
}
}
func WrapWithSignal(f func(chan bool), signal chan bool) func() {
return func() {
defer func() {
if r := recover(); r != nil {
fmt.Printf("RECOVERED - %v\r\n", r)
signal <- false
}
}()
f(signal)
}
}
What will be the use of WrapWithSignal function, and how can I use it?
Is there another way to wrap functions in order to recover that is less verbose?
When implementing long-running processes, it's possible that certain code paths will result in a panic. This is usually common for things such as uninitialized maps and pointers, as well as division by zero problems in the case of poorly validated user input.
Having a program crash completely in these cases is frequently much worse than the panic itself, and so it can be helpful to catch and handle panics.
In most web applications, it's common to catch panics and emit an http.InternalServerError message when a panic occurs.
Within the recover, you can, essentially, do anything you want, although emitting a log is common.
1) Wrap with Signal can be used to recover from panic and send a signal to the given bool channel.
The warp with signal function implements the wrap pattern and can be used to wrap a signal channel and emit an event to the channel even when a panic occurs. In this case, if the recover was not called, you would get a deadlock.
package main
import (
"errors"
"fmt"
"math/rand"
"time"
)
func main() {
signal:= make(chan bool)
go WrapWithSignal(play, signal)()
if <-signal{
fmt.Println("Congratulations, you win!")
return
}
fmt.Println("You died.")
}
func play(signal chan bool){
fmt.Println("Playing russian roulette")
rand.Seed(time.Now().UnixNano())
if rand.Intn(2) == 1{
panic(errors.New("got a bullet"))
}
signal <- true
}
2) Is there another way to wrap functions in order to recover that is less verbose?
Short answer: NO, unless you use a package. A package theoretically is always less verbose.
Long answer: Depending the situation, you can for example, create a go routine wrapper and create your own panic group based on a waiting group. It can be useful while validating user input or doing concurrent operations and collect all the errors at the end. Split it into a package and you will get verbosity.
Just a sample: https://play.golang.org/p/on9AwZkvyIZ
I have the code as below and defer wasn't executed.
Doesn't defer work if we put it after panic?
package main
import (
"fmt"
)
func main() {
fmt.Println("begining of main")
panic("stop here")
defer fmt.Println("end of main")
}
nghiatran#nghiatran-VB:~/go/src/defer$ go run main.go
begining of main
panic: stop here
goroutine 1 [running]:
main.main()
/home/nghiatran/go/src/defer/main.go:9 +0x96
exit status 2
nghiatran#nghiatran-VB:~/go/src/defer$
The order of statements is wrong. Defer pushes function call to the stack. At the end of the function execution stacked calls are taken in reverse order and are executed. It does not matter whether function panics or not.
You need to push the function call first and then panic.
package main
import (
"fmt"
)
func main() {
defer fmt.Println("end of main") // push the call to the stack
fmt.Println("begining of main")
panic("stop here")
// the deffered functions are called as if they where here
}
The defer statement works differently than catch and finally blocks but it offer the same functionality.
See Use of defer in Go
defer won't work after panic because the control never reached the statement, hence it was never registered. This is like printing something after a return statement in a function, it's basically an unreachable code.
It's easy to redirect a golang panic to a file, just use recover() to capture that and use syscall.Dup2() to redirect. But when it comes to a C panic, it seems to be useless, just like the image, the console will show the error message like
"fatal error: unexpected signal during runtime execution" and some stack message. How to redirect these error message to a file
package main
/*
#include <stdio.h>
void sayHi(int a, int b) {
int c = a/b;
}
*/
import "C"
import (
"runtime/debug"
"syscall"
"os"
"log"
)
func main() {
logFile, logErr := os.OpenFile("/home/error.log", os.O_CREATE|os.O_RDWR|os.O_APPEND, 0666)
if logErr != nil {
log.Println("Fail to find", *logFile)
os.Exit(1)
}
log.SetOutput(logFile)
defer func() {
if r := recover(); r != nil {
syscall.Dup2(int(logFile.fd()), 2)
debug.PrintStack()
}
}()
C.sayHi(1, 0)
}
ps:The key point is how to redirect the error message on the terminal screen to a file?
Integer division by zero at runtime in C does not result in a traditional panic() that you can recover() from. It turns out the C standard does not define a specific action to be taken in this case; this is called "undefined behavior", and you should be working to avoid it (for instance, by checking your denominators).
But I assume you are only using that as an example; it just turns out to not be a very good example.
What can happen in C, and what you're getting in your case, is that the operating system can throw a signal. Signals are a way for the operating system to notify a traditional C program that something went wrong, for instance that the user hit control-C (asking to terminate the process) or that a floating-point exception occurred. You can catch these from Go with the os/signal package, but note that only the Interrupt and Kill signals are available on all OSs (and only Interrupt can be caught; Kill cannot). See the package documentation for details.