What happens when defer is called twice on same variable? - go

What happened when defer called twice when the struct of that method has been changed?
For example:
rows := Query(`SELECT FROM whatever`)
defer rows.Close()
for rows.Next() {
// do something
}
rows = Query(`SELECT FROM another`)
defer rows.Close()
for rows.Next() {
// do something else
}
which rows when the last rows.Close() called?

It depends on the method receiver and on the type of the variable.
Short answer: if you're using the database/sql package, your deferred Rows.Close() methods will properly close both of your Rows instances because Rows.Close() has pointer receiver and because DB.Query() returns a pointer (and so rows is a pointer). See reasoning and explanation below.
To avoid confusion, I recommend using different variables and it will be clear what you want and what will be closed:
rows := Query(`SELECT FROM whatever`)
defer rows.Close()
// ...
rows2 := Query(`SELECT FROM whatever`)
defer rows2.Close()
I'd like to point out an important fact that comes from the deferred function and its parameters being evaluated immedately which is stated in the Effective Go blog post and in the Language Spec: Deferred statements too:
Each time a "defer" statement executes, the function value and parameters to the call are evaluated as usual and saved anew but the actual function is not invoked. Instead, deferred functions are invoked immediately before the surrounding function returns, in the reverse order they were deferred.
If variable is not a pointer: You will observe different results when calling a method deferred, depending if the method has a pointer receiver.
If variable is a pointer, you will see always the "desired" result.
See this example:
type X struct {
S string
}
func (x X) Close() {
fmt.Println("Value-Closing", x.S)
}
func (x *X) CloseP() {
fmt.Println("Pointer-Closing", x.S)
}
func main() {
x := X{"Value-X First"}
defer x.Close()
x = X{"Value-X Second"}
defer x.Close()
x2 := X{"Value-X2 First"}
defer x2.CloseP()
x2 = X{"Value-X2 Second"}
defer x2.CloseP()
xp := &X{"Pointer-X First"}
defer xp.Close()
xp = &X{"Pointer-X Second"}
defer xp.Close()
xp2 := &X{"Pointer-X2 First"}
defer xp2.CloseP()
xp2 = &X{"Pointer-X2 Second"}
defer xp2.CloseP()
}
Output:
Pointer-Closing Pointer-X2 Second
Pointer-Closing Pointer-X2 First
Value-Closing Pointer-X Second
Value-Closing Pointer-X First
Pointer-Closing Value-X2 Second
Pointer-Closing Value-X2 Second
Value-Closing Value-X Second
Value-Closing Value-X First
Try it on the Go Playground.
Using a pointer variable the result is always good (as expected).
Using a non-pointer variable and using pointer receiver we see the same printed results (the latest) but if we have value receiver, it prints 2 different results.
Explanation for non-pointer variable:
As stated, deferred function including the receiver is evaluated when the defer executes. In case of a pointer receiver it will be the address of the local variable. So when you assign a new value to it and call another defer, the pointer receiver will be again the same address of the local variable (just the pointed value is different). So later when the function is executed, both will use the same address twice but the pointed value will be the same, the one assigned later.
In case of value receiver, the receiver is a copy which is made when the defer executed, so if you assign a new value to the variable and call another defer, another copy will be made which is different from the previous one.

Effective Go mentions:
The arguments to the deferred function (which include the receiver if the function is a method) are evaluated when the defer executes, not when the call executes.
Besides avoiding worries about variables changing values as the function executes, this means that a single deferred call site can defer multiple function executions
In your case, the defer would reference the second rows instance.
The two deferred functions are executed in LIFO order (as mentioned also in "Defer, Panic, and Recover").
As icza mentions in his answer and in the comments:
The 2 deferred Close() methods will refer to the 2 distinct Rows values and both will be properly closed because rows is a pointer, not a value type.

Ah I see, the rows always refer to the last one, http://play.golang.org/p/_xzxHnbFSz
package main
import "fmt"
type X struct {
A string
}
func (x *X) Close() {
fmt.Println(x.A)
}
func main() {
rows := X{`1`}
defer rows.Close()
rows = X{`2`}
defer rows.Close()
}
Output:
2
2
So maybe the best way to preserve the object is to pass it to a function: http://play.golang.org/p/TIMCliUn60
package main
import "fmt"
type X struct {
A string
}
func (x *X) Close() {
fmt.Println(x.A)
}
func main() {
rows := X{`1`}
defer func(r X) { r.Close() }(rows)
rows = X{`2`}
defer func(r X) { r.Close() }(rows)
}
Output:
2
1

Most of the time, you should be able to just add a block, that way you don't have to worry about thinking of a new variable name, and you don't have to worry about any of the items not being closed:
rows := Query(`SELECT FROM whatever`)
defer rows.Close()
for rows.Next() {
// do something
}
{
rows := Query(`SELECT FROM another`)
defer rows.Close()
for rows.Next() {
// do something else
}
}
https://golang.org/ref/spec#Blocks

Related

Check named return error using defer function

Hi I want to write a generic function to trace error message when a function returns error. So I wrote this:
func TraceError1(err *error) {
if err != nil && *err != nil {
pc := make([]uintptr, 15)
n := runtime.Callers(2, pc)
frames := runtime.CallersFrames(pc[:n])
frame, _ := frames.Next()
fmt.Printf("%s:%d %s\n", frame.File, frame.Line, frame.Function)
}
}
func TraceError2(err error) {
if err != nil {
pc := make([]uintptr, 15)
n := runtime.Callers(2, pc)
frames := runtime.CallersFrames(pc[:n])
frame, _ := frames.Next()
fmt.Printf("%s:%d %s\n", frame.File, frame.Line, frame.Function)
}
}
func foo() (err error) {
defer TraceError1(&err)
defer TraceError2(err)
fmt.Println("do something")
return fmt.Errorf("haha")
}
TraceError1 works but TraceError2 didn't. In my understanding, error is an interface so it is a pointer/address, why do I need to pass its address? Why TraceError2 cannot work? Thanks.
In case of TraceError1 you are passing a pointer to the named return value err. The pointer is non-nil, but the value it points at (err) is nil (at the time of defer). However, it is not yet evaluated (dereferenced) because TraceError1 has not yet been called. By the time the function does run (after foo returns) and the pointer gets dereferenced, the value of err has been updated (by the return statement inside foo).
However, in case of TraceError2, a nil interface value is passed, which will stay nil even when TraceError2 executes eventually.
Here is a simpler example:
package main
import "fmt"
func intByValue(i int) {
fmt.Printf("i = %d\n", i)
// ^--- `i` is an integer value
// --- whatever i was passed to the function, gets printed
}
func intByRef(i *int) {
var v int = *i // i is a pointer to an int, which gets dereferenced here
// the *address* where the actual value resides was passed
// while the address stays the same, its value can change before
// i is dereferenced, and its value stored in v.
fmt.Printf("i = %d\n", v)
}
func main() {
var i int
defer intByValue(i) // passed the *value* of i, which is 0 right now
defer intByRef(&i) // passed a *pointer* to i, which contains 0 right now
i = 100 // before intByRef could "dereference" its argument, the value that it
// contained has been updated
// intByRef gets called, dereferences the value, finds 100, prints it.
// intByValue gets called, finds 0, prints it
// result should be:
// i = 100
// i = 0
}
So unfortunately, if you want the ability to update the error (e.g. by returning a named return value) before it gets used by the deferred function, you are going to have to pass around pointers to the variable.
In other words, TraceError2 is simply not suited for your use case.
Edit: use correct terminology and (questionably) improve example code.
As go blog explained
The behavior of defer statements is straightforward and predictable.
There are three simple rules:
A deferred function's arguments are evaluated when the defer statement is evaluated.
Deferred function calls are executed in Last In First Out order after the surrounding function returns.
Deferred functions may read and assign to the returning function's named return values.
According to first point, when you call defer TraceError2(err) , that err = nil and that is the value pass to the TraceError2 function.
TraceError1(err *error) works because it is getting a pointer to err, and that pointer value is assigned before defer func TraceError1 is executed.
Simple example code to explain the behaviour.
package main
import (
"fmt"
"runtime"
)
func main() {
i := 0
defer func(i int) {
fmt.Printf("%d\n",i) //Output: 0
}(i)
defer func(i *int) {
defer fmt.Printf("%d\n",*i) //Output: 1
}(&i)
i++
}

Data race within Go function with named return values?

Question
Is there a non-obvious data race within the following code between qux() writing to the err at the same time when a new error baz is being returned? Is it possible that Go internal handling of the named return values treats explicit returning of a new variable as a de facto assignment to the named return value and thus allowing a data race here?
func foo(baz time.Duration) (bar *Bar, err error) {
done := make(chan struct{})
go func() {
bar, err = qux()
close(done)
}()
select {
case <-done:
return
case <-time.After(baz):
return nil, errors.New("baz")
}
}
This does not make any sense for me, as I would expect that result of errors.New() in the above code should be allocated to a different address than the pre-initialized err. However, I have reasons to suspect that this is exactly what happened to me.
Background
One of our services is executing a bit less than a half of a million HTTP requests per second over multiple pods in a cluster. Averagely once per 500 million or so requests we got an irrecoverable fault panic happening during checking of a wrapped error here in the standard library. It happens usually when a non-nil pointer points to a faulty memory address, which suggests concurrent writes.
unexpected fault address 0x0
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x80 addr=0x0 pc=0x40ebfa]
goroutine 7431988 [running]:
runtime.throw(0x1bcf57c, 0x5)
/usr/local/go/src/runtime/panic.go:1116 +0x72 fp=0xc0026e8d38 sp=0xc0026e8d08 pc=0x43bef2
runtime.sigpanic()
/usr/local/go/src/runtime/signal_unix.go:727 +0x405 fp=0xc0026e8d68 sp=0xc0026e8d38 pc=0x452805
runtime.getitab(0x195cae0, 0xe93824548b483024, 0x1, 0x7f7fc8037af8)
/usr/local/go/src/runtime/iface.go:39 +0x3a fp=0xc0026e8db8 sp=0xc0026e8d68 pc=0x40ebfa
runtime.assertI2I2(0x195cae0, 0xdcd388, 0x0, 0x7f7fc8037af8, 0xc018808a20, 0xc012bbaa01)
/usr/local/go/src/runtime/iface.go:472 +0x6a fp=0xc0026e8de8 sp=0xc0026e8db8 pc=0x41004a
errors.Is(0x1dd3f20, 0xc0120b6780, 0x1dd2a60, 0xc00003c1c0, 0xc019510c60)
/usr/local/go/src/runtime/wrap.go:49 +0xd8 fp=0xc0026e8e50 sp=0xc0026e8de8 pc=0xaacbf8
The code we run looks more or less like this:
func (c *client) Get(ctx context.Context, query Query) (Results, error) {
b, err := c.fetchAndLearn(ctx, query)
if errors.Is(err, context.Canceled) { // Panic happens on this line
// ...
}
// ...
}
func (c *client) fetchAndLearn(ctx context.Context, query Query) (response *http.Response, err error) {
done := make(chan struct{})
go func() {
response, err = c.fetch(ctx, query)
close(done)
}()
select {
case <-done:
return
case <-clock.After(c.getTimeout(query)):
return nil, xerrors.New("timeout exceeded")
}
}
After fruitless debugging I wondered if Go was doing something unintuitive under the hood due to the fact that named return values were used here. Having this faint intuition I fixed our issue with irrecoverable panics simply by removing named return values and declaring separate err variables.
func (c *client) fetchAndLearn(ctx context.Context, query Query) (*http.Response, error) {
done := make(chan struct{})
var response *http.Response
var err error
go func() {
response, err = c.fetch(ctx, query)
close(done)
}()
select {
case <-done:
if err != nil {
return nil, xerrors.Errorf(": %w", err)
}
return response, nil
case <-clock.After(c.getTimeout(query)):
return nil, xerrors.New("timeout exceeded during request duration learning")
}
}
But it still does not make that much sense for me and I would really appreciate if someone with a good understanding of Go inner workings either confirm or deny my suspicion.
Answer
Yes, explicit return in a named return function is written to preinitialized named results as per documentation.
Background
If you run following code you will notice that even though in foo() the preinitialized val is never manipulated directly after initial assignment it changes contents after val2 is returned.
package main
import (
"fmt"
"time"
)
func main() {
val := foo()
fmt.Printf("Main: %+v\n", val.Bar) // explicit
time.Sleep(2 * time.Second)
}
type Foo struct {
Bar string
}
func foo() (val *Foo) {
val = &Foo{Bar: "named"}
go func() {
time.Sleep(time.Second)
fmt.Printf("Goroutine: %+v\n", val.Bar) // explicit
}()
val2 := &Foo{Bar: "explicit"}
return val2
}
This happens because in Go a zero-valued return parameters are initialized on each function start. Then upon return any returned values are assigned to the preinitialized ones. In our example upon return val2 is assigned to val, which then is returned by the function. This opens the possibility of a data race, because even if you are assigning value to a val2 it will eventually be assigned to the val too.
My only excuse is that as of the time of writing this the official effective Go guide is slightly unintuitive as it says that named results are initialized and tied to an unadorned return which falsely presupposes that an explicit return is not tied. Credit goes to helpful guys behind go101.

Is there an efficient way to calculate execution time in golang?

I'm looking for the best way to calculate execution time in go.
func main() {
start := time.Now()
time.Sleep(time.Second * 2)
//something doing here
elapsed := time.Since(start)
fmt.Printf("page took %s", elapsed)
}
The code above works fine.
But when I use templates, I have to write it again for each template function.
Is there an efficient way of calculating execution time, including with templates?
If you are timing an entire function, then you can use defer to eliminate some of the repetitive code.
// timer returns a function that prints the name argument and
// the elapsed time between the call to timer and the call to
// the returned function. The returned function is intended to
// be used in a defer statement:
//
// defer timer("sum")()
func timer(name string) func() {
start := time.Now()
return func() {
fmt.Printf("%s took %v\n", name, time.Since(start))
}
}
func main() {
defer timer("main")() // <-- The trailing () is the deferred call
time.Sleep(time.Second * 2)
} // prints: main took 2s
Run the example on the playground.
The specification says this about deferred calls:
Each time a "defer" statement executes, the function value and parameters to the call are evaluated as usual and saved anew but the actual function is not invoked. Instead, deferred functions are invoked immediately before the surrounding function returns,
The function value timer("main") is evaluated at the defer statement. The timer function records the current time and returns an anonymous function. The returned anonymous function is invoked immediately before the surrounding function returns. The anonymous function computes and prints the elapsed time.
Use runtime.Callers and runtime.CallersFrames to automatically get the name of the calling function.
// callerName returns the name of the function skip frames up the call stack.
func callerName(skip int) string {
const unknown = "unknown"
pcs := make([]uintptr, 1)
n := runtime.Callers(skip+2, pcs)
if n < 1 {
return unknown
}
frame, _ := runtime.CallersFrames(pcs).Next()
if frame.Function == "" {
return unknown
}
return frame.Function
}
// timer returns a function that prints the name of the calling
// function and the elapsed time between the call to timer and
// the call to the returned function. The returned function is
// intended to be used in a defer statement:
//
// defer timer()()
func timer() func() {
name := callerName(1)
start := time.Now()
return func() {
fmt.Printf("%s took %v\n", name, time.Since(start))
}
}
func main() {
defer timer()()
time.Sleep(time.Second * 2)
} // prints: main.main took 2s
Note that there is a runtime cost for getting the function name compared to using a string literal as in the first part of this answer. To avoid measuring the cost of getting the function name, timer gets the name before recording the start time.
Run the example on the playground.
The solution provided by Cerise Limón is perfect.
In addition, if you don't want to pass function name explicitly, you could accomplish it like this:
func SomeFunction(list *[]string) {
defer TimeTrack(time.Now())
// Do whatever you want.
}
func TimeTrack(start time.Time) {
elapsed := time.Since(start)
// Skip this function, and fetch the PC and file for its parent.
pc, _, _, _ := runtime.Caller(1)
// Retrieve a function object this functions parent.
funcObj := runtime.FuncForPC(pc)
// Regex to extract just the function name (and not the module path).
runtimeFunc := regexp.MustCompile(`^.*\.(.*)$`)
name := runtimeFunc.ReplaceAllString(funcObj.Name(), "$1")
log.Println(fmt.Sprintf("%s took %s", name, elapsed))
}
As a result, you would get:
SomeFunction took 15.483µs
For more information, Refer this article: Go Function Tracing
Share the knowledge. :)
Use init function
package main
import (
"fmt"
"time"
)
var start time.Time
func init() {
start = time.Now()
}
func getChars(s string) {
for _, c := range s {
fmt.Printf("%c at time %v\n", c, time.Since(start))
time.Sleep(10 * time.Millisecond)
}
}
func main() {
fmt.Println("main execution started at time", time.Since(start))
getChars("Hello")
fmt.Println("\nmain execution stopped at time", time.Since(start))
}
Efficient way to calculate execution time in golang
You can easily get the execution time on your console using a defer function
defer functions execute even if the code get an error so you always get the execution time.
time package is used to get the time difference.
func main() {
now := time.Now()
defer func() {
fmt.Println(time.Now().Sub(now))
}()
// Here you can do whatever you want
}
Or you can use this code
func main() {
now := time.Now()
defer func() {
fmt.Println(time.Since(now))
}()
// Here you can do whatever you want
}
check the code in Playground for more. I added some functionality to recover from an error at the same time print the execution time, even if in the case of a panic error.
Use this as a side reference for an average computation time in Golang
(for 10,100,1000 elements accordingly).
Accessing an element within an array (1/1/1 sec)
Binary/dictionary search (1/2/3 sec)
Simple loop/linear search (10/100/1,000 sec)
Quick Sort/Merge Sort/Heap Sort/Tim Sort (10/200/3,000 sec)
Bubble Sort/Insertion Sort/Selection Sort (100/10,000/1,000,000 sec)

Data passed by reference to "defer"

There is a passage in the "Mastering Concurrency in Go" book which made me think I might be missing something about "defer" functions.
You should also take note that any data passed by reference may be in an unexpected state.
func main() {
aValue := new(int)
defer fmt.Println(*aValue)
for i := 0; i < 100; i++ {
*aValue++
}
}
This prints 0, I thought, because according to spec:
Each time a "defer" statement executes, the function value and parameters to the call are evaluated as usual and saved anew
That is, *aValue is 0 when defer is called and that's why at the end it prints 0. Whether or not a pointer is passed to the differ function in this case is irrelevant.
Is my understanding correct or am I missing something?
Consider a situation using structs.
type User struct {
Name string
}
func main() {
user := User{}
defer fmt.Printf("%#v\n", user)
user.Name = "AJ"
}
You know defer should run at the end, so you might expect to see User{Name: "AJ"} but instead you get User{Name: ""} because defer binds parameters.
If you use a pointer it works.
user := &User{}
If you use a closure, it works.
defer func() {
fmt.Printf("%#v\n", user)
}()
The defer statement is "evaluating" the parameters and saving the result, and the result of evaluating *aValue is 0 at the time of the defer call. Something like this may be what you're looking for:
func main() {
aValue := new(int)
defer func() { fmt.Println(*aValue) }()
for i := 0; i < 100; i++ {
*aValue++
}
}

Passing parameters to function closure

I'm trying to understand the difference in Go between creating an anonymous function which takes a parameter, versus having that function act as a closure. Here is an example of the difference.
With parameter:
func main() {
done := make(chan bool, 1)
go func(c chan bool) {
time.Sleep(50 * time.Millisecond)
c <- true
}(done)
<-done
}
As closure:
func main() {
done := make(chan bool, 1)
go func() {
time.Sleep(50 * time.Millisecond)
done <- true
}()
<-done
}
My question is, when is the first form better than the second? Would you ever use a parameter for this kind of thing? The only time I can see the first form being useful is when returning a func(x, y) from another function.
The difference between using a closure vs using a function parameter has to do with sharing the same variable vs getting a copy of the value. Consider these two examples below.
In the Closure all function calls will use the value stored in i. This value will most likely already reach 3 before any of the goroutines has had time to print it's value.
In the Parameter example each function call will get passed a copy of the value of i when the call was made, thus giving us the result we more likely wanted:
Closure:
for i := 0; i < 3; i++ {
go func() {
fmt.Println(i)
}()
}
Result:
3
3
3
Parameter:
for i := 0; i < 3; i++ {
go func(v int) {
fmt.Println(v)
}(i)
}
Result:
0
1
2
Playground: http://play.golang.org/p/T5rHrIKrQv
When to use parameters
Definitely the first form is preferred if you plan to change the value of the variable which you don't want to observe in the function.
This is the typical case when the anonymous function is inside a for loop and you intend to use the loop's variables, for example:
for i := 0; i < 10; i++ {
go func(i int) {
fmt.Println(i)
}(i)
}
Without passing the variable i you might observe printing 10 ten times. With passing i, you will observe numbers printed from 0 to 9.
When not to use parameters
If you don't want to change the value of the variable, it is cheaper not to pass it and thus not create another copy of it. This is especially true for large structs. Although if you later alter the code and modify the variable, you may easily forget to check its effect on the closure and get unexpected results.
Also there might be cases when you do want to observe changes made to "outer" variables, such as:
func GetRes(name string) (Res, error) {
res, err := somepack.OpenRes(name)
if err != nil {
return nil, err
}
closeres := true
defer func() {
if closeres {
res.Close()
}
}()
// Do other stuff
if err = otherStuff(); err != nil {
return nil, err // res will be closed
}
// Everything went well, return res, but
// res must not be closed, it will be the responsibility of the caller
closeres = false
return res, nil // res will not be closed
}
In this case the GetRes() is to open some resource. But before returning it other things have to be done which might also fail. If those fail, res must be closed and not returned. If everything goes well, res must not be closed and returned.
This is a example of parameter from net/Listen
package main
import (
"io"
"log"
"net"
)
func main() {
// Listen on TCP port 2000 on all available unicast and
// anycast IP addresses of the local system.
l, err := net.Listen("tcp", ":2000")
if err != nil {
log.Fatal(err)
}
defer l.Close()
for {
// Wait for a connection.
conn, err := l.Accept()
if err != nil {
log.Fatal(err)
}
// Handle the connection in a new goroutine.
// The loop then returns to accepting, so that
// multiple connections may be served concurrently.
go func(c net.Conn) {
// Echo all incoming data.
io.Copy(c, c)
// Shut down the connection.
c.Close()
}(conn)
}
}

Resources