Register multiple routes using range for loop slices/map - for-loop

Consider I have slice of string paths:
paths := []string{"/path0", "/path1", "/path2" /*... "/path-n"*/ }
// where n is the last path
Using package net/http, I want to register handler for this path using for loop with range clause. This is how I do this:
for _, path := range paths {
http.HandleFunc(path, handler)
}
// in this case every handler is print the path to the console or to the browser
EDIT: Basically the asker used this code:
for _, path := range paths {
http.HandleFunc(path, func(w http.ResponseWriter, req *http.Request) {
fmt.Fprintf(w, path)
})
}
But I ended up with same output which is the last element of slice, so when I go to /path1, the output is /path-n. Same behavior with other element, always print /path-n.
But if I use this:
http.HandleFunc(paths[0], handler)
http.HandleFunc(paths[1], handler)
http.HandleFunc(paths[2], handler)
// ...
http.HandleFunc(paths[n], handler)
The output is correct.
What's going on, did I miss something? I need for loop for registration given by slice of paths or map, so I can't do the second code.
Can you give me the alternative to accomplished this task?

So the problem was that you actually used this code:
for _, path := range paths {
http.HandleFunc(path, func(w http.ResponseWriter, req *http.Request) {
fmt.Fprintf(w, path)
})
}
You used a function literal, a closure as the handler function to register. Closures capture the context they refer to, in your case the path loop variable.
But there is only a single path loop variable, its value is overwritten in each iterations of the loop, and its final value will be the last path. Relevant section from the spec: For statements with range clause:
The iteration variables may be declared by the "range" clause using a form of short variable declaration (:=). In this case their types are set to the types of the respective iteration values and their scope is the block of the "for" statement; they are re-used in each iteration. If the iteration variables are declared outside the "for" statement, after execution their values will be those of the last iteration.
Once the for loop is finished, and you start making requests, each registered handler function will send back the value of this single path variable. That's why you see the last path returned for all requested paths.
Solution is easy: create a new variable in each iteration, and use that in the handler function:
for _, path := range paths {
path2 := path
http.HandleFunc(path2, func(w http.ResponseWriter, req *http.Request) {
fmt.Fprintf(w, path2)
})
}
What happens here is that we use a short variable declaration in each iteration to create a new variable, initialized with the value of the path loop variable. And the handler function we register will refer to this new variable, unique only to one registered path.
Another, equally good solution is to use an anonymous function with a parameter to pass the path string. Might be harder to understand though:
for _, path := range paths {
func(p string) {
http.HandleFunc(p, func(w http.ResponseWriter, req *http.Request) {
fmt.Fprintf(w, p)
})
}(path)
}
What happens here is that we call an anonymous function, passing the current path value to it, and it registers the handler function, using only the parameter of this anonymous function (and there's a new, distinct local variable allocated for each call).

Related

Parallel table - driven testing in go fails miserably

I have the following test function
func TestIntegrationAppsWithProductionSelf(t *testing.T) {
// here is where the apps array that will act as my test suite is being populated
myapps, err := RetrieveApps(fs)
for _, v := range apps {
v := v
t.Run("", func(t *testing.T) {
t.Parallel()
expectedOutput = `=` + v + `
`
cmpOpts.SingleApp = v
t.Logf("\t\tTesting %s\n", v)
buf, err := VarsCmp(output, cmpOpts)
if err != nil {
t.Fatalf("ERROR executing var comparison for %s: %s\n", v, err)
}
assert.Equal(t, expectedOutput, buf.String())
})
}
}
The test fails, despite the fact that when I remove t.Parallel() (even keeping the sub-testing structure) it succeeds.
The failure (happens as said before only when t.Parallel() is incorporated) has to do with the fact that the values to be compared passed to assert are out of sync, i.e. the assert method compares values that it shouldn't)
Why is that?
I also perform this cryptic re-assignment of the test suite variable (v := v) which I do not understand)
edit: Wandering if it was the usage of the assert method from this package, I made the following substitution, nonetheless the end result is the same,
//assert.Equal(t, expectedOutput, buf.String())
if expectedOutput != buf.String() {
t.Errorf("Failed! Expected %s - Actual: %s\n", expectedOutput, buf.String())
}
Let's dissect the case.
First, let's refer to the docs on testing.T.Run:
Run runs f as a subtest of t called name.
It runs f in a separate goroutine <…>
(Emphasis mine.)
So, when you call t.Run("some_name", someFn), that SomeFn is being run by the test suite as if you would manually do something like
go someFn(t)
Next, let's notice that you do not pass a named function into your call to t.Run, but rather you pass it a so-called function literal; let's cite the language spec on them:
Function literals are closures: they may refer to variables defined in a surrounding function. Those variables are then shared between the surrounding function and the function literal, and they survive as long as they are accessible.
In your case, it means when the compiler compiles the body of your function literal, it makes the function "close over" any variable its body mentions, and which is not one of the formal function parameters; in your case, the only function parameter is t *testing.T, hence every other accessed variable is captured by the created closure.
In Go, when a function literal closes over a variable, it does so by retaining a reference to that variable — which is explicitly mentioned in the spec as («Those variables are then shared between the surrounding function and the function literal <…>», again, emphasis mine.)
Now notice that loops in Go reuse iteration variables on each iteration; that is, when you write
for _, v := range apps {
that variable v is created once in the "outer" scope of the loop and then gets reassigned on each iteration of the loop. To recap: the same variable, whose storage is located at some fixed point in memory, gets assigned a new value on each iteration.
Now, since a function literal closes over external variables by keeping references to them — as opposed to copying their values at the "time" of its definition into itself, — without that funky-looking v := v "trick" each function literal created at each call to t.Run in your loop would reference exactly the same iteration variable v of the loop.
The v := v construct declares another variable named v which is local to the loop's body and at the same time assigns it the value of the loop iteration variable v. Since the local v "shadows" loop iterator's v, the function literal declared afterwards would close over that local variable, and hence each function literal created on each iteration will close over a distinct, separate variable v.
Why is this needed, you may ask?
This is needed because of a subtle problem with the interaction of loop iteration variable and goroutines, which is detailed on the Go wiki:
when one does something like
for _, v := range apps {
go func() {
// use v
}()
}
A function literal closing over v is created, and then it's run with the go statement—in parallel both with the goroutine which runs the loop and with all the other goroutines started on the len(apps)-1 other iterations.
These goroutines running our function literals all refer to the same v and so they all have a data race over that variable: the goroutine running the looop writes to it, and the goroutines running function literals read from it—concurrently and without any synchronization.
I hope, by now you should see the puzzle's pieces coming together: in the code
for _, v := range apps {
v := v
t.Run("", func(t *testing.T) {
expectedOutput = `=` + v + `
// ...
the function literal passed to t.Run closes over v, expectedOutput,
cmpOpts.SingleApp (and may be something else),
and then t.Run() makes that function literal run in a separate goroutine, as documented,—producing the classic data race on expectedOutput and cmpOpts.SingleApp, and whatever else which is not v (a fresh variable on each iteration) or t (passed to the call of the function literal).
You might run go test -race -run=TestIntegrationAppsWithProductionSelf ./... to see the engaged race detector crashing your test case's code.
I am going to post what actually worked, but (unless the question is closed) I will accept the answer that actually elaborates on it.
The problem was that the variable used to store the expectedOutput was declared with a declaration inside the TestIntegrationAppsWithProductionSelf function but outside the for loop (this is now reflected in the code snippet of the initial question).
What worked was to remove the var expectedOutput string statement and do within the for loop
for _, v := range apps {
v := v
expectedOutput := `=` + v + `
`
t.Run("", func(t *testing.T) {
t.Parallel()
cmpOpts.SingleApp = v
t.Logf("\t\tTesting %s\n", v)
buf, err := VarsCmp(output, cmpOpts)
if err != nil {
t.Fatalf("ERROR executing var comparison for %s: %s\n", v, err)
}
//assert.Equal(t, expectedOutput, buf.String())
if expectedOutput != buf.String() {
t.Errorf("Failed! Expected %s - Actual: %s\n", expectedOutput, buf.String())
}
})
}

Using Pointers in a for loop

I'm struggling to understand why I have a bug in my code in one state but not the other. It's been a while since I've covered pointers, so I'm probably rusty!
Basically I have a repository structure I'm using to store an object in memory, that has a Store function.
type chartsRepository struct {
mtx sync.RWMutex
charts map[ChartName]*Chart
}
func (r *chartsRepository) Store(c *Chart) error {
r.mtx.Lock()
defer r.mtx.Unlock()
r.charts[c.Name] = c
return nil
}
So all it does is put a RW mutex lock on and adds the pointer to a map, referenced by an identifier.
Then I've got a function that will basically loop through a slice of these objects, storing them all in the repository.
type service struct {
charts Repository
}
func (svc *service) StoreCharts(arr []Chart) error {
hasError := false
for _, chart := range arr {
err := svc.repo.Store(&chart)
// ... error handling
}
if hasError {
// ... Deals with the error object
return me
}
return nil
}
The above doesn't work, it looks like everything works fine at first, but on trying to access the data later, the entries in the map all point to the same Chart object, despite having different keys.
If I do the following and move the pointer reference to another function, everything works as expected:
func (svc *service) StoreCharts(arr []Chart) error {
// ...
for _, chart := range arr {
err := svc.storeChart(chart)
}
// ...
}
func (svc *service) storeChart(c Chart) error {
return svc.charts.Store(&c)
}
I'm assuming the issue is that because the loop overwrites the reference to the chart in the for loop, the pointer reference also changes. When the pointer is generated in an independent function, that reference is never overwritten. Is that right?
I feel like I'm being stupid, but shouldn't the pointer be generated by &chart and that's independent of the chart reference? I also tried creating a new variable for the pointer p := &chart in the for loop and that didn't work either.
Should I just avoid generating pointers in loops?
This is because there is only a single loop variable chart, and in each iteration just a new value is assigned to it. So if you attempt to take the address of the loop variable, it will be the same in each iteration, so you will store the same pointer, and the pointed object (the loop variable) is overwritten in each iteration (and after the loop it will hold the value assigned in the last iteration).
This is mentioned in Spec: For statements: For statements with range clause:
The iteration variables may be declared by the "range" clause using a form of short variable declaration (:=). In this case their types are set to the types of the respective iteration values and their scope is the block of the "for" statement; they are re-used in each iteration. If the iteration variables are declared outside the "for" statement, after execution their values will be those of the last iteration.
Your second version works, because you pass the loop variable to a function, so a copy will be made of it, and then you store the address of the copy (which is detached from the loop variable).
You can achieve the same effect without a function though: just create a local copy and use the address of that:
for _, chart := range arr {
chart2 := chart
err := svc.repo.Store(&chart2) // Address of the local var
// ... error handling
}
Also note that you may also store the address of the slice elements:
for i := range arr {
err := svc.repo.Store(&arr[i]) // Address of the slice element
// ... error handling
}
The disadvantage of this is that since you store pointers to the slice elements, the whole backing array of the slice would have to be kept in memory for as long as you keep any of the pointers (the array cannot be garbage collected). Moreover, the pointers you store would share the same Chart values as the slice, so if someone would modify a chart value of the passed slice, that would effect the charts whose pointers you stored.
See related questions:
Golang: Register multiple routes using range for loop slices/map
Why do these two for loop variations give me different behavior?
I faced a similar issue today and creating this simple example helped me understand the problem.
// Input array of string values
inputList := []string {"1", "2", "3"}
// instantiate empty list
outputList := make([]*string, 0)
for _, value := range inputList {
// print memory address on each iteration
fmt.Printf("address of %v: %v\n", value, &value)
outputList = append(outputList, &value)
}
// show memory address of all variables
fmt.Printf("%v", outputList)
This printed out:
address of 1: 0xc00008e1e0
address of 2: 0xc00008e1e0
address of 3: 0xc00008e1e0
[0xc00008e1e0 0xc00008e1e0 0xc00008e1e0]
As you can see, the address of value in each iteration was always the same even though the actual value was different ("1", "2", and "3"). This is because value was getting reassigned.
In the end, every value in the outputList was pointing to the same address which is now storing the value "3".

How to understand this behavior of goroutine?

package main
import (
"fmt"
"time"
)
type field struct {
name string
}
func (p *field) print() {
fmt.Println(p.name)
}
func main() {
data := []field{ {"one"},{"two"},{"three"} }
for _,v := range data {
go v.print()
}
<-time.After(1 * time.Second)
}
why does this code print 3 "three" instead of "one" "two" "three" in any order?
There is a data race.
The code implicitly takes address of variable v when evaluating arguments to the goroutine function. Note that the call v.print() is shorthand for the call (&v).print().
The loop changes the value of variable v.
When goroutines execute, it so happens that v has the last value of the loop. That's not guaranteed. It could execute as you expected.
It's helpful and easy to run programs with the race detector. This data race is detected and reported by the detector.
One fix is to create another variable scoped to the inside of the loop:
for _, v := range data {
v := v // short variable declaration of new variable `v`.
go v.print()
}
With this change, the address of the inner variable v is taken when evaluating the arguments to the goroutine. There is a unique inner variable v for each iteration of the loop.
Yet another way to fix the problem is use a slice of pointers:
data := []*field{ {"one"},{"two"},{"three"} } // note '*'
for _, v := range data {
go v.print()
}
With this change, the individual pointers in the slice are passed to the goroutine, not the address of the range variable v.
Another fix is to use the address of the slice element:
data := []field{ {"one"},{"two"},{"three"} } // note '*'
for i:= range data {
v := &data[i]
go v.print()
}
Because pointer values are typically used with types having a pointer receiver, this subtle issue does not come up often in practice. Because field has a pointer receiver, it would be typical to use []*field instead of []field for the type of data in the question.
If the goroutine function is in an anonymous function, then a common approach for avoiding the issue is to pass the range variables as an argument to the anonymous function:
for _, v := range data {
go func(v field) {
v.print() // take address of argument v, not range variable v.
}(v)
}
Because the code in the question does not already use an anonymous function for the goroutine, the first approach used in this answer is simpler.
As stated above there’s a race condition it’s result depends on delays on different processes and not well defined and predictable.
For example if you add time.Sleep(1*time.Seconds) you likely to get a correct result. Because usually goroutine prints faster than 1second and will have correct variable v but it’s a very bad way.
Golang has a special race detector tool which helps to find such situations. I recommend read about it while reading testing. Definitely it’s worth it.
There’s another way - explicitly pass variable value at goroutine start:
for _, v := range data {
go func(iv field) {
iv.print()
}(v)
}
Here v will be copied to iv (“internal v”) on every iteration and each goroutine will use correct value.

Meaning of underscore in a Go function parameter

Came accross the below function here. I noticed the last parameter is identified with _. What is the intent of this pattern?
func Index(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
fmt.Fprint(w, "Welcome!\n")
}
It means "ignore that parameter", the reason that they still need the last parameter here is because they want to pass it as type Handle to the function GET, which has the signature:
type Handle func(http.ResponseWriter, *http.Request, Params)
If you simply pass something like func Index(w http.ResponseWriter, r *http.Request) it will not be treated as type Handle.
_ is the blank identifier. It's in the signature to show that the value doesn't get used, so the signature will still match the methods of the interface.
Using "_" in place of a parameter name fulfills the obligations of a higher-level "function as a parameter" without getting a warning about an unused parameter. In your case, I believe the compiler is told to ignore all incoming "POST" data, thus in effect reducing the request to the functionality of a "GET".
As others pointed out, it is a blank identifier. For instance, consider the following example:
func main() {
nums := []int{5, 3, 4}
max := nums[0]
for _, num := range nums {
if num > max {
max = num
}
}
fmt.Println("max:", max)
}
If you aren't going to use an index value, you can just ignore storing it by using _ instead of a variable name.=.

Key into map with undefined integer

I'm using Gorilla mux for my handlers and using mux.Vars. I'm trying to write a test for one of the handlers that uses mux.Vars so what I do is
var vars = map[string]string{
"id": user.ID,
}
context.Set(req, 0, vars)
In mux the key (an integer) is undefined so by default 0. I've logged the key when mux.Vars gets called and it prints 0. I should be able to key into this map
map[0:map[id:522d14f5b1b92235d6000002]]
by doing map[key] but that returns nil. However, I get the correct value back if I hardcode map[0]. Any thoughts?
I'm not entirely sure I understand the question, but it looks like you might be confusing mux.Vars with mux.context. The two are separate entities. The former returns route variables that are parsed from the URL path. For instance, you could do:
r := mux.NewRouter()
r.HandleFunc("/blah/{foo}/", MyHandler)
...
func MyHandler(w http.ResponseWriter, r *http.Request) {
vars := mux.Vars(r)
...
}
The latter contains context variables you set yourself. For instance:
func MyHandler(w http.ResponseWriter, r *http.Request) {
context.Set(r, 0, map[string]string{"id": "myid"})
myMap := context.Get(r, 0)
...
}
You might check out some usage examples of how others use both to see what is most appropriate for your use case:
mux.Vars: https://sourcegraph.com/github.com/gorilla/mux/symbols/go/github.com/gorilla/mux/Vars
mux.context: https://sourcegraph.com/github.com/gorilla/context/symbols/go/github.com/gorilla/context

Resources