I'm creating a priority queue using Go's heap package. There is an example of one in the documentation.
The queue I'm creating needs to be based around a struct rather than a slice because it requires other properties like a mutex.
type PQueue struct {
queue []*Item
sync.Mutex
}
I implement all the methods that heap.Interface requires.
The issue is that my PQueue.Push method seems not to be permanently adding a value to PQueue.queue.
func (p PQueue) Push(x interface{}) {
p.Lock()
defer p.Unlock()
item := x.(*Item)
item.place = len(p.queue) // the index of an item in the queue
p.queue = append(p.queue, item)
// len(p.queue) does increase
// after the functions exits, the queues length has not increased
}
If I print the length of p.queue at the end of this function, the length has increased. After the functions exits however, it seems the original struct does not get updated.
I think it might be happening because of func (p PQueue) not being a pointer. Why might that be? Is there a way to fix it? If I were to use func (p *PQeueue) Push(x interface{}) instead, I would need to implement my own heap because heap.Interface specifically requires no pointer. Is that my only option?
The problem is that you are appending to a copy of your slice. Thus the change shows within the function, but is lost once you return from the function.
In this blog article from the section Passing slices to functions:
It's important to understand that even though a slice contains a
pointer, it is itself a value. Under the covers, it is a struct value
holding a pointer and a length. It is not a pointer to a struct.
With append you are modifying the slice header. And
Thus if we want to write a function that modifies the header, we must
return it as a result parameter
Or:
Another way to have a function modify the slice header is to pass a
pointer to it.
As a result you need to pass a pointer if you want to modify it with append. Simply change the method to use a pointer receiver. And for that to work you need to call init with a pointer like heap.Init(&pq) as shown in the example that you linked to which does just that and also uses pointer receivers.
From the spec on Method Sets:
The method set of the corresponding pointer type *T is the set of all methods
declared with receiver *T or T (that is, it also contains the method
set of T).
So using a pointer type will work with value and pointer receivers and still implement the interface.
You are right about the problem being related to the receiver of your Push method: the method will receive a copy of the PQueue, so any changes made to the struct will not persist.
Changing the method to use a pointer as a receiver is the correct change, but this also means that PQueue no longer implements heap.Interface. This is due to the fact that Go does not let you take a pointer to the value stored inside an interface variable, so the automatic translation of q.Push() to (&q).Push() does not occur.
This isn't a dead end though, since *PQueue should still implement the heap.Interface. So if you were previously calling heap.Init(q), just change it to heap.Init(&q).
I think it might be happening because of func (p PQueue) not being a pointer
That's right. Quoting Effective Go:
invoking [the method] on a value would cause the method to receive a
copy of the value, so any modifications would be discarded.
You say:
heap.Interface specifically requires no pointer
I'm confused, the example you point to is, in fact, using a pointer:
func (pq *PriorityQueue) Push(x interface{}) {
n := len(*pq)
item := x.(*Item)
item.index = n
*pq = append(*pq, item)
}
Maybe something else is going on?
Related
The Go FAQ answers a question regarding the choice of by-value vs. by-pointer receiver definition in methods. One of the statements in that answer is:
If some of the methods of the type must have pointer receivers, the rest should too, so the method set is consistent regardless of how the type is used.
This implies that if I have for a data type a few methods that mutate the data, thus require by-pointer receiver, I should use by-pointer receiver for all the methods defined for that data type.
On the other hand, the "fmt" package invokes the String() method as defined in the Stringer interface by value. If one defines the String() method with a receiver by-pointer it would not be invoked when the associated data type is given as a parameter to fmt.Println (or other fmt formatting methods). This leaves one no choice but to implement the String() method with a receiver by value.
How can one be consistent with the choice of by-value vs. by-pointer, as the FAQ suggests, while fulfilling fmt requirements for the Stringer interface?
EDIT:
In order to emphasize the essence of the problem I mention, consider a case where one has a data type with a set of methods defined with receiver by-value (including String()). Then one wishes to add an additional method that mutates that data type - so he defines it with receiver by-pointer, and (in order to be consistent, per FAQ answer) he also updates all the other methods of that data type to use by-pointer receiver. This change has zero impact on any code that uses the methods of this data type - BUT for invocations of fmt formatting functions (that now require passing a pointer to a variable instead of its value, as before the change). So consistency requirements are only problematic in the context of fmt. The need to adjust the manner one provides a variable to fmt.Println (or similar function) based on the receiver type breaks the capability to easily refactor one's package.
If you define your methods with pointer receiver, then you should use and pass pointer values and not non-pointer values. Doing so the passed value does indeed implement Stringer, and the fmt package will have no problem "detecting" and calling your String() method.
Example:
type Person struct {
Name string
}
func (p *Person) String() string {
return fmt.Sprintf("Person[%s]", p.Name)
}
func main() {
p := &Person{Name: "Bob"}
fmt.Println(p)
}
Output (try it on the Go Playground):
Person[Bob]
If you would pass a value of type Person to fmt.Println() instead of a pointer of type *Person, yes, indeed the Person.String() would not be called. But if all methods of Person has pointer receiver, that's a strong indication that you should use the type and its values as pointers (unless you don't intend its methods to be used).
Yes, you have to know whether you have to use Person or *Person. Deal with it. If you want to write correct and efficient programs, you have to know a lot more than just whether to use pointer or non-pointer values, I don't know why this is a big deal for you. Look it up if you don't know, and if you're lazy, use a pointer as the method set of (the type of) a pointer value contains methods with both pointer and non-pointer receiver.
Also the author of Person may provide you a NewPerson() factory function which you can rely on to return the value of the correct type (e.g. Person if methods have value receivers, and *Person if the methods have pointer receivers), and so you won't have to know which to use.
Answer to later adding a method with pointer receiver to a type which previously only had methods with value receiver:
Yes, as you described in the question, that might not break existing code, yet continuing to use a non-pointer value may not profit from the later added method with pointer receiver.
We might ask: is this a problem? When the type was used, the new method you just added didn't exist. So the original code made no assumption about its existence. So it shouldn't be a problem.
Second consideration: the type only had methods with value receiver, so one could easily assume that by their use, the value was immutable as methods with value receiver cannot alter the value. Code that used the type may have built on this, assuming it was not changed by its methods, so using it from multiple goroutines may have omitted certain synchronization rightfully.
So I do think that adding a new method with pointer receiver to a type that previously only had methods with value receiver should not be "opaque", the person who adds this new method has the responsibility to either modify uses of this type to "switch" to pointers and make sure the code remains safe and correct, or deal with the fact that non-pointer values will not have this new method.
Tips:
If there's a chance that a type may have mutator methods in the future, you should start creating it with methods with pointer receivers. Doing so you avoid later having to go through the process described above.
Another tip could be to hide the type entirely, and only publish interfaces. Doing so, the users of this type don't have to know whether the interface wraps a pointer or not, it just doesn't matter. They receive an interface value, and they call methods of the interface. It's the responsibility of the package author to take care of proper method receivers, and return the appropriate type that implements the interface. The clients don't see this and they don't depend on this. All they see and use is the interface.
In order to emphasize the essence of the problem I mention, consider a case where one has a data type with a set of methods defined with receiver by-value (including String()). Then one wishes to add an additional method that mutates that data type - so he defines it with receiver by-pointer, and (in order to be consistent, per FAQ answer) he also updates all the other methods of that data type to use by-pointer receiver. This change has zero impact on any code that uses the methods of this data type - BUT for invocations of fmt formatting functions (that now require passing a pointer to a variable instead of its value, as before the change).
This is not true. All interface of it and some of type assertions will be affected as well - that is why fmt is affected. e.g. :
package main
import (
"fmt"
)
type I interface {
String() string
}
func (t t) String() string { return "" }
func (p *p) String() string { return "" }
type t struct{}
type p struct{}
func S(i I) {}
func main() {
fmt.Println("Hello, playground")
T := t{}
P := p{}
_ = P
S(T)
//S(P) //fail
}
To understand this from the root, you should know that a pointer method and a value method is different from the very base. However, for convenience, like the omit of ;, golang compiler looks for cases using pointer methods without a pointer and change it back.
As explained here: https://tour.golang.org/methods/6
So back to the orignal question: consistency of pointer methods. If you read the faq more carefully, you will find it is the very last part of considering to use a value or pointer methods. And you can find counter-example in standard lib examples, in container/heap :
// A PriorityQueue implements heap.Interface and holds Items.
type PriorityQueue []*Item
func (pq PriorityQueue) Len() int { return len(pq) }
func (pq PriorityQueue) Less(i, j int) bool {
// We want Pop to give us the highest, not lowest, priority so we use greater than here.
return pq[i].priority > pq[j].priority
}
func (pq PriorityQueue) Swap(i, j int) {
pq[i], pq[j] = pq[j], pq[i]
pq[i].index = i
pq[j].index = j
}
func (pq *PriorityQueue) Push(x interface{}) {
n := len(*pq)
item := x.(*Item)
item.index = n
*pq = append(*pq, item)
}
func (pq *PriorityQueue) Pop() interface{} {
old := *pq
n := len(old)
item := old[n-1]
item.index = -1 // for safety
*pq = old[0 : n-1]
return item
}
// update modifies the priority and value of an Item in the queue.
func (pq *PriorityQueue) update(item *Item, value string, priority int) {
item.value = value
item.priority = priority
heap.Fix(pq, item.index)
}
So, indeed, as the FAQ say, to determine whether to use a pointer methods, take the following consideration in order:
Does the method need to modify the receiver? If yes, use a pointer. If not, there should be a good reason or it makes confusion.
Efficiency. If the receiver is large, a big struct for instance, it will be much cheaper to use a pointer receiver. However, efficiency is not easy to discuss. If you think it is an issue, profile and/or benchmark it before doint so.
Consistency. If some of the methods of the type must have pointer receivers, the rest should too, so the method set is consistent regardless of how the type is used. This, to me, means that if the type shall be used as a pointer (e.g., frequent modify), it should use the method set to mark so. It does not mean one type can only have solely pointer methods or the other way around.
The previous answers here do not address the underlying issue, although the answer from leaf bebop is solid advice.
Given a value, you can in fact invoke either pointer or value receiver methods on it, the compiler will do that for you. However, that does not apply when invoking via interface implementations.
This boils down to this dicussion about interface implementations.
In that discussion the discussion is about implementing interfaces with nil pointers. But the underlying discussion revolves around the same issue: when implementing an interface you must choose the pointer or the value type, and there will be no attempt by the compiler, nor can there be any attempt in golang code, to figure out exactly what type it is, and adjust the interface call accordingly.
So for example, when calling
fmt.Println(object)
you are implementing the arg of type interface{} with object of type X. The fmt code within has no interest in knowing whether the type of object is a pointer type or not. It will not even be able to tell without using reflection. It will simply call String() on whatever type that is.
So if you supplied a value of type X, and there just so happens to be a (*X) String() string method, that does not matter, that method will not be called, it will only type-assert whether that type X implements Stringer, it has no interest if type *X asserts Stringer. Since there is no (X) String() string method, it will move on. It will not attempt to check what X may happen to be, whether it's a pointer type, and if not, whether the associated pointer type implements Stringer, and call that String() method instead.
So this is not really a pointer vs value methods issue, this is an interface implementation issue when implementing interface{} in calls to fmt methods.
What is the best way to "build" an object.
Leme write some code:
type Car struct {
Wheels int
Doors int
}
This cars are stored somewhere, somehow. So should my interface be the type of
func (s Store) GetCar() *Car
or should I make it
func (s Store) GetCar(*Car)
and pass a reference to a variable?
Im looking for some sort of rule of thumb.
Thanks!
Go manages the heap/stack, keeping track when the reference goes outside of scope. So, you can return the pointer without any worries.
func (s *Store) GetCar() *Car {
return &Car{Store: s}
}
In many cases this is preferable
func (s Store) GetCar() *Car
because it is more convenient and readable, but has consequences. All variables such as Car are created inside the function which means they are placed onto stack. After function return this memory of stack is marked as invalid and can be used again. It a bit differs for pointer values such as *Car. Because pointer is virtually means you want to share the value with other scope and return an address, the value has to be stored somewhere in order to be available for calling function. It is copied onto heap memory and stays there until garbage collection finds no references to it.
It implies overheads:
copying values from stack to heap
additional work for garbage collection
The overheads is not significant if the value is relatively small. This is a reason why we have to pass an argument in io.Reader and io.Writer rather than have the value in return.
If you'd like to dive yourself into guts follow the links:
Language Mechanics On Stacks And Pointers
and
Bad Go: pointer returns
The most common way to do that would be to write it as:
func (s Store) GetCar() *Car
Or, if you don't want to use pointers, you can do it like:
func (s Store) GetCar() Car
The other alternative, making it GetCar(aCar *Car) might work, but it will not be as clear since it's not obvious that aCar should be sent empty and then populated by the function.
Create them on heap from beginning by new.
Let's say we have this kind of a struct (one of the simplest ever):
type some struct{
I uint32
}
And we want to have a variable of that type and to atomically increment in for loop (possibly in another goroutine but now the story is different). I do the following:
q := some{0}
for i := 0; i < 10; i++ {
atomic.AddUint32(&q.I,1) // increment [1]
fmt.Println(q.I)
}
We're getting what we'd expect, so far so good, but if we declare a function for that type as follows:
func (sm some) Add1(){
atomic.AddUint32(&sm.I,1)
}
and call this function in the above sample (line [1]) the value isn't incremented and we just get zeros. The question is obvious - why?
This has to be something basic but since I am new to go I don't realize it.
The Go Programming Language Specification
Calls
In a function call, the function value and arguments are evaluated in
the usual order. After they are evaluated, the parameters of the call
are passed by value to the function and the called function begins
execution. The return parameters of the function are passed by value
back to the calling function when the function returns.
The receiver sm some is passed by value to the method and the copy is discarded when you return from the method. Use a pointer receiver.
For example,
package main
import (
"fmt"
"sync/atomic"
)
type some struct {
I uint32
}
func (sm *some) Add1() {
atomic.AddUint32(&sm.I, 1)
}
func main() {
var s some
s.Add1()
fmt.Println(s)
}
Output:
{1}
Go Frequently Asked Questions (FAQ)
When are function parameters passed by value?
As in all languages in the C family, everything in Go is passed by
value. That is, a function always gets a copy of the thing being
passed, as if there were an assignment statement assigning the value
to the parameter. For instance, passing an int value to a function
makes a copy of the int, and passing a pointer value makes a copy of
the pointer, but not the data it points to.
Should I define methods on values or pointers?
func (s *MyStruct) pointerMethod() { } // method on pointer
func (s MyStruct) valueMethod() { } // method on value
For programmers unaccustomed to pointers, the distinction between
these two examples can be confusing, but the situation is actually
very simple. When defining a method on a type, the receiver (s in the
above examples) behaves exactly as if it were an argument to the
method. Whether to define the receiver as a value or as a pointer is
the same question, then, as whether a function argument should be a
value or a pointer. There are several considerations.
First, and most important, does the method need to modify the
receiver? If it does, the receiver must be a pointer. (Slices and maps
act as references, so their story is a little more subtle, but for
instance to change the length of a slice in a method the receiver must
still be a pointer.) In the examples above, if pointerMethod modifies
the fields of s, the caller will see those changes, but valueMethod is
called with a copy of the caller's argument (that's the definition of
passing a value), so changes it makes will be invisible to the caller.
By the way, pointer receivers are identical to the situation in Java,
although in Java the pointers are hidden under the covers; it's Go's
value receivers that are unusual.
Second is the consideration of efficiency. If the receiver is large, a
big struct for instance, it will be much cheaper to use a pointer
receiver.
Next is consistency. If some of the methods of the type must have
pointer receivers, the rest should too, so the method set is
consistent regardless of how the type is used. See the section on
method sets for details.
For types such as basic types, slices, and small structs, a value
receiver is very cheap so unless the semantics of the method requires
a pointer, a value receiver is efficient and clear.
Your function need to receive a pointer for the value to be incremented, that way you are not passing a copy of the struct and on next iteration the I can be incremented.
package main
import (
"sync/atomic"
"fmt"
)
type some struct{
I uint32
}
func main() {
q := &some{0}
for i := 0; i < 10; i++ {
q.Add1()
fmt.Println(q.I)
}
}
func (sm *some) Add1(){
atomic.AddUint32(&sm.I,1)
}
I was writing a program that reads data from a io.Reader and caches them in a bytes.Buffer.
type SecureReader struct {
pipe io.Reader
shared *[32]byte
decrypted bytes.Buffer
}
func (s SecureReader) Read(b []byte) (int, error) {
s.decryptPipeIntoBuffer()
return s.decrypted.Read(b)
}
func (s SecureReader) decryptPipeIntoBuffer() (int, error) {/*Lots of code...*/}
I first used a value receiver, because I thought they were the same. However, I noticed my method does not do anything when invoked: SecureReader.Read() would always return io.EOF.
I banged my head around and changed the receiver type to
func (s *SecureReader) decryptPipeIntoBuffer() (int, error) {/*Lots of code...*/}
Now my code magically works. What is going on?
A value receiver operates on a copy of the SecureReader instance s.
If the method mutates any part of the copy of the instance (like modify s.decrypted), it is not visible on the original instance of the receiver, once the method exit.
That changes with a pointer receiver, where the method operates and can mutates the actual SecureReader instance s, since a copy of the pointer is passed to the method.
See more examples in "Don't Get Bitten by Pointer vs Non-Pointer Method Receivers in Golang".
Simply stated: you can treat the receiver as if it was an argument being passed to the method. All the same reasons why you might want to pass by value or pass by reference apply.
Reasons why you would want to pass by reference as opposed to by value:
You want to actually modify the receiver (“read/write” as opposed to just “read”)
The struct is very large and a deep copy is expensive
Consistency: if some of the methods on the struct have pointer receivers, the rest should too. This allows predictability of behavior
If you need these characteristics on your method call, use a pointer receiver.
I've read a Tour of Go and Effective Go, http://golang.org/doc/effective_go.html#pointers_vs_values, but still have a difficult time understanding when you would define a method on a struct using a value method receiver instead of a pointer method receiver. In other words, when would this:
type ByteSlice []byte
func (slice ByteSlice) Append(data []byte) []byte {
}
be preferable over this?
func (p *ByteSlice) Append(data []byte) {
slice := *p
*p = slice
}
Slices are one place where it's not always obvious at first. The Slice header is small, so copying it is cheap, and the underlying array is referenced via a pointer, so you can manipulate the contents of a slice with a value receiver. You can see this in the sort package, where the methods for the sortable types are defined without pointers.
The only time you need to use a pointer with a slice, is if you're going to manipulate the slice header, which means changing the length or capacity. For an Append method, you would want:
func (p *ByteSlice) Append(data []byte) {
*p = append(*p, data...)
}
There is an FAQ entry on that matter:
Should I define methods on values or pointers?
First, and most important, does the method need to modify the receiver? If it does, the receiver must be a pointer.
...
Second is the consideration of efficiency. If the receiver is large, a big struct for instance, it will be much cheaper to use a pointer receiver.