I often see a type like (int, int) -<cloptr1> bool. What does it really mean?
How does it compare to a type like (int, int) -<cloref1> bool?
The -<cloptr1> part of the type involves the decorated arrow notation to signify effects for the given function.
See this ATS Wiki page for a list of available effects; cloptr1 means that the function is a linear closure that must be explicitly freed (and thus does not require garbage collection to be enabled, in contrast to the cloref1-denoted closure functions). See the chapter on linear closures from A Tutorial on Programming Features in ATS for the detailed description and explanation.
Related
I will clarify the question a bit. I have read (almost completely) the Go specification, FAQ, Effective Go, and, of course, Tour of Go.
I know that Go is a "pass by value" language and even managed to reason about this behavior and understand all the implications.
All assignments in Go also create copies. In some cases, it's just a value, in some -- a pointer. For some data structures, it's a bit trickier in that the whole structure is copied and might include an implicit pointer to another data structure.
The question is: what in the language specification says explicitly that assignments always create copies?
I feel like it doesn't even need to be mentioned once you understand that there are no references in Go, but the section on assignment statements in the specification doesn't even mention the pass-by-value semantics.
I feel like there must be something in the documentation that describes the behavior in detail, and I, due to lack of some foundational misunderstanding, fail to realize the explanation is there.
What in the language specification says explicitly that assignment always creates copies?
Nothing explicit, but you can maybe deduce this from Variables, which nicely addresses also the case of function signatures:
A variable declaration or, for function parameters and results, the signature of a function declaration or function literal reserves storage for a named variable.
If storage is reserved, when later you assign the result of a unary expression to it — e.g. another variable —, then it must be a copy, otherwise you would have memory aliasing. Which is what Dave Cheney is talking about in There is no pass-by-reference in Go.
Unlike C++, each variable defined in a Go program occupies a unique memory location.
This also has one more important implication, which is the zero value. If you don't provide an expression to initialize a variable in its declaration, the storage reserved for it is given the zero value as default value.
Without going into too much detail, these excerpts from the spec should provide some clarity:
A variable is a storage location for holding a value.
A variable's value [...] is the most recent value assigned to the variable.
At the language level, defining "copying" of values isn't really necessitated. The important implication of copying as we commonly understand it, is that modifying the "copied to" value will not alter the "copied from" value *. This property can be inferred from the above quotations.
(important to note here that "value" and "data structure" are not the same thing. A data structure may be comprised of multiple values.)
The spec actually explicitly talks about this here:
https://golang.org/ref/spec#Calls
In particular:
After they are evaluated, the parameters of the call are passed by value to the function and the called function begins execution. The return parameters of the function are passed by value back to the caller when the function returns.
About assignments: All assignments in most of the languages I know of create copies of values (see Python exception in the comments). A copy of the value on the RHS is assigned to the LHS. If the RHS is a pointer, then a copy of that pointer is assigned to the LHS.
Go uses escape analysis and garbage collection to manage memory allocation on stack and heap. Go's FAQ also says:
How do I know whether a variable is allocated on the heap or the stack?
From a correctness standpoint, you don't need to know. Each variable in Go exists as long as there are references to it. The storage location chosen by the implementation is irrelevant to the semantics of the language.
So Go allocates a memory for a variable, and reserves it at least until it's needed.
My question is: Is this (abstract) behavior written in The Go Programming Language Specification? I found the allocation part is written, for example, in Allocation section:
The built-in function new takes a type T, allocates storage for a variable of that type at run time, and returns a value of type *T pointing to it.
But is there any description of the reservation part? Can we confirm the fact "Each variable in Go exists as long as there are references to it"? If not, is there any reasons?
For example, I want to confirm the following program must not throw SIGSEGV or similar exceptions if a Go compiler has no bugs.
func foo() *int {
x := 42
return &x
}
func main() {
px := foo()
fmt.Println(*px)
}
To be more precise, I expected that the two part, "Go allocates memory when new or something" and "Go reserves the allocated memory at least it's needed", should be written in the specification. I don't care about its implementation details, even though https://github.com/golang/go uses escape analysis and garbage collection.
If the latter part does not exist, then in an extreme case, it's valid implementation according to the spec that the memory is un-allocated immediately after it is allocated. But this is ridiculous, so I think the spec should invalidate that.
Edit for close: I don't think this question is opinion-based. This question is a simple yes/no-question, asking for the description in the specification. The reason for the existence/non-existence can be answered with citations. If not, please show/comment which points are opinion-based. I'll improve that.
The specification uses the term variable for storage location. The specification does not distinguish between storage locations on the heap or the stack. The terms heap and stack are absent from specification.
The section on variables says:
A variable's value is retrieved by referring to the variable in an expression; it is the most recent value assigned to the variable. If a variable has not yet been assigned a value, its value is the zero value for its type.
If a variable can be referenced, then the variable's value can be retrieved. The compiler and runtime must retain a variable's value when there are extant references to the variable.
But is there any description of the preservation part? Can we confirm the fact "Each variable in Go exists as long as there are references to it"? If not, is there any reasons?
Not in the language specification, no; that is a quality of the runtime, not the language. We can confirm the fact that memory is not collected as long as there are references to it by simply observing that Go programs actually work. If that assumption were not valid, most of the standard library would be invalid, along with pretty much all code written by any Go developer. The Go compiler's escape analysis and garbage collector definitely work.
The FAQ entry you found is canonical and can be relied upon, same as the spec.
The thing that you could imagine would cause an issue is the *px in the main function. If the thing pointed to by px does not exist any more. However, According to this section: https://golang.org/ref/spec#Address_operators
For an operand x of pointer type *T, the pointer indirection *x denotes the variable of type T pointed to by x. If x is nil, an attempt to evaluate *x will cause a run-time panic.
This basically says that an implementation of Go is bound to give you the value pointed to, unless the pointer is nil in which case it will panic. The specification does not say how an implementation is to do this, but you can count on any implementation of Go doing this some how.
This matches what your first quote says.
In go, there's a "prefix" that you can put on a function. How is this useful? What are the use cases for this?
Example:
type a struct {
Thing string
}
func (something a) b() {
fmt.Println(something.Thing)
}
Programming is communication: you are communicating to the machine what it should do, and communicating to other programmers (including your future self). Many higher-level programming constructs serve the purpose of making code more expressive--that is, more clearly stating the programmer's intent.
Go's function receivers are like the "self" object in more traditional object-oriented languages. It's a way of grouping a set of functions together and saying "these methods exist primarily to operate on objects of this type", rather than just being general utility methods that happen to take an argument of that type. In other words, they exist to describe the behavior of the abstract object whose state is described by the structure.
It means that the function is attached to the struct, in this case, so that you can do a.b() somewhere else.
if i had an instance of the following struct
type Node struct {
id string
name string
address string
conn net.Conn
enc json.Encoder
dec json.Decoder
in chan *Command
out chan *Command
clients map[string]ClientNodesContainer
}
i am failing to understand when i should send a struct by reference and when should i send it by value(considering that i do not want to make any changes to that instance), is there a rule of thumb that makes it easier to decide?
all i could find is send a struct by value when its small or inexpensive to copy, but does small really mean smaller than 64bit address for example?
would be glad if someone can point some more obvious rules
The rule is very simple:
There is no concept of "pass/send by reference" in Go, all you can do is pass by value.
Regarding the question whether to pass the value of your struct or a pointer to your struct (this is not call by reference!):
If you want to modify the value inside the function or method: Pass a pointer.
If you do not want to modify the value:
If your struct is large: Use pointer.
Otherwise: It just doesn't matter.
All this thinking about how much a copy costs you is wasted time. Copies are cheap, even for medium sized structs. Passing a pointer might be a suitable optimization after profiling.
Your struct is not large. A large struct contains fields like wholeWorldBuf [1000000]uint64.
Tiny structs like yours might or might not benefit from passing a pointer and anybody who gives advice which one is better is lying: It all depends on your code and call patterns.
If you run out of sensible options and profiling shows that time is spent copying your structs: Experiment with pointers.
The principle of "usually pass values for small structs you don't intend to mutate" I agree with, but this struct, right now, is 688 bytes on x64, most of those in the embedded json structs. The breakdown is:
16*4=64 for the three strings (pointer/len pairs) and the net.Conn (an interface value)
208 for the embedded json.Encoder
392 for the embedded json.Decoder
8*3=24 for the three chan/map values (must be pointers)
Here's the code I used to get that, though you need to save it locally to run it because it uses unsafe.Sizeof.
You could embed *Encoder/*Decoder instead of pointers, leaving you at 104 bytes. I think it's reasonable to keep as-is and pass *Nodes around, though.
The Go code review comments on receiver type say "How large is large? Assume it's equivalent to passing all its elements as arguments to the method. If that feels too large, it's also too large for the receiver." There is room for differences of opinion here, but for me, nine values, some multiple words, "feels large" even before getting precise numbers.
In the "Pass Values" section the review comments doc says "This advice does not apply to large structs, or even small structs that might grow." It doesn't say that the same standard for "large" applies to normal parameters as to receivers, but I think that's a reasonable approach.
Part of the subtlety of determining largeness, alluded to in the code review doc, is that many things in Go are internally pointers or small, reference-containing structs: slices (but not fixed-size arrays), strings, interface values, function values, channels, maps. Your struct may own a []byte pointing to a 64KB buffer, or a big struct via an interface, but still be cheap to copy. I wrote some about this in another answer, but nothing I said there prevents one from having to make some less-than-scientific judgement calls.
It's interesting to see what standard library methods do. bytes.Replace has ten words' worth of args, so that's at least sometimes OK to copy onto the stack. On the other hand go/ast tends to pass around references (either * pointers or interface values) even for non-mutating methods on its not-huge structs. Conclusion: it can be hard to reduce it to a couple simple rules. :)
Most of the time you should use passing by reference. Like:
func (n *Node) exampleFunc() {
...
}
Only situation when you would like to use passing instance by value is when you would like to be sure that your instance is safe from changes.
This from Bjarne Stroustrup's The C++ Programming Language, Fourth Edition 3.3.2.
We didn’t really want a copy; we just wanted to get the result out of
a function: we wanted to move a Vector rather than to copy it.
Fortunately, we can state that intent:
class Vector {
// ...
Vector(const Vector& a); // copy constructor
Vector& operator=(const Vector& a); // copy assignment
Vector(Vector&& a); // move constructor
Vector& operator=(Vector&& a); // move assignment
};
Given that definition, the compiler will choose the move constructor
to implement the transfer of the return value out of the function.
This means that r=x+y+z will involve no copying of Vectors. Instead,
Vectors are just moved.As is typical, Vector’s move constructor is
trivial to define...
I know Golang supports traditional passing by value and passing by reference using Go style pointers.
Does Go support "move semantics" the way C++11 does, as described by Stroustrup above, to avoid the useless copying back and forth? If so, is this automatic, or does it require us to do something in our code to make it happen.
Note: A few answers have been posted - I have to digest them a bit, so I haven't accepted one yet - thanks.
The breakdown is like here:
Everything in Go is passed by value.
But there are five built-in "reference types" which are passed by value as well but internally they hold references to separately maintained data structure: maps, slices, channels, strings and function values (there is no way to mutate the data the latter two reference).
Your own answer, #Vector, is incorrect is that nothing in Go is passed by reference. Rather, there are types with reference semantics. Values of them are still passed by value (sic!).
Your confusion suppsedly stems from the fact your mind is supposedly currently burdened by C++, Java etc while these things in Go are done mostly "as in C".
Take arrays and slices for instance. An array is passed by value in Go, but a slice is a packed struct containing a pointer (to an underlying array) and two platform-sized integers (the length and the capacity of the slice), and it's the value of this structure which is copied — a pointer and two integers — when it's assigned or returned etc. Should you copy a "bare" array, it would be copied literally — with all its elements.
The same applies to channels and maps. You can think of types defining channels and maps as declared something like this:
type Map struct {
impl *mapImplementation
}
type Slice struct {
impl *sliceImplementation
}
(By the way, if you know C++, you should be aware that some C++ code uses this trick to lower exposure of the details into header files.)
So when you later have
m := make(map[int]string)
you could think of it as m having the type Map and so when you later do
x := m
the value of m gets copied, but it contains just a single pointer, and so both x and m now reference the same underlying data structure. Was m copied by reference ("move semantics")? Surely not! Do values of type map and slice and channel have reference semantincs? Yes!
Note that these three types of this kind are not at all special: implementing your custom type by embedding in it a pointer to some complicated data structure is a rather common pattern.
In other words, Go allows the programmer to decide what semantics they want for their types. And Go happens to have five built-in types which have reference semantics already (while all the other built-in types have value semantics). Picking one semantics over the other does not affect the rule of copying everything by value in any way. For instance, it's fine to have pointers to values of any kind of type in Go, and assign them (so long they have compatible types) — these pointers will be copied by value.
Another angle to look at this is that many Go packages (standard and 3rd-party) prefer to work with pointers to (complex) values. One example is os.Open() (which opens a file on a filesystem) returning a value of the type *os.File. That is, it returns a pointer and expects the calling code to pass this pointer around. Surely, the Go authors might have declared os.File to be a struct containing a single pointer, essentially making this value have reference semantics but they did not do that. I think the reason for this is that there's no special syntax to work with the values of this type so there's no reason to make them work as maps, channels and slices. KISS, in other words.
Recommended reading:
"Go Data Structures"
"Go Slices: Usage and Internals"
Arrays, slices (and strings): The mechanics of 'append'"
A thead on golang-nuts — pay close attention to the reply by Rob Pike.
The Go Programming Language Specification
Calls
In a function call, the function value and arguments are evaluated in
the usual order. After they are evaluated, the parameters of the call
are passed by value to the function and the called function begins
execution. The return parameters of the function are passed by value
back to the calling function when the function returns.
In Go, everything is passed by value.
Rob Pike
In Go, everything is passed by value. Everything.
There are some types (pointers, channels, maps, slices) that have
reference-like properties, but in those cases the relevant data
structure (pointer, channel pointer, map header, slice header) holds a
pointer to an underlying, shared object (pointed-to thing, channel
descriptor, hash table, array); the data structure itself is passed by
value. Always.
Always.
-rob
It is my understanding that Go, as well as Java and C# never had the excessive copying costs of C++, but do not solve ownership transference to containers. Therefore there is still copying involved. As C++ becomes more of a value-semantics language, with references/pointers being relegated to i) smart-pointer managed objects inside classes and ii) dependence references, move semantics solves the problem of excessive copying. Note that this has nothing to do with "pass by value", nowadays everyone passes objects by Reference (&) or Const Reference (const &) in C++.
Let's look at this (1) :
BigObject BO(big,stuff,inside);
vector<BigObject> vo;
vo.reserve(1000000);
vo.push_back(BO);
Or (2)
vector<BigObject> vo;
vo.reserve(1000000);
vo.push_back(BigObject(big,stuff,inside));
Although you're passing by reference to the vector vo, in C++03 there was a copy inside the vector code.
In the second case, there is a temporary object that has to be constructed and then is copied inside the vector. Since it can only be accessed by the vector, that is a wasteful copy.
However, in the first case, our intent could be just to give control of BO to the vector itself. C++17 allows this:
(1, C++17)
vector<BigObject> vo;
vo.reserve(1000000);
vo.emplace_back(big,stuff,inside);
Or (2, C++17)
vector<BigObject> vo;
vo.reserve(1000000);
vo.push_back(BigObject(big,stuff,inside));
From what I've read, it is not clear that Java, C# or Go are exempt from the same copy duplication that C++03 suffered from in the case of containers.
The old-fashioned COW (copy-on-write) technique, also had the same problems, since the resources will be copied as soon as the object inside the vector is duplicated.
Stroustrup is talking about C++, which allows you to pass containers, etc by value - so the excessive copying becomes an issue.
In Go, (like in Delphi, Java, etc) when you pass a container type, etc they are always references, so it's a non-issue. Regardless, you don't have to deal with it or worry about in GoLang - the compiler just does what it needs to do, and from what I've seen thus far, it's doing it right.
Tnx to #KerrekSB for putting me on the right track.
#KerrekSB - I hope this is the right answer. If it's wrong, you bear no responsibility.:)