I'm learning Go and have a C/C++ background. In the following example, is it safe to append the address of a into slice? When I run this example, the correct value (2) is printed, but wanted to be sure. If this is wrong, how should I do it?
func add(mapping map[string]*[]*int) {
sliceptr := &[]*int{}
mapping["foo"] = sliceptr
ele := mapping["foo"]
a := 2
// won't address of `a` go out of scope?
ele2 := append(*ele, &a)
mapping["foo"] = &ele2
}
func main() {
mapping := map[string]*[]*int{}
add(mapping)
fmt.Println(*(*mapping["foo"])[0])
}
It's safe to reference a after the function declaring it ends, because go does escape analysis. If the compiler can prove it can be accessed safely, it puts it on the stack, if not, it allocates it on the heap.
Build flags can give some insight into the escape analysis:
go build -gcflags "-m" main.go
...
./main.go:10:2: moved to heap: a
...
This might be helpful: Allocation efficiency.
Also, it's less common to see pointers to slices, since a slice is small: a pointer, length and capacity. See slice internals.
Related
I have the following code snippet which "go vet" complains about with the warning "possible misuse of reflect.SliceHeader". I can not find very much information about this warning other then this. After reading that it is not very clear to me what is needed to do this in a way that makes go vet happy - and without possible gc issues.
The goal of the snippet is to have a go function copy data to memory which is managed by an opaque C library. The Go function expects a []byte as a parameter.
func Callback(ptr unsafe.Pointer, buffer unsafe.Pointer, size C.longlong) C.longlong {
...
sh := &reflect.SliceHeader{
Data: uintptr(buffer),
Len: int(size),
Cap: int(size),
}
buf := *(*[]byte)(unsafe.Pointer(sh))
err := CopyToSlice(buf)
if err != nil {
log.Fatal("failed to copy to slice")
}
...
}
https://pkg.go.dev/unsafe#go1.19.4#Pointer
Pointer represents a pointer to an arbitrary type. There are four
special operations available for type Pointer that are not available
for other types:
A pointer value of any type can be converted to a Pointer.
A Pointer can be converted to a pointer value of any type.
A uintptr can be converted to a Pointer.
A Pointer can be converted to a uintptr.
Pointer therefore allows a program to defeat the type system and read
and write arbitrary memory. It should be used with extreme care.
The following patterns involving Pointer are valid. Code not using
these patterns is likely to be invalid today or to become invalid in
the future. Even the valid patterns below come with important caveats.
Running "go vet" can help find uses of Pointer that do not conform to
these patterns, but silence from "go vet" is not a guarantee that the
code is valid.
(6) Conversion of a reflect.SliceHeader or reflect.StringHeader Data
field to or from Pointer.
As in the previous case, the reflect data structures SliceHeader and
StringHeader declare the field Data as a uintptr to keep callers from
changing the result to an arbitrary type without first importing
"unsafe". However, this means that SliceHeader and StringHeader are
only valid when interpreting the content of an actual slice or string
value.
var s string
hdr := (*reflect.StringHeader)(unsafe.Pointer(&s)) // case 1
hdr.Data = uintptr(unsafe.Pointer(p)) // case 6 (this case)
hdr.Len = n
In this usage hdr.Data is really an alternate way to refer to the
underlying pointer in the string header, not a uintptr variable
itself.
In general, reflect.SliceHeader and reflect.StringHeader should be used only as *reflect.SliceHeader and *reflect.StringHeader pointing at actual slices or strings, never as plain structs. A program should not declare or allocate variables of these struct types.
// INVALID: a directly-declared header will not hold Data as a reference.
var hdr reflect.StringHeader
hdr.Data = uintptr(unsafe.Pointer(p))
hdr.Len = n
s := *(*string)(unsafe.Pointer(&hdr)) // p possibly already lost
It looks like JimB (from the comments) hinted upon the most correct answer, though he didn't post it as an answer and he didn't include an example. The following passes go vet, staticcheck, and golangci-lint - and doesn't segfault so I think it is the correct answer.
func Callback(ptr unsafe.Pointer, buffer unsafe.Pointer, size C.longlong) C.longlong {
...
buf := unsafe.Slice((*byte)(buffer), size)
err := CopyToSlice(buf)
if err != nil {
log.Fatal("failed to copy to slice")
}
...
}
i read the golang FAQ:https://go.dev/doc/faq#stack_or_heap，i want to know when golang allocate variable on stack or heap. so i write code like below :
package main
import (
"fmt"
)
type Object struct {
Field int
}
func main() {
A := Object{1}
B := Object{2}
fmt.Println(A,B)
//fmt.Printf("A:%p;B:%p\n",&A,&B)
//m := testStackOrHeap()
//C:=m[A]
//D:=m[B]
//fmt.Printf("C:%p;D:%p\n",&C,&D)
}
//go:noinline
func testStackOrHeap() map[Object]Object {
one:=1
two:=2
A := Object{one}
B := Object{two}
C:= Object{one}
D := Object{two}
fmt.Println(C,D)
fmt.Printf("A:%p;B:%p\n",&A,&B)
m := map[Object]Object{A: A, B: B}
return m
}
then see how the compiler allocate the memory .the cmd is go tool compile "-m" main.go
the output is below :
main.go:15:13: inlining call to fmt.Println
main.go:30:13: inlining call to fmt.Println
main.go:31:12: inlining call to fmt.Printf
main.go:15:13: A escapes to heap
main.go:15:13: B escapes to heap
main.go:15:13: []interface {} literal does not escape
main.go:26:2: moved to heap: A
main.go:27:2: moved to heap: B
main.go:30:13: C escapes to heap
main.go:30:13: D escapes to heap
main.go:30:13: []interface {} literal does not escape
main.go:31:12: []interface {} literal does not escape
main.go:32:24: map[Object]Object literal escapes to heap
<autogenerated>:1: .this does not escape
my question is:
why not golang allocate variable A B in testStackOrHeap() to the stack ,they can not escape to stackframe ,if it allocate to heap , the gcworker need to collect it,but if it allocate in stack, it will release when function return.
As #Volker pointed out in a comment, the heap/stack distinction is an implementation detail, and the rules for escape analysis are defined by the compiler, not by the language. Correctness is the most important trait of a compiler, so a compiler's rules will frequently favor simplicity and performance over absolute "optimalness".
In this case, it's quite likely that the compiler doesn't know what fmt.Printf() will do with the pointers it receives. Therefore, it has to assume that the pointers might be stored somewhere on the heap by that function and that the references to those two objects might thus survive the call to testStackOrHeap(). Therefore, it errs on the side of caution and promotes those two variables to the heap.
(Note that your conclusion that they do not escape was presumably based on an assumption that fmt.Printf() won't store the pointers. Did you actually read the source code of that function to learn that it doesn't? If not, you can't actually be sure that it doesn't - just like the compiler isn't sure. And even if the current version of that function doesn't, future versions might.)
In plain C, if I want a shallow heap copy of a struct, I would malloc() and memcpy() it.
In Go, I guess I have to do something like this:
original := Data{...}
copy := &Data{} // malloc
*copy = original // memcpy
But it doesn't look nice to me, nor idiomatic. What's the "right" way to do it?
The idiomatic way is to do a simple assignment and let the compiler allocate copy on the heap after performing escape analysis:
original := Data{...}
copy := original
return © // Or call some function with © as a parameter
Upon noticing that copy is used by reference and outlives the stack, Go will automatically allocate it on the heap rather than on the stack (the copy is still done properly of course)
We effectively no longer care about the heap, letting the compiler allocate it there as needed based on escape analysis. Our only concern is the copy itself.
You can see an example in action on godbolt:
Given the following simple code:
func main() {
type Data struct{
foo string
}
original := Data{"hi"}
copy := original
copyPtr := ©
fmt.Println(copyPtr)
}
Go will automatically allocate copy on the heap:
call runtime.newobject(SB)
We can also see this in action by passing extra flags at compile time showing escape and inlining decisions:
$ go build -gcflags '-m' .
...
./main.go:11:2: moved to heap: copy
...
Note: copy is a builtin function. It might be a good idea to avoid reusing the name (it works just fine, but it's not great practice).
A struct variable in Golang can be copied to another simply by an assignment statement:
https://play.golang.org/p/4Zcbxhy5UoB
package main
import (
"fmt"
)
type User struct {
name string
}
func main() {
u1 := User{name: "foo"}
u2 := u1
u2.name = "bar"
fmt.Println("u1: ", u1)
fmt.Println("u2: ", u2)
}
output:
u1: {foo}
u2: {bar}
A related questions is here https://stackoverflow.com/a/12965872/6421681.
In go, you can do:
func numsInFactorial(n int) (nums []int) {
// `nums := make([]int)` is not needed
for i := 1; i <= n; i++ {
nums = append(nums, i)
}
return
}
However,the following doesn't work:
func mapWithOneKeyAndValue(k int, v int) (m map[int]int) {
m[k] = v
return
}
An error is thrown:
panic: assignment to entry in nil map
Instead, you must:
func mapWithOneKeyAndValue(k int, v int) map[int]int {
m := make(map[int]int)
m[k] = v
return
}
I can't find the documentation for this behavior.
I have read through all of effective go, and there's no mention of it there either.
I know that named return values are defined (i.e. memory is allocated; close to what new does) but not initialized (so make behavior isn't replicated).
After some experimenting, I believe this behavior can be reduced into understanding the behavior of the following code:
func main() {
var s []int // len and cap are both 0
var m map[int]int
fmt.Println(s) // works... prints an empty slice
fmt.Println(m) // works... prints an empty map
s = append(s, 10) // returns a new slice, so underlying array gets allocated
fmt.Println(s) // works... prints [10]
m[10] = 10 // program crashes, with "assignment to entry in nil map"
fmt.Println(m)
}
The issue seems that append likely calls make and allocates a new slice detecting that the capacity of s is 0. However, map never gets an explicit initialization.
The reason for this SO question is two-pronged. First, I would like to document the behavior on SO. Second, why would the language allow non-initializing definitions of slice and map? With my experience with go so far, it seems to be a pragmatic language (i.e. unused variables lead to compilation failure, gofmt forces proper formatting), so it would make sense for it to prevent the code from compiling.
Try to assign in nil slice by index - you will get "panic: runtime error: index out of range" (example: https://play.golang.org/p/-XHh1jNyn5g)
The only reason why append function works with nil, is that append function can do reallocation for the given slice.
For example, if you trying to to append 6th element to slice of 5 elements with current capacity 5, it will create the new array with new capacity, copy all the info from old one, and swap the data array pointers in the given slice. In my understanding, it is just golang implementation of dynamic arrays.
So, the nil slice is just a special case of slice with not enough capacity, so it would be reallocated on any append operation.
More details on https://blog.golang.org/go-slices-usage-and-internals
From https://blog.golang.org/go-maps-in-action
A nil map behaves like an empty map when reading, but attempts to write to a nil map will cause a runtime panic; don't do that. To initialize a map, use the built in make function
It seems like a nil map is considered a valid empty map and that's the reason they don't allocate memory for it automatically.
Consider the following code
package main
import (
"fmt"
)
func main() {
x := []byte("a")
fmt.Println(x)
fmt.Println(cap(x) == cap([]byte("a"))) // prints false
y := []byte("a")
fmt.Println(cap(y) == cap([]byte("a"))) // prints true
}
https://play.golang.org/p/zv8KQekaxH8
Calling simple Println with a slice variable, changes its capacity. I suspect calling any function with variadic parameters of ...interface{} produces the same effect. Is there any sane explanation for such behavior?
The explanation is, like bradfitz point in github, if you don't use make to create a slice, the compiler will use the cap it believes convenient. Creating multiple slices in different versions, or even the same, can result on slices of different capacities.
In short, if you need a concrete capacity, use make([]byte, len, cap). Otherwise you can't trust on a fixed capacity.