The C part:
struct Person {...}
struct Person * get_team(int * n)
The Go part:
n := C.int(0)
var team *C.struct_Person = C.get_team(&n)
defer C.free(unsafe.Pointer(team))
I can get the first element of the array in this way. But how to get the whole array with n elements?
and how to free them safely?
First, even though you’re using Go, when you add cgo there is no longer any "safe". It's up to you to determine when and how you free the memory, just as if you were programming in C.
The easiest way to use a C array in go is to convert it to a slice through an array:
team := C.get_team()
defer C.free(unsafe.Pointer(team))
teamSlice := (*[1 << 30]C.struct_Person)(unsafe.Pointer(team))[:teamSize:teamSize]
The max-sized array isn't actually allocated, but Go requires constant size arrays, and 1<<30 is going to be large enough. That array is immediately converted to a slice, with the length and capacity properly set.
Related
In the Golang wiki, under "Turning C arrays into Go slices", there is a block of code that demonstrates how to create a Go slice backed by a C array.
import "C"
import "unsafe"
...
var theCArray *C.YourType = C.getTheArray()
length := C.getTheArrayLength()
slice := (*[1 << 30]C.YourType)(unsafe.Pointer(theCArray))[:length:length]
Can anyone explain exactly what (*[1 << 30]C.YourType) does? How does it turn an unsafe.Pointer into a Go slice?
*[1 << 30]C.YourType doesn't do anything itself, it's a type. Specifically, it's a pointer to an array of size 1 << 30, of C.YourType values. The size is arbitrary, and only represents an upper bound that needs to be valid on the host system.
What you're doing in the third expression is a type conversion.
This converts the unsafe.Pointer to a *[1 << 30]C.YourType.
Then, you're taking that converted array value, and turning it into a slice with a full slice expression (Array values don't need to be dereferenced for a slice expression, so there is no need to prefix the value with a *, even though it is a pointer).
You could expand this out a bit like so:
// unsafe.Pointer to the C array
unsafePtr := unsafe.Pointer(theCArray)
// convert unsafePtr to a pointer of the type *[1 << 30]C.YourType
arrayPtr := (*[1 << 30]C.YourType)(unsafePtr)
// slice the array into a Go slice, with the same backing array
// as theCArray, making sure to specify the capacity as well as
// the length.
slice := arrayPtr[:length:length]
This construct has been replaced by a generalized unsafe.Slice function in go1.17:
slice := unsafe.Slice(theCArray, length)
I've got a newbie CGO question I was wondering if someone could help me with. When running with GODEBUG set to cgocheck=2 my application crashes with the following
write of Go pointer 0xc0003b72c0 to non-Go memory 0x7fefa0016810
fatal error: Go pointer stored into non-Go memory
The code causing the issue is
cArray := C.malloc(C.size_t(len(fd.Faces)) * C.size_t(unsafe.Sizeof(uintptr(0))))
defer C.free(unsafe.Pointer(cArray))
a := (*[1<<30 - 1]*C.struct_Box)(cArray)
for index, value := range fd.GetFaceRectangles() {
box := &C.struct_Box{
left: C.int(value.Min.X),
top: C.int(value.Min.Y),
right: C.int(value.Max.X),
bottom: C.int(value.Max.Y),
}
a[index] = box
}
cBoxCount := C.int(len(fd.Faces))
ret := C.facerec_compute_multi(rec.ptr, cImgData, cLen, &a[0], cBoxCount)
Specifically this row:
a[index] = box
I understand the memory for the array is allocated in C using malloc. I'm trying to add C Box to the array before passing it to a C function. Would the fix for this be for me to write a function in C which can receive the array and the items needed to create the struct and I do that part in C instead? I'm trying to minimise the number of calls to C so If I can create the array from Go that would be great. Really struggling with how to pass an array of data through to a function in C safely.
You've allocated C memory to hold many individual pointers. Each individual pointer can be set to a separate value. That's fine as far as it goes, but as you noted, the a[index] = box is a problem, because box itself holds a pointer to a Go-memory C.struct_Box.
I'm trying to add C Box to the array before passing it to a C function.
To do that literally, you need a C.malloc call. As written, you'll need one per C.struct_Box instance.
Would the fix for this be for me to write a function in C which can receive the array and the items needed to create the struct and I do that part in C instead?
While you could try that, it seems ... suboptimal.
I'm trying to minimise the number of calls to C so If I can create the array from Go that would be great. Really struggling with how to pass an array of data through to a function in C safely.
C is never really safe 😀 but I suspect your best bet here is to make use of the way C implements "arrays", which is as a sort of poor-man's slice. (Or, perhaps more accurately, a Go slice is a "C array" done right.) Remember that a slice is implemented in Go as a three-element header:
pointer to base of storage area;
current length within storage area; and
capacity of storage area
which can then hold some run-time computed number n of items of some type. C's dynamic "arrays" work the same way, only without the safety of a proper slice header.
What this means is that you can write your C code—provided you have control of this C code—to take a pointer to a base of a storage area, and an int or size_t value that counts the number n of items in that area. The signature of such a function is:
void f(T *base, size_t n);
where n is the number of items. (Add a second size_t if appropriate, if you want to provide a capacity as well, though if the memory has capacity, and the function modifies it to contain a different number of items, then either the function must return the new number of items—i.e., f would not be void—or n itself has to be provided by reference or something along those lines.)
In this case, if you can arrange the C code to take a pointer to the first of n struct Box-es (instead of the first of n pointers to struct Box-es), you can allocate that in a single C.malloc call.
If you can't rework the C code, you're stuck with a minimum of two C.malloc calls. You can, however, allocate all the struct Boxes in one C.malloc again, and make them all adjacent—i.e., a C "array" / poor-man's-slice—and then make each pointer point to the next struct Box:
cArray := C.malloc(C.size_t(len(fd.Faces)) * C.size_t(unsafe.Sizeof(uintptr(0))))
defer C.free(unsafe.Pointer(cArray))
cBoxes := C.malloc(C.size_t(len(fd.Faces)) * C.size_t(unsafe.Sizeof(C.struct_Box)))
defer C.free(unsafe.Pointer(cBoxes))
a := (*[1<<30 - 1]*C.struct_Box)(cArray)
b := (*[1<<30 - 1]C.struct_Box)(cBoxes)
for index, value := range fd.GetFaceRectangles() {
b[index] = C.struct_Box{
left: C.int(value.Min.X),
top: C.int(value.Min.Y),
right: C.int(value.Max.X),
bottom: C.int(value.Max.Y),
}
a[index] = &b[index]
}
cBoxCount := C.int(len(fd.Faces))
ret := C.facerec_compute_multi(rec.ptr, cImgData, cLen, &a[0], cBoxCount)
(I've tried to keep the code structure as similar as possible here, and this is quite untested since I don't have some of the parts.)
Aside: you have len(fd.Faces) and fd.GetFaceRectangles() and the code here assumes that the number of iterations of a for ... range fd.GetFaceRectangles() is always len(fd.Faces). I'm calling this out because I have made the same assumption, without anything to back it other than your example.
With some help on the Slack channel I ended up with this, just adding the struct directly instead of a pointer.
// Create an array in C
arr := C.malloc(C.size_t(len(fd.Faces)) * C.sizeof_Box)
defer C.free(unsafe.Pointer(arr))
var boxes []C.struct_Box
hdr := (*reflect.SliceHeader)(unsafe.Pointer(&boxes))
hdr.Data, hdr.Cap, hdr.Len = uintptr(arr), len(fd.Faces), len(fd.Faces)
for i, v := range fd.GetFaceRectangles() {
boxes[i] = C.struct_Box{
left: C.int(v.Min.X),
top: C.int(v.Min.Y),
right: C.int(v.Max.X),
bottom: C.int(v.Max.Y),
}
}
cBoxCount := C.int(len(fd.Faces))
ret := C.facerec_compute_multi(rec.ptr, cImgData, cLen, &(boxes[0]), cBoxCount)
I'm learning Go and have a C/C++ background. In the following example, is it safe to append the address of a into slice? When I run this example, the correct value (2) is printed, but wanted to be sure. If this is wrong, how should I do it?
func add(mapping map[string]*[]*int) {
sliceptr := &[]*int{}
mapping["foo"] = sliceptr
ele := mapping["foo"]
a := 2
// won't address of `a` go out of scope?
ele2 := append(*ele, &a)
mapping["foo"] = &ele2
}
func main() {
mapping := map[string]*[]*int{}
add(mapping)
fmt.Println(*(*mapping["foo"])[0])
}
It's safe to reference a after the function declaring it ends, because go does escape analysis. If the compiler can prove it can be accessed safely, it puts it on the stack, if not, it allocates it on the heap.
Build flags can give some insight into the escape analysis:
go build -gcflags "-m" main.go
...
./main.go:10:2: moved to heap: a
...
This might be helpful: Allocation efficiency.
Also, it's less common to see pointers to slices, since a slice is small: a pointer, length and capacity. See slice internals.
I have read on the golang blog: https://blog.golang.org/go-maps-in-action that:
var m map[string]int
Map types are reference types, like pointers or slices, and so the
value of m above is nil; it doesn't point to an initialized map. A nil
map behaves like an empty map when reading, but attempts to write to a
nil map will cause a runtime panic; don't do that. To initialize a
map, use the built in make function:
m = make(map[string]int)
The make function allocates and initializes a hash map data structure
and returns a map value that points to it.
I have a hard time understanding some parts of this:
What does var m map[string]int do?
Why do I need to write m = make(map[string]int) but not i = make(int)
What does var m map[string]int do?
It tells the compiler that m is a variable of type map[string]int, and assigns "The Zero Value" of the type map[string] int to m (that's why m is nil as nil is The Zero Value of any map).
Why do I need to write m = make(map[string]int) but not i = make(int)
You don't need to. You can create a initialized map also like this:
m = map[string]int{}
which does exactly the same.
The difference between maps and ints is: A nil map is perfectly fine. E.g. len() of a nil map works and is 0. The only thing you cannot do with a nil map is store key-value-pairs. If you want to do this you'll have to prepare/initialize the map. This preparation/initialization in Go is done through the builtin make (or by a literal map as shown above). This initialization process is not needed for ints. As there are no nil ints this initialization would be total noise.
Note that you do not initialize the variable m: The variable m is a map of strings to ints, initialized or not. Like i is a variable for ints. Now ints are directly usable while maps require one more step because the language works that way.
What does var m map[string]int do?
You can think about it like pointer with nil value, it does not point to anything yet but able to point to concrete value.
Why do I need to write m = make(map[string]int) but not i = make(int)
https://golang.org/doc/effective_go.html#allocation_make
Back to allocation. The built-in function make(T, args) serves a purpose different from new(T). It creates slices, maps, and channels only, and it returns an initialized (not zeroed) value of type T (not *T). The reason for the distinction is that these three types represent, under the covers, references to data structures that must be initialized before use. A slice, for example, is a three-item descriptor containing a pointer to the data (inside an array), the length, and the capacity, and until those items are initialized, the slice is nil. For slices, maps, and channels, make initializes the internal data structure and prepares the value for use. For instance,
make([]int, 10, 100)
allocates an array of 100 ints and then creates a slice structure with length 10 and a capacity of 100 pointing at the first 10 elements of the array. (When making a slice, the capacity can be omitted; see the section on slices for more information.) In contrast, new([]int) returns a pointer to a newly allocated, zeroed slice structure, that is, a pointer to a nil slice value.
These examples illustrate the difference between new and make.
var p *[]int = new([]int) // allocates slice structure; *p == nil; rarely useful
var v []int = make([]int, 100) // the slice v now refers to a new array of 100 ints
// Unnecessarily complex:
var p *[]int = new([]int)
*p = make([]int, 100, 100)
// Idiomatic:
v := make([]int, 100)
Remember that make applies only to maps, slices and channels and does not return a pointer. To obtain an explicit pointer allocate with new or take the address of a variable explicitly.
All words have the same length of 32 bits (4 bytes) or 64 bits (8 bytes),
depending on the processor and the operating system. They are identified by their memory address (represented as a hexadecimal number).
All variables of primitive types like int, float, bool, string ... are value types, they point directly to the values contained in the memory. Also composite types like arrays and structs are value types. When assigning with = the value of a value type to another variable: j = i, a copy of the original value i is made in memory.
More complex data which usually needs several words are treated as reference types. A reference type variable r1 contains the address (a number) of the memory location where the value of r1 is stored (or at least the 1st word of it):
For reference types when assigning r2 = r1, only the reference (the address) is copied and not the value!!. If the value of r1 is modified, all references of that value (like r1 and r2) will be reflected.
In Go pointers are reference types, as well as slices, maps and channels. The variables that are referenced are stored in the heap, which is garbage collected.
In the light of the above statements it's clear why the article states:
To initialize a map, use the built in make function.
The make function allocates and initializes a hash map data structure and returns a map value that points to it. This means you can write into it, compare to
var m map[string]int
which is readable, resulting a nil map, but an attempt to write to a nil map will cause a runtime panic. This is the reason why it's important to initialize the map with make.
m = make(map[string]int)
example code (edited code snippet): http://play.golang.org/p/eZV4WL-4N_
Why is it that
for x, _ := range body.Personality {
body.Personality[x].Mutate()
}
successfully mutates the structs' contents, but
for _, pf := range body.Personality{
pf.Mutate()
}
does not?
Is it that range creates new instances of each item it iterates over? because the struct does in fact mutate but it doesn't persist.
The range keyword copies the results of the array, thus making it impossible to alter the
contents using the value of range. You have to use the index or a slice of pointers instead of values
if you want to alter the original array/slice.
This behaviour is covered by the spec as stated here. The crux is that an assignment line
x := a[i] copies the value a[i] to x as it is no pointer. Since range is defined to use
a[i], the values are copied.
Your second loop is roughly equivalent to:
for x := range body.Personality {
pf := body.Personality[x]
pf.Mutate()
}
Since body.Personality is an array of structs, the assignment to pf makes a copy of the struct, and that is what we call Mutate() on.
If you want to range over your array in the way you do in the example, one option would be to make it an array of pointers to structs (i.e. []*PFile). That way the assignment in the loop will just take a pointer to the struct allowing you to modify it.