In go, I'm trying to iterate through an array that was created in C. I have the length of the array (int arrc) and its pointer (mytype *arrv).
I have found a way, but it involves transferring back and forth between go and c and is super super hacky.
// void *nextelement(void *p, int i, int size) {
// return (void*)((uint64_t)p+i*size);
// }
#import "C"
...
for i := 0; i < protoc; i++ {
adr := (*C.mytype)(C.nextelement(unsafe.Pointer(myarr), C.int(i), C.sizeof_mytype))
All that code just to get myarr[i]... it doesn't feel right.
Note: this is simply copied from the CGo documentation.
Turning C arrays into Go slices
C arrays are typically either null-terminated or have a length kept
elsewhere.
Go provides the following function to make a new Go byte slice from a
C array:
func C.GoBytes(cArray unsafe.Pointer, length C.int) []byte
To create a Go slice backed by a C array (without copying the original
data), one needs to acquire this length at runtime and use a type
conversion to a pointer to a very big array and then slice it to the
length that you want (also remember to set the cap if you're using Go
1.2 or later), for example (see http://play.golang.org/p/XuC0xqtAIC for a runnable example):
var theCArray *C.YourType = C.getTheArray()
length := C.getTheArrayLength()
slice := (*[1 << 28]C.YourType)(unsafe.Pointer(theCArray))[:length:length]
It is important to keep in mind that the Go garbage collector will not
interact with this data, and that if it is freed from the C side of
things, the behavior of any Go code using the slice is
nondeterministic.
The reason for the magic array size constant is described in What does (*[1 << 30]C.YourType) do exactly in CGo? The actual value of the constant, whether it's 1 << 30 or 1 << 28 or whatever, is not crucial, but it must be at least as large as the largest length value will be.
Related
I'm still new to programming. Forgive my lack of computer science knowledge. Not sure if this question is specific to Golang or computer science in general...
I always thought that functions do not alter variables/data held outside their own scope unless you use a return statement back into the other scope, or unless they are higher in the hierarchy of scopes. One may argue that functions f1 and f2 in this example are called from a lower scope. However, this still doesn't explain why I'm getting different results for variable num and nums.
package main
import "fmt"
func f1(a int) {
a = 50 // this will not work, as it shouldn't
}
func f2(a ...int) {
a[0] = 50 // this will work without return statement
a[1] = 50 // this will work without return statement
}
func main() {
num := 2
nums := []int{2, 2}
f1(num)
f2(nums...)
fmt.Printf("function f1 doesn't affect the variable num and stays: %v\n", num)
fmt.Printf("function f2 affects the variable nums and results in: %v", nums)
Questions:
Why doesn't f2 require a return statement to modify nums like num
would within f1?
Golang functions are said to pass values (rather than reference),
shouldn't that force the function to return copies?
Can this happen in other languages? (I think I may have
seen this in other languages).
This is the correct behaviour, since a ...int is equal to a slice e.g.: a []int
func f2(a []int) {
a[0] = 50
a[1] = 50
}
func main() {
b := []int{2, 2}
f2(b)
fmt.Println(b) // [50 50]
}
And a slice is a view to the original data, here 'b'.
"Why doesn't f2 require a return statement to modify nums like num would in f1?"
In f2 you are using the slice, which has a pointer to the original array, so f2 can change the outside array.
"Golang functions are said to pass values (not reference), shouldn't that force to return copies? (If the question is related...)"
In f2 the slice itself is passed by value, meaning pointer and length and capacity of the original array.
"Can this happen in other languages? (I think I may have seen this in other langues)"
Too broad to answer, there are many languages and in general if you have a pointer to the outside world array, yes.
Edit:
package main
import "fmt"
func sum(a ...int) int {
s := 0
for _, v := range a {
s += v
}
return s
}
func f2(a []int) {
c := make([]int, len(a))
copy(c, a)
c[0] = 50
fmt.Println(sum(c...)) // 52
}
func main() {
b := []int{2, 2}
fmt.Println(sum(1, 2, 3, 4)) // 10
fmt.Println(sum(b...)) // 4
f2(b)
fmt.Println(b) // [2 2]
}
Notes:
The sum() function above is a pure function, since it has no side effect.
The new f2 function above is a pure function, since it has no side effect: it makes a copy of a into c then calls the sum.
In go, function arguments are passed by value. That means, if you pass an int (like in f1), compiler will pass the value of f1, essentially copying it. If the function takes a *int and you pass &num, then the compiler passes the value of &num, which is a pointer to num. When the function changes *num, the value of the variable outside the function will change. If the function changes num, the pointer value of num will change, and it will point to a different variable.
As a contrast, Java passes all primitive values as value, and all objects by reference. That is, if you pass an int, there is no way for the function to modify the value of that int that is visible to the caller. If you want to pass an int the function can modify, you put that in a class and pass an instance of that class in Java.
A slice (as in f2) contains a pointer to the underlying array. When you call a function with a slice, the slice header (containing a pointer to the underlying array) is copied, so when the function changes the slice elements, the underlying array elements change.
The question of scope is somewhat different. Scope of a function is all the variables it can see. Those are the global variables (if from different packages, exported global variables), function arguments, and if the function is declared nested within another function, all the variables visible in that function at that point.
1 & 2) Both questions can be answered when looking at how slices work in Go. There's a blog article on it.
In general, all variables are passed by value in Go. You can use pointers (e.g. *int for f1) to pass by reference (or more correct, the address of the pointer).
However, slices are technically also passed by value.
When we look here, we can get an idea how they work:
type SliceHeader struct {
Data uintptr
Len int
Cap int
}
Len and Cap are integers, but Data is a pointer.
When this struct is copied (when passing by value), a copy of Len, Cap and Data will be made. Since Data is a pointer, any modifications made to the value it's pointing to will be visible after your function returns.
You can also read this
Suppose I have a slice slice of type int. While declaring, I set the third argument to size, which I believe reserves memory for at least size ints by setting the cap parameter of the slice.
slice:=make([]int,0,size)
Now, suppose I have an integer variable value. To add it to the slice at the end, I use
slice=append(slice,value)
If the number of elements currently in the slice is less than size, then there will be no need to copy the entire underlying array to a new location in order to add the new element.
Further, if I want to prepend value to slice, as suggested here and here, I use
slice=append([]int{value},slice...)
My question is, what happens in this case? If the number of elements is still less than size, how are the elements stored in the memory? Assuming a contiguous allocation when the make() function was invoked, are all existing elements right shifted to free the first space for value? Or is memory reallocated and all elements copied?
The reason for asking is that I would like my program to be as fast as possible, and would like to know if this is a possible cause for slowing it down. If it is so, is there any alternative way of prepending that would be more time efficient?
With reslicing and copying
The builtin append() always appends elements to a slice. You cannot use it (alone) to prepend elements.
Having said that, if you have a slice capacity bigger than length (has "free" space after its elements) to which you want to prepend an element, you may reslice the original slice, copy all elements to an index one higher to make room for the new element, then set the element to the 0th index. This will require no new allocation. This is how it could look like:
func prepend(dest []int, value int) []int {
if cap(dest) > len(dest) {
dest = dest[:len(dest)+1]
copy(dest[1:], dest)
dest[0] = value
return dest
}
// No room, new slice need to be allocated:
// Use some extra space for future:
res := make([]int, len(dest)+1, len(dest)+5)
res[0] = value
copy(res[1:], dest)
return res
}
Testing it:
s := make([]int, 0, 5)
s = append(s, 1, 2, 3, 4)
fmt.Println(s)
s = prepend(s, 9)
fmt.Println(s)
s = prepend(s, 8)
fmt.Println(s)
Output (try it on the Go Playground):
[1 2 3 4]
[9 1 2 3 4]
[8 9 1 2 3 4]
Note: if no room for the new element, since performance does matter now, we didn't just do:
res := append([]int{value}, dest...)
Because it does more allocations and copying than needed: allocates a slice for the literal ([]int{value}), then append() allocates a new when appending dest to it.
Instead our solution allocates just one new array (by make(), even reserving some space for future growth), then just set value as the first element and copy dest (the previous elements).
With linked list
If you need to prepend many times, a normal slice may not be the right choice. A faster alternative would be to use a linked list, to which prepending an element requires no allocations of slices / arrays and copying, you just create a new node element, and you designate it to be the root by pointing it to the old root (first element).
The standard library provides a general implementation in the container/list package.
With manually managing a larger backing array
Sticking to normal slices and arrays, there is another solution.
If you're willing to manage a larger backing array (or slice) yourself, you can do so by leaving free space before the slice you use. When prepending, you can create a new slice value from the backing larger array or slice which starts at an index that leaves room for 1 element to be prepended.
Without completeness, just for demonstration:
var backing = make([]int, 15) // 15 elements
var start int
func prepend(dest []int, value int) []int {
if start == 0 {
// No more room for new value, must allocate bigger backing array:
newbacking := make([]int, len(backing)+5)
start = 5
copy(newbacking[5:], backing)
backing = newbacking
}
start--
dest = backing[start : start+len(dest)+1]
dest[0] = value
return dest
}
Testing / using it:
start = 5
s := backing[start:start] // empty slice, starting at idx=5
s = append(s, 1, 2, 3, 4)
fmt.Println(s)
s = prepend(s, 9)
fmt.Println(s)
s = prepend(s, 8)
fmt.Println(s)
// Prepend more to test reallocation:
for i := 10; i < 15; i++ {
s = prepend(s, i)
}
fmt.Println(s)
Output (try it on the Go Playground):
[1 2 3 4]
[9 1 2 3 4]
[8 9 1 2 3 4]
[14 13 12 11 10 8 9 1 2 3 4]
Analysis: this solution makes no allocations and no copying when there is room in the backing slice to prepend the value! All that happens is it creates a new slice from the backing slice that covers the destination +1 space for the value to be prepended, sets it and returns this slice value. You can't really get better than this.
If there is no room, then it allocates a larger backing slice, copies over the content of the old, and then does the "normal" prepending.
With tricky slice usage
Idea: imagine that you always store elements in a slice in backward order.
Storing your elements in backward order in a slice means a prepand becomes append!
So to "prepand" an element, you can simply use append(s, value). And that's all.
Yes, this has its limited uses (e.g. append to a slice stored in reverse order has the same issues and complexity as a "normal" slice and prepand operation), and you lose many conveniences (ability to list elements using for range just to name one), but performance wise nothing beats prepanding a value just by using append().
Note: iterating over the elements that stores elements in backward order has to use a downward loop, e.g.:
for i := len(s) - 1; i >= 0; i-- {
// do something with s[i]
}
Final note: all these solutions can easily be extended to prepend a slice instead of just a value. Generally the additional space when reslicing is not +1 but +len(values), and not simply setting dst[0] = value but instead a call to copy(dst, values).
The "prepend" call will need to allocate an array and copy all elements because a slice in Go is defined as a starting point, a size and an allocation (with the allocation being counted from the starting point).
There is no way a slice can know that the element before the first one can be used to extend the slice.
What will happen with
slice = append([]int{value}, slice...)
is
a new array of a single element value is allocated (probably on stack)
a slice is created to map this element (start=0, size=1, alloc=1)
the append call is done
append sees that there is not enough room to extend the single-element slice so allocates a new array and copies all the elements
a new slice object is created to refer to this array
If appending/removing at both ends of a large container is the common use case for your application then you need a deque container. It is unfortunately unavailable in Go and impossible to write efficiently for generic contained types while maintaining usability (because Go still lacks generics).
You can however implement a deque for your specific case and this is easy (for example if you have a large container with a known upper bound may be a circular buffer is all you need and that is just a couple of lines of code away).
I'm very new to Go, so may be the following is very bad Go code... but it's an attempt to implement a deque using a growing circular buffer (depending on the use case this may be or may be not a good solution)
type Deque struct {
buffer []interface{}
f, b, n int
}
func (d *Deque) resize() {
new_buffer := make([]interface{}, 2*(1+d.n))
j := d.f
for i := 0; i < d.n; i++ {
new_buffer[i] = d.buffer[j]
d.buffer[j] = nil
j++
if j == len(d.buffer) {
j = 0
}
}
d.f = 0
d.b = d.n
d.buffer = new_buffer
}
func (d *Deque) push_back(x interface{}) {
if d.n == len(d.buffer) {
d.resize()
}
d.buffer[d.b] = x
d.b++
if d.b == len(d.buffer) {
d.b = 0
}
d.n++
}
func (d *Deque) push_front(x interface{}) {
if d.n == len(d.buffer) {
d.resize()
}
if d.f == 0 {
d.f = len(d.buffer)
}
d.f--
d.buffer[d.f] = x
d.n++
}
func (d *Deque) pop_back() interface{} {
if d.n == 0 {
panic("Cannot pop from an empty deque")
}
if d.b == 0 {
d.b = len(d.buffer)
}
d.b--
x := d.buffer[d.b]
d.buffer[d.b] = nil
d.n--
return x
}
func (d *Deque) pop_front() interface{} {
if d.n == 0 {
panic("Cannot pop from an empty deque")
}
x := d.buffer[d.f]
d.buffer[d.f] = nil
d.f++
if d.f == len(d.buffer) {
d.f = 0
}
d.n--
return x
}
I have the following variables:
var foo int8
var bar [5]int8
I want to count number of bytes in both variables and output the sum - so right here it should be 1 + 5 = 6. Is this possible?
You can use reflect.Size, this returns a uintptr for some reason, but is actually just the exact same thing as unsafe.SizeOf without having to use unsafe.
var x [8]byte
t := reflect.TypeOf(x)
fmt.Println(t.Size())
The snags are going to be maps, slices, strings, and pointers, which you'll get the size of the header/metadata (or just the pointer size for pointers). If that's okay, great! If not, you can handle this in various ways, this is the best I have off the top of my head:
func UnderlyingSize(x interface{}) uintptr {
v := reflect.ValueOf(x)
t := v.Type()
var size uintptr;
switch t.Kind() {
// For the builtin collections, we just multiply the len by the
// element size, for maps also do the key
case reflect.Map:
l := uintptr(v.Len())
size = t.Key().Size()*l + t.Elem().Size()*l
case reflect.Slice:
t := t.Elem();
size = t.Size() * uintptr(v.Len())
case reflect.Ptr:
t := t.Elem();
size = t.Size()
// Strings are just byte arrays, so it's just the len
case reflect.String:
size = uintptr(v.Len())
// For an interface, we need to find the underlying type
case reflect.Interface:
v := v.Elem()
size = UnderlyingSize(v)
// For anything else, including arrays, Size returns the correct value
default:
size = t.Size();
}
return size
}
There is an argument to be made for using Cap rather than Len, but it's easy enough to change yourself. You can also add t.Size() to any of these values if you want the size of the header information AND the underlying size. Note a word of warning that the real map probably takes more memory than just the key+value+header size, since there's probably some extra information under the hood.
If you have a data structure that's a collection, you'll have to implement something like this yourself, but if it's a simple struct (i.e. only made of POD structs and builtin types), you can simply add up UnderlyingSize of all the members.
You can use unsafe.Sizeof (https://play.golang.org/p/FroasKud7I):
unsafe.Sizeof(foo) + unsafe.Sizeof(bar)
I have a question about the reader interface, the definition looks like:
type Reader interface {
Read(p []byte) (n int, err error)
}
I have following code that use the reader interface:
package main
import (
"fmt"
"os"
)
// Reading files requires checking most calls for errors.
// This helper will streamline our error checks below.
func check(e error) {
if e != nil {
panic(e)
}
}
func main() {
// You'll often want more control over how and what
// parts of a file are read. For these tasks, start
// by `Open`ing a file to obtain an `os.File` value.
f, err := os.Open("configuration.ini")
check(err)
// Read some bytes from the beginning of the file.
// Allow up to 5 to be read but also note how many
// actually were read.
b1 := make([]byte, 10)
n1, err := f.Read(b1)
check(err)
fmt.Printf("%d bytes: %s\n", n1, string(b1))
f.Close()
}
As you can see the code above, b1 is defined as byte slice and it passed to the Read method as value argument. After the Read method, the b1 contains the first 10 letters from file.
What for me very confusing about the code above is, why does b1 contains suddenly values after the Read method.
In Golang, when I pass a value to the method, it will be passed as value and not as reference. To clarify, what I talking about, I made a sample application:
package main
import (
"fmt"
)
func passAsValue(p []byte) {
c := []byte("Foo")
p = c
}
func main() {
b := make([]byte, 10)
passAsValue(b)
fmt.Println(string(b))
}
After passAsValue function, b does not contain any values and that what I expected in golang, arguments will be pass as value to the function or method.
Why then, the first code snippet can change the content of the passed argument? If the Read method expects a pointer of []byte slice, then I would be agreed, but on this case not.
Everything is passed by value (by creating a copy of the value being passed).
But since slices in Go are just descriptors for a contiguous segment of an underlying array, the descriptor will be copied which will refer to the same underlying array, so if you modify the contents of the slice, the same underlying array is modified.
If you modify the slice value itself in the function, that is not reflected at the calling place, because the slice value is just a copy and the copy will be modified (not the original slice descriptor value).
If you pass a pointer, the value of the pointer is also passed by value (the pointer value will be copied), but in this case if you modify the pointed value, that will be the same as at the calling place (the copy of the pointer and the original pointer points to the same object/value).
Related blog articles:
Go Slices: usage and internals
Arrays, slices (and strings): The mechanics of 'append'
The slice header in Go contains in itself a pointer to the underlaying array.
You can read from the official blog post: https://blog.golang.org/slices
Even though the slice header is passed by value, the header includes a pointer to elements of an array, so both the original slice header and the copy of the header passed to the function describe the same array. Therefore, when the function returns, the modified elements can be seen through the original slice variable.
It is the exact same behaviour as passing a pointer in C :
#include <stdio.h>
#include <stdlib.h>
// p is passed by value ; however, this function does not modify p,
// it modifies the values pointed by p.
void read(int* p) {
int i;
for( i=0; i<10; i++) {
p[i] = i+1;
}
}
// p is passed by value, so changing p itself has no effect out
// of the function's scope
void passAsValue(int*p) {
int* c = (int*)malloc(3*sizeof(int));
c[0] = 15; // 'F' in hex is 15 ...
c[1] = 0;
c[2] = 0;
p = c;
}
int main() {
int* p = (int*)malloc(10*sizeof(int));
int i;
for( i=0; i<10; i++) {
p[i] = 0;
}
printf(" init : p[0] = %d\n", p[0]);
read(p);
printf(" after read : p[0] = %d\n", p[0]);
passAsValue(p);
printf("after passAsValue : p[0] = %d\n", p[0]);
return 0;
}
output :
// init : p[0] = 0
// after read : p[0] = 1
//after passAsValue : p[0] = 1 // <- not 15, the modification from
// // within passAsValue is not persistent
(for the record : this C program leaks the int* c array)
A Go slice contains more info than just the pointer : it is a small struct, which contains the pointer, the length, and the max capacity of the allocated array (see the link mentioned in other answers : https://blog.golang.org/slices ).
But from the code's perspective, it behaves exactly like the C pointer.
The code snippet below is the library implementation of the push methods for a priority queue. I am wondering why the line with the code a = a[0 : n+1] does not throw an out of bounds errors.
func (pq *PriorityQueue) Push(x interface{}) {
// Push and Pop use pointer receivers because they modify the slice's length,
// not just its contents.
// To simplify indexing expressions in these methods, we save a copy of the
// slice object. We could instead write (*pq)[i].
a := *pq
n := len(a)
a = a[0 : n+1]
item := x.(*Item)
item.index = n
a[n] = item
*pq = a
}
a slice is not an array; it is a view onto an existing array. The slice in question is backed by an array larger than itself. When you define a slice of an existing slice, you're actually slicing the underlying array, but the indexes referenced are relative to the source slice.
That's a mouthful. Let's prove this in the following way: we'll create a slice of zero length, but we'll force the underlying array to be larger. When creating a slice with make, the third parameter will set the size of the underlying array. The expression make([]int, 0, 2) will allocate an array of size 2, but it evaluates to a size-zero slice.
package main
import ("fmt")
func main() {
// create a zero-width slice over an initial array of size 2
a := make([]int, 0, 2)
fmt.Println(a)
// expand the slice. Since we're not beyond the size of the initial
// array, this isn't out of bounds.
a = a[0:len(a)+1]
a[0] = 1
fmt.Println(a)
fmt.Println(a[0:len(a)+1])
}
see here. You can use the cap keyword to reference the size of the array that backs a given slice.
The specific code that you asked about loops over cap(pq) in the calling context (container/heap/example_test.go line 90). If you modify the code at the call site and attempt to push another item into the queue, it will panic like you expect. I ... probably wouldn't suggest writing code like this. Although the code in the standard library executes, I would be very sour if I found that in my codebase. It's generally safer to use the append keyword.
Because it works in a specific example program. Here are the important parts from the original/full example source)
const nItem = 10
and
pq := make(PriorityQueue, 0, nItem)
and
for i := 0; i < cap(pq); i++ {
item := &Item{
value: values[i],
priority: priorities[i],
}
heap.Push(&pq, item)
}
Is it an example from container/heap? If yes, then it doesn't throws an exception because capacity is big enough (see how the Push method is used). If you change the example to Push more items then the capacity, then it'll throw.
It does in general; it doesn't in the container/heap example. Here's the general fix I already gave you some time ago.
func (pq *PriorityQueue) Push(x interface{}) {
a := *pq
n := len(a)
item := x.(*Item)
item.index = n
a = append(a, item)
*pq = a
}
Golang solution to Project Euler problem #81