I need to populate a struct that has a member of type [8]uint8. This needs be populated with a byte array of type []byte initialized to length 8. The simplistic approach does not work:
Data: [8]uint8(RequestFrame(0x180, r)),
gives
cannot convert .. (type []byte) to type [8]uint8
Since both arrays are structurally identical it would be nice if this could be done with casting/assignment rather than copying?
Background
The problem with your "simplistic approach" is that a slice
(of any type) is a struct-typed value consisting of a pointer
and two integers; the pointer contains the address of the
underlying (backing) data array, and the integers contain
what len() and cap() builtins return for that slice.
In other words, a slice is sort of a view into an array.
Then, in Go, there is no concept of a type cast; there are only
type conversions, and these conversions may only happen between
types with the same underlying representation¹.
Since a slice and an array may not have the same underlying
representation (array is literally a contiguous block of memory
of the size just enough to contain all the array's elements),
your alleged type conversion may not be legal.
Possible solutions
There are two possible solutions.
The simplest is to just copy the data from the slice's
backing array into a newly-allocated array:
var (
src = []byte{1, 2, 3, 4, 5, 6, 7, 8}
dst [8]uint8
)
copy(dst[:], src[:8])
Note that there exists an inherent disparity between slice an
array types: an array type encodes both the type of its elements
and its length (that is, the length is a part of the type),
while a slice type only encodes the type of its elements
(and may be of any length at runtime).
This means that you might need to have a check before such
copying that makes sure the source slice has exactly 8
elements, that is, len(src) == len(dst).
This invariant may be enforced by some other code, but I think
I'd warn you up front about this: if src has less than 8
elements, the src[:8] expression will panic at runtime,
and if it contains more, then there's the question of whether
copying just the first 8 of them is exactly what's needed.
The second approach (admittedly messier) is to just re-use
the underlying array of the slice:
import "unsafe"
var (
src = []byte{1, 2, 3, 4, 5, 6, 7, 8}
dstPtr *[8]uint8
)
if len(src) != len(*dstPtr) {
panic("boom")
}
dstPtr = (*[8]uint8)(unsafe.Pointer(&src[0]))
Here, we've just taken the address of the first element
contained in the slice's underlying array and peformed
a "dirty" two-phase type-conversion to make the obtained
pointer to be of type *[8]uint8—that is, "an address of
an array of 8 uint8s".
Note two caveats:
The resulting pointer now points to
the same memory block the original slice does.
It means it's now possible to mutate that memory both through the
slice and the pointer we obtained.
As soon as you'll decide to assign the array's data
to a variable of type [8]uint8 (and passing it as an argument
to a function's parameter of that type), you will dereference
that pointer (like with *dstPtr), and at that moment
the array's data will be copied.
I'm specifically mentioning this as often people resort
to hacks like this one to pull the backing array out of
a slice precisely in an attempt to not copy the memory.
TL;DR
Copy the data (after supposedly verifying the
len(src) == len(dst) invariant holds).
Copying 8 bytes is fast (on a typical 64-bit CPU this will be
a single MOV instruction, or two at most), and the code will
be straightforward.
Only resort to hacks from the second solution when you really
need to optimize on some critical hot path.
In that case, comment the solution extensively and watch for
not accidentally dereferencing your pointer.
¹ There are notable exceptions to this rule:
A []byte is type-convertible to string, and vice-versa.
A string is type-convertible to []rune, and vice-versa.
An int is type-convertible to string (but since Go 1.15 go vet gives a warning about it, and this feature may probably be prohibited in the future).
You can copy the contents of your byte slice into your uint8 array very simply by using copy, like this:
package main
import (
"fmt"
)
func main() {
slice := []byte{1, 2, 3, 4, 5, 6, 7, 8}
array := [8]uint8{}
copy(array[:], slice)
fmt.Println(array)
}
Outputs
[1 2 3 4 5 6 7 8]
Try it out on the playground.
But may I ask why you are using an array? It's usually better to just use slices, unless you have a really good reason.
Starting from Go 1.17 you are able to use type conversion directly, from a slice to an array pointer:
a := make([]byte, 8)
b := (*[8]uint8)(a) // b is pointer to [8]uint8
The you can just dereference to obtain a non-pointer [8]uint8 type.
a := make([]byte, 8)
b := *(*[8]uint8)(a) // b is [8]uint8
Notes:
unlike copy, the conversion approach does not incur in extra allocations (not yours, nor any possibly done by copy), because it simply yields a pointer to the existing backing array. Though dereferencing the array pointer will make a copy.
the conversion panics if the length of the array is greater than the slice's
a := make([]byte, 5)
b := (*[10]byte)(a) // panics
the pointer points to the slice's underlying array, therefore the same values will be visible by indexing either:
a := []byte{0xa1, 0xa2}
b := (*[2]uint8)(a)
fmt.Printf("%x\n", a[0]) // a1
b[0] = 0xff
fmt.Printf("%x\n", a[0]) // ff
you can convert from byte to uint8, including type literals derived from them, because byte is an alias (identical to) of uint8.
Related: How do you convert a slice into an array?
Related
For example:
func test(requiredIp []int, i int) []int {
requiredIp = append(requiredIp[0:i], requiredIp[i+1:]...)
return requiredIp
}
func main(){
requiredIp := []int{1,2,4,5,6}
fmt.Println(test(requiredIp,0)) // output:[2 4 5 6]
fmt.Println("original", requiredIp) // output:[2 4 5 6,6]
}
Why original slice has two 6 ?
A slice is a struct type with three fields:
A pointer to (an address of) the underlying array holding the data.
The length — how many elements are there in the slice.
The capacity — how many elements it's possible to store into the underlying array without reallocating it.
As everything in Go, slices are passed by value. This happens when you assign a slice value to a variable or when you pass it as a parameter in a function/method call.
What is pased by value (that is, copied) is that structure with three fields.
The pointer which is copied, obviously points at the same data block as the one in the original slice.
What happens in your code is the following:
The original slice is []int{1,2,4,5,6}.
It has length and capacity equal to 5.
It is passed to a functon, test.
The slice available there via the function's parameter requiredIp is initially identical to the one which is passed in a call.
You reslice that slice by evaluating requiredIp[0:i], and since in your call i equals to 0, you evaluate requiredIp[0:0].
That expression creates a slice with the backing array and the capacity of the original slice and length 0.
You then reslice the original slice once again — with the expression requiredIp[i+1:], which, in your call is requiredIp[1:].
The result shares the backing array with the original, has the capacity 4 and the contents []int{2,4,5,6}.
You then append the slice obtained on the previous step to the one obtained in the step before the previous one. That's where it gets interesting.
Consider that the slice being appended to points at the 0th element of the original slice and has length 0 and capacity 5. It means it has room for 5 elements.
That is, the slice's backing array still holds [1, 2, 4, 5, 6].
The slice being appended shares the same backing array with the slice being appended to, just it points at the 1st element, not the 0th.
The code of append sees it's told to append 4 elements and checks to see whether the target slice has enough capacity to hold them, and it has.
So append merely copies [2, 4, 5, 6] from and to the same backing array, overwriting 4 elements starting from index 0 in it. Effectively, it's elements [2, 4, 5, 6] moved one element left.
The resulting backing array now contains [2, 4, 5, 6, 6]: the first 4 element are the 4 elements which were the last, moved by one element left, overwriting what there were at indices 0 through 3.
You now return the reslting slice to the caller. That new slice value shares the backing array with all the slice values involved in the example, but remember that it has its length set to 4 — because the append appended 4 elements to the slice of length 0.
You print the original slice and the one returned from your function. They differ only in their length: the original has it equals to 5, and the returned from the function — to 4. The rest of their fields are the same: they both have capacity 5 and share the same backing array by both pointing its 0th element.
The difference in the lengths explains why "the original" slice seemingly has "extra" 6 in it. In fact in has it exactly where it were left off before test was called.
If, for some reason you really wanted to "detach" the slice produced by test from the original one by forcing it to allocate the new backing array, there are several possibilities:
Merely append to an unallocated slice — for instance,
append([]int(nil), 1, 2, 3, 4) would allocate a fresh backing array.
In your particular case that would not really be a solution as with i > 0 the append would have to operate on a non-empty slice. This can be dealt, for exmaple, with two appends:
s := append([]int(nil), input[0:i]...)
s = append(s, input[i+1:]...)
…or with allocating a new slice and copying:
s := make([]int, len(input)-1)
copy(s, input[0:i])
copy(s[i:], input[i+1:])
When reslicing the original slice, be sure to also artifically reset its capacity:
return append(input[0:i:i], input[i+1:]...)
Here, the first slice would have the same capacity as its length, i, and appending even a single element to it would force the append to allocate a new backing array, copy over these i elements to it and then copy over what is being appended.
Further reading:
https://blog.golang.org/slices-intro
https://blog.golang.org/slices
https://blog.golang.org/strings
…and actually consider starting with Effective Go.
I have the following multivariate array:
x := [2][3]int{
{3, 2, 1},
{3, 2, 1},
}
Both rows and columns are fixed-size.
I'm trying to check that the rows are sorted, and I undertand the sort function requires arrays without a known size. How can I ask the go to treat the item with a fixed-, known size as if it had an unknown size?
var allTrue bool = true
for i := range x {
t := sort.Ints(x[i]) == []int{1, 2, 3}
allTrue = allTrue && t
}
I get:
./main.go:xx:yy: sort.Ints(x[i]) used as value
./main.go:xx:yy: cannot use x[i] (type [3]int) as type []int in argument to sort.Ints
Am I reading this error message correctly?
Notwithstanding this other answer, which provides guidance regarding appropriate slicing to use the sort package, I have added this answer to provide more description on some further issues with the code you have posted. I hope this aids your understanding of the Go language.
Introduction to Slices
I undertand the sort function requires arrays without a known size [sic]
As others have said, this is not the terminology to describe this concept in Go. All Go arrays are of fixed size, as defined by the language spec. As you know, arrays are of type [N]T for an array which contains some non-negative number N of elements of type T. This is fixed at compile time and never changes during runtime of your program.
"Arrays without a known size" most closely maps to slices. Slices are distinct types in Go which allow for representation of sequences of data of a particular type, where their length is managed dynamically by the Go runtime. They are of type []T for elements of type T. In particular, their size is not part of their type definition and can change at runtime. For some slice variable x []T, the implementation provides:
an internal backing array of similar elemental type, where the implementation manages the allocation of memory and expansion of the array as the slice length increases
its length len(x) – denoting the number of elements the slice currently contains
its capacity cap(x) – the total length of the slice plus the additional extent of the backing array, which may extend beyond the length due to slicing operations restricting the view on the array or a larger array being allocated by the runtime to allow for appending more items to the slice.
See the Tour of Go and the language spec on slices for more details.
Resolving the problem with the code
As noted above, slices are of distinct type to arrays, so you cannot use something of type [N]T for some N and T where something of type []T is required.
sort.Ints sorts a slice of integers in-place – it has type signature func Ints(a []int). Your call sort.Ints(x[i]) indexes the array x at index i, which will return an array of type [3]int. This is incompatible with the sort function and leads to the compile-time error you observe.
To obtain a slice from an array, you use a slice expression. Such expressions allow arrays, slices and some other types to be used to construct new slices.
Slice expressions are given in the form a[low : high] where low and high are optional integers providing indices into the backing array or slice which specify the range to return in the new slice. The language spec link above has more details which I recommend you read; suffice to say the simplest slice expression a[:] for some array or slice a is syntactic sugar to mean a[0:len(a)-1], i.e. transform the array/slice into a slice of the same length.
Using this trick, obtain a slice of type []int from your multi-dimensional array by slicing: x[i][:]:
x[i] returns an array of type [3]int, as before
slicing the returned array returns a slice of type []int, which is compatible with sort.Ints.
sort.Ints does not return a value, and slices are not comparable
Even if you fix these issues with your code, there remain two issues with the following line:
t := sort.Ints(x[i]) == []int{1, 2, 3}
sort.Ints sorts in-place; it does not return a value, so the equality test is meaningless.
sort.Ints operates on slices, which are not comparable. It is not possible to call A == B where either A or B is a slice, unless either A or B is the special identifier nil. This is a subtle point which is covered in the language spec. (Aside: read that page, as you will note arrays are comparable.)
As you cannot compare slices directly using the == equality operator, verifying element-wise equality of slices requires:
Slices to be of the same length (dissimilar lengths implies one slice has more elements than the other)
Elements at each index of one slice are identical to other slices.
(I am ignoring the fact that one slice may have a dissimilar capacity to another, as we only care about the element-wise equality.)
This can be verified by looping through one of the slices and verifying elements at each index correspond at the same index in the other slice. This example code provides an example of that (playground link):
package main
import (
"fmt"
)
func CheckEquality(a, b []int) bool {
// Slices of dissimilar length are not equal
if len(a) != len(b) {
return false
}
for i, el := range a {
if b[i] != el {
return false
}
}
return true
}
func main() {
var mySlice = []int{1, 2, 3, 4, 5}
var mySlice2 = []int{1, 2, 3, 4, 5} // same as mySlice
var otherSlice = []int{5, 6, 7, 8, 9} // dissimilar slice
var longSlice = []int{1, 2, 3, 4, 5, 6, 7, 8, 9}
fmt.Println(CheckEquality(mySlice, mySlice2)) // Expect true
fmt.Println(CheckEquality(mySlice, otherSlice)) // Expect false
fmt.Println(CheckEquality(mySlice, longSlice)) // Expect false
}
You have to slice the array with the [:] operator before you can use it with the sort package. See: A Tour of Go: Slices and the "sort" package godoc.
Also, you can check whether the slice is sorted more efficiently than sorting it, using sort.IntsAreSorted([]int), like this.
var allTrue bool = true
for i := range x {
if !sort.IntsAreSorted(x[i][:]) {
allTrue = false
break
}
}
You can get a slice from an array by using the [:] operator, e.g.:
arr := [3]int{1, 2, 3}
slc := arr[:] // []int{1, 2, 3}, backed by arr
As such, you could use the sort.Ints(...) function:
sort.Ints(x[0][:])
sort.Ints(x[1][:])
// Now both elements of "x" are sorted.
I have the following definition:
func (c *Collector) RegisterSource(f func() []interface{}) {
c.source = f
}
I attempt to call it as follows but get an error:
func source() []int {
return []int{ 0, 1, 2, 3, 4 }
}
...
c.RegisterSource(source)
This is met with:
cannot use source (type func() []int) as type func() []interface {} in argument to c.RegisterSource
The relevant Go FAQ entry states that []T and []interface{} «do not have the same representation in memory».
To understand why, let's dissect two things:
A slice is a backing storage array plus several integers containing the size and capacity
of the slice.
In Go, arrays are not somehow "high-level"; on the contrary,
the layout of the elements they contain is strictly defined: they are all
contained in a contiguous region of memory, adjacent to each other.
This means, that in the backing array of a slice []T, elements are of
type T, and each of them occupies a region of memory of a size natural
for that type T, and all these regions are all adjacent to each other
in a single contiguous memory block.
That means, each element of a slice []int occupies exactly 64 bits
(8 bytes) on a 64-bit platform — the amount of memory a single int value
occupies.
A value of any interface type, including the empty interface,
interface{}, is represented as a structure containing two pointers,
something like:
type iface struct {
realType *typeInfo
realValue *dataType
}
(More on how interfaces are represented — here).
All of the above means in a slice []interface{} each element occupies the
region of memory the size of two pointers, and these two pointers contain
addresses of other variables in memory — as opposed to mere integer values
contained by elements of []int.
And this, in turn, means you can't just "cast" []int "to" []interface{} — simply because a value stored in any element of []int (an int) is
incompatible in its structure (memory layout) with an element
of []interface{} (a struct containing two pointers).
To produce one from another, you need to allocate a slice and convert each
element of the source slice to a matching element of the destination one.
Finally, this means if a function returns a slice of type []int,
that slice cannot be directly processed by code expecting slices of
type []interface{} (and vice-versa), and this explains why
the two function signatures from your question represent incompatible
types.
Because []int and []interface are 2 different types of slices and go does not allow automatic conversion between the 2.
https://github.com/golang/go/wiki/InterfaceSlice
all you can do is change this
func source() []int {
return []int{ 0, 1, 2, 3, 4 }
}
into that
func source() []interface{} {
return []interface{}{ 0, 1, 2, 3, 4 }
}
in order to fit the function signature you defined for RegisterSource.
I've been trying out Go for some time and this question keeps bugging me. Say I build up a somewhat large dataset in a slice (say, 10 million int64s).
package main
import (
"math"
"fmt"
)
func main() {
var a []int64
var i int64;
upto := int64(math.Pow10(7))
for i = 0; i < upto; i++ {
a = append(a, i)
}
fmt.Println(cap(a))
}
But then I decide I don't want most of them so I want to end up with a slice of just 10 of those. I've tried both slicing and delete techniques on Go's wiki but none of them seem to reduce the slice's capacity.
So that's my question: does Go has no real way of shrinking the capacity of a slice that would be similar to realloc()-ing with a smaller size argument than in your previous call on the same pointer in C? Is that an issue and how should one deal with it?
To perform an, in effect, a realloc of a slice:
a = append([]T(nil), a[:newSize]...) // Thanks to #Dijkstra for pointing out the missing ellipsis.
If it does a copy of newSize elements to a new memory place or if it does an actual in place resize as in realloc(3) is at complete discretion of the compiler. You might want to investigate the current state and perhaps raise an issue if there's a room for improvement in this.
However, this is likely a micro-optimization. The first source of performance enhancements lies almost always in selecting a better algorithm and/or a better data structure. Using a hugely sized vector to finally keep a few items only is probably not the best option wrt to memory consumption.
EDIT: The above is only partially correct. The compiler cannot, in the general case, derive if there are other pointers to the slice's backing array. Thus the realloc is not applicable. The above snippet is actually guaranteed to peform a copy of 'newSize' elements. Sorry for any confusion possibly created.
Go does not have a way of shrinking slices. This isn't a problem in most cases, but if you profile your memory use and find you're using too much, you can do something about it:
Firstly, you can just create a slice of the size you need and copy your data into it. The garbage collector will then free the large slice. Copy built-in
Secondly, you could re-use the big slice each time you wish to generate it, so you never allocate it more than once.
On a final note, you can use 1e7 instead of math.Pow10(7).
Let's see this example:
func main() {
s := []string{"A", "B", "C", "D", "E", "F", "G", "H"}
fmt.Println(s, len(s), cap(s)) // slice, length, capacity
t := s[2:4]
fmt.Println(t, len(t), cap(t))
u := make([]string, len(t))
copy(u, t)
fmt.Println(u, len(u), cap(u))
}
It produces the following output:
[A B C D E F G H] 8 8
[C D] 2 6
[C D] 2 2
s is a slice that holds 8 pieces of strings. t is a slice that keeps the part [C D]. The length of t is 2, but since it uses the same hidden array of s, its capacity is 6 (from "C" to "H"). The question is: how to have a slice of [C D] that is independent from the hidden array of s? Simply create a new slice of strings with length 2 (slice u) and copy the content of t to u. u's underlying hidden array is different from the hidden array of s.
The initial problem was this: you have a big slice and you create a new smaller slice on it. Since the smaller slice uses the same hidden array, the garbage collector won't delete the hidden array.
See the bottom of this post for more info: http://blog.golang.org/go-slices-usage-and-internals .
Additionally you can re-use most of the allocated memory during work of yours app, take a look at: bufs package
PS if you re-alocate new memory for smaller slice, old memory may not be freed in same time, it will be freed when garbage collector decides to.
You can do that by re-assigning the slice's value to a portion of itself
a := []int{1,2,3}
fmt.Println(len(a), a) // 3 [1 2 3]
a = a[:len(a)-1]
fmt.Println(len(a), a) //2 [1 2]
There is a new feature called 3-index slice in Go 1.2, which means to get part of a slice in this way:
slice[a:b:c]
In which the len for the returned slice whould be b-a, and the cav for the new slice would be c-a.
Tips: no copy is down in the whole process, it only returns a new slice which points to &slice[a] and has the len as b-a and cav as c-a.
And that's the only thing you have to do:
slice= slice[0:len(slice):len(slice)];
Then the cav of the slice would be changed to len(slice) - 0, which is the same as the len of it, and no copy is done.
What is the difference between var a [4]int and b := make([]int, 4)? The b can be extended, but not a, right? But if I know that I need really i.e. 4 elements, then is an array faster then a slice?
Is there any performance difference between var d []int and e := make([]int)? Would f := make([]int, 5) provide more performance than without the length for the first i.e. 5 elements?
Would this c := make([]int, 5, 10) not allocate more memory than I can access?
a is an array, and b is a slice. What makes slices different from arrays is that a slice is a pointer to an array; slices are reference types, which means that if you assign one slice
to another, both refer to the same underlying array. For instance, if a function takes a
slice argument, changes it makes to the elements of the slice will be visible to the caller,
analogous to passing a pointer to the underlying array(Above from Learning Go). You can easily use append and copy with slice. Array should be a little faster than slice, but it doesn't make much difference. Unless you know the size exactly, it would be better to use slice which make things easy.
make([]type,length, capacity), you can estimate the size and possible capacity to improve the performance.
More details, you can refer:Go Slices: usage and internals