What is the difference between var a [4]int and b := make([]int, 4)? The b can be extended, but not a, right? But if I know that I need really i.e. 4 elements, then is an array faster then a slice?
Is there any performance difference between var d []int and e := make([]int)? Would f := make([]int, 5) provide more performance than without the length for the first i.e. 5 elements?
Would this c := make([]int, 5, 10) not allocate more memory than I can access?
a is an array, and b is a slice. What makes slices different from arrays is that a slice is a pointer to an array; slices are reference types, which means that if you assign one slice
to another, both refer to the same underlying array. For instance, if a function takes a
slice argument, changes it makes to the elements of the slice will be visible to the caller,
analogous to passing a pointer to the underlying array(Above from Learning Go). You can easily use append and copy with slice. Array should be a little faster than slice, but it doesn't make much difference. Unless you know the size exactly, it would be better to use slice which make things easy.
make([]type,length, capacity), you can estimate the size and possible capacity to improve the performance.
More details, you can refer:Go Slices: usage and internals
Related
I need to populate a struct that has a member of type [8]uint8. This needs be populated with a byte array of type []byte initialized to length 8. The simplistic approach does not work:
Data: [8]uint8(RequestFrame(0x180, r)),
gives
cannot convert .. (type []byte) to type [8]uint8
Since both arrays are structurally identical it would be nice if this could be done with casting/assignment rather than copying?
Background
The problem with your "simplistic approach" is that a slice
(of any type) is a struct-typed value consisting of a pointer
and two integers; the pointer contains the address of the
underlying (backing) data array, and the integers contain
what len() and cap() builtins return for that slice.
In other words, a slice is sort of a view into an array.
Then, in Go, there is no concept of a type cast; there are only
type conversions, and these conversions may only happen between
types with the same underlying representation¹.
Since a slice and an array may not have the same underlying
representation (array is literally a contiguous block of memory
of the size just enough to contain all the array's elements),
your alleged type conversion may not be legal.
Possible solutions
There are two possible solutions.
The simplest is to just copy the data from the slice's
backing array into a newly-allocated array:
var (
src = []byte{1, 2, 3, 4, 5, 6, 7, 8}
dst [8]uint8
)
copy(dst[:], src[:8])
Note that there exists an inherent disparity between slice an
array types: an array type encodes both the type of its elements
and its length (that is, the length is a part of the type),
while a slice type only encodes the type of its elements
(and may be of any length at runtime).
This means that you might need to have a check before such
copying that makes sure the source slice has exactly 8
elements, that is, len(src) == len(dst).
This invariant may be enforced by some other code, but I think
I'd warn you up front about this: if src has less than 8
elements, the src[:8] expression will panic at runtime,
and if it contains more, then there's the question of whether
copying just the first 8 of them is exactly what's needed.
The second approach (admittedly messier) is to just re-use
the underlying array of the slice:
import "unsafe"
var (
src = []byte{1, 2, 3, 4, 5, 6, 7, 8}
dstPtr *[8]uint8
)
if len(src) != len(*dstPtr) {
panic("boom")
}
dstPtr = (*[8]uint8)(unsafe.Pointer(&src[0]))
Here, we've just taken the address of the first element
contained in the slice's underlying array and peformed
a "dirty" two-phase type-conversion to make the obtained
pointer to be of type *[8]uint8—that is, "an address of
an array of 8 uint8s".
Note two caveats:
The resulting pointer now points to
the same memory block the original slice does.
It means it's now possible to mutate that memory both through the
slice and the pointer we obtained.
As soon as you'll decide to assign the array's data
to a variable of type [8]uint8 (and passing it as an argument
to a function's parameter of that type), you will dereference
that pointer (like with *dstPtr), and at that moment
the array's data will be copied.
I'm specifically mentioning this as often people resort
to hacks like this one to pull the backing array out of
a slice precisely in an attempt to not copy the memory.
TL;DR
Copy the data (after supposedly verifying the
len(src) == len(dst) invariant holds).
Copying 8 bytes is fast (on a typical 64-bit CPU this will be
a single MOV instruction, or two at most), and the code will
be straightforward.
Only resort to hacks from the second solution when you really
need to optimize on some critical hot path.
In that case, comment the solution extensively and watch for
not accidentally dereferencing your pointer.
¹ There are notable exceptions to this rule:
A []byte is type-convertible to string, and vice-versa.
A string is type-convertible to []rune, and vice-versa.
An int is type-convertible to string (but since Go 1.15 go vet gives a warning about it, and this feature may probably be prohibited in the future).
You can copy the contents of your byte slice into your uint8 array very simply by using copy, like this:
package main
import (
"fmt"
)
func main() {
slice := []byte{1, 2, 3, 4, 5, 6, 7, 8}
array := [8]uint8{}
copy(array[:], slice)
fmt.Println(array)
}
Outputs
[1 2 3 4 5 6 7 8]
Try it out on the playground.
But may I ask why you are using an array? It's usually better to just use slices, unless you have a really good reason.
Starting from Go 1.17 you are able to use type conversion directly, from a slice to an array pointer:
a := make([]byte, 8)
b := (*[8]uint8)(a) // b is pointer to [8]uint8
The you can just dereference to obtain a non-pointer [8]uint8 type.
a := make([]byte, 8)
b := *(*[8]uint8)(a) // b is [8]uint8
Notes:
unlike copy, the conversion approach does not incur in extra allocations (not yours, nor any possibly done by copy), because it simply yields a pointer to the existing backing array. Though dereferencing the array pointer will make a copy.
the conversion panics if the length of the array is greater than the slice's
a := make([]byte, 5)
b := (*[10]byte)(a) // panics
the pointer points to the slice's underlying array, therefore the same values will be visible by indexing either:
a := []byte{0xa1, 0xa2}
b := (*[2]uint8)(a)
fmt.Printf("%x\n", a[0]) // a1
b[0] = 0xff
fmt.Printf("%x\n", a[0]) // ff
you can convert from byte to uint8, including type literals derived from them, because byte is an alias (identical to) of uint8.
Related: How do you convert a slice into an array?
I'm wondering about best practices when initializing empty arrays.
i.e. Is there any difference here between arr1, arr2, and arr3?
myArr1 := []int{}
myArr2 := make([]int,0)
var myArr3 []int
I know that they make empty []int but I wonder, is one syntax preferable to the others? Personally I find the first to be most readable but that's beside the point here. One key point of contention may be the array capacity, presumably the default capacity is the same between the three as it is unspecified. Is declaring arrays of unspecified capacity "bad"? I can assume it comes with some performance cost but how "bad" is it really?
/tldr:
Is there any difference between the 3 ways to make an empty
array?
What is the default capacity of an array when unspecified?
What is the performance cost of using arrays with unspecified capacity?
First, it's a slice not an array. Arrays and slices in Go are very different, arrays have a fixed size that is part of the type. I had trouble with this at first too :)
Not really. Any if the three is correct, and any difference should be too small to worry about. In my own code I generally use whatever is easiest in a particular case.
0
Nothing, until you need to add an item, then whatever it costs to allocate the storage needed.
What is the performance cost of using arrays with unspecified capacity?
There is certainly a cost when you start populating the slice. If you know how big the slice should grow, you can allocate capacity of the underlying array from the very begging as opposed to reallocating every time the underlying array fills up.
Here is a simple example with timing:
package main
import "fmt"
func main() {
limit := 500 * 1000 * 1000
mySlice := make([]int, 0, limit) //vs mySlice := make([]int, 0)
for i := 0; i < limit; i++ {
mySlice = append(mySlice, i)
}
fmt.Println(len(mySlice))
}
On my machine:
time go run my_file.go
With preallocation:
real 0m2.129s
user 0m2.073s
sys 0m1.357s
Without preallocation
real 0m7.673s
user 0m9.095s
sys 0m3.462s
Is there any difference between the 3 ways to make an empty array?
if empty array means len(array)==0, the answer is no, but actually only myArr3==nil is true.
What is the default capacity of an array when unspecified?
the default capacity will be same with the len you specify.
What is the performance cost of using arrays with unspecified capacity?
none
I have a program in which I'm going to make lots and lots of slices, some of which might be empty:
nb := something() // something might return 0
slices = append(slices, make([]int, nb))
Does make([]int, 0) allocates some memory and is, thus, less memory efficient than a nil slice although they share the same behavior ? By how much ?
If so, is it worth doing a test to avoid useless allocations, or is the CPU time cost of the test not worth the saving in memory (or any other reason not to do so) ?
var sl slice
nb := something()
if nb > 0 {
sl = make([]int, nb)
}
slices = append(slices, sl)
There is no difference in the allocated memory between
var a []T // nil slice
and
a := make([]T, 0) // zero-length, zero-capacity, non-nil slice
The difference is in the slice header content. In the second case the slice pointer contains some fixed address, same for all 0-sized allocations.
If this code is in a performance critical part of the program, the difference makes ... quite a difference. In the first case you do zero the slice header, in the second case you go through 3-4 function calls, some range checks for cap and length, etc. before malloc returns a pointer to the zero base.
Does make([]int, 0) allocates some memory
Yes, it allocates a slice header but no backing array. If the slice header doesn't escape the current scope it may be allocated on the stack.
less memory efficient than a nil slice
In terms of memory used, they're the same.
is it worth doing a test to avoid useless allocations
In general yes, the 3 or 4 instructions it takes to compare an int are nothing compared to the cycles you'd need to do a memory allocation and initialization.
What is the difference between:
x := make([]int, 5, 10)
x := make([]int, 5)
x := [5]int{}
I know that make allocates an array and returns a slice that refers to that array. I don't understand where it can be used?
I can't find a good example that will clarify the situation.
x := make([]int, 5) Makes slice of int with length 5 and capacity 5 (same as length).
x := make([]int, 5, 10) Makes slice of int with length 5 and capacity 10.
x := [5]int{} Makes array of int with length 5.
Slices
If you need to append more items than capacity of slice using append function, go runtime will allocate new underlying array and copy existing one to it. So if you know about estimated length of your slice, better to use explicit capacity declaration. It will consume more memory for underlying array at the beginning, but safe cpu time for many allocations and array copying.
You can explore how len and cap changes while append, using that simple test on Go playground
Every time when cap value changed, new array allocated
Arrays
Array size is fixed, so if you need to grow array you have to create new one with new length and copy your old array into it by your own.
There are some great articles about slices and arrays in go:
http://blog.golang.org/go-slices-usage-and-internals
http://blog.golang.org/slices
The second line will allocate 10 int's worth memory at the very beginning, but returning you a slice of 5 int's. The second line does not stand less memory, it saves you another memory allocation if you need to expand the slice to anything not more than 10 * load_factor.
I've been trying out Go for some time and this question keeps bugging me. Say I build up a somewhat large dataset in a slice (say, 10 million int64s).
package main
import (
"math"
"fmt"
)
func main() {
var a []int64
var i int64;
upto := int64(math.Pow10(7))
for i = 0; i < upto; i++ {
a = append(a, i)
}
fmt.Println(cap(a))
}
But then I decide I don't want most of them so I want to end up with a slice of just 10 of those. I've tried both slicing and delete techniques on Go's wiki but none of them seem to reduce the slice's capacity.
So that's my question: does Go has no real way of shrinking the capacity of a slice that would be similar to realloc()-ing with a smaller size argument than in your previous call on the same pointer in C? Is that an issue and how should one deal with it?
To perform an, in effect, a realloc of a slice:
a = append([]T(nil), a[:newSize]...) // Thanks to #Dijkstra for pointing out the missing ellipsis.
If it does a copy of newSize elements to a new memory place or if it does an actual in place resize as in realloc(3) is at complete discretion of the compiler. You might want to investigate the current state and perhaps raise an issue if there's a room for improvement in this.
However, this is likely a micro-optimization. The first source of performance enhancements lies almost always in selecting a better algorithm and/or a better data structure. Using a hugely sized vector to finally keep a few items only is probably not the best option wrt to memory consumption.
EDIT: The above is only partially correct. The compiler cannot, in the general case, derive if there are other pointers to the slice's backing array. Thus the realloc is not applicable. The above snippet is actually guaranteed to peform a copy of 'newSize' elements. Sorry for any confusion possibly created.
Go does not have a way of shrinking slices. This isn't a problem in most cases, but if you profile your memory use and find you're using too much, you can do something about it:
Firstly, you can just create a slice of the size you need and copy your data into it. The garbage collector will then free the large slice. Copy built-in
Secondly, you could re-use the big slice each time you wish to generate it, so you never allocate it more than once.
On a final note, you can use 1e7 instead of math.Pow10(7).
Let's see this example:
func main() {
s := []string{"A", "B", "C", "D", "E", "F", "G", "H"}
fmt.Println(s, len(s), cap(s)) // slice, length, capacity
t := s[2:4]
fmt.Println(t, len(t), cap(t))
u := make([]string, len(t))
copy(u, t)
fmt.Println(u, len(u), cap(u))
}
It produces the following output:
[A B C D E F G H] 8 8
[C D] 2 6
[C D] 2 2
s is a slice that holds 8 pieces of strings. t is a slice that keeps the part [C D]. The length of t is 2, but since it uses the same hidden array of s, its capacity is 6 (from "C" to "H"). The question is: how to have a slice of [C D] that is independent from the hidden array of s? Simply create a new slice of strings with length 2 (slice u) and copy the content of t to u. u's underlying hidden array is different from the hidden array of s.
The initial problem was this: you have a big slice and you create a new smaller slice on it. Since the smaller slice uses the same hidden array, the garbage collector won't delete the hidden array.
See the bottom of this post for more info: http://blog.golang.org/go-slices-usage-and-internals .
Additionally you can re-use most of the allocated memory during work of yours app, take a look at: bufs package
PS if you re-alocate new memory for smaller slice, old memory may not be freed in same time, it will be freed when garbage collector decides to.
You can do that by re-assigning the slice's value to a portion of itself
a := []int{1,2,3}
fmt.Println(len(a), a) // 3 [1 2 3]
a = a[:len(a)-1]
fmt.Println(len(a), a) //2 [1 2]
There is a new feature called 3-index slice in Go 1.2, which means to get part of a slice in this way:
slice[a:b:c]
In which the len for the returned slice whould be b-a, and the cav for the new slice would be c-a.
Tips: no copy is down in the whole process, it only returns a new slice which points to &slice[a] and has the len as b-a and cav as c-a.
And that's the only thing you have to do:
slice= slice[0:len(slice):len(slice)];
Then the cav of the slice would be changed to len(slice) - 0, which is the same as the len of it, and no copy is done.