My question is about slice length and capacity. I'm learning about Go here: https://tour.golang.org/moretypes/11.
(My question was marked as a possible duplicate of this; however, this is not the case. My question is specifically about the cutting off the first few elements of a slice and the implications of that.)
Why does the line s = s[2:] decrease the capacity when s = s[:4] and s = s[:0] do not? The only difference I see is that there is a number before the colon in s = s[2:] while there is a number after the colon in the other two lines.
Is there any way to recover the first two elements that we cut off with s = s[2:]?
package main
import "fmt"
func main() {
s := []int{2, 3, 5, 7, 11, 13}
printSlice(s)
// Slice the slice to give it zero length.
s = s[:0]
printSlice(s)
// Extend its length.
s = s[:4]
printSlice(s)
// Drop its first two values.
s = s[2:]
printSlice(s)
}
func printSlice(s []int) {
fmt.Printf("len=%d cap=%d %v\n", len(s), cap(s), s)
}
After clicking the Run button, we get the following.
len=6 cap=6 [2 3 5 7 11 13]
len=0 cap=6 []
len=4 cap=6 [2 3 5 7]
len=2 cap=4 [5 7]
You can read more about slices here. But I think this passage answers your question:
Slicing does not copy the slice's data. It creates a new slice value that points to the original array. This makes slice operations as efficient as manipulating array indices. Therefore, modifying the elements (not the slice itself) of a re-slice modifies the elements of the original slice.
So you cannot recover the slice data if you are assigning it to the same variable.
The capacity decrease is because by dropping the first 2 elements you are changing the pointer to the new slice (slices are referenced by the pointer to the first element).
How slices are represented in the memory:
make([]byte, 5)
s = s[2:4]
You can use a full slice expression:
package main
func main() {
s := []int{2, 3, 5, 7, 11, 13}
{ // example 1
t := s[:0]
println(cap(t) == 6)
}
{ // example 2
t := s[:0:0]
println(cap(t) == 0)
}
}
https://golang.org/ref/spec#Slice_expressions
The slices.Clip of "golang.org/x/exp/slices" could reduce the capacity of slice through Full slice expressions.
Clip removes unused capacity from the slice, returning s[:len(s):len(s)].
func main() {
s := []int{2, 3, 5, 7, 11, 13}
printSlice(s)
s = s[:4]
printSlice(s)
s = slices.Clip(s)
printSlice(s)
}
func printSlice(s []int) {
fmt.Printf("len=%d cap=%d %v\n", len(s), cap(s), s)
}
len=6 cap=6 [2 3 5 7 11 13]
len=4 cap=6 [2 3 5 7]
len=4 cap=4 [2 3 5 7]
Playground
Related
I am going through the tour of Go and wanted to know the following:
package main
import "fmt"
func main() {
s := []int{2, 3, 5, 7, 11, 13}
printSlice(s)
// Drop its last two values
s = s[:len(s)-2]
printSlice(s)
// Drop its first two values.
s = s[2:]
printSlice(s)
}
func printSlice(s []int) {
fmt.Printf("len=%d cap=%d %v\n", len(s), cap(s), s)
}
Result:
len=6 cap=6 [2 3 5 7 11 13]
len=4 cap=6 [2 3 5 7]
len=2 cap=4 [5 7]
Why does the capacity of the slice stay the same when you drop the last 2 items but changes when you drop the first 2?
https://play.golang.org/p/ZNKwOYKDqOi
Go slices are implemented as a struct:
src/runtime/slice.go:
type slice struct {
array unsafe.Pointer
len int
cap int
}
Revise your printSlice function to show the pointer to the underlying array:
package main
import "fmt"
func main() {
s := []int{2, 3, 5, 7, 11, 13}
printSlice(s)
// Drop its last two values
s = s[:len(s)-2]
printSlice(s)
// Drop its first two values.
s = s[2:]
printSlice(s)
}
func printSlice(s []int) {
var ptr *int
if cap(s) >= 1 {
ptr = &s[:cap(s)][0]
}
fmt.Printf("ptr=%p len=%d cap=%d %v\n", ptr, len(s), cap(s), s)
}
Playground: https://play.golang.org/p/pk3cpE_LsUV
Output:
ptr=0x450000 len=6 cap=6 [2 3 5 7 11 13]
ptr=0x450000 len=4 cap=6 [2 3 5 7]
ptr=0x450008 len=2 cap=4 [5 7]
See how your slicing operations adjust the pointer, the length, and the capacity. A slice is simply a view of or window into the underlying array.
References:
The Go Blog: Go Slices: usage and internals
The Go Programming Language Specification:
Slice types
Slice expressions
A slice has a buffer under the covers. The capacity is the size of that buffer. Removing items from the end does not alter the buffer size.
You are creating a new slice from an existing slice but starting the index of the new slice at 2. The new slice still points to the same underlying buffer (but offset by 2). The buffer size (and capacity) for this new slice is thus 2 less.
Capacity makes sense when you see how append works. Append tries to reuse the same underlying buffer - but if the capacity is filled, then a reallocation and copy is performed.
I have the following code which causes a weird result. I cannot understand why:
func main() {
var s = []int{2, 3}
var s1 = append(s, 4)
var a = append(s1, 5)
var b = append(s1, 6)
fmt.Println(s)
fmt.Println(s1)
fmt.Println(a)
fmt.Println(b)
}
This then results in:
[2 3]
[2 3 4]
[2 3 4 6]
[2 3 4 6]
My question is: why a is not [2 3 4 5] but [2 3 4 6]? I know append to b changes a, but how. Is this a bug because I never changed a directly and obviously I don't want this happen?
Keep in mind that a slice is a structure of 3 fields.
a pointer to the underlying array
length of the slice
capacity of the slice
append() function may either modify its argument in-place or return a copy of its argument with an additional entry, depending on the size and capacity of its input. append() function creates a new slice, if the length the slice is greater than the length of the array pointed by the slice.
If I have a Python list, it is simple to duplicate the last element: l.append(l[-1]). Is there an elegant way to do this with Go slices?
The best I have so far is data = append(data, data[len(data) - 1]).
That is the proper way to do it.
In Go you can't use negative indices, so the index of the last element is len(data) -1.
And in Go append() is a builtin function and not a method of slices, and it returns a new slice value which you have to assign or store if you need the extended slice, so there's nothing you can make shorter in your code in general.
Note that if you were on to create your own type, you could attach methods to it which could simpify things. For example:
type ints []int
func (i *ints) append(es ...int) {
*i = append(*i, es...)
}
func (i *ints) appendLast() {
*i = append(*i, (*i)[len(*i)-1])
}
Using it:
is := ints{1, 2, 3}
fmt.Println(is)
is.append(4, 5) // Append some elements
fmt.Println(is)
is.append(is[len(is)-1]) // Append last element explicitly
fmt.Println(is)
is.appendLast() // Append last element using appendLast method
fmt.Println(is)
// You can also use it for []int values, with conversion:
is2 := []int{1, 2, 3}
(*ints)(&is2).appendLast()
fmt.Println(is2)
Output (try it on the Go Playground):
[1 2 3]
[1 2 3 4 5]
[1 2 3 4 5 5]
[1 2 3 4 5 5 5]
[1 2 3 3]
You can also use simple util functions without creating a new type:
func appendLast(i []int) []int {
return append(i, i[len(i)-1])
}
func appendLast2(i *[]int) {
*i = append(*i, (*i)[len(*i)-1])
}
func main() {
is := []int{1, 2, 3}
fmt.Println(is)
is = appendLast(is)
fmt.Println(is)
appendLast2(&is)
fmt.Println(is)
}
Output (try it on the Go Playground):
[1 2 3]
[1 2 3 3]
[1 2 3 3 3]
data = append(data, data[len(data) - 1])
Yes. It's the best way to do in Go. It's not possible to use a negative index in Go.
I tried to append a new element to a slice, just like appending a new element to a list in Python, but there is an error
package main
import "fmt"
func main() {
s := []int{2, 3, 5, 7, 11, 13}
printSlice(s)
s[len(s)] = 100
printSlice(s)
}
func printSlice(s []int) {
fmt.Printf("%v\n", s)
}
and the output is
[2 3 5 7 11 13]
panic: runtime error: index out of range
How can I append a new element to a slice then?
Is it correct that slice in Go is the closet data type to list in Python?
Thanks.
Appending in Go uses the builtin method append. For example:
s := []byte{1, 2, 3}
s = append(s, 4, 5, 6)
fmt.Println(s)
// Output: [1 2 3 4 5 6]
You are correct, slices in Go are close to Python's list, and, in fact, many of the same indexing and slicing operations work on them.
When you do:
s[len(s)] = 100
it panics because you cannot access a value beyond the length of the slice. There appears to be some confusion here, because in Python this works the exact same way. Attempting the above in Python (unless you've already expanded the list) will give:
IndexError: list assignment index out of range
Go's append() function only allocates new slice data, when the capacity of the given slice is not sufficient (see also: https://stackoverflow.com/a/28143457/802833). This can lead to unexpected behavior (at least for me as a golang newbie):
package main
import (
"fmt"
)
func main() {
a1 := make([][]int, 3)
a2 := make([][]int, 3)
b := [][]int{{1, 1, 1}, {2, 2, 2}, {3, 3, 3}}
common1 := make([]int, 0)
common2 := make([]int, 0, 12) // provide sufficient capacity
common1 = append(common1, []int{10, 20}...)
common2 = append(common2, []int{10, 20}...)
idx := 0
for _, k := range b {
a1[idx] = append(common1, k...) // new slice is allocated
a2[idx] = append(common2, k...) // no allocation
idx++
}
fmt.Println(a1)
fmt.Println(a2) // surprise!!!
}
output:
[[10 20 1 1 1] [10 20 2 2 2] [10 20 3 3 3]]
[[10 20 3 3 3] [10 20 3 3 3] [10 20 3 3 3]]
https://play.golang.org/p/8PEqFxAsMt
So, what ist the (idomatic) way in Go to force allocation of new slice data or more precisely to make sure that the slice argument to append() remains unchanged?
You might maintain a wrong idea of how slices work in Go.
When you append elements to a slice, the call to append() returns a new slice. If reallocation did not happen, both slice values — the one you called append() on and the one it returned back — share the same backing array but they will have different lengths; observe:
package main
import "fmt"
func main() {
a := make([]int, 0, 10)
b := append(a, 1, 2, 3)
c := append(a, 4, 3, 2)
fmt.Printf("a=%#v\nb=%#v\nc=%#v\n", a, b, c)
}
outputs:
a=[]int{}
b=[]int{4, 3, 2}
c=[]int{4, 3, 2}
So, len(a) == 0, len(b) == 3, len(c) == 3, and the second call to append() owerwrote what the first one did because all the slices share the same underlying array.
As to reallocation of the backing array, the spec is clear:
If the capacity of s is not large enough to fit the additional values, append allocates a new, sufficiently large underlying array that fits both the existing slice elements and the additional values. Otherwise, append re-uses the underlying array.
From this, it follows that:
append() never copies the underlying storage if the capacity of the slice being appeneded to is sufficient.
If there's not enough capacity, the array will be reallocated.
That is, given a slice s to which you want to append N elements, the reallocation won't be done iff cap(s) - len(s) ≥ N.
Hence I suspect your problem is not about unexpected reallocation results but rather about the concept of slices as implemented in Go. The code idea to absorb is that append() returns the resulting slice value, which you're supposed to be using after the call unless you fully understand the repercussions.
I recommend starting with this to fully understand them.
Thanx for your feedback.
So the solution to gain control of the memory allocation is to do it explicitely (which remembers me that Go is a more a system language than other (scripting) langs):
package main
import (
"fmt"
)
func main() {
a1 := make([][]int, 3)
a2 := make([][]int, 3)
b := [][]int{{1, 1, 1}, {2, 2, 2}, {3, 3, 3}}
common1 := make([]int, 0)
common2 := make([]int, 0, 12) // provide sufficient capacity
common1 = append(common1, []int{10, 20}...)
common2 = append(common2, []int{10, 20}...)
idx := 0
for _, k := range b {
a1[idx] = append(common1, k...) // new slice is allocated
a2[idx] = make([]int, len(common2), len(common2)+len(k))
copy(a2[idx], common2) // copy & append could probably be
a2[idx] = append(a2[idx], k...) // combined into a single copy step
idx++
}
fmt.Println(a1)
fmt.Println(a2)
}
output:
[[10 20 1 1 1] [10 20 2 2 2] [10 20 3 3 3]]
[[10 20 1 1 1] [10 20 2 2 2] [10 20 3 3 3]]
https://play.golang.org/p/Id_wSZwb84