Why can't I duplicate a slice with `copy()`? - go

I need to make a copy of a slice in Go and reading the docs there is a copy function at my disposal.
The copy built-in function copies elements from a source slice into a
destination slice. (As a special case, it also will copy bytes from a
string to a slice of bytes.) The source and destination may overlap.
Copy returns the number of elements copied, which will be the minimum
of len(src) and len(dst).
But when I do:
arr := []int{1, 2, 3}
tmp := []int{}
copy(tmp, arr)
fmt.Println(tmp)
fmt.Println(arr)
My tmp is empty as it was before (I even tried to use arr, tmp):
[]
[1 2 3]
You can check it on go playground. So why can not I copy a slice?

The builtin copy(dst, src) copies min(len(dst), len(src)) elements.
So if your dst is empty (len(dst) == 0), nothing will be copied.
Try tmp := make([]int, len(arr)) (Go Playground):
arr := []int{1, 2, 3}
tmp := make([]int, len(arr))
copy(tmp, arr)
fmt.Println(tmp)
fmt.Println(arr)
Output (as expected):
[1 2 3]
[1 2 3]
Unfortunately this is not documented in the builtin package, but it is documented in the Go Language Specification: Appending to and copying slices:
The number of elements copied is the minimum of len(src) and len(dst).
Edit:
Finally the documentation of copy() has been updated and it now contains the fact that the minimum length of source and destination will be copied:
Copy returns the number of elements copied, which will be the minimum of len(src) and len(dst).

Another simple way to do this is by using append which will allocate the slice in the process.
arr := []int{1, 2, 3}
tmp := append([]int(nil), arr...) // Notice the ... splat
fmt.Println(tmp)
fmt.Println(arr)
Output (as expected):
[1 2 3]
[1 2 3]
As pointed out in the comments below, append may allocate excess memory if the slice isn't sized correctly to begin with. A nice solution to this is to preallocate a slice of the right capacity, like so:
tmp := append(make([]int, 0, len(arr)), arr...)
So a shorthand for copying array arr would be append(make([]int, 0, len(arr)), arr...)
https://play.golang.org/p/xwevI1chGrd

The copy() runs for the least length of dst and src, so you must initialize the dst to the desired length.
A := []int{1, 2, 3}
B := make([]int, 3)
copy(B, A)
C := make([]int, 2)
copy(C, A)
fmt.Println(A, B, C)
Output:
[1 2 3] [1 2 3] [1 2]
You can initialize and copy all elements in one line using append() to a nil slice.
x := append([]T{}, []...)
Example:
A := []int{1, 2, 3}
B := append([]int{}, A...)
C := append([]int{}, A[:2]...)
fmt.Println(A, B, C)
Output:
[1 2 3] [1 2 3] [1 2]
Comparing with allocation+copy(), for greater than 1,000 elements, use append. Actually bellow 1,000 the difference may be neglected, make it a go for rule of thumb unless you have many slices.
BenchmarkCopy1-4 50000000 27.0 ns/op
BenchmarkCopy10-4 30000000 53.3 ns/op
BenchmarkCopy100-4 10000000 229 ns/op
BenchmarkCopy1000-4 1000000 1942 ns/op
BenchmarkCopy10000-4 100000 18009 ns/op
BenchmarkCopy100000-4 10000 220113 ns/op
BenchmarkCopy1000000-4 1000 2028157 ns/op
BenchmarkCopy10000000-4 100 15323924 ns/op
BenchmarkCopy100000000-4 1 1200488116 ns/op
BenchmarkAppend1-4 50000000 34.2 ns/op
BenchmarkAppend10-4 20000000 60.0 ns/op
BenchmarkAppend100-4 5000000 240 ns/op
BenchmarkAppend1000-4 1000000 1832 ns/op
BenchmarkAppend10000-4 100000 13378 ns/op
BenchmarkAppend100000-4 10000 142397 ns/op
BenchmarkAppend1000000-4 2000 1053891 ns/op
BenchmarkAppend10000000-4 200 9500541 ns/op
BenchmarkAppend100000000-4 20 176361861 ns/op

If your slices were of the same size, it would work:
arr := []int{1, 2, 3}
tmp := []int{0, 0, 0}
i := copy(tmp, arr)
fmt.Println(i)
fmt.Println(tmp)
fmt.Println(arr)
Would give:
3
[1 2 3]
[1 2 3]
From "Go Slices: usage and internals":
The copy function supports copying between slices of different lengths (it will copy only up to the smaller number of elements)
The usual example is:
t := make([]byte, len(s), (cap(s)+1)*2)
copy(t, s)
s = t

The best way to clone as slice is
sClone = append(s[:0:0], s...)
This implementation has two advantage:
make sure that the result sClone is nil if s is nil, and is not nil
if s is not nil.
No need to import the containing package of type T even if T is declared in
another package

The Go Programming Language Specification
Appending to and copying slices
The function copy copies slice elements from a source src to a
destination dst and returns the number of elements copied. Both
arguments must have identical element type T and must be assignable to
a slice of type []T. The number of elements copied is the minimum of
len(src) and len(dst). As a special case, copy also accepts a
destination argument assignable to type []byte with a source argument
of a string type. This form copies the bytes from the string into the
byte slice.
copy(dst, src []T) int
copy(dst []byte, src string) int
tmp needs enough room for arr. For example,
package main
import "fmt"
func main() {
arr := []int{1, 2, 3}
tmp := make([]int, len(arr))
copy(tmp, arr)
fmt.Println(tmp)
fmt.Println(arr)
}
Output:
[1 2 3]
[1 2 3]

Sweet, Simple, Performant, No need to be careful of length, No Memory overlap, Different copies
slice2 := append([]int{}, slice1...)

NOTE: This is an incorrect solution as #benlemasurier proved
Here is a way to copy a slice. I'm a bit late, but there is a simpler, and faster answer than #Dave's. This are the instructions generated from a code like #Dave's. These is the instructions generated by mine. As you can see there are far fewer instructions. What is does is it just does append(slice), which copies the slice. This code:
package main
import "fmt"
func main() {
var foo = []int{1, 2, 3, 4, 5}
fmt.Println("foo:", foo)
var bar = append(foo)
fmt.Println("bar:", bar)
bar = append(bar, 6)
fmt.Println("foo after:", foo)
fmt.Println("bar after:", bar)
}
Outputs this:
foo: [1 2 3 4 5]
bar: [1 2 3 4 5]
foo after: [1 2 3 4 5]
bar after: [1 2 3 4 5 6]

If you don't care about the speed:
import "golang.org/x/exp/slices"
tmp := slices.Clone(arr)
With Go 1.18 and generics, any slices now could be copied with slices.Clone from package "golang.org/x/exp/slices". Playground

Just do benchmark for those three methods which implement slice copy
with append on CloneWithAppend
with copy on CloneWithCopy
with append for generic any on CloneWithAny
func CloneWithAppend(b []byte) []byte {
if b == nil {
return nil
}
return append([]byte{}, b...)
}
func CloneWithCopy(b []byte) []byte {
if b == nil {
return nil
}
tmp := make([]byte, len(b))
copy(tmp, b)
return tmp
}
func CloneWithAny[B ~[]T, T any](b B) B {
if b == nil {
return nil
}
return append([]T{}, b...)
}
Benchmark codes
var testSlice = []byte("qwertyuiopasdfghjklzxcvbnmQWERTYUIOPASDFGHJKLZXCVBNM098765432112345678901234567890")
func BenchmarkCloneWithAppend(b *testing.B) {
for i := 0; i < b.N; i++ {
CloneWithAppend(testSlice)
}
}
func BenchmarkCloneWithCopy(b *testing.B) {
for i := 0; i < b.N; i++ {
CloneWithCopy(testSlice)
}
}
func BenchmarkCloneWithAny(b *testing.B) {
for i := 0; i < b.N; i++ {
CloneWithAny(testSlice)
}
}
Results
goarch: amd64
pkg: test
cpu: Intel(R) Core(TM) i7-8850H CPU # 2.60GHz
BenchmarkCloneWithAppend-12 28700232 41.50 ns/op
BenchmarkCloneWithCopy-12 32453222 30.98 ns/op
BenchmarkCloneWithAny-12 31737926 41.68 ns/op
It seems the with copy method has better performance.
Note, func Clone([]uint8) []uint8 of pkg bytes would be added in Golang next release per this commit and related proposal bytes, strings: add Clone
// Clone returns a copy of b[:len(b)].
// The result may have additional unused capacity.
// Clone(nil) returns nil.
func Clone(b []byte) []byte {
if b == nil {
return nil
}
return append([]byte{}, b...)
}

Related

How to sort a slice in go without mutating the original slice

In Golang, I am trying to sort a slice without mutating the original value
func main() {
originalArray := []int{4, 2, 1, 1, 2}
newArray := originalArray
sort.Ints(newArray)
fmt.Printf("%v", originalArray) // prints [1 1 2 2 4]
}
How can I sort a slice in golang without mutating the original value?
You need to make a copy of the original slice.
Use:
newArray := make([]int, len(originalArray))
copy(newArray, originalArray)
or:
newArray := append([]int{}, originalArray...)

Change golang slice in another function

I have a slice, if i remove one element from it directly in a main function the length of slice would be cut by one. But do the remove in another function and called it in main, the length of the slice is still keep origin. Who can explain it for me? Thanks!
package main
import "fmt"
func main() {
a := []int{1, 2, 3, 4}
i := 0
//copy(a[i:], a[i+1:])
//a[len(a)-1] = 0
//a = a[:len(a)-1]
//fmt.Println(a) //outputs: [2 3 4], this is correct
f(a, i)
fmt.Println(a) //outputs: [2 3 4 0], this is wrong!
}
func f(a []int, i int) {
copy(a[i:], a[i+1:])
a[len(a)-1] = 0
a = a[:len(a)-1]
fmt.Println(a) //outputs: [2 3 4], here still correct
}
Go Playground Link
The slice is passed by value, so changing it in your function f won't change it in function main. You can pass by pointer, like this:
package main
import "fmt"
func main() {
a := []int{1, 2, 3, 4}
i := 0
f(&a, i)
fmt.Println(a) //outputs: [2 3 4], correct
}
func f(a *[]int, i int) {
b := *a
copy(b[i:], b[i+1:])
// The following line seems pointless, but ok...
b[len(b)-1] = 0
b = b[:len(b)-1]
fmt.Println(b) //outputs: [2 3 4], here still correct
*a = b
}
Go Playground
As suggested by #zerkms in the comments, you could also return the new slice, avoiding the use of pointers:
package main
import "fmt"
func main() {
a := []int{1, 2, 3, 4}
i := 0
a = f(a, i)
fmt.Println(a)
}
func f(a []int, i int) []int {
copy(a[i:], a[i+1:])
// The following line seems pointless, but ok...
a[len(a)-1] = 0
a = a[:len(a)-1]
fmt.Println(a) //outputs: [2 3 4], here still correct
return a
}
Not providing new solution, just trying to explain why your program is behaving the way you asked:
Let us try to understand first how the built in function ‘copy’ works

Ref: [https://golang.org/pkg/builtin/#copy]
func copy(dst, src []Type) int
The copy built-in function copies elements from a source slice into a destination slice. (As a special case, it also will copy bytes from a string to a slice of bytes.) The source and destination may overlap. Copy returns the number of elements copied, which will be the minimum of len(src) and len(dst).
Two things:
1. 
First comment the line : //a[len(a)-1] = 0
Second: As you are using the same array i.e as source and destination you are getting [2,3,4,4] as output as the destination array is {1,2,3,4} which got overwritten to {2,3,4,4(which is already present)}

you can try with different array’s to make it more clear to you

How does Go slice capacity change on append? [duplicate]

This question already has answers here:
How the slice is enlarged by append? Is the capacity always doubled?
(2 answers)
Closed 3 years ago.
Running the example The Go Tour on server (currently on version 1.12.7), I find the capacity of slice doubling to the next power of 2, if the new slice length is larger than current backing array's length.
If I run the same program on my machine (version 1.10.3 on windows), the slice capacity changes to next multiple of two.
Why are they different? Is it because of Go version or run-time implementations? Is the capacity change deterministic?
The output on remote server is this
len=0 cap=0 []
len=1 cap=2 [0]
len=2 cap=2 [0 1]
len=5 cap=8 [0 1 2 3 4]
The output on local machine is this
len=0 cap=0 []
len=1 cap=1 [0]
len=2 cap=2 [0 1]
len=5 cap=6 [0 1 2 3 4]
This is the code for reference
package main
import "fmt"
func main() {
var s []int
printSlice(s)
// append works on nil slices.
s = append(s, 0)
printSlice(s)
// The slice grows as needed.
s = append(s, 1)
printSlice(s)
// We can add more than one element at a time.
s = append(s, 2, 3, 4)
printSlice(s)
}
func printSlice(s []int) {
fmt.Printf("len=%d cap=%d %v\n", len(s), cap(s), s)
}
TL;DR: It depends on the size of the elements stored in the array
The implementation can be seen here:
https://github.com/golang/go/blob/master/src/runtime/slice.go
But as you can see looking into the history it can't be relied to stay the same over time.
That might also explain the difference you may note on different versions of Go.
Making some tests show how a 0-size struct will increment capacity by just 1 element, and int or string will duplicate on each growth, and a 3-byte struct "roughly" doubles on each growth.
You can execute code like this using different types to see these different cases in action:
arr := []struct{}{}
oldCap := 0
for i := 0; i < 100; i++ {
arr = append(arr, struct{}{})
if cap(arr) != oldCap {
oldCap = cap(arr)
fmt.Println("arr", cap(arr))
}
}
Playground showing the cases described above:
https://play.golang.org/p/OKtCFskbp2t
According to information from Go slice internal The implementation behaviour of append is as follows.
It is just a multiple of the (len(source slice) + len(new data)) * 2
func AppendByte(slice []byte, data ...byte) []byte {
m := len(slice)
n := m + len(data)
if n > cap(slice) { // if necessary, reallocate
// allocate double what's needed, for future growth.
newSlice := make([]byte, (n+1)*2)
copy(newSlice, slice)
slice = newSlice
}
slice = slice[0:n]
copy(slice[m:n], data)
return slice
}

Does slice assignment in Go copy memory

Purpose: I have a big buffer, and I would like to have an array/slice of pointer pointing to different loc in the buffer.
What I am doing:
datPtrs := make([][]byte, n)
for i:=0; i<n; i++{
datPtrs[i] = bigBuf[i*m:(i+1)*m]
}
My Question:
Will this copy memory? My guess is not, but I cannot find anywhere to confirm this.
What is the best way/tool to find out whether there is memory copy or not?
Go slices are implemented as a struct:
src/runtime/slice.go:
type slice struct {
array unsafe.Pointer
len int
cap int
}
You are assigning/copying the slice struct, which does not copy the underlying array, only its pointer.
A simple illustration:
package main
import (
"fmt"
)
func main() {
buf := make([]byte, 8)
for i := range buf {
buf[i] = byte(i)
}
sub := buf[1:3]
fmt.Println(buf)
fmt.Println(sub)
for i := range sub {
sub[i] += 43
}
fmt.Println(buf)
fmt.Println(sub)
}
Playground: https://play.golang.org/p/4OzPwuNmUlY
Output:
[0 1 2 3 4 5 6 7]
[1 2]
[0 44 45 3 4 5 6 7]
[44 45]
See The Go Blog: Go Slices: usage and internals,
No
Slice is just a pointer to memory + len and cap
see: Why can not I duplicate a slice with `copy()` in Golang?
Like so:
package main
import (
"fmt"
)
func main() {
bigBuf := []byte{1, 2, 3, 4, 5}
datPtrs := make([][]byte, 2)
for i := 0; i < 2; i++ {
datPtrs[i] = bigBuf[i : i+1]
}
fmt.Println(bigBuf) // [1 2 3 4 5]
datPtrs[0][0] = 10
fmt.Println(bigBuf) // [10 2 3 4 5]
datPtrs[1][0] = 20
fmt.Println(bigBuf) // [10 20 3 4 5]
}

Golang: Problems when using append on slice

I am using golang. Here is my code:
func main() {
quanPailie([]int{1, 2})
}
func quanPailie(nums []int) [][]int {
COUNT := len(nums)
//only one item
if COUNT == 1 {
return [][]int{nums}
}
insertItem(quanPailie(nums[:COUNT-1]), nums[COUNT-1])
return [][]int{}
}
func insertItem(res [][]int, insertNum int) {
fmt.Println("insertItem,res:", res, "insertNum", insertNum) //insertItem,res: [[1]] insertNum 2
for _, v := range res {
for i := 0; i < len(v); i++ {
fmt.Println("===before,v:", v)
c := append(v[:i], append([]int{insertNum}, v[i:]...)...)
fmt.Println("===after,v:", v)
fmt.Println("ccc", c)
}
}
}
What makes me very confused is the output:
===before,v: [1]
===after,v: [2]
Why did the value of v change? Hope someone can help me. Thanks a lot.
Go playground: https://play.golang.org/p/wITYsGpX7U
EDIT:
Thanks for icza's great help, I think I have understood this problem.
And, here is a simple code to show this issue.
func test1() {
nums := []int{1, 2, 3}
_ = append(nums[:2], 4)
fmt.Println("test1:", nums)
//nums changes because the cap is big enought, the original array is modified.
}
func test2() {
nums := []int{1, 2, 3}
c := append(nums[:2], []int{4, 5, 6}...)
fmt.Println("test2:", nums)
fmt.Println("cc:", c)
//nums dont't change because the cap isn't big enought.
//a new array is allocated while the nums still points to the old array.
//Of course, the return value of append points to the new array.
}
Go playground: https://play.golang.org/p/jBNFsCqUn3
This is the code in question:
fmt.Println("===before,v:", v)
c := append(v[:i], append([]int{insertNum}, v[i:]...)...)
fmt.Println("===after,v:", v)
You ask why v changes between the 2 Println() statements.
Because you are using the builtin append() function, quoting from its doc:
The append built-in function appends elements to the end of a slice. If it has sufficient capacity, the destination is resliced to accommodate the new elements. If it does not, a new underlying array will be allocated. Append returns the updated slice.
So if the slice you append to has enough room (capacity) to accomodate the elements you want to append, no new slice will be allocated, instead the destination slice will be re-sliced (which will use the same underlying array) and append will happen in that.
Let's check the capacity:
fmt.Println("===before,v:", v, cap(v))
c := append(v[:i], append([]int{insertNum}, v[i:]...)...)
fmt.Println("===after,v:", v, cap(v))
Output:
===before,v: [1] 2
===after,v: [2] 2
The v slice has a capacity of 2. When for loop starts, i=0, v[:i] is v[:0] which is an empty slice (but has capacity 2) and so appending 1 or 2 elements will not allocate a new array/slice, it will be done "in place". This "in place" is the 0th element of v, since v[:i] is shorthand for v[0:i]. Hence the elements will be appended starting from v[0] in the underlying array which is shared, so the element denoted by v[0] will change.
Note that slicing a slice results in a slice which shares its underlying backing array with the original slice (does not make a copy of the elements).
If you want to avoid this, use or allocate a new slice, copy original content and append to the new slice, e.g.:
src := []int{1, 2}
c := make([]int, len(src))
copy(c, src)
// Append something:
c = append(c, 3, 4)
fmt.Println(src) // [1 2] - src doesn't change
fmt.Println(c) // [1 2 3 4]

Resources