Golang: Problems when using append on slice - go

I am using golang. Here is my code:
func main() {
quanPailie([]int{1, 2})
}
func quanPailie(nums []int) [][]int {
COUNT := len(nums)
//only one item
if COUNT == 1 {
return [][]int{nums}
}
insertItem(quanPailie(nums[:COUNT-1]), nums[COUNT-1])
return [][]int{}
}
func insertItem(res [][]int, insertNum int) {
fmt.Println("insertItem,res:", res, "insertNum", insertNum) //insertItem,res: [[1]] insertNum 2
for _, v := range res {
for i := 0; i < len(v); i++ {
fmt.Println("===before,v:", v)
c := append(v[:i], append([]int{insertNum}, v[i:]...)...)
fmt.Println("===after,v:", v)
fmt.Println("ccc", c)
}
}
}
What makes me very confused is the output:
===before,v: [1]
===after,v: [2]
Why did the value of v change? Hope someone can help me. Thanks a lot.
Go playground: https://play.golang.org/p/wITYsGpX7U
EDIT:
Thanks for icza's great help, I think I have understood this problem.
And, here is a simple code to show this issue.
func test1() {
nums := []int{1, 2, 3}
_ = append(nums[:2], 4)
fmt.Println("test1:", nums)
//nums changes because the cap is big enought, the original array is modified.
}
func test2() {
nums := []int{1, 2, 3}
c := append(nums[:2], []int{4, 5, 6}...)
fmt.Println("test2:", nums)
fmt.Println("cc:", c)
//nums dont't change because the cap isn't big enought.
//a new array is allocated while the nums still points to the old array.
//Of course, the return value of append points to the new array.
}
Go playground: https://play.golang.org/p/jBNFsCqUn3

This is the code in question:
fmt.Println("===before,v:", v)
c := append(v[:i], append([]int{insertNum}, v[i:]...)...)
fmt.Println("===after,v:", v)
You ask why v changes between the 2 Println() statements.
Because you are using the builtin append() function, quoting from its doc:
The append built-in function appends elements to the end of a slice. If it has sufficient capacity, the destination is resliced to accommodate the new elements. If it does not, a new underlying array will be allocated. Append returns the updated slice.
So if the slice you append to has enough room (capacity) to accomodate the elements you want to append, no new slice will be allocated, instead the destination slice will be re-sliced (which will use the same underlying array) and append will happen in that.
Let's check the capacity:
fmt.Println("===before,v:", v, cap(v))
c := append(v[:i], append([]int{insertNum}, v[i:]...)...)
fmt.Println("===after,v:", v, cap(v))
Output:
===before,v: [1] 2
===after,v: [2] 2
The v slice has a capacity of 2. When for loop starts, i=0, v[:i] is v[:0] which is an empty slice (but has capacity 2) and so appending 1 or 2 elements will not allocate a new array/slice, it will be done "in place". This "in place" is the 0th element of v, since v[:i] is shorthand for v[0:i]. Hence the elements will be appended starting from v[0] in the underlying array which is shared, so the element denoted by v[0] will change.
Note that slicing a slice results in a slice which shares its underlying backing array with the original slice (does not make a copy of the elements).
If you want to avoid this, use or allocate a new slice, copy original content and append to the new slice, e.g.:
src := []int{1, 2}
c := make([]int, len(src))
copy(c, src)
// Append something:
c = append(c, 3, 4)
fmt.Println(src) // [1 2] - src doesn't change
fmt.Println(c) // [1 2 3 4]

Related

How to implement remove range on string array golang?

How can I implement RemoveRange method in golang? It is a method in C# as shown here
I want to implement RemoveRange method on my hashCode string array and return new modified array back if possible with those ranges remove.
func removeRange(hashCode []string, idx int, count int) []string {
var temp []string
for i, s := range hashCode {
fmt.Println(i, s)
// confuse here on what to do
}
return temp
}
Simply slice the slice up until idx, skip count elements and append the rest to the result of the first slicing:
func removeRange(hashCode []string, idx int, count int) []string {
return append(hashCode[:idx], hashCode[idx+count:]...)
}
Testing it:
s := []string{"0", "1", "2", "3", "4", "5"}
fmt.Println(s)
s = removeRange(s, 1, 2)
fmt.Println(s)
Which outputs (try it on the Go Playground):
[0 1 2 3 4 5]
[0 3 4 5]
Note: the above implementation does not check whether indices are valid (whether they are in range). If not, the code could panic. Add necessary checks if you need to.
Note #2: the above implementation modifies the elements of the passed slice, the returned slice will share the backing array of the parameter. If you want to avoid this, if you want to leave the input intact and allocate a new slice for the result, then do so:
func removeRange(hashCode []string, idx int, count int) []string {
result := make([]string, 0, len(hashCode)-count)
result = append(result, hashCode[:idx]...)
result = append(result, hashCode[idx+count:]...)
return result
}
Try this one on the Go Playground.
You don't need a method or function for this at all in golang. Go slices can be subsliced and appended in place, which is how you can quickly and easily remove subsets from any slice.
Say you want to remove 2 elements, starting at index 2, you'd simply write:
Sub := append(original [:2], original [4:]...)
Demo
How this works:
original[:2] creates a sub-slice starting at 0, with a length of 2 elements (so index 0 and 1)
append because to this first part, we want to add the rest of the slice, minus the range we want to skip/remove
original[4:] creates another sub-slice, this time starting at index 4, and ending wherever original ends. Just like we don't explicitly mention 0 as the starting point in the first sub-slice, by not specifying a number of elements here, golang will just include all of the remaining elements in the slice.
... because append is a variadic function (built-in, but you get the point), we need to pass in every element we want to append as a new argument. The ... operator expands the sub-slice and passes in every element as a separate argument.
Because we assigned the new slice to a new variable, original will remain unchanged, so if you want to overwrite the slice, you just assign it to the same variable.
Note I wrote this on my phone, so markup and code may not be quite right, but this should answer your question at least
I've explained the code using // comments and if not commented, code is self explanatory.
package main
import (
"fmt"
"os"
)
func RemoveRange(s []string, index, count int) []string {
sLen := len(s)
// Similar semantics to match (similar) the behavior of
// C# implementation
switch {
case index < 0, count < 0: // arguments are not valid
fmt.Fprintln(os.Stderr, "error: argument out of range error")
return s
case index+count-1 >= sLen: // range results in exceeding the limit
fmt.Fprintln(os.Stderr, "error: argument error")
return s
}
// Create a slice p and pre-allocate the size required
// to store the resultant slice after removing range.
// Result := s[:] -> s[:index] + s[index+count:]
// Remove := s[index:index+count-1]
p := make([]string, 0, sLen-count)
p = append(p, s[:index]...)
p = append(p, s[index+count:]...)
return p
}
func main() {
s := []string{"0", "1", "2", "3", "4", "5"}
fmt.Println(s)
r := RemoveRange(s, 1, 3)
fmt.Println(r)
}
Output:
[0 1 2 3 4 5]
[0 4 5]

Flipping a slice with a for loop logic error

So I am trying to write a method that takes two slices, flips both of them and then gives them to each other.
Ex.
s1 = {1,2,3,4,5}
s2 = {6,7,8,9,10}
Should return:
s1 = {10,9,8,7,6}
s2 = {5,4,3,2,1}
Here is my code:
package main
import(
"fmt"
)
func main(){
f:= [5]int{1,2,3,4,5}
h:= [5]int{6,7,8,9,10}
var sliceF []int = f[0:5]
var sliceH []int = h[0:5]
fmt.Println(reverseReverse(sliceF,sliceH))
}
func reverseReverse(first []int, second []int) ([]int, []int){
//creating temp arrays to hold the traversed arrays before swapping.
var tempArr1 []int = first
var tempArr2 []int = second
//count is used for counting up the tempArrays in the correct order in the For loops
var count int= 0
//goes through the first array and sets the values starting from the end equal to the temp array
//which increases normally from left to right.
for i :=len(first)-1; i>=0;i--{
tempArr1[count] = first[i]
fmt.Println(i)
count++
}
count =0
//same as first for loop just on the second array
for i :=len(second)-1; i>=0;i--{
tempArr2[count] = second[i]
count++
}
//trying to replace the values of the param arrays to be equal to the temp arrays
first=tempArr2
second = tempArr1
//returning the arrays
return first,second
}
When run here is the output:
4
3
2
1
0
[10 9 8 9 10]
[5 4 3 4 5]
*Not I included a print statement in the for loop to check if the index is decreasing properly.
I understand there are better ways to do this but for proof of concept I want to use a for loop.
Any help appreciated. I am new to go and tend to have java habits so I assume somehow my problem is related to that.
This can be done much simpler by realizing there's no need to actually swap the individual elements. Instead, reverse each array and swap their order. Much simpler!
func reverseReverse( a, b []int ) ([]int, []int) {
return reverse(b), reverse(a)
}
func reverse( a []int ) []int {
end := len(a) - 1
// Allocate a new array slice of the same length to copy to.
ret := make( []int, len(a) )
// Copy each element of a into ret, reversed.
for i := range a {
ret[end-i] = a[i]
}
return ret
}
With that revelation, there's little need for the very specialized reverseReverse function. Swap the order yourself.
fmt.Println(reverse(sliceH), reverse(sliceF))
Note that if you just want to take a slice of an array, it's sufficient to write sliceH []int := h[:] without specifying the start and end. The start is assumed to be 0 and the end is the end. Also note there's no need to declare the type, := takes care of that for you.
Even better, you can declare and initialize them directly.
sliceF:= []int{1,2,3,4,5}
sliceH:= []int{6,7,8,9,10}
Short answer:
tempArr1[count] = first[i]
This line is logically identical to:
first[count] = first[i]
Detailed answer:
x := [5]int{} and x := []int{} are in fact two very different assignments. In the first case x is actually a static array. In the second case x is a slice which is in fact a data structure which has a length, capacity and a pointer to the underlying array. Therefore, var tempArr1 []int = first means copy the pointer to the underlying array of first into the tempArr1, so any modification to first[i] will be reflected in tempArr1 and vice versa
For example,
package main
import "fmt"
func reverse(s []int) []int {
for i := 0; i < len(s)/2; i++ {
s[i], s[len(s)-1-i] = s[len(s)-1-i], s[i]
}
return s
}
func main() {
s1, s2 := []int{1, 2, 3, 4, 5}, []int{6, 7, 8, 9, 10}
fmt.Println(s1, s2)
s1, s2 = reverse(s2), reverse(s1)
fmt.Println(s1, s2)
}
Output:
[1 2 3 4 5] [6 7 8 9 10]
[10 9 8 7 6] [5 4 3 2 1]

What is the mechanism of using append to prepend in Go?

Suppose I have a slice slice of type int. While declaring, I set the third argument to size, which I believe reserves memory for at least size ints by setting the cap parameter of the slice.
slice:=make([]int,0,size)
Now, suppose I have an integer variable value. To add it to the slice at the end, I use
slice=append(slice,value)
If the number of elements currently in the slice is less than size, then there will be no need to copy the entire underlying array to a new location in order to add the new element.
Further, if I want to prepend value to slice, as suggested here and here, I use
slice=append([]int{value},slice...)
My question is, what happens in this case? If the number of elements is still less than size, how are the elements stored in the memory? Assuming a contiguous allocation when the make() function was invoked, are all existing elements right shifted to free the first space for value? Or is memory reallocated and all elements copied?
The reason for asking is that I would like my program to be as fast as possible, and would like to know if this is a possible cause for slowing it down. If it is so, is there any alternative way of prepending that would be more time efficient?
With reslicing and copying
The builtin append() always appends elements to a slice. You cannot use it (alone) to prepend elements.
Having said that, if you have a slice capacity bigger than length (has "free" space after its elements) to which you want to prepend an element, you may reslice the original slice, copy all elements to an index one higher to make room for the new element, then set the element to the 0th index. This will require no new allocation. This is how it could look like:
func prepend(dest []int, value int) []int {
if cap(dest) > len(dest) {
dest = dest[:len(dest)+1]
copy(dest[1:], dest)
dest[0] = value
return dest
}
// No room, new slice need to be allocated:
// Use some extra space for future:
res := make([]int, len(dest)+1, len(dest)+5)
res[0] = value
copy(res[1:], dest)
return res
}
Testing it:
s := make([]int, 0, 5)
s = append(s, 1, 2, 3, 4)
fmt.Println(s)
s = prepend(s, 9)
fmt.Println(s)
s = prepend(s, 8)
fmt.Println(s)
Output (try it on the Go Playground):
[1 2 3 4]
[9 1 2 3 4]
[8 9 1 2 3 4]
Note: if no room for the new element, since performance does matter now, we didn't just do:
res := append([]int{value}, dest...)
Because it does more allocations and copying than needed: allocates a slice for the literal ([]int{value}), then append() allocates a new when appending dest to it.
Instead our solution allocates just one new array (by make(), even reserving some space for future growth), then just set value as the first element and copy dest (the previous elements).
With linked list
If you need to prepend many times, a normal slice may not be the right choice. A faster alternative would be to use a linked list, to which prepending an element requires no allocations of slices / arrays and copying, you just create a new node element, and you designate it to be the root by pointing it to the old root (first element).
The standard library provides a general implementation in the container/list package.
With manually managing a larger backing array
Sticking to normal slices and arrays, there is another solution.
If you're willing to manage a larger backing array (or slice) yourself, you can do so by leaving free space before the slice you use. When prepending, you can create a new slice value from the backing larger array or slice which starts at an index that leaves room for 1 element to be prepended.
Without completeness, just for demonstration:
var backing = make([]int, 15) // 15 elements
var start int
func prepend(dest []int, value int) []int {
if start == 0 {
// No more room for new value, must allocate bigger backing array:
newbacking := make([]int, len(backing)+5)
start = 5
copy(newbacking[5:], backing)
backing = newbacking
}
start--
dest = backing[start : start+len(dest)+1]
dest[0] = value
return dest
}
Testing / using it:
start = 5
s := backing[start:start] // empty slice, starting at idx=5
s = append(s, 1, 2, 3, 4)
fmt.Println(s)
s = prepend(s, 9)
fmt.Println(s)
s = prepend(s, 8)
fmt.Println(s)
// Prepend more to test reallocation:
for i := 10; i < 15; i++ {
s = prepend(s, i)
}
fmt.Println(s)
Output (try it on the Go Playground):
[1 2 3 4]
[9 1 2 3 4]
[8 9 1 2 3 4]
[14 13 12 11 10 8 9 1 2 3 4]
Analysis: this solution makes no allocations and no copying when there is room in the backing slice to prepend the value! All that happens is it creates a new slice from the backing slice that covers the destination +1 space for the value to be prepended, sets it and returns this slice value. You can't really get better than this.
If there is no room, then it allocates a larger backing slice, copies over the content of the old, and then does the "normal" prepending.
With tricky slice usage
Idea: imagine that you always store elements in a slice in backward order.
Storing your elements in backward order in a slice means a prepand becomes append!
So to "prepand" an element, you can simply use append(s, value). And that's all.
Yes, this has its limited uses (e.g. append to a slice stored in reverse order has the same issues and complexity as a "normal" slice and prepand operation), and you lose many conveniences (ability to list elements using for range just to name one), but performance wise nothing beats prepanding a value just by using append().
Note: iterating over the elements that stores elements in backward order has to use a downward loop, e.g.:
for i := len(s) - 1; i >= 0; i-- {
// do something with s[i]
}
Final note: all these solutions can easily be extended to prepend a slice instead of just a value. Generally the additional space when reslicing is not +1 but +len(values), and not simply setting dst[0] = value but instead a call to copy(dst, values).
The "prepend" call will need to allocate an array and copy all elements because a slice in Go is defined as a starting point, a size and an allocation (with the allocation being counted from the starting point).
There is no way a slice can know that the element before the first one can be used to extend the slice.
What will happen with
slice = append([]int{value}, slice...)
is
a new array of a single element value is allocated (probably on stack)
a slice is created to map this element (start=0, size=1, alloc=1)
the append call is done
append sees that there is not enough room to extend the single-element slice so allocates a new array and copies all the elements
a new slice object is created to refer to this array
If appending/removing at both ends of a large container is the common use case for your application then you need a deque container. It is unfortunately unavailable in Go and impossible to write efficiently for generic contained types while maintaining usability (because Go still lacks generics).
You can however implement a deque for your specific case and this is easy (for example if you have a large container with a known upper bound may be a circular buffer is all you need and that is just a couple of lines of code away).
I'm very new to Go, so may be the following is very bad Go code... but it's an attempt to implement a deque using a growing circular buffer (depending on the use case this may be or may be not a good solution)
type Deque struct {
buffer []interface{}
f, b, n int
}
func (d *Deque) resize() {
new_buffer := make([]interface{}, 2*(1+d.n))
j := d.f
for i := 0; i < d.n; i++ {
new_buffer[i] = d.buffer[j]
d.buffer[j] = nil
j++
if j == len(d.buffer) {
j = 0
}
}
d.f = 0
d.b = d.n
d.buffer = new_buffer
}
func (d *Deque) push_back(x interface{}) {
if d.n == len(d.buffer) {
d.resize()
}
d.buffer[d.b] = x
d.b++
if d.b == len(d.buffer) {
d.b = 0
}
d.n++
}
func (d *Deque) push_front(x interface{}) {
if d.n == len(d.buffer) {
d.resize()
}
if d.f == 0 {
d.f = len(d.buffer)
}
d.f--
d.buffer[d.f] = x
d.n++
}
func (d *Deque) pop_back() interface{} {
if d.n == 0 {
panic("Cannot pop from an empty deque")
}
if d.b == 0 {
d.b = len(d.buffer)
}
d.b--
x := d.buffer[d.b]
d.buffer[d.b] = nil
d.n--
return x
}
func (d *Deque) pop_front() interface{} {
if d.n == 0 {
panic("Cannot pop from an empty deque")
}
x := d.buffer[d.f]
d.buffer[d.f] = nil
d.f++
if d.f == len(d.buffer) {
d.f = 0
}
d.n--
return x
}

unexpected slice append behaviour

I encountered weird behaviour in go code today: when I append elements to slice in loop and then try to create new slices based on the result of the loop, last append overrides slices from previous appends.
In this particular example it means that sliceFromLoop j,g and h slice's last element are not 100,101 and 102 respectively, but...always 102!
Second example - sliceFromLiteral behaves as expected.
package main
import "fmt"
func create(iterations int) []int {
a := make([]int, 0)
for i := 0; i < iterations; i++ {
a = append(a, i)
}
return a
}
func main() {
sliceFromLoop()
sliceFromLiteral()
}
func sliceFromLoop() {
fmt.Printf("** NOT working as expected: **\n\n")
i := create(11)
fmt.Println("initial slice: ", i)
j := append(i, 100)
g := append(i, 101)
h := append(i, 102)
fmt.Printf("i: %v\nj: %v\ng: %v\nh:%v\n", i, j, g, h)
}
func sliceFromLiteral() {
fmt.Printf("\n\n** working as expected: **\n")
i := []int{0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10}
fmt.Println("initial slice: ", i)
j := append(i, 100)
g := append(i, 101)
h := append(i, 102)
fmt.Printf("i: %v\nj: %v\ng: %v\nh:%v\n", i, j, g, h)
}
link to play.golang:
https://play.golang.org/p/INADVS3Ats
After some reading, digging and experimenting I found that this problem is originated in slices referencing the same underlaying array values and can be solved by copying slice to new one before appending anything, however it looks quite... hesitantly.
What's the idomatic way for creating many new slices based on old ones and not worrying about changing values of old slices?
Don't assign append to anything other than itself.
As you mention in the question, the confusion is due to the fact that append both changes the underlying array and returns a new slice (since the length might be changed). You'd imagine that it copies that backing array, but it doesn't, it just allocates a new slice object that points at it. Since i never changes, all those appends keep changing the value of backingArray[12] to a different number.
Contrast this to appending to an array, which allocates a new literal array every time.
So yes, you need to copy the slice before you can work on it.
func makeFromSlice(sl []int) []int {
result := make([]int, len(sl))
copy(result, sl)
return result
}
func main() {
i := make([]int, 0)
for ii:=0; ii<11; ii++ {
i = append(i, ii)
}
j := append(makeFromSlice(i), 100) // works fine
}
The slice literal behavior is explained because a new array is allocated if the append would exceed the cap of the backing array. This has nothing to do with slice literals and everything to do with the internals of how exceeding the cap works.
a := []int{1,2,3,4,5,6,7}
fmt.Printf("len(a) %d, cap(a) %d\n", a, len(a), cap(a))
// len(a) 7, cap(a) 7
b := make([]int, 0)
for i:=1; i<8, i++ {
b = append(b, i)
} // b := []int{1,2,3,4,5,6,7}
// len(b) 7, cap(b) 8
b = append(b, 1) // any number, just so it hits cap
i := append(b, 100)
j := append(b, 101)
k := append(b, 102) // these work as expected now
If you need a copy of a slice, there's no other way to do it other than, copying the slice. You should almost never assign the result of append to a variable other than the first argument of append. It leads to hard to find bugs, and will behave differently depending on whether the slice has the required capacity or not.
This isn't a commonly needed pattern, but as with all things of this nature if you need to repeate a few lines of code multiple times, then you can use a small helper function:
func copyAndAppend(i []int, vals ...int) []int {
j := make([]int, len(i), len(i)+len(vals))
copy(j, i)
return append(j, vals...)
}
https://play.golang.org/p/J99_xEbaWo
There is also a little bit simpler way to implement copyAndAppend function:
func copyAndAppend(source []string, items ...string) []string {
l := len(source)
return append(source[:l:l], items...)
}
Here we just make sure that source has no available capacity and so copying is forced.

Do Clearing slices in golang guarantees garbage collection?

I wanted to implement time based slots for holding data using golang slices. I managed to come up with a go program like this and it also works. But I have few questions regarding garbage collection and the general performance of this program. Does this program guarantee garbage collection of items once slice is equated to nil? And while shuffling slices, I hope this program does not do any deep copying.
type DataSlots struct {
slotDuration int //in milliseconds
slots [][]interface{}
totalDuration int //in milliseconds
}
func New(slotDur int, totalDur int) *DataSlots {
dat := &DataSlots{slotDuration: slotDur,
totalDuration: totalDur}
n := totalDur / slotDur
dat.slots = make([][]interface{}, n)
for i := 0; i < n; i++ {
dat.slots[i] = make([]interface{}, 0)
}
go dat.manageSlots()
return dat
}
func (self *DataSlots) addData(data interface{}) {
self.slots[0] = append(self.slots[0], data)
}
// This should be a go routine
func (self *DataSlots) manageSlots() {
n := self.totalDuration / self.slotDuration
for {
time.Sleep(time.Duration(self.slotDuration) * time.Millisecond)
for i := n - 1; i > 0; i-- {
self.slots[i] = self.slots[i-1]
}
self.slots[0] = nil
}
}
I removed critical section handling in this snippet to make it concise.
Once your slice is set too nil, any values contained in the slice are available for garbage collection, provided that the underlying array isn't shared with another slice.
Since there are no slice operations in your program, you never have multiple references to the same array, nor are you leaving data in any inaccessible portions of the underlying array.
What you need to be careful of, is when you're using slice operations:
a := []int{1, 2, 3, 4}
b := a[1:3]
a = nil
// the values 1 and 4 can't be collected, because they are
// still contained in b's underlying array
c := []int{1, 2, 3, 4}
c = append(c[1:2], 5)
// c is now []int{2, 5}, but again the values 1 and 4 are
// still in the underlying array. The 4 may be overwritten
// by a later append, but the 1 is inaccessible and won't
// be collected until the underlying array is copied.
While append does copy values when the capacity of the slice in insufficient, only the values contained in the slice are copied. There is no deep copy of any of the values.

Resources