Does the internal operation to slice influence the original slice outside? - go

For example:
func test(requiredIp []int, i int) []int {
requiredIp = append(requiredIp[0:i], requiredIp[i+1:]...)
return requiredIp
}
func main(){
requiredIp := []int{1,2,4,5,6}
fmt.Println(test(requiredIp,0)) // output:[2 4 5 6]
fmt.Println("original", requiredIp) // output:[2 4 5 6,6]
}
Why original slice has two 6 ?

A slice is a struct type with three fields:
A pointer to (an address of) the underlying array holding the data.
The length ­— how many elements are there in the slice.
The capacity — how many elements it's possible to store into the underlying array without reallocating it.
As everything in Go, slices are passed by value. This happens when you assign a slice value to a variable or when you pass it as a parameter in a function/method call.
What is pased by value (that is, copied) is that structure with three fields.
The pointer which is copied, obviously points at the same data block as the one in the original slice.
What happens in your code is the following:
The original slice is []int{1,2,4,5,6}.
It has length and capacity equal to 5.
It is passed to a functon, test.
The slice available there via the function's parameter requiredIp is initially identical to the one which is passed in a call.
You reslice that slice by evaluating requiredIp[0:i], and since in your call i equals to 0, you evaluate requiredIp[0:0].
That expression creates a slice with the backing array and the capacity of the original slice and length 0.
You then reslice the original slice once again — with the expression requiredIp[i+1:], which, in your call is requiredIp[1:].
The result shares the backing array with the original, has the capacity 4 and the contents []int{2,4,5,6}.
You then append the slice obtained on the previous step to the one obtained in the step before the previous one. That's where it gets interesting.
Consider that the slice being appended to points at the 0th element of the original slice and has length 0 and capacity 5. It means it has room for 5 elements.
That is, the slice's backing array still holds [1, 2, 4, 5, 6].
The slice being appended shares the same backing array with the slice being appended to, just it points at the 1st element, not the 0th.
The code of append sees it's told to append 4 elements and checks to see whether the target slice has enough capacity to hold them, and it has.
So append merely copies [2, 4, 5, 6] from and to the same backing array, overwriting 4 elements starting from index 0 in it. Effectively, it's elements [2, 4, 5, 6] moved one element left.
The resulting backing array now contains [2, 4, 5, 6, 6]: the first 4 element are the 4 elements which were the last, moved by one element left, overwriting what there were at indices 0 through 3.
You now return the reslting slice to the caller. That new slice value shares the backing array with all the slice values involved in the example, but remember that it has its length set to 4 — because the append appended 4 elements to the slice of length 0.
You print the original slice and the one returned from your function. They differ only in their length: the original has it equals to 5, and the returned from the function — to 4. The rest of their fields are the same: they both have capacity 5 and share the same backing array by both pointing its 0th element.
The difference in the lengths explains why "the original" slice seemingly has "extra" 6 in it. In fact in has it exactly where it were left off before test was called.
If, for some reason you really wanted to "detach" the slice produced by test from the original one by forcing it to allocate the new backing array, there are several possibilities:
Merely append to an unallocated slice — for instance,
append([]int(nil), 1, 2, 3, 4) would allocate a fresh backing array.
In your particular case that would not really be a solution as with i > 0 the append would have to operate on a non-empty slice. This can be dealt, for exmaple, with two appends:
s := append([]int(nil), input[0:i]...)
s = append(s, input[i+1:]...)
…or with allocating a new slice and copying:
s := make([]int, len(input)-1)
copy(s, input[0:i])
copy(s[i:], input[i+1:])
When reslicing the original slice, be sure to also artifically reset its capacity:
return append(input[0:i:i], input[i+1:]...)
Here, the first slice would have the same capacity as its length, i, and appending even a single element to it would force the append to allocate a new backing array, copy over these i elements to it and then copy over what is being appended.
Further reading:
https://blog.golang.org/slices-intro
https://blog.golang.org/slices
https://blog.golang.org/strings
…and actually consider starting with Effective Go.

Related

Golang append changing append parameter [duplicate]

This question already has answers here:
Why does append modify passed slice
(5 answers)
Closed 10 months ago.
When running the following program:
package main
import "fmt"
func main() {
edges := [][]int{{1,2}, {2,3}, {3,4}, {1,4}, {1,5}}
printSlice2d(edges)
_ = append(edges[:0], edges[1:]...)
printSlice2d(edges)
}
func printSlice2d(s [][]int) {
fmt.Printf("len=%d cap=%d %v\n", len(s), cap(s), s)
}
I get the following output:
len=5 cap=5 [[1 2] [2 3] [3 4] [1 4] [1 5]]
len=5 cap=5 [[2 3] [3 4] [1 4] [1 5] [1 5]]
I don't understand why edges is being changed by calling append. I would expect the first and second lines to be the same. I've checked the specification, but I can't seem to find anything that would explain this behaviour this kind of input.
Slice uses array internally to store the data.
append function does not create a new array if it is not necessary to increase array size beyond its current capacity. Instead it copies new data elements into existing array. After that function returns reference to a new slice that internally uses the same array.
You can read more in this article - https://blog.golang.org/go-slices-usage-and-internals
As mentioned earlier, re-slicing a slice doesn't make a copy of the
underlying array. The full array will be kept in memory until it is no
longer referenced. Occasionally this can cause the program to hold all
the data in memory when only a small piece of it is needed.
This is what is going on in this line:
_ = append(edges[:0], edges[1:]...)
second append function argument (edges[1:]...) copies 4 items of edges into temp var. Its value - [{2,3}, {3,4}, {1,4}, {1,5}]
these values are copied into array that edges uses internally to store data. That overrides all items except of last one. This is where edges is mutated.
append returns reference to a new slice that internally uses the same array to store the data as edges
returned slice is ignored and will be garbage collected, but that does not matter for edges.
That is why you see changed values when you check edges after performing append on it.
edges[:0] is a slice of length 0 starting at index 0 of the underlying array of edges.
To this slice, you append another slice, the slice of length 4 starting at index of the underlying array ofedges`. That gives you the first 4 elements of the result you see in the second line.
Then you print edges, which is a slice with an underlying array of 5 elements, whose last 4 you just shifted one element lower. The last element is duplicated. in the edges array.
If you look at the result of the append, then you'd see a slice with length 4, cap 5, the first 4 elements of the underlying array of edges.
If you expected the two lines to be the same, maybe you tried to do:
append(edges[:1],edges[1:]...)

How can I cast []byte to [8]uint8

I need to populate a struct that has a member of type [8]uint8. This needs be populated with a byte array of type []byte initialized to length 8. The simplistic approach does not work:
Data: [8]uint8(RequestFrame(0x180, r)),
gives
cannot convert .. (type []byte) to type [8]uint8
Since both arrays are structurally identical it would be nice if this could be done with casting/assignment rather than copying?
Background
The problem with your "simplistic approach" is that a slice
(of any type) is a struct-typed value consisting of a pointer
and two integers; the pointer contains the address of the
underlying (backing) data array, and the integers contain
what len() and cap() builtins return for that slice.
In other words, a slice is sort of a view into an array.
Then, in Go, there is no concept of a type cast; there are only
type conversions, and these conversions may only happen between
types with the same underlying representation¹.
Since a slice and an array may not have the same underlying
representation (array is literally a contiguous block of memory
of the size just enough to contain all the array's elements),
your alleged type conversion may not be legal.
Possible solutions
There are two possible solutions.
The simplest is to just copy the data from the slice's
backing array into a newly-allocated array:
var (
src = []byte{1, 2, 3, 4, 5, 6, 7, 8}
dst [8]uint8
)
copy(dst[:], src[:8])
Note that there exists an inherent disparity between slice an
array types: an array type encodes both the type of its elements
and its length (that is, the length is a part of the type),
while a slice type only encodes the type of its elements
(and may be of any length at runtime).
This means that you might need to have a check before such
copying that makes sure the source slice has exactly 8
elements, that is, len(src) == len(dst).
This invariant may be enforced by some other code, but I think
I'd warn you up front about this: if src has less than 8
elements, the src[:8] expression will panic at runtime,
and if it contains more, then there's the question of whether
copying just the first 8 of them is exactly what's needed.
The second approach (admittedly messier) is to just re-use
the underlying array of the slice:
import "unsafe"
var (
src = []byte{1, 2, 3, 4, 5, 6, 7, 8}
dstPtr *[8]uint8
)
if len(src) != len(*dstPtr) {
panic("boom")
}
dstPtr = (*[8]uint8)(unsafe.Pointer(&src[0]))
Here, we've just taken the address of the first element
contained in the slice's underlying array and peformed
a "dirty" two-phase type-conversion to make the obtained
pointer to be of type *[8]uint8—that is, "an address of
an array of 8 uint8s".
Note two caveats:
The resulting pointer now points to
the same memory block the original slice does.
It means it's now possible to mutate that memory both through the
slice and the pointer we obtained.
As soon as you'll decide to assign the array's data
to a variable of type [8]uint8 (and passing it as an argument
to a function's parameter of that type), you will dereference
that pointer (like with *dstPtr), and at that moment
the array's data will be copied.
I'm specifically mentioning this as often people resort
to hacks like this one to pull the backing array out of
a slice precisely in an attempt to not copy the memory.
TL;DR
Copy the data (after supposedly verifying the
len(src) == len(dst) invariant holds).
Copying 8 bytes is fast (on a typical 64-bit CPU this will be
a single MOV instruction, or two at most), and the code will
be straightforward.
Only resort to hacks from the second solution when you really
need to optimize on some critical hot path.
In that case, comment the solution extensively and watch for
not accidentally dereferencing your pointer.
¹ There are notable exceptions to this rule:
A []byte is type-convertible to string, and vice-versa.
A string is type-convertible to []rune, and vice-versa.
An int is type-convertible to string (but since Go 1.15 go vet gives a warning about it, and this feature may probably be prohibited in the future).
You can copy the contents of your byte slice into your uint8 array very simply by using copy, like this:
package main
import (
"fmt"
)
func main() {
slice := []byte{1, 2, 3, 4, 5, 6, 7, 8}
array := [8]uint8{}
copy(array[:], slice)
fmt.Println(array)
}
Outputs
[1 2 3 4 5 6 7 8]
Try it out on the playground.
But may I ask why you are using an array? It's usually better to just use slices, unless you have a really good reason.
Starting from Go 1.17 you are able to use type conversion directly, from a slice to an array pointer:
a := make([]byte, 8)
b := (*[8]uint8)(a) // b is pointer to [8]uint8
The you can just dereference to obtain a non-pointer [8]uint8 type.
a := make([]byte, 8)
b := *(*[8]uint8)(a) // b is [8]uint8
Notes:
unlike copy, the conversion approach does not incur in extra allocations (not yours, nor any possibly done by copy), because it simply yields a pointer to the existing backing array. Though dereferencing the array pointer will make a copy.
the conversion panics if the length of the array is greater than the slice's
a := make([]byte, 5)
b := (*[10]byte)(a) // panics
the pointer points to the slice's underlying array, therefore the same values will be visible by indexing either:
a := []byte{0xa1, 0xa2}
b := (*[2]uint8)(a)
fmt.Printf("%x\n", a[0]) // a1
b[0] = 0xff
fmt.Printf("%x\n", a[0]) // ff
you can convert from byte to uint8, including type literals derived from them, because byte is an alias (identical to) of uint8.
Related: How do you convert a slice into an array?

Golang: Make function and third param

What is the difference between:
x := make([]int, 5, 10)
x := make([]int, 5)
x := [5]int{}
I know that make allocates an array and returns a slice that refers to that array. I don't understand where it can be used?
I can't find a good example that will clarify the situation.
x := make([]int, 5) Makes slice of int with length 5 and capacity 5 (same as length).
x := make([]int, 5, 10) Makes slice of int with length 5 and capacity 10.
x := [5]int{} Makes array of int with length 5.
Slices
If you need to append more items than capacity of slice using append function, go runtime will allocate new underlying array and copy existing one to it. So if you know about estimated length of your slice, better to use explicit capacity declaration. It will consume more memory for underlying array at the beginning, but safe cpu time for many allocations and array copying.
You can explore how len and cap changes while append, using that simple test on Go playground
Every time when cap value changed, new array allocated
Arrays
Array size is fixed, so if you need to grow array you have to create new one with new length and copy your old array into it by your own.
There are some great articles about slices and arrays in go:
http://blog.golang.org/go-slices-usage-and-internals
http://blog.golang.org/slices
The second line will allocate 10 int's worth memory at the very beginning, but returning you a slice of 5 int's. The second line does not stand less memory, it saves you another memory allocation if you need to expand the slice to anything not more than 10 * load_factor.

Slices in Go: why does it allow appending more than the capacity allows?

The capacity parameter in making a slice in Go does not make much sense to me. For example,
aSlice := make([]int, 2, 2) //a new slice with length and cap both set to 2
aSlice = append(aSlice, 1, 2, 3, 4, 5) //append integers 1 through 5
fmt.Println("aSlice is: ", aSlice) //output [0, 0, 1, 2, 3, 4, 5]
If the slice allows inserting more elements than the capacity allows, why do we need to set it in the make() function?
The builtin append() function uses the specified slice to append elements to if it has a big enough capacity to accomodate the specified elements.
But if the passed slice is not big enough, it allocates a new, big enough slice, copies the elements from the passed slice to the new slice and append the elements to that new slice. And returns this new slice. Quoting from the append() documentation:
The append built-in function appends elements to the end of a slice. If it has sufficient capacity, the destination is resliced to accommodate the new elements. If it does not, a new underlying array will be allocated. Append returns the updated slice. It is therefore necessary to store the result of append, often in the variable holding the slice itself:
When making a slice with make if the length and capacity are the same, the capacity can be omitted, in which case it is defaulted to the specified length:
// These 2 declarations are equivalent:
s := make([]int, 2, 2)
s := make([]int, 2)
Also note that append() appends elements after the last element of the slice. And the above slices already have len(s) == 2 right after declaration so if you append even just 1 element to it, it will cause a reallocation as seen in this example:
s := make([]int, 2, 2)
fmt.Println(s, len(s), cap(s))
s = append(s, 1)
fmt.Println(s, len(s), cap(s))
Output:
[0 0] 2 2
[0 0 1] 3 4
So in your example what you should do is something like this:
s := make([]int, 0, 10) // Create a slice with length=0 and capacity=10
fmt.Println(s, len(s), cap(s))
s = append(s, 1)
fmt.Println(s, len(s), cap(s))
Output:
[] 0 10
[1] 1 10
I recommend the following blog articles if you want to understand slices in more details:
Go Slices: usage and internals
Arrays, slices (and strings): The mechanics of 'append'
It is mainly an optimization, and it is not unique to go, similar structures in other languages have this as well.
When you append more than the capacity, the runtime needs to allocate more memory for the new elements. This is costly and can also cause memory fragmentation.
By specifying the capacity, the runtime allocates what is needed in advance, and avoids reallocations. However if you do not know the estimated capacity in advance or it changes, you do not have to set it, and the runtime reallocates what is needed and grows the capacity itself.

golang slice, slicing a slice with slice[a:b:c]

I read go slice usage and internals and Slice and Effective go#slice but there is nothing about slicing a slice with 3 number like this : slice[a:b:c]
For example this code :
package main
import "fmt"
func main() {
var s = []string{"a", "b", "c", "d", "e", "f", "g"}
fmt.Println(s[1:2:6], len(s[1:2:6]), cap(s[1:2:6]))
fmt.Println(s[1:2:5], len(s[1:2:5]), cap(s[1:2:5]))
fmt.Println(s[1:2], len(s[1:2]), cap(s[1:2]))
}
go playground result is this :
[b] 1 5
[b] 1 4
[b] 1 6
I can understand that the third one is something about capacity, but what is the exact meaning of this?
Do I miss something in documents?
The syntax has been introduced in Go 1.2, as I mentioned in "Re-slicing slices in Golang".
It is documented in Full slice expressions:
a[low : high : max]
constructs a slice of the same type, and with the same length and elements as the simple slice expression a[low : high].
Additionally, it controls the resulting slice's capacity by setting it to max - low.
Only the first index may be omitted; it defaults to 0.
After slicing the array a:
a := [5]int{1, 2, 3, 4, 5}
t := a[1:3:5]
the slice t has type []int, length 2, capacity 4, and elements
t[0] == 2
t[1] == 3
The design document for that feature had the following justification:
It would occasionally be useful, for example in custom []byte allocation managers, to be able to hand a slice to a caller and know that the caller cannot edit values beyond a given subrange of the true array.
The addition of append to the language made this somewhat more important, because append lets programmers overwrite entries between len and cap without realizing it or even mentioning cap.
2022: svitanok adds for Go 1.19+:
while the capacity of a "derivative" slice doesn't exceed the one specified by the third index during its creation the slice is still "from" the same spot in the memory as its original ("true") slice, so the changes applied to it will affect the original slice.
And if then, for example, you append to this derivative slice the amount of elements that would cause its capacity to be increased, this new slice will occupy a different place in the memory, and so the changes made to it will not affect the slice it originated from.
In a slice expression slice[a:b:c] or aSlice[1:3:5]
a:b or 1:3 -> gives length
a:c or 1:5 -> gives capacity
We can extract both length and capacity from a slice expression with 3 numbers/indices, without looking at the source slice/array.
expression| aSlice[low:high:max] or aSlice[a:b:c] or aSlice[1:3:7]
------------------------------------------------------------------------
Length | len(aSlice[low:high]) or len(aSlice[a:b]) or len(aSlice[1:3])
Capacity | len(aSlice[low:max]) or len(aSlice[a:c]) or len(aSlice[1:7])
------------------------------------------------------------------------
Read more here at Slice Expressions
Playground
Actually Go slice have a pointer and pointing to the array and it holds length and capacity of the array and we can show it like will be
pointer:length:capacity
and append is used for adding same new length.
sl1 := make([]int, 6)
fmt.Println(sl1)
sl2 := append(sl1, 1)
fmt.Println(sl2)
[0 0 0 0 0 0]
[0 0 0 0 0 0 1]

Resources