I wrote some code where a function f takes a slice s as an argument and modifies it without returning it.
Since slices can be considered as references to underlying arrays, I thought that the slice would be actually modified outside of that function's scope, but it's not the case.
An example would be the code below (https://play.golang.org/p/Y5JUmDtRXrz).
package main
import (
"fmt"
)
func pop(s []int) int {
first, s := s[0], s[1:]
return first
}
func main() {
s := []int{0, 1, 2, 3}
first := pop(s)
fmt.Println(first, s)
}
pop(s) actually returns 0, which is expected. But then in the output s still has 0 as its first element.
0 [0 1 2 3]
Program exited.
Why? And how could I solve this ?
Two separate things are happening here, both of which prevent this from behaving as you expect:
func pop(s []int) int {
first, s := s[0], s[1:]
The first (and simpler) issue is you're defining a new local variable s here, which shadows you function parameter s.
Second, slices do point to an underlying array, but the slice is still passed by copy, just like everything else. That means that:
s = s[1:]
Modifies your copy of s to have a different window on the underlying array. That doesn't change the slice in the caller. However, if you change the values in the underlying array, that will be reflected in the caller, e.g.:
s[1] = 42
You can learn more about this throughout the Tour and on the Go blog.
The line first, s := s[0], s[1:] creates a new variable s since you are using :=. On top of that if you want to modify the slice you need to pass it by pointer.
Passing it by value, it will refer to the same underlying array, but the slice itself is a copy. So changes to the underlying array would be reflected in main, but changes to the slice itself would not.
Here is an example of passing the slice by pointer.
package main
import (
"fmt"
)
func pop(s *[]int) int {
first := (*s)[0]
*s = (*s)[1:]
return first
}
func main() {
s := []int{0, 1, 2, 3}
first := pop(&s)
fmt.Println(first, s)
}
Related
I've had difficulty learning the basics of reflect, pointers and interface in go, so here's another entry level question I can't seem to figure out.
This code does what I want it to do - I'm using reflect to add another record to a slice that's typed as an interface.
package main
import (
"reflect"
"log"
)
type Person struct {
Name string
}
func Add(slice interface{}) {
s := reflect.ValueOf(slice).Elem()
// in my actual code, p is declared via the use of reflect.New([Type])
p := Person{Name:"Sam"}
s.Set(reflect.Append(s,reflect.ValueOf(p)))
}
func main() {
p := []Person{}
Add(&p)
log.Println(p)
}
If I changed the Add and main function to this, things don't work the way I want it to.
func Add(slice interface{}) {
s := reflect.ValueOf(&slice).Elem()
p := Person{Name:"Sam"}
s.Set(reflect.Append(reflect.ValueOf(slice),reflect.ValueOf(p)))
log.Println(s)
}
func main() {
p := []Person{}
Add(p)
log.Println(p)
}
That is, the log.Println(p) at the end doesn't show a slice with the record Sam in it like the way I had hoped. So my question is whether it's possible for me to have Add() receive a slice that is not a pointer, and for me to still write some code in Add() that will produce the outcome shown in my first scenario?
A lot of my recent questions dance around this kind of subject, so it's still taking me a while to figure out how to use the reflect package effectively.
No, it's not possible to append to a slice in a function without passing in a pointer to the slice. This isn't related to reflection, but to how variables are passed in to functions. Here's the same code, modified to not use reflection:
package main
import (
"log"
)
type Person struct {
Name string
}
func AddWithPtr(slicep interface{}) {
sp := slicep.(*[]Person)
// This modifies p1 itself, since *sp IS p1
*sp = append(*sp, Person{"Sam"})
}
func Add(slice interface{}) {
// s is now a copy of p2
s := slice.([]Person)
sp := &s
// This modifies a copy of p2 (i.e. s), not p2 itself
*sp = append(*sp, Person{"Sam"})
}
func main() {
p1 := []Person{}
// This passes a reference to p1
AddWithPtr(&p1)
log.Println("Add with pointer: ", p1)
p2 := []Person{}
// This passes a copy of p2
Add(p2)
log.Println("Add without pointer:", p2)
}
(Above, when it says 'copy' of the slice, it doesn't mean the copy of the underlying data - just the slice)
When you pass in a slice, the function effectively gets a new slice that refers to the same data as the original. Appending to the slice in the function increases the length of the new slice, but doesn't change the length of the original slice that was passed in. That's why the original slice remains unchanged.
I know everything is passed by value in Go, meaning if I give a slice to a function and that function appends to the slice using the builtin append function, then the original slice will not have the values that were appended in the scope of the function.
For instance:
nums := []int{1, 2, 3}
func addToNumbs(nums []int) []int {
nums = append(nums, 4)
fmt.Println(nums) // []int{1, 2, 3, 4}
}
fmt.Println(nums) // []int{1, 2, 3}
This causes a problem for me, because I am trying to do recursion on an accumulated slice, basically a reduce type function except the reducer calls itself.
Here is an example:
func Validate(obj Validatable) ([]ValidationMessage, error) {
messages := make([]ValidationMessage, 0)
if err := validate(obj, messages); err != nil {
return messages, err
}
return messages, nil
}
func validate(obj Validatable, accumulator []ValidationMessage) error {
// If something is true, recurse
if something {
if err := validate(obj, accumulator); err != nil {
return err
}
}
// Append to the accumulator passed in
accumulator = append(accumulator, message)
return nil
}
The code above gives me the same error as the first example, in that the accumulator does not get all the appended values because they only exist within the scope of the function.
To solve this, I pass in a pointer struct into the function, and that struct contains the accumulator. That solution works nicely.
My question is, is there a better way to do this, and is my approach idiomatic to Go?
Updated solution (thanks to icza):
I just return the slice in the recursed function. Such a facepalm, should have thought of that.
func Validate(obj Validatable) ([]ValidationMessage, error) {
messages := make([]ValidationMessage, 0)
return validate(obj, messages)
}
func validate(obj Validatable, messages []ValidationMessage) ([]ValidationMessage, error) {
err := v.Struct(obj)
if _, ok := err.(*validator.InvalidValidationError); ok {
return []ValidationMessage{}, errors.New(err.Error())
}
if _, ok := err.(validator.ValidationErrors); ok {
messageMap := obj.Validate()
for _, err := range err.(validator.ValidationErrors) {
f := err.StructField()
t := err.Tag()
if v, ok := err.Value().(Validatable); ok {
return validate(v, messages)
} else if _, ok := messageMap[f]; ok {
if _, ok := messageMap[f][t]; ok {
messages = append(messages, ValidationMessage(messageMap[f][t]))
}
}
}
}
return messages, nil
}
If you want to pass a slice as a parameter to a function, and have that function modify the original slice, then you have to pass a pointer to the slice:
func myAppend(list *[]string, value string) {
*list = append(*list, value)
}
I have no idea if the Go compiler is naive or smart about this; performance is left as an exercise for the comment section.
For junior coders out there, please note that this code is provided without error checking. For example, this code will panic if list is nil.
Slice grows dynamically as required if the current size of the slice is not sufficient to append new value thereby changing the underlying array. If this new slice is not returned, your append change will not be visible.
Example:
package main
import (
"fmt"
)
func noReturn(a []int, data ...int) {
a = append(a, data...)
}
func returnS(a []int, data ...int) []int {
return append(a, data...)
}
func main() {
a := make([]int, 1)
noReturn(a, 1, 2, 3)
fmt.Println(a) // append changes will not visible since slice size grew on demand changing underlying array
a = returnS(a, 1, 2, 3)
fmt.Println(a) // append changes will be visible here since your are returning the new updated slice
}
Result:
[0]
[0 1 2 3]
Note:
You don't have to return the slice if you are updating items in the slice without adding new items to slice
Slice you passed is an reference to an array, which means the size is fixed. If you just modified the stored values, that's ok, the value will be updated outside the called function.
But if you added new element to the slice, it will reslice to accommodate new element, in other words, a new slice will be created and old slice will not be overwritten.
As a summary, if you need to extend or cut the slice, pass the pointer to the slice.Otherwise, use slice itself is good enough.
Update
I need to explain some important facts. For adding new elements to a slice which was passed as a value to a function, there are 2 cases:
A
the underlying array reached its capacity, a new slice created to replace the origin one, obviously the origin slice will not be modified.
B
the underlying array has not reached its capacity, and was modified. BUT the field len of the slice was not overwritten because the slice was passed by value. As a result, the origin slice will not aware its len was modified, which result in the slice not modified.
When appending data into slice, if the underlying array of the slice doesn't have enough space, a new array will be allocated. Then the elements in old array will be copied into this new memory, accompanied with adding new data behind
package main
import "fmt"
func main() {
a := SomeType{myslice: []int{1, 2, 3}, decimal: 2.33}
for _, i := range a.myslice {
fmt.Println(i)
}
fmt.Println(a.decimal)
addOne(a)
for _, i := range a.myslice {
fmt.Println(i)
}
fmt.Println(a.decimal)
}
type SomeType struct {
myslice []int
decimal float32
}
func addOne(s SomeType) {
s.myslice[0]++
s.decimal += 1.2
}
The output for the code above is:
1
2
3
2.33
2
2
3
2.33
Even though i have not passed the SomeType object a by reference the myslice field is being modified in the original object. Why is this happening? Is there anyway to pass the entire object by value without having to create a copy of the original object?
The slice is not really being passed by reference; if you append to it in addOne, its length will not change. But a slice contains a reference (or pointer) to its backing array. So when you copy a slice, the new one shares the same backing array with the old one.
The fact that the slice is inside a struct doesn't make any difference. You would see the same thing if you changed addOne to just take a slice instead of the whole struct.
In Go, I am trying to make a scramble slice function for my traveling salesman problem. While doing this I noticed when I started editing the slice I gave the scramble function was different every time I passed it in.
After some debugging I found out it was due to me editing the slice inside the function. But since Go is supposed to be a "pass by value" language, how is this possible?
https://play.golang.org/p/mMivoH0TuV
I have provided a playground link to show what I mean.
By removing line 27 you get a different output than leaving it in, this should not make a difference since the function is supposed to make its own copy of the slice when passed in as an argument.
Can someone explain the phenomenon?
Everything in Go is passed by value, slices too. But a slice value is a header, describing a contiguous section of a backing array, and a slice value only contains a pointer to the array where the elements are actually stored. The slice value does not include its elements (unlike arrays).
So when you pass a slice to a function, a copy will be made from this header, including the pointer, which will point to the same backing array. Modifying the elements of the slice implies modifying the elements of the backing array, and so all slices which share the same backing array will "observe" the change.
To see what's in a slice header, check out the reflect.SliceHeader type:
type SliceHeader struct {
Data uintptr
Len int
Cap int
}
See related / possible duplicate question:
Performance of function slice parameter vs global variable?
Read blog post: Go Slices: usage and internals
Please note that when you pass a slice to a function, if the function modifies the "existing" elements of the slice, the caller will see / observe the changes. If the function adds new elements to the slice, that requires changing the slice header (the length at a minimum, but may also involve allocating a new backing array), which the caller will not see (not without returning the new slice header).
Not with maps, because maps are pointers under the hood, and if you pass a map to a function and the function adds a new entry to the map, the map pointer will not change so the caller will see the changed map (the new entry) without returning the map after change.
Also regarding slices and maps, see Map initialization in Go and why slice values can sometimes go stale but never map values?
You can find an example below. Briefly slices is also passed by value but original slice and copied slice are linked to the same underlying array. If one of this slice changes, then underlying array changes, then other slice changes.
package main
import "fmt"
func main() {
x := []int{1, 10, 100, 1000}
double(x)
fmt.Println(x) // ----> 3 will print [2, 20, 200, 2000] (original slice changed)
}
func double(y []int) {
fmt.Println(y) // ----> 1 will print [1, 10, 100, 1000]
for i := 0; i < len(y); i++ {
y[i] *= 2
}
fmt.Println(y) // ----> 2 will print [2, 20, 200, 2000] (copy slice + under array changed)
}
Slices when its passed it’s passed with the pointer to underlying array, so a slice is a small structure that points to an underlying array. The small structure is copied, but it still points to the same underlying array. the memory block containing the slice elements is passed by "reference". The slice information triplet holding the capacity, the number of element and the pointer to the elements is passed by value.
The best way to handle slices passing to function (if the elements of the slice are manipulated into the function, and we do not want this to be reflected at the elements memory block is to copy them using copy(s, *c) as:
package main
import "fmt"
type Team []Person
type Person struct {
Name string
Age int
}
func main() {
team := Team{
Person{"Hasan", 34}, Person{"Karam", 32},
}
fmt.Printf("original before clonning: %v\n", team)
team_cloned := team.Clone()
fmt.Printf("original after clonning: %v\n", team)
fmt.Printf("clones slice: %v\n", team_cloned)
}
func (c *Team) Clone() Team {
var s = make(Team, len(*c))
copy(s, *c)
for index, _ := range s {
s[index].Name = "change name"
}
return s
}
But be careful, if this slice is containing a sub slice further copying is required, as we'll still have the sub slice elements sharing pointing to the same memory block elements, an example is:
type Inventories []Inventory
type Inventory struct { //instead of: map[string]map[string]Pairs
Warehouse string
Item string
Batches Lots
}
type Lots []Lot
type Lot struct {
Date time.Time
Key string
Value float64
}
func main() {
ins := Inventory{
Warehouse: "DMM",
Item: "Gloves",
Batches: Lots{
Lot{mustTime(time.Parse(custom, "1/7/2020")), "Jan", 50},
Lot{mustTime(time.Parse(custom, "2/1/2020")), "Feb", 70},
},
}
inv2 := CloneFrom(c Inventories)
}
func (i *Inventories) CloneFrom(c Inventories) {
inv := new(Inventories)
for _, v := range c {
batches := Lots{}
for _, b := range v.Batches {
batches = append(batches, Lot{
Date: b.Date,
Key: b.Key,
Value: b.Value,
})
}
*inv = append(*inv, Inventory{
Warehouse: v.Warehouse,
Item: v.Item,
Batches: batches,
})
}
(*i).ReplaceBy(inv)
}
func (i *Inventories) ReplaceBy(x *Inventories) {
*i = *x
}
Slice will work with pass by value to the function, But we should not use append to add values to slice in the function, instead we should use the assignment directly. Reason being that append will create new memory and copy values to that. Here is the example.
Go playground
// Go program to illustrate how to
// pass a slice to the function
package main
import "fmt"
// Function in which slice
// is passed by value
func myfun(element []string) {
// Here we only modify the slice
// Using append function
// Here, this function only modifies
// the copy of the slice present in
// the function not the original slice
element = append(element, "blackhole")
fmt.Println("Modified slice: ", element)
}
func main() {
// Creating a slice
slc := []string{"rocket", "galaxy", "stars", "milkyway"}
fmt.Println("Initial slice: ", slc)
//slice pass by value
myfun(slc)
fmt.Println("Final slice: ", slc)
}
Output-
Initial slice: [rocket galaxy stars milkyway]
Modified slice: [rocket galaxy stars milkyway blackhole]
Final slice: [rocket galaxy stars milkyway]
Go Playground
// Go program to illustrate how to
// pass a slice to the function
package main
import "fmt"
// Function in which slice
// is passed by value
func myfun(element []string) {
// Here we only modify the slice
// Using append function
// Here, this function only modifies
// the copy of the slice present in
// the function not the original slice
element[0] = "Spaceship"
element[4] = "blackhole"
element[5] = "cosmos"
fmt.Println("Modified slice: ", element)
}
func main() {
// Creating a slice
slc := []string{"rocket", "galaxy", "stars", "milkyway", "", ""}
fmt.Println("Initial slice: ", slc)
//slice pass by value
myfun(slc)
fmt.Println("Final slice: ", slc)
}
Output-
Initial slice: [rocket galaxy stars milkyway ]
Modified slice: [Spaceship galaxy stars milkyway blackhole cosmos]
Final slice: [Spaceship galaxy stars milkyway blackhole cosmos]
To complement this post, here is an example of passing by reference for the Golang PlayGround you shared:
type point struct {
x int
y int
}
func main() {
data := []point{{1, 2}, {3, 4}, {5, 6}, {7, 8}}
makeRandomDatas(&data)
}
func makeRandomDatas(dataPoints *[]point) {
for i := 0; i < 10; i++ {
if len(*dataPoints) > 0 {
fmt.Println(makeRandomData(dataPoints))
} else {
fmt.Println("no more elements")
}
}
}
func makeRandomData(cities *[]point) []point {
solution := []point{(*cities)[0]} //create a new slice with the first item from the old slice
*cities = append((*cities)[:0], (*cities)[1:]...) //remove the first item from the old slice
return solution
}
I've started learning Go these days and got stuck in trying to pass a struct property's value (a slice) to a function. Apparently it's being passed as a reference (or it holds a pointer to its slice) and changes made inside the function affect it.
Here is my code, in which testFunction is supposed to receive a slice, remove its first 3 elements and print the updated values, but without affecting it externally:
package main
import (
"fmt"
)
type testStruct struct {
testArray []float64
}
var test = testStruct {
testArray: []float64{10,20,30,40,50},
}
func main() {
fmt.Println(test.testArray)
testFunction(test.testArray)
fmt.Println(test.testArray)
}
func testFunction(array []float64) {
for i:=0; i<3; i++ {
array = removeFrom(array, 0)
}
fmt.Println(array)
}
func removeFrom(array []float64, index int) []float64 {
return append(array[:index], array[index+1:]...)
}
That outputs:
[10 20 30 40 50]
[40 50]
[40 50 50 50 50]
My question is: what is causing the third fmt.Println to print this strange result?
Playground: https://play.golang.org/p/G8W3H085In
p.s.: This code is only an example. It's not my goal to remove the first elements of something. I just wanna know what is causing this strange behaviour.
Usually we don't know whether a given call to append will cause a reallocation, so we can't assume that the original slice refers to the same array as the resulting slice, nor that it refers to a different one.
To use slices correctly, it's important to remember that although the elements of the underlying array are indirect, the slice's pointer, length and capacity are not.
As a result, it's usual to assign the result of a call to append to the same slice variable:
array = append(array, ...)
So to sum up, to receive the desired result always remember to assign the append function to a new or the same slice variable.
Here is the corrected and working code:
package main
import (
"fmt"
)
type testStruct struct {
testArray []float64
}
var test = testStruct {
testArray: []float64{10,20,30,40,50},
}
func main() {
fmt.Println(test.testArray)
a := testFunction(test.testArray)
fmt.Println(a)
}
func testFunction(array []float64)[]float64 {
for i:=0; i<3; i++ {
array = removeFrom(array, 0)
}
fmt.Println(array)
return array
}
func removeFrom(array []float64, index int) []float64 {
return append(array[:index], array[index+1:]...)
}
Check it the working code on Go Playground.
Another solution is to pass the array argument via pointer reference:
func testFunction(array *[]float64) {
for i:=0; i<3; i++ {
*array = removeFrom(*array, 0)
}
fmt.Println(*array)
}
Go Playground
The slice is a composite type. It has a pointer to the data, the length and the capacity. When you pass it as an argument you're passing those values, the pointer, the length and the capacity; they are copies, always.
In your case you modify the data within the slice when you call removeFrom(), which you can do because you've copied the value of a pointer to the original data into the func, but the length and capacity remain unchanged outside the scope of that function as those are not pointers.
So, when you print it again from main() you see the altered values but it still uses the original length and capacity as any changes made to those within the scope of the other funcs were actually on copies of those values.
Here is a useful blog post about slices https://blog.golang.org/slices. It states this in particular.
It's important to understand that even though a slice contains a
pointer, it is itself a value. Under the covers, it is a struct value
holding a pointer and a length. It is not a pointer to a struct.
The reason you see [40 50 50 50 50] is because you changed the values in the slice, but you did not alter the slice itself(it's cap and len)