Pass slice as function argument, and modify the original slice - go

I know everything is passed by value in Go, meaning if I give a slice to a function and that function appends to the slice using the builtin append function, then the original slice will not have the values that were appended in the scope of the function.
For instance:
nums := []int{1, 2, 3}
func addToNumbs(nums []int) []int {
nums = append(nums, 4)
fmt.Println(nums) // []int{1, 2, 3, 4}
}
fmt.Println(nums) // []int{1, 2, 3}
This causes a problem for me, because I am trying to do recursion on an accumulated slice, basically a reduce type function except the reducer calls itself.
Here is an example:
func Validate(obj Validatable) ([]ValidationMessage, error) {
messages := make([]ValidationMessage, 0)
if err := validate(obj, messages); err != nil {
return messages, err
}
return messages, nil
}
func validate(obj Validatable, accumulator []ValidationMessage) error {
// If something is true, recurse
if something {
if err := validate(obj, accumulator); err != nil {
return err
}
}
// Append to the accumulator passed in
accumulator = append(accumulator, message)
return nil
}
The code above gives me the same error as the first example, in that the accumulator does not get all the appended values because they only exist within the scope of the function.
To solve this, I pass in a pointer struct into the function, and that struct contains the accumulator. That solution works nicely.
My question is, is there a better way to do this, and is my approach idiomatic to Go?
Updated solution (thanks to icza):
I just return the slice in the recursed function. Such a facepalm, should have thought of that.
func Validate(obj Validatable) ([]ValidationMessage, error) {
messages := make([]ValidationMessage, 0)
return validate(obj, messages)
}
func validate(obj Validatable, messages []ValidationMessage) ([]ValidationMessage, error) {
err := v.Struct(obj)
if _, ok := err.(*validator.InvalidValidationError); ok {
return []ValidationMessage{}, errors.New(err.Error())
}
if _, ok := err.(validator.ValidationErrors); ok {
messageMap := obj.Validate()
for _, err := range err.(validator.ValidationErrors) {
f := err.StructField()
t := err.Tag()
if v, ok := err.Value().(Validatable); ok {
return validate(v, messages)
} else if _, ok := messageMap[f]; ok {
if _, ok := messageMap[f][t]; ok {
messages = append(messages, ValidationMessage(messageMap[f][t]))
}
}
}
}
return messages, nil
}

If you want to pass a slice as a parameter to a function, and have that function modify the original slice, then you have to pass a pointer to the slice:
func myAppend(list *[]string, value string) {
*list = append(*list, value)
}
I have no idea if the Go compiler is naive or smart about this; performance is left as an exercise for the comment section.
For junior coders out there, please note that this code is provided without error checking. For example, this code will panic if list is nil.

Slice grows dynamically as required if the current size of the slice is not sufficient to append new value thereby changing the underlying array. If this new slice is not returned, your append change will not be visible.
Example:
package main
import (
"fmt"
)
func noReturn(a []int, data ...int) {
a = append(a, data...)
}
func returnS(a []int, data ...int) []int {
return append(a, data...)
}
func main() {
a := make([]int, 1)
noReturn(a, 1, 2, 3)
fmt.Println(a) // append changes will not visible since slice size grew on demand changing underlying array
a = returnS(a, 1, 2, 3)
fmt.Println(a) // append changes will be visible here since your are returning the new updated slice
}
Result:
[0]
[0 1 2 3]
Note:
You don't have to return the slice if you are updating items in the slice without adding new items to slice

Slice you passed is an reference to an array, which means the size is fixed. If you just modified the stored values, that's ok, the value will be updated outside the called function.
But if you added new element to the slice, it will reslice to accommodate new element, in other words, a new slice will be created and old slice will not be overwritten.
As a summary, if you need to extend or cut the slice, pass the pointer to the slice.Otherwise, use slice itself is good enough.
Update
I need to explain some important facts. For adding new elements to a slice which was passed as a value to a function, there are 2 cases:
A
the underlying array reached its capacity, a new slice created to replace the origin one, obviously the origin slice will not be modified.
B
the underlying array has not reached its capacity, and was modified. BUT the field len of the slice was not overwritten because the slice was passed by value. As a result, the origin slice will not aware its len was modified, which result in the slice not modified.

When appending data into slice, if the underlying array of the slice doesn't have enough space, a new array will be allocated. Then the elements in old array will be copied into this new memory, accompanied with adding new data behind

Related

How to slice a slice for eliminating matching values from identical slice

I'm looking for an easy way to iterate through a slice and on every value that's present in the current slice, remove the element from another slice.
I have a struct:
a := enter{
uid: 1234,
status: []StatusEntry{
{
rank: 1,
iterate: ierationState_Ongoing,
},
{
rank: 2,
iterate: ierationState_Completed,
},
},
}
In my .go file, I have a constant
Steps = [5]int64{0,1,2,3,4}
According to my requirement I want to copy the Steps in another variable and perform remove operation :
Steps2 := Steps // Make a copy of Steps
for _, element := enter.status {
// Remove that element from Steps
}
But I find it difficult to do so since Golang doesn't give me direct method to iterate and remove every element from enter.status from Steps.
I tried multiple things like creating a removeIndex function as posted on various stackoverflow answers like this:
for i, element := enter.status {
Steps2 = removeIndex(enter.status, i)
}
func removeIndex(s []int, index int) []int {
ret := make([]int, 0)
ret = append(ret, s[:index]...)
return append(ret, s[index+1:]...)
}
But it doesn't make sense to use this because I'm trying to remove a matching value (element) and not a specific index (for eg index 5) from Steps2.
Basically, for every element that's in slice enter.status, I want to remove that element/value from slice Steps2
Careful:
[5]int64{0,1,2,3,4}
This is an array (of 5 ints), not a slice. And:
Steps2 := Steps
If Steps were a slice, this would copy the slice header without copying the underlying array.
In any case, given some slice s of type T and length len(s), if you are allowed to modify s in place and order is relevant, you generally want to use this algorithm:
func trim(s []T) []T {
out := 0
for i := range s {
if keep(s[i]) {
s[out] = s[i]
out++
}
}
return s[:out]
}
where keep is your boolean function to decide whether to keep an element. To make this produce a new slice, allocate an output slice of the appropriate length (len(s)) at the start and optionally shrink it later, or, if you expect to throw out most elements, make it empty at the start and use append.
When the keep function is "the value of some field in the output slice does not match the value of any earlier kept field" and the type of that field is usable as a key type, you can use a simple map[T2]struct{} to determine whether the value has occurred yet:
seen := make(map[T2]struct{}, len(s))
and then the keep test and copy sequence becomes:
_, ok := seen[s[i].field]
if !ok {
seen[s[i].field] = struct{}{}
s[out] = s[i]
out++
}
The initial size of seen here is optimized on the theory that most values will be kept; if most values will be discarded, make the map initially empty, or small.

Mutate a slice passed as an argument to a function in Go

I wrote some code where a function f takes a slice s as an argument and modifies it without returning it.
Since slices can be considered as references to underlying arrays, I thought that the slice would be actually modified outside of that function's scope, but it's not the case.
An example would be the code below (https://play.golang.org/p/Y5JUmDtRXrz).
package main
import (
"fmt"
)
func pop(s []int) int {
first, s := s[0], s[1:]
return first
}
func main() {
s := []int{0, 1, 2, 3}
first := pop(s)
fmt.Println(first, s)
}
pop(s) actually returns 0, which is expected. But then in the output s still has 0 as its first element.
0 [0 1 2 3]
Program exited.
Why? And how could I solve this ?
Two separate things are happening here, both of which prevent this from behaving as you expect:
func pop(s []int) int {
first, s := s[0], s[1:]
The first (and simpler) issue is you're defining a new local variable s here, which shadows you function parameter s.
Second, slices do point to an underlying array, but the slice is still passed by copy, just like everything else. That means that:
s = s[1:]
Modifies your copy of s to have a different window on the underlying array. That doesn't change the slice in the caller. However, if you change the values in the underlying array, that will be reflected in the caller, e.g.:
s[1] = 42
You can learn more about this throughout the Tour and on the Go blog.
The line first, s := s[0], s[1:] creates a new variable s since you are using :=. On top of that if you want to modify the slice you need to pass it by pointer.
Passing it by value, it will refer to the same underlying array, but the slice itself is a copy. So changes to the underlying array would be reflected in main, but changes to the slice itself would not.
Here is an example of passing the slice by pointer.
package main
import (
"fmt"
)
func pop(s *[]int) int {
first := (*s)[0]
*s = (*s)[1:]
return first
}
func main() {
s := []int{0, 1, 2, 3}
first := pop(&s)
fmt.Println(first, s)
}

Golang fill slice in function not working

I am trying to add stuff to my slice but somehow the slice is never updated.
endpointsList := make([]string, 3)
for _, route := range routes {
if len(route.Endpoints) > 0 {
waitGroup.Add(1)
go endpointRoutine(route, template, route.Protected, &waitGroup, &endpointsList)
}
}
I pass the endpointsList by reference, meaning I should be able to assign new things to its memory location I think.
In the function endpointRoutine I do this:
list := make([]string, 3)
for _, r := range route.Endpoints {
list = append(list, "some data comes here...")
}
endpointsList = &list
When I do a printr after this (below my first bit of code and AFTER the waitGroup.Wait() part) the slice is still empty.
Obviously, I am overwriting the slice now and my final goal is to ADD to the slice. But when I try to add with this code:
endpointsList = append(endpointsList, "fdssdfsdfsdf")
It gives me the error:
cannot use endpointsList (type *[]string) as []Type
Can someone please explain to me what might be wrong?
With endpointsList = &list, you are assigning the pointer pointing to the slice to some other slice. To set the slice, do this instead:
*endpointsList=list

Difference in behavior between slices and maps

A related questions is here https://stackoverflow.com/a/12965872/6421681.
In go, you can do:
func numsInFactorial(n int) (nums []int) {
// `nums := make([]int)` is not needed
for i := 1; i <= n; i++ {
nums = append(nums, i)
}
return
}
However,the following doesn't work:
func mapWithOneKeyAndValue(k int, v int) (m map[int]int) {
m[k] = v
return
}
An error is thrown:
panic: assignment to entry in nil map
Instead, you must:
func mapWithOneKeyAndValue(k int, v int) map[int]int {
m := make(map[int]int)
m[k] = v
return
}
I can't find the documentation for this behavior.
I have read through all of effective go, and there's no mention of it there either.
I know that named return values are defined (i.e. memory is allocated; close to what new does) but not initialized (so make behavior isn't replicated).
After some experimenting, I believe this behavior can be reduced into understanding the behavior of the following code:
func main() {
var s []int // len and cap are both 0
var m map[int]int
fmt.Println(s) // works... prints an empty slice
fmt.Println(m) // works... prints an empty map
s = append(s, 10) // returns a new slice, so underlying array gets allocated
fmt.Println(s) // works... prints [10]
m[10] = 10 // program crashes, with "assignment to entry in nil map"
fmt.Println(m)
}
The issue seems that append likely calls make and allocates a new slice detecting that the capacity of s is 0. However, map never gets an explicit initialization.
The reason for this SO question is two-pronged. First, I would like to document the behavior on SO. Second, why would the language allow non-initializing definitions of slice and map? With my experience with go so far, it seems to be a pragmatic language (i.e. unused variables lead to compilation failure, gofmt forces proper formatting), so it would make sense for it to prevent the code from compiling.
Try to assign in nil slice by index - you will get "panic: runtime error: index out of range" (example: https://play.golang.org/p/-XHh1jNyn5g)
The only reason why append function works with nil, is that append function can do reallocation for the given slice.
For example, if you trying to to append 6th element to slice of 5 elements with current capacity 5, it will create the new array with new capacity, copy all the info from old one, and swap the data array pointers in the given slice. In my understanding, it is just golang implementation of dynamic arrays.
So, the nil slice is just a special case of slice with not enough capacity, so it would be reallocated on any append operation.
More details on https://blog.golang.org/go-slices-usage-and-internals
From https://blog.golang.org/go-maps-in-action
A nil map behaves like an empty map when reading, but attempts to write to a nil map will cause a runtime panic; don't do that. To initialize a map, use the built in make function
It seems like a nil map is considered a valid empty map and that's the reason they don't allocate memory for it automatically.

Are slices passed by value?

In Go, I am trying to make a scramble slice function for my traveling salesman problem. While doing this I noticed when I started editing the slice I gave the scramble function was different every time I passed it in.
After some debugging I found out it was due to me editing the slice inside the function. But since Go is supposed to be a "pass by value" language, how is this possible?
https://play.golang.org/p/mMivoH0TuV
I have provided a playground link to show what I mean.
By removing line 27 you get a different output than leaving it in, this should not make a difference since the function is supposed to make its own copy of the slice when passed in as an argument.
Can someone explain the phenomenon?
Everything in Go is passed by value, slices too. But a slice value is a header, describing a contiguous section of a backing array, and a slice value only contains a pointer to the array where the elements are actually stored. The slice value does not include its elements (unlike arrays).
So when you pass a slice to a function, a copy will be made from this header, including the pointer, which will point to the same backing array. Modifying the elements of the slice implies modifying the elements of the backing array, and so all slices which share the same backing array will "observe" the change.
To see what's in a slice header, check out the reflect.SliceHeader type:
type SliceHeader struct {
Data uintptr
Len int
Cap int
}
See related / possible duplicate question:
Performance of function slice parameter vs global variable?
Read blog post: Go Slices: usage and internals
Please note that when you pass a slice to a function, if the function modifies the "existing" elements of the slice, the caller will see / observe the changes. If the function adds new elements to the slice, that requires changing the slice header (the length at a minimum, but may also involve allocating a new backing array), which the caller will not see (not without returning the new slice header).
Not with maps, because maps are pointers under the hood, and if you pass a map to a function and the function adds a new entry to the map, the map pointer will not change so the caller will see the changed map (the new entry) without returning the map after change.
Also regarding slices and maps, see Map initialization in Go and why slice values can sometimes go stale but never map values?
You can find an example below. Briefly slices is also passed by value but original slice and copied slice are linked to the same underlying array. If one of this slice changes, then underlying array changes, then other slice changes.
package main
import "fmt"
func main() {
x := []int{1, 10, 100, 1000}
double(x)
fmt.Println(x) // ----> 3 will print [2, 20, 200, 2000] (original slice changed)
}
func double(y []int) {
fmt.Println(y) // ----> 1 will print [1, 10, 100, 1000]
for i := 0; i < len(y); i++ {
y[i] *= 2
}
fmt.Println(y) // ----> 2 will print [2, 20, 200, 2000] (copy slice + under array changed)
}
Slices when its passed it’s passed with the pointer to underlying array, so a slice is a small structure that points to an underlying array. The small structure is copied, but it still points to the same underlying array. the memory block containing the slice elements is passed by "reference". The slice information triplet holding the capacity, the number of element and the pointer to the elements is passed by value.
The best way to handle slices passing to function (if the elements of the slice are manipulated into the function, and we do not want this to be reflected at the elements memory block is to copy them using copy(s, *c) as:
package main
import "fmt"
type Team []Person
type Person struct {
Name string
Age int
}
func main() {
team := Team{
Person{"Hasan", 34}, Person{"Karam", 32},
}
fmt.Printf("original before clonning: %v\n", team)
team_cloned := team.Clone()
fmt.Printf("original after clonning: %v\n", team)
fmt.Printf("clones slice: %v\n", team_cloned)
}
func (c *Team) Clone() Team {
var s = make(Team, len(*c))
copy(s, *c)
for index, _ := range s {
s[index].Name = "change name"
}
return s
}
But be careful, if this slice is containing a sub slice further copying is required, as we'll still have the sub slice elements sharing pointing to the same memory block elements, an example is:
type Inventories []Inventory
type Inventory struct { //instead of: map[string]map[string]Pairs
Warehouse string
Item string
Batches Lots
}
type Lots []Lot
type Lot struct {
Date time.Time
Key string
Value float64
}
func main() {
ins := Inventory{
Warehouse: "DMM",
Item: "Gloves",
Batches: Lots{
Lot{mustTime(time.Parse(custom, "1/7/2020")), "Jan", 50},
Lot{mustTime(time.Parse(custom, "2/1/2020")), "Feb", 70},
},
}
inv2 := CloneFrom(c Inventories)
}
func (i *Inventories) CloneFrom(c Inventories) {
inv := new(Inventories)
for _, v := range c {
batches := Lots{}
for _, b := range v.Batches {
batches = append(batches, Lot{
Date: b.Date,
Key: b.Key,
Value: b.Value,
})
}
*inv = append(*inv, Inventory{
Warehouse: v.Warehouse,
Item: v.Item,
Batches: batches,
})
}
(*i).ReplaceBy(inv)
}
func (i *Inventories) ReplaceBy(x *Inventories) {
*i = *x
}
Slice will work with pass by value to the function, But we should not use append to add values to slice in the function, instead we should use the assignment directly. Reason being that append will create new memory and copy values to that. Here is the example.
Go playground
// Go program to illustrate how to
// pass a slice to the function
package main
import "fmt"
// Function in which slice
// is passed by value
func myfun(element []string) {
// Here we only modify the slice
// Using append function
// Here, this function only modifies
// the copy of the slice present in
// the function not the original slice
element = append(element, "blackhole")
fmt.Println("Modified slice: ", element)
}
func main() {
// Creating a slice
slc := []string{"rocket", "galaxy", "stars", "milkyway"}
fmt.Println("Initial slice: ", slc)
//slice pass by value
myfun(slc)
fmt.Println("Final slice: ", slc)
}
Output-
Initial slice: [rocket galaxy stars milkyway]
Modified slice: [rocket galaxy stars milkyway blackhole]
Final slice: [rocket galaxy stars milkyway]
Go Playground
// Go program to illustrate how to
// pass a slice to the function
package main
import "fmt"
// Function in which slice
// is passed by value
func myfun(element []string) {
// Here we only modify the slice
// Using append function
// Here, this function only modifies
// the copy of the slice present in
// the function not the original slice
element[0] = "Spaceship"
element[4] = "blackhole"
element[5] = "cosmos"
fmt.Println("Modified slice: ", element)
}
func main() {
// Creating a slice
slc := []string{"rocket", "galaxy", "stars", "milkyway", "", ""}
fmt.Println("Initial slice: ", slc)
//slice pass by value
myfun(slc)
fmt.Println("Final slice: ", slc)
}
Output-
Initial slice: [rocket galaxy stars milkyway ]
Modified slice: [Spaceship galaxy stars milkyway blackhole cosmos]
Final slice: [Spaceship galaxy stars milkyway blackhole cosmos]
To complement this post, here is an example of passing by reference for the Golang PlayGround you shared:
type point struct {
x int
y int
}
func main() {
data := []point{{1, 2}, {3, 4}, {5, 6}, {7, 8}}
makeRandomDatas(&data)
}
func makeRandomDatas(dataPoints *[]point) {
for i := 0; i < 10; i++ {
if len(*dataPoints) > 0 {
fmt.Println(makeRandomData(dataPoints))
} else {
fmt.Println("no more elements")
}
}
}
func makeRandomData(cities *[]point) []point {
solution := []point{(*cities)[0]} //create a new slice with the first item from the old slice
*cities = append((*cities)[:0], (*cities)[1:]...) //remove the first item from the old slice
return solution
}

Resources