Given two nested types
type Inner struct {
InnerVal int
}
type Outer struct {
InnerStruct *Inner
OuterVal int
}
I need to duplicate a slice of pointers to Outer
originalSlice := []*Outer{<plenty_of_items>}
with itself, but having updated field values in the duplicates, including the Outer.InnerStruct.InnerVal.
To do so I create a new slice of the same type and length as originalSlice, append pointers to newly created structs with altered values to it, and finally append these items to the originalSlice
duplicateSlice := make([]*Outer, len(originalSlice))
for _, originalItem := range originalSlice {
duplicateSlice = append(duplicateSlice, &Outer{
InnerStruct: &Inner{
InnerVal: originalItem.InnerStruct.InnerVal + 1
},
OuterVal: originalItem.OuterVal + 1,
})
}
originalSlice = append(originalSlice, duplicateSlice...)
While this is verbose enough to follow the pointers around, or so I thought, when passed to a function right after as nowDoubledSlice, and accessed via loop
someOtherSlice := make([]*types.Inner, len(nowDoubledSlice))
for i, doubledItem := range nowDoubledSlice {
someOtherSlice[i] = doubledItem.InnerStruct
}
I get a
runtime error: invalid memory address or nil pointer dereference
Why is that? And is there a more elegant or idiomatic way to duplicate a slice of pointers to structs, while altering the duplicates' fields?
It's nothing to do with your pointer creation, it's your slice allocation. This line:
duplicateSlice := make([]*Outer, len(originalSlice))
Creates a new slice of length len(originalSlice), filled with zero-value elements. What you likely want instead is:
duplicateSlice := make([]*Outer, 0, len(originalSlice))
to create a slice of length 0 but capacity of len(originalSlice). This works fine, as you can see here.
Alternatively, you could keep make([]*Outer, len(originalSlice)) and use indexing instead of append in your loop:
for i, originalItem := range originalSlice {
duplicateSlice[i] =&Outer{
InnerStruct: &Inner{
InnerVal: originalItem.InnerStruct.InnerVal + 1,
},
OuterVal: originalItem.OuterVal + 1,
}
}
Which works just as well, as you can see here.
Related
I'm looking for an easy way to iterate through a slice and on every value that's present in the current slice, remove the element from another slice.
I have a struct:
a := enter{
uid: 1234,
status: []StatusEntry{
{
rank: 1,
iterate: ierationState_Ongoing,
},
{
rank: 2,
iterate: ierationState_Completed,
},
},
}
In my .go file, I have a constant
Steps = [5]int64{0,1,2,3,4}
According to my requirement I want to copy the Steps in another variable and perform remove operation :
Steps2 := Steps // Make a copy of Steps
for _, element := enter.status {
// Remove that element from Steps
}
But I find it difficult to do so since Golang doesn't give me direct method to iterate and remove every element from enter.status from Steps.
I tried multiple things like creating a removeIndex function as posted on various stackoverflow answers like this:
for i, element := enter.status {
Steps2 = removeIndex(enter.status, i)
}
func removeIndex(s []int, index int) []int {
ret := make([]int, 0)
ret = append(ret, s[:index]...)
return append(ret, s[index+1:]...)
}
But it doesn't make sense to use this because I'm trying to remove a matching value (element) and not a specific index (for eg index 5) from Steps2.
Basically, for every element that's in slice enter.status, I want to remove that element/value from slice Steps2
Careful:
[5]int64{0,1,2,3,4}
This is an array (of 5 ints), not a slice. And:
Steps2 := Steps
If Steps were a slice, this would copy the slice header without copying the underlying array.
In any case, given some slice s of type T and length len(s), if you are allowed to modify s in place and order is relevant, you generally want to use this algorithm:
func trim(s []T) []T {
out := 0
for i := range s {
if keep(s[i]) {
s[out] = s[i]
out++
}
}
return s[:out]
}
where keep is your boolean function to decide whether to keep an element. To make this produce a new slice, allocate an output slice of the appropriate length (len(s)) at the start and optionally shrink it later, or, if you expect to throw out most elements, make it empty at the start and use append.
When the keep function is "the value of some field in the output slice does not match the value of any earlier kept field" and the type of that field is usable as a key type, you can use a simple map[T2]struct{} to determine whether the value has occurred yet:
seen := make(map[T2]struct{}, len(s))
and then the keep test and copy sequence becomes:
_, ok := seen[s[i].field]
if !ok {
seen[s[i].field] = struct{}{}
s[out] = s[i]
out++
}
The initial size of seen here is optimized on the theory that most values will be kept; if most values will be discarded, make the map initially empty, or small.
I have a JSON file named test.json which contains:
[
{
"name" : "john",
"interests" : ["hockey", "jockey"]
},
{
"name" : "lima",
"interests" : ["eating", "poker"]
}
]
Now I have written a golang script which reads the JSON file to an slice of structs, and then upon a condition check, modifies a struct fields by iterating over the slice.
Here is what I've tried so far:
package main
import (
"log"
"strings"
"io/ioutil"
"encoding/json"
)
type subDB struct {
Name string `json:"name"`
Interests []string `json:"interests"`
}
var dbUpdate []subDB
func getJSON() {
// open the file
filename := "test.json"
val, err := ioutil.ReadFile(filename)
if err != nil {
log.Fatal(err)
}
err = json.Unmarshal(val, &dbUpdate)
}
func (v *subDB) Change(newresponse []string) {
v.Interests = newresponse
}
func updater(name string, newinterest string) {
// iterating over the slice of structs
for _, item := range dbUpdate {
// checking if name supplied matches to the current struct
if strings.Contains(item.Name, name) {
flag := false // declare a flag variable
// item.Interests is a slice, so we iterate over it
for _, intr := range item.Interests {
// check if newinterest is within any one of slice value
if strings.Contains(intr, newinterest) {
flag = true
break // if we find one, we terminate the loop
}
}
// if flag is false, then we change the Interests field
// of the current struct
if !flag {
// Interests holds a slice of strings
item.Change([]string{newinterest}) // passing a slice of string
}
}
}
}
func main() {
getJSON()
updater("lima", "jogging")
log.Printf("%+v\n", dbUpdate)
}
The output I'm getting is:
[{Name:john Interests:[hockey jockey]} {Name:lima Interests:[eating poker]}]
However I should be getting an output like:
[{Name:john Interests:[hockey jockey]} {Name:lima Interests:[jogging]}]
My understanding was that since Change() has a pointer passed, it should directly modify the field. Can anyone point me out what I'm doing wrong?
The problem
Let's cite what the language specification says on the for ... range loops:
A "for" statement with a "range" clause iterates through all entries
of an array, slice, string or map, or values received on a channel.
For each entry it assigns iteration values to corresponding iteration
variables if present and then executes the block.
So, in
for _, item := range dbUpdate { ... }
the whole statement forms a scope in which a variable named item is declared and it gets assigned a value of each element of dbUpdate, in turn, form the first to the last — as the statement performs its iterations.
All assignments in Go, always and everywhere do copy the value of the expression being assigned, into a variable receiving that value.
So, when you have
type subDB struct {
Name string `json:"name"`
Interests []string `json:"interests"`
}
var dbUpdate []subDB
you have a slice whose backing array contains a set of elements, each of which has type subDB.
Consequently, when for ... range iterates over your slice, on each iteration a shallow copy of the fields of a subDB value contained in the current slice element is done: the values of those fields are copied into the variable item.
We could re-write what happes as this:
for i := 0; i < len(dbUpdate); i++ {
var item subDB
item = dbUpdate[i]
...
}
As you can see, if you mutate item in the loop's body, the changes you do to it do not in any way affect the collection's element currently being iterated over.
The solutions
Broadly speaking, the solution is to become fully acquainted with the fact that Go is very simple in most of the stuff it implements, and so range is no magic to: the iteration variable is just a variable, and assignment to it is just an assignment.
As to solving the particular problem, there are multiple ways.
Refer to a collection element by its index
Do
for i := range dbUpdate {
dbUpdate[i].FieldName = value
}
A corollary to this is that sometimes, when the element is complex or you'd like to delegate its mutation to some function, you may take a pointer to it:
for i := range dbUpdate {
p := &dbUpdate[i]
mutateSubDB(p)
}
...
func mutateSubDB(p *subDB) {
p.SomeField = someValue
}
Keep pointers in the slice
If your slice were declated like
var dbUpdates []*subDB
…and you'd keep pointers to (usually heap-allocated) SubDB values,
the
for _, ptr := range dbUpdate { ... }
statement would naturally copy a pointer to a SubDB (anonymous) variable into ptr as the slice contains pointers and so the assignment copies a pointer.
Since all pointers containing the same address are pointing to the same value, mutating the target variable through the pointer kept in the iteration variable would mutate the same thing which is pointed to by the slice's element.
Which approach to select should usually depend on considerations other than thinking about how one would iterate over the elements — simply because once you understand why your code did not work, you do not have this problem anymore.
As usually: if your values are really big, consider keeping pointers to them.
If you values need to be referenced from multiple places at the same time, keep pointers to them. In other cases keep the values directly — this greatly improves CPU data cache locality (simply put, by the time you're about to access the next element its contents will most likely have been already fetched from the memory, which does not occur when the CPU has to chase a pointer to access some arbitrary memory location through it).
Assume I have []struct{} and I need to know whether an element with id = A exists in the slice. If exists, the element will be removed or moved to index 0 according to request in user input. So, how to find an element in golang slice in optimal way without check each element? Or, is using slice.contains(obj) enough? Then, if the element exists, I will do action according to request in user input. If the request is remove, I will remove it without broke the elements order. But if the request is add, I will move the element to index 0.
Note: The function will be often called.
Thank you.
It is not difficult to write function to find element by iterating over slice:
func contains(s []your_struct, e int) (bool, int) {
for idx, a := range s {
if a.id == e {
return true, idx
}
}
return false, -1
}
If you a going to call the function often it may be useful to sort the slice by id field and implement binary search over slice of your_struct.
If the slice is not very big you can create additional data structure - map[int]int and keep the indexes of elements of the slice in this map. But in this case you need to synchronize content of your slice and the map when you are modifying one of them:
your_map := make(map[int]int)
if idx, ok := your_map[id]; ok {
// ...
}
If you need to check many times then
it's better to create a map[string]int of id field one time.
And every time just check map contains that id or not
Here,id as key and slice index as value
mp := make(map[string]int)
for idx, a := range yourStuctSlice {
mp[a.id] = idx
}
if idx, ok := mp[id]; ok {
// remove the element using idx
}
If new element added in slice then update the map also
mp[newElement.id] = true
If you want to remove searched element you can remove by slice index
func RemoveIndex(s []yourStuct, index int) []int {
return append(s[:index], s[index+1:]...)
}
if idx, ok := mp[id]; ok {
yourStuctSlice = RemoveIndex(yourStuctSlice , idx)
delete(mp , id); // Remove from map also for next search
}
In Go, I am trying to make a scramble slice function for my traveling salesman problem. While doing this I noticed when I started editing the slice I gave the scramble function was different every time I passed it in.
After some debugging I found out it was due to me editing the slice inside the function. But since Go is supposed to be a "pass by value" language, how is this possible?
https://play.golang.org/p/mMivoH0TuV
I have provided a playground link to show what I mean.
By removing line 27 you get a different output than leaving it in, this should not make a difference since the function is supposed to make its own copy of the slice when passed in as an argument.
Can someone explain the phenomenon?
Everything in Go is passed by value, slices too. But a slice value is a header, describing a contiguous section of a backing array, and a slice value only contains a pointer to the array where the elements are actually stored. The slice value does not include its elements (unlike arrays).
So when you pass a slice to a function, a copy will be made from this header, including the pointer, which will point to the same backing array. Modifying the elements of the slice implies modifying the elements of the backing array, and so all slices which share the same backing array will "observe" the change.
To see what's in a slice header, check out the reflect.SliceHeader type:
type SliceHeader struct {
Data uintptr
Len int
Cap int
}
See related / possible duplicate question:
Performance of function slice parameter vs global variable?
Read blog post: Go Slices: usage and internals
Please note that when you pass a slice to a function, if the function modifies the "existing" elements of the slice, the caller will see / observe the changes. If the function adds new elements to the slice, that requires changing the slice header (the length at a minimum, but may also involve allocating a new backing array), which the caller will not see (not without returning the new slice header).
Not with maps, because maps are pointers under the hood, and if you pass a map to a function and the function adds a new entry to the map, the map pointer will not change so the caller will see the changed map (the new entry) without returning the map after change.
Also regarding slices and maps, see Map initialization in Go and why slice values can sometimes go stale but never map values?
You can find an example below. Briefly slices is also passed by value but original slice and copied slice are linked to the same underlying array. If one of this slice changes, then underlying array changes, then other slice changes.
package main
import "fmt"
func main() {
x := []int{1, 10, 100, 1000}
double(x)
fmt.Println(x) // ----> 3 will print [2, 20, 200, 2000] (original slice changed)
}
func double(y []int) {
fmt.Println(y) // ----> 1 will print [1, 10, 100, 1000]
for i := 0; i < len(y); i++ {
y[i] *= 2
}
fmt.Println(y) // ----> 2 will print [2, 20, 200, 2000] (copy slice + under array changed)
}
Slices when its passed it’s passed with the pointer to underlying array, so a slice is a small structure that points to an underlying array. The small structure is copied, but it still points to the same underlying array. the memory block containing the slice elements is passed by "reference". The slice information triplet holding the capacity, the number of element and the pointer to the elements is passed by value.
The best way to handle slices passing to function (if the elements of the slice are manipulated into the function, and we do not want this to be reflected at the elements memory block is to copy them using copy(s, *c) as:
package main
import "fmt"
type Team []Person
type Person struct {
Name string
Age int
}
func main() {
team := Team{
Person{"Hasan", 34}, Person{"Karam", 32},
}
fmt.Printf("original before clonning: %v\n", team)
team_cloned := team.Clone()
fmt.Printf("original after clonning: %v\n", team)
fmt.Printf("clones slice: %v\n", team_cloned)
}
func (c *Team) Clone() Team {
var s = make(Team, len(*c))
copy(s, *c)
for index, _ := range s {
s[index].Name = "change name"
}
return s
}
But be careful, if this slice is containing a sub slice further copying is required, as we'll still have the sub slice elements sharing pointing to the same memory block elements, an example is:
type Inventories []Inventory
type Inventory struct { //instead of: map[string]map[string]Pairs
Warehouse string
Item string
Batches Lots
}
type Lots []Lot
type Lot struct {
Date time.Time
Key string
Value float64
}
func main() {
ins := Inventory{
Warehouse: "DMM",
Item: "Gloves",
Batches: Lots{
Lot{mustTime(time.Parse(custom, "1/7/2020")), "Jan", 50},
Lot{mustTime(time.Parse(custom, "2/1/2020")), "Feb", 70},
},
}
inv2 := CloneFrom(c Inventories)
}
func (i *Inventories) CloneFrom(c Inventories) {
inv := new(Inventories)
for _, v := range c {
batches := Lots{}
for _, b := range v.Batches {
batches = append(batches, Lot{
Date: b.Date,
Key: b.Key,
Value: b.Value,
})
}
*inv = append(*inv, Inventory{
Warehouse: v.Warehouse,
Item: v.Item,
Batches: batches,
})
}
(*i).ReplaceBy(inv)
}
func (i *Inventories) ReplaceBy(x *Inventories) {
*i = *x
}
Slice will work with pass by value to the function, But we should not use append to add values to slice in the function, instead we should use the assignment directly. Reason being that append will create new memory and copy values to that. Here is the example.
Go playground
// Go program to illustrate how to
// pass a slice to the function
package main
import "fmt"
// Function in which slice
// is passed by value
func myfun(element []string) {
// Here we only modify the slice
// Using append function
// Here, this function only modifies
// the copy of the slice present in
// the function not the original slice
element = append(element, "blackhole")
fmt.Println("Modified slice: ", element)
}
func main() {
// Creating a slice
slc := []string{"rocket", "galaxy", "stars", "milkyway"}
fmt.Println("Initial slice: ", slc)
//slice pass by value
myfun(slc)
fmt.Println("Final slice: ", slc)
}
Output-
Initial slice: [rocket galaxy stars milkyway]
Modified slice: [rocket galaxy stars milkyway blackhole]
Final slice: [rocket galaxy stars milkyway]
Go Playground
// Go program to illustrate how to
// pass a slice to the function
package main
import "fmt"
// Function in which slice
// is passed by value
func myfun(element []string) {
// Here we only modify the slice
// Using append function
// Here, this function only modifies
// the copy of the slice present in
// the function not the original slice
element[0] = "Spaceship"
element[4] = "blackhole"
element[5] = "cosmos"
fmt.Println("Modified slice: ", element)
}
func main() {
// Creating a slice
slc := []string{"rocket", "galaxy", "stars", "milkyway", "", ""}
fmt.Println("Initial slice: ", slc)
//slice pass by value
myfun(slc)
fmt.Println("Final slice: ", slc)
}
Output-
Initial slice: [rocket galaxy stars milkyway ]
Modified slice: [Spaceship galaxy stars milkyway blackhole cosmos]
Final slice: [Spaceship galaxy stars milkyway blackhole cosmos]
To complement this post, here is an example of passing by reference for the Golang PlayGround you shared:
type point struct {
x int
y int
}
func main() {
data := []point{{1, 2}, {3, 4}, {5, 6}, {7, 8}}
makeRandomDatas(&data)
}
func makeRandomDatas(dataPoints *[]point) {
for i := 0; i < 10; i++ {
if len(*dataPoints) > 0 {
fmt.Println(makeRandomData(dataPoints))
} else {
fmt.Println("no more elements")
}
}
}
func makeRandomData(cities *[]point) []point {
solution := []point{(*cities)[0]} //create a new slice with the first item from the old slice
*cities = append((*cities)[:0], (*cities)[1:]...) //remove the first item from the old slice
return solution
}
I am trying to create a generic function that can handle actions on slices in Go... for instance, append an item of any type to a slice of that same type. This is simply a generic purpose for a more complex solution, but overall the issue boils down to this example:
package main
type car struct {
make string
color string
}
type submarine struct {
name string
length int
}
func genericAppender(thingList interface{}, thing interface{}) []interface{} {
return append(thingList, thing)
}
func main() {
cars := make([]car, 0, 10)
cars[0] = car{make: "ford", color: "red"}
cars[1] = car{make: "chevy", color: "blue"}
subs := make([]submarine, 0, 10)
subs[0] = submarine{name: "sally", length: 100}
subs[1] = submarine{name: "matilda", length: 200}
newCar := car{make: "bmw", color: "white"}
genericAppender(&cars, newCar)
}
The code playground is at this location
The above errors as follows:
prog.go:14: first argument to append must be slice; have interface {}
After this change you're still getting a runtime error (index out of range) however the problem is that thingList is not of type []interface{} but rather interface{} so you can't append to it. Here's an updated version of your code on playground that does a type assertion to convert it to an []interface{} in line with the append. In reality you need to do that on a separate line and check for errors.
https://play.golang.org/p/YMed0VDZrv
So to put some code here;
func genericAppender(thingList interface{}, thing interface{}) []interface{} {
return append(thingList.([]interface{}), thing)
}
will solve the basic problem you're facing. As noted, you still get runtime errors when indexing into the slice. Also, you could change the argument to avoid this by making it;
func genericAppender(thingList []interface{}, thing interface{}) []interface{} {
return append(thingList, thing)
}
Here's a complete example of the second type; https://play.golang.org/p/dIuW_UG7XY
Note I also corrected the runtime error. When you use make with 3 args they are, in this order, type, length, capacity. This means the length of the array is 0 so when you try to assign to indexes 0 and 1 it was causing a panic for IndexOutoFRange. Instead I removed the middle argument so it's make([]interface{}, 10) meaning the length is initially set to 10 so you can assign to those indexes.
In the answer above if you do the following then it throws error. This is what the original question was about:
//genericAppender(subs, newCar). // Throws "cannot use subs (type []submarine) as type []interface {} in argument to genericAppender"
The trick is to convert your slice of specific type into a generic []interface{}.
func convertToGeneric(thingList interface{}) []interface{} {
input := reflect.ValueOf(thingList)
length := input.Len()
out := make([]interface{},length)
for i:=0 ;i < length; i++ {
out[i] = input.Index(i).Interface()
}
return out
}
This you can call the function like this:
genericAppender(convertToGeneric(subs), newCar)
You can check modified working code here: https://play.golang.org/p/0_Zmme3c8lT
With Go 1.19 (Q4 2022), no need for interface, or "convert your slice of specific type into a generic []interface{}"
CL 363434 comes with a new slices packages:
// Package slices defines various functions useful with slices of any type.
// Unless otherwise specified, these functions all apply to the elements
// of a slice at index 0 <= i < len(s).
package slices
import "constraints"
// Grow increases the slice's capacity, if necessary, to guarantee space for
// another n elements. After Grow(n), at least n elements can be appended
// to the slice without another allocation. If n is negative or too large to
// allocate the memory, Grow panics.
func Grow[S ~[]T, T any](s S, n int) S {
return append(s, make(S, n)...)[:len(s)]
}
// Equal reports whether two slices are equal: the same length and all
// elements equal. If the lengths are different, Equal returns false.
// Otherwise, the elements are compared in index order, and the
// comparison stops at the first unequal pair.
// Floating point NaNs are not considered equal.
func Equal[T comparable](s1, s2 []T) bool {
if len(s1) != len(s2) {
return false
}
for i, v1 := range s1 {
v2 := s2[i]
if v1 != v2 {
return false
}
}
return true
}
// ...
Ian Lance Taylor confirms in issue 45955:
This package is now available at golang.org/x/exp/slices.
Per this thread, it will not be put into standard library until the 1.19 release.
We may of course adjust it based on anything we learn about having it in x/exp.