I'm trying to implement a very simple queue in Go using slices. This is the code that I have to enqueue five values and then discard the first two values:
package main
import (
"fmt"
)
var (
localQ []int
)
func main() {
fmt.Printf("%v %v\n", localQ, len(localQ))
for i := 0; i< 5; i++ {
localQ = enqueue(localQ, i)
fmt.Printf("%v %v\n", localQ, len(localQ))
}
localQ = dequeue(localQ, 2)
fmt.Printf("%v %v\n", localQ, len(localQ))
}
func enqueue(q []int, n int) ([]int) {
q = append(q, n)
return q
}
func dequeue(q []int, s int) ([]int) {
r := q[s:]
q = nil
return r
}
Two questions regarding the dequeue func:
1- I'm trying to ensure that the popped items are discarded and garbage collected. Does this function result them to be garbage collected?
2- What are the time and space complexities of r := q[s:]? I know there is an array under each slice. Are the array values being copied? Or is it just a pointer being copied?
Does this function result them to be garbage collected?
If the application enqueues a sufficient number of elements to cause the slice backing array to be reallocated, then the previous backing array (and its elements) will be eligible for collection.
What are the time and space complexities of r := q[s:]?
This is an O(1) operation. The operation does not allocate memory on the heap.
Related
I have a map[string]int
I want to get the x top values from it and store them in another data structure, another map or a slice.
From https://blog.golang.org/go-maps-in-action#TOC_7. I understood that:
When iterating over a map with a range loop, the iteration order is
not specified and is not guaranteed to be the same from one iteration
to the next.
so the result structure will be a slice then.
I had a look at several related topics but none fits my problem:
related topic 1
related topic 2
related topic 3
What would be the most efficient way to do this please?
Thanks,
Edit:
My solution would be to turn my map into a slice and sort it, then extract the first x values.
But is there a better way ?
package main
import (
"fmt"
"sort"
)
func main() {
// I want the x top values
x := 3
// Here is the map
m := make(map[string]int)
m["k1"] = 7
m["k2"] = 31
m["k3"] = 24
m["k4"] = 13
m["k5"] = 31
m["k6"] = 12
m["k7"] = 25
m["k8"] = -8
m["k9"] = -76
m["k10"] = 22
m["k11"] = 76
// Turning the map into this structure
type kv struct {
Key string
Value int
}
var ss []kv
for k, v := range m {
ss = append(ss, kv{k, v})
}
// Then sorting the slice by value, higher first.
sort.Slice(ss, func(i, j int) bool {
return ss[i].Value > ss[j].Value
})
// Print the x top values
for _, kv := range ss[:x] {
fmt.Printf("%s, %d\n", kv.Key, kv.Value)
}
}
Link to golang playground example
If I want to have a map at the end with the x top values, then with my solution I would have to turn the slice into a map again. Would this still be the most efficient way to do it?
Creating a slice and sorting is a fine solution; however, you could also use a heap. The Big O performance should be equal for both implementations (n log n) so this is a viable alternative with the advantage that if you want to add new entries you can still efficiently access the top N items without repeatedly sorting the entire set.
To use a heap, you would implement the heap.Interface for the kv type with a Less function that compares Values as greater than (h[i].Value > h[j].Value), add all of the entries from the map, and then pop the number of items you want to use.
For example (Go Playground):
func main() {
m := getMap()
// Create a heap from the map and print the top N values.
h := getHeap(m)
for i := 1; i <= 3; i++ {
fmt.Printf("%d) %#v\n", i, heap.Pop(h))
}
// 1) main.kv{Key:"k11", Value:76}
// 2) main.kv{Key:"k2", Value:31}
// 3) main.kv{Key:"k5", Value:31}
}
func getHeap(m map[string]int) *KVHeap {
h := &KVHeap{}
heap.Init(h)
for k, v := range m {
heap.Push(h, kv{k, v})
}
return h
}
// See https://golang.org/pkg/container/heap/
type KVHeap []kv
// Note that "Less" is greater-than here so we can pop *larger* items.
func (h KVHeap) Less(i, j int) bool { return h[i].Value > h[j].Value }
func (h KVHeap) Swap(i, j int) { h[i], h[j] = h[j], h[i] }
func (h KVHeap) Len() int { return len(h) }
func (h *KVHeap) Push(x interface{}) {
*h = append(*h, x.(kv))
}
func (h *KVHeap) Pop() interface{} {
old := *h
n := len(old)
x := old[n-1]
*h = old[0 : n-1]
return x
}
I need to create a 2 dimensional string array as shown below -
matrix = [['cat,'cat','cat'],['dog','dog']]
Code:-
package main
import (
"fmt"
)
func main() {
{ // using append
var matrix [][]string
matrix[0] = append(matrix[0],'cat')
fmt.Println(matrix)
}
}
Error:-
panic: runtime error: index out of range
goroutine 1 [running]:
main.main()
/tmp/sandbox863026592/main.go:11 +0x20
You have a slice of slices, and the outer slice is nil until it's initialized:
matrix := make([][]string, 1)
matrix[0] = append(matrix[0],'cat')
fmt.Println(matrix)
Or:
var matrix [][]string
matrix = append(matrix, []string{"cat"})
fmt.Println(matrix)
Or:
var matrix [][]string
var row []string
row = append(row, "cat")
matrix = append(matrix, row)
The problem with doing two-dimensional arrays with Go is that you have to initialise each part individually, e.g., if you have a [][]bool, you gotta allocate []([]bool) first, and then allocate the individual []bool afterwards; this is the same logic regardless of whether you're using make() or append() to perform the allocations.
In your example, the matrix[0] doesn't exist yet after a mere var matrix [][]string, hence you're getting the index out of range error.
For example, the code below would create another slice based on the size of an existing slice of a different type:
func solve(board [][]rune, …) {
x := len(board)
y := len(board[0])
visited := make([][]bool, x)
for i := range visited {
visited[i] = make([]bool, y)
}
…
If you simply want to initialise the slice based on a static array you have, you can do it directly like this, without even having to use append() or make():
package main
import (
"fmt"
)
func main() {
matrix := [][]string{{"cat", "cat", "cat"}, {"dog", "dog"}}
fmt.Println(matrix)
}
https://play.golang.org/p/iWgts-m7c4u
Is it possible to LoadOrStore into a Go sync.Map without creating a new structure every time? If not, what alternatives are available?
The use case here is if I'm using the sync.Map as a cache where cache misses are rare (but possible) and on a cache miss I want to add to the map, I need to initialize a structure every single time LoadOrStore is called rather than just creating the struct when needed. I'm worried this will hurt the GC, initializing hundreds of thousands of structures that will not be needed.
In Java this can be done using computeIfAbsent.
you can try:
var m sync.Map
s, ok := m.Load("key")
if !ok {
s, _ = m.LoadOrStore("key", "value")
}
fmt.Println(s)
play demo
This is my solution: use sync.Map and sync.One
type syncData struct {
data interface{}
once *sync.Once
}
func LoadOrStore(m *sync.Map, key string, f func() (interface{}, error)) (interface{}, error) {
temp, _ := m.LoadOrStore(key, &syncData{
data: nil,
once: &sync.Once{},
})
d := temp.(*syncData)
var err error
if d.data == nil {
d.once.Do(func() {
d.data, err = f()
if err != nil {
//if failed, will try again by new sync.Once
d.once = &sync.Once{}
}
})
}
return d.data, err
}
Package sync
import "sync"
type Map
Map is like a Go map[interface{}]interface{} but is safe for
concurrent use by multiple goroutines without additional locking or
coordination. Loads, stores, and deletes run in amortized constant
time.
The Map type is specialized. Most code should use a plain Go map
instead, with separate locking or coordination, for better type safety
and to make it easier to maintain other invariants along with the map
content.
The Map type is optimized for two common use cases: (1) when the entry
for a given key is only ever written once but read many times, as in
caches that only grow, or (2) when multiple goroutines read, write,
and overwrite entries for disjoint sets of keys. In these two cases,
use of a Map may significantly reduce lock contention compared to a Go
map paired with a separate Mutex or RWMutex.
The usual way to solve these problems is to construct a usage model and then benchmark it.
For example, since "cache misses are rare", assume that Load wiil work most of the time and only LoadOrStore (with value allocation and initialization) when necessary.
$ go test map_test.go -bench=. -benchmem
BenchmarkHit-4 2 898810447 ns/op 44536 B/op 1198 allocs/op
BenchmarkMiss-4 1 2958103053 ns/op 483957168 B/op 43713042 allocs/op
$
map_test.go:
package main
import (
"strconv"
"sync"
"testing"
)
func BenchmarkHit(b *testing.B) {
for N := 0; N < b.N; N++ {
var m sync.Map
for i := 0; i < 64*1024; i++ {
for k := 0; k < 256; k++ {
// Assume cache hit
v, ok := m.Load(k)
if !ok {
// allocate and initialize value
v = strconv.Itoa(k)
a, loaded := m.LoadOrStore(k, v)
if loaded {
v = a
}
}
_ = v
}
}
}
}
func BenchmarkMiss(b *testing.B) {
for N := 0; N < b.N; N++ {
var m sync.Map
for i := 0; i < 64*1024; i++ {
for k := 0; k < 256; k++ {
// Assume cache miss
// allocate and initialize value
var v interface{} = strconv.Itoa(k)
a, loaded := m.LoadOrStore(k, v)
if loaded {
v = a
}
_ = v
}
}
}
}
What's the most efficient way of inserting an element to a sorted slice?
I tried a couple of things but all ended up using at least 2 appends which as I understand makes a new copy of the slice
Here is how to insert into a sorted slice of strings:
Go Playground Link to full example: https://play.golang.org/p/4RkVgEpKsWq
func Insert(ss []string, s string) []string {
i := sort.SearchStrings(ss, s)
ss = append(ss, "")
copy(ss[i+1:], ss[i:])
ss[i] = s
return ss
}
If the slice has enough capacity then there's no need for a new copy.
The elements after the insert position can be shifted to the right.
Only when the slice doesn't have enough capacity,
a new slice and copying all values will be necessary.
Keep in mind that slices are not designed for fast insertion.
So there won't be a miracle solution here using slices.
You could create a custom data structure to make this more efficient,
but obviously there will be other trade-offs.
One point that can be optimized in the process is finding the insertion point quickly. If the slice is sorted, then you can use binary search to perform this in O(log n) time.
However, this might not matter much,
considering the expensive operation of copying the end of the slice,
or reallocating when necessary.
I like #likebike's answer but it only works for strings. Here is the generic version that will work for a slice of any ordered type (requires Go 1.18):
func Insert[T constraints.Ordered](ts []T, t T) []T {
var dummy T
ts = append(ts, dummy) // extend the slice
i, _ := slices.BinarySearch(ts, t) // find slot
copy(ts[i+1:], ts[i:]) // make room
ts[i] = t
return ts
}
Note that this uses the package golang.org/x/exp/slices but this will almost certainly be included in the std Go library in Go 1.19.
Try it in the Go Playground
There are two parts to the problem: finding where to insert the value and inserting the value.
Use the sort package search functions to efficiently find the insertion index using binary search.
Use a single call to append to efficiently insert a value into a slice:
// insertAt inserts v into s at index i and returns the new slice.
func insertAt(data []int, i int, v int) []int {
if i == len(data) {
// Insert at end is the easy case.
return append(data, v)
}
// Make space for the inserted element by shifting
// values at the insertion index up one index. The call
// to append does not allocate memory when cap(data) is
// greater than len(data).
data = append(data[:i+1], data[i:]...)
// Insert the new element.
data[i] = v
// Return the updated slice.
return data
}
Here's the code for inserting a value a sorted slice:
func insertSorted(data []int, v int) []int {
i := sort.Search(len(data), func(i int) bool { return data[i] >= v })
return insertAt(data, i, v)
}
The code in this answer uses a slice of int. Adjust the type to match your actual data.
The call to sort.Search in this answer can be replaced with a call to the helper function sort.SearchInts. I show sort.Search in this answer because the function applies to a slice of any type.
If you do not want to add duplicate values, check the value at the search index before inserting:
func insertSortedNoDups(data []int, v int) []int {
i := sort.Search(len(data), func(i int) bool { return data[i] >= v })
if i < len(data) && data[i] == v {
return data
}
return insertAt(data, i, v)
}
You could use a heap:
package main
import (
"container/heap"
"sort"
)
type slice struct { sort.IntSlice }
func (s slice) Pop() interface{} { return 0 }
func (s *slice) Push(x interface{}) {
(*s).IntSlice = append((*s).IntSlice, x.(int))
}
func main() {
s := &slice{
sort.IntSlice{11, 10, 14, 13},
}
heap.Init(s)
heap.Push(s, 12)
println(s.IntSlice[0] == 10)
}
Note that a heap is not strictly sorted, but the "minimum element" is guaranteed
to be the first element. Also I did not implement the Pop function in my
example, you would want to do that.
https://golang.org/pkg/container/heap
There are two approaches mentioned here to insert into the slice when the position i is known:
data = append(data, "")
copy(data[i+1:], data[i:])
data[i] = s
and
data = append(data[:i+1], data[i:]...)
data[i] = s
I just benchmarked both with go1.18beta2, and the first solution is approximately 10% faster.
no dependency, generic data type with duplicated options. (go 1.18)
time complexity : Log2(n) + 1
import "golang.org/x/exp/constraints"
import "golang.org/x/exp/slices"
func InsertionSort[T constraints.Ordered](array []T, value T, canDupicate bool) []T {
pos, isFound := slices.BinarySearch(array, value)
if canDupicate || !isFound {
array = slices.Insert(array, pos, value)
}
return array
}
full version : https://go.dev/play/p/P2_ou2Fqs37
play : https://play.golang.org/p/dUGmPurouxA
array1 := []int{1, 3, 4, 5}
//want to insert at index 1
insertAtIndex := 1
temp := append([]int{}, array1[insertAtIndex:]...)
array1 = append(array1[0:insertAtIndex], 2)
array1 = append(array1, temp...)
fmt.Println(array1)
You can try the below code. It basically uses the golang sort package
package main
import "sort"
import "fmt"
func main() {
data := []int{20, 21, 22, 24, 25, 26, 28, 29, 30, 31, 32}
var items = []int{23, 27}
for _, x := range items {
i := sort.Search(len(data), func(i int) bool { return data[i] >= x })
if i < len(data) && data[i] == x {
fmt.Println(i)
} else {
data = append(data, 0)
copy(data[i+1:], data[i:])
data[i] = x
}
fmt.Println(data)
}
}
If I have two pointers to slices s1, s2, which initially point to the same slice, is it possible to append to one of the slices and have the other slice also point to the updated slice? This seems to be a problem because, appending to slices might make a new slice with the entries copied over if the capacity is insufficient.
The following is a slightly complicated version on the Go Playground that is closer to my use case. Namely, I have nodes that have a pointer to a (global) queue which is implemented by a slice. When one node updates the global queue, I want that to be reflected in the slice that the other node points to.
https://play.golang.org/p/NG11HbLBrI
Add a indirect layer, i.e. a pointer, can help you work this out
https://play.golang.org/p/R7JILCbYTF
package main
import (
"fmt"
)
type queue *[]int
type node struct {
id int
list *queue
}
func (n *node) mutate(){
newSlice := append(**n.list, 1)
*n.list = &newSlice
}
func (n *node) get(idx int) int{
return (*(*[]int)(*n.list))[idx]
}
func main() {
/* Make empty queue */
s:=make([]int, 0)
q1:=queue(&s)
fmt.Println(q1)
/* Make new nodes */
n1 := new(node)
n1.id = 0
n1.list = &q1
fmt.Println(n1)
n2 := new(node)
n2.id = 1
n2.list = n1.list
fmt.Println(n2)
/* Mutate node in n1 */
n1.mutate()
fmt.Println(n1, n2) // n1.list != n2.list
fmt.Println(n1.list, n2.list)
fmt.Println(n1.get(0))
n2.mutate()
fmt.Println(n2.get(0))
}