Minimum value of set in idiomatic Go - algorithm

How do I write a function that returns the minimum value of a set in go? I am not just looking for a solution (I know I could just initialize the min value when iterating over the first element and then set a boolean variable that I initialized the min value) but rather an idiomatic solution. Since go doesn't have native sets, assume we have a map[Cell]bool.

Maps are the idiomatic way to implement sets in Go. Idiomatic code uses either bool or struct{} as the map's value type. The latter uses less storage, but requires a little more typing at the keyboard to use.
Assuming that the maximum value for a cell is maxCell, then this function will compute the min:
func min(m map[Cell]bool) Cell {
min := maxCell
for k := range m {
if k < min {
min = k
}
}
return min
}
If Cell is a numeric type, then maxCell can be set to one of the math constants.
Any solution using a map will require a loop over the keys.
You can keep a heap in addition to the map to find a minimum. This will require more storage and code, but can be more efficient depending on the size of the set and how often the minimum function is called.

A different approach and depending on how big your set is, using a self-sorting-slice can be more efficient:
type Cell uint64
type CellSet struct {
cells []Cell
}
func (cs *CellSet) Len() int {
return len(cs.cells)
}
func (cs *CellSet) Swap(i, j int) {
cs.cells[i], cs.cells[j] = cs.cells[j], cs.cells[i]
}
func (cs *CellSet) Less(i, j int) bool {
return cs.cells[i] < cs.cells[j]
}
func (cs *CellSet) Add(c Cell) {
for _, v := range cs.cells {
if v == c {
return
}
}
cs.cells = append(cs.cells, c)
sort.Sort(cs)
}
func (cs *CellSet) Min() Cell {
if cs.Len() > 0 {
return cs.cells[0]
}
return 0
}
func (cs *CellSet) Max() Cell {
if l := cs.Len(); l > 0 {
return cs.cells[l-1]
}
return ^Cell(0)
}
playground // this is a test file, copy it to set_test.go and run go test -bench=. -benchmem -v
BenchmarkSlice 20 75385089 ns/op 104 B/op 0 allocs/op
BenchmarkMap 20 77541424 ns/op 158 B/op 0 allocs/op
BenchmarkSliceAndMin 20 77155563 ns/op 104 B/op 0 allocs/op
BenchmarkMapAndMin 1 1827782378 ns/op 2976 B/op 8 allocs/op

Related

How to create recursive concurrent tree search in go

Is there a proper way to implement concurrency inside recursive tree search in go?
The following code shows basic recursive tree search for a tree of nodes having id and children properties.
If I find a result I can close the channel but in case there is no result how to deal with that?
// Node is a struct containing int field ID and []*Node field Children.
type Node struct {
ID int
Children []*Node
}
// Find search for a record by id inside an array of records
func Find(node *Node, id int) *Node {
if node.ID == id {
return node
}
for _, c := range node.Children {
found := Find(c, id)
if found != nil {
return found
}
}
return nil
}
here is a simple and stupid benchmark that i hope will be helpful to OP to better understand impacts of simplistic implementations of concurrent programming.
package main_test
import (
"sync"
"testing"
)
func SearchNonConcurrently(v int, mySlice []int) int {
for idx, i := range mySlice {
if i == v {
return idx
}
}
return -1
}
func SearchConcurrently(v int, mySlice []int) int {
var wg sync.WaitGroup
wg.Add(len(mySlice))
res := make(chan int)
for idx, i := range mySlice {
go func(idx, i int) {
defer wg.Done()
if i == v {
res <- idx
}
}(idx, i)
}
go func() {
wg.Wait()
close(res)
}()
idx, ok := <-res
if !ok {
return -1
}
return idx
}
func BenchmarkSearchNonConcurrently(b *testing.B) {
mySlice := make([]int, 10000)
mySlice[len(mySlice)-1] = 1
for n := 0; n < b.N; n++ {
SearchNonConcurrently(1, mySlice)
}
}
func BenchmarkSearchConcurrently(b *testing.B) {
mySlice := make([]int, 10000)
mySlice[len(mySlice)-1] = 1
for n := 0; n < b.N; n++ {
SearchConcurrently(1, mySlice)
}
}
The results:
$ go test -bench=. -benchmem -count=4
goos: linux
goarch: amd64
pkg: test/bencslice
BenchmarkSearchNonConcurrently-4 85780 13831 ns/op 0 B/op 0 allocs/op
BenchmarkSearchNonConcurrently-4 83619 13880 ns/op 0 B/op 0 allocs/op
BenchmarkSearchNonConcurrently-4 87128 14050 ns/op 0 B/op 0 allocs/op
BenchmarkSearchNonConcurrently-4 87366 13837 ns/op 0 B/op 0 allocs/op
BenchmarkSearchConcurrently-4 235 4867257 ns/op 2200 B/op 6 allocs/op
BenchmarkSearchConcurrently-4 244 4896937 ns/op 3360 B/op 8 allocs/op
BenchmarkSearchConcurrently-4 240 4871934 ns/op 453 B/op 2 allocs/op
BenchmarkSearchConcurrently-4 244 4884617 ns/op 3352 B/op 8 allocs/op
PASS
ok test/bencslice 12.087s
$ go version
go version go1.15.2 linux/amd64
I created the follwing code. My only regret is that it keeps creating go routines even after finding the solution. I mean for a sorted tree with 1000 elements if I search for node with ID = 5 it keeps opening cases with above 900.
func FindConcurrent(node *Node, id int, c chan *Node) {
select {
case <-c:
return
default:
}
if node.ID == id {
c <- node
close(c)
return
}
for _, child := range node.Children {
go FindConcurrent(child, id, c)
}
}

go maps non-performant for large number of keys

I discovered very strange behaviour with go maps recently. The use case is to create a group of integers and have O(1) check for IsMember(id int).
The current implementation is :
func convertToMap(v []int64) map[int64]void {
out := make(map[int64]void, len(v))
for _, i := range v {
out[i] = void{}
}
return out
}
type Group struct {
members map[int64]void
}
type void struct{}
func (g *Group) IsMember(input string) (ok bool) {
memberID, _ := strconv.ParseInt(input, 10, 64)
_, ok = g.members[memberID]
return
}
When i benchmark the IsMember method, until 6 million members, everything looks fine. But above that the map look up is taking 1 second for each lookup!!
The benchmark test:
func BenchmarkIsMember(b *testing.B) {
b.ReportAllocs()
b.ResetTimer()
g := &Group{}
g.members = convertToMap(benchmarkV)
for N := 0; N < b.N && N < sizeOfGroup; N++ {
g.IsMember(benchmarkKVString[N])
}
}
var benchmarkV, benchmarkKVString = func(size int) ([]int64, []string{
v := make([]int64, size)
s := make([]string, size)
for i := range v {
val := rand.Int63()
v[i] = val
s[i] = strconv.FormatInt(val, 10)
}
return v, s
}(sizeOfGroup)
Benchmark numbers:
const sizeOfGroup = 6000000
BenchmarkIsMember-8 2000000 568 ns/op 50 B/op 0 allocs/op
const sizeOfGroup = 6830000
BenchmarkIsMember-8 1 1051725455 ns/op 178767208 B/op 25 allocs/op
Anything above group size of 6.8 million gives the same result.
Can someone help me to explain why this is happening, and can anything be done to make this performant while still using maps?
Also, i dont understand why so much memory is being allocated? Even if the time taken is due to collision and then linked list traversal, there shouldn't be any mem allocation, is my thought process wrong?
No need to measure extra allocation for converting slice to map because we just want to measure the lookup operation.
I've slightly modify the benchmark:
func BenchmarkIsMember(b *testing.B) {
fn := func(size int) ([]int64, []string) {
v := make([]int64, size)
s := make([]string, size)
for i := range v {
val := rand.Int63()
v[i] = val
s[i] = strconv.FormatInt(val, 10)
}
return v, s
}
for _, size := range []int{
6000000,
6800000,
6830000,
60000000,
} {
b.Run(fmt.Sprintf("size=%d", size), func(b *testing.B) {
var benchmarkV, benchmarkKVString = fn(size)
g := &deltaGroup{}
g.members = convertToMap(benchmarkV)
b.ReportAllocs()
b.ResetTimer()
for N := 0; N < b.N && N < size; N++ {
g.IsMember(benchmarkKVString[N])
}
})
}
}
And got the following results:
go test ./... -bench=. -benchtime=10s -cpu=1
goos: linux
goarch: amd64
pkg: trash
BenchmarkIsMember/size=6000000 2000000000 0.55 ns/op 0 B/op 0 allocs/op
BenchmarkIsMember/size=6800000 1000000000 1.27 ns/op 0 B/op 0 allocs/op
BenchmarkIsMember/size=6830000 1000000000 1.23 ns/op 0 B/op 0 allocs/op
BenchmarkIsMember/size=60000000 100000000 136 ns/op 0 B/op 0 allocs/op
PASS
ok trash 167.578s
Degradation isn't so significant as in your example.

How to check if []byte is all zeros in go

Is there a way to check if a byte slice is empty or 0 without checking each element or using reflect?
theByteVar := make([]byte, 128)
if "theByteVar is empty or zeroes" {
doSomething()
}
One solution which seems weird that I found was to keep an empty byte array for comparison.
theByteVar := make([]byte, 128)
emptyByteVar := make([]byte, 128)
// fill with anything
theByteVar[1] = 2
if reflect.DeepEqual(theByteVar,empty) == false {
doSomething(theByteVar)
}
For sure there must be a better/quicker solution.
Thanks
UPDATE did some comparison for 1000 loops and the reflect way is the worst by far...
Equal Loops: 1000 in true in 19.197µs
Contains Loops: 1000 in true in 34.507µs
AllZero Loops: 1000 in true in 117.275µs
Reflect Loops: 1000 in true in 14.616277ms
Comparing it with another slice containing only zeros, that requires reading (and comparing) 2 slices.
Using a single for loop will be more efficient here:
for _, v := range theByteVar {
if v != 0 {
doSomething(theByteVar)
break
}
}
If you do need to use it in multiple places, wrap it in a utility function:
func allZero(s []byte) bool {
for _, v := range s {
if v != 0 {
return false
}
}
return true
}
And then using it:
if !allZero(theByteVar) {
doSomething(theByteVar)
}
Another solution borrows an idea from C. It could be achieved by using the unsafe package in Go.
The idea is simple, instead of checking each byte from []byte, we can check the value of byte[i:i+8], which is a uint64 value, in each steps. By doing this we can check 8 bytes instead of checking only one byte in each iteration.
Below codes are not best practice but only show the idea.
const (
len8 int = 0xFFFFFFF8
)
func IsAllBytesZero(data []byte) bool {
n := len(data)
// Magic to get largest length which could be divided by 8.
nlen8 := n & len8
i := 0
for ; i < nlen8; i += 8 {
b := *(*uint64)(unsafe.Pointer(uintptr(unsafe.Pointer(&data[0])) + 8*uintptr(i)))
if b != 0 {
return false
}
}
for ; i < n; i++ {
if data[i] != 0 {
return false
}
}
return true
}
Benchmark
Testcases:
Only test for worst cases (all elements are zero)
Methods:
IsAllBytesZero: unsafe package solution
NaiveCheckAllBytesAreZero: a loop to iterate the whole byte array and check it.
CompareAllBytesWithFixedEmptyArray: using bytes.Compare solution with pre-allocated fixed size empty byte array.
CompareAllBytesWithDynamicEmptyArray: using bytes.Compare solution without pre-allocated fixed size empty byte array.
Results
BenchmarkIsAllBytesZero10-8 254072224 4.68 ns/op
BenchmarkIsAllBytesZero100-8 132266841 9.09 ns/op
BenchmarkIsAllBytesZero1000-8 19989015 55.6 ns/op
BenchmarkIsAllBytesZero10000-8 2344436 507 ns/op
BenchmarkIsAllBytesZero100000-8 1727826 679 ns/op
BenchmarkNaiveCheckAllBytesAreZero10-8 234153582 5.15 ns/op
BenchmarkNaiveCheckAllBytesAreZero100-8 30038720 38.2 ns/op
BenchmarkNaiveCheckAllBytesAreZero1000-8 4300405 291 ns/op
BenchmarkNaiveCheckAllBytesAreZero10000-8 407547 2666 ns/op
BenchmarkNaiveCheckAllBytesAreZero100000-8 43382 27265 ns/op
BenchmarkCompareAllBytesWithFixedEmptyArray10-8 415171356 2.71 ns/op
BenchmarkCompareAllBytesWithFixedEmptyArray100-8 218871330 5.51 ns/op
BenchmarkCompareAllBytesWithFixedEmptyArray1000-8 56569351 21.0 ns/op
BenchmarkCompareAllBytesWithFixedEmptyArray10000-8 6592575 177 ns/op
BenchmarkCompareAllBytesWithFixedEmptyArray100000-8 567784 2104 ns/op
BenchmarkCompareAllBytesWithDynamicEmptyArray10-8 64215448 19.8 ns/op
BenchmarkCompareAllBytesWithDynamicEmptyArray100-8 32875428 35.4 ns/op
BenchmarkCompareAllBytesWithDynamicEmptyArray1000-8 8580890 140 ns/op
BenchmarkCompareAllBytesWithDynamicEmptyArray10000-8 1277070 938 ns/op
BenchmarkCompareAllBytesWithDynamicEmptyArray100000-8 121256 10355 ns/op
Summary
Assumed that we're talking about the condition in sparse zero byte array. According to the benchmark, if performance is an issue, the naive check solution would be a bad idea. And, if you don't want to use unsafe package in your project, then consider using bytes.Compare solution with pre-allocated empty array as an alternative.
An interesting point could be pointed out is that the performance comes from unsafe package varies a lot, but it basically outperform all other solution mentioned above. I think it was relevant to the CPU cache mechanism.
You can possibly use bytes.Equal or bytes.Contains to compare with a zero initialized byte slice, see https://play.golang.org/p/mvUXaTwKjP, I haven't checked for performance, but hopefully it's been optimized. You might want to try out other solutions and compare the performance numbers, if needed.
I think it is better (faster) if binary or is used instead of if condition inside loop:
func isZero(bytes []byte) bool {
b := byte(0)
for _, s := range bytes {
b |= s
}
return b == 0
}
One can optimize this even more by using idea with uint64 mentioned in previous answers

in golang, is there any performance difference between maps initialized using make vs {}

as we know there are two ways to initialize a map (as listed below). I'm wondering if there is any performance difference between the two approaches.
var myMap map[string]int
then
myMap = map[string]int{}
vs
myMap = make(map[string]int)
On my machine they appear to be about equivalent.
You can easily make a benchmark test to compare. For example:
package bench
import "testing"
var result map[string]int
func BenchmarkMakeLiteral(b *testing.B) {
var m map[string]int
for n := 0; n < b.N; n++ {
m = InitMapLiteral()
}
result = m
}
func BenchmarkMakeMake(b *testing.B) {
var m map[string]int
for n := 0; n < b.N; n++ {
m = InitMapMake()
}
result = m
}
func InitMapLiteral() map[string]int {
return map[string]int{}
}
func InitMapMake() map[string]int {
return make(map[string]int)
}
Which on 3 different runs yielded results that are close enough to be insignificant:
First Run
$ go test -bench=.
testing: warning: no tests to run
PASS
BenchmarkMakeLiteral-8 10000000 160 ns/op
BenchmarkMakeMake-8 10000000 171 ns/op
ok github.com/johnweldon/bench 3.664s
Second Run
$ go test -bench=.
testing: warning: no tests to run
PASS
BenchmarkMakeLiteral-8 10000000 182 ns/op
BenchmarkMakeMake-8 10000000 173 ns/op
ok github.com/johnweldon/bench 3.945s
Third Run
$ go test -bench=.
testing: warning: no tests to run
PASS
BenchmarkMakeLiteral-8 10000000 170 ns/op
BenchmarkMakeMake-8 10000000 170 ns/op
ok github.com/johnweldon/bench 3.751s
When allocating empty maps there is no difference but with make you can pass second parameter to pre-allocate space in map. This will save a lot of reallocations when maps are being populated.
Benchmarks
package maps
import "testing"
const SIZE = 10000
func fill(m map[int]bool, size int) {
for i := 0; i < size; i++ {
m[i] = true
}
}
func BenchmarkEmpty(b *testing.B) {
for n := 0; n < b.N; n++ {
m := make(map[int]bool)
fill(m, SIZE)
}
}
func BenchmarkAllocated(b *testing.B) {
for n := 0; n < b.N; n++ {
m := make(map[int]bool, 2*SIZE)
fill(m, SIZE)
}
}
Results
go test -benchmem -bench .
BenchmarkEmpty-8 500 2988680 ns/op 431848 B/op 625 allocs/op
BenchmarkAllocated-8 1000 1618251 ns/op 360949 B/op 11 allocs/op
A year ago I actually stumped on the fact that using make with explicitly allocated space is better then using map literal if your values are not static
So doing
return map[string]float {
"key1": SOME_COMPUTED_ABOVE_VALUE,
"key2": SOME_COMPUTED_ABOVE_VALUE,
// more keys here
"keyN": SOME_COMPUTED_ABOVE_VALUE,
}
is slower then
// some code above
result := make(map[string]float, SIZE) // SIZE >= N
result["key1"] = SOME_COMPUTED_ABOVE_VALUE
result["key2"] = SOME_COMPUTED_ABOVE_VALUE
// more keys here
result["keyN"] = SOME_COMPUTED_ABOVE_VALUE
return result
for N which are quite big (N=300 in my use case).
The reason is the compiler fails to understand that one needs to allocate at least N slots in the first case.
I wrote a blog post about it
https://trams.github.io/golang-map-literal-performance/
and I reported a bug to the community
https://github.com/golang/go/issues/43020
As of golang 1.17 it is still an issue.

Compare string and byte slice in Go without copy

What is the best way to check that Go string and a byte slice contain the same bytes? The simplest str == string(byteSlice) is inefficient as it copies byteSlice first.
I was looking for a version of Equal(a, b []byte) that takes a string as its argument, but could not find anything suitable.
Starting from Go 1.5 the compiler optimizes string(bytes) when comparing to a string using a stack-allocated temporary. Thus since Go 1.5
str == string(byteSlice)
became a canonical and efficient way to compare string to a byte slice.
The Go Programming Language Specification
String types
A string type represents the set of string values. A string value is a
(possibly empty) sequence of bytes. The predeclared string type is
string.
The length of a string s (its size in bytes) can be discovered using
the built-in function len. A string's bytes can be accessed by integer
indices 0 through len(s)-1.
For example,
package main
import "fmt"
func equal(s string, b []byte) bool {
if len(s) != len(b) {
return false
}
for i, x := range b {
if x != s[i] {
return false
}
}
return true
}
func main() {
s := "equal"
b := []byte(s)
fmt.Println(equal(s, b))
s = "not" + s
fmt.Println(equal(s, b))
}
Output:
true
false
If you're comfortable enough with the fact that this can break on a later release (doubtful though), you can use unsafe:
func unsafeCompare(a string, b []byte) int {
abp := *(*[]byte)(unsafe.Pointer(&a))
return bytes.Compare(abp, b)
}
func unsafeEqual(a string, b []byte) bool {
bbp := *(*string)(unsafe.Pointer(&b))
return a == bbp
}
playground
Benchmarks:
// using:
// aaa = strings.Repeat("a", 100)
// bbb = []byte(strings.Repeat("a", 99) + "b")
// go 1.5
BenchmarkCopy-8 20000000 75.4 ns/op
BenchmarkPetersEqual-8 20000000 83.1 ns/op
BenchmarkUnsafe-8 100000000 12.2 ns/op
BenchmarkUnsafeEqual-8 200000000 8.94 ns/op
// go 1.4
BenchmarkCopy 10000000 233 ns/op
BenchmarkPetersEqual 20000000 72.3 ns/op
BenchmarkUnsafe 100000000 15.5 ns/op
BenchmarkUnsafeEqual 100000000 10.7 ns/op
There is no reason to use the unsafe package or something just to compare []byte and string. The Go compiler is clever enough now, and it can optimize such conversions.
Here's a benchmark:
BenchmarkEqual-8 172135624 6.96 ns/op <--
BenchmarkUnsafe-8 179866616 6.65 ns/op <--
BenchmarkUnsafeEqual-8 175588575 6.85 ns/op <--
BenchmarkCopy-8 23715144 47.3 ns/op
BenchmarkPetersEqual-8 24709376 47.3 ns/op
Just convert a byte slice to a string and compare:
var (
aaa = strings.Repeat("a", 100)
bbb = []byte(strings.Repeat("a", 99) + "b")
)
func BenchmarkEqual(b *testing.B) {
for i := 0; i < b.N; i++ {
_ = aaa == string(bbb)
}
}
👉 Here is more information about the optimization, and this.

Resources