go random generator stops working after some calls - go

Question

I want shuffle db ids so that none of the id refer to themselves, but with this piece of code:
package main
import (
"log"
"math/rand"
"time"
)
func main() {
seed := time.Now().UnixNano() & 999999999
log.Print("seed: ", seed)
rand.Seed(seed)
ordered := []int{0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19}
randomized := shufflePreventCollision(ordered)
log.Print("Final Result")
log.Print("ordered: ", ordered)
log.Print("random: ", randomized)
}
func shufflePreventCollision(ordered []int) []int {
randomized := rand.Perm(len(ordered))
for i, o := range ordered {
if o == randomized[i] {
log.Printf("Doing it again because ordered[%d] (%d) is == randomized[%d] (%d)", i, o, i, randomized[i])
log.Print(ordered)
log.Print(randomized)
shufflePreventCollision(ordered)
}
}
return randomized
}
I find a strange behaviour, when it runs often it at some point hangs and cannot find non-colliding sequences anymore. I tried
go build -o rand_example3 rand_example3.go && time (for i in $(seq 10000) ; do ./rand_example3 ; done)
And it seems to never end. Am I missing some understanding here or is there really something fishy with math/rand?

Answer 1

"With this piece of code, I want shuffle db ids so that none of the
ids refer to themselves. [Sometimes] it doesn't end even when I let it
run for an hour or so."
tl;dr There is a faster solution that is more than a thousand times faster.
Your code:
slaxor.go:
package main
import (
"log"
"math/rand"
"time"
)
func main() {
seed := time.Now().UnixNano() & 999999999
log.Print("seed: ", seed)
rand.Seed(seed)
ordered := []int{0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19}
randomized := shufflePreventCollisionSlaxor(ordered)
log.Print("Final Result")
log.Print("ordered: ", ordered)
log.Print("random: ", randomized)
}
func shufflePreventCollisionSlaxor(ordered []int) []int {
randomized := rand.Perm(len(ordered))
for i, o := range ordered {
if o == randomized[i] {
log.Printf("Doing it again because ordered[%d] (%d) is == randomized[%d] (%d)", i, o, i, randomized[i])
log.Print(ordered)
log.Print(randomized)
shufflePreventCollisionSlaxor(ordered)
}
}
return randomized
}
Playground: https://play.golang.org/p/JI5rJGcAAz
It is not obvious how, or even if, the code fulfills its purpose.
The termination condition is probabalistic, not deterministic.
Let's leave aside the issue of whether the code fulfills its purpose.
This benchmark has been modified so that stderr is limited by the speed of the sink /dev/null, not a terminal.
slaxor.bash:
go build -o slaxor slaxor.go && time (for i in $(seq 10000) ; do ./slaxor 2> /dev/null ; done)
The benchmark measures the execution of a program and a single execution of the algorithm. The benchmark times are inconsistent because the pseudorandom seed value changes for each program execution. The benchmark sometimes "doesn't end even when I let it run for an hour or so."
There is a faster solution that runs and terminates in a few seconds, despite the program execution overhead.
peterso.bash:
go build -o peterso peterso.go && time (for i in $(seq 10000) ; do ./peterso 2> /dev/null ; done)
Output:
$ ./peterso.bash
real 0m5.290s
user 0m5.224s
sys 0m1.128s
$ ./peterso.bash
real 0m7.462s
user 0m7.109s
sys 0m1.922s
peterso.go:
package main
import (
"fmt"
"log"
"math/rand"
"time"
)
func main() {
seed := time.Now().UnixNano() & 999999999
log.Print("seed: ", seed)
r = rand.New(rand.NewSource(seed))
ordered := []int{0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19}
randomized := shufflePreventCollisionPeterSO(ordered)
log.Print("Final Result")
log.Print("ordered: ", ordered)
log.Print("random: ", randomized)
if randomized == nil {
err := "Shuffle Error!"
fmt.Print(err)
log.Fatal(err)
}
}
var r *rand.Rand
func isNoCollision(a, b []int) bool {
if len(a) == len(b) {
for i, ai := range a {
if ai == b[i] {
return false
}
}
return true
}
return false
}
func shufflePreventCollisionPeterSO(ordered []int) []int {
const guard = 4 * 1024 // deterministic, finite time
for n := 1; n <= guard; n++ {
randomized := r.Perm(len(ordered))
if isNoCollision(ordered, randomized) {
return randomized
}
}
return nil
}
The guard provides a deterministic, finite time, termination condition.
Playground: https://play.golang.org/p/ZT-sfDW5Mi
Let's put aside the distorted program execution benchmarks, let's look at function execution times. Because this is important, the Go standard library has the testing package for testing and benchmarking functions.
Tests:
$ go test shuffle_test.go -v -count=1 -run=. -bench=!
=== RUN TestTimeSlaxor
=== RUN TestTimeSlaxor/1K
=== RUN TestTimeSlaxor/2K
=== RUN TestTimeSlaxor/3K
--- PASS: TestTimeSlaxor (13.78s)
--- PASS: TestTimeSlaxor/1K (1.18s)
--- PASS: TestTimeSlaxor/2K (1.27s)
--- PASS: TestTimeSlaxor/3K (11.33s)
=== RUN TestTimePeterSO
=== RUN TestTimePeterSO/1K
=== RUN TestTimePeterSO/2K
=== RUN TestTimePeterSO/3K
=== RUN TestTimePeterSO/1M
=== RUN TestTimePeterSO/2M
=== RUN TestTimePeterSO/3M
--- PASS: TestTimePeterSO (6.57s)
--- PASS: TestTimePeterSO/1K (0.00s)
--- PASS: TestTimePeterSO/2K (0.00s)
--- PASS: TestTimePeterSO/3K (0.00s)
--- PASS: TestTimePeterSO/1M (1.13s)
--- PASS: TestTimePeterSO/2M (2.25s)
--- PASS: TestTimePeterSO/3M (3.19s)
PASS
ok command-line-arguments 20.347s
$
In a fraction of the rapidly increasing time it takes to run 3K (3,000) iterations of shufflePreventCollisionSlaxor, 3M (3,000,000) iterations of shufflePreventCollisionPeterSO run, a more than thousand-fold improvement.
Benchmarks:
$ go test shuffle_test.go -v -count=1 -run=! -bench=.
goos: linux
goarch: amd64
BenchmarkTimePeterSO-8 1000000 1048 ns/op 434 B/op 2 allocs/op
BenchmarkTimeSlaxor-8 10000 2256271 ns/op 636894 B/op 3980 allocs/op
PASS
ok command-line-arguments 23.643s
$
It's easy to see that the shufflePreventCollisionPeterSO average per iteration cost of 1,000,000 iterations is small, 1,048 nanoseconds, especially when compared to only 10,000 iterations of shufflePreventCollisionSlaxor at 2,256,271 nanoseconds average per iteration.
Also, note the sparing use of memory by shufflePreventCollisionPeterSO, on average 2 allocations for a total allocation of 434 bytes per iteration, versus the profligate use of memory by shufflePreventCollisionSlaxor, on average 3,980 allocations for a total allocation of 636,894 bytes per iteration.
shuffle_test.go:
package main
import (
"fmt"
"math/rand"
"strconv"
"testing"
)
func shufflePreventCollisionSlaxor(ordered []int) []int {
randomized := rand.Perm(len(ordered))
for i, o := range ordered {
if o == randomized[i] {
shufflePreventCollisionSlaxor(ordered)
}
}
return randomized
}
var r *rand.Rand
func isNoCollision(a, b []int) bool {
if len(a) == len(b) {
for i, ai := range a {
if ai == b[i] {
return false
}
}
return true
}
return false
}
func shufflePreventCollisionPeterSO(ordered []int) []int {
const guard = 4 * 1024 // deterministic, finite time
for n := 1; n <= guard; n++ {
randomized := r.Perm(len(ordered))
if isNoCollision(ordered, randomized) {
return randomized
}
}
return nil
}
const testSeed = int64(60309766)
func testTime(t *testing.T, ordered, randomized []int, shuffle func([]int) []int) {
shuffled := shuffle(ordered)
want := fmt.Sprintf("%v", randomized)
got := fmt.Sprintf("%v", shuffled)
if want != got {
t.Errorf("Error:\n from: %v\n want: %s\n got: %s\n", ordered, want, got)
}
}
func testTimeSlaxor(t *testing.T, n int) {
rand.Seed(testSeed)
ordered := []int{0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19}
randomized := []int{3, 1, 17, 15, 10, 16, 14, 19, 7, 6, 11, 2, 0, 12, 8, 18, 13, 4, 9, 5}
testTime(t, ordered, randomized, shufflePreventCollisionSlaxor)
for i := 1; i < n; i++ {
shufflePreventCollisionSlaxor(ordered)
}
}
func TestTimeSlaxor(t *testing.T) {
for k := 1; k <= 3; k++ {
n := 1000 * k
t.Run(strconv.Itoa(k)+"K", func(t *testing.T) { testTimeSlaxor(t, n) })
}
}
func testTimePeterSo(t *testing.T, n int) {
r = rand.New(rand.NewSource(testSeed))
ordered := []int{0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19}
randomized := []int{10, 7, 15, 14, 8, 6, 18, 17, 19, 11, 5, 16, 2, 12, 1, 13, 3, 0, 9, 4}
testTime(t, ordered, randomized, shufflePreventCollisionPeterSO)
for i := 1; i < n; i++ {
shufflePreventCollisionPeterSO(ordered)
}
}
func TestTimePeterSO(t *testing.T) {
for k := 1; k <= 3; k++ {
n := 1000 * k
t.Run(strconv.Itoa(k)+"K", func(t *testing.T) { testTimePeterSo(t, n) })
}
for m := 1; m <= 3; m++ {
n := 1000 * 1000 * m
t.Run(strconv.Itoa(m)+"M", func(t *testing.T) { testTimePeterSo(t, n) })
}
}
func benchTime(b *testing.B, ordered, randomized []int, shuffle func([]int) []int) {
shuffled := shuffle(ordered)
want := fmt.Sprintf("%v", randomized)
got := fmt.Sprintf("%v", shuffled)
if want != got {
b.Errorf("Error:\n from: %v\n want: %s\n got: %s\n", ordered, want, got)
}
}
func BenchmarkTimePeterSO(b *testing.B) {
b.ReportAllocs()
r = rand.New(rand.NewSource(testSeed))
ordered := []int{0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19}
randomized := []int{10, 7, 15, 14, 8, 6, 18, 17, 19, 11, 5, 16, 2, 12, 1, 13, 3, 0, 9, 4}
benchTime(b, ordered, randomized, shufflePreventCollisionPeterSO)
r = rand.New(rand.NewSource(testSeed))
b.ResetTimer()
for i := 0; i < b.N; i++ {
shufflePreventCollisionPeterSO(ordered)
}
}
func BenchmarkTimeSlaxor(b *testing.B) {
b.ReportAllocs()
r = rand.New(rand.NewSource(testSeed))
ordered := []int{0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19}
randomized := []int{10, 7, 15, 14, 8, 6, 18, 17, 19, 11, 5, 16, 2, 12, 1, 13, 3, 0, 9, 4}
benchTime(b, ordered, randomized, shufflePreventCollisionPeterSO)
r = rand.New(rand.NewSource(testSeed))
b.ResetTimer()
for i := 0; i < b.N; i++ {
shufflePreventCollisionSlaxor(ordered)
}
}
Playground: https://play.golang.org/p/ozazWGNZsu