let's say that I have generator of fibonachi numbers, and I would like to use enumerate(get_next_fibs(10)) and I would like to have generator of pairs index, number_from_generator, I am struggling to find solution with "named return values"
and it's not how it should be done but it's for purpose of learning specific things about generators
package main
import "fmt"
func get_next_fibs(ii int) func() int {
i := 0
a, b := 0, 1
fc := func() int {
i++
a, b = b, a+b
if ii <= i {
return -1
}
return a
}
return fc
}
func enumerate(iter func() int) func() (index, v int) {
index := 0
fc := func() (index, v int) {
v := iter()
return
index++
}
return fc
}
func main() {
iter := enumerate(get_next_fibs(10))
// iter := get_next_fibs(10)
fmt.Printf("iter = %T\n", iter)
for tuple := iter(); tuple != -1; tuple = iter() {
fmt.Println("tuple:", tuple)
}
}
You have few issues in this code sample:
You can't have index++ after return statement. Use defer if you need to do something after return-ing.
You're missing how variable shadowing works in go. Thus, you're trying to modify a wrong index variable.
Go doesn't have tuples.
...
func enumerate(iter func() int) func() (index, v int) {
counter := 0
return func() (index, v int) {
i := counter
counter++
return i, iter()
}
}
...
func main() {
iter := enumerate(get_next_fibs(10))
fmt.Printf("iter = %T\n", iter)
for i, v := iter(); v != -1; i, v = iter() {
fmt.Printf("i: %d, v: %d\n", i, v)
}
}
Playground link
Related
I want to call a number of function names stored in a slice. The code snippet below works so far but I need to return a value from those functions. Unfortunately I don't get it to work because I don't know to to call those functions and store the return value. Any ideas?
This is the code I'm currently working on:
package main
func A(x int) int {
return x + 1
}
func B(x int) int {
return x + 2
}
func C(x int) int {
return x + 3
}
func main() {
x := 10
type fs func(x int) int
f := []fs{A, B, C}
fns := make([]func(), 3)
for a, _ := range f {
a := a
fns[a] = func() {
f[a](x)
}
}
for _, f := range fns {
f()
}
}
Go Playground
You have call it...
for a, _ := range f {
a := a
fns[a] = func() {
f[a](x) // in this
}
}
here is the playground
I have 3 merge sort implementations:
MergeSort: simple one without concurrency;
MergeSortSmart: with concurrency limited by buffered channel size limit. If buffer is full, calls the simple implementation;
MergeSortSmartBug: same strategy as the previous one, but with a small "refactor", passing wg pointer to a function reducing code duplication.
The first two works as expected, but the third one returns an empty slice instead of the sorted input. I couldn't understand what happened and found no answers as well.
Here is the playground link for the code: https://play.golang.org/p/DU1ypbanpVi
package main
import (
"fmt"
"math/rand"
"runtime"
"sync"
)
type pass struct{}
var semaphore = make(chan pass, runtime.NumCPU())
func main() {
rand.Seed(10)
s := make([]int, 16)
for i := 0; i < 16; i++ {
s[i] = int(rand.Int31n(1000))
}
fmt.Println(s)
fmt.Println(MergeSort(s))
fmt.Println(MergeSortSmart(s))
fmt.Println(MergeSortSmartBug(s))
}
func merge(l, r []int) []int {
tmp := make([]int, 0, len(l)+len(r))
for len(l) > 0 || len(r) > 0 {
if len(l) == 0 {
return append(tmp, r...)
}
if len(r) == 0 {
return append(tmp, l...)
}
if l[0] <= r[0] {
tmp = append(tmp, l[0])
l = l[1:]
} else {
tmp = append(tmp, r[0])
r = r[1:]
}
}
return tmp
}
func MergeSort(s []int) []int {
if len(s) <= 1 {
return s
}
n := len(s) / 2
l := MergeSort(s[:n])
r := MergeSort(s[n:])
return merge(l, r)
}
func MergeSortSmart(s []int) []int {
if len(s) <= 1 {
return s
}
n := len(s) / 2
var wg sync.WaitGroup
wg.Add(2)
var l, r []int
select {
case semaphore <- pass{}:
go func() {
l = MergeSortSmart(s[:n])
<-semaphore
wg.Done()
}()
default:
l = MergeSort(s[:n])
wg.Done()
}
select {
case semaphore <- pass{}:
go func() {
r = MergeSortSmart(s[n:])
<-semaphore
wg.Done()
}()
default:
r = MergeSort(s[n:])
wg.Done()
}
wg.Wait()
return merge(l, r)
}
func MergeSortSmartBug(s []int) []int {
if len(s) <= 1 {
return s
}
n := len(s) / 2
var wg sync.WaitGroup
wg.Add(2)
l := mergeSmart(s[:n], &wg)
r := mergeSmart(s[n:], &wg)
wg.Wait()
return merge(l, r)
}
func mergeSmart(s []int, wg *sync.WaitGroup) []int {
var tmp []int
select {
case semaphore <- pass{}:
go func() {
tmp = MergeSortSmartBug(s)
<-semaphore
wg.Done()
}()
default:
tmp = MergeSort(s)
wg.Done()
}
return tmp
}
Why does the Bug version returns an empty slice? How can I refactor the Smart version without doing two selects one after the other?
Sorry for I couldn't reproduce this behavior in a smaller example.
The problem is not with the WaitGroup itself. It's with your concurrency handling. Your mergeSmart function lunches a go routine and returns the tmp variable without waiting for the go routine to finish.
You might want to try a pattern more like this:
leftchan := make(chan []int)
rightchan := make(chan []int)
go mergeSmart(s[:n], leftchan)
go mergeSmart(s[n:], rightchan)
l := <-leftchan
r := <-rightchan
Or you can use a single channel if order doesn't matter.
mergeSmart doesn't wait on the wg, so it returns a tmp that hasn't received a value yet. You could probably repair it by passing a reference to the destination slice in to the function, instead of returning a slice.
Look at the mergeSmart function. When the select enter into the first case, the goroutine is launched and imediatly returns tmp (which is an empty array). In that case there is no way to get the right value. (See advanced debugging prints here https://play.golang.org/p/IedaY3muso2)
Maybe passing arrays preallocated by reference?
I implemented both suggestions (passing slice by reference and using channels) and the (working!) result is here: https://play.golang.org/p/DcDC_-NjjAH
package main
import (
"fmt"
"math/rand"
"runtime"
"sync"
)
type pass struct{}
var semaphore = make(chan pass, runtime.NumCPU())
func main() {
rand.Seed(10)
s := make([]int, 16)
for i := 0; i < 16; i++ {
s[i] = int(rand.Int31n(1000))
}
fmt.Println(s)
fmt.Println(MergeSort(s))
fmt.Println(MergeSortSmart(s))
fmt.Println(MergeSortSmartPointer(s))
fmt.Println(MergeSortSmartChan(s))
}
func merge(l, r []int) []int {
tmp := make([]int, 0, len(l)+len(r))
for len(l) > 0 || len(r) > 0 {
if len(l) == 0 {
return append(tmp, r...)
}
if len(r) == 0 {
return append(tmp, l...)
}
if l[0] <= r[0] {
tmp = append(tmp, l[0])
l = l[1:]
} else {
tmp = append(tmp, r[0])
r = r[1:]
}
}
return tmp
}
func MergeSort(s []int) []int {
if len(s) <= 1 {
return s
}
n := len(s) / 2
l := MergeSort(s[:n])
r := MergeSort(s[n:])
return merge(l, r)
}
func MergeSortSmart(s []int) []int {
if len(s) <= 1 {
return s
}
n := len(s) / 2
var wg sync.WaitGroup
wg.Add(2)
var l, r []int
select {
case semaphore <- pass{}:
go func() {
l = MergeSortSmart(s[:n])
<-semaphore
wg.Done()
}()
default:
l = MergeSort(s[:n])
wg.Done()
}
select {
case semaphore <- pass{}:
go func() {
r = MergeSortSmart(s[n:])
<-semaphore
wg.Done()
}()
default:
r = MergeSort(s[n:])
wg.Done()
}
wg.Wait()
return merge(l, r)
}
func MergeSortSmartPointer(s []int) []int {
if len(s) <= 1 {
return s
}
n := len(s) / 2
var l, r []int
var wg sync.WaitGroup
wg.Add(2)
mergeSmartPointer(&l, s[:n], &wg)
mergeSmartPointer(&r, s[n:], &wg)
wg.Wait()
return merge(l, r)
}
func mergeSmartPointer(tmp *[]int, s []int, wg *sync.WaitGroup) {
select {
case semaphore <- pass{}:
go func() {
*tmp = MergeSortSmartPointer(s)
<-semaphore
wg.Done()
}()
default:
*tmp = MergeSort(s)
wg.Done()
}
}
func MergeSortSmartChan(s []int) []int {
if len(s) <= 1 {
return s
}
n := len(s) / 2
lchan := make(chan []int)
rchan := make(chan []int)
go mergeSmartChan(s[:n], lchan)
go mergeSmartChan(s[n:], rchan)
l := <-lchan
r := <-rchan
return merge(l, r)
}
func mergeSmartChan(s []int, c chan []int) {
select {
case semaphore <- pass{}:
go func() {
c <- MergeSortSmartChan(s)
<-semaphore
}()
default:
c <- MergeSort(s)
}
}
I understood 100% what I was doing wrong, thanks!
And for future references, here's the benchmark of sorting a slice of 100,000 elems:
$ go test -bench=.
goos: linux
goarch: amd64
cpu: Intel(R) Core(TM) i5-9300H CPU # 2.40GHz
BenchmarkMergeSort-8 97 12230309 ns/op
BenchmarkMergeSortSmart-8 181 7209844 ns/op
BenchmarkMergeSortSmartPointer-8 163 7483136 ns/op
BenchmarkMergeSortSmartChan-8 156 8149585 ns/op
Suppose I have a helper function helper(n int) which returns a slice of integers of variable length. I would like to run helper(n) in parallel for various values of n and collect the output in one big slice. My first attempt at this is the following:
package main
import (
"fmt"
"golang.org/x/sync/errgroup"
)
func main() {
out := make([]int, 0)
ch := make(chan int)
go func() {
for i := range ch {
out = append(out, i)
}
}()
g := new(errgroup.Group)
for n := 2; n <= 3; n++ {
n := n
g.Go(func() error {
for _, i := range helper(n) {
ch <- i
}
return nil
})
}
if err := g.Wait(); err != nil {
panic(err)
}
close(ch)
// time.Sleep(time.Second)
fmt.Println(out) // should have the same elements as [0 1 0 1 2]
}
func helper(n int) []int {
out := make([]int, 0)
for i := 0; i < n; i++ {
out = append(out, i)
}
return out
}
However, if I run this example I do not get all 5 expected values, instead I get
[0 1 0 1]
(If I uncomment the time.Sleep I do get all five values, [0 1 2 0 1], but this is not an acceptable solution).
It seems that the problem with this is that out is being updated in a goroutine, but the main function returns before it is done updating.
One thing that would work is using a buffered channel of size 5:
func main() {
ch := make(chan int, 5)
g := new(errgroup.Group)
for n := 2; n <= 3; n++ {
n := n
g.Go(func() error {
for _, i := range helper(n) {
ch <- i
}
return nil
})
}
if err := g.Wait(); err != nil {
panic(err)
}
close(ch)
out := make([]int, 0)
for i := range ch {
out = append(out, i)
}
fmt.Println(out) // should have the same elements as [0 1 0 1 2]
}
However, although in this simplified example I know what the size of the output should be, in my actual application this is not known a priori. Essentially what I would like is an 'infinite' buffer such that sending to the channel never blocks, or a more idiomatic way to achieve the same thing; I've read https://blog.golang.org/pipelines but wasn't able to find a close match to my use case. Any ideas?
In this version of the code, the execution is blocked until ch is closed.
ch is always closed at the end of a routine that is responsible to push into ch. Because the program pushes to ch in a routine, it is not needed to use a buffered channel.
package main
import (
"fmt"
"golang.org/x/sync/errgroup"
)
func main() {
ch := make(chan int)
go func() {
g := new(errgroup.Group)
for n := 2; n <= 3; n++ {
n := n
g.Go(func() error {
for _, i := range helper(n) {
ch <- i
}
return nil
})
}
if err := g.Wait(); err != nil {
panic(err)
}
close(ch)
}()
out := make([]int, 0)
for i := range ch {
out = append(out, i)
}
fmt.Println(out) // should have the same elements as [0 1 0 1 2]
}
func helper(n int) []int {
out := make([]int, 0)
for i := 0; i < n; i++ {
out = append(out, i)
}
return out
}
Here is the fixed version of the first code, it is convoluted but demonstrates the usage of sync.WaitGroup.
package main
import (
"fmt"
"sync"
"golang.org/x/sync/errgroup"
)
func main() {
out := make([]int, 0)
ch := make(chan int)
var wg sync.WaitGroup
wg.Add(1)
go func() {
defer wg.Done()
for i := range ch {
out = append(out, i)
}
}()
g := new(errgroup.Group)
for n := 2; n <= 3; n++ {
n := n
g.Go(func() error {
for _, i := range helper(n) {
ch <- i
}
return nil
})
}
if err := g.Wait(); err != nil {
panic(err)
}
close(ch)
wg.Wait()
// time.Sleep(time.Second)
fmt.Println(out) // should have the same elements as [0 1 0 1 2]
}
func helper(n int) []int {
out := make([]int, 0)
for i := 0; i < n; i++ {
out = append(out, i)
}
return out
}
This question already has answers here:
Golang channel output order
(4 answers)
Closed 4 years ago.
Why following codes always return 2,1, not 1,2.
func test(x int, c chan int) {
c <- x
}
func main() {
c := make(chan int)
go test(1, c)
go test(2, c)
x, y := <-c, <-c // receive from c
fmt.Println(x, y)
}
If you want to know what the order is, then let your program include ordering information
This example uses a function closure to generate a sequence
The channel returns a struct of two numbers, one of which is a sequence order number
The sequence incrementer should be safe across go routines as there is a mutex lock on the sequence counter
package main
import (
"fmt"
"sync"
)
type value_with_order struct {
v int
order int
}
var (
mu sync.Mutex
)
func orgami(x int, c chan value_with_order, f func() int) {
v := new(value_with_order)
v.v = x
v.order = f()
c <- *v
}
func seq() func() int {
i := 0
return func() int {
mu.Lock()
defer mu.Unlock()
i++
return i
}
}
func main() {
c := make(chan value_with_order)
sequencer := seq()
for n := 0; n < 10; n++ {
go orgami(1, c, sequencer)
go orgami(2, c, sequencer)
go orgami(3, c, sequencer)
}
received := 0
for q := range c {
fmt.Printf("%v\n", q)
received++
if received == 30 {
close(c)
}
}
}
second version where the sequence is called from the main loop to make the sequence numbers come out in the order that the function is called
package main
import (
"fmt"
"sync"
)
type value_with_order struct {
v int
order int
}
var (
mu sync.Mutex
)
func orgami(x int, c chan value_with_order, seqno int) {
v := new(value_with_order)
v.v = x
v.order = seqno
c <- *v
}
func seq() func() int {
i := 0
return func() int {
mu.Lock()
defer mu.Unlock()
i++
return i
}
}
func main() {
c := make(chan value_with_order)
sequencer := seq()
for n := 0; n < 10; n++ {
go orgami(1, c, sequencer())
go orgami(2, c, sequencer())
go orgami(3, c, sequencer())
}
received := 0
for q := range c {
fmt.Printf("%v\n", q)
received++
if received == 30 {
close(c)
}
}
}
How to realize a nested iterator that takes a depth argument. A simple iterator would be when depth = 1. it is a simple iterator which runs like a simple for loop.
func Iter () chan int {
ch := make(chan int);
go func () {
for i := 1; i < 60; i++ {
ch <- i
}
close(ch)
} ();
return ch
}
Output is 1,2,3...59
For depth = 2 Output would be "1,1" "1,2" ... "1,59" "2,1" ... "59,59"
For depth = 3 Output would be "1,1,1" ... "59,59,59"
I want to avoid a nested for loop. What is the solution here ?
I don't know if it is possible to avoid nested loops, but one solution is to use a pipeline of channels. For example:
const ITER_N = 60
// ----------------
func _goFunc1(out chan string) {
for i := 1; i < ITER_N; i++ {
out <- fmt.Sprintf("%d", i)
}
close(out)
}
func _goFuncN(in chan string, out chan string) {
for j := range in {
for i := 1; i < ITER_N; i++ {
out <- fmt.Sprintf("%s,%d", j, i)
}
}
close(out)
}
// ----------------
// create the pipeline
func IterDepth(d int) chan string {
c1 := make(chan string)
go _goFunc1(c1)
var c2 chan string
for ; d > 1; d-- {
c2 = make(chan string)
go _goFuncN(c1, c2)
c1 = c2
}
return c1
}
You can test it with:
func main() {
c := IterDepth(2)
for i := range c {
fmt.Println(i)
}
}
I usually implement iterators using closures. Multiple dimensions don't make the problem much harder. Here's one example of how to do this:
package main
import "fmt"
func iter(min, max, depth int) func() ([]int, bool) {
s := make([]int, depth)
for i := range s {
s[i] = min
}
s[0] = min - 1
return func() ([]int, bool) {
s[0]++
for i := 0; i < depth-1; i++ {
if s[i] >= max {
s[i] = min
s[i+1]++
}
}
if s[depth-1] >= max {
return nil, false
}
return s, true
}
}
func main() {
// Three dimensions, ranging between [1,4)
i := iter(1, 4, 3)
for s, ok := i(); ok; s, ok = i() {
fmt.Println(s)
}
}
Try it out on the Playground.
It'd be a simple change for example to give arguments as a single int slice instead, so that you could have per-dimension limits, if such a thing were necessary.