This is homework
I'm working on a project, and a very small (very small, once I get this working...it's basically the pre-req for the rest of the project) part of it is to generate some combinations using a Go routine.
The code I have is thusly:
package bruteforce
// GenerateCombinations is an iterator function. Given an alphabet and a
// length, it will generate every possible combination of the letters in
// alphabet of the specified length.
//
// It is meant to be consumed by using the range clause:
//
// for combination := range GenerateCombinations(alphabet, length) {
// process(combination)
// }
//
func GenerateCombinations(alphabet string, length int) <-chan string {
GenerateCombinations(alphabet, length):
if length == 0:
yield ""
else:
for i := range alphabet{
for j := range GenerateCombinations(alphabet, length-1){
i + j
}
}
return nil
}
I seriously do not get this at all. As you can see, there is some instructor-provided pseudo code there, but the implementation of it is frying my brain.
Example I/O would be something like this:
If the alphabet is {0, 1} and the password was length 2, then it would need to generate {0, 1, 00, 01, 10, 11}.
I appreciate all suggestions, but please recognize that the term "beginner" doesn't begin to describe my competency with go. Saying things like "use channels" doesn't help me at all. Explanations are my friend... something that I haven't had a lot of luck getting out of my professor aside from "use channels."
Your teacher has already hinted that you should use a channel instead of returning a big array. By solving it like that, you will not have to store this big chunk of data containing all combination, but rather feed your function with different iterations and process them one at a time.
We can see your teachers hint in that the GenerateCombinations returns a chan string and not a []string:
func GenerateCombinations(alphabet string, length int) <-chan string
This also means that the function must 1) create a channel to return, and 2) start a goroutine that feeds iterations to the channel. This function would look something like this:
func GenerateCombinations(alphabet string, length int) <-chan string {
c := make(chan string)
// Starting a separate goroutine that will create all the combinations,
// feeding them to the channel c
go func(c chan string) {
defer close(c) // Once the iteration function is finished, we close the channel
// This is where the iteration will take place
// Your teacher's pseudo code uses recursion
// which mean you might want to create a separate function
// that can call itself.
}(c)
return c // Return the channel to the calling function
}
While I will leave the actual iteration to you, every loop should result in you putting a combination string into the channel. Because it is not a buffered channel, the iterating function will wait til the main function has read the value before continue to process the next iteration.
A playground version including the main function: http://play.golang.org/p/CBkSjpmQ0t
A full working solution using recursion: http://play.golang.org/p/0bWDCibSUJ
This is a tricky problem if you are entirely unfamiliar with Go and/or how to generate permutations. Below is a full implementation of the solution. I suggest you only look at this if you really get stuck, because doing so will obviously remove the learning experience.
You can run it on the go playground to see it work.
This approach is not recursive as your instructor's example suggests, but it gets the job done quite nicely.
package main
import "fmt"
import "sync"
func main() {
// Make sure the alphabet is sorted.
const alphabet = "abcde"
for str := range generate(alphabet) {
fmt.Println(str)
}
}
func generate(alphabet string) <-chan string {
c := make(chan string, len(alphabet))
go func() {
defer close(c)
if len(alphabet) == 0 {
return
}
// Use a sync.WaitGroup to spawn permutation
// goroutines and allow us to wait for them all
// to finish.
var wg sync.WaitGroup
wg.Add(len(alphabet))
for i := 1; i <= len(alphabet); i++ {
go func(i int) {
// Perform permutation on a subset of
// the alphabet.
Word(alphabet[:i]).Permute(c)
// Signal Waitgroup that we are done.
wg.Done()
}(i)
}
// Wait for all routines to finish.
wg.Wait()
}()
return c
}
type Word []rune
// Permute generates all possible combinations of
// the current word. This assumes the runes are sorted.
func (w Word) Permute(out chan<- string) {
if len(w) <= 1 {
out <- string(w)
return
}
// Write first result manually.
out <- string(w)
// Find and print all remaining permutations.
for w.next() {
out <- string(w)
}
}
// next performs a single permutation by shuffling characters around.
// Returns false if there are no more new permutations.
func (w Word) next() bool {
var left, right int
left = len(w) - 2
for w[left] >= w[left+1] && left >= 1 {
left--
}
if left == 0 && w[left] >= w[left+1] {
return false
}
right = len(w) - 1
for w[left] >= w[right] {
right--
}
w[left], w[right] = w[right], w[left]
left++
right = len(w) - 1
for left < right {
w[left], w[right] = w[right], w[left]
left++
right--
}
return true
}
Related
A few days ago, I posted this topic on the Code Review site. In it, I detailed my first attempt at implementing goroutines into my mergesort code and while it worked fine, I was hoping for a better implementation. As I thought about it more, I had what I thought was a solid idea: instead of constantly waiting for both the left and right side to complete before merging both sides together, why not take the (presumably) sorted singleton chunks you get from the left side as it is sorting itself and sorted single chunks you get from the right and sort those as they come?
I attempted to restructure my code but I've run into a bit of an issue: from what I can tell, my implementation of the base case has caused massive problems, or I am misunderstanding the scope of goroutines and am telling channels to close when they, in a different sort block, are still being used. I was hoping someone could help me refine my understanding or, on the chance that my code is broken in a simple fashion, help me to understand an issue I will present after this code:
package main
import (
"crypto/rand"
"fmt"
"os"
"strconv"
)
var (
nums []byte //The slice of numbers we want to sort
numVals int = -1
)
//User can optionally add a parameter that determines how many random numbers will be sorted
//If none are provided, 100 will be used
func main() {
if len(os.Args) >= 2 {
numVals, _ = strconv.Atoi(os.Args[1])
} else {
numVals = 2
}
nums = initSlice()
ms := make(chan byte)
go mergeSort(nums, ms)
pos := 0
for val := range ms {
nums[pos] = val
pos++
}
for _, value := range nums {
fmt.Printf("%d\n", value)
}
}
func initSlice() []byte {
vals := make([]byte, numVals)
_, err := rand.Read(vals)
if err != nil {
panic(err)
}
return vals
}
func mergeSort(arr []byte, ms chan byte) {
if len(arr) <= 1 {
if len(arr) == 1 { //base case
ms <- arr[0]
}
close(ms)
return
}
leftMS := make(chan byte)
go mergeSort(arr[:len(arr)/2], leftMS)
rightMS := make(chan byte)
go mergeSort(arr[len(arr)/2:], rightMS)
left, lOK := <-leftMS
right, rOK := <-rightMS
for lOK && rOK {
leftLeast := left <= right
if leftLeast {
ms <- left
left, lOK = <-leftMS
} else {
ms <- right
right, lOK = <-rightMS
}
}
if lOK {
ms <- left
for val := range leftMS {
ms <- val
}
}
if rOK {
ms <- right
for val := range rightMS {
ms <- val
}
}
close(ms)
}
Overall, my biggest question would be, let's say we have the following sort:
If I am currently working through '38' and '27' pairing and I close that ms channel, I would expect it is not the same channel as the channel that starts everything off in main? If not, is there a way I can create new channels recursively while still keeping the name?
Hope this all makes sense and thanks for the help.
Your channel use is not your problem. There are two problems with your program.
First, you have to collect the results in a separate array in the main goroutine, otherwise, you'll be modifying the array you're sorting as it is being sorted.
Second, this block:
} else {
ms <- right
right, lOK = <-rightMS
It should be
right, rOK = <-rightMS
You're setting lOK with rightMS, not rOK.
I have a loop throwing multiple go routines, they call a function that makes a http get petition and calculate and object.
I want to recover the result of all those routines.
I tried using channels, but hey are empty, even if I force wait for all the routines to be done.
This is the code that starts the routines:
func main() {
pairs := getPairs() //Returns an array of strings
c := make(chan result)
for _, product := range pairs {
go getScore(product.Symbol, 1, c)
}
fmt.Println(len(c))
time.Sleep(5000 * time.Millisecond)
fmt.Println(len(c))
}
And at the end of getScore() I do this, c being the name of the channel in the function and res the result of the function
c <- res
The length of the channel is 0 in both prints.
What's the best way to get the result of the functions?
A channel is a synchronization prototype against a shared memory (in simple point of view). A buffered channel has a length but not a regular channel. Buffered channel is useful in little bit cases but not as a general approaches.
The simplest way to Just add a loop by range of pair or len of pairs:
// start processing
for _, product := range pairs {
go getScore(product.Symbol, 1, c)
}
// getting a result
for i:=0; i<len(pairs); i ++ {
result := <-c
// process a result value
}
Or another way is collecting result in another grouting:
// result and sync variable
var (
wait sync.WaitGroup
result int32
)
// start processing
for _, product := range pairs {
wait.Add(1)
go getScore(product.Symbol, 1, c)
go func() {
defer wait.Done()
// simple accumulate or maybe more complicated actions
atomic.AddInt32(&result, <-c)
}()
}
// wait finishing
wait.Wait()
c := make(chan result)
Creates an unbuffered channel. Therefore send statements, such as
c <- res
cannot proceed until another goroutine is attempting a receive operation.
In other words, execute the number of receive operations in your main goroutine matching the number of sends that will be attempted from other goroutines. Like this:
for _, product := range pairs {
go getScore(product.Symbol, 1, c)
}
for x := 0; x < len(pairs); x++ {
fmt.Println(<-c)
}
See the Go Tour section on channels, and the Effective Go section on channels for more information.
I was trying to implement Heap's Algorithm in go using channels. The code below is working fine when just printing the slices on the screen, but when using channels to delivery the arrays to a for/range loop on main function some unexpected behaviour occurs and the slices/arrays are printed in duplicity and not all permutations are sent. I thought that maybe i'm closing the channel earlier than the main function is able to print the results but i wouldn't expect that double print. Why is this happening and how can i make it work.
package main
import "fmt"
func perm(a []int64) {
var n = len(a)
var c = make([]int, n)
fmt.Println(a)
i := 0
for i < n {
if c[i] < i {
if i%2 == 0 {
a[0], a[i] = a[i], a[0]
} else {
a[c[i]], a[i] = a[i], a[c[i]]
}
fmt.Println(a)
c[i]++
i = 0
} else {
c[i] = 0
i++
}
}
}
func permch(a []int64, ch chan<- []int64) {
var n = len(a)
var c = make([]int, n)
ch <- a
i := 0
for i < n {
if c[i] < i {
if i%2 == 0 {
a[0], a[i] = a[i], a[0]
} else {
a[c[i]], a[i] = a[i], a[c[i]]
}
ch <- a
c[i]++
i = 0
} else {
c[i] = 0
i++
}
}
close(ch)
}
func main() {
var i = []int64{1, 2, 3}
fmt.Println("Without Channels")
perm(i)
ch := make(chan []int64)
go permch(i, ch)
fmt.Println("With Channels")
for slc := range ch {
fmt.Println(slc)
}
}
Your problem is that slices are reference types, and are being accessed in multiple goroutines. In perm, you're printing a directly as you finish processing it at each step. In permch, you're sending a over the channel, but then immediate starting to modify it again. Since each slice sent over the channel refers to the same underlying array, you have a race condition as to whether your next loop iteration alters a or your Println() call in main gets to that array first.
In general, if you're running into unexpected behavior in any program using goroutines, you probably have a race condition. Run the program with the -race flag to see where.
Edit: also, closing a channel has no effect on a routine reading from the channel. A channel can continue to be read until its buffer is empty, at which point it will start returning the zero value for that type instead. Range loops over a channel will only terminate once the channel is closed and its buffer is empty.
It looks like permch is modifying a at the same time as main is printing it, so your output is garbled.
I can think of three easy fixes:
Guard access to a with a mutex.
Put a copy of a on the channel:
Have some kind of return signal from main that it has printed and permch can continue. (don't really recommend this, but it works).
Number 2 is pretty simple:
a2 := make([]int64, len(a))
copy(a2,a)
ch <- a2
and is what I would recommend.
For #1, just declare a var m sync.Mutex as a package variable and Lock is anytime you read or modify a. This is a race condition though between the reader and the next modification, as you pointed out, so it probably isn't a good solution after all.
fixed version on playground using #3
I'm a Go learner. In order better to understand the care and feeding of channels and goroutines, I'm trying to build a Sieve of Eratosthenes as a set of goroutines connected into a pipeline by channels.
Here's what I have so far:
// esieve implements a Sieve of Eratosthenes
// as a series of channels connected together
// by goroutines
package main
import "fmt"
func sieve(mine int, inch chan int) {
start := true // First-number switch
ouch := make(chan int) // Output channel for this instance
fmt.Printf("%v\n", mine) // Print this instance's prime
for next := <-inch; next > 0; next = <-inch { // Read input channel
fmt.Printf("%v <- %v\n",mine,next) // (Trace)
if (next % mine) > 0 { // Divisible by my prime?
if start { // No; is it the first number through?
go sieve(next, ouch) // First number - create instance for it
start = false // First time done
} else { // Not first time
ouch <- next // Pass it to the next instance
}
}
}
}
func main() {
lim := 30 // Let's do up to 30
fmt.Printf("%v\n", 2) // Treat 2 as a special case
ouch := make(chan int) // Create the first segment of the pipe
go sieve(3, ouch) // Create the instance for '3'
for prime := 3; prime < lim; prime += 2 { // Generate 3, 5, ...
fmt.Printf("Send %v\n", prime) // Trace
ouch <- prime // Send it down the pipe
}
}
And as far as it goes, it works nicely.
However, when I finish the main loop, main exits before all the numbers still in the pipeline of sieve instances have propagated down to the end.
What is the simplest, most elegant, or generally accepted way to make a main routine wait for a set of goroutines (about which it only 'knows' of the first one) to complete?
As for your title question, killing worker goroutines when you don't need them anymore:
You could use the Done idiom. Reads from a closed channel yield the zero value.
Make a new channel done. When reads from this channel succeed, the goroutines know they should quit. Close the channel in main when you have all the values you need.
Check if you can read from a channel done, and exit by returning, or read from next when that's available. This partially replaces the assignment to next in you for loop:
select {
case <-done:
return
case next = <- inch:
}
Ranging over a channel also works, since closing that channel exits the loop.
As for the reverse, your body question, waiting for a set of goroutines to finish:
Use sync.WaitGroup.
var wg sync.WaitGroup
wg.Add(goroutineCount)
And when each goroutine finishes:
wg.Done()
Or use defer:
defer wg.Done()
To wait for all of them to report as Done:
wg.Wait()
In your example, simply call wg.Add(1) when you start a new goroutine, before you call wg.Done() and return. As long as you only reach zero once, wg.Wait() works as expected, so wg.Add(1) before wg.Done.
After #izca unblocked my logjam, and after a few false starts involving deadlocks when everything finished, here's my solution working correctly:
// esieve implements a Sieve of Eratosthenes
// as a series of channels connected together
// by goroutines
package main
import "fmt"
func sieve(mine int, // This instance's own prime
inch chan int, // Input channel from lower primes
done chan int, // Channel for signalling shutdown
count int) { // Number of primes - counter
start := true // First-number switch
ouch := make(chan int) // Output channel, this instance
fmt.Printf("%v ", mine) // Print this instance's prime
for next := <-inch; next > 0; next = <-inch { // Read input channel
if (next % mine) > 0 { // Divisible by my prime?
if start { // No; first time through?
go sieve(next, ouch, done, count+1) // First number,
// create instance for it
start = false // First time done
} else { // Not first time
ouch <- next // Pass to next instance
}
}
}
if start { // Just starting?
close(done) // Yes - we're last in pipe - signal done
print("\n",count," primes\n") // Number of primes/goroutines
} else {
close(ouch) // No - send the signal down the pipe
}
}
func main() {
lim := 100 // Let's do up to 100
done := make(chan int) // Create the done return channel
ouch := make(chan int) // Create the first segment of the pipe
go sieve(2, ouch, done, 1) // Create the first instance for '2'
for prime := 3; prime < lim; prime += 1 { // Generate odd numbers
ouch <- prime // Send numbers down the pipe
}
close(ouch) // Send the done signal down the pipe
<- done // and wait for it to come back
}
I'm tremendously impressed with the elegance and simplicity of Go for this kind of programming, when compared with many other languages. Of course, the warts I claim for myself.
If appropriate here, I'd welcome critical comments.
I'm currently working through the Tour of Go, and I thought that goroutines have been used similarly to Python generators, particularly with Question 66. I thought 66 looked complex, so I rewrote it to this:
package main
import "fmt"
func fibonacci(c chan int) {
x, y := 1, 1
for {
c <- x
x, y = y, x + y
}
}
func main() {
c := make(chan int)
go fibonacci(c)
for i := 0; i < 10; i++ {
fmt.Println(<-c)
}
}
This seems to work. A couple of questions:
If I turn up the buffer size on the channel to say, 10, fibonacci would fill up 10 further spots, as quickly as possible, and main would eat up the spots as quickly as it could go. Is this right? This would be more performant than a buffer size of 1 at the expense of memory, correct?
As the channel doesn't get closed by the fibonacci sender, what happens memory-wise when we go out of scope here? My expectation is that once c and go fibonacci is out of scope, the channel and everything on it gets garbage-collected. My gut tells me this is probably not what happens.
Yes, increasing the buffer size might drastically increase the execution speed of your program, because it will reduce the number of context switches. Goroutines aren't garbage-collected, but channels are. In your example, the fibonacci goroutine will run forever (waiting for another goroutine to read from the channel c), and the channel c will never be destroyed, because the fib-goroutine is still using it.
Here is another, sightly different program, which does not leak memory and is imho more similar to Python's generators:
package main
import "fmt"
func fib(n int) chan int {
c := make(chan int)
go func() {
x, y := 0, 1
for i := 0; i <= n; i++ {
c <- x
x, y = y, x+y
}
close(c)
}()
return c
}
func main() {
for i := range fib(10) {
fmt.Println(i)
}
}
Alternatively, if you do not know how many Fibonacci numbers you want to generate, you have to use another quit channel so that you can send the generator goroutine a signal when it should stop. This is whats explained in golang's tutorial https://tour.golang.org/concurrency/4.
I like #tux21b's answer; having the channel created in the fib() function makes the calling code nice and clean. To elaborate a bit, you only need a separate 'quit' channel if there's no way to tell the function when to stop when you call it. If you only ever care about "numbers up to X", you can do this:
package main
import "fmt"
func fib(n int) chan int {
c := make(chan int)
go func() {
x, y := 0, 1
for x < n {
c <- x
x, y = y, x+y
}
close(c)
}()
return c
}
func main() {
// Print the Fibonacci numbers less than 500
for i := range fib(500) {
fmt.Println(i)
}
}
If you want the ability to do either, this is a little sloppy, but I personally like it better than testing the condition in the caller and then signalling a quit through a separate channel:
func fib(wanted func (int, int) bool) chan int {
c := make(chan int)
go func() {
x, y := 0, 1
for i := 0; wanted(i, x); i++{
c <- x
x, y = y, x+y
}
close(c)
}()
return c
}
func main() {
// Print the first 10 Fibonacci numbers
for n := range fib(func(i, x int) bool { return i < 10 }) {
fmt.Println(n)
}
// Print the Fibonacci numbers less than 500
for n := range fib(func(i, x int) bool { return x < 500 }) {
fmt.Println(n)
}
}
I think it just depends on the particulars of a given situation whether you:
Tell the generator when to stop when you create it by
Passing an explicit number of values to generate
Passing a goal value
Passing a function that determines whether to keep going
Give the generator a 'quit' channel, test the values yourself, and tell it to quit when appropriate.
To wrap up and actually answer your questions:
Increasing the channel size would help performance due to fewer context switches. In this trivial example, neither performance nor memory consumption are going to be an issue, but in other situations, buffering the channel is often a very good idea. The memory used by make (chan int, 100) hardly seems significant in most cases, but it could easily make a big performance difference.
You have an infinite loop in your fibonacci function, so the goroutine running it will run (block on c <- x, in this case) forever. The fact that (once c goes out of scope in the caller) you won't ever again read from the channel you share with it doesn't change that. And as #tux21b pointed out, the channel will never be garbage collected since it's still in use. This has nothing to do with closing the channel (the purpose of which is to let the receiving end of the channel know that no more values will be coming) and everything to do with not returning from your function.
You could use closures to simulate a generator. Here is the example from golang.org.
package main
import "fmt"
// fib returns a function that returns
// successive Fibonacci numbers.
func fib() func() int {
a, b := 0, 1
return func() int {
a, b = b, a+b
return a
}
}
func main() {
f := fib()
// Function calls are evaluated left-to-right.
fmt.Println(f(), f(), f(), f(), f())
}
Using channels to emulate Python generators kind of works, but they introduce concurrency where none is needed, and it adds more complication than's probably needed. Here, just keeping the state explicitly is easier to understand, shorter, and almost certainly more efficient. It makes all your questions about buffer sizes and garbage collection moot.
type fibState struct {
x, y int
}
func (f *fibState) Pop() int {
result := f.x
f.x, f.y = f.y, f.x + f.y
return result
}
func main() {
fs := &fibState{1, 1}
for i := 0; i < 10; i++ {
fmt.Println(fs.Pop())
}
}